Hello All:
I'm trying to run the memcached example locally but it fails pretty quick and
claims "protobuf-java-2.5.0.jar does not exist".
Consider the following script (test-slider.sh):
#!/bin/bash
# this does not work when run during docker-build, we need to do this once we
are logged in
echo "memcached built, registering memcached with slider and then running it"
/usr/lib/slider/bin/slider install-package --package
/tmp/mybuild/incubator-slider/app-packages/memcached/jmemcached-1.0.0.zip
--name jmemcached --debug
echo "Listing applications - before starting jmemcached"
/usr/lib/slider/bin/slider list --manager localhost:8032
# startup memcached properly, yarn task name will be mymemcached
/usr/lib/slider/bin/slider create mymemcached --template
/tmp/mybuild/incubator-slider/app-packages/memcached/appConfig.json --resources
/tmp/mybuild/incubator-slider/app-packages/memcached/resources-default.json
--manager localhost:8032 --debug
echo "Listing applications - after starting jmemcached"
/usr/lib/slider/bin/slider list --manager localhost:8032
echo "Finished $0"
I am having difficulty in locating the logs after the failure, any idea where
they would be? yarn logs cannot find them, and I'm not having luck with the
name node. I did modify the yarn-site.xml read at startup to include the lines:
<!-- Begin modifications for debugging slider -->
<!-- 60 minutes after a failure to see what is left in the directory-->
<property>
<name>yarn.nodemanager.delete.debug-delay-sec</name>
<value>3600</value>
</property>
<!--time before the process gets a -9 (Should it be 30 seconds?)-->
<property>
<name>yarn.nodemanager.sleep-delay-before-sigkill.ms</name>
<value>3600000</value>
</property>
<!-- End modifications for debugging slider -->
Regarding the error, I'm seeing, the slider install-package appears to work
correctly from what I can tell, but the slider create encounters run time
errors when launched, it seems to be looking for protobuf-java-2.5.0.jar. I
tried making fat jars for the memcached application and the various slider
packages, to no avail. Any ideas how to resolve these errors reported in the
resource manager:
Application application_1492557996590_0002 failed 2 times due to AM Container
for appattempt_1492557996590_0002_000002 exited with exitCode: -1000
For more detailed output, check application tracking
page:http://quickstart.cloudera:8088/proxy/application_1492557996590_0002/Then,
click on links to logs of each attempt.
Diagnostics: File
file:/root/.slider/cluster/mymemcached/tmp/application_1492557996590_0002/am/lib/protobuf-java-2.5.0.jar
does not exist
java.io.FileNotFoundException: File
file:/root/.slider/cluster/mymemcached/tmp/application_1492557996590_0002/am/lib/protobuf-java-2.5.0.jar
does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:542)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:755)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:532)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:425)
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Failing this attempt. Failing the application.
The test-slider.sh script's output is as follows:
# bash -x /tmp/mybuild/test_slider.sh
+ echo 'memcached built, registering memcached with slider and then running it'
memcached built, registering memcached with slider and then running it
+ /usr/lib/slider/bin/slider install-package --package
/tmp/mybuild/incubator-slider/app-packages/memcached/jmemcached-1.0.0.zip
--name jmemcached --debug
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/lib/slider/lib/slider-core-0.60.0-incubating.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/slider/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2017-04-19 01:10:44,080 [main] INFO client.RMProxy - Connecting to
ResourceManager at /0.0.0.0:8032
2017-04-19 01:10:44,243 [main] ERROR main.ServiceLauncher - Unable to access
supplied pkg file at
/tmp/mybuild/incubator-slider/app-packages/memcached/jmemcached-1.0.0.zip
2017-04-19 01:10:44,247 [main] INFO util.ExitUtil - Exiting with status 40
+ echo 'Listing applications - before starting jmemcached'
Listing applications - before starting jmemcached
+ /usr/lib/slider/bin/slider list --manager localhost:8032
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/lib/slider/lib/slider-core-0.60.0-incubating.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/slider/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2017-04-19 01:10:46,066 [main] INFO client.RMProxy - Connecting to
ResourceManager at localhost/127.0.0.1:8032
2017-04-19 01:10:46,217 [main] INFO util.ExitUtil - Exiting with status 0
+ /usr/lib/slider/bin/slider create mymemcached --template
/tmp/mybuild/incubator-slider/app-packages/memcached/appConfig.json --resources
/tmp/mybuild/incubator-slider/app-packages/memcached/resources-default.json
--manager localhost:8032 --debug
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/lib/slider/lib/slider-core-0.60.0-incubating.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/slider/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2017-04-19 01:10:48,070 [main] INFO client.RMProxy - Connecting to
ResourceManager at localhost/127.0.0.1:8032
2017-04-19 01:10:48,647 [main] INFO agent.AgentClientProvider - Validating app
definition
/tmp/mybuild/incubator-slider/app-packages/memcached/jmemcached-1.0.1.zip
2017-04-19 01:10:48,648 [main] INFO agent.AgentUtils - Reading metainfo at
/tmp/mybuild/incubator-slider/app-packages/memcached/jmemcached-1.0.1.zip
2017-04-19 01:10:48,675 [main] INFO tools.SliderUtils - Reading metainfo.xml
of size 2202
2017-04-19 01:10:48,893 [main] INFO client.SliderClient - No credentials
requested
2017-04-19 01:10:48,938 [main] INFO agent.AgentUtils - Reading metainfo at
/tmp/mybuild/incubator-slider/app-packages/memcached/jmemcached-1.0.1.zip
2017-04-19 01:10:48,939 [main] INFO tools.SliderUtils - Reading metainfo.xml
of size 2202
2017-04-19 01:10:48,976 [main] INFO launch.AbstractLauncher - Log include
patterns:
2017-04-19 01:10:48,976 [main] INFO launch.AbstractLauncher - Log exclude
patterns:
2017-04-19 01:10:49,861 [main] INFO slideram.SliderAMClientProvider - Loading
all dependencies for AM.
2017-04-19 01:10:49,862 [main] INFO tools.SliderUtils - Loading all
dependencies from /usr/lib/slider/lib
2017-04-19 01:10:51,324 [main] INFO agent.AgentClientProvider - Automatically
uploading the agent tarball at
file:/root/.slider/cluster/mymemcached/tmp/application_1492557996590_0003/agent
2017-04-19 01:10:51,361 [main] INFO agent.AgentClientProvider - Validating app
definition
/tmp/mybuild/incubator-slider/app-packages/memcached/jmemcached-1.0.1.zip
2017-04-19 01:10:51,361 [main] INFO agent.AgentUtils - Reading metainfo at
/tmp/mybuild/incubator-slider/app-packages/memcached/jmemcached-1.0.1.zip
2017-04-19 01:10:51,364 [main] INFO tools.SliderUtils - Reading metainfo.xml
of size 2202
2017-04-19 01:10:51,429 [main] INFO Configuration.deprecation -
slider.registry.path is deprecated. Instead, use hadoop.registry.zk.root
2017-04-19 01:10:51,436 [main] INFO launch.AppMasterLauncher - Submitting
application to Resource Manager
2017-04-19 01:10:51,480 [main] INFO impl.YarnClientImpl - Submitted
application application_1492557996590_0003
2017-04-19 01:10:51,484 [main] INFO util.ExitUtil - Exiting with status 0
+ echo 'Listing applications - after starting jmemcached'
Listing applications - after starting jmemcached
+ /usr/lib/slider/bin/slider list --manager localhost:8032
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/lib/slider/lib/slider-core-0.60.0-incubating.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/slider/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2017-04-19 01:10:53,893 [main] INFO client.RMProxy - Connecting to
ResourceManager at localhost/127.0.0.1:8032
mymemcached ACCEPTED application_1492557996590_0003
2017-04-19 01:10:54,186 [main] INFO util.ExitUtil - Exiting with status 0
+ echo 'Finished /tmp/mybuild/test_slider.sh'
Finished /tmp/mybuild/test_slider.sh
Thanks:
Bill
