I found the container logs via the containers web UI (on Cloudera VM it is
http://quickstart.cloudera:8042/node/allContainers)
There are 2 containers for my application, first just shows the logs I was
looking at earlier indicating whether the container succeeded or failed;
second has many logs with useful info (command / errors / slider-agent /
status_command).
They are transient, but I was able to look at them before the application
terminated.
slider-agent.out just has this line in it:
*No handlers could be found for logger "root"*
However *slider-agent.log* gave me the info I was looking for, basically
the stderr / stdout from executing the Java command line so that is very
helpful.
INFO 2015-08-19 14:07:28,422 AgentToggleLogger.py:40 - Queue result:
{'componentStatus': [],
'reports': [{'actionId': u'4-1',
'clusterName': u'myapp1',
'exitcode': 1,
'reportResult': True,
'role': u'MYAPP',
'roleCommand': u'START',
'serviceName': u'myapp1',
'status': 'FAILED',
'stderr': '2015-08-19 14:07:28,268 - Error while executing
command ...<removed for brevity>,
'stdout': '2015-08-19 14:07:23,261 -
Execute[\'/usr/java/latest/bin/java -Xmx256m -classpath ...<removed for
brevity>,
'structuredOut': '{}',
'taskId': 4}]}
Thanks, put me on the path to solving this.
Dave
On Wed, Aug 19, 2015 at 12:53 PM, Gour Saha <[email protected]> wrote:
> Can you find the slider-agent.out available in the container logs for
> container_1439926335194_0003_01_000002 and see what¹s in there?
>
> -Gour
>
> On 8/19/15, 12:46 PM, "Dave Rugh" <[email protected]> wrote:
>
> >I went through the Slider Memcached Tutorial
> ><
> http://slider.incubator.apache.org/docs/slider_specs/hello_world_slider_a
> >pp.html>
> >and
> >was able to package/deploy/start the memcached container successfully;
> >however when I package up a custom application, basically a Java jar plus
> >dependencies, the container never launches succssfully.
> >
> >
> >
> >The application page show the app is in a FINISHED/FAILED state with this
> >diagnostic:
> >
> >
> http://quickstart.cloudera:8088/cluster/app/application_1439926335194_0001
> >
> >
> >
> >Diagnostics: Unstable Application Instance : - failed with component
> >MYAPP failed 'recently' 6 times (4 in startup); threshold is 5 - last
> >failure: Failure container_1439926335194_0001_01_000008 on host
> >quickstart.cloudera (0):
> >
> http://quickstart.cloudera:19888/jobhistory/logs//quickstart.cloudera:8041
> >/container_1439926335
> ><
> http://quickstart.cloudera:19888/jobhistory/logs//quickstart.cloudera:804
> >1/container_1439926335194_0001_01_000008/ctx/C4>
> >
> >
> >Looking at the container logs - nothing is in the slider-out.txt file
> >indicating that the container never ran; no errors in the slider-err.txt
> >file (see below) and the slider-log.txt file is quite lengthy but the only
> >error I see is this:
> >
> >
> >
> >2015-08-19 12:14:05,744 [AMRM Callback Handler Thread] ERROR
> >appmaster.SliderAppMaster - Role instance RoleInstance{role='MYAPP',
> >id='container_1439926335194_0003_01_000002',
> >container=ContainerID=container_1439926335194_0003_01_000002
> >nodeID=quickstart.cloudera:8041 http=quickstart.cloudera:8042
> >priority=1073741825 resource=<memory:1024, vCores:1>,
> >createTime=1440011595168, startTime=1440011595313, released=false,
> >roleId=1, host=quickstart.cloudera,
> >hostURL=http://quickstart.cloudera:8042,
> >state=5, placement=null, exitCode=0, command='python
> >./infra/agent/slider-agent/agent/main.py --label
> >container_1439926335194_0003_01_000002___MYAPP --zk-quorum localhost:2181
> >--zk-reg-path /registry/users/myuser/services/org-apache-slider/myapp1 >
> ><LOG_DIR>/slider-agent.out 2>&1 ; ', diagnostics='', output=null,
> >environment=[LANGUAGE="en_US.UTF-8", AGENT_WORK_ROOT="$PWD",
> >HADOOP_USER_NAME="MYUSER", AGENT_LOG_ROOT="<LOG_DIR>",
> >PYTHONPATH="./infra/agent/slider-agent/", LC_ALL="en_US.UTF-8",
> >SLIDER_PASSPHRASE="<redacted>", LANG="en_US.UTF-8"]} failed
> >
> >
> >What's my next step to debug this?
> >
> >
> >Do I want to attach to the code that launches the container and step
> >through? I have Netbeans available in the environment.
> >
> >
> >Thanks,
> >
> >
> >Dave
> >
> >
> >
> >*slider-err.txt*
> >
> >
> >*Aug 19, 2015 12:13:07 PM com.sun.jersey.api.core.PackagesResourceConfig
> >init*
> >
> >*INFO: Scanning for root resource and provider classes in the packages:*
> >
> >* org.apache.slider.server.appmaster.web.rest.agent*
> >
> >*Aug 19, 2015 12:13:07 PM com.sun.jersey.api.core.ScanningResourceConfig
> >logClasses*
> >
> >*INFO: Root resource classes found:*
> >
> >* class
> >org.apache.slider.server.appmaster.web.rest.agent.AgentWebServices*
> >
> >*Aug 19, 2015 12:13:07 PM com.sun.jersey.api.core.ScanningResourceConfig
> >init*
> >
> >*INFO: No provider classes found.*
> >
> >*Aug 19, 2015 12:13:07 PM
> >com.sun.jersey.server.impl.application.WebApplicationImpl _initiate*
> >
> >*INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011
> >11:17
> >AM'*
> >
> >*Aug 19, 2015 12:13:18 PM com.sun.jersey.spi.inject.Errors
> >processErrorMessages*
> >
> >*WARNING: The following warnings have been detected with resource and/or
> >provider classes:*
> >
> >* WARNING: A sub-resource method, public javax.ws.rs.core.Response
> >org.apache.slider.server.appmaster.web.rest.agent.AgentResource.endpointRo
> >ot(),
> >with URI template, "/", is treated as a resource method*
>
>
