Do you see any exception logged in files inside failed/killed container. Most probably you will find log in logs/userlogs directory inside hadoop installation directory, or in $YARN_LOG_DIR.
Do check master container log for any out of memory errors, the master container id will be container_1464766676472_0003_01_*001 Regards, - Tushar. On Wed, Jun 1, 2016 at 1:38 PM, Junguk Cho <[email protected]> wrote: > Hi. > > Sorry for many emails. > However, I think something is wrong. > It continuously created log files > container_1464766676472_0003_01_xxxx, which meant java process keep > dead. > > Is it right? > Was it supposed to do? > > Thanks in advance. > Junguk > > 2016-06-01 1:57 GMT-06:00 Junguk Cho <[email protected]>: >> Hi, Tushar. >> >> Thanks. >> Yes. It worked. >> >> I have one more question. >> When I access to hadoop web UI and there is menu "ApplicationMaster" >> in Tracking UI menu. >> When I click this, it gave me error. >> >> Sorry, got error 404 >> >> Please consult RFC 2616 for meanings of the error code. >> >> Error Details >> >> org.apache.hadoop.yarn.webapp.WebAppException: /: controller for >> default not found >> at org.apache.hadoop.yarn.webapp.Router.resolveDefault(Router.java:232) >> at org.apache.hadoop.yarn.webapp.Router.resolve(Router.java:140) >> at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:134) >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >> at >> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) >> at >> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) >> at >> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) >> at >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) >> at >> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900) >> at >> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834) >> at >> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795) >> at >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) >> at >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> at >> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) >> at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) >> at >> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >> at >> org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) >> at >> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >> at >> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1243) >> at >> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) >> at >> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) >> at >> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) >> at >> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) >> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) >> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) >> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) >> at >> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) >> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >> at org.mortbay.jetty.Server.handle(Server.java:326) >> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) >> at >> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) >> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) >> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) >> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) >> at >> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) >> at >> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) >> >> The URL is "http://<IP address>:8088/proxy/application_1464766676472_0002/. >> >> With this menu, what we can see? >> >> >> Thanks, >> Junguk >> >> 2016-06-01 1:24 GMT-06:00 Tushar Gosavi <[email protected]>: >>> Hi Junguk, >>> >>> Please check your yarn configuration settings. I normally set >>> minimum-allocation-mb to 256MB for single node install, you can add >>> following lines to your yarn-site.xml and restart the yarn. >>> >>> <property> >>> <name>yarn.scheduler.minimum-allocation-mb</name> >>> <value>256</value> >>> </property> >>> >>> I am able to run YahooFinanceDemo app with above setting on my laptop >>> with 5GB memory usage. >>> >>> Regards, >>> -Tushar. >>> >>> On Wed, Jun 1, 2016 at 12:40 PM, Sandeep Deshmukh >>> <[email protected]> wrote: >>>> Thanks for details. >>>> >>>> Typically the "Waiting for pending status" is followed by "Deploy request" >>>> as highlighted below. Are you getting the highlighted line in your log? If >>>> not, the operators is not getting deployed correctly. Please let us know. >>>> >>>> >>>> 2016-05-23 14:46:51,649 DEBUG >>>> com.datatorrent.stram.engine.StreamingContainer: buffer server memory 512 >>>> 2016-05-23 14:46:51,655 INFO >>>> com.datatorrent.bufferserver.storage.DiskStorage: using >>>> /data/hadoop-2.7.2/tmp/nm-local-dir/usercache/sandeep/appcache/application_1463980780220_0003/container_1463980780220_0003_01_000002/tmp >>>> as the basepath for spooling. >>>> 2016-05-23 14:46:51,663 INFO com.datatorrent.bufferserver.server.Server: >>>> Server started listening at /0:0:0:0:0:0:0:0:42358 >>>> 2016-05-23 14:46:51,665 DEBUG >>>> com.datatorrent.stram.engine.StreamingContainer: Buffer server started: >>>> /0:0:0:0:0:0:0:0:42358 >>>> 2016-05-23 14:46:51,700 DEBUG >>>> com.datatorrent.stram.engine.StreamingContainer: Entering heartbeat loop >>>> (interval is 1000 ms) >>>> 2016-05-23 14:46:52,750 INFO >>>> com.datatorrent.stram.engine.StreamingContainer: Waiting for pending >>>> request. >>>> 2016-05-23 14:46:53,255 INFO >>>> com.datatorrent.stram.engine.StreamingContainer: Waiting for pending >>>> request. >>>> 2016-05-23 14:46:56,287 INFO >>>> com.datatorrent.stram.engine.StreamingContainer: Waiting for pending >>>> request. >>>> 2016-05-23 14:46:56,791 INFO >>>> com.datatorrent.stram.engine.StreamingContainer: Waiting for pending >>>> request. >>>> 2016-05-23 14:46:57,295 INFO >>>> com.datatorrent.stram.engine.StreamingContainer: Waiting for pending >>>> request. >>>> 2016-05-23 14:46:57,809 INFO >>>> com.datatorrent.stram.engine.StreamingContainer: Deploy request: >>>> [OperatorDeployInfo[id=1,name=Input,type=INPUT,checkpoint={ffffffffffffffff, >>>> 0, >>>> 0},inputs=[],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=enrich,bufferServer=goofy]]]] >>>> 2016-05-23 14:46:57,811 DEBUG >>>> com.datatorrent.stram.engine.StreamingContainer: Restoring operator 1 to >>>> checkpoint ffffffffffffffff stateless=false. >>>> 2016-05-23 14:46:57,876 DEBUG com.datatorrent.stram.engine.Node: >>>> type=INPUT, >>>> operator class=class com.datatorrent.InputFileReader >>>> 2016-05-23 14:46:57,889 DEBUG >>>> com.datatorrent.stram.engine.StreamingContainer: Marking operator 1 as >>>> deployed. >>>> >>>> >>>> Regards, >>>> Sandeep >>>> >>>> Regards, >>>> Sandeep >>>> >>>> On Wed, Jun 1, 2016 at 12:29 PM, Junguk Cho <[email protected]> wrote: >>>>> >>>>> Hi, >>>>> >>>>> 1. I used only one machine and use docker image to run it. >>>>> When I access to ip:8088, it shows I use 8 GB memory. >>>>> (Is there a configuration file I can check?) >>>>> >>>>> 2. Hadoop version : 2.7.1 >>>>> I tested both Apex 3.2.2 and Apex 3.3. >>>>> >>>>> Thanks, >>>>> Junguk >>>>> >>>>> >>>>> 2016-05-31 22:09 GMT-06:00 Sandeep Deshmukh <[email protected]>: >>>>> > Hi Junguk, >>>>> > >>>>> > Could you please give us some details: >>>>> > >>>>> > Hadoop cluster configuration: no of machines and their configuration >>>>> > (memory >>>>> > in particular) >>>>> > Version of Hadoop and Apex >>>>> > >>>>> > Regards, >>>>> > Sandeep >>>>> > >>>>> > On Wed, Jun 1, 2016 at 5:04 AM, Junguk Cho <[email protected]> wrote: >>>>> >> >>>>> >> Hi, >>>>> >> >>>>> >> I am new to apex. >>>>> >> I tried to run demo examples in incubator-apex-malhar. >>>>> >> >>>>> >> When I ran yahoofinance example, I think it was stuck. >>>>> >> Log file said >>>>> >> 2016-05-31 20:34:00,628 INFO >>>>> >> com.datatorrent.bufferserver.server.Server: Server started listening >>>>> >> at /0:0:0:0:0:0:0:0:45546 >>>>> >> 2016-05-31 20:34:01,668 INFO >>>>> >> com.datatorrent.stram.engine.StreamingContainer: Waiting for pending >>>>> >> request. >>>>> >> >>>>> >> I think binding IP address has a problem. >>>>> >> >>>>> >> Does anyone help me? >>>>> >> >>>>> >> >>>>> >> Thanks, >>>>> >> Junguk >>>>> > >>>>> > >>>> >>>>
