Hello, This was a Hadoop configuration issue. yarn.nodemanager.resource.cpu-vcores was set to 1. So total vCores available were 3 but the requirement was 4 and hence containers were not getting allocated. On increasing to 8, app got the required resources.
Thanks, Shubham On Wed, Jun 15, 2016 at 10:20 AM, Shubham Pathak <[email protected]> wrote: > Thank you for the replies. > > Will go through Yarn Configuration and Yarn logs to see if we get any > clues. > > Total memory shown in YARN UI is 24 GB . yarn.nodemanager.resource.memory-mb > = 8 GB > > Thanks, > Shubham > > On Tue, Jun 14, 2016 at 10:06 PM, Pradeep A. Dalvi <[email protected]> > wrote: > >> From logs, it appears that container requests are being resubmitted. >> Doubt that it is related to APEXCORE-471. >> >> How much Total Memory is shown in DT or YARN UI? >> >> On Tue, Jun 14, 2016 at 8:22 AM, Sandesh Hegde <[email protected]> >> wrote: >> >>> There is one known issue, >>> >>> https://issues.apache.org/jira/browse/APEXCORE-471 >>> >>> >>> On Tue, Jun 14, 2016 at 7:58 AM Munagala Ramanath <[email protected]> >>> wrote: >>> >>>> Could you take a look at the YARN logs ? >>>> >>>> Ram >>>> >>>> On Tue, Jun 14, 2016 at 5:43 AM, Shubham Pathak < >>>> [email protected]> wrote: >>>> >>>>> Hello, >>>>> >>>>> I have setup a 3 node hadoop cluster using Cloudera Express 5.7.0. and >>>>> installed Dt RTS community edition datatorrent-rts-3.3.0.bin. >>>>> Total memory available is 24 GB. >>>>> >>>>> However when I launch PiDemo, containers are not getting allocated. >>>>> Just 2 / 3 containers are being allocated. This is error in Stram logs. >>>>> >>>>> 2016-06-14 17:41:48,762 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Removed container: >>>>> Capability[<memory:1024, vCores:1>]Priority[2] >>>>> 2016-06-14 17:41:48,762 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: >>>>> [Capability[<memory:1024, vCores:1>]Priority[2]] >>>>> 2016-06-14 17:41:48,762 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Requested container: >>>>> Capability[<memory:1024, vCores:1>]Priority[2] >>>>> 2016-06-14 17:42:20,185 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Removing container >>>>> request: [Capability[<memory:1024, vCores:1>]Priority[2]] >>>>> 2016-06-14 17:42:20,185 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Removed container: >>>>> Capability[<memory:1024, vCores:1>]Priority[2] >>>>> 2016-06-14 17:42:20,185 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: >>>>> [Capability[<memory:1024, vCores:1>]Priority[2]] >>>>> 2016-06-14 17:42:20,185 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Requested container: >>>>> Capability[<memory:1024, vCores:1>]Priority[2] >>>>> 2016-06-14 17:42:35,822 WARN >>>>> com.datatorrent.common.util.PubSubWebSocketClient: Ignoring unparseable >>>>> JSON message: �Idle for 300289ms > 300000ms >>>>> org.codehaus.jackson.JsonParseException: Illegal character >>>>> ((CTRL-CHAR, code 3)): only regular white space (\r, \n, \t) is allowed >>>>> between tokens >>>>> at [Source: java.io.StringReader@7ab8225e; line: 1, column: 2] >>>>> at >>>>> org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1432) >>>>> at >>>>> org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385) >>>>> at >>>>> org.codehaus.jackson.impl.JsonParserMinimalBase._throwInvalidSpace(JsonParserMinimalBase.java:331) >>>>> at >>>>> org.codehaus.jackson.impl.ReaderBasedParser._skipWSOrEnd(ReaderBasedParser.java:1485) >>>>> at >>>>> org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:362) >>>>> at >>>>> org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2761) >>>>> at >>>>> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2709) >>>>> at >>>>> org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1854) >>>>> at >>>>> com.datatorrent.common.util.PubSubMessageCodec.parseMessage(PubSubMessageCodec.java:66) >>>>> at >>>>> com.datatorrent.common.util.PubSubWebSocketClient$PubSubWebSocket.onMessage(PubSubWebSocketClient.java:79) >>>>> at >>>>> com.ning.http.client.providers.netty.NettyWebSocket.onTextFragment(NettyWebSocket.java:183) >>>>> at >>>>> com.ning.http.client.providers.netty.NettyAsyncHttpProvider$WebSocketProtocol.handle(NettyAsyncHttpProvider.java:2369) >>>>> at >>>>> com.ning.http.client.providers.netty.NettyAsyncHttpProvider.messageReceived(NettyAsyncHttpProvider.java:1142) >>>>> at >>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) >>>>> at >>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) >>>>> at >>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) >>>>> at >>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) >>>>> at >>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) >>>>> at >>>>> org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) >>>>> at >>>>> org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) >>>>> at >>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) >>>>> at >>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) >>>>> at >>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) >>>>> at >>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) >>>>> at >>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) >>>>> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88) >>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) >>>>> at >>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> 2016-06-14 17:42:51,522 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Removing container >>>>> request: [Capability[<memory:1024, vCores:1>]Priority[2]] >>>>> 2016-06-14 17:42:51,523 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Removed container: >>>>> Capability[<memory:1024, vCores:1>]Priority[2] >>>>> 2016-06-14 17:42:51,523 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: >>>>> [Capability[<memory:1024, vCores:1>]Priority[2]] >>>>> 2016-06-14 17:42:51,523 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Requested container: >>>>> Capability[<memory:1024, vCores:1>]Priority[2] >>>>> 2016-06-14 17:43:22,829 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Removing container >>>>> request: [Capability[<memory:1024, vCores:1>]Priority[2]] >>>>> 2016-06-14 17:43:22,830 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Removed container: >>>>> Capability[<memory:1024, vCores:1>]Priority[2] >>>>> 2016-06-14 17:43:22,830 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: >>>>> [Capability[<memory:1024, vCores:1>]Priority[2]] >>>>> 2016-06-14 17:43:22,830 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Requested container: >>>>> Capability[<memory:1024, vCores:1>]Priority[2] >>>>> 2016-06-14 17:43:54,218 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Removing container >>>>> request: [Capability[<memory:1024, vCores:1>]Priority[2]] >>>>> 2016-06-14 17:43:54,219 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Removed container: >>>>> Capability[<memory:1024, vCores:1>]Priority[2] >>>>> 2016-06-14 17:43:54,219 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: >>>>> [Capability[<memory:1024, vCores:1>]Priority[2]] >>>>> 2016-06-14 17:43:54,219 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Requested container: >>>>> Capability[<memory:1024, vCores:1>]Priority[2] >>>>> 2016-06-14 17:44:25,543 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Removing container >>>>> request: [Capability[<memory:1024, vCores:1>]Priority[2]] >>>>> 2016-06-14 17:44:25,543 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Removed container: >>>>> Capability[<memory:1024, vCores:1>]Priority[2] >>>>> 2016-06-14 17:44:25,543 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: >>>>> [Capability[<memory:1024, vCores:1>]Priority[2]] >>>>> 2016-06-14 17:44:25,543 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Requested container: >>>>> Capability[<memory:1024, vCores:1>]Priority[2] >>>>> 2016-06-14 17:44:56,885 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Removing container >>>>> request: [Capability[<memory:1024, vCores:1>]Priority[2]] >>>>> 2016-06-14 17:44:56,886 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Removed container: >>>>> Capability[<memory:1024, vCores:1>]Priority[2] >>>>> 2016-06-14 17:44:56,886 INFO >>>>> com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: >>>>> [Capability[<memory:1024, vCores:1>]Priority[2]] >>>>> >>>>> >>>>> Any idea what could be the issue ? >>>>> >>>>> Thanks, >>>>> Shubham >>>>> >>>> >>>> >> >
