Hi, What are YARN memory configurations?
This should help you tune those configurations: http://www.cloudera.com/documentation/enterprise/5-5-x/topics/cdh_ig_yarn_tuning.html Thanks, Aniruddha On Tue, Jun 14, 2016 at 6:13 PM, Shubham Pathak <[email protected]> wrote: > Hello, > > I have setup a 3 node hadoop cluster using Cloudera Express 5.7.0. and > installed Dt RTS community edition datatorrent-rts-3.3.0.bin. > Total memory available is 24 GB. > > However when I launch PiDemo, containers are not getting allocated. Just 2 > / 3 containers are being allocated. This is error in Stram logs. > > 2016-06-14 17:41:48,762 INFO > com.datatorrent.stram.StreamingAppMasterService: Removed container: > Capability[<memory:1024, vCores:1>]Priority[2] > 2016-06-14 17:41:48,762 INFO > com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: > [Capability[<memory:1024, vCores:1>]Priority[2]] > 2016-06-14 17:41:48,762 INFO > com.datatorrent.stram.StreamingAppMasterService: Requested container: > Capability[<memory:1024, vCores:1>]Priority[2] > 2016-06-14 17:42:20,185 INFO > com.datatorrent.stram.StreamingAppMasterService: Removing container > request: [Capability[<memory:1024, vCores:1>]Priority[2]] > 2016-06-14 17:42:20,185 INFO > com.datatorrent.stram.StreamingAppMasterService: Removed container: > Capability[<memory:1024, vCores:1>]Priority[2] > 2016-06-14 17:42:20,185 INFO > com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: > [Capability[<memory:1024, vCores:1>]Priority[2]] > 2016-06-14 17:42:20,185 INFO > com.datatorrent.stram.StreamingAppMasterService: Requested container: > Capability[<memory:1024, vCores:1>]Priority[2] > 2016-06-14 17:42:35,822 WARN > com.datatorrent.common.util.PubSubWebSocketClient: Ignoring unparseable > JSON message: �Idle for 300289ms > 300000ms > org.codehaus.jackson.JsonParseException: Illegal character ((CTRL-CHAR, > code 3)): only regular white space (\r, \n, \t) is allowed between tokens > at [Source: java.io.StringReader@7ab8225e; line: 1, column: 2] > at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1432) > at > org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385) > at > org.codehaus.jackson.impl.JsonParserMinimalBase._throwInvalidSpace(JsonParserMinimalBase.java:331) > at > org.codehaus.jackson.impl.ReaderBasedParser._skipWSOrEnd(ReaderBasedParser.java:1485) > at > org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:362) > at > org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2761) > at > org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2709) > at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1854) > at > com.datatorrent.common.util.PubSubMessageCodec.parseMessage(PubSubMessageCodec.java:66) > at > com.datatorrent.common.util.PubSubWebSocketClient$PubSubWebSocket.onMessage(PubSubWebSocketClient.java:79) > at > com.ning.http.client.providers.netty.NettyWebSocket.onTextFragment(NettyWebSocket.java:183) > at > com.ning.http.client.providers.netty.NettyAsyncHttpProvider$WebSocketProtocol.handle(NettyAsyncHttpProvider.java:2369) > at > com.ning.http.client.providers.netty.NettyAsyncHttpProvider.messageReceived(NettyAsyncHttpProvider.java:1142) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) > at > org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) > at > org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88) > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > at > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 2016-06-14 17:42:51,522 INFO > com.datatorrent.stram.StreamingAppMasterService: Removing container > request: [Capability[<memory:1024, vCores:1>]Priority[2]] > 2016-06-14 17:42:51,523 INFO > com.datatorrent.stram.StreamingAppMasterService: Removed container: > Capability[<memory:1024, vCores:1>]Priority[2] > 2016-06-14 17:42:51,523 INFO > com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: > [Capability[<memory:1024, vCores:1>]Priority[2]] > 2016-06-14 17:42:51,523 INFO > com.datatorrent.stram.StreamingAppMasterService: Requested container: > Capability[<memory:1024, vCores:1>]Priority[2] > 2016-06-14 17:43:22,829 INFO > com.datatorrent.stram.StreamingAppMasterService: Removing container > request: [Capability[<memory:1024, vCores:1>]Priority[2]] > 2016-06-14 17:43:22,830 INFO > com.datatorrent.stram.StreamingAppMasterService: Removed container: > Capability[<memory:1024, vCores:1>]Priority[2] > 2016-06-14 17:43:22,830 INFO > com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: > [Capability[<memory:1024, vCores:1>]Priority[2]] > 2016-06-14 17:43:22,830 INFO > com.datatorrent.stram.StreamingAppMasterService: Requested container: > Capability[<memory:1024, vCores:1>]Priority[2] > 2016-06-14 17:43:54,218 INFO > com.datatorrent.stram.StreamingAppMasterService: Removing container > request: [Capability[<memory:1024, vCores:1>]Priority[2]] > 2016-06-14 17:43:54,219 INFO > com.datatorrent.stram.StreamingAppMasterService: Removed container: > Capability[<memory:1024, vCores:1>]Priority[2] > 2016-06-14 17:43:54,219 INFO > com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: > [Capability[<memory:1024, vCores:1>]Priority[2]] > 2016-06-14 17:43:54,219 INFO > com.datatorrent.stram.StreamingAppMasterService: Requested container: > Capability[<memory:1024, vCores:1>]Priority[2] > 2016-06-14 17:44:25,543 INFO > com.datatorrent.stram.StreamingAppMasterService: Removing container > request: [Capability[<memory:1024, vCores:1>]Priority[2]] > 2016-06-14 17:44:25,543 INFO > com.datatorrent.stram.StreamingAppMasterService: Removed container: > Capability[<memory:1024, vCores:1>]Priority[2] > 2016-06-14 17:44:25,543 INFO > com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: > [Capability[<memory:1024, vCores:1>]Priority[2]] > 2016-06-14 17:44:25,543 INFO > com.datatorrent.stram.StreamingAppMasterService: Requested container: > Capability[<memory:1024, vCores:1>]Priority[2] > 2016-06-14 17:44:56,885 INFO > com.datatorrent.stram.StreamingAppMasterService: Removing container > request: [Capability[<memory:1024, vCores:1>]Priority[2]] > 2016-06-14 17:44:56,886 INFO > com.datatorrent.stram.StreamingAppMasterService: Removed container: > Capability[<memory:1024, vCores:1>]Priority[2] > 2016-06-14 17:44:56,886 INFO > com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: > [Capability[<memory:1024, vCores:1>]Priority[2]] > > > Any idea what could be the issue ? > > Thanks, > Shubham >
