Hi,
I am trying to input values from Hbase using HbaseScan operator but getting
container killed messages repeatedly. Seems like heart beat is getting
timed out and window of HbaseOperator is not moving.
Please let me know what should i do to get this fixed.
Attaching log for reference.
HADOOP_JOB_HISTORYSERVER_OPTS=
2016-07-27 12:33:30,478 INFO com.datatorrent.stram.StreamingAppMaster:
Initializing Application Master.
2016-07-27 12:33:30,494 INFO com.datatorrent.stram.StreamingAppMasterService:
Application master, appId=179, clustertimestamp=1468312847328, attemptId=1
2016-07-27 12:33:31,820 INFO com.datatorrent.common.util.AsyncFSStorageAgent:
using
/yarn/nm/usercache/admin/appcache/application_1468312847328_0179/container_1468312847328_0179_01_000001/tmp/chkp6268938737245191560
as the basepath for checkpointing.
2016-07-27 12:33:33,102 INFO com.datatorrent.stram.FSRecoveryHandler: Creating
hdfs://dev-dwh-hadoop-namenode.snapdeal.com:8020/user/admin/datatorrent/apps/application_1468312847328_0179/recovery/log
2016-07-27 12:33:33,179 INFO com.datatorrent.stram.StreamingAppMasterService:
Starting application with 2 operators in 2 containers
2016-07-27 12:33:33,190 INFO
org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Upper bound of
the thread pool size is 500
2016-07-27 12:33:33,191 INFO
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
yarn.client.max-cached-nodemanagers-proxies : 0
2016-07-27 12:33:33,205 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting
to ResourceManager at dev-dwh-hadoop-namenode.snapdeal.com/10.41.122.23:8030
2016-07-27 12:33:33,231 INFO com.datatorrent.stram.StreamingContainerParent:
Config: Configuration: core-default.xml, core-site.xml, yarn-default.xml,
yarn-site.xml, hdfs-default.xml, hdfs-site.xml
2016-07-27 12:33:33,231 INFO com.datatorrent.stram.StreamingContainerParent:
Listener thread count 30
2016-07-27 12:33:33,237 INFO org.apache.hadoop.ipc.CallQueueManager: Using
callQueue class java.util.concurrent.LinkedBlockingQueue
2016-07-27 12:33:33,240 INFO org.apache.hadoop.ipc.Server: Starting Socket
Reader #1 for port 36854
2016-07-27 12:33:33,254 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2016-07-27 12:33:33,254 INFO org.apache.hadoop.ipc.Server: IPC Server listener
on 36854: starting
2016-07-27 12:33:33,268 INFO com.datatorrent.stram.StreamingContainerParent:
Container callback server listening at
dev-dwh-hadoop-data2.snapdeal.com/10.41.122.20:36854
2016-07-27 12:33:33,335 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2016-07-27 12:33:33,401 INFO org.apache.hadoop.http.HttpRequestLog: Http
request log for http.requests.stram is not defined
2016-07-27 12:33:33,412 INFO org.apache.hadoop.http.HttpServer2: Added global
filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2016-07-27 12:33:33,414 INFO org.apache.hadoop.http.HttpServer2: Added filter
static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context stram
2016-07-27 12:33:33,414 INFO org.apache.hadoop.http.HttpServer2: Added filter
static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context logs
2016-07-27 12:33:33,414 INFO org.apache.hadoop.http.HttpServer2: Added filter
static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context static
2016-07-27 12:33:33,417 INFO org.apache.hadoop.http.HttpServer2: adding path
spec: /stram/*
2016-07-27 12:33:33,417 INFO org.apache.hadoop.http.HttpServer2: adding path
spec: /ws/*
2016-07-27 12:33:33,425 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to
port 46989
2016-07-27 12:33:33,863 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app
/stram started at 46989
2016-07-27 12:33:34,138 INFO org.apache.hadoop.yarn.webapp.WebApps: Registered
webapp guice modules
2016-07-27 12:33:34,138 INFO com.datatorrent.stram.StreamingAppMasterService:
Started web service at port: 46989
2016-07-27 12:33:34,138 INFO com.datatorrent.stram.StreamingAppMasterService:
Setting tracking URL to: dev-dwh-hadoop-data2.snapdeal.com:46989
2016-07-27 12:33:34,148 INFO com.datatorrent.stram.StreamingAppMasterService:
Starting ApplicationMaster
2016-07-27 12:33:34,148 INFO com.datatorrent.stram.StreamingAppMasterService:
number of tokens: 1
2016-07-27 12:33:34,183 INFO com.datatorrent.stram.StreamingAppMasterService:
Max mem 16384m, Min mem 1024m, Max vcores 4 and Min vcores 1 capabililty of
resources in this cluster
2016-07-27 12:33:34,183 INFO com.datatorrent.stram.StreamingAppMasterService:
Blacklist removal time in millis = 3600000, max consecutive node failure count
= 2147483647
2016-07-27 12:33:34,186 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting
to ResourceManager at dev-dwh-hadoop-namenode.snapdeal.com/10.41.122.23:8032
2016-07-27 12:33:34,322 INFO com.datatorrent.stram.ResourceRequestHandler: Node
report:
rackName=/default,nodeid=dev-dwh-hadoop-data2.snapdeal.com:8041,numContainers=1,capability=<memory:16384,
vCores:4>used=<memory:1024, vCores:1>state=RUNNING
2016-07-27 12:33:34,322 INFO com.datatorrent.stram.ResourceRequestHandler: Node
report:
rackName=/default,nodeid=dev-dwh-hadoop-data3.snapdeal.com:8041,numContainers=0,capability=<memory:16384,
vCores:4>used=<memory:0, vCores:0>state=RUNNING
2016-07-27 12:33:34,322 INFO com.datatorrent.stram.ResourceRequestHandler: Node
report:
rackName=/default,nodeid=dev-dwh-hadoop-data1.snapdeal.com:8041,numContainers=0,capability=<memory:16384,
vCores:4>used=<memory:0, vCores:0>state=RUNNING
2016-07-27 12:33:35,332 INFO com.datatorrent.stram.ResourceRequestHandler:
Strict anti-affinity = [] for container with operators PTOperator[id=2,name=wc]
2016-07-27 12:33:35,332 INFO com.datatorrent.stram.ResourceRequestHandler:
Found host null
2016-07-27 12:33:35,335 INFO
com.datatorrent.stram.BlacklistBasedResourceRequestHandler: No node specific
request
2016-07-27 12:33:35,335 INFO com.datatorrent.stram.ResourceRequestHandler:
Strict anti-affinity = [] for container with operators
PTOperator[id=1,name=input]
2016-07-27 12:33:35,335 INFO com.datatorrent.stram.ResourceRequestHandler:
Found host null
2016-07-27 12:33:35,335 INFO
com.datatorrent.stram.BlacklistBasedResourceRequestHandler: No node specific
request
2016-07-27 12:33:35,337 INFO com.datatorrent.stram.StreamingAppMasterService:
Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[1],
Capability[<memory:8048, vCores:1>]Priority[0]]
2016-07-27 12:33:35,337 INFO com.datatorrent.stram.StreamingAppMasterService:
Requested container: Capability[<memory:1536, vCores:1>]Priority[1] on host:
[null]
2016-07-27 12:33:35,340 INFO com.datatorrent.stram.StreamingAppMasterService:
Requested container: Capability[<memory:8048, vCores:1>]Priority[0] on host:
[null]
2016-07-27 12:33:36,385 INFO
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for :
dev-dwh-hadoop-data3.snapdeal.com:8041
2016-07-27 12:33:36,385 INFO
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for :
dev-dwh-hadoop-data1.snapdeal.com:8041
2016-07-27 12:33:36,387 INFO com.datatorrent.stram.StreamingAppMasterService:
Got new container., containerId=container_1468312847328_0179_01_000002,
containerNode=dev-dwh-hadoop-data3.snapdeal.com:8041,
containerNodeURI=dev-dwh-hadoop-data3.snapdeal.com:8042,
containerResourceMemory8192, priority0
2016-07-27 12:33:36,422 INFO com.datatorrent.stram.LaunchContainerRunnable:
Setting up container launch context for
containerid=container_1468312847328_0179_01_000002
2016-07-27 12:33:36,428 INFO com.datatorrent.stram.LaunchContainerRunnable:
CLASSPATH:
./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*:.
2016-07-27 12:33:36,803 INFO
com.datatorrent.common.util.BasicContainerOptConfigurator: property map for
operator {Generic=null, -Xmx=7024m}
2016-07-27 12:33:36,803 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm
opts -Xmx7365197824 for container container_1468312847328_0179_01_000002
2016-07-27 12:33:36,803 INFO com.datatorrent.stram.LaunchContainerRunnable:
Launching on node: dev-dwh-hadoop-data3.snapdeal.com:8041 command:
$JAVA_HOME/bin/java -Xmx7365197824
-Ddt.attr.APPLICATION_PATH=hdfs://dev-dwh-hadoop-namenode.snapdeal.com:8020/user/admin/datatorrent/apps/application_1468312847328_0179
-Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1468312847328_0179_01_000002
-Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR>
com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout
2><LOG_DIR>/stderr
2016-07-27 12:33:36,813 INFO com.datatorrent.stram.StreamingAppMasterService:
Got new container., containerId=container_1468312847328_0179_01_000003,
containerNode=dev-dwh-hadoop-data1.snapdeal.com:8041,
containerNodeURI=dev-dwh-hadoop-data1.snapdeal.com:8042,
containerResourceMemory1536, priority1
2016-07-27 12:33:36,815 INFO
org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing
Event EventType: START_CONTAINER for Container
container_1468312847328_0179_01_000002
2016-07-27 12:33:36,816 INFO com.datatorrent.stram.LaunchContainerRunnable:
Setting up container launch context for
containerid=container_1468312847328_0179_01_000003
2016-07-27 12:33:36,816 INFO com.datatorrent.stram.LaunchContainerRunnable:
CLASSPATH:
./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*:.
2016-07-27 12:33:36,818 INFO
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
Opening proxy : dev-dwh-hadoop-data3.snapdeal.com:8041
2016-07-27 12:33:37,134 INFO
com.datatorrent.common.util.BasicContainerOptConfigurator: property map for
operator {Generic=null, -Xmx=768m}
2016-07-27 12:33:37,134 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm
opts -Xmx1342177280 for container container_1468312847328_0179_01_000003
2016-07-27 12:33:37,135 INFO com.datatorrent.stram.LaunchContainerRunnable:
Launching on node: dev-dwh-hadoop-data1.snapdeal.com:8041 command:
$JAVA_HOME/bin/java -Xmx1342177280
-Ddt.attr.APPLICATION_PATH=hdfs://dev-dwh-hadoop-namenode.snapdeal.com:8020/user/admin/datatorrent/apps/application_1468312847328_0179
-Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1468312847328_0179_01_000003
-Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR>
com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout
2><LOG_DIR>/stderr
2016-07-27 12:33:37,136 INFO
org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing
Event EventType: START_CONTAINER for Container
container_1468312847328_0179_01_000003
2016-07-27 12:33:37,136 INFO
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
Opening proxy : dev-dwh-hadoop-data1.snapdeal.com:8041
2016-07-27 12:33:41,380 INFO com.datatorrent.stram.StreamingContainerParent:
child msg: [container_1468312847328_0179_01_000003] Entering heartbeat loop..
context:
PTContainer[id=1(container_1468312847328_0179_01_000003),state=ALLOCATED,operators=[PTOperator[id=1,name=input]]]
2016-07-27 12:33:41,740 INFO com.datatorrent.stram.StreamingContainerParent:
child msg: [container_1468312847328_0179_01_000002] Entering heartbeat loop..
context:
PTContainer[id=2(container_1468312847328_0179_01_000002),state=ALLOCATED,operators=[PTOperator[id=2,name=wc]]]
2016-07-27 12:33:42,394 INFO com.datatorrent.stram.StreamingContainerManager:
Container container_1468312847328_0179_01_000003 buffer server:
dev-dwh-hadoop-data1.snapdeal.com:51801
2016-07-27 12:33:42,757 INFO com.datatorrent.stram.StreamingContainerManager:
Container container_1468312847328_0179_01_000002 buffer server:
dev-dwh-hadoop-data3.snapdeal.com:36366
2016-07-27 12:34:13,339 INFO com.datatorrent.stram.StreamingContainerManager:
Container
container_1468312847328_0179_01_000...@dev-dwh-hadoop-data1.snapdeal.com:8041
heartbeat timeout (30438 ms).
2016-07-27 12:34:14,340 INFO com.datatorrent.stram.StreamingAppMasterService:
Requested stop container container_1468312847328_0179_01_000003
2016-07-27 12:34:14,340 INFO
org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing
Event EventType: STOP_CONTAINER for Container
container_1468312847328_0179_01_000003
2016-07-27 12:34:14,341 INFO
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
Opening proxy : dev-dwh-hadoop-data1.snapdeal.com:8041
2016-07-27 12:34:14,346 INFO com.datatorrent.stram.StreamingContainerManager:
Container
container_1468312847328_0179_01_000...@dev-dwh-hadoop-data1.snapdeal.com:8041
heartbeat timeout (31445 ms).
2016-07-27 12:34:15,351 INFO com.datatorrent.stram.StreamingContainerManager:
Container
container_1468312847328_0179_01_000...@dev-dwh-hadoop-data1.snapdeal.com:8041
heartbeat timeout (32450 ms).
2016-07-27 12:34:16,360 INFO com.datatorrent.stram.StreamingAppMasterService:
Completed containerId=container_1468312847328_0179_01_000003, state=COMPLETE,
exitStatus=-105, diagnostics=Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
2016-07-27 12:34:16,360 INFO com.datatorrent.stram.StreamingContainerManager:
Initiating recovery for
container_1468312847328_0179_01_000...@dev-dwh-hadoop-data1.snapdeal.com:8041
2016-07-27 12:34:16,360 INFO com.datatorrent.stram.StreamingContainerManager:
Affected operators [PTOperator[id=1,name=input], PTOperator[id=2,name=wc]]
2016-07-27 12:34:17,383 INFO com.datatorrent.stram.ResourceRequestHandler:
Strict anti-affinity = [] for container with operators
PTOperator[id=1,name=input]
2016-07-27 12:34:17,383 INFO com.datatorrent.stram.ResourceRequestHandler:
Found host null
2016-07-27 12:34:17,383 INFO
com.datatorrent.stram.BlacklistBasedResourceRequestHandler: No node specific
request
2016-07-27 12:34:17,383 INFO com.datatorrent.stram.StreamingAppMasterService:
Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[2]]
2016-07-27 12:34:17,383 INFO com.datatorrent.stram.StreamingAppMasterService:
Requested container: Capability[<memory:1536, vCores:1>]Priority[2] on host:
[null]
2016-07-27 12:34:18,396 INFO com.datatorrent.stram.StreamingAppMasterService:
Got new container., containerId=container_1468312847328_0179_01_000004,
containerNode=dev-dwh-hadoop-data1.snapdeal.com:8041,
containerNodeURI=dev-dwh-hadoop-data1.snapdeal.com:8042,
containerResourceMemory1536, priority2
2016-07-27 12:34:18,411 INFO com.datatorrent.stram.StreamingContainerManager:
Removing container agent container_1468312847328_0179_01_000003
2016-07-27 12:34:18,414 INFO com.datatorrent.stram.LaunchContainerRunnable:
Setting up container launch context for
containerid=container_1468312847328_0179_01_000004
2016-07-27 12:34:18,414 INFO com.datatorrent.stram.LaunchContainerRunnable:
CLASSPATH:
./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*:.
2016-07-27 12:34:18,755 INFO
com.datatorrent.common.util.BasicContainerOptConfigurator: property map for
operator {Generic=null, -Xmx=768m}
2016-07-27 12:34:18,756 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm
opts -Xmx1342177280 for container container_1468312847328_0179_01_000004
2016-07-27 12:34:18,756 INFO com.datatorrent.stram.LaunchContainerRunnable:
Launching on node: dev-dwh-hadoop-data1.snapdeal.com:8041 command:
$JAVA_HOME/bin/java -Xmx1342177280
-Ddt.attr.APPLICATION_PATH=hdfs://dev-dwh-hadoop-namenode.snapdeal.com:8020/user/admin/datatorrent/apps/application_1468312847328_0179
-Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1468312847328_0179_01_000004
-Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR>
com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout
2><LOG_DIR>/stderr
2016-07-27 12:34:18,757 INFO
org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing
Event EventType: START_CONTAINER for Container
container_1468312847328_0179_01_000004
2016-07-27 12:34:18,757 INFO
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
Opening proxy : dev-dwh-hadoop-data1.snapdeal.com:8041
2016-07-27 12:34:21,188 INFO com.datatorrent.stram.StreamingContainerParent:
child msg: [container_1468312847328_0179_01_000004] Entering heartbeat loop..
context:
PTContainer[id=1(container_1468312847328_0179_01_000004),state=ALLOCATED,operators=[PTOperator[id=1,name=input]]]
2016-07-27 12:34:22,202 INFO com.datatorrent.stram.StreamingContainerManager:
Container container_1468312847328_0179_01_000004 buffer server:
dev-dwh-hadoop-data1.snapdeal.com:39487
2016-07-27 12:34:52,942 INFO com.datatorrent.stram.StreamingContainerManager:
Container
container_1468312847328_0179_01_000...@dev-dwh-hadoop-data1.snapdeal.com:8041
heartbeat timeout (30740 ms).
2016-07-27 12:34:53,957 INFO com.datatorrent.stram.StreamingAppMasterService:
Requested stop container container_1468312847328_0179_01_000004
2016-07-27 12:34:53,958 INFO
org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing
Event EventType: STOP_CONTAINER for Container
container_1468312847328_0179_01_000004
2016-07-27 12:34:53,959 INFO
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
Opening proxy : dev-dwh-hadoop-data1.snapdeal.com:8041
2016-07-27 12:34:53,962 INFO com.datatorrent.stram.StreamingContainerManager:
Container
container_1468312847328_0179_01_000...@dev-dwh-hadoop-data1.snapdeal.com:8041
heartbeat timeout (31760 ms).
2016-07-27 12:34:54,967 INFO com.datatorrent.stram.StreamingContainerManager:
Container
container_1468312847328_0179_01_000...@dev-dwh-hadoop-data1.snapdeal.com:8041
heartbeat timeout (32765 ms).
2016-07-27 12:34:55,973 INFO com.datatorrent.stram.StreamingAppMasterService:
Completed containerId=container_1468312847328_0179_01_000004, state=COMPLETE,
exitStatus=-105, diagnostics=Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
2016-07-27 12:34:55,973 INFO com.datatorrent.stram.StreamingContainerManager:
Initiating recovery for
container_1468312847328_0179_01_000...@dev-dwh-hadoop-data1.snapdeal.com:8041
2016-07-27 12:34:55,973 INFO com.datatorrent.stram.StreamingContainerManager:
Affected operators [PTOperator[id=1,name=input], PTOperator[id=2,name=wc]]
2016-07-27 12:34:56,993 INFO com.datatorrent.stram.ResourceRequestHandler:
Strict anti-affinity = [] for container with operators
PTOperator[id=1,name=input]
2016-07-27 12:34:56,993 INFO com.datatorrent.stram.ResourceRequestHandler:
Found host null
2016-07-27 12:34:56,993 INFO
com.datatorrent.stram.BlacklistBasedResourceRequestHandler: No node specific
request
2016-07-27 12:34:56,993 INFO com.datatorrent.stram.StreamingAppMasterService:
Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[3]]
2016-07-27 12:34:56,994 INFO com.datatorrent.stram.StreamingAppMasterService:
Requested container: Capability[<memory:1536, vCores:1>]Priority[3] on host:
[null]
2016-07-27 12:34:58,006 INFO com.datatorrent.stram.StreamingAppMasterService:
Got new container., containerId=container_1468312847328_0179_01_000005,
containerNode=dev-dwh-hadoop-data1.snapdeal.com:8041,
containerNodeURI=dev-dwh-hadoop-data1.snapdeal.com:8042,
containerResourceMemory1536, priority3
2016-07-27 12:34:58,006 INFO com.datatorrent.stram.StreamingContainerManager:
Removing container agent container_1468312847328_0179_01_000004
2016-07-27 12:34:58,008 INFO com.datatorrent.stram.LaunchContainerRunnable:
Setting up container launch context for
containerid=container_1468312847328_0179_01_000005
2016-07-27 12:34:58,009 INFO com.datatorrent.stram.LaunchContainerRunnable:
CLASSPATH:
./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*:.
2016-07-27 12:34:58,308 INFO
com.datatorrent.common.util.BasicContainerOptConfigurator: property map for
operator {Generic=null, -Xmx=768m}
2016-07-27 12:34:58,309 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm
opts -Xmx1342177280 for container container_1468312847328_0179_01_000005
2016-07-27 12:34:58,309 INFO com.datatorrent.stram.LaunchContainerRunnable:
Launching on node: dev-dwh-hadoop-data1.snapdeal.com:8041 command:
$JAVA_HOME/bin/java -Xmx1342177280
-Ddt.attr.APPLICATION_PATH=hdfs://dev-dwh-hadoop-namenode.snapdeal.com:8020/user/admin/datatorrent/apps/application_1468312847328_0179
-Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1468312847328_0179_01_000005
-Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR>
com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout
2><LOG_DIR>/stderr
2016-07-27 12:34:58,309 INFO
org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing
Event EventType: START_CONTAINER for Container
container_1468312847328_0179_01_000005
2016-07-27 12:34:58,310 INFO
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
Opening proxy : dev-dwh-hadoop-data1.snapdeal.com:8041
2016-07-27 12:35:00,480 INFO com.datatorrent.stram.StreamingContainerParent:
child msg: [container_1468312847328_0179_01_000005] Entering heartbeat loop..
context:
PTContainer[id=1(container_1468312847328_0179_01_000005),state=ALLOCATED,operators=[PTOperator[id=1,name=input]]]
2016-07-27 12:35:01,491 INFO com.datatorrent.stram.StreamingContainerManager:
Container container_1468312847328_0179_01_000005 buffer server:
dev-dwh-hadoop-data1.snapdeal.com:43257
2016-07-27 12:35:32,495 INFO com.datatorrent.stram.StreamingContainerManager:
Container
container_1468312847328_0179_01_000...@dev-dwh-hadoop-data1.snapdeal.com:8041
heartbeat timeout (31004 ms).
2016-07-27 12:35:33,496 INFO com.datatorrent.stram.StreamingAppMasterService:
Requested stop container container_1468312847328_0179_01_000005
2016-07-27 12:35:33,496 INFO
org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing
Event EventType: STOP_CONTAINER for Container
container_1468312847328_0179_01_000005
2016-07-27 12:35:33,497 INFO
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
Opening proxy : dev-dwh-hadoop-data1.snapdeal.com:8041
2016-07-27 12:35:33,502 INFO com.datatorrent.stram.StreamingContainerManager:
Container
container_1468312847328_0179_01_000...@dev-dwh-hadoop-data1.snapdeal.com:8041
heartbeat timeout (32011 ms).
2016-07-27 12:35:34,507 INFO com.datatorrent.stram.StreamingContainerManager:
Container
container_1468312847328_0179_01_000...@dev-dwh-hadoop-data1.snapdeal.com:8041
heartbeat timeout (33016 ms).
2016-07-27 12:35:35,513 INFO com.datatorrent.stram.StreamingAppMasterService:
Completed containerId=container_1468312847328_0179_01_000005, state=COMPLETE,
exitStatus=-105, diagnostics=Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
2016-07-27 12:35:35,513 INFO com.datatorrent.stram.StreamingContainerManager:
Initiating recovery for
container_1468312847328_0179_01_000...@dev-dwh-hadoop-data1.snapdeal.com:8041
2016-07-27 12:35:35,513 INFO com.datatorrent.stram.StreamingContainerManager:
Affected operators [PTOperator[id=1,name=input], PTOperator[id=2,name=wc]]
2016-07-27 12:35:36,533 INFO com.datatorrent.stram.ResourceRequestHandler:
Strict anti-affinity = [] for container with operators
PTOperator[id=1,name=input]
2016-07-27 12:35:36,533 INFO com.datatorrent.stram.ResourceRequestHandler:
Found host null
2016-07-27 12:35:36,533 INFO
com.datatorrent.stram.BlacklistBasedResourceRequestHandler: No node specific
request
2016-07-27 12:35:36,533 INFO com.datatorrent.stram.StreamingAppMasterService:
Asking RM for containers: [Capability[<mem
