Hi team, Trying to access live data from couchbase using input Operators. It is running fine on my local. The tuples are emitting as Pojo through the output port of inputOperator. But when I upload the .apa file on data torrent, it gets launched but soon it goes into Failed State just after been accepted. Can you please help me to resolve the issue ? I am sending logs as an attached file.
Regards, Hitesh Goyal Simpli5d Technologies Cont No.: 9996588220
2016-09-22 12:40:30,308 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Updating application attempt appattempt_1474540741175_0001_000001 with final state: FAILED, and exit status: 1 2016-09-22 12:40:30,308 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1474540741175_0001_000001 State change from LAUNCHED to FINAL_SAVING 2016-09-22 12:40:30,308 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Unregistering app attempt : appattempt_1474540741175_0001_000001 2016-09-22 12:40:30,309 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Application finished, removing password for appattempt_1474540741175_0001_000001 2016-09-22 12:40:30,309 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1474540741175_0001_000001 State change from FINAL_SAVING to FAILED 2016-09-22 12:40:30,309 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: The number of failed attempts is 1. The max attempts is 2 2016-09-22 12:40:30,309 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application Attempt appattempt_1474540741175_0001_000001 is done. finalState=FAILED 2016-09-22 12:40:30,309 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1474540741175_0001_000002 2016-09-22 12:40:30,309 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1474540741175_0001_000002 State change from NEW to SUBMITTED 2016-09-22 12:40:30,309 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: Application application_1474540741175_0001 requests cleared 2016-09-22 12:40:30,310 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application removed - appId: application_1474540741175_0001 user: dtadmin queue: default #user-pending-applications: 0 #user-active-applications: 0 #queue-pending-applications: 0 #queue-active-applications: 0 2016-09-22 12:40:30,310 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application application_1474540741175_0001 from user: dtadmin activated in queue: default 2016-09-22 12:40:30,310 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application added - appId: application_1474540741175_0001 user: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue$User@77d0922b, leaf-queue: default #user-pending-applications: 0 #user-active-applications: 1 #queue-pending-applications: 0 #queue-active-applications: 1 2016-09-22 12:40:30,310 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Added Application Attempt appattempt_1474540741175_0001_000002 to scheduler from user dtadmin in queue default 2016-09-22 12:40:30,310 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1474540741175_0001_000002 State change from SUBMITTED to SCHEDULED 2016-09-22 12:40:31,308 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Null container completed... 2016-09-22 12:40:31,308 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1474540741175_0001_02_000001 Container Transitioned from NEW to ALLOCATED 2016-09-22 12:40:31,308 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dtadmin OPERATION=AM Allocated Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1474540741175_0001 CONTAINERID=container_1474540741175_0001_02_000001 2016-09-22 12:40:31,308 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Assigned container container_1474540741175_0001_02_000001 of capacity <memory:1024, vCores:1> on host localhost.localdomain:35189, which has 1 containers, <memory:1024, vCores:1> used and <memory:28976, vCores:7> available after allocation 2016-09-22 12:40:31,308 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: assignedContainer application attempt=appattempt_1474540741175_0001_000002 container=Container: [ContainerId: container_1474540741175_0001_02_000001, NodeId: localhost.localdomain:35189, NodeHttpAddress: localhost.localdomain:8042, Resource: <memory:1024, vCores:1>, Priority: 0, Token: null, ] queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=0 clusterResource=<memory:30000, vCores:8> 2016-09-22 12:40:31,309 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Re-sorting assigned queue: root.default stats: default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:1024, vCores:1>, usedCapacity=0.034133334, absoluteUsedCapacity=0.034133334, numApps=1, numContainers=1 2016-09-22 12:40:31,309 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.034133334 absoluteUsedCapacity=0.034133334 used=<memory:1024, vCores:1> cluster=<memory:30000, vCores:8> 2016-09-22 12:40:31,309 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: Sending NMToken for nodeId : localhost.localdomain:35189 for container : container_1474540741175_0001_02_000001 2016-09-22 12:40:31,310 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1474540741175_0001_02_000001 Container Transitioned from ALLOCATED to ACQUIRED 2016-09-22 12:40:31,310 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: Clear node set for appattempt_1474540741175_0001_000002 2016-09-22 12:40:31,310 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Storing attempt: AppId: application_1474540741175_0001 AttemptId: appattempt_1474540741175_0001_000002 MasterContainer: Container: [ContainerId: container_1474540741175_0001_02_000001, NodeId: localhost.localdomain:35189, NodeHttpAddress: localhost.localdomain:8042, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.0.41:35189 }, ] 2016-09-22 12:40:31,310 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1474540741175_0001_000002 State change from SCHEDULED to ALLOCATED_SAVING 2016-09-22 12:40:31,310 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1474540741175_0001_000002 State change from ALLOCATED_SAVING to ALLOCATED 2016-09-22 12:40:31,311 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Launching masterappattempt_1474540741175_0001_000002 2016-09-22 12:40:31,312 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Setting up container Container: [ContainerId: container_1474540741175_0001_02_000001, NodeId: localhost.localdomain:35189, NodeHttpAddress: localhost.localdomain:8042, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.0.41:35189 }, ] for AM appattempt_1474540741175_0001_000002 2016-09-22 12:40:31,312 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Command to launch container container_1474540741175_0001_02_000001 : ${JAVA_HOME}/bin/java -Djava.io.tmpdir=$PWD/tmp -Xmx768m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/dt-heap-1.bin -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> -Ddt.attr.APPLICATION_PATH=hdfs://localhost:54310/var/datatorrent/apps/application_1474540741175_0001 com.datatorrent.stram.StreamingAppMaster 1><LOG_DIR>/AppMaster.stdout 2><LOG_DIR>/AppMaster.stderr 2016-09-22 12:40:31,312 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Create AMRMToken for ApplicationAttempt: appattempt_1474540741175_0001_000002 2016-09-22 12:40:31,312 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Creating password for appattempt_1474540741175_0001_000002 2016-09-22 12:40:31,321 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Done launching container Container: [ContainerId: container_1474540741175_0001_02_000001, NodeId: localhost.localdomain:35189, NodeHttpAddress: localhost.localdomain:8042, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.0.41:35189 }, ] for AM appattempt_1474540741175_0001_000002 2016-09-22 12:40:31,321 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1474540741175_0001_000002 State change from ALLOCATED to LAUNCHED 2016-09-22 12:40:32,310 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1474540741175_0001_02_000001 Container Transitioned from ACQUIRED to COMPLETED 2016-09-22 12:40:32,310 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp: Completed container: container_1474540741175_0001_02_000001 in state: COMPLETED event:FINISHED 2016-09-22 12:40:32,310 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dtadmin OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1474540741175_0001 CONTAINERID=container_1474540741175_0001_02_000001 2016-09-22 12:40:32,310 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Released container container_1474540741175_0001_02_000001 of capacity <memory:1024, vCores:1> on host localhost.localdomain:35189, which currently has 0 containers, <memory:0, vCores:0> used and <memory:30000, vCores:8> available, release resources=true 2016-09-22 12:40:32,310 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Updating application attempt appattempt_1474540741175_0001_000002 with final state: FAILED, and exit status: 1 2016-09-22 12:40:32,310 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1474540741175_0001_000002 State change from LAUNCHED to FINAL_SAVING 2016-09-22 12:40:32,310 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: default used=<memory:0, vCores:0> numContainers=0 user=dtadmin user-resources=<memory:0, vCores:0> 2016-09-22 12:40:32,310 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Unregistering app attempt : appattempt_1474540741175_0001_000002 2016-09-22 12:40:32,310 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: completedContainer container=Container: [ContainerId: container_1474540741175_0001_02_000001, NodeId: localhost.localdomain:35189, NodeHttpAddress: localhost.localdomain:8042, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.0.41:35189 }, ] queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=0 cluster=<memory:30000, vCores:8> 2016-09-22 12:40:32,310 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: completedContainer queue=root usedCapacity=0.0 absoluteUsedCapacity=0.0 used=<memory:0, vCores:0> cluster=<memory:30000, vCores:8> 2016-09-22 12:40:32,311 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Re-sorting completed queue: root.default stats: default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=0 2016-09-22 12:40:32,311 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application attempt appattempt_1474540741175_0001_000002 released container container_1474540741175_0001_02_000001 on node: host: localhost.localdomain:35189 #containers=0 available=30000 used=0 with event: FINISHED 2016-09-22 12:40:32,311 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Application finished, removing password for appattempt_1474540741175_0001_000002 2016-09-22 12:40:32,311 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1474540741175_0001_000002 State change from FINAL_SAVING to FAILED 2016-09-22 12:40:32,311 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: The number of failed attempts is 2. The max attempts is 2 2016-09-22 12:40:32,312 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Updating application application_1474540741175_0001 with final state: FAILED 2016-09-22 12:40:32,312 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1474540741175_0001 State change from ACCEPTED to FINAL_SAVING 2016-09-22 12:40:32,312 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application Attempt appattempt_1474540741175_0001_000002 is done. finalState=FAILED 2016-09-22 12:40:32,312 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: Application application_1474540741175_0001 requests cleared 2016-09-22 12:40:32,312 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating info for app: application_1474540741175_0001 2016-09-22 12:40:32,313 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application removed - appId: application_1474540741175_0001 user: dtadmin queue: default #user-pending-applications: 0 #user-active-applications: 0 #queue-pending-applications: 0 #queue-active-applications: 0 2016-09-22 12:40:32,313 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Application application_1474540741175_0001 failed 2 times due to AM Container for appattempt_1474540741175_0001_000002 exited with exitCode: 1 For more detailed output, check application tracking page:http://data1:8088/proxy/application_1474540741175_0001/Then, click on links to logs of each attempt. Diagnostics: Exception from container-launch. Container id: container_1474540741175_0001_02_000001 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 Failing this attempt. Failing the application. 2016-09-22 12:40:32,314 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1474540741175_0001 State change from FINAL_SAVING to FAILED 2016-09-22 12:40:32,314 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Application removed - appId: application_1474540741175_0001 user: dtadmin leaf-queue of parent: root #applications: 0 2016-09-22 12:40:32,315 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dtadmin OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1474540741175_0001 failed 2 times due to AM Container for appattempt_1474540741175_0001_000002 exited with exitCode: 1 For more detailed output, check application tracking page:http://data1:8088/proxy/application_1474540741175_0001/Then, click on links to logs of each attempt. Diagnostics: Exception from container-launch. Container id: container_1474540741175_0001_02_000001 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 Failing this attempt. Failing the application. APPID=application_1474540741175_0001 2016-09-22 12:40:32,317 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary: appId=application_1474540741175_0001,name=MyFirstApplication,user=dtadmin,queue=default,state=FAILED,trackingUrl=http://data1:8088/cluster/app/application_1474540741175_0001,appMasterHost=N/A,startTime=1474540826765,finishTime=1474540832312,finalStatus=FAILED 16-09-22 12:40:33,311 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Null container completed...