hi kun! I mean check your hadoop mapreduce task status & log, there may have something goes wrong.
notice that hbase client will try many times before throw Exception out, which means you need to wait for a while(on my case, over 30 mins), before you can see the error in mapreduce task log! best regards! [email protected] From: zhangrongkun Date: 2016-05-05 18:55 To: dev Subject: Re: when build kylin cube,the job stopping at 77.78%,can't go on the job No! My kylin‘s log haven't any exception,and My HBase works well. there is my MRApp’s log： 2016-05-05 18:32:37,273 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 2 2016-05-05 18:32:37,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned to reduce 2016-05-05 18:32:37,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1461663034262_1777_01_000010 to attempt_1461663034262_1777_r_000000_0 2016-05-05 18:32:37,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned to reduce 2016-05-05 18:32:37,275 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1461663034262_1777_01_000011 to attempt_1461663034262_1777_r_000001_0 2016-05-05 18:32:37,275 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:7 AssignedReds:2 CompletedMaps:1 CompletedReds:0 ContAlloc:10 ContRel:0 HostLocal:2 RackLocal:6 2016-05-05 18:32:37,290 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved hadoop22 to /default-rack 2016-05-05 18:32:37,291 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1461663034262_1777_r_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2016-05-05 18:32:37,292 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved hadoop24 to /default-rack 2016-05-05 18:32:37,293 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1461663034262_1777_r_000001_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2016-05-05 18:32:37,294 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1461663034262_1777_01_000011 taskAttempt attempt_1461663034262_1777_r_000001_0 2016-05-05 18:32:37,294 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1461663034262_1777_r_000001_0 2016-05-05 18:32:37,295 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1461663034262_1777_01_000010 taskAttempt attempt_1461663034262_1777_r_000000_0 2016-05-05 18:32:37,295 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1461663034262_1777_r_000000_0 2016-05-05 18:32:37,301 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1461663034262_1777_r_000001_0 : 13562 2016-05-05 18:32:37,301 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1461663034262_1777_r_000000_0 : 13562 2016-05-05 18:32:37,301 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1461663034262_1777_r_000001_0] using containerId: [container_1461663034262_1777_01_000011 on NM: [hadoop24:45083] 2016-05-05 18:32:37,302 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1461663034262_1777_r_000001_0 TaskAttempt Transitioned from ASSIGNED to RUNNING 2016-05-05 18:32:37,302 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1461663034262_1777_r_000000_0] using containerId: [container_1461663034262_1777_01_000010 on NM: [hadoop22:43623] 2016-05-05 18:32:37,303 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1461663034262_1777_r_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING 2016-05-05 18:32:37,303 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: ATTEMPT_START task_1461663034262_1777_r_000001 2016-05-05 18:32:37,304 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1461663034262_1777_r_000001 Task Transitioned from SCHEDULED to RUNNING 2016-05-05 18:32:37,304 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: ATTEMPT_START task_1461663034262_1777_r_000000 2016-05-05 18:32:37,304 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1461663034262_1777_r_000000 Task Transitioned from SCHEDULED to RUNNING 2016-05-05 18:32:37,968 INFO [IPC Server handler 1 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1461663034262_1777_m_000007_0 is : 0.0 2016-05-05 18:32:38,092 INFO [IPC Server handler 3 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1461663034262_1777_m_000004_0 is : 0.0 2016-05-05 18:32:38,101 INFO [IPC Server handler 7 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1461663034262_1777_m_000003_0 is : 0.0 2016-05-05 18:32:38,128 INFO [IPC Server handler 2 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1461663034262_1777_m_000007_0 is : 1.0 2016-05-05 18:32:38,133 INFO [IPC Server handler 5 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1461663034262_1777_m_000007_0 2016-05-05 18:32:38,134 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1461663034262_1777_m_000007_0 TaskAttempt Transitioned from RUNNING to SUCCESS_CONTAINER_CLEANUP 2016-05-05 18:32:38,135 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1461663034262_1777_01_000009 taskAttempt attempt_1461663034262_1777_m_000007_0 2016-05-05 18:32:38,135 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1461663034262_1777_m_000007_0 2016-05-05 18:32:38,138 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1461663034262_1777_m_000007_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2016-05-05 18:32:38,138 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1461663034262_1777_m_000007_0 2016-05-05 18:32:38,138 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1461663034262_1777_m_000007 Task Transitioned from RUNNING to SUCCEEDED 2016-05-05 18:32:38,139 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 2 2016-05-05 18:32:38,277 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:7 AssignedReds:2 CompletedMaps:2 CompletedReds:0 ContAlloc:10 ContRel:0 HostLocal:2 RackLocal:6 2016-05-05 18:32:38,279 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1461663034262_1777: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:0, vCores:0> knownNMs=34 2016-05-05 18:32:38,392 INFO [IPC Server handler 12 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1461663034262_1777_m_000001_0 is : 0.0 2016-05-05 18:32:38,416 INFO [IPC Server handler 13 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1461663034262_1777_m_000004_0 is : 1.0 2016-05-05 18:32:38,421 INFO [IPC Server handler 11 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1461663034262_1777_m_000004_0 2016-05-05 18:32:38,422 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1461663034262_1777_m_000004_0 TaskAttempt Transitioned from RUNNING to SUCCESS_CONTAINER_CLEANUP 2016-05-05 18:32:38,423 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1461663034262_1777_01_000006 taskAttempt attempt_1461663034262_1777_m_000004_0 2016-05-05 18:32:38,423 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1461663034262_1777_m_000004_0 2016-05-05 18:32:38,426 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1461663034262_1777_m_000004_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2016-05-05 18:32:38,427 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1461663034262_1777_m_000004_0 2016-05-05 18:32:38,427 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1461663034262_1777_m_000004 Task Transitioned from RUNNING to SUCCEEDED 2016-05-05 18:32:38,428 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 3 2016-05-05 18:32:38,620 INFO [IPC Server handler 9 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1461663034262_1777_m_000003_0 is : 1.0 2016-05-05 18:32:38,625 INFO [IPC Server handler 10 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1461663034262_1777_m_000003_0 2016-05-05 18:32:38,627 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1461663034262_1777_m_000003_0 TaskAttempt Transitioned from RUNNING to SUCCESS_CONTAINER_CLEANUP 2016-05-05 18:32:38,627 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1461663034262_1777_01_000005 taskAttempt attempt_1461663034262_1777_m_000003_0 2016-05-05 18:32:38,627 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1461663034262_1777_m_000003_0 2016-05-05 18:32:38,630 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1461663034262_1777_m_000003_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2016-05-05 18:32:38,631 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1461663034262_1777_m_000003_0 2016-05-05 18:32:38,631 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1461663034262_1777_m_000003 Task Transitioned from RUNNING to SUCCEEDED 2016-05-05 18:32:38,632 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 4 2016-05-05 18:32:38,755 INFO [IPC Server handler 8 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1461663034262_1777_m_000000_0 is : 0.0 2016-05-05 18:32:38,879 INFO [IPC Server handler 15 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1461663034262_1777_m_000001_0 is : 1.0 2016-05-05 18:32:38,884 INFO [IPC Server handler 18 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1461663034262_1777_m_000001_0 2016-05-05 18:32:38,885 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1461663034262_1777_m_000001_0 TaskAttempt Transitioned from RUNNING to SUCCESS_CONTAINER_CLEANUP 2016-05-05 18:32:38,885 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1461663034262_1777_01_000004 taskAttempt attempt_1461663034262_1777_m_000001_0 2016-05-05 18:32:38,886 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1461663034262_1777_m_000001_0 2016-05-05 18:32:38,888 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1461663034262_1777_m_000001_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2016-05-05 18:32:38,889 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1461663034262_1777_m_000001_0 2016-05-05 18:32:38,889 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1461663034262_1777_m_000001 Task Transitioned from RUNNING to SUCCEEDED 2016-05-05 18:32:38,890 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 5 2016-05-05 18:32:38,954 INFO [DefaultSpeculator background processing] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: DefaultSpeculator.addSpeculativeAttempt -- we are speculating task_1461663034262_1777_m_000000 2016-05-05 18:32:38,955 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Scheduling a redundant attempt for task task_1461663034262_1777_m_000000 2016-05-05 18:32:38,955 INFO [DefaultSpeculator background processing] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: We launched 1 speculations. Sleeping 15000 milliseconds. 2016-05-05 18:32:38,955 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved hadoop143 to /default-rack 2016-05-05 18:32:38,955 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved hadoop51 to /default-rack 2016-05-05 18:32:38,956 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1461663034262_1777_m_000000_1 TaskAttempt Transitioned from NEW to UNASSIGNED 2016-05-05 18:32:39,076 INFO [Socket Reader #1 for port 45950] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1461663034262_1777 (auth:SIMPLE) 2016-05-05 18:32:39,091 INFO [Socket Reader #1 for port 45950] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1461663034262_1777 (auth:SIMPLE) 2016-05-05 18:32:39,092 INFO [IPC Server handler 3 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1461663034262_1777_r_000010 asked for a task 2016-05-05 18:32:39,092 INFO [IPC Server handler 3 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1461663034262_1777_r_000010 given task: attempt_1461663034262_1777_r_000000_0 2016-05-05 18:32:39,110 INFO [IPC Server handler 2 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1461663034262_1777_r_000011 asked for a task 2016-05-05 18:32:39,111 INFO [IPC Server handler 2 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1461663034262_1777_r_000011 given task: attempt_1461663034262_1777_r_000001_0 2016-05-05 18:32:39,113 INFO [IPC Server handler 5 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1461663034262_1777_m_000005_0 is : 0.0 2016-05-05 18:32:39,187 INFO [IPC Server handler 4 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1461663034262_1777_m_000002_0 is : 0.0 2016-05-05 18:32:39,282 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:7 AssignedReds:2 CompletedMaps:5 CompletedReds:0 ContAlloc:10 ContRel:0 HostLocal:2 RackLocal:6 2016-05-05 18:32:39,284 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1461663034262_1777: ask=4 release= 0 newContainers=0 finishedContainers=1 resourcelimit=<memory:0, vCores:0> knownNMs=34 2016-05-05 18:32:39,285 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1461663034262_1777_01_000009 2016-05-05 18:32:39,285 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:6 AssignedReds:2 CompletedMaps:5 CompletedReds:0 ContAlloc:10 ContRel:0 HostLocal:2 RackLocal:6 2016-05-05 18:32:39,285 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1461663034262_1777_m_000007_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2016-05-05 18:32:39,301 INFO [IPC Server handler 6 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1461663034262_1777_m_000000_0 is : 1.0 2016-05-05 18:32:39,306 INFO [IPC Server handler 12 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1461663034262_1777_m_000000_0 2016-05-05 18:32:39,307 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1461663034262_1777_m_000000_0 TaskAttempt Transitioned from RUNNING to SUCCESS_CONTAINER_CLEANUP 2016-05-05 18:32:39,308 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1461663034262_1777_01_000003 taskAttempt attempt_1461663034262_1777_m_000000_0 2016-05-05 18:32:39,308 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1461663034262_1777_m_000000_0 2016-05-05 18:32:39,311 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1461663034262_1777_m_000000_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2016-05-05 18:32:39,311 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1461663034262_1777_m_000000_0 2016-05-05 18:32:39,312 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Issuing kill to other attempt attempt_1461663034262_1777_m_000000_1 2016-05-05 18:32:39,312 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1461663034262_1777_m_000000 Task Transitioned from RUNNING to SUCCEEDED 2016-05-05 18:32:39,313 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 6 2016-05-05 18:32:39,314 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1461663034262_1777_m_000000_1 TaskAttempt Transitioned from UNASSIGNED to KILLED 2016-05-05 18:32:39,314 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Processing the event EventType: CONTAINER_DEALLOCATE 2016-05-05 18:32:39,378 INFO [IPC Server handler 13 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1461663034262_1777_m_000005_0 is : 1.0 2016-05-05 18:32:39,383 INFO [IPC Server handler 11 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1461663034262_1777_m_000005_0 2016-05-05 18:32:39,384 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1461663034262_1777_m_000005_0 TaskAttempt Transitioned from RUNNING to SUCCESS_CONTAINER_CLEANUP 2016-05-05 18:32:39,384 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1461663034262_1777_01_000007 taskAttempt attempt_1461663034262_1777_m_000005_0 2016-05-05 18:32:39,385 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1461663034262_1777_m_000005_0 2016-05-05 18:32:39,388 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1461663034262_1777_m_000005_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2016-05-05 18:32:39,388 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1461663034262_1777_m_000005_0 2016-05-05 18:32:39,388 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1461663034262_1777_m_000005 Task Transitioned from RUNNING to SUCCEEDED 2016-05-05 18:32:39,389 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 7 2016-05-05 18:32:39,693 INFO [IPC Server handler 8 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1461663034262_1777_m_000002_0 is : 1.0 2016-05-05 18:32:39,699 INFO [IPC Server handler 16 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1461663034262_1777_m_000002_0 2016-05-05 18:32:39,700 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1461663034262_1777_m_000002_0 TaskAttempt Transitioned from RUNNING to SUCCESS_CONTAINER_CLEANUP 2016-05-05 18:32:39,700 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1461663034262_1777_01_000008 taskAttempt attempt_1461663034262_1777_m_000002_0 2016-05-05 18:32:39,700 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1461663034262_1777_m_000002_0 2016-05-05 18:32:39,704 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1461663034262_1777_m_000002_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2016-05-05 18:32:39,704 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1461663034262_1777_m_000002_0 2016-05-05 18:32:39,704 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1461663034262_1777_m_000002 Task Transitioned from RUNNING to SUCCEEDED 2016-05-05 18:32:39,705 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 8 2016-05-05 18:32:40,288 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:6 AssignedReds:2 CompletedMaps:8 CompletedReds:0 ContAlloc:10 ContRel:0 HostLocal:2 RackLocal:6 2016-05-05 18:32:40,291 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1461663034262_1777: ask=4 release= 0 newContainers=1 finishedContainers=3 resourcelimit=<memory:0, vCores:0> knownNMs=34 2016-05-05 18:32:40,291 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1461663034262_1777_01_000006 2016-05-05 18:32:40,291 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1461663034262_1777_01_000005 2016-05-05 18:32:40,292 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1461663034262_1777_01_000004 2016-05-05 18:32:40,292 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1461663034262_1777_m_000004_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2016-05-05 18:32:40,292 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1 2016-05-05 18:32:40,292 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1461663034262_1777_m_000003_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2016-05-05 18:32:40,292 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Cannot assign container Container: [ContainerId: container_1461663034262_1777_01_000012, NodeId: hadoop34:51310, NodeHttpAddress: hadoop34:8042, Resource: <memory:8192, vCores:1>, Priority: 20, Token: Token { kind: ContainerToken, service: 10.40.30.34:51310 }, ] for a map as either container memory less than required 8192 or no pending map tasks - maps.isEmpty=true 2016-05-05 18:32:40,292 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1461663034262_1777_m_000001_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2016-05-05 18:32:40,292 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:3 AssignedReds:2 CompletedMaps:8 CompletedReds:0 ContAlloc:11 ContRel:1 HostLocal:2 RackLocal:6 2016-05-05 18:32:40,458 INFO [IPC Server handler 9 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: MapCompletionEvents request from attempt_1461663034262_1777_r_000000_0. startIndex 0 maxEvents 10000 2016-05-05 18:32:40,574 INFO [IPC Server handler 10 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: MapCompletionEvents request from attempt_1461663034262_1777_r_000001_0. startIndex 0 maxEvents 10000 2016-05-05 18:32:40,869 INFO [IPC Server handler 15 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1461663034262_1777_r_000000_0 is : 0.0 2016-05-05 18:32:40,989 INFO [IPC Server handler 3 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1461663034262_1777_r_000001_0 is : 0.0 2016-05-05 18:32:41,297 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1461663034262_1777: ask=0 release= 1 newContainers=0 finishedContainers=4 resourcelimit=<memory:0, vCores:0> knownNMs=34 2016-05-05 18:32:41,297 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1461663034262_1777_01_000003 2016-05-05 18:32:41,298 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1461663034262_1777_01_000007 2016-05-05 18:32:41,298 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1461663034262_1777_m_000000_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2016-05-05 18:32:41,298 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1461663034262_1777_01_000008 2016-05-05 18:32:41,298 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1461663034262_1777_m_000005_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2016-05-05 18:32:41,298 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1461663034262_1777_01_000012 2016-05-05 18:32:41,299 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Container complete event for unknown container id container_1461663034262_1777_01_000012 2016-05-05 18:32:41,299 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1461663034262_1777_m_000002_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2016-05-05 18:32:41,299 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:2 CompletedMaps:8 CompletedReds:0 ContAlloc:11 ContRel:1 HostLocal:2 RackLocal:6 2016-05-05 18:32:41,684 INFO [IPC Server handler 8 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1461663034262_1777_r_000000_0 is : 0.0 2016-05-05 18:32:41,837 INFO [IPC Server handler 15 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1461663034262_1777_r_000001_0 is : 0.0 2016-05-05 18:32:46,394 INFO [IPC Server handler 11 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1461663034262_1777_r_000000_0 is : 0.6666833 2016-05-05 18:32:46,485 INFO [IPC Server handler 10 on 45950] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1461663034262_1777_r_000001_0 is : 0.66666937 -- View this message in context: http://apache-kylin.74782.x6.nabble.com/when-build-kylin-cube-the-job-stopping-at-77-78-can-t-go-on-the-job-tp4421p4423.html Sent from the Apache Kylin mailing list archive at Nabble.com.
