[
https://issues.apache.org/jira/browse/TEZ-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14170283#comment-14170283
]
Rajesh Balamohan commented on TEZ-1629:
---------------------------------------
[~zjffdu] - Yes, this happens during AM shutdown. May be we could log the
status of the threadpool executor as well in the log of
CustomizedRejectedExecutionHandler.
> ContainerLauncherImpl's event handler thread should check for threadpool's
> status before submitting a task
> ----------------------------------------------------------------------------------------------------------
>
> Key: TEZ-1629
> URL: https://issues.apache.org/jira/browse/TEZ-1629
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Rajesh Balamohan
> Assignee: Jeff Zhang
> Attachments: Tez-1629.patch
>
>
> 2014-09-19 10:18:42,654 INFO [AsyncDispatcher event handler]
> org.apache.tez.dag.app.DAGAppMaster: On DAG completion. Old state: RUNNING
> new state: SUCCEEDED
> 2014-09-19 10:18:42,654 INFO [AsyncDispatcher event handler]
> org.apache.tez.dag.app.DAGAppMaster: Shutting down on completion of
> dag:dag_1410976766734_0795_1
> 2014-09-19 10:18:42,654 INFO [AsyncDispatcher event handler]
> org.apache.tez.dag.app.DAGAppMaster: DAGAppMasterShutdownHandler invoked
> 2014-09-19 10:18:42,655 INFO [AsyncDispatcher event handler]
> org.apache.tez.dag.app.DAGAppMaster: Handling DAGAppMaster shutdown
> 2014-09-19 10:18:42,656 INFO [AMShutdownThread]
> org.apache.tez.dag.app.DAGAppMaster: Sleeping for 5 seconds before shutting
> down
> 2014-09-19 10:18:47,661 INFO [AMShutdownThread]
> org.apache.tez.dag.app.DAGAppMaster: Calling stop for all the services
> 2014-09-19 10:18:47,662 INFO [AMShutdownThread]
> org.apache.tez.dag.history.HistoryEventHandler: Stopping HistoryEventHandler
> 2014-09-19 10:18:47,663 INFO [AMShutdownThread]
> org.apache.tez.dag.history.recovery.RecoveryService: Stopping RecoveryService
> 2014-09-19 10:18:47,663 INFO [AMShutdownThread]
> org.apache.tez.dag.history.recovery.RecoveryService: Closing Summary Stream
> 2014-09-19 10:18:47,663 INFO [RecoveryEventHandlingThread]
> org.apache.tez.dag.history.recovery.RecoveryService: EventQueue take
> interrupted. Returning
> 2014-09-19 10:18:47,699 INFO [AMShutdownThread]
> org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService: Stopping
> ATSService, eventQueueBacklog=0
> 2014-09-19 10:18:47,699 INFO [HistoryEventHandlingThread]
> org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService: EventQueue
> take interrupted. Returning
> 2014-09-19 10:18:47,700 INFO [AMShutdownThread]
> org.apache.tez.dag.app.launcher.ContainerLauncherImpl: Sending a stop request
> to the NM for ContainerId: container_1410976766734_0795_01_000002
> 2014-09-19 10:18:47,711 INFO [AsyncDispatcher event handler]
> org.apache.tez.dag.history.HistoryEventHandler:
> [HISTORY][DAG:dag_1410976766734_0795_1][Event:CONTAINER_STOPPED]:
> containerId=container_1410976766734_0795_01_000002,
> stoppedTime=1411121927711, exitStatus=-100
> 2014-09-19 10:18:47,712 INFO [AsyncDispatcher event handler]
> org.apache.tez.dag.app.rm.container.AMContainerImpl: AMContainer
> container_1410976766734_0795_01_000002 transitioned from IDLE to
> STOP_REQUESTED via event C_NM_STOP_SENT
> 2014-09-19 10:18:47,713 INFO [DelayedContainerManager]
> org.apache.tez.dag.app.rm.YarnTaskSchedulerService: AllocatedContainerManager
> Thread interrupted
> 2014-09-19 10:18:47,713 ERROR [ContainerLauncher Event Handler]
> org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread
> Thread[ContainerLauncher Event Handler,5,main] threw an Exception.
> java.util.concurrent.RejectedExecutionException: Task
> org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor@8b1f182
> rejected from java.util.concurrent.ThreadPoolExecutor@5a669243[Terminated,
> pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
> at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
> at
> org.apache.tez.dag.app.launcher.ContainerLauncherImpl$1.run(ContainerLauncherImpl.java:310)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
