[
https://issues.apache.org/jira/browse/SPARK-14846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-14846.
-------------------------------
Resolution: Not A Problem
OK SGTM
> Driver process fails to terminate when graceful shutdown is used
> ----------------------------------------------------------------
>
> Key: SPARK-14846
> URL: https://issues.apache.org/jira/browse/SPARK-14846
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.6.1
> Reporter: Mattias Aspholm
>
> During shutdown, the job scheduler in Streaming (JobScheduler.stop) spends
> some time waiting for all queued work to complete. If graceful shutdown is
> used, the time is 1 hour, for non-graceful shutdown it's 2 seconds.
> The wait is implemented using the ThreadPoolExecutor.awaitTermination method
> in java.util.concurrent. The problem is that instead of looping over the
> method for the desired period of time, the wait period is passed in as the
> timeout parameter to awaitTermination.
> The result is that if the termination condition is false the first time, the
> method will sleep for the timeout period before trying again. In the case of
> graceful shutdown this means at least an hour's wait before the condition is
> checked again, even though all work is completed in just a few seconds. The
> driver process will continue to live during this time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
