[
https://issues.apache.org/jira/browse/SPARK-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen updated SPARK-11296:
------------------------------
Issue Type: Improvement (was: Bug)
> FutureAction.onComplete consumes a thread while waiting for job
> ---------------------------------------------------------------
>
> Key: SPARK-11296
> URL: https://issues.apache.org/jira/browse/SPARK-11296
> Project: Spark
> Issue Type: Improvement
> Components: Scheduler, Spark Core
> Affects Versions: 1.4.0, 1.4.1, 1.5.0, 1.5.1
> Reporter: Richard W. Eggert II
> Labels: performance
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The implementations of the onComplete callback (and, by extension, the
> callbacks based on it) of FutureAction (SimpleFutureAction and
> ComplexFutureAction) each consume a thread that blocks waiting for the
> underlying job(s) to complete. This is unnecessary and is likely to be very
> surprising to anyone accustomed to the usual behavior of Scala Futures (i.e.,
> that callbacks only occupy threads AFTER the underlying task has completed),
> and it runs the risk of causing thread starvation for any processes utilizing
> the same thread pool (especially if scala.concurrent.ExecutionContext.global
> is used). The implementations can and should be reworked to operate in a
> non-blocking manner, by modifying the underlying JobWaiter class to expose a
> non-blocking API, so that the onComplete (as well as onSuccess and onFailure)
> callbacks only utilize threads while they are actually executing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
