Richard W. Eggert II created SPARK-11296:
--------------------------------------------
Summary: FutureAction.onComplete consumes a thread while waiting
for job
Key: SPARK-11296
URL: https://issues.apache.org/jira/browse/SPARK-11296
Project: Spark
Issue Type: Bug
Components: Scheduler, Spark Core
Affects Versions: 1.5.1, 1.5.0, 1.4.1, 1.4.0
Reporter: Richard W. Eggert II
The implementations of the onComplete callback (and, by extension, the
callbacks based on it) of FutureAction (SimpleFutureAction and
ComplexFutureAction) each consume a thread that blocks waiting for the
underlying job(s) to complete. This is unnecessary and is likely to be very
surprising to anyone accustomed to the usual behavior of Scala Futures (i.e.,
that callbacks only occupy threads AFTER the underlying task has completed),
and it runs the risk of causing thread starvation for any processes utilizing
the same thread pool (especially if scala.concurrent.ExecutionContext.global is
used). The implementations can and should be reworked to operate in a
non-blocking manner, by modifying the underlying JobWaiter class to expose a
non-blocking API, so that the onComplete (as well as onSuccess and onFailure)
callbacks only utilize threads while they are actually executing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
