GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/7385
[SPARK-9026] Refactor SimpleFutureAction.onComplete to not launch separate
thread for every callback
This commit refactors SimpleFutureAction so that registering a callback
with `onComplete` does not immediately tie up a thread in the provided
execution context.
As @zsxwing noticed in
https://github.com/apache/spark/pull/7276#issuecomment-121097747, the existing
implementation of SimpleFutureAction.onComplete creates a separate thread to
wrap the blocking `awaitResult()` callback:
```scala
override def onComplete[U](func: (Try[T]) => U)(implicit executor:
ExecutionContext) {
executor.execute(new Runnable {
override def run() {
func(awaitResult())
}
})
}
```
This PR addresses this issue by adding a future to JobWaiter and using that
future in the implementation of SimpleFutureAction's future methods. The idea
of adding a future to JobWaiter is borrowed from #7276.
This patch was pair-programmed with @tdas.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark simplefutureaction-refactoring
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7385.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7385
----
commit 11c745095bbaa382a15e6ad1ee1afd2e933e4da8
Author: Josh Rosen <[email protected]>
Date: 2015-07-14T02:37:38Z
Refactor SimpleFutureAction to not block threads for every onComplete
callback.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
