GitHub user reggert opened a pull request:
https://github.com/apache/spark/pull/9264
[SPARK-11296] Modifications to JobWaiter, FutureAction, and AsyncRDDActions
to support non-blocking operation
These changes rework the implementations of SimpleFutureAction,
ComplexFutureAction, JobWaiter, and AsyncRDDActions such that asynchronous
callbacks on the generated Futures NEVER block waiting for a job to complete. A
small amount of mutex synchronization is necessary to protect the internal
fields that manage cancellation, but these locks are only held very briefly and
in practice should almost never cause any blocking to occur. The existing
blocking APIs of these classes are retained, but they simply delegate to the
underlying non-blocking API and `Await` the results with indefinite timeouts.
Associated JIRA ticket: https://issues.apache.org/jira/browse/SPARK-11296
This pull request contains all my own original work, which I release to the
Spark project under its open source license.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/reggert/spark fix-futureaction
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9264.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9264
----
commit 4210aa6dc4d837988f101689bab6117404c00eb2
Author: Richard W. Eggert II <richard.egg...@gmail.com>
Date: 2015-10-24T02:30:18Z
reworked FutureAction and JobWaiter to avoid blocking or consuming threads
while waiting for jobs
commit 1ad1abdd995a6347b2a18a31260168a987fee06b
Author: Richard W. Eggert II <richard.egg...@gmail.com>
Date: 2015-10-24T16:17:09Z
reworked ComplexFutureAction and AsyncRDDActions.takeAsync to be
non-blocking
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
