GitHub user jiangxb1987 opened a pull request:
https://github.com/apache/spark/pull/21494
[WIP][SPARK-24375][Prototype] Support barrier scheduling
## What changes were proposed in this pull request?
Add new RDDBarrier and BarrierTaskContext to support barrier scheduling in
Spark, also modify how the job scheduling works a bit to accommodate the new
feature.
This is a prototype to facilitate discussion, it's not meant for the final
design or anything, just shows one way that might works.
## How was this patch tested?
Simple unit test and integration test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jiangxb1987/spark barrierSync
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21494.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21494
----
commit 6bdc5fe27dd2c28971e61119ef63dbee0385138d
Author: Xingbo Jiang <xingbo.jiang@...>
Date: 2018-05-17T03:20:17Z
Implement support for barrier scheduling
commit 84cdc68c33e913335d8a0a26a388a32ba6e9cf18
Author: Xingbo Jiang <xingbo.jiang@...>
Date: 2018-06-04T23:45:06Z
add TODOs
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
