GitHub user jiangxb1987 opened a pull request:
https://github.com/apache/spark/pull/21898
[SPARK-24817][Core] Implement BarrierTaskContext.barrier()
## What changes were proposed in this pull request?
Implement BarrierTaskContext.barrier(), to support global sync between all
the tasks in a barrier stage.
The function set a global barrier and waits until all tasks in this stage
hit this barrier. Similar to MPI_Barrier function in MPI, the barrier()
function call blocks until all tasks in the same stage have reached this
routine. The global sync shall finish immediately once all tasks in the same
barrier stage reaches the same barrier.
This PR implements BarrierTaskContext.barrier() based on netty-based RPC
client, introduces new `BarrierCoordinator` and new
`BarrierCoordinatorMessage`, and new config to handle timeout issue.
## How was this patch tested?
TBD: Add test cases.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jiangxb1987/spark taskcontext.barrier
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21898.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21898
----
commit de517f58ff9fb6cd0c870c9e51bc59d9236dc7fb
Author: Xingbo Jiang <xingbo.jiang@...>
Date: 2018-07-27T18:12:01Z
implement BarrierTaskContext.barrier()
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
