GitHub user jiangxb1987 opened a pull request:

    [SPARK-24795][CORE] Implement barrier execution mode

    ## What changes were proposed in this pull request?
    Propose new APIs and modify job/task scheduling to support barrier 
execution mode, which requires all tasks in a same barrier stage start at the 
same time, and retry all tasks in case some tasks fail in the middle. The 
barrier execution mode is useful for some ML/DL workloads.
    The proposed API changes include:
    `RDDBarrier` that marks an RDD as barrier (Spark must launch all the tasks 
together for the current stage).
    `BarrierTaskContext` that support global sync of all tasks in a barrier 
stage, and provide extra `BarrierTaskInfo`s.
    In DAGScheduler, we retry all tasks of a barrier stage in case some tasks 
fail in the middle, this is achieved by unregistering map outputs for a 
shuffleId (for ShuffleMapStage) or clear the finished partitions in an active 
job (for ResultStage).
    ## How was this patch tested?
    Add `RDDBarrierSuite` to ensure we convert RDDs correctly;
    Add new test cases in `DAGSchedulerSuite` to ensure we do task scheduling 
    Add new test cases in `SparkContextSuite` to ensure the barrier execution 
mode actually works (both under local mode and local cluster mode).

You can merge this pull request into a Git repository by running:

    $ git pull barrier-execution-mode

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #21758
commit c25ec473ff078c071aec513953f56c64e6a228a4
Author: Xingbo Jiang <xingbo.jiang@...>
Date:   2018-07-12T17:38:58Z

    implement barrier execution mode.



To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to