[
https://issues.apache.org/jira/browse/SPARK-24817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566278#comment-16566278
]
Jiang Xingbo commented on SPARK-24817:
--------------------------------------
Actually the current implementation of _barrier_ function doesn't requires
communications between executors, all executors just talk to a
_BarrierCoordinator_ which is in the driver. But to allow launching ML
workloads we do need to enable executors to communicate with each other
directly, IIUC that shall be investigated under SPARK-24724 . Maybe [~mengxr]
can provide more context here.
> Implement BarrierTaskContext.barrier()
> --------------------------------------
>
> Key: SPARK-24817
> URL: https://issues.apache.org/jira/browse/SPARK-24817
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Affects Versions: 2.4.0
> Reporter: Jiang Xingbo
> Priority: Major
>
> Implement BarrierTaskContext.barrier(), to support global sync between all
> the tasks in a barrier stage. The global sync shall finish immediately once
> all tasks in the same barrier stage reaches the same barrier.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
