Xiangrui Meng created SPARK-30667:
-------------------------------------
Summary: Support simple all gather in barrier task context
Key: SPARK-30667
URL: https://issues.apache.org/jira/browse/SPARK-30667
Project: Spark
Issue Type: New Feature
Components: PySpark, Spark Core
Affects Versions: 3.0.0
Reporter: Xiangrui Meng
Currently we offer task.barrier() to coordinate tasks in barrier mode. Tasks
can see all IP addresses from BarrierTaskContext. It would be simpler to
integrate with distributed frameworks like TensorFlow DistributionStrategy if
we provide all gather that can let tasks share additional information with
others, e.g., an available port.
Note that with all gather, tasks are share their IP addresses as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
