Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/21898#discussion_r206710204
--- Diff: core/src/main/scala/org/apache/spark/BarrierTaskContext.scala ---
@@ -27,6 +27,33 @@ trait BarrierTaskContext extends TaskContext {
* Sets a global barrier and waits until all tasks in this stage hit
this barrier. Similar to
* MPI_Barrier function in MPI, the barrier() function call blocks until
all tasks in the same
* stage have reached this routine.
+ *
+ * This function is expected to be called by EVERY tasks in the same
barrier stage in the SAME
+ * pattern, otherwise you may get a SparkException. Some examples of
misuses listed below:
+ * 1. Only call barrier() function on a subset of all the tasks in the
same barrier stage, it
+ * shall lead to time out of the function call.
+ * rdd.barrier().mapPartitions { (iter, context) =>
--- End diff --
This won't be rendered correctly in ScalaDoc/JavaDoc. See
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L160.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
