Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/22001#discussion_r208067143
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -402,6 +403,18 @@ class DAGScheduler(
}
}
+ /**
+ * Check whether the barrier stage requires more slots (to be able to
launch all tasks in the
+ * barrier stage together) than the total number of active slots
currently. Fail fast if trying
+ * to submit a barrier stage that requires more slots than current total
number.
+ */
+ private def checkBarrierStageWithNumSlots(rdd: RDD[_]): Unit = {
+ if (rdd.isBarrier() && rdd.getNumPartitions > sc.getNumSlots) {
+ throw new SparkException(
--- End diff --
We should tolerate temporarily unavailability here by adding a wait.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
