Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/21927#discussion_r206595897
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -340,6 +340,18 @@ class DAGScheduler(
}
}
+ /**
+ * Check to make sure we are not launching a barrier stage that contains
PartitionPruningRDD,
+ * which may launch tasks on partial partitions.
+ */
+ private def checkBarrierStageWithPartitionPruningRDD(rdd: RDD[_]): Unit
= {
+ if (rdd.isBarrier() &&
+ !traverseParentRDDsWithinStage(rdd, (r =>
!r.isInstanceOf[PartitionPruningRDD[_]]))) {
+ throw new SparkException("Don't support run a barrier stage that
contains " +
--- End diff --
* Since the error message is used in the test, it would be nice to make it
a package private constant and use it in the test.
* "Barrier execution mode does not support partition pruning
(PartitionPruningRDD)." should be sufficient.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
