This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new ccd8426579b8 [SPARK-60496][CORE] Limit the pipelined resource profile
check to group members
ccd8426579b8 is described below
commit ccd8426579b8694b15d87a7003fd4734d1741fca
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Oct 23 05:29:18 2026 -0700
[SPARK-60496][CORE] Limit the pipelined resource profile check to group
members
### What changes were proposed in this pull request?
This PR aims to limit the resource profile check of
`DAGScheduler.checkPipelinedJobSupported` to the members of a pipelined group.
Currently, the check runs inside the whole-graph walk of the method. Since
the walk also follows regular shuffle dependencies, it checks the RDDs of a
fully-materialized regular prefix too.
```scala
traverseRDDGraph(finalRDD) { (rdd, enqueue) =>
...
val rp = rdd.getResourceProfile()
if (rp != null && rp.id != ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID) {
hasNonDefaultResourceProfile = true
}
rdd.dependencies.foreach {
...
case sd: ShuffleDependency[_, _, _] =>
shuffleRoots.getOrElseUpdate(sd.shuffleId, sd.rdd)
enqueue(sd.rdd) // Walks into the materialized regular prefix too
...
}
}
```
This PR moves the check after the fan-out check and walks only the
within-stage chains of the group members, which are the result stage and each
pipelined producer. Like the producer side of the reliable checkpoint check, it
uses `traverseParentRDDsWithinStage` with a visited set shared across the
roots. The error message and the order of the checks are unchanged.
### Why are the changes needed?
SPARK-57399 (apache/spark#58097) allows a pipelined job whose regular
shuffles are a fully-materialized prefix below the pipelined shuffles (the
materialized-prefix mixed shape of `JobShuffleShape`). The RDDs in such a
prefix are not members of the pipelined group.
- The prefix stages are skipped because all of their map outputs are
already registered.
- A member stage's resource profile comes only from its own within-stage
chain because `getShuffleDependenciesAndResourceProfiles` stops at shuffle
boundaries.
- The gang admission counts only the result stage and the pipelined
producers (`pipelinedJobConcurrentTaskDemand`).
However, the resource profile check was added by SPARK-58398
(apache/spark#57361) when only all-pipelined jobs were accepted, so the whole
RDD graph was the group at that time. It was not narrowed when the
materialized-prefix shape was added. As a result, a job whose pipelined group
runs entirely on the default profile fails up front only because its
materialized prefix was computed with `RDD.withResources(...)`.
```
[PIPELINED_SHUFFLE_UNSUPPORTED] A pipelined shuffle stage group cannot be
scheduled because it uses an unsupported feature: a pipelined group member with
a non-default resource profile (the whole group must run on the default
profile). ...
```
The other member checks of the same method, the reliable checkpoint check
and the repeated-read check, already stop at regular shuffle boundaries.
### Does this PR introduce _any_ user-facing change?
No for the released Apache Spark versions because the pipelined shuffle
scheduling exists only in the `master` branch.
In the `master` branch, a pipelined job is no longer rejected when only its
materialized regular prefix uses a non-default resource profile. A job is still
rejected when any group member uses one. This case happens only through the RDD
API (`PipelinedShuffleDependency` is a `DeveloperApi`) because SQL does not
produce the materialized-prefix shape. It also requires a non-local deployment
with dynamic allocation disabled, since local mode rejects non-default resource
profiles outside t [...]
### How was this patch tested?
Pass the CIs with the newly added test cases.
Without this fix, the following test case fails with the above error.
- `SPARK-60496: pipelined shuffle: a non-default resource profile on a
materialized regular prefix does not reject the pipelined suffix`
The other new test case, `SPARK-60496: pipelined shuffle: a non-default
resource profile on an RDD shared by a materialized prefix and a member stage
is rejected`, makes sure that an RDD is still checked when it is both the
map-side RDD of the prefix and a narrow parent of a pipelined producer.
In addition, this PR increases the expected edge visits of `pipelined
checkpoint preflight: shared ancestry is visited once` by one because the new
check walks the members' within-stage chains once. The number of visits still
does not grow with the number of roots.
```
$ build/sbt 'core/testOnly org.apache.spark.scheduler.DAGSchedulerSuite --
-z "pipelined" -z "profile"'
...
[info] Tests: succeeded 140, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59737 from dongjoon-hyun/SPARK-60496.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../org/apache/spark/scheduler/DAGScheduler.scala | 22 ++++---
.../apache/spark/scheduler/DAGSchedulerSuite.scala | 68 +++++++++++++++++++++-
2 files changed, 78 insertions(+), 12 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
index 06bed7cad55e..c021d26d2802 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
@@ -1436,19 +1436,10 @@ private[spark] class DAGScheduler(
val consumerRDDs = new HashSet[RDD[_]] // RDDs that READ a
pipelined shuffle
val narrowChildren = new HashMap[RDD[_], ListBuffer[RDD[_]]]
val reliablyCheckpointed = new HashSet[RDD[_]] // RDDs with a reliable
checkpoint pending
- var hasNonDefaultResourceProfile = false // any member RDD with a
non-default RP
traverseRDDGraph(finalRDD) { (rdd, enqueue) =>
if
(rdd.checkpointData.exists(_.isInstanceOf[ReliableRDDCheckpointData[_]])) {
reliablyCheckpointed += rdd
}
- // An RDD's EFFECTIVE profile is its explicit one, or the default when
unset. The slot check
- // measures capacity against the default profile (see
rejectUnadmittablePipelinedGroup), so
- // the whole group must run on the default profile; any explicit
non-default profile on a
- // member makes the group span profiles (against the default the rest
use) and is rejected.
- val rp = rdd.getResourceProfile()
- if (rp != null && rp.id != ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID) {
- hasNonDefaultResourceProfile = true
- }
rdd.dependencies.foreach {
case pd: PipelinedShuffleDependency[_, _, _] =>
consumersByShuffleId.getOrElseUpdate(pd.shuffleId, new HashSet[Int])
+= rdd.id
@@ -1478,6 +1469,19 @@ private[spark] class DAGScheduler(
// profile -- that both spans profiles (against the default the other
members use) and would be
// admitted against the wrong (default) capacity pool. This is a real
check, not just a
// documented assumption: nothing else enforces it.
+ //
+ // Check only the members' within-stage chains, rooted at the result stage
and at each
+ // pipelined producer, the only stages this job runs. A materialized
regular prefix (see
+ // JobShuffleShape) never runs, so its profile is not a member's. An RDD's
EFFECTIVE profile is
+ // its explicit one, or the default when unset.
+ val usesDefaultProfile = (r: RDD[_]) => {
+ val rp = r.getResourceProfile()
+ rp == null || rp.id == ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID
+ }
+ val memberVisited = new HashSet[RDD[_]]
+ val hasNonDefaultResourceProfile = (Iterator.single(finalRDD) ++
producerRoots).exists {
+ root => !traverseParentRDDsWithinStage(root, usesDefaultProfile,
memberVisited)
+ }
if (hasNonDefaultResourceProfile) {
throw pipelinedUnsupportedError(
"a pipelined group member with a non-default resource profile (the
whole group must run " +
diff --git
a/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
b/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
index 715efc831813..f09b7d160ea7 100644
--- a/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
@@ -9581,9 +9581,10 @@ class DAGSchedulerSuite extends SparkFunSuite with
TempLocalSparkContext with Ti
edgeReads.foreach(_.set(0))
submit(result, Array(0, 1))
assert(failure === null)
- // Classification, admission, and group collection each visit once.
Only the search
- // rooted on this shared chain adds a visit, regardless of the number
of roots.
- val expected = if (roots == "no-checkpoint") 3 else 4
+ // Classification, admission, group collection, and the member
resource-profile search
+ // each visit once. Only the checkpoint search rooted on this shared
chain adds a visit,
+ // regardless of the number of roots.
+ val expected = if (roots == "no-checkpoint") 4 else 5
assert(preflightEdgeReads === Seq.fill(edgeReads.size)(expected))
pipelined.foreach { dep =>
completeShuffleMapStageSuccessfully(
@@ -9995,6 +9996,67 @@ class DAGSchedulerSuite extends SparkFunSuite with
TempLocalSparkContext with Ti
assertDataStructuresEmpty()
}
+ test("SPARK-60496: pipelined shuffle: a non-default resource profile on a
materialized " +
+ "regular prefix does not reject the pipelined suffix") {
+ // The materialized prefix never runs in the pipelined job (see
JobShuffleShape), so its
+ // profile is not a group member's profile. A task-only profile is the
only non-default kind
+ // a pipelined job can use, since dynamic allocation is rejected for it.
+ val taskRp = new TaskResourceProfile(new
TaskResourceRequests().cpus(1).requests)
+ val regularRoot = new MyRDD(sc, 2, Nil).withResources(taskRp)
+ val regularDep = new ShuffleDependency(regularRoot, new HashPartitioner(3))
+ val prefixReader = new MyRDD(sc, 3, List(regularDep), tracker =
mapOutputTracker)
+ // A non-default profile needs non-null job properties when its stage is
submitted.
+ submit(prefixReader, Array(0, 1, 2), properties = new Properties())
+ completeShuffleMapStageSuccessfully(taskSets.head.stageId, 0, 3)
+ complete(taskSets(1), Seq((Success, 1), (Success, 2), (Success, 3)))
+ assert(results === Map(0 -> 1, 1 -> 2, 2 -> 3))
+ results.clear()
+ val taskSetsBefore = taskSets.size
+
+ // producer(reads the materialized regular prefix) --[pipelined]-->
consumer (result stage).
+ // Both members are on the default profile.
+ val producerRdd = new MyRDD(sc, 3, List(regularDep), tracker =
mapOutputTracker)
+ val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new
HashPartitioner(2))
+ val consumerRdd = new MyRDD(sc, 2, List(pipelinedDep), tracker =
mapOutputTracker)
+ submit(consumerRdd, Array(0, 1))
+
+ assert(failure === null, s"the prefix's profile must not reject the job,
got: $failure")
+ assert(taskSets.size === taskSetsBefore + 2,
+ s"expected producer and consumer submitted concurrently, got
${taskSets.size} task sets")
+ assert(!scheduler.runningStages.exists(_.rdd eq regularRoot),
+ "the materialized prefix stage must not re-run")
+ completeShuffleMapStageSuccessfully(taskSets(taskSetsBefore).stageId, 0, 2)
+ complete(taskSets(taskSetsBefore + 1), Seq((Success, 42), (Success, 43)))
+ assert(results === Map(0 -> 42, 1 -> 43))
+ assertDataStructuresEmpty()
+ }
+
+ test("SPARK-60496: pipelined shuffle: a non-default resource profile on an
RDD shared by a " +
+ "materialized prefix and a member stage is rejected") {
+ // `shared` is the materialized prefix's map RDD and also a narrow parent
of the pipelined
+ // producer, so it runs inside that member stage too. The member check
must reach it through
+ // the narrow edge even though the prefix itself is not checked.
+ val taskRp = new TaskResourceProfile(new
TaskResourceRequests().cpus(1).requests)
+ val shared = new MyRDD(sc, 2, Nil).withResources(taskRp)
+ val regularDep = new ShuffleDependency(shared, new HashPartitioner(2))
+ val prefixReader = new MyRDD(sc, 2, List(regularDep), tracker =
mapOutputTracker)
+ // A non-default profile needs non-null job properties when its stage is
submitted.
+ submit(prefixReader, Array(0, 1), properties = new Properties())
+ completeShuffleMapStageSuccessfully(taskSets.head.stageId, 0, 2)
+ complete(taskSets(1), Seq((Success, 1), (Success, 2)))
+ assert(results === Map(0 -> 1, 1 -> 2))
+ val taskSetsBefore = taskSets.size
+
+ val producerRdd = new MyRDD(sc, 2, List(regularDep, new
OneToOneDependency(shared)),
+ tracker = mapOutputTracker)
+ val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new
HashPartitioner(2))
+ val consumerRdd = new MyRDD(sc, 2, List(pipelinedDep), tracker =
mapOutputTracker)
+ assertPipelinedUnsupported(
+ submitAndCaptureFailure(consumerRdd, Array(0, 1)), "non-default resource
profile")
+ assert(taskSets.size === taskSetsBefore, "no stage should be submitted for
a rejected job")
+ assertDataStructuresEmpty()
+ }
+
test("regular shuffle idioms are NOT rejected (inertness of the pipelined
fail-fast)") {
// The pipelined fail-fast checks (indeterminate producer,
reliable-checkpoint-in-chain) must be
// inert for a job with NO pipelined dependency: a regular shuffle whose
producer is BOTH
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]