This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new ccd8426579b8 [SPARK-60496][CORE] Limit the pipelined resource profile 
check to group members
ccd8426579b8 is described below

commit ccd8426579b8694b15d87a7003fd4734d1741fca
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Oct 23 05:29:18 2026 -0700

    [SPARK-60496][CORE] Limit the pipelined resource profile check to group 
members
    
    ### What changes were proposed in this pull request?
    
    This PR aims to limit the resource profile check of 
`DAGScheduler.checkPipelinedJobSupported` to the members of a pipelined group.
    
    Currently, the check runs inside the whole-graph walk of the method. Since 
the walk also follows regular shuffle dependencies, it checks the RDDs of a 
fully-materialized regular prefix too.
    
    ```scala
    traverseRDDGraph(finalRDD) { (rdd, enqueue) =>
      ...
      val rp = rdd.getResourceProfile()
      if (rp != null && rp.id != ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID) {
        hasNonDefaultResourceProfile = true
      }
      rdd.dependencies.foreach {
        ...
        case sd: ShuffleDependency[_, _, _] =>
          shuffleRoots.getOrElseUpdate(sd.shuffleId, sd.rdd)
          enqueue(sd.rdd) // Walks into the materialized regular prefix too
        ...
      }
    }
    ```
    
    This PR moves the check after the fan-out check and walks only the 
within-stage chains of the group members, which are the result stage and each 
pipelined producer. Like the producer side of the reliable checkpoint check, it 
uses `traverseParentRDDsWithinStage` with a visited set shared across the 
roots. The error message and the order of the checks are unchanged.
    
    ### Why are the changes needed?
    
    SPARK-57399 (apache/spark#58097) allows a pipelined job whose regular 
shuffles are a fully-materialized prefix below the pipelined shuffles (the 
materialized-prefix mixed shape of `JobShuffleShape`). The RDDs in such a 
prefix are not members of the pipelined group.
    - The prefix stages are skipped because all of their map outputs are 
already registered.
    - A member stage's resource profile comes only from its own within-stage 
chain because `getShuffleDependenciesAndResourceProfiles` stops at shuffle 
boundaries.
    - The gang admission counts only the result stage and the pipelined 
producers (`pipelinedJobConcurrentTaskDemand`).
    
    However, the resource profile check was added by SPARK-58398 
(apache/spark#57361) when only all-pipelined jobs were accepted, so the whole 
RDD graph was the group at that time. It was not narrowed when the 
materialized-prefix shape was added. As a result, a job whose pipelined group 
runs entirely on the default profile fails up front only because its 
materialized prefix was computed with `RDD.withResources(...)`.
    
    ```
    [PIPELINED_SHUFFLE_UNSUPPORTED] A pipelined shuffle stage group cannot be 
scheduled because it uses an unsupported feature: a pipelined group member with 
a non-default resource profile (the whole group must run on the default 
profile). ...
    ```
    
    The other member checks of the same method, the reliable checkpoint check 
and the repeated-read check, already stop at regular shuffle boundaries.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No for the released Apache Spark versions because the pipelined shuffle 
scheduling exists only in the `master` branch.
    
    In the `master` branch, a pipelined job is no longer rejected when only its 
materialized regular prefix uses a non-default resource profile. A job is still 
rejected when any group member uses one. This case happens only through the RDD 
API (`PipelinedShuffleDependency` is a `DeveloperApi`) because SQL does not 
produce the materialized-prefix shape. It also requires a non-local deployment 
with dynamic allocation disabled, since local mode rejects non-default resource 
profiles outside t [...]
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test cases.
    
    Without this fix, the following test case fails with the above error.
    - `SPARK-60496: pipelined shuffle: a non-default resource profile on a 
materialized regular prefix does not reject the pipelined suffix`
    
    The other new test case, `SPARK-60496: pipelined shuffle: a non-default 
resource profile on an RDD shared by a materialized prefix and a member stage 
is rejected`, makes sure that an RDD is still checked when it is both the 
map-side RDD of the prefix and a narrow parent of a pipelined producer.
    
    In addition, this PR increases the expected edge visits of `pipelined 
checkpoint preflight: shared ancestry is visited once` by one because the new 
check walks the members' within-stage chains once. The number of visits still 
does not grow with the number of roots.
    
    ```
    $ build/sbt 'core/testOnly org.apache.spark.scheduler.DAGSchedulerSuite -- 
-z "pipelined" -z "profile"'
    ...
    [info] Tests: succeeded 140, failed 0, canceled 0, ignored 0, pending 0
    [info] All tests passed.
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #59737 from dongjoon-hyun/SPARK-60496.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../org/apache/spark/scheduler/DAGScheduler.scala  | 22 ++++---
 .../apache/spark/scheduler/DAGSchedulerSuite.scala | 68 +++++++++++++++++++++-
 2 files changed, 78 insertions(+), 12 deletions(-)

diff --git a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala 
b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
index 06bed7cad55e..c021d26d2802 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
@@ -1436,19 +1436,10 @@ private[spark] class DAGScheduler(
     val consumerRDDs = new HashSet[RDD[_]]            // RDDs that READ a 
pipelined shuffle
     val narrowChildren = new HashMap[RDD[_], ListBuffer[RDD[_]]]
     val reliablyCheckpointed = new HashSet[RDD[_]]     // RDDs with a reliable 
checkpoint pending
-    var hasNonDefaultResourceProfile = false          // any member RDD with a 
non-default RP
     traverseRDDGraph(finalRDD) { (rdd, enqueue) =>
       if 
(rdd.checkpointData.exists(_.isInstanceOf[ReliableRDDCheckpointData[_]])) {
         reliablyCheckpointed += rdd
       }
-      // An RDD's EFFECTIVE profile is its explicit one, or the default when 
unset. The slot check
-      // measures capacity against the default profile (see 
rejectUnadmittablePipelinedGroup), so
-      // the whole group must run on the default profile; any explicit 
non-default profile on a
-      // member makes the group span profiles (against the default the rest 
use) and is rejected.
-      val rp = rdd.getResourceProfile()
-      if (rp != null && rp.id != ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID) {
-        hasNonDefaultResourceProfile = true
-      }
       rdd.dependencies.foreach {
         case pd: PipelinedShuffleDependency[_, _, _] =>
           consumersByShuffleId.getOrElseUpdate(pd.shuffleId, new HashSet[Int]) 
+= rdd.id
@@ -1478,6 +1469,19 @@ private[spark] class DAGScheduler(
     // profile -- that both spans profiles (against the default the other 
members use) and would be
     // admitted against the wrong (default) capacity pool. This is a real 
check, not just a
     // documented assumption: nothing else enforces it.
+    //
+    // Check only the members' within-stage chains, rooted at the result stage 
and at each
+    // pipelined producer, the only stages this job runs. A materialized 
regular prefix (see
+    // JobShuffleShape) never runs, so its profile is not a member's. An RDD's 
EFFECTIVE profile is
+    // its explicit one, or the default when unset.
+    val usesDefaultProfile = (r: RDD[_]) => {
+      val rp = r.getResourceProfile()
+      rp == null || rp.id == ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID
+    }
+    val memberVisited = new HashSet[RDD[_]]
+    val hasNonDefaultResourceProfile = (Iterator.single(finalRDD) ++ 
producerRoots).exists {
+      root => !traverseParentRDDsWithinStage(root, usesDefaultProfile, 
memberVisited)
+    }
     if (hasNonDefaultResourceProfile) {
       throw pipelinedUnsupportedError(
         "a pipelined group member with a non-default resource profile (the 
whole group must run " +
diff --git 
a/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala 
b/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
index 715efc831813..f09b7d160ea7 100644
--- a/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
@@ -9581,9 +9581,10 @@ class DAGSchedulerSuite extends SparkFunSuite with 
TempLocalSparkContext with Ti
         edgeReads.foreach(_.set(0))
         submit(result, Array(0, 1))
         assert(failure === null)
-        // Classification, admission, and group collection each visit once. 
Only the search
-        // rooted on this shared chain adds a visit, regardless of the number 
of roots.
-        val expected = if (roots == "no-checkpoint") 3 else 4
+        // Classification, admission, group collection, and the member 
resource-profile search
+        // each visit once. Only the checkpoint search rooted on this shared 
chain adds a visit,
+        // regardless of the number of roots.
+        val expected = if (roots == "no-checkpoint") 4 else 5
         assert(preflightEdgeReads === Seq.fill(edgeReads.size)(expected))
         pipelined.foreach { dep =>
           completeShuffleMapStageSuccessfully(
@@ -9995,6 +9996,67 @@ class DAGSchedulerSuite extends SparkFunSuite with 
TempLocalSparkContext with Ti
     assertDataStructuresEmpty()
   }
 
+  test("SPARK-60496: pipelined shuffle: a non-default resource profile on a 
materialized " +
+      "regular prefix does not reject the pipelined suffix") {
+    // The materialized prefix never runs in the pipelined job (see 
JobShuffleShape), so its
+    // profile is not a group member's profile. A task-only profile is the 
only non-default kind
+    // a pipelined job can use, since dynamic allocation is rejected for it.
+    val taskRp = new TaskResourceProfile(new 
TaskResourceRequests().cpus(1).requests)
+    val regularRoot = new MyRDD(sc, 2, Nil).withResources(taskRp)
+    val regularDep = new ShuffleDependency(regularRoot, new HashPartitioner(3))
+    val prefixReader = new MyRDD(sc, 3, List(regularDep), tracker = 
mapOutputTracker)
+    // A non-default profile needs non-null job properties when its stage is 
submitted.
+    submit(prefixReader, Array(0, 1, 2), properties = new Properties())
+    completeShuffleMapStageSuccessfully(taskSets.head.stageId, 0, 3)
+    complete(taskSets(1), Seq((Success, 1), (Success, 2), (Success, 3)))
+    assert(results === Map(0 -> 1, 1 -> 2, 2 -> 3))
+    results.clear()
+    val taskSetsBefore = taskSets.size
+
+    // producer(reads the materialized regular prefix) --[pipelined]--> 
consumer (result stage).
+    // Both members are on the default profile.
+    val producerRdd = new MyRDD(sc, 3, List(regularDep), tracker = 
mapOutputTracker)
+    val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+    val consumerRdd = new MyRDD(sc, 2, List(pipelinedDep), tracker = 
mapOutputTracker)
+    submit(consumerRdd, Array(0, 1))
+
+    assert(failure === null, s"the prefix's profile must not reject the job, 
got: $failure")
+    assert(taskSets.size === taskSetsBefore + 2,
+      s"expected producer and consumer submitted concurrently, got 
${taskSets.size} task sets")
+    assert(!scheduler.runningStages.exists(_.rdd eq regularRoot),
+      "the materialized prefix stage must not re-run")
+    completeShuffleMapStageSuccessfully(taskSets(taskSetsBefore).stageId, 0, 2)
+    complete(taskSets(taskSetsBefore + 1), Seq((Success, 42), (Success, 43)))
+    assert(results === Map(0 -> 42, 1 -> 43))
+    assertDataStructuresEmpty()
+  }
+
+  test("SPARK-60496: pipelined shuffle: a non-default resource profile on an 
RDD shared by a " +
+      "materialized prefix and a member stage is rejected") {
+    // `shared` is the materialized prefix's map RDD and also a narrow parent 
of the pipelined
+    // producer, so it runs inside that member stage too. The member check 
must reach it through
+    // the narrow edge even though the prefix itself is not checked.
+    val taskRp = new TaskResourceProfile(new 
TaskResourceRequests().cpus(1).requests)
+    val shared = new MyRDD(sc, 2, Nil).withResources(taskRp)
+    val regularDep = new ShuffleDependency(shared, new HashPartitioner(2))
+    val prefixReader = new MyRDD(sc, 2, List(regularDep), tracker = 
mapOutputTracker)
+    // A non-default profile needs non-null job properties when its stage is 
submitted.
+    submit(prefixReader, Array(0, 1), properties = new Properties())
+    completeShuffleMapStageSuccessfully(taskSets.head.stageId, 0, 2)
+    complete(taskSets(1), Seq((Success, 1), (Success, 2)))
+    assert(results === Map(0 -> 1, 1 -> 2))
+    val taskSetsBefore = taskSets.size
+
+    val producerRdd = new MyRDD(sc, 2, List(regularDep, new 
OneToOneDependency(shared)),
+      tracker = mapOutputTracker)
+    val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+    val consumerRdd = new MyRDD(sc, 2, List(pipelinedDep), tracker = 
mapOutputTracker)
+    assertPipelinedUnsupported(
+      submitAndCaptureFailure(consumerRdd, Array(0, 1)), "non-default resource 
profile")
+    assert(taskSets.size === taskSetsBefore, "no stage should be submitted for 
a rejected job")
+    assertDataStructuresEmpty()
+  }
+
   test("regular shuffle idioms are NOT rejected (inertness of the pipelined 
fail-fast)") {
     // The pipelined fail-fast checks (indeterminate producer, 
reliable-checkpoint-in-chain) must be
     // inert for a job with NO pipelined dependency: a regular shuffle whose 
producer is BOTH


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to