This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.3 by this push:
     new 589cbc5d49cc [SPARK-60497][CORE] Reject a map-stage job that depends 
on a pipelined shuffle
589cbc5d49cc is described below

commit 589cbc5d49ccf90601ef0cc2602bf32a2c175760
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Oct 23 05:39:10 2026 -0700

    [SPARK-60497][CORE] Reject a map-stage job that depends on a pipelined 
shuffle
    
    ### What changes were proposed in this pull request?
    
    This PR makes `DAGScheduler.handleMapStageSubmitted` reject a map-stage job 
whose shuffle dependency has a `PipelinedShuffleDependency` anywhere in its RDD 
lineage. The job fails with `PIPELINED_SHUFFLE_UNSUPPORTED` before any stage is 
created, in the same way that `handleJobSubmitted` rejects the same graph for a 
result job.
    
    ```scala
    if (rddGraphHasPipelinedDependency(dependency.rdd)) {
      val e = pipelinedUnsupportedError("a map-stage job that depends on a 
pipelined shuffle")
      logWarning(...)
      listener.jobFailed(e)
      return
    }
    ```
    
    This adds one RDD graph walk per map-stage job, similar to the walk that 
`handleJobSubmitted` already does for every result job.
    
    ### Why are the changes needed?
    
    `PipelinedShuffleDependency` is added newly at 4.3.0.
    - #57286
    
    `handleMapStageSubmitted` rejected only a dependency that is itself a 
`PipelinedShuffleDependency`. A regular `ShuffleDependency` whose map side 
reads a pipelined shuffle, directly or further down its lineage, was accepted. 
Such a job skipped every pipelined job check of `handleJobSubmitted`:
    
    - the shuffle shape check, which rejects a pipelined shuffle below a 
regular shuffle
    - gang admission (`rejectUnadmittablePipelinedGroup`)
    - `checkPipelinedJobSupported` and the speculation check
    - the run epoch (`SPARK_PIPELINED_RUN_EPOCH` and 
`PipelinedShuffleManager.startRun`)
    - `ActiveJob.hasPipelinedDependency`
    
    However, `submitStage` still co-scheduled the pipelined producer with the 
stage reading it, as ordinary task sets (`TaskSet.isPipelined = false`).
    
    With the in-process channel transport (`PipelinedChannelShuffleManager`), 
this causes a hang or wrong results.
    
    - **Hang.** The tasks have no run epoch, so every channel access throws 
`TaskKilledException("Pipelined shuffle run has ended")`. A kill does not count 
as a task failure in a non-pipelined task set, so the tasks are relaunched 
forever. In a `local[4]` test, 73,470 task attempts were killed in 10 seconds, 
each with a WARN log line, and the job never finished.
    - **Correctness.** If run epoch 0 is active, because job 0 of the 
`SparkContext` is a pipelined job that is still running, the channel accesses 
succeed. A producer attempt that is retried, or relaunched after a kill, then 
delivers its rows again. The channel reader counts end-of-stream markers 
without tracking writer attempts, so the map-stage job can succeed with 
duplicated rows, or stop early and drop the rows of other producers. In local 
tests, the reader saw 5,120 rows instead of  [...]
    
    With the RPC streaming transport, the job runs without gang admission or 
group-atomic failure handling, and speculation is not rejected.
    
    `SparkContext.submitMapStage` is `private[spark]`, and its only in-tree 
caller is AQE (`ShuffleExchangeExec`). `AQEEnablePipelinedShuffle` keeps each 
plan all-pipelined or all-regular, so SQL does not create this shape within one 
plan. It does when an RDD that contains a pipelined shuffle is wrapped in a new 
DataFrame and shuffled under AQE. For example, in local mode with 
`spark.shuffle.manager.incremental=org.apache.spark.shuffle.local.pipelined.PipelinedChannelShuffleManager`
 and ` [...]
    
    ```scala
    val inner = spark.range(0, 1000, 1, 2).withColumn("k", $"id" % 
7).groupBy($"k").count()
    val rows = inner.queryExecution.toRdd.map(r => Row(r.getLong(0), 
r.getLong(1)))
    spark.createDataFrame(rows, 
inner.schema).groupBy($"count").count().collect()
    ```
    
    A user RDD that creates a `PipelinedShuffleDependency` (a `DeveloperApi`) 
reaches it the same way.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, within the unreleased branches only. A map-stage job that depends on a 
pipelined shuffle now fails up front instead of hanging or completing with 
wrong results. For example, the query above fails with:
    
    ```
    [PIPELINED_SHUFFLE_UNSUPPORTED] A pipelined shuffle stage group cannot be 
scheduled because it uses an unsupported feature: a map-stage job that depends 
on a pipelined shuffle. ...
    ```
    
    No released Spark version has `PipelinedShuffleDependency`. `branch-4.3` 
has it with the RPC streaming transport, and `branch-4.x` and `master` also 
have the in-process channel transport.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test cases.
    
    - `DAGSchedulerSuite`: the job is rejected when its map stage reads a 
pipelined shuffle directly and when the pipelined shuffle sits below another 
regular shuffle, and no task set or partial state is left.
    - `PipelinedChannelShuffleSuite`: the job is rejected before any task 
starts, and the same pipelined shuffle still runs as a result job.
    - `AQEPipelinedShuffleSuite`: the query above fails fast instead of 
hanging, and the inner pipelined query still runs.
    
    Without the fix, the first test fails because the job is accepted, and the 
other two time out. The duplicated and dropped rows above were reproduced with 
temporary tests that keep job 0 running, which are not included.
    
    ```
    $ build/sbt 'core/testOnly *DAGSchedulerSuite *PipelinedChannelShuffleSuite 
-- -z SPARK-60497'
    $ build/sbt 'sql/testOnly *AQEPipelinedShuffleSuite -- -z SPARK-60497'
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #59738 from dongjoon-hyun/SPARK-60497.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit 84dce6db0c8e2dbc86df49a32eb09e5d340c7b61)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../org/apache/spark/scheduler/DAGScheduler.scala  | 12 +++++++++
 .../apache/spark/scheduler/DAGSchedulerSuite.scala | 29 ++++++++++++++++++++++
 2 files changed, 41 insertions(+)

diff --git a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala 
b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
index cf2f21e6448f..d59e734761e3 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
@@ -2271,6 +2271,18 @@ private[spark] class DAGScheduler(
           "map output to produce statistics from. This is not supported."))
       return
     }
+    // A map-stage job also runs its missing ancestor stages, and submitStage 
co-schedules a
+    // pipelined producer with the stage reading it, but none of 
handleJobSubmitted's pipelined
+    // setup applies here (shape check, gang admission, idiom checks, run 
epoch, group-atomic
+    // failure). A pipelined shuffle anywhere below `dependency` is the 
pipelined-below-regular
+    // shape that handleJobSubmitted rejects, so reject it here too, before 
any stage is created.
+    if (rddGraphHasPipelinedDependency(dependency.rdd)) {
+      val e = pipelinedUnsupportedError("a map-stage job that depends on a 
pipelined shuffle")
+      logWarning(log"Rejecting map-stage job ${MDC(JOB_ID, jobId)}: 
unsupported " +
+        log"pipelined-shuffle idiom", e)
+      listener.jobFailed(e)
+      return
+    }
     // Submitting this map stage might still require the creation of some 
parent stages, so make
     // sure that happens.
     var finalStage: ShuffleMapStage = null
diff --git 
a/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala 
b/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
index 6b31458abebf..27dfd62effba 100644
--- a/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
@@ -6911,6 +6911,35 @@ class DAGSchedulerSuite extends SparkFunSuite with 
TempLocalSparkContext with Ti
     assertDataStructuresEmpty()
   }
 
+  test("SPARK-60497: a map-stage job that depends on a pipelined shuffle is 
rejected") {
+    // A map-stage job runs its missing ancestor stages, so a pipelined 
producer below its
+    // dependency would be co-scheduled without any of the pipelined-group 
setup of a result job.
+    // As in the result-job path, a pipelined shuffle below a regular one is 
rejected up front,
+    // both when the map stage reads it directly and when it sits below 
another regular shuffle.
+    // No stage is submitted and no partial state remains.
+    val producerRdd = new MyRDD(sc, 2, Nil)
+    val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+    val consumerRdd = new MyRDD(sc, 2, List(pipelinedDep), tracker = 
mapOutputTracker)
+    // consumer --[regular, the map-stage job's dependency]-->
+    val directDep = new ShuffleDependency(consumerRdd, new HashPartitioner(2))
+    // consumer --[regular]--> mid --[regular, the map-stage job's 
dependency]-->
+    val midRdd = new MyRDD(sc, 2, List(new ShuffleDependency(consumerRdd, new 
HashPartitioner(2))),
+      tracker = mapOutputTracker)
+    val deepDep = new ShuffleDependency(midRdd, new HashPartitioner(2))
+    for (dep <- Seq(directDep, deepDep)) {
+      val failure = new 
java.util.concurrent.atomic.AtomicReference[Exception]()
+      val failListener = new JobListener {
+        override def taskSucceeded(index: Int, result: Any): Unit = {}
+        override def jobFailed(exception: Exception): Unit = 
failure.set(exception)
+      }
+      submitMapStage(dep, listener = failListener)
+      assertPipelinedUnsupported(
+        failure.get(), "a map-stage job that depends on a pipelined shuffle")
+    }
+    assert(taskSets.isEmpty)
+    assertDataStructuresEmpty()
+  }
+
   test("regular shuffle map-stage job with speculation enabled is NOT rejected 
(inertness)") {
     // The map-stage speculation fail-fast must be inert for a regular 
ShuffleDependency.
     sc.conf.set(config.SPECULATION_ENABLED, true)


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to