This is an automated email from the ASF dual-hosted git repository.

cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 9a979187528f [SPARK-58263][CORE] Concurrently schedule 
pipelined-shuffle stage groups in the DAGScheduler
9a979187528f is described below

commit 9a979187528f7ba19923616a79729fae99cfc586
Author: Boyang Jerry Peng <[email protected]>
AuthorDate: Tue Jul 28 13:41:45 2026 +0800

    [SPARK-58263][CORE] Concurrently schedule pipelined-shuffle stage groups in 
the DAGScheduler
    
    ### What changes were proposed in this pull request?
    
    This PR adds native `DAGScheduler` support for **concurrently scheduling 
stages connected by a
    `PipelinedShuffleDependency`** (added earlier in the stack), together with 
the admission and
    completion semantics that make co-scheduling correct.
    
    A pipelined shuffle is incrementally readable: a consumer stage may begin 
reading the producer's
    output while the producer is still running. The stock scheduler runs a 
consumer only after its
    producer has fully materialized; this PR teaches the scheduler to 
co-schedule a producer and its
    pipelined consumer as a **pipelined group** instead. Every new path is 
gated on a job actually
    using a pipelined dependency, so behavior is unchanged for all existing 
jobs (the existing
    `DAGSchedulerSuite` is unaffected).
    
    This PR supports a job that is **all-regular or all-pipelined** (a job 
mixing the two is rejected
    up front). So an all-pipelined job's whole stage graph is one pipelined 
group, which lets admission
    be decided once, up front.
    
    - **Up-front gang admission (`handleJobSubmitted`).** All members of a 
pipelined group must run
      concurrently, so a group that cannot fit would deadlock (a consumer holds 
slots waiting for
      producer output while the producer cannot get slots to produce). Before 
any stage is created, the
      group's total task demand (computed from the RDD graph) is compared 
against the currently **free**
      slots of its resource profile -- total capacity (`maxNumConcurrentTasks`) 
minus the *outstanding*
      (running **plus enqueued**) task demand of other work in the same 
profile. Counting enqueued, not
      just running, demand prevents two groups from each passing the check yet 
failing to co-fit. If the
      group does not fit, the job fails fast with 
`CONCURRENT_SCHEDULER_INSUFFICIENT_SLOT` before any
      member runs -- true all-or-nothing admission that leaves no partial 
scheduler state (like the
      barrier slot check). There is no scheduler-side retry: a transient 
shortfall is the caller's to
      retry (a streaming query's batch loop reruns the batch). The check can be 
disabled with
      `spark.scheduler.pipelinedGroup.slotCheck.enabled=false` for deployments 
that admit capacity
      out-of-band (e.g. a slot reservation).
    
    - **Co-scheduling (`submitStage`).** A stage's missing parents are 
classified by shuffle dependency
      type; a parent reached through a `PipelinedShuffleDependency` is a 
"pipelined parent". The stage is
      co-scheduled with its producers (tasks submitted immediately) only if 
every missing parent is
      pipelined **and** each is already running; otherwise it parks in 
`waitingStages` exactly as before.
      `submitWaitingPipelinedChildStages` is the "producer started running" 
analog of the existing
      "producer completed" hook: when a pipelined producer starts, its waiting 
consumers are reconsidered
      immediately, cascading transitively down a chain.
    
    - **Deferred completion for a co-scheduled consumer (`handleTaskCompletion` 
/
      `markStageAsFinished`).** A consumer co-scheduled with a still-running 
producer can finish first.
      Advancing its stage/job completion early would end the job and cancel the 
still-running producer,
      or make the consumer's output observable before the producer's. So the 
consumer's completion is
      deferred until the producer's outcome is final: its whole 
`CompletionEvent` is buffered and
      returns before any side effect runs (accumulator updates, 
`SparkListenerTaskEnd`, stage/job
      completion), which makes those side effects run exactly once, at replay. 
The buffered event is
      replayed on genuine producer success (applied normally), or dropped on 
producer failure -- on the
      drop the buffered tasks' `TaskEnd` events are still emitted (the tasks 
did finish, so active-task
      listeners must see them end) but no stage/job success is applied, since 
the group reruns. Deferrals
      are cleaned up on job end/abort so none outlive their job, flushing any 
still-buffered `TaskEnd`
      events on the way out.
    
    - **Other guards.** A job using a pipelined dependency is rejected up front 
when speculation is
      enabled (a speculative producer copy would race a consumer already 
reading partial output, with no
      commit barrier), when dynamic allocation is enabled (gang admission needs 
a stable slot set), or
      when a group member carries a **non-default resource profile** (gang 
admission measures capacity
      against the default profile, so the whole group must run on it -- 
per-profile admission is a
      follow-up); and a `PipelinedShuffleDependency` cannot be submitted as a 
map-stage job (no durable
      map output to compute statistics from).
    
    Main changes:
    
    - `DAGScheduler.scala` -- job classification, up-front gang admission, 
co-scheduling, and
      completion deferral. A one-pass RDD-graph walk 
(`rddGraphHasPipelinedDependency` /
      `classifyJobShuffleKinds`) keeps every new path inert for a job with no 
pipelined dependency.
    - `TaskSchedulerImpl.outstandingTasksForOtherWorkInProfile` -- a 
resource-profile-scoped
      outstanding-task (running + enqueued) count for the admission check 
(`private[scheduler]`).
    - `spark.scheduler.pipelinedGroup.slotCheck.enabled` -- a new `internal()` 
config (default `true`).
    - `CONCURRENT_SCHEDULER_INSUFFICIENT_SLOT` -- a new error condition.
    
    This is a follow-up to `PipelinedShuffleDependency` and the 
dependency-typed shuffle-manager
    routing (SPARK-58185, already in `master`); it is the first PR that makes 
the scheduler behave
    differently for a pipelined dependency. Group-atomic failure/rerun and 
additional fail-fast checks
    for unsupported idioms follow in later PRs of the stack.
    
    ### Why are the changes needed?
    
    `PipelinedShuffleDependency` and its incremental shuffle routing let a 
consumer read a producer's
    output as it is produced, but nothing takes advantage of that until the 
scheduler co-schedules the
    two stages -- otherwise the consumer still waits for the producer to fully 
materialize and the
    pipelining is never realized. Co-scheduling in turn requires admission 
control (a group that cannot
    co-fit must fail fast, not deadlock) and completion control (a 
fast-finishing consumer must not end
    the job or cancel its producer). This PR provides both.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. All new behavior is gated on a job using a 
`PipelinedShuffleDependency`, which nothing constructs
    yet, so for every existing job the scheduler behaves exactly as before. The 
new
    `spark.scheduler.pipelinedGroup.slotCheck.enabled` config is `internal()` 
and defaults to `true`,
    and `CONCURRENT_SCHEDULER_INSUFFICIENT_SLOT` is a new error condition that 
can only surface for a job
    that uses a pipelined dependency.
    
    ### How was this patch tested?
    
    New unit tests in `DAGSchedulerSuite` cover:
    
    - concurrent submission of a pipelined producer/consumer; inertness for a 
regular shuffle; a job
      mixing pipelined + regular shuffles rejected up front; a deep 
all-pipelined chain co-scheduled;
      transitive cascade when a producer starts; no double-submission on 
producer completion;
    - speculation, dynamic-allocation, and non-default-resource-profile 
rejection for a pipelined job
      (and that the corresponding regular jobs are not rejected -- including a 
regular job that merely
      attaches a non-default profile via `RDD.withResources`); a pipelined 
dependency submitted as a
      map-stage job rejected;
    - up-front admission: a group too large to co-fit failing fast with
      `CONCURRENT_SCHEDULER_INSUFFICIENT_SLOT`; a group that fits total 
capacity but not free slots
      failing fast; a group whose demand exactly equals free capacity admitted; 
other work's outstanding
      demand charged against admission; the slot check disabled admitting an 
over-capacity group;
    - deferred completion: an early-finishing consumer not ending the job or 
cancelling its running
      producer; its completion buffered until the producer finishes, then 
applied exactly once at replay
      (no buffer+replay `TaskEnd` duplication); normal producer-then-consumer 
ordering; the deferral
      dropped on producer failure; the deferral released only when the producer 
is genuinely available;
      an explicit job cancellation cleaning up the buffered deferral; and job 
teardown flushing a
      buffered consumer's `TaskEnd` events even when the release path never 
drained it (so active-task
      listeners do not leak the tasks as running).
    
    `TaskSchedulerImplSuite` covers `outstandingTasksForOtherWorkInProfile` 
counting running + enqueued
    tasks, excluding given stages, being resource-profile-scoped, and not 
double-counting a
    zombie + live attempt.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Co-authored with: Claude Code (Opus 4.8)
    
    Closes #57341 from jerrypeng/stack/pipelined-shuffle-pr3-scheduling.
    
    Authored-by: Boyang Jerry Peng <[email protected]>
    Signed-off-by: Wenchen Fan <[email protected]>
    (cherry picked from commit 35309c5336dbf08781212623d2ce05e7a1ba86f9)
    Signed-off-by: Wenchen Fan <[email protected]>
---
 .../src/main/resources/error/error-conditions.json |   6 +
 .../org/apache/spark/internal/config/package.scala |  15 +
 .../org/apache/spark/scheduler/DAGScheduler.scala  | 568 ++++++++++++-
 .../apache/spark/scheduler/TaskSchedulerImpl.scala |  31 +
 .../apache/spark/scheduler/DAGSchedulerSuite.scala | 878 +++++++++++++++++++++
 .../spark/scheduler/TaskSchedulerImplSuite.scala   |  58 ++
 6 files changed, 1554 insertions(+), 2 deletions(-)

diff --git a/common/utils/src/main/resources/error/error-conditions.json 
b/common/utils/src/main/resources/error/error-conditions.json
index 04aa72c8cc8a..54a65e965867 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -1188,6 +1188,12 @@
     ],
     "sqlState" : "0A000"
   },
+  "CONCURRENT_SCHEDULER_INSUFFICIENT_SLOT" : {
+    "message" : [
+      "Cannot run the pipelined stage group: it needs <numTasks> concurrent 
task slots to run all its stages together, but only <numSlots> are currently 
free. The stages of a pipelined group must run concurrently, so the cluster 
must have enough free slots for all of them at once. Provision more executors, 
or reduce the parallelism of the query."
+    ],
+    "sqlState" : "53000"
+  },
   "CONCURRENT_STREAM_LOG_UPDATE" : {
     "message" : [
       "Concurrent update to the log. Multiple streaming jobs detected for 
<batchId>.",
diff --git a/core/src/main/scala/org/apache/spark/internal/config/package.scala 
b/core/src/main/scala/org/apache/spark/internal/config/package.scala
index 9ddb0a025ab7..56e384ab49c6 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/package.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/package.scala
@@ -2017,6 +2017,21 @@ package object config {
       .checkValue(v => v > 0, "The max failures should be a positive value.")
       .createWithDefault(40)
 
+  private[spark] val PIPELINED_GROUP_SLOT_CHECK_ENABLED =
+    ConfigBuilder("spark.scheduler.pipelinedGroup.slotCheck.enabled")
+      .internal()
+      .doc("When true, before co-scheduling a pipelined-shuffle stage group 
the DAGScheduler " +
+        "checks that the group's total task demand fits in the currently free 
slots of its " +
+        "resource profile (total capacity minus the outstanding -- running 
plus enqueued -- " +
+        "tasks of other work), and fails the job with 
CONCURRENT_SCHEDULER_INSUFFICIENT_SLOT " +
+        "rather than co-scheduling a group that cannot fit and deadlocking. 
Set to false for " +
+        "deployments that admit capacity out-of-band (e.g. a slot 
reservation), which then own " +
+        "admission. Only applies to jobs that use a pipelined shuffle 
dependency.")
+      .version("4.3.0")
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+      .booleanConf
+      .createWithDefault(true)
+
   private[spark] val NUM_CANCELLED_JOB_GROUPS_TO_TRACK =
     ConfigBuilder("spark.scheduler.numCancelledJobGroupsToTrack")
       .doc("The maximum number of tracked job groups that are cancelled with " 
+
diff --git a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala 
b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
index b87ae41c4bc4..bfd24ceb2f89 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
@@ -169,6 +169,24 @@ private[spark] class DAGScheduler(
   // Stages that must be resubmitted due to fetch failures
   private[scheduler] val failedStages = new HashSet[Stage]
 
+  /**
+   * Deferred completion for pipelined groups. When a stage (a "pipelined 
consumer") is co-scheduled
+   * with a pipelined producer that is still running, the stage/job-completion 
decision from
+   * its successful task-completion events must not be processed yet: doing so 
could finish the
+   * consumer's job and cancel the still-running producer, or make the 
consumer's output observable
+   * before the producer's. We buffer such events here and replay them once 
every pipelined producer
+   * the consumer was waiting on has finished.
+   *
+   * Keyed by the consumer stage. `parents` is the set of its pipelined 
producer stages not yet
+   * finished; `delayedTaskCompletionEvents` are its Success CompletionEvents 
held until then.
+   * Entries exist only for stages that were co-scheduled with a 
not-yet-finished pipelined
+   * producer, so this is empty for any job without a pipelined dependency.
+   */
+  private[scheduler] case class DependentStageInfo(
+      parents: HashSet[Stage] = new HashSet[Stage],
+      delayedTaskCompletionEvents: ListBuffer[CompletionEvent] = new 
ListBuffer[CompletionEvent])
+  private[scheduler] val dependentStageMap = new HashMap[Stage, 
DependentStageInfo]
+
   private[scheduler] val activeJobs = new HashSet[ActiveJob]
 
   // Track all the jobs submitted by the same query execution, will clean up 
after
@@ -945,6 +963,103 @@ private[spark] class DAGScheduler(
     missing.toList
   }
 
+  /**
+   * Whether the RDD graph rooted at `finalRDD` contains a 
[[PipelinedShuffleDependency]] anywhere.
+   * Walks the RDD dependency graph directly (not the stage graph), so it can 
be checked before any
+   * stages are created -- letting a job be rejected up front without leaving 
partial scheduler
+   * state behind.
+   */
+  private def rddGraphHasPipelinedDependency(finalRDD: RDD[_]): Boolean = {
+    // traverseRDDGraphUntil stops and returns false as soon as the visitor 
returns false; we use
+    // that to short-circuit on the first pipelined dependency found. It 
returns true if the whole
+    // graph was visited without stopping (i.e. none found), so negate the 
result.
+    !traverseRDDGraphUntil(finalRDD) { (rdd, enqueue) =>
+      val hasPipelined = rdd.dependencies.exists {
+        case _: PipelinedShuffleDependency[_, _, _] => true
+        case dep =>
+          enqueue(dep.rdd)
+          false
+      }
+      !hasPipelined // keep traversing while none found; stop (return false) 
when one is found
+    }
+  }
+
+  /**
+   * Classifies the shuffle boundaries in the RDD graph rooted at `finalRDD` 
by kind, walking the
+   * RDD dependency graph directly (before any stages are created). Returns 
whether the graph
+   * contains any [[PipelinedShuffleDependency]] and whether it contains any 
regular (non-pipelined)
+   * `ShuffleDependency`. Narrow dependencies are not boundaries and are 
ignored.
+   *
+   * A job must be either ALL-regular or ALL-pipelined: a job whose shuffle 
graph mixes a pipelined
+   * shuffle with a regular one is rejected fail-fast (see 
`handleJobSubmitted`). Restricting to a
+   * single kind makes the whole job one pipelined group when pipelined, so 
gang admission can be
+   * decided up front before any stage is submitted.
+   */
+  private def classifyJobShuffleKinds(finalRDD: RDD[_]): (Boolean, Boolean) = {
+    var hasPipelined = false
+    var hasRegular = false
+    traverseRDDGraph(finalRDD) { (rdd, enqueue) =>
+      rdd.dependencies.foreach { dep =>
+        dep match {
+          case _: PipelinedShuffleDependency[_, _, _] => hasPipelined = true
+          case _: ShuffleDependency[_, _, _] => hasRegular = true
+          case _ => // narrow dependency: not a boundary
+        }
+        // Descend through every edge (shuffle and narrow) so a pipelined 
boundary behind a regular
+        // one -- or vice versa -- anywhere in the graph is still detected. 
traverseRDDGraph dedups.
+        enqueue(dep.rdd)
+      }
+    }
+    (hasPipelined, hasRegular)
+  }
+
+  /**
+   * Reject a job that uses a pipelined shuffle in combination with a cluster 
feature that a
+   * pipelined group cannot support. Checked up front, before any stage is 
created, so a rejection
+   * leaves no partial scheduler state. Used by the result-job path 
(handleJobSubmitted); the
+   * map-stage-job path rejects a pipelined dependency outright (see 
handleMapStageSubmitted), which
+   * subsumes these. Returns true (and fails the job via `listener`) if 
rejected; false otherwise.
+   * The RDD-graph walk runs only when a relevant feature is enabled and is 
inert for jobs without a
+   * pipelined dependency.
+   *
+   * Rejected combinations:
+   *  - Speculation: a speculative copy of a producer would race a consumer 
already reading the
+   *    producer's partial output, with no commit barrier protecting the read.
+   *  - Dynamic allocation: a pipelined group is gang-scheduled (all member 
stages must run at
+   *    once), and the free-slot admission check measures currently-active 
executors. Under dynamic
+   *    allocation a job can start before any executor has spun up, so the 
group would be failed
+   *    with CONCURRENT_SCHEDULER_INSUFFICIENT_SLOT against a transient 0-slot 
snapshot even though
+   *    the cluster would soon have capacity. Barrier scheduling forbids the 
same combination
+   *    (checkBarrierStageWithDynamicAllocation); pipelined groups do likewise.
+   */
+  private def rejectUnsupportedPipelinedJob(
+      jobId: Int,
+      finalRDD: RDD[_],
+      listener: JobListener): Boolean = {
+    // Only walk the RDD graph if a relevant feature is on, then only once.
+    val speculationOn = sc.conf.get(config.SPECULATION_ENABLED)
+    val dynAllocOn = Utils.isDynamicAllocationEnabled(sc.conf)
+    if ((speculationOn || dynAllocOn) && 
rddGraphHasPipelinedDependency(finalRDD)) {
+      if (speculationOn) {
+        logWarning(log"Rejecting job ${MDC(JOB_ID, jobId)}: speculation is 
incompatible with " +
+          log"pipelined shuffle dependencies")
+        listener.jobFailed(new SparkException(
+          "Speculative execution is not supported for a job that uses a 
pipelined shuffle " +
+            s"dependency. Disable ${config.SPECULATION_ENABLED.key} for such 
jobs."))
+        return true
+      }
+      // dynAllocOn
+      logWarning(log"Rejecting job ${MDC(JOB_ID, jobId)}: dynamic allocation 
is incompatible " +
+        log"with pipelined shuffle dependencies")
+      listener.jobFailed(new SparkException(
+        "Dynamic allocation is not supported for a job that uses a pipelined 
shuffle dependency: " +
+          "a pipelined stage group must be co-scheduled against a known 
cluster size. Disable " +
+          s"${config.DYN_ALLOCATION_ENABLED.key} for such jobs."))
+      return true
+    }
+    false
+  }
+
   /** Invoke `.partitions` on the given RDD and all of its ancestors  */
   private def eagerlyComputePartitionsForRddAndAncestors(rdd: RDD[_]): Unit = {
     val startTime = System.nanoTime
@@ -1016,6 +1131,23 @@ private[spark] class DAGScheduler(
                   logDebug("Removing stage %d from failed 
set.".format(stageId))
                   failedStages -= stage
                 }
+                // Drop any pipelined-completion deferral keyed on this stage 
(as consumer), and
+                // remove it from other consumers' pending-producer sets (as 
producer), so no
+                // deferral outlives its job (e.g. on job abort before the 
producers finished).
+                // A consumer's buffered completion events hold TaskEnds that 
have not been emitted
+                // yet (the whole event is deferred until group completion). 
This teardown is a job
+                // failure / cancellation, so the buffered successes must NOT 
be applied as results
+                // -- but their TaskEnds must still be flushed, exactly as the 
producer-failed drop
+                // path does (see releaseDeferredPipelinedConsumers), or a 
listener tracking active
+                // tasks (e.g. AppStatusListener) would leak these tasks as 
perpetually running.
+                // Normally releaseDeferredPipelinedConsumers has already 
drained the deferral by
+                // the time cleanup runs (cancelRunningIndependentStages 
finishes each running
+                // producer first); this covers the case where it has not -- 
e.g. a producer left in
+                // resubmit limbo (finished but not available) that 
cancelRunningIndependentStages
+                // skips because it is neither running nor failed.
+                
dependentStageMap.remove(stage).foreach(_.delayedTaskCompletionEvents.foreach(
+                  postTaskEnd))
+                dependentStageMap.values.foreach(_.parents -= stage)
               }
               // data structures based on StageId
               stageIdToStage -= stageId
@@ -1346,6 +1478,227 @@ private[spark] class DAGScheduler(
     }
   }
 
+  /**
+   * Reconsider waiting stages that read `runningParent` through a pipelined 
edge, now that it has
+   * started running. A pipelined edge is non-sequencing, so such a consumer 
can be co-scheduled as
+   * soon as its producer is running -- it need not wait for the producer to 
finish. This is the
+   * "producer started" analog of `submitWaitingChildStages` (which fires on 
producer *completion*):
+   * without it, a pipelined consumer parked because its producer was not yet 
runnable (e.g. the
+   * producer sat behind a regular shuffle) would not be co-scheduled until 
the producer finished,
+   * losing the pipelining. Inert unless `runningParent` is the pipelined 
producer of a waiting
+   * stage.
+   */
+  private def submitWaitingPipelinedChildStages(runningParent: Stage): Unit = {
+    val pipelinedChildren = waitingStages.filter { child =>
+      child.parents.contains(runningParent) && 
isPipelinedProducer(runningParent)
+    }.toArray
+    // Remove them from waitingStages before resubmitting, or submitStage's 
`!waitingStages(stage)`
+    // guard would treat them as already-scheduled and no-op (mirrors 
submitWaitingChildStages).
+    waitingStages --= pipelinedChildren
+    for (child <- pipelinedChildren.sortBy(_.firstJobId)) {
+      logInfo(log"Reconsidering ${MDC(STAGE, child)} now that its pipelined 
producer " +
+        log"${MDC(STAGE, runningParent)} is running")
+      submitStage(child)
+    }
+  }
+
+  /**
+   * Whether `stage` produces its output through a 
[[PipelinedShuffleDependency]] -- i.e. it is a
+   * pipelined producer, whose consumers may run concurrently with it. Callers 
that need the
+   * producer/consumer relationship check `child.parents.contains(stage)` 
separately.
+   */
+  private def isPipelinedProducer(stage: Stage): Boolean = stage match {
+    case m: ShuffleMapStage => 
m.shuffleDep.isInstanceOf[PipelinedShuffleDependency[_, _, _]]
+    case _ => false
+  }
+
+  /**
+   * The total concurrent-task demand of an all-pipelined job, computed from 
the RDD graph BEFORE
+   * any stage is created (so a rejection based on it leaves no partial 
scheduler state, exactly as
+   * the barrier slot check and the speculation/DA reject do). Because a job 
is either all-regular
+   * or all-pipelined, an all-pipelined job's whole stage graph is one 
pipelined group; its members
+   * are the final result stage plus every pipelined producer. Each member's 
task count is its RDD's
+   * partition count (`rdd.partitions.length`), matching how 
`createShuffleMapStage` derives
+   * `numTasks`. `finalNumPartitions` is the result stage's task count (the 
number of partitions the
+   * job runs, which may be a subset of `finalRDD.partitions`).
+   *
+   * Count each producer once per SHUFFLE ID, matching what execution 
schedules: one stage is
+   * created per shuffle ID (`getOrCreateShuffleMapStage`), not per dependency 
edge. A fan-out or
+   * diamond graph references one `PipelinedShuffleDependency` from more than 
one consumer RDD, so
+   * charging its producer per edge would over-count and could reject a group 
whose stages actually
+   * fit. Dedup on `shuffleId`, not `pd.rdd`: two distinct dependencies can 
share a producer RDD yet
+   * carry distinct shuffle IDs and produce distinct stages, and both must be 
counted.
+   */
+  private def pipelinedJobConcurrentTaskDemand(finalRDD: RDD[_], 
finalNumPartitions: Int): Int = {
+    var demand = finalNumPartitions
+    val countedShuffleIds = new HashSet[Int]
+    traverseRDDGraph(finalRDD) { (rdd, enqueue) =>
+      rdd.dependencies.foreach {
+        case pd: PipelinedShuffleDependency[_, _, _] =>
+          if (countedShuffleIds.add(pd.shuffleId)) {
+            demand += pd.rdd.partitions.length
+          }
+        case _ => // regular/narrow deps do not occur in an all-pipelined 
job's group
+      }
+      rdd.dependencies.foreach(dep => enqueue(dep.rdd))
+    }
+    demand
+  }
+
+  /**
+   * Up-front gang admission for an all-pipelined job, checked BEFORE any 
stage exists. Because a
+   * job is either all-regular or all-pipelined (mixed jobs are already 
rejected), an all-pipelined
+   * job's whole stage graph is one pipelined group with no regular prefix, so 
its full demand is
+   * known up front and the group is ready to admit immediately. Checking here 
(rather than in
+   * `submitStage` once a producer is already running) is true all-or-nothing 
gang admission: the
+   * whole group is admitted, or the job is failed before any member runs, so 
a member is never left
+   * running while a sibling cannot get slots -- and, like the barrier slot 
check, a rejection
+   * leaves no partial scheduler state.
+   *
+   * Free-slot accounting: demand vs. total capacity (`maxNumConcurrentTasks` 
for the default
+   * profile -- a group is required to be single-profile) minus what OTHER 
work (other jobs) has
+   * outstanding -- running plus enqueued -- in that profile. Counting 
enqueued, not just running,
+   * tasks charges a busy neighbor's queued backlog against capacity too, so a 
group is not admitted
+   * against slots other work is already committed to. Fails the job via 
`listener` with no retry --
+   * a transient shortfall is the caller's to retry (e.g. the streaming batch 
loop reruns it).
+   * Returns true (job failed) if it does not fit.
+   *
+   * The likeness to barrier's slot check is only that both reject before any 
stage is created
+   * (leaving no partial state) and compute demand from the RDD graph. Retry 
behavior differs
+   * deliberately: barrier RE-POSTS the job and re-runs its check on a timer 
up to
+   * `spark.scheduler.barrier.maxConcurrentTasksCheck.maxFailures` times; 
pipelined admission is
+   * TERMINAL (one check, then fail), delegating transient-shortfall retry to 
the caller.
+   *
+   * The check can be turned off with 
`spark.scheduler.pipelinedGroup.slotCheck.enabled=false` (for
+   * deployments that admit capacity out-of-band, e.g. via a slot reservation).
+   */
+  private def rejectUnadmittablePipelinedGroup(
+      jobId: Int, finalRDD: RDD[_], partitions: Array[Int], listener: 
JobListener): Boolean = {
+    // Reject up-front (before any stage is created) two group members the 
admission model cannot
+    // support, walking the group's RDD graph once:
+    //  - A barrier member. A barrier stage exposes its output only after a 
global sync, which
+    //    contradicts a pipelined consumer reading the producer's output 
incrementally as it runs;
+    //    and its failure recovery resubmits the stage, which the group's 
atomic completion cannot
+    //    accommodate (a resubmitted producer would drop a co-scheduled 
consumer's buffered
+    //    completions). Reject it here rather than let it be co-scheduled.
+    //  - A member on a non-default resource profile. Admission below measures 
capacity and
+    //    occupancy against the default profile, but each stage derives its 
profile from its RDDs
+    //    (see createShuffleMapStage/createResultStage). A member carrying a 
non-default profile would be
+    //    admitted against the default profile's free slots yet run in a 
different, often smaller
+    //    pool -- and could then queue or deadlock there.
+    var offendingBarrier = false
+    var offendingRp: Option[ResourceProfile] = None
+    traverseRDDGraph(finalRDD) { (rdd, enqueue) =>
+      if (rdd.isBarrier()) {
+        offendingBarrier = true
+      }
+      val rp = rdd.getResourceProfile()
+      if (offendingRp.isEmpty && rp != null && rp.id != 
DEFAULT_RESOURCE_PROFILE_ID) {
+        offendingRp = Some(rp)
+      }
+      rdd.dependencies.foreach(dep => enqueue(dep.rdd))
+    }
+    if (offendingBarrier) {
+      logWarning(log"Rejecting job ${MDC(JOB_ID, jobId)}: a pipelined stage 
group contains a " +
+        log"barrier member")
+      listener.jobFailed(new SparkException(
+        "A pipelined shuffle job with a barrier stage in the pipelined group 
is not supported: a " +
+          "barrier stage exposes its output only after a global sync, which is 
incompatible with " +
+          "a pipelined consumer reading its output incrementally."))
+      return true
+    }
+    if (offendingRp.nonEmpty) {
+      logWarning(log"Rejecting job ${MDC(JOB_ID, jobId)}: a pipelined stage 
group member uses a " +
+        log"non-default resource profile")
+      listener.jobFailed(new SparkException(
+        "A pipelined shuffle job with a member on a non-default resource 
profile is not " +
+          s"supported (resource profile id ${offendingRp.get.id}): the whole 
pipelined stage " +
+          "group must run on the default resource profile."))
+      return true
+    }
+    if (!sc.conf.get(config.PIPELINED_GROUP_SLOT_CHECK_ENABLED)) {
+      return false
+    }
+    val rp = sc.resourceProfileManager.defaultResourceProfile
+    val demand = pipelinedJobConcurrentTaskDemand(finalRDD, partitions.length)
+    val totalSlots = maxConcurrentTasksForProfile(rp.id)
+    // No stage of this job exists yet, so it has no outstanding tasks of its 
own to exclude; only
+    // other concurrent jobs' outstanding demand is charged, 
resource-profile-scoped.
+    val outstandingForOthers = outstandingTasksForOtherWork(rp.id, Set.empty)
+    val freeSlots = math.max(0, totalSlots - outstandingForOthers)
+    if (demand > freeSlots) {
+      logWarning(log"Rejecting job ${MDC(JOB_ID, jobId)}: pipelined stage 
group needs " +
+        log"${MDC(NUM_TASKS, demand)} concurrent task slots but only " +
+        log"${MDC(NUM_SLOTS, freeSlots)} are free")
+      listener.jobFailed(new SparkException(
+        errorClass = "CONCURRENT_SCHEDULER_INSUFFICIENT_SLOT",
+        messageParameters = scala.collection.immutable.Map(
+          "numTasks" -> demand.toString, "numSlots" -> freeSlots.toString),
+        cause = null))
+      true
+    } else {
+      false
+    }
+  }
+
+  /**
+   * The cluster's total concurrent-task capacity for the given resource 
profile. Extracted as a
+   * seam so tests can control it without changing the cluster's core count. 
Production reads it
+   * from the scheduler backend, exactly as barrier's slot check does.
+   */
+  protected def maxConcurrentTasksForProfile(rpId: Int): Int = {
+    val rp = sc.resourceProfileManager.resourceProfileFromId(rpId)
+    sc.maxNumConcurrentTasks(rp)
+  }
+
+  /**
+   * Outstanding tasks (running plus enqueued) in the given resource profile, 
for work OTHER than
+   * the stages in `excludeStageIds`. Counting enqueued tasks, not just 
running ones, means a busy
+   * neighbor's queued backlog is charged against capacity too, so a group is 
not admitted against
+   * slots that other work is already committed to using. 
Resource-profile-scoped to match
+   * `maxConcurrentTasksForProfile`. Extracted as a seam so tests can control 
occupancy without
+   * launching real tasks; returns 0 for a non-TaskSchedulerImpl backend.
+   */
+  protected def outstandingTasksForOtherWork(rpId: Int, excludeStageIds: 
Set[Int]): Int =
+    taskScheduler match {
+      case impl: TaskSchedulerImpl =>
+        impl.outstandingTasksForOtherWorkInProfile(rpId, excludeStageIds)
+      case _ => 0
+    }
+
+  /**
+   * Whether `stage` is a member of a pipelined group -- i.e. it is connected 
to another stage by a
+   * [[PipelinedShuffleDependency]], either as the producer (it writes such a 
shuffle) or as a
+   * consumer (one of its direct parent shuffle dependencies is pipelined). 
Members run
+   * concurrently over a transient shuffle that cannot be re-read in 
isolation, so a member's task
+   * failure must fail the whole group rather than resubmit one stage; the 
task scheduler keys its
+   * fail-fast behavior off this (via TaskSet.isPipelined). False for any 
stage in a job with no
+   * pipelined dependency.
+   *
+   * NOTE: this is deliberately defined here alongside the other 
group-topology helper
+   * (`isPipelinedProducer`). Its call sites are the group-atomic failure 
handling: tagging the
+   * member's `TaskSet.isPipelined` at submission and routing a member 
FetchFailed to a whole-group
+   * abort.
+   */
+  private def isPipelinedGroupMember(stage: Stage): Boolean = {
+    if (isPipelinedProducer(stage)) {
+      return true
+    }
+    // Consumer check: does this stage read through a 
PipelinedShuffleDependency at one of its
+    // shuffle boundaries? Walk the stage's own RDD graph (descending narrow 
deps, stopping at every
+    // shuffle boundary) -- the same edges that define the stage -- and look 
for a pipelined one.
+    !traverseRDDGraphUntil(stage.rdd) { (rdd, enqueue) =>
+      val hasPipelinedBoundary = rdd.dependencies.exists {
+        case _: PipelinedShuffleDependency[_, _, _] => true
+        case _: ShuffleDependency[_, _, _] => false // regular shuffle 
boundary: do not descend
+        case narrowDep =>
+          enqueue(narrowDep.rdd)
+          false
+      }
+      !hasPipelinedBoundary // keep walking until a pipelined boundary is found
+    }
+  }
+
   /** Finds the earliest-created active job that needs the stage */
   // TODO: Probably should actually find among the active jobs that need this
   // stage the one with the highest priority (highest-priority pool, earliest 
created).
@@ -1489,6 +1842,40 @@ private[spark] class DAGScheduler(
       return
     }
 
+    // A job that uses a pipelined shuffle co-schedules its producer and 
consumer stages, which is
+    // incompatible with speculation (a speculative producer copy would race a 
consumer reading its
+    // partial output) and with dynamic allocation (the gang-scheduling slot 
check would fail
+    // against a not-yet-warmed cluster). Reject such a job up front -- before 
any stages are
+    // created, so no partial scheduler state is left behind. (Inert for jobs 
without any pipelined
+    // dependency.)
+    if (rejectUnsupportedPipelinedJob(jobId, finalRDD, listener)) {
+      return
+    }
+
+    // A job must be either ALL-regular or ALL-pipelined, not a mix: a job 
whose shuffle graph
+    // combines a pipelined shuffle with a regular one is rejected up front 
(before any stage is
+    // created). Restricting to a single kind makes an all-pipelined job's 
whole stage graph one
+    // pipelined group with no regular prefix, so gang admission is decided up 
front (see the slot
+    // check below) rather than mid-DAG once a producer is already running.
+    val (hasPipelined, hasRegular) = classifyJobShuffleKinds(finalRDD)
+    if (hasPipelined && hasRegular) {
+      logWarning(log"Rejecting job ${MDC(JOB_ID, jobId)}: a job mixing a 
pipelined shuffle with " +
+        log"a regular shuffle is not supported")
+      listener.jobFailed(new SparkException(
+        "A job that mixes a pipelined shuffle dependency with a regular 
shuffle dependency is " +
+          "not supported: a job must be either all-regular or all-pipelined."))
+      return
+    }
+
+    // Gang admission for an all-pipelined job: the whole stage graph is one 
pipelined group, so
+    // check up front (before any stage is created) that the cluster can run 
the entire group
+    // concurrently. If it cannot fit, fail the job now -- no partial 
scheduler state, and no member
+    // ever left running while a sibling waits on slots (true all-or-nothing 
gang admission). Inert
+    // for a regular job (no pipelined dependency).
+    if (hasPipelined && rejectUnadmittablePipelinedGroup(jobId, finalRDD, 
partitions, listener)) {
+      return
+    }
+
     var finalStage: ResultStage = null
     try {
       // New stage creation may throw an exception if, for example, jobs are 
run on a
@@ -1564,6 +1951,24 @@ private[spark] class DAGScheduler(
       listener: JobListener,
       artifacts: JobArtifactSet,
       properties: Properties): Unit = {
+    // A map-stage job (SparkContext.submitMapStage, e.g. AQE's 
ShuffleExchangeExec) materializes a
+    // shuffle to produce its map-output statistics. A 
PipelinedShuffleDependency cannot serve that:
+    // it is transient with no durable, addressable map output, so it 
registers no map outputs and
+    // getStatistics would return all-zero stats (misleading the map-stage/AQE 
consumer); and its
+    // producer/consumer co-scheduling makes speculation unsafe. Reject a 
pipelined
+    // dependency submitted as a map-stage job outright, up front, before any 
stage is created so no
+    // partial scheduler state is left behind. Inert for a regular 
ShuffleDependency. (The
+    // result-job path, handleJobSubmitted, rejects the speculation and 
dynamic-allocation cases via
+    // rejectUnsupportedPipelinedJob, but a pipelined dependency is otherwise 
a legitimate internal
+    // edge there.)
+    if (dependency.isInstanceOf[PipelinedShuffleDependency[_, _, _]]) {
+      logWarning(log"Rejecting map-stage job ${MDC(JOB_ID, jobId)}: a 
pipelined shuffle dependency " +
+        log"cannot be materialized as a map-stage job")
+      listener.jobFailed(new SparkException(
+        "A pipelined shuffle dependency cannot be submitted as a map-stage 
job: it has no durable " +
+          "map output to produce statistics from. This is not supported."))
+      return
+    }
     // Submitting this map stage might still require the creation of some 
parent stages, so make
     // sure that happens.
     var finalStage: ShuffleMapStage = null
@@ -1630,11 +2035,63 @@ private[spark] class DAGScheduler(
             logInfo(log"Submitting ${MDC(STAGE, stage)} (${MDC(RDD_ID, 
stage.rdd)}), " +
                     log"which has no missing parents")
             submitMissingTasks(stage, jobId.get)
+            // If this stage is the pipelined producer of a waiting consumer, 
co-schedule it now.
+            submitWaitingPipelinedChildStages(stage)
           } else {
             for (parent <- missing) {
               submitStage(parent)
             }
-            waitingStages += stage
+
+            // Submitting a parent can abort the job during this recursion 
(e.g. a parent that has
+            // exhausted its stage attempts hits abortStage, which cleans up 
all of the job's stages
+            // including this one and fails the job). If that happened, 
`stage` is no longer
+            // registered; do not proceed to co-schedule or re-park it 
(re-parking would re-insert a
+            // job-less stage into waitingStages -- a scheduler-state leak). 
Inert for a
+            // non-aborting submit. (Pipelined group admission is decided up 
front in
+            // handleJobSubmitted, so it cannot abort the group from within 
this recursion.)
+            if (!stageIdToStage.contains(stage.id)) {
+              logInfo(log"${MDC(STAGE, stage)} was removed during parent 
submission (its job was " +
+                log"aborted); not co-scheduling or re-parking it")
+              return
+            }
+
+            // A missing parent reached through a PipelinedShuffleDependency 
("pipelined parent")
+            // is incrementally readable: this stage may run before that 
parent materializes, so
+            // the two are co-scheduled. `missing` already holds the direct 
parent shuffle-map
+            // stages (from getMissingParentStages), so classify them by their 
shuffle dependency
+            // type -- no extra graph walk. For a job with no pipelined 
dependency, pipelinedMissing
+            // is empty and this stage simply parks in waitingStages, exactly 
as before.
+            val (pipelinedMissing, regularMissing) = 
missing.partition(isPipelinedProducer)
+            // Co-schedule only if EVERY missing parent is pipelined AND each 
is actually running
+            // now. submitStage above may have parked a pipelined parent in 
waitingStages (e.g. it
+            // has its own regular missing parent); running this stage against 
a not-yet-running
+            // producer would strand it. If any parent is regular or not yet 
runnable, park this
+            // stage. It is then resubmitted and co-scheduled once its parents 
become runnable --
+            // via submitWaitingChildStages when a regular parent completes, 
or via
+            // submitWaitingPipelinedChildStages when a pipelined parent 
starts running.
+            val allPipelinedParentsRunning =
+              pipelinedMissing.nonEmpty && 
pipelinedMissing.forall(runningStages.contains)
+
+            if (regularMissing.isEmpty && allPipelinedParentsRunning) {
+              // The whole group's capacity was already admitted up front 
(handleJobSubmitted ->
+              // rejectUnadmittablePipelinedGroup) before any member was 
submitted, so the group is
+              // known to fit; just co-schedule this consumer with its running 
producer(s). No slot
+              // check here -- that would re-measure capacity against a 
mid-flight snapshot and is
+              // unnecessary once admission is decided up front (gang 
admission).
+              logInfo(log"Submitting ${MDC(STAGE, stage)} concurrently with 
its running " +
+                log"pipelined producer(s) ${MDC(MISSING_PARENT_STAGES, 
pipelinedMissing)}")
+              // Record that this stage is co-scheduled with still-running 
pipelined producers,
+              // so its successful completions are deferred until those 
producers finish.
+              val deferral =
+                dependentStageMap.getOrElseUpdate(stage, DependentStageInfo())
+              deferral.parents ++= pipelinedMissing
+              submitMissingTasks(stage, jobId.get)
+              // This stage is now running; if it is itself the pipelined 
producer of a waiting
+              // consumer, co-schedule that consumer too.
+              submitWaitingPipelinedChildStages(stage)
+            } else {
+              waitingStages += stage
+            }
           }
         }
       }
@@ -2488,6 +2945,34 @@ private[spark] class DAGScheduler(
 
     val stage = stageIdToStage(task.stageId)
 
+    // Group-observable completion (spec S5): if this stage is a pipelined 
consumer co-scheduled
+    // with a still-running pipelined producer, defer its *successful* 
completion in full until the
+    // producer(s) finish. Buffer the whole CompletionEvent and return before 
ANY of its side
+    // effects run. Buffering the whole event DEFERS all three of those 
effects -- the accumulator
+    // update, the task-end listener event, and stage/job completion -- until 
the event is replayed;
+    // it is NOT just the stage/job-completion bookkeeping that waits. (A 
listener-visible
+    // consequence: a consumer task that finishes ahead of its producer emits 
no SparkListenerTaskEnd
+    // until the producer finishes, so the Spark UI reports that 
already-succeeded task as still
+    // running for the producer's remaining lifetime.) Else a consumer 
finishing ahead of its
+    // producer would advance job completion and cancel the still-running 
producer (via
+    // cancelRunningIndependentStages), or expose its output early.
+    // Deferring the entire event is what makes the side effects run exactly 
ONCE, at replay:
+    // releaseDeferredPipelinedConsumers re-posts the buffered event once the 
last producer completes
+    // (or drops it if a producer fails, S6), and it then re-enters here and 
runs the side effects
+    // normally. This deferral check must therefore precede updateAccumulators 
and postTaskEnd. Inert
+    // for jobs with no pipelined dependency (the map is empty), so the 
regular path is unchanged.
+    if (event.reason == Success) {
+      dependentStageMap.get(stage) match {
+        case Some(deferral) if deferral.parents.nonEmpty =>
+          logInfo(log"Deferring completion of task ${MDC(TASK_ID, 
event.taskInfo.taskId)} in " +
+            log"pipelined consumer ${MDC(STAGE, stage)} until its producer(s) 
" +
+            log"${MDC(MISSING_PARENT_STAGES, deferral.parents.toSeq)} finish")
+          deferral.delayedTaskCompletionEvents += event
+          return
+        case _ =>
+      }
+    }
+
     // Make sure the task's accumulators are updated before any other 
processing happens, so that
     // we can post a task end event before any jobs or stages are updated. The 
accumulators are
     // only updated in certain cases.
@@ -3513,8 +3998,12 @@ private[spark] class DAGScheduler(
 
   /**
    * Marks a stage as finished and removes it from the list of running stages.
+   *
+   * `private[scheduler]` (not `private`) so tests can drive the 
pipelined-consumer release/retain
+   * decision below directly, including the retained branch (a 
not-yet-available pipelined producer
+   * about to resubmit), which is otherwise brittle to reach through the mock 
backend.
    */
-  private def markStageAsFinished(
+  private[scheduler] def markStageAsFinished(
       stage: Stage,
       errorMessage: Option[String] = None,
       willRetry: Boolean = false): Unit = {
@@ -3543,6 +4032,81 @@ private[spark] class DAGScheduler(
     }
     listenerBus.post(SparkListenerStageCompleted(stage.latestInfo))
     runningStages -= stage
+
+    // Release any pipelined consumers whose completion was deferred while 
this stage (a pipelined
+    // producer) was running. Only act when the producer's outcome is final:
+    //  - willRetry: the stage is being retried, not finished -- leave 
consumers deferred.
+    //  - a ShuffleMapStage that finished "successfully" (no errorMessage) but 
is NOT yet available
+    //    (e.g. a bogus-epoch task left an output missing) is about to be 
resubmitted by
+    //    processShuffleMapStageCompletion -- it is not truly done, so do NOT 
replay its consumers
+    //    against soon-to-be-recomputed output; the release happens when its 
reattempt completes and
+    //    it becomes available.
+    // Otherwise: producerFailed = the stage failed (errorMessage set) -> drop 
the consumers'
+    // buffered successes; producer succeeded -> replay them.
+    if (!willRetry) {
+      val producerFailed = errorMessage.isDefined
+      val producerAboutToResubmit = stage match {
+        case m: ShuffleMapStage => !producerFailed && !m.isAvailable
+        case _ => false
+      }
+      if (!producerAboutToResubmit) {
+        releaseDeferredPipelinedConsumers(stage, producerFailed = 
producerFailed)
+      }
+    }
+  }
+
+  /**
+   * Called when a stage finishes. If `finishedStage` is a pipelined producer 
that some co-scheduled
+   * consumer was deferred on, resolve that consumer's deferral:
+   *  - Producer FAILED: drop the consumer's buffered completions immediately, 
regardless of whether
+   *    it still has other pending producers. A pipelined group is failed as a 
unit (S6), so once a
+   *    producer fails the consumer's buffered successes must not be applied 
-- and dropping now,
+   *    rather than remembering the failure until the last producer finishes, 
means no failure state
+   *    outlives this call (a surviving producer that later finishes just 
finds the consumer gone).
+   *  - Producer SUCCEEDED: remove it from the consumer's pending-producer 
set, and replay the
+   *    buffered completions only once the LAST producer has succeeded (the 
set becomes empty).
+   * Inert unless `finishedStage` is a tracked producer. Recovery from a drop, 
if any, is a caller
+   * rerun of the whole group (a new job, S6) -- the scheduler does not rerun 
the group in place.
+   */
+  private def releaseDeferredPipelinedConsumers(
+      finishedStage: Stage, producerFailed: Boolean): Unit = {
+    if (dependentStageMap.isEmpty) {
+      return
+    }
+    // Consumers waiting on this producer.
+    val released = dependentStageMap.filter {
+      case (_, d) => d.parents.contains(finishedStage)
+    }.keys.toArray
+    for (consumer <- released) {
+      val deferral = dependentStageMap(consumer)
+      deferral.parents -= finishedStage
+      if (producerFailed) {
+        // Drop immediately: the group is being failed as a unit, so this 
consumer's buffered
+        // successes must not be applied no matter what its other producers 
do. Removing the entry
+        // now leaves no deferral state to go stale for a later 
re-co-scheduling of this stage.
+        // Treating producerFailed as terminal is sound because every failure 
path that reaches a
+        // pipelined member's markStageAsFinished(errorMessage) here is 
terminal for the group: the
+        // one base path that fails a stage and then resubmits it (a barrier 
task failure) cannot
+        // occur, because a barrier member is rejected up front 
(rejectUnadmittablePipelinedGroup),
+        // so a pipelined group never contains one.
+        dependentStageMap -= consumer
+        val events = deferral.delayedTaskCompletionEvents.toList
+        logInfo(log"Dropping ${MDC(NUM_EVENTS, events.size.toLong)} deferred 
completion(s) for " +
+          log"pipelined consumer ${MDC(STAGE, consumer)} because a producer 
failed; failing the " +
+          log"group")
+        // The buffered tasks genuinely succeeded, so still emit their TaskEnd 
events -- otherwise
+        // listeners that track active tasks (e.g. AppStatusListener) would 
believe these tasks are
+        // still running. We deliberately do NOT run the stage/job completion 
bookkeeping: the group
+        // is being failed as a unit, so the consumer's results must not be 
applied.
+        events.foreach(postTaskEnd)
+      } else if (deferral.parents.isEmpty) {
+        dependentStageMap -= consumer
+        val events = deferral.delayedTaskCompletionEvents.toList
+        logInfo(log"Replaying ${MDC(NUM_EVENTS, events.size.toLong)} deferred 
completion(s) " +
+          log"for pipelined consumer ${MDC(STAGE, consumer)} now that its 
producers finished")
+        events.foreach(eventProcessLoop.post)
+      }
+    }
   }
 
   /**
diff --git 
a/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala 
b/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
index d79d69cf65b7..e5fdfc68701f 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
@@ -162,6 +162,37 @@ private[spark] class TaskSchedulerImpl(
     executorIdToRunningTaskIds.toMap.transform((_, v) => v.size)
   }
 
+  /**
+   * The number of outstanding tasks for the given resource profile that 
belong to work OTHER than
+   * the stages in `excludeStageIds`. "Outstanding" is the not-yet-completed 
demand of each task set
+   * -- running plus enqueued (`numTasks - tasksSuccessful`) -- not just the 
tasks actively running,
+   * so a neighbor's queued backlog is charged against capacity too. Used by 
the pipelined-group
+   * slot admission check (see DAGScheduler): it compares the group's demand 
against the slots left
+   * free after accounting for everything else in the SAME resource profile, 
so it excludes the
+   * group's own members (whose tasks would otherwise be charged against the 
group's own admission).
+   *
+   * Computed from the TaskSetManagers so it is resource-profile-scoped, skips 
zombie (superseded)
+   * attempts so a retried stage is not double-counted, and is taken under a 
single lock so the
+   * count is one consistent snapshot (both the per-profile total and the 
excluded members are read
+   * together).
+   */
+  private[scheduler] def outstandingTasksForOtherWorkInProfile(
+      resourceProfileId: Int, excludeStageIds: Set[Int]): Int = synchronized {
+    taskSetsByStageIdAndAttempt.iterator.flatMap { case (stageId, attempts) =>
+      if (excludeStageIds.contains(stageId)) {
+        Iterator.empty
+      } else {
+        attempts.valuesIterator
+          // Skip zombie attempts (superseded by a retry/kill): a stage can 
have both a zombie and a
+          // live attempt in this map at once, and the live attempt already 
re-runs the zombie's
+          // outstanding tasks -- counting both would double-count that 
stage's demand. Matches the
+          // !isZombie filtering used elsewhere on this map.
+          .filter(tsm => !tsm.isZombie && tsm.taskSet.resourceProfileId == 
resourceProfileId)
+          .map(tsm => math.max(0, tsm.numTasks - tsm.tasksSuccessful))
+      }
+    }.sum
+  }
+
   // The set of executors we have on each host; this is used to compute 
hostsAlive, which
   // in turn is used to decide when we can attain data locality on a given host
   protected val hostToExecutors = new HashMap[String, HashSet[String]]
diff --git 
a/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala 
b/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
index fd6f32919fac..f43354c80865 100644
--- a/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
@@ -380,6 +380,20 @@ class DAGSchedulerSuite extends SparkFunSuite with 
TempLocalSparkContext with Ti
       shuffleMergeRegister: Boolean = true
   ) extends DAGScheduler(
       sc, taskScheduler, listenerBus, mapOutputTracker, blockManagerMaster, 
env, clock) {
+
+    // Capacity reported to the pipelined-group slot check. Defaults high so 
scheduling-logic tests
+    // are not gated by the real (local[2]) backend capacity; a test can lower 
it to exercise the
+    // fail-fast path.
+    @volatile var maxConcurrentTasksForTest: Int = 1000
+    override protected def maxConcurrentTasksForProfile(rpId: Int): Int = 
maxConcurrentTasksForTest
+
+    // Seam for the free-slot admission check: outstanding (running + 
enqueued) task
+    // demand of OTHER work. Default 0 so existing tests see a fully-free 
cluster; a test can set it
+    // to model a busy neighbor.
+    @volatile var outstandingTasksForOtherWorkForTest: (Int, Set[Int]) => Int 
= (_, _) => 0
+    override protected def outstandingTasksForOtherWork(rpId: Int, 
excludeStageIds: Set[Int]): Int =
+      outstandingTasksForOtherWorkForTest(rpId, excludeStageIds)
+
     /**
      * Schedules shuffle merge finalize.
      */
@@ -6309,6 +6323,7 @@ class DAGSchedulerSuite extends SparkFunSuite with 
TempLocalSparkContext with Ti
     assert(scheduler.runningStages.isEmpty)
     assert(scheduler.shuffleIdToMapStage.isEmpty)
     assert(scheduler.waitingStages.isEmpty)
+    assert(scheduler.dependentStageMap.isEmpty)
     assert(scheduler.outputCommitCoordinator.isEmpty)
   }
 
@@ -6341,6 +6356,869 @@ class DAGSchedulerSuite extends SparkFunSuite with 
TempLocalSparkContext with Ti
     }
     CompletionEvent(task, reason, result, accumUpdates ++ extraAccumUpdates, 
metricPeaks, taskInfo)
   }
+
+  // 
==========================================================================================
+  // Pipelined shuffle dependency: group formation + concurrent submission
+  // 
==========================================================================================
+
+  test("pipelined shuffle: consumer stage is submitted concurrently with its 
producer") {
+    // producer (shuffle map) --[pipelined]--> consumer (result)
+    val producerRdd = new MyRDD(sc, 2, Nil)
+    val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+    val consumerRdd = new MyRDD(sc, 2, List(pipelinedDep), tracker = 
mapOutputTracker)
+    submit(consumerRdd, Array(0, 1))
+
+    // Both the producer's and the consumer's task sets must be submitted 
before the producer
+    // completes: the pipelined edge is non-sequencing, so the consumer does 
not wait.
+    assert(taskSets.size === 2,
+      s"expected producer and consumer submitted concurrently, got 
${taskSets.size} task sets")
+    val producerStageId = taskSets.head.stageId
+    val consumerStageId = taskSets(1).stageId
+    assert(producerStageId != consumerStageId)
+    // The consumer is actually running (not parked in waitingStages), 
co-resident with the
+    // producer -- both are in runningStages.
+    assert(!scheduler.waitingStages.exists(_.rdd eq consumerRdd),
+      "consumer should not be waiting; it is co-scheduled with its pipelined 
producer")
+    assert(scheduler.runningStages.exists(_.rdd eq consumerRdd))
+    assert(scheduler.runningStages.exists(_.rdd eq producerRdd))
+
+    // Now complete the producer, then the consumer, and the job finishes.
+    completeShuffleMapStageSuccessfully(producerStageId, 0, 2)
+    complete(taskSets(1), Seq((Success, 42), (Success, 43)))
+    assert(results === Map(0 -> 42, 1 -> 43))
+    assertDataStructuresEmpty()
+  }
+
+  test("regular shuffle still waits (pipelined change is inert without a 
pipelined dependency)") {
+    // Identical shape but a REGULAR shuffle dependency: the consumer must NOT 
be submitted until
+    // the producer materializes -- verifying the concurrent-submission path 
is inert here.
+    val producerRdd = new MyRDD(sc, 2, Nil)
+    val regularDep = new ShuffleDependency(producerRdd, new HashPartitioner(2))
+    val consumerRdd = new MyRDD(sc, 2, List(regularDep), tracker = 
mapOutputTracker)
+    submit(consumerRdd, Array(0, 1))
+
+    // Only the producer is submitted initially; the consumer waits.
+    assert(taskSets.size === 1, s"expected only the producer submitted, got 
${taskSets.size}")
+    val producerStageId = taskSets.head.stageId
+    completeShuffleMapStageSuccessfully(producerStageId, 0, 2)
+    // Now the consumer is submitted.
+    assert(taskSets.size === 2)
+    complete(taskSets(1), Seq((Success, 42), (Success, 43)))
+    assert(results === Map(0 -> 42, 1 -> 43))
+    assertDataStructuresEmpty()
+  }
+
+  test("pipelined shuffle: a job mixing a pipelined and a regular shuffle is 
rejected up front") {
+    // A job must be either all-regular or all-pipelined, not a mix. A consumer
+    // depending on BOTH a pipelined producer AND a regular producer is a 
mixed job and must be
+    // rejected up front (before any stage is submitted), leaving no scheduler 
state behind.
+    val pipelinedProducerRdd = new MyRDD(sc, 2, Nil)
+    val pipelinedDep = new PipelinedShuffleDependency(pipelinedProducerRdd, 
new HashPartitioner(2))
+    val regularProducerRdd = new MyRDD(sc, 2, Nil)
+    val regularDep = new ShuffleDependency(regularProducerRdd, new 
HashPartitioner(2))
+    val consumerRdd =
+      new MyRDD(sc, 2, List(pipelinedDep, regularDep), tracker = 
mapOutputTracker)
+    val failure = new java.util.concurrent.atomic.AtomicReference[Exception]()
+    val failListener = new JobListener {
+      override def taskSucceeded(index: Int, result: Any): Unit = {}
+      override def jobFailed(exception: Exception): Unit = 
failure.set(exception)
+    }
+    submit(consumerRdd, Array(0, 1), listener = failListener)
+
+    assert(failure.get() != null, "a mixed pipelined+regular job must fail")
+    assert(failure.get().getMessage.contains("all-regular or all-pipelined"),
+      s"expected a mixed-job rejection, got: ${failure.get().getMessage}")
+    assert(taskSets.isEmpty, "no stage should be submitted for a rejected 
mixed job")
+    assertDataStructuresEmpty()
+  }
+
+  test("pipelined shuffle: deep chain A->B->C is submitted fully 
concurrently") {
+    // A --pipelined--> B --pipelined--> C : all three co-scheduled (each edge 
is non-sequencing).
+    val rddA = new MyRDD(sc, 2, Nil)
+    val psdAB = new PipelinedShuffleDependency(rddA, new HashPartitioner(2))
+    val rddB = new MyRDD(sc, 2, List(psdAB), tracker = mapOutputTracker)
+    val psdBC = new PipelinedShuffleDependency(rddB, new HashPartitioner(2))
+    val rddC = new MyRDD(sc, 2, List(psdBC), tracker = mapOutputTracker)
+    submit(rddC, Array(0, 1))
+
+    // All three stages must be running concurrently; none parked.
+    assert(taskSets.size === 3, s"expected A, B, C co-scheduled, got 
${taskSets.size}")
+    assert(scheduler.waitingStages.isEmpty, "no stage should be waiting in an 
all-pipelined chain")
+    Seq(rddA, rddB, rddC).foreach { rdd =>
+      assert(scheduler.runningStages.exists(_.rdd eq rdd), s"stage for $rdd 
should be running")
+    }
+
+    // Drain in dependency order: A, then B, then the result stage C.
+    val idA = taskSets.find(ts => scheduler.stageIdToStage(ts.stageId).rdd eq 
rddA).get.stageId
+    completeShuffleMapStageSuccessfully(idA, 0, 2)
+    val idB = taskSets.find(ts => scheduler.stageIdToStage(ts.stageId).rdd eq 
rddB).get.stageId
+    completeShuffleMapStageSuccessfully(idB, 0, 2)
+    val tsC = taskSets.find(ts => scheduler.stageIdToStage(ts.stageId).rdd eq 
rddC).get
+    complete(tsC, Seq((Success, 42), (Success, 43)))
+    assert(results === Map(0 -> 42, 1 -> 43))
+    assertDataStructuresEmpty()
+  }
+
+  test("regular shuffle job with speculation enabled is NOT rejected 
(rejection path is inert)") {
+    // The speculation fail-fast must apply only to jobs with a pipelined 
dependency; a plain
+    // regular-shuffle job with speculation on runs normally.
+    sc.conf.set(config.SPECULATION_ENABLED, true)
+    try {
+      val producerRdd = new MyRDD(sc, 2, Nil)
+      val regularDep = new ShuffleDependency(producerRdd, new 
HashPartitioner(2))
+      val consumerRdd = new MyRDD(sc, 2, List(regularDep), tracker = 
mapOutputTracker)
+      submit(consumerRdd, Array(0, 1))
+      // Not rejected: the producer is submitted and the job proceeds normally.
+      assert(taskSets.nonEmpty, "regular-shuffle job must not be rejected 
under speculation")
+      completeShuffleMapStageSuccessfully(taskSets.head.stageId, 0, 2)
+      complete(taskSets(1), Seq((Success, 42), (Success, 43)))
+      assert(results === Map(0 -> 42, 1 -> 43))
+      assertDataStructuresEmpty()
+    } finally {
+      sc.conf.set(config.SPECULATION_ENABLED, false)
+    }
+  }
+
+  test("pipelined shuffle: speculation is rejected for a job with a pipelined 
dependency") {
+    // Speculation races a producer copy against a consumer reading its 
partial output; reject.
+    // The scheduler reads sc.conf live, so toggling it here takes effect for 
the submit below.
+    sc.conf.set(config.SPECULATION_ENABLED, true)
+    try {
+      val producerRdd = new MyRDD(sc, 2, Nil)
+      val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+      val consumerRdd = new MyRDD(sc, 2, List(pipelinedDep), tracker = 
mapOutputTracker)
+      val failure = new 
java.util.concurrent.atomic.AtomicReference[Exception]()
+      val failListener = new JobListener {
+        override def taskSucceeded(index: Int, result: Any): Unit = {}
+        override def jobFailed(exception: Exception): Unit = 
failure.set(exception)
+      }
+      submit(consumerRdd, Array(0, 1), listener = failListener)
+      assert(failure.get() != null,
+        "job with pipelined dependency + speculation should be rejected")
+      assert(failure.get().getMessage.contains("Speculative execution is not 
supported"))
+      // No task sets were submitted for the rejected job.
+      assert(taskSets.isEmpty)
+      assertDataStructuresEmpty()
+    } finally {
+      sc.conf.set(config.SPECULATION_ENABLED, false)
+    }
+  }
+
+  test("pipelined shuffle: dynamic allocation is rejected for a job with a 
pipelined dependency") {
+    // A pipelined group is gang-scheduled and its slot check measures 
currently-active executors;
+    // under dynamic allocation a job can start before any executor is up, so 
the check would fail
+    // the group against a transient 0-slot snapshot. Reject the combo 
(barrier does the same).
+    // DYN_ALLOCATION_TESTING lets isDynamicAllocationEnabled report true 
under a local master.
+    sc.conf.set(config.DYN_ALLOCATION_ENABLED, true)
+    sc.conf.set(config.DYN_ALLOCATION_TESTING, true)
+    try {
+      val producerRdd = new MyRDD(sc, 2, Nil)
+      val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+      val consumerRdd = new MyRDD(sc, 2, List(pipelinedDep), tracker = 
mapOutputTracker)
+      val failure = new 
java.util.concurrent.atomic.AtomicReference[Exception]()
+      val failListener = new JobListener {
+        override def taskSucceeded(index: Int, result: Any): Unit = {}
+        override def jobFailed(exception: Exception): Unit = 
failure.set(exception)
+      }
+      submit(consumerRdd, Array(0, 1), listener = failListener)
+      assert(failure.get() != null,
+        "job with pipelined dependency + dynamic allocation should be 
rejected")
+      assert(failure.get().getMessage.contains("Dynamic allocation is not 
supported"))
+      assert(taskSets.isEmpty)
+      assertDataStructuresEmpty()
+    } finally {
+      sc.conf.set(config.DYN_ALLOCATION_ENABLED, false)
+      sc.conf.set(config.DYN_ALLOCATION_TESTING, false)
+    }
+  }
+
+  test("pipelined shuffle: a member on a non-default resource profile is 
rejected up front") {
+    // Admission measures capacity/occupancy against the DEFAULT resource 
profile, but a stage
+    // derives its profile from its RDDs. A pipelined group member on a custom 
profile would be
+    // admitted against the default pool's free slots yet run in a different 
(often smaller) pool
+    // and could deadlock there. Reject such a job before any stage is created.
+    // Ensure the default profile exists (id 0) before building a custom one, 
so the custom profile
+    // gets a distinct, non-default id regardless of test-execution order 
(profile ids come from a
+    // process-wide counter, and the default profile occupies id 0).
+    sc.resourceProfileManager.defaultResourceProfile
+    val ereqs = new ExecutorResourceRequests().cores(4)
+    val treqs = new TaskResourceRequests().cpus(2)
+    val customRp = new 
ResourceProfileBuilder().require(ereqs).require(treqs).build()
+    val producerRdd = new MyRDD(sc, 2, Nil)
+    val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+    val consumerRdd = new MyRDD(sc, 2, List(pipelinedDep), tracker = 
mapOutputTracker)
+      .withResources(customRp)
+    // Sanity: the consumer really carries a non-default profile (otherwise 
the test is vacuous).
+    assert(consumerRdd.getResourceProfile() != null &&
+      consumerRdd.getResourceProfile().id != 
ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID,
+      s"test setup: expected a non-default profile, got id " +
+        s"${Option(consumerRdd.getResourceProfile()).map(_.id)}")
+    val failure = new java.util.concurrent.atomic.AtomicReference[Exception]()
+    val failListener = new JobListener {
+      override def taskSucceeded(index: Int, result: Any): Unit = {}
+      override def jobFailed(exception: Exception): Unit = 
failure.set(exception)
+    }
+    submit(consumerRdd, Array(0, 1), listener = failListener)
+    assert(failure.get() != null,
+      "a pipelined job with a non-default-resource-profile member should be 
rejected")
+    assert(failure.get().getMessage.contains("non-default resource profile"))
+    assert(taskSets.isEmpty, "no stage should be created for a rejected job")
+    assertDataStructuresEmpty()
+  }
+
+  test("pipelined shuffle: a barrier group member is rejected up front") {
+    // A barrier stage exposes its output only after a global sync, which is 
incompatible with a
+    // pipelined consumer reading the producer's output incrementally. It is 
also a recovery hazard:
+    // a barrier task failure fails the stage and RESUBMITS it 
(markStageAsFinished without
+    // willRetry), and if such a producer were co-scheduled, the resubmit 
would drop a fan-in
+    // consumer's buffered completions and the job could never complete. 
Reject any job whose
+    // pipelined group contains a barrier member before any stage is created, 
so that path is
+    // unreachable.
+    val producerRdd = new MyRDD(sc, 2, Nil).barrier().mapPartitions(iter => 
iter)
+    assert(producerRdd.isBarrier(), "test setup: the producer must be a 
barrier RDD")
+    val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+    val consumerRdd = new MyRDD(sc, 2, List(pipelinedDep), tracker = 
mapOutputTracker)
+    val failure = new java.util.concurrent.atomic.AtomicReference[Exception]()
+    val failListener = new JobListener {
+      override def taskSucceeded(index: Int, result: Any): Unit = {}
+      override def jobFailed(exception: Exception): Unit = 
failure.set(exception)
+    }
+    submit(consumerRdd, Array(0, 1), listener = failListener)
+    assert(failure.get() != null,
+      "a pipelined job with a barrier group member should be rejected")
+    assert(failure.get().getMessage.contains("barrier"))
+    assert(taskSets.isEmpty, "no stage should be created for a rejected job")
+    assertDataStructuresEmpty()
+  }
+
+  test("regular shuffle job with dynamic allocation enabled is NOT rejected 
(path is inert)") {
+    // The dynamic-allocation fail-fast must apply only to jobs with a 
pipelined dependency; a plain
+    // regular-shuffle job with dynamic allocation on runs normally.
+    sc.conf.set(config.DYN_ALLOCATION_ENABLED, true)
+    sc.conf.set(config.DYN_ALLOCATION_TESTING, true)
+    try {
+      val shuffleMapRdd = new MyRDD(sc, 2, Nil)
+      val shuffleDep = new ShuffleDependency(shuffleMapRdd, new 
HashPartitioner(2))
+      val reduceRdd = new MyRDD(sc, 2, List(shuffleDep), tracker = 
mapOutputTracker)
+      submit(reduceRdd, Array(0, 1))
+      assert(taskSets.nonEmpty, "regular-shuffle job must not be rejected 
under dynamic allocation")
+    } finally {
+      sc.conf.set(config.DYN_ALLOCATION_ENABLED, false)
+      sc.conf.set(config.DYN_ALLOCATION_TESTING, false)
+    }
+  }
+
+  test("pipelined shuffle: submitting a pipelined dependency as a map-stage 
job is rejected") {
+    // A map-stage job (SparkContext.submitMapStage, e.g. AQE's 
ShuffleExchangeExec) materializes a
+    // shuffle to produce map-output statistics. A PipelinedShuffleDependency 
cannot serve that --
+    // it
+    // is transient with no durable map output (getStatistics would return 
all-zero stats), and its
+    // producer/consumer co-scheduling makes speculation unsafe. It is 
rejected outright, regardless
+    // of speculation. No partial scheduler state is left behind.
+    val producerRdd = new MyRDD(sc, 2, Nil)
+    val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+    val failure = new java.util.concurrent.atomic.AtomicReference[Exception]()
+    val failListener = new JobListener {
+      override def taskSucceeded(index: Int, result: Any): Unit = {}
+      override def jobFailed(exception: Exception): Unit = 
failure.set(exception)
+    }
+    submitMapStage(pipelinedDep, listener = failListener)
+    assert(failure.get() != null,
+      "submitMapStage with a pipelined dependency must be rejected")
+    assert(failure.get().getMessage.contains("cannot be submitted as a 
map-stage job"),
+      s"expected a map-stage-rejection error, got: 
${failure.get().getMessage}")
+    // No task sets were submitted for the rejected map-stage job, and no 
partial state remains.
+    assert(taskSets.isEmpty)
+    assertDataStructuresEmpty()
+  }
+
+  test("regular shuffle map-stage job with speculation enabled is NOT rejected 
(inertness)") {
+    // The map-stage speculation fail-fast must be inert for a regular 
ShuffleDependency.
+    sc.conf.set(config.SPECULATION_ENABLED, true)
+    try {
+      val producerRdd = new MyRDD(sc, 2, Nil)
+      val regularDep = new ShuffleDependency(producerRdd, new 
HashPartitioner(2))
+      submitMapStage(regularDep)
+      // Not rejected: the map stage is submitted and runs normally.
+      assert(taskSets.nonEmpty,
+        "regular-shuffle map-stage job must not be rejected under speculation")
+      completeShuffleMapStageSuccessfully(taskSets.head.stageId, 0, 2)
+      assertDataStructuresEmpty()
+    } finally {
+      sc.conf.set(config.SPECULATION_ENABLED, false)
+    }
+  }
+
+  // 
==========================================================================================
+  // Gang admission / slot check
+  // 
==========================================================================================
+
+  test("pipelined shuffle: an all-pipelined group that fits is admitted up 
front and runs") {
+    // Whole-group demand producer(2) + consumer(2) = 4 <= capacity 4, 
other-work occupancy 0, so
+    // the up-front gang admission check (handleJobSubmitted) admits the job 
before any stage runs,
+    // and both stages are then co-scheduled and complete normally.
+    val producerRdd = new MyRDD(sc, 2, Nil)
+    val myScheduler = scheduler.asInstanceOf[MyDAGScheduler]
+    myScheduler.maxConcurrentTasksForTest = 4
+    myScheduler.outstandingTasksForOtherWorkForTest = (_, _) => 0
+    try {
+      val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+      val consumerRdd = new MyRDD(sc, 2, List(pipelinedDep), tracker = 
mapOutputTracker)
+      submit(consumerRdd, Array(0, 1))
+      assert(taskSets.size === 2, "a group that fits must be admitted and 
co-scheduled")
+      completeShuffleMapStageSuccessfully(taskSets.head.stageId, 0, 2)
+      complete(taskSets(1), Seq((Success, 42), (Success, 43)))
+      assert(results === Map(0 -> 42, 1 -> 43))
+      assertDataStructuresEmpty()
+    } finally {
+      myScheduler.maxConcurrentTasksForTest = 1000
+      myScheduler.outstandingTasksForOtherWorkForTest = (_, _) => 0
+    }
+  }
+
+  test("pipelined shuffle: a group too large to co-fit fails fast with 
INSUFFICIENT_SLOT") {
+    // Constrain reported capacity to 3 slots. A pipelined group of 
producer(2) + consumer(2) = 4
+    // tasks exceeds it and can never be co-resident, so it must fail fast 
rather than deadlock.
+    val producerRdd = new MyRDD(sc, 2, Nil) // touch sc first so `scheduler` 
is initialized
+    scheduler.asInstanceOf[MyDAGScheduler].maxConcurrentTasksForTest = 3
+    try {
+      val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+      val consumerRdd = new MyRDD(sc, 2, List(pipelinedDep), tracker = 
mapOutputTracker)
+      val failure = new 
java.util.concurrent.atomic.AtomicReference[Exception]()
+      val failListener = new JobListener {
+        override def taskSucceeded(index: Int, result: Any): Unit = {}
+        override def jobFailed(exception: Exception): Unit = 
failure.set(exception)
+      }
+      submit(consumerRdd, Array(0, 1), listener = failListener)
+
+      assert(failure.get() != null, "an over-large pipelined group must fail 
the job")
+      
assert(failure.get().getMessage.contains("CONCURRENT_SCHEDULER_INSUFFICIENT_SLOT")
 ||
+        failure.get().getMessage.contains("concurrent task slots"),
+        s"expected an insufficient-slot error, got: 
${failure.get().getMessage}")
+      assertDataStructuresEmpty()
+    } finally {
+      scheduler.asInstanceOf[MyDAGScheduler].maxConcurrentTasksForTest = 1000
+    }
+  }
+
+  test("pipelined shuffle: an over-capacity 3-stage all-pipelined chain is 
rejected up front") {
+    // A 3-stage all-pipelined chain A->B->C has whole-group demand 2+2+2 = 6. 
With capacity 3 it
+    // cannot co-fit, so the up-front gang admission check 
(handleJobSubmitted) fails the job before
+    // any stage is created -- leaving no scheduler state behind 
(assertDataStructuresEmpty). This
+    // covers a multi-stage group in addition to the 2-stage "group too large" 
case.
+    val rddA = new MyRDD(sc, 2, Nil) // touch sc first so `scheduler` is 
initialized
+    scheduler.asInstanceOf[MyDAGScheduler].maxConcurrentTasksForTest = 3
+    try {
+      val depAB = new PipelinedShuffleDependency(rddA, new HashPartitioner(2))
+      val rddB = new MyRDD(sc, 2, List(depAB), tracker = mapOutputTracker)
+      val depBC = new PipelinedShuffleDependency(rddB, new HashPartitioner(2))
+      val rddC = new MyRDD(sc, 2, List(depBC), tracker = mapOutputTracker)
+      val failure = new 
java.util.concurrent.atomic.AtomicReference[Exception]()
+      val failListener = new JobListener {
+        override def taskSucceeded(index: Int, result: Any): Unit = {}
+        override def jobFailed(exception: Exception): Unit = 
failure.set(exception)
+      }
+      submit(rddC, Array(0, 1), listener = failListener)
+
+      assert(failure.get() != null, "an over-large 3-stage pipelined chain 
must fail the job")
+      
assert(failure.get().getMessage.contains("CONCURRENT_SCHEDULER_INSUFFICIENT_SLOT")
 ||
+        failure.get().getMessage.contains("concurrent task slots"),
+        s"expected an insufficient-slot error, got: 
${failure.get().getMessage}")
+      // No stage is created for a job rejected up front.
+      assert(taskSets.isEmpty, "no stage should be submitted for an 
up-front-rejected group")
+      assertDataStructuresEmpty()
+    } finally {
+      scheduler.asInstanceOf[MyDAGScheduler].maxConcurrentTasksForTest = 1000
+    }
+  }
+
+  test("pipelined shuffle: a group that fits is co-scheduled (slot check 
passes)") {
+    // Producer (2) + consumer (2) = 4 tasks <= 16 slots: co-scheduled 
normally, no abort.
+    val producerRdd = new MyRDD(sc, 2, Nil)
+    val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+    val consumerRdd = new MyRDD(sc, 2, List(pipelinedDep), tracker = 
mapOutputTracker)
+    submit(consumerRdd, Array(0, 1))
+    assert(taskSets.size === 2, "group fits, so producer and consumer are 
co-scheduled")
+    completeShuffleMapStageSuccessfully(taskSets.head.stageId, 0, 2)
+    complete(taskSets(1), Seq((Success, 42), (Success, 43)))
+    assert(results === Map(0 -> 42, 1 -> 43))
+    assertDataStructuresEmpty()
+  }
+
+  test("pipelined shuffle: a group that fits total capacity but not FREE slots 
fails fast") {
+    // Admission is decided against currently-FREE slots, not total capacity. 
A group of
+    // producer(2) + consumer(2) = 4 tasks fits a 10-slot cluster in 
principle, but if 8 slots are
+    // already occupied by OTHER work only 2 are free -- the group cannot 
co-fit right now and must
+    // fail fast rather than queue forever. Under the old total-capacity check 
this would wrongly be
+    // admitted (4 <= 10) and then hang. outstandingTasksForOtherWork already 
excludes the group's
+    // own members, so we report 8 directly.
+    val producerRdd = new MyRDD(sc, 2, Nil) // touch sc first so `scheduler` 
is initialized
+    val myScheduler = scheduler.asInstanceOf[MyDAGScheduler]
+    myScheduler.maxConcurrentTasksForTest = 10
+    // 8 of 10 slots busy elsewhere -> 2 free
+    myScheduler.outstandingTasksForOtherWorkForTest = (_, _) => 8
+    try {
+      val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+      val consumerRdd = new MyRDD(sc, 2, List(pipelinedDep), tracker = 
mapOutputTracker)
+      val failure = new 
java.util.concurrent.atomic.AtomicReference[Exception]()
+      val failListener = new JobListener {
+        override def taskSucceeded(index: Int, result: Any): Unit = {}
+        override def jobFailed(exception: Exception): Unit = 
failure.set(exception)
+      }
+      submit(consumerRdd, Array(0, 1), listener = failListener)
+
+      assert(failure.get() != null,
+        "a group that fits total but not free slots must fail the job 
(free-slot check)")
+      
assert(failure.get().getMessage.contains("CONCURRENT_SCHEDULER_INSUFFICIENT_SLOT")
 ||
+        failure.get().getMessage.contains("currently free"),
+        s"expected a free-slot insufficient-slot error, got: 
${failure.get().getMessage}")
+      assertDataStructuresEmpty()
+    } finally {
+      myScheduler.maxConcurrentTasksForTest = 1000
+      myScheduler.outstandingTasksForOtherWorkForTest = (_, _) => 0
+    }
+  }
+
+  test("pipelined shuffle: slot check can be disabled, admitting a group that 
exceeds capacity") {
+    // With spark.scheduler.pipelinedGroup.slotCheck.enabled=false the 
admission check is skipped
+    // entirely, so a group whose demand exceeds free slots is co-scheduled 
instead of failing fast.
+    // Deployments that admit capacity out-of-band (e.g. a slot reservation) 
rely on this.
+    val producerRdd = new MyRDD(sc, 2, Nil)
+    val myScheduler = scheduler.asInstanceOf[MyDAGScheduler]
+    myScheduler.maxConcurrentTasksForTest = 2 // demand 4 > 2 slots: would 
fail if the check ran
+    myScheduler.outstandingTasksForOtherWorkForTest = (_, _) => 0
+    val prev = sc.conf.get(config.PIPELINED_GROUP_SLOT_CHECK_ENABLED)
+    sc.conf.set(config.PIPELINED_GROUP_SLOT_CHECK_ENABLED, false)
+    try {
+      val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+      val consumerRdd = new MyRDD(sc, 2, List(pipelinedDep), tracker = 
mapOutputTracker)
+      submit(consumerRdd, Array(0, 1))
+      assert(taskSets.size === 2,
+        "with the slot check disabled, the over-capacity group is 
co-scheduled, not rejected")
+      completeShuffleMapStageSuccessfully(taskSets.head.stageId, 0, 2)
+      complete(taskSets(1), Seq((Success, 42), (Success, 43)))
+      assert(results === Map(0 -> 42, 1 -> 43))
+      assertDataStructuresEmpty()
+    } finally {
+      myScheduler.maxConcurrentTasksForTest = 1000
+      sc.conf.set(config.PIPELINED_GROUP_SLOT_CHECK_ENABLED, prev)
+    }
+  }
+
+  test("pipelined shuffle: a group whose demand exactly equals free capacity 
is admitted") {
+    // Boundary: demand == free capacity must be admitted, not rejected by an 
off-by-one (the check
+    // is `demand > freeSlots`, not `>=`). 4 total slots, other-work occupancy 
0 -> free 4 >= demand
+    // 4 -> admitted. NOTE: the DAGScheduler up-front admission 
(rejectUnadmittablePipelinedGroup)
+    // passes excludeStageIds = Set.empty (no job stage exists yet at 
admission time), so the
+    // group's-own-members exclusion is NOT exercised here; that exclusion is 
genuinely covered by
+    // TaskSchedulerImplSuite ("outstandingTasksForOtherWorkInProfile excludes 
the given stages").
+    val producerRdd = new MyRDD(sc, 2, Nil)
+    val myScheduler = scheduler.asInstanceOf[MyDAGScheduler]
+    myScheduler.maxConcurrentTasksForTest = 4
+    myScheduler.outstandingTasksForOtherWorkForTest = (_, _) => 0
+    try {
+      val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+      val consumerRdd = new MyRDD(sc, 2, List(pipelinedDep), tracker = 
mapOutputTracker)
+      submit(consumerRdd, Array(0, 1))
+      assert(taskSets.size === 2,
+        "a group whose demand equals free capacity must be co-scheduled, not 
rejected")
+      completeShuffleMapStageSuccessfully(taskSets.head.stageId, 0, 2)
+      complete(taskSets(1), Seq((Success, 42), (Success, 43)))
+      assert(results === Map(0 -> 42, 1 -> 43))
+      assertDataStructuresEmpty()
+    } finally {
+      myScheduler.maxConcurrentTasksForTest = 1000
+      myScheduler.outstandingTasksForOtherWorkForTest = (_, _) => 0
+    }
+  }
+
+  test("pipelined shuffle: a diamond reusing one producer counts its tasks 
once for admission") {
+    // Fan-out/diamond: ONE PipelinedShuffleDependency (one producer, one 
shuffle id) is read by TWO
+    // consumers that are then narrow-joined into the result. Execution 
creates ONE producer stage
+    // (getOrCreateShuffleMapStage keys on shuffle id), so the group's real 
concurrent demand is
+    // producer(2) + result(2) = 4 -- NOT 6. Counting the producer once per 
consumer EDGE would make
+    // demand 6 and wrongly reject this group at capacity 4. Pinning capacity 
to exactly 4 admits
+    // the correctly-deduped group and would fail if the producer were 
double-counted.
+    val producerRdd = new MyRDD(sc, 2, Nil)
+    val myScheduler = scheduler.asInstanceOf[MyDAGScheduler]
+    myScheduler.maxConcurrentTasksForTest = 4
+    myScheduler.outstandingTasksForOtherWorkForTest = (_, _) => 0
+    try {
+      val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+      val consumer1 = new MyRDD(sc, 2, List(pipelinedDep), tracker = 
mapOutputTracker)
+      val consumer2 = new MyRDD(sc, 2, List(pipelinedDep), tracker = 
mapOutputTracker)
+      // Narrow-join the two consumers into the result stage; the same 
pipelined dependency is thus
+      // referenced from two distinct consumer RDDs, exercising the per-edge 
double-count.
+      val resultRdd = new MyRDD(
+        sc, 2, List(new OneToOneDependency(consumer1), new 
OneToOneDependency(consumer2)),
+        tracker = mapOutputTracker)
+      submit(resultRdd, Array(0, 1))
+      assert(taskSets.size === 2,
+        s"diamond group is producer + result (both consumers); got 
${taskSets.size} task sets")
+      // Drain to completion: producer stage first, then the result stage.
+      completeShuffleMapStageSuccessfully(taskSets.head.stageId, 0, 2)
+      complete(taskSets(1), Seq((Success, 42), (Success, 43)))
+      assert(results === Map(0 -> 42, 1 -> 43))
+      assertDataStructuresEmpty()
+    } finally {
+      myScheduler.maxConcurrentTasksForTest = 1000
+      myScheduler.outstandingTasksForOtherWorkForTest = (_, _) => 0
+    }
+  }
+
+  test("pipelined shuffle: a wide fan-out reusing one producer counts its 
tasks once") {
+    // Wider fan-out: ONE producer feeds THREE consumers, all narrow-joined 
into the result. Real
+    // demand is still producer(2) + result(2) = 4; a per-edge count would be 
2 + 3*2 = 8. Capacity
+    // 4 admits the deduped group and would reject the double-counted one.
+    val producerRdd = new MyRDD(sc, 2, Nil)
+    val myScheduler = scheduler.asInstanceOf[MyDAGScheduler]
+    myScheduler.maxConcurrentTasksForTest = 4
+    myScheduler.outstandingTasksForOtherWorkForTest = (_, _) => 0
+    try {
+      val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+      val consumers = (0 until 3).map { _ =>
+        new OneToOneDependency(new MyRDD(sc, 2, List(pipelinedDep), tracker = 
mapOutputTracker))
+      }.toList
+      val resultRdd = new MyRDD(sc, 2, consumers, tracker = mapOutputTracker)
+      submit(resultRdd, Array(0, 1))
+      assert(taskSets.size === 2,
+        s"wide fan-out group is producer + result; got ${taskSets.size} task 
sets")
+      completeShuffleMapStageSuccessfully(taskSets.head.stageId, 0, 2)
+      complete(taskSets(1), Seq((Success, 42), (Success, 43)))
+      assert(results === Map(0 -> 42, 1 -> 43))
+      assertDataStructuresEmpty()
+    } finally {
+      myScheduler.maxConcurrentTasksForTest = 1000
+      myScheduler.outstandingTasksForOtherWorkForTest = (_, _) => 0
+    }
+  }
+
+  test("pipelined shuffle: two distinct shuffles over one producer RDD are 
both counted") {
+    // Mirror case guarding against OVER-deduping. Two SEPARATE 
PipelinedShuffleDependency objects
+    // are built over the SAME producer RDD; each calls newShuffleId() so they 
carry DISTINCT
+    // shuffle ids and produce DISTINCT stages. Both must be counted: demand 
is producer1(2) +
+    // producer2(2) + result(2) = 6. Deduping on producer RDD (instead of 
shuffle id) would drop one
+    // and undercount to 4. Capacity 5 must REJECT the true 6-task group; if 
it were wrongly deduped
+    // to 4 it would be admitted. The two producer stages share an RDD but are 
separate shuffles.
+    val producerRdd = new MyRDD(sc, 2, Nil)
+    val myScheduler = scheduler.asInstanceOf[MyDAGScheduler]
+    myScheduler.maxConcurrentTasksForTest = 5
+    myScheduler.outstandingTasksForOtherWorkForTest = (_, _) => 0
+    try {
+      val dep1 = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+      val dep2 = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+      assert(dep1.shuffleId != dep2.shuffleId, "the two deps must carry 
distinct shuffle ids")
+      val consumer1 = new MyRDD(sc, 2, List(dep1), tracker = mapOutputTracker)
+      val consumer2 = new MyRDD(sc, 2, List(dep2), tracker = mapOutputTracker)
+      val resultRdd = new MyRDD(
+        sc, 2, List(new OneToOneDependency(consumer1), new 
OneToOneDependency(consumer2)),
+        tracker = mapOutputTracker)
+      val failure = new 
java.util.concurrent.atomic.AtomicReference[Exception]()
+      val failListener = new JobListener {
+        override def taskSucceeded(index: Int, result: Any): Unit = {}
+        override def jobFailed(exception: Exception): Unit = 
failure.set(exception)
+      }
+      submit(resultRdd, Array(0, 1), listener = failListener)
+      assert(failure.get() != null,
+        "a 6-task group over 5 slots must fail fast; both distinct shuffles 
must be counted")
+      
assert(failure.get().getMessage.contains("CONCURRENT_SCHEDULER_INSUFFICIENT_SLOT")
 ||
+        failure.get().getMessage.contains("concurrent task slots"),
+        s"expected an insufficient-slot error, got: 
${failure.get().getMessage}")
+      assert(taskSets.isEmpty, "no stage should be submitted for an 
up-front-rejected group")
+      assertDataStructuresEmpty()
+    } finally {
+      myScheduler.maxConcurrentTasksForTest = 1000
+      myScheduler.outstandingTasksForOtherWorkForTest = (_, _) => 0
+    }
+  }
+
+  // 
==========================================================================================
+  // Group-observable completion
+  // 
==========================================================================================
+
+  test("pipelined shuffle: a consumer that finishes early does not end the job 
or cancel its " +
+      "still-running producer") {
+    // producer (shuffle map) --[pipelined]--> consumer (result). Consumer is 
co-scheduled; if its
+    // result tasks finish before the producer, its completion MUST be 
deferred -- otherwise the job
+    // would end and cancelRunningIndependentStages would cancel the 
still-running producer.
+    val producerRdd = new MyRDD(sc, 2, Nil)
+    val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+    val consumerRdd = new MyRDD(sc, 2, List(pipelinedDep), tracker = 
mapOutputTracker)
+    submit(consumerRdd, Array(0, 1))
+    assert(taskSets.size === 2)
+    val producerStageId = taskSets.head.stageId
+    val consumerTaskSet = taskSets(1)
+
+    // Consumer finishes FIRST (both result tasks succeed) while the producer 
is still running.
+    complete(consumerTaskSet, Seq((Success, 42), (Success, 43)))
+    // The job must NOT be finished yet, and the producer must still be 
running (not cancelled).
+    assert(results.isEmpty, "consumer completion must be deferred until the 
producer finishes")
+    assert(scheduler.runningStages.exists(_.rdd eq producerRdd),
+      "the still-running producer must not be cancelled by the early-finishing 
consumer")
+    assert(cancelledStages.isEmpty)
+
+    // Now the producer finishes -> the deferred consumer completions replay 
-> the job completes.
+    completeShuffleMapStageSuccessfully(producerStageId, 0, 2)
+    assert(results === Map(0 -> 42, 1 -> 43),
+      "deferred consumer completions should replay once the producer finishes")
+    assertDataStructuresEmpty()
+  }
+
+  test("pipelined shuffle: producer-then-consumer ordering completes normally 
(no deferral " +
+      "needed)") {
+    // Sanity: when the producer finishes before the consumer's tasks 
complete, there is nothing to
+    // defer and the job completes as usual.
+    val producerRdd = new MyRDD(sc, 2, Nil)
+    val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+    val consumerRdd = new MyRDD(sc, 2, List(pipelinedDep), tracker = 
mapOutputTracker)
+    submit(consumerRdd, Array(0, 1))
+    val producerStageId = taskSets.head.stageId
+    val consumerTaskSet = taskSets(1)
+
+    completeShuffleMapStageSuccessfully(producerStageId, 0, 2)
+    complete(consumerTaskSet, Seq((Success, 42), (Success, 43)))
+    assert(results === Map(0 -> 42, 1 -> 43))
+    assertDataStructuresEmpty()
+  }
+
+  test("pipelined shuffle: deferred consumer completions are dropped when the 
producer fails") {
+    // If a producer fails while a co-scheduled consumer's completions are 
buffered, those buffered
+    // successes must be DROPPED (not applied): the group is torn down / the 
job fails, and applying
+    // a partial consumer success would be incorrect (S6). The buffered tasks 
genuinely succeeded,
+    // so their TaskEnd events are still emitted on the drop path (otherwise 
listeners that track
+    // active tasks would believe these tasks are still running); only the 
stage/job-completion
+    // bookkeeping is withheld.
+    val taskEndCount = new java.util.concurrent.atomic.AtomicInteger(0)
+    val countingListener = new SparkListener {
+      override def onTaskEnd(taskEnd: SparkListenerTaskEnd): Unit = 
taskEndCount.incrementAndGet()
+    }
+    sc.addSparkListener(countingListener)
+    try {
+      val producerRdd = new MyRDD(sc, 2, Nil)
+      val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+      val consumerRdd = new MyRDD(sc, 2, List(pipelinedDep), tracker = 
mapOutputTracker)
+      val failure = new 
java.util.concurrent.atomic.AtomicReference[Exception]()
+      val failListener = new JobListener {
+        override def taskSucceeded(index: Int, result: Any): Unit = 
results.put(index, result)
+        override def jobFailed(exception: Exception): Unit = 
failure.set(exception)
+      }
+      submit(consumerRdd, Array(0, 1), listener = failListener)
+      val producerTaskSet = taskSets.head
+      val consumerTaskSet = taskSets(1)
+
+      // Consumer finishes early -> its whole completion events are buffered 
(deferred): no result
+      // and no TaskEnd yet, since the entire event is withheld until the 
producer's fate is known.
+      complete(consumerTaskSet, Seq((Success, 42), (Success, 43)))
+      assert(results.isEmpty, "consumer completions should be buffered while 
the producer runs")
+      sc.listenerBus.waitUntilEmpty(10000)
+      assert(taskEndCount.get() === 0,
+        "buffered consumer completions must not fire TaskEnd until released; 
got " +
+          taskEndCount.get())
+
+      // The producer now FAILS its whole task set. The job fails; the 
buffered consumer successes
+      // must NOT be applied as results, but their TaskEnd events ARE emitted 
on the drop path so
+      // active-task-tracking listeners see the tasks finish.
+      failed(producerTaskSet, "producer blew up")
+      assert(failure.get() != null, "job should fail when the producer fails")
+      assert(results.isEmpty,
+        "buffered consumer successes must be dropped when the producer fails, 
not applied")
+      sc.listenerBus.waitUntilEmpty(10000)
+      assert(taskEndCount.get() === 2,
+        "drop path must emit the buffered consumer TaskEnd events; got " + 
taskEndCount.get())
+      assertDataStructuresEmpty()
+    } finally {
+      sc.removeSparkListener(countingListener)
+    }
+  }
+
+  test("pipelined shuffle: a fan-in consumer's buffered successes are dropped 
as soon as one " +
+      "producer fails, even if another later succeeds") {
+    // Fan-in: a consumer co-scheduled with TWO pipelined producers (A and B). 
If A fails, the
+    // consumer's buffered successes must be DROPPED -- they were computed 
against A's now-invalid
+    // output, and the group is failed as a unit (S6). The drop happens 
immediately when A fails,
+    // not deferred until the last producer finishes: so even if B later 
succeeds, the consumer has
+    // already been dropped and B's completion is a clean no-op (it must NOT 
replay the successes).
+    val producerA = new MyRDD(sc, 2, Nil)
+    val psdA = new PipelinedShuffleDependency(producerA, new 
HashPartitioner(2))
+    val producerB = new MyRDD(sc, 2, Nil)
+    val psdB = new PipelinedShuffleDependency(producerB, new 
HashPartitioner(2))
+    val consumerRdd = new MyRDD(sc, 2, List(psdA, psdB), tracker = 
mapOutputTracker)
+    submit(consumerRdd, Array(0, 1))
+    assert(taskSets.size === 3, s"A, B and the consumer must be co-scheduled; 
got ${taskSets.size}")
+    val consumerTaskSet =
+      taskSets.find(ts => scheduler.stageIdToStage(ts.stageId).rdd eq 
consumerRdd).get
+    val stageA = scheduler.stageIdToStage(
+      taskSets.find(ts => scheduler.stageIdToStage(ts.stageId).rdd eq 
producerA).get.stageId)
+    val stageB = scheduler.stageIdToStage(
+      taskSets.find(ts => scheduler.stageIdToStage(ts.stageId).rdd eq 
producerB).get.stageId)
+    val consumerStage = scheduler.stageIdToStage(consumerTaskSet.stageId)
+
+    // Consumer finishes early -> its successes are buffered against BOTH A 
and B.
+    complete(consumerTaskSet, Seq((Success, 42), (Success, 43)))
+    assert(results.isEmpty, "consumer completions should be buffered while its 
producers run")
+    
assert(scheduler.dependentStageMap.get(consumerStage).exists(_.parents.size == 
2),
+      "the consumer must be deferred against both producers")
+
+    // A fails: the consumer is dropped immediately (not retained pending B), 
and no result applied.
+    // (Driven via markStageAsFinished directly; the job-failure teardown that 
would normally follow
+    // is exercised by the group-atomic-failure tests in a later PR -- here we 
assert only the
+    // drop-vs-replay outcome.)
+    scheduler.markStageAsFinished(stageA, errorMessage = Some("producer A blew 
up"))
+    assert(!scheduler.dependentStageMap.contains(consumerStage),
+      "the consumer's deferral must be dropped as soon as a producer fails")
+    assert(results.isEmpty, "a fan-in consumer's buffered successes must not 
be applied on failure")
+
+    // B now succeeds: the consumer is already gone, so this is a clean no-op 
-- it must NOT replay.
+    scheduler.runningStages += stageB
+    completeShuffleMapStageSuccessfully(stageB.id, 0, 2)
+    assert(results.isEmpty,
+      s"a dropped fan-in consumer must not replay when a surviving producer 
succeeds; got $results")
+  }
+
+
+  test("pipelined shuffle: a deferred consumer task fires its TaskEnd exactly 
once (at replay)") {
+    // A deferred CompletionEvent must have its side effects (task-end 
listener event, accumulator
+    // update) applied exactly once -- at replay -- not once when buffered and 
again when replayed.
+    // The producer (2 tasks) and consumer (2 tasks) yield exactly 4 TaskEnd 
events total; a
+    // buffer+replay double-post would inflate that count.
+    val taskEndCount = new java.util.concurrent.atomic.AtomicInteger(0)
+    val countingListener = new SparkListener {
+      override def onTaskEnd(taskEnd: SparkListenerTaskEnd): Unit = 
taskEndCount.incrementAndGet()
+    }
+    sc.addSparkListener(countingListener)
+    try {
+      val producerRdd = new MyRDD(sc, 2, Nil)
+      val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+      val consumerRdd = new MyRDD(sc, 2, List(pipelinedDep), tracker = 
mapOutputTracker)
+      submit(consumerRdd, Array(0, 1))
+      val producerStageId = taskSets.head.stageId
+      val consumerTaskSet = taskSets(1)
+
+      // Consumer finishes first -> its two completions are buffered 
(deferred, no TaskEnd yet).
+      complete(consumerTaskSet, Seq((Success, 42), (Success, 43)))
+      assert(results.isEmpty, "consumer job result must be deferred until the 
producer finishes")
+      sc.listenerBus.waitUntilEmpty(10000)
+      assert(taskEndCount.get() === 0,
+        "deferred consumer completions must not fire TaskEnd until replay; got 
" +
+          taskEndCount.get())
+
+      // Producer finishes -> the two deferred consumer completions replay.
+      completeShuffleMapStageSuccessfully(producerStageId, 0, 2)
+      assert(results === Map(0 -> 42, 1 -> 43))
+      sc.listenerBus.waitUntilEmpty(10000)
+
+      // Exactly 4 TaskEnd events (2 producer + 2 consumer). A double-post 
from buffer+replay would
+      // make it 6 (the 2 consumer tasks counted twice).
+      assert(taskEndCount.get() === 4,
+        s"expected 4 TaskEnd events (no buffer+replay duplication), got 
${taskEndCount.get()}")
+      assertDataStructuresEmpty()
+    } finally {
+      sc.removeSparkListener(countingListener)
+    }
+  }
+
+  test("pipelined shuffle: releasing a deferral is gated on the producer being 
genuinely done") {
+    // Regression for the "producer finished but not available -> resubmitted" 
hazard: a
+    // ShuffleMapStage producer that reaches markStageAsFinished with no error 
but is NOT available
+    // (some output missing, so it will be resubmitted) must NOT 
release/replay its deferred
+    // consumers -- doing so would apply the consumer's results against 
soon-to-be-recomputed
+    // output.
+    // This drives BOTH branches of the gate: (1) the RETAINED branch -- 
markStageAsFinished on a
+    // not-yet-available pipelined producer must leave the deferral in place; 
and (2) the RELEASED
+    // branch -- once the producer is genuinely available, the deferral is 
released and results
+    // applied.
+    val producerRdd = new MyRDD(sc, 2, Nil)
+    val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+    val consumerRdd = new MyRDD(sc, 2, List(pipelinedDep), tracker = 
mapOutputTracker)
+    submit(consumerRdd, Array(0, 1))
+    val producerStageId = taskSets.head.stageId
+    val consumerTaskSet = taskSets(1)
+
+    complete(consumerTaskSet, Seq((Success, 42), (Success, 43)))
+    // Deferred while the producer runs.
+    assert(scheduler.dependentStageMap.keys.exists(_.rdd eq consumerRdd))
+    assert(results.isEmpty)
+
+    val producerStage =
+      scheduler.stageIdToStage(producerStageId).asInstanceOf[ShuffleMapStage]
+
+    // (1) RETAINED branch: the producer has completed NO partitions yet, so 
it is not available.
+    // markStageAsFinished with no error and willRetry=false hits the 
producerAboutToResubmit guard
+    // (ShuffleMapStage && !producerFailed && !isAvailable) and must NOT 
release the consumer.
+    assert(!producerStage.isAvailable, "precondition: producer not yet 
available")
+    scheduler.markStageAsFinished(producerStage, errorMessage = None, 
willRetry = false)
+    assert(scheduler.dependentStageMap.keys.exists(_.rdd eq consumerRdd),
+      "a not-yet-available pipelined producer must NOT release its deferred 
consumers (it is " +
+        "about to resubmit; releasing would apply results against 
soon-to-be-recomputed output)")
+    assert(results.isEmpty,
+      "consumer results must not be applied while the producer is not 
available")
+
+    // (2) RELEASED branch: producer completes for real and IS available -> 
deferral released,
+    // consumer results applied. (Re-add it to runningStages since (1) removed 
it, so the normal
+    // completion path proceeds as in production.)
+    scheduler.runningStages += producerStage
+    completeShuffleMapStageSuccessfully(producerStageId, 0, 2)
+    assert(producerStage.isAvailable)
+    assert(!scheduler.dependentStageMap.keys.exists(_.rdd eq consumerRdd),
+      "deferral released once the producer is genuinely done (available)")
+    assert(results === Map(0 -> 42, 1 -> 43))
+    assertDataStructuresEmpty()
+  }
+
+  test("pipelined shuffle: job teardown flushes a buffered consumer's TaskEnds 
even when the " +
+      "release path never drained it") {
+    // A consumer's whole completion event is buffered while its producer runs 
(coarse model), so
+    // its TaskEnds are held, not yet emitted. Normally 
releaseDeferredPipelinedConsumers drains
+    // them (cancelRunningIndependentStages finishes each running producer 
before cleanup). But a
+    // producer left in resubmit limbo -- finished with no error yet NOT 
available, so
+    // producerAboutToResubmit held the deferral AND markStageAsFinished 
removed it from
+    // runningStages -- is skipped by cancelRunningIndependentStages (neither 
running nor failed).
+    // If the job is then cancelled, cleanup must still flush the buffered 
TaskEnds (results stay
+    // dropped) so a listener tracking active tasks does not leak them as 
perpetually running.
+    val taskEndCount = new java.util.concurrent.atomic.AtomicInteger(0)
+    val countingListener = new SparkListener {
+      override def onTaskEnd(taskEnd: SparkListenerTaskEnd): Unit = 
taskEndCount.incrementAndGet()
+    }
+    sc.addSparkListener(countingListener)
+    try {
+      val producerRdd = new MyRDD(sc, 2, Nil)
+      val pipelinedDep = new PipelinedShuffleDependency(producerRdd, new 
HashPartitioner(2))
+      val consumerRdd = new MyRDD(sc, 2, List(pipelinedDep), tracker = 
mapOutputTracker)
+      val jobId = submit(consumerRdd, Array(0, 1))
+      val producerStageId = taskSets.head.stageId
+      val consumerTaskSet = taskSets(1)
+
+      // Consumer finishes early -> whole completion buffered; no TaskEnd yet.
+      complete(consumerTaskSet, Seq((Success, 42), (Success, 43)))
+      sc.listenerBus.waitUntilEmpty()
+      assert(taskEndCount.get() === 0, "buffered consumer TaskEnds are held 
while producer runs")
+
+      // Drive the producer into resubmit limbo: finished, no error, but not 
available -> the
+      // deferral is retained and the producer leaves runningStages. No task 
failed, so it also has
+      // no failedAttemptIds -- cancelRunningIndependentStages will skip it.
+      val producerStage = 
scheduler.stageIdToStage(producerStageId).asInstanceOf[ShuffleMapStage]
+      assert(!producerStage.isAvailable, "precondition: producer not 
available")
+      scheduler.markStageAsFinished(producerStage, errorMessage = None, 
willRetry = false)
+      assert(scheduler.dependentStageMap.keys.exists(_.rdd eq consumerRdd),
+        "precondition: the deferral is retained (producer about to resubmit)")
+      assert(!scheduler.runningStages.contains(producerStage) &&
+        producerStage.failedAttemptIds.isEmpty,
+        "precondition: producer is neither running nor failed, so cancel will 
skip it")
+
+      // Cancel the job. The release path never drains this consumer, so 
cleanup must flush it.
+      cancel(jobId)
+      sc.listenerBus.waitUntilEmpty()
+      assert(scheduler.dependentStageMap.isEmpty, "the deferral must not 
outlive the job")
+      assert(results.isEmpty, "a cancelled job's buffered consumer success 
must not be applied")
+      assert(taskEndCount.get() === 2,
+        s"teardown must flush the 2 buffered consumer TaskEnds (else active 
tasks leak); got " +
+          taskEndCount.get())
+      assertDataStructuresEmpty()
+    } finally {
+      sc.removeSparkListener(countingListener)
+    }
+  }
+
 }
 
 class DAGSchedulerAbortStageOffSuite extends DAGSchedulerSuite {
diff --git 
a/core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala 
b/core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala
index 377c8afe134d..81782df283ee 100644
--- 
a/core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala
+++ 
b/core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala
@@ -2808,4 +2808,62 @@ class TaskSchedulerImplSuite extends SparkFunSuite with 
LocalSparkContext
     assert(!tsm.isInstanceOf[StructuredStreamingIdAwareSchedulerLogging])
   }
 
+  test("outstandingTasksForOtherWorkInProfile counts running + enqueued, 
excludes given stages, " +
+      "and is resource-profile-scoped") {
+    // Backs the pipelined-group slot admission check (DAGScheduler): it must 
report the OUTSTANDING
+    // (running + enqueued, i.e. numTasks - tasksSuccessful) task demand of 
OTHER work in a given
+    // resource profile, excluding the group's own member stages.
+    val taskScheduler = setupScheduler()
+    // Only 3 total cores, but each stage has 4 tasks -> some tasks launch, 
the rest stay enqueued.
+    val workerOffers = IndexedSeq(
+      new WorkerOffer("executor0", "host0", 2),
+      new WorkerOffer("executor1", "host1", 1))
+
+    // Stage 0 and stage 1 both in the DEFAULT profile, 4 tasks each (8 total) 
but only 3 cores, so
+    // 3 launch and 5 remain enqueued. Distinct stageIds let us exclude one 
and count the other.
+    taskScheduler.submitTasks(FakeTask.createTaskSet(4, stageId = 0, 
stageAttemptId = 0))
+    taskScheduler.submitTasks(FakeTask.createTaskSet(4, stageId = 1, 
stageAttemptId = 0))
+    val launched = taskScheduler.resourceOffers(workerOffers).flatten
+    assert(launched.length === 3, "only three tasks can run (3 cores); the 
other five are enqueued")
+
+    val defaultRp = ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID
+    // Nothing excluded: all 8 tasks are outstanding (3 running + 5 enqueued) 
though only 3 run.
+    // The old running-only count would have reported 3 here; outstanding 
demand is what can hang a
+    // co-scheduled group, so it must be 8.
+    assert(taskScheduler.outstandingTasksForOtherWorkInProfile(defaultRp, 
Set.empty) === 8)
+    // Exclude stage 0 (as if it were the group's own member): only stage 1's 
4 tasks remain.
+    assert(taskScheduler.outstandingTasksForOtherWorkInProfile(defaultRp, 
Set(0)) === 4)
+    // Exclude both: zero "other" tasks.
+    assert(taskScheduler.outstandingTasksForOtherWorkInProfile(defaultRp, 
Set(0, 1)) === 0)
+    // A DIFFERENT (non-existent here) resource profile has no outstanding 
tasks -- other-profile
+    // load must not be charged against this profile (RP-scoping).
+    val otherRpId = defaultRp + 12345
+    assert(taskScheduler.outstandingTasksForOtherWorkInProfile(otherRpId, 
Set.empty) === 0,
+      "tasks in the default profile must not be counted against a different 
profile")
+  }
+
+  test("outstandingTasksForOtherWorkInProfile does not double-count a zombie + 
live attempt") {
+    // A retried/killed stage can have both a zombie (superseded) attempt and 
a live attempt in the
+    // map at once; the live attempt re-runs the zombie's outstanding tasks, 
so only the live
+    // attempt's demand should count. Counting both would inflate occupancy 
and could spuriously
+    // fail a pipelined group with CONCURRENT_SCHEDULER_INSUFFICIENT_SLOT.
+    val taskScheduler = setupScheduler()
+    val defaultRp = ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID
+
+    // Attempt 0 of stage 0: 4 tasks, none started. Mark it zombie (as if 
superseded).
+    val attempt0 = FakeTask.createTaskSet(4, stageId = 0, stageAttemptId = 0)
+    taskScheduler.submitTasks(attempt0)
+    taskScheduler.taskSetManagerForAttempt(0, 0).get.isZombie = true
+
+    // Attempt 1 of the SAME stage: the live retry, also 4 tasks.
+    val attempt1 = FakeTask.createTaskSet(4, stageId = 0, stageAttemptId = 1)
+    taskScheduler.submitTasks(attempt1)
+
+    // Both attempts are in the map, but only the live attempt's 4 tasks are 
outstanding -- not 8.
+    assert(taskScheduler.outstandingTasksForOtherWorkInProfile(defaultRp, 
Set.empty) === 4,
+      "a zombie attempt must not be counted alongside its live retry")
+    // Excluding the stage (as the group's own member) drops both attempts.
+    assert(taskScheduler.outstandingTasksForOtherWorkInProfile(defaultRp, 
Set(0)) === 0)
+  }
+
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to