This is an automated email from the ASF dual-hosted git repository.

cloud-fan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new d37297d1b133 [SPARK-58454][CORE] Fully decouple pipelined shuffles 
from the MapOutputTracker
d37297d1b133 is described below

commit d37297d1b133baa73df5a4e5d41c80d66226cc0e
Author: Boyang Jerry Peng <[email protected]>
AuthorDate: Mon Aug 3 09:46:07 2026 +0800

    [SPARK-58454][CORE] Fully decouple pipelined shuffles from the 
MapOutputTracker
    
    ### What changes were proposed in this pull request?
    
    This PR fully decouples a **pipelined (streaming) shuffle** from the 
`MapOutputTracker` (MOT). It is part of the pipelined-shuffle stack and builds 
on top of #57361 (group-atomic failure + fail-fast rejection).
    
    Background: a `PipelinedShuffleDependency` produces a transient, 
once-through stream and has no durable, addressable map output. Up to and 
including #57361, such a shuffle was still *registered* in the 
`MapOutputTracker` as an empty shell entry (so `containsShuffle` / 
`unregisterShuffle` bookkeeping stayed uniform), and only its map outputs were 
never populated. This PR removes that vestigial coupling: a pipelined shuffle 
is now served **solely** by the driver-only `StreamingShuffleOu [...]
    
    Concretely:
    
    - **Registration (`DAGScheduler.createShuffleMapStage`)**: a pipelined 
shuffle is registered only with the `StreamingShuffleOutputTracker`; a regular 
shuffle registers with the `MapOutputTracker` exactly as before. If a pipelined 
dependency is created but no streaming tracker is configured, creation now 
**fails loud before mutating any scheduler state** (rather than leaving a 
half-created stage cached), so a re-submission re-throws the same error instead 
of reusing stale state.
    - **Availability (`ShuffleMapStage`)**: a pipelined stage's 
`numAvailableOutputs` / `findMissingPartitions` / `isAvailable` are read from 
the stage-local, monotonic `pipelinedCompletedPartitions` set (a partition is 
added on map-task success and never removed on executor/host loss), not from 
the MOT. This is what avoids the streaming-writer resubmit hang.
    - **Cleanup (`ContextCleaner`)**: `doCleanupShuffle` now cleans up 
whichever tracker holds the shuffle. The two branches are independent, each 
keyed on its own tracker's membership. For a pipelined shuffle the branch calls 
`shuffleDriverComponents.removeShuffle` (balancing the `registerShuffle` the 
`ShuffleDependency` constructor issues for every shuffle -- the default 
implementation turns this into a `RemoveShuffle` RPC, which finds no blocks 
since a pipelined shuffle has none) and t [...]
    - **Every MOT call site keyed on a pipelined `shuffleId`** (FetchFailed 
invalidation, indeterminate-stage rollback in `rollbackShuffleMapStage`, merge 
finalization, `getStatistics`) is now either type-guarded (`!isPipelined`), 
inside a non-pipelined branch, or unreachable because the shape is rejected up 
front (barrier / push-merge / map-stage-job / indeterminate producer). None can 
reach a `getShuffleStatusOrError` throw for an absent pipelined entry.
    
    Note: this PR does **not** activate pipelined shuffles for any query -- no 
production code constructs a `PipelinedShuffleDependency` yet. Activation 
(wiring `ShuffleExchangeExec` and Real-Time Mode) is the follow-up PR stacked 
on this one.
    
    ### Why are the changes needed?
    
    Keeping a pipelined shuffle in the `MapOutputTracker` -- even as an empty 
shell -- left latent coupling that is unsound once the shuffle is transient: 
any MOT path that invalidates or reads outputs by `shuffleId` (executor-loss 
stripping, indeterminate rollback, FetchFailed handling) could either strip a 
completed pipelined output (triggering an invalid producer resubmit that hangs 
the streaming writer) or throw `ShuffleStatusNotFoundException` on the 
scheduler event thread once regis [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. There is no production path that creates a pipelined shuffle yet; these 
changes are internal to the scheduler and shuffle-tracking layers.
    
    ### How was this patch tested?
    
    New and existing unit tests under `core`:
    
    - `ContextCleanerSuite`: cleanup unregisters a pipelined shuffle from the 
streaming tracker; cleanup of a regular shuffle leaves the streaming tracker 
untouched.
    - `StreamingShuffleOutputTrackerSuite`: `containsShuffle` reflects 
register/unregister.
    - `PipelinedShuffleRoutingSuite`: `createShuffleMapStage` fails loud 
(leaving no partial scheduler state) when a pipelined dependency is created 
with no streaming tracker configured.
    - `DAGSchedulerSuite`: existing pipelined group-atomic and FetchFailed 
tests continue to pass with the MOT fully decoupled.
    
    `build/sbt "core/testOnly ... "` (core main + test compile clean; 
scalastyle clean).
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Co-authored with Claude Code (Opus 4.8)
    
    Closes #57646 from jerrypeng/stack/pipelined-shuffle-pr7-mot-decouple.
    
    Authored-by: Boyang Jerry Peng <[email protected]>
    Signed-off-by: Wenchen Fan <[email protected]>
---
 .../scala/org/apache/spark/ContextCleaner.scala    |  19 ++
 .../scala/org/apache/spark/MapOutputTracker.scala  |  10 +-
 .../spark/StreamingShuffleOutputTracker.scala      |  23 +-
 .../org/apache/spark/scheduler/DAGScheduler.scala  | 244 +++++++++++++--------
 .../apache/spark/scheduler/ShuffleMapStage.scala   |  17 +-
 .../streaming/StreamingShuffleManager.scala        |   4 +-
 .../org/apache/spark/ContextCleanerSuite.scala     |  90 ++++++++
 .../spark/StreamingShuffleOutputTrackerSuite.scala |  14 ++
 .../apache/spark/scheduler/DAGSchedulerSuite.scala |  40 ++++
 .../spark/scheduler/TaskSetManagerSuite.scala      |   5 +-
 .../shuffle/PipelinedShuffleRoutingSuite.scala     |  40 ++++
 11 files changed, 397 insertions(+), 109 deletions(-)

diff --git a/core/src/main/scala/org/apache/spark/ContextCleaner.scala 
b/core/src/main/scala/org/apache/spark/ContextCleaner.scala
index 0b3c22a22cb4..6c1b49157cc0 100644
--- a/core/src/main/scala/org/apache/spark/ContextCleaner.scala
+++ b/core/src/main/scala/org/apache/spark/ContextCleaner.scala
@@ -236,6 +236,10 @@ private[spark] class ContextCleaner(
   /** Perform shuffle cleanup. */
   def doCleanupShuffle(shuffleId: Int, blocking: Boolean): Unit = {
     try {
+      // A shuffle lives in exactly one tracker, split by dependency type: a 
regular shuffle in the
+      // MapOutputTracker, a pipelined shuffle in the driver-only 
StreamingShuffleOutputTracker (see
+      // DAGScheduler.createShuffleMapStage). Clean up whichever holds it -- 
the two branches are
+      // independent, each keyed on its own tracker's membership, so neither 
depends on the other.
       if (mapOutputTrackerMaster.containsShuffle(shuffleId)) {
         logDebug("Cleaning shuffle " + shuffleId)
         // Shuffle must be removed before it's unregistered from the output 
tracker
@@ -244,6 +248,19 @@ private[spark] class ContextCleaner(
         mapOutputTrackerMaster.unregisterShuffle(shuffleId)
         listeners.asScala.foreach(_.shuffleCleaned(shuffleId))
         logDebug("Cleaned shuffle " + shuffleId)
+      } else if 
(streamingShuffleOutputTrackerMaster.exists(_.containsShuffle(shuffleId))) {
+        // A pipelined shuffle's output state is driver-only (the worker 
tracker caches nothing), so
+        // it is unregistered directly from the StreamingShuffleOutputTracker 
rather than the
+        // MapOutputTracker. Still call shuffleDriverComponents.removeShuffle 
to balance the
+        // registerShuffle that the ShuffleDependency constructor issues for 
EVERY shuffle (incl. a
+        // pipelined one): a custom components impl may track per-shuffle 
driver state that would
+        // otherwise leak. For the default impl this is just a RemoveShuffle 
RPC that finds no
+        // blocks (a pipelined shuffle has none), matching the regular 
branch's ordering.
+        logDebug("Cleaning pipelined shuffle " + shuffleId)
+        shuffleDriverComponents.removeShuffle(shuffleId, blocking)
+        
streamingShuffleOutputTrackerMaster.foreach(_.unregisterShuffle(shuffleId))
+        listeners.asScala.foreach(_.shuffleCleaned(shuffleId))
+        logDebug("Cleaned pipelined shuffle " + shuffleId)
       } else {
         logDebug("Asked to cleanup non-existent shuffle (maybe it was already 
removed)")
       }
@@ -308,6 +325,8 @@ private[spark] class ContextCleaner(
 
   private def broadcastManager = sc.env.broadcastManager
   private def mapOutputTrackerMaster = 
sc.env.mapOutputTracker.asInstanceOf[MapOutputTrackerMaster]
+  private def streamingShuffleOutputTrackerMaster: 
Option[StreamingShuffleOutputTrackerMaster] =
+    
sc.env.streamingShuffleOutputTracker.map(_.asInstanceOf[StreamingShuffleOutputTrackerMaster])
 }
 
 private object ContextCleaner {
diff --git a/core/src/main/scala/org/apache/spark/MapOutputTracker.scala 
b/core/src/main/scala/org/apache/spark/MapOutputTracker.scala
index 742baf40ecbf..933da9477272 100644
--- a/core/src/main/scala/org/apache/spark/MapOutputTracker.scala
+++ b/core/src/main/scala/org/apache/spark/MapOutputTracker.scala
@@ -799,7 +799,7 @@ private[spark] class MapOutputTrackerMaster(
     conf: SparkConf,
     private[spark] val broadcastManager: BroadcastManager,
     private[spark] val isLocal: Boolean)
-  extends MapOutputTracker(conf) {
+  extends MapOutputTracker(conf) with ShuffleOutputTrackerMaster {
 
   // The size at which we use Broadcast to send the map output statuses to the 
executors
   private val minSizeForBroadcast = 
conf.get(SHUFFLE_MAPOUTPUT_MIN_SIZE_FOR_BROADCAST).toInt
@@ -943,6 +943,10 @@ private[spark] class MapOutputTrackerMaster(
     }
   }
 
+  // ShuffleOutputTrackerMaster: a regular shuffle has no per-job 
registration, so jobId is ignored.
+  override def registerShuffle(shuffleId: Int, numMaps: Int, numReduces: Int, 
jobId: Int): Unit =
+    registerShuffle(shuffleId, numMaps, numReduces)
+
   def updateMapOutput(shuffleId: Int, mapId: Long, bmAddress: BlockManagerId): 
Unit = {
     shuffleStatuses.get(shuffleId) match {
       case Some(shuffleStatus) =>
@@ -1030,7 +1034,7 @@ private[spark] class MapOutputTrackerMaster(
   }
 
   /** Unregister shuffle data */
-  def unregisterShuffle(shuffleId: Int): Unit = {
+  override def unregisterShuffle(shuffleId: Int): Unit = {
     shuffleStatuses.remove(shuffleId).foreach { shuffleStatus =>
       shuffleStatus.invalidateSerializedMapOutputStatusCache()
       shuffleStatus.invalidateSerializedMergeOutputStatusCache()
@@ -1077,7 +1081,7 @@ private[spark] class MapOutputTrackerMaster(
   }
 
   /** Check if the given shuffle is being tracked */
-  def containsShuffle(shuffleId: Int): Boolean = 
shuffleStatuses.contains(shuffleId)
+  override def containsShuffle(shuffleId: Int): Boolean = 
shuffleStatuses.contains(shuffleId)
 
   def getNumAvailableOutputs(shuffleId: Int): Int = {
     shuffleStatuses.get(shuffleId).map(_.numAvailableMapOutputs).getOrElse(0)
diff --git 
a/core/src/main/scala/org/apache/spark/StreamingShuffleOutputTracker.scala 
b/core/src/main/scala/org/apache/spark/StreamingShuffleOutputTracker.scala
index 090be7f691d4..df4e76b017db 100644
--- a/core/src/main/scala/org/apache/spark/StreamingShuffleOutputTracker.scala
+++ b/core/src/main/scala/org/apache/spark/StreamingShuffleOutputTracker.scala
@@ -28,6 +28,23 @@ import 
org.apache.spark.internal.config.SHUFFLE_MAPOUTPUT_DISPATCHER_NUM_THREADS
 import org.apache.spark.rpc.{RpcCallContext, RpcEndpoint, RpcEndpointRef, 
RpcEnv}
 import org.apache.spark.util.ThreadUtils
 
+/**
+ * The driver-side registry for a shuffle's output. A regular shuffle is 
served by the
+ * `MapOutputTrackerMaster`, a pipelined (streaming) shuffle by the
+ * `StreamingShuffleOutputTrackerMaster` -- split by dependency type, with no 
overlap. This is the
+ * small common surface the DAGScheduler and ContextCleaner drive 
polymorphically, so they select
+ * the right tracker by shuffle-dependency type (see 
`DAGScheduler.outputTrackerMaster`) rather than
+ * special-casing pipelined shuffles at each call site.
+ */
+private[spark] trait ShuffleOutputTrackerMaster {
+  /** Register a shuffle so its outputs can be tracked. `jobId` is used by the 
streaming tracker. */
+  def registerShuffle(shuffleId: Int, numMaps: Int, numReduces: Int, jobId: 
Int): Unit
+  /** Whether the given shuffle is registered with this tracker. */
+  def containsShuffle(shuffleId: Int): Boolean
+  /** Unregister a shuffle and release its tracked state. */
+  def unregisterShuffle(shuffleId: Int): Unit
+}
+
 private[spark] sealed trait StreamingShuffleTaskLocationTrackerMessage
 
 private[spark] case class UpdateStreamingShuffleTaskLocation(
@@ -196,7 +213,7 @@ private[spark] abstract class 
StreamingShuffleOutputTracker(conf: SparkConf) ext
 private[spark] case class StreamingShuffleInfo(numMaps: Int, numReduces: Int, 
jobId: Int)
 
 private[spark] class StreamingShuffleOutputTrackerMaster(conf: SparkConf)
-  extends StreamingShuffleOutputTracker(conf) {
+  extends StreamingShuffleOutputTracker(conf) with ShuffleOutputTrackerMaster {
 
   // map that stores task location information organized in the following 
fashion
   // shuffle id -> {mapId -> location}
@@ -220,7 +237,7 @@ private[spark] class 
StreamingShuffleOutputTrackerMaster(conf: SparkConf)
     pool
   }
 
-  def registerShuffle(shuffleId: Int, numMaps: Int, numReduces: Int, jobId: 
Int): Unit = {
+  override def registerShuffle(shuffleId: Int, numMaps: Int, numReduces: Int, 
jobId: Int): Unit = {
     logInfo(log"Registering shuffleId ${MDC(LogKeys.SHUFFLE_ID, shuffleId)} 
with ${
       MDC(LogKeys.NUM_MAPPERS, numMaps)} mappers and ${
       MDC(LogKeys.NUM_REDUCERS, numReduces)} reducers")
@@ -230,6 +247,8 @@ private[spark] class 
StreamingShuffleOutputTrackerMaster(conf: SparkConf)
     }
   }
 
+  override def containsShuffle(shuffleId: Int): Boolean = 
shuffleInfos.containsKey(shuffleId)
+
   // for testing purposes
   private[spark] def getShuffleInfo(shuffleId: Int): 
Option[StreamingShuffleInfo] = {
     Option(shuffleInfos.get(shuffleId))
diff --git a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala 
b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
index a72af257ad73..05c142558a43 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
@@ -188,7 +188,7 @@ private[spark] class DAGScheduler(
   private[scheduler] val dependentStageMap = new HashMap[Stage, 
DependentStageInfo]
 
   // Whether we have already logged that the pipelined-group slot check is 
disabled. Only the
-  // (single-threaded) event loop touches this, so a plain var is safe. Keeps 
the warning to once
+  // (single-threaded) event loop touches this, so a plain var is safe. Limits 
the warning to once
   // per scheduler rather than once per submitted batch job.
   private var warnedPipelinedSlotCheckDisabled = false
 
@@ -692,6 +692,11 @@ private[spark] class DAGScheduler(
     checkBarrierStageWithNumSlots(rdd, resourceProfile)
     checkBarrierStageWithRDDChainPattern(rdd, rdd.getNumPartitions)
     checkPipelinedProducerSupported(shuffleDep)
+    // Resolve the tracker that owns this shuffle's output, by dependency type 
(see
+    // outputTrackerMaster). Resolve it up front, BEFORE any stage-map 
mutation below, so that a
+    // fail-loud on a misconfigured pipelined shuffle (no 
StreamingShuffleOutputTracker) leaves no
+    // partial scheduler state and a re-submit re-throws the same error.
+    val outputTracker = outputTrackerMaster(shuffleDep)
     val numTasks = rdd.partitions.length
     val parents = getOrCreateParentStages(shuffleDeps, jobId)
     val id = nextStageId.getAndIncrement()
@@ -703,18 +708,51 @@ private[spark] class DAGScheduler(
     shuffleIdToMapStage(shuffleDep.shuffleId) = stage
     updateJobIdStageIdMaps(jobId, stage)
 
-    if (!mapOutputTracker.containsShuffle(shuffleDep.shuffleId)) {
-      // Kind of ugly: need to register RDDs with the cache and map output 
tracker here
-      // since we can't do it in the RDD constructor because # of partitions 
is unknown
+    // Register the shuffle with its own tracker (a pipelined shuffle in the
+    // StreamingShuffleOutputTracker, a regular one in the MapOutputTracker -- 
split by dependency
+    // type, no overlap). Self-guarded on the tracker's own membership: 
createShuffleMapStage runs
+    // once per shuffleId via getOrCreateShuffleMapStage, but guard 
defensively against re-entry.
+    // (A pipelined shuffle is never registered with the MapOutputTracker; its 
availability is
+    // tracked on the stage via pipelinedCompletedPartitions and its writers 
are located through the
+    // StreamingShuffleOutputTracker. jobId is used only by the streaming 
tracker.) The
+    // MapOutputTracker.getStatistics paths, which WOULD throw 
ShuffleStatusNotFoundException on the
+    // absent entry, are unreachable for a pipelined dependency: 
markMapStageJobsAsFinished calls it
+    // only when mapStageJobs is non-empty, but handleMapStageSubmitted 
rejects a pipelined dep
+    // before addActiveJob (its sole populator) runs, so a pipelined stage's 
mapStageJobs is always
+    // empty; and checkAndScheduleShuffleMergeFinalize's getStatistics is on 
the push-based-merge
+    // path, which a pipelined dependency rejects up front 
(checkPipelinedProducerSupported).
+    if (!outputTracker.containsShuffle(shuffleDep.shuffleId)) {
       logInfo(log"Registering RDD ${MDC(RDD_ID, rdd.id)} " +
         log"(${MDC(CREATION_SITE, rdd.getCreationSite)}) as input to " +
         log"shuffle ${MDC(SHUFFLE_ID, shuffleDep.shuffleId)}")
-      mapOutputTracker.registerShuffle(shuffleDep.shuffleId, 
rdd.partitions.length,
-        shuffleDep.partitioner.numPartitions)
+      outputTracker.registerShuffle(shuffleDep.shuffleId, 
rdd.partitions.length,
+        shuffleDep.partitioner.numPartitions, jobId)
     }
     stage
   }
 
+  /**
+   * The driver-side output tracker that owns a shuffle's outputs, selected by 
dependency type: the
+   * StreamingShuffleOutputTracker for a pipelined shuffle, the 
MapOutputTracker otherwise. The two
+   * are split with no overlap. A pipelined shuffle REQUIRES a 
StreamingShuffleOutputTracker
+   * (created with a streaming-capable shuffle manager, see
+   * SparkEnv.initializeStreamingShuffleOutputTracker), so fail loud if one is 
not configured rather
+   * than silently register it nowhere (a consumer would then find no writer 
locations; the reader
+   * enforces the same invariant, see StreamingShuffleReader).
+   */
+  private def outputTrackerMaster(
+      shuffleDep: ShuffleDependency[_, _, _]): ShuffleOutputTrackerMaster = {
+    if (shuffleDep.isInstanceOf[PipelinedShuffleDependency[_, _, _]]) {
+      sc.env.streamingShuffleOutputTracker
+        .getOrElse(throw new IllegalStateException(
+          s"A pipelined shuffle (id ${shuffleDep.shuffleId}) requires a " +
+            "StreamingShuffleOutputTracker, but none is configured"))
+        .asInstanceOf[StreamingShuffleOutputTrackerMaster]
+    } else {
+      mapOutputTracker
+    }
+  }
+
   private def pipelinedUnsupportedError(reason: String): 
PipelinedShuffleUnsupportedException =
     new PipelinedShuffleUnsupportedException(reason)
 
@@ -760,7 +798,7 @@ private[spark] class DAGScheduler(
     if (shuffleDep.checksumMismatchFullRetryEnabled) {
       throw pipelinedUnsupportedError("checksum-mismatch full retry with a 
pipelined shuffle")
     }
-    // Push-based shuffle merge as the pipelined shuffle: exposes output only 
after a
+    // Push-based shuffle merge on a pipelined shuffle: exposes output only 
after a
     // post-completion finalize step, the opposite of incremental reads. A
     // PipelinedShuffleDependency disables merge in its constructor, so this 
is a defensive
     // backstop against that being bypassed.
@@ -3046,8 +3084,17 @@ private[spark] class DAGScheduler(
     // will be re-executed.
     if (clearShuffle) {
       logInfo(log"Cleaning up shuffle for stage ${MDC(STAGE, sms)} to ensure 
re-execution")
-      mapOutputTracker.unregisterAllMapAndMergeOutput(sms.shuffleDep.shuffleId)
-      sms.shuffleDep.newShuffleMergeState()
+      // A pipelined shuffle is not registered with the MapOutputTracker, so 
unregistering there
+      // would throw ShuffleStatusNotFoundException. Not reachable today -- an 
indeterminate
+      // pipelined producer is rejected up front 
(checkPipelinedProducerSupported), and a job is
+      // all-regular or all-pipelined (mixed rejected), so a pipelined stage 
is never a succeeding
+      // stage of a regular indeterminate producer that rolls back -- but 
guard defensively, like
+      // the pipelined branch on the FetchFailed base path. (A transient 
pipelined producer cannot
+      // be rolled back and recomputed anyway; a genuine member failure fails 
the group, not this.)
+      if (!sms.isPipelined) {
+        
mapOutputTracker.unregisterAllMapAndMergeOutput(sms.shuffleDep.shuffleId)
+        sms.shuffleDep.newShuffleMergeState()
+      }
     }
   }
 
@@ -3357,8 +3404,8 @@ private[spark] class DAGScheduler(
             (isPipelinedGroupMember(failedStage) || 
isPipelinedGroupMember(mapStage))) {
           // Failure is group-atomic for a pipelined group. The base scheduler 
handles a
           // FetchFailed by resubmitting just the map stage in isolation and 
recomputing serially,
-          // but a transient pipelined shuffle cannot be re-read and its 
members are co-scheduled, so
-          // a lone-stage resubmit is never valid and would deadlock the 
group. Abort the
+          // but a transient pipelined shuffle cannot be re-read and its 
members are co-scheduled,
+          // so a lone-stage resubmit is never valid and would deadlock the 
group. Abort the
           // whole group instead: aborting the failed stage tears down its 
running co-scheduled
           // members and fails the job, and the caller (e.g. the streaming 
batch loop) reruns the
           // batch from scratch. This is distinct from the maxTaskFailures=1 
lever (which handles
@@ -3411,86 +3458,101 @@ private[spark] class DAGScheduler(
               "longer running")
           }
 
-          if (mapStage.rdd.isBarrier()) {
-            // Mark all the map as broken in the map stage, to ensure retry 
all the tasks on
-            // resubmitted stage attempt.
-            // TODO: SPARK-35547: Clean all push-based shuffle metadata like 
merge enabled and
-            // TODO: finalized as we are clearing all the merge results.
-            mapOutputTracker.unregisterAllMapAndMergeOutput(shuffleId)
-          } else if (mapIndex != -1) {
-            // Mark the map whose fetch failed as broken in the map stage
-            mapOutputTracker.unregisterMapOutput(shuffleId, mapIndex, 
bmAddress)
-            if (pushBasedShuffleEnabled) {
-              // Possibly unregister the merge result <shuffleId, reduceId>, 
if the FetchFailed
-              // mapIndex is part of the merge result of <shuffleId, reduceId>
-              mapOutputTracker.
-                unregisterMergeResult(shuffleId, reduceId, bmAddress, 
Option(mapIndex))
-            }
+          if (mapStage.isPipelined) {
+            // Defense-in-depth for a pipelined-shuffle FetchFailed that 
reaches this base path
+            // rather than the group-atomic branch above (e.g. a job whose 
hasPipelinedDependency
+            // flag was not propagated through the group check). The base path 
is invalid for a
+            // pipelined shuffle in two ways: (a) the MapOutputTracker 
invalidation below would
+            // throw ShuffleStatusNotFoundException (a pipelined shuffle is 
never registered there,
+            // see createShuffleMapStage); (b) the resubmit branch would 
enqueue a lone-stage
+            // resubmit of the transient producer, which -- as the 
group-atomic branch's comment
+            // explains -- is never valid and would deadlock the group. So 
abort the group here
+            // instead, matching the group-atomic branch's outcome, and skip 
both.
+            abortStage(failedStage,
+              s"A pipelined group member failed with a fetch failure: 
$failureMessage", None)
           } else {
-            // Unregister the merge result of <shuffleId, reduceId> if there 
is a FetchFailed event
-            // and is not a  MetaDataFetchException which is signified by 
bmAddress being null
-            if (bmAddress != null &&
-              
bmAddress.executorId.equals(BlockManagerId.SHUFFLE_MERGER_IDENTIFIER)) {
-              assert(pushBasedShuffleEnabled, "Push based shuffle expected to 
" +
-                "be enabled when handling merge block fetch failure.")
-              mapOutputTracker.
-                unregisterMergeResult(shuffleId, reduceId, bmAddress, None)
+            if (mapStage.rdd.isBarrier()) {
+              // Mark all the map as broken in the map stage, to ensure retry 
all the tasks on
+              // resubmitted stage attempt.
+              // TODO: SPARK-35547: Clean all push-based shuffle metadata like 
merge enabled and
+              // TODO: finalized as we are clearing all the merge results.
+              mapOutputTracker.unregisterAllMapAndMergeOutput(shuffleId)
+            } else if (mapIndex != -1) {
+              // Mark the map whose fetch failed as broken in the map stage
+              mapOutputTracker.unregisterMapOutput(shuffleId, mapIndex, 
bmAddress)
+              if (pushBasedShuffleEnabled) {
+                // Possibly unregister the merge result <shuffleId, reduceId>, 
if the FetchFailed
+                // mapIndex is part of the merge result of <shuffleId, 
reduceId>
+                mapOutputTracker.
+                  unregisterMergeResult(shuffleId, reduceId, bmAddress, 
Option(mapIndex))
+              }
+            } else {
+              // Unregister the merge result of <shuffleId, reduceId> if there 
is a FetchFailed
+              // event and is not a MetaDataFetchException (signified by 
bmAddress being null)
+              if (bmAddress != null &&
+                
bmAddress.executorId.equals(BlockManagerId.SHUFFLE_MERGER_IDENTIFIER)) {
+                assert(pushBasedShuffleEnabled, "Push based shuffle expected 
to " +
+                  "be enabled when handling merge block fetch failure.")
+                mapOutputTracker.
+                  unregisterMergeResult(shuffleId, reduceId, bmAddress, None)
+              }
             }
-          }
-
-          if (failedStage.rdd.isBarrier()) {
-            failedStage match {
-              case failedMapStage: ShuffleMapStage =>
-                // Mark all the map as broken in the map stage, to ensure 
retry all the tasks on
-                // resubmitted stage attempt.
-                
mapOutputTracker.unregisterAllMapAndMergeOutput(failedMapStage.shuffleDep.shuffleId)
 
-              case failedResultStage: ResultStage =>
-                // Abort the failed result stage since we may have committed 
output for some
-                // partitions.
-                val reason = "Could not recover from a failed barrier 
ResultStage. Most recent " +
-                  s"failure reason: $failureMessage"
-                abortStage(failedResultStage, reason, None)
+            if (failedStage.rdd.isBarrier()) {
+              failedStage match {
+                case failedMapStage: ShuffleMapStage =>
+                  // Mark all the map as broken in the map stage, to ensure 
retry all the tasks on
+                  // resubmitted stage attempt.
+                  mapOutputTracker.unregisterAllMapAndMergeOutput(
+                    failedMapStage.shuffleDep.shuffleId)
+
+                case failedResultStage: ResultStage =>
+                  // Abort the failed result stage since we may have committed 
output for some
+                  // partitions.
+                  val reason = "Could not recover from a failed barrier 
ResultStage. Most recent " +
+                    s"failure reason: $failureMessage"
+                  abortStage(failedResultStage, reason, None)
+              }
             }
-          }
 
-          if (shouldAbortStage) {
-            abortStage(failedStage, abortReason.get, None)
-          } else { // update failedStages and make sure a ResubmitFailedStages 
event is enqueued
-            // TODO: Cancel running tasks in the failed stage -- cf. 
SPARK-17064
-            val noResubmitEnqueued = !failedStages.contains(failedStage)
-            failedStages += failedStage
-            failedStages += mapStage
-            if (noResubmitEnqueued) {
-              // For statically indeterminate stages, trigger rollback early 
(here and in
-              // submitMissingTasks) rather than deferring to task completion. 
This is more
-              // efficient because it clears shuffle outputs before the retry 
is submitted,
-              // ensuring findMissingPartitions() returns all partitions.
-              //
-              // For runtime detection (checksum mismatch), rollback is 
triggered at task
-              // completion when the mismatch is discovered.
-              //
-              // The `rollbackCurrentStage = true` parameter ensures the 
failed map stage is
-              // included in the cleanup: clearing its shuffle outputs, 
marking old task results
-              // to be ignored, and creating a new shuffle merge state for the 
upcoming retry.
-              if (mapStage.isStaticallyIndeterminate &&
-                  !mapStage.shuffleDep.checksumMismatchFullRetryEnabled) {
-                rollbackSucceedingStages(mapStage, rollbackCurrentStage = true)
-              }
+            if (shouldAbortStage) {
+              abortStage(failedStage, abortReason.get, None)
+            } else { // update failedStages and make sure a 
ResubmitFailedStages event is enqueued
+              // TODO: Cancel running tasks in the failed stage -- cf. 
SPARK-17064
+              val noResubmitEnqueued = !failedStages.contains(failedStage)
+              failedStages += failedStage
+              failedStages += mapStage
+              if (noResubmitEnqueued) {
+                // For statically indeterminate stages, trigger rollback early 
(here and in
+                // submitMissingTasks) rather than deferring to task 
completion. This is more
+                // efficient because it clears shuffle outputs before the 
retry is submitted,
+                // ensuring findMissingPartitions() returns all partitions.
+                //
+                // For runtime detection (checksum mismatch), rollback is 
triggered at task
+                // completion when the mismatch is discovered.
+                //
+                // The `rollbackCurrentStage = true` parameter ensures the 
failed map stage is
+                // included in the cleanup: clearing its shuffle outputs, 
marking old task results
+                // to be ignored, and creating a new shuffle merge state for 
the upcoming retry.
+                if (mapStage.isStaticallyIndeterminate &&
+                    !mapStage.shuffleDep.checksumMismatchFullRetryEnabled) {
+                  rollbackSucceedingStages(mapStage, rollbackCurrentStage = 
true)
+                }
 
-              // We expect one executor failure to trigger many FetchFailures 
in rapid succession,
-              // but all of those task failures can typically be handled by a 
single resubmission of
-              // the failed stage.  We avoid flooding the scheduler's event 
queue with resubmit
-              // messages by checking whether a resubmit is already in the 
event queue for the
-              // failed stage.  If there is already a resubmit enqueued for a 
different failed
-              // stage, that event would also be sufficient to handle the 
current failed stage, but
-              // producing a resubmit for each failed stage makes debugging 
and logging a little
-              // simpler while not producing an overwhelming number of 
scheduler events.
-              logInfo(
-                log"Resubmitting ${MDC(STAGE, mapStage)} " +
-                log"(${MDC(STAGE_NAME, mapStage.name)}) and 
${MDC(FAILED_STAGE, failedStage)} " +
-                log"(${MDC(FAILED_STAGE_NAME, failedStage.name)}) due to fetch 
failure")
-              scheduleResubmit()
+                // We expect one executor failure to trigger many 
FetchFailures in rapid succession,
+                // but all of those task failures can typically be handled by 
a single resubmission
+                // of the failed stage.  We avoid flooding the scheduler's 
event queue with resubmit
+                // messages by checking whether a resubmit is already in the 
event queue for the
+                // failed stage.  If there is already a resubmit enqueued for 
a different failed
+                // stage, that event would also be sufficient to handle the 
current failed stage,
+                // but producing a resubmit for each failed stage makes 
debugging and logging a
+                // little simpler while not producing an overwhelming number 
of scheduler events.
+                logInfo(
+                  log"Resubmitting ${MDC(STAGE, mapStage)} " +
+                  log"(${MDC(STAGE_NAME, mapStage.name)}) and 
${MDC(FAILED_STAGE, failedStage)} " +
+                  log"(${MDC(FAILED_STAGE_NAME, failedStage.name)}) due to 
fetch failure")
+                scheduleResubmit()
+              }
             }
           }
 
@@ -4038,12 +4100,12 @@ private[spark] class DAGScheduler(
   /**
    * On a FetchFailed, unregister the shuffle outputs of the executor (or its 
whole host) whose
    * fetch failed, treating the FetchFailed as authoritative evidence that its 
shuffle data is gone.
-   * Extracted from the base FetchFailed handler so the pipelined-group-abort 
branch can also run it:
-   * aborting the group fails only this job's stages, but a dead executor's 
REGULAR outputs must
-   * still be cleaned up for other/concurrent jobs (with an external shuffle 
service, an ExecutorLost
-   * does not clean them, so FetchFailed is the only proactive channel). No-op 
when `bmAddress` is
-   * null. Safe for a pipelined shuffle: it registers no map outputs in the 
tracker, so this can only
-   * strip regular/durable outputs.
+   * Extracted from the base FetchFailed handler so the pipelined-group-abort 
branch can also run
+   * it: aborting the group fails only this job's stages, but a dead 
executor's REGULAR outputs must
+   * still be cleaned up for other/concurrent jobs (with an external shuffle 
service, an
+   * ExecutorLost does not clean them, so FetchFailed is the only proactive 
channel). No-op when
+   * `bmAddress` is null. Safe for a pipelined shuffle: it registers no map 
outputs in the tracker,
+   * so this can only strip regular/durable outputs.
    */
   private def unregisterOutputsOnFetchFailedExecutor(
       bmAddress: BlockManagerId, task: Task[_]): Unit = {
diff --git 
a/core/src/main/scala/org/apache/spark/scheduler/ShuffleMapStage.scala 
b/core/src/main/scala/org/apache/spark/scheduler/ShuffleMapStage.scala
index 94078d698cf4..11cec94f2b31 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/ShuffleMapStage.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/ShuffleMapStage.scala
@@ -64,13 +64,12 @@ private[spark] class ShuffleMapStage(
 
   /**
    * Availability tracking for a pipelined shuffle. A pipelined shuffle 
produces no durable,
-   * addressable map output. `createShuffleMapStage` still calls 
`MapOutputTracker.registerShuffle`
-   * for it (it is a `ShuffleDependency`, and an empty shuffle-status entry 
keeps the tracker's
-   * `containsShuffle` / `unregisterShuffle` bookkeeping uniform), but no map 
output is ever
-   * registered into that entry -- `registerMapOutput` runs only on the 
non-pipelined completion
-   * path -- so the entry stays empty. Its map-stage availability therefore 
cannot be read from the
-   * tracker; instead we track completed partitions here, monotonically: a 
partition is added when
-   * its map task succeeds and is NEVER removed on executor/host loss.
+   * addressable map output and is NOT registered with the `MapOutputTracker` 
at all --
+   * `createShuffleMapStage` registers it only with the 
`StreamingShuffleOutputTracker` (the two
+   * trackers are split by dependency type, with no overlap). Its map-stage 
availability therefore
+   * cannot be read from the `MapOutputTracker`; instead we track completed 
partitions here,
+   * monotonically: a partition is added when its map task succeeds and is 
NEVER removed on
+   * executor/host loss.
    *
    * This is the crux of avoiding the streaming-writer resubmit hang: if a 
pipelined shuffle's
    * availability were read from the `MapOutputTracker`, losing an executor 
that held a completed
@@ -110,8 +109,8 @@ private[spark] class ShuffleMapStage(
    * When this reaches [[numPartitions]], this map stage is ready.
    */
   def numAvailableOutputs: Int = {
-    // A pipelined shuffle's outputs are never populated in the 
MapOutputTracker (its entry stays
-    // empty; see pipelinedCompletedPartitions); read its locally-tracked, 
monotonic completed set.
+    // A pipelined shuffle is not registered with the MapOutputTracker at all 
(see
+    // pipelinedCompletedPartitions); read its locally-tracked, monotonic 
completed set instead.
     if (isPipelined) pipelinedCompletedPartitions.size
     else mapOutputTrackerMaster.getNumAvailableOutputs(shuffleDep.shuffleId)
   }
diff --git 
a/core/src/main/scala/org/apache/spark/shuffle/streaming/StreamingShuffleManager.scala
 
b/core/src/main/scala/org/apache/spark/shuffle/streaming/StreamingShuffleManager.scala
index 03ca257353c5..b83bd33552b9 100644
--- 
a/core/src/main/scala/org/apache/spark/shuffle/streaming/StreamingShuffleManager.scala
+++ 
b/core/src/main/scala/org/apache/spark/shuffle/streaming/StreamingShuffleManager.scala
@@ -113,8 +113,8 @@ private[spark] class StreamingShuffleManager extends 
PipelinedShuffleManager wit
 
   override def unregisterShuffle(shuffleId: Int): Boolean = {
     // No manager-side state to release here: the driver's 
StreamingShuffleOutputTracker is
-    // unregistered in BlockManagerStorageEndpoint's RemoveShuffle handler, 
and per-task writer
-    // and reader resources are released via task completion listeners.
+    // unregistered in ContextCleaner.doCleanupShuffle (its state is 
driver-only), and per-task
+    // writer and reader resources are released via task completion listeners.
     true
   }
 
diff --git a/core/src/test/scala/org/apache/spark/ContextCleanerSuite.scala 
b/core/src/test/scala/org/apache/spark/ContextCleanerSuite.scala
index bb2d7d5c4d8e..0bd0ee103a19 100644
--- a/core/src/test/scala/org/apache/spark/ContextCleanerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/ContextCleanerSuite.scala
@@ -20,6 +20,7 @@ package org.apache.spark
 import scala.collection.mutable.HashSet
 import scala.util.Random
 
+import org.apache.logging.log4j.Level
 import org.scalatest.BeforeAndAfter
 import org.scalatest.concurrent.Eventually._
 import org.scalatest.concurrent.PatienceConfiguration
@@ -127,6 +128,52 @@ class ContextCleanerSuite extends ContextCleanerSuiteBase {
     assert(rdd.collect().toList.equals(collected))
   }
 
+  test("cleanup shuffle unregisters a pipelined shuffle from the streaming 
output tracker") {
+    // A pipelined shuffle lives ONLY in the driver-only 
StreamingShuffleOutputTracker -- it is
+    // never registered with the MapOutputTracker (see 
DAGScheduler.createShuffleMapStage). So the
+    // MapOutputTracker.containsShuffle guard is false for it, and 
doCleanupShuffle must still clean
+    // it up via its own streaming branch; otherwise the tracker grows without 
bound across
+    // micro-batches in Real-Time Mode.
+    val streamingTracker = sc.env.streamingShuffleOutputTracker.get
+      .asInstanceOf[StreamingShuffleOutputTrackerMaster]
+    val mapOutputTracker = 
sc.env.mapOutputTracker.asInstanceOf[MapOutputTrackerMaster]
+
+    // Register a shuffle id in the streaming tracker ONLY, exactly as the 
scheduler now does for a
+    // pipelined shuffle (nothing is put in the MapOutputTracker).
+    val shuffleId = 1000
+    streamingTracker.registerShuffle(shuffleId, numMaps = 2, numReduces = 2, 
jobId = 0)
+    assert(streamingTracker.containsShuffle(shuffleId))
+    assert(!mapOutputTracker.containsShuffle(shuffleId),
+      "a pipelined shuffle must not be registered with the MapOutputTracker")
+
+    cleaner.doCleanupShuffle(shuffleId, blocking = true)
+
+    // The streaming tracker entry is gone -- cleanup fired even though the 
shuffle was never in the
+    // MapOutputTracker, proving the streaming cleanup branch is independent 
of MapOutputTracker.
+    assert(!streamingTracker.containsShuffle(shuffleId))
+  }
+
+  test("cleanup shuffle leaves a regular shuffle's streaming tracker untouched 
and quiet") {
+    // A regular (non-pipelined) shuffle is registered only with the 
MapOutputTracker, never the
+    // streaming tracker. doCleanupShuffle must take the MapOutputTracker 
branch and never touch the
+    // streaming tracker -- in particular it must not emit the streaming 
tracker's "attempting to
+    // unregister a shuffle that hasn't been registered" warning.
+    val streamingTracker = sc.env.streamingShuffleOutputTracker.get
+      .asInstanceOf[StreamingShuffleOutputTrackerMaster]
+
+    val (rdd, shuffleDeps) = newRDDWithShuffleDependencies()
+    rdd.collect()
+    shuffleDeps.foreach(s => 
assert(!streamingTracker.containsShuffle(s.shuffleId)))
+
+    val logAppender = new LogAppender("unregister streaming shuffle")
+    logAppender.setThreshold(Level.WARN)
+    withLogAppender(logAppender, level = Some(Level.WARN)) {
+      shuffleDeps.foreach(s => cleaner.doCleanupShuffle(s.shuffleId, blocking 
= true))
+    }
+    assert(!logAppender.loggingEvents.exists(
+      _.getMessage.getFormattedMessage.contains("hasn't been registered")))
+  }
+
   test("cleanup broadcast") {
     val broadcast = newBroadcast()
     val tester = new CleanerTester(sc, broadcastIds = Seq(broadcast.id))
@@ -174,6 +221,49 @@ class ContextCleanerSuite extends ContextCleanerSuiteBase {
     postGCTester.assertCleanup()
   }
 
+  test("automatically cleanup pipelined shuffle from the streaming output 
tracker") {
+    // The load-bearing leak guarantee for a pipelined (streaming) shuffle: it 
is registered ONLY
+    // with the driver-only StreamingShuffleOutputTracker (never the 
MapOutputTracker), and its only
+    // cleanup channel is the ContextCleaner weak-reference path fired when the
+    // PipelinedShuffleDependency is garbage-collected 
(registerShuffleForCleanup in the
+    // ShuffleDependency constructor). If that GC path did not reach the 
streaming tracker,
+    // the tracker would grow without bound across Real-Time Mode 
micro-batches. The sibling
+    // doCleanupShuffle tests call cleanup directly; this one proves the 
end-to-end
+    // GC -> ContextCleaner -> streaming-tracker unregister link.
+    val streamingTracker = sc.env.streamingShuffleOutputTracker.get
+      .asInstanceOf[StreamingShuffleOutputTrackerMaster]
+    val mapOutputTracker = 
sc.env.mapOutputTracker.asInstanceOf[MapOutputTrackerMaster]
+
+    // Build a real PipelinedShuffleDependency (registers itself for cleanup 
on construction) and
+    // register its shuffleId in the streaming tracker exactly as 
DAGScheduler.createShuffleMapStage
+    // does for a pipelined shuffle (nothing is put in the MapOutputTracker).
+    var pipelinedDep: PipelinedShuffleDependency[Int, Int, Int] =
+      new PipelinedShuffleDependency(newPairRDD(), new HashPartitioner(2))
+    val shuffleId = pipelinedDep.shuffleId
+    streamingTracker.registerShuffle(shuffleId, numMaps = 2, numReduces = 2, 
jobId = 0)
+    assert(streamingTracker.containsShuffle(shuffleId))
+    assert(!mapOutputTracker.containsShuffle(shuffleId),
+      "a pipelined shuffle must not be registered with the MapOutputTracker")
+
+    // A strong reference to the dependency must prevent GC-triggered cleanup.
+    runGC()
+    intercept[Exception] {
+      eventually(timeout(1.second), interval(100.milliseconds)) {
+        assert(!streamingTracker.containsShuffle(shuffleId),
+          "cleanup must NOT fire while the dependency is strongly referenced")
+      }
+    }
+
+    // Dereference the dependency; GC must then drive ContextCleaner to 
unregister it from the
+    // streaming tracker (and only there -- it was never in the 
MapOutputTracker).
+    pipelinedDep = null
+    runGC()
+    eventually(timeout(10.seconds), interval(100.milliseconds)) {
+      assert(!streamingTracker.containsShuffle(shuffleId),
+        "GC of the PipelinedShuffleDependency must unregister it from the 
streaming tracker")
+    }
+  }
+
   test("automatically cleanup broadcast") {
     var broadcast = newBroadcast()
 
diff --git 
a/core/src/test/scala/org/apache/spark/StreamingShuffleOutputTrackerSuite.scala 
b/core/src/test/scala/org/apache/spark/StreamingShuffleOutputTrackerSuite.scala
index 3c8d9b2f14e2..a9351630ad8d 100644
--- 
a/core/src/test/scala/org/apache/spark/StreamingShuffleOutputTrackerSuite.scala
+++ 
b/core/src/test/scala/org/apache/spark/StreamingShuffleOutputTrackerSuite.scala
@@ -270,6 +270,20 @@ class StreamingShuffleOutputTrackerSuite
     assert(tracker.getAllShuffleWriterTaskLocations(0).isEmpty)
   }
 
+  test("StreamingShuffleOutputTrackerMaster - containsShuffle reflects 
register/unregister") {
+    val conf = new SparkConf(false)
+    val tracker = newTrackerMaster(conf)
+
+    // A shuffle that was never registered is absent.
+    assert(!tracker.containsShuffle(0))
+
+    tracker.registerShuffle(shuffleId = 0, numMaps = 1, numReduces = 1, jobId 
= 1)
+    assert(tracker.containsShuffle(0))
+
+    tracker.unregisterShuffle(0)
+    assert(!tracker.containsShuffle(0))
+  }
+
   test("StreamingShuffleOutputTrackerMaster - register writer before shuffle 
fails") {
     val conf = new SparkConf(false)
     val tracker = newTrackerMaster(conf)
diff --git 
a/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala 
b/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
index 05ba990e2d63..a0428d8d8bd4 100644
--- a/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
@@ -7078,6 +7078,46 @@ class DAGSchedulerSuite extends SparkFunSuite with 
TempLocalSparkContext with Ti
       s"a dropped fan-in consumer must not replay when a surviving producer 
succeeds; got $results")
   }
 
+  test("pipelined shuffle: a fan-in group leaks no scheduler state when a 
producer fails and the " +
+      "job is torn down") {
+    // Leak check for the multi-producer (fan-in) deferral: a consumer 
deferred against TWO
+    // producers buffers its completions; when one producer fails and the job 
is torn down, ALL
+    // scheduler state -- the consumer's dependentStageMap deferral keyed on 
both producers --
+    // must be cleaned up. The sibling "dropped as soon as one producer fails" 
test stops at the
+    // drop-vs-replay outcome; this one drives the failure to full job 
teardown and asserts no leak.
+    val producerA = new MyRDD(sc, 2, Nil)
+    val psdA = new PipelinedShuffleDependency(producerA, new 
HashPartitioner(2))
+    val producerB = new MyRDD(sc, 2, Nil)
+    val psdB = new PipelinedShuffleDependency(producerB, new 
HashPartitioner(2))
+    val consumerRdd = new MyRDD(sc, 2, List(psdA, psdB), tracker = 
mapOutputTracker)
+    val failure = new java.util.concurrent.atomic.AtomicReference[Exception]()
+    val failListener = new JobListener {
+      override def taskSucceeded(index: Int, result: Any): Unit = 
results.put(index, result)
+      override def jobFailed(exception: Exception): Unit = 
failure.set(exception)
+    }
+    submit(consumerRdd, Array(0, 1), listener = failListener)
+    assert(taskSets.size === 3, s"A, B and the consumer must be co-scheduled; 
got ${taskSets.size}")
+    val consumerTaskSet =
+      taskSets.find(ts => scheduler.stageIdToStage(ts.stageId).rdd eq 
consumerRdd).get
+    val consumerStage = scheduler.stageIdToStage(consumerTaskSet.stageId)
+    val producerATaskSet =
+      taskSets.find(ts => scheduler.stageIdToStage(ts.stageId).rdd eq 
producerA).get
+
+    // Consumer finishes early -> buffered (deferred) against BOTH producers.
+    complete(consumerTaskSet, Seq((Success, 42), (Success, 43)))
+    assert(results.isEmpty, "consumer completions should be buffered while its 
producers run")
+    
assert(scheduler.dependentStageMap.get(consumerStage).exists(_.parents.size == 
2),
+      "the consumer must be deferred against both producers")
+
+    // Producer A fails its whole task set -> the job fails and the group is 
torn down.
+    failed(producerATaskSet, "producer A blew up")
+    assert(failure.get() != null, "the job must fail when a fan-in producer 
fails")
+    assert(results.isEmpty, "buffered consumer successes must be dropped, not 
applied, on failure")
+    sc.listenerBus.waitUntilEmpty(10000)
+    // The whole point: no leak. Every scheduler map (including the two-parent 
deferral) empties.
+    assertDataStructuresEmpty()
+  }
+
   test("pipelined shuffle: a multi-producer consumer's buffered successes are 
dropped when one " +
       "producer fails (no stale replay via a sibling)") {
     // Group-atomic drop across MULTIPLE producers. Consumer C reads TWO 
pipelined producers P1 and
diff --git 
a/core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala 
b/core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
index 4c7e8efc4eab..5fc7c842c54b 100644
--- a/core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
@@ -3139,8 +3139,9 @@ class TaskSetManagerSuite
     // The TaskSetManager.executorLost "Resubmitted" loop re-enqueues an 
already-successful
     // ShuffleMapTask when its executor is lost and the map output looks gone 
-- for a decommission,
     // it looks gone whenever MapOutputTracker.getMapOutputLocation returns 
None. A pipelined
-    // shuffle is NEVER registered in the MapOutputTracker, so 
getMapOutputLocation is always None
-    // and this loop would resubmit the lone producer task -- the exact 
streaming-writer hang. This
+    // shuffle is registered only with the StreamingShuffleOutputTracker, 
never the
+    // MapOutputTracker, so getMapOutputLocation is always None and this loop 
would resubmit the
+    // lone producer task -- the exact streaming-writer hang. This
     // loop bypasses handleFailedTask, so the group-atomic abort would never 
see it. The guard
     // (!taskSet.isPipelined on maybeShuffleMapOutputLoss) must keep a 
pipelined set out of the
     // loop.
diff --git 
a/core/src/test/scala/org/apache/spark/shuffle/PipelinedShuffleRoutingSuite.scala
 
b/core/src/test/scala/org/apache/spark/shuffle/PipelinedShuffleRoutingSuite.scala
index da74803a10eb..8e0f9e04c1bd 100644
--- 
a/core/src/test/scala/org/apache/spark/shuffle/PipelinedShuffleRoutingSuite.scala
+++ 
b/core/src/test/scala/org/apache/spark/shuffle/PipelinedShuffleRoutingSuite.scala
@@ -240,6 +240,46 @@ class PipelinedShuffleRoutingSuite extends SparkFunSuite 
with LocalSparkContext
     assert(SparkEnv.get.streamingShuffleOutputTracker.isEmpty)
   }
 
+  test("createShuffleMapStage fails loud for a pipelined dependency when no 
streaming tracker") {
+    // A pipelined shuffle is served solely by the 
StreamingShuffleOutputTracker. With a non-
+    // streaming incremental manager configured, that tracker is absent 
(verified above), so a job
+    // whose stage graph contains a PipelinedShuffleDependency cannot be 
scheduled.
+    // createShuffleMapStage must fail loud rather than silently register the 
shuffle in no tracker
+    // (which would strand a consumer with no writer locations). This guards 
the decoupling's
+    // invariant that a pipelined dependency implies a streaming tracker.
+    // local[4]: the pipelined group's up-front slot admission (producer 2 + 
result 2 = 4) must pass
+    // so that stage creation is actually reached -- otherwise the job is 
rejected for slots first.
+    sc = new SparkContext("local[4]", "test", newConf())
+    assert(SparkEnv.get.streamingShuffleOutputTracker.isEmpty)
+    val producer = sc.parallelize(1 to 4, 2).map(x => (x, x))
+    val pipelined = new PipelinedShuffleDependency[Int, Int, Int](producer, 
new HashPartitioner(2))
+    // A minimal reduce-side RDD whose single dependency is the pipelined 
shuffle, so submitting an
+    // action forces createShuffleMapStage for the pipelined producer.
+    val consumer = new RDD[(Int, Int)](sc, Seq(pipelined)) {
+      override def compute(split: Partition, ctx: TaskContext): Iterator[(Int, 
Int)] =
+        Iterator.empty
+      override protected def getPartitions: Array[Partition] =
+        Array.tabulate(2)(i => new Partition { override def index: Int = i })
+    }
+    val ex = intercept[Exception] {
+      consumer.count()
+    }
+    assert(findCause[IllegalStateException](ex).exists(
+      _.getMessage.contains("requires a StreamingShuffleOutputTracker")),
+      s"expected a fail-loud IllegalStateException about the missing tracker, 
got: $ex")
+
+    // The fail-loud throw happens BEFORE createShuffleMapStage mutates 
stageIdToStage /
+    // shuffleIdToMapStage, so it leaves no partial scheduler state behind. If 
it left a
+    // half-created stage cached in shuffleIdToMapStage, a re-submission would 
reuse that stale
+    // stage instead of re-throwing. Re-submitting the same job must therefore 
re-throw the error.
+    val ex2 = intercept[Exception] {
+      consumer.count()
+    }
+    assert(findCause[IllegalStateException](ex2).exists(
+      _.getMessage.contains("requires a StreamingShuffleOutputTracker")),
+      s"a re-submission must re-throw the fail-loud error (no leaked partial 
stage), got: $ex2")
+  }
+
   test("spark.shuffle.manager.incremental resolves the same short aliases as 
the default manager") {
     // "sort" is a short alias the incremental slot must resolve (to 
SortShuffleManager) rather than
     // treat as a class name. SortShuffleManager is blocking, so it is 
rejected from the pipelined


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to