This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 781b7edc675e [SPARK-58454][CORE] Fully decouple pipelined shuffles
from the MapOutputTracker
781b7edc675e is described below
commit 781b7edc675e32500327f6ff193494b3ad545a96
Author: Boyang Jerry Peng <[email protected]>
AuthorDate: Mon Aug 3 09:46:07 2026 +0800
[SPARK-58454][CORE] Fully decouple pipelined shuffles from the
MapOutputTracker
### What changes were proposed in this pull request?
This PR fully decouples a **pipelined (streaming) shuffle** from the
`MapOutputTracker` (MOT). It is part of the pipelined-shuffle stack and builds
on top of #57361 (group-atomic failure + fail-fast rejection).
Background: a `PipelinedShuffleDependency` produces a transient,
once-through stream and has no durable, addressable map output. Up to and
including #57361, such a shuffle was still *registered* in the
`MapOutputTracker` as an empty shell entry (so `containsShuffle` /
`unregisterShuffle` bookkeeping stayed uniform), and only its map outputs were
never populated. This PR removes that vestigial coupling: a pipelined shuffle
is now served **solely** by the driver-only `StreamingShuffleOu [...]
Concretely:
- **Registration (`DAGScheduler.createShuffleMapStage`)**: a pipelined
shuffle is registered only with the `StreamingShuffleOutputTracker`; a regular
shuffle registers with the `MapOutputTracker` exactly as before. If a pipelined
dependency is created but no streaming tracker is configured, creation now
**fails loud before mutating any scheduler state** (rather than leaving a
half-created stage cached), so a re-submission re-throws the same error instead
of reusing stale state.
- **Availability (`ShuffleMapStage`)**: a pipelined stage's
`numAvailableOutputs` / `findMissingPartitions` / `isAvailable` are read from
the stage-local, monotonic `pipelinedCompletedPartitions` set (a partition is
added on map-task success and never removed on executor/host loss), not from
the MOT. This is what avoids the streaming-writer resubmit hang.
- **Cleanup (`ContextCleaner`)**: `doCleanupShuffle` now cleans up
whichever tracker holds the shuffle. The two branches are independent, each
keyed on its own tracker's membership. For a pipelined shuffle the branch calls
`shuffleDriverComponents.removeShuffle` (balancing the `registerShuffle` the
`ShuffleDependency` constructor issues for every shuffle -- the default
implementation turns this into a `RemoveShuffle` RPC, which finds no blocks
since a pipelined shuffle has none) and t [...]
- **Every MOT call site keyed on a pipelined `shuffleId`** (FetchFailed
invalidation, indeterminate-stage rollback in `rollbackShuffleMapStage`, merge
finalization, `getStatistics`) is now either type-guarded (`!isPipelined`),
inside a non-pipelined branch, or unreachable because the shape is rejected up
front (barrier / push-merge / map-stage-job / indeterminate producer). None can
reach a `getShuffleStatusOrError` throw for an absent pipelined entry.
Note: this PR does **not** activate pipelined shuffles for any query -- no
production code constructs a `PipelinedShuffleDependency` yet. Activation
(wiring `ShuffleExchangeExec` and Real-Time Mode) is the follow-up PR stacked
on this one.
### Why are the changes needed?
Keeping a pipelined shuffle in the `MapOutputTracker` -- even as an empty
shell -- left latent coupling that is unsound once the shuffle is transient:
any MOT path that invalidates or reads outputs by `shuffleId` (executor-loss
stripping, indeterminate rollback, FetchFailed handling) could either strip a
completed pipelined output (triggering an invalid producer resubmit that hangs
the streaming writer) or throw `ShuffleStatusNotFoundException` on the
scheduler event thread once regis [...]
### Does this PR introduce _any_ user-facing change?
No. There is no production path that creates a pipelined shuffle yet; these
changes are internal to the scheduler and shuffle-tracking layers.
### How was this patch tested?
New and existing unit tests under `core`:
- `ContextCleanerSuite`: cleanup unregisters a pipelined shuffle from the
streaming tracker; cleanup of a regular shuffle leaves the streaming tracker
untouched.
- `StreamingShuffleOutputTrackerSuite`: `containsShuffle` reflects
register/unregister.
- `PipelinedShuffleRoutingSuite`: `createShuffleMapStage` fails loud
(leaving no partial scheduler state) when a pipelined dependency is created
with no streaming tracker configured.
- `DAGSchedulerSuite`: existing pipelined group-atomic and FetchFailed
tests continue to pass with the MOT fully decoupled.
`build/sbt "core/testOnly ... "` (core main + test compile clean;
scalastyle clean).
### Was this patch authored or co-authored using generative AI tooling?
Co-authored with Claude Code (Opus 4.8)
Closes #57646 from jerrypeng/stack/pipelined-shuffle-pr7-mot-decouple.
Authored-by: Boyang Jerry Peng <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit d37297d1b133baa73df5a4e5d41c80d66226cc0e)
Signed-off-by: Wenchen Fan <[email protected]>
---
.../scala/org/apache/spark/ContextCleaner.scala | 19 ++
.../scala/org/apache/spark/MapOutputTracker.scala | 10 +-
.../spark/StreamingShuffleOutputTracker.scala | 23 +-
.../org/apache/spark/scheduler/DAGScheduler.scala | 244 +++++++++++++--------
.../apache/spark/scheduler/ShuffleMapStage.scala | 17 +-
.../streaming/StreamingShuffleManager.scala | 4 +-
.../org/apache/spark/ContextCleanerSuite.scala | 90 ++++++++
.../spark/StreamingShuffleOutputTrackerSuite.scala | 14 ++
.../apache/spark/scheduler/DAGSchedulerSuite.scala | 40 ++++
.../spark/scheduler/TaskSetManagerSuite.scala | 5 +-
.../shuffle/PipelinedShuffleRoutingSuite.scala | 40 ++++
11 files changed, 397 insertions(+), 109 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/ContextCleaner.scala
b/core/src/main/scala/org/apache/spark/ContextCleaner.scala
index 0b3c22a22cb4..6c1b49157cc0 100644
--- a/core/src/main/scala/org/apache/spark/ContextCleaner.scala
+++ b/core/src/main/scala/org/apache/spark/ContextCleaner.scala
@@ -236,6 +236,10 @@ private[spark] class ContextCleaner(
/** Perform shuffle cleanup. */
def doCleanupShuffle(shuffleId: Int, blocking: Boolean): Unit = {
try {
+ // A shuffle lives in exactly one tracker, split by dependency type: a
regular shuffle in the
+ // MapOutputTracker, a pipelined shuffle in the driver-only
StreamingShuffleOutputTracker (see
+ // DAGScheduler.createShuffleMapStage). Clean up whichever holds it --
the two branches are
+ // independent, each keyed on its own tracker's membership, so neither
depends on the other.
if (mapOutputTrackerMaster.containsShuffle(shuffleId)) {
logDebug("Cleaning shuffle " + shuffleId)
// Shuffle must be removed before it's unregistered from the output
tracker
@@ -244,6 +248,19 @@ private[spark] class ContextCleaner(
mapOutputTrackerMaster.unregisterShuffle(shuffleId)
listeners.asScala.foreach(_.shuffleCleaned(shuffleId))
logDebug("Cleaned shuffle " + shuffleId)
+ } else if
(streamingShuffleOutputTrackerMaster.exists(_.containsShuffle(shuffleId))) {
+ // A pipelined shuffle's output state is driver-only (the worker
tracker caches nothing), so
+ // it is unregistered directly from the StreamingShuffleOutputTracker
rather than the
+ // MapOutputTracker. Still call shuffleDriverComponents.removeShuffle
to balance the
+ // registerShuffle that the ShuffleDependency constructor issues for
EVERY shuffle (incl. a
+ // pipelined one): a custom components impl may track per-shuffle
driver state that would
+ // otherwise leak. For the default impl this is just a RemoveShuffle
RPC that finds no
+ // blocks (a pipelined shuffle has none), matching the regular
branch's ordering.
+ logDebug("Cleaning pipelined shuffle " + shuffleId)
+ shuffleDriverComponents.removeShuffle(shuffleId, blocking)
+
streamingShuffleOutputTrackerMaster.foreach(_.unregisterShuffle(shuffleId))
+ listeners.asScala.foreach(_.shuffleCleaned(shuffleId))
+ logDebug("Cleaned pipelined shuffle " + shuffleId)
} else {
logDebug("Asked to cleanup non-existent shuffle (maybe it was already
removed)")
}
@@ -308,6 +325,8 @@ private[spark] class ContextCleaner(
private def broadcastManager = sc.env.broadcastManager
private def mapOutputTrackerMaster =
sc.env.mapOutputTracker.asInstanceOf[MapOutputTrackerMaster]
+ private def streamingShuffleOutputTrackerMaster:
Option[StreamingShuffleOutputTrackerMaster] =
+
sc.env.streamingShuffleOutputTracker.map(_.asInstanceOf[StreamingShuffleOutputTrackerMaster])
}
private object ContextCleaner {
diff --git a/core/src/main/scala/org/apache/spark/MapOutputTracker.scala
b/core/src/main/scala/org/apache/spark/MapOutputTracker.scala
index 742baf40ecbf..933da9477272 100644
--- a/core/src/main/scala/org/apache/spark/MapOutputTracker.scala
+++ b/core/src/main/scala/org/apache/spark/MapOutputTracker.scala
@@ -799,7 +799,7 @@ private[spark] class MapOutputTrackerMaster(
conf: SparkConf,
private[spark] val broadcastManager: BroadcastManager,
private[spark] val isLocal: Boolean)
- extends MapOutputTracker(conf) {
+ extends MapOutputTracker(conf) with ShuffleOutputTrackerMaster {
// The size at which we use Broadcast to send the map output statuses to the
executors
private val minSizeForBroadcast =
conf.get(SHUFFLE_MAPOUTPUT_MIN_SIZE_FOR_BROADCAST).toInt
@@ -943,6 +943,10 @@ private[spark] class MapOutputTrackerMaster(
}
}
+ // ShuffleOutputTrackerMaster: a regular shuffle has no per-job
registration, so jobId is ignored.
+ override def registerShuffle(shuffleId: Int, numMaps: Int, numReduces: Int,
jobId: Int): Unit =
+ registerShuffle(shuffleId, numMaps, numReduces)
+
def updateMapOutput(shuffleId: Int, mapId: Long, bmAddress: BlockManagerId):
Unit = {
shuffleStatuses.get(shuffleId) match {
case Some(shuffleStatus) =>
@@ -1030,7 +1034,7 @@ private[spark] class MapOutputTrackerMaster(
}
/** Unregister shuffle data */
- def unregisterShuffle(shuffleId: Int): Unit = {
+ override def unregisterShuffle(shuffleId: Int): Unit = {
shuffleStatuses.remove(shuffleId).foreach { shuffleStatus =>
shuffleStatus.invalidateSerializedMapOutputStatusCache()
shuffleStatus.invalidateSerializedMergeOutputStatusCache()
@@ -1077,7 +1081,7 @@ private[spark] class MapOutputTrackerMaster(
}
/** Check if the given shuffle is being tracked */
- def containsShuffle(shuffleId: Int): Boolean =
shuffleStatuses.contains(shuffleId)
+ override def containsShuffle(shuffleId: Int): Boolean =
shuffleStatuses.contains(shuffleId)
def getNumAvailableOutputs(shuffleId: Int): Int = {
shuffleStatuses.get(shuffleId).map(_.numAvailableMapOutputs).getOrElse(0)
diff --git
a/core/src/main/scala/org/apache/spark/StreamingShuffleOutputTracker.scala
b/core/src/main/scala/org/apache/spark/StreamingShuffleOutputTracker.scala
index 090be7f691d4..df4e76b017db 100644
--- a/core/src/main/scala/org/apache/spark/StreamingShuffleOutputTracker.scala
+++ b/core/src/main/scala/org/apache/spark/StreamingShuffleOutputTracker.scala
@@ -28,6 +28,23 @@ import
org.apache.spark.internal.config.SHUFFLE_MAPOUTPUT_DISPATCHER_NUM_THREADS
import org.apache.spark.rpc.{RpcCallContext, RpcEndpoint, RpcEndpointRef,
RpcEnv}
import org.apache.spark.util.ThreadUtils
+/**
+ * The driver-side registry for a shuffle's output. A regular shuffle is
served by the
+ * `MapOutputTrackerMaster`, a pipelined (streaming) shuffle by the
+ * `StreamingShuffleOutputTrackerMaster` -- split by dependency type, with no
overlap. This is the
+ * small common surface the DAGScheduler and ContextCleaner drive
polymorphically, so they select
+ * the right tracker by shuffle-dependency type (see
`DAGScheduler.outputTrackerMaster`) rather than
+ * special-casing pipelined shuffles at each call site.
+ */
+private[spark] trait ShuffleOutputTrackerMaster {
+ /** Register a shuffle so its outputs can be tracked. `jobId` is used by the
streaming tracker. */
+ def registerShuffle(shuffleId: Int, numMaps: Int, numReduces: Int, jobId:
Int): Unit
+ /** Whether the given shuffle is registered with this tracker. */
+ def containsShuffle(shuffleId: Int): Boolean
+ /** Unregister a shuffle and release its tracked state. */
+ def unregisterShuffle(shuffleId: Int): Unit
+}
+
private[spark] sealed trait StreamingShuffleTaskLocationTrackerMessage
private[spark] case class UpdateStreamingShuffleTaskLocation(
@@ -196,7 +213,7 @@ private[spark] abstract class
StreamingShuffleOutputTracker(conf: SparkConf) ext
private[spark] case class StreamingShuffleInfo(numMaps: Int, numReduces: Int,
jobId: Int)
private[spark] class StreamingShuffleOutputTrackerMaster(conf: SparkConf)
- extends StreamingShuffleOutputTracker(conf) {
+ extends StreamingShuffleOutputTracker(conf) with ShuffleOutputTrackerMaster {
// map that stores task location information organized in the following
fashion
// shuffle id -> {mapId -> location}
@@ -220,7 +237,7 @@ private[spark] class
StreamingShuffleOutputTrackerMaster(conf: SparkConf)
pool
}
- def registerShuffle(shuffleId: Int, numMaps: Int, numReduces: Int, jobId:
Int): Unit = {
+ override def registerShuffle(shuffleId: Int, numMaps: Int, numReduces: Int,
jobId: Int): Unit = {
logInfo(log"Registering shuffleId ${MDC(LogKeys.SHUFFLE_ID, shuffleId)}
with ${
MDC(LogKeys.NUM_MAPPERS, numMaps)} mappers and ${
MDC(LogKeys.NUM_REDUCERS, numReduces)} reducers")
@@ -230,6 +247,8 @@ private[spark] class
StreamingShuffleOutputTrackerMaster(conf: SparkConf)
}
}
+ override def containsShuffle(shuffleId: Int): Boolean =
shuffleInfos.containsKey(shuffleId)
+
// for testing purposes
private[spark] def getShuffleInfo(shuffleId: Int):
Option[StreamingShuffleInfo] = {
Option(shuffleInfos.get(shuffleId))
diff --git a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
index a72af257ad73..05c142558a43 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
@@ -188,7 +188,7 @@ private[spark] class DAGScheduler(
private[scheduler] val dependentStageMap = new HashMap[Stage,
DependentStageInfo]
// Whether we have already logged that the pipelined-group slot check is
disabled. Only the
- // (single-threaded) event loop touches this, so a plain var is safe. Keeps
the warning to once
+ // (single-threaded) event loop touches this, so a plain var is safe. Limits
the warning to once
// per scheduler rather than once per submitted batch job.
private var warnedPipelinedSlotCheckDisabled = false
@@ -692,6 +692,11 @@ private[spark] class DAGScheduler(
checkBarrierStageWithNumSlots(rdd, resourceProfile)
checkBarrierStageWithRDDChainPattern(rdd, rdd.getNumPartitions)
checkPipelinedProducerSupported(shuffleDep)
+ // Resolve the tracker that owns this shuffle's output, by dependency type
(see
+ // outputTrackerMaster). Resolve it up front, BEFORE any stage-map
mutation below, so that a
+ // fail-loud on a misconfigured pipelined shuffle (no
StreamingShuffleOutputTracker) leaves no
+ // partial scheduler state and a re-submit re-throws the same error.
+ val outputTracker = outputTrackerMaster(shuffleDep)
val numTasks = rdd.partitions.length
val parents = getOrCreateParentStages(shuffleDeps, jobId)
val id = nextStageId.getAndIncrement()
@@ -703,18 +708,51 @@ private[spark] class DAGScheduler(
shuffleIdToMapStage(shuffleDep.shuffleId) = stage
updateJobIdStageIdMaps(jobId, stage)
- if (!mapOutputTracker.containsShuffle(shuffleDep.shuffleId)) {
- // Kind of ugly: need to register RDDs with the cache and map output
tracker here
- // since we can't do it in the RDD constructor because # of partitions
is unknown
+ // Register the shuffle with its own tracker (a pipelined shuffle in the
+ // StreamingShuffleOutputTracker, a regular one in the MapOutputTracker --
split by dependency
+ // type, no overlap). Self-guarded on the tracker's own membership:
createShuffleMapStage runs
+ // once per shuffleId via getOrCreateShuffleMapStage, but guard
defensively against re-entry.
+ // (A pipelined shuffle is never registered with the MapOutputTracker; its
availability is
+ // tracked on the stage via pipelinedCompletedPartitions and its writers
are located through the
+ // StreamingShuffleOutputTracker. jobId is used only by the streaming
tracker.) The
+ // MapOutputTracker.getStatistics paths, which WOULD throw
ShuffleStatusNotFoundException on the
+ // absent entry, are unreachable for a pipelined dependency:
markMapStageJobsAsFinished calls it
+ // only when mapStageJobs is non-empty, but handleMapStageSubmitted
rejects a pipelined dep
+ // before addActiveJob (its sole populator) runs, so a pipelined stage's
mapStageJobs is always
+ // empty; and checkAndScheduleShuffleMergeFinalize's getStatistics is on
the push-based-merge
+ // path, which a pipelined dependency rejects up front
(checkPipelinedProducerSupported).
+ if (!outputTracker.containsShuffle(shuffleDep.shuffleId)) {
logInfo(log"Registering RDD ${MDC(RDD_ID, rdd.id)} " +
log"(${MDC(CREATION_SITE, rdd.getCreationSite)}) as input to " +
log"shuffle ${MDC(SHUFFLE_ID, shuffleDep.shuffleId)}")
- mapOutputTracker.registerShuffle(shuffleDep.shuffleId,
rdd.partitions.length,
- shuffleDep.partitioner.numPartitions)
+ outputTracker.registerShuffle(shuffleDep.shuffleId,
rdd.partitions.length,
+ shuffleDep.partitioner.numPartitions, jobId)
}
stage
}
+ /**
+ * The driver-side output tracker that owns a shuffle's outputs, selected by
dependency type: the
+ * StreamingShuffleOutputTracker for a pipelined shuffle, the
MapOutputTracker otherwise. The two
+ * are split with no overlap. A pipelined shuffle REQUIRES a
StreamingShuffleOutputTracker
+ * (created with a streaming-capable shuffle manager, see
+ * SparkEnv.initializeStreamingShuffleOutputTracker), so fail loud if one is
not configured rather
+ * than silently register it nowhere (a consumer would then find no writer
locations; the reader
+ * enforces the same invariant, see StreamingShuffleReader).
+ */
+ private def outputTrackerMaster(
+ shuffleDep: ShuffleDependency[_, _, _]): ShuffleOutputTrackerMaster = {
+ if (shuffleDep.isInstanceOf[PipelinedShuffleDependency[_, _, _]]) {
+ sc.env.streamingShuffleOutputTracker
+ .getOrElse(throw new IllegalStateException(
+ s"A pipelined shuffle (id ${shuffleDep.shuffleId}) requires a " +
+ "StreamingShuffleOutputTracker, but none is configured"))
+ .asInstanceOf[StreamingShuffleOutputTrackerMaster]
+ } else {
+ mapOutputTracker
+ }
+ }
+
private def pipelinedUnsupportedError(reason: String):
PipelinedShuffleUnsupportedException =
new PipelinedShuffleUnsupportedException(reason)
@@ -760,7 +798,7 @@ private[spark] class DAGScheduler(
if (shuffleDep.checksumMismatchFullRetryEnabled) {
throw pipelinedUnsupportedError("checksum-mismatch full retry with a
pipelined shuffle")
}
- // Push-based shuffle merge as the pipelined shuffle: exposes output only
after a
+ // Push-based shuffle merge on a pipelined shuffle: exposes output only
after a
// post-completion finalize step, the opposite of incremental reads. A
// PipelinedShuffleDependency disables merge in its constructor, so this
is a defensive
// backstop against that being bypassed.
@@ -3046,8 +3084,17 @@ private[spark] class DAGScheduler(
// will be re-executed.
if (clearShuffle) {
logInfo(log"Cleaning up shuffle for stage ${MDC(STAGE, sms)} to ensure
re-execution")
- mapOutputTracker.unregisterAllMapAndMergeOutput(sms.shuffleDep.shuffleId)
- sms.shuffleDep.newShuffleMergeState()
+ // A pipelined shuffle is not registered with the MapOutputTracker, so
unregistering there
+ // would throw ShuffleStatusNotFoundException. Not reachable today -- an
indeterminate
+ // pipelined producer is rejected up front
(checkPipelinedProducerSupported), and a job is
+ // all-regular or all-pipelined (mixed rejected), so a pipelined stage
is never a succeeding
+ // stage of a regular indeterminate producer that rolls back -- but
guard defensively, like
+ // the pipelined branch on the FetchFailed base path. (A transient
pipelined producer cannot
+ // be rolled back and recomputed anyway; a genuine member failure fails
the group, not this.)
+ if (!sms.isPipelined) {
+
mapOutputTracker.unregisterAllMapAndMergeOutput(sms.shuffleDep.shuffleId)
+ sms.shuffleDep.newShuffleMergeState()
+ }
}
}
@@ -3357,8 +3404,8 @@ private[spark] class DAGScheduler(
(isPipelinedGroupMember(failedStage) ||
isPipelinedGroupMember(mapStage))) {
// Failure is group-atomic for a pipelined group. The base scheduler
handles a
// FetchFailed by resubmitting just the map stage in isolation and
recomputing serially,
- // but a transient pipelined shuffle cannot be re-read and its
members are co-scheduled, so
- // a lone-stage resubmit is never valid and would deadlock the
group. Abort the
+ // but a transient pipelined shuffle cannot be re-read and its
members are co-scheduled,
+ // so a lone-stage resubmit is never valid and would deadlock the
group. Abort the
// whole group instead: aborting the failed stage tears down its
running co-scheduled
// members and fails the job, and the caller (e.g. the streaming
batch loop) reruns the
// batch from scratch. This is distinct from the maxTaskFailures=1
lever (which handles
@@ -3411,86 +3458,101 @@ private[spark] class DAGScheduler(
"longer running")
}
- if (mapStage.rdd.isBarrier()) {
- // Mark all the map as broken in the map stage, to ensure retry
all the tasks on
- // resubmitted stage attempt.
- // TODO: SPARK-35547: Clean all push-based shuffle metadata like
merge enabled and
- // TODO: finalized as we are clearing all the merge results.
- mapOutputTracker.unregisterAllMapAndMergeOutput(shuffleId)
- } else if (mapIndex != -1) {
- // Mark the map whose fetch failed as broken in the map stage
- mapOutputTracker.unregisterMapOutput(shuffleId, mapIndex,
bmAddress)
- if (pushBasedShuffleEnabled) {
- // Possibly unregister the merge result <shuffleId, reduceId>,
if the FetchFailed
- // mapIndex is part of the merge result of <shuffleId, reduceId>
- mapOutputTracker.
- unregisterMergeResult(shuffleId, reduceId, bmAddress,
Option(mapIndex))
- }
+ if (mapStage.isPipelined) {
+ // Defense-in-depth for a pipelined-shuffle FetchFailed that
reaches this base path
+ // rather than the group-atomic branch above (e.g. a job whose
hasPipelinedDependency
+ // flag was not propagated through the group check). The base path
is invalid for a
+ // pipelined shuffle in two ways: (a) the MapOutputTracker
invalidation below would
+ // throw ShuffleStatusNotFoundException (a pipelined shuffle is
never registered there,
+ // see createShuffleMapStage); (b) the resubmit branch would
enqueue a lone-stage
+ // resubmit of the transient producer, which -- as the
group-atomic branch's comment
+ // explains -- is never valid and would deadlock the group. So
abort the group here
+ // instead, matching the group-atomic branch's outcome, and skip
both.
+ abortStage(failedStage,
+ s"A pipelined group member failed with a fetch failure:
$failureMessage", None)
} else {
- // Unregister the merge result of <shuffleId, reduceId> if there
is a FetchFailed event
- // and is not a MetaDataFetchException which is signified by
bmAddress being null
- if (bmAddress != null &&
-
bmAddress.executorId.equals(BlockManagerId.SHUFFLE_MERGER_IDENTIFIER)) {
- assert(pushBasedShuffleEnabled, "Push based shuffle expected to
" +
- "be enabled when handling merge block fetch failure.")
- mapOutputTracker.
- unregisterMergeResult(shuffleId, reduceId, bmAddress, None)
+ if (mapStage.rdd.isBarrier()) {
+ // Mark all the map as broken in the map stage, to ensure retry
all the tasks on
+ // resubmitted stage attempt.
+ // TODO: SPARK-35547: Clean all push-based shuffle metadata like
merge enabled and
+ // TODO: finalized as we are clearing all the merge results.
+ mapOutputTracker.unregisterAllMapAndMergeOutput(shuffleId)
+ } else if (mapIndex != -1) {
+ // Mark the map whose fetch failed as broken in the map stage
+ mapOutputTracker.unregisterMapOutput(shuffleId, mapIndex,
bmAddress)
+ if (pushBasedShuffleEnabled) {
+ // Possibly unregister the merge result <shuffleId, reduceId>,
if the FetchFailed
+ // mapIndex is part of the merge result of <shuffleId,
reduceId>
+ mapOutputTracker.
+ unregisterMergeResult(shuffleId, reduceId, bmAddress,
Option(mapIndex))
+ }
+ } else {
+ // Unregister the merge result of <shuffleId, reduceId> if there
is a FetchFailed
+ // event and is not a MetaDataFetchException (signified by
bmAddress being null)
+ if (bmAddress != null &&
+
bmAddress.executorId.equals(BlockManagerId.SHUFFLE_MERGER_IDENTIFIER)) {
+ assert(pushBasedShuffleEnabled, "Push based shuffle expected
to " +
+ "be enabled when handling merge block fetch failure.")
+ mapOutputTracker.
+ unregisterMergeResult(shuffleId, reduceId, bmAddress, None)
+ }
}
- }
-
- if (failedStage.rdd.isBarrier()) {
- failedStage match {
- case failedMapStage: ShuffleMapStage =>
- // Mark all the map as broken in the map stage, to ensure
retry all the tasks on
- // resubmitted stage attempt.
-
mapOutputTracker.unregisterAllMapAndMergeOutput(failedMapStage.shuffleDep.shuffleId)
- case failedResultStage: ResultStage =>
- // Abort the failed result stage since we may have committed
output for some
- // partitions.
- val reason = "Could not recover from a failed barrier
ResultStage. Most recent " +
- s"failure reason: $failureMessage"
- abortStage(failedResultStage, reason, None)
+ if (failedStage.rdd.isBarrier()) {
+ failedStage match {
+ case failedMapStage: ShuffleMapStage =>
+ // Mark all the map as broken in the map stage, to ensure
retry all the tasks on
+ // resubmitted stage attempt.
+ mapOutputTracker.unregisterAllMapAndMergeOutput(
+ failedMapStage.shuffleDep.shuffleId)
+
+ case failedResultStage: ResultStage =>
+ // Abort the failed result stage since we may have committed
output for some
+ // partitions.
+ val reason = "Could not recover from a failed barrier
ResultStage. Most recent " +
+ s"failure reason: $failureMessage"
+ abortStage(failedResultStage, reason, None)
+ }
}
- }
- if (shouldAbortStage) {
- abortStage(failedStage, abortReason.get, None)
- } else { // update failedStages and make sure a ResubmitFailedStages
event is enqueued
- // TODO: Cancel running tasks in the failed stage -- cf.
SPARK-17064
- val noResubmitEnqueued = !failedStages.contains(failedStage)
- failedStages += failedStage
- failedStages += mapStage
- if (noResubmitEnqueued) {
- // For statically indeterminate stages, trigger rollback early
(here and in
- // submitMissingTasks) rather than deferring to task completion.
This is more
- // efficient because it clears shuffle outputs before the retry
is submitted,
- // ensuring findMissingPartitions() returns all partitions.
- //
- // For runtime detection (checksum mismatch), rollback is
triggered at task
- // completion when the mismatch is discovered.
- //
- // The `rollbackCurrentStage = true` parameter ensures the
failed map stage is
- // included in the cleanup: clearing its shuffle outputs,
marking old task results
- // to be ignored, and creating a new shuffle merge state for the
upcoming retry.
- if (mapStage.isStaticallyIndeterminate &&
- !mapStage.shuffleDep.checksumMismatchFullRetryEnabled) {
- rollbackSucceedingStages(mapStage, rollbackCurrentStage = true)
- }
+ if (shouldAbortStage) {
+ abortStage(failedStage, abortReason.get, None)
+ } else { // update failedStages and make sure a
ResubmitFailedStages event is enqueued
+ // TODO: Cancel running tasks in the failed stage -- cf.
SPARK-17064
+ val noResubmitEnqueued = !failedStages.contains(failedStage)
+ failedStages += failedStage
+ failedStages += mapStage
+ if (noResubmitEnqueued) {
+ // For statically indeterminate stages, trigger rollback early
(here and in
+ // submitMissingTasks) rather than deferring to task
completion. This is more
+ // efficient because it clears shuffle outputs before the
retry is submitted,
+ // ensuring findMissingPartitions() returns all partitions.
+ //
+ // For runtime detection (checksum mismatch), rollback is
triggered at task
+ // completion when the mismatch is discovered.
+ //
+ // The `rollbackCurrentStage = true` parameter ensures the
failed map stage is
+ // included in the cleanup: clearing its shuffle outputs,
marking old task results
+ // to be ignored, and creating a new shuffle merge state for
the upcoming retry.
+ if (mapStage.isStaticallyIndeterminate &&
+ !mapStage.shuffleDep.checksumMismatchFullRetryEnabled) {
+ rollbackSucceedingStages(mapStage, rollbackCurrentStage =
true)
+ }
- // We expect one executor failure to trigger many FetchFailures
in rapid succession,
- // but all of those task failures can typically be handled by a
single resubmission of
- // the failed stage. We avoid flooding the scheduler's event
queue with resubmit
- // messages by checking whether a resubmit is already in the
event queue for the
- // failed stage. If there is already a resubmit enqueued for a
different failed
- // stage, that event would also be sufficient to handle the
current failed stage, but
- // producing a resubmit for each failed stage makes debugging
and logging a little
- // simpler while not producing an overwhelming number of
scheduler events.
- logInfo(
- log"Resubmitting ${MDC(STAGE, mapStage)} " +
- log"(${MDC(STAGE_NAME, mapStage.name)}) and
${MDC(FAILED_STAGE, failedStage)} " +
- log"(${MDC(FAILED_STAGE_NAME, failedStage.name)}) due to fetch
failure")
- scheduleResubmit()
+ // We expect one executor failure to trigger many
FetchFailures in rapid succession,
+ // but all of those task failures can typically be handled by
a single resubmission
+ // of the failed stage. We avoid flooding the scheduler's
event queue with resubmit
+ // messages by checking whether a resubmit is already in the
event queue for the
+ // failed stage. If there is already a resubmit enqueued for
a different failed
+ // stage, that event would also be sufficient to handle the
current failed stage,
+ // but producing a resubmit for each failed stage makes
debugging and logging a
+ // little simpler while not producing an overwhelming number
of scheduler events.
+ logInfo(
+ log"Resubmitting ${MDC(STAGE, mapStage)} " +
+ log"(${MDC(STAGE_NAME, mapStage.name)}) and
${MDC(FAILED_STAGE, failedStage)} " +
+ log"(${MDC(FAILED_STAGE_NAME, failedStage.name)}) due to
fetch failure")
+ scheduleResubmit()
+ }
}
}
@@ -4038,12 +4100,12 @@ private[spark] class DAGScheduler(
/**
* On a FetchFailed, unregister the shuffle outputs of the executor (or its
whole host) whose
* fetch failed, treating the FetchFailed as authoritative evidence that its
shuffle data is gone.
- * Extracted from the base FetchFailed handler so the pipelined-group-abort
branch can also run it:
- * aborting the group fails only this job's stages, but a dead executor's
REGULAR outputs must
- * still be cleaned up for other/concurrent jobs (with an external shuffle
service, an ExecutorLost
- * does not clean them, so FetchFailed is the only proactive channel). No-op
when `bmAddress` is
- * null. Safe for a pipelined shuffle: it registers no map outputs in the
tracker, so this can only
- * strip regular/durable outputs.
+ * Extracted from the base FetchFailed handler so the pipelined-group-abort
branch can also run
+ * it: aborting the group fails only this job's stages, but a dead
executor's REGULAR outputs must
+ * still be cleaned up for other/concurrent jobs (with an external shuffle
service, an
+ * ExecutorLost does not clean them, so FetchFailed is the only proactive
channel). No-op when
+ * `bmAddress` is null. Safe for a pipelined shuffle: it registers no map
outputs in the tracker,
+ * so this can only strip regular/durable outputs.
*/
private def unregisterOutputsOnFetchFailedExecutor(
bmAddress: BlockManagerId, task: Task[_]): Unit = {
diff --git
a/core/src/main/scala/org/apache/spark/scheduler/ShuffleMapStage.scala
b/core/src/main/scala/org/apache/spark/scheduler/ShuffleMapStage.scala
index 94078d698cf4..11cec94f2b31 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/ShuffleMapStage.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/ShuffleMapStage.scala
@@ -64,13 +64,12 @@ private[spark] class ShuffleMapStage(
/**
* Availability tracking for a pipelined shuffle. A pipelined shuffle
produces no durable,
- * addressable map output. `createShuffleMapStage` still calls
`MapOutputTracker.registerShuffle`
- * for it (it is a `ShuffleDependency`, and an empty shuffle-status entry
keeps the tracker's
- * `containsShuffle` / `unregisterShuffle` bookkeeping uniform), but no map
output is ever
- * registered into that entry -- `registerMapOutput` runs only on the
non-pipelined completion
- * path -- so the entry stays empty. Its map-stage availability therefore
cannot be read from the
- * tracker; instead we track completed partitions here, monotonically: a
partition is added when
- * its map task succeeds and is NEVER removed on executor/host loss.
+ * addressable map output and is NOT registered with the `MapOutputTracker`
at all --
+ * `createShuffleMapStage` registers it only with the
`StreamingShuffleOutputTracker` (the two
+ * trackers are split by dependency type, with no overlap). Its map-stage
availability therefore
+ * cannot be read from the `MapOutputTracker`; instead we track completed
partitions here,
+ * monotonically: a partition is added when its map task succeeds and is
NEVER removed on
+ * executor/host loss.
*
* This is the crux of avoiding the streaming-writer resubmit hang: if a
pipelined shuffle's
* availability were read from the `MapOutputTracker`, losing an executor
that held a completed
@@ -110,8 +109,8 @@ private[spark] class ShuffleMapStage(
* When this reaches [[numPartitions]], this map stage is ready.
*/
def numAvailableOutputs: Int = {
- // A pipelined shuffle's outputs are never populated in the
MapOutputTracker (its entry stays
- // empty; see pipelinedCompletedPartitions); read its locally-tracked,
monotonic completed set.
+ // A pipelined shuffle is not registered with the MapOutputTracker at all
(see
+ // pipelinedCompletedPartitions); read its locally-tracked, monotonic
completed set instead.
if (isPipelined) pipelinedCompletedPartitions.size
else mapOutputTrackerMaster.getNumAvailableOutputs(shuffleDep.shuffleId)
}
diff --git
a/core/src/main/scala/org/apache/spark/shuffle/streaming/StreamingShuffleManager.scala
b/core/src/main/scala/org/apache/spark/shuffle/streaming/StreamingShuffleManager.scala
index 03ca257353c5..b83bd33552b9 100644
---
a/core/src/main/scala/org/apache/spark/shuffle/streaming/StreamingShuffleManager.scala
+++
b/core/src/main/scala/org/apache/spark/shuffle/streaming/StreamingShuffleManager.scala
@@ -113,8 +113,8 @@ private[spark] class StreamingShuffleManager extends
PipelinedShuffleManager wit
override def unregisterShuffle(shuffleId: Int): Boolean = {
// No manager-side state to release here: the driver's
StreamingShuffleOutputTracker is
- // unregistered in BlockManagerStorageEndpoint's RemoveShuffle handler,
and per-task writer
- // and reader resources are released via task completion listeners.
+ // unregistered in ContextCleaner.doCleanupShuffle (its state is
driver-only), and per-task
+ // writer and reader resources are released via task completion listeners.
true
}
diff --git a/core/src/test/scala/org/apache/spark/ContextCleanerSuite.scala
b/core/src/test/scala/org/apache/spark/ContextCleanerSuite.scala
index bb2d7d5c4d8e..0bd0ee103a19 100644
--- a/core/src/test/scala/org/apache/spark/ContextCleanerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/ContextCleanerSuite.scala
@@ -20,6 +20,7 @@ package org.apache.spark
import scala.collection.mutable.HashSet
import scala.util.Random
+import org.apache.logging.log4j.Level
import org.scalatest.BeforeAndAfter
import org.scalatest.concurrent.Eventually._
import org.scalatest.concurrent.PatienceConfiguration
@@ -127,6 +128,52 @@ class ContextCleanerSuite extends ContextCleanerSuiteBase {
assert(rdd.collect().toList.equals(collected))
}
+ test("cleanup shuffle unregisters a pipelined shuffle from the streaming
output tracker") {
+ // A pipelined shuffle lives ONLY in the driver-only
StreamingShuffleOutputTracker -- it is
+ // never registered with the MapOutputTracker (see
DAGScheduler.createShuffleMapStage). So the
+ // MapOutputTracker.containsShuffle guard is false for it, and
doCleanupShuffle must still clean
+ // it up via its own streaming branch; otherwise the tracker grows without
bound across
+ // micro-batches in Real-Time Mode.
+ val streamingTracker = sc.env.streamingShuffleOutputTracker.get
+ .asInstanceOf[StreamingShuffleOutputTrackerMaster]
+ val mapOutputTracker =
sc.env.mapOutputTracker.asInstanceOf[MapOutputTrackerMaster]
+
+ // Register a shuffle id in the streaming tracker ONLY, exactly as the
scheduler now does for a
+ // pipelined shuffle (nothing is put in the MapOutputTracker).
+ val shuffleId = 1000
+ streamingTracker.registerShuffle(shuffleId, numMaps = 2, numReduces = 2,
jobId = 0)
+ assert(streamingTracker.containsShuffle(shuffleId))
+ assert(!mapOutputTracker.containsShuffle(shuffleId),
+ "a pipelined shuffle must not be registered with the MapOutputTracker")
+
+ cleaner.doCleanupShuffle(shuffleId, blocking = true)
+
+ // The streaming tracker entry is gone -- cleanup fired even though the
shuffle was never in the
+ // MapOutputTracker, proving the streaming cleanup branch is independent
of MapOutputTracker.
+ assert(!streamingTracker.containsShuffle(shuffleId))
+ }
+
+ test("cleanup shuffle leaves a regular shuffle's streaming tracker untouched
and quiet") {
+ // A regular (non-pipelined) shuffle is registered only with the
MapOutputTracker, never the
+ // streaming tracker. doCleanupShuffle must take the MapOutputTracker
branch and never touch the
+ // streaming tracker -- in particular it must not emit the streaming
tracker's "attempting to
+ // unregister a shuffle that hasn't been registered" warning.
+ val streamingTracker = sc.env.streamingShuffleOutputTracker.get
+ .asInstanceOf[StreamingShuffleOutputTrackerMaster]
+
+ val (rdd, shuffleDeps) = newRDDWithShuffleDependencies()
+ rdd.collect()
+ shuffleDeps.foreach(s =>
assert(!streamingTracker.containsShuffle(s.shuffleId)))
+
+ val logAppender = new LogAppender("unregister streaming shuffle")
+ logAppender.setThreshold(Level.WARN)
+ withLogAppender(logAppender, level = Some(Level.WARN)) {
+ shuffleDeps.foreach(s => cleaner.doCleanupShuffle(s.shuffleId, blocking
= true))
+ }
+ assert(!logAppender.loggingEvents.exists(
+ _.getMessage.getFormattedMessage.contains("hasn't been registered")))
+ }
+
test("cleanup broadcast") {
val broadcast = newBroadcast()
val tester = new CleanerTester(sc, broadcastIds = Seq(broadcast.id))
@@ -174,6 +221,49 @@ class ContextCleanerSuite extends ContextCleanerSuiteBase {
postGCTester.assertCleanup()
}
+ test("automatically cleanup pipelined shuffle from the streaming output
tracker") {
+ // The load-bearing leak guarantee for a pipelined (streaming) shuffle: it
is registered ONLY
+ // with the driver-only StreamingShuffleOutputTracker (never the
MapOutputTracker), and its only
+ // cleanup channel is the ContextCleaner weak-reference path fired when the
+ // PipelinedShuffleDependency is garbage-collected
(registerShuffleForCleanup in the
+ // ShuffleDependency constructor). If that GC path did not reach the
streaming tracker,
+ // the tracker would grow without bound across Real-Time Mode
micro-batches. The sibling
+ // doCleanupShuffle tests call cleanup directly; this one proves the
end-to-end
+ // GC -> ContextCleaner -> streaming-tracker unregister link.
+ val streamingTracker = sc.env.streamingShuffleOutputTracker.get
+ .asInstanceOf[StreamingShuffleOutputTrackerMaster]
+ val mapOutputTracker =
sc.env.mapOutputTracker.asInstanceOf[MapOutputTrackerMaster]
+
+ // Build a real PipelinedShuffleDependency (registers itself for cleanup
on construction) and
+ // register its shuffleId in the streaming tracker exactly as
DAGScheduler.createShuffleMapStage
+ // does for a pipelined shuffle (nothing is put in the MapOutputTracker).
+ var pipelinedDep: PipelinedShuffleDependency[Int, Int, Int] =
+ new PipelinedShuffleDependency(newPairRDD(), new HashPartitioner(2))
+ val shuffleId = pipelinedDep.shuffleId
+ streamingTracker.registerShuffle(shuffleId, numMaps = 2, numReduces = 2,
jobId = 0)
+ assert(streamingTracker.containsShuffle(shuffleId))
+ assert(!mapOutputTracker.containsShuffle(shuffleId),
+ "a pipelined shuffle must not be registered with the MapOutputTracker")
+
+ // A strong reference to the dependency must prevent GC-triggered cleanup.
+ runGC()
+ intercept[Exception] {
+ eventually(timeout(1.second), interval(100.milliseconds)) {
+ assert(!streamingTracker.containsShuffle(shuffleId),
+ "cleanup must NOT fire while the dependency is strongly referenced")
+ }
+ }
+
+ // Dereference the dependency; GC must then drive ContextCleaner to
unregister it from the
+ // streaming tracker (and only there -- it was never in the
MapOutputTracker).
+ pipelinedDep = null
+ runGC()
+ eventually(timeout(10.seconds), interval(100.milliseconds)) {
+ assert(!streamingTracker.containsShuffle(shuffleId),
+ "GC of the PipelinedShuffleDependency must unregister it from the
streaming tracker")
+ }
+ }
+
test("automatically cleanup broadcast") {
var broadcast = newBroadcast()
diff --git
a/core/src/test/scala/org/apache/spark/StreamingShuffleOutputTrackerSuite.scala
b/core/src/test/scala/org/apache/spark/StreamingShuffleOutputTrackerSuite.scala
index 3c8d9b2f14e2..a9351630ad8d 100644
---
a/core/src/test/scala/org/apache/spark/StreamingShuffleOutputTrackerSuite.scala
+++
b/core/src/test/scala/org/apache/spark/StreamingShuffleOutputTrackerSuite.scala
@@ -270,6 +270,20 @@ class StreamingShuffleOutputTrackerSuite
assert(tracker.getAllShuffleWriterTaskLocations(0).isEmpty)
}
+ test("StreamingShuffleOutputTrackerMaster - containsShuffle reflects
register/unregister") {
+ val conf = new SparkConf(false)
+ val tracker = newTrackerMaster(conf)
+
+ // A shuffle that was never registered is absent.
+ assert(!tracker.containsShuffle(0))
+
+ tracker.registerShuffle(shuffleId = 0, numMaps = 1, numReduces = 1, jobId
= 1)
+ assert(tracker.containsShuffle(0))
+
+ tracker.unregisterShuffle(0)
+ assert(!tracker.containsShuffle(0))
+ }
+
test("StreamingShuffleOutputTrackerMaster - register writer before shuffle
fails") {
val conf = new SparkConf(false)
val tracker = newTrackerMaster(conf)
diff --git
a/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
b/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
index 05ba990e2d63..a0428d8d8bd4 100644
--- a/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
@@ -7078,6 +7078,46 @@ class DAGSchedulerSuite extends SparkFunSuite with
TempLocalSparkContext with Ti
s"a dropped fan-in consumer must not replay when a surviving producer
succeeds; got $results")
}
+ test("pipelined shuffle: a fan-in group leaks no scheduler state when a
producer fails and the " +
+ "job is torn down") {
+ // Leak check for the multi-producer (fan-in) deferral: a consumer
deferred against TWO
+ // producers buffers its completions; when one producer fails and the job
is torn down, ALL
+ // scheduler state -- the consumer's dependentStageMap deferral keyed on
both producers --
+ // must be cleaned up. The sibling "dropped as soon as one producer fails"
test stops at the
+ // drop-vs-replay outcome; this one drives the failure to full job
teardown and asserts no leak.
+ val producerA = new MyRDD(sc, 2, Nil)
+ val psdA = new PipelinedShuffleDependency(producerA, new
HashPartitioner(2))
+ val producerB = new MyRDD(sc, 2, Nil)
+ val psdB = new PipelinedShuffleDependency(producerB, new
HashPartitioner(2))
+ val consumerRdd = new MyRDD(sc, 2, List(psdA, psdB), tracker =
mapOutputTracker)
+ val failure = new java.util.concurrent.atomic.AtomicReference[Exception]()
+ val failListener = new JobListener {
+ override def taskSucceeded(index: Int, result: Any): Unit =
results.put(index, result)
+ override def jobFailed(exception: Exception): Unit =
failure.set(exception)
+ }
+ submit(consumerRdd, Array(0, 1), listener = failListener)
+ assert(taskSets.size === 3, s"A, B and the consumer must be co-scheduled;
got ${taskSets.size}")
+ val consumerTaskSet =
+ taskSets.find(ts => scheduler.stageIdToStage(ts.stageId).rdd eq
consumerRdd).get
+ val consumerStage = scheduler.stageIdToStage(consumerTaskSet.stageId)
+ val producerATaskSet =
+ taskSets.find(ts => scheduler.stageIdToStage(ts.stageId).rdd eq
producerA).get
+
+ // Consumer finishes early -> buffered (deferred) against BOTH producers.
+ complete(consumerTaskSet, Seq((Success, 42), (Success, 43)))
+ assert(results.isEmpty, "consumer completions should be buffered while its
producers run")
+
assert(scheduler.dependentStageMap.get(consumerStage).exists(_.parents.size ==
2),
+ "the consumer must be deferred against both producers")
+
+ // Producer A fails its whole task set -> the job fails and the group is
torn down.
+ failed(producerATaskSet, "producer A blew up")
+ assert(failure.get() != null, "the job must fail when a fan-in producer
fails")
+ assert(results.isEmpty, "buffered consumer successes must be dropped, not
applied, on failure")
+ sc.listenerBus.waitUntilEmpty(10000)
+ // The whole point: no leak. Every scheduler map (including the two-parent
deferral) empties.
+ assertDataStructuresEmpty()
+ }
+
test("pipelined shuffle: a multi-producer consumer's buffered successes are
dropped when one " +
"producer fails (no stale replay via a sibling)") {
// Group-atomic drop across MULTIPLE producers. Consumer C reads TWO
pipelined producers P1 and
diff --git
a/core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
b/core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
index 4c7e8efc4eab..5fc7c842c54b 100644
--- a/core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
@@ -3139,8 +3139,9 @@ class TaskSetManagerSuite
// The TaskSetManager.executorLost "Resubmitted" loop re-enqueues an
already-successful
// ShuffleMapTask when its executor is lost and the map output looks gone
-- for a decommission,
// it looks gone whenever MapOutputTracker.getMapOutputLocation returns
None. A pipelined
- // shuffle is NEVER registered in the MapOutputTracker, so
getMapOutputLocation is always None
- // and this loop would resubmit the lone producer task -- the exact
streaming-writer hang. This
+ // shuffle is registered only with the StreamingShuffleOutputTracker,
never the
+ // MapOutputTracker, so getMapOutputLocation is always None and this loop
would resubmit the
+ // lone producer task -- the exact streaming-writer hang. This
// loop bypasses handleFailedTask, so the group-atomic abort would never
see it. The guard
// (!taskSet.isPipelined on maybeShuffleMapOutputLoss) must keep a
pipelined set out of the
// loop.
diff --git
a/core/src/test/scala/org/apache/spark/shuffle/PipelinedShuffleRoutingSuite.scala
b/core/src/test/scala/org/apache/spark/shuffle/PipelinedShuffleRoutingSuite.scala
index da74803a10eb..8e0f9e04c1bd 100644
---
a/core/src/test/scala/org/apache/spark/shuffle/PipelinedShuffleRoutingSuite.scala
+++
b/core/src/test/scala/org/apache/spark/shuffle/PipelinedShuffleRoutingSuite.scala
@@ -240,6 +240,46 @@ class PipelinedShuffleRoutingSuite extends SparkFunSuite
with LocalSparkContext
assert(SparkEnv.get.streamingShuffleOutputTracker.isEmpty)
}
+ test("createShuffleMapStage fails loud for a pipelined dependency when no
streaming tracker") {
+ // A pipelined shuffle is served solely by the
StreamingShuffleOutputTracker. With a non-
+ // streaming incremental manager configured, that tracker is absent
(verified above), so a job
+ // whose stage graph contains a PipelinedShuffleDependency cannot be
scheduled.
+ // createShuffleMapStage must fail loud rather than silently register the
shuffle in no tracker
+ // (which would strand a consumer with no writer locations). This guards
the decoupling's
+ // invariant that a pipelined dependency implies a streaming tracker.
+ // local[4]: the pipelined group's up-front slot admission (producer 2 +
result 2 = 4) must pass
+ // so that stage creation is actually reached -- otherwise the job is
rejected for slots first.
+ sc = new SparkContext("local[4]", "test", newConf())
+ assert(SparkEnv.get.streamingShuffleOutputTracker.isEmpty)
+ val producer = sc.parallelize(1 to 4, 2).map(x => (x, x))
+ val pipelined = new PipelinedShuffleDependency[Int, Int, Int](producer,
new HashPartitioner(2))
+ // A minimal reduce-side RDD whose single dependency is the pipelined
shuffle, so submitting an
+ // action forces createShuffleMapStage for the pipelined producer.
+ val consumer = new RDD[(Int, Int)](sc, Seq(pipelined)) {
+ override def compute(split: Partition, ctx: TaskContext): Iterator[(Int,
Int)] =
+ Iterator.empty
+ override protected def getPartitions: Array[Partition] =
+ Array.tabulate(2)(i => new Partition { override def index: Int = i })
+ }
+ val ex = intercept[Exception] {
+ consumer.count()
+ }
+ assert(findCause[IllegalStateException](ex).exists(
+ _.getMessage.contains("requires a StreamingShuffleOutputTracker")),
+ s"expected a fail-loud IllegalStateException about the missing tracker,
got: $ex")
+
+ // The fail-loud throw happens BEFORE createShuffleMapStage mutates
stageIdToStage /
+ // shuffleIdToMapStage, so it leaves no partial scheduler state behind. If
it left a
+ // half-created stage cached in shuffleIdToMapStage, a re-submission would
reuse that stale
+ // stage instead of re-throwing. Re-submitting the same job must therefore
re-throw the error.
+ val ex2 = intercept[Exception] {
+ consumer.count()
+ }
+ assert(findCause[IllegalStateException](ex2).exists(
+ _.getMessage.contains("requires a StreamingShuffleOutputTracker")),
+ s"a re-submission must re-throw the fail-loud error (no leaked partial
stage), got: $ex2")
+ }
+
test("spark.shuffle.manager.incremental resolves the same short aliases as
the default manager") {
// "sort" is a short alias the incremental slot must resolve (to
SortShuffleManager) rather than
// treat as a class name. SortShuffleManager is blocking, so it is
rejected from the pipelined
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]