jerrypeng opened a new pull request, #57179:
URL: https://github.com/apache/spark/pull/57179
### What changes were proposed in this pull request?
This PR adds `PipelinedShuffleDependency`, a new `ShuffleDependency`
subtype that marks a
shuffle's output as **incrementally readable**: a consumer stage may begin
reading the output
while the producer stage is still running, rather than waiting for the
producer's full,
materialized output.
Class hierarchy:
Dependency[T]
ShuffleDependency[K, V, C]
PipelinedShuffleDependency[K, V, C] <- new
This PR adds **only the type**. On its own, `PipelinedShuffleDependency`
behaves exactly like its
parent `ShuffleDependency` — construction goes through the normal path
(shuffle id allocation,
`ShuffleManager.registerShuffle`, cleaner registration), and any code that
matches on
`ShuffleDependency` continues to treat it as an ordinary,
fully-materialized shuffle. It introduces
**no behavior change**. It is the marker that a later `DAGScheduler`
change will use to (a)
co-schedule the producer and consumer stages connected by this edge (a
"pipelined group") and (b)
select an incremental shuffle implementation for this edge. That
concurrent-scheduling and
incremental-shuffle behavior is added in follow-up PRs.
The parent's `checksumMismatchFullRetryEnabled` /
`checksumMismatchQueryLevelRollbackEnabled`
constructor parameters are intentionally not exposed by the subclass, so
they stay at their `false`
defaults. Their checksum-mismatch model recomputes and re-runs succeeding
stages after a mismatch;
in the pipelined-group failure model any failure aborts the whole group
and the caller reruns from
scratch, so that stage-level recompute never fires, and it would also
conflict with a consumer that
has already read the output incrementally.
This is the first PR in a stack that incrementally introduces pipelined
shuffle dependencies and
concurrent stage scheduling.
### Why are the changes needed?
Today a multi-stage job runs one stage at a time: each shuffle is fully
materialized before the next
stage starts. Some workloads (initially Structured Streaming real-time
mode) need the stages of a
single job to run concurrently, connected by a shuffle whose consumer
reads the producer's output as
it is produced. Expressing that requires a first-class, distinguishable
dependency kind that the
scheduler and shuffle layers can key off of. This PR introduces that
primitive as the foundation the
subsequent scheduler and shuffle-layer changes build on, split out on its
own so it can be reviewed
in isolation without any behavioral risk.
### Does this PR introduce _any_ user-facing change?
No. The new class is a `@DeveloperApi` addition that changes no existing
behavior; nothing constructs
a `PipelinedShuffleDependency` yet, and it is handled identically to an
ordinary `ShuffleDependency`
everywhere.
### How was this patch tested?
New unit tests in `ShuffleDependencySuite` (all passing) verify that a
`PipelinedShuffleDependency`:
- **is-a** `ShuffleDependency` (so existing `ShuffleDependency` matches
continue to apply) and
preserves its fields (partitioner, key/value class names, rdd);
- registers its shuffle through the normal path (a `shuffleHandle` is
produced) and a second instance
receives a distinct `shuffleId`;
- keeps `checksumMismatchFullRetryEnabled` and
`checksumMismatchQueryLevelRollbackEnabled` at `false`;
- forwards non-default constructor arguments (`aggregator`,
`mapSideCombine`) to `ShuffleDependency`
correctly (positional-forwarding guard);
and that an ordinary `ShuffleDependency` is **not** a
`PipelinedShuffleDependency` (negative case).
build/sbt 'core/testOnly org.apache.spark.shuffle.ShuffleDependencySuite'
...
Tests: succeeded 6, failed 0
### Was this patch authored or co-authored using generative AI tooling?
co-authored: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
