jerrypeng commented on code in PR #57092: URL: https://github.com/apache/spark/pull/57092#discussion_r3583038897
########## PIPELINED_SHUFFLE_DEPENDENCY_SPEC.md: ########## @@ -0,0 +1,386 @@ +# Pipelined Shuffle Dependency & Concurrent Stage Scheduling + +A spec for running data-dependent stages of a single job concurrently, connected by a shuffle the +consumer reads incrementally. + +--- + +## 1. Motivation + +Today, when two stages of a job are connected by a shuffle, they run one after another: the consumer +stage does not start until the producer's shuffle output is fully materialized. (Independent stages — +ones not connected by a shuffle — already run concurrently.) Some workloads need even +shuffle-connected stages to run **concurrently**, with the consumer reading the producer's output +**as it is produced** rather than after the producer finishes. This spec introduces the scheduler +primitives to express and run that. + +"Run these stages concurrently" and "the connecting shuffle is incremental" are the same decision +seen from two sides: co-scheduling a producer and consumer is only useful if the edge is readable +before the producer completes. + +--- + +## 2. Primitives + +### 2.1 Pipelined shuffle dependency (PSD) + +A shuffle dependency declared **incrementally readable**: a consumer stage may begin reading its +output while the producer stage is still running. + +- It is a shuffle dependency (has a `shuffleId`, partitioner, map/reduce sides); the *pipelined* Review Comment: Sounds good will add a explicit section on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
