gaoyajun02 created SPARK-52923:
----------------------------------
Summary: Allow ShuffleManager to control shuffle merge during
registration
Key: SPARK-52923
URL: https://issues.apache.org/jira/browse/SPARK-52923
Project: Spark
Issue Type: Improvement
Components: Shuffle
Affects Versions: 4.0.0, 3.5.0
Reporter: gaoyajun02
Background:
Although spark.shuffle.push.enabled can globally control push-based shuffle,
there are scenarios that require more fine-grained control:
- Mass job migration scenarios where different jobs may need different shuffle
strategies
- External remote shuffle services that need shuffle-level fallback from
push-based shuffle
- Dynamic decision making based on shuffle characteristics during registration
Currently, the _shuffleMergeAllowed flag in ShuffleDependency is initialized
before shuffle registration, preventing ShuffleManager implementations from
modifying this capability based on their specific requirements or runtime
conditions.
This change moves the initialization of numPartitions and _shuffleMergeAllowed
before the shuffleManager.registerShuffle() call, allowing ShuffleManager
implementations to dynamically disable push-based shuffle merge during
registration if needed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
