Wan Kun created SPARK-44243:
-------------------------------
Summary: Local shuffle reader should not respect
SHUFFLE_REDUCE_LOCALITY_ENABLE
Key: SPARK-44243
URL: https://issues.apache.org/jira/browse/SPARK-44243
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.5.0
Reporter: Wan Kun
Local shuffle reader can achieve better performance with preferred locations.
If we disable SHUFFLE_REDUCE_LOCALITY_ENABLE in queries that include reduce
shuffles and local shuffles, local shuffle readers can not get preferred
locations.
Add new parameter LOCAL_SHUFFLE_LOCALITY_ENABLE to determine whether to get the
preferred locations of the current partitionSpec.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
