[
https://issues.apache.org/jira/browse/SPARK-44243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wan Kun updated SPARK-44243:
----------------------------
Summary: Add a parameter to determine the locality of local shuffle reader
(was: Local shuffle reader should not respect SHUFFLE_REDUCE_LOCALITY_ENABLE)
> Add a parameter to determine the locality of local shuffle reader
> -----------------------------------------------------------------
>
> Key: SPARK-44243
> URL: https://issues.apache.org/jira/browse/SPARK-44243
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.5.0
> Reporter: Wan Kun
> Priority: Major
>
> Local shuffle reader can achieve better performance with preferred locations.
> If we disable SHUFFLE_REDUCE_LOCALITY_ENABLE in queries that include reduce
> shuffles and local shuffles, local shuffle readers can not get preferred
> locations.
> Add new parameter LOCAL_SHUFFLE_LOCALITY_ENABLE to determine whether to get
> the preferred locations of the current partitionSpec.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
