[
https://issues.apache.org/jira/browse/SPARK-35426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qi Zhu updated SPARK-35426:
---------------------------
Description:
Now When addMergerLocation exceed the maxRetainedMergerLocations , we just
remove the oldest merger, but we'd better remove the merger based on merged
shuffle data size.
was:
Now When addMergerLocation exceed the maxRetainedMergerLocations , we just
remove the oldest merger, but we'd better remove the merger based on merged
shuffle data size.
The oldest merger may have big merged shuffle data size, it will not be a good
choice to do so.
> When addMergerLocation exceed the maxRetainedMergerLocations , we should
> remove the merger based on merged shuffle data size.
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-35426
> URL: https://issues.apache.org/jira/browse/SPARK-35426
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 3.2.0
> Reporter: Qi Zhu
> Priority: Major
>
> Now When addMergerLocation exceed the maxRetainedMergerLocations , we just
> remove the oldest merger, but we'd better remove the merger based on merged
> shuffle data size.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
