[ https://issues.apache.org/jira/browse/SPARK-35426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Qi Zhu updated SPARK-35426: --------------------------- Description: Now When addMergerLocation exceed the maxRetainedMergerLocations , we just remove the oldest merger, but we'd better remove the merger based on merged shuffle data size. was: Now When addMergerLocation exceed the maxRetainedMergerLocations , we just remove the oldest merger, but we'd better remove the merger based on merged shuffle data size. The oldest merger may have big merged shuffle data size, it will not be a good choice to do so. > When addMergerLocation exceed the maxRetainedMergerLocations , we should > remove the merger based on merged shuffle data size. > ----------------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-35426 > URL: https://issues.apache.org/jira/browse/SPARK-35426 > Project: Spark > Issue Type: Sub-task > Components: Spark Core > Affects Versions: 3.2.0 > Reporter: Qi Zhu > Priority: Major > > Now When addMergerLocation exceed the maxRetainedMergerLocations , we just > remove the oldest merger, but we'd better remove the merger based on merged > shuffle data size. > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org