[
https://issues.apache.org/jira/browse/SPARK-28607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Vanzin resolved SPARK-28607.
------------------------------------
Fix Version/s: 3.0.0
Resolution: Fixed
Issue resolved by pull request 25341
[https://github.com/apache/spark/pull/25341]
> Don't hold a reference to two partitionLengths arrays
> -----------------------------------------------------
>
> Key: SPARK-28607
> URL: https://issues.apache.org/jira/browse/SPARK-28607
> Project: Spark
> Issue Type: Sub-task
> Components: Shuffle
> Affects Versions: 3.0.0
> Reporter: Matt Cheah
> Assignee: Matt Cheah
> Priority: Major
> Fix For: 3.0.0
>
>
> SPARK-28209 introduced the new shuffle writer API and its usage in
> BypassMergeSortShuffleWriter. However, the design of the API forces the
> partition lengths to be tracked both in the implementation of the plugin and
> also by the higher-level writer. This leads to redundant memory usage. We
> should only track the lengths of the partitions in the implementation of the
> plugin and propagate this information back up to the writer as the return
> value of {{commitAllPartitions}}.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
