[
https://issues.apache.org/jira/browse/SPARK-29072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-29072:
-------------------------------
Description: From SPARK-28209, SPARK-28570, and SPARK-28571, we used the
new shuffle writer plugin API across all the shuffle writers. However, we
accidentally lost time tracking metrics for shuffle writes in the process,
particularly for UnsafeShuffleWriter when writing with streams (without
transferTo), as well as the SortShuffleWriter. (was: From SPARK-28209,
SPARK-28570, and SPARK-28571, we used the new shuffle writer plugin API across
all the shuffle writers. However, some mistakes were made in the process:
* We lost shuffle write time metrics for the Unsafe and Sort shuffle writer
implementations
* There was a duplicate implementation of ShufflePartitionPairsWriter
introduced
This issue tracks rectifying both situations; the second is related to the
first.)
> Properly track shuffle write time with refactor
> -----------------------------------------------
>
> Key: SPARK-29072
> URL: https://issues.apache.org/jira/browse/SPARK-29072
> Project: Spark
> Issue Type: Bug
> Components: Shuffle
> Affects Versions: 3.0.0
> Reporter: Matt Cheah
> Priority: Major
>
> From SPARK-28209, SPARK-28570, and SPARK-28571, we used the new shuffle
> writer plugin API across all the shuffle writers. However, we accidentally
> lost time tracking metrics for shuffle writes in the process, particularly
> for UnsafeShuffleWriter when writing with streams (without transferTo), as
> well as the SortShuffleWriter.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
