Josh Rosen created SPARK-7413:
---------------------------------
Summary: Time to write shuffle spill files is not captured in
ShuffleWriteMetrics
Key: SPARK-7413
URL: https://issues.apache.org/jira/browse/SPARK-7413
Project: Spark
Issue Type: Bug
Components: Shuffle
Reporter: Josh Rosen
In ExternalSorter's {{spillToMergeableFile()}} method, we pass
ShuffleWriteMetrics instances to the disk writers, but discard the
{{shuffleWriteTime}} metrics captured here. I think that we should account for
this IO time, possibly by introducing new metrics to distinguish time spent
writing spills vs. writing final shuffle output and extending the UI to break
down the overall IO write time in terms of these two components.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
