Kay Ousterhout created SPARK-5762:
-------------------------------------
Summary: Shuffle write time is incorrect for sort-based shuffle
Key: SPARK-5762
URL: https://issues.apache.org/jira/browse/SPARK-5762
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.2.1
Reporter: Kay Ousterhout
Assignee: Kay Ousterhout
For the sort-based shuffle, when bypassing merge sort, one file is written for
each partition, and then a final file is written that concatenates all of the
existing files together. The time to write this final file is not included in
the shuffle write time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
