Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23106#discussion_r236432889
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java ---
@@ -242,8 +243,13 @@ private void writeSortedFile(boolean isLastFile) {
// Note that we intentionally ignore the value of
`writeMetricsToUse.shuffleWriteTime()`.
// Consistent with ExternalSorter, we do not count this IO towards
shuffle write time.
// This means that this IO time is not accounted for anywhere;
SPARK-3577 will fix this.
- writeMetrics.incRecordsWritten(writeMetricsToUse.recordsWritten());
-
taskContext.taskMetrics().incDiskBytesSpilled(writeMetricsToUse.bytesWritten());
+
+ // This is guaranteed to be a ShuffleWriteMetrics based on the if
check in the beginning
+ // of this file.
--- End diff --
ah yes. nice catch
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
