Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/23106#discussion_r236432889 --- Diff: core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java --- @@ -242,8 +243,13 @@ private void writeSortedFile(boolean isLastFile) { // Note that we intentionally ignore the value of `writeMetricsToUse.shuffleWriteTime()`. // Consistent with ExternalSorter, we do not count this IO towards shuffle write time. // This means that this IO time is not accounted for anywhere; SPARK-3577 will fix this. - writeMetrics.incRecordsWritten(writeMetricsToUse.recordsWritten()); - taskContext.taskMetrics().incDiskBytesSpilled(writeMetricsToUse.bytesWritten()); + + // This is guaranteed to be a ShuffleWriteMetrics based on the if check in the beginning + // of this file. --- End diff -- ah yes. nice catch
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org