Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/22062#discussion_r209337484 --- Diff: core/src/main/java/org/apache/spark/shuffle/sort/ShuffleInMemorySorter.java --- @@ -94,12 +94,20 @@ public int numRecords() { } public void reset() { + // Reset `pos` here so that `spill` triggered by the below `allocateArray` will be no-op. + pos = 0; --- End diff -- We also need to set `usableCapacity` to `0`. Otherwise, https://github.com/apache/spark/blob/9b8521e53e56a53b44c02366a99f8a8ee1307bbf/core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java#L343 will not rethrow SparkOutOfMemoryError. ShuffleExternalSorter will keep running and finally touch `array`. Setting `array` to `null` is just for safety so that anything incorrect use will fail with NPE.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org