Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/23225#discussion_r239860885
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java ---
@@ -161,6 +161,10 @@ private void writeSortedFile(boolean isLastFile) {
final ShuffleInMemorySorter.ShuffleSorterIterator sortedRecords =
inMemSorter.getSortedIterator();
+ // If there are no sorted records, so we don't need to create an empty
spill file.
+ if (!sortedRecords.hasNext()) {
+ return;
+ }
--- End diff --
@wangjiaochun . Shall we move line 160 ~ 167 to the beginning of this
function (line 147)?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
