cloud-fan commented on code in PR #44709:
URL: https://github.com/apache/spark/pull/44709#discussion_r1450540492
##########
core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java:
##########
@@ -440,8 +445,9 @@ public void insertRecord(Object recordBase, long
recordOffset, int length, int p
*/
public SpillInfo[] closeAndGetSpills() throws IOException {
if (inMemSorter != null) {
- // Do not count the final file towards the spill count.
- writeSortedFile(true);
+ // Here we are spilling the remaining data in the buffer. If there is no
spill before, this
+ // final spill file will be the final shuffle output file.
+ writeSortedFile(/* isFinalFile = */spills.isEmpty());
Review Comment:
This is the actual change of this PR. We should only set this flag if we
have not spilled before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
