cloud-fan commented on code in PR #44709:
URL: https://github.com/apache/spark/pull/44709#discussion_r1450538991
##########
core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java:
##########
@@ -150,11 +150,21 @@ public long[] getChecksums() {
* Sorts the in-memory records and writes the sorted records to an on-disk
file.
* This method does not free the sort data structures.
*
- * @param isLastFile if true, this indicates that we're writing the final
output file and that the
- * bytes written should be counted towards shuffle spill
metrics rather than
- * shuffle write metrics.
+ * @param isFinalFile if true, this indicates that we're writing the final
output file and that
+ * the bytes written should be counted towards shuffle
write metrics rather
+ * than shuffle spill metrics.
*/
- private void writeSortedFile(boolean isLastFile) {
+ private void writeSortedFile(boolean isFinalFile) {
+ // Only emit the log if this is an actual spilling.
+ if (!isFinalFile) {
+ logger.info(
Review Comment:
move the logging here so that it applies to the last spilling as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
