zml1206 commented on code in PR #57815:
URL: https://github.com/apache/spark/pull/57815#discussion_r3757091395
##########
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java:
##########
@@ -547,6 +577,8 @@ public void insertKVRecord(Object keyBase, long keyOffset,
int keyLen,
Object valueBase, long valueOffset, int valueLen, long prefix, boolean
prefixIsNull)
throws IOException {
+ spillIfThresholdReached();
Review Comment:
Good point. I have called this behavior change out explicitly in the
user-facing changes section of the PR description. The default behavior remains
unchanged because both thresholds default to their maximum values, but lowered
thresholds may now cause additional spills in ObjectHashAggregateExec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
