advancedxy commented on code in PR #445:
URL: https://github.com/apache/incubator-uniffle/pull/445#discussion_r1058072348
##########
server/src/main/java/org/apache/uniffle/server/buffer/ShuffleBufferManager.java:
##########
@@ -427,22 +432,35 @@ private Map<String, Set<Integer>> pickFlushedShuffle() {
// the remaining data in buffer maybe more than lowWaterMark
// because shuffle server is still receiving data, but it should be ok
long expectedFlushSize = highWaterMark - lowWaterMark;
+ long atLeastFlushSizeIgnoreThreshold = expectedFlushSize >>> 1;
Review Comment:
Despite of the `shuffleFlushThreshold` setting, at least half of
`highWaterMark - lowWaterMark`'s shuffle would be picked to flush. This is a
precaution of extremely case:
Shuffle Server reached high watermark and all the shuffle are small I/Os.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
