zuston commented on code in PR #428:
URL: https://github.com/apache/incubator-uniffle/pull/428#discussion_r1055983057
##########
server/src/main/java/org/apache/uniffle/server/ShuffleTaskManager.java:
##########
@@ -579,4 +602,11 @@ void removeShuffleDataSync(String appId, int shuffleId) {
public ShuffleDataDistributionType getDataDistributionType(String appId) {
return shuffleTaskInfos.get(appId).getDataDistType();
}
+
+ private void triggerFlush() {
+ synchronized (this.shuffleBufferManager) {
Review Comment:
> In current codebase, the triggerFlush is only called per
rss.server.shuffleBufferManager.trigger.flush.interval, which should be 2-5
mins on server side. That shouldn't be too much overhead?
Yes, it wont.
> As for why triggerFlush is introduced in this PR, its main reason is to
reduce UT changes for the behavior changes as preAllocatedSize is decreased in
Grpc layer, rather than the cache data layer.
Got it.
> And also, I thought it would be better if we decoupled the flush logic
with cacheShuffleData, which causes flush spikes, and frequent full gc.
+1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
