advancedxy commented on code in PR #471:
URL: https://github.com/apache/incubator-uniffle/pull/471#discussion_r1070806030
##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java:
##########
@@ -342,6 +342,20 @@ public class ShuffleServerConf extends RssBaseConf {
.noDefaultValue()
.withDescription("The env key to get json source of local storage media
provider");
+ public static final ConfigOption<Long> HUGE_PARTITION_SIZE_THRESHOLD =
ConfigOptions
+ .key("rss.server.huge-partition.size.threshold")
+ .longType()
+ .defaultValue(20 * 1024 * 1024 * 1024L)
Review Comment:
Is 20GB has some meaningful background? It would be great if you could add
some suggestion values in the doc description.
Also could you please update the configuration doc?
##########
server/src/main/java/org/apache/uniffle/server/ShuffleTaskManager.java:
##########
@@ -200,7 +199,13 @@ public StatusCode registerShuffle(
public StatusCode cacheShuffleData(
String appId, int shuffleId, boolean isPreAllocated,
ShufflePartitionedData spd) {
refreshAppId(appId);
- return shuffleBufferManager.cacheShuffleData(appId, shuffleId,
isPreAllocated, spd);
+ return shuffleBufferManager.cacheShuffleData(
+ appId,
+ shuffleId,
+ isPreAllocated,
+ spd,
+ this::getPartitionDataSize
Review Comment:
Why couldn't we pass the partitionDataSize directly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
