swuferhong commented on a change in pull request #3169:
URL: https://github.com/apache/hudi/pull/3169#discussion_r660232507
##########
File path:
hudi-flink/src/main/java/org/apache/hudi/sink/compact/FlinkCompactionConfig.java
##########
@@ -83,6 +83,12 @@
@Parameter(names = {"--compaction-max-memory"}, description = "Max memory in
MB for compaction spillable map, default 100MB.", required = false)
public Integer compactionMaxMemory = 100;
+ @Parameter(names = {"--compaction-target-io"}, description = "Target IO per
compaction (both read and write) for batching compaction, default 512000M.",
required = false)
Review comment:
> Should FlinkOptions#COMPACTION_TARGET_IO default value change to
512000 to align with
HoodieCompactionConfig#DEFAULT_TARGET_IO_PER_COMPACTION_IN_MB and here?
After we used a large amount of data to test writing, we found that the
online compaction setting FlinkOptions#COMPACTION_TARGET_IO cannot be setted
too large, which would seriously affect the running of the writing tasks. In
contrast, the offline compaction target io default value can be set larger
while we have enough memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
