vinothchandar commented on a change in pull request #1654:
URL: https://github.com/apache/hudi/pull/1654#discussion_r429605435
##########
File path:
hudi-client/src/main/java/org/apache/hudi/config/HoodieCompactionConfig.java
##########
@@ -147,11 +147,6 @@ public Builder withInlineCompaction(Boolean
inlineCompaction) {
return this;
}
- public Builder inlineCompactionEvery(int deltaCommits) {
- props.setProperty(INLINE_COMPACT_PROP, String.valueOf(deltaCommits));
Review comment:
this is wrong nonetheless.. we can remove this method..
but the issue you faced may not be due to this? I see
`LogFileSizeBasedCompactionStrategy` is the default strategy, which chooses the
largest log files for compaction (like I mentioned, we check if compaction has
to be scheduled at the end of each delta commit), upto a certain size .. This
default is very large (probably due to the data volumes at uber)//
` public static final String DEFAULT_TARGET_IO_PER_COMPACTION_IN_MB =
String.valueOf(500 * 1024);`
And it keeps compacting all the logs, thus..
Could you try setting `hoodie.compaction.target.io=1` (i.e allow upto just
1MB, artifically low threshold) and see if the log files are there.. We can
also revisit if we can lower the limit for that in this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
