minihippo opened a new pull request #4152:
URL: https://github.com/apache/hudi/pull/4152
…ite amplification by using LogFileSizeThresholdBasedCompactionStrategy.
## What is the purpose of the pull request
For huge table, it has many file groups and each file group size can be more
than GB. Therefore, we have to increase the compaction target io size to
compact more file groups. However , this parameter is difficult to adjust. If
the target io size is large, file groups with small log file size will be
compacted, which results in write amplification. If the target io size is
small, more file groups with large log file will wait for compaction and reduce
the read performance.
Base on the `LogFileSizeBasedCompactionStrategy`, the new compaction
strategy filters the file groups with log file size greater than the
threshold(500M) when planning the compaction.
## Brief change log
- add `LogFileSizeThresholdBasedCompactionStrategy`
## Verify this pull request
- add test case to `TestHoodieCompactionStrategy`
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
