chinmay291 opened a new pull request, #20549:
URL: https://github.com/apache/hudi/pull/20549
### Describe the issue this Pull Request addresses
Closes #20548
Count-based compaction triggers count every delta commit since the last
compaction, including delta commits that wrote no files. Streaming writers make
such commits routinely, for example when only the source checkpoint advances.
On the metadata table, every empty data table commit still creates a delta
commit, so the metadata table is compacted far more often than its write volume
needs. With the record-level index enabled, each of those inline compactions
rewrites every record index file group.
### Summary and Changelog
Adds an opt-in setting, off by default, that excludes empty delta commits
from count-based compaction triggers.
- **New configs**
- `hoodie.compact.inline.ignore.empty.commits` (default `false`) and
`hoodie.compact.inline.max.total.delta.commits` (default `-1`, meaning 5 x
`hoodie.compact.inline.max.delta.commits`) in `HoodieCompactionConfig`.
- `hoodie.metadata.compact.ignore.empty.commits` and
`hoodie.metadata.compact.max.total.delta.commits` in `HoodieMetadataConfig`,
passed to the metadata table write config in `HoodieMetadataWriteUtils`.
- **`CompactionUtils.filterOutEmptyDeltaCommits`**
- **Data table:** a delta commit is empty when it has no write stats.
- **Metadata table:** a delta commit is empty when the data table commit
at the same instant time has no write stats. The metadata table's own write
stats can't tell, because an empty data commit still writes a `files` partition
record.
- Cleans, rollbacks, and instants without a readable data table commit
count as non-empty, so the trigger errs towards compacting.
- **`ScheduleCompactionActionExecutor.needCompact`**: when enabled,
`NUM_COMMITS`, `NUM_COMMITS_AFTER_LAST_REQUEST`, `NUM_AND_TIME` and
`NUM_OR_TIME` use the count of non-empty delta commits. Once total delta
commits reach the cap, every delta commit counts again. The log compaction and
time-only triggers are unchanged.
- **`TimelineArchiverV1` / `TimelineArchiverV2`**: when enabled, every delta
commit since the latest compaction is retained
(`CompactionUtils.getEarliestInstantToRetainForCompaction` overload). Otherwise
archival could shrink the non-empty count and postpone compaction indefinitely.
### Impact
- New opt-in configs only. With the default (`false`), behaviour is
unchanged.
- When enabled, scheduling a compaction reads the commit metadata of each
delta commit since the last compaction. On the metadata table it also reads the
data table's active timeline. The number of reads is bounded by the
total-delta-commits cap.
### Risk Level
Low. The feature is off by default. When enabled, the trigger errs towards
compacting whenever an instant can't be classified, and the cap bounds how long
compaction can be postponed. Tests cover data and metadata table filtering, the
trigger count with and without the cap, archival retention, and an end-to-end
metadata table scenario: empty data commits don't trigger the compaction when
the flag is on, and do when it's off.
### Documentation Update
Config descriptions are included in the new `ConfigProperty` definitions. A
website update for the new configs can follow once the approach is agreed.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]