chinmay291 opened a new issue, #20548:
URL: https://github.com/apache/hudi/issues/20548

   ### Feature Description
   
   **What the feature achieves:**
   
   An opt-in way for count-based compaction triggers to ignore delta commits 
that wrote no records, for both merge-on-read data tables and the metadata 
table.
   
   **Why this feature is needed:**
   
   The count-based compaction triggers (`NUM_COMMITS`, 
`NUM_COMMITS_AFTER_LAST_REQUEST`, `NUM_AND_TIME`, `NUM_OR_TIME`) count every 
completed delta commit since the last compaction, including delta commits that 
wrote no records.
   
   Streaming writers produce such commits routinely. For example, Hudi Streamer 
makes an empty commit whenever the source checkpoint advances but no records 
match, just to persist the new checkpoint. With a short sync interval, there 
can be many empty commits for every commit that carries data.
   
   This hurts most on the metadata table, which is compacted inline:
   - Every data table commit, empty or not, produces a metadata table delta 
commit. Even for an empty data commit, that delta commit writes a record to the 
`files` partition.
   - Empty data commits therefore count towards 
`hoodie.metadata.compact.max.delta.commits` like real ones.
   - With the record-level index enabled, every metadata table compaction 
rewrites all record index file groups. For large tables this is expensive, and 
inline compaction blocks the writer while it runs.
   
   As a result, the metadata table can be compacted several times more often 
than its actual write volume warrants. For example, with 
`hoodie.metadata.compact.max.delta.commits=10`, a writer that makes 3 data 
commits and 7 empty commits triggers a compaction even though only 3 delta 
commits carried changes.
   
   Raising `max.delta.commits` reduces the frequency, but it also delays 
compaction for tables with real write activity, because it doesn't tell empty 
commits from real ones.
   
   ### User Experience
   
   **How users will use this feature:**
   
   New configs, all off by default:
   
   | Config | Default | Purpose |
   |---|---|---|
   | `hoodie.compact.inline.ignore.empty.commits` | `false` | MoR data tables: 
don't count empty delta commits towards 
`hoodie.compact.inline.max.delta.commits` |
   | `hoodie.metadata.compact.ignore.empty.commits` | `false` | Metadata table: 
don't count delta commits whose data table commit wrote no records |
   | `hoodie.compact.inline.max.total.delta.commits` / 
`hoodie.metadata.compact.max.total.delta.commits` | `-1` (5 x 
`max.delta.commits`) | Safety cap: once total delta commits, empty ones 
included, reach this, compaction triggers anyway. Bounds the number of log 
files and the length of the active timeline |
   
   Behaviour when enabled:
   - **Data table:** a delta commit is empty when it wrote and deleted no 
records.
   - **Metadata table:** a delta commit is empty when the data table commit at 
the same instant time wrote and deleted no records. The metadata table's own 
write stats can't tell, because empty data commits still write a `files` 
partition record.
   - Cleans, rollbacks, and any instant whose metadata can't be read still 
count, so the trigger errs towards compacting.
   - **Archival:** while the flag is on, the archiver keeps every delta commit 
since the latest compaction, so archiving can't shrink the count and postpone 
compaction indefinitely.
   - Time-based triggers and the log compaction trigger are unchanged.
   
   ### Hudi RFC Requirements
   
   **RFC PR link:** N/A
   
   **Why RFC is/isn't needed:** The change is opt-in and limited to how the 
existing compaction trigger counts delta commits. It adds no storage format, 
timeline or API changes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to