shangxinli opened a new pull request, #19868: URL: https://github.com/apache/hudi/pull/19868
### Describe the issue this Pull Request addresses The metadata table reports `baseFileCount` and `logFileCount` per partition, but nothing reports whether MDT compaction is keeping up. When compaction stops being scheduled or fails quietly, log files accumulate and every metadata lookup merges more log blocks — operators only notice via slow lookups, long after compaction stopped. ### Summary and Changelog Adds one gauge, `delta_commits_since_last_compaction`: completed delta commits on the metadata table since the last completed compaction. It counts the same instants `ScheduleCompactionActionExecutor` compares against `hoodie.metadata.compact.max.delta.commits`, so it is directly comparable to the compaction trigger. Sampled in the `finally` block of `performTableServices()`, so a repeatedly failing pending compaction — which throws before compaction is reached — still reports. The timeline is reloaded because neither `runAnyPendingCompactions()` nor `writeClient.compact()` refreshes this meta client. Reporting is best effort: failures are logged and swallowed so they cannot fail a table service run or displace an in-flight exception. Alert condition: the value staying at or above the threshold across consecutive cycles. It does not grow without bound — once the backlog passes the threshold, MDT archival eventually drops the last compaction and the count plateaus near `hoodie.keep.min.commits`. Not emitted under `hoodie.metadata.streaming.write.enabled`, where Flink compacts MDT in a separate pipeline; left to a follow-up. An earlier revision also added `totalBaseFileCount`, `totalLogFileCount` and `logToBaseFileRatioPercent`. Dropped — they are arithmetic over gauges the same call already emits, and none measure compaction health. Thanks @voonhous for the review that surfaced this. ### Impact One additional gauge and one metadata timeline reload per table service cycle. One series per table. No write-path behavior change. ### Risk Level low Metrics-only change, covered by unit tests in `TestHoodieBackedTableMetadataWriter`. ### Documentation Update None. The gauge is documented on the constant in `HoodieMetadataMetrics`. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
