shangxinli opened a new pull request, #19868: URL: https://github.com/apache/hudi/pull/19868
### Describe the issue this Pull Request addresses The metadata table reports `baseFileCount` and `logFileCount` per partition, but nothing reports whether MDT compaction is keeping up with the delta commits being written to it. When compaction stops being scheduled, or fails quietly because `hoodie.metadata.write.fail.on.table.service.failures` is false and the exception is swallowed, log files accumulate against a fixed set of base files. Every metadata lookup then merges more log blocks per file slice. The first thing an operator actually observes is slower lookups and higher memory use on the read path, long after the point where compaction stopped. The per-partition counts do not surface it, because a table with a healthy file count and a table that has not compacted in a thousand delta commits can report identical `baseFileCount`. ### Summary and Changelog Adds four gauges to `HoodieMetadataMetrics` so that a stalled MDT compaction is visible while it is still cheap to fix. - `deltaCommitsSinceLastCompaction` — **completed** delta commits on the metadata table since the last completed compaction. Emitted from `HoodieBackedTableMetadataWriter.performTableServices()`. This counts the same instants that `ScheduleCompactionActionExecutor.getLatestDeltaCommitInfoSinceCompaction()` compares against `hoodie.compact.inline.max.delta.commits`. It is sampled *before* the compaction in the same method runs, so on a healthy table the gauge sawtooths: it peaks at that config value on the cycle where compaction fires, then drops. Alert on a multiple of the config value, not the value itself. A stalled compaction climbs past the peak without bound, which is the signal. Reset to 0 on the path where the metadata timeline has no completed delta commit yet, so a stale value is not left latched. - `totalBaseFileCount` / `totalLogFileCount` — the existing per-partition counts summed over the enabled metadata partitions. - `logToBaseFileRatioPercent` — log files per base file, scaled by 100 because `HoodieGauge` is long valued. 500 means five log files per base file. The ratio returns 0 when there are no base files to divide by. That case is not healthy — a metadata table holding log files and no base file has never been compacted — which is why the two totals are emitted alongside it rather than the ratio alone. Changelog: - `HoodieMetadataMetrics`: four new metric-name constants; `updateDeltaCommitsSinceLastCompaction(long)`; `updateSizeMetrics()` now also emits the two totals and the ratio; `sumStat()` and `logToBaseFileRatioPercent()` extracted as package-private statics so the arithmetic is unit-testable without standing up a file system view. - `HoodieBackedTableMetadataWriter.performTableServices()`: emits `deltaCommitsSinceLastCompaction`, reusing `CompactionUtils.getCompletedDeltaCommitsSinceLatestCompaction()`. - New `TestHoodieMetadataMetrics` in `hudi-common`, covering gauge emission through a real `Metrics` instance with the `INMEMORY` reporter, the ×100 scaling, the zero-base-file case, and stat summation over enabled partitions only. - Three cases added to `TestHoodieBackedTableMetadataWriter` covering the compacted, never-compacted, and empty-timeline paths. These run against a real `ActiveTimelineV2` rather than a static mock of `CompactionUtils`, so the completed-versus-inflight distinction is actually asserted: the main case has three completed delta commits and one inflight after the last compaction, and expects 3. Three notes on choices reviewers may want to push back on, stated up front: - The new gauges are **not** table-name prefixed, unlike `TABLE_SERVICE_EXECUTION_DURATION` in the same method. They follow the size-metric family instead. `Metrics.getBasePath()` strips the metadata-table suffix, so a metadata writer deliberately shares one registry with its own data-table writer, and distinct tables have distinct base paths — meaning cross-table collision is not possible and the prefix is not load-bearing here. The names are also distinct from the data-writer metric names in that shared registry. Happy to prefix them if the project prefers consistency within the method over consistency within the metric family. - The delta-commit gauge is sampled before table services rather than re-sampled after a synchronous compaction. Re-sampling would give a cleaner post-compaction value, but needs another `reloadActiveTimeline()` on the write path, and `compactIfNecessary` is `void` and has three outcomes (skipped, delegated to the table service manager, compacted inline) so "did it actually compact" is not currently observable at the call site. Flagging it as a deliberate trade-off; if reviewers would rather pay the reload, it is a small follow-up. - `sumStat` and `logToBaseFileRatioPercent` are package-private with `@VisibleForTesting` rather than private, so the arithmetic can be tested without constructing a `HoodieTableFileSystemView`. Can be inlined back to private if preferred. No code was copied from another project. ### Impact No public API change, no config change, no behavior change to writes or reads. Cost per commit is negligible and adds no listing or scan. `deltaCommitsSinceLastCompaction` reuses `CompactionUtils.getCompletedDeltaCommitsSinceLatestCompaction()` over the already-loaded active timeline. The totals are summed from the stats map `updateSizeMetrics()` already builds, which is O(enabled partitions) — roughly seven entries. Gauge cardinality is fixed: four new series per table, independent of partition or file count. Users on a metrics reporter gain four new series. Users with metrics disabled see nothing, since every emission goes through the existing `Option<HoodieMetadataMetrics>` guard. ### Risk Level low Metrics-only change on paths that already emit gauges. The one behavioral edge is the early return in `performTableServices()` when the metadata timeline has no completed delta commit; the gauge is set to 0 before returning so the value is not left stale, and that path is covered by a test. ### Documentation Update No new configs, so no config description update is required. The four metric names could reasonably be added to the metadata-table section of the metrics page on the Hudi website. Happy to open the companion website PR if reviewers want them listed. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
