suryaprasanna opened a new pull request, #17929:
URL: https://github.com/apache/hudi/pull/17929
### Describe the issue this Pull Request addresses
This PR adds metrics publishing for deltastreamer's incremental source to
track commit processing progress. This enables better monitoring and
observability of deltastreamer ingestion jobs by exposing metrics about how
many commits are being processed and how many remain unprocessed.
### Summary and Changelog
**Summary:**
Users can now monitor deltastreamer's incremental source progress through
metrics that track the number of commits in progress and unprocessed commits.
**Changelog:**
- Added `updateHoodieIncrSourceMetrics` method to `HoodieIngestionMetrics`
abstract class and `HoodieStreamerMetrics` implementation
- Updated `IncrSourceHelper.generateQueryInfo` to accept an optional metrics
parameter
- Added metrics publishing in `IncrSourceHelper.publishIncrSourceMetrics` to
track:
- Number of commits in progress (commits being processed in current batch)
- Number of unprocessed commits (commits remaining after current batch)
- Updated all callers of `generateQueryInfo` to pass metrics:
`GcsEventsHoodieIncrSource`, `S3EventsHoodieIncrSource`, and `HoodieIncrSource`
- Updated tests in `TestIncrSourceHelper` to accommodate new method signature
### Impact
This change adds two new metrics gauges for deltastreamer:
- `numCommitsInProgress`: Number of commits being processed in the current
incremental batch
- `numUnprocessedCommits`: Number of commits remaining to be processed after
the current batch
No existing functionality is affected. The metrics parameter is optional
(wrapped in `Option`), so this is backward compatible.
### Risk Level
**Low**
The changes are additive and do not modify existing behavior. The metrics
publishing is optional and only happens when metrics are enabled and the
metrics option is present.
### Documentation Update
None - this is an internal metrics improvement. The new metrics will
automatically be available through the existing metrics reporting
infrastructure when metrics are enabled.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
