skywalker0618 opened a new pull request, #20735: URL: https://github.com/apache/hudi/pull/20735
### Describe the issue this Pull Request addresses Part 2 of #20531. The Source V2 streaming reader has no metric that says how far behind the table it is, the way Kafka consumer lag does. This PR adds read lag and pending work metrics to the continuous split enumerator, for append-only and copy-on-write tables. Merge-on-read is left for future work. ### Summary and Changelog The continuous split enumerator now tracks every discovered split until a reader reports it finished, and registers these gauges under the table metric group: | Gauge | Meaning | |---|---| | `readLagSeconds` | Seconds since the oldest pending commit completed, 0 when nothing is pending | | `pendingCommits` | `undiscoveredCommits` + `inFlightCommits` | | `undiscoveredCommits` | Completed commits after the issued offset that discovery has not turned into splits yet | | `inFlightCommits` | Discovered commits that still have queued or unfinished splits | | `pendingSplits` | Discovered splits not finished yet, queued or being read | | `pendingRecords` | Expected records still to be emitted by the pending splits | | `pendingBytes` | Size of the files the pending splits still have to read | **What a split reads.** At discovery, each split records the commits within the batch that wrote its file group, taken from the commit write stats that discovery already loads. For copy-on-write and append-only tables, these are the only commits whose records the split emits, because the reader filters rows by commit time. When they are unknown (the initial snapshot read from earliest, or a split restored from a version 1 checkpoint), the split falls back to the instants of its exact-match range, or to its latest commit. **How the metrics are maintained.** All state lives in `FlinkStreamReadLagMetrics` on the enumerator. Discovery, restore and splits added back by a failed reader call `addSplits`; the finished split ids sent with each split request call `finishSplits`. Both are synchronized, so a gauge read never sees half an update. - **`readLagSeconds`**: each split carries the completion time of the oldest commit it reads. A sorted map counts the pending splits per completion time; finishing a split decrements its entry and removes it at 0. The gauge returns now minus the first key, or minus the oldest undiscovered commit when no split is pending. Completion times come from the timeline, and the value is computed when the gauge is read, so it keeps growing while the readers stall. - **`inFlightCommits`**: a map counts the pending splits that read each commit; a split reading two commits counts under both. A commit is removed once every split reading it is finished. The gauge returns the number of commits in the map. - **`undiscoveredCommits`**: after each discovery pass, the completed commits after the new offset are counted from the timeline that pass already loaded. When discovery is paused because of `read.splits.limit`, the timeline is reloaded instead. With the default `read.commits.limit` and `read.splits.limit` every pass discovers all completed commits, so this is 0; it is non-zero for jobs that throttle discovery. Archived commits are not counted. - **`pendingSplits`**: the number of tracked splits. One split is one base file for copy-on-write and append-only tables, and one file slice for merge-on-read. - **`pendingRecords`**: a running sum of each split's expected records, which are the inserts plus updates its write commits wrote to its file group. Exact for append-only tables. For upserts it is an estimate: a record written by two commits of the same batch counts twice. Reading the file footer would not help, since a copy-on-write file holds every row of the file group but the split only emits the rows of its commits. 0 for the initial snapshot read. - **`pendingBytes`**: a running sum of each split's file size: the size of the latest file written for the file group, from the write stats, or the file slice size for the initial snapshot read. For merge-on-read this is only the last log file, which under-reports. **Changelog** - `HoodieSourceSplit`: new fields for the write commits, the oldest commit completion time, the estimated records and the estimated bytes, and `getReadCommits()`. - `HoodieSourceSplitSerializer` and `HoodieEnumeratorStateSerializer` move to version 2. Both still read version 1, which is covered by tests; they move together because the state serializer passes its version to the split serializer. - `IncrementalInputSplits.Result` carries the instant completion times and per file group read estimates, filled by `HoodieContinuousSplitBatch.fromResult`. - `HoodieContinuousSplitDiscover.getUndiscoveredCommits` (default empty), implemented by `DefaultHoodieSplitDiscover`. - `AbstractHoodieSplitEnumerator` gets hooks for finished and added-back splits, used by `HoodieContinuousSplitEnumerator`. - New `FlinkStreamReadLagMetrics`. ### Impact New enumerator metrics for Source V2 streaming reads. No new configs and no change to read behavior. Performance: no new storage requests in the normal case. Undiscovered commits reuse the timeline loaded by discovery, and write commits, records and bytes come from commit metadata discovery already parses. A paused discovery pass reloads the timeline once per check interval, which is still cheaper than the pass it replaces. On the enumerator, each split adds a few map updates on discovery and on finish, and the JobManager keeps a small list of write commits per split. Each checkpointed split grows by roughly 50 to 70 bytes, which matters only when many splits are queued (about 6 MB of enumerator state per 100K queued splits). ### Risk Level low. The split and enumerator state formats change, but version 1 checkpoints still restore, and both formats are covered by tests. The metrics only observe the enumerator and don't change split assignment. Known limitation: splits that the readers restore from their own state are not known to the enumerator, so right after a restore up to about one split per subtask is not counted until it finishes or is added back. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
