skywalker0618 opened a new issue, #20531: URL: https://github.com/apache/hudi/issues/20531
### Task Description **What needs to be done:** **Part 1: fix the existing stream read metrics (`FlinkStreamReadMetrics`)** The FLIP-27 source (`read.source-v2.enabled=true`) registers the existing stream read metrics, but they don't reflect the reader's state: - `splitLatestCommit` / `splitLatestCommitDelay` are registered by `HoodieSourceSplitReader` but never set, so they always report 0. Only the legacy `StreamReadOperator` calls `setSplitLatestCommit`. - `issuedInstantDelay` and `splitLatestCommitDelay` are computed once, inside the setter (`Duration.between(instant, Instant.now())`), and the gauge returns the stored value. If the reader stalls or discovery stops finding commits, the delay freezes instead of growing. Fixes (no metric names change; values stay based on instant time): - Store the instant and compute the delay when the gauge is read. - Set `splitLatestCommit` when a Source V2 reader starts a split. - Report a `splitLatestCommitDelay` of 0 while a reader has no split, so idle, caught-up readers don't look like they're lagging. **Part 2: add read lag metrics to the continuous split enumerator** The existing metrics measure discovery, not reading. `issuedInstantDelay` only covers the newest discovered commit, so it can show a reader as nearly caught up while it's still hours into reading earlier commits. There is no Hudi equivalent of Kafka's consumer lag. Add, on the enumerator: | Metric | Definition | |---|---| | `readLagSeconds` | "now minus completion time of the oldest completed commit that isn't fully read" (undiscovered, or discovered with unfinished splits); 0 when caught up | | `pendingCommits` | completed commits not fully read, split into undiscovered and in-flight | | `pendingSplits` | splits discovered but not yet finished | | `pendingRecords` | expected records still to be emitted by unfinished splits | | `pendingBytes` | base and log file bytes still to be read by unfinished splits | Implementation notes: - **Tracking:** the enumerator already receives `finishedSplitIds` in `SplitRequestEvent`. Keep a count of unfinished splits per commit. A split whose instant range covers several commits counts against each of them. After a restore, rebuild the counts from the checkpointed pending splits, so no new enumerator state is needed. - **Undiscovered commits:** taken from the completed commits after the issued offset in the timeline that discovery already loads. No extra commit metadata is read. - **Records and bytes:** attach an expected record count and the file size to each split at discovery, from the write stats already being read (`numInserts` for append-only; `numInserts + numUpdateWrites` for upserts). This adds a field to `HoodieSourceSplit`, so it needs a `HoodieSourceSplitSerializer` version bump that still reads old splits. For upsert tables, `pendingRecords` is an estimate: a record updated more than once within one instant range is counted more than once. - **Completion time:** all new metrics use commit completion time, so the writer's commit duration isn't counted as reader lag. For table versions before 8, completion time comes from the timeline's existing completion-time lookup. - **Scope:** append-only and COW upsert tables. MOR is out of scope for now, will leave it for future work. **Why this task is needed:** Kafka sources give Flink users record- and time-based consumer lag, which is the standard way to tell whether a streaming job is keeping up and to alert when it falls behind. The Hudi streaming source has nothing comparable: the reader-side metrics on Source V2 always report 0, the delay metrics freeze, and no metric reports how far reading (as opposed to discovery) is behind. This matters most for COW upsert tables, where reading a commit can take far longer than discovering it. In production we stream a COW table of ~10B rows (~28 TB) where each commit rewrites ~197K files. The reader discovers a commit within minutes and then takes several hours to read it, and existing metrics show no lag during that time. Falling behind also has a hard limit: once the reader is more than `hoodie.cleaner.commits.retained` commits behind, the files it's about to read are cleaned and the job fails with `FileNotFoundException`. `readLagSeconds` and `pendingCommits` let users alert well before that, for example on `pendingCommits` relative to cleaner retention. The per-subtask `splitLatestCommit*` metrics, once fixed, also show stragglers: a subtask whose split is much older than the others' is the one holding a commit back. I plan to open two PRs, the Part 1 fixes first, then the Part 2 metrics. ### Task Type Code improvement/refactoring ### Related Issues **Parent feature issue:** (if applicable ) **Related issues:** NOTE: Use `Relationships` button to add parent/blocking issues after issue is created. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
