skywalker0618 opened a new issue, #20531:
URL: https://github.com/apache/hudi/issues/20531

   ### Task Description
   
   **What needs to be done:**
   
   **Part 1: fix the existing stream read metrics (`FlinkStreamReadMetrics`)**
   
   The FLIP-27 source (`read.source-v2.enabled=true`) registers the existing 
stream read metrics, but they don't reflect the reader's state:
   
   - `splitLatestCommit` / `splitLatestCommitDelay` are registered by 
`HoodieSourceSplitReader` but never set, so they always report 0. Only the 
legacy `StreamReadOperator` calls `setSplitLatestCommit`.
   - `issuedInstantDelay` and `splitLatestCommitDelay` are computed once, 
inside the setter (`Duration.between(instant, Instant.now())`), and the gauge 
returns the stored value. If the reader stalls or discovery stops finding 
commits, the delay freezes instead of growing.
   
   Fixes (no metric names change; values stay based on instant time):
   - Store the instant and compute the delay when the gauge is read.
   - Set `splitLatestCommit` when a Source V2 reader starts a split.
   - Report a `splitLatestCommitDelay` of 0 while a reader has no split, so 
idle, caught-up readers don't look like they're lagging.
   
   **Part 2: add read lag metrics to the continuous split enumerator**
   
   The existing metrics measure discovery, not reading. `issuedInstantDelay` 
only covers the newest discovered commit, so it can show a reader as nearly 
caught up while it's still hours into reading earlier commits. There is no Hudi 
equivalent of Kafka's consumer lag. Add, on the enumerator:
   
   | Metric | Definition |
   |---|---|
   | `readLagSeconds` | "now minus completion time of the oldest completed 
commit that isn't fully read" (undiscovered, or discovered with unfinished 
splits); 0 when caught up |
   | `pendingCommits` | completed commits not fully read, split into 
undiscovered and in-flight |
   | `pendingSplits` | splits discovered but not yet finished |
   | `pendingRecords` | expected records still to be emitted by unfinished 
splits |
   | `pendingBytes` | base and log file bytes still to be read by unfinished 
splits |
   
   Implementation notes:
   - **Tracking:** the enumerator already receives `finishedSplitIds` in 
`SplitRequestEvent`. Keep a count of unfinished splits per commit. A split 
whose instant range covers several commits counts against each of them. After a 
restore, rebuild the counts from the checkpointed pending splits, so no new 
enumerator state is needed.
   - **Undiscovered commits:** taken from the completed commits after the 
issued offset in the timeline that discovery already loads. No extra commit 
metadata is read.
   - **Records and bytes:** attach an expected record count and the file size 
to each split at discovery, from the write stats already being read 
(`numInserts` for append-only; `numInserts + numUpdateWrites` for upserts). 
This adds a field to `HoodieSourceSplit`, so it needs a 
`HoodieSourceSplitSerializer` version bump that still reads old splits. For 
upsert tables, `pendingRecords` is an estimate: a record updated more than once 
within one instant range is counted more than once.
   - **Completion time:** all new metrics use commit completion time, so the 
writer's commit duration isn't counted as reader lag. For table versions before 
8, completion time comes from the timeline's existing completion-time lookup.
   - **Scope:** append-only and COW upsert tables. MOR is out of scope for now, 
will leave it for future work.
   
   **Why this task is needed:**
   
   Kafka sources give Flink users record- and time-based consumer lag, which is 
the standard way to tell whether a streaming job is keeping up and to alert 
when it falls behind. The Hudi streaming source has nothing comparable: the 
reader-side metrics on Source V2 always report 0, the delay metrics freeze, and 
no metric reports how far reading (as opposed to discovery) is behind.
   
   This matters most for COW upsert tables, where reading a commit can take far 
longer than discovering it. In production we stream a COW table of ~10B rows 
(~28 TB) where each commit rewrites ~197K files. The reader discovers a commit 
within minutes and then takes several hours to read it, and existing metrics 
show no lag during that time.
   
   Falling behind also has a hard limit: once the reader is more than 
`hoodie.cleaner.commits.retained` commits behind, the files it's about to read 
are cleaned and the job fails with `FileNotFoundException`. `readLagSeconds` 
and `pendingCommits` let users alert well before that, for example on 
`pendingCommits` relative to cleaner retention.
   
   The per-subtask `splitLatestCommit*` metrics, once fixed, also show 
stragglers: a subtask whose split is much older than the others' is the one 
holding a commit back.
   
   I plan to open two PRs, the Part 1 fixes first, then the Part 2 metrics.
   
   
   ### Task Type
   
   Code improvement/refactoring
   
   ### Related Issues
   
   **Parent feature issue:** (if applicable )
   **Related issues:**
   NOTE: Use `Relationships` button to add parent/blocking issues after issue 
is created.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to