skywalker0618 opened a new pull request, #20564:
URL: https://github.com/apache/hudi/pull/20564

   ### Describe the issue this Pull Request addresses
   
   Part 1 of #20531. The stream read metrics (`FlinkStreamReadMetrics`) don't 
reflect the reader's state:
   
   - On the FLIP-27 source (`read.source-v2.enabled=true`), `splitLatestCommit` 
and `splitLatestCommitDelay` always report 0. `HoodieSourceSplitReader` 
registers them but never sets them; only the legacy `StreamReadOperator` does.
   - `issuedInstantDelay` and `splitLatestCommitDelay` are computed once, in 
the setter, and the gauge returns the stored value. If the reader stalls on a 
split, or discovery stops finding commits, the delay freezes instead of growing.
   
   There is also a registration problem on Source V2. Flink closes an idle 
split fetcher and creates a new `HoodieSourceSplitReader` when new splits 
arrive, and each split reader created and registered its own 
`FlinkStreamReadMetrics`. Flink keeps the first metric registered under a name 
and ignores later ones, so even once set, the reported values would have come 
from the first split reader only.
   
   ### Summary and Changelog
   
   The stream read delays are now computed when the gauges are read, and the 
Source V2 reader reports `splitLatestCommit` and `splitLatestCommitDelay`. No 
metric names change, and values are still based on instant time.
   
   - `FlinkStreamReadMetrics` stores the issued instant and the split's latest 
commit, and computes `issuedInstantDelay` and `splitLatestCommitDelay` in the 
gauges. A new `clearCurrentSplit()` makes `splitLatestCommitDelay` report 0 
until the next split starts, so an idle, caught-up reader doesn't look like 
it's lagging. `splitLatestCommit` keeps reporting the last split's commit.
   - `HoodieSourceReader` creates and registers one `FlinkStreamReadMetrics` 
and passes it to every split reader it creates. `HoodieSourceSplitReader` now 
takes the metrics instead of the table name and reader context.
   - `HoodieSourceSplitReader` sets the split's latest commit when it opens a 
split and clears it when the split finishes.
   - `StreamReadOperator` clears the current split when it finishes one. 
Without this, the delay computed at read time would keep growing after the 
legacy reader went idle.
   - Tests: new `TestFlinkStreamReadMetrics` covering the delays growing with 
time, the idle reader, the values before anything is set and invalid instants; 
a new `TestHoodieSourceSplitReader` case checking that the split reader sets 
and clears the split's commit.
   
   ### Impact
   
   Metric values change; names don't.
   - `issuedInstantDelay` now keeps growing while discovery finds no new 
commits, rather than holding the delay from when the last commit was 
discovered. On a table that commits rarely it rises between commits; it still 
resets when the next commit is issued.
   - On Source V2, `splitLatestCommit` and `splitLatestCommitDelay` now report 
real values instead of 0.
   - On both readers, `splitLatestCommitDelay` grows while a split is being 
read and reports 0 while the reader has no split.
   
   `HoodieSourceSplitReader`'s constructor changes, but it's internal to the 
Flink source.
   
   ### Risk Level
   
   low. The changes are limited to metric values; reading is unaffected.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to