skywalker0618 commented on code in PR #20564:
URL: https://github.com/apache/hudi/pull/20564#discussion_r4532744903
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/reader/HoodieSourceSplitReader.java:
##########
@@ -181,6 +186,7 @@ public void close() throws Exception {
private RecordsWithSplitIds<HoodieRecordWithPosition<T>> finishSplit() {
RecordsWithSplitIds<HoodieRecordWithPosition<T>> records =
BatchRecords.lastBatchRecords(currentSplit.splitId());
currentSplit = null;
+ readerMetrics.clearCurrentSplit();
Review Comment:
Agreed, fixed in bc41326be5. `FlinkStreamReadMetrics` now tracks the active
splits by id (`startSplit` / `finishSplit`), and
`HoodieSourceReader.onSplitFinished` finishes them, on the task thread after
all of the split's records are emitted. The split reader no longer clears
anything when it fetches the last batch. Since the fetcher may already have
started the next split by then, `splitLatestCommitDelay` reports the oldest
active split, so finishing an older split doesn't clear a newer one. Tests:
`TestHoodieSourceSplitReader#testMetricsSplitIsNotFinishedWhenItsLastBatchIsFetched`,
`TestHoodieSourceReader#testMetricsFinishSplitsOnceTheirRecordsAreEmitted`,
and
`TestFlinkStreamReadMetrics#testFinishingAnOlderSplitKeepsTheNewerSplitActive`
/ `#testSplitLatestCommitDelayReportsTheOldestActiveSplit`.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/reader/HoodieSourceSplitReader.java:
##########
@@ -109,6 +113,7 @@ public RecordsWithSplitIds<HoodieRecordWithPosition<T>>
fetch() throws IOExcepti
}
currentSplit = nextSplit;
readerFunction.open(currentSplit);
+ readerMetrics.setSplitLatestCommit(currentSplit.getLatestCommit());
Review Comment:
Good point, done in bc41326be5: the split is started in the metrics before
`readerFunction.open()`, so a slow open or recovery replay counts as reading.
`TestHoodieSourceSplitReader#testMetricsStartTheSplitBeforeOpeningIt` checks
inside `open()` that the split was already started; it fails if the order is
reversed. The legacy `StreamReadOperator` now also starts the split before
`format.open()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]