danny0405 commented on code in PR #20564:
URL: https://github.com/apache/hudi/pull/20564#discussion_r4520556348


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/reader/HoodieSourceSplitReader.java:
##########
@@ -181,6 +186,7 @@ public void close() throws Exception {
   private RecordsWithSplitIds<HoodieRecordWithPosition<T>> finishSplit() {
     RecordsWithSplitIds<HoodieRecordWithPosition<T>> records = 
BatchRecords.lastBatchRecords(currentSplit.splitId());
     currentSplit = null;
+    readerMetrics.clearCurrentSplit();

Review Comment:
   [P2] Could we keep the delay active until the source reader has consumed the 
split's buffered records? This runs on the fetcher thread, before Flink's 
FetchTask enqueues the finished-split marker. Earlier batches can still be 
queued or blocked in downstream emission, so clearing here makes 
splitLatestCommitDelay report 0 while the source is still backpressured, 
potentially indefinitely. Completion should be tracked at the 
source-reader/emission boundary, taking care that completion of an older split 
cannot clear the metric for a newer active split. A test that blocks emission 
after fetching the final batch would cover this case.



##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/reader/HoodieSourceSplitReader.java:
##########
@@ -109,6 +113,7 @@ public RecordsWithSplitIds<HoodieRecordWithPosition<T>> 
fetch() throws IOExcepti
       }
       currentSplit = nextSplit;
       readerFunction.open(currentSplit);
+      readerMetrics.setSplitLatestCommit(currentSplit.getLatestCommit());

Review Comment:
   [P2] Could we set the split metric before calling 
readerFunction.open(currentSplit)? AbstractSplitReaderFunction.open() creates 
the underlying iterator and synchronously replays/skips the consumed records 
when restoring a split. Slow initialization or a large recovery replay 
therefore leaves readingSplit false and splitLatestCommitDelay at 0 for the 
whole operation, despite the reader actively working on an old split. Setting 
it before open() would make those stalls visible too; a test with a blocked 
open() would verify the gauge keeps growing during initialization.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to