JNSimba commented on code in PR #66409:
URL: https://github.com/apache/doris/pull/66409#discussion_r3709374419
##########
fe/fe-core/src/main/java/org/apache/doris/job/offset/jdbc/JdbcSourceOffsetProvider.java:
##########
@@ -292,14 +301,18 @@ public void fetchRemoteMeta(Map<String, String>
properties) throws Exception {
"Failed to get end offset from backend," +
result.getStatus().getErrorMsgs(0) + ", response: "
+ result.getResponse());
}
- Map<String, String> newEndOffset = parseCdcResponseData(
- result.getResponse(), new TypeReference<Map<String,
String>>() {});
+ FetchEndOffsetResult fetchResult = parseCdcResponseData(
+ result.getResponse(), new
TypeReference<FetchEndOffsetResult>() {});
+ Map<String, String> newEndOffset = fetchResult.getEndOffset();
synchronized (splitsLock) {
// null→value also counts as a change: upstream may have
advanced while fetch was blocked.
if (endBinlogOffset == null ||
!endBinlogOffset.equals(newEndOffset)) {
hasMoreData = true;
}
endBinlogOffset = newEndOffset;
+ if (!isSnapshotOnlyMode()) {
+ updateLagBytes(fetchResult.getLagBytes());
Review Comment:
[P2] Do not publish a lag computed for an offset generation that has already
been committed past. `getLagReferenceOffset()` copies `O0` before this RPC
releases `splitsLock`; while the request is in flight, `afterCommitted()` or
`commitOffset()` can advance `currentOffset` to `O1`. This block then
unconditionally publishes the response calculated from `O0`, so even `end ==
O1` can be shown with a positive backlog, and PAUSE or a later fetch failure
can retain it indefinitely. Carry the reference/generation back (or snapshot it
locally) and only publish when the current reference still matches, like the
stale-result identity check in `hasMoreDataToConsume()`; please add a
latch-based concurrency test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
