JNSimba opened a new pull request, #65688:
URL: https://github.com/apache/doris/pull/65688
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Streaming CDC jobs periodically fetch the source end offset, while a
successful task commits its current offset independently. A task can advance
beyond the previously fetched end offset. The FE previously converted the
offset comparison into a boolean, so it could stop scheduling but could not
distinguish equality from the current offset being ahead, leaving CurrentOffset
greater than EndOffset in job output.
This change preserves the three-way comparison result and advances EndOffset
to CurrentOffset when the fetched end is behind. End-offset publication is
synchronized so an obsolete comparison result cannot overwrite a newer fetched
end. The PostgreSQL latest-offset credential regression case now waits for the
first task to commit the resolved latest offset before inserting new rows.
### Release note
Keep the displayed streaming CDC end offset from falling behind committed
current progress after offset comparison.
### Check List (For Author)
- Test
- [ ] Regression test
- [ ] Unit Test
- Verification was not run at user request. Focused FE unit coverage and
regression coverage are included in this PR.
- Behavior changed:
- [x] Yes. EndOffset is advanced to CurrentOffset when the committed
current position is ahead of the fetched end position.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
