JNSimba opened a new pull request, #66409:
URL: https://github.com/apache/doris/pull/66409
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
Streaming job lag was calculated from the current time and the timestamp of
the last consumed event. The value kept increasing when the source was idle, so
it could not distinguish source inactivity from an actual synchronization
backlog.
This change reports source-log backlog in bytes. MySQL-compatible sources
calculate the remaining bytes from the committed binlog file and position to
the current end offset. PostgreSQL calculates the WAL byte difference from the
replication slot confirmed position to the current WAL position. Snapshot-only
jobs report -1, and transient lag calculation failures retain the latest
successful observation. SHOW output remains numeric and the per-job metric now
uses the BYTES unit.
### Release note
Streaming job lag and the per-job lag metric now report source-log backlog
in bytes. A value of -1 means that no valid observation is available.
### Check List (For Author)
- Test
- [ ] Regression test (cases were updated but not run locally)
- [x] Unit Test
- [ ] Manual test
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Unit tests run:
- FE lag tests: 10 passed, 0 failed
- CDC lag calculator tests: 12 passed, 0 failed
MySQL/PostgreSQL version integration assertions and
MySQL/PostgreSQL/OceanBase regression assertions were added but were not run
locally because the required container and deployed regression environments
were unavailable.
- Behavior changed:
- [ ] No.
- [x] Yes. Streaming job lag changes from event-time seconds to
source-log backlog bytes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
