JNSimba opened a new pull request, #69129:
URL: https://github.com/apache/doris/pull/69129
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
FROM-TO CDC offset commits repeatedly serialize the complete cached table
schema into scheduler-job journals, even when the CDC client reports no schema
update. A roughly 1 MiB schema can therefore add roughly 1 MiB to every
ordinary offset journal.
Keep the complete provider state in the Job and checkpoint images, but
include schemas in offset journals only when the client supplies them. Clear
the pending-schema flag after journal acknowledgement so failed writes can
retry. Merge schema-omitting records during replay, including cold ALTER and
cloud offset recovery, while retaining full-replacement semantics for TVF
providers. Reuse the existing scheduler-job operation and freeze the journal
payload before queueing it.
Compatibility: the operation and outer encoding are unchanged, but older FE
binaries replace provider state wholesale. Replaying compact journals with an
old FE during a rolling upgrade or downgrade can clear its cached schema. This
change does not add a writer compatibility switch.
### Release note
Reduce redundant schema bytes in FROM-TO CDC offset journals while retaining
complete checkpoint images. Compact journals require the updated FE replay
behavior; older FE readers may lose cached schemas when replaying them.
### Check List (For Author)
- Test
- [x] Unit Test
- [x] Regression test
- [x] Manual test
Validation:
- Final baseline `078e667a7ba`: FE build passed; Checkstyle passed for all
eight changed Java files, including test sources. The repository-wide
Checkstyle attempt was stopped after prolonged parser processing, then rerun
with includes limited to the changed files.
- `run-fe-ut.sh --run
StreamingInsertJobSchemaPersistenceTest,StreamingInsertJobOffsetPersistenceTest,JdbcSourceOffsetProviderOffsetTest`
(fully qualified class names): 54 tests passed, zero failures/errors/skips.
Includes byte-for-byte equivalence with the original Job writer, schema-update
acknowledgement/retry, legacy and compact replay, cold ALTER, cloud recovery,
TVF behavior and complete image recovery.
- Before the master sync, existing MySQL FROM-TO, advanced DDL and TVF
schema-change regression suites passed on the development machine. The advanced
DDL suite passed after retrying a cold CDC-client startup failure.
- Two real single-FE restarts were verified before the master sync:
journal-only recovery, then checkpoint-image recovery followed by DDL and
schema-omitting offset journals. JVM breakpoints inspected the complete
Job/provider schema before CDC access; full-schema SHA-256 values and latest
offsets matched.
- Actual BDB ordinary offset records omitted the schema and measured 1291
and 1302 bytes in the small-table scenario. A separate synthetic 1 MiB-schema
unit test checks that subsequent ordinary records remain below 10,000 bytes.
- Runtime validation reused an existing BE with `be_exec_version=15`.
Mixed-version FE rolling upgrades, downgrade replay and real multi-FE failover
were not exercised.
- Behavior changed:
- [x] Yes. Ordinary FROM-TO CDC offset journals omit unchanged schemas;
replay preserves the previously persisted schema. Images remain complete.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
