Ripening opened a new pull request, #70823:
URL: https://github.com/apache/doris/pull/70823
### What problem does this PR solve?
Issue Number: ref #69410
Related PR:
Problem Summary:
During Unique Key Merge-on-Write (MoW) schema change, a shadow tablet in
`TABLET_NOTREADY` state enforces strict version continuity when scheduling
publish tasks. When the base tablet still has earlier uncommitted transactions
and the shadow tablet receives later double-write versions first, publish
scheduling blocks on the version gap until the old transactions finish.
Conversely, if scheduling admits gaps without recording them, subsequent
history selection may only cover `request.alter_version` and miss those gaps.
This PR coordinates publish scheduling and history conversion for MoW shadow
tablets:
1. **Tablet state (`be/src/storage/tablet/tablet.h`)**: Add
`_schema_change_max_missing_version` (monotonically increasing watermark,
initialized to `-1`) and `_schema_change_conversion_started` (initialized to
`false`), protected by the header lock (`_meta_lock`).
2. **Publish scheduling
(`be/src/storage/task/engine_publish_version_task.cpp`)**: For
`TABLET_NOTREADY`, non-restore, Unique Key MoW shadow tablets before history
conversion starts, temporarily defer the continuity check and raise
`_schema_change_max_missing_version` to `version.first - 1` when `version.first
> max_version + 1`. Once history conversion starts, resume the original
continuity check.
3. **History selection (`be/src/storage/schema_change/schema_change.cpp`)**:
Require base tablet continuous history to reach `max(request.alter_version,
_new_tablet->schema_change_max_missing_version_unlocked())`. Mark
`_schema_change_conversion_started = true` under the header write lock once
base history preparation and reader initialization succeed.
4. **BE restart recovery (`be/src/storage/storage_engine.cpp`)**: After
`load_data_dirs()` completes in `StorageEngine::_open()` and before background
threads start, rebuild `_schema_change_max_missing_version` for
`TABLET_NOTREADY`, non-restore Unique Key MoW tablets via
`get_missed_versions_unlocked(max_version)`.
### Check List (For Committer)
- Test
- [x] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- Added
`regression-test/suites/schema_change_p0/test_mow_schema_change_history_boundary.groovy`
covering shadow tablet version gaps during `WAITING_TXN` and post-alter
publish consistency.
- Verified BE crash (`SIGKILL`) & restart gap watermark rebuild
(`[0,1], v3, v5` -> `4`, conversion started = `false`), post-restart
publish/history concurrency, conversion-boundary crashes, and 3-replica repair
consistency.
- [ ] No need to test or manual test.
- Behavior changed:
- [ ] No.
- [x] Yes. Unique Key MoW `TABLET_NOTREADY` shadow tablets (non-restore)
temporarily defer publish continuity checks before history selection succeeds,
while requiring history conversion to cover all admitted version gaps.
- Does this need documentation?
- [x] No.
- [ ] Yes.
- Release note
Fix Unique Key MoW schema change publish blocking by coordinating shadow
tablet version gaps and history coverage boundary.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]