Ripening opened a new pull request, #70823:
URL: https://github.com/apache/doris/pull/70823

   ### What problem does this PR solve?
   
   Issue Number: ref #69410
   
   Related PR:
   
   Problem Summary:
   During Unique Key Merge-on-Write (MoW) schema change, a shadow tablet in 
`TABLET_NOTREADY` state enforces strict version continuity when scheduling 
publish tasks. When the base tablet still has earlier uncommitted transactions 
and the shadow tablet receives later double-write versions first, publish 
scheduling blocks on the version gap until the old transactions finish. 
Conversely, if scheduling admits gaps without recording them, subsequent 
history selection may only cover `request.alter_version` and miss those gaps.
   
   This PR coordinates publish scheduling and history conversion for MoW shadow 
tablets:
   1. **Tablet state (`be/src/storage/tablet/tablet.h`)**: Add 
`_schema_change_max_missing_version` (monotonically increasing watermark, 
initialized to `-1`) and `_schema_change_conversion_started` (initialized to 
`false`), protected by the header lock (`_meta_lock`).
   2. **Publish scheduling 
(`be/src/storage/task/engine_publish_version_task.cpp`)**: For 
`TABLET_NOTREADY`, non-restore, Unique Key MoW shadow tablets before history 
conversion starts, temporarily defer the continuity check and raise 
`_schema_change_max_missing_version` to `version.first - 1` when `version.first 
> max_version + 1`. Once history conversion starts, resume the original 
continuity check.
   3. **History selection (`be/src/storage/schema_change/schema_change.cpp`)**: 
Require base tablet continuous history to reach `max(request.alter_version, 
_new_tablet->schema_change_max_missing_version_unlocked())`. Mark 
`_schema_change_conversion_started = true` under the header write lock once 
base history preparation and reader initialization succeed.
   4. **BE restart recovery (`be/src/storage/storage_engine.cpp`)**: After 
`load_data_dirs()` completes in `StorageEngine::_open()` and before background 
threads start, rebuild `_schema_change_max_missing_version` for 
`TABLET_NOTREADY`, non-restore Unique Key MoW tablets via 
`get_missed_versions_unlocked(max_version)`.
   
   ### Check List (For Committer)
   
   - Test
   
       - [x] Regression test
       - [ ] Unit Test
       - [x] Manual test (add detailed scripts or steps below)
           - Added 
`regression-test/suites/schema_change_p0/test_mow_schema_change_history_boundary.groovy`
 covering shadow tablet version gaps during `WAITING_TXN` and post-alter 
publish consistency.
           - Verified BE crash (`SIGKILL`) & restart gap watermark rebuild 
(`[0,1], v3, v5` -> `4`, conversion started = `false`), post-restart 
publish/history concurrency, conversion-boundary crashes, and 3-replica repair 
consistency.
       - [ ] No need to test or manual test.
   
   - Behavior changed:
   
       - [ ] No.
       - [x] Yes. Unique Key MoW `TABLET_NOTREADY` shadow tablets (non-restore) 
temporarily defer publish continuity checks before history selection succeeds, 
while requiring history conversion to cover all admitted version gaps.
   
   - Does this need documentation?
   
       - [x] No.
       - [ ] Yes.
   
   - Release note
   
       Fix Unique Key MoW schema change publish blocking by coordinating shadow 
tablet version gaps and history coverage boundary.
   
   ### Check List (For Reviewer who merge this PR)
   
   - [ ] Confirm the release note
   - [ ] Confirm test cases
   - [ ] Confirm document
   - [ ] Add branch pick label
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to