Yukang-Lian opened a new pull request, #68398:
URL: https://github.com/apache/doris/pull/68398
### What problem does this PR solve?
Related PR: #48399 (introduced the schema change `STOP_TOKEN`), #49275
(unregisters the token when `commit_tablet_job` fails), #66082 (a load must not
preempt the schema change delete bitmap lock)
Problem Summary:
A heavy schema change on a cloud MOW table gets CANCELLED when its BE
restarts while the delete bitmap is being recalculated, and loads on the table
fail with delete bitmap lock conflicts for a while afterwards.
`CloudSchemaChangeJob::_process_delete_bitmap` registers a compaction
`STOP_TOKEN` on the new tablet before it recalculates the delete bitmap. When
the BE dies at that point, nothing unregisters the token: it stays in the meta
service until its lease expires (`lease_compaction_interval_seconds * 4` =
80s), and the dead run's initiator stays in the table's schema change delete
bitmap lock (`lock_id=-2`).
FE re-sends the ALTER task a couple of seconds after the BE comes back. The
schema change job record survives that restart on purpose
(`start_schema_change_job` takes the "same job restarts" path because the job
id and the BE-address initiator are unchanged), but the new run uses a fresh
`delete_bitmap_lock_initiator`, and `start_compaction_job` rejects its
`STOP_TOKEN` because the stale one is still there:
```
failed to start tablet job: compactions are not allowed on tablet_id=...
currently, blocked by schema change job delete_bitmap_initiator=...
```
The BE reports that as `INTERNAL_ERROR`, which `AlterJobV2.getRetryTimes`
does not retry, so FE cancels the whole schema change job. The new run also
cannot release the stale lock (`failed to remove delete bitmap update lock:
lock initiator not exist`), so until it expires
(`delete_bitmap_lock_expiration_seconds`) loads fail with `Failed to get delete
bitmap lock due to conflict`: a load may force-take a load/compaction lock but
never a schema change lock (#66082).
The same check ordering also breaks idempotency: a retried `STOP_TOKEN`
registration of the same run hits the `STOP_TOKEN` conflict check before the
same-id check and gets `JOB_TABLET_BUSY`.
### Changes
`start_compaction_job` no longer rejects an incoming `STOP_TOKEN` because of
an existing `STOP_TOKEN`. The incoming token replaces any stale token whose
`delete_bitmap_lock_initiator` differs, and the schema change delete bitmap
lock held by that stale initiator is released in the same transaction
(`remove_delete_bitmap_update_lock` with `SCHEMA_CHANGE_DELETE_BITMAP_LOCK_ID`,
respecting the instance's v1/v2 lock version, which `start_tablet_job` now
passes in). FE guarantees at most one schema change job per tablet, so the
incoming token always belongs to the live run. Regular compactions are still
blocked by a live token, and the BE/FE sides are unchanged.
New unit test `MetaServiceJobTest.StopTokenReplacesStaleStopToken`: a stale
token plus a `-2` lock held by the dead initiator; a regular compaction is
still rejected; the new token is accepted and the stale token and lock are
gone; re-registering the same token is idempotent; the live token still blocks
compactions.
### Testing
- `./run-cloud-ut.sh --run
--filter='meta_service_test:MetaServiceJobTest.*'` passes, including the new
test.
- Reproduced and verified end to end on a single FE/BE cloud cluster with a
debug point that blocks the schema change right after it acquires the delete
bitmap lock, then restarting the BE: without this change the re-sent ALTER task
is rejected and the job is CANCELLED every time; with it the meta service logs
`replace stale STOP_TOKEN ...` followed by `remove delete bitmap lock ...
lock_id=-2 initiator=<stale>`, the same job finishes within a few seconds of
the restart, and loads on the table are not blocked.
### Release note
Cloud MOW: a heavy schema change no longer gets CANCELLED when its BE
restarts while the delete bitmap is being recalculated, and loads on the table
are no longer blocked by the dead run's schema change lock.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. The meta service accepts a schema change `STOP_TOKEN` that
replaces a stale one on the same tablet and releases the stale run's schema
change delete bitmap lock.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
