Ryan19929 opened a new pull request, #67143:
URL: https://github.com/apache/doris/pull/67143
### What problem does this PR solve?
Issue Number: close #67120
Related PR: None
Problem Summary:
In Shared-Nothing CCR incremental replication, `IngestBinlog` discarded both
the status returned by `BaseTablet::commit_phase_update_delete_bitmap()` and
the asynchronous delete bitmap token's `wait()` status.
For UNIQUE KEY merge-on-write tablets, a delete bitmap calculation failure
could therefore be logged but not propagated. The downloaded rowset could
still be committed and published with an incomplete delete bitmap, exposing
historical rows and producing duplicate unique keys.
The failure cleanup also captured `download_success_files` by value before
downloads occurred, so downloaded segment and index files were not removed.
Additionally, the result of `batch_delete()` was discarded and cleanup could
be logged as successful even when deletion failed.
This PR:
- preserves and propagates synchronous task-submission failures;
- waits for already-submitted delete bitmap tasks and propagates asynchronous
worker failures;
- returns before `commit_txn()` so the prepared tablet transaction is
aborted;
- captures downloaded files by reference for failure cleanup;
- reports `batch_delete()` failures without replacing the original ingest
error;
- adds a CCR MOW fault-injection regression case that verifies the failed
version remains invisible and the same binlog can be retried safely.
### Release note
Fix CCR replay for MOW tables so delete bitmap calculation failures abort
`IngestBinlog` instead of publishing a rowset with an incomplete delete
bitmap.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- `test_mow_ingest_binlog`, including the new asynchronous delete
bitmap failure case, passed.
- CCR P0 cases covering MOW and non-MOW `_ingest_binlog` paths
passed.
- [ ] Unit Test
- [x] Manual test
- Full FE and BE ASAN build passed with clang 20.1.7.
- A Docker two-cluster CCR environment was used, with separate source
and target clusters and binlog enabled.
- Injected `BaseTablet::calc_segment_delete_bitmap.inject_err` on the
target BE.
- Verified that ingest failed before `commit_txn()`, the failed
version
remained invisible, no duplicate key became visible, and retrying
the same binlog succeeded.
- `build-support/clang-format.sh` and
`build-support/check-format.sh` passed with clang-format 16.
- Changed-lines clang-tidy completed with zero warnings.
- Cleanup failure reporting was validated by compilation and static
analysis; a filesystem `batch_delete()` failure was not injected.
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Known environment-only failures unrelated to this change:
- Two `test_create_table_with_binlog_config` cases conflict with the Docker
image's forced `force_enable_feature_binlog` setting.
- `test_is_being_synced` requires an unavailable external S3 bucket.
- clang-tidy 20 reports a pre-existing orphan `NOLINTEND` at `types.h:576`;
no warning was reported on the lines changed by this PR.
- Behavior changed:
- [ ] No.
- [x] Yes. CCR `IngestBinlog` now returns delete bitmap calculation
errors
and aborts before committing the rowset. Cleanup failures are also
reported accurately.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
