bobhan1 opened a new pull request, #69719:
URL: https://github.com/apache/doris/pull/69719
### What problem does this PR solve?
A late sequence=90 write is rejected while a sequence=100 delete-sign row
exists, but can be accepted after compaction removes that row. Independent
compaction on replicas can therefore produce different results for the same
write.
Add the mutable, nonnegative BE configuration
`sequence_delete_sign_retention_version_gap` for shared-nothing MOW tables with
a row-level sequence column:
- Base/full/cold compaction prunes a delete-sign row only when the local
tablet's maximum published version minus its original row version reaches the
configured gap.
- Reuse `__DORIS_VERSION_COL__`, snapshot the configuration per reader, read
the local maximum under the existing tablet metadata lock, and share the filter
between horizontal and vertical readers. Read VERSION in the vertical key group
without making it a sort key.
- Keep legacy rows with missing/unknown versions conservatively. Default `0`
preserves immediate pruning. Query/checksum filtering and MOR/cloud behavior
remain unchanged.
This is bounded protection for monotonically increasing business sequences
with short ingestion disorder. All BEs must use the same setting, and the gap
must cover normal ingestion/publish/compaction lag. It does not guarantee
consistency for long-unavailable replicas or replay beyond the window. Enable
it after all BEs are upgraded, accounting for compactions still using the old
setting.
### Release note
Add `sequence_delete_sign_retention_version_gap` (default `0`) to delay
physical deletion of sequence delete-sign rows by a local published-version
gap. No new storage or wire format is introduced.
### Validation
- FE + ASAN BE build: `./build.sh --be --fe -j100`; final BE refresh:
`./build.sh --be -j100`.
- 53 ASAN BE tests: `./run-be-ut.sh --run
--filter='SequenceDeleteSignRetentionTest.*:TabletReaderTest.*:TestBaseCompaction.*:SparseColumnOptimizationTest.*'
-j100`.
- Local shared-nothing 1 FE + 1 ASAN BE: `./run-regression-test.sh --run -d
compaction -s
test_sequence_delete_sign_retention_version_gap,test_config_prune_delete_sign`
(2 suites passed). The new suite covers horizontal/vertical cumulative, base
and full compaction, original-version preservation, retention, exact expiry,
and lower-sequence writes before/after cleanup. Expected outputs were generated
by the runner and then compared in a separate run.
- Formatting, build hygiene and whitespace checks passed, retaining the
runner-generated `.out` trailing block separator. Clang-tidy checks changed
lines; config.h is analyzed separately without a PCH that already includes it.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- Behavior changed:
- [x] Yes, when the new configuration is positive; default `0` preserves
pruning behavior.
- Does this need documentation?
- [x] Yes: [configuration, prerequisites and upgrade
guide](https://github.com/bobhan1/doris/blob/feature/sequence-delete-sign-retention-version-gap/docs/sequence-delete-sign-retention.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
