bobhan1 opened a new pull request, #70625: URL: https://github.com/apache/doris/pull/70625
### Background MOW tables use sequence values to choose the current row for a primary key. A row with delete_sign=1 also participates in that comparison. Removing that row during base compaction can change whether the same later input is accepted. ### What problem does this PR solve? Issue Number: TODO(author): add a public feature issue before marking this draft ready Related PR: None PR Type: Feature Problem Summary: Add an opt-in partition-version retention window for delete_sign=1 rows on new MOW tables with row-level sequence. Fix the expiration cutoff for each transaction, validate its actual commit-version span, and use delete-bitmap-aware compaction for physical cleanup. The core implementation and targeted unit tests are present; this is a draft because production builds, cluster regression and the remaining integration paths are not complete. ### Details - **Change map:** 105 files: production code +2781/-79 lines in 83 files, tests +2198/-4 lines in 15 files, and optional protocol fields +90 lines in 7 files. No generated files or documents are included. The main parts are FE table/planning/transaction handling, BE conflict decisions and compaction, and MS planning-context/commit validation. - **Scope:** New Unique Key MOW tables with row-level sequence, in shared-nothing and cloud modes. Column-group sequence_mapping and enabling existing tables through ALTER are outside this core scope. The implementation includes unit-tested fixed/flexible partial-update and changelog decision paths, without claiming full integration coverage. - **Design:** For a configured gap N, capture the partition visible version V_plan once per transaction/table/partition and fix the expiration cutoff at max(0, V_plan - N). Validate 0 < C - V_plan <= N against the actual assigned commit version C. Base/full compaction freezes a cleanup cutoff max(0, L - 2N), where L is its local complete, bitmap-ready published version; cumulative compaction retains delete signs. Readers apply the delete bitmap before merging rows and preserve their original VERSION values. - **Design:** Keep original write candidates and recompute final conflicts against the preceding published state. After a delete expires, a lower sequence may become current, so an input that lost an early comparison can win at the final decision. Discarding that input early would make flush timing observable. Finalization also rebuilds partial-update/changelog outputs when needed. - **Usage:** Set sequence_delete_retention_version_gap to a positive BIGINT in CREATE TABLE properties. Creation ensures the hidden VERSION column is present even when its global default is disabled. SHOW CREATE and existing table-property persistence retain the setting. The unit is partition versions, not wall-clock time or individual API requests. - **Default behavior:** The property defaults to zero. Existing tables retain their behavior. Immutable new-table policies let MS commit reuse the captured transaction policy; disabled inputs do not introduce per-tablet policy reads. - **Compatibility:** New protocol/storage fields are optional. All participating FE/BE/MS implementations must support the feature before it is enabled. No WAL changes are included. Online activation synchronization, history scans, heartbeat capability negotiation and mutable-policy generation fencing are not part of this draft. ### Testing - FE: five SequenceRetentionPlannerTest tests, four transaction SequenceRetentionTest tests and two CreateTableTest methods passed. CREATE coverage includes unpartitioned/partitioned tables, zero mode, hidden VERSION with the global default disabled, Gson persistence, SHOW CREATE and invalid prerequisites. The nine planner/transaction tests and the two successful CREATE methods were run in separate invocations. - BE ASAN: 55 tests passed through run-be-ut.sh with filter *SequenceRetention*:KeyProbeTest.*. Coverage includes expiration boundaries, early-loser/final-winner concurrency, partial update, changelog, and horizontal/vertical compaction. - MS ASAN: seven SequenceRetentionGuardTest, ten SequenceRetentionServiceTest and nine native UpdateDeleteBitmap tests passed through run-cloud-ut.sh. Coverage includes captured context, actual pending input, immediate/lazy/2PC version-window checks, retries and native bitmap behavior. - FE Checkstyle, clang-format v16, header hygiene and git diff --check passed. clang-tidy was not run. - These results precede the history squash; the squashed commit has the identical source tree. Production builds and real-cluster regression have not been run. No regression-test suites have been added yet. ### Notes for Reviewers This draft records the core implementation, not the complete expanded design. Remaining production-readiness work includes cluster regression and recovery/clone/schema-change handling that preserves committed lower-sequence decisions. The internal ignore_sequence_delete flag exists, but its HTTP/SQL user interface and Group Commit option handling are not complete. Replacement/restore version mapping and broader restart, failover and load-mode coverage also remain. Existing-table ALTER activation and the physical-time TTL alternative are deferred. Review should start with the fixed planning/commit window, final conflict decisions, and bitmap-aware compaction; do not infer correctness of all lifecycle paths from the unit-test totals. ### Release note New MOW tables with row-level sequence can opt into version-based retention of delete_sign=1 rows through sequence_delete_retention_version_gap. The expiration decision is independent of physical compaction cleanup. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: Yes; public table-property documentation remains to be added. - Pre-submission status: Not ready: targeted unit tests passed; production build, cluster regression and the integration work above remain incomplete. A public feature issue is also pending. ### AI Disclosure - Tool: OpenAI Codex - Generated: Code, tests and this description - Human review: Not yet performed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
