bobhan1 opened a new pull request, #70625:
URL: https://github.com/apache/doris/pull/70625

   ### Background
   
   MOW tables use sequence values to choose the current row for a primary key. 
A row with delete_sign=1 also participates in that comparison. Removing that 
row during base compaction can change whether the same later input is accepted.
   
   ### What problem does this PR solve?
   
   Issue Number: TODO(author): add a public feature issue before marking this 
draft ready
   Related PR: None
   PR Type: Feature
   
   Problem Summary:
   
   Add an opt-in partition-version retention window for delete_sign=1 rows on 
new MOW tables with row-level sequence. Fix the expiration cutoff for each 
transaction, validate its actual commit-version span, and use 
delete-bitmap-aware compaction for physical cleanup. The core implementation 
and targeted unit tests are present; this is a draft because production builds, 
cluster regression and the remaining integration paths are not complete.
   
   ### Details
   
   - **Change map:** 105 files: production code +2781/-79 lines in 83 files, 
tests +2198/-4 lines in 15 files, and optional protocol fields +90 lines in 7 
files. No generated files or documents are included. The main parts are FE 
table/planning/transaction handling, BE conflict decisions and compaction, and 
MS planning-context/commit validation.
   - **Scope:** New Unique Key MOW tables with row-level sequence, in 
shared-nothing and cloud modes. Column-group sequence_mapping and enabling 
existing tables through ALTER are outside this core scope. The implementation 
includes unit-tested fixed/flexible partial-update and changelog decision 
paths, without claiming full integration coverage.
   - **Design:** For a configured gap N, capture the partition visible version 
V_plan once per transaction/table/partition and fix the expiration cutoff at 
max(0, V_plan - N). Validate 0 < C - V_plan <= N against the actual assigned 
commit version C. Base/full compaction freezes a cleanup cutoff max(0, L - 2N), 
where L is its local complete, bitmap-ready published version; cumulative 
compaction retains delete signs. Readers apply the delete bitmap before merging 
rows and preserve their original VERSION values.
   - **Design:** Keep original write candidates and recompute final conflicts 
against the preceding published state. After a delete expires, a lower sequence 
may become current, so an input that lost an early comparison can win at the 
final decision. Discarding that input early would make flush timing observable. 
Finalization also rebuilds partial-update/changelog outputs when needed.
   - **Usage:** Set sequence_delete_retention_version_gap to a positive BIGINT 
in CREATE TABLE properties. Creation ensures the hidden VERSION column is 
present even when its global default is disabled. SHOW CREATE and existing 
table-property persistence retain the setting. The unit is partition versions, 
not wall-clock time or individual API requests.
   - **Default behavior:** The property defaults to zero. Existing tables 
retain their behavior. Immutable new-table policies let MS commit reuse the 
captured transaction policy; disabled inputs do not introduce per-tablet policy 
reads.
   - **Compatibility:** New protocol/storage fields are optional. All 
participating FE/BE/MS implementations must support the feature before it is 
enabled. No WAL changes are included. Online activation synchronization, 
history scans, heartbeat capability negotiation and mutable-policy generation 
fencing are not part of this draft.
   
   ### Testing
   
   - FE: five SequenceRetentionPlannerTest tests, four transaction 
SequenceRetentionTest tests and two CreateTableTest methods passed. CREATE 
coverage includes unpartitioned/partitioned tables, zero mode, hidden VERSION 
with the global default disabled, Gson persistence, SHOW CREATE and invalid 
prerequisites. The nine planner/transaction tests and the two successful CREATE 
methods were run in separate invocations.
   - BE ASAN: 55 tests passed through run-be-ut.sh with filter 
*SequenceRetention*:KeyProbeTest.*. Coverage includes expiration boundaries, 
early-loser/final-winner concurrency, partial update, changelog, and 
horizontal/vertical compaction.
   - MS ASAN: seven SequenceRetentionGuardTest, ten 
SequenceRetentionServiceTest and nine native UpdateDeleteBitmap tests passed 
through run-cloud-ut.sh. Coverage includes captured context, actual pending 
input, immediate/lazy/2PC version-window checks, retries and native bitmap 
behavior.
   - FE Checkstyle, clang-format v16, header hygiene and git diff --check 
passed. clang-tidy was not run.
   - These results precede the history squash; the squashed commit has the 
identical source tree. Production builds and real-cluster regression have not 
been run. No regression-test suites have been added yet.
   
   ### Notes for Reviewers
   
   This draft records the core implementation, not the complete expanded 
design. Remaining production-readiness work includes cluster regression and 
recovery/clone/schema-change handling that preserves committed lower-sequence 
decisions. The internal ignore_sequence_delete flag exists, but its HTTP/SQL 
user interface and Group Commit option handling are not complete. 
Replacement/restore version mapping and broader restart, failover and load-mode 
coverage also remain.
   
   Existing-table ALTER activation and the physical-time TTL alternative are 
deferred. Review should start with the fixed planning/commit window, final 
conflict decisions, and bitmap-aware compaction; do not infer correctness of 
all lifecycle paths from the unit-test totals.
   
   ### Release note
   
   New MOW tables with row-level sequence can opt into version-based retention 
of delete_sign=1 rows through sequence_delete_retention_version_gap. The 
expiration decision is independent of physical compaction cleanup.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: Yes; public table-property documentation 
remains to be added.
   - Pre-submission status: Not ready: targeted unit tests passed; production 
build, cluster regression and the integration work above remain incomplete. A 
public feature issue is also pending.
   
   ### AI Disclosure
   
   - Tool: OpenAI Codex
   - Generated: Code, tests and this description
   - Human review: Not yet performed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to