Userwhite opened a new pull request, #63110: URL: https://github.com/apache/doris/pull/63110
### What problem does this PR solve? Issue Number: close https://github.com/apache/doris/issues/61956 This PR support write and read path of binlog<row>. #### Write Path 1. TabletsChannel splits the request based on **OlapTableSchemaParam** and concurrently submits Data and Binlog tasks to the executor via GroupFlushContext. 2. At the storage layer, data is flushed by SegmentWriter and RowBinlogSegmentWriter respectively. RowBinlogSegmentWriter fills in partial column updates and "Before" data via Historical RetrieveContext. 3. During the Commit/Publish stages, the `delete_bitmap` is calculated synchronously. 4. In the Publish stage, the `delete_bitmap` inside the current Rowset is copied to `binlog_delvec`. 5. Finally, TxnManager uniformly persists the Rowset Meta. <img width="2654" height="1426" alt="image" src="https://github.com/user-attachments/assets/ddc8aef2-6804-40a2-8428-592b88334b62"; /> <img width="2100" height="546" alt="image" src="https://github.com/user-attachments/assets/a3ba33b7-e79c-4277-a8d3-cf241440a603"; /> #### Write Implementation Details - Concurrent Flush Model: Allows GroupFlushContext to submit both `DATA_IN_GROUP` and `BINLOG_IN_GROUP` tasks concurrently for the same SharedMemtable. - Reuse delete bitmap： Segment ID allocations for both writers are strictly synchronized during the flush phase. So `binlog_delvec` can resue tablet delete_bitmap(seq update, partial update conflict) #### Read Path 1. A query is initiated via TableBinlogFunction, and the READER_BINLOG flag is pushed down along the OlapScanNode to the underlying SegmentIterator. 2. After iterating through the physical data, the 64-bit Version/LSN and Commit TSO are injected into the result set in real-time via _update_lsn_col_if_needed and _update_tso_col_if_needed. <img width="1664" height="1362" alt="image" src="https://github.com/user-attachments/assets/aadc6fd2-7ae1-4a3f-80e3-c342e38740c5"; /> #### How to do simple test ```bash CREATE TABLE test_row_binlog_simple ( k1 INT, v1 INT, v2 STRING ) UNIQUE KEY(k1) DISTRIBUTED BY HASH(k1) BUCKETS 1 PROPERTIES ( "replication_num" = "1", "enable_unique_key_merge_on_write" = "true", "binlog.enable" = "true", "binlog.format" = "ROW", "binlog.need_historical_value" = "true" ); INSERT INTO test_row_binlog_simple VALUES (1, 10, '10'), (2, 20, '20'); UPDATE test_row_binlog_simple SET v1 = 11, v2 = '11' WHERE k1 = 1; SELECT __DORIS_BINLOG_LSN__ DIV 18446744073709551616 AS version, __DORIS_BINLOG_LSN__ % 18446744073709551616 AS row_id, __DORIS_BINLOG_OP__ AS op, k1, v1, v2, __BEFORE__v1__, __BEFORE__v2__ FROM binlog("table" = "test_row_binlog_simple") ORDER BY __DORIS_BINLOG_LSN__; ``` ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [x] Regression test - [x] Unit Test - [x] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
