Userwhite opened a new pull request, #65810:
URL: https://github.com/apache/doris/pull/65810
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
This PR implements Row Binlog based on an independent row-binlog tablet
model for both non-cloud mode and cloud mode.
Before this change, Row Binlog data was tightly coupled with the base
tablet write/compaction path, and cloud mode did not have a complete end-
to-end implementation. This made it difficult to keep Row Binlog behavior
consistent across storage-compute integrated and storage-compute
separated deployments, especially for MOW tables, historical values,
partial update/upsert publish conflicts, and compaction.
The new implementation introduces row-binlog companion tablets and wires
them through FE metadata, BE write/publish/compaction/query paths, and
Cloud Meta Service. In non-cloud mode, Row Binlog write and compaction
are refactored around the independent tablet model. In cloud mode, the
same model is extended with cloud-specific rowset commit, publish,
delete-bitmap, metadata, and compaction handling.
Main changes:
1. Independent Row Binlog tablet model
- Store Row Binlog data in companion row-binlog tablets instead of
coupling it directly with base tablet rowsets.
- Track the relationship between base tablets and row-binlog tablets
through aligned tablet ids.
- Represent Row Binlog indexes with dedicated materialized-index states.
- Keep Row Binlog schema aligned with master: `__DORIS_BINLOG_TSO__`,
`__DORIS_BINLOG_LSN__`, `__DORIS_BINLOG_OP__`.
2. Non-cloud write, publish, and compaction path
- Support writing Row Binlog data through independent tablets in non-
cloud mode.
- Propagate Row Binlog tablet information through FE sink planning and BE
load/write context.
- Refactor Row Binlog compaction to reuse cumulative-compaction style
logic.
- Add `CumulativeCompactionBinlogPolicy` for non-cloud Row Binlog
compaction.
- Preserve MOW historical-value behavior, partial update/upsert
semantics, and delete-bitmap correctness.
3. Cloud write, publish, and metadata path
- Add cloud Row Binlog companion tablet creation and metadata flow.
- Add paired base-rowset + row-binlog-rowset commit through Meta Service.
- Support Cloud publish conflict handling for partial update and upsert.
- Make Cloud delete-bitmap cache, rowset builder/writer, tablet manager,
and transaction paths aware of Row Binlog rowsets.
- Persist Row Binlog tablet statistics and compaction metadata in Cloud
Meta Service.
4. Cloud Row Binlog compaction
- Add `CloudBinlogCumulativeCompactionPolicy`.
- Keep binlog-specific candidate selection separate from normal
compaction paths.
- Add `filter_new_visible_rowsets` to avoid compacting newly visible
rowsets too early unless backlog requires it.
- Fix cloud candidate selection so cloud rowsets are not incorrectly
filtered as non-local rowsets.
5. Query, schema change, restore, and observability
- Update Row Binlog scan/read paths to use TSO/LSN correctly.
- Support Row Binlog behavior across schema change, restore, tablet
report, SHOW/proc paths, and compaction actions.
- Expose Row Binlog tablet/index information where needed for debugging
and operation.
6. Tests
- Add/update BE unit tests for non-cloud and cloud Row Binlog write,
rowset builder/writer, transaction, and compaction policy behavior.
- Add/update regression tests for Row Binlog basic behavior, compaction,
schema change, multi-segment reads, Cloud compaction, and Cloud publish
conflict.
- Update regression ordering to use `__DORIS_BINLOG_TSO__,
__DORIS_BINLOG_LSN__` where result order depends on commit/visible TSO
semantics.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
