Ryan19929 opened a new issue, #66265: URL: https://github.com/apache/doris/issues/66265
## Background Doris **Cross-Cluster Replication (CCR)** keeps a target cluster in sync with a source cluster: full initialization via `BACKUP`/`RESTORE`, then continuous incremental sync by pulling binlog commit sequences from the source and applying them on the target through transactional `ingest binlog`. The open-source syncer lives in [`selectdb/ccr-syncer`](https://github.com/selectdb/ccr-syncer); Doris FE/BE provide backup/restore, binlog, and ingest APIs. This is an **umbrella / tracking issue** to consolidate CCR-related work — bug fixes, feature requests, and full-sync performance improvements — across **apache/doris** and **selectdb/ccr-syncer**, so contributors and maintainers have a single place to see status and pick up work. Related umbrella: **#66252**. Please comment below to add items or volunteer. --- ## Open bugs & compatibility breakage - [ ] **selectdb/ccr-syncer#654** — Table-level CCR silently ignores upstream `DROP TABLE`; task keeps polling with no warning. → fix in **selectdb/ccr-syncer#660** *(open, lint CI failing — needs review)* - [ ] **selectdb/ccr-syncer#677** — Table-level CCR may miss table-id switch after `REPLACE TABLE … PROPERTIES('swap'='true')`; `REPLACE_TABLE` binlog `tableIds` only attaches `origTblId`. *(no Doris PR yet — needs `BinlogManager.addReplaceTable` fix)* - [ ] **#61063** — `get_binlog_info` returns not found for existing binlog files after tablet clone; CCR incremental ingest fails. *(no PR yet)* - [ ] **selectdb/ccr-syncer#648** — Deleting a job and immediately recreating one with the same name can corrupt or wipe task metadata. → fixed in **selectdb/ccr-syncer#659** *(merged 2025-12 — issue still open, pending verification & close)* - [ ] **#65632** — FE backup staging directories under `tmp_dir` may not be cleaned up after backup jobs. → fix in **#65642** *(open, needs review)* ## Open feature requests ### Storage medium sync Cross-cluster clusters often differ in SSD/HDD policy; restore and `MODIFY PARTITION` need explicit medium control on both sides. - [ ] **#58897** — Enhance restore storage medium control. *(open, needs review — large PR)* - [ ] **#63011** — Support medium allocation mode (part 1 of #58897 series). *(open, needs review)* - [ ] **selectdb/ccr-syncer#624** — Medium sync policy in CCR tasks (CREATE TABLE retry + MODIFY PARTITION medium sync). *(draft, conflicting — blocked on #58897 / #63011)* ### Replication-num heterogeneity - [ ] **#65321** — Strip replica info from backup meta. *(draft — complements selectdb/ccr-syncer#649, merged)* - [x] **selectdb/ccr-syncer#649** — Custom `replication_num` for downstream cluster. *(merged)* - [x] **selectdb/ccr-syncer#658** — Legacy job compatibility for `replication_num`. *(merged)* - [x] **selectdb/ccr-syncer#651** — Remove BE caching so restore sees current backend set after scale-out. *(merged)* - [x] Allow `atomic_restore` with different replica num. *(done — not yet in apache/doris)* ### CCR full-sync optimization CCR full sync is `BACKUP → RESTORE`; at scale FE memory, concurrency limits, and staging cleanup become bottlenecks. - [ ] Reduce FE memory usage of backup jobs with massive tablets (Gson DOM serialization, replica amplification, double journal serialization). *(no apache/doris PR yet)* - [ ] **#64492** — Streaming Gson adapter for RestoreJob Guava Table fields (restore-side memory). *(closed without merge — re-submit needed)* - [x] Streaming Gson adapter for RestoreJob Guava Table fields (restore-side memory). *(done — not yet in apache/doris)* - [ ] **#61710** — Support concurrent table-level backup/restore. *(open, needs review)* - [x] **#59580** — Preserve cross-database references when restoring views. *(merged)* - [x] **#63831** — Handle incomplete dynamic partition properties on restore. *(merged)* - [x] Clean backup local job dir after completion. *(done — not yet in apache/doris)* - [x] Centralize backup staging directory cleanup on checkpoint replay and job eviction. *(done — not yet in apache/doris)* ### Single-replica ingest binlog - [ ] Single-replica ingest binlog for 1-replica target clusters (paired FE/BE + syncer changes). *(no upstream PRs yet)* ### CCR ops & FE connectivity - [x] **selectdb/ccr-syncer#645** — JobCollector RPC connection reuse. *(merged — closes selectdb/ccr-syncer#644)* - [x] **selectdb/ccr-syncer#653** — Fix `/job_status` not updating after Pause/Resume. *(merged)* - [x] Return valid master address for `NOT_MASTER` errors. *(done — not yet in apache/doris)* - [x] Return master address for snapshot RPCs. *(done — not yet in apache/doris)* ## Open pull requests awaiting review | PR | Repo | Summary | Closes | |----|------|---------|--------| | **selectdb/ccr-syncer#660** | ccr-syncer | Auto-recover table sync when upstream table is recreated | selectdb/ccr-syncer#654 | | **selectdb/ccr-syncer#624** | ccr-syncer | Medium sync policy in CCR tasks *(draft)* | — | | **#65642** | doris | Clean up FE backup staging directories | #65632 | | **#58897** | doris | Enhance restore storage medium control | — | | **#63011** | doris | Support medium allocation mode | — | | **#61710** | doris | Support concurrent backup/restore | — | | **#65321** | doris | Strip replica info from backup meta *(draft)* | — | --- ## Recently completed (last ~12 months) ### ccr-syncer - [x] **selectdb/ccr-syncer#676** — Fix nil panic when transaction becomes visible. - [x] **selectdb/ccr-syncer#672** — Persist `Dest.TableId` after `partialSync` in TableSync mode. - [x] **selectdb/ccr-syncer#659** — Optimize job deletion to prevent same-name recreate races. Closes **selectdb/ccr-syncer#648** *(pending issue close)*. - [x] **selectdb/ccr-syncer#658** — Legacy job compatibility for `replication_num`. - [x] **selectdb/ccr-syncer#653** — Fix `/job_status` not updating after Pause/Resume. - [x] **selectdb/ccr-syncer#651** — Remove BE caching; fix restore with stale backend info after scale-out. - [x] **selectdb/ccr-syncer#649** — Support custom `replication_num` for downstream cluster. - [x] **selectdb/ccr-syncer#645** — JobCollector RPC connection reuse. Closes **selectdb/ccr-syncer#644**. - [x] **selectdb/ccr-syncer#638** — `tableRecord` add delta rows. - [x] **selectdb/ccr-syncer#486** — Handle dummy binlog. - [x] **selectdb/ccr-syncer#662** (w41ter) — Fix INSERT OVERWRITE in TableSync mode. Closes **selectdb/ccr-syncer#664**. ### apache/doris - [x] **#59580** — Preserve cross-database references when restoring views. - [x] **#63831** — Handle incomplete dynamic partition properties on restore. <details> <summary>Earlier / closed without merge (for reference)</summary> - **#59044** — Detect dropped tables early in binlog API *(stale-closed Jun 2026; referenced by selectdb/ccr-syncer#660)* - **#64492** — RestoreJob Gson streaming *(closed without merge)* - **#52391** / **#53120** — Earlier storage-medium attempts, superseded by #58897 / #63011 - **selectdb/ccr-syncer#647** — `atomic_restore` replica mismatch discussion → doris#42337 - **selectdb/ccr-syncer#652** / **#374** — Cross-version feasibility & getBinlog Q&A *(closed)* </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
