Ryan19929 opened a new issue, #66982: URL: https://github.com/apache/doris/issues/66982
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description Backup and restore jobs persist metadata such as BackupMeta, RestoreJob tablet/partition mappings, and job state through FE edit logs, images, and backup metadata files as JSON. For ordinary catalogs this is usually acceptable. With hundreds of thousands or millions of tablets and multiple replicas per tablet, however, one persistence or replay operation can retain several complete-payload-sized representations at the same time: 1. a complete Gson JsonElement DOM for polymorphic objects and large Guava Table/Multimap values; 2. the complete JSON as a Java String and then as a UTF-8 byte array; 3. another complete payload buffer used only by the BDB journal-size preflight; 4. source-cluster Replica objects retained in BackupMeta even though Restore reconstructs replicas from the target ReplicaAllocation. These allocations scale linearly with tablet count, can approach the signed-int payload limit, and may cause long Full GC pauses or FE OOM during large backup/restore operations and replay after FE restart. The investigation also found two correctness problems: - After TableRef was migrated to TableRefInfo, affected versions could persist selected table references as empty JSON objects. A replayed PENDING BackupJob then repeatedly failed with a context-free NullPointerException. The original selected scope cannot be inferred safely from an already-corrupted journal. - A streaming table-metadata deep-copy failure could be converted into null and later surface as a context-free NullPointerException in TRUNCATE, hiding the original serialization, spill, or read failure. This issue intentionally does not claim to bound every FE metadata allocation. The detached table-copy object graph and the current BDB writer's full JournalEntity buffer still scale with payload size. Other full-String catalog paths, spill quotas/metrics/startup cleanup, enabling streaming by default, the independent backup-timeout fix, and final million-tablet capacity benchmarking are follow-up work. ### Solution This issue tracks six strictly ordered PRs. Each PR will be submitted after its prerequisite has merged so that every review remains focused and every intermediate master revision remains independently buildable. ## Implementation plan - [ ] PR0 — Preserve backup table references across replay - [ ] PR1 — Strip replica info from backup metadata ([draft #65321](https://github.com/apache/doris/pull/65321)) - [ ] PR2 — Count backup journal serialized bytes without buffering - [ ] PR3 — Add streaming Gson adapter foundation - [ ] PR4 — Stream RestoreJob JSON persistence - [ ] PR5 — Bound backup metadata JSON memory usage ### PR0 — Preserve backup table references across replay Restore stable serialized names, accept historical table-name keys, and cancel already-corrupted PENDING jobs once with an actionable resubmission error instead of guessing their requested scope. Later job states are unaffected. ### PR1 — Strip replica info from backup metadata Remove source-cluster Replica objects from the detached BackupMeta table copy by default. Restore already creates new replicas from the target cluster's ReplicaAllocation. Historical backup metadata remains readable, and backup_meta_reserve_replica_info=true preserves the previous representation for subsequent backups. ### PR2 — Count backup journal serialized bytes without buffering Replace the O(payload) BDB journal-size preflight buffer with an O(1)-heap counting stream using the real serialization path. Persist only the valid DatabaseEntry byte range; old padded and new compact entries remain readable. ### PR3 — Add streaming Gson adapter foundation Add schema-preserving streaming adapters for polymorphic types and Guava Table/Multimap. Canonical output remains byte-compatible; historical non-first type fields use a single-object fallback. This PR is infrastructure only. ### PR4 — Stream RestoreJob JSON persistence Apply the streaming infrastructure to polymorphic Backup/Restore jobs and RestoreJob's large Table fields behind enable_backup_restore_job_streaming_json, default false. Legacy and streaming reader/writer combinations remain compatible, including replay redo and cleanup states. ### PR5 — Bound backup metadata JSON memory usage Stream BackupMeta, Table/OlapTable, BackupJobInfo, and selected job payloads through an approximately 8 MiB heap buffer per active writer, spilling larger payloads to unique temporary files. Preserve int32-length + UTF-8 framing, bounded reads, cleanup on failure, and original exception causes. Table metadata streaming is gated by enable_table_meta_streaming_json, default false. ## Compatibility - No new opcode, metadata version, FE/BE protocol, compression marker, or canonical subtype label. - Legacy and streaming reader/writer combinations use the same logical JSON schema. - Both streaming switches default to off. - PR0 accepts historical field names. - PR2 removes only unread BDB backing-array padding; old and new entries are mutually readable. - Replica stripping affects newly created BackupMeta by default and can be reverted for future backups with backup_meta_reserve_replica_info=true. ## Validation The six-layer local stack was rebased onto upstream/master@6e887178b994. The final stack passed git diff --check, the standard ./build.sh --fe build, and 87/87 targeted FE unit tests across 15 test classes. Backup/restore regression compatibility and cloud restore-cancellation E2E tests passed on a patch-equivalent pre-rebase stack. They were not rerun after the final rebase, so they are retained as historical evidence rather than claimed as final-head E2E results. Memory benefits are currently described as structural removal of DOM, String, byte-array, preflight-buffer, and Replica allocations. A final large-catalog capacity benchmark is follow-up work and is not claimed as a measured result here. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
