[I] [Feature][Task] 基于 RocksDB Checkpoint 实现 hugegraph-tools 物理增量备份与恢复支持 (Phase 1: 单机版) [hugegraph-toolchain]

2026-09-11 Thread via GitHub


imbajin opened a new issue, #770:
URL: https://github.com/apache/hugegraph-toolchain/issues/770

   ### Search before asking
   
   - [x] I had searched in the 
[feature](https://github.com/apache/hugegraph-toolchain/issues?q=is%3Aissue+label%3A%22Feature%22)
 and found no similar feature requirement.
   
   ### Feature Description (功能描述)
   
   ## 📌 1. 任务背景与目标 (Goal & Motivation)
   
   目前 `hugegraph-tools` 的备份与恢复(`backup` / `restore`)走的是**通用逻辑图数据(Logical 
Backup)**路线:通过 REST API 分片扫描点边并序列化为 JSON 文本。这种方式通用性好(可跨异构存储迁移),但对于主流的单机 
RocksDB(以及后续演进的 hstore)生产环境,存在显著瓶颈:
   1. **无法做增量备份**:缺少系统级修改时间戳与变更日志流,无法识别增量变动;
   2. **开销大、耗时长**:大图场景下全量 Shard 扫描并经由 HTTP 传输大量 JSON,网络与磁盘 I/O 极高;
   3. **物理删除(DELETE)无法捕获**:逻辑导出无法感知哪些点边已被删除,导致增量合并时无法重放删除操作,目标图残留脏数据。
   
   ### 本任务目标:
   利用 LSM-Tree 的 **不可变 SST 文件(Immutable SST)** 与 **RocksDB 
Checkpoint(秒级硬链接快照)** 原生能力,为 `hugegraph-tools` 打造生产级**物理增量备份与恢复(Physical 
Incremental Backup & Restore)**能力。
   
   > **阶段规划说明**:
   > - **Phase 1(本任务范围)**:聚焦**单机 RocksDB + Server 一体**架构下的**同一个图**物理增量备份与精准恢复。
   > - **Phase 2(后续规划)**:进一步联动 PD 协调器,扩展支持 **hstore(Multi-Raft + 
RocksDB)分布式集群**的增量物理快照备份。
   
   ---
   
   ## 🔍 2. 方案对比 (Before vs After)
   
   | 比较维度 | Before: 逻辑备份 (`tools backup`) | After: 物理增量备份 (`snapshot-backup`) |
   | :--- | :--- | :--- |
   | **底层原理** | REST API 分片扫描点边并序列化为 JSON | 调用 Server 接口触发底层 RocksDB Checkpoint 
硬链接 |
   | **增量机制** | **不支持**(Server 存储层硬编码禁止 Scan 下推条件过滤) | **原生增量**(仅同步增量产生的不可变 
`.sst` 数据文件) |
   | **删除感知** | **完全无法感知 DELETE 操作** | **完美包含所有 DELETE**(RocksDB 底层 Tombstone 
原样保留) |
   | **生成耗时** | 线性依赖数据量(千万/亿级点边耗时数十分钟甚至数小时) | **毫秒级 O(1) 完成**(系统级 Hard 
Link,与图大小无关) |
   | **业务影响** | 长时间占用 Server CPU、磁盘 IO 和网络带宽 | **零写入阻塞**,对在线业务读写几乎无影响 |
   | **恢复速度** | 逐条调用 `addVertices`/`addEdges` 重新解析写入,慢 | 直接加载/软链接 SST 文件,秒级完成恢复 
|
   
   ---
   
   ## 💡 3. 核心设计与执行时序 (Architecture & Flow)
   
   ```mermaid
   sequenceDiagram
   autonumber
   actor Admin as 管理员 / 定时调度任务
   participant Tools as hugegraph-tools
   participant Client as hugegraph-client
   participant Server as hugegraph-server
   participant RocksDB as RocksDB 引擎
   participant Storage as 备份归档目录
   
   rect rgb(240, 248, 255)
   Note over Admin, Storage: 增量备份阶段 (Incremental Backup)
   Admin->>Tools: 执行 snapshot-backup --graph g -d /backup
   Tools->>Client: client.graphs().createSnapshot("g")
   Client->>Server: PUT /graphs/{name}/snapshot_create
   Server->>RocksDB: sessions.createSnapshot() -> Checkpoint.create()
   RocksDB-->>Server: 秒级生成快照目录 (SST 硬链接)
   Server-->>Tools: 返回快照创建成功与路径
   Tools->>Tools: 对比上次备份元数据 (Manifest),找出新增/变更的 .sst 文件
   Tools->>Storage: 仅增量拷贝新增 .sst 文件 + 写入新的备份元数据 (metadata.json)
   Tools->>Server: 清理临时 Checkpoint 目录
   Tools-->>Admin: 增量备份完成 (耗时秒级,仅传输增量数据)
   end
   
   rect rgb(255, 245, 245)
   Note over Admin, Storage: 恢复阶段 (Restore)
   Admin->>Tools: 执行 snapshot-restore --graph g -d /backup --backup-id 
   Tools->>Storage: 根据 backup-id 回溯基线与增量 SST 链条
   Tools->>Server: 停止图服务 或 设置维护模式
   Tools->>Server: 将对应版本的 SST 集合恢复至 Server 数据目录
   Tools->>Client: client.graphs().resumeSnapshot("g") 或 重启服务
   Tools-->>Admin: 精准恢复到指定时间点状态 (包含新增/修改/删除)
   end
   ```
   
   ---
   
   ## 🛠️ 4. 详细开发拆解与实现步骤 (Implementation Steps)
   
   ### Step 1: `hugegraph-client` 补全快照管理 API
   在 `hugegraph-client` 的 `GraphsManager` 中封装快照 REST 请求(目前 Server 端已具备该接口):
   - `createSnapshot(String graph)`: 对应调用 `PUT /graphs/{name}/snapshot_create`
   - `resumeSnapshot(String graph)`: 对应调用 `PUT /graphs/{name}/snapshot_resume`
   
   > **服务端已有实现参考**:
   > - REST API 
端点:[`GraphsAPI.java#L612-L640`](https://github.com/apache/hugegraph/blob/master/hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/api/profile/GraphsAPI.java#L612-L640)
   > - RocksDB Checkpoint 
调用:[`RocksDBStdSessions.java#L254-L256`](https://github.com/apache/hugegraph/blob/master/hugegraph-server/hugegraph-rocksdb/src/main/java/org/apache/hugegraph/backend/store/rocksdb/RocksDBStdSessions.java#L254-L256)
   
   ### Step 2: `hugegraph-tools` 新增命令行参数定义
   在 
[`SubCommands.java`](https://github.com/apache/hugegraph-toolchain/blob/master/hugegraph-tools/src/main/java/org/apache/hugegraph/cmd/SubCommands.java)
 中新增快照备份与恢复命令参数:
   - `snapshot-backup`:
 - `--directory, -d`: 备份文件存放根目录(必填)
 - `--mode, -m`: 备份模式,支持 `full`(强制全量)或 `incremental`(增量,默认)
 - `--keep-num`: 保留的历史快照版本数量(默认 7 份)
   - `snapshot-restore`:
 - `--directory, -d`: 备份根目录(必填)
 - `--backup-id`: 指定要恢复的历史快照版本 ID(不传则恢复最新一份)
   
   ### Step 3: 实现快照与增量文件管理器 (`SnapshotManager`)
   新建 `org.apache.hugegraph.manager.SnapshotManager`:
   1. **元数据管理 (`metadata.json`)**:
  记录每次备份的快照清单:
  ```json
  {
"backup_id": "backup_20260911_18",
"timestamp": 178912080,
"graph": "hugegraph",
"base_backup_id": "backup_20260911_12",
"files": [
  {"name": "000124.sst", "size": 1048576, "checksum": 

Re: [I] [Feature][Task] 基于 RocksDB Checkpoint 实现 hugegraph-tools 物理增量备份与恢复支持 (Phase 1: 单机版) [hugegraph-toolchain]

2026-09-11 Thread via GitHub


imbajin commented on issue #770:
URL: 
https://github.com/apache/hugegraph-toolchain/issues/770#issuecomment-5632932256

   Created by mistake in apache org instead of hugegraph org. Closing this 
issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [I] [Feature][Task] 基于 RocksDB Checkpoint 实现 hugegraph-tools 物理增量备份与恢复支持 (Phase 1: 单机版) [hugegraph-toolchain]

2026-09-11 Thread via GitHub


imbajin closed issue #770: [Feature][Task] 基于 RocksDB Checkpoint 实现 
hugegraph-tools 物理增量备份与恢复支持 (Phase 1: 单机版)
URL: https://github.com/apache/hugegraph-toolchain/issues/770


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]