imbajin commented on code in PR #510: URL: https://github.com/apache/hugegraph-doc/pull/510#discussion_r4179655899
########## content/cn/docs/guides/toplingdb.md: ########## @@ -0,0 +1,153 @@ +--- +title: "ToplingDB 开发版集成" +linkTitle: "ToplingDB（开发版）" +weight: 11 +description: "从开发源码构建可选 ToplingDB 运行时，隔离数据目录，并了解启动与恢复边界。" +--- + +> **尚未发布的开发接口。** 本文介绍用于替换 2025 年 ToplingDB 集成的候选实现。 +> 需要使用包含该集成及其生命周期、快照恢复前置修改的完整源码。 +> 这些修改仍在审查中；已有 HugeGraph 发行版和镜像标签不能证明其支持本文接口。 +> 交付状态见[集成提案](https://github.com/hugegraph/hugegraph/pull/264)。 + +## 确认哪个进程持有数据库 + +| 部署方式 | 加载 Topling JNI 的进程 | 运行时配置 | +|---|---|---| +| 单机 RocksDB Server | Server | `conf/graphs/hugegraph.properties` 中的 `rocksdb.provider=topling` | +| HStore PD | 每个 PD | `conf/application.yml` 中 `rocksdb` 下的 `provider: topling` | +| HStore Store | 每个 Store | `conf/application-pd.yml` 中 `rocksdb` 下的 `provider: topling` | +| 使用 HStore 的 Server | 无 | 不配置本地 Topling 运行时 | + +PD 和 Store 分别选择自己的运行时。使用 HStore 的 Server 是远程客户端，不需要 Topling JAR 或原生库。 +运行时在进程启动时选定。同一 Server 进程中的所有本地 RocksDB 图必须使用相同的运行时。 + +历史文档中的 `rocksdb.option_path` 和 `rocksdb.open_http` 示例属于另一套集成方式。 +不要将这些配置复制到本文的开发接口；组件启动脚本负责选择 Easy Migrate YAML 文件。 + +## 构建可选发行包 + +在 Linux x86_64 上构建，需要 Java 11+、Maven 3.5+、`rsync`、`unzip` 和 `tar`。 +Server 挂载预检查还要求 `PATH` 中有 util-linux 2.37+ 的 `mountpoint`。 +此集成不支持在 macOS 原生运行；macOS 用户应使用 Linux x86_64 容器。 + +从生产方获取兼容的 Topling Easy Migrate JNI JAR，以及对应源码版本、依赖许可证、声明文件和可信 SHA-256。 +HugeGraph 源码不包含该二进制文件。校验和只能识别产物，不能证明来源、许可证或运行时兼容性。 +分发衍生包或镜像前，核查再分发权限并保留所需许可证和声明文件。 + +在包含此功能的完整源码目录中执行： + +```bash +VERSION=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout) +mvn clean package \ + -pl hugegraph-server/hugegraph-dist,hugegraph-pd/hg-pd-dist,hugegraph-store/hg-store-dist \ + -am -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -ntp + +export TOPLING_JNI_JAR=/absolute/path/to/rocksdbjni-topling.jar +export TOPLING_JNI_SHA256='<trusted 64-character SHA-256>' +install-dist/scripts/build-topling-distribution.sh server "$VERSION" +# 仅在运行 HStore 时生成以下两个组件： +install-dist/scripts/build-topling-distribution.sh pd "$VERSION" +install-dist/scripts/build-topling-distribution.sh store "$VERSION" +``` + +构建命令先生成标准发行包。生成脚本在其旁边创建独立的 `-topling` 目录及压缩包。 +脚本将外部 JAR 复制到私有暂存目录，校验该副本后再准备原生库。 +缺少输入、校验和不一致或 JAR 结构不符合要求时，生成过程会停止；脚本不会搜索 Maven 缓存或下载替代文件。 +构建成功不等于原生运行时验收通过。 + +每个 Topling 包包含 `lib/topling/rocksdbjni-topling.jar`、`lib/topling/runtime.properties`、 +`library/librocksdbjni-linux64.so` 以及组件的准备和预加载脚本。 +更换已安装的 Topling JAR 后，重新运行 `bin/prepare-topling.sh`。 + +构建容器时，使用同一源码中的 `docker/README.md` 和 `docker/bake.hcl`，并传入相同的外部 JNI 参数。 +先在本地完成构建，再使用示例中的 `topling` 标签，本地部署选择 `pull_policy: never`。 +使用镜像仓库中的产物前，先核对源码版本、JNI 身份和镜像摘要。标签或容器健康状态本身不能证明实际运行的 JNI。 + +## 配置独立数据目录 + +切换运行时不会迁移或转换数据。为 Topling 设置独立目录，保留标准 RocksDB 目录不变。 +各组件使用以下配置： + +| 组件 | 数据配置 | Topling 根目录示例 | +|---|---|---| +| Server | `rocksdb.data_path` | `/srv/hugegraph/topling/server` | +| PD | `pd.data-path` | `/srv/hugegraph/topling/pd` | +| Store | `app.data-path` | `/srv/hugegraph/topling/store` | + +单机 Server 修改生成包中的 `conf/graphs/hugegraph.properties`： + +```properties +backend=rocksdb +rocksdb.provider=topling +rocksdb.data_path=/srv/hugegraph/topling/server +``` + +启动前，挂载或创建实际的 Topling 根目录，不要使用包含符号链接的路径。 +挂载父数据根目录，不要单独挂载 `data/g` 等数据库目录，确保所有遵循恢复协议的进程都能看到同级恢复文件。 +单机 Server 启动脚本在 Java 启动前拒绝单独挂载的数据库目录，包括 Linux 上同一文件系统的绑定挂载。 +启动和恢复期间保持挂载布局不变。 + +`.hugegraph-rocksdb-provider` 标记用于防止误用目录，不负责数据转换。 +标记冲突时停止启动。Topling 也会拒绝没有标记的非空数据目录。 +标准 RocksDB 为兼容历史数据而接受已有的无标记目录，但这不意味着可以打开经 Topling 修改的目录。 Review Comment: 已同步收窄中英文指南并配对新的 TP 核心 hugegraph/hugegraph#266：普通发行包使用显式 runtime 选择，启动时核对实际 JNI 与业务 provider；不再承诺 marker 认领、已有数据库归属识别或 Docker 目录保护。指南要求使用独立、初始为空的数据目录，并配置包外 data/WAL/Raft。目录归属强校验和 Docker 属于独立后续，当前文档没有把裸包路径说成已受这些保护。 ########## content/en/docs/guides/toplingdb.md: ########## @@ -0,0 +1,163 @@ +--- +title: "ToplingDB development integration" +linkTitle: "ToplingDB (development)" +weight: 11 +description: "Build the optional ToplingDB runtime from development source, isolate its data, and understand startup and recovery boundaries." +--- + +> **Unreleased development interface.** This guide describes the proposed replacement for the 2025 ToplingDB integration. +> It requires a complete source checkout containing the integration and its lifecycle and snapshot-recovery prerequisites. +> These changes are under review; existing HugeGraph releases and registry tags do not establish support for this interface. +> Follow the [integration proposal](https://github.com/hugegraph/hugegraph/pull/264) for its delivery status. + +## Choose the process that owns the database + +| Deployment | Process that loads Topling JNI | Provider setting | +|---|---|---| +| Standalone RocksDB Server | Server | `rocksdb.provider=topling` in `conf/graphs/hugegraph.properties` | +| HStore PD | Each PD | `provider: topling` under `rocksdb` in `conf/application.yml` | +| HStore Store | Each Store | `provider: topling` under `rocksdb` in `conf/application-pd.yml` | +| HStore-backed Server | None | No local Topling setting | + +PD and Store select their providers independently. The HStore-backed Server is a remote client and does not need a Topling JAR or native library. +Provider selection happens at process startup. All local RocksDB graphs within one Server process must agree on the provider. + +The historical `rocksdb.option_path` and `rocksdb.open_http` examples describe a different integration. +Do not copy that configuration into this development interface. The component's launch scripts select the Easy Migrate YAML file. + +## Build an optional distribution + +Build on Linux x86_64 with Java 11+, Maven 3.5+, `rsync`, `unzip`, and `tar`. +The Server mount preflight additionally requires util-linux 2.37+ `mountpoint` on `PATH`. +Native macOS execution is outside this integration's support matrix; use a Linux x86_64 container when working on macOS. + +Obtain a compatible Topling Easy Migrate JNI JAR from its producer, together with its source revision, dependency licenses, notices, and trusted SHA-256. +The HugeGraph source tree does not include this binary. A checksum identifies an artifact; it does not establish its origin, license, or runtime compatibility. +Verify redistribution rights and retain the required license and notice files before distributing a derived package or image. + +From the complete source checkout containing this feature: + +```bash +VERSION=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout) +mvn clean package \ + -pl hugegraph-server/hugegraph-dist,hugegraph-pd/hg-pd-dist,hugegraph-store/hg-store-dist \ + -am -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -ntp + +export TOPLING_JNI_JAR=/absolute/path/to/rocksdbjni-topling.jar +export TOPLING_JNI_SHA256='<trusted 64-character SHA-256>' +install-dist/scripts/build-topling-distribution.sh server "$VERSION" +# Generate these only when running HStore: +install-dist/scripts/build-topling-distribution.sh pd "$VERSION" +install-dist/scripts/build-topling-distribution.sh store "$VERSION" +``` + +The Maven build creates standard packages. The generator creates separate `-topling` directories and archives beside them. +It copies the external JAR into private staging, verifies that copy, and prepares the native library. +Missing inputs, a checksum mismatch, or incompatible JAR structure stop generation; the generator does not search a Maven cache or download a substitute. +Build success is not a native runtime acceptance test. + +Each Topling package contains `lib/topling/rocksdbjni-topling.jar`, `lib/topling/runtime.properties`, +`library/librocksdbjni-linux64.so`, and the component's preparation and preload scripts. +To change the JNI JAR, regenerate the component distribution with the new trusted JAR and SHA-256, then run preparation. +`prepare-topling.sh` alone does not refresh the JAR checksum in `runtime.properties`. + +For container builds, use the matching checkout's `docker/README.md` and `docker/bake.hcl` with the same external JNI inputs. +Build locally before using the example `topling` tags and select `pull_policy: never` for that local deployment. +For registry images, verify the source revision, JNI identity, and digest first. A tag or a healthy container alone does not prove which JNI is running. + +## Configure isolated data + +Changing the provider does not migrate or convert data. Give Topling a separate data directory and keep the standard RocksDB directory unchanged. +Use these component-specific settings: + +| Component | Data setting | Example Topling root | +|---|---|---| +| Server | `rocksdb.data_path` | `/srv/hugegraph/topling/server/data` | +| Server WAL | `rocksdb.wal_path` | `/srv/hugegraph/topling/server/wal` | +| PD | `pd.data-path` | `/srv/hugegraph/topling/pd` | +| Store | `app.data-path` | `/srv/hugegraph/topling/store` | + +For a standalone Server, edit the generated distribution's `conf/graphs/hugegraph.properties`: + +```properties +backend=rocksdb +rocksdb.provider=topling +rocksdb.data_path=/srv/hugegraph/topling/server/data +rocksdb.wal_path=/srv/hugegraph/topling/server/wal +``` + +Both Server data and WAL paths must sit under the mounted `/srv/hugegraph/topling/server` root, outside the generated distribution. Do not leave WAL at its +default path inside the distribution; regenerating the package replaces that directory. +Mount or create the configured Topling root as a real directory before startup. Do not use symlinked path components. Review Comment: Both guides now direct the service account to create persistent, initially empty directories outside the distribution before startup, and explicitly set both Server data and WAL paths. The example paths are /srv/hugegraph/topling/server/data and /srv/hugegraph/topling/server/wal. The guide now pairs with hugegraph/hugegraph#266, which omits the old shell ownership guards rather than claiming them for bare launchers. The actual standard/Topling Server init, write/read, clear/rewrite, normal stop/restart and dump-store acceptance has passed for that scoped implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
