bitflicker64 commented on code in PR #510: URL: https://github.com/apache/hugegraph-doc/pull/510#discussion_r4178842090
########## content/en/docs/guides/toplingdb.md: ########## @@ -0,0 +1,163 @@ +--- +title: "ToplingDB development integration" +linkTitle: "ToplingDB (development)" +weight: 11 +description: "Build the optional ToplingDB runtime from development source, isolate its data, and understand startup and recovery boundaries." +--- + +> **Unreleased development interface.** This guide describes the proposed replacement for the 2025 ToplingDB integration. +> It requires a complete source checkout containing the integration and its lifecycle and snapshot-recovery prerequisites. +> These changes are under review; existing HugeGraph releases and registry tags do not establish support for this interface. +> Follow the [integration proposal](https://github.com/hugegraph/hugegraph/pull/264) for its delivery status. + +## Choose the process that owns the database + +| Deployment | Process that loads Topling JNI | Provider setting | +|---|---|---| +| Standalone RocksDB Server | Server | `rocksdb.provider=topling` in `conf/graphs/hugegraph.properties` | +| HStore PD | Each PD | `provider: topling` under `rocksdb` in `conf/application.yml` | +| HStore Store | Each Store | `provider: topling` under `rocksdb` in `conf/application-pd.yml` | +| HStore-backed Server | None | No local Topling setting | + +PD and Store select their providers independently. The HStore-backed Server is a remote client and does not need a Topling JAR or native library. +Provider selection happens at process startup. All local RocksDB graphs within one Server process must agree on the provider. + +The historical `rocksdb.option_path` and `rocksdb.open_http` examples describe a different integration. +Do not copy that configuration into this development interface. The component's launch scripts select the Easy Migrate YAML file. + +## Build an optional distribution + +Build on Linux x86_64 with Java 11+, Maven 3.5+, `rsync`, `unzip`, and `tar`. +The Server mount preflight additionally requires util-linux 2.37+ `mountpoint` on `PATH`. +Native macOS execution is outside this integration's support matrix; use a Linux x86_64 container when working on macOS. + +Obtain a compatible Topling Easy Migrate JNI JAR from its producer, together with its source revision, dependency licenses, notices, and trusted SHA-256. +The HugeGraph source tree does not include this binary. A checksum identifies an artifact; it does not establish its origin, license, or runtime compatibility. +Verify redistribution rights and retain the required license and notice files before distributing a derived package or image. + +From the complete source checkout containing this feature: + +```bash +VERSION=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout) +mvn clean package \ + -pl hugegraph-server/hugegraph-dist,hugegraph-pd/hg-pd-dist,hugegraph-store/hg-store-dist \ + -am -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -ntp + +export TOPLING_JNI_JAR=/absolute/path/to/rocksdbjni-topling.jar +export TOPLING_JNI_SHA256='<trusted 64-character SHA-256>' +install-dist/scripts/build-topling-distribution.sh server "$VERSION" +# Generate these only when running HStore: +install-dist/scripts/build-topling-distribution.sh pd "$VERSION" +install-dist/scripts/build-topling-distribution.sh store "$VERSION" +``` + +The Maven build creates standard packages. The generator creates separate `-topling` directories and archives beside them. +It copies the external JAR into private staging, verifies that copy, and prepares the native library. +Missing inputs, a checksum mismatch, or incompatible JAR structure stop generation; the generator does not search a Maven cache or download a substitute. +Build success is not a native runtime acceptance test. + +Each Topling package contains `lib/topling/rocksdbjni-topling.jar`, `lib/topling/runtime.properties`, +`library/librocksdbjni-linux64.so`, and the component's preparation and preload scripts. +To change the JNI JAR, regenerate the component distribution with the new trusted JAR and SHA-256, then run preparation. +`prepare-topling.sh` alone does not refresh the JAR checksum in `runtime.properties`. + +For container builds, use the matching checkout's `docker/README.md` and `docker/bake.hcl` with the same external JNI inputs. +Build locally before using the example `topling` tags and select `pull_policy: never` for that local deployment. +For registry images, verify the source revision, JNI identity, and digest first. A tag or a healthy container alone does not prove which JNI is running. + +## Configure isolated data + +Changing the provider does not migrate or convert data. Give Topling a separate data directory and keep the standard RocksDB directory unchanged. +Use these component-specific settings: + +| Component | Data setting | Example Topling root | +|---|---|---| +| Server | `rocksdb.data_path` | `/srv/hugegraph/topling/server/data` | +| Server WAL | `rocksdb.wal_path` | `/srv/hugegraph/topling/server/wal` | +| PD | `pd.data-path` | `/srv/hugegraph/topling/pd` | +| Store | `app.data-path` | `/srv/hugegraph/topling/store` | + +For a standalone Server, edit the generated distribution's `conf/graphs/hugegraph.properties`: + +```properties +backend=rocksdb +rocksdb.provider=topling +rocksdb.data_path=/srv/hugegraph/topling/server/data +rocksdb.wal_path=/srv/hugegraph/topling/server/wal +``` + +Both Server data and WAL paths must sit under the mounted `/srv/hugegraph/topling/server` root, outside the generated distribution. Do not leave WAL at its +default path inside the distribution; regenerating the package replaces that directory. +Mount or create the configured Topling root as a real directory before startup. Do not use symlinked path components. Review Comment: Important: Creating only the mounted root is not enough for the standalone Server. With `rocksdb.provider=topling`, `bin/init-store.sh` and `bin/start-hugegraph.sh` run `server_verify_graph_roots` before Java starts. For an unmarked path it calls `verify-rocksdb-provider.sh` on the configured `data_path` and `wal_path`, which fails with `RocksDB data path must be an existing directory` when either is missing. Only the standard-provider branch runs `mkdir -p`. Evidence: hugegraph/hugegraph#264 at 36058b0, `bin/rocksdb-server-config.sh` and `bin/verify-rocksdb-provider.sh`. This follows up the WAL thread (r4175878816). Please tell readers to run `mkdir -p /srv/hugegraph/topling/server/data /srv/hugegraph/topling/server/wal` before the first `bin/init-store.sh`; on the first run both must be empty real directories. Mirror this at line 90 of content/cn/docs/guides/toplingdb.md. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
