imbajin commented on code in PR #510: URL: https://github.com/apache/hugegraph-doc/pull/510#discussion_r4207998311
########## content/en/docs/guides/toplingdb.md: ########## @@ -0,0 +1,118 @@ +--- +title: "ToplingDB development integration" +linkTitle: "ToplingDB (development)" +weight: 11 +description: "Build the optional ToplingDB runtime from development source, isolate its data, and understand startup and recovery boundaries." +--- + +> **Unreleased development interface.** This guide follows the [three-component runtime integration](https://github.com/hugegraph/hugegraph/pull/266). +> Use source containing that change. Existing releases and image tags do not establish support, and runtime acceptance remains a deployment prerequisite. + +## Choose the database owner + +HugeGraph defaults to standard RocksDB. Enable Topling separately for each process that owns local storage: + +| Component | Business provider setting | Start command | +| --- | --- | --- | +| Standalone RocksDB Server | `rocksdb.provider=topling` in each selected graph properties file | `bin/start-hugegraph.sh` | +| PD | `provider: topling` under `rocksdb` in `conf/application.yml` | `bin/start-hugegraph-pd.sh` | +| Store | `provider: topling` under `rocksdb` in `conf/application-pd.yml` | `bin/start-hugegraph-store.sh` | + +A standalone Server keeps `backend=rocksdb`; an HStore-backed Server uses `backend=hstore` and needs no local Topling runtime. +All local RocksDB graphs in one Server process must use the same provider. The business setting must match the runtime selected by its launcher. + +## Build and prepare a standard distribution + +Build the normal distributions from the source repository root using Java 17 and Maven 3.6.3 or later: + +```bash +mvn clean package -Dmaven.test.skip=true -Dmaven.javadoc.skip=true +``` + +Obtain a trusted Topling Easy Migrate JNI JAR and its SHA-256 through your artifact channel. Preparation requires a Java 17 JDK, Linux x86_64, +`sha256sum`, `unzip`, `od`, `ldd`, and GNU `mv`. The selected JAR's native dependencies, including its glibc and libaio requirements, must be installed. + +Stop the component, unpack a fresh standard distribution, and run from that component directory: + +```bash +export TOPLING_JNI_JAR=/absolute/path/to/rocksdbjni-topling.jar +export TOPLING_JNI_SHA256='<trusted-64-character-sha256>' +bash bin/prepare-topling.sh +``` + +The script verifies the copied JAR's hash, Topling Java marker, one Linux x86_64 JNI entry, ELF architecture and native dependencies. +Before installation, a disposable database verifies that the selected JNI actually applies EasyMigrate configuration to its persisted write-buffer size. +Unsupported builds are rejected even when their marker and hash match. The probe does not certify existing-data compatibility or every production setting. +A checksum receipt binds this verification to the installed JAR and native library; launchers reject missing or changed receipts. Use a fresh distribution +when upgrading from an older preparation script. Launch rejects inherited RocksDB JNI preloads in default/standard mode and competing JNI in Topling mode, while preserving unrelated preloads. +It installs `topling/rocksdbjni.jar`, its native library and optional web resources outside `lib`. Preparation refuses an existing `topling` directory; +replace a runtime by preparing another stopped, fresh distribution. Repeat preparation separately for Server, PD and Store as applicable. + +## Keep data outside the distribution + +Changing providers does not convert an existing database. Use separate, initially empty Topling directories and retain the standard-provider data. +Create persistent directories writable by the service account, outside source checkouts, build outputs and unpacked distribution directories: + +| Component | Configuration file and key | Example persistent path | +| --- | --- | --- | +| Server data | Graph properties: `rocksdb.data_path` | `/srv/hugegraph/topling/server/data` | +| Server WAL | Graph properties: `rocksdb.wal_path` | `/srv/hugegraph/topling/server/wal` | +| PD | `conf/application.yml`: `pd.data-path` | `/srv/hugegraph/topling/pd` | +| Store data | `conf/application.yml`: `app.data-path` | `/srv/hugegraph/topling/store/data` | +| Store Raft | `conf/application.yml`: `app.raft-path` | `/srv/hugegraph/topling/store/raft` | + +For a standalone Server, edit every selected graph configuration, for example `conf/graphs/hugegraph.properties`: + +```properties +backend=rocksdb +rocksdb.provider=topling +rocksdb.data_path=/srv/hugegraph/topling/server/data +rocksdb.wal_path=/srv/hugegraph/topling/server/wal +``` + +Set **both** Server data and WAL paths. Leaving WAL at its package-local default risks losing it when a build or package replacement removes that directory. +Keep the external data, WAL and Raft roots across runtime/package changes. + +For PD and Store, edit the existing `rocksdb` mapping in the provider configuration file listed above, retaining its other options: + +```yaml +rocksdb: + provider: topling +``` + +Set their persistent paths separately in `conf/application.yml`; keep the normal network addresses and cluster configuration. + +The published runtime directories use 0755 and regular files use 0644, so a service account different from the preparer can read the JNI. The service account also needs traversal permission on the component's parent directories; preparation does not change those parents. + +## Select, start and verify + +In the startup shell of each component that owns a Topling database, explicitly select the runtime: + +```bash +export TOPLINGDB_ROCKSDB_PROVIDER=topling +# Run this component's start command from the first table. +``` + +The launcher uses that component's `conf/toplingdb.yaml`. Set `TOPLINGDB_EASY_MIGRATE_CONF=/absolute/path/config.yaml` for an explicit override. +The shipped native profile disables its HTTP server and keeps `memtable_as_log_index=false`, which the Java write path requires. +Tune that profile for the host; it is separate from the business provider and data-path configuration. + +For a standalone Server, `bin/init-store.sh` and `bin/dump-store.sh` use the same explicit selection as `bin/start-hugegraph.sh`. Review Comment: Updated both English and Chinese standalone Server instructions to run `init-store.sh` once before the first `start-hugegraph.sh` for a fresh graph. Published in [5363a1](https://github.com/apache/hugegraph-doc/commit/5363a1); bilingual link validation passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
