imbajin commented on code in PR #3024:
URL: https://github.com/apache/hugegraph/pull/3024#discussion_r3328903899
##########
hugegraph-store/hg-store-rocksdb/src/main/java/org/apache/hugegraph/rocksdb/access/RocksDBSession.java:
##########
@@ -450,8 +451,11 @@ private void openRocksDB(String dbDataPath, long version) {
new
ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, cfOptions));
}
List<ColumnFamilyHandle> columnFamilyHandleList = new
ArrayList<>();
- this.rocksDB = RocksDB.open(dbOptions, dbPath,
columnFamilyDescriptorList,
- columnFamilyHandleList);
+ this.rocksDB =
+ RocksDBProviderLoader.openRocksDB(dbOptions, dbPath,
columnFamilyDescriptorList,
Review Comment:
‼️ **Store can start the Topling HTTP server once per RocksDB session**
Server already gates `open_http` to one logical DB, but Store passes the
configured flag into every `RocksDBSession`:
```text
RocksDBFactory
-> createGraphDB(db1) -> RocksDBSession -> startHttpServer()
-> createGraphDB(db2) -> RocksDBSession -> startHttpServer()
-> same YAML listening_ports
```
**Evidence**
- This call forwards `hugeConfig.get(RocksDBOptions.OPEN_HTTP)` directly.
- `RocksDBFactory.createGraphDB()` can create multiple sessions in the same
Store process.
- `ToplingRocksDBProvider.startHttpServerIfNeeded()` calls
`startHttpServer()` whenever `openHttp` is true.
- Server code has a `GRAPH_STORE` gate, but Store has no equivalent
process-level gate.
**Impact**
Multiple Store RocksDB instances can compete for the same Topling HTTP port.
Startup then depends on open order and may fail partway through DB
initialization.
**Suggested fix**
Move Topling HTTP lifecycle to a process-level singleton, or add an explicit
Store-side gating rule so only one chosen DB can start the HTTP server. Please
cover the multi-session case in a regression test.
##########
pom.xml:
##########
@@ -91,6 +91,7 @@
<hugegraph-commons.version>1.7.0</hugegraph-commons.version>
<lombok.version>1.18.30</lombok.version>
<release.name>hugegraph</release.name>
+ <rocksdbjni.version>8.10.2-SNAPSHOT</rocksdbjni.version>
Review Comment:
‼️ **Default RocksDB now depends on a Topling snapshot**
The current dependency flow makes ToplingDB part of the default RocksDB
path, not an opt-in provider:
```text
standard RocksDB user
-> root pom selects rocksdbjni:8.10.2-SNAPSHOT
-> build needs GitHub Packages
-> Linux startup scripts source Topling preload logic
-> native preload/runtime assumptions affect default startup
```
**Evidence**
- This line changes the global `rocksdbjni.version` to `8.10.2-SNAPSHOT`.
- `.github/configs/settings.xml` adds GitHub Packages as the source for that
snapshot.
- `start-hugegraph.sh` / `init-store.sh` source `preload-topling.sh`
unconditionally on the server startup path.
**Impact**
Users who did not configure `rocksdb.option_path` can still depend on a
mutable snapshot artifact and Topling native preload behavior. That is risky
for normal contributor builds, offline deployments, and release reproducibility.
**Suggested fix**
Keep the default dependency on a released standard `rocksdbjni` artifact.
Select the Topling snapshot only through an explicit profile/configuration
path, and only enter native preload when Topling is explicitly enabled.
##########
hugegraph-store/hg-store-rocksdb/src/main/java/org/apache/hugegraph/rocksdb/access/RocksDBSession.java:
##########
@@ -450,8 +451,11 @@ private void openRocksDB(String dbDataPath, long version) {
new
ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, cfOptions));
}
List<ColumnFamilyHandle> columnFamilyHandleList = new
ArrayList<>();
- this.rocksDB = RocksDB.open(dbOptions, dbPath,
columnFamilyDescriptorList,
- columnFamilyHandleList);
+ this.rocksDB =
+ RocksDBProviderLoader.openRocksDB(dbOptions, dbPath,
columnFamilyDescriptorList,
Review Comment:
‼️ **Snapshot verification still bypasses the provider path**
Normal open now goes through `RocksDBProviderLoader`, but this recovery path
still uses raw RocksDB APIs:
```text
Topling DB / snapshot
-> verifySnapshot()
-> RocksDB.listColumnFamilies()
-> RocksDB.openReadOnly()
-> no option_path / SidePluginRepo initialization
```
**Evidence**
- Snapshot verification calls `RocksDB.listColumnFamilies()` directly here.
- It then calls `RocksDB.openReadOnly()` directly below.
- Store startup has a similar pre-open direct `listColumnFamilies()` path,
and Server `listCFs()` also bypasses the provider.
**Impact**
If a Topling database or snapshot relies on YAML-defined factories/options,
restart or snapshot restore can fail before the provider gets a chance to
import the Topling configuration. This leaves lifecycle paths inconsistent with
the new normal open path.
**Suggested fix**
Add provider-aware helpers for CF listing and read-only snapshot
verification. At minimum, when `rocksdb.option_path` is set, initialize/import
the Topling config before CF discovery and read-only verification. Please add
regression coverage for Topling DB restart and snapshot verify/load.
##########
hugegraph-rocksdb-provider/src/main/java/org/apache/hugegraph/rocksdb/provider/ToplingRocksDBProvider.java:
##########
@@ -0,0 +1,614 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hugegraph.rocksdb.provider;
+
+import org.rocksdb.*;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Locale;
+import java.util.List;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+import net.minidev.json.JSONObject;
+
+import org.yaml.snakeyaml.Yaml;
+import org.yaml.snakeyaml.constructor.BaseConstructor;
+import org.yaml.snakeyaml.constructor.SafeConstructor;
+import org.yaml.snakeyaml.LoaderOptions;
+
+/**
+ * ToplingRocksDBProvider provides ToplingDB-specific RocksDB functionality.
+ * This provider supports advanced ToplingDB features including:
+ * - YAML-based configuration via optionPath
+ * - HTTP server for monitoring and management
+ * - SidePluginRepo integration for enhanced performance
+ */
+public class ToplingRocksDBProvider extends AbstractRocksDBProvider {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(ToplingRocksDBProvider.class);
+
+ private static final String PROVIDER_NAME = "topling";
+ private static final int PROVIDER_PRIORITY = 200; // Higher priority than
standard
+ private static final String SIDE_PLUGIN_REPO_CLASS =
"org.rocksdb.SidePluginRepo";
+
+ // Validation constants migrated from RocksDBOptions
+ private static final Pattern SAFE_PATH_PATTERN =
+ Pattern.compile("^[a-zA-Z0-9/_.-]+\\.ya?ml$");
+ private static final String ALLOWED_CONFIG_DIR = "./conf/";
+ private static final long MAX_CONFIG_FILE_SIZE = 1024 * 1024 * 10; // 10 MB
+
+ // Store repo objects for proper cleanup
+ private final Map<RocksDB, Object> rocksDBToRepoMap = new
ConcurrentHashMap<>();
+
+ @Override
+ public String getProviderName() {
+ return PROVIDER_NAME;
+ }
+
+ @Override
+ public int getPriority() {
+ return PROVIDER_PRIORITY;
+ }
+
+ @Override
+ public boolean isAvailable() {
+ try {
+ // Check if SidePluginRepo class is available
+ Class.forName(SIDE_PLUGIN_REPO_CLASS);
+ LOG.info("ToplingDB SidePluginRepo found, ToplingRocksDBProvider
is available");
+ return true;
+ } catch (ClassNotFoundException e) {
+ LOG.debug(
+ "ToplingDB SidePluginRepo not found,
ToplingRocksDBProvider is not available:" +
+ " {}",
+ e.getMessage());
+ return false;
+ }
+ }
+
+ @Override
+ protected RocksDB doOpenRocksDB(Options options, String dataPath) throws
RocksDBException {
+ LOG.info("Opening RocksDB with Options at path: {}", dataPath);
+
+ // For simple Options-based opening without optionPath, use standard
RocksDB.open
+ return RocksDB.open(options, dataPath);
+ }
+
+ @Override
+ public RocksDB openRocksDB(Options options, String dataPath, String
optionPath,
+ Boolean openHttp) throws RocksDBException {
+ initialize();
+ return doOpenRocksDB(options, dataPath, optionPath, openHttp);
+ }
+
+ @Override
+ protected RocksDB doOpenRocksDB(Options options, String dataPath, String
optionPath,
+ Boolean openHttp) throws RocksDBException {
+ // Check if we should use ToplingDB features
+ boolean useTopling = validateConfiguration(optionPath);
+
+ if (useTopling) {
+ return openWithToplingFeatures(options, dataPath, optionPath,
openHttp);
+ } else {
+ logStandardFallback(optionPath);
+ return RocksDB.open(options, dataPath);
+ }
+ }
+
+ @Override
+ protected RocksDB doOpenRocksDB(DBOptions dbOptions, String dataPath,
+ List<ColumnFamilyDescriptor> cfDescriptors,
+ List<ColumnFamilyHandle> cfHandles) throws
RocksDBException {
+ LOG.info("Opening RocksDB with DBOptions and column families at path:
{}", dataPath);
+
+ // For column family opening without ToplingDB features, use standard
RocksDB.open
+ return RocksDB.open(dbOptions, dataPath, cfDescriptors, cfHandles);
+ }
+
+ @Override
+ protected RocksDB doOpenRocksDB(DBOptions dbOptions, String dataPath,
+ List<ColumnFamilyDescriptor> cfDescriptors,
+ List<ColumnFamilyHandle> cfHandles,
+ String optionPath, Boolean openHttp)
throws RocksDBException {
+ // Check if we should use ToplingDB features
+ boolean useTopling = validateConfiguration(optionPath);
+
+ if (useTopling) {
+ // For ToplingDB with column families, we need to use the standard
RocksDB.open method
+ // but with ToplingDB-specific initialization through
SidePluginRepo
+ return openWithToplingFeaturesAndCF(dbOptions, dataPath,
cfDescriptors, cfHandles,
+ optionPath, openHttp);
+ } else {
+ logStandardFallback(optionPath);
+ return RocksDB.open(dbOptions, dataPath, cfDescriptors, cfHandles);
+ }
+ }
+
+ /**
+ * Opens RocksDB using ToplingDB features with SidePluginRepo
+ */
+ private RocksDB openWithToplingFeatures(Options options, String dataPath,
+ String optionPath, Boolean
openHttp)
+ throws RocksDBException {
+ Object repo = null;
+ RocksDB opened = null;
+ boolean registered = false;
+ try {
+ // Initialize ToplingDB repo with common operations
+ repo = initializeToplingRepo(options, dataPath, optionPath);
+
+ // Open database with default column families
+ Class<?> sidePluginRepoClass = repo.getClass();
+ Method openDBMethod = sidePluginRepoClass.getMethod("openDB",
String.class);
+ Object result = openDBMethod.invoke(repo,
converseOptionsToJsonString(dataPath, null));
+
+ // Validate and store result before starting HTTP server
+ opened = validateAndStoreResult(result, repo, dataPath, 0);
+ registered = true;
+
+ // Start HTTP server if needed
+ startHttpServerIfNeeded(repo, dataPath, openHttp, optionPath);
+
+ return opened;
+
+ } catch (InvocationTargetException e) {
+ cleanupFailedOpen(opened, repo, registered, null);
+ Throwable cause = e.getCause();
+ if (cause instanceof RocksDBException) {
+ throw (RocksDBException) cause;
+ }
+ throw new RocksDBException(
+ "Failed to open DB with SidePluginRepo: " +
cause.getMessage());
+ } catch (Exception e) {
+ cleanupFailedOpen(opened, repo, registered, null);
+ throw new RocksDBException("Failed to open ToplingDB: " +
e.getMessage());
+ }
+ }
+
+ /**
+ * Open RocksDB with ToplingDB features and column families support
+ */
+ private RocksDB openWithToplingFeaturesAndCF(DBOptions dbOptions, String
dataPath,
+ List<ColumnFamilyDescriptor>
cfDescriptors,
+ List<ColumnFamilyHandle>
cfHandles,
+ String optionPath, Boolean
openHttp)
+ throws RocksDBException {
+ Object repo = null;
+ RocksDB opened = null;
+ boolean registered = false;
+ try {
+ // Initialize ToplingDB repo with common operations
+ repo = initializeToplingRepo(dbOptions, dataPath, optionPath);
+
+ // Prepare column family names for JSON
+ List<String> cfNames = new java.util.ArrayList<>();
+ for (ColumnFamilyDescriptor cfDescriptor : cfDescriptors) {
+ cfNames.add(new String(cfDescriptor.getName(),
StandardCharsets.UTF_8));
+ }
+
+ // Open database with column families
+ Class<?> sidePluginRepoClass = repo.getClass();
+ Method openDBMethod = sidePluginRepoClass.getMethod("openDB",
String.class, List.class);
+ Object result = openDBMethod.invoke(repo,
+
converseOptionsToJsonString(dataPath, cfNames),
+ cfHandles);
+
+ // Validate and store result before starting HTTP server
+ opened = validateAndStoreResult(result, repo, dataPath,
cfDescriptors.size());
+ registered = true;
+
+ // Start HTTP server if needed
+ startHttpServerIfNeeded(repo, dataPath, openHttp, optionPath);
+
+ return opened;
+
+ } catch (InvocationTargetException e) {
+ cleanupFailedOpen(opened, repo, registered, cfHandles);
+ Throwable cause = e.getCause();
+ if (cause instanceof RocksDBException) {
+ throw (RocksDBException) cause;
+ }
+ throw new RocksDBException(
+ "Failed to open DB with SidePluginRepo: " +
cause.getMessage());
+ } catch (Exception e) {
+ cleanupFailedOpen(opened, repo, registered, cfHandles);
+ LOG.error("Failed to open ToplingDB with column families", e);
+ throw new RocksDBException("Failed to open ToplingDB: " +
e.getMessage());
+ }
+ }
+
+ /**
+ * Common operations for ToplingDB SidePluginRepo initialization and setup
+ */
+ private Object initializeToplingRepo(Object options, String dataPath,
String optionPath)
+ throws RocksDBException {
+ try {
+ // Dynamically load the SidePluginRepo class by its name at
runtime.
+ Class<?> sidePluginRepoClass =
Class.forName(SIDE_PLUGIN_REPO_CLASS);
+ Object repo = sidePluginRepoClass.getConstructor().newInstance();
+
+ String dbName = getDbName(dataPath);
+
+ // Put options into repo - handle both Options and DBOptions
+ if (options instanceof Options) {
+ Method putMethod =
+ sidePluginRepoClass.getMethod("put", String.class,
Options.class);
+ putMethod.invoke(repo, dbName, options);
+ } else if (options instanceof DBOptions) {
+ Method putMethod =
+ sidePluginRepoClass.getMethod("put", String.class,
DBOptions.class);
+ putMethod.invoke(repo, dbName, options);
+ } else {
+ throw new RocksDBException(
+ "Unsupported options type: " +
options.getClass().getName());
+ }
+
+ // Import auto file
+ Method importAutoFileMethod =
+ sidePluginRepoClass.getMethod("importAutoFile",
String.class);
+ importAutoFileMethod.invoke(repo, optionPath);
+
+ return repo;
+
+ } catch (ClassNotFoundException e) {
+ LOG.error(
+ "SidePluginRepo not found. This version of rocksdbjni does
not support " +
+ "topling.",
+ e);
+ throw new IllegalStateException(
+ "Topling features (SidePluginRepo) are required but not
found in the " +
+ "rocksdbjni library.",
+ e);
+ } catch (Exception e) {
+ LOG.error("Failed to initialize ToplingDB SidePluginRepo", e);
+ throw new RocksDBException("SidePluginRepo reflection error: " +
e.getMessage());
+ }
+ }
+
+ /**
+ * Start HTTP server if conditions are met
+ */
+ private void startHttpServerIfNeeded(Object repo, String dataPath, Boolean
openHttp,
+ String optionPath)
+ throws RocksDBException {
+ try {
+ if (Boolean.TRUE.equals(openHttp)) {
+ Class<?> sidePluginRepoClass = repo.getClass();
+ Method openHttpMethod =
sidePluginRepoClass.getMethod("startHttpServer");
+ openHttpMethod.invoke(repo);
+ LOG.info("Topling HTTP Server has been started according to
the " +
+ "listening_ports specified in " + optionPath);
+ }
+ } catch (Exception e) {
+ LOG.error("Failed to start HTTP server", e);
+ throw new RocksDBException("Failed to start HTTP server: " +
e.getMessage());
+ }
+ }
+
+ /**
+ * Validate and store RocksDB result with repo mapping
+ */
+ private RocksDB validateAndStoreResult(Object result, Object repo, String
dataPath, int cfCount)
+ throws RocksDBException {
+ if (result instanceof RocksDB) {
+ RocksDB rocksDB = (RocksDB) result;
+ // Store the repo reference for later cleanup
+ rocksDBToRepoMap.put(rocksDB, repo);
+ if (cfCount > 0) {
+ LOG.info("Successfully opened ToplingDB with {} column
families at path: {}",
+ cfCount, dataPath);
+ } else {
+ LOG.info("Successfully opened ToplingDB with default column
families at path: {}",
+ dataPath);
+ }
+ return rocksDB;
+ } else {
+ throw new RocksDBException("ToplingDB openDB returned unexpected
result type: " +
+ (result != null ?
result.getClass().getName() : "null"));
+ }
+ }
+
+ /**
+ * Cleanup resources when opening RocksDB fails after successful openDB
call.
+ * This prevents resource leaks when HTTP server startup or other
post-open operations fail.
+ *
+ * @param opened The RocksDB instance that was opened (may be null)
+ * @param repo The SidePluginRepo instance (may be null)
+ * @param registered Whether the RocksDB was registered in rocksDBToRepoMap
+ * @param cfHandles List of column family handles to close (may be null)
+ */
+ private void cleanupFailedOpen(RocksDB opened, Object repo, boolean
registered,
+ List<ColumnFamilyHandle> cfHandles) {
+ if (opened == null && repo == null) {
+ // Nothing to clean up
+ return;
+ }
+
+ LOG.warn("Cleaning up resources after failed RocksDB open operation");
+
+ // Remove from map if registered
+ if (registered && opened != null) {
+ rocksDBToRepoMap.remove(opened);
+ }
+
+ // Close column family handles if provided
+ if (cfHandles != null && !cfHandles.isEmpty()) {
+ for (ColumnFamilyHandle cfHandle : cfHandles) {
+ if (cfHandle != null) {
+ try {
+ cfHandle.close();
+ } catch (Exception e) {
+ LOG.warn("Failed to close column family handle during
cleanup", e);
+ }
+ }
+ }
+ }
+
+ // Close RocksDB instance
+ if (opened != null) {
+ try {
+ opened.close();
+ } catch (Exception e) {
+ LOG.warn("Failed to close RocksDB instance during cleanup", e);
+ }
+ }
+
+ // Close SidePluginRepo
+ if (repo != null) {
+ try {
+ Class<?> sidePluginRepoClass = repo.getClass();
+ Method closeAllDBMethod =
sidePluginRepoClass.getMethod("closeAllDB");
+ closeAllDBMethod.invoke(repo);
+ LOG.debug("Successfully called closeAllDB() on SidePluginRepo
during cleanup");
+ } catch (Exception e) {
+ LOG.warn("Failed to call closeAllDB() on SidePluginRepo during
cleanup", e);
+ }
+ }
+ }
+
+ /**
+ * Utility function to convert options to JSON string for ToplingDB
+ * Moved from RocksDBStdSessions
+ */
+ private static String converseOptionsToJsonString(String dataPath,
List<String> cfs)
+ throws RocksDBException {
+ if (dataPath == null || dataPath.trim().isEmpty()) {
+ throw new RocksDBException("RocksDB dataPath cannot be null or
empty");
+ }
+ // sanitize path to avoid trailing slash causing empty namepart in
native side
+ String sanitizedPath = sanitizePath(dataPath);
+ // construct CFOptions
+ JSONObject columnFamilies = new JSONObject();
+ // multi CFs
+ if (cfs != null) {
+ for (String cf : cfs) {
+ columnFamilies.put(cf, "$default");
+ }
+ } else { // single default CF
+ columnFamilies.put("default", "$default");
+ }
+
+ // construct params
+ JSONObject params = new JSONObject();
+ params.put("db_options", "$dbo");
+ params.put("cf_options", "$default");
+ params.put("column_families", columnFamilies);
+ params.put("path", sanitizedPath);
+
+ // construct wrapper
+ JSONObject wrapper = new JSONObject();
+ wrapper.put("method", "DB::Open");
+ wrapper.put("params", params);
+
+ return wrapper.toString();
+ }
+
+ /**
+ * Utility function to get database name from path
+ * Moved from RocksDBStdSessions
+ */
+ private static String getDbName(String dataPath) {
+ String sanitizedPath = sanitizePath(dataPath);
+ return Paths.get(sanitizedPath).getFileName().toString();
+ }
+
+ /**
+ * Ensure path has no trailing separators and is normalized
+ */
+ private static String sanitizePath(String dataPath) {
+ String p = dataPath.trim();
+ // remove trailing separators
+ while (p.endsWith("/") || p.endsWith(java.io.File.separator)) {
+ p = p.substring(0, p.length() - 1);
+ }
+ // normalize using Paths to collapse redundant parts
+ try {
+ return Paths.get(p).normalize().toString();
+ } catch (Exception e) {
+ // fallback to original trimmed path
+ return p;
+ }
+ }
+
+ private static boolean validateConfiguration(String optionPath) {
+ boolean result = false;
+ if (optionPath != null && !optionPath.isEmpty()) {
+ try {
+ // Validate option path first
+ validateOptionPath(optionPath);
+
+ // Read and validate YAML content
+ String yamlContent = Files.readString(Paths.get(optionPath),
+ StandardCharsets.UTF_8);
+ validateYamlContent(yamlContent);
+
+ Class.forName(SIDE_PLUGIN_REPO_CLASS);
+ result = true;
+ LOG.info(
+ "SidePluginRepo found. Will attempt to open default CF
RocksDB using " +
+ "Topling.");
+ } catch (ClassNotFoundException e) {
+ LOG.warn("SidePluginRepo not found, even though 'optionPath'
was provided. " +
+ "Falling back to the standard RocksDB default CF
opening method. " +
+ "The configuration in '{}' will be ignored.",
optionPath);
+ } catch (Exception e) {
Review Comment:
‼️ **Explicit Topling config errors should fail fast, not silently fall
back**
When users configure `rocksdb.option_path`, the current behavior treats
invalid Topling config as a warning and then opens standard RocksDB:
```text
rocksdb.option_path is set
-> path/YAML validation fails
-> warn only
-> validateConfiguration() returns false
-> standard RocksDB opens successfully
```
**Evidence**
- Path traversal, missing file, unreadable file, and invalid YAML are all
caught here as generic exceptions.
- The method then returns `false`, which makes the caller use the standard
RocksDB path.
- The provider module also hard-codes `./conf/` as the allowed deployment
directory.
**Impact**
A typo or malformed YAML can make the service start successfully while not
using ToplingDB at all. That is dangerous operationally because the user
explicitly requested Topling behavior but gets a silent downgrade.
**Suggested fix**
Only allow fallback when Topling was not requested, or when the platform is
explicitly unsupported by design. If `option_path` is explicitly configured and
validation fails, startup should fail fast with a clear error. Path policy
should be resolved in Server/PD/Store config layers and passed to the provider
as a normalized path, rather than hard-coded in this shared provider.
##########
hugegraph-server/hugegraph-dist/src/assembly/static/bin/common-topling.sh:
##########
@@ -0,0 +1,299 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+set -Eeuo pipefail
+IFS=$'\n\t'
+trap 'echo "[common-topling] error at line ${LINENO}: ${BASH_COMMAND}" >&2' ERR
+
+GITHUB="https://github.com";
+
+function abs_path() {
+ local SOURCE
+ SOURCE="${BASH_SOURCE[0]}"
+ while [[ -h "$SOURCE" ]]; do
+ local DIR
+ DIR="$(cd -P "$(dirname "$SOURCE")" && pwd)"
+ SOURCE="$(readlink "$SOURCE")"
+ [[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE"
+ done
+ cd -P "$(dirname "$SOURCE")" && pwd
+}
+
+function extract_so_with_jar() {
+ local jar_file="$1"
+ local dest_dir="$2"
+ local abs_jar_path
+
+ if [ ! -f "$jar_file" ]; then
+ echo "'$jar_file' Not Exist" >&2
+ return 1
+ fi
+
+ mkdir -p "$dest_dir" || {
+ echo "Cannot mkdir '$dest_dir'" >&2
+ return 1
+ }
+
+ if command -v realpath >/dev/null 2>&1; then
+ abs_jar_path="$(realpath "$jar_file")"
+ else
+ abs_jar_path="$(readlink -f "$jar_file")"
+ fi
+ if ! command -v unzip >/dev/null 2>&1; then
+ echo "Error: 'unzip' command not found. Please install unzip." >&2
+ return 1
+ fi
+ unzip -j -o "$abs_jar_path" "*.so" -d "$dest_dir" > /dev/null 2>&1 || {
+ local code=$?
+ if [ $code -eq 11 ]; then
+ echo "Error: No .so files found in '$abs_jar_path' (unzip exit
11)" >&2
+ else
+ echo "Error: unzip failed (exit $code) for '$abs_jar_path'" >&2
+ fi
+ return $code
+ }
+}
+
+function extract_html_css_from_jar() {
+ local jar_file="$1"
+ local dest_dir="$2"
+ local abs_jar_path
+ # Prefer /dev/shm on Linux for speed; fallback to TMPDIR or /tmp
+ local resource_target
+ if [ "$(uname -s)" = "Linux" ] && [ -d /dev/shm ]; then
+ resource_target="/dev/shm/rocksdb_resource"
+ else
+ resource_target="${TMPDIR:-/tmp}/rocksdb_resource"
+ fi
+
+ if [ ! -f "$jar_file" ]; then
+ echo "Error: JAR file '$jar_file' does not exist." >&2
+ return 1
+ fi
+
+ mkdir -p "$dest_dir" || {
+ echo "Error: Cannot create destination directory '$dest_dir'." >&2
+ return 1
+ }
+
+ if command -v realpath >/dev/null 2>&1; then
+ abs_jar_path="$(realpath "$jar_file")"
+ else
+ abs_jar_path="$(readlink -f "$jar_file")"
+ fi
+ if ! command -v unzip >/dev/null 2>&1; then
+ echo "Error: 'unzip' command not found. Please install unzip." >&2
+ return 1
+ fi
+ unzip -j -o "$abs_jar_path" "*.html" "*.css" -d "$dest_dir" > /dev/null ||
{
+ local code=$?
+ if [ $code -eq 11 ]; then
+ echo "Notice: No .html or .css files found in '$jar_file'." >&2
+ return 0
+ else
+ echo "Error: unzip failed with exit code $code" >&2
+ return $code
+ fi
+ }
+
+ mkdir -p "$resource_target" || {
+ echo "Error: Cannot create target directory '$resource_target'." >&2
+ return 1
+ }
+
+ if compgen -G "$dest_dir"/*.html >/dev/null 2>&1; then
+ cp -f "$dest_dir"/*.html "$resource_target"/
+ fi
+ if compgen -G "$dest_dir"/*.css >/dev/null 2>&1; then
+ cp -f "$dest_dir"/*.css "$resource_target"/
+ fi
+}
+
+function ensure_libaio_symlink() {
+ # Check for Ubuntu 24.04+ and create a symlink for libaio if needed.
+ # This is a workaround for software expecting the old libaio.so.1 name,
+ # as it was renamed to libaio.so.1t64 in the new release.
+ # https://askubuntu.com/questions/1512196/libaio1-on-noble/1516639#1516639
+ if [ -f /etc/os-release ]; then
+ . /etc/os-release
+ if [ "${ID:-}" = "ubuntu" ] && command -v dpkg >/dev/null 2>&1 && dpkg
--compare-versions "${VERSION_ID:-0}" "ge" "24.04"; then
+ local libaio_link_target="/usr/lib/x86_64-linux-gnu/libaio.so.1"
+ if [ ! -e "$libaio_link_target" ]; then
+ echo "Ubuntu ${VERSION_ID:-?} detected. Creating compatibility
symlink for libaio."
+ if [ -e /usr/lib/x86_64-linux-gnu/libaio.so.1t64 ]; then
+ if [ "$EUID" -eq 0 ]; then
+ ln -sf /usr/lib/x86_64-linux-gnu/libaio.so.1t64
"$libaio_link_target" || true
+ elif command -v sudo >/dev/null 2>&1; then
+ sudo ln -sf /usr/lib/x86_64-linux-gnu/libaio.so.1t64
"$libaio_link_target" || true
+ else
+ echo "Warn: sudo not available, skip creating
$libaio_link_target" >&2
+ fi
+ else
+ echo "Warn: libaio.so.1t64 not found, skip creating compat
symlink" >&2
+ fi
+ fi
+ fi
+ fi
+}
+
+function download_and_verify() {
+ local url=$1
+ local filepath=$2
+ local expected_sha256=$3
+ local actual_sha256
+
+ if [[ -f "$filepath" ]]; then
+ echo "File $filepath exists. Verifying SHA-256 checksum..."
+ actual_sha256=$(sha256sum "$filepath" | awk '{ print $1 }')
+ if [[ "$actual_sha256" != "$expected_sha256" ]]; then
+ echo "SHA-256 checksum verification failed for $filepath.
Expected: $expected_sha256, but got: $actual_sha256"
+ echo "Deleting $filepath..."
+ rm -f "$filepath"
+ else
+ echo "SHA-256 checksum verification succeeded for $filepath."
+ return 0
+ fi
+ fi
+
+ echo "Downloading $filepath..."
+ curl -fL -o "$filepath" "$url"
+
+ actual_sha256=$(sha256sum "$filepath" | awk '{ print $1 }')
+ if [[ "$actual_sha256" != "$expected_sha256" ]]; then
+ echo "SHA-256 checksum verification failed for $filepath after
download. Expected: $expected_sha256, but got: $actual_sha256"
+ return 1
+ fi
+
+ return 0
+}
+
+function download_and_setup_jemalloc() {
+ local arch lib_file download_url expected_sha256 system_lib top
+ top=$1
+
+ # Prefer system-installed jemalloc if available
+ # Try ldconfig first to locate the shared object
+ if command -v ldconfig >/dev/null 2>&1; then
+ system_lib=$(ldconfig -p 2>/dev/null | awk '/jemalloc/{print $4}' |
head -n1)
+ fi
+ # Fallback to common library paths if ldconfig is not available or found
nothing
+ if [[ -z "$system_lib" ]]; then
+ for p in \
+ /usr/lib/libjemalloc.so \
+ /usr/lib/libjemalloc.so.2 \
+ /usr/lib64/libjemalloc.so \
+ /usr/lib64/libjemalloc.so.2 \
+ /usr/local/lib/libjemalloc.so \
+ /usr/local/lib/libjemalloc.so.2 \
+ /usr/lib/x86_64-linux-gnu/libjemalloc.so \
+ /usr/lib/x86_64-linux-gnu/libjemalloc.so.2 \
+ /usr/lib/aarch64-linux-gnu/libjemalloc.so \
+ /usr/lib/aarch64-linux-gnu/libjemalloc.so.2; do
+ if [[ -f "$p" ]]; then
+ system_lib="$p"
+ break
+ fi
+ done
+ fi
+
+ # If found, set LD_PRELOAD and return immediately
+ if [[ -n "$system_lib" ]]; then
+ if [[ ":${LD_PRELOAD:-}:" != *"libjemalloc"* ]]; then
+ export LD_PRELOAD="${system_lib}${LD_PRELOAD:+:$LD_PRELOAD}"
+ fi
+ return 0
+ fi
+
+ # Detect system architecture
+ arch=$(uname -m)
+
+ # System jemalloc not found, try to download the correct library for the
architecture
+ if [[ $arch == "aarch64" || $arch == "arm64" ]]; then
+ lib_file="$top/bin/libjemalloc_aarch64.so"
+
download_url="${GITHUB}/apache/hugegraph-doc/raw/binary-1.5/dist/server/libjemalloc_aarch64.so"
+
expected_sha256="6b7e6099b6da798829c6ce6fcb55a787508841edd52446332a73300889dcd1dc"
+ elif [[ $arch == "x86_64" ]]; then
+ lib_file="$top/bin/libjemalloc.so"
+
download_url="${GITHUB}/apache/hugegraph-doc/raw/binary-1.5/dist/server/libjemalloc.so"
+
expected_sha256="53b25e8626e1605cbd8b60befb3431cabc1b8851a54285e0dda412796feab67d"
+ else
+ echo "Unsupported architecture: $arch"
+ return 1
+ fi
+
+ # Download and verify jemalloc library (fallback when system lib not found)
+ if download_and_verify "$download_url" "$lib_file" "$expected_sha256"; then
+ if [[ ":${LD_PRELOAD:-}:" != *":${lib_file}:"* ]]; then
+ export LD_PRELOAD="${lib_file}${LD_PRELOAD:+:$LD_PRELOAD}"
+ fi
+ else
+ echo "Failed to verify or download jemalloc for $arch, skipping"
+ return 1
+ fi
+}
+
+function preload_toplingdb() {
+ local lib_dir="$1"
+ local dest_dir="$2"
+ local os_name
+
+ # NOTE: The current ToplingDB rocksdbjni snapshot bundles Linux x86_64
native libraries.
Review Comment:
‼️ **Linux arm64 does not actually fall back to standard RocksDB**
The comment says the current ToplingDB snapshot bundles Linux x86_64 native
libraries, but the implementation only checks the OS:
```text
Linux arm64/aarch64
-> os_name == Linux
-> extract .so from rocksdbjni snapshot
-> preload librocksdbjni-linux64.so
-> Java provider may still select Topling by class presence
```
**Evidence**
- This script skips only non-Linux platforms.
- It later preloads `librocksdbjni-linux64.so` without checking `uname -m`.
- `ToplingRocksDBProvider.isAvailable()` only checks for
`org.rocksdb.SidePluginRepo`, not CPU architecture.
- The root POM only has a macOS profile that falls back to standard
`rocksdbjni`; Linux arm64 still gets the Topling snapshot.
**Impact**
Linux arm64/aarch64 will not follow the documented fallback behavior. It can
try to load x86_64 native libraries and fail during startup instead of using
standard RocksDB.
**Suggested fix**
Add the same platform boundary in all three places:
```text
Maven dependency selection: Linux x86_64 -> Topling snapshot, otherwise
standard rocksdbjni
Shell preload: return unless uname -s == Linux and uname -m is x86_64/amd64
Java provider availability: return false on unsupported OS/arch
```
Please also add at least one fallback check for Linux arm64 so this does not
regress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
