dpol1 opened a new issue, #3238:
URL: https://github.com/apache/hugegraph/issues/3238

   Status: draft for review. Parent: apache/hugegraph#3110. Groundwork: #3145, 
#3146. Snapshot: apache/hugegraph `83ef9f3fa`. Evidence repository: 
[dpol1/hugegraph-rust-rfc](https://github.com/dpol1/hugegraph-rust-rfc/tree/5b05babdc89c2f0803cbd50db18e48a08cf61dc8)
 (commit `5b05bab`); paths below in its `rfc`, `spike` and `results` 
directories link to it, and its README explains how to reproduce the runs. 
Benchmarks and harness: 
[`benchmark/rfc`](https://github.com/dpol1/hugegraph/tree/1c62d1fe7bbed08c1e0d45b81a289df20ceb8ae9/hugegraph-server/hugegraph-test/src/test/java/org/apache/hugegraph/benchmark/rfc)
 on the branch `prototype/rfc-spike` of dpol1/hugegraph (commit `1c62d1fe7`), 
not proposed for merge. Paths that start with a HugeGraph module name are in 
apache/hugegraph at `83ef9f3fa`.
   
   ## 1. Objectives, scope, non-goals
   
   Question: which Server-core candidate offers a verifiable benefit large 
enough to justify the interoperability, compatibility, and maintenance cost of 
a Rust implementation? The latency criterion, committed before any measurement 
([`rfc/thresholds.md`](https://github.com/dpol1/hugegraph-rust-rfc/blob/5b05babdc89c2f0803cbd50db18e48a08cf61dc8/rfc/thresholds.md)),
 is T_native / T_Java <= 0.5, where T_native covers the crossing, conversions, 
copies, the operation, and building the result.
   
   Scope: the candidates of the #3145 inventory (`LongEncoding`, `NumericUtil`, 
both `BytesBuffer` copies, `IntSet`/`IntMap`, `IntMapByDynamicHash`) and one 
non-codec candidate, the Jieba analyzer. Server core is the perimeter proposed 
here, not a choice the maintainers have made.
   
   Non-goals: no Rust code in `apache/hugegraph`; no storage baselines (#3223 
covers them); no toolchain client work; no proof of concept on `BytesBuffer` 
before #3090 is decided.
   
   ## 2. Candidates and decision
   
   Outcomes per candidate. Numbers are JMH average time per call on one Linux 
x86_64 laptop (section 6); "stop" is the exploratory criterion of 
[`rfc/thresholds.md`](https://github.com/dpol1/hugegraph-rust-rfc/blob/5b05babdc89c2f0803cbd50db18e48a08cf61dc8/rfc/thresholds.md)
 for the declared path and load, not a statement about every possible native 
implementation.
   
   | Candidate | Outcome | Evidence |
   |---|---|---|
   | `LongEncoding` encoders (`encodeNumber`, `encodeSortable`, 
`encodeSignedB64`) | Evidence contrary for the declared path and loads | Java 
38 to 182 ns; creating the result string across the boundary alone costs 166 to 
171 ns 
([`rfc/stage-c-gate.md`](https://github.com/dpol1/hugegraph-rust-rfc/blob/5b05babdc89c2f0803cbd50db18e48a08cf61dc8/rfc/stage-c-gate.md))
 |
   | `LongEncoding.decodeSignedB64` | Against an improved Java decoder (same 
behaviour on the verified cases): evidence contrary for the declared path on 
all loads. Against the current Java: stop on small and large; on mixed, 
evidence insufficient (the gate says explore; stage C was not run) | Current 
Java 25 / 280 / 263 ns; improved Java 9.5 / 21.7 / 23.2 ns (small / mixed / 
large, same campaign); on the verified cases the improved decoder gives the 
same values and exception classes as the corpus, and the same values, exception 
classes, and messages as the original, except the NullPointerException message; 
native read calibration 128 to 143 ns 
([`rfc/stage-c-gate-d1.md`](https://github.com/dpol1/hugegraph-rust-rfc/blob/5b05babdc89c2f0803cbd50db18e48a08cf61dc8/rfc/stage-c-gate-d1.md))
 |
   | `LongEncoding.validB64Char` | Evidence contrary for the declared path | 
Java 5.7 to 25.4 ns; calibration 61 ns 
([`rfc/stage-c-gate.md`](https://github.com/dpol1/hugegraph-rust-rfc/blob/5b05babdc89c2f0803cbd50db18e48a08cf61dc8/rfc/stage-c-gate.md))
 |
   | `NumericUtil` (`numberToSortableLong`, `numberToSortableBytes`, 
`sortableBytesToNumber`) | Evidence contrary for the declared path; one case 
not distinguishable from the harness | Java 3.8 to 11.5 ns; calibrations 59 to 
158 ns 
([`rfc/stage-c-gate.md`](https://github.com/dpol1/hugegraph-rust-rfc/blob/5b05babdc89c2f0803cbd50db18e48a08cf61dc8/rfc/stage-c-gate.md))
 |
   | `BytesBuffer` (core and struct) | Not evaluated | Blocked by #3090. The 
two copies already meet on one live path: the store `ScanPartition` service 
decodes server-written rows with the struct copy 
(`GraphStoreIterator.getElement` → `AbstractSelectIterator.parseEntry` → 
`BinaryElementSerializer.parseVertex`/`parseEdge`, created at 
`hugegraph-store/hg-store-core/.../business/BusinessHandlerImpl.java:549`) and 
passes no schema (`AbstractSelectIterator.java:42`), while the core property 
format relies on schema metadata. Vermeer loads graphs through it; 
apache/hugegraph-computer#361 reports `Can't construct Cardinality from code 0` 
in the struct `readProperty` there, an error consistent with the mismatch that 
was not reproduced here (the report lists Store and PD images tagged `latest`) |
   | `IntSet` / `IntMap` | Evidence insufficient for Rust | Motivation is 
maintenance (`sun.misc.Unsafe`) and memory, outside the latency criterion; JEP 
471 maps every member they use to `VarHandle` (https://openjdk.org/jeps/471), a 
Java path that was not measured here |
   | `IntMapByDynamicHash` | Not a candidate today | No production 
instantiation at `83ef9f3fa` 
([`rfc/matrix.md`](https://github.com/dpol1/hugegraph-rust-rfc/blob/5b05babdc89c2f0803cbd50db18e48a08cf61dc8/rfc/matrix.md))
 |
   | Jieba analyzer | Evidence insufficient | Tokens are persisted as 
search-index row keys; the jieba-rs README says nothing about compatibility 
with jieba-analysis 1.0.2; a token-equality run on real text is missing 
([`rfc/matrix.md`](https://github.com/dpol1/hugegraph-rust-rfc/blob/5b05babdc89c2f0803cbd50db18e48a08cf61dc8/rfc/matrix.md))
 |
   
   Recommendation: for the measured loads, the Java optimization makes it 
unjustified to continue the studied JNI path with respect to the 2x objective. 
On this path every mapped operation and load stops, with two exceptions: 
`numberToSortableLong(Long)` on the small load, whose cost is not 
distinguishable from the harness, and the current `decodeSignedB64` on the 
mixed load, which ends as evidence insufficient (the gate says explore; stage C 
was not run). Against the improved Java decoder (21.7 ns instead of 280 ns on 
that load), the declared path stops on every load, so the spike on 
`decodeSignedB64` ends here: no stage C, and the wrapper question stays 
deferred. The other candidates keep their own outcomes above. The conclusion is 
unfavourable to the measured JNI path, on the measured loads, with respect to 
the 2x objective; it does not extend to every fine-grained call or to every 
native implementation. It is compatible with the boundary principle on the 
`refactor/rust-rewrite-de
 sign` branch (explicit RPC or a C ABI, avoiding fine-grained cross-language 
calls; principle 5 of its design document, linked in the next section). 
Coarse-grained or batched boundaries were not measured here; a later candidate 
of that kind would need its own evidence, against the #3223 baselines.
   
   These results exclude the studied path and loads with respect to the 2x 
objective. They do not show that Rust is without value for codecs in general.
   
   ## Relation to `refactor/rust-rewrite-design`
   
   The upstream branch `refactor/rust-rewrite-design` (apache/hugegraph, head 
`a92da16a3` of 2026-09-13) is earlier work on the same roadmap. This RFC does 
not extend to PD or storage, which that work covers.
   
   - **What it proposes** (review drafts, in Chinese): a phased rewrite in 
which Rust services coexist with Java behind gRPC 
([`docs/rust-rewrite-design.md`](https://github.com/apache/hugegraph/blob/a92da16a31a27a7835b8b03932519b54159e1521/docs/rust-rewrite-design.md),
 sections 3 and 5); the boundary principle "explicit RPC or C ABI, avoiding 
fine-grained cross-language calls" (principle 5 in section 2, line 21); a phase 
0 of baselines and contracts, with golden samples generated by the Java version 
that a Rust version must pass (section 8.1); data semantics compared through 
normalized comparators rather than internal bytes (section 8.2); an engineering 
standard for Rust code 
([`docs/rust-engineering-standard.md`](https://github.com/apache/hugegraph/blob/a92da16a31a27a7835b8b03932519b54159e1521/docs/rust-engineering-standard.md)).
   - **What it contains**: a standalone model of PD partition ranges 
([`tools/rust-partition-poc`](https://github.com/apache/hugegraph/tree/a92da16a31a27a7835b8b03932519b54159e1521/tools/rust-partition-poc)),
 a separate process that reads RocksDB files 
([`tools/rust-store-reader`](https://github.com/apache/hugegraph/tree/a92da16a31a27a7835b8b03932519b54159e1521/tools/rust-store-reader)),
 cluster smoke scripts 
([`tools/rust-rewrite-cluster`](https://github.com/apache/hugegraph/tree/a92da16a31a27a7835b8b03932519b54159e1521/tools/rust-rewrite-cluster)),
 and a linearizability checker 
([`tools/raft-linearizability`](https://github.com/apache/hugegraph/tree/a92da16a31a27a7835b8b03932519b54159e1521/tools/raft-linearizability)).
   
   Where the two meet: this RFC's measurements are compatible with principle 5 
for the measured path and loads. The byte-exact corpus of #3146 and the 
branch's normalized comparison answer different questions: encoded keys that 
are persisted need exact bytes; external semantics can be compared after 
normalization.
   
   ## 3. Boundary
   
   Each candidate faced three options:
   
   - **Improved Java.** Argued first, measured when it decides the 
recommendation. It did for `decodeSignedB64` (section 2). For `IntSet`/`IntMap` 
it is argued only (JEP 471).
   - **In-process JNI.** The declared path: the jni-rs 0.22.4 `with_env` 
wrapper, which catches Rust panics that unwind and turns them into Java 
exceptions, around raw JNI calls on stack buffers 
([`rfc/thresholds.md`](https://github.com/dpol1/hugegraph-rust-rfc/blob/5b05babdc89c2f0803cbd50db18e48a08cf61dc8/rfc/thresholds.md)).
 FFM is final only from JDK 22 (JEP 454), so it needs an explicit baseline 
decision and was not measured. JNI obligations are stated per call in 
[`spike/rust/src/lib.rs`](https://github.com/dpol1/hugegraph-rust-rfc/blob/5b05babdc89c2f0803cbd50db18e48a08cf61dc8/spike/rust/src/lib.rs);
 the functional check ran with and without `-Xcheck:jni`, which reported 
nothing 
([`spike/VALIDATION.md`](https://github.com/dpol1/hugegraph-rust-rfc/blob/5b05babdc89c2f0803cbd50db18e48a08cf61dc8/spike/VALIDATION.md)).
 The out-of-memory branches (a NULL result from `NewStringUTF` or 
`NewByteArray`) were not exercised.
   - **Out-of-process.** Excluded from the initial perimeter of scalar 
operations as a scope choice, not by measurement.
   
   A crossing through the declared path with no JNI call inside the closure 
(`passLong`) costs 59 ns (57.9 to 60.1) against a harness baseline of 3.6 ns 
([`results/20260923T134724Z-stage-b`](https://github.com/dpol1/hugegraph-rust-rfc/tree/5b05babdc89c2f0803cbd50db18e48a08cf61dc8/results/20260923T134724Z-stage-b)).
 That cost belongs to the measured path; it is not attributed to JNI or to the 
panic capture. It sets the verdicts for the scalar operations, whose budgets 
(half the Java cost) are about 2 to 13 ns. Removing `with_env` would not force 
`panic = "abort"`: a directly written boundary could keep `catch_unwind` with 
`panic = "unwind"`, but its exception handling, references, and error paths 
would need their own design and checks, and nothing here shows it would be 
faster or as robust.
   
   If a future candidate is recommended, its shape would be: the existing Java 
class keeps its interface; a Java implementation and a native one sit behind an 
internal seam chosen at class initialisation; the Java implementation is the 
fallback for availability and compatibility, not for recovery from a fatal 
native failure. The native library would load where `HugeSecurityManager` 
allows it (it refuses loads from Gremlin workers, 
`hugegraph-server/hugegraph-core/.../security/HugeSecurityManager.java:144-149`);
 loading at startup is one option. Unpaired surrogates: on the declared read 
path (`GetStringRegion`) UTF-16 units arrive unchanged 
([`results/20260923T134724Z-stage-b/check.txt`](https://github.com/dpol1/hugegraph-rust-rfc/blob/5b05babdc89c2f0803cbd50db18e48a08cf61dc8/results/20260923T134724Z-stage-b/check.txt)).
   
   ## 4. Validation
   
   Verified by the spike 
([`spike/VALIDATION.md`](https://github.com/dpol1/hugegraph-rust-rfc/blob/5b05babdc89c2f0803cbd50db18e48a08cf61dc8/spike/VALIDATION.md),
 
[`results/`](https://github.com/dpol1/hugegraph-rust-rfc/tree/5b05babdc89c2f0803cbd50db18e48a08cf61dc8/results)):
   
   - The #3146 corpus is an oracle of recorded outcomes: on its 740 
`decodeSignedB64` cases the improved decoder gives the recorded values and 
exception classes (the corpus records classes only). The differential test is a 
separate comparison with the original on 20,020 generated cases, including 
invalid input, overflow, null, and surrogates: same values, exception classes, 
and messages, except the NullPointerException message, which the check does not 
compare. This covers the verified cases, not every possible input.
   - Measurement guards: runs refuse uncommitted changes and rebuild from the 
recorded revision; the set of benchmarks JMH discovers must equal the set 
derived from the mapping, and every benchmark must have results for the same 
parameters as its class baseline. Every JMH fork prints where the measured Java 
classes were loaded from (stage A) or the native library path it was given 
(stage B), and the scripts check these records against the recorded build; 
stage B also records the library's SHA-256 and checks its exported symbols. 
Each result directory records the commits of thresholds, code, and scripts.
   - Boundary checks: deterministic results, null references, a panic, and an 
out-of-range array each give the expected Java outcome, with and without 
`-Xcheck:jni`, which reported nothing.
   
   Criteria for a future proof of concept (not verified here): concurrency and 
resource release for stateful candidates; native library absent or of the wrong 
version (fallback); observability of which implementation runs; upgrade and 
rollback where a persisted format is involved.
   
   ## 5. CI, packaging, security
   
   With no candidate recommended, this section sizes nothing. Constraints for 
any later candidate: external GitHub Actions pinned to commit SHAs 
(https://infra.apache.org/github-actions-policy.html; the rust-toolchain action 
is allowlisted only at a fixed SHA in `approved_patterns.yml` of 
https://github.com/apache/infrastructure-actions); convenience binaries contain 
only what is built from the source release 
(https://www.apache.org/legal/release-policy.html); `LICENSE`/`NOTICE` of the 
jar match its contents; Rust dependency licences listed, with MPL-2.0 crates 
treated as Category B (https://www.apache.org/legal/resolved.html#category-b); 
`--enable-native-access` documented (JEP 472, JDK 24, 
https://openjdk.org/jeps/472). The engineering standard on 
`refactor/rust-rewrite-design` 
([`docs/rust-engineering-standard.md`](https://github.com/apache/hugegraph/blob/a92da16a31a27a7835b8b03932519b54159e1521/docs/rust-engineering-standard.md))
 already lists toolchain pinning, `clippy -D warnin
 gs`, `cargo audit`, `cargo deny`, an `unsafe` policy, and no panic across FFI.
   
   ## 6. Risks, limits, open questions
   
   Limits of the evidence: one laptop (Intel Core Ultra 7 155H, `powersave` 
governor, no CPU pinning), one JDK (Temurin 11.0.30), one Rust toolchain 
(1.85.0); aarch64 not measured. Every measurement ran on JDK 11: if #3117 moves 
the baseline to Java 17, the Java costs, and with them the budgets, need 
measuring again; FFM would still need JDK 22. Stage A and stage B ran in 
different campaigns, and other work ran on the machine during stage A 
([`rfc/appendix-method.md`](https://github.com/dpol1/hugegraph-rust-rfc/blob/5b05babdc89c2f0803cbd50db18e48a08cf61dc8/rfc/appendix-method.md));
 the borderline `decodeSignedB64` large case changed verdict between two 
campaigns (`explore` in the first, `stop` in the second).
   
   The `mixed` load covers the full `long` range and the `large` load the 1000 
values below `Long.MAX_VALUE`. On the index read path `decodeSignedB64` decodes 
the index label id 
(`hugegraph-server/hugegraph-core/.../structure/HugeIndex.java:240`, encoded 
with `IdGenerator.asStoredString` at `:213`). That id is a schema id: 
`SchemaElement.schemaId` limits it to the `int` range 
(`hugegraph-server/hugegraph-core/.../schema/SchemaElement.java:184-189`), user 
schema ids are positive 
(`hugegraph-server/hugegraph-core/.../backend/tx/SchemaTransaction.java:675`), 
and system ids are small negative constants (`SchemaElement.java:38-48`), so 
user ids need at most 6 characters in this encoding. The other 
`ofStoredString(..., LONG)` callers (`EdgeId.java:296-297`, `IdUtil.java:52`) 
are reached only through `TableSerializer.readId`, and no concrete 
`TableSerializer` exists at `83ef9f3fa`. This bounds the length, not the 
distribution of lengths in production. The `small` load draws values from 0 to
  999 (1 to 2 characters); on that load the improved Java costs 9.5 ns instead 
of 25.2 ns, a saving measured on that load and not established per index row in 
production.
   
   Java-side findings, not Rust questions 
([`rfc/java-findings.md`](https://github.com/dpol1/hugegraph-rust-rfc/blob/5b05babdc89c2f0803cbd50db18e48a08cf61dc8/rfc/java-findings.md)):
 secondary-index prefixes convert n(n+1)/2 elements per update; 
`CollectionFactory` always returns the concurrent segmented `IntSet`/`IntMap`; 
server and store declare different analyzer versions, with no server path that 
triggers a store-side index build in the repository.
   
   Open questions for the maintainers:
   
   1. Is Server core still the preferred first area, given the storage-side 
exploration on `refactor/rust-rewrite-design` and the toolchain priority in 
apache/hugegraph-toolchain#748?
   2. Mentor and Rust reviewers: to be defined; no one is named here as 
responsible.
   3. Is the improved `decodeSignedB64` wanted as a small optimization? A 
checked prototype exists 
([`ImprovedLongEncoding`](https://github.com/dpol1/hugegraph/blob/1c62d1fe7bbed08c1e0d45b81a289df20ceb8ae9/hugegraph-server/hugegraph-test/src/test/java/org/apache/hugegraph/benchmark/rfc/ImprovedLongEncoding.java),
 a class in `hugegraph-test` with a copy of the alphabet); no application-level 
benefit is claimed. The prefix conversion has a different motivation and would 
be evaluated separately.
   4. Is a Jieba token-equality run worth doing?
   5. #3117 timing: when the Java baseline moves, should the Java measurements 
be repeated before the maintainers decide? A raw `extern "system"` no-op 
control without `with_env` would also show whether the `validB64Char` verdicts 
on the late and absent loads depend on the wrapper; it is deferred (section 3).
   6. How do the maintainers want to bring this proposal to discussion and 
decision on the public mailing list?
   
   ## Guardrails from #3110
   
   | Guardrail | Where |
   |---|---|
   | Scope, owner, mentor, non-goals | Sections 1 and 6 (mentor open) |
   | API, protocol, storage-format, operational compatibility | Sections 2 and 
3; persisted formats (`BytesBuffer` rows, `LongEncoding`/`NumericUtil` outputs 
inside index keys, analyzer tokens) not changed |
   | Differential, concurrency, fault-injection, upgrade/rollback tests | 
Section 4; concurrency, fault injection, and upgrade are criteria for a future 
PoC, out of scope here |
   | Latency, throughput, memory, recovery benchmarks | Latency measured; 
throughput, memory, and recovery out of scope for a first selection (#3223 
covers storage) |
   | Security/dependency checks and platforms | Section 5; x86_64 only measured 
|
   | Observability, deployment, coexistence, migration, rollback | Section 3 
(fallback shape) and section 4 (criteria); no deployment proposed |
   | Acceptance criteria and documentation | 
[`rfc/thresholds.md`](https://github.com/dpol1/hugegraph-rust-rfc/blob/5b05babdc89c2f0803cbd50db18e48a08cf61dc8/rfc/thresholds.md);
 this document and 
[`rfc/appendix-method.md`](https://github.com/dpol1/hugegraph-rust-rfc/blob/5b05babdc89c2f0803cbd50db18e48a08cf61dc8/rfc/appendix-method.md)
 |
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to