imbajin commented on issue #3145: URL: https://github.com/apache/hugegraph/issues/3145#issuecomment-5719094030
## References: compatibility baselines and a later Java/Rust PoC The Java-only groundwork here has useful precedents. `LongEncoding` remains the first fixture target; choosing a native component and demonstrating its value are later decisions. ### Shared fixtures, golden outputs and differential testing | Reference | Existing practice | What is useful here | |---|---|---| | [SQLite — sqllogictest](https://www.sqlite.org/sqllogictest/doc/trunk/about.wiki) | A reference engine fills in expected query results; validation replays the completed scripts against an engine under test. Independently developed engines provide additional oracles. | The closest example of recording reference results and replaying them elsewhere. Later Java/Rust differential tests can supplement the fixed corpus. This checks query results, not performance or concurrency. | | [SQLite — regression and fuzz testing](https://www.sqlite.org/testing.html) | Historical fuzz cases are retained and replayed by `fuzzcheck`; boundary, malformed-input and fault tests complement ordinary regression tests. | Keep minimized failures as permanent regression inputs, alongside deliberately selected boundary cases. A finite corpus complements rather than replaces other test methods. | | [Apache Parquet — interoperability fixtures](https://github.com/apache/parquet-format#modules) / [parquet-testing](https://github.com/apache/parquet-testing) | Shared data files verify that implementations in different languages can read and write each other's files; the repository also keeps bad-data reproducers. | Reuse one fixture collection across languages. If serialization becomes a candidate later, test old-writer/new-reader and new-writer/old-reader paths explicitly. | | [Apache Arrow — integration testing](https://arrow.apache.org/docs/format/Integration.html) | Human-readable JSON datasets and pre-generated IPC files support cross-language integration tests. | Separate portable test data from language-specific runners. The fixture format need not resemble the implementation's internal objects. | | [Protocol Buffers — conformance suite](https://github.com/protocolbuffers/protobuf/blob/main/conformance/README.md) | A common tester communicates with small adapters for individual language implementations. | Keep a common contract and thin runners. This is a conformance example, not a universal byte-equality rule: [protobuf serialization is not canonical](https://protobuf.dev/programming-guides/serialization-not-canonical/). | | [Go — go/doc golden files](https://go.dev/src/go/doc/testdata/) | Expected outputs are checked in as readable `.golden` files beside test inputs. | A lightweight precedent for reviewable expected-output files; this is an output-regression example, not a storage-format guarantee. | | [Rust Insta — snapshot files](https://insta.rs/docs/snapshot-files/) / [review workflow](https://insta.rs/docs/cli/) | Committed snapshots are compared with new results, with an explicit review/accept workflow. Text snapshot line endings are normalized to LF. | Changes to expected results deserve review. The LF policy also illustrates why fixture transport details should not cause platform-dependent failures. | | [Project Wycheproof](https://github.com/C2SP/wycheproof) | Implementation-agnostic JSON vectors and schemas encode expected behavior for cryptographic edge cases and known bugs. Language-specific tests consume the same vectors. | Label the intent of unusual cases and keep the oracle reusable. The cryptographic requirements themselves are separate from HugeGraph's codec contract. | | [RFC 4648 — encoding test vectors](https://www.rfc-editor.org/rfc/rfc4648.html#section-10) | A small set of explicit input/output examples anchors encoding behavior. | Keep a few easy-to-explain examples beside the larger corpus. HugeGraph's ordered alphabet and sortable prefixes are its own format; RFC 4648 is a testing precedent, not the format to substitute. | The comparison rule must follow the contract: exact encoded strings for HugeGraph's canonical sortable keys; semantic equivalence where a format permits multiple representations; and explicit compatibility decisions for historical malformed-input behavior. ### Java/Rust integration and operational boundaries | Reference | Existing practice | What is useful for a later PoC | |---|---|---| | [Apache DataFusion Comet — architecture](https://datafusion.apache.org/comet/) / [compatibility](https://datafusion.apache.org/comet/user-guide/latest/compatibility/index.html) | Spark/JVM integration delegates supported work to a Rust native engine, with documented compatibility and fallback behavior. | Evaluate a bounded component while preserving observable behavior. A native engine's success does not establish a speedup for individual `LongEncoding` calls. | | [Apache OpenDAL — Java binding](https://opendal.apache.org/docs/bindings/java/) | A Java API uses a Rust core and distributes platform-specific native artifacts. | A concrete reference for classifiers, native loading, platform support and release packaging. HugeGraph's preferred Java fallback still needs its own design. | | [jni-rs](https://docs.rs/jni/latest/jni/) | Rust bindings expose Java native methods with explicit reference lifetimes and error/panic handling at the boundary. | Define ownership and Java-visible error mapping before integration; failures must not unwind across the FFI boundary. | | [JNI design/performance guidance](https://developer.android.com/ndk/guides/jni-tips#general_tips) | The official Android JNI guide recommends reducing marshalling volume and frequency. | Benchmark the complete call path, including crossing, conversion/copying and returned objects. Compare single-item and batched calls where appropriate; Android guidance is not a HugeGraph benchmark. | | [Rust wrapping arithmetic](https://doc.rust-lang.org/std/primitive.i64.html#method.wrapping_add) / [Rust string invariants](https://doc.rust-lang.org/std/primitive.str.html) | Wrapping operations are explicit, while Rust `str` requires valid UTF-8. | Preserve the documented overflow behavior and decide how arbitrary Java UTF-16 inputs, including lone surrogates, reach the Rust boundary. Ordinary string conversion must not silently change the test input. | ### Applying these references without expanding the current scope - Keep the committed corpus and the existing hand-written, ordering and round-trip tests. Record the baseline revision and fixture-format version; inspect behavior changes before regenerating expected results. - A future native candidate should demonstrate compatibility and end-to-end value against the Java baseline on representative workloads. Java fallback should cover defined availability/compatibility cases; it is not recovery from a fatal native-process failure. - One coverage clarification for the earlier inventory: #3146 currently covers `encodeSortable`/`decodeSortable`, `encodeB64`/`decodeB64`, and `encodeSignedB64`/`decodeSignedB64`. It does **not** include `encodeNumber` or a shared `NumericUtil` corpus. Their existing Java unit tests and a future portable corpus are distinct forms of coverage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
