linliu-code opened a new pull request, #700:
URL: https://github.com/apache/hudi-rs/pull/700
## Description
A JVM cannot call hudi-rs today. The `cpp` bridge is generated by `cxx`, so
its symbols are C++-mangled and its arguments are `CxxString`/`CxxVector`;
neither binds from Java. This adds `crates/jvm-ffi`, a plain C ABI over the
reads that already exist, and one new read the metadata table needed.
That gap matters most for the metadata table, because Gluten offloads
`SparkPlan` nodes and a metadata read is not one: `HoodieFileIndex` consults
the metadata table while the plan is being built, and its output is what
produces the scan Gluten later offloads. There is no node to hand over, so a
JVM caller wanting hudi-rs's metadata reader currently has no route at all.
**The core change is a split, not new behaviour.** Reading the `files`
partition built a `RecordBatch` and immediately decoded it into
`FilesPartitionRecord`s. An Arrow consumer would have taken those structs and
re-encoded them, paying for a decode it did not want. `read_files_partition` is
now `read_files_partition_batch` plus the decode that was already in it, and
`Table::read_metadata_table_files_partition_arrow` exposes the batch. Slice
resolution is shared by both, so neither path can drift from the other.
**The shim holds no reading logic.** Null-terminated strings in, opaque
handles and an `ArrowArrayStream` out. Every export routes through a guard that
turns an error or a panic into a null plus a message retrievable with
`hudi_ffi_last_error`, because a panic unwinding across `extern "C"` aborts the
process that called us.
Two things a reviewer should know rather than infer. `hudi_ffi_read_slice`
reads a slice through the generic file group reader, which refuses an HFile
base file by design, so it serves parquet slices and not metadata ones; the
metadata read is the separate entry point above. And a JVM consumer must run
with `--add-opens=java.base/java.nio=ALL-UNNAMED`, which arrow-java requires
and fails loudly without.
Stacked on #696. Review that first; this diff is the last commit.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [x] Manual tests
- [x] Details are described below
Six tests cover the boundary, and three are mutation-checked, since a guard
that is never exercised is not a guard:
| Mutation | Result |
| --- | --- |
| remove `catch_unwind` from the guard |
`a_panic_becomes_a_null_and_a_message` fails |
| remove the null check in `hudi_ffi_free_stream` |
`freeing_null_is_a_no_op` aborts with SIGABRT |
| weaken the UTF-8 error message | `invalid_utf8_is_refused` fails |
The others cover an error becoming a null plus a message, null arguments
being refused rather than dereferenced, and the error slot being per-thread so
one thread's failure is not another's.
Manually verified end to end from Java 25, using FFM to call the shim and
arrow-java 15 to import the stream, against a real metadata table:
```
reader opened
columns: 12
key=city=chennai type=2
key=city=san_francisco type=2
key=__all_partitions__ type=1
key=city=sao_paulo type=2
rows: 4
```
Those are the same four records
`hudi_table_read_metadata_table_files_partition` asserts in Rust, with the
`type` discriminator intact.
Locally: 1386 `hudi-core` lib tests pass, 6 in `hudi-jvm-ffi`, clippy clean
on both with `-D warnings`, fmt clean.
**Not covered, and stated rather than implied.** The JVM half of the
ownership contract is untested here: that the stream is freed exactly once
including on an exception path, and that a panic does not abort a real JVM, are
asserted on the Rust side only. Proving them from Java needs a test module that
does not exist yet in this repo. The panicking export used for that is behind a
default-off `ffi-test-hooks` feature so it is absent from a shipped library.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
