yihua opened a new issue, #848:
URL: https://github.com/apache/hudi-rs/issues/848
hudi-rs now exposes its reader through four non-Rust interfaces, and new
reader features increasingly have to be wired through each of them. I'd like to
agree on a direction for 0.7.0 and beyond so binding work converges on one
place.
Nothing here blocks current work: open PRs against any of the bindings can
go in as they are in the interim.
## Where we are
| Interface | Crate | Consumers | Published |
|---|---|---|---|
| Python (PyO3) | `python/` | PyPI wheel | yes |
| DataFusion FFI | `python/` | `datafusion-python` | through the wheel |
| C++ (cxx) | `cpp/` (`hudi-cpp`) | demo app | no |
| C ABI | `crates/jvm-ffi` (`hudi-ffi` after #847) | JVM through JNA; C++
engines with #834 | no |
The same concerns are being solved once per binding:
- Filters: Python takes `(field, op, value)` tuples, #767 adds `HudiFilter`
to the cxx bridge, and #834 adds `HudiFfiFilter` to the C ABI.
- Runtimes: the cxx bridge creates a current-thread Tokio runtime per
reader, #834 moves the C ABI to one process-wide runtime, and Python has its
own (#829).
- Panic guards landed separately: #700 for the C ABI, #766 for the cxx
bridge.
- Read options are parsed and documented per binding, and they already
differ in which keys apply per read.
## Proposal
1. Keep Python as the high-level binding. PyO3 is the right tool there.
2. Make the C ABI (`hudi-ffi`) the one native boundary for every other
caller: JVMs, C++ engines like DuckDB, and anything else with a C FFI. Plain C
plus the Arrow C Data Interface has no C++ toolchain coupling, which matters
for engines built across many platforms, and a JVM can bind it where it cannot
bind cxx.
3. Stop maintaining `hudi-cpp` as a second implementation of the reader
bindings. Two ways to get there:
- Deprecate and remove it; C++ callers use the C ABI directly.
- Keep a C++ API with the same entry points
(`new_file_group_reader_with_options`, `read_file_slice`, ...), but as a thin
header-only wrapper over the C ABI (RAII handles, exceptions raised from
`hudi_ffi_last_error`). Existing C++ callers keep working with small changes,
such as cxx types like `rust::Box` becoming plain C++ types, while hudi-rs
maintains only the C ABI behind it.
4. Move shared parsing into `hudi-core` (filter tuples already live there;
`key=value` options could join them) so each binding only converts types.
## Rough timeline
- 0.7.0: publish `hudi-ffi` with a C header and a stated ABI compatibility
policy; mark `hudi-cpp` deprecated.
- 1.0.0: remove `hudi-cpp`, or ship it only as the wrapper over the C ABI.
## Open questions
- Who uses `hudi-cpp` today, and would the C ABI cover those cases? @codope,
you've been closest to the cxx bridge.
- @linliu-code, does the Gluten work load the library by name, and is JNA
still the plan for the JVM side?
- Header: hand-written, or generated with `cbindgen`?
- Distribution: crates.io only, or prebuilt shared libraries per platform?
Comments welcome. If there's agreement, I'll summarize the outcome on dev@
so the decision is recorded there too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
