zhouyc-ali opened a new pull request, #376:
URL: https://github.com/apache/paimon-cpp/pull/376
### Purpose
Linked issue: close #319
Adds optional read and write support for the Apache Paimon Vortex file
format, integrated through
the upstream Vortex Rust FFI and gated by `PAIMON_ENABLE_VORTEX`.
- Add the Vortex file reader, writer, format factory, stats extractor, and
Arrow C Data Interface
conversion.
- Add a callback-based IO bridge so Vortex reads and writes directly through
paimon's
`InputStream`/`OutputStream` (works with any `FileSystem`: local, OSS, S3,
Jindo), instead of
buffering whole files in memory on read or staging a local temporary file
on write. The stock
Vortex C API cannot route IO through paimon's filesystem, so the bridge is
added to `vortex-ffi`
as an injected Rust module (`crates/vortex_callback_io/`) plus a one-line
patch
(`cmake_modules/vortex.diff`).
- Support projection and configurable output batch sizes; reset the reader
to the beginning when
`SetReadSchema` is called.
- Read the file row count for `FileInfo`; Vortex exposes no per-column
statistics to Paimon
(consistent with the Java implementation).
- Validate Vortex-incompatible schema types (MAP, MULTISET, VARIANT, BLOB).
- Register Vortex in the file-format factory and bypass the generic
`PrefetchFileBatchReader`
(Vortex performs random-access reads through its own callbacks).
- Enable Vortex in CI: add `-DPAIMON_ENABLE_VORTEX=ON` to `build_paimon.sh`,
install the `stable`
Rust toolchain (Vortex MSRV 1.95), and build `flatc` 25.12.19
(`ci/scripts/setup_flatc.sh`), which
Vortex needs to compile its FlatBuffers schemas at build time.
Predicate/projection pushdown into Vortex is not implemented; all rows and
columns are scanned and
projected to the read schema.
### Tests
- Vortex unit tests: round-trip read/write across multiple batch sizes;
primitive, string, binary,
date, timestamp, decimal and list types; row-count statistics extraction;
projected reads; and IO
fault injection at every read and write position (verifying paimon IO
errors surface through the
FFI callback boundary rather than an opaque Vortex error).
- Rust unit tests (`cargo test -p vortex-ffi callback_io`): callback read
round-trip, callback
write→read round-trip, and context release on open failure.
- Integration test `TestVortexJavaCompatibility`: reads an append table
written by Paimon Java 2.0.0
(`paimon-vortex-format` + `paimon-vortex-jni`).
- Full CTest suite passes (33/33 targets); clang-format, cpplint,
cmake-format and rustfmt clean.
### API and Format
Adds the optional Vortex file-format factory and the `PAIMON_ENABLE_VORTEX`
CMake option. It reads
and writes the existing Apache Paimon Vortex file format through the
upstream Vortex Rust FFI and
does not change the Vortex binary format or any existing public API under
`include/`. Building Vortex
requires a Rust toolchain (>= 1.95) and `flatc` 25.12.19; both are set up in
CI.
### Documentation
Updated `docs/source/user_guide/format_table.rst` to list Vortex among the
formats a format table
does not reach, and added `crates/vortex_callback_io/README.md` describing
the injected module and
its build wiring.
### Generative AI tooling
Generated-by: Qoder
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
