yihua opened a new pull request, #834:
URL: https://github.com/apache/hudi-rs/pull/834
## Description
A C++ query engine needs more from the C ABI than one slice by its paths:
the table's schema, a planned read whose file slices are pruned with the
engine's filters, and each slice's rows as a stream projected to the engine's
columns. The DuckDB extension (alitrack/duckdb-hudi#3) carries that interface
in its own Rust crate today; this puts it here so engines share one surface.
The crate gains `hudi_ffi_table_schema`, a `HudiFfiFilter` struct (`column
op values`, text literals the kernel casts), `hudi_ffi_plan_create` with
`as_of` or a `start`/`end` window, the filters and the read settings the table
was opened with, plan getters (slice, partition and record counts), and
`hudi_ffi_plan_read_slice`, which streams one slice through
`FileGroupReader::read_file_slice_stream` with projection, batch size and the
plan's filters as a row mask. A batch whose columns or types differ from the
declared schema is a stream error, and every top-level field is declared
nullable.
Every handle now runs on one process-wide multi-thread runtime instead of a
current-thread runtime per handle: a stream's `get_next` blocks on the engine's
thread while the kernel's spawned tasks (object store requests among them) keep
running on the runtime's workers. A call from a thread that is already inside a
Tokio runtime fails with a message instead of panicking in `block_on`.
`hudi_ffi_read_slice` keeps its eager read; moving it onto the stream could be
a small follow-up once this shape looks right.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [x] Manual tests
- [x] Details are described below
Unit tests on the sample tables cover the schema with and without meta
fields, projection order, a batch size of one, partition pruning and row
masking on copy-on-write and merge-on-read tables (asserting the latter reads a
slice with a log file), read-optimized mode passed at open, time travel and an
incremental window, a stream outliving its plan, a call from inside a Tokio
runtime, and the error paths. Manually, the same calls back the DuckDB
extension, where all 22 TPC-H SF1 queries match `read_parquet` and the
sqllogictests compare pruned reads against unpruned copies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
