XiaoHongbo-Hope opened a new pull request, #1009: URL: https://github.com/apache/paimon-rust/pull/1009
### Purpose Avoid materializing a complete wide Variant column in Python when a reader only needs many typed numeric paths. The existing Variant extraction read type evaluates each row/path independently. This change reuses the batched layout/slot cache added by #997 for the common case of top-level `FLOAT` extractions, while preserving the existing read type and output schema. ### Brief change log - Batch plain, top-level Variant-to-`FLOAT` extractions through the existing typed Variant read path. - Preserve path order, duplicate paths, null rows, missing fields, numeric casts, and `fail_on_error` behavior. - Fall back to the existing general implementation for nested paths, mixed target types, and shredded Variant values. - Add a low-level Python `ReadBuilder.with_variant_projection(...)` hook that composes with normal column projection. A PyPaimon public API is intentionally out of scope. ### Tests - `cargo test -p paimon arrow::shredding::variant::tests --lib` - `cargo test -p paimon arrow::variant::tests --lib` - `cargo test -p paimon variant::numeric::tests --lib` - `cargo test -p paimon-datafusion --test variant_pushdown` - `pytest -q bindings/python/tests/test_read.py` (107 passed) - `cargo clippy -p paimon -p pypaimon_rust --all-targets -- -D warnings` An earlier F-zone prototype used the same #997 batched extraction kernel immediately before the Python FFI boundary on the target workload (8 ranks x 8 workers, batch size 16, three fresh runs): | metric | control median | batched prototype median | | --- | ---: | ---: | | supply throughput | 276.67 samples/s | 337.95 samples/s | | pose p50 | 1.337 s | 1.106 s | | Python-visible topic Arrow bytes | 426.3 MB | 1.21 MB | | process-tree peak RSS | 106.8 GiB | 106.5 GiB | The throughput change was +22.1%. The prototype returned a `FixedSizeList<float32>` from a benchmark-only post-read hook; this PR instead integrates the same kernel with the existing typed Struct read path. These numbers motivate the change but are not presented as an exact A/B of the final API. ### API and Format Adds a Python binding method. No storage-format change. ### Documentation The low-level binding is documented in its type stub. Higher-level PyPaimon API work is separate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
