JingsongLi opened a new pull request, #460: URL: https://github.com/apache/paimon-rust/pull/460
## Summary This PR adds DataFusion-side Variant extraction pushdown for simple `variant_get` / `try_variant_get` expressions over Paimon `VARIANT` columns. Instead of always reading and rebuilding the full Variant value, eligible projections and filters can request connector-defined extraction read types and evaluate on extracted typed fields. ## Changes - Add Java-compatible Variant extraction metadata markers and allow `ReadBuilder` / `PaimonTableScan` to carry full read types, not only top-level projections. - Teach shredding reads to assemble extraction structs from both plain Variant arrays and shredded Variant storage. - Add a DataFusion optimizer rule and extension planner that rewrite eligible `variant_get` calls to scan-level Variant extraction plus `get_field` access. - Keep correctness conservative when the same query also needs the full Variant column. - Update SQL documentation to distinguish extraction pushdown from predicate/statistics pushdown. ## Testing - `cargo test -p paimon variant_extraction -- --nocapture` - `cargo test -p paimon read_data_fields -- --nocapture` - `cargo test -p paimon-datafusion --test variant_pushdown -- --nocapture` - `cargo check -p paimon-datafusion` ## Notes Predicate translation through `variant_get` is still not pushed into Paimon/Parquet statistics. DataFusion evaluates those predicates after reading the extracted field. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
