JingsongLi opened a new pull request, #1248: URL: https://github.com/apache/paimon-rust/pull/1248
### Purpose A read type that selects only fields inside an ARRAY or MAP must reach the Parquet leaf-column projection. For example, reading only `items[].wanted` from `ARRAY<ROW<wanted, unused>>` should avoid decoding `unused`, including through mixed ROW/ARRAY/MAP nesting. This also makes formats without nested projection support read complete physical fields and restore the requested shape outside the decoder, while preserving Java Paimon's field-ID and null/anchor semantics. ### Brief change log - Add a format-level nested-projection capability. Parquet opts in; other formats expand nested fields and use outer reconciliation. - Recursively prune ROW, ARRAY elements, and MAP keys/values, preserving file names, IDs, physical types, defaults, and Java's all-missing-field behavior. - Build Parquet leaf masks for standard and legacy LIST layouts, MAP keys/values, and shared-shredding MAP value projections. - Preserve omitted descendants during fallback reconciliation so decoding a complete ROW cannot resurrect a container that logical pruning removed. - Fix Avro child resolution when the first nested ROW is null. Add a Vortex 0.75 read-side fallback that reconstructs each ROW with its independent validity and selects rows before Arrow export. - Add mixed-container, null/empty, schema-evolution, duplicate-key, physical-I/O, Java-written LIST fixture, and ORC fallback regressions. Keep Java's complete-value restriction for selected-key MAP projections. ### Tests - `cargo fmt --all -- --check`: passed. - Paimon library suite with `vortex`: **4671 passed, 6 ignored**. - **79** other Paimon integration/example targets passed in batches; the initial aggregate build exceeded local disk capacity. - Java golden warehouse suites: **82 passed** (11 append-table tests and 71 read-table tests). - `cargo clippy --locked --all-targets --workspace --features vortex -- -D warnings -A clippy::nonminimal_bool`: passed. Strict Clippy is blocked by four pre-existing warnings in unchanged `de_vector_read.rs` (two), `table_merge_into.rs`, and `vector_search_common.rs`. - Multiple independent review/fix rounds completed without remaining findings. Local full-suite builds used `CARGO_INCREMENTAL=0` and `CARGO_PROFILE_TEST_DEBUG=0` to limit build artifacts. ### API and Format No public API, table option, or on-disk format changes. Nested requests continue to use `with_read_type`. Readers that cannot prune nested fields keep full decoder types and apply projection outside the format reader. ### Documentation Implementation comments document the format capability, Java-compatible pruning behavior, and Vortex validity fallback. Fixture generators are included alongside the saved test files. ### AI assistance Codex assisted with implementation, regression testing, and review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
