darshpatel1052 opened a new pull request, #3624: URL: https://github.com/apache/iceberg-rust/pull/3624
## Which issue does this PR close? - Closes #3609. ## What changes are included in this PR? This PR fixes incorrect Parquet column projection and predicate lookup caused by relying on Arrow's leaf numbering, which can differ from the actual Parquet column order for nested types such as `ListView` and `LargeListView`. The changes include: - **Parquet-based column indexing:** Resolve field IDs to leaf indices using the Parquet schema instead of `Fields::filter_leaves`, ensuring projection masks select the correct physical columns. - **Name mapping:** Replace Arrow-based leaf index mapping with a recursive Parquet schema walk. Handle nested structs, maps, and different Parquet list encodings, including legacy repeated-field layouts. - **Predicate pushdown:** Use Parquet-derived field ID mappings for predicate evaluation, row-group statistics, bloom filters, and page-index filtering. - **Nested list views:** Recursively traverse `ListView` and `LargeListView` elements during schema mapping and projected-field collection. - **Missing columns:** Remove the fallback to `ProjectionMask::all()` when no requested columns exist in the file. This prevents unrelated file values from being returned under newly added column names. - **Zero-column batches:** Preserve the original row count when transforming batches with no physical columns, allowing missing fields to be populated with NULLs or initial defaults. - **Partially annotated schemas:** Allow Parquet schemas containing both ID-bearing and ID-less leaves without discarding valid field ID mappings. ## Are these changes tested? Added and extended regression tests covering: - `test_projection_after_nested_list_view`: Tests nested lists with multiple element leaves, including regular/large lists and list views, with embedded field IDs and name mapping. Covers projection and predicate filtering through row-group, bloom-filter, and page-index configurations. - `test_name_mapping_parquet_list_encodings`: Verifies correct leaf indices across different Parquet list encodings, including legacy repeated fields. - `test_schema_evolution_add_column`: Verifies that existing data is preserved while missing columns return NULLs or their configured initial defaults, including projections containing only absent fields. - Extended name-mapping tests for nested fields, unmapped fields, and duplicate aliases. The reader-level regression tests also exercise the related pipeline and record-batch transformation changes. ## AI Disclosure OpenAI Codex (GPT-6.1 Sol, xhigh reasoning effort) assisted with regression-test development, and code review. I reviewed the resulting changes and take responsibility for their correctness. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
