c-thiel opened a new issue, #3609: URL: https://github.com/apache/iceberg-rust/issues/3609
### Is your feature request related to a problem or challenge? `ArrowReader` builds its `ProjectionMask` from leaf indices numbered over the Arrow schema with arrow-schema's `Fields::filter_leaves`, and passes them to `ProjectionMask::leaves` as Parquet leaf indices. For name-mapped files, predicates use the same numbering (`build_field_id_map_from_arrow_schema`), and so do the row filter, row-group statistics, bloom filters and the page index. This is only correct while arrow's numbering matches Parquet's leaf order for every type the reader can produce. #3393 shows what happens when it doesn't. `filter_leaves` counts a `ListView` or `LargeListView` as one leaf. When the list view's element spans several Parquet leaves, a struct for example, every later column is read from the wrong Parquet leaf. #3393 reports NULLs. When the misread column has the same Arrow type and nullability as the requested one, the reader instead returns that column's **values** under the requested name. This happens because `RecordBatchTransformer::compare_schemas` doesn't compare top-level field ids, so the batch counts as a pure rename. #3467 fixes the ListView case by reading list views as lists. The underlying dependency on arrow's internal numbering remains. #2880 adds a parallel walk for variants that reproduces `filter_leaves`, with a test that pins it. ### Describe the solution you'd like Choose leaves from the Parquet schema, so the indices can't drift from arrow's numbering: - Walk the Parquet type tree under `SchemaDescriptor::root_schema()`. Its depth-first leaf order is the order of `SchemaDescriptor::columns()`, which is also the order `ProjectionMask::leaves` indexes. - Match field ids on each node (`get_basic_info().id()`), and pass the selected leaf indices to `ProjectionMask::leaves`. - A variant group then simply contributes every leaf under it. - For name-mapped files, apply the mapping to the Parquet names during the walk, as iceberg-java's `ApplyNameMapping` does on the `MessageType`. Predicate column lookup then uses the file's own numbering too. This matches iceberg-java, which never computes Parquet leaf indices: - `ParquetSchemaUtil.pruneColumns` / `PruneColumns` prune the file's `MessageType` by Iceberg field id. - parquet-java selects column chunks from the pruned schema by `ColumnPath` (`ParquetFileReader.setRequestedSchema`). - Each reader binds to the `ColumnDescriptor` at its Parquet path. For #2880's variant support this would: - remove the parallel `collect_variant_leaves` walk; its `validate_unshredded_variant` check moves into the new walk; - if the type-promotion check also stops converting the pruned Arrow schema (for example by checking each selected leaf's type), remove the step that tags projected variants with the `arrow.parquet.variant` extension. The tag exists only so that `arrow_schema_to_schema` on the pruned Arrow schema doesn't fail on the variant's id-less children. Its result for the variant is unused, because variant leaves skip the type-promotion check. This also fixes a related case on main. When none of the projected columns exist in a file, the mask falls back to `ProjectionMask::all()`. If the file's columns then match the projection in count, type and nullability, the file's data is relabelled as the new columns. For example, take a file with only `a: long (id 1) = [1, 2, 3]` and a scan projecting only an added `b: long (id 2)`. The scan returns `b = [1, 2, 3]`; the result should be NULLs. Related: #3393, #3467, #2880 (review discussion on the hand-written leaf numbering). ### Willingness to contribute None -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
