comphead opened a new pull request, #3467: URL: https://github.com/apache/iceberg-rust/pull/3467
## Which issue does this PR close? - Closes #3393. ## What changes are included in this PR? parquet-rs reads a Parquet LIST as `ListView` or `LargeListView` when the file's embedded Arrow schema says so. The projection mask is built with arrow-schema's `Fields::filter_leaves`, which treats list views as leaves. So a list view column read as NULL whenever another column was projected, and a list view of structs shifted the leaf index of every later column, which was then read from the wrong Parquet leaf. `configure_arrow_reader_metadata` now rewrites every `ListView` and `LargeListView` to `List` and `LargeList` in the schema hint it passes to parquet-rs, at any depth and keeping field metadata. Name mapping, fallback IDs, and INT96 coercion then start from that schema. A schema without list views is not copied. The transformer then handles these columns as it already handles `List` and `LargeList`. ## Are these changes tested? Yes. - `test_read_list_view_columns` reads the file from #3393, a `list_view<struct<a, b>>` followed by an `int`, and checks both columns. On `main` that read fails with `cannot promote ListView(...) to List(...)`, and projecting only the `int` returns NULL. - `test_list_views_as_lists` checks the rewrite at every nesting position, and that a schema without list views is left alone. As a spot check, I broke the change in 12 ways, one at a time: skipping the rewrite in the pipeline, not using it when no other fix-up applies, each list kind and container arm, dropping field metadata, missing structs or maps when detecting list views, and rebuilding a schema that has none. Each mutation failed at least one of the two tests. ## AI Disclosure Drafted with Claude Code. The behavior on `main` was checked against `main` at `b3470c846`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
