adriangb opened a new pull request, #25342: URL: https://github.com/apache/datafusion/pull/25342
# Match Parquet map key/value by position in nested schema coercion ## Which issue does this PR close? - Follow-up to #25193 (which closed #25192). No separate issue. ## Rationale for this change #25193 made `apply_file_schema_type_coercions` recurse into nested types so that, for example, a nested `Utf8` column in a Parquet file is decoded directly as `Utf8View` when the table schema asks for it, instead of being cast afterwards. For `Map` columns the recursion delegated the `key`/`value` children to the by-name `Struct` path, even though the code comment said container children match by position. Parquet readers always name map children `key_value`/`key`/`value`, while Arrow producers commonly use `entries`/`keys`/`values` (e.g. `MapBuilder` defaults). With that naming mismatch the view coercion was silently skipped for maps and queries kept the slower cast path. ## What changes are included in this PR? - `Map` key and value children are now matched by position, regardless of their names. Field names, nullability, metadata, the container kind, `FixedSizeList` width and `Map` ordering continue to come from the file schema. - `apply_file_schema_type_coercions` is restructured around a recursive `coerce_data_type` helper that operates on `DataType`s directly. This removes the throwaway single-field `Schema`s built for each nested level, the `needs_*` pre-scan flags (the result is compared to the input instead), and an unreachable match arm in the container branch (the one uncovered line in the #25193 codecov report). - The doc comment now describes the nested matching rules (structs by name, lists and maps by position) and which parts of the result always come from the file. Behaviour is otherwise unchanged; the existing nested parquet `.slt` files (`schema_evolution_nested`, `parquet_nested_schema_pruning`, `parquet_filter_pushdown`) pass. ## What is the testing strategy for this PR? Unit tests in `schema_coercion.rs`: - `nested_coercion_preserves_list_containers` now uses a table-side child that differs from the file-side child in name, nullability, metadata, `FixedSizeList` width and `Map` ordering, and asserts the file's values win. Mutation testing against #25193 showed that taking the width, ordering or child field from the table survived the previous version of this test; a wrong `FixedSizeList` width is accepted silently by parquet-rs and would decode data with the wrong list size. - `nested_coercion_matches_map_entries_by_position`: Arrow-named table map vs Parquet-named file map, plus a map with a different number of entry children that must be left alone. - `nested_coercion_map_reader_produces_string_views`: writes a file with Parquet's map naming and no embedded Arrow schema, coerces against an Arrow-named table schema, and checks the Parquet reader accepts the schema and produces `StringViewArray` keys and values. - `nested_coercion_list_inside_struct`. The two map tests fail on `main` without the fix. ## Are there any user-facing changes? No API changes. Map columns whose table schema uses `keys`/`values` naming now get the same nested view/string coercion as structs and lists, avoiding a cast during scan. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01LA4Rks1XaN8wxpszj1hXQQ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
