init-js opened a new pull request, #12234: URL: https://github.com/apache/arrow-rs/pull/12234
# Which issue does this PR close? - Closes #12233. This change was first reviewed in init-js/arrow-rs#1. (https://github.com/init-js/arrow-rs/pull/1) # Rationale for this change We have hit issues trying to place extension/annotation types onto elements of the legacy 2-level lists. The symptom is that the annotation gets placed on the resulting list wrapper, rather than the element contained in the list. When you then try to convert the Parquet schema to Arrow, arrow bails because it finds types in the wrong place. It will complain, for instance, that "UUID" cannot be placed on a List. This affects every logical type that the converter maps to an Arrow extension type ([extension.rs#L29-L93](https://github.com/apache/arrow-rs/blob/30c5c3287cd5b99ca9aa0aec57509d076eb07347/parquet/src/arrow/schema/extension.rs#L29-L93)): | Logical type | Annotates | Extension | Feature | |---|---|---|---| | `UUID` | `FIXED_LEN_BYTE_ARRAY(16)` | `arrow.uuid` | `arrow_canonical_extension_types` | | `JSON` | `BYTE_ARRAY` | `arrow.json` | `arrow_canonical_extension_types` | | `GEOMETRY`, `GEOGRAPHY` | `BYTE_ARRAY` | `geoarrow.wkb` | `geospatial` | | `VARIANT` | group | `arrow.parquet.variant` | `variant_experimental` | | `FILE` | group | `arrow.parquet.file` | `file_experimental` | Other logical types, such as `STRING`, `DATE` or `DECIMAL`, map to a native Arrow data type and carry no extension type, so they are not affected. The two forms of 2-level list fail differently: - A bare repeated field gets the extension type attached to the list field, and `Field::try_with_extension_type` rejects it because the extension's `supports_data_type` does not accept a `List` ([field.rs#L578](https://github.com/apache/arrow-rs/blob/30c5c3287cd5b99ca9aa0aec57509d076eb07347/arrow-schema/src/field.rs#L578), [uuid.rs#L55-L62](https://github.com/apache/arrow-rs/blob/30c5c3287cd5b99ca9aa0aec57509d076eb07347/arrow-schema/src/extension/canonical/uuid.rs#L55-L62)). Example schema that would trigger the bug: ``` message test_schema { repeated fixed_len_byte_array(16) col (UUID); } ``` It fails with `Uuid data type mismatch, expected FixedSizeBinary(16), found List(non-null FixedSizeBinary(16), field: 'col', metadata: {"ARROW:extension:name": "arrow.uuid"})`. Each of the other types fails with its own storage check, for example `Json data type mismatch, expected one of Utf8, LargeUtf8, Utf8View, found List(...)`. - A LIST group whose repeated child is the element converts without error, but the element silently loses its extension type: ``` message test_schema { optional group col (LIST) { repeated fixed_len_byte_array(16) element (UUID); } } ``` converts to `List<FixedSizeBinary(16)>` with no `arrow.uuid` on the element. 3-level lists are not affected: their element is a separate, non-repeated node, converted on its own ([complex.rs#L626](https://github.com/apache/arrow-rs/blob/30c5c3287cd5b99ca9aa0aec57509d076eb07347/parquet/src/arrow/schema/complex.rs#L626)). # What changes are included in this PR? - `convert_field` no longer attaches a repeated node's extension type to the list it converts to. This fixes the bare repeated field. - `ParquetField::into_list` takes the repeated node's `Type` and attaches its extension type to the element field, as `into_list_with_arrow_list_hint` already does. This fixes the LIST group whose repeated child is the element. # Are these changes tested? Yes. New tests in `arrow::schema::tests` cover UUID in a bare repeated primitive, in a LIST group whose repeated child is the element, in a 3-level list, and in a list of 2-level lists, plus JSON in both 2-level forms. All but the plain 3-level test fail on `main` and pass with this change. # Are there any significant user-facing changes? No API change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
