sdf-jkl opened a new issue, #11937: URL: https://github.com/apache/arrow-rs/issues/11937
### Is your feature request related to a problem or challenge? This came up while implementing SQL casts to Variant in [datafusion-variant](https://github.com/datafusion-contrib/datafusion-variant). `cast_to_variant` and `cast_to_variant_with_options` cannot distinguish an Arrow UUID column from ordinary `FixedSizeBinary(16)`. The `arrow.uuid` annotation belongs to the input `Field`, but these APIs accept only `&dyn Array`. The conversion dispatches on the storage data type and converts fixed-size binary values to `Variant::Binary`. A caller implementing `CAST(uuid_column AS VARIANT)` therefore cannot preserve UUID semantics through this kernel, even when it has the correctly annotated input field. This follows from the implementation in Arrow 60.0.0 and remains the case on [current main](https://github.com/apache/arrow-rs/blob/dc59495122884e4aced624e6744f268131472094/parquet-variant-compute/src/cast_to_variant.rs#L57-L69): the [fixed-size binary path](https://github.com/apache/arrow-rs/blob/dc59495122884e4aced624e6744f268131472094/parquet-variant-compute/src/arrow_to_variant.rs#L234-L239) receives no extension metadata. ### Describe the solution you'd like Allow callers to supply input field information, for example through a conversion entry point accepting `&Field` and `&dyn Array`. Use the canonical `arrow.uuid` annotation to distinguish UUID from ordinary binary: | Input | Expected Variant value | | --- | --- | | `FixedSizeBinary(16)` annotated with `arrow.uuid` | `Variant::Uuid` | | Unannotated `FixedSizeBinary(16)` | `Variant::Binary` | | Arrow null | Arrow null | Honor UUID annotations on nested fields in lists, structs, and maps as well. Existing array-only callers should retain their binary conversion behavior. The exact API shape is open for discussion. ### Additional context Related work addressed UUID metadata in shredded Variant storage: - [#10015](https://github.com/apache/arrow-rs/pull/10015): merged UUID metadata preservation for shredded fields and Parquet writing. In [#10015's discussion](https://github.com/apache/arrow-rs/pull/10015#discussion_r3349401253), the justification was that a shredded `FixedSizeBinary(16)` field can only represent UUID: Arrow uses dedicated decimal types, and the [shredding specification](https://parquet.apache.org/docs/file-format/types/variantshredding/#shredded-value-types) stores binary as `BYTE_ARRAY`. The merged change therefore attaches UUID metadata when constructing or canonicalizing those shredded fields. That inference relies on the input already being shredded Variant storage. Ordinary Arrow inputs to `cast_to_variant` need their field metadata to distinguish UUID from binary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
