codewarnab commented on issue #8730: URL: https://github.com/apache/arrow-rs/issues/8730#issuecomment-6611528377
I'd like to pick this up again. I'll start from @tustvold's question about nested types (Jan 20). Nobody answered it. To check that question, I built a small local prototype. It covers one path: `cast_to_variant` with an `arrow.uuid` field (#11937). **Nested types were the easy part.** - The parent's `DataType` already holds each child `Field`. Examples: `List(FieldRef)` and `Struct(Fields)`. - The code dropped that `Field` and passed only `values.data_type()`. - I changed 6 places to pass the child `Field` down. These are: list types, struct, map values and union children. - Now a UUID inside a `List` or `Struct` becomes `Variant::Uuid`. The existing `cast_to_variant(&dyn Array)` does this, with no API change. **Dictionary and run-end encoded values were also easy.** They only needed the outer `Field` passed through (2 places). **The top-level call was the only hard part.** It has no `Field` at all. It needed a new function: `cast_to_variant_with_field(array, Option<&Field>, options)`. **Size:** - About 125 lines of code and about 60 lines of tests. - All changes are in `parquet-variant-compute`. - No changes to `Field`, `DataType` or the array structs. - All existing tests pass. **What I think this shows.** For this function, nesting is not the problem. The problem is top-level functions that receive only an `ArrayRef` or a `DataType`. These functions have the same gap: - `cast`, `cast_with_options` and `can_cast_types` - `ArrayFormatter::try_new` - `SortField::new` #12044 works around a similar case. It passes the `Field` as an extra argument. I can share the branch if it helps. @tustvold, please correct me if I misread your concern. @sdf-jkl @paleolimbot, this may help with the Variant side and the implementation side. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
