r3stl355 commented on issue #9158: URL: https://github.com/apache/arrow-datafusion/issues/9158#issuecomment-1962319469
I've done some digging but did not find an easy fix, only few options listed below. Happy to follow up but need a decision on which fix to attempt. The following works `select arrow_cast([null], 'FixedSizeList(1, Null)');` so it's logical to use `FixedSizeList(0, Null)` when casting an empty array (`select arrow_cast([], 'FixedSizeList(0, Null)');`). However, that doesn't work because of the following: - Docstring for `datafusion_common::ScalarValue::FixedSizeList` says "The array must be a FixedSizeListArray with length 1." (the same applies to other Scalar::List* types) so any length other than 1 would be invalid. https://github.com/r3stl355/arrow-datafusion/blob/3b355c798a3258f118016b33f26c5a55fed36220/datafusion/common/src/scalar/mod.rs#L231 - During the cast, `datafusion_common::ScalarValue::FixedSizeList(0, Null)` is converted to `arrow_schema::datatype::DataType::FixedSizeList(FieldRef, 0)` before being passed to `arrow::compute::kernels::cast::cast_with_options` for evaluation of the `arrow_cast` - `arrow::compute::kernels::cast::cast_with_options` returns a FixedSizeListArray<0> of length 0 when called with `arrow_schema::datatype::DataType::FixedSizeList(FieldRef, 0)`. Note that this is different for any length greater than 0 used in `FixedSizeList` (i.e. the return value will always be of length 1), e.g. called with `FixedSizeList(FieldRef, 2)` as cast type, `arrow::compute::kernels::cast::cast_with_options` which returns a `FixedSizeListArray<2>` with a length 1. The possible fix options are: 1. Raise an exception if 0 is used as a cast target type (i.e. `FixedSizeList(0, Null)')`) 2. Try to convert `FixedSizeList(FieldRef, 0)` to `FixedSizeList(FieldRef, 1)` before calling `cast_with_options` but A. this feels really wrong and B. may still not work 3. Raise an issue in Arrow asking to return a non-empty array when `cast_with_options` is called with `FixedSizeList(FieldRef, 0)`. I'll do some digging there to see if it's possible, e.g if `FixedSizeListArray<0>[NullArray(0),]` would be a valid type Lastly, this error happens when displaying the result but not when applying some other functions to it, e.g. this following works but its the only function I tested it with: ``` select arrow_typeof(arrow_cast([], 'FixedSizeList(0, Null)')); ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
