gstvg commented on issue #15896: URL: https://github.com/apache/datafusion/issues/15896#issuecomment-2840980768
After https://github.com/apache/datafusion/pull/15149, flatten stopped working for `List(FixedSizeList)` because it expected the inner fixed size list to be casted to list, which wasn't happening anymore. Looking back, https://github.com/apache/datafusion/pull/15160 correctly stopped using [ArrayFunctionSignature::RecursiveArray](https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.ArrayFunctionSignature.html#variant.RecursiveArray), which I expected would make only the top level list be coerced, and it would break, but it didn't. It appers that recursive lists continued to be coerced And then https://github.com/apache/datafusion/pull/15149 refactored that code path with the correct behavior, exposing the issue in flatten I opened #15898 with a potential fix. Other options include: 1. Reverting to [ArrayFunctionSignature::RecursiveArray](https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.ArrayFunctionSignature.html#variant.RecursiveArray) again, but for deeply recursive fixed size lists, it would unnecessarily coerce some of them(which includes building offsets buffers). 2. Creating a new [ArrayFunctionSignature](https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.ArrayFunctionSignature.html) variant for this case 3. Manually handling fixed size list in `flatten` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
