andygrove opened a new issue, #314: URL: https://github.com/apache/datafusion-spark/issues/314
### Describe the bug `concat_ws` panics when every argument is a scalar and `number_rows` is greater than 1. `spark_concat_ws` ([string/concat_ws.rs#L176](https://github.com/apache/datafusion-spark/blob/21ab12776f55d5646431d81f76b55f6bd0b3ea1e/crates/spark/src/function/string/concat_ws.rs#L176)) calls `ColumnarValue::values_to_arrays`, which returns 1-row arrays when no argument is an array. It then loops over `0..number_rows` and indexes past the end of those arrays. With the default optimizer this is hidden, because a literal-only call is constant-folded during planning and evaluated on a one-row batch. It panics when the expression is not folded, for example when an engine such as Comet builds the physical expression directly. ### To Reproduce Run with the `simplify_expressions` optimizer rule removed (`Optimizer::new().rules` without it, passed to `SessionStateBuilder::with_optimizer_rules`): ```sql SELECT concat_ws(',', 'a', 'b') AS result FROM VALUES (1), (2), (3) AS t(x); -- panic: Trying to access an element at index 1 from a StringArray of length 1 ``` ### Expected behavior Three rows of `a,b`. Either compute one row and return a `ColumnarValue::Scalar` when every argument is a scalar, or expand each argument with `to_array(number_rows)`. #68 fixed the same pattern in `hash` and `xxhash64`. ### Additional context Found by the new `evaluate` mode of the sqllogictest runner, which runs the `.slt` files without `simplify_expressions`. That query is skipped in `evaluate` mode until this is fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
