andygrove opened a new issue, #314:
URL: https://github.com/apache/datafusion-spark/issues/314

   ### Describe the bug
   
   `concat_ws` panics when every argument is a scalar and `number_rows` is 
greater than 1. `spark_concat_ws` 
([string/concat_ws.rs#L176](https://github.com/apache/datafusion-spark/blob/21ab12776f55d5646431d81f76b55f6bd0b3ea1e/crates/spark/src/function/string/concat_ws.rs#L176))
 calls `ColumnarValue::values_to_arrays`, which returns 1-row arrays when no 
argument is an array. It then loops over `0..number_rows` and indexes past the 
end of those arrays.
   
   With the default optimizer this is hidden, because a literal-only call is 
constant-folded during planning and evaluated on a one-row batch. It panics 
when the expression is not folded, for example when an engine such as Comet 
builds the physical expression directly.
   
   ### To Reproduce
   
   Run with the `simplify_expressions` optimizer rule removed 
(`Optimizer::new().rules` without it, passed to 
`SessionStateBuilder::with_optimizer_rules`):
   
   ```sql
   SELECT concat_ws(',', 'a', 'b') AS result FROM VALUES (1), (2), (3) AS t(x);
   -- panic: Trying to access an element at index 1 from a StringArray of 
length 1
   ```
   
   ### Expected behavior
   
   Three rows of `a,b`. Either compute one row and return a 
`ColumnarValue::Scalar` when every argument is a scalar, or expand each 
argument with `to_array(number_rows)`. #68 fixed the same pattern in `hash` and 
`xxhash64`.
   
   ### Additional context
   
   Found by the new `evaluate` mode of the sqllogictest runner, which runs the 
`.slt` files without `simplify_expressions`. That query is skipped in 
`evaluate` mode until this is fixed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to