viirya opened a new issue, #6470:
URL: https://github.com/apache/datafusion-comet/issues/6470
### Describe the bug
On Spark 4.x with the default
`spark.comet.exec.scalaUDF.codegen.enabled=true`, `array_contains`,
`arrays_overlap`, `array_distinct` and `array_union` return wrong results when
their array elements are strings with a non-`UTF8_BINARY` collation.
A cast to a collated string type has no native path, so Comet runs it
through the JVM codegen dispatcher. The `array(...)` above it and the four
array functions still run natively, and their native kernels compare strings by
raw bytes, so the collation is ignored.
### Steps to reproduce
Parquet table with columns `_1`, `_2` and rows `("a", "A")`, `("x ", "x")`,
`("b", "c")`:
```sql
SELECT array_contains(array(CAST(_1 AS STRING COLLATE UTF8_LCASE)), CAST(_2
AS STRING COLLATE UTF8_LCASE)) FROM t;
SELECT arrays_overlap(array(CAST(_1 AS STRING COLLATE UTF8_LCASE)),
array(CAST(_2 AS STRING COLLATE UTF8_LCASE))) FROM t;
```
- `UTF8_LCASE`: Spark returns `true` for `('a', 'A')`, Comet returns `false`.
- `UTF8_BINARY_RTRIM`: Spark returns `true` for `('x ', 'x')`, Comet returns
`false`.
- `UNICODE_CI` behaves like `UTF8_LCASE`.
- `array_distinct(array(a, b))` and `array_union(array(a), array(b))` keep
both `'a'` and `'A'` (or `'x '` and `'x'`) where Spark keeps one.
The plan shows `cast` dispatched and `array` plus the array function native.
### Expected behavior
The results match Spark. Other collation-sensitive expressions already route
collated input through the codegen dispatcher (for example `=`, `IN`,
`array_intersect`, `array_except`, `array_max`) or fall back (`array_position`,
`array_remove`).
### Additional context
Reproduced on Spark 4.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
