viirya opened a new issue, #6758:
URL: https://github.com/apache/datafusion-comet/issues/6758
### Describe the bug
With `spark.sql.caseSensitive=false`, `greatest` and `least` return a
different struct from Spark when the arguments' struct field names differ only
in case.
Spark accepts such arguments because `TypeCoercion.haveSameType` compares
the types ignoring case and nullability. It then compares the structs field by
field by position, and the result keeps the first argument's field names. Comet
passes the arguments through DataFusion's `greatest`/`least` coercion. When
both structs hold the same set of field names (here `x` and `X`, in different
orders), DataFusion's struct coercion matches the fields by name, and Arrow's
struct cast reorders the second argument's values by name. So Comet compares
different pairs of fields than Spark does.
This affects every supported Spark version (3.4 to 4.1), for float and
non-float fields, for structs nested in arrays, and for three or more
arguments. With three arguments that also differ in field nullability, the
native cast fails instead: `Cannot cast nullable struct field 'X' to
non-nullable field`.
### Steps to reproduce
```sql
SET spark.sql.caseSensitive=false;
SELECT id, greatest(named_struct('x', CAST(id AS DOUBLE), 'X', 1D),
named_struct('X', 1D, 'x', CAST(id AS DOUBLE)))
FROM range(4);
```
With `spark.comet.exec.range.enabled=true`,
`spark.comet.sparkToColumnar.enabled=true`, and
`spark.comet.sparkToColumnar.supportedOperatorList=Range`, Comet runs this
natively. For `id = 0` Spark returns `[1.0, 0.0]` and Comet returns `[0.0,
1.0]`. `least` has the same problem: for `id = 2` it returns `[2.0, 1.0]` where
Spark returns `[1.0, 2.0]`.
### Expected behavior
Comet compares struct arguments by position, as Spark does, and returns
`[1.0, 0.0]` for `id = 0`.
### Additional context
`coalesce`, `IF`, `CASE WHEN`, `array`, `map`, `concat`, `map_concat`,
`array_append`, `array_insert`, `array_contains`, `array_intersect`,
`array_except`, `arrays_overlap`, `=`, `<`, `IN` and `nullif` match Spark for
the same arguments. `array_union` with integer struct fields fails natively
with `Union is not implemented for 'ListArray'` rather than returning a wrong
answer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
