peterxcli opened a new issue, #5191:
URL: https://github.com/apache/datafusion-comet/issues/5191
## Describe the bug
Nested floating-point elements use total-order comparison in Comet, where
`-0.0` and `0.0` are distinct. Spark uses `ordering.equiv` for nested elements,
which checks numeric equality first and therefore treats both signed zeros as
equal.
This affects:
- `arrays_overlap` when the array elements are themselves arrays or structs
containing floats.
- The nested fallback in `array_position`.
The flat `arrays_overlap` path is intentionally different: Spark uses
`Double.equals` there and distinguishes signed zeros.
## Steps to reproduce
```sql
CREATE TABLE t(
a ARRAY<ARRAY<DOUBLE>>,
b ARRAY<ARRAY<DOUBLE>>,
v ARRAY<DOUBLE>
) USING parquet;
INSERT INTO t VALUES (
array(array(0.0D)),
array(array(-0.0D)),
array(-0.0D)
);
SELECT arrays_overlap(a, b), array_position(a, v) FROM t;
```
Spark returns `true` and `1`; Comet returns `false` and `0`.
## Expected behavior
Nested `arrays_overlap` and `array_position` should treat `-0.0` as equal to
`0.0`, matching Spark, while continuing to treat NaN as equal to NaN.
## Additional context
Tracked separately from #5176 because the mismatch predates that refactor.
Arrow's `make_comparator` uses total ordering; a potential fix is to normalize
negative zero in nested float leaves before constructing the comparator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
