sunchao opened a new issue, #6019:
URL: https://github.com/apache/datafusion-comet/issues/6019
### Describe the bug
`IN` / `InSet` floating-point normalization added in #5472 applies only to
top-level
`FLOAT` and `DOUBLE` operands. `ARRAY<DOUBLE>` and structs containing
floating-point
leaves fall through unnormalized.
Spark evaluates `IN` with `TypeUtils.getInterpretedOrdering`, which treats
`-0.0` and
`0.0` as equal at floating-point leaves. The native nested membership
comparator can
distinguish those bit patterns, so Comet can silently return `false` where
Spark returns
`true`.
### Steps to reproduce
Use a Parquet table so the signed-zero bits survive input materialization:
```sql
CREATE TABLE comet_nested_in_zero (
id INT,
a ARRAY<DOUBLE>,
s STRUCT<v: DOUBLE>
) USING parquet;
INSERT INTO comet_nested_in_zero VALUES (
1,
array(CAST('-0.0' AS DOUBLE)),
named_struct('v', CAST('-0.0' AS DOUBLE))
);
SELECT
id,
a IN (array(CAST('0.0' AS DOUBLE))) AS array_match,
s IN (named_struct('v', CAST('0.0' AS DOUBLE))) AS struct_match
FROM comet_nested_in_zero;
```
Compare with `spark.comet.enabled=false`, and require the native membership
path in the
Comet run so fallback cannot hide the mismatch. Repeat for `FLOAT`, `InSet`,
and a
column candidate such as `a IN (b)`.
### Expected behavior
Nested floating-point leaves use Spark-compatible membership equality:
`-0.0` and `0.0`
match. NaN representations should also be checked and either normalized
consistently or
routed through Spark fallback.
### Additional context
- #5472 fixes scalar `FLOAT` / `DOUBLE` comparison and membership paths, but
deliberately
does not recurse into nested types.
- #5507 tracks nested `ORDER BY` and window-rank semantics, not `IN`.
- #5191 covered `arrays_overlap` / `array_position` and is now closed.
- This is a pre-existing limitation, not a regression introduced by #5472.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
