andygrove opened a new issue, #6138:
URL: https://github.com/apache/datafusion-comet/issues/6138
### Describe the bug
The native Iceberg writer puts rows with `-0.0` and `0.0` in a float/double
identity partition column into the same partition. iceberg-java keeps them
apart.
iceberg-rust holds float partition values as `OrderedFloat`, whose `Eq` and
`Hash` treat `-0.0` and `0.0` as equal. Both places the native writer groups
rows by partition compare those values:
- the fanout path keys its per-partition state on the partition struct
(`native/core/src/execution/operators/iceberg_write.rs:643-645`), and
iceberg-rust's `FanoutWriter` keys `partition_writers: HashMap<Struct, _>` the
same way
- the clustered path splits a batch into runs with `*current == value`
(`iceberg_write.rs:937`)
So whichever of the two values arrives first names the partition, and the
other value's rows are written into it. iceberg-java compares partition keys
with `Float.compare` / `Double.compare`, which orders `-0.0` before `0.0`, so
it writes two partitions.
This was found by reading the code (iceberg-rust pin `665c64e`,
ordered-float 4.6), not by a reproduction.
### Steps to reproduce
```sql
CREATE TABLE t (id INT, f DOUBLE) USING iceberg PARTITIONED BY (f);
-- with spark.comet.iceberg.write.enabled=true and the split operator on,
-- in a single task:
INSERT INTO t VALUES (1, CAST('-0.0' AS DOUBLE)), (2, 0.0D);
SELECT partition, record_count FROM t.partitions;
SELECT * FROM t WHERE f = 0.0D;
```
### Expected behavior
Two partitions, `f=-0.0` and `f=0.0`, as iceberg-java writes. With the
native writer, one data file holds both rows under a single partition value, so
a filter that prunes on the other value's partition can drop rows.
Possible fixes: group rows by the float's bit pattern rather than by
`OrderedFloat` equality, or decline native writes for float/double identity
partitions until iceberg-rust distinguishes signed zeros.
### Additional context
Found in an audit of the native Iceberg write path before enabling it by
default. Part of #5649.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
