mhatipog commented on issue #18723:
URL: https://github.com/apache/iceberg/issues/18723#issuecomment-6916031458

   End-to-end confirmation on Spark 4.0.1 with 
`iceberg-spark-runtime-4.0_2.13:1.12.0` (Hadoop catalog, default table 
properties). There's also a partitioned case where Spark returns extra rows, 
not only missing ones.
   
   ```sql
   CREATE TABLE local.db.zero (id BIGINT, d DOUBLE) USING iceberg;            
-- and a second table with PARTITIONED BY (d)
   INSERT INTO local.db.zero VALUES (1, CAST('-0.0' AS DOUBLE));
   INSERT INTO local.db.zero VALUES (2, 5.0);
   ```
   
   | query | unpartitioned Iceberg | `PARTITIONED BY (d)` Iceberg | same rows 
as a Parquet table |
   | --- | --- | --- | --- |
   | `WHERE d = 0.0` | `[]` | `[]` | `[1]` |
   | `WHERE d >= 0.0` | `[2]` | | `[1, 2]` |
   | `WHERE d IN (0.0, 7.0)` | `[]` | | `[1]` |
   | `WHERE d < 0.0` | | `[1]` | `[]` |
   | `WHERE d <> 0.0` | | `[1, 2]` | `[2]` |
   
   On the identity-partitioned table, the partition predicate is evaluated with 
total order, so `-0.0 < 0.0` holds. Spark then treats the filter as fully 
handled by the partition and does not re-check it, which is how `d < 0.0` and 
`d <> 0.0` return row 1.
   
   In a wider sweep (float and double columns, files from Spark, `add_files` 
over DuckDB and pyarrow files, and pyiceberg 0.12.0 appends), the same 
signed-zero mismatch also drops data in two other places:
   - row groups through the Parquet dictionary filter, even when the file 
bounds are `[-0.0, 0.0]` (consistent with #18887)
   - files through `add_files` metrics
   
   The mirror case fails as well: `+0.0` rows are missed by `= -0.0`, `<= 
-0.0`, `<=> -0.0` and `BETWEEN -0.0 AND -0.0`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to