hanke580 opened a new issue, #26491:
URL: https://github.com/apache/datafusion/issues/26491
### Describe the bug
A Parquet bloom filter stores hashes of the plain-encoded value, so `-0.0`
(`0x8000000000000000`) and `+0.0` are
different members. When DataFusion prunes row groups with the bloom filter
for `c = 0.0` (or `c = -0.0`,
`c IN (…)`), it probes only the literal's own bit pattern. A row group whose
zeros all have the other sign is answered
"absent" and skipped. SQL `=` treats `-0.0 = 0.0` as true (as does
DataFusion's own evaluation without the bloom
filter), so matching rows are silently missing from the result.
`bloom_filter_on_read` is on by default. So every file with bloom filters on
a float column is affected, including
files written by DataFusion itself (`bloom_filter_on_write`),
parquet-java/Spark and parquet-rs. pyarrow and DuckDB return
all matching rows from the same file.
The probe is in `datafusion/datasource-parquet/src/bloom_filter.rs` (lines
107–108):
`ScalarValue::Float64(Some(v)) => sbbf.check(v)` and the same for Float32.
For a zero literal, both `0.0` and `-0.0`
should be probed (a row group is kept if either may be present).
Alternatively, writers could normalise `-0.0` to
`+0.0` before hashing, but files written that way already exist.
### To Reproduce
```python
import pyarrow as pa, pyarrow.parquet as pq
pq.write_table(pa.table({'id': [1, 2, 3, 4, 5, 6],
'c': pa.array([-0.0, -0.0, 0.0, 1.0, -0.0, 2.0],
pa.float64())}), 'src.parquet')
```
```sql
SET datafusion.execution.parquet.bloom_filter_on_write = true;
SET datafusion.execution.parquet.max_row_group_size = 2;
CREATE EXTERNAL TABLE s STORED AS PARQUET LOCATION 'src.parquet';
COPY (SELECT * FROM s) TO 'bloom.parquet' STORED AS PARQUET; -- row
groups: (-0.0, -0.0), (0.0, 1.0), (-0.0, 2.0)
SELECT id FROM 'bloom.parquet' WHERE c = 0.0 ORDER BY id; -- 1, 2,
3 <-- row 5 (-0.0) missing
SELECT id FROM 'bloom.parquet' WHERE c = -0.0 ORDER BY id; -- 1, 2,
5 <-- row 3 (0.0) missing
SELECT id FROM 'bloom.parquet' WHERE c IN (0.0) ORDER BY id; -- 1, 2, 3
SET datafusion.execution.parquet.bloom_filter_on_read = false;
SELECT id FROM 'bloom.parquet' WHERE c = 0.0 ORDER BY id; -- 1, 2,
3, 5 (correct)
```
pyarrow (`pq.read_table('bloom.parquet', filters=[('c', '==', 0.0)])`) and
DuckDB return `1, 2, 3, 5`.
### Expected behavior
`WHERE c = 0.0` and `WHERE c = -0.0` both return rows 1, 2, 3 and 5, with or
without bloom filters.
### Additional context
- DataFusion main 290ad15 (2026-10-18), `datafusion-cli 55.1.0` built from
source; also 91cd7ffb (2026-10-15).
- The min/max path is fine: with `bloom_filter_on_read = false` all matching
rows come back, although the arrow-rs 60
writer (`IEEE_754_TOTAL_ORDER`) records the signed zeros as they are (row
groups: `min = max = -0.0`, `min = 0.0`,
`min = -0.0`). Only the bloom-filter probe tells the two zeros apart. A
smaller case: `COPY` of `(1, -0.0), (2, 7.0),
(3, 0.0), (4, 8.0)` with 2-row groups returns only id 3 for `c = 0.0` and
only id 1 for `c = -0.0`.
### Willingness to contribute
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
