akshat-lakhera opened a new pull request, #26978: URL: https://github.com/apache/datafusion/pull/26978
## Which issue does this PR close? - Closes #26491. ## Rationale for this change A Parquet bloom filter hashes the plain-encoded value, so 0.0 and -0.0 are different members. DataFusion probed only the literal's own bit pattern, so a row group holding only -0.0 was skipped for WHERE c = 0.0, and a row group holding only 0.0 was skipped for WHERE c = -0.0. SQL treats the two as equal, and the query returns those rows once bloom_filter_on_read is turned off, so rows were silently missing. Since bloom_filter_on_read is on by default, this affects any file with a bloom filter on a float column. ## What changes are included in this PR? In BloomFilterStatistics::check_scalar, a Float64 or Float32 zero literal now probes both +0.0 and -0.0, and the row group is kept if either may be present. Every path that reaches the probe (=, IN lists, OR of equalities) goes through this function, so one change covers them. Float16 isn't probed at all and decimals have no signed zero, so neither needed a change. ## Are these changes tested? Yes. I added unit tests in bloom_filter.rs that build files whose bloom filter holds only one sign of zero, for Float64 and Float32, and check that =, IN and OR keep the row group. I also added a sqllogictest in predicates.slt that writes with bloom_filter_on_write and max_row_group_size = 2 and checks that WHERE c = 0.0 and WHERE c = -0.0 both return all four signed-zero rows. The new tests fail without the fix. While testing I noticed that c IN (0.0) also misses -0.0 rows when it is evaluated in memory, even with bloom filters off. That looks like a separate problem in the IN list evaluation, so the sqllogictest only covers equality and I left it out of this PR. ## Are there any user-facing changes? Queries with = or IN on float columns over Parquet files with bloom filters now return rows that were wrongly pruned before. No API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
