etseidl commented on issue #15742: URL: https://github.com/apache/datafusion/issues/15742#issuecomment-2815595171
BTW, this issue is somewhat tied up with https://github.com/apache/parquet-format/pull/221. Take for example ```sql > select * from 'parquet-testing/data/float16_nonzeros_and_nans.parquet' where x > 2.0; +-----+ | x | +-----+ | NaN | +-----+ 1 row(s) fetched. ``` This return a single `NaN` row because the lack of pruning will not filter out pages/column chunks. Rewriting the query to avoid the LHS cast: ```sql > select * from 'parquet-testing/data/float16_nonzeros_and_nans.parquet' where x > arrow_cast(2.0, 'Float16'); +---+ | x | +---+ +---+ 0 row(s) fetched. ``` Now filters on statistics that have `max == 2.0`. Changing `verify_support_type_for_prune` to support floating point types will now also return 0 rows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
