rahulsmahadev opened a new pull request, #4350: URL: https://github.com/apache/iceberg-python/pull/4350
# Rationale for this change The manifest and metrics evaluators apply `rewrite_not`, so `not value < 6.0` decides which files are read as `value >= 6.0`. The rows of those files were then filtered with the original expression, which Arrow evaluates as `~(value < 6.0)`. The two disagree on NaN: `value >= 6.0` is false for NaN, `~(value < 6.0)` is true. A NaN row was therefore returned only when another value in its file kept the file from being skipped: ```python # file 1: value [NaN, 7.0] (ids 1, 2); file 2: value [NaN, 5.0] (ids 3, 4) tbl.scan(row_filter="not value < 6.0").to_arrow() # ids [1, 2]: the NaN of file 2 is skipped with its file tbl.scan(row_filter="value >= 6.0").to_arrow() # ids [2] ``` This change applies `rewrite_not` to the bound file filter in `_task_to_record_batches` too, so rows are filtered with the same predicates that decided which files to read, and NOT no longer reaches Arrow. Only negated range comparisons on NaN change: the other negations (`=`, `IN`, `IS NULL`, `IS NaN`, `STARTS WITH`) give the same Arrow result before and after the rewrite. Follow-up from the review of #4346, as suggested by @Fokko. ## Are these changes tested? Yes. `test_scan_negated_comparison_matches_file_pruning_for_nan` in `tests/catalog/test_catalog_behaviors.py` scans a file that is read and a file that is skipped, both holding a NaN row, with `not value < 6.0`, `not (value < 6.0 and id = 1)` and their rewritten forms, on each catalog of the `catalog` fixture. It fails without this change. ## Are there any user-facing changes? Yes: a scan whose filter negates a range comparison on a float or double column no longer returns NaN rows, which matches the file-level filtering. No API changes. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
