namanjain24-sudo commented on PR #24804: URL: https://github.com/apache/datafusion/pull/24804#issuecomment-6641469401
I tested this branch using a file containing one finite value and one NaN. The column chunk has `min == max == 1.0` and `null_count == 0`. I also compared the results with the commit immediately before this one. | Query | Before | This PR | | --- | --- | --- | | `SELECT x` | `1.0, 1.0` | `1.0, NaN` | | `WHERE x > 2.0` | 0 rows | `NaN` | | `WHERE x = 1.0` | `1.0, 1.0` | `1.0` | When I ran the same three queries with `collect_statistics`, `parquet.pruning`, and `enable_page_index` all disabled, the results matched the right-hand column exactly. In other words, with this PR the default path returns the same rows as a scan that reads no statistics at all. There are two additional points worth mentioning in the PR description. First, the first and third rows are related to #26490, which this PR fixes but does not currently mention. There is also a downstream report in delta-io/delta-rs#5007, where `delete` and `update` write the substituted value back into the table instead of simply returning it. Second, `enable_page_index` can independently prune the second case. This means that setting `parquet.pruning = false` alone is not sufficient as a workaround. One caveat: all these tests were run against this branch's current base rather than after rebasing. The branch is 987 commits behind `main` and has conflicts in `row_group_pruning.rs`, `metadata.rs`, and `parquet_statistics.slt`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
