rdblue opened a new pull request, #18147: URL: https://github.com/apache/iceberg/pull/18147
This implements stats filtering in `V4ManifestReader` using an `InclusiveStatsFilter`. This required updating `CloseableIterable.filter` to be able to pass a callback for updating metrics based on a skipped item. A reasonable follow up would be to rewrite the partition filtering to use this, but #18108 is changing partition handling so I thought it would be better to wait and avoid conflicts. Test plan: * `InclusiveStatsFilter` is thoroughly tested, so it should not be exhaustively tested here * Ensure that the inputs to `InclusiveStatsFilter.eval` are passed correctly (content stats and record count) * Test content stats using bounds filter * Test record count filters files with 0 records * Test record count filter does not filter files with -1 records (old Avro bug) * Test filtering data files and manifest files * Test that files with null stats are not filtered * Test filtering with `forScanPlanning` that projects no stats by default * Test filtering with `project` and `select` that do not request stats columns * Test filter binding correctly applies case sensitivity setting * Test file skipping updates scan metrics I also added a test for bucket partition skipping, which will eventually move to stats-based skipping. This currently passes based on partition filtering. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
