dshogan-amzn opened a new issue, #19014: URL: https://github.com/apache/iceberg/issues/19014
### Feature Request / Improvement `InclusiveMetricsEvaluator`, `ManifestEvaluator` and `ParquetMetricsRowGroupFilter` each stop evaluating an `IN` predicate once it has more than 200 literals and treat the file, manifest or row group as a possible match. The limit was added in #1588 because each check compared every literal against the bounds, so a large list cost O(literals) per file during planning. For a table sorted or clustered by the `IN` column, this means a query with 201 values keeps every file in its window, while the same query with 200 values keeps only the files whose bounds contain a value. For example, on a manifest of 10,000 files with disjoint key ranges, an `IN` with 200 spread-out values selects 200 files and an `IN` with 201 values selects all 10,000. Sorting the literals once per predicate and binary-searching each file's `[lower, upper]` makes the check O(log literals) per file, which removes the planning-cost reason for the limit. ### Query engine None ### Willingness to contribute - [x] I can contribute this improvement/feature independently - [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community - [ ] I cannot contribute this improvement/feature at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
