avinash-git-24 commented on issue #4003: URL: https://github.com/apache/iceberg-python/issues/4003#issuecomment-5875268461
I've gone through the issue and inspected `_InclusiveMetricsEvaluationVisitor.visit_in` and `_ManifestEvalVisitor.visit_in` in `pyiceberg/expressions/visitors.py`. My current understanding is that both visitors short-circuit with `if len(literals) > IN_PREDICATE_LIMIT: return ROWS_MIGHT_MATCH` before comparing against the file/partition `lower_bound` and `upper_bound`, which completely disables manifest and data-file pruning for `IN` predicates with more than 200 values. I think this could be approached by checking whether the file's `[lower_bound, upper_bound]` is completely disjoint from `[min(literals), max(literals)]` before bailing out on `IN_PREDICATE_LIMIT` (or evaluating against the sorted literal bounds), so non-overlapping files and manifests are still pruned even for large `IN` sets. I'd be interested in implementing this and adding unit tests in `tests/expressions/test_visitors.py`. If the issue is still available, I'd be happy to take it up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
