dossett opened a new pull request, #3664: URL: https://github.com/apache/iceberg-python/pull/3664
Partition pruning currently binds the same projected expression for every data file because `_ExpressionEvaluator` stores the current record as mutable instance state. That repeated preparation is expensive, but sharing the existing evaluator across workers would allow concurrent calls to mix records. This separates preparation from per-record evaluation. `_ExpressionEvaluator` binds the expression once, while every call creates a private `_ExpressionEvaluationVisitor` containing that call's record. Manifest planning can therefore prepare one evaluator per partition spec and safely share it across manifests and workers. This provides the construction-reuse benefit targeted by #3656 without depending on files within a manifest remaining sequential, and it also benefits one-file manifests. ## Summary - Split the prepared expression evaluator from the call-local mutable visitor. - Reuse one prepared partition evaluator per partition spec during manifest planning. - Add deterministic concurrent-use, prepared-state, and planner-sharing coverage. - Add a benchmark using a single equality predicate across dense and one-file manifests. ## Testing - `pytest tests/expressions/test_visitors.py tests/table/test_partition_evaluator_planning.py tests/table/test_init.py` (200 passed) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
