dossett opened a new pull request, #3666: URL: https://github.com/apache/iceberg-python/pull/3666
Residual planning currently creates a mutable evaluator per data file because evaluating one partition changes evaluator state. That avoids unsafe sharing, but it repeatedly prepares equivalent evaluation state and recalculates identical residuals for files whose relevant partition values are the same. This change makes residual evaluators safe to share by keeping partition-specific state in a short-lived visitor. Scan planning then reuses one prepared evaluator per partition spec and maintains a bounded residual cache keyed by the spec and only the partition fields that can affect the filter. This reduces repeated planning work without allowing high-cardinality, unrelated partition fields to grow or defeat the cache. ## Commit structure 1. [`bca46fc3` — Make residual evaluation stateless](https://github.com/dossett/iceberg-python/commit/bca46fc3): moves partition-local state into `_ResidualEvaluationVisitor`, prepares the partition schema once, and adds state-isolation and concurrency tests. 2. [`5676096d` — Cache residuals during scan planning](https://github.com/dossett/iceberg-python/commit/5676096d): reuses one evaluator per spec and adds a bounded 128-entry cache keyed by relevant partition values, with planner tests covering spec isolation, irrelevant fields, multiple transforms, and eviction. 3. [`e9fc9107` — Benchmark residual planning](https://github.com/dossett/iceberg-python/commit/e9fc9107): adds a benchmark using a realistic 15-leaf predicate and both repeated and unique relevant partition values. ## Performance Mean of three runs planning 2,000 files with a 15-leaf predicate: | Relevant partition values | `main` | This PR | Improvement | |---|---:|---:|---:| | 7 repeated values | 1.117 s | 0.004 s | ~99.6% | | 2,000 unique values | 1.170 s | 0.456 s | ~61.0% | The unique-values case intentionally forces cache misses and exercises evaluator reuse independently of cache hits. ## Testing - `pytest tests/expressions tests/table -q` — 896 passed - `pytest tests/benchmark/test_residual_evaluator_benchmark.py -q -s -m benchmark` — 2 passed - `UV_NO_CONFIG=1 .venv/bin/prek run -a` — all hooks passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
