dossett opened a new pull request, #3665: URL: https://github.com/apache/iceberg-python/pull/3665
Metrics filtering currently combines expensive, input-independent expression binding with mutable per-file metrics state. `main` constructs a new `_InclusiveMetricsEvaluator` for every file because concurrent calls on one instance could mix bounds and counts from different files. #3655 avoids most of that setup within a manifest, but its safety and performance depend on manifest entries remaining sequential and it cannot reuse preparation across one-file manifests. This separates preparation from evaluation. The inclusive and strict evaluators retain only the bound expression and options, while each call creates a private visitor containing that file's metrics. A single prepared inclusive evaluator can therefore be shared across every manifest task in one planning call. Strict evaluation uses the same pattern so snapshot updates and other consumers do not retain the same latent concurrency hazard. ## Summary - Split inclusive and strict metrics evaluation into prepared facades and call-local visitors. - Share one prepared inclusive evaluator across all manifest tasks in a plan. - Remove the unused `schema.as_struct()` construction from inclusive evaluation. - Add deterministic concurrent-use, prepared-state, record-count, planner-sharing, and downstream-consumer coverage. - Add a realistic single-equality benchmark covering dense, fragmented, and fully pruned manifests. ## Performance I compared `main`, #3655, and this branch using 1,000 files, a 102-column schema, and a single `x = 5` predicate. The serial manifest executor isolates planning work from thread scheduling. Values are median times. | Manifest layout | `main` | #3655 | This PR | |---|---:|---:|---:| | 1 manifest × 1,000 files | 48.690 ms | 1.519 ms | 1.558 ms | | 1,000 manifests × 1 file | 50.093 ms | 50.150 ms | 1.933 ms | The dense case remains within 3% of #3655 and is 31× faster than `main`. With one file per manifest, sharing across manifests makes this branch about 26× faster than both alternatives. Eager preparation adds approximately 5 microseconds when one manifest has every file rejected before metrics evaluation (`0.024 ms` on `main` versus `0.029 ms` here). With 1,000 fully pruned manifests, this branch is faster because it avoids constructing 1,000 per-manifest callables (`0.406 ms` versus `0.650 ms` on `main`). ## Testing - `pytest tests/expressions/test_evaluator.py tests/table/test_metrics_evaluator_planning.py tests/table/test_init.py tests/table/test_delete_file_index.py tests/table/test_snapshots.py tests/table/test_validate.py` (241 passed) - `pytest tests/benchmark/test_metrics_evaluator_benchmark.py -m benchmark` (4 passed) - `UV_NO_CONFIG=1 .venv/bin/prek run -a` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
