goutamadwant opened a new pull request, #24836:
URL: https://github.com/apache/datafusion/pull/24836
## Which issue does this PR close?
- Closes #24707.
## Rationale for this change
The string IN-list pruning benchmark only covered domain sizes 20, 21, 256,
and 1,024 with a fixed 4,096 pruning containers. This did not provide the
small-list or container-scaling measurements needed by the follow-up work in
#24709 and #24710.
This PR extends the benchmark before making any production threshold change,
following the contributor guide's recommendation to isolate benchmark additions
from the optimization they are intended to evaluate.
## What changes are included in this PR?
- Add domain sizes 1, 2, 4, 8, and 16 while retaining the existing boundary
and large-domain cases.
- Evaluate each domain at 16, 256, and 4,096 pruning containers.
- Keep construction benchmarks independent of container count.
- Preserve the existing 4,096-container Criterion benchmark IDs so
historical baselines remain comparable.
- Validate both representations produce the same pruning results for every
evaluation matrix cell outside the timed loop.
The production pruning implementation and its current threshold are
unchanged.
## What is the testing strategy for this PR?
- `./dev/rust_lint.sh`
- `cargo test --locked -p datafusion-pruning` (95 tests passed)
- Extended workspace test suite:
```shell
RUST_BACKTRACE=1 cargo test --locked --profile ci --exclude
datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli
--workspace --lib --tests --bins --features
avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption
```
- Full 72-case Criterion matrix:
```shell
cargo bench --locked --profile release-nonlto -p datafusion-pruning
--bench string_in_list_pruning -- --sample-size 10 --warm-up-time 0.1
--measurement-time 0.2 --noplot
```
## Are there any user-facing changes?
No. This PR only extends benchmark coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
