This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/main by this push:
new 8c230e6499 Add filter benchmarks for predicates that select no null
rows (#11256)
8c230e6499 is described below
commit 8c230e6499a7d6405457006f1dcb85ee28b07339
Author: Bharadwaj Pendyala <[email protected]>
AuthorDate: Wed Sep 30 18:55:13 2026 +0000
Add filter benchmarks for predicates that select no null rows (#11256)
# Which issue does this PR close?
- Part of #11200.
# Rationale for this change
#11200 is about `FilterPredicate::filter_nulls` building and then
throwing away a validity bitmap when the filter keeps no null rows. None
of the existing `filter_kernels` cases hit that: every nullable input is
50% random nulls filtered by a mask that ignores them, so the output
always has nulls.
CONTRIBUTING asks for new benchmarks in their own PR so the bench runner
can compare against them, so these come ahead of the kernel change.
# What changes are included in this PR?
Five cases in `arrow/benches/filter_kernels.rs`, next to the existing
`i32 w NULLs` ones:
- `filter context i32 w NULLs, only valid` at kept 1/4, 1023/2048 and
1/2048. Each mask is the existing 1/2, dense or sparse mask ANDed with
`is_not_null` of the data, so it never selects a null.
- `filter context i32 w NULLs at end` at kept 1/2 and 1023/1024, over an
array whose only nulls are the last 64 rows. These are the adverse case
the issue asks for: any check for a selected null has to scan to the
last word before it finds one.
# Are these changes tested?
Benchmarks only. `cargo clippy -p arrow --bench filter_kernels
--features test_utils -- -D warnings` is clean and all five run.
Baseline on main at `fa337f8`, M1 laptop, criterion median:
```
i32 w NULLs, only valid (kept 1/4) ~120 µs
i32 w NULLs, only valid high selectivity (kept 1023/2048) ~221 µs
i32 w NULLs, only valid low selectivity (kept 1/2048) ~1.10 µs
i32 w NULLs at end (kept 1/2) ~221 µs
i32 w NULLs at end high selectivity (kept 1023/1024) ~42 µs
```
This machine is noisy: the same binary moved between 73 µs and 135 µs on
the first case across runs, so read these as rough.
# Are there any user-facing changes?
No.
# AI usage
Claude wrote the benchmarks and Codex reviewed them adversarially; its
first pass is what asked for the late-null cases. I ran every number
above myself.
---
arrow/benches/filter_kernels.rs | 35 +++++++++++++++++++++++++++++++++++
1 file changed, 35 insertions(+)
diff --git a/arrow/benches/filter_kernels.rs b/arrow/benches/filter_kernels.rs
index e8b2495549..8d7a973605 100644
--- a/arrow/benches/filter_kernels.rs
+++ b/arrow/benches/filter_kernels.rs
@@ -22,6 +22,7 @@ use arrow::util::bench_util::*;
use arrow::array::*;
use arrow::compute::filter;
+use arrow::compute::{and, is_not_null};
use arrow::datatypes::{Field, Float32Type, Int32Type, Int64Type, Schema,
UInt8Type};
use arrow_array::types::Decimal128Type;
@@ -115,6 +116,40 @@ fn add_benchmark(c: &mut Criterion) {
|b| b.iter(|| bench_built_filter(&data_array, &sparse_filter)),
);
+ let valid = is_not_null(&data_array).unwrap();
+ let valid_filter = FilterBuilder::new(&and(&filter_array, &valid).unwrap())
+ .optimize()
+ .build();
+ let valid_dense_filter = FilterBuilder::new(&and(&dense_filter_array,
&valid).unwrap())
+ .optimize()
+ .build();
+ let valid_sparse_filter = FilterBuilder::new(&and(&sparse_filter_array,
&valid).unwrap())
+ .optimize()
+ .build();
+ c.bench_function("filter context i32 w NULLs, only valid (kept 1/4)", |b| {
+ b.iter(|| bench_built_filter(&data_array, &valid_filter))
+ });
+ c.bench_function(
+ "filter context i32 w NULLs, only valid high selectivity (kept
1023/2048)",
+ |b| b.iter(|| bench_built_filter(&data_array, &valid_dense_filter)),
+ );
+ c.bench_function(
+ "filter context i32 w NULLs, only valid low selectivity (kept 1/2048)",
+ |b| b.iter(|| bench_built_filter(&data_array, &valid_sparse_filter)),
+ );
+
+ // Nulls only in the last word, so the filters below find a selected null
late
+ let data_array: Int32Array = (0..size)
+ .map(|i| (i < size - 64).then_some(i as i32))
+ .collect();
+ c.bench_function("filter context i32 w NULLs at end (kept 1/2)", |b| {
+ b.iter(|| bench_built_filter(&data_array, &filter))
+ });
+ c.bench_function(
+ "filter context i32 w NULLs at end high selectivity (kept 1023/1024)",
+ |b| b.iter(|| bench_built_filter(&data_array, &dense_filter)),
+ );
+
let data_array = create_primitive_array::<UInt8Type>(size, 0.5);
c.bench_function("filter context u8 w NULLs (kept 1/2)", |b| {
b.iter(|| bench_built_filter(&data_array, &filter))
