This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion.git
The following commit(s) were added to refs/heads/main by this push:
new 2fbc758263 perf: avoid intermediate slice allocation in Spark slice
function (#23481)
2fbc758263 is described below
commit 2fbc7582639ee33eeffe83b8b43c354bf05fdd18
Author: Andy Grove <[email protected]>
AuthorDate: Sat Jul 11 11:14:28 2026 -0600
perf: avoid intermediate slice allocation in Spark slice function (#23481)
## Which issue does this PR close?
- N/A (small performance improvement)
## Rationale for this change
The Spark `slice` function's `calculate_start_end` helper reads the
length of each list element in a per-row hot loop via
`values.value(row).len()`. `GenericListArray::value(row)` materializes a
new `ArrayRef` for the sublist on every iteration purely to call
`.len()` on it. The length is already available from the offset buffer,
so this allocation is pure overhead.
Replacing it with `values.value_length(row)` reads the length directly
from the offsets with no allocation.
Benchmarked with the existing `datafusion/spark/benches/slice.rs` over
1M rows:
| case | before | after | improvement |
|------|--------|-------|-------------|
| List(Int64), array args | 54.0 ms | 40.0 ms | ~26% faster |
| List(Int64), scalar args | 88.4 ms | 69.8 ms | ~21% faster |
## What changes are included in this PR?
A single-line change in `datafusion/spark/src/function/array/slice.rs`
replacing `values.value(row).len() as i64` with
`values.value_length(row) as i64`.
## Are these changes tested?
Covered by existing unit tests in
`datafusion/spark/src/function/array/slice.rs` and slt coverage for the
Spark `slice` function; behavior is unchanged. The performance impact
was measured with the existing `slice` criterion benchmark.
## Are there any user-facing changes?
No.
---
datafusion/spark/src/function/array/slice.rs | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/datafusion/spark/src/function/array/slice.rs
b/datafusion/spark/src/function/array/slice.rs
index 5c65f899a0..f471565c40 100644
--- a/datafusion/spark/src/function/array/slice.rs
+++ b/datafusion/spark/src/function/array/slice.rs
@@ -157,7 +157,7 @@ fn calculate_start_end(args: &[ArrayRef]) ->
Result<(ArrayRef, ArrayRef)> {
}
let start = start.value(row);
let length = length.value(row);
- let value_length = values.value(row).len() as i64;
+ let value_length = values.value_length(row) as i64;
if start == 0 {
return exec_err!("Start index must not be zero");
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
