This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-23481-a29c58f105cbdfd98fcb1996386e3d3fee7189a2 in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit 2fbc7582639ee33eeffe83b8b43c354bf05fdd18 Author: Andy Grove <[email protected]> AuthorDate: Sat Jul 11 11:14:28 2026 -0600 perf: avoid intermediate slice allocation in Spark slice function (#23481) ## Which issue does this PR close? - N/A (small performance improvement) ## Rationale for this change The Spark `slice` function's `calculate_start_end` helper reads the length of each list element in a per-row hot loop via `values.value(row).len()`. `GenericListArray::value(row)` materializes a new `ArrayRef` for the sublist on every iteration purely to call `.len()` on it. The length is already available from the offset buffer, so this allocation is pure overhead. Replacing it with `values.value_length(row)` reads the length directly from the offsets with no allocation. Benchmarked with the existing `datafusion/spark/benches/slice.rs` over 1M rows: | case | before | after | improvement | |------|--------|-------|-------------| | List(Int64), array args | 54.0 ms | 40.0 ms | ~26% faster | | List(Int64), scalar args | 88.4 ms | 69.8 ms | ~21% faster | ## What changes are included in this PR? A single-line change in `datafusion/spark/src/function/array/slice.rs` replacing `values.value(row).len() as i64` with `values.value_length(row) as i64`. ## Are these changes tested? Covered by existing unit tests in `datafusion/spark/src/function/array/slice.rs` and slt coverage for the Spark `slice` function; behavior is unchanged. The performance impact was measured with the existing `slice` criterion benchmark. ## Are there any user-facing changes? No. --- datafusion/spark/src/function/array/slice.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/datafusion/spark/src/function/array/slice.rs b/datafusion/spark/src/function/array/slice.rs index 5c65f899a0..f471565c40 100644 --- a/datafusion/spark/src/function/array/slice.rs +++ b/datafusion/spark/src/function/array/slice.rs @@ -157,7 +157,7 @@ fn calculate_start_end(args: &[ArrayRef]) -> Result<(ArrayRef, ArrayRef)> { } let start = start.value(row); let length = length.value(row); - let value_length = values.value(row).len() as i64; + let value_length = values.value_length(row) as i64; if start == 0 { return exec_err!("Start index must not be zero"); --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
