neilconway opened a new pull request, #25121:
URL: https://github.com/apache/datafusion/pull/25121
## Which issue does this PR close?
- Closes ##25119.
## Rationale for this change
`array_length` worked by forming a slice of values for each logical row in
the given dimension, and then appending the length of the slice to the result.
That is inefficient; if computing the length of the first array dimension, we
can get that from the offset buffer instead. That avoids a lot of per-row work
and is significantly faster.
Benchmarks: (M4 Max)
- list/default/8192, 175.379 µs -> 1.137 µs, -99.35%
- list/dimension_1/8192, 173.643 µs -> 1.159 µs, -99.33%
- fixed_size_list/default/8192, 174.770 µs -> 0.963 µs, -99.45%
- list/default/1, 0.171 µs -> 0.088 µs, -48.48%
## What changes are included in this PR?
* Compute `array_length` of dimension 1 for `List` and `LargeList` from
offsets
* Compute `array_length` of dimension 1 for `FixedSizeList` from the FSL's
size
* Treat `array_length(x, 1)` as equivalent to `array_length(x)`
* Simplify the code to remove a no-longer-necessary macro
* Add unit tests
* Add benchmark for `array_length`
## What is the testing strategy for this PR?
Existing tests pass; new unit tests added.
## Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
