neilconway opened a new pull request, #25079:
URL: https://github.com/apache/datafusion/pull/25079
## Which issue does this PR close?
- Closes #25078
## Rationale for this change
`cardinality` was implemented by iterating over the `values` of the input
`GenericListArray`, creating a slice for each logical row, and then fetching
the length of the slice. This incurs a lot of unnecessary per-row overhead; we
can instead determine the cardinality of each list by looking at the
`OffsetBuffer`. This is roughly 100x-500x faster.
For non-nested lists, this is straightforward. It would be possible to use a
similar approach to compute the cardinality of nested lists, but for now this
PR falls back to the previous approach for that case.
Benchmarks: (M4 Max)
list/valid/8192x32, 203.951 µs → 0.984 µs, -99.52%
large_list/nullable/8192x32, 172.699 µs → 1.049 µs, -99.39%
map/valid/8192x32, 471.074 µs → 0.958 µs, -99.80%
## What changes are included in this PR?
* Add optimized implementation of `cardinality` for non-nested lists/maps
* Add unit tests
* Add benchmark
## What is the testing strategy for this PR?
Existing tests pass; new tests added.
## Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
