0lai0 opened a new issue, #5266:
URL: https://github.com/apache/datafusion-comet/issues/5266
### What is the problem the feature request solves?
Follow-up from #5233
`spark_size` for `List` / `LargeList` / `FixedSizeList` now reuses Arrow's
`length` kernel.
The `Map` branch in `native/spark-expr/src/array_funcs/size.rs` is still a
per-row loop that appends `MapArray::value_length(i)` into an `Int32Builder`.
`length` does not accept `MapArray`, so Map cannot reuse the kernel. It was
kept out of #5233 on purpose: different technique, and its own correctness and
benchmark surface.
### Describe the potential solution
`MapArray::offsets()` is the entry-range offset buffer; for each row,
`offsets[i+1] - offsets[i]` is the entry count (same value `value_length(i)`
returns one row at a time). Vectorize that and apply Spark's null → `-1`
rewrite with the same `set_indices` patch the list path uses
### Additional context
Before merging:
- Add a Map shape (with and without nulls) to `benches/array_size.rs` —
there is no Map coverage today. The win is expected to be smaller than for
List, since `value_length` already reads the offsets; only per-row dispatch and
builder overhead go away. If the benchmark comes back flat, closing this as not
worth the churn is fine.
- Add a sliced-`MapArray` test, mirroring the sliced-List test added in
#5233. `MapArray::offsets()` returns the sliced offsets so `windows(2)` stays
correct, but pin the invariant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
