0lai0 opened a new issue, #5272: URL: https://github.com/apache/datafusion-comet/issues/5272
### What is the problem the feature request solves? Follow-up from #5233 `spark_size` for `LargeList` reuses Arrow's `length` kernel, which returns `Int64`, then `cast_with_options(..., Int32, safe: false)`, then (when nulls are present) `to_vec()` to patch null slots to `-1`. That is three allocations on the hot path. On the `array_size` bench after #5233, `LargeList (10% null)` is only ~1.3x faster than `main` (7.54 µs vs 9.70 µs), while no-null `List` is ~12x. The gap is the Int64 → cast → copy chain, not the kernel idea itself. ### Describe the potential solution Build an `Int32` length array straight from the `LargeList` i64 offset buffer (`offsets[i+1] - offsets[i]`, with a checked conversion that errors on overflow rather than wrapping or becoming `-1`), then apply the same null → `-1` rewrite as the List path (`null_count == 0` fast path; otherwise `into_parts` + `set_indices`). Avoid allocating the intermediate Int64 `length` output and the Int32 cast result when those are only used to derive i32 sizes ### Additional context - Bench coverage already exists: `spark_size: LargeList (10% null)` in `benches/array_size.rs`. Consider adding a no-null LargeList shape for parity with the List production path. - Keep `safe: false` / checked overflow semantics: a length that does not fit in `i32` must error, not become null then `-1` - Called out in review on #5233; left out of that PR so Map (#5266) and this path can land separately without blocking the List win -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
