andygrove opened a new issue, #5099: URL: https://github.com/apache/datafusion-comet/issues/5099
### What is the problem the feature request solves? `spark_size_array` / `append_list_sizes` in `native/spark-expr/src/array_funcs/size.rs` compute Spark's `size()` with a per-row loop reading `value_length(i)` into an `Int32Builder`, emitting -1 for null rows in legacy mode, for List/LargeList/FixedSizeList/Map. Arrow's `length` kernel computes list lengths vectorized from the offset buffer (List to Int32, LargeList to Int64, FixedSizeList supported), preserving nulls. ### Describe the potential solution Use `arrow::compute::kernels::length::length`, then apply the Spark legacy "-1 for null" semantics with one `zip(is_null_mask, scalar(-1), lengths)`. LargeList needs a checked Int64-to-Int32 cast (Spark arrays are capped near `Integer.MAX_VALUE` anyway). The Map branch is not covered by the length kernel and stays manual, or wraps the map's offsets in a ListArray view first. ### Additional context Found during an audit of native code that replicates existing arrow-rs kernels. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
