andygrove opened a new pull request, #4877: URL: https://github.com/apache/datafusion-comet/pull/4877
> ⚠️ **Autonomously generated by an LLM.** This PR was created end-to-end by an automated agent with no human author. It has passed automated correctness (unit tests + differential fuzz) and micro-benchmark gates, but **requires full human review**. Do not merge without expert scrutiny. ## What changed Optimizes `spark_size` in the native `spark-expr` crate. Compute list row sizes from the offset buffer instead of allocating a sliced ArrayRef per row via list_array.value(i), removing one heap allocation per row. ## Evidence - Correctness: unit tests + seeded differential fuzz (bit-identical Arrow output vs `main`). - Benchmark (criterion, Rust-only): see numbers below; gate required ≥ 5.0% mean improvement with non-overlapping confidence intervals and no regressions. ``` spark_size_ list of short arrays: 93.333% faster (base 151223ns -> cand 10082ns) spark_unhex_ valid mixed hex: 19.964% faster (base 125189ns -> cand 100197ns) spark_unhex_ mixed hex column: 13.986% faster (base 109637ns -> cand 94303ns) spark_size_ list of long arrays: 94.086% faster (base 156323ns -> cand 9245ns) spark_unhex_ all valid: 42.128% faster (base 201069ns -> cand 116362ns) spark_unhex_ with nulls: 36.465% faster (base 179663ns -> cand 114150ns) spark_unhex_ valid hex: 22.177% faster (base 152680ns -> cand 118821ns) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
