jackylee-ch opened a new pull request, #23721: URL: https://github.com/apache/datafusion/pull/23721
## Which issue does this PR close? - Closes #23674. ## Rationale for this change Follow-on from review feedback on #23473. `hex_encode_bytes` in `datafusion/spark/src/function/math/hex.rs` rebuilt the output `NullBuffer` row by row via `NullBufferBuilder`, even though hex encoding maps rows 1:1 (a null in maps to a null out), so the output nulls are always identical to the input's. This PR applies both optimizations suggested in the issue: 1. **Reuse the input `NullBuffer`.** The input array's null buffer is now cloned (a cheap `Arc`-based clone) and attached to the output array directly, removing the per-row `append_null`/`append_non_null` bookkeeping. 2. **Special-case the no-nulls path.** When the input has no null buffer, a tight loop encodes every value without the per-row `Option` check. ## What changes are included in this PR? - `hex_encode_bytes` now takes the input array's `Option<NullBuffer>` and reuses it for the output array instead of rebuilding it with `NullBufferBuilder`; all call sites (top-level arrays and dictionary values) pass `array.nulls().cloned()`. - The encoding loop is split into a no-nulls fast path and a has-nulls path. Benchmark results from the existing `datafusion/spark/benches/hex.rs` (null_density = 0.1) against `main` on an M-series MacBook: ``` hex_utf8/size=1024 time: [17.466 µs 17.507 µs 17.552 µs] change: −4.98% hex_utf8/size=4096 time: [73.478 µs 73.592 µs 73.727 µs] change: −7.16% hex_utf8/size=8192 time: [147.63 µs 148.19 µs 148.82 µs] change: −6.97% hex_binary/size=1024 time: [17.583 µs 17.613 µs 17.650 µs] change: −5.48% hex_binary/size=4096 time: [74.234 µs 74.326 µs 74.435 µs] change: −7.38% hex_binary/size=8192 time: [148.91 µs 149.04 µs 149.19 µs] change: −7.19% ``` ## Are these changes tested? Yes. Existing unit tests cover the string/binary/dictionary paths, and a new test `test_spark_hex_utf8_with_nulls` exercises the has-nulls path with interleaved nulls and empty strings to verify the reused null buffer stays aligned with the encoded values. ## Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
