shinzoxD opened a new pull request, #24494: URL: https://github.com/apache/datafusion/pull/24494
## Which issue does this PR close? - Closes #15986 ## Rationale for this change Review of #15947 asked Spark `hex` to return `Utf8View` instead of `Utf8`, with a fast path for encodings that fit in the StringView inline prefix (12 bytes). The lookup table (#21836) and criterion benches (#19738) already landed. This finishes the remaining item on #15986. Short hex results are common (small integers, short strings/binaries). Building a `Utf8` `StringArray` for those rows pays for offset buffers and a data buffer even when Arrow can store the digits in the view itself. Returning `Utf8View` avoids that for encodings of at most 12 bytes and still stores longer encodings out-of-line. ## What changes are included in this PR? - `SparkHex::return_type` is now `Utf8View`, including dictionary values (`Dictionary(K, Utf8View)`). - Encodings of at most 12 hex digits are inlined via `make_view` and never touch the StringView data buffer. - Longer encodings are written once into that buffer and referenced by offset. - Input null buffers are still reused (`ptr_eq` on the inner validity buffer). - `BinaryView` inputs are handled (the signature already accepted them). - Criterion bench adds `hex_utf8_inline` for the short-string path. ## Are these changes tested? Yes. - Unit tests in `datafusion/spark/src/function/math/hex.rs` cover Utf8View / LargeUtf8 / BinaryView inputs, null reuse, the 12- vs 13-digit inline boundary, and `return_type`. - SQL logic tests in `datafusion/sqllogictest/test_files/spark/math/hex.slt` assert values plus `arrow_typeof(hex(...)) = Utf8View` (and `Dictionary(Int32, Utf8View)` for dictionary input). - `cargo test -p datafusion-spark --lib hex`: 19 passed. - `cargo clippy -p datafusion-spark --all-targets -- -D warnings`: clean. ## Are there any user-facing changes? `hex` now returns `Utf8View` instead of `Utf8`. Encoded values are unchanged. Callers that match on the physical type (for example `ScalarValue::Utf8`) need to accept `Utf8View`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
