David Mollitor created SPARK-59000:
--------------------------------------
Summary: Use java.util.Arrays.compareUnsigned for BinaryType
compariso
Key: SPARK-59000
URL: https://issues.apache.org/jira/browse/SPARK-59000
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: David Mollitor
Migrate the production {{BinaryType}} comparison call sites off the hand-rolled
word-at-a-time {{ByteArray.compareBinary(byte[], byte[])}} onto the
{{java.util.Arrays.compareUnsigned(byte[], byte[])}} intrinsic, which the JIT
compiles to vectorized (SIMD) code on JDK 9+:
* *{{CodegenContext.genComp}}* ({{{}case BinaryType{}}}) - the generated
row/ordering comparators used by {{GenerateOrdering}} (ORDER BY, sort-merge join
keys, array/map element comparison).
* *{{PhysicalBinaryType.ordering}}* - the interpreted
{{{}Ordering[Array[Byte]]{}}}.
This mirrors the sibling {{{}genComp{}}}/{{{}genEqual{}}} path, which already
emits
{{java.util.Arrays.equals}} for {{{}BinaryType{}}}, so it extends an existing
pattern.
*Semantics are unchanged.* The previous implementation compared bytes as
unsigned
({{{}& 0xFF{}}}) and returned the length difference on a common prefix - exactly
{{{}Arrays.compareUnsigned{}}}'s contract. The returned magnitude on a mismatch
may differ,
but the sign is identical, and all callers are {{Ordering}} that only use the
sign.
The now-unused 2-arg {{ByteArray.compareBinary(byte[], byte[])}} overload is
retained as a benchmark baseline, with a JavaDoc pointing callers to
{{{}Arrays.compareUnsigned{}}}; the off-heap overload (base object + offset) is
unchanged and still used by {{UTF8String}} / {{{}BinaryView{}}}.
{{ByteArrayBenchmark}} gains an old-vs-intrinsic A/B case per size bucket. Local
measurements (OpenJDK 17 / 21) show the intrinsic wins for all input sizes *>=
8 bytes*
(the realistic {{BinaryType}} range: hashes, UUIDs, digests, blobs), with the
largest gains
in the 8-32 byte range; sub-8-byte arrays regress slightly (~13-18%) since
there is no
full word to vectorize over.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
