[
https://issues.apache.org/jira/browse/SPARK-59000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated SPARK-59000:
-----------------------------------
Summary: Use java.util.Arrays.compareUnsigned for BinaryType comparison
(was: Use java.util.Arrays.compareUnsigned for BinaryType compariso)
> Use java.util.Arrays.compareUnsigned for BinaryType comparison
> --------------------------------------------------------------
>
> Key: SPARK-59000
> URL: https://issues.apache.org/jira/browse/SPARK-59000
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.1.0
> Reporter: David Mollitor
> Priority: Minor
> Labels: pull-request-available
>
> Migrate the production {{BinaryType}} comparison call sites off the
> hand-rolled
> word-at-a-time {{ByteArray.compareBinary(byte[], byte[])}} onto the
> {{java.util.Arrays.compareUnsigned(byte[], byte[])}} intrinsic, which the JIT
> compiles to vectorized (SIMD) code on JDK 9+:
> * *{{CodegenContext.genComp}}* ({{{}case BinaryType{}}}) - the generated
> row/ordering comparators used by {{GenerateOrdering}} (ORDER BY, sort-merge
> join
> keys, array/map element comparison).
> * *{{PhysicalBinaryType.ordering}}* - the interpreted
> {{{}Ordering[Array[Byte]]{}}}.
> This mirrors the sibling {{{}genComp{}}}/{{{}genEqual{}}} path, which already
> emits
> {{java.util.Arrays.equals}} for {{{}BinaryType{}}}, so it extends an existing
> pattern.
> *Semantics are unchanged.* The previous implementation compared bytes as
> unsigned
> ({{{}& 0xFF{}}}) and returned the length difference on a common prefix -
> exactly
> {{{}Arrays.compareUnsigned{}}}'s contract. The returned magnitude on a
> mismatch may differ,
> but the sign is identical, and all callers are {{Ordering}} that only use the
> sign.
> The now-unused 2-arg {{ByteArray.compareBinary(byte[], byte[])}} overload is
> retained as a benchmark baseline, with a JavaDoc pointing callers to
> {{{}Arrays.compareUnsigned{}}}; the off-heap overload (base object + offset)
> is unchanged and still used by {{UTF8String}} / {{{}BinaryView{}}}.
> {{ByteArrayBenchmark}} gains an old-vs-intrinsic A/B case per size bucket.
> Local
> measurements (OpenJDK 17 / 21) show the intrinsic wins for all input sizes
> *>= 8 bytes*
> (the realistic {{BinaryType}} range: hashes, UUIDs, digests, blobs), with the
> largest gains
> in the 8-32 byte range; sub-8-byte arrays regress slightly (~13-18%) since
> there is no
> full word to vectorize over.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
