[
https://issues.apache.org/jira/browse/HIVE-12348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Teddy Choi updated HIVE-12348:
------------------------------
Attachment: HIVE-12348.patch
I compared FastByteComparisons#UnsafeComparer and Java 8's String#compare,
Array#equals. However, String.compare requires a char array. It needs a
conversion from a byte array to a char array, which takes enough time to cancel
performance improvement. So I decided to use FastByteComparison#UnsafeComparer.
This patch brings 2.3x performance in StringColEqualStringColumnBench and 2.8x
performance in StringColGreaterEqualStringColumnBench.
There also are several dozens of classes that use StringExpr#equal, including
VectorMapJoinInnerBigOnlyStringOperator, CuckooSetBytes, VectorHashKeyWrapper.
These classes will get performance benefit also.
> Byte array comparison optimization for SIMD
> -------------------------------------------
>
> Key: HIVE-12348
> URL: https://issues.apache.org/jira/browse/HIVE-12348
> Project: Hive
> Issue Type: Sub-task
> Reporter: Teddy Choi
> Assignee: Teddy Choi
> Priority: Minor
> Attachments: HIVE-12348.patch
>
>
> The current byte comparison implementation in Hive is basic. It handles a
> byte (1 byte) at a time, so it's slow.
> There's FastByteComparisons class in Hadoop with sun.misc.Unsafe class.
> (https://github.com/hanborq/hadoop/blob/master/src/core/org/apache/hadoop/io/FastByteComparisons.java)
> It handles a long integer (8 bytes) at a time, so it's faster.
> Java 8 has String.compare, String.equalTo intrinsics with AVX2 and SSE4.2.
> (http://hg.openjdk.java.net/jdk8/jdk8/hotspot/rev/038dd2875b94) It handles
> 128~256 bits (16~32 bytes) at a time, so it's much faster.
> However, Unsafe.getLong and String.compare intrinsic needs additional data
> copies, so the actual performance increase is smaller than "1 byte : 32
> bytes" comparison.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
