GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/10974
[SPARK-13072][SQL] simplify and improve murmur3 hash expression codegen
simplify(remove several unnecessary local variables) the generated code of
hash expression, and avoid null check if possible.
generated code comparison for `hash(int, double, string, array<string>)`:
**before:**
```
public UnsafeRow apply(InternalRow i) {
/* hash(input[0, int],input[1, double],input[2, string],input[3,
array<int>],42) */
int value1 = 42;
/* input[0, int] */
int value3 = i.getInt(0);
if (!false) {
value1 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashInt(value3,
value1);
}
/* input[1, double] */
double value5 = i.getDouble(1);
if (!false) {
value1 =
org.apache.spark.unsafe.hash.Murmur3_x86_32.hashLong(Double.doubleToLongBits(value5),
value1);
}
/* input[2, string] */
boolean isNull6 = i.isNullAt(2);
UTF8String value7 = isNull6 ? null : (i.getUTF8String(2));
if (!isNull6) {
value1 =
org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(value7.getBaseObject(),
value7.getBaseOffset(), value7.numBytes(), value1);
}
/* input[3, array<int>] */
boolean isNull8 = i.isNullAt(3);
ArrayData value9 = isNull8 ? null : (i.getArray(3));
if (!isNull8) {
int result10 = value1;
for (int index11 = 0; index11 < value9.numElements(); index11++) {
if (!value9.isNullAt(index11)) {
final int element12 = value9.getInt(index11);
result10 =
org.apache.spark.unsafe.hash.Murmur3_x86_32.hashInt(element12, result10);
}
}
value1 = result10;
}
}
```
**after:**
```
public UnsafeRow apply(InternalRow i) {
/* hash(input[0, int],input[1, double],input[2, string],input[3,
array<int>],42) */
int value1 = 42;
/* input[0, int] */
int value3 = i.getInt(0);
value1 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashInt(value3,
value1);
/* input[1, double] */
double value5 = i.getDouble(1);
value1 =
org.apache.spark.unsafe.hash.Murmur3_x86_32.hashLong(Double.doubleToLongBits(value5),
value1);
/* input[2, string] */
boolean isNull6 = i.isNullAt(2);
UTF8String value7 = isNull6 ? null : (i.getUTF8String(2));
if (!isNull6) {
value1 =
org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(value7.getBaseObject(),
value7.getBaseOffset(), value7.numBytes(), value1);
}
/* input[3, array<int>] */
boolean isNull8 = i.isNullAt(3);
ArrayData value9 = isNull8 ? null : (i.getArray(3));
if (!isNull8) {
for (int index10 = 0; index10 < value9.numElements(); index10++) {
final int element11 = value9.getInt(index10);
value1 =
org.apache.spark.unsafe.hash.Murmur3_x86_32.hashInt(element11, value1);
}
}
rowWriter14.write(0, value1);
return result12;
}
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark codegen
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10974.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10974
----
commit f6a3cccab507c66f29f12132c9656bb409263a13
Author: Wenchen Fan <[email protected]>
Date: 2016-01-29T02:13:25Z
simplify and improve murmur3 hash expression codegen
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
