Repository: spark
Updated Branches:
refs/heads/branch-2.0 198b0426e -> d226dce12
[SPARK-16699][SQL] Fix performance bug in hash aggregate on long string keys
## What changes were proposed in this pull request?
In the following code in `VectorizedHashMapGenerator.scala`:
```
def hashBytes(b: String): String = {
val hash = ctx.freshName("hash")
s"""
|int $result = 0;
|for (int i = 0; i < $b.length; i++) {
| ${genComputeHash(ctx, s"$b[i]", ByteType, hash)}
| $result = ($result ^ (0x9e3779b9)) + $hash + ($result << 6) +
($result >>> 2);
|}
""".stripMargin
}
```
when b=input.getBytes(), the current 2.0 code results in getBytes() being
called n times, n being length of input. getBytes() involves memory copy is
thus expensive and causes a performance degradation.
Fix is to evaluate getBytes() before the for loop.
## How was this patch tested?
Performance bug, no additional test added.
Author: Qifan Pu <[email protected]>
Closes #14337 from ooq/SPARK-16699.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d226dce1
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d226dce1
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d226dce1
Branch: refs/heads/branch-2.0
Commit: d226dce12babcd9f30db033417b2b9ce79f44312
Parents: 198b042
Author: Qifan Pu <[email protected]>
Authored: Sun Jul 24 21:53:21 2016 -0700
Committer: Reynold Xin <[email protected]>
Committed: Sun Jul 24 21:53:21 2016 -0700
----------------------------------------------------------------------
.../sql/execution/aggregate/VectorizedHashMapGenerator.scala | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/d226dce1/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/VectorizedHashMapGenerator.scala
----------------------------------------------------------------------
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/VectorizedHashMapGenerator.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/VectorizedHashMapGenerator.scala
index 8a3f466..b4a9059 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/VectorizedHashMapGenerator.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/VectorizedHashMapGenerator.scala
@@ -313,10 +313,12 @@ class VectorizedHashMapGenerator(
def hashLong(l: String): String = s"long $result = $l;"
def hashBytes(b: String): String = {
val hash = ctx.freshName("hash")
+ val bytes = ctx.freshName("bytes")
s"""
|int $result = 0;
- |for (int i = 0; i < $b.length; i++) {
- | ${genComputeHash(ctx, s"$b[i]", ByteType, hash)}
+ |byte[] $bytes = $b;
+ |for (int i = 0; i < $bytes.length; i++) {
+ | ${genComputeHash(ctx, s"$bytes[i]", ByteType, hash)}
| $result = ($result ^ (0x9e3779b9)) + $hash + ($result << 6) +
($result >>> 2);
|}
""".stripMargin
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
