Repository: spark Updated Branches: refs/heads/branch-2.0 198b0426e -> d226dce12
[SPARK-16699][SQL] Fix performance bug in hash aggregate on long string keys ## What changes were proposed in this pull request? In the following code in `VectorizedHashMapGenerator.scala`: ``` def hashBytes(b: String): String = { val hash = ctx.freshName("hash") s""" |int $result = 0; |for (int i = 0; i < $b.length; i++) { | ${genComputeHash(ctx, s"$b[i]", ByteType, hash)} | $result = ($result ^ (0x9e3779b9)) + $hash + ($result << 6) + ($result >>> 2); |} """.stripMargin } ``` when b=input.getBytes(), the current 2.0 code results in getBytes() being called n times, n being length of input. getBytes() involves memory copy is thus expensive and causes a performance degradation. Fix is to evaluate getBytes() before the for loop. ## How was this patch tested? Performance bug, no additional test added. Author: Qifan Pu <qifan...@gmail.com> Closes #14337 from ooq/SPARK-16699. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d226dce1 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d226dce1 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d226dce1 Branch: refs/heads/branch-2.0 Commit: d226dce12babcd9f30db033417b2b9ce79f44312 Parents: 198b042 Author: Qifan Pu <qifan...@gmail.com> Authored: Sun Jul 24 21:53:21 2016 -0700 Committer: Reynold Xin <r...@databricks.com> Committed: Sun Jul 24 21:53:21 2016 -0700 ---------------------------------------------------------------------- .../sql/execution/aggregate/VectorizedHashMapGenerator.scala | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/d226dce1/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/VectorizedHashMapGenerator.scala ---------------------------------------------------------------------- diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/VectorizedHashMapGenerator.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/VectorizedHashMapGenerator.scala index 8a3f466..b4a9059 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/VectorizedHashMapGenerator.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/VectorizedHashMapGenerator.scala @@ -313,10 +313,12 @@ class VectorizedHashMapGenerator( def hashLong(l: String): String = s"long $result = $l;" def hashBytes(b: String): String = { val hash = ctx.freshName("hash") + val bytes = ctx.freshName("bytes") s""" |int $result = 0; - |for (int i = 0; i < $b.length; i++) { - | ${genComputeHash(ctx, s"$b[i]", ByteType, hash)} + |byte[] $bytes = $b; + |for (int i = 0; i < $bytes.length; i++) { + | ${genComputeHash(ctx, s"$bytes[i]", ByteType, hash)} | $result = ($result ^ (0x9e3779b9)) + $hash + ($result << 6) + ($result >>> 2); |} """.stripMargin --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org