Wenchen Fan created SPARK-25317:
-----------------------------------
Summary: MemoryBlock performance regression
Key: SPARK-25317
URL: https://issues.apache.org/jira/browse/SPARK-25317
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.4.0
Reporter: Wenchen Fan
There is a performance regression when calculating hash code for UTF8String:
{code}
test("hashing") {
import org.apache.spark.unsafe.hash.Murmur3_x86_32
import org.apache.spark.unsafe.types.UTF8String
val hasher = new Murmur3_x86_32(0)
val str = UTF8String.fromString("b" * 10001)
val numIter = 100000
val start = System.nanoTime
for (i <- 0 until numIter) {
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
Murmur3_x86_32.hashUTF8String(str, 0)
}
val duration = (System.nanoTime() - start) / 1000 / numIter
println(s"duration $duration us")
}
{code}
To run this test in 2.3, we need to add
{code}
public static int hashUTF8String(UTF8String str, int seed) {
return hashUnsafeBytes(str.getBaseObject(), str.getBaseOffset(),
str.numBytes(), seed);
}
{code}
to `Murmur3_x86_32`
In my laptop, the result for master vs 2.3 is: 120 us vs 40 us
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
