subject:"\[jira\] \[Updated\] \(SPARK\-25317\) MemoryBlock performance regression"

[jira] [Updated] (SPARK-25317) MemoryBlock performance regression

2018-09-03 Thread Kazuaki Ishizaki (JIRA)



 [ 
https://issues.apache.org/jira/browse/SPARK-25317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kazuaki Ishizaki updated SPARK-25317:
-
Description: 
eThere is a performance regression when calculating hash code for UTF8String:
{code:java}
  test("hashing") {
import org.apache.spark.unsafe.hash.Murmur3_x86_32
import org.apache.spark.unsafe.types.UTF8String
val hasher = new Murmur3_x86_32(0)
val str = UTF8String.fromString("b" * 10001)
val numIter = 10
val start = System.nanoTime
for (i <- 0 until numIter) {
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
}
val duration = (System.nanoTime() - start) / 1000 / numIter
println(s"duration $duration us")
  }
{code}
To run this test in 2.3, we need to add
{code:java}
public static int hashUTF8String(UTF8String str, int seed) {
return hashUnsafeBytes(str.getBaseObject(), str.getBaseOffset(), 
str.numBytes(), seed);
  }
{code}
to `Murmur3_x86_32`

In my laptop, the result for master vs 2.3 is: 120 us vs 40 us

  was:
There is a performance regression when calculating hash code for UTF8String:

{code}
  test("hashing") {
import org.apache.spark.unsafe.hash.Murmur3_x86_32
import org.apache.spark.unsafe.types.UTF8String
val hasher = new Murmur3_x86_32(0)
val str = UTF8String.fromString("b" * 10001)
val numIter = 10
val start = System.nanoTime
for (i <- 0 until numIter) {
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
  Murmur3_x86_32.hashUTF8String(str, 0)
}
val duration = (System.nanoTime() - start) / 1000 / numIter
println(s"duration $duration us")
  }
{code}

To run this test in 2.3, we need to add
{code}
public static int hashUTF8String(UTF8String str, int seed) {
return hashUnsafeBytes(str.getBaseObject(), str.getBaseOffset(), 
str.numBytes(), seed);
  }
{code}
to `Murmur3_x86_32`

In my laptop, the result for master vs 2.3 is: 120 us vs 40 us


> MemoryBlock performance regression
> --
>
> Key: SPARK-25317
> URL: https://issues.apache.org/jira/browse/SPARK-25317
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.4.0
>Reporter: Wenchen Fan
>Priority: Blocker
>
> eThere is a performance regression when calculating hash code for UTF8String:
> {code:java}
>   test("hashing") {
> import

[jira] [Updated] (SPARK-25317) MemoryBlock performance regression

2018-09-03 Thread Wenchen Fan (JIRA)



 [ 
https://issues.apache.org/jira/browse/SPARK-25317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wenchen Fan updated SPARK-25317:

Priority: Blocker  (was: Major)

> MemoryBlock performance regression
> --
>
> Key: SPARK-25317
> URL: https://issues.apache.org/jira/browse/SPARK-25317
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.4.0
>Reporter: Wenchen Fan
>Priority: Blocker
>
> There is a performance regression when calculating hash code for UTF8String:
> {code}
>   test("hashing") {
> import org.apache.spark.unsafe.hash.Murmur3_x86_32
> import org.apache.spark.unsafe.types.UTF8String
> val hasher = new Murmur3_x86_32(0)
> val str = UTF8String.fromString("b" * 10001)
> val numIter = 10
> val start = System.nanoTime
> for (i <- 0 until numIter) {
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
>   Murmur3_x86_32.hashUTF8String(str, 0)
> }
> val duration = (System.nanoTime() - start) / 1000 / numIter
> println(s"duration $duration us")
>   }
> {code}
> To run this test in 2.3, we need to add
> {code}
> public static int hashUTF8String(UTF8String str, int seed) {
> return hashUnsafeBytes(str.getBaseObject(), str.getBaseOffset(), 
> str.numBytes(), seed);
>   }
> {code}
> to `Murmur3_x86_32`
> In my laptop, the result for master vs 2.3 is: 120 us vs 40 us



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-25317) MemoryBlock performance regression

[jira] [Updated] (SPARK-25317) MemoryBlock performance regression

2 matches

Site Navigation

Mail list logo

Footer information