[GitHub] [spark] mgaido91 commented on issue #24636: [SPARK-27684][SQL] Avoid conversion overhead for primitive types

Tue, 21 May 2019 02:42:45 -0700 

mgaido91 commented on issue #24636: [SPARK-27684][SQL] Avoid conversion 
overhead for primitive types
URL: https://github.com/apache/spark/pull/24636#issuecomment-494319192
 
 
   Thanks for you comment @kiszk .
   
   I added the warm-up, and the result is barely the same. Here is the code:
   ```
   test("SPARK-27684: perf evaluation") {
       val intLongUdf = ScalaUDF(
         (a: Int, b: Long) => a + b, LongType,
         Literal(1) :: Literal(1L) :: Nil,
         true :: true :: Nil,
         nullable = false)
   
       val plan = generateProject(
         MutableProjection.create(Alias(intLongUdf, s"udf")() :: Nil),
         intLongUdf)
       plan.initialize(0)
   
       var i = 0
       val N = 100000000
       while(i < 1000) {
         plan(EmptyRow).get(0, intLongUdf.dataType)
         plan(EmptyRow).get(0, intLongUdf.dataType)
         plan(EmptyRow).get(0, intLongUdf.dataType)
         plan(EmptyRow).get(0, intLongUdf.dataType)
         plan(EmptyRow).get(0, intLongUdf.dataType)
         plan(EmptyRow).get(0, intLongUdf.dataType)
         plan(EmptyRow).get(0, intLongUdf.dataType)
         plan(EmptyRow).get(0, intLongUdf.dataType)
         plan(EmptyRow).get(0, intLongUdf.dataType)
         plan(EmptyRow).get(0, intLongUdf.dataType)
         i += 1
       }
       i = 0
       val t0 = System.nanoTime()
       while(i < N) {
         plan(EmptyRow).get(0, intLongUdf.dataType)
         plan(EmptyRow).get(0, intLongUdf.dataType)
         plan(EmptyRow).get(0, intLongUdf.dataType)
         plan(EmptyRow).get(0, intLongUdf.dataType)
         plan(EmptyRow).get(0, intLongUdf.dataType)
         plan(EmptyRow).get(0, intLongUdf.dataType)
         plan(EmptyRow).get(0, intLongUdf.dataType)
         plan(EmptyRow).get(0, intLongUdf.dataType)
         plan(EmptyRow).get(0, intLongUdf.dataType)
         plan(EmptyRow).get(0, intLongUdf.dataType)
         i += 1
       }
       val t1 = System.nanoTime()
       println(s"Avg time: ${(t1 - t0).toDouble / N} ns")
     }
   ```
   
   and the results are:
   ```
   Old Avg time: 49.58303799 ns
   New Avg time: 12.66588096 ns
   ```
   
   Any more comments/concerns?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[email protected]


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to