Github user Dooyoung-Hwang commented on the issue:
https://github.com/apache/spark/pull/22347
I tested in my local PC. 3.3 GHz Intel Core i5, and selected 400,000 rows x
25 times.
I took a total execution time between decodeUnsafeRows.
My tested data is skewed, so gathered rows from executor are distributed
between 400000 and 800000.
The average execution time decreased from 175.92ms to 93.52ms.
Memory usage also improved, and total GC Time is decreased from 13.883 sec
to 10.764 sec.
## Before Patch
### GC statics
S0 | S1 | E | O | M | CCS | YGC | YGCT | FGC | FGCT | GCT
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
0 | 100 | 24.92 | 48.66 | 96.92 | 88.25 | 150 | 13.883 | 0 | 0 | 13.883
### Wall time : AVG 175.92 ms
Row Count | Decode Time(ms)
-- | --
428942 | 73
473726 | 106
476322 | 78
509996 | 83
510590 | 124
556896 | 94
556896 | 362
595272 | 193
595272 | 175
642478 | 120
644970 | 279
679544 | 269
693354 | 116
723532 | 124
729912 | 136
730218 | 120
730246 | 184
773640 | 183
774148 | 380
810198 | 128
811606 | 131
859090 | 138
895474 | 314
895954 | 339
939636 | 149
## After Patch : 93.52ms
### GC statics
S0 | S1 | E | O | M | CCS | YGC | YGCT | FGC | FGCT | GCT
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
0 | 100 | 81.37 | 33.34 | 97.21 | 88.35 | 127 | 10.764 | 0 | 0 | 10.764
### Wall time : AVG 93.52 ms
Row Count | Decode time (ms)
-- | --
421922 | 61
422516 | 180
422850 | 110
473218 | 62
473218 | 103
473438 | 115
507198 | 60
554606 | 144
557202 | 119
601392 | 71
642652 | 61
645276 | 64
679036 | 64
679036 | 63
729624 | 242
729652 | 62
729912 | 131
773814 | 122
774234 | 62
807908 | 59
810198 | 64
814900 | 72
844772 | 59
858582 | 127
858582 | 61
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
