yaooqinn commented on issue #26491: [SPARK-29870][SQL] Unify the logic of multi-units interval string to CalendarInterval URL: https://github.com/apache/spark/pull/26491#issuecomment-553266844 # benchmark test with ``` val N = 5000000```, performace seems not to be affected that much ## before ```java [info] Java HotSpot(TM) 64-Bit Server VM 1.8.0_65-b17 on Mac OS X 10.15.1 [info] Intel(R) Core(TM) i5-5287U CPU @ 2.90GHz [info] cast strings to intervals: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative [info] ------------------------------------------------------------------------------------------------------------------------ [info] prepare string w/ interval 2198 2231 46 2.3 439.6 1.0X [info] prepare string w/o interval 2189 2636 388 2.3 437.8 1.0X [info] 1 units w/ interval 2256 2315 76 2.2 451.2 1.0X [info] 1 units w/o interval 2106 2182 95 2.4 421.2 1.0X [info] 2 units w/ interval 3137 3597 792 1.6 627.5 0.7X [info] 2 units w/o interval 3038 3299 241 1.6 607.6 0.7X [info] 3 units w/ interval 6747 7466 638 0.7 1349.3 0.3X [info] 3 units w/o interval 6905 7583 605 0.7 1381.0 0.3X [info] 4 units w/ interval 7621 8126 785 0.7 1524.2 0.3X [info] 4 units w/o interval 7323 8823 1305 0.7 1464.5 0.3X [info] 5 units w/ interval 8297 8522 378 0.6 1659.4 0.3X [info] 5 units w/o interval 8215 8220 6 0.6 1642.9 0.3X [info] 6 units w/ interval 9022 9111 78 0.6 1804.4 0.2X [info] 6 units w/o interval 9642 12338 NaN 0.5 1928.4 0.2X [info] 7 units w/ interval 9343 48099 1662 0.5 1868.6 0.2X [info] 7 units w/o interval 8879 8941 54 0.6 1775.7 0.2X [info] 8 units w/ interval 10401 10537 225 0.5 2080.1 0.2X [info] 8 units w/o interval 10398 10413 20 0.5 2079.6 0.2X [info] 9 units w/ interval 10206 10373 189 0.5 2041.2 0.2X [info] 9 units w/o interval 10100 10151 45 0.5 2020.0 0.2X [info] 10 units w/ interval 10887 11881 1596 0.5 2177.5 0.2X [info] 10 units w/o interval 11430 13329 1754 0.4 2286.1 0.2X [info] 11 units w/ interval 13046 14058 877 0.4 2609.2 0.2X [info] 11 units w/o interval 12059 12297 265 0.4 2411.7 0.2X [info] ``` ## after ``` info] Java HotSpot(TM) 64-Bit Server VM 1.8.0_65-b17 on Mac OS X 10.15.1 [info] Intel(R) Core(TM) i5-5287U CPU @ 2.90GHz [info] cast strings to intervals: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative [info] ------------------------------------------------------------------------------------------------------------------------ [info] prepare string w/ interval 2055 2230 202 2.4 411.1 1.0X [info] prepare string w/o interval 1894 1938 71 2.6 378.8 1.1X [info] 1 units w/ interval 2235 2285 68 2.2 447.1 0.9X [info] 1 units w/o interval 2046 2076 38 2.4 409.1 1.0X [info] 2 units w/ interval 3024 3624 978 1.7 604.8 0.7X [info] 2 units w/o interval 4622 4963 336 1.1 924.3 0.4X [info] 3 units w/ interval 6794 7369 499 0.7 1358.7 0.3X [info] 3 units w/o interval 6090 7030 815 0.8 1217.9 0.3X [info] 4 units w/ interval 8100 10666 NaN 0.6 1619.9 0.3X [info] 4 units w/o interval 8063 10349 NaN 0.6 1612.5 0.3X [info] 5 units w/ interval 8429 8630 318 0.6 1685.8 0.2X [info] 5 units w/o interval 8448 9335 1365 0.6 1689.7 0.2X [info] 6 units w/ interval 8671 8707 31 0.6 1734.2 0.2X [info] 6 units w/o interval 8613 8638 38 0.6 1722.5 0.2X [info] 7 units w/ interval 8924 8981 68 0.6 1784.8 0.2X [info] 7 units w/o interval 8858 8889 52 0.6 1771.5 0.2X [info] 8 units w/ interval 10640 11768 979 0.5 2127.9 0.2X [info] 8 units w/o interval 10990 11551 554 0.5 2197.9 0.2X [info] 9 units w/ interval 10783 11849 1110 0.5 2156.5 0.2X [info] 9 units w/o interval 10408 12267 1609 0.5 2081.6 0.2X [info] 10 units w/ interval 11245 13406 851 0.4 2249.1 0.2X [info] 10 units w/o interval 11331 11444 102 0.4 2266.2 0.2X [info] 11 units w/ interval 12480 13403 919 0.4 2496.0 0.2X [info] 11 units w/o interval 12001 12009 7 0.4 2400.2 0.2X ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
