LuciferYang commented on PR #38427:
URL: https://github.com/apache/spark/pull/38427#issuecomment-1296208816
The bench result by GA as follows:
**Scala 2.12**
```
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test size of Seq with buffer size 1 and call .size 1 time(s): Best Time(ms)
Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------------------
toSeq + Size 1
1 0 96.2 10.4 1.0X
toIndexedSeq + Size 13
14 1 7.4 134.4 0.1X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test size of Seq with buffer size 10 and call .size 1 time(s): Best
Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------------------------
toSeq + Size 1
1 0 93.3 10.7 1.0X
toIndexedSeq + Size 21
21 1 4.7 212.3 0.1X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test size of Seq with buffer size 100 and call .size 1 time(s): Best
Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------------------------
toSeq + Size
1 1 0 93.3 10.7 1.0X
toIndexedSeq + Size
87 88 1 1.1 870.9 0.0X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test size of Seq with buffer size 1000 and call .size 1 time(s): Best
Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------------------------
toSeq + Size
1 1 0 93.3 10.7 1.0X
toIndexedSeq + Size
687 689 2 0.1 6870.1 0.0X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test size of Seq with buffer size 10000 and call .size 1 time(s): Best
Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------------------------------
toSeq + Size
1 1 0 93.3 10.7 1.0X
toIndexedSeq + Size
6738 6738 0 0.0 67378.4 0.0X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test size of Seq with buffer size 100000 and call .size 1 time(s): Best
Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------------------------
toSeq + Size
1 1 0 93.3 10.7 1.0X
toIndexedSeq + Size
67357 67452 135 0.0 673566.0 0.0X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test size of Seq with buffer size 1000 and call .size 2 time(s): Best
Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------------------------
toSeq + Size
1 1 0 78.6 12.7 1.0X
toIndexedSeq + Size
692 695 2 0.1 6921.7 0.0X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test size of Seq with buffer size 1000 and call .size 3 time(s): Best
Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------------------------
toSeq + Size
1 1 0 67.9 14.7 1.0X
toIndexedSeq + Size
695 697 2 0.1 6945.6 0.0X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test size of Seq with buffer size 1000 and call .size 4 time(s): Best
Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------------------------
toSeq + Size
2 2 0 57.6 17.4 1.0X
toIndexedSeq + Size
698 699 1 0.1 6983.1 0.0X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test size of Seq with buffer size 1000 and call .size 5 time(s): Best
Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------------------------
toSeq + Size
2 2 0 49.8 20.1 1.0X
toIndexedSeq + Size
702 703 1 0.1 7016.7 0.0X
```
**Scala 2.13**
```
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test size of Seq with buffer size 1 and call .size 1 time(s): Best Time(ms)
Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------------------
toSeq + Size 9
9 0 11.6 86.2 1.0X
toIndexedSeq + Size 17
18 1 5.8 173.8 0.5X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test size of Seq with buffer size 10 and call .size 1 time(s): Best
Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------------------------
toSeq + Size 20
21 1 4.9 202.2 1.0X
toIndexedSeq + Size 20
20 0 5.1 195.9 1.0X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test size of Seq with buffer size 100 and call .size 1 time(s): Best
Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------------------------
toSeq + Size
129 132 2 0.8 1294.2 1.0X
toIndexedSeq + Size
76 77 1 1.3 763.8 1.7X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test size of Seq with buffer size 1000 and call .size 1 time(s): Best
Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------------------------
toSeq + Size
1240 1241 2 0.1 12397.0 1.0X
toIndexedSeq + Size
577 579 2 0.2 5769.4 2.1X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test size of Seq with buffer size 10000 and call .size 1 time(s): Best
Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------------------------------
toSeq + Size
12279 12285 8 0.0 122791.8 1.0X
toIndexedSeq + Size
5641 5646 8 0.0 56405.9 2.2X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test size of Seq with buffer size 100000 and call .size 1 time(s): Best
Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------------------------
toSeq + Size
130320 130553 330 0.0 1303202.4 1.0X
toIndexedSeq + Size
59120 59237 166 0.0 591196.0 2.2X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test size of Seq with buffer size 1000 and call .size 2 time(s): Best
Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------------------------
toSeq + Size
1675 1675 1 0.1 16747.3 1.0X
toIndexedSeq + Size
614 614 1 0.2 6136.6 2.7X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test size of Seq with buffer size 1000 and call .size 3 time(s): Best
Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------------------------
toSeq + Size
2100 2104 6 0.0 21000.9 1.0X
toIndexedSeq + Size
614 615 1 0.2 6143.8 3.4X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test size of Seq with buffer size 1000 and call .size 4 time(s): Best
Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------------------------
toSeq + Size
2544 2546 3 0.0 25442.4 1.0X
toIndexedSeq + Size
616 616 0 0.2 6161.8 4.1X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test size of Seq with buffer size 1000 and call .size 5 time(s): Best
Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------------------------
toSeq + Size
2983 2988 8 0.0 29829.5 1.0X
toIndexedSeq + Size
617 618 1 0.2 6165.7 4.8X
```
The overall trend is consistent with local tests. Using `toIndexedSeq`
improves the performance of Scala 2.13 by 2 to 4 times, but degrades the
performance of Scala 2.12 by more than 10 times
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
