LuciferYang commented on PR #37609:
URL: https://github.com/apache/spark/pull/37609#issuecomment-1245153126
To check `val map: Map[K, V] = data.zip(data)(collection.breakOut)`, update
bench result
```
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Test zip to map with collectionSize = 1: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use zip + toMap 16 18
2 6.4 155.5 1.0X
Use zip + collection.breakOut 3 4
1 29.3 34.1 4.6X
Use Manual builder 3 3
1 33.3 30.0 5.2X
Use Manual map 3 3
1 39.1 25.6 6.1X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Test zip to map with collectionSize = 5: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use zip + toMap 91 98
6 1.1 910.1 1.0X
Use zip + collection.breakOut 74 77
3 1.4 737.3 1.2X
Use Manual builder 72 77
4 1.4 722.1 1.3X
Use Manual map 43 46
1 2.3 426.1 2.1X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Test zip to map with collectionSize = 10: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use zip + toMap 114 120
4 0.9 1138.3 1.0X
Use zip + collection.breakOut 95 100
3 1.0 954.1 1.2X
Use Manual builder 94 101
4 1.1 942.4 1.2X
Use Manual map 85 91
4 1.2 851.7 1.3X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Test zip to map with collectionSize = 20: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use zip + toMap 156 162
4 0.6 1560.9 1.0X
Use zip + collection.breakOut 136 140
3 0.7 1356.4 1.2X
Use Manual builder 132 143
8 0.8 1317.6 1.2X
Use Manual map 166 170
3 0.6 1657.2 0.9X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Test zip to map with collectionSize = 50: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use zip + toMap 628 636
10 0.2 6277.4 1.0X
Use zip + collection.breakOut 572 583
10 0.2 5720.1 1.1X
Use Manual builder 574 584
16 0.2 5741.6 1.1X
Use Manual map 466 477
12 0.2 4660.8 1.3X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Test zip to map with collectionSize = 100: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------
Use zip + toMap 1127 1138
15 0.1 11269.1 1.0X
Use zip + collection.breakOut 1060 1073
18 0.1 10600.8 1.1X
Use Manual builder 1050 1073
32 0.1 10500.7 1.1X
Use Manual map 1004 1017
19 0.1 10039.2 1.1X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Test zip to map with collectionSize = 500: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------
Use zip + toMap 4634 4665
44 0.0 46338.7 1.0X
Use zip + collection.breakOut 4772 4792
28 0.0 47723.3 1.0X
Use Manual builder 4517 4597
112 0.0 45173.1 1.0X
Use Manual map 6473 6487
20 0.0 64726.3 0.7X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Test zip to map with collectionSize = 1000: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------
Use zip + toMap 12355 12366
16 0.0 123550.7 1.0X
Use zip + collection.breakOut 12585 12593
11 0.0 125846.1 1.0X
Use Manual builder 12076 12101
35 0.0 120764.3 1.0X
Use Manual map 14641 14664
34 0.0 146406.5 0.8X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Test zip to map with collectionSize = 5000: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------
Use zip + toMap 68054 68354
425 0.0 680539.3 1.0X
Use zip + collection.breakOut 73307 73316
13 0.0 733073.3 0.9X
Use Manual builder 70887 71129
342 0.0 708867.4 1.0X
Use Manual map 91936 91980
63 0.0 919357.2 0.7X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Test zip to map with collectionSize = 10000: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
Use zip + toMap 142306 142993
971 0.0 1423062.3 1.0X
Use zip + collection.breakOut 148545 148635
127 0.0 1485454.3 1.0X
Use Manual builder 143287 144215
1313 0.0 1432866.1 1.0X
Use Manual map 198459 198995
758 0.0 1984586.7 0.7X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Test zip to map with collectionSize = 20000: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
Use zip + toMap 318022 318528
716 0.0 3180223.5 1.0X
Use zip + collection.breakOut 333891 337354
2352 0.0 3338910.7 1.0X
Use Manual builder 319468 320649
1670 0.0 3194676.5 1.0X
Use Manual map 423019 423164
204 0.0 4230194.5 0.8X
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
