yaooqinn commented on code in PR #45654:
URL: https://github.com/apache/spark/pull/45654#discussion_r1534987046
##########
sql/hive/benchmarks/OrcReadBenchmark-jdk21-results.txt:
##########
@@ -2,221 +2,221 @@
SQL Single Numeric Column Scan
================================================================================================
-OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
+OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 6.5.0-1016-azure
AMD EPYC 7763 64-Core Processor
SQL Single TINYINT Column Scan: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Hive built-in ORC 640 693
75 24.6 40.7 1.0X
-Native ORC MR 719 733
24 21.9 45.7 0.9X
-Native ORC Vectorized 90 112
15 175.3 5.7 7.1X
+Hive built-in ORC 662 691
46 23.8 42.1 1.0X
+Native ORC MR 784 812
28 20.1 49.9 0.8X
+Native ORC Vectorized 92 120
35 171.5 5.8 7.2X
-OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
+OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 6.5.0-1016-azure
AMD EPYC 7763 64-Core Processor
SQL Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Hive built-in ORC 633 698
79 24.9 40.2 1.0X
-Native ORC MR 726 757
32 21.7 46.1 0.9X
-Native ORC Vectorized 74 91
12 212.5 4.7 8.5X
+Hive built-in ORC 692 724
31 22.7 44.0 1.0X
+Native ORC MR 789 803
15 19.9 50.1 0.9X
+Native ORC Vectorized 73 94
17 214.7 4.7 9.4X
-OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
+OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 6.5.0-1016-azure
AMD EPYC 7763 64-Core Processor
SQL Single INT Column Scan: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Hive built-in ORC 732 737
5 21.5 46.5 1.0X
-Native ORC MR 832 872
38 18.9 52.9 0.9X
-Native ORC Vectorized 94 116
29 168.0 6.0 7.8X
+Hive built-in ORC 685 708
21 23.0 43.5 1.0X
+Native ORC MR 811 854
40 19.4 51.5 0.8X
+Native ORC Vectorized 79 94
18 199.8 5.0 8.7X
-OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
+OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 6.5.0-1016-azure
AMD EPYC 7763 64-Core Processor
SQL Single BIGINT Column Scan: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Hive built-in ORC 763 778
18 20.6 48.5 1.0X
-Native ORC MR 844 858
14 18.6 53.7 0.9X
-Native ORC Vectorized 110 134
26 143.5 7.0 7.0X
+Hive built-in ORC 733 771
39 21.5 46.6 1.0X
+Native ORC MR 803 819
15 19.6 51.1 0.9X
+Native ORC Vectorized 113 128
16 138.8 7.2 6.5X
-OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
+OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 6.5.0-1016-azure
AMD EPYC 7763 64-Core Processor
SQL Single FLOAT Column Scan: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Hive built-in ORC 852 871
16 18.5 54.2 1.0X
-Native ORC MR 848 853
4 18.6 53.9 1.0X
-Native ORC Vectorized 163 178
24 96.6 10.4 5.2X
+Hive built-in ORC 804 815
15 19.6 51.1 1.0X
+Native ORC MR 873 904
27 18.0 55.5 0.9X
+Native ORC Vectorized 138 169
29 114.2 8.8 5.8X
-OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
+OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 6.5.0-1016-azure
AMD EPYC 7763 64-Core Processor
SQL Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Hive built-in ORC 860 885
27 18.3 54.7 1.0X
-Native ORC MR 948 968
31 16.6 60.3 0.9X
-Native ORC Vectorized 242 258
15 65.0 15.4 3.6X
+Hive built-in ORC 812 825
12 19.4 51.6 1.0X
+Native ORC MR 934 943
13 16.8 59.4 0.9X
+Native ORC Vectorized 188 213
24 83.6 12.0 4.3X
================================================================================================
Int and String Scan
================================================================================================
-OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
+OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 6.5.0-1016-azure
AMD EPYC 7763 64-Core Processor
Int and String Scan: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Hive built-in ORC 1814 1836
31 5.8 173.0 1.0X
-Native ORC MR 1573 1633
84 6.7 150.0 1.2X
-Native ORC Vectorized 763 775
10 13.7 72.8 2.4X
+Hive built-in ORC 1547 1557
15 6.8 147.5 1.0X
+Native ORC MR 1469 1478
13 7.1 140.1 1.1X
+Native ORC Vectorized 573 601
32 18.3 54.6 2.7X
================================================================================================
Partitioned Table Scan
================================================================================================
-OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
+OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 6.5.0-1016-azure
AMD EPYC 7763 64-Core Processor
Partitioned Table: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Data column - Hive built-in ORC 1028 1035
9 15.3 65.4 1.0X
-Data column - Native ORC MR 1016 1023
10 15.5 64.6 1.0X
-Data column - Native ORC Vectorized 110 135
11 143.4 7.0 9.4X
-Partition column - Hive built-in ORC 665 677
19 23.7 42.3 1.5X
-Partition column - Native ORC MR 553 573
23 28.4 35.2 1.9X
-Partition column - Native ORC Vectorized 32 43
8 491.4 2.0 32.1X
-Both columns - Hive built-in ORC 919 939
33 17.1 58.4 1.1X
-Both columns - Native ORC MR 1032 1040
12 15.2 65.6 1.0X
-Both columns - Native ORC Vectorized 121 144
21 129.8 7.7 8.5X
+Data column - Hive built-in ORC 827 859
48 19.0 52.6 1.0X
+Data column - Native ORC MR 977 1025
77 16.1 62.1 0.8X
+Data column - Native ORC Vectorized 109 124
15 144.3 6.9 7.6X
+Partition column - Hive built-in ORC 702 715
12 22.4 44.6 1.2X
+Partition column - Native ORC MR 556 562
9 28.3 35.3 1.5X
+Partition column - Native ORC Vectorized 31 41
8 510.7 2.0 26.8X
+Both columns - Hive built-in ORC 888 933
40 17.7 56.5 0.9X
+Both columns - Native ORC MR 1076 1083
10 14.6 68.4 0.8X
+Both columns - Native ORC Vectorized 126 144
17 125.3 8.0 6.6X
================================================================================================
Repeated String Scan
================================================================================================
-OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
+OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 6.5.0-1016-azure
AMD EPYC 7763 64-Core Processor
Repeated String: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Hive built-in ORC 873 890
21 12.0 83.3 1.0X
-Native ORC MR 789 858
75 13.3 75.3 1.1X
-Native ORC Vectorized 121 134
16 86.8 11.5 7.2X
+Hive built-in ORC 832 886
48 12.6 79.3 1.0X
+Native ORC MR 821 825
6 12.8 78.3 1.0X
+Native ORC Vectorized 173 189
14 60.7 16.5 4.8X
================================================================================================
String with Nulls Scan
================================================================================================
-OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
+OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 6.5.0-1016-azure
AMD EPYC 7763 64-Core Processor
String with Nulls Scan (0.0%): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Hive built-in ORC 1723 1732
13 6.1 164.3 1.0X
-Native ORC MR 1311 1327
23 8.0 125.0 1.3X
-Native ORC Vectorized 370 377
6 28.4 35.3 4.7X
+Hive built-in ORC 1421 1433
16 7.4 135.6 1.0X
+Native ORC MR 1205 1215
14 8.7 114.9 1.2X
+Native ORC Vectorized 285 314
29 36.8 27.2 5.0X
-OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
+OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 6.5.0-1016-azure
AMD EPYC 7763 64-Core Processor
String with Nulls Scan (50.0%): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Hive built-in ORC 1297 1327
43 8.1 123.7 1.0X
-Native ORC MR 1145 1156
15 9.2 109.2 1.1X
-Native ORC Vectorized 391 419
19 26.8 37.3 3.3X
+Hive built-in ORC 1261 1273
17 8.3 120.3 1.0X
+Native ORC MR 1121 1131
14 9.4 106.9 1.1X
+Native ORC Vectorized 341 357
13 30.8 32.5 3.7X
-OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
+OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 6.5.0-1016-azure
AMD EPYC 7763 64-Core Processor
String with Nulls Scan (95.0%): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Hive built-in ORC 741 766
21 14.1 70.7 1.0X
-Native ORC MR 743 751
8 14.1 70.9 1.0X
-Native ORC Vectorized 154 175
20 67.9 14.7 4.8X
+Hive built-in ORC 840 842
2 12.5 80.1 1.0X
+Native ORC MR 750 751
1 14.0 71.5 1.1X
+Native ORC Vectorized 144 160
9 72.7 13.8 5.8X
================================================================================================
Single Column Scan From Wide Columns
================================================================================================
-OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
+OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 6.5.0-1016-azure
AMD EPYC 7763 64-Core Processor
Single Column Scan from 100 columns: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Hive built-in ORC 571 611
43 1.8 544.5 1.0X
-Native ORC MR 91 102
8 11.5 86.8 6.3X
-Native ORC Vectorized 39 48
6 27.0 37.0 14.7X
+Hive built-in ORC 669 688
23 1.6 638.3 1.0X
+Native ORC MR 85 100
13 12.3 81.4 7.8X
+Native ORC Vectorized 36 46
9 29.3 34.1 18.7X
-OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
+OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 6.5.0-1016-azure
AMD EPYC 7763 64-Core Processor
Single Column Scan from 200 columns: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Hive built-in ORC 1066 1076
15 1.0 1016.6 1.0X
-Native ORC MR 102 114
9 10.3 96.9 10.5X
-Native ORC Vectorized 50 63
8 21.0 47.7 21.3X
+Hive built-in ORC 1221 1230
13 0.9 1164.4 1.0X
+Native ORC MR 100 111
12 10.5 95.7 12.2X
+Native ORC Vectorized 50 68
11 20.9 47.8 24.4X
-OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
+OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 6.5.0-1016-azure
AMD EPYC 7763 64-Core Processor
Single Column Scan from 300 columns: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Hive built-in ORC 1532 1562
42 0.7 1461.1 1.0X
-Native ORC MR 114 124
7 9.2 109.0 13.4X
-Native ORC Vectorized 62 72
9 17.0 59.0 24.8X
+Hive built-in ORC 1834 1884
70 0.6 1749.1 1.0X
+Native ORC MR 112 135
23 9.3 107.2 16.3X
+Native ORC Vectorized 61 70
8 17.3 58.0 30.2X
================================================================================================
Struct scan
================================================================================================
-OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
+OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 6.5.0-1016-azure
AMD EPYC 7763 64-Core Processor
Single Struct Column Scan with 10 Fields: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Hive built-in ORC 447 499
32 2.3 425.8 1.0X
-Native ORC MR 224 267
41 4.7 213.7 2.0X
-Native ORC Vectorized 99 115
21 10.6 94.5 4.5X
+Hive built-in ORC 279 338
41 3.8 266.0 1.0X
+Native ORC MR 220 268
44 4.8 209.8 1.3X
+Native ORC Vectorized 97 115
18 10.8 92.5 2.9X
-OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
+OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 6.5.0-1016-azure
AMD EPYC 7763 64-Core Processor
Single Struct Column Scan with 100 Fields: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------
-Hive built-in ORC 3468 3546
110 0.3 3307.7 1.0X
-Native ORC MR 1623 1655
47 0.6 1547.4 2.1X
-Native ORC Vectorized 795 868
85 1.3 757.8 4.4X
+Hive built-in ORC 1869 1890
29 0.6 1782.7 1.0X
+Native ORC MR 1684 1729
64 0.6 1606.4 1.1X
+Native ORC Vectorized 869 913
56 1.2 828.5 2.2X
-OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
+OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 6.5.0-1016-azure
AMD EPYC 7763 64-Core Processor
Single Struct Column Scan with 300 Fields: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------
-Hive built-in ORC 9046 9092
66 0.1 8626.7 1.0X
-Native ORC MR 6233 6397
232 0.2 5944.5 1.5X
-Native ORC Vectorized 5378 5407
41 0.2 5129.1 1.7X
+Hive built-in ORC 5329 5433
148 0.2 5082.1 1.0X
+Native ORC MR 5514 5545
43 0.2 5259.0 1.0X
+Native ORC Vectorized 5450 5502
74 0.2 5197.3 1.0X
-OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
+OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 6.5.0-1016-azure
AMD EPYC 7763 64-Core Processor
Single Struct Column Scan with 600 Fields: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------
-Hive built-in ORC 22437 22692
361 0.0 21397.7 1.0X
-Native ORC MR 12624 12694
99 0.1 12039.1 1.8X
-Native ORC Vectorized 12680 12860
255 0.1 12092.1 1.8X
+Hive built-in ORC 11352 11377
36 0.1 10826.0 1.0X
+Native ORC MR 12781 12796
22 0.1 12188.7 0.9X
+Native ORC Vectorized 12673 12779
151 0.1 12085.7 0.9X
================================================================================================
Nested Struct scan
================================================================================================
-OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
+OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 6.5.0-1016-azure
AMD EPYC 7763 64-Core Processor
Nested Struct Scan with 10 Elements, 10 Fields: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------------
-Hive built-in ORC 3599 3644
64 0.3 3432.4 1.0X
-Native ORC MR 1912 1953
58 0.5 1823.8 1.9X
-Native ORC Vectorized 603 629
35 1.7 574.7 6.0X
+Hive built-in ORC 1793 1818
36 0.6 1710.3 1.0X
+Native ORC MR 2053 2069
22 0.5 1958.2 0.9X
+Native ORC Vectorized 539 569
37 1.9 513.7 3.3X
-OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
+OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 6.5.0-1016-azure
AMD EPYC 7763 64-Core Processor
Nested Struct Scan with 30 Elements, 10 Fields: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------------
-Hive built-in ORC 9213 9334
171 0.1 8786.1 1.0X
-Native ORC MR 4514 4529
20 0.2 4305.0 2.0X
-Native ORC Vectorized 1521 1550
41 0.7 1450.8 6.1X
+Hive built-in ORC 4477 4533
78 0.2 4270.0 1.0X
+Native ORC MR 4467 4484
25 0.2 4259.6 1.0X
+Native ORC Vectorized 1470 1490
29 0.7 1402.0 3.0X
-OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
+OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 6.5.0-1016-azure
AMD EPYC 7763 64-Core Processor
Nested Struct Scan with 10 Elements, 30 Fields: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------------
-Hive built-in ORC 9739 9740
2 0.1 9287.4 1.0X
-Native ORC MR 5553 5613
84 0.2 5296.0 1.8X
-Native ORC Vectorized 2090 2108
26 0.5 1993.1 4.7X
+Hive built-in ORC 4417 4430
19 0.2 4212.1 1.0X
+Native ORC MR 5039 5069
42 0.2 4805.9 0.9X
+Native ORC Vectorized 1726 1737
16 0.6 1645.8 2.6X
Review Comment:
Thanks for the information. @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
