gengliangwang commented on a change in pull request #24783:
[SPARK-27105][SQL][test-hadoop3.2] Optimize away exponential complexity in ORC
predicate conversion
URL: https://github.com/apache/spark/pull/24783#discussion_r291471197
##########
File path: sql/core/benchmarks/FilterPushdownBenchmark-results.txt
##########
@@ -2,669 +2,698 @@
Pushdown for many distinct value case
================================================================================================
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 0 string row (value IS NULL): Best/Avg Time(ms) Rate(M/s) Per
Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 11405 / 11485 1.4
725.1 1.0X
-Parquet Vectorized (Pushdown) 675 / 690 23.3
42.9 16.9X
-Native ORC Vectorized 7127 / 7170 2.2
453.1 1.6X
-Native ORC Vectorized (Pushdown) 519 / 541 30.3
33.0 22.0X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 0 string row ('7864320' < value < '7864320'): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 11457 / 11473 1.4
728.4 1.0X
-Parquet Vectorized (Pushdown) 656 / 686 24.0
41.7 17.5X
-Native ORC Vectorized 7328 / 7342 2.1
465.9 1.6X
-Native ORC Vectorized (Pushdown) 539 / 565 29.2
34.2 21.3X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 1 string row (value = '7864320'): Best/Avg Time(ms) Rate(M/s) Per
Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 11878 / 11888 1.3
755.2 1.0X
-Parquet Vectorized (Pushdown) 630 / 654 25.0
40.1 18.9X
-Native ORC Vectorized 7342 / 7362 2.1
466.8 1.6X
-Native ORC Vectorized (Pushdown) 519 / 537 30.3
33.0 22.9X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 1 string row (value <=> '7864320'): Best/Avg Time(ms) Rate(M/s)
Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 11423 / 11440 1.4
726.2 1.0X
-Parquet Vectorized (Pushdown) 625 / 643 25.2
39.7 18.3X
-Native ORC Vectorized 7315 / 7335 2.2
465.1 1.6X
-Native ORC Vectorized (Pushdown) 507 / 520 31.0
32.2 22.5X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 1 string row ('7864320' <= value <= '7864320'): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 11440 / 11478 1.4
727.3 1.0X
-Parquet Vectorized (Pushdown) 634 / 652 24.8
40.3 18.0X
-Native ORC Vectorized 7311 / 7324 2.2
464.8 1.6X
-Native ORC Vectorized (Pushdown) 517 / 548 30.4
32.8 22.1X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select all string rows (value IS NOT NULL): Best/Avg Time(ms) Rate(M/s)
Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 20750 / 20872 0.8
1319.3 1.0X
-Parquet Vectorized (Pushdown) 21002 / 21032 0.7
1335.3 1.0X
-Native ORC Vectorized 16714 / 16742 0.9
1062.6 1.2X
-Native ORC Vectorized (Pushdown) 16926 / 16965 0.9
1076.1 1.2X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 0 int row (value IS NULL): Best/Avg Time(ms) Rate(M/s) Per
Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 10510 / 10532 1.5
668.2 1.0X
-Parquet Vectorized (Pushdown) 642 / 665 24.5
40.8 16.4X
-Native ORC Vectorized 6609 / 6618 2.4
420.2 1.6X
-Native ORC Vectorized (Pushdown) 502 / 512 31.4
31.9 21.0X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 0 int row (7864320 < value < 7864320): Best/Avg Time(ms) Rate(M/s)
Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 10505 / 10514 1.5
667.9 1.0X
-Parquet Vectorized (Pushdown) 659 / 673 23.9
41.9 15.9X
-Native ORC Vectorized 6634 / 6641 2.4
421.8 1.6X
-Native ORC Vectorized (Pushdown) 513 / 526 30.7
32.6 20.5X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 1 int row (value = 7864320): Best/Avg Time(ms) Rate(M/s) Per
Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 10555 / 10570 1.5
671.1 1.0X
-Parquet Vectorized (Pushdown) 651 / 668 24.2
41.4 16.2X
-Native ORC Vectorized 6721 / 6728 2.3
427.3 1.6X
-Native ORC Vectorized (Pushdown) 508 / 519 31.0
32.3 20.8X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 1 int row (value <=> 7864320): Best/Avg Time(ms) Rate(M/s) Per
Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 10556 / 10566 1.5
671.1 1.0X
-Parquet Vectorized (Pushdown) 647 / 654 24.3
41.1 16.3X
-Native ORC Vectorized 6716 / 6728 2.3
427.0 1.6X
-Native ORC Vectorized (Pushdown) 510 / 521 30.9
32.4 20.7X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 1 int row (7864320 <= value <= 7864320): Best/Avg Time(ms) Rate(M/s)
Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 10556 / 10565 1.5
671.1 1.0X
-Parquet Vectorized (Pushdown) 649 / 654 24.2
41.3 16.3X
-Native ORC Vectorized 6700 / 6712 2.3
426.0 1.6X
-Native ORC Vectorized (Pushdown) 509 / 520 30.9
32.3 20.8X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 1 int row (7864319 < value < 7864321): Best/Avg Time(ms) Rate(M/s)
Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 10547 / 10566 1.5
670.5 1.0X
-Parquet Vectorized (Pushdown) 649 / 653 24.2
41.3 16.3X
-Native ORC Vectorized 6703 / 6713 2.3
426.2 1.6X
-Native ORC Vectorized (Pushdown) 510 / 520 30.8
32.5 20.7X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 10% int rows (value < 1572864): Best/Avg Time(ms) Rate(M/s) Per
Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 11478 / 11525 1.4
729.7 1.0X
-Parquet Vectorized (Pushdown) 2576 / 2587 6.1
163.8 4.5X
-Native ORC Vectorized 7633 / 7657 2.1
485.3 1.5X
-Native ORC Vectorized (Pushdown) 2076 / 2096 7.6
132.0 5.5X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 50% int rows (value < 7864320): Best/Avg Time(ms) Rate(M/s) Per
Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 14785 / 14802 1.1
940.0 1.0X
-Parquet Vectorized (Pushdown) 9971 / 9977 1.6
633.9 1.5X
-Native ORC Vectorized 11082 / 11107 1.4
704.6 1.3X
-Native ORC Vectorized (Pushdown) 8061 / 8073 2.0
512.5 1.8X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 90% int rows (value < 14155776): Best/Avg Time(ms) Rate(M/s) Per
Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 18174 / 18214 0.9
1155.5 1.0X
-Parquet Vectorized (Pushdown) 17387 / 17403 0.9
1105.5 1.0X
-Native ORC Vectorized 14465 / 14492 1.1
919.7 1.3X
-Native ORC Vectorized (Pushdown) 14024 / 14041 1.1
891.6 1.3X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select all int rows (value IS NOT NULL): Best/Avg Time(ms) Rate(M/s) Per
Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 19004 / 19014 0.8
1208.2 1.0X
-Parquet Vectorized (Pushdown) 19219 / 19232 0.8
1221.9 1.0X
-Native ORC Vectorized 15266 / 15290 1.0
970.6 1.2X
-Native ORC Vectorized (Pushdown) 15469 / 15482 1.0
983.5 1.2X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select all int rows (value > -1): Best/Avg Time(ms) Rate(M/s) Per
Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 19036 / 19052 0.8
1210.3 1.0X
-Parquet Vectorized (Pushdown) 19287 / 19306 0.8
1226.2 1.0X
-Native ORC Vectorized 15311 / 15371 1.0
973.5 1.2X
-Native ORC Vectorized (Pushdown) 15517 / 15590 1.0
986.5 1.2X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select all int rows (value != -1): Best/Avg Time(ms) Rate(M/s) Per
Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 19072 / 19102 0.8
1212.6 1.0X
-Parquet Vectorized (Pushdown) 19288 / 19318 0.8
1226.3 1.0X
-Native ORC Vectorized 15277 / 15293 1.0
971.3 1.2X
-Native ORC Vectorized (Pushdown) 15479 / 15499 1.0
984.1 1.2X
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 0 string row (value IS NULL): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 8100 8441
338 1.9 515.0 1.0X
+Parquet Vectorized (Pushdown) 508 554
30 31.0 32.3 16.0X
+Native ORC Vectorized 5747 5950
195 2.7 365.4 1.4X
+Native ORC Vectorized (Pushdown) 585 699
76 26.9 37.2 13.8X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 0 string row ('7864320' < value < '7864320'): Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 8001 8183
142 2.0 508.7 1.0X
+Parquet Vectorized (Pushdown) 498 536
37 31.6 31.7 16.1X
+Native ORC Vectorized 5907 6064
160 2.7 375.6 1.4X
+Native ORC Vectorized (Pushdown) 542 593
34 29.0 34.5 14.8X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 1 string row (value = '7864320'): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 7999 8356
247 2.0 508.6 1.0X
+Parquet Vectorized (Pushdown) 471 484
11 33.4 30.0 17.0X
+Native ORC Vectorized 5856 5921
45 2.7 372.3 1.4X
+Native ORC Vectorized (Pushdown) 531 541
17 29.6 33.7 15.1X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 1 string row (value <=> '7864320'): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 8061 8398
451 2.0 512.5 1.0X
+Parquet Vectorized (Pushdown) 462 479
17 34.1 29.4 17.5X
+Native ORC Vectorized 5778 5843
86 2.7 367.4 1.4X
+Native ORC Vectorized (Pushdown) 620 684
39 25.4 39.4 13.0X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 1 string row ('7864320' <= value <= '7864320'): Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 8056 8216
163 2.0 512.2 1.0X
+Parquet Vectorized (Pushdown) 465 488
19 33.8 29.6 17.3X
+Native ORC Vectorized 5793 5844
68 2.7 368.3 1.4X
+Native ORC Vectorized (Pushdown) 538 579
26 29.2 34.2 15.0X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select all string rows (value IS NOT NULL): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 14531 14600
52 1.1 923.8 1.0X
+Parquet Vectorized (Pushdown) 15076 15285
189 1.0 958.5 1.0X
+Native ORC Vectorized 12828 13009
233 1.2 815.6 1.1X
+Native ORC Vectorized (Pushdown) 13072 13253
201 1.2 831.1 1.1X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 0 int row (value IS NULL): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 7431 7880
418 2.1 472.4 1.0X
+Parquet Vectorized (Pushdown) 453 456
2 34.7 28.8 16.4X
+Native ORC Vectorized 5157 5669
447 3.0 327.9 1.4X
+Native ORC Vectorized (Pushdown) 559 572
14 28.1 35.5 13.3X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 0 int row (7864320 < value < 7864320): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 7441 7552
143 2.1 473.1 1.0X
+Parquet Vectorized (Pushdown) 466 470
5 33.8 29.6 16.0X
+Native ORC Vectorized 5238 5307
51 3.0 333.0 1.4X
+Native ORC Vectorized (Pushdown) 557 571
15 28.3 35.4 13.4X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 1 int row (value = 7864320): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 7447 7502
33 2.1 473.5 1.0X
+Parquet Vectorized (Pushdown) 460 465
6 34.2 29.2 16.2X
+Native ORC Vectorized 5212 5320
88 3.0 331.4 1.4X
+Native ORC Vectorized (Pushdown) 647 764
120 24.3 41.1 11.5X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 1 int row (value <=> 7864320): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 7573 7877
429 2.1 481.5 1.0X
+Parquet Vectorized (Pushdown) 468 474
4 33.6 29.8 16.2X
+Native ORC Vectorized 5338 5471
122 2.9 339.4 1.4X
+Native ORC Vectorized (Pushdown) 560 595
25 28.1 35.6 13.5X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 1 int row (7864320 <= value <= 7864320): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 7589 7680
102 2.1 482.5 1.0X
+Parquet Vectorized (Pushdown) 471 472
2 33.4 29.9 16.1X
+Native ORC Vectorized 5380 5764
344 2.9 342.0 1.4X
+Native ORC Vectorized (Pushdown) 576 594
16 27.3 36.6 13.2X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 1 int row (7864319 < value < 7864321): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 7540 7585
32 2.1 479.4 1.0X
+Parquet Vectorized (Pushdown) 469 471
2 33.5 29.8 16.1X
+Native ORC Vectorized 5437 5499
73 2.9 345.7 1.4X
+Native ORC Vectorized (Pushdown) 653 705
34 24.1 41.5 11.6X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 10% int rows (value < 1572864): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 8989 9867
981 1.7 571.5 1.0X
+Parquet Vectorized (Pushdown) 2197 2224
17 7.2 139.7 4.1X
+Native ORC Vectorized 6612 7005
454 2.4 420.4 1.4X
+Native ORC Vectorized (Pushdown) 2060 2136
66 7.6 131.0 4.4X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 50% int rows (value < 7864320): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 11024 11815
959 1.4 700.9 1.0X
+Parquet Vectorized (Pushdown) 7368 7556
325 2.1 468.4 1.5X
+Native ORC Vectorized 9328 9436
120 1.7 593.0 1.2X
+Native ORC Vectorized (Pushdown) 6613 6662
35 2.4 420.4 1.7X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 90% int rows (value < 14155776): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 13358 13383
16 1.2 849.3 1.0X
+Parquet Vectorized (Pushdown) 12788 13091
336 1.2 813.0 1.0X
+Native ORC Vectorized 11443 12081
574 1.4 727.5 1.2X
+Native ORC Vectorized (Pushdown) 11158 11180
16 1.4 709.4 1.2X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select all int rows (value IS NOT NULL): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 14090 14979
693 1.1 895.8 1.0X
+Parquet Vectorized (Pushdown) 14540 15283
474 1.1 924.5 1.0X
+Native ORC Vectorized 12602 12754
280 1.2 801.2 1.1X
+Native ORC Vectorized (Pushdown) 12633 13019
367 1.2 803.2 1.1X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select all int rows (value > -1): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 14068 14383
217 1.1 894.4 1.0X
+Parquet Vectorized (Pushdown) 14228 15038
781 1.1 904.6 1.0X
+Native ORC Vectorized 12845 13208
413 1.2 816.7 1.1X
+Native ORC Vectorized (Pushdown) 12985 13223
190 1.2 825.6 1.1X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select all int rows (value != -1): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 14607 15004
284 1.1 928.7 1.0X
+Parquet Vectorized (Pushdown) 15216 15625
670 1.0 967.4 1.0X
+Native ORC Vectorized 12779 13432
814 1.2 812.5 1.1X
+Native ORC Vectorized (Pushdown) 13001 13294
232 1.2 826.6 1.1X
================================================================================================
Pushdown for few distinct value case (use dictionary encoding)
================================================================================================
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 0 distinct string row (value IS NULL): Best/Avg Time(ms) Rate(M/s)
Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 10250 / 10274 1.5
651.7 1.0X
-Parquet Vectorized (Pushdown) 571 / 576 27.5
36.3 17.9X
-Native ORC Vectorized 8651 / 8660 1.8
550.0 1.2X
-Native ORC Vectorized (Pushdown) 909 / 933 17.3
57.8 11.3X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 0 distinct string row ('100' < value < '100'): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 10420 / 10426 1.5
662.5 1.0X
-Parquet Vectorized (Pushdown) 574 / 579 27.4
36.5 18.2X
-Native ORC Vectorized 8973 / 8982 1.8
570.5 1.2X
-Native ORC Vectorized (Pushdown) 916 / 955 17.2
58.2 11.4X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 1 distinct string row (value = '100'): Best/Avg Time(ms) Rate(M/s)
Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 10428 / 10441 1.5
663.0 1.0X
-Parquet Vectorized (Pushdown) 789 / 809 19.9
50.2 13.2X
-Native ORC Vectorized 9042 / 9055 1.7
574.9 1.2X
-Native ORC Vectorized (Pushdown) 1130 / 1145 13.9
71.8 9.2X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 1 distinct string row (value <=> '100'): Best/Avg Time(ms) Rate(M/s)
Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 10402 / 10416 1.5
661.3 1.0X
-Parquet Vectorized (Pushdown) 791 / 806 19.9
50.3 13.2X
-Native ORC Vectorized 9042 / 9055 1.7
574.9 1.2X
-Native ORC Vectorized (Pushdown) 1112 / 1145 14.1
70.7 9.4X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 1 distinct string row ('100' <= value <= '100'): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 10548 / 10563 1.5
670.6 1.0X
-Parquet Vectorized (Pushdown) 790 / 796 19.9
50.2 13.4X
-Native ORC Vectorized 9144 / 9153 1.7
581.3 1.2X
-Native ORC Vectorized (Pushdown) 1117 / 1148 14.1
71.0 9.4X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select all distinct string rows (value IS NOT NULL): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 20445 / 20469 0.8
1299.8 1.0X
-Parquet Vectorized (Pushdown) 20686 / 20699 0.8
1315.2 1.0X
-Native ORC Vectorized 18851 / 18953 0.8
1198.5 1.1X
-Native ORC Vectorized (Pushdown) 19255 / 19268 0.8
1224.2 1.1X
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 0 distinct string row (value IS NULL): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 7446 7615
210 2.1 473.4 1.0X
+Parquet Vectorized (Pushdown) 422 458
73 37.3 26.8 17.7X
+Native ORC Vectorized 6842 7387
585 2.3 435.0 1.1X
+Native ORC Vectorized (Pushdown) 1313 1437
145 12.0 83.5 5.7X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 0 distinct string row ('100' < value < '100'): Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 7669 8303
494 2.1 487.6 1.0X
+Parquet Vectorized (Pushdown) 426 434
9 36.9 27.1 18.0X
+Native ORC Vectorized 7791 8283
318 2.0 495.3 1.0X
+Native ORC Vectorized (Pushdown) 1368 1389
21 11.5 87.0 5.6X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 1 distinct string row (value = '100'): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 7680 7743
97 2.0 488.3 1.0X
+Parquet Vectorized (Pushdown) 595 605
10 26.4 37.8 12.9X
+Native ORC Vectorized 7314 7785
320 2.2 465.0 1.1X
+Native ORC Vectorized (Pushdown) 1646 1847
161 9.6 104.6 4.7X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 1 distinct string row (value <=> '100'): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 8254 8356
111 1.9 524.7 1.0X
+Parquet Vectorized (Pushdown) 640 649
9 24.6 40.7 12.9X
+Native ORC Vectorized 7206 7780
473 2.2 458.2 1.1X
+Native ORC Vectorized (Pushdown) 1706 1795
57 9.2 108.5 4.8X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 1 distinct string row ('100' <= value <= '100'): Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 8482 9206
752 1.9 539.3 1.0X
+Parquet Vectorized (Pushdown) 623 669
31 25.2 39.6 13.6X
+Native ORC Vectorized 7762 7962
272 2.0 493.5 1.1X
+Native ORC Vectorized (Pushdown) 1523 1578
36 10.3 96.8 5.6X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select all distinct string rows (value IS NOT NULL): Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 16069 16332
214 1.0 1021.7 1.0X
+Parquet Vectorized (Pushdown) 16387 16448
57 1.0 1041.8 1.0X
+Native ORC Vectorized 15257 15783
813 1.0 970.0 1.1X
+Native ORC Vectorized (Pushdown) 17282 18955
NaN 0.9 1098.7 0.9X
================================================================================================
Pushdown benchmark for StringStartsWith
================================================================================================
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-StringStartsWith filter: (value like '10%'): Best/Avg Time(ms) Rate(M/s)
Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 14265 / 15213 1.1
907.0 1.0X
-Parquet Vectorized (Pushdown) 4228 / 4870 3.7
268.8 3.4X
-Native ORC Vectorized 10116 / 10977 1.6
643.2 1.4X
-Native ORC Vectorized (Pushdown) 10653 / 11376 1.5
677.3 1.3X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-StringStartsWith filter: (value like '1000%'): Best/Avg Time(ms) Rate(M/s)
Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 11499 / 11539 1.4
731.1 1.0X
-Parquet Vectorized (Pushdown) 669 / 672 23.5
42.5 17.2X
-Native ORC Vectorized 7343 / 7363 2.1
466.8 1.6X
-Native ORC Vectorized (Pushdown) 7559 / 7568 2.1
480.6 1.5X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-StringStartsWith filter: (value like '786432%'): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 11463 / 11468 1.4
728.8 1.0X
-Parquet Vectorized (Pushdown) 647 / 651 24.3
41.1 17.7X
-Native ORC Vectorized 7322 / 7338 2.1
465.5 1.6X
-Native ORC Vectorized (Pushdown) 7533 / 7544 2.1
478.9 1.5X
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+StringStartsWith filter: (value like '10%'): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 10929 11386
525 1.4 694.8 1.0X
+Parquet Vectorized (Pushdown) 3649 3934
254 4.3 232.0 3.0X
+Native ORC Vectorized 8790 9574
1093 1.8 558.8 1.2X
+Native ORC Vectorized (Pushdown) 8094 9051
1366 1.9 514.6 1.4X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+StringStartsWith filter: (value like '1000%'): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 8090 8297
283 1.9 514.4 1.0X
+Parquet Vectorized (Pushdown) 472 475
4 33.3 30.0 17.1X
+Native ORC Vectorized 6023 6123
78 2.6 383.0 1.3X
+Native ORC Vectorized (Pushdown) 6291 6696
352 2.5 400.0 1.3X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+StringStartsWith filter: (value like '786432%'): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 8210 8447
245 1.9 522.0 1.0X
+Parquet Vectorized (Pushdown) 463 466
4 34.0 29.4 17.7X
+Native ORC Vectorized 5822 5867
28 2.7 370.1 1.4X
+Native ORC Vectorized (Pushdown) 6122 6146
15 2.6 389.2 1.3X
================================================================================================
Pushdown benchmark for decimal
================================================================================================
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 1 decimal(9, 2) row (value = 7864320): Best/Avg Time(ms) Rate(M/s)
Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 5543 / 5564 2.8
352.4 1.0X
-Parquet Vectorized (Pushdown) 168 / 174 93.7
10.7 33.0X
-Native ORC Vectorized 4992 / 5052 3.2
317.4 1.1X
-Native ORC Vectorized (Pushdown) 840 / 850 18.7
53.4 6.6X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 10% decimal(9, 2) rows (value < 1572864): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 7312 / 7358 2.2
464.9 1.0X
-Parquet Vectorized (Pushdown) 3008 / 3078 5.2
191.2 2.4X
-Native ORC Vectorized 6775 / 6798 2.3
430.7 1.1X
-Native ORC Vectorized (Pushdown) 6819 / 6832 2.3
433.5 1.1X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 50% decimal(9, 2) rows (value < 7864320): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 13232 / 13241 1.2
841.3 1.0X
-Parquet Vectorized (Pushdown) 12555 / 12569 1.3
798.2 1.1X
-Native ORC Vectorized 12597 / 12627 1.2
800.9 1.1X
-Native ORC Vectorized (Pushdown) 12677 / 12711 1.2
806.0 1.0X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 90% decimal(9, 2) rows (value < 14155776): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 14725 / 14729 1.1
936.2 1.0X
-Parquet Vectorized (Pushdown) 14781 / 14800 1.1
939.7 1.0X
-Native ORC Vectorized 15360 / 15453 1.0
976.5 1.0X
-Native ORC Vectorized (Pushdown) 15444 / 15466 1.0
981.9 1.0X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 1 decimal(18, 2) row (value = 7864320): Best/Avg Time(ms) Rate(M/s)
Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 5746 / 5763 2.7
365.3 1.0X
-Parquet Vectorized (Pushdown) 166 / 169 94.8
10.6 34.6X
-Native ORC Vectorized 5007 / 5023 3.1
318.3 1.1X
-Native ORC Vectorized (Pushdown) 2629 / 2640 6.0
167.1 2.2X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 10% decimal(18, 2) rows (value < 1572864): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 6827 / 6864 2.3
434.0 1.0X
-Parquet Vectorized (Pushdown) 1809 / 1827 8.7
115.0 3.8X
-Native ORC Vectorized 6287 / 6296 2.5
399.7 1.1X
-Native ORC Vectorized (Pushdown) 6364 / 6377 2.5
404.6 1.1X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 50% decimal(18, 2) rows (value < 7864320): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 11315 / 11342 1.4
719.4 1.0X
-Parquet Vectorized (Pushdown) 8431 / 8450 1.9
536.0 1.3X
-Native ORC Vectorized 11591 / 11611 1.4
736.9 1.0X
-Native ORC Vectorized (Pushdown) 11424 / 11475 1.4
726.3 1.0X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 90% decimal(18, 2) rows (value < 14155776): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 15703 / 15712 1.0
998.4 1.0X
-Parquet Vectorized (Pushdown) 14982 / 15009 1.0
952.5 1.0X
-Native ORC Vectorized 16887 / 16955 0.9
1073.7 0.9X
-Native ORC Vectorized (Pushdown) 16518 / 16530 1.0
1050.2 1.0X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 1 decimal(38, 2) row (value = 7864320): Best/Avg Time(ms) Rate(M/s)
Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 8101 / 8130 1.9
515.1 1.0X
-Parquet Vectorized (Pushdown) 184 / 187 85.6
11.7 44.1X
-Native ORC Vectorized 4998 / 5027 3.1
317.8 1.6X
-Native ORC Vectorized (Pushdown) 165 / 168 95.6
10.5 49.2X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 10% decimal(38, 2) rows (value < 1572864): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 9405 / 9447 1.7
597.9 1.0X
-Parquet Vectorized (Pushdown) 2269 / 2275 6.9
144.2 4.1X
-Native ORC Vectorized 6167 / 6203 2.6
392.1 1.5X
-Native ORC Vectorized (Pushdown) 1783 / 1787 8.8
113.3 5.3X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 50% decimal(38, 2) rows (value < 7864320): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 14700 / 14707 1.1
934.6 1.0X
-Parquet Vectorized (Pushdown) 10699 / 10712 1.5
680.2 1.4X
-Native ORC Vectorized 10687 / 10703 1.5
679.5 1.4X
-Native ORC Vectorized (Pushdown) 8364 / 8415 1.9
531.8 1.8X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 90% decimal(38, 2) rows (value < 14155776): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 19780 / 19894 0.8
1257.6 1.0X
-Parquet Vectorized (Pushdown) 19003 / 19025 0.8
1208.1 1.0X
-Native ORC Vectorized 15385 / 15404 1.0
978.2 1.3X
-Native ORC Vectorized (Pushdown) 15032 / 15060 1.0
955.7 1.3X
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 1 decimal(9, 2) row (value = 7864320): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 3863 4034
224 4.1 245.6 1.0X
+Parquet Vectorized (Pushdown) 127 139
12 123.4 8.1 30.3X
+Native ORC Vectorized 3905 4028
83 4.0 248.3 1.0X
+Native ORC Vectorized (Pushdown) 742 776
27 21.2 47.2 5.2X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 10% decimal(9, 2) rows (value < 1572864): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 5416 5640
149 2.9 344.3 1.0X
+Parquet Vectorized (Pushdown) 2444 2492
50 6.4 155.4 2.2X
+Native ORC Vectorized 5364 5392
40 2.9 341.1 1.0X
+Native ORC Vectorized (Pushdown) 5442 5487
56 2.9 346.0 1.0X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 50% decimal(9, 2) rows (value < 7864320): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 10072 10157
76 1.6 640.4 1.0X
+Parquet Vectorized (Pushdown) 9638 9889
184 1.6 612.7 1.0X
+Native ORC Vectorized 10192 10224
33 1.5 648.0 1.0X
+Native ORC Vectorized (Pushdown) 10287 10381
173 1.5 654.1 1.0X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 90% decimal(9, 2) rows (value < 14155776): Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 11307 11681
311 1.4 718.9 1.0X
+Parquet Vectorized (Pushdown) 11409 11509
94 1.4 725.4 1.0X
+Native ORC Vectorized 11613 12703
NaN 1.4 738.3 1.0X
+Native ORC Vectorized (Pushdown) 11938 12334
525 1.3 759.0 0.9X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 1 decimal(18, 2) row (value = 7864320): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 4109 4344
168 3.8 261.2 1.0X
+Parquet Vectorized (Pushdown) 134 150
13 117.7 8.5 30.7X
+Native ORC Vectorized 3927 4163
192 4.0 249.7 1.0X
+Native ORC Vectorized (Pushdown) 2159 2337
120 7.3 137.3 1.9X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 10% decimal(18, 2) rows (value < 1572864): Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 5049 5301
183 3.1 321.0 1.0X
+Parquet Vectorized (Pushdown) 1379 1467
89 11.4 87.7 3.7X
+Native ORC Vectorized 4727 5344
637 3.3 300.5 1.1X
+Native ORC Vectorized (Pushdown) 4862 5101
216 3.2 309.1 1.0X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 50% decimal(18, 2) rows (value < 7864320): Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 7877 8107
216 2.0 500.8 1.0X
+Parquet Vectorized (Pushdown) 5771 5836
88 2.7 366.9 1.4X
+Native ORC Vectorized 7684 7704
18 2.0 488.5 1.0X
+Native ORC Vectorized (Pushdown) 7976 8248
262 2.0 507.1 1.0X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 90% decimal(18, 2) rows (value < 14155776): Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 10875 11744
792 1.4 691.4 1.0X
+Parquet Vectorized (Pushdown) 10535 11057
418 1.5 669.8 1.0X
+Native ORC Vectorized 10825 11382
760 1.5 688.2 1.0X
+Native ORC Vectorized (Pushdown) 11319 11950
701 1.4 719.6 1.0X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 1 decimal(38, 2) row (value = 7864320): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 5764 6018
262 2.7 366.5 1.0X
+Parquet Vectorized (Pushdown) 138 148
8 114.0 8.8 41.8X
+Native ORC Vectorized 4030 4068
42 3.9 256.2 1.4X
+Native ORC Vectorized (Pushdown) 174 191
24 90.5 11.0 33.2X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 10% decimal(38, 2) rows (value < 1572864): Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 6790 7280
466 2.3 431.7 1.0X
+Parquet Vectorized (Pushdown) 1677 1802
197 9.4 106.6 4.0X
+Native ORC Vectorized 5077 5429
333 3.1 322.8 1.3X
+Native ORC Vectorized (Pushdown) 1510 1645
213 10.4 96.0 4.5X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 50% decimal(38, 2) rows (value < 7864320): Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 10372 11031
543 1.5 659.4 1.0X
+Parquet Vectorized (Pushdown) 7581 7959
330 2.1 482.0 1.4X
+Native ORC Vectorized 8409 9048
680 1.9 534.6 1.2X
+Native ORC Vectorized (Pushdown) 6551 6564
16 2.4 416.5 1.6X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.14.5
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
+Select 90% decimal(38, 2) rows (value < 14155776): Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+Parquet Vectorized 13819 14830
924 1.1 878.6 1.0X
+Parquet Vectorized (Pushdown) 14227 14856
440 1.1 904.5 1.0X
+Native ORC Vectorized 12257 12701
481 1.3 779.3 1.1X
+Native ORC Vectorized (Pushdown) 12108 12785
863 1.3 769.8 1.1X
================================================================================================
Pushdown benchmark for InSet -> InFilters
================================================================================================
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-InSet -> InFilters (values count: 5, distribution: 10): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 10521 / 10534 1.5
668.9 1.0X
-Parquet Vectorized (Pushdown) 677 / 691 23.2
43.1 15.5X
-Native ORC Vectorized 6768 / 6776 2.3
430.3 1.6X
-Native ORC Vectorized (Pushdown) 501 / 512 31.4
31.8 21.0X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-InSet -> InFilters (values count: 5, distribution: 50): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 10531 / 10538 1.5
669.5 1.0X
-Parquet Vectorized (Pushdown) 677 / 718 23.2
43.0 15.6X
-Native ORC Vectorized 6765 / 6773 2.3
430.1 1.6X
-Native ORC Vectorized (Pushdown) 499 / 507 31.5
31.7 21.1X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-InSet -> InFilters (values count: 5, distribution: 90): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 10540 / 10553 1.5
670.1 1.0X
-Parquet Vectorized (Pushdown) 678 / 710 23.2
43.1 15.5X
-Native ORC Vectorized 6787 / 6794 2.3
431.5 1.6X
-Native ORC Vectorized (Pushdown) 501 / 509 31.4
31.9 21.0X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-InSet -> InFilters (values count: 10, distribution: 10): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 10551 / 10559 1.5
670.8 1.0X
-Parquet Vectorized (Pushdown) 703 / 708 22.4
44.7 15.0X
-Native ORC Vectorized 6791 / 6802 2.3
431.7 1.6X
-Native ORC Vectorized (Pushdown) 519 / 526 30.3
33.0 20.3X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-InSet -> InFilters (values count: 10, distribution: 50): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------
-Parquet Vectorized 10561 / 10565 1.5
671.4 1.0X
-Parquet Vectorized (Pushdown) 711 / 716 22.1
45.2 14.9X
-Native ORC Vectorized 6791 / 6806 2.3
431.8 1.6X
-Native ORC Ve
Review comment:
All the benchmark results of `FilterPushdownBenchmark` will be in this file,
unless we move the new benchmarks to another micro benchmark.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
