Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21603#discussion_r200559860
--- Diff: sql/core/benchmarks/FilterPushdownBenchmark-results.txt ---
@@ -417,120 +417,120 @@ Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 5, distribution: 10): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 7477 / 7587 2.1
475.4 1.0X
-Parquet Vectorized (Pushdown) 7862 / 8346 2.0
499.9 1.0X
-Native ORC Vectorized 6447 / 7021 2.4
409.9 1.2X
-Native ORC Vectorized (Pushdown) 983 / 1003 16.0
62.5 7.6X
+Parquet Vectorized 7993 / 8104 2.0
508.2 1.0X
+Parquet Vectorized (Pushdown) 507 / 532 31.0
32.2 15.8X
+Native ORC Vectorized 6922 / 7163 2.3
440.1 1.2X
+Native ORC Vectorized (Pushdown) 1017 / 1058 15.5
64.6 7.9X
Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6
Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 5, distribution: 50): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 7107 / 7290 2.2
451.9 1.0X
-Parquet Vectorized (Pushdown) 7196 / 7258 2.2
457.5 1.0X
-Native ORC Vectorized 6102 / 6222 2.6
388.0 1.2X
-Native ORC Vectorized (Pushdown) 926 / 958 17.0
58.9 7.7X
+Parquet Vectorized 7855 / 7963 2.0
499.4 1.0X
+Parquet Vectorized (Pushdown) 503 / 516 31.3
32.0 15.6X
+Native ORC Vectorized 6825 / 6954 2.3
433.9 1.2X
+Native ORC Vectorized (Pushdown) 1019 / 1044 15.4
64.8 7.7X
Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6
Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 5, distribution: 90): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 7374 / 7692 2.1
468.8 1.0X
-Parquet Vectorized (Pushdown) 7771 / 7848 2.0
494.1 0.9X
-Native ORC Vectorized 6184 / 6356 2.5
393.2 1.2X
-Native ORC Vectorized (Pushdown) 920 / 963 17.1
58.5 8.0X
+Parquet Vectorized 7858 / 7928 2.0
499.6 1.0X
+Parquet Vectorized (Pushdown) 490 / 519 32.1
31.1 16.0X
+Native ORC Vectorized 7079 / 7966 2.2
450.1 1.1X
+Native ORC Vectorized (Pushdown) 1276 / 1673 12.3
81.1 6.2X
Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6
Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 10, distribution: 10): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 7073 / 7326 2.2
449.7 1.0X
-Parquet Vectorized (Pushdown) 7304 / 7647 2.2
464.4 1.0X
-Native ORC Vectorized 6222 / 6579 2.5
395.6 1.1X
-Native ORC Vectorized (Pushdown) 958 / 994 16.4
60.9 7.4X
+Parquet Vectorized 8007 / 11155 2.0
509.0 1.0X
+Parquet Vectorized (Pushdown) 519 / 540 30.3
33.0 15.4X
+Native ORC Vectorized 6848 / 7072 2.3
435.4 1.2X
+Native ORC Vectorized (Pushdown) 1026 / 1050 15.3
65.2 7.8X
Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6
Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 10, distribution: 50): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 7121 / 7501 2.2
452.7 1.0X
-Parquet Vectorized (Pushdown) 7751 / 8334 2.0
492.8 0.9X
-Native ORC Vectorized 6225 / 6680 2.5
395.8 1.1X
-Native ORC Vectorized (Pushdown) 998 / 1020 15.8
63.5 7.1X
+Parquet Vectorized 7876 / 7956 2.0
500.7 1.0X
+Parquet Vectorized (Pushdown) 521 / 535 30.2
33.1 15.1X
+Native ORC Vectorized 7051 / 7368 2.2
448.3 1.1X
+Native ORC Vectorized (Pushdown) 1014 / 1035 15.5
64.5 7.8X
Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6
Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 10, distribution: 90): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 7157 / 7399 2.2
455.1 1.0X
-Parquet Vectorized (Pushdown) 7806 / 7911 2.0
496.3 0.9X
-Native ORC Vectorized 6548 / 6720 2.4
416.3 1.1X
-Native ORC Vectorized (Pushdown) 1016 / 1050 15.5
64.6 7.0X
+Parquet Vectorized 7897 / 8229 2.0
502.1 1.0X
+Parquet Vectorized (Pushdown) 513 / 530 30.7
32.6 15.4X
+Native ORC Vectorized 6730 / 6990 2.3
427.9 1.2X
+Native ORC Vectorized (Pushdown) 1003 / 1036 15.7
63.8 7.9X
Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6
Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 50, distribution: 10): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 7662 / 7805 2.1
487.1 1.0X
-Parquet Vectorized (Pushdown) 7590 / 7861 2.1
482.5 1.0X
-Native ORC Vectorized 6840 / 8073 2.3
434.9 1.1X
-Native ORC Vectorized (Pushdown) 1041 / 1075 15.1
66.2 7.4X
+Parquet Vectorized 7967 / 8175 2.0
506.5 1.0X
+Parquet Vectorized (Pushdown) 8155 / 8434 1.9
518.5 1.0X
--- End diff --
I think this benchmark numbers are more meaningful if we show IN predicate
is pushdown or not.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
