Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22501#discussion_r226439834
--- Diff: sql/core/benchmarks/WideSchemaBenchmark-results.txt ---
@@ -1,117 +1,145 @@
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_92-b14 on Mac OS X 10.11.6
-Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz
+================================================================================================
+parsing large select expressions
+================================================================================================
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
parsing large select: Best/Avg Time(ms) Rate(M/s)
Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-1 select expressions 2 / 4 0.0
2050147.0 1.0X
-100 select expressions 6 / 7 0.0
6123412.0 0.3X
-2500 select expressions 135 / 141 0.0
134623148.0 0.0X
+1 select expressions 2 / 4 0.0
1934953.0 1.0X
+100 select expressions 4 / 5 0.0
3659399.0 0.5X
+2500 select expressions 68 / 76 0.0
68278937.0 0.0X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_92-b14 on Mac OS X 10.11.6
-Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz
+================================================================================================
+many column field read and write
+================================================================================================
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
many column field r/w: Best/Avg Time(ms) Rate(M/s)
Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-1 cols x 100000 rows (read in-mem) 16 / 18 6.3
158.6 1.0X
-1 cols x 100000 rows (exec in-mem) 17 / 19 6.0
166.7 1.0X
-1 cols x 100000 rows (read parquet) 24 / 26 4.3
235.1 0.7X
-1 cols x 100000 rows (write parquet) 81 / 85 1.2
811.3 0.2X
-100 cols x 1000 rows (read in-mem) 17 / 19 6.0
166.2 1.0X
-100 cols x 1000 rows (exec in-mem) 25 / 27 4.0
249.2 0.6X
-100 cols x 1000 rows (read parquet) 23 / 25 4.4
226.0 0.7X
-100 cols x 1000 rows (write parquet) 83 / 87 1.2
831.0 0.2X
-2500 cols x 40 rows (read in-mem) 132 / 137 0.8
1322.9 0.1X
-2500 cols x 40 rows (exec in-mem) 326 / 330 0.3
3260.6 0.0X
-2500 cols x 40 rows (read parquet) 831 / 839 0.1
8305.8 0.0X
-2500 cols x 40 rows (write parquet) 237 / 245 0.4
2372.6 0.1X
-
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_92-b14 on Mac OS X 10.11.6
-Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz
+1 cols x 100000 rows (read in-mem) 22 / 25 4.6
219.4 1.0X
+1 cols x 100000 rows (exec in-mem) 22 / 28 4.5
223.8 1.0X
+1 cols x 100000 rows (read parquet) 45 / 49 2.2
449.6 0.5X
+1 cols x 100000 rows (write parquet) 204 / 223 0.5
2044.4 0.1X
--- End diff --
This might be a little regression on Parquet writer from Spark 2.1.0
(SPARK-17335).
cc @cloud-fan and @gatorsmile , @rdblue
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
