dongjoon-hyun commented on a change in pull request #23964: [SPARK-26975][SQL] Support nested-column pruning over limit/sample/repartition URL: https://github.com/apache/spark/pull/23964#discussion_r262661446
########## File path: sql/core/benchmarks/ParquetNestedSchemaPruningBenchmark-results.txt ########## @@ -9,19 +9,19 @@ Selection: Best Time(ms) Avg Time(ms) Stdev(m Top-level column 88 114 16 11.4 87.5 1.0X Nested column 201 223 27 5.0 200.5 0.4X -Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3 -Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz +Java HotSpot(TM) 64-Bit Server VM 1.8.0_201-b09 on Mac OS X 10.14.3 +Intel(R) Core(TM) i9-8950HK CPU @ 2.90GHz Limiting: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ -Top-level column 263 315 36 3.8 263.2 1.0X -Nested column 2111 2622 613 0.5 2111.1 0.1X +Top-level column 230 236 4 4.4 229.5 1.0X +Nested column 356 367 9 2.8 355.9 0.6X -Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3 -Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz +Java HotSpot(TM) 64-Bit Server VM 1.8.0_201-b09 on Mac OS X 10.14.3 +Intel(R) Core(TM) i9-8950HK CPU @ 2.90GHz Repartitioning: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ -Top-level column 222 250 34 4.5 222.2 1.0X -Nested column 2084 2339 266 0.5 2084.2 0.1X +Top-level column 197 201 6 5.1 196.6 1.0X +Nested column 337 345 7 3.0 336.6 0.6X Review comment: This is a result from the master branch a few minutes ago. I intentionally excluded the other irrelevant changes and kept the relevant ones like above. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
