Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/13701 @viirya Yeah, it is not easy to get a full performance picture. I do not know how Spark community did it in the past. When I working for the mainframe team, we had dedicated PQAs for measuring the performance-related topics. We need to present the performance to the customers and give them guides about it. How about doing one more test? Changing the way we generate the data set. Insert into the parquet table row by row. Then, you will get many many tiny parquet files. This scenario is not rare in Big Data. I am not sure what is the performance gain and loss in all your test cases.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org