Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/13701
@viirya Yeah, it is not easy to get a full performance picture. I do not
know how Spark community did it in the past. When I working for the mainframe
team, we had dedicated PQAs for measuring the performance-related topics. We
need to present the performance to the customers and give them guides about it.
How about doing one more test? Changing the way we generate the data set.
Insert into the parquet table row by row. Then, you will get many many tiny
parquet files. This scenario is not rare in Big Data. I am not sure what is the
performance gain and loss in all your test cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
