Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/8125#issuecomment-130263451
Reading a large number of small files is not recommended and shouldn't be a
common case. This is especially true for columnar file formats like Parquet and
ORC. Basically you loose almost all the benefits brought by a columnar file
format when dealing with small files. Besides that, this PR introduces multiple
major problems, please see my inline comments for details. And I don't think a
`coalesce(n)` call is really that inconvenient...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
