Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/5403#issuecomment-94905121
My two cents: I think the main reason to merge this is that one doesn't
need to maintain patches out of tree that'll become outdated when the interface
changes :) Also I think this patch completely gets rid of all filesystem
overheads (like file open, close, flush etc.) which is a pretty useful design
point. We've seen filesystem types being a big issue in the past and thus it'll
be useful not just for research benchmarks but also for spark-perf with things
like 'how much time does a small shuffle using files takes compared to this
etc.'
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
