Github user jegonzal commented on the pull request:
https://github.com/apache/spark/pull/10550#issuecomment-168850430
@davies thanks for taking look! I will open a JIRA issue later today.
With respect to the disk based design, I had considered it but it has a few
limitations. First it breaks the lazy evaluation model which (while not
critical) was something I wanted to avoid. Second, and perhaps more
importantly, I wanted to avoid writing both relations completely to disk since
it is possible that one may only need to be partially processed or require
subsequent disk processing. I should add that the Python performance is
actually pretty reasonable :-) for basic data-processing tasks so I wanted a
design that would retain the current level of performance.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
