Github user pwendell commented on the pull request:
https://github.com/apache/incubator-spark/pull/180#issuecomment-35966227
Hey @kellrott - I started to do a review on this focused on the tests and
smaller stuff. But I realized, this makes a fairly major change to the block
manager API in that it changes it to accept iterators instead of existing
buffers. This means you do a copy into a new buffer in the case where an
iterator is not used - which is expensive and will regress behavior for
existing users. That I think blocks this patch as-is from being merged.
I need to think about this a bit more and see if there is a more
surgical/simple solution to fixing this. Since this is not a super common issue
(although agreed it would be way better to pipeline this write directly to
Disk) it would be nice if we could avoid changing the codepath for normal
users.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
