Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/1499#issuecomment-49924823
@colorant @mridulm for the serializing part, one issue is that as we merge
streams we already need a heap of O(num streams) objects, so if you're worried
about the objects being large, deferring the read won't help tremendously. I'm
hoping to fix that by adding a max merge factor and supporting hierarchical
merges later, but I think we can do it after this first PR. (If possible I
would like to get it in 1.1 too, since it's not that much code. But I don't
want to have a giant PR waiting that blocks other things.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
