Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/4450#issuecomment-96485873
Hey Sandy,
I've now taken a pretty thorough look at this patch. There are a lot of low
level comments and it would be nice if you could do a pass to bring this up to
date and address them. The two major new classes seem good ChainedBuffer and
PartitionedSerializedPairBuffer at a high level.
The only issue on which I have some remaining concern is just the overall
complexity this adds to the already fairly convoluted ExternalSorter and that
the new WritablePartitionedPairCollection is a little clunky. However, I don't
have concrete suggestions to improve that issue at present. Curious if you have
any ideas.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
