Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/4706#issuecomment-75502698
@viirya As I mentioned, the partitioning scheme you proposed will make the
first partition (which contains the most frequent items) very heavy load and
slows down the entire process. Please check the example I gave. Using a random
partitioning scheme helps distribute the work. It may increase the total
shuffle size in some cases but the work loads are more balanced.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
