Github user syedhashmi commented on the pull request:
https://github.com/apache/spark/pull/876#issuecomment-44152706
You are right there are routines which make this assumption but this is
becoming a pain point for users as they end up with lopsided partitions and
especially, if their dataset is huge, some larger partitions become bottleneck
and extend the tail of processing time. This partitioner is explicitly
targeting such scenarios. If agree upon general idea of partitioner itself, I
can add checks to functions assuming Hash or Range partitioning behavior to
classify Balanced partitioner as general case. User ends up with exactly
balanced partitions and sacrifices a bit at lookup type routines.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
