Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/3694#issuecomment-75563929
I am also not clear this is a good thing. As a default, it doesn't change
anything. There is probably not a globally correct ratio, even if it's not 1,
but this implies there is. Is there evidence that a default besides 1.0 is
better in most cases? The docs don't even suggest what the tradeoff is here.
Won't this potentially cause more shuffles when the ratio is not 1? I think
this is something that must be set on a case-by-case basis, and that can
already be done, even as a function of the parent RDD partitions, by the caller.
Can we elaborate on this or close it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
