Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/4899#issuecomment-77538334
@hhbyyh that's right, you can coalesce to a smaller number of partitions
without a shuffle. However, `defaultParallelism` could be a larger number of
partitions. If `defaultParallelism` is just a little bit smaller than the
source partitioning, you'll get some uneven partitions and could actually slow
it down.
It's a larger question, really -- how much can you hide these details from
the user, and how much do we expect people can or should understand these
things to use Spark effectively? I think an example is not held forth as an
example of "fastest possible", but just of "essential, basic usage".
Partitioning and shuffles are documented elsewhere, and are not essential to
LDA usage. That is, putting it in the example implies you must perform this
step, and that's not so.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
