Github user hhbyyh commented on the pull request:
https://github.com/apache/spark/pull/4899#issuecomment-77491892
@srowen @jkbradley Thanks for review.
I'm not sure about that `coalesce` always introduce shuffle, given its
signature is
`def coalesce(numPartitions: Int, shuffle: Boolean = false)`. In this case,
I thought there will be no shuffle.
Any user trying the LDA example will probably meet the performance
downgrade since they could have too big or too small partition number
(corresponding to input file number), and they will have to spend time
investigating or just rush to a conclusion that LDA is slow. I believe only
part of the users is aware of the partition behavior of `sc.textFile`. (That's
why it's better to add partitions number check/warning in `LDA.run`)
Surely I can close the PR if still it's regarded improper. Right now I
believe it helps. Looking forward to your comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
