GitHub user zhuoliu opened a pull request:
https://github.com/apache/spark/pull/11060
[SPARK-11316] setupGroups in coalescedRDD causes super long delay.
In coalescedRDD, the setupGroups causes super long delay due to the O(n^2)
loop in the second while. That while is used to make sure that each
PartitionGroup contains at least one partition. In some cases, this while will
take O(n^2) to complete. If number of partitions is very large, this would take
tens of minutes or even hours to complete.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhuoliu/spark 11316
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11060.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11060
----
commit e202f579c994be49b255edd4014c0e7cfebaad78
Author: zhuol <[email protected]>
Date: 2016-02-03T21:21:50Z
[SPARK-11316] setupGroups in coalesceRDD causes super long delay due to the
O(n^2) loop.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
