GitHub user yingjieMiao opened a pull request:
https://github.com/apache/spark/pull/2648
[Spark] RDD take() method: overestimate too much
In the comment (Line 1083), it says: "Otherwise, interpolate the number of
partitions we need to try, but overestimate it by 50%."
`(1.5 * num * partsScanned / buf.size).toInt` is the guess of "num of total
partitions needed". In every iteration, we should consider the increment `(1.5
* num * partsScanned / buf.size).toInt - partsScanned`
Existing implementation 'exponentially' grows `partsScanned ` ( roughly:
`x_{n+1} >= (1.5 + 1) x_n`)
This could be a performance problem. (unless this is the intended behavior)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yingjieMiao/spark rdd_take
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2648.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2648
----
commit a2aa36b6838ff71941dab1d4af5c8e5f79fd4b4f
Author: yingjieMiao <[email protected]>
Date: 2014-10-03T22:26:01Z
RDD take method: overestimate too much
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
