GitHub user ankurdave opened a pull request:
https://github.com/apache/spark/pull/2106
[SPARK-3190] Avoid overflow in VertexRDD.count()
VertexRDDs with more than 4 billion elements are counted incorrectly due to
integer overflow when summing partition sizes. This PR fixes the issue by
converting partition sizes to Longs before summing them.
The following code previously returned -10000000. After applying this PR,
it returns the correct answer of 5000000000 (5 billion).
```scala
val pairs = sc.parallelize(0L until 500L).map(_ * 10000000)
.flatMap(start => start until (start + 10000000)).map(x => (x, x))
VertexRDD(pairs).count()
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ankurdave/spark SPARK-3190
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2106.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2106
----
commit 641f46899b0c82564a02a2483caa204e3943beb2
Author: Ankur Dave <[email protected]>
Date: 2014-08-23T23:49:41Z
Avoid overflow in VertexRDD.count()
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
