GitHub user advancedxy reopened a pull request:
https://github.com/apache/spark/pull/4002
[SPARK-5201][CORE] deal with int overflow in the
ParallelCollectionRDD.slice method
There is an int overflow in the ParallelCollectionRDD.slice method. That's
originally reported by SaintBacchus.
```
sc.makeRDD(1 to (Int.MaxValue)).count // result = 0
sc.makeRDD(1 to (Int.MaxValue - 1)).count // result = 2147483646 =
Int.MaxValue - 1
sc.makeRDD(1 until (Int.MaxValue)).count // result = 2147483646 =
Int.MaxValue - 1
```
see https://github.com/apache/spark/pull/2874 for more details.
This pr try to fix the overflow. However, There's another issue I don't
address.
```
val largeRange = Int.MinValue to Int.MaxValue
largeRange.length // throws java.lang.IllegalArgumentException: -2147483648
to 2147483647 by 1: seqs cannot contain more than Int.MaxValue elements.
```
So, the range we feed to sc.makeRDD cannot contain more than Int.MaxValue
elements. This is the limitation of Scala. However I think we may want to
support that kind of range. But the fix is beyond this pr.
@srowen @andrewor14 would you mind take a look at this pr?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/advancedxy/spark SPARk-5201
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4002.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4002
----
commit e66e60a9f2f37a7845ab09a165ee960f830d7b4e
Author: Ye Xianjin <[email protected]>
Date: 2015-01-12T07:23:39Z
Deal with inclusive and exclusive ranges in one case. If the range is
inclusive
and the end of the range is (Int.MaxValue or Int.MinValue), we should use
inclusive range instead of exclusive
commit 196f8a88ad7a26f285a74529303bd76aa46e9a25
Author: Ye Xianjin <[email protected]>
Date: 2015-01-12T07:28:45Z
Add test cases for ranges end with Int.MaxValue or Int.MinValue
commit 651c959529a59a6f31032ba5ba8dcc459d509453
Author: Ye Xianjin <[email protected]>
Date: 2015-01-12T12:59:03Z
rename sign to needsInclusiveRange. add some comments
commit 7d39b9e9b9c0582993eaf585899ec972a1bb9a17
Author: Ye Xianjin <[email protected]>
Date: 2015-01-13T10:28:41Z
Convert the two cases pattern matching to one case.
commit b3f55773a305d874b0a4db8258bbd8561e3c070b
Author: Ye Xianjin <[email protected]>
Date: 2015-01-14T01:37:13Z
We can include the last element in the last slice in general for inclusive
range, hence eliminate the need to check Int.MaxValue or Int.MinValue.
commit e143d7aee8853d030bf8a1d795a1d13523280b33
Author: Ye Xianjin <[email protected]>
Date: 2015-01-14T01:49:07Z
Update inclusive range check for splitting inclusive range.
commit 96265a1607cf08b0109f8ee21d069d1637589283
Author: Ye Xianjin <[email protected]>
Date: 2015-01-14T02:21:39Z
Update slice method comment and some responding docs.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
