GitHub user darabos opened a pull request:
https://github.com/apache/spark/pull/7285
[SPARK-8893] Add runtime checks against non-positive number of partitions
https://issues.apache.org/jira/browse/SPARK-8893
> What does `sc.parallelize(1 to 3).repartition(p).collect` return? I would
expect `Array(1, 2, 3)` regardless of `p`. But if `p` < 1, it returns
`Array()`. I think instead it should throw an `IllegalArgumentException`.
> I think the case is pretty clear for `p` < 0. But the behavior for `p` =
0 is also error prone. In fact that's how I found this strange behavior. I used
`rdd.repartition(a/b)` with positive `a` and `b`, but `a/b` was rounded down to
zero and the results surprised me. I'd prefer an exception instead of
unexpected (corrupt) results.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/darabos/spark patch-1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7285.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7285
----
commit 897c628a5b0aad31d00beb32a2c3ef85eb5fae9c
Author: Daniel Darabos <[email protected]>
Date: 2015-07-08T10:43:54Z
Require positive maxPartitions in CoalescedRDD
commit d5e3df8555654c54c2095b0285c3ded373345f9f
Author: Daniel Darabos <[email protected]>
Date: 2015-07-08T10:48:35Z
Require positive number of partitions in HashPartitioner
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
