GitHub user fidato13 opened a pull request:
https://github.com/apache/spark/pull/15327
[SPARK-16575] [spark core] partition calculation mismatch with
sc.binaryFiles
## What changes were proposed in this pull request?
This Pull request comprises of the critical bug SPARK-16575 changes. This
change rectifies the issue with BinaryFileRDD partition calculations as upon
creating an RDD with sc.binaryFiles, the resulting RDD always just consisted of
two partitions only.
## How was this patch tested?
The original issue ie. getNumPartitions on binary Files RDD (always having
two partitions) was first replicated and then tested upon the changes. Also the
unit tests have been checked and passed.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/fidato13/spark SPARK-16575
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15327.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15327
----
commit 823aba185c562d08474c1aca846bf8de467beee4
Author: fidato <[email protected]>
Date: 2016-10-02T22:21:13Z
[SPARK-16575] [spark core] partition calculation mismatch with
sc.binaryFiles
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
