GitHub user aarondav opened a pull request:
https://github.com/apache/spark/pull/973
SPARK-2028: Expose mapPartitionsWithInputSplit in HadoopRDD
This allows users to gain access to the InputSplit which backs each
partition.
An alternative solution would have been to have a .withInputSplit() method
which returns a new RDD[(InputSplit, (K, V))], but this is confusing because
you could not cache this RDD or shuffle it, as InputSplit is not inherently
serializable.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aarondav/spark hadoop
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/973.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #973
----
commit c12556bfacda50a707522b428bb86b656b58c218
Author: Aaron Davidson <[email protected]>
Date: 2014-06-05T02:17:13Z
SPARK-2028: Expose mapPartitionsWithInputSplit in HadoopRDD
This allows users to gain access to the InputSplit which backs
each partition.
An alternative solution would have been to have a .withInputSplit()
method which returns a new RDD[(InputSplit, (K, V))], but this is
confusing because you could not cache this RDD or shuffle it, as
InputSplit is not inherently serializable.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
