I'm now making the Backwards RDD take the previous RDD's partitions and
then using those to iterate. Passes my test. Is it kosher?
On Thu, Sep 11, 2014 at 5:00 PM, Victor Tso-Guillen <v...@paxata.com> wrote:
> Iterating an RDD gives you each partition in order of their split index.
> I'd like to be able to get each partition in reverse order, but I'm having
> difficultly implementing the compute() method. I thought I could do
> something like this:
>
> override def getDependencies: Seq[Dependency[_]] = {
> Seq(new NarrowDependency[T](prev) {
> def getParents(partitionId: Int): Seq[Int] = {
> Seq(prev.partitions.size - partitionId - 1)
> }
> })
> }
>
> override def compute(split: Partition, context: TaskContext):
> Iterator[T] = {
> firstParent[T].iterator(split, context).toArray.reverseIterator
> }
>
> But that doesn't work. How do I get one split to depend on exactly one
> split from the parent that does not match indices?
>
