Hi all – a utility that I’ve found useful several times now when working with
RDDs is to be able to reason about segments of the RDD.
For example, if I have two large RDDs and I want to combine them in a way that
would be intractable in terms of memory or disk storage (e.g. A cartesian) but
a piece-wise approach is tractable, there are not many methods that enable this.
Existing relevant methods are :
takeSample – Which allows me to take a number of random values from an RDD but
does not let me process the entire RDD
randomSplit – Which segments the RDD into an array of smaller RDDs – with this,
however, I’ve seen numerous memory issues during execution on larger datasets
or when splitting into many RDDs
forEach or collect – Which require the RDD to fit into memory – which is not
possible for larger datasets
What I am proposing is the addition of a simple method to operate on an RDD
which would operate equivalently to a substring operation in String:
For an RDD[T]:
def sample(startIdx : Int, endIdx : Int) : RDD[T] = {
val zipped = this.zipWithIndex()
val sampled = filter(idx => (idx >= startIdx) && (idx < endIdx))
sampled.map(_._1)
}
I would appreciate any feedback – thank you!
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to
Capital One and/or its affiliates. The information transmitted herewith is
intended only for use by the individual or entity to which it is addressed. If
the reader of this message is not the intended recipient, you are hereby
notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is
strictly prohibited. If you have received this communication in error, please
contact the sender and delete the material from your computer.
