No, but you can write a couple lines of code that do this. It's not optimized of course. This is actually a long and interesting side discussion, but I'm not sure how much it could be given that the computation is "pull" rather than "push"; there is no concept of one pass over the data resulting in many RDDs. However you can cache / persist the source RDD to at least make sure it is not recomputed.
I don't think groupByKey is quite a solution since it means one RDD for which all values for one key must fit in memory, and because the desired output is an RDD, I am not sure that is suitable. On Wed, Jun 10, 2015 at 1:56 PM, dgoldenberg <[email protected]> wrote: > Hi, > > I'm gathering that the typical approach for splitting an RDD is to apply > several filters to it. > > rdd1 = rdd.filter(func1); > rdd2 = rdd.filter(func2); > ... > > Is there/should there be a way to create 'buckets' like these in one go? > > List<RDD> rddList = rdd.filter(func1, func2, ..., funcN) > > Another angle here is, when applying a filter(func), is there a way to get > two RDD's back, one for which func returned true for all elements of the > original RDD (the one being filtered), and the other one for which func > returned false for all the elements? > > Pair<RDD> pair = rdd.filterTrueFalse(func); > > Right now I'm doing > > RDD x = rdd.filter(func); > RDD y = rdd.filter(reverseOfFunc); > > This seems a bit tautological to me, though Spark must be optimizing this > out (?) > > Thanks. > > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Split-RDD-based-on-criteria-tp23254.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
