Hi Reza, Yes, I just found RDD.cartesian(). Very useful.
Thanks, David On Sun, Mar 22, 2015 at 5:08 PM Reza Zadeh <r...@databricks.com> wrote: > You can do this with the 'cartesian' product method on RDD. For example: > > val rdd1 = ... > val rdd2 = ... > > val combinations = rdd1.cartesian(rdd2).filter{ case (a,b) => a < b } > > Reza > > On Sat, Mar 21, 2015 at 10:37 PM, Xi Shen <davidshe...@gmail.com> wrote: > >> Hi, >> >> I have two big RDD, and I need to do some math against each pair of them. >> Traditionally, it is like a nested for-loop. But for RDD, it cause a nested >> RDD which is prohibited. >> >> Currently, I am collecting one of them, then do a nested for-loop, so to >> avoid nested RDD. But would like to know if there's spark-way to do this. >> >> >> Thanks, >> David >> >> >