Hi Sameer,
If you set those two IDs to be a Tuple2 in the key of the RDD, then you can
join on that tuple.
Example:
val rdd1: RDD[Tuple3[Int, Int, String]] = ...
val rdd2: RDD[Tuple3[Int, Int, String]] = ...
val resultRDD = rdd1.map(k = ((k._1, k._2), k._3)).join(
rdd2.map(k = ((k._1, k._2), k.)3)))
Note that when using .join though, that is an inner join so you only get
results from (id1, id2) pairs that have BOTH a score1 and a score2.
Andrew
On Wed, Jul 2, 2014 at 5:12 PM, Sameer Tilak ssti...@live.com wrote:
Hi everyone,
Is it possible to join RDDs using composite keys? I would like to join
these two RDDs with RDD1.id1 = RDD2.id1 and RDD1.id2 RDD2.id2
RDD1 (id1, id2, scoretype1)
RDD2 (id1, id2, scoretype2)
I want the result to be ResultRDD = (id1, id2, (score1, score2))
Would really appreciate if you can point me in the right direction.
