How to join two PairRDD together?

2014-08-25 Thread Gefei Li
, IterableTuple2Tuple2key1, value1, Tuple2key2, value2, and tried to use values() method and map the keys out, but it gives me an out of memory error. I think the out of memory error is caused by the few entries of my RDD, but I have no idea how to solve it. Can you help me? Regards, Gefei Li

how to use the method saveAsTextFile of a RDD like javaRDDmyOwnClass[]

2014-08-14 Thread Gefei Li
Hello, I wrote a class named BooleanPair: public static class BooleanPairet implements Serializable{ public Boolean elementBool1; public Boolean elementBool2; BooleanPair(Boolean bool1, Boolean bool2){elementBool1 = bool1; elementBool2 = bool2;} public String

Re: how to use the method saveAsTextFile of a RDD like javaRDDmyOwnClass[]

2014-08-14 Thread Gefei Li
Thank you! It works so well for me! Regards, Gefei On Thu, Aug 14, 2014 at 4:25 PM, Tathagata Das tathagata.das1...@gmail.com wrote: FlatMap the JavaRDDBooleanPair[] to JavaRDDBooleanPair. Then it should work. TD On Thu, Aug 14, 2014 at 1:23 AM, Gefei Li gefeili.2...@gmail.com wrote

Re: how to use the method saveAsTextFile of a RDD like javaRDDmyOwnClass[]

2014-08-14 Thread Gefei Li
It is interesting to save a RDD on a disk or HDFS or somethings else as a set of objects, but I think it's more useful to save it as a text file for debugging or just as an output file. If we want to reuse a RDD, text file also works, but perhaps a set of object files will bring a decrease on