Perfect! Thank you Josh. From: Josh Rosen <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Wednesday, October 23, 2013 2:19 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: JavaPairRDD unpersist
This is a bug; all three Java*RDD classes should have had unpersist() methods added to them, instead of just JavaRDD (https://github.com/apache/incubator-spark/commit/4a318774088f829fe54c3ef0b5f565a845631b4e). I'll submit a pull request to fix this. In 0.8, you can access the underlying Scala RDD and call unpersist() on that: javaPairRDD.rdd().unpersist() If you want to know the details of why we need to add unpersist() to each Java*RDD class, rather than adding it once in JavaRDDLike, check out the "implementation traits" section of https://cwiki.apache.org/confluence/display/SPARK/Java+API+Internals - Josh On Wed, Oct 23, 2013 at 2:01 PM, Yann Luppo <[email protected]<mailto:[email protected]>> wrote: Hi, I've noticed that in 0.8.0 JavaRDD got a new "unpersist" method. I was wondering if this method was going to be ported eventually to JavaPairRDD. Is this somewhere on the road map? Or should we just change all of our JavaPairRDD to JavaRDDs w/ tuples for now? Thanks, Yann
