Thanks Liquan, that makes sense, but if I am only doin the computation once, there will essentially be no difference, correct?
I had second question related to mapPartitions 1) All of the records of the Iterator[T] that a single function call in mapPartitions process must fit into memory, correct? 2) Is there someway to process that iterator in sorted order? Thanks! Arun On Tue, Sep 23, 2014 at 5:21 PM, Liquan Pei <liquan...@gmail.com> wrote: > Hi Arun, > > The intermediate results like keyedRecordPieces will not be > materialized. This indicates that if you run > > partitoned = keyedRecordPieces.partitionBy(KeyPartitioner) > > partitoned.mapPartitions(doComputation).save() > > again, the keyedRecordPieces will be re-computed . In this case, cache or > persist keyedRecordPieces is a good idea to eliminate unnecessary expensive > computation. What you can probably do is > > keyedRecordPieces = records.flatMap( record => Seq(key, > recordPieces)).cache() > > Which will cache the RDD referenced by keyedRecordPieces in memory. For > more options on cache and persist, take a look at > http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.RDD. > There are two APIs you can use to persist RDDs and one allows you to > specify storage level. > > Thanks, > Liquan > > > > On Tue, Sep 23, 2014 at 2:08 PM, Arun Ahuja <aahuj...@gmail.com> wrote: > >> I have a general question on when persisting will be beneficial and when >> it won't: >> >> I have a task that runs as follow >> >> keyedRecordPieces = records.flatMap( record => Seq(key, recordPieces)) >> partitoned = keyedRecordPieces.partitionBy(KeyPartitioner) >> >> partitoned.mapPartitions(doComputation).save() >> >> Is there value in having a persist somewhere here? For example if the >> flatMap step is particularly expensive, will it ever be computed twice when >> there are no failures? >> >> Thanks >> >> Arun >> > > > > -- > Liquan Pei > Department of Physics > University of Massachusetts Amherst >
