You can store either in serialized form(butter array) or just save it in a string format like tsv or csv. There are different RDD save apis for that.
Sent from my iPhone > On Aug 23, 2016, at 12:26 PM, kant kodali <[email protected]> wrote: > > > ok now that I understand RDD can be stored to the disk. My last question on > this topic would be this. > > Storing RDD to disk is nothing but storing JVM byte code to disk (in case of > Java or Scala). am I correct? > > > > > >> On Tue, Aug 23, 2016 12:19 PM, RK Aduri [email protected] wrote: >> On an other note, if you have a streaming app, you checkpoint the RDDs so >> that they can be accessed in case of a failure. And yes, RDDs are persisted >> to DISK. You can access spark’s UI and see it listed under Storage tab. >> >> If RDDs are persisted in memory, you avoid any disk I/Os so that any lookups >> will be cheap. RDDs are reconstructed based on a graph (DAG - available in >> Spark UI ) >> >>> On Aug 23, 2016, at 12:10 PM, <[email protected]> >>> <[email protected]> wrote: >>> >>> RAM or Virtual memory is finite, so data size needs to be considered before >>> persist. Please see below documentation when to choose the persistency >>> level. >>> >>> http://spark.apache.org/docs/latest/programming-guide.html#which-storage-level-to-choose >>> >>> Thanks, >>> Sreekanth Jella >>> >>> From: kant kodali >>> Sent: Tuesday, August 23, 2016 2:42 PM >>> To: [email protected] >>> Cc: [email protected] >>> Subject: Re: Are RDD's ever persisted to disk? >>> >>> so when do we ever need to persist RDD on disk? given that we don't need to >>> worry about RAM(memory) as virtual memory will just push pages to the disk >>> when memory becomes scarce. >>> >>> >>> >>> >>> >>> On Tue, Aug 23, 2016 11:23 AM, [email protected] wrote: >>> Hi Kant Kodali, >>> >>> >>> >>> Based on the input parameter to persist() method either it will be cached >>> on memory or persisted to disk. In case of failures Spark will reconstruct >>> the RDD on a different executor based on the DAG. That is how failures are >>> handled. Spark Core does not replicate the RDDs as they can be >>> reconstructed from the source (let’s say HDFS, Hive or S3 etc.) but not >>> from memory (which is lost already). >>> >>> >>> >>> Thanks, >>> Sreekanth Jella >>> >>> >>> >>> From: kant kodali >>> Sent: Tuesday, August 23, 2016 2:12 PM >>> To: [email protected] >>> Subject: Are RDD's ever persisted to disk? >>> >>> >>> >>> I am new to spark and I keep hearing that RDD's can be persisted to memory >>> or disk after each checkpoint. I wonder why RDD's are persisted in memory? >>> In case of node failure how would you access memory to reconstruct the RDD? >>> persisting to disk make sense because its like persisting to a Network file >>> system (in case of HDFS) where a each block will have multiple copies >>> across nodes so if a node goes down RDD's can still be reconstructed by the >>> reading the required block from other nodes and recomputing it but my >>> biggest question is Are RDD's ever persisted to disk? >>> >>> >>> >> >> >> Collective[i] dramatically improves sales and marketing performance using >> technology, applications and a revolutionary network designed to provide >> next generation analytics and decision-support directly to business users. >> Our goal is to maximize human potential and minimize mistakes. In most >> cases, the results are astounding. We cannot, however, stop emails from >> sometimes being sent to the wrong person. If you are not the intended >> recipient, please notify us by replying to this email's sender and deleting >> it (and any attachments) permanently from your system. If you are, please >> respect the confidentiality of this communication's contents. > > > > > >
