You can look at the logic for offloading data from Memory by looking at ensureFreeSpace <https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/MemoryStore.scala#L416> call. And dropFromMemory <https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L1011> is the call responsible for putting the data into disk (if applicable).
Thanks Best Regards On Fri, May 22, 2015 at 1:01 PM, swaranga <sarma.swara...@gmail.com> wrote: > Experts, > > This is an academic question. Since Spark runs on the JVM, how it is able > to > do things like offloading RDDs from memory to disk when the data cannot fit > into memory. How are the calculations performed? Does it use the methods > availabe in the java.lang.Runtime class to get free/available memory? How > accurate are these calculations? > > Thanks for any inputs. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Memory-management-tp22992.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
