To answer your first question, caching in Spark is lazy, meaning that Spark will not actually try to cache the RDD you've targeted until you take some sort of action on that RDD (like a count).
That might be why you don't see any error at first. On Thu, May 8, 2014 at 2:46 AM, vinay Bajaj <vbajaj2...@gmail.com> wrote: > > Hello > > I have few questions regarding shark. > > 1) I have a table of 60 GB and i have total memory of 50 GB but when i try > to cache the table it get cached successfully. How shark caches the table > there was not enough memory to get the table in memory. And how cache > eviction policies (FIFO and LRU) works while caching the table. While > creating tables I am using cache type property as MEMORY (storage level: > memory and disk) > > 2) Sometime while running queries I get JavaOutOfMemory Exception but all > tables are cached successfully. Can you tell me the cases or some example > due to which that error can come. > > Regards > Vinay Bajaj > >
