Just to close out this one, I noticed that the cache partition size was quite
low for each of the RDDs (1 - 14). Increasing the number of partitions
(~400) resolved this for me.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/RDD-Cache-Cleanup-tp19772p19908
10 GB. There is
enough memory available (around 1 TB). The /spark.storage.memoryFraction/
property is set at 0.7.
How does the cleanup work? Any help is appreciated.
- Ranga
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/RDD-Cache-Cleanup-tp19771.html
