Hi, This problem is very annoying for me and I'm tired of surfing the network without any good advice to follow.
I have a complex job. It has been worked fine until I needed to save partial results (RDDs) to files. So I tried to cache the RDDs and then call a saveAsText method and follow the workflow as usual. The first problem I noticed was that RDD was not totally cached . So I changed the cache() method for a persist(StorageLevel.MEMORY_AND_DISK_SER()) hoping this should persist 100% of RDD. But it didn't at all. That not make any sense to me. Is not suposed that with that storage level, the fractions which don't fit in memory will be persisted in disk? [image: Imagen integrada 3] Even insignificant RDDs of about ~5MB was cached only at 82%. The last one in the previous image, which had 6628 cached partitions, is distributed in the following way: [image: Imagen integrada 4] The executors Storage Memory wee far away from be filled: [image: Imagen integrada 5] [image: Imagen integrada 6] The only thing I noticed is near to be exhausted is "Memory" in hadoop Cluster Memory: [image: Imagen integrada 7] I don't know the relation between this "memory used" column and the memory described in Spark UI (Storage memory was almost empty). Finally, the job accumulated a lot of stages (~100) for recalculation of RDDs not cached and the cluster failed with an enigmatic and apparently known error: 16/12/14 21:03:54 ERROR LiveListenerBus: Dropping SparkListenerEvent because no remaining room in event queue. This likely means one of the SparkListeners is too slow and cannot keep up with the rate at which tasks are being started by the scheduler. 16/12/14 21:03:54 WARN LiveListenerBus: Dropped 1 SparkListenerEvents since Thu Jan 01 01:00:00 CET 1970 mentioned here <https://issues.apache.org/jira/browse/SPARK-18838> . Please, any clue, any comment indeed, will be much appreciated. Thanks, Pedro image.png (48K) <http://apache-spark-user-list.1001560.n3.nabble.com/attachment/29006/0/image.png> image.png (294K) <http://apache-spark-user-list.1001560.n3.nabble.com/attachment/29006/1/image.png> image.png (57K) <http://apache-spark-user-list.1001560.n3.nabble.com/attachment/29006/2/image.png> image.png (53K) <http://apache-spark-user-list.1001560.n3.nabble.com/attachment/29006/3/image.png> image.png (197K) <http://apache-spark-user-list.1001560.n3.nabble.com/attachment/29006/4/image.png> -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Persisting-RDD-Low-Percentage-with-a-lot-of-memory-available-tp29006.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
