Have you tried repartition to larger number of partitions? Also, I would suggest increase number of executors and give them smaller amount of memory each. On 15 Feb 2016 06:49, "gustavolacerdas" <gustavolacer...@gmail.com> wrote:
> I have a machine with 96GB and 24 cores. I'm trying to run a k-means > algorithm with 30GB of input data. My spark-defaults.conf file are > configured like that: spark.driver.memory 80g spark.executor.memory 70g > spark.network.timeout 1200s spark.rdd.compress true > spark.broadcast.compress true But i always get this error Spark Error: Not > enough space to cache partition rdd I changed the k-means code to persist > the rdd.cache(MEMORY_AND_DISK) but didn't work. > ------------------------------ > View this message in context: Spark Error: Not enough space to cache > partition rdd > <http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Error-Not-enough-space-to-cache-partition-rdd-tp26222.html> > Sent from the Apache Spark User List mailing list archive > <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com. >
