Hi Supun, Did you look at https://spark.apache.org/docs/latest/tuning.html?
In addition to the info there, if you're partitioning by some key where you've got a lot of data skew, one of the task's memory requirements may be larger than the RAM of a given executor, where the rest of the tasks may be just fine. If you're partitioning by some key, you may want to see if some key has way more data than the others. Matt -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
