1.At begining of reduce task , mask will deliver map output info to every excutor. You can check stderr to find size of map output info . It should be : "spark.MapOutputTrackerMaster: Size of output statuses for shuffle 0 is xxx bytes"
2.Erery excutor will processing 10+TB/2000 = 5G data. ReduceByKey will create a hashtable of unique lines form this 5G data and keep it in memory. it is maybe exceeed 16G . -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/AppMaster-OOME-on-YARN-tp12612p12627.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
