Re: Advantage of using cache()

[Nieyuan]

Because map-reduce tasks like join will save shuffle data to disk . So the
only diffrence with caching or no-caching version is :
  >> .map { case (x, (n, i)) => (x, n)}



-
Thanks,
Nieyuan
--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Advantage-of-using-cache-tp12480p12634.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: AppMaster OOME on YARN

[Nieyuan]

1.At begining of reduce task , mask will deliver map output info to every
excutor. You can check stderr to find size of map output info . It should be
:
   "spark.MapOutputTrackerMaster: Size of output statuses for shuffle 0 is
xxx bytes"

2.Erery excutor will processing 10+TB/2000 = 5G data. ReduceByKey will
create a hashtable of unique lines form this 5G data and keep it in memory.
it is maybe exceeed 16G .



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/AppMaster-OOME-on-YARN-tp12612p12627.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org