Spark DIMSUM Memory requirement?
Hi All, I am trying to run RowMatrix.similarity(0.5) on 60K users (n) with 130k features (m) on spark 1.3.0. Using 4 m3.2xlarge 30GB RAM and 8 cores but getting lots of ERROR YarnScheduler: Lost executor 1 on XXX.internal: remote Akka client disassociate What could be the reason? Is it shuffle memory that I should increase? Thank You Parin Choganwala
Stack Overflow Question
EMR 4.1.0 + Spark 1.5.0 + YARN Resource Allocation http://stackoverflow.com/q/33488869/1366507?sem=2
