Not quiet sure, but one assumption would be that you are not having sufficient memory to hold that much of data and the process gets busy in cleaning the garbage and it could be the reason it works when you set MEMORY_AND_DISK_SER_2.
Thanks Best Regards On Mon, Feb 9, 2015 at 8:38 PM, Jong Wook Kim <jongw...@nyu.edu> wrote: > replying to my own thread; I realized that this only happens when the > replication level is 1. > > Regardless of whether setting memory_only or disk or deserialized, I had to > make the replication level >= 2 to make the streaming work properly on > YARN. > > I still don't get it why, because intuitively less replication should imply > faster computation, and testing on a cloudera VM everything worked fine on > YARN. > > If I am missing something important, please let me know. I am going to > settle down to '..._2' variants for now. > > > Jong Wook > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Custom-streaming-receiver-slow-on-YARN-tp21544p21553.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
