What are you comparing in your last experiment? Time spent in writeObject0 on a 100-core machine (!) vs a cluster?
On Sat, Jul 26, 2014 at 11:59 PM, lokesh.gidra <lokesh.gi...@gmail.com> wrote: > Thanks a lot for clarifying this. This explains why there is less > serialization happening with lesser parallelism. There would be less > network > communication, and hence less serialization right? > > But then if we compare 100 cores in local mode vs. 10 nodes of 10 cores > each > in standalone mode, then am I seeing huge improvement in the standalone > mode > as compared to local mode? Is the amount of network communication in both > the cases not same? > > > Thanks, > Lokesh > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spilling-in-memory-messages-in-log-even-with-MEMORY-ONLY-tp10723p10739.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >
