Thanks guys, that did it. On Thu, Aug 13, 2015 at 6:49 PM, Akhil Das <ak...@sigmoidanalytics.com> wrote:
> Have a look at spark.shuffle.manager, You can switch between sort and hash > with this configuration. > > spark.shuffle.managersortImplementation to use for shuffling data. There > are two implementations available:sort and hash. Sort-based shuffle is > more memory-efficient and is the default option starting in 1.2. > > Thanks > Best Regards > > On Thu, Aug 13, 2015 at 2:56 PM, cheez <11besemja...@seecs.edu.pk> wrote: > >> I understand that the current master branch of Spark uses Sort based >> shuffle. >> Is there a way to change that to Hash based shuffle, just for experimental >> purposes by modifying the source code ? >> >> >> >> -- >> View this message in context: >> http://apache-spark-developers-list.1001551.n3.nabble.com/Switch-from-Sort-based-to-Hash-based-shuffle-tp13661.html >> Sent from the Apache Spark Developers List mailing list archive at >> Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> For additional commands, e-mail: dev-h...@spark.apache.org >> >> >
