Have a look at spark.shuffle.manager, You can switch between sort and hash with this configuration.
spark.shuffle.managersortImplementation to use for shuffling data. There are two implementations available:sort and hash. Sort-based shuffle is more memory-efficient and is the default option starting in 1.2. Thanks Best Regards On Thu, Aug 13, 2015 at 2:56 PM, cheez <11besemja...@seecs.edu.pk> wrote: > I understand that the current master branch of Spark uses Sort based > shuffle. > Is there a way to change that to Hash based shuffle, just for experimental > purposes by modifying the source code ? > > > > -- > View this message in context: > http://apache-spark-developers-list.1001551.n3.nabble.com/Switch-from-Sort-based-to-Hash-based-shuffle-tp13661.html > Sent from the Apache Spark Developers List mailing list archive at > Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > >
