Re: Do we need to enabled Tungsten sort in Spark 1.6?

Yeah, this confused me, as well. Good question, Umesh. As Ted pointed out: between Spark 1.5 and 1.6, o.a.s.shuffle.unsafe.UnsafeShuffleManager no longer exists as a separate shuffle manager. Here's the old code (notice the o.a.s.shuffle.unsafe package):

Re: Do we need to enabled Tungsten sort in Spark 1.6?

For "spark.shuffle.manager", the default is "sort" >From core/src/main/scala/org/apache/spark/SparkEnv.scala : val shuffleMgrName = conf.get("spark.shuffle.manager", "sort") "tungsten-sort" is the same as "sort" : val shortShuffleMgrNames = Map( "hash" ->

Do we need to enabled Tungsten sort in Spark 1.6?

Hi I was using Spark 1.5 with Tungsten sort and now I have using Spark 1.6 I dont see any difference I was expecting Spark 1.6 to be faster. Anyways do we need to enable Tunsten and unsafe options or they are enabled by default I see in documentation that default sort manager is sort I though it

Re: Do we need to enabled Tungsten sort in Spark 1.6?

>From sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala : case Some((SQLConf.Deprecated.TUNGSTEN_ENABLED, Some(value))) => val runFunc = (sqlContext: SQLContext) => { logWarning( s"Property ${SQLConf.Deprecated.TUNGSTEN_ENABLED} is deprecated and "

Re: Do we need to enabled Tungsten sort in Spark 1.6?

ok thanks so it will be enabled by default always if yes then in documentation why default shuffle manager is mentioned as sort? On Sat, Jan 9, 2016 at 1:55 AM, Ted Yu wrote: > From sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala > : > > case