t;>
>>
>>
>> BTW: spark.shuffle.reduceLocality.enabled is the configuration of Spark, not
>> Spark SQL.
>>
>>
>>
>> From: Todd [mailto:bit1...@163.com]
>> Sent: Friday, September 11, 2015 3:39 PM
>> To: Todd
>> Cc: Cheng, Hao; Jesse F Ch
Thanks Hao for the reply.
I turn the merge sort join off, the physical plan is below, but the performance
is roughly the same as it on...
== Physical Plan ==
TungstenProject
[ss_quantity#10,ss_list_price#12,ss_coupon_amt#19,ss_cdemo_sk#4,ss_item_sk#2,ss_promo_sk#8,ss_sold_date_sk#0]
.
From: Todd [mailto:bit1...@163.com]
Sent: Friday, September 11, 2015 2:17 PM
To: Cheng, Hao
Cc: Jesse F Chen; Michael Armbrust; user@spark.apache.org
Subject: Re:RE: spark 1.5 SQL slows down dramatically by 50%+ compared with
spark 1.4.1 SQL
Thanks Hao for the reply.
I turn the merge sort join off
5, and it’s true by
default, but we found it probably causes the performance reduce dramatically.
From: Todd [mailto:bit1...@163.com]
Sent: Friday, September 11, 2015 2:17 PM
To: Cheng, Hao
Cc: Jesse F Chen; Michael Armbrust; user@spark.apache.org
Subject: Re:RE: spark 1.5 SQL slows down dr
ark SQL.
>
>
>
> From: Todd [mailto:bit1...@163.com]
> Sent: Friday, September 11, 2015 3:39 PM
> To: Todd
> Cc: Cheng, Hao; Jesse F Chen; Michael Armbrust; user@spark.apache.org
> Subject: Re:Re:RE: Re:RE: spark 1.5 SQL slows down dramatically by 50
, September 11, 2015 3:39 PM
To: Todd
Cc: Cheng, Hao; Jesse F Chen; Michael Armbrust; user@spark.apache.org
Subject: Re:Re:RE: Re:RE: spark 1.5 SQL slows down dramatically by 50%+
compared with spark 1.4.1 SQL
I add the following two options:
spark.sql.planner.sortMergeJoin=false
.
From: Todd [mailto:bit1...@163.com]
Sent: Friday, September 11, 2015 2:17 PM
To: Cheng, Hao
Cc: Jesse F Chen; Michael Armbrust; user@spark.apache.org
Subject: Re:RE: spark 1.5 SQL slows down dramatically by 50%+ compared with
spark 1.4.1 SQL
Thanks Hao for the reply.
I turn the merge
.@intel.com>, Todd <bit1...@163.com>, Michael
> Armbrust <mich...@databricks.com>, "user@spark.apache.org"
> <user@spark.apache.org>
> Date: 09/11/2015 10:41 AM
> Subject: Re: Re:Re:RE: Re:RE: spark 1.5 SQL slows down dramatically by 50%+
> compared w
e? Not the local mode. Can you print the c
>
> From: "Cheng, Hao" <hao.ch...@intel.com>
> To: Todd <bit1...@163.com>
> Cc: Jesse F Chen/San Francisco/IBM@IBMUS, Michael Armbrust
<mich...@databricks.com>, "user@spark.apache.org" <user@spark.ap
@intel.com>
> To: Todd <bit1...@163.com>
> Cc: Jesse F Chen/San Francisco/IBM@IBMUS, Michael Armbrust
> <mich...@databricks.com>, "user@spark.apache.org" <user@spark.apache.org>
> Date: 09/11/2015 01:00 AM
> Subject: RE: Re:Re:RE: Re:RE: spark 1.5
: Jesse F Chen/San Francisco/IBM@IBMUS, Michael Armbrust
<mich...@databricks.com>, "user@spark.apache.org"
<user@spark.apache.org>
Date: 09/11/2015 01:00 AM
Subject: RE: Re:Re:RE: Re:RE: spark 1.5 SQL slows down dramatically by
5
Thanks Michael for the reply.
Below is the sql plan for 1.5 and 1.4. 1.5 is using SortMergeJoin, while 1.4.1
is using shuffled hash join.
In this case, it seems hash join performs better than sort join.
12 matches
Mail list logo
