Re: [DISCUSS] Adaptive execution in Spark SQL

numbers in near future. Thanks, Yucai From: "Wang, Carson" Date: Tuesday, July 31, 2018 at 4:06 PM To: Marco Gaido , Wenchen Fan Cc: "xyliyuanj...@gmail.com" , Spark dev list Subject: RE: [DISCUSS] Adaptive execution in Spark SQL Thanks Marco and Wenchen for reviewing. It s

RE: [DISCUSS] Adaptive execution in Spark SQL

sql-adaptive-execution-at-100-tb>. Thanks, Carson From: Marco Gaido [mailto:marcogaid...@gmail.com] Sent: Tuesday, July 31, 2018 3:00 PM To: Wenchen Fan Cc: xyliyuanj...@gmail.com; Wang, Carson ; Spark dev list Subject: Re: [DISCUSS] Adaptive execution in Spark SQL Hi all, I also like thi

Re: [DISCUSS] Adaptive execution in Spark SQL

Hi all, I also like this idea very much and I think it may bring also other performance improvements in the future. Thanks to everybody who worked on this. I agree to target this feature for 3.0. Thanks everybody, Bests. Marco On Tue, 31 Jul 2018, 08:39 Wenchen Fan, wrote: > Hi Carson and

Re: [DISCUSS] Adaptive execution in Spark SQL

Hi Carson and Yuanjian, Thanks for contributing to this project and sharing the production use cases! I believe the adaptive execution will be a very important feature of Spark SQL and will definitely benefit a lot of users. I went through the design docs and the high-level design totally makes

Re: [DISCUSS] Adaptive execution in Spark SQL

Thanks Carson, great note! Actually Baidu has ported this patch in our internal folk. I collected some user cases and performance improve effect during Baidu internal usage of this patch, summarize as following 3 scenario: 1. SortMergeJoin to BroadcastJoin The SortMergeJoin transform to

[DISCUSS] Adaptive execution in Spark SQL

Dear all, The initial support of adaptive execution[SPARK-9850] in Spark SQL has been there since Spark 1.6, but there is no more update since then. One of the key features in adaptive execution is to determine the number of reducer