GitHub user carsonwang opened a pull request:
https://github.com/apache/spark/pull/20303
[SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL
## What changes were proposed in this pull request?
This is the co-work with @yucai , @gczsjdy , @chenghao-intel ,
@xuanyuanking
We'd like to introduce a new approach to do adaptive execution in Spark
SQL. The idea is described at
https://docs.google.com/document/d/1mpVjvQZRAkD-Ggy6-hcjXtBPiQoVbZGe3dLnAKgtJ4k/edit?usp=sharing
## How was this patch tested?
Updated ExchangeCoordinatorSuite.
We also tested this with all queries in TPC-DS.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/carsonwang/spark AE_1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20303.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20303
----
commit 7e9e57859ae35f70b57ece2fd907464392d88c0f
Author: Carson Wang <carson.wang@...>
Date: 2017-11-22T08:49:01Z
Introduce QueryStage for adapative execution
commit 2aa185ddec4616b60e7596153c3dd62188e4d145
Author: Carson Wang <carson.wang@...>
Date: 2017-12-12T02:19:53Z
Add QueryPlan for ExecutedCommandExec
commit 7c3d182f6b6c21f437fbef1ea5ebb2843934dbc6
Author: Carson Wang <carson.wang@...>
Date: 2017-12-13T05:31:52Z
Add config spark.sql.adaptive.maxNumPostShufflePartitions
commit e9b3075c7799b8de6038cc6492cd9e54c12f108e
Author: Carson Wang <carson.wang@...>
Date: 2017-12-13T07:40:08Z
update commetns
commit 9666c5fa5b42ebabc4e2b099a1f545945e959e6f
Author: Carson Wang <carson.wang@...>
Date: 2017-12-18T08:38:32Z
Fix style
commit 9b29a3c5eb20b76c1ef7f897c07ccdd3a98a27b4
Author: Carson Wang <carson.wang@...>
Date: 2017-12-26T02:22:45Z
fix bug
commit e0b98fbed96c1c07ebf2a1a6846576f705cf2c24
Author: Carson Wang <carson.wang@...>
Date: 2018-01-17T05:59:51Z
update doc and style
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
