GitHub user maryannxue opened a pull request:
https://github.com/apache/spark/pull/21876
[SPARK-24802][SQL][FOLLOW-UP] Add a new config for Optimization Rule
Exclusion
## What changes were proposed in this pull request?
This is an extension to the original PR, in which rule exclusion did not
work for classes derived from Optimizer, e.g., SparkOptimizer.
To solve this issue, Optimizer and its derived classes will define/override
`defaultBatches` and `nonExcludableRules` in order to define its default rule
set as well as rules that cannot be excluded by the SQL config. In the
meantime, Optimizer's `batches` method is dedicated to the rule exclusion logic
and is defined "final".
## How was this patch tested?
Added UT.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/maryannxue/spark rule-exclusion
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21876.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21876
commit eaec2f5f2b4e3193de41655b84a1dc936b0e50a3
Author: maryannxue
Date: 2018-07-13T21:32:01Z
[SPARK-24802] Optimization Rule Exclusion
commit 84f1a6b5cba08df8684179e9d7195545be655e76
Author: maryannxue
Date: 2018-07-18T06:13:50Z
Address review comments
commit ff23edf81a4a78d1589ed582a1802b94a8ebf4c6
Author: maryannxue
Date: 2018-07-21T02:37:54Z
Address review comments
commit b154979236e211dc7185ca8e450493f0c6b0f469
Author: maryannxue
Date: 2018-07-21T02:41:21Z
change test name
commit 87afe4fbcaf71d303b07612f9ceb9ad25dd3dcda
Author: maryannxue
Date: 2018-07-23T00:35:01Z
address review comments
commit 39b6ce9548c99363e81cb246b4cbe5534d710f3e
Author: maryannxue
Date: 2018-07-23T04:28:00Z
address review comments
commit a2161ef1f333f2cc039df0ecc8c96e5ec27e00ff
Author: maryannxue
Date: 2018-07-25T18:52:13Z
Merge remote-tracking branch 'origin/master' into rule-exclusion
commit 3730053d7386188042b2f2d4bd6784c3de722df6
Author: maryannxue
Date: 2018-07-25T20:08:19Z
Extend rule-exclusion to Optimizer sub-classes, esp. SparkOptimizer
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
