GitHub user maryannxue opened a pull request:
https://github.com/apache/spark/pull/21360
[SPARK-24288] Enable preventing predicate pushdown
## What changes were proposed in this pull request?
1. Add DataSet interface "withOptimizerBarrier()"
2. Modify AnalysisBarrier to accommodate two scenarios: 1) analysis-only
barriers; and 2) optimizer barriers.
3. Add handling of Barrier in Optimizer (logical plan optimization).
4. Add handling of Barrier in SparkStrategies (logical-to-physical plan
translation).
## How was this patch tested?
1. Add DataFrameOptimizerBarrierSuite to ensure:
a) Barriers isolate optimization rule applications.
b) Plans with optimization Barriers get resolved correctly, same as
analysis-only Barriers.
c) Barriers preserves ordering.
d) Barriers preserves constraints.
2. Add one test in JDBCSuite to verify scenario raised by the user (avoid
filter push-down to data source).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/maryannxue/spark spark-24288
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21360.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21360
----
commit d57d89ad98d554c020a59c5c44a5f6725d6e916e
Author: maryannxue <maryann.xue@...>
Date: 2018-05-18T06:34:38Z
[SPARK-24288] Enable preventing predicate pushdown
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
