GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/10221
[SPARK-12236][SQL] JDBC filter tests all pass if filters are not really
pushed down
https://issues.apache.org/jira/browse/SPARK-12236
Currently JDBC filters are not tested properly. All the tests pass even if
the filters are not pushed down due to Spark-side filtering.
In this PR,
Firstly, I corrected the tests to properly check the pushed down filters by
removing Spark-side filtering.
Also, `!=` was being tested which is actually not pushed down. So I removed
them.
Lastly, I moved the `stripSparkFilter()` function to `SQLTestUtils` as this
functions would be shared for all tests for pushed down filters. This function
would be also shared with ORC datasource as the filters for that are also not
being tested properly.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark SPARK-12236
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10221.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10221
----
commit 3294b31303d4f67d6a7e7b9125c3f38df0f01fda
Author: hyukjinkwon <[email protected]>
Date: 2015-12-09T07:31:36Z
Correct the test for checking pushed down filters
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
