GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/9659
[SPARK-11676][SQL] Parquet filter tests all pass if filters are not really
pushed down
Currently Parquet predicate tests all pass even if filters are not pushed
down or this is disabled.
In this PR, For checking evaluating filters, Simply it makes the expression
from `expression.Filter` and then try to create filters just like Spark does.
For checking the results, this manually accesses to the child rdd (of
`expression.Filter`) and produces the results which should be filtered
properly, and then compares it to expected values.
Now, if filters are not pushed down or this is disabled, this throws
exceptions.
cc @liancheng
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark SPARK-11676
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9659.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9659
----
commit 1c6a5480df53d16fe38d14146deedaf035a6024c
Author: hyukjinkwon <[email protected]>
Date: 2015-11-12T09:31:41Z
[SPARK-11676][SQL] Parquet filter tests all pass if filters are not really
pushed down
commit 6b722ef4ad3a29a1bd0cde0961f6bc12638e2f31
Author: hyukjinkwon <[email protected]>
Date: 2015-11-12T09:37:27Z
[SPARK-11676][SQL] Update indentation and comments
commit d007c3f996f1fd4e093a590741b92a4bc4072d41
Author: hyukjinkwon <[email protected]>
Date: 2015-11-12T09:39:05Z
[SPARK-11676][SQL] Remove unused imports
commit 8cc842fa3f97c39e61fa072a916f7056a85be1b7
Author: hyukjinkwon <[email protected]>
Date: 2015-11-12T09:49:11Z
[SPARK-111676][SQL] Set true to pushing down filters
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
