GitHub user marmbrus opened a pull request:
https://github.com/apache/spark/pull/3258
[SPARK-4391][SQL] Configure parquet filters using SQLConf
This is more uniform with the rest of SQL configuration and allows it to be
turned on and off without restarting the SparkContext. In this PR I also turn
off filter pushdown by default due to a number of outstanding issues (in
particular SPARK-4258). When those are fixed we should turn it back on by
default.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/marmbrus/spark parquetFilters
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3258.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3258
----
commit e7f9e163ed5d5a3eb72adc118f4310b0d4620a22
Author: Michael Armbrust <[email protected]>
Date: 2014-11-13T23:30:31Z
First draft of correctly configuring parquet filter pushdown
commit 78fa02d1b00f41e961b4d150de5739057fa94ff9
Author: Michael Armbrust <[email protected]>
Date: 2014-11-13T23:37:23Z
off by default
commit 75afd39ba2a034fb67792c2773ba53dd92e92a71
Author: Michael Armbrust <[email protected]>
Date: 2014-11-14T00:01:24Z
Fix comments
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
