GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/13728
[SPARK-16010] [SQL] Code Refactoring, Test Case Improvement and Description
Updates for SQLConf spark.sql.parquet.filterPushdown
#### What changes were proposed in this pull request?
Starting Spark 2.0, vectorized decoding is introduced for parquet reading.
This feature changes the filter pushdown behavior of parquet reading. Thus,
this PR updates the out-of-dated description of two external `SQLConf`:
`spark.sql.parquet.filterPushdown` and
`spark.sql.parquet.enableVectorizedReader`.
The PR also slightly simplifies the code for building `parquetReader`. cc
@davies @liancheng @marmbrus
Because the current test cases do not verify the behavior when
`spark.sql.parquet.filterPushdown` is set to `false`, added a test case for
improving the test case coverage. Also, improved the test case when the parquet
file path points to either non-existent files or non-existent hosts.
#### How was this patch tested?
Added the related test cases.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark addTestForParquet
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13728.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13728
----
commit a7b89bd414874fc62576f7fb54e9f5e2ffe5f397
Author: gatorsmile <[email protected]>
Date: 2016-06-17T02:59:00Z
fix
commit a1da7981638723e07753e0aa97686602d3bb38a3
Author: gatorsmile <[email protected]>
Date: 2016-06-17T03:19:52Z
update the comment
commit ad1f18cf4ebf189581997876cd13614ec940b961
Author: gatorsmile <[email protected]>
Date: 2016-06-17T03:33:16Z
update the document
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
