GitHub user dbtsai opened a pull request:
https://github.com/apache/spark/pull/22574
[SPARK-25556][SQL] Just remove the unsupported predicates in Parquet
## What changes were proposed in this pull request?
Currently, in `ParquetFilters`, if one of the children predicates is not
supported by Parquet, the entire predicates will be thrown away. In fact, if
the unsupported predicate is in the top level `And` condition or in the child
before hitting `Not` or `Or` condition, it can be safely removed.
## How was this patch tested?
Tests are added.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dbtsai/spark
removeUnsupportedPredicatesInParquet
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22574.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22574
----
commit d49d63bc40a7752990583f9afbd10c68025510b3
Author: DB Tsai <d_tsai@...>
Date: 2018-09-27T22:12:44Z
Remove unsupported predicates in parquet
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
