GitHub user yucai opened a pull request:
https://github.com/apache/spark/pull/22197
[SPARK-25207][SQL] Case-insensitve field resolution for filter pushdown
when reading Parquet
## What changes were proposed in this pull request?
Currently, filter pushdown will not work if Parquet schema and Hive
metastore schema are in different letter cases even spark.sql.caseSensitive is
false.
Like the below case:
```scala
spark.range(10).write.parquet("/tmp/data")
sql("DROP TABLE t")
sql("CREATE TABLE t (ID LONG) USING parquet LOCATION '/tmp/data'")
sql("select * from t where id > 0").show
```
No filter will be generated, this PR provides a case-insensitive field
resolution to filter pushdown.
## How was this patch tested?
Added UTs.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yucai/spark SPARK-25207
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22197.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22197
----
commit 5902afe6fb6e88f98fb4f2649e59156264bc3e4d
Author: yucai <yyu1@...>
Date: 2018-08-23T07:16:42Z
[SPARK-25207][SQL] Case-insensitve field resolution for filter pushdown
when reading Parquet
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
