MaxGekk opened a new pull request #28272:
URL: https://github.com/apache/spark/pull/28272
### What changes were proposed in this pull request?
Always translate date values of pushed down filters to `java.sql.Date`
independently from the SQL config `spark.sql.datetime.java8API.enabled`.
Closes #28259
Closes #28261
### Why are the changes needed?
1. For backward compatibility with existing implementations of datasources.
For example, the following exception is thrown by ORC datasource when
`spark.sql.datetime.java8API.enabled` is set to `true`:
```
Wrong value class java.time.LocalDate for DATE.EQUALS leaf
java.lang.IllegalArgumentException: Wrong value class java.time.LocalDate
for DATE.EQUALS leaf
at
org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.checkLiteralType(SearchArgumentImpl.java:192)
at
org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.<init>(SearchArgumentImpl.java:75)
at
org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$BuilderImpl.equals(SearchArgumentImpl.java:352)
at
org.apache.spark.sql.execution.datasources.orc.OrcFilters$.buildLeafSearchArgument(OrcFilters.scala:229)
```
2. Before the changes, date filters are not pushed down to Parquet
datasource when `spark.sql.datetime.java8API.enabled` is `true`.
### Does this PR introduce any user-facing change?
Yes
### How was this patch tested?
Added a test to `ParquetFilterSuite` and to `OrcFilterSuite`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
