[
https://issues.apache.org/jira/browse/KUDU-2821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836760#comment-16836760
]
Grant Henke commented on KUDU-2821:
-----------------------------------
Based on some initial experimentation/research it looks like Spark doesn't pass
this filter to a data source and give it an opportunity to try and handle it.
This looks to have been found by other projects as well:
[https://github.com/databricks/spark-redshift/issues/155]
SPARK-26165 also has some context.
> Push down string range filter from Spark on timestamp column
> ------------------------------------------------------------
>
> Key: KUDU-2821
> URL: https://issues.apache.org/jira/browse/KUDU-2821
> Project: Kudu
> Issue Type: Improvement
> Components: spark
> Affects Versions: 1.9.0
> Reporter: Jeremy Beard
> Priority: Major
>
> In the Spark-Kudu integration, a range filter on a Kudu timestamp column
> using a string representation will not be pushed down to Kudu. It would be
> good if it did!
> For example:
> {code:java}
> df.where("time > '2019-05-09 19:04:17' and time < '2019-05-09
> 19:04:26'").count
> {code}
> will not push down to Kudu, but
> {code:java}
> df.where("time > cast('2019-05-09 19:04:17' as timestamp) and time <
> cast('2019-05-09 19:04:26' as timestamp)").count
> {code}
> will push down to Kudu.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
