[
https://issues.apache.org/jira/browse/KUDU-2821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeremy Beard updated KUDU-2821:
-------------------------------
Description:
In the Spark-Kudu integration, a range filter on a Kudu timestamp column using
a string representation will not be pushed down to Kudu. It would be good if it
did!
For example:
{code:java}
df.where("time > '2019-05-09 19:04:17' and time < '2019-05-09 19:04:26'").count
{code}
will not push down to Kudu, but
{code:java}
df.where("time > cast('2019-05-09 19:04:17' as timestamp) and time <
cast('2019-05-09 19:04:26' as timestamp)").count
{code}
will push down to Kudu.
was:
In the Spark-Kudu integration, a range filter on a Kudu timestamp column using
a string representation will not be pushed down to Kudu. It would be good if it
did!
For example:
{code:java}
df.where("time > '2019-05-09 19:04:17' and time < '2019-05-09 19:04:26'").count
{code}
will not push down to Kudu, but
{code:java}
df.where("time >= cast('2019-05-09 19:04:17' as timestamp) and time <=
cast('2019-05-09 19:04:26' as timestamp)").count
{code}
will push down to Kudu.
> Push down string range filter from Spark on timestamp column
> ------------------------------------------------------------
>
> Key: KUDU-2821
> URL: https://issues.apache.org/jira/browse/KUDU-2821
> Project: Kudu
> Issue Type: Improvement
> Components: spark
> Affects Versions: 1.9.0
> Reporter: Jeremy Beard
> Priority: Major
>
> In the Spark-Kudu integration, a range filter on a Kudu timestamp column
> using a string representation will not be pushed down to Kudu. It would be
> good if it did!
> For example:
> {code:java}
> df.where("time > '2019-05-09 19:04:17' and time < '2019-05-09
> 19:04:26'").count
> {code}
> will not push down to Kudu, but
> {code:java}
> df.where("time > cast('2019-05-09 19:04:17' as timestamp) and time <
> cast('2019-05-09 19:04:26' as timestamp)").count
> {code}
> will push down to Kudu.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
