[
https://issues.apache.org/jira/browse/SPARK-7095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Liu closed SPARK-7095.
---------------------------
Resolution: Not A Problem
Spark SQL only pushes filter to datasource if the value is the same data type
as the column
> Pass DataType to source.Filter classes
> --------------------------------------
>
> Key: SPARK-7095
> URL: https://issues.apache.org/jira/browse/SPARK-7095
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 1.3.0
> Reporter: Alex Liu
> Priority: Minor
>
> source.Filter class doesn't pass any data type information.
> e.g.
> {code}
> case class GreaterThan(attribute: String, value: Any) extends Filter
> {code}
> GreaterThan requires data are in the same type.
> Data type matching check are required during Implementing PrunedFilteredScan
> where there are comparison filters.
> Data type can be obtained by a case matching method. Since the method is
> called each time filter a row, it could improve performance if the data type
> is included in Filter.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
