[
https://issues.apache.org/jira/browse/SPARK-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Armbrust resolved SPARK-6647.
-------------------------------------
Resolution: Fixed
Fix Version/s: 1.4.0
Issue resolved by pull request 5309
[https://github.com/apache/spark/pull/5309]
> Make trait StringComparison as BinaryPredicate and throw error when Predicate
> can't translate to data source Filter
> -------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-6647
> URL: https://issues.apache.org/jira/browse/SPARK-6647
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Liang-Chi Hsieh
> Fix For: 1.4.0
>
>
> Now trait {{StringComparison}} is a {{BinaryExpression}}. In fact, it should
> be a {{BinaryPredicate}}.
> By making {{StringComparison}} as {{BinaryPredicate}}, we can throw error
> when a {{expressions.Predicate}} can't translate to a data source {{Filter}}
> in function {{selectFilters}}.
> Without this modification, because we will wrap a {{Filter}} outside the
> scanned results in {{pruneFilterProjectRaw}}, we can't detect about something
> is wrong in translating predicates to filters in {{selectFilters}}.
> The unit test of SPARK-6625 demonstrates such problem. In that pr, even
> {{expressions.Contains}} is not properly translated to
> {{sources.StringContains}}, the filtering is still performed by the
> {{Filter}} and so the test passes.
> Of course, by doing this modification, all {{expressions.Predicate}} classes
> need to have its data source {{Filter}} correspondingly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
