[
https://issues.apache.org/jira/browse/SPARK-25127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16581638#comment-16581638
]
Ryan Blue commented on SPARK-25127:
-----------------------------------
FYI [~rxin].
> DataSourceV2: Remove SupportsPushDownCatalystFilters
> ----------------------------------------------------
>
> Key: SPARK-25127
> URL: https://issues.apache.org/jira/browse/SPARK-25127
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 2.3.0, 2.3.1, 2.4.0
> Reporter: Ryan Blue
> Priority: Major
>
> Discussion about adding TableCatalog on the dev list focused around whether
> Expression should be used in the public DataSourceV2 API, with
> SupportsPushDownCatalystFilters as an example of where it is already exposed.
> The early consensus is that Expression should not be exposed in the public
> API.
> From [~rxin]:
> bq. I completely disagree with using Expression in critical public APIs that
> we expect a lot of developers to use . . . If we are depending on Expressions
> on the more common APIs in dsv2 already, we should revisit that.
> The main use of this API is to pass Expression to FileFormat classes that
> used Expression instead of Filter. External sources also use it for more
> complex push-down, like {{to_date(ts) = '2018-05-13'}}, but those uses can be
> done with Analyzer rules or when translating to Filters.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
