cloud-fan commented on a change in pull request #25955: [SPARK-29277][SQL] Add
early DSv2 filter and projection pushdown
URL: https://github.com/apache/spark/pull/25955#discussion_r338163995
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -243,17 +247,36 @@ class FindDataSourceTable(sparkSession: SparkSession)
extends Rule[LogicalPlan]
override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
case i @ InsertIntoStatement(UnresolvedCatalogRelation(tableMeta), _, _,
_, _)
if DDLUtils.isDatasourceTable(tableMeta) =>
- i.copy(table = readDataSourceTable(tableMeta))
+ if (DataSource.isV2Provider(tableMeta.provider.get,
sparkSession.sessionState.conf)) {
Review comment:
> No, it fixes an existing problem -- that rules were order dependent.
Yes, this is an existing problem, but does it affect this patch? IIUC this
PR only needs to add resolving logic of `DescribeTable` and `AlterTable` to
`ResolveTables`, so it doesn't matter that rules are order dependent.
The refactor that is done in this PR, exposes the problem. I think we should
revert that refactor, and only keep the "early pushdown change".
More specifically, let's revert the changes made in `ResolveCatalogs` and
`ResolveSessionCatalog`. We should keep `ResolveTables` to still resolve both
`UnresolvedRelation` and `UnresolvedV2Relation`. Please treat this as a -1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
