cloud-fan commented on a change in pull request #25955: [SPARK-29277][SQL] Add
early DSv2 filter and projection pushdown
URL: https://github.com/apache/spark/pull/25955#discussion_r337848276
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -243,17 +247,36 @@ class FindDataSourceTable(sparkSession: SparkSession)
extends Rule[LogicalPlan]
override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
case i @ InsertIntoStatement(UnresolvedCatalogRelation(tableMeta), _, _,
_, _)
if DDLUtils.isDatasourceTable(tableMeta) =>
- i.copy(table = readDataSourceTable(tableMeta))
+ if (DataSource.isV2Provider(tableMeta.provider.get,
sparkSession.sessionState.conf)) {
Review comment:
Yes it's currently working. But it adds a refactor and a hack to fix the
issue introduced by the refactoring.
We should not wait until we find a different solution, but it's better to
not expose the problem in this PR.
That said, the refactor in this PR may be problematic. More specifically,
converting `UnresolvedRelation` to `UnresolvedV2Relation` in `ResolveCatalogs`
may not be a good idea. I think we should either 1) revert the refactor in this
PR, or 2) wait for #26214 until we reach an agreement about how
`UnresolvedRelation` should be resolved.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
