aokolnychyi commented on a change in pull request #30558:
URL: https://github.com/apache/spark/pull/30558#discussion_r533123744
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -189,6 +189,9 @@ abstract class Optimizer(catalogManager: CatalogManager)
// plan may contain nodes that do not report stats. Anything that uses
stats must run after
// this batch.
Batch("Early Filter and Projection Push-Down", Once,
earlyScanPushDownRules: _*) :+
+ // This batch rewrites plans for v2 tables. It should be run after the
operator
+ // optimization batch and before any batches that depend on stats.
+ Batch("V2 Source Rewrite Rules", Once, v2SourceRewriteRules: _*) :+
Review comment:
Right now, scan construction and pushdown happens in the same method. We
could split that later so I agree with moving this batch before early filter
and projection pushdown.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
