rdblue commented on a change in pull request #30558:
URL: https://github.com/apache/spark/pull/30558#discussion_r532907901
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -189,6 +189,9 @@ abstract class Optimizer(catalogManager: CatalogManager)
// plan may contain nodes that do not report stats. Anything that uses
stats must run after
// this batch.
Batch("Early Filter and Projection Push-Down", Once,
earlyScanPushDownRules: _*) :+
+ // This batch rewrites plans for v2 tables. It should be run after the
operator
+ // optimization batch and before any batches that depend on stats.
+ Batch("V2 Source Rewrite Rules", Once, v2SourceRewriteRules: _*) :+
Review comment:
If these rules create scan nodes, then shouldn't this run before the
early filter and projection pushdown? Then we can rewrite to normal logical
plans and have those rules perform early pushdown like normal.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
