rdblue commented on issue #25955: [SPARK-29277][SQL] Add early DSv2 filter and projection pushdown URL: https://github.com/apache/spark/pull/25955#issuecomment-544685621 @cloud-fan, I've reverted the change to the `SessionCatalog` and `NonSessionCatalog` extractors. Now, the change to remove `Analyzer.lookupV2Relation` is a refactor. It is okay if temporary table identifiers are converted to v2 plans, so that plans that should ignore existing temp tables will succeed. Instead, I've added additional cases to `CheckAnalysis` to catch plans that are still unresolved because of views. I also found that the latest set of test failures was caused by analysis depending on the order of rules and had to fix this. V2 `ResolveTables` runs before `ResolveRelations`, so `UnresolvedIdentifier` for v2 tables would get converted by `lookupV2Relation`. Now this process is two steps, `ResolveCatalogs` or `ResolveSessionCatalog`, followed by `ResolveTables`. And `ResolveSessionCatalog` is added later, so it doesn't run before `ResolveRelations`. To fix the rule order problem, I had to update the rule that resolves `UnresolvedCatalogRelation` to support v2 plans. Now, `UnresolvedRelation` can be converted to `UnresolvedCatalogTable` and then converted to a v2 relation. This is safe because `ResolveRelations` only runs on session catalog identifiers.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
