cloud-fan commented on issue #25955: [SPARK-29277][SQL] Add early DSv2 filter and projection pushdown URL: https://github.com/apache/spark/pull/25955#issuecomment-544617970 This is a tricky problem and we need to think through it carefully. The table lookup logic is not consistently defined within Spark SQL: 1. for SELECT/INSERT, we try to lookup temp view first, then lookup table/view(these two share the same namespace so doesn't matter which one goes first). This is a little weird that Spark can insert into a temp view, but that's what it is. 2. for some commands like DESC TABLE, we follow the previous one and also try to lookup temp view first, because these commands support temp view. 3. for other commands like ALTER TABLE, we try to lookup table/view, because they don't support temp view. Seems like this PR is trying to unify the table lookup logic, which may change the behavior. For example, if there are a table and a temp view have the same name, ALTER TABLE should still work, but it fails now since both `ResolveCatalog` and `ResolveCatalogs` skip temp view.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
