[
https://issues.apache.org/jira/browse/SPARK-29481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954661#comment-16954661
]
Wenchen Fan commented on SPARK-29481:
-------------------------------------
I really appreciate it if more people can help on this ticket. There are still
many commands need to be handled, e.g. REPAIR TABLE, REFRESH TABLE, ADD
PARTITION, etc. We can take a look at `SparkSqlAstBuilder` and pick the
commands that need to resolve a table.
https://github.com/apache/spark/pull/26129 is a good example about how to add a
v2 command:
1. create a statement plan for the command.
2. update `SqlBase.g4` and `AstBuilder`, use multi-part table name for the
command and create statement plan after parsing.
3. create a logical plan and physical plan for the command, if this command can
be implemented via v2 APIs (e.g. REFRESH TABLE).
4. update `ResolveCatalogs`, convert the statement plan to the logical plan, if
we create such a logical plan in step 3.
5. update `ResolveSessionCatalog`, convert the statement plan to the old v1
command plan.
6. add tests in `DDLSuite` and `DataSourceV2SQLSuite`
Please create a sub-task for this ticket if you want to work on one command.
cc [~imback82] [~rdblue] [~dongjoon] [~viirya]
> all the commands should look up catalog/table like v2 commands
> --------------------------------------------------------------
>
> Key: SPARK-29481
> URL: https://issues.apache.org/jira/browse/SPARK-29481
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Wenchen Fan
> Priority: Major
>
> The newly added v2 commands support multi-catalog and respect the current
> catalog/namespace. However, it's not true for old v1 commands.
> This leads to very confusing behaviors, for example
> {code}
> USE my_catalog
> DESC t // success and describe the table t from my_catalog
> ANALYZE TABLE t // report table not found as there is no table t in the
> session catalog
> {code}
> We should make sure all the commands have the same behavior regarding table
> resolution
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
