Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/12579#issuecomment-218379687
Hey @xwu0226, sorry that I didn't explain why I opened another PR for the
same issue, was in code rush for 2.0...
So one of the considerations for all the native DDL commands is that we
don't want these DDL commands to rely on Hive anymore. This is because we'd
like to remove Hive dependency from Spark SQL core and gradually make Hive a
separate data source in the future. This means, we shouldn't add new code in
places like `HiveClientImpl`. These new DDL command should be implemented upon
interfaces like `CatalogTable`.
One apparent problem of this approach is that, current Spark SQL interfaces
don't capture all semantics of Hive. For example, some table metadata like skew
spec is not covered in `CatalogTable` yet. Our general strategies are:
1. For easy ones, like "owner" and "compressed" in #12844, we may just add
them to the interface and leverage them.
2. For features that are not supported in Spark SQL, for example, skew
spec, we can simply ignore them for now, since Spark can't handle them anyway.
There will be a follow-up of #12781 to add support for Hive tables. After
offline discussion with @yhuai, we decided to add a flag in `CatalogTable` to
indicate that whether there unrecognized metadata provided by the underlying
external catalog, but not translated and included in `CatalogTable`. In this
way, when applying `SHOW CREATE TABLE` to tables containing such metadata, this
flag can be set to true, and we can simply refuse to output anything by
checking this flag. This makes sense because even if you add things like skew
spec in the result of `SHOW CREATE TABLE`, Spark can't handle the generated DDL
statement
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
