Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/5733#issuecomment-101358044
Here are my two cents:
1. Short term: we simply don't allow data source tables persisted in Hive
metastore to be accessible from Hive client API.
2. Mid term: maybe a Hive storage handler similar to the one for HBase,
responsible for translating Spark SQL data source table properties stored as
Hive metastore SerDe properties to corresponding Hive concepts.
3. Long term:
- Migrate Spark SQL Hive support to a separate external data source
- Have Spark SQL's own metastore service
- The storage handler mentioned above can be used to interacting with
this metastore service
In a word, ideally, Hive is an external data source of Spark SQL via
Spark SQL's external data sources API, while Spark SQL can also be viewed as a
data source of Hive via Hive's storage handler API.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
