Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/6459#discussion_r31279071
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -717,10 +717,23 @@ private[hive] case class MetastoreRelation
}
// Must be a stable value since new attributes are born here.
- val partitionKeys = hiveQlTable.getPartitionKeys.map(_.toAttribute)
-
+ // Since we create the TTable inside hiveQlTable manually, we can use
TTbale's interface
+ // to get Partition keys at here. We do this to make sure Hive will not
try to use
+ // any metastore utility functions. All of interactions between
metastore and related
+ // parts should be done through our ClientWrapper.
+ // Without the guard of ClientWrapper, we cannot touch metastore client
(e.g. Hive class) and
+ // the conf associated with the metastore client (Hive.get().getConf()).
+ /** PartitionKey attributes */
+ val partitionKeys =
hiveQlTable.getTTable.getPartitionKeys.map(_.toAttribute)
+
+ // Since we create the TTable inside hiveQlTable manually, we can use
TTbale's interface
+ // to get non-partition columns at here. We do this to make sure Hive
will not try to use
+ // any metastore utility functions. All of interactions between
metastore and related
+ // parts should be done through our ClientWrapper.
+ // Without the guard of ClientWrapper, we cannot touch metastore client
(e.g. Hive class) and
+ // the conf associated with the metastore client (Hive.get().getConf()).
/** Non-partitionKey attributes */
- val attributes = hiveQlTable.getCols.map(_.toAttribute)
+ val attributes = hiveQlTable.getTTable.getSd.getCols.map(_.toAttribute)
--- End diff --
Can we just get both of these from the spark sql `HiveTable` instead?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
