wankunde commented on code in PR #38495:
URL: https://github.com/apache/spark/pull/38495#discussion_r1028741472
##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala:
##########
@@ -721,19 +721,18 @@ private[spark] class HiveExternalCatalog(conf: SparkConf,
hadoopConf: Configurat
table: String,
stats: Option[CatalogStatistics]): Unit = withClient {
requireTableExists(db, table)
- val rawTable = getRawTable(db, table)
-
- // convert table statistics to properties so that we can persist them
through hive client
- val statsProperties =
+ val rawHiveTable = client.getRawHiveTable(db, table)
+ val oldProps =
+ client.hiveTableProps(rawHiveTable, containsStats = false)
+ .filterKeys(!_.startsWith(STATISTICS_PREFIX))
+ val newProps =
if (stats.isDefined) {
- statsToProperties(stats.get)
+ oldProps ++ statsToProperties(stats.get)
} else {
- new mutable.HashMap[String, String]()
+ oldProps
}
Review Comment:
@cloud-fan
* Get all old properties from the hive table without table stats.
(HiveStatisticsProperties is defined in HiveClientImpl, so do this in
HiveClientImpl)
* Filter out spark sql table and columns properties. (STATISTICS_PREFIX is
defined in HiveExternalCatalog, so do this in HiveExternalCatalog)
* Add the new table stats and then save the new table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
