Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/15959#discussion_r88857244
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -627,26 +629,43 @@ private[spark] class HiveExternalCatalog(conf:
SparkConf, hadoopConf: Configurat
}
}
+ // Get the original table properties as defined by Hive.
+ val tableWithProps = tableWithSchema.copy(
+ properties = table.properties.filterNot { case (key, _) =>
key.startsWith(SPARK_SQL_PREFIX) })
+
// construct Spark's statistics from information in Hive metastore
val statsProps =
tableWithSchema.properties.filterKeys(_.startsWith(STATISTICS_PREFIX))
- val tableWithStats = if (statsProps.nonEmpty) {
- val colStatsProps =
statsProps.filterKeys(_.startsWith(STATISTICS_COL_STATS_PREFIX))
- .map { case (k, v) => (k.drop(STATISTICS_COL_STATS_PREFIX.length),
v) }
- val colStats: Map[String, ColumnStat] =
tableWithSchema.schema.collect {
- case f if colStatsProps.contains(f.name) =>
- val numFields = AnalyzeColumnCommand.numStatFields(f.dataType)
- (f.name, ColumnStat(numFields, colStatsProps(f.name)))
- }.toMap
+
+ if (statsProps.nonEmpty) {
+ val colStats = new scala.collection.mutable.HashMap[String,
ColumnStat]
+
+ // For each column, recover its column stats. Note that this is
currently a O(n^2) operation,
+ // but given the number of columns it usually not enormous, this is
probably OK as a start.
+ // If we want to map this a linear operation, we'd need a stronger
contract between the
+ // naming convention used for serialization.
+ tableWithSchema.schema.foreach { field =>
+ // TODO: Do we need to consider case sensivitiy here?
--- End diff --
I don't think case sensitivity matters here, the column stats is an
internal data structure, and we can guarantee the column name in column stats
is always the actual column name.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
