Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19479#discussion_r147946241
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -1032,7 +1032,21 @@ private[spark] class HiveExternalCatalog(conf:
SparkConf, hadoopConf: Configurat
schema.fields.map(f => (f.name, f.dataType)).toMap
stats.colStats.foreach { case (colName, colStat) =>
colStat.toMap(colName, colNameTypeMap(colName)).foreach { case (k,
v) =>
- statsProperties += (columnStatKeyPropName(colName, k) -> v)
+ if (k == ColumnStat.KEY_HISTOGRAM) {
+ // In Hive metastore, the length of value in table properties
cannot be larger than 4000,
+ // so we need to split histogram into multiple key-value
properties if it's too long.
+ val maxValueLen = 4000
--- End diff --
use `SCHEMA_STRING_LENGTH_THRESHOLD` instead of hardcode, please follow
`tableMetaToTableProps`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
