Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19479#discussion_r150011624
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -1034,11 +1034,18 @@ private[spark] class HiveExternalCatalog(conf:
SparkConf, hadoopConf: Configurat
schema.fields.map(f => (f.name, f.dataType)).toMap
stats.colStats.foreach { case (colName, colStat) =>
colStat.toMap(colName, colNameTypeMap(colName)).foreach { case (k,
v) =>
- statsProperties += (columnStatKeyPropName(colName, k) -> v)
+ val statKey = columnStatKeyPropName(colName, k)
+ val threshold = conf.get(SCHEMA_STRING_LENGTH_THRESHOLD)
+ if (v.length > threshold) {
+ throw new AnalysisException(s"Cannot persist '$statKey' into
hive metastore as " +
--- End diff --
what if we don't do it? will hive give us an exception?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
