Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19479#discussion_r150192871
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -1034,11 +1034,18 @@ private[spark] class HiveExternalCatalog(conf:
SparkConf, hadoopConf: Configurat
schema.fields.map(f => (f.name, f.dataType)).toMap
stats.colStats.foreach { case (colName, colStat) =>
colStat.toMap(colName, colNameTypeMap(colName)).foreach { case (k,
v) =>
- statsProperties += (columnStatKeyPropName(colName, k) -> v)
+ val statKey = columnStatKeyPropName(colName, k)
+ val threshold = conf.get(SCHEMA_STRING_LENGTH_THRESHOLD)
+ if (v.length > threshold) {
+ throw new AnalysisException(s"Cannot persist '$statKey' into
hive metastore as " +
--- End diff --
can we try catch it in `HiveExternalCatalog.withClient`? This can also help
if users set a very long table property.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
