dongjoon-hyun commented on code in PR #59734:
URL: https://github.com/apache/spark/pull/59734#discussion_r4360115263
##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala:
##########
@@ -270,6 +271,14 @@ private[spark] class HiveExternalCatalog(conf: SparkConf,
hadoopConf: Configurat
hiveTable.withNewStorage(locationUri = tableLocation),
ignoreIfExists)
} else {
+ // Hive incompatible partition column types are stored with a
placeholder type in Hive
+ // metastore (see `HiveClientImpl.toHiveTable`), which only data source
tables can restore.
+ schemaWithNoCollation
+ .filter(f => tableDefinition.partitionColumnNames.contains(f.name))
+ .find(f => !isHiveCompatibleDataType(f.dataType))
+ .foreach { f =>
+ throw
QueryCompilationErrors.dataTypeUnsupportedByDataSourceError("Hive", f)
Review Comment:
Thank you. You're right. In 3e200d3774c, I moved the check into
`HiveExternalCatalog.verifyHiveSerdePartitionColumnTypes`, which
`HiveExternalCatalog.createTable` still uses, and
`CreateHiveTableAsSelectCommand` now runs it on the new table definition before
it drops the existing table. The new table definition is computed once and
passed to `createTable` too. I also added a test case which overwrites an
existing table through `saveAsTable` and checks that the table and its rows are
kept.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
