Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18824#discussion_r131210013
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -616,15 +616,24 @@ private[spark] class HiveExternalCatalog(conf:
SparkConf, hadoopConf: Configurat
// Add table metadata such as table schema, partition columns, etc. to
table properties.
val updatedTable = withNewSchema.copy(
properties = withNewSchema.properties ++
tableMetaToTableProps(withNewSchema))
+
+ // If it's a data source table, make sure the original schema is left
unchanged; the
+ // actual schema is recorded as a table property.
+ val tableToStore = if (DDLUtils.isDatasourceTable(updatedTable)) {
+ updatedTable.copy(schema = rawTable.schema)
--- End diff --
Hmm, I see that this will break DS tables created with
`newHiveCompatibleMetastoreTable` instead of
`newSparkSQLSpecificMetastoreTable`.
For the former, the only thing I can see that could be used to identify the
case is the presence of serde properties in the table metadata. That could
replace the `DDLUtils.isDatasourceTable(updatedTable)` check to see whether the
schema needs to be updated.
For the latter case, I see that `newSparkSQLSpecificMetastoreTable` stores
the partition schema as the table's schema (which sort of explains the weird
exception handling I saw). So this code is only correct if the partition schema
cannot change. Where is the partition schema for a DS table defined? Is that
under control of the user (or the data source implementation)? Because if it
can change you can run into pretty much the same issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
