peter-toth commented on code in PR #59757:
URL: https://github.com/apache/spark/pull/59757#discussion_r4370977080
##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateHiveTableAsSelectCommand.scala:
##########
@@ -65,20 +66,24 @@ case class CreateHiveTableAsSelectCommand(
tableDesc.storage.locationUri.foreach { p =>
DataWritingCommand.assertEmptyRootPath(p, mode,
sparkSession.sessionState.newHadoopConf())
}
+ val outputColumns = DataWritingCommand.logicalPlanOutputWithNames(query,
outputColumnNames)
+ val tableSchema = CharVarcharUtils.getRawSchema(
+ outputColumns.toStructType, sparkSession.sessionState.conf)
+ assert(tableDesc.schema.isEmpty)
+ val newTableDesc = tableDesc.copy(schema = tableSchema)
if (tableExists) {
// `SaveAsV1TableCommand` runs this command with `SaveMode.Overwrite`
to replace the
- // table. Drop it here, after the analysis, because the new table is
created before the
- // write.
+ // table. Drop it here, after the analysis and the checks of the new
table, because the
+ // new table is created before the write.
+
sparkSession.sharedState.externalCatalog.unwrapped.asInstanceOf[HiveExternalCatalog]
Review Comment:
**Finding 1.** The dynamic-partition strict-mode check still runs after this
drop. So with the default `hive.exec.dynamic.partition.mode=strict`, a
partitioned Hive-format overwrite loses the table:
```scala
spark.range(3).write.format("hive").saveAsTable("t")
spark.range(5).selectExpr("id", "id % 2 AS p").write.format("hive")
.partitionBy("p").mode("overwrite").saveAsTable("t")
// SparkException: Dynamic partition strict mode requires at least one
static partition column. ...
spark.catalog.tableExists("t") // false
```
I ran it with `textfile` and `parquet`, both at the head and with the new
check turned off. Every run ends with `t` gone, so this isn't a regression.
`parquet` hits it too, because a partitioned CTAS is never converted
([`HiveStrategies.scala:282`](https://github.com/apache/spark/blob/415dc8789ad83db301ac3caeebe31414eb77c89d/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala#L282)).
`hive.exec.dynamic.partition=false` loses the table the same way.
The check is in
[`V1WritesHiveUtils.getDynamicPartitionColumns`](https://github.com/apache/spark/blob/415dc8789ad83db301ac3caeebe31414eb77c89d/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/V1WritesHiveUtils.scala#L74-L84).
`InsertIntoHiveTable` calls it for the table that `createTable` just created.
It only reads the new table's partition columns and the Hive conf. So it could
run before the drop too.
Fine as a follow-up. I didn't find a ticket for it. Could you file one if
you'd rather not fix it here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
