dongjoon-hyun commented on code in PR #59783:
URL: https://github.com/apache/spark/pull/59783#discussion_r4379687689
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala:
##########
@@ -1155,8 +1155,14 @@ object DDLUtils extends Logging {
_.toLowerCase(Locale.ROOT) match {
case HIVE_PROVIDER =>
val serde = table.storage.serde
+ // Hive has no TIMESTAMP_NTZ type. Check the nested types too.
+ val timestampNTZColumn =
+
schema.find(_.dataType.existsRecursively(_.isInstanceOf[TimestampNTZType]))
if (schema.exists(_.dataType.isInstanceOf[AnsiIntervalType])) {
throw hiveTableWithAnsiIntervalsError(table.identifier)
+ } else if (timestampNTZColumn.isDefined) {
+ throw QueryCompilationErrors.dataTypeUnsupportedByDataSourceError(
Review Comment:
Thank you for measuring it, @peter-toth. You're right. I moved the check
from `DDLUtils.checkTableColumns` to `HiveExternalCatalog`, where Spark creates
or alters the Hive serde table, like SPARK-60493 does for the partition columns.
`SessionCatalog.createTable` calls `HiveExternalCatalog.createTable` only
when the table doesn't exist, and `CreateHiveTableAsSelectCommand` returns
earlier for `SaveMode.Ignore`. So, all three variants are no-ops again for an
existing table. I added a test case for them.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala:
##########
@@ -1155,8 +1155,14 @@ object DDLUtils extends Logging {
_.toLowerCase(Locale.ROOT) match {
case HIVE_PROVIDER =>
val serde = table.storage.serde
+ // Hive has no TIMESTAMP_NTZ type. Check the nested types too.
+ val timestampNTZColumn =
+
schema.find(_.dataType.existsRecursively(_.isInstanceOf[TimestampNTZType]))
Review Comment:
Thank you. SPARK-60493 (#59734) landed first, and I rebased this PR onto it.
So, such a partition column is now rejected before the existing table is
dropped and before the new table is created.
This PR also no longer adds the check to `DDLUtils.checkTableColumns`, so
`RelationConversions` doesn't run it after `createTable`. I ran your example
with `parquet` and `orc`. Both `saveAsTable` with `mode("overwrite")` and the
CTAS fail with `UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE` for `p`, and `t` keeps
its rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
