brkyvz commented on a change in pull request #25305: [SPARK-28572][SQL] Simple
analyzer checks for v2 table creation code paths
URL: https://github.com/apache/spark/pull/25305#discussion_r312122520
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala
##########
@@ -236,6 +237,46 @@ case class PreprocessTableCreation(sparkSession:
SparkSession) extends Rule[Logi
c.copy(tableDesc = normalizedTable.copy(schema = reorderedSchema))
}
+
+ case create: V2TableCreation =>
+ val schema = create.tableSchema
+ val partitioning = create.partitioning
+ val identifier = create.tableName
+ val isCaseSensitive =
sparkSession.sessionState.conf.caseSensitiveAnalysis
+ // Check that columns are not duplicated in the schema
+ val flattenedSchema = SchemaUtils.explodeNestedFieldNames(schema)
+ SchemaUtils.checkColumnNameDuplication(
+ flattenedSchema,
+ s"in the table definition of $identifier",
+ isCaseSensitive)
+
+ // Check that columns are not duplicated in the partitioning statement
+ SchemaUtils.checkTransformDuplication(
+ partitioning, "in the partitioning", isCaseSensitive)
+
+ if (schema.isEmpty) {
Review comment:
no. Existing data may exist for a table, and all that you may want is to
create a pointer to that table (Think of creating a pointer to a JDBC table in
the MetaStore)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
