rdblue commented on a change in pull request #24832: [SPARK-27845][SQL]
DataSourceV2: InsertTable
URL: https://github.com/apache/spark/pull/24832#discussion_r300786324
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceResolution.scala
##########
@@ -118,6 +122,95 @@ case class DataSourceResolution(
if newColumns.forall(_.name.size == 1) =>
// only top-level adds are supported using AlterTableAddColumnsCommand
AlterTableAddColumnsCommand(table, newColumns.map(convertToStructField))
+
+ case i @
InsertTableStatement(UnresolvedRelation(CatalogObjectIdentifier(Some(catalog),
ident)),
+ _, _, _, _) if i.query.resolved =>
+ loadTable(catalog, ident)
+ .map(DataSourceV2Relation.create)
+ .map(table => {
+ // ifPartitionNotExists is append with validation, but validation is
not supported
+ if (i.ifPartitionNotExists) {
+ throw new AnalysisException(
+ s"Cannot write, IF NOT EXISTS is not supported for table:
${table.table.name}")
+ }
+
+ val staticPartitions =
i.partition.filter(_._2.isDefined).mapValues(_.get)
+
+ val resolver = conf.resolver
+
+ // add any static value as a literal column
+ val staticPartitionProjectList = {
+ // check that the data column counts match
+ val numColumns = table.output.size
+ if (numColumns > staticPartitions.size + i.query.output.size) {
Review comment:
The `ResolveOutputRelation` rule will check this and produces a more useful
error message:
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L2336-L2339
I think this rule should simply transform the query assuming that it will
work, and let `ResolveOutputRelation` ensure that the types and number of
columns align.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
