rdblue commented on a change in pull request #25348: [RFC][SPARK-28554][SQL]
Adds a v1 fallback writer implementation for v2 data source codepaths
URL: https://github.com/apache/spark/pull/25348#discussion_r310743932
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala
##########
@@ -83,12 +85,17 @@ case class CreateTableAsSelectExec(
catalog.createTable(
ident, query.schema, partitioning.toArray, properties.asJava) match {
case table: SupportsWrite =>
- val batchWrite = table.newWriteBuilder(writeOptions)
+ val writer = table.newWriteBuilder(writeOptions)
.withInputDataSchema(query.schema)
.withQueryId(UUID.randomUUID().toString)
- .buildForBatch()
- doWrite(batchWrite)
+ writer match {
+ case v1: V1WriteBuilder =>
+ val mode = if (ifNotExists) SaveMode.Ignore else
SaveMode.ErrorIfExists
Review comment:
The table was just created above, so both of these modes are incorrect. This
should be `SaveMode.Append` because the table already exists.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
