dongjoon-hyun opened a new pull request, #59643:
URL: https://github.com/apache/spark/pull/59643
[SPARK-60387][SQL] Keep the existing table when `saveAsTable` overwrite
fails before writing
### What changes were proposed in this pull request?
This PR makes `saveAsTable` with `SaveMode.Overwrite` drop the existing V1
table only after the checks that run before writing, such as the analysis, the
data schema check, and the write options.
Previously, `SaveAsV1TableCommand` dropped the table before running the
CTAS. Now the CTAS drops it:
- `CreateDataSourceTableAsSelectCommand` drops it right before the write job
starts, in the `beforeSetupJob` callback of `FileFormatWriter.write`
(SPARK-60360), or right before `CreatableRelationProvider.createRelation`.
- `CreateHiveTableAsSelectCommand` drops it after the analysis, because it
creates the new table before writing.
### Why are the changes needed?
When these checks fail, the table has already been dropped. A managed table
loses its data too.
```scala
spark.range(3).write.saveAsTable("t")
spark.range(5).write.mode("overwrite").option("compression",
"bogus").saveAsTable("t")
// SparkIllegalArgumentException:
[CODEC_NOT_AVAILABLE.WITH_AVAILABLE_CODECS_SUGGESTION] ...
spark.table("t")
// AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view `t` cannot
be found. ...
```
The same happens with an unsupported data type (e.g. a map column in CSV) or
a nonexistent partition column. This has been the behavior since Spark 1.3.0.
### Does this PR introduce _any_ user-facing change?
Yes. Such an overwrite keeps the existing table and its data. Valid writes
are unchanged.
### How was this patch tested?
Added tests to `DataFrameReaderWriterSuite` and `MetastoreDataSourcesSuite`.
They fail without this fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
