rdblue commented on issue #23836: [SPARK-26915][SQL] DataFrameWriter.save() should write without schema validation URL: https://github.com/apache/spark/pull/23836#issuecomment-466100319 > the current translation is not precise: append mode doesn't mean append, it's actually "create table if not exist or append table" Agreed. This is why we need to get CTAS finished. > The next problem is, how to implement "create table if not exist or append table" with ds v2 APIs. My understanding is that the plan is to do both. If a catalog supports staged tables, then Spark uses them to perform an atomic operation. If it doesn't, then Spark uses the create/append/drop-on-error strategy. I agree that option 2 is "better" in that the operation is atomic. But sources are not required to support atomic CTAS. We need both options, so they are not mutually exclusive.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
