karuppayya commented on issue #25995: SPARK-29324: Fix overwrite behaviour for saveAsTable URL: https://github.com/apache/spark/pull/25995#issuecomment-537824060 Without the change, an external table become a managed table, the file format could change as well(Say table was created using Orc file format, after overwrite it will change to parquet which is the default FF) Few issues 1. Any other user who is consuming the table will be querying the data at an older location, instead of the new data. 2. In case using overwriting a partitioned table with `partitionOverwriteMode` set to `dynamic`, I think the expectation would be to overwrite the data of the only affected partitions with the same file format(Please correct me if I am wrong). In this case, we will end up writing in a completely different location and maybe with different file format.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
