sryza opened a new pull request, #51497: URL: https://github.com/apache/spark/pull/51497
This reverts commit 8b437570fd1e93ae3f3fad09482f62727cca77db. ### What changes were proposed in this pull request? Reverts SPARK-52576. I.e. truncates + alters instead of drop + recreate, for materialized views and full refreshes. ### Why are the changes needed? Some pipeline runs result in wiping out and replacing all the data for a table: - Every run of a materialized view - Runs of streaming tables that have the "full refresh" flag Prior to SPARK-52576, this "wipe out and replace" was implemented by: - Truncating the table - Altering the table to drop/update/add columns that don't match the columns in the DataFrame for the current run However, we discovered that this didn't work on Hive. So we moved to drop + recreate, which did work on Hive. However, compared to truncate + alter, drop + recreate has some undesirable effects. E.g. it interrupts readers of the table and wipes away things like ACLs. This Hive behavior was fixed here: https://github.com/apache/spark/pull/51007. So now we can switch back to truncate + alter. ### Does this PR introduce _any_ user-facing change? Yes, described above ### How was this patch tested? Existing tests ### Was this patch authored or co-authored using generative AI tooling? <!-- If generative AI tooling has been used in the process of authoring this patch, please include the phrase: 'Generated-by: ' followed by the name of the tool and its version. If no, write 'No'. Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details. --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
