parthchandra opened a new issue, #7428: URL: https://github.com/apache/datafusion-comet/issues/7428
### What is the problem the feature request solves? On Spark 3.4, `CREATE TABLE AS SELECT` and `REPLACE TABLE AS SELECT` into Iceberg plan `AtomicCreateTableAsSelectExec` / `AtomicReplaceTableAsSelectExec`, and the staged table writes inline. These writes are not eligible for native Iceberg writes, so the write and the operators above the last shuffle stay in Spark. ### Describe the potential solution Plan the staged CTAS/RTAS write on Spark 3.4 through the same native Iceberg write path used for `AppendData`, and commit through the staged table. ### Additional context The Iceberg guide notes that CTAS/RTAS on Spark 3.4 writes inline today. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
