unikdahal opened a new pull request, #6587: URL: https://github.com/apache/datafusion-comet/pull/6587
Closes #6586 ## Summary Spark 4.2 injects catalog transactions into physical `TransactionalExec` nodes. Comet's split Iceberg write path currently replaces the normal Spark commit node with `IcebergCommitExec`, which means the attached transaction is not preserved. This PR: - adds a Spark 4.2-specific transaction-aware Iceberg commit node - preserves the transaction across Spark plan copies/clones - keeps the commit ordering as write commit → driver metrics → transaction commit → cache refresh - leaves Spark 3.x, 4.0, and 4.1 behavior unchanged through version-specific shims ## Tests - adds Spark 4.2 coverage verifying split Iceberg writes use a transaction-aware commit node - existing Iceberg split-write coverage continues to pass -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
