unikdahal opened a new issue, #6586: URL: https://github.com/apache/datafusion-comet/issues/6586
### What is the problem? Spark 4.2 injects catalog transactions into physical write nodes implementing `TransactionalExec`. Comet's split Iceberg write path replaces Spark's normal V2 commit node with `IcebergCommitExec`. Since this node is not transaction-aware, the transaction attached by Spark is not propagated or committed. This creates a correctness gap for Iceberg writes using transactional catalogs on Spark 4.2. The commit sequence should remain consistent with Spark: 1. commit the Iceberg `BatchWrite` 2. post driver metrics 3. commit the attached catalog transaction 4. refresh the table cache Earlier Spark versions are unaffected. ### Proposed fix Add a Spark 4.2-specific transaction-aware wrapper around `IcebergCommitExec` that implements `TransactionalExec` and preserves the attached transaction through Spark plan copies/clones. Keep the existing commit node unchanged for Spark 3.x, 4.0, and 4.1. ### Testing Add Spark 4.2 coverage verifying that split Iceberg writes expose a transaction-aware physical commit node and complete successfully. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
