viirya opened a new pull request, #6821: URL: https://github.com/apache/datafusion-comet/pull/6821
## Which issue does this PR close? Closes #6820. ## Rationale for this change On Spark 3.x `CometNativeWriteExec` stands in for the whole `InsertIntoHadoopFsRelationCommand`, but only runs the commit lifecycle. After a native write, cached DataFrames and cached tables over the output path return stale rows, the table relation's file listing is stale (the Spark 3.x side of #3521), and table stats are neither updated nor cleared. It also skips the command's duplicate-column check, so a write Spark rejects succeeds natively. Spark 4.0+ is not affected because Spark's command still runs there. ## What changes are included in this PR? `CometNativeWriteExec` now does what the command it replaces does, using the command it already keeps as `originalPlan`: - before writing, it rejects duplicate output column names with `SchemaUtils.checkColumnNameDuplication` - after the job commits, it refreshes the command's `fileIndex`, calls `cacheManager.recacheByPath` for the output path, and calls `CommandUtils.updateTableStats` when the command has a catalog table These steps run after the commit block, so a failure there does not abort a job that has already committed. As in Spark, they are skipped when `SaveMode.Ignore` finds an existing target. Partition metadata updates are not needed because partitioned writes are never converted. ## How are these changes tested? New tests in `CometParquetWriterSuite`, which run on every Spark profile: - a cached path is recached after a native overwrite - a cached table is recached after a native `INSERT` - table stats after native `INSERT INTO` / `INSERT OVERWRITE`, with `spark.sql.statistics.size.autoUpdate.enabled` on (stats equal the data size) and off (stale stats cleared) - a write with case-insensitively duplicate column names fails with `COLUMN_ALREADY_EXISTS` and writes nothing The existing "INSERT INTO ... SELECT is visible to subsequent reads" test no longer assumes Spark 4.0+. On Spark 3.5 all of these fail without the fix and pass with it. `CometParquetWriterSuite`, `CometNativeWriteSuite` and `RevertNativeForTransitionHeavyStagesSuite` pass on Spark 3.4, 3.5 and 4.1. On 4.x the new tests pass through Spark's own command. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
