pratham76 opened a new pull request, #59777: URL: https://github.com/apache/spark/pull/59777
### What changes were proposed in this pull request? This PR enables `spark.sql.streaming.checkpoint.renamedFileCheck.enabled` by default: - `SQLConf`: the default changes from `false` to `true`, and the doc string now describes what the check does. - `ss-migration-guide.md`: a 4.3 to 4.4 note describing the new default and how to restore the previous behavior. - `CheckpointFileManagerSuite`: a test with a file system whose rename reports success without moving the file. With the default, `createAtomic` now fails with the existing "does not exist" error. With the flag set to `false`, the write still succeeds silently and no file is created. ### Why are the changes needed? I encountered this in production with a Structured Streaming query on Azure Data Lake Storage Gen2 (ABFS). A state store delta file was never created under its final name, while its temporary file was left behind, and no error was raised. The batch was committed, and the query kept running because executors serve recent state versions from memory. On the next restart, rebuilding state from disk failed with `Error reading delta file ... does not exist`, and every later restart failed the same way, so the checkpoint needed manual repair. Checkpoint files are written to a temporary file and then renamed into place, and the rename is trusted as long as it does not throw. SPARK-40826 added a check that the renamed file exists, but it is off by default. With the check enabled, the same failure is caught at commit time, before the batch is recorded in the commit log, so the task is retried and the query recovers without intervention. More details and logs are in SPARK-60533. ### Does this PR introduce _any_ user-facing change? Yes. By default, a checkpoint write whose rename does not produce the final file now fails with `IllegalStateException: Renamed temp file ... But ... does not exist.` instead of succeeding silently and leaving a checkpoint that cannot be restarted. Each checkpoint file write does one extra existence check. Setting `spark.sql.streaming.checkpoint.renamedFileCheck.enabled` to `false` restores the previous behavior. ### How was this patch tested? - New test in `CheckpointFileManagerSuite` + Existing UTs ### Was this patch authored or co-authored using generative AI tooling? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
