james-willis opened a new pull request, #57976: URL: https://github.com/apache/spark/pull/57976
### What changes were proposed in this pull request? `ReliableCheckpointRDD.writePartitionToCheckpointFile` renames the attempt-temp file onto the final part file and only handles a rename that reports failure by returning `false` (HDFS semantics). This PR additionally catches `FileAlreadyExistsException` from that rename and routes it into the same existing handling: if the final part file exists, another attempt of this task already committed it, so the temp file is deleted and the write is treated as successful. If the destination does not exist, the existing `checkpointFailedToSaveError` is still thrown. ### Why are the changes needed? Since [HADOOP-16721](https://issues.apache.org/jira/browse/HADOOP-16721) (Hadoop 3.3.1), S3A deliberately raises `FileAlreadyExistsException` when the rename destination is an existing file, instead of returning `false` as HDFS does. ABFS behaves the same way. The Hadoop FileSystem specification does not guarantee HDFS-style `false` reporting. Under speculative execution (or a zombie attempt racing a retry), two attempts of the same checkpoint task race to rename onto the same final part file. On HDFS the loser sees `rename() == false` and Spark correctly treats it as "some other copy of this task must've finished before us". On S3A/ABFS the loser gets an unhandled `FileAlreadyExistsException`, which fails the task — and because the destination now permanently exists, **every retry of that task fails on the same rename**, so `spark.task.maxFailures` is always exhausted and the job aborts, even though the checkpoint data was written successfully by the winning attempt. Observed in production (Spark 4.0.1, Hadoop 3.4.1, `spark.speculation=true`): ``` org.apache.hadoop.fs.FileAlreadyExistsException: Failed to rename s3://<bucket>/<prefix>/spark-checkpoints/<uuid>/rdd-207/.part-00379-attempt-25364 to s3://<bucket>/<prefix>/spark-checkpoints/<uuid>/rdd-207/part-00379; destination file exists at org.apache.hadoop.fs.s3a.S3AFileSystem.initiateRename(S3AFileSystem.java:2468) at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:2392) at org.apache.spark.rdd.ReliableCheckpointRDD$.writePartitionToCheckpointFile(ReliableCheckpointRDD.scala:229) ... ERROR TaskSetManager: Task 379 in stage 105.0 failed 4 times; aborting job ``` See [SPARK-58750](https://issues.apache.org/jira/browse/SPARK-58750) for full details. Structured Streaming's `CheckpointFileManager` was already hardened for divergent rename semantics; the RDD checkpoint writer is the remaining caller assuming HDFS semantics. ### Does this PR introduce _any_ user-facing change? No. RDD checkpointing to S3A/ABFS under speculative execution (or task retry after a committed rename) now succeeds instead of unrecoverably failing the job, which is the bug fix itself. ### How was this patch tested? Added a regression test to `CheckpointStorageSuite` that writes the same checkpoint partition from two task attempts against a `FileSystem` mimicking S3A's rename semantics (raises `FileAlreadyExistsException` when the destination file exists). Without the fix, the second attempt throws and would fail the task; with the fix, it succeeds and exactly one committed part file remains, with the attempt-temp file cleaned up. Ran `build/sbt "core/testOnly org.apache.spark.CheckpointStorageSuite"` locally. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (model claude-fable-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
