anshulbaliga7 opened a new pull request, #58167: URL: https://github.com/apache/spark/pull/58167
### What changes were proposed in this pull request? Persist the original RDD's partition count to a `_num_partitions` file in the checkpoint directory at write time, and validate it in `ReliableCheckpointRDD.getPartitions` on read. A missing file (pre-existing checkpoints) or unreadable file is tolerated for backward compatibility. ### Why are the changes needed? `getPartitions` only checks that part-* files are contiguous from `part-00000`. Deleting a middle file breaks contiguity and is caught, but deleting the trailing file leaves the rest contiguous, so the check passes and the RDD is silently read back with fewer partitions and no error. This mainly affects `SparkContext.checkpointFile`, which Spark Streaming recovery uses to rebuild `generatedRDDs`, since that path has no original RDD to compare against. Split out of SPARK-58770 (#58004), which deliberately left this out as needing its own on-disk format. ### Does this PR introduce any user-facing change? Yes. Checkpoint directories now get one extra small metadata file. Reading back a directory missing its trailing partition file(s) now throws `CHECKPOINT_RDD_PARTITION_COUNT_MISMATCH` instead of silently returning fewer partitions. Directories written before this change read exactly as before. ### How was this patch tested? Added three tests to `CheckpointStorageSuite`: - reading a checkpoint with the trailing partition file deleted now throws - a checkpoint missing `_num_partitions` (pre-existing checkpoint) still reads back correctly - a corrupted `_num_partitions` file is tolerated and still reads back correctly ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
