anshulbaliga7 commented on code in PR #58167:
URL: https://github.com/apache/spark/pull/58167#discussion_r3862061394
##########
core/src/main/scala/org/apache/spark/rdd/ReliableCheckpointRDD.scala:
##########
@@ -83,6 +85,16 @@ private[spark] class ReliableCheckpointRDD[T: ClassTag](
throw SparkCoreErrors.invalidCheckpointDirectoryError(path,
expectedFileName)
}
}
+ // If a partition-count metadata file is present, verify no trailing files
are missing.
+ // Directories written by earlier Spark versions have no such file; a
missing file is
+ // silently tolerated for backward compatibility. See SPARK-58883.
+ ReliableCheckpointRDD.readPartitionCountFromCheckpointDir(context,
checkpointPath)
+ .foreach { expected =>
+ if (inputFiles.length != expected) {
Review Comment:
No bypass for a fatal check. Added
`spark.checkpoint.verifyPartitionCount.enabled`, .internal(), defaults to true,
following the `spark.checkpoint.local.verifyChecksum.enabled` precedent you
pointed to.
Setting it to false loads a truncated directory as-is instead of throwing..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
