juliuszsompolski opened a new pull request, #57310: URL: https://github.com/apache/spark/pull/57310
### What changes were proposed in this pull request? Follow-up to [SPARK-58107](https://issues.apache.org/jira/browse/SPARK-58107) (dedup-and-seal for non-deterministic local checkpoints). That change added `BlockManagerMaster.sealRddChecksums`, which picks one authoritative content checksum per RDD block, evicts divergent replicas, and rejects future divergent registrations. This adds a read-only companion to inspect the result of a seal: - `BlockManagerMaster.verifyRddChecksumSeal(rddId): Option[String]` (with the endpoint handler and the `VerifyRddChecksumSeal` message). It checks, for every block of `rddId` in the directory, that the block is sealed and that every replica the directory tracks for it carries the sealed content checksum. Returns `None` when the invariant holds, or `Some(diagnostic)` naming the first block that violates it (present-but-unsealed, or a tracked replica whose checksum differs from the seal). It runs on the master's message-handler thread, so it observes a consistent snapshot of `blockLocations`, `blockChecksums`, and `sealedChecksums`. It also makes `LOCAL_CHECKPOINT_VERIFY_CHECKSUM_ENABLED` a non-`private[spark]` config `val`, so callers outside the `org.apache.spark` packages can gate behavior on whether the seal feature is enabled without hardcoding the config key. ### Why are the changes needed? `sealRddChecksums` mutates the block directory (evicts and reseals), but there was no way to assert afterward that the directory ended up in the intended consistent state — every tracked replica agreeing on the sealed checksum. `verifyRddChecksumSeal` gives callers and tests a cheap invariant check: under correct operation it always returns `None`, so a non-`None` result is a tripwire for a seal-machinery regression or an unexpected directory state. The existing seal tests only asserted on locations and the sealed value; they did not assert the post-seal directory is internally consistent. ### Does this PR introduce _any_ user-facing change? No. `verifyRddChecksumSeal` is an internal master API with no public surface, and `LOCAL_CHECKPOINT_VERIFY_CHECKSUM_ENABLED` is an internal (`.internal()`) config; widening its Scala visibility does not change any user-facing behavior or default. ### How was this patch tested? `build/sbt "core/testOnly org.apache.spark.storage.BlockManagerSuite"`: - A new unit test drives the branches directly: no blocks for an RDD -> `None`; two agreeing replicas after a seal -> `None`; a present-but-unsealed block -> `Some(...)`. - The existing seal tests (agreeing disk and `_SER` replicas survive the seal; divergent copies converge after eviction; a replicated seal-path block) now additionally assert `verifyRddChecksumSeal === None`, exercising the check on genuinely sealed block state. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
