anshulbaliga7 commented on PR #58167: URL: https://github.com/apache/spark/pull/58167#issuecomment-5910785074
Thanks @cloud-fan for the review , have addressed all five: - Setting doc: Rewritten as a read-time switch where publication is always attempted best-effort regardless of the value, the flag skips only the comparison and it has no effect when the file is absent for either reason. - Direction-aware mismatch: Replaced `CHECKPOINT_TRUNCATED_DIRECTORY` with `CHECKPOINT_DIRECTORY_PARTITION_COUNT_MISMATCH` and two sub-conditions, `MISSING_FILES` and `UNEXPECTED_FILES`, selected by comparing found against recorded. The parent message is direction-neutral with path and both counts; each sub-condition carries the direction-specific diagnosis and the bypass hint. I renamed rather than only rewording since "truncated" is wrong for the surplus case. - Surplus coverage: New test with recorded count 4 and contiguous part-00000 through part-00004 asserts `UNEXPECTED_FILES` with expected 4, found 5, and that the disabled switch loads all 5. The existing missing-file tests now assert `MISSING_FILES`. - Missing metadata: The `getPartitions` scaladoc and comment, the reader scaladoc and the debug log now name both causes an older Spark version or a failed best-effort write by a current one. - Delete result: A false return from `delete` now logs the cleanup warning, same as an exception. The four existing publication-failure cases additionally assert no cleanup warning is emitted. A new case injects a throwing rename plus a false delete and asserts both the publication and cleanup warnings, the leftover temp file, and successful recovery with 4 partitions. PR description updated for the renamed condition and the current test coverage. Can you PTAL again? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
