manuzhang opened a new issue, #18951: URL: https://github.com/apache/iceberg/issues/18951
### Feature Request / Improvement Track the checkpoint failure paths explicitly left untested by #18340, #18339, and #18294. Those verification runs exercised the default local checkpoint manager and ordinary restart behavior. They did not inject an S3A/abortable checkpoint stream close failure or exercise Spark async progress-tracking write failures. [TestStructuredStreaming on main](https://github.com/apache/iceberg/blob/234d55774d3c7355f92c8e93f8e117b70726e556/spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java) has local checkpoint/restart coverage, but that does not establish these failure contracts. Two scenarios need an applicability check and focused integration coverage: - A failed close of the abortable checkpoint output stream: verify the original I/O failure reaches the query and an incomplete write is not treated as a valid checkpoint. - Async offset/commit-log failures, following [SPARK-56720](https://github.com/apache/spark/pull/55676): after the first failure, queued later writes must not create durable checkpoint entries, and later errors must not replace the original cause. Acceptance criteria: - [ ] Establish a supported Iceberg source/sink query and checkpoint-manager configuration for each scenario. Document Spark restrictions where a combination is unsupported. - [ ] Inject failures deterministically using an appropriate test fixture; do not require live cloud credentials. - [ ] Assert query failure, preserved root cause, and the durable offset/commit-log state. - [ ] Check restart behavior under the supported tracking mode's documented delivery guarantees. - [ ] Cover applicable Spark 4.0/4.1/4.2 lines without conflating Spark async progress tracking with Iceberg async scan planning. This is a request to close the stated verification gaps; the drafts do not demonstrate an Iceberg failure in these paths. The checkpointed batch-scan join tests from #18340 belong to the SPJ coverage follow-up. ### Query engine Spark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
