manuzhang opened a new issue, #18951:
URL: https://github.com/apache/iceberg/issues/18951

   ### Feature Request / Improvement
   
   Track the checkpoint failure paths explicitly left untested by #18340, 
#18339, and #18294.
   
   Those verification runs exercised the default local checkpoint manager and 
ordinary restart behavior. They did not inject an S3A/abortable checkpoint 
stream close failure or exercise Spark async progress-tracking write failures. 
[TestStructuredStreaming on 
main](https://github.com/apache/iceberg/blob/234d55774d3c7355f92c8e93f8e117b70726e556/spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java)
 has local checkpoint/restart coverage, but that does not establish these 
failure contracts.
   
   Two scenarios need an applicability check and focused integration coverage:
   
   - A failed close of the abortable checkpoint output stream: verify the 
original I/O failure reaches the query and an incomplete write is not treated 
as a valid checkpoint.
   - Async offset/commit-log failures, following 
[SPARK-56720](https://github.com/apache/spark/pull/55676): after the first 
failure, queued later writes must not create durable checkpoint entries, and 
later errors must not replace the original cause.
   
   Acceptance criteria:
   
   - [ ] Establish a supported Iceberg source/sink query and checkpoint-manager 
configuration for each scenario. Document Spark restrictions where a 
combination is unsupported.
   - [ ] Inject failures deterministically using an appropriate test fixture; 
do not require live cloud credentials.
   - [ ] Assert query failure, preserved root cause, and the durable 
offset/commit-log state.
   - [ ] Check restart behavior under the supported tracking mode's documented 
delivery guarantees.
   - [ ] Cover applicable Spark 4.0/4.1/4.2 lines without conflating Spark 
async progress tracking with Iceberg async scan planning.
   
   This is a request to close the stated verification gaps; the drafts do not 
demonstrate an Iceberg failure in these paths. The checkpointed batch-scan join 
tests from #18340 belong to the SPJ coverage follow-up.
   
   ### Query engine
   
   Spark
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to