snmvaughan opened a new pull request, #59644: URL: https://github.com/apache/spark/pull/59644
### What changes were proposed in this pull request? Backport of #55676 to branch-4.1. The cherry-pick conflicted only in the import list of `AsyncProgressTrackingMicroBatchExecutionSuite`, because this branch already imports `OffsetSeq` there. The resolved imports match master, and the rest of the change is identical to #55676. `AsyncOffsetSeqLog` and `AsyncCommitLog` now share the query's `ErrorNotifier`. After a write has failed: - Offset and commit log write tasks that run later fail with the first error and don't write to the checkpoint. - `ErrorNotifier.markError` keeps the first error and attaches later ones to it as suppressed exceptions, so a cascading failure can't replace the root cause. ### Why are the changes needed? With async progress tracking, offset and commit log entries are written by a single-threaded executor. When the write of `offsets/N` fails, a `commits/N` write queued after it can still succeed. That leaves a commit log entry without its offset log entry in the checkpoint. The query can also fail with a later, cascading error instead of the I/O error that started it. The same gap exists on branch-4.1. SPARK-60264 (#59518), which is already on branch-4.1, makes `AbortableStreamBasedCheckpointFileManager` report failed writes instead of only logging them. With async progress tracking, a commit log write queued behind such a failure can still go through without this change. ### Does this PR introduce _any_ user-facing change? No. As in #55676, a query that hits a write failure still stops with a `StreamingQueryException`, which now reports the original error. ### How was this patch tested? Includes the regression test from #55676, "async log writes record first error and gate subsequent writes", and its assertion change in `testAsyncWriteErrorsPermissionsIssue`, both in `AsyncProgressTrackingMicroBatchExecutionSuite`. The changed main and test sources compile against the Spark 4.1.3 release jars with this branch's scalac options, and pass scalastyle. The suite was not run locally; this PR's CI runs it. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
