shlomitubul commented on PR #3706: URL: https://github.com/apache/celeborn/pull/3706#issuecomment-4642196186
Thanks @SteNicholas — your blocking analysis was exactly right. Tracing it through `commitFiles` and the driver confirmed that queued/interrupted tasks land in none of the `committed` / `empty` / `failed` sets, and `checkDataLost` keys only off the failed sets, so an in-flight partition was indistinguishable from an empty one and would silently produce wrong results with no `FetchFailure`. I've opened #3721 targeting `main` with the fix: - failed lists computed as `requested − committed − empty` (your suggested fix), so every not-actually-committed partition is recomputed by the driver; - the response-building extracted into `Controller.buildCommitFilesResponseOnCancel` and covered by a new `ControllerSuite` (committed preserved, in-flight → failed, empty → not failed; and the nothing-committed → `COMMIT_FILE_EXCEPTION` case); - `context.reply()` + `stopTimer()` on the error path, as in this PR. Closing this one in favor of #3721. Thanks again for the careful review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
