waitinfuture commented on PR #2373: URL: https://github.com/apache/celeborn/pull/2373#issuecomment-2078878530
> The data correctness issues, unless we can mitigate it, is what makes it risky for me. Totally Agree! Correctness is NO.1 important thing. Based on previous discussion, correctness will only happen when stage rerun, I think we both agree that if this PR takes affects, we will not allow stage rerun, right? > Based on my current read, this does have correctness implications. I would suggest we should do either or all of the following: > > a) If recomputation happens, we should fail the stage and not allow retries - this will prevent data loss. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
