CodingCat commented on PR #3109: URL: https://github.com/apache/celeborn/pull/3109#issuecomment-2855937384
> @CodingCat: > > > we already know for determinant stages we should not fail/restart certain stages in some scenario > > We are not failing/restarting for determinate stages - only for indeterminate stages. We dont need to do that for determinate stages - shuffle can be reused across all cases. > > Please do let me know if I am missing something/scenarios ! You have been looking into this for Celeborn in this PR much longer than me :-) Hi, @mridulm I think what you said makes sense, I simplified the code but still keeps the basic structure like keeping shuffleCleaner in spark modules, etc. the reason is that I think it won't leak the Spark-specific assumption like the format of appShuffleIdentifier to LifecycleManager layer, -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
