mridulm commented on PR #3569: URL: https://github.com/apache/celeborn/pull/3569#issuecomment-3698657768
@CodingCat I understand the motivation and rationale behind the proposal better now - and there are existing alternatives like checkpoint'ing, temp materialization, etc. This will be more on the application design/impl side though - and apps will have to do their own tradeoffs (given there are costs involved - it is not free). As I mentioned earlier, this proposal itself is inherently unsound, and given additional details provided, I am not in favor of introducing it into Apache Celeborn - my rationale/analysis would be the same if a variant of the proposal was made to Apache Spark for "vanilla" shuffle as well :-). I am open to being corrected ofcourse if there are valid usecases and/or requirements I am missing ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
