mridulm commented on PR #37922: URL: https://github.com/apache/spark/pull/37922#issuecomment-1251349810
> The push-based shuffle service will auto clean up the old shuffle merge data Consider the case I mentioned above - stage retry for an `INDETERMINATE` stage. We cleanup previous attempts only if it happens to use the same merger in new attempt as well. The previous attempt's mergers are not reused for the next attempt - and so the previous mergers will continue to hold stale data without cleaning them up - until application terminates. Note - any merger which happens to be reused in new stage attempt will cleanup - I am referring to those which are not used: old attempt `mergerLocs` `--` new attempts `mergerLocs`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
