agrawaldevesh commented on pull request #28370: URL: https://github.com/apache/spark/pull/28370#issuecomment-642362076
Hi @holdenk and @prakharjain09 I am trying to follow along the great improvements you have been making to the Spark's decommissioning story. I had two basic questions please: - Is my understanding correct that this PR is only for the blocks cached using the `persist()` API and not the shuffle blocks ? I see that there is another PR in the community https://github.com/apache/spark/pull/27636# which deals with shuffle files elegantly by forgetting about them and forcing the driver to recompute them. I am curious how that fits in with this approach ? > otherwise I'll get that started after the shuffle block stuff :) - Is this the PR https://github.com/apache/spark/pull/28708 that is working on improving shuffle files + decommission story (so called Part 2 and 3 in the [design doc](https://docs.google.com/document/d/1yMulI51dQqqrBO5IcSPKCiaNllkmX3NSAgsIvfdoDx4/edit#heading=h.427iwkc3ai3g). Would you say that we would be done with Parts 2 and 3, as mentioned in the design doc above after the PR https://github.com/apache/spark/pull/28708 is merged in ? Thanks for improving this key aspect of Spark in cloud environments ! ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
