Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/21390
Thanks @JoshRosen for very detailed and thoughtful reply.
Agreed TTL could be fragile, but I was very concern with what point
> There is a related issue where shuffle files can be leaked indefinitely
following executor death because the external shuffle service is never directly
told that shuffles are safe to remove (the context cleaner sends RPCs to
executors and executors clean up their own shuffle files). That issue is
substantially harder to fix, though, since it likely requires protocol changes
to the shuffle service or an inversion-of-control where the shuffle service can
periodically ask the driver "do any of these shuffle IDs correspond to cleaned
shuffles?".
So will probably follow up with you at some point.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
