Github user danny0405 commented on the issue:
I agree with you that the race condition is between nimbus deleting the
blobs and the supervisor fully processing the topology being killed.
But i still think we should handle this race condition reasonably as a
normal and expectant case. It's a distribution system, so can the
AsyncLocalizer decide based on master blob keys' existence when update local
resources? A thrown stack trace it really confusing and non-necessary,
especially when the blob-client try as configured max times to fetch a blob
already removed from cluster.
At lease we should not make a downloading request when we have removed the
blobs, and should not decide the blob's existence based on a thrown
KeyNotFoundException[ confusing too, because user have already killed/removed
them on their own initiative ].