rzo1 commented on issue #7653: URL: https://github.com/apache/storm/issues/7653#issuecomment-5616042679
Thanks for testing and for the detailed logs. Reopening, since the blob can still leak. The cleanup does run and deletes the blob. The two `set-path` lines afterwards (`…:0-0`, then `…:0-1`) suggest that the leader recreates it about 30 seconds later: a supervisor's blob update check (default every 30s) calls `getBlobMeta` on the leader, `checkForBlobOrDownload` finds the key registered in ZooKeeper by another Nimbus, and the leader downloads the blob again from that Nimbus. The version numbers `0` and then `1` only fit if another Nimbus had registered the key again (at version `1`) after the leader deleted it. After that, nothing deletes the dependency blob, because the topology is already gone. To confirm this and find out which step re-registers the key, could you please provide: 1. How many Nimbus instances you run (is there a `stormnimbus-0` besides `stormnimbus-1`?), and which one was the leader at that time. 2. The full `nimbus.log` of **all** Nimbus instances for a failing case, covering the upload through a few minutes after `Removing dependency blobs of …`. Please don't filter by the key: some relevant lines don't contain it (e.g. `Exception` errors logged by `KeySequenceNumber`). 3. If you can, the `supervisor.log` of a supervisor that ran a worker of that topology, for the same time window. For your excerpt above, that would be roughly 05:13–05:25 for `dep-apeframework-0.0.11710-SNAPSHOT-f2d87a33-a8ad-4a95-a567-71e96a2b8695.jar`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
