Hi all, My name is Diogo and I am a dev for Paddy Power Betfair in Porto, Portugal. We're running Storm 1.x.x in production for a couple of years and the time has come for us to upgrade to 2.0.0. We use *LocalCluster* to run topologies in our local machines to perform manually tests.
So, going to the point: I was trying to launch a topology that I'm developing (in 2.0.0) and noticed that the worker was getting restarted each ~30 seconds. I placed a breakpoint in the *kill* method of *LocalContainer* ( https://github.com/apache/storm/blob/2ba95bbd1c911d4fc6363b1c4b9c4c6d86ac9aae/storm-server/src/main/java/org/apache/storm/daemon/supervisor/LocalContainer.java#L66) to try and understand why the worker was getting restarted. The call stack was: kill:66, LocalContainer (org.apache.storm.daemon.supervisor) killContainerFor:269, Slot (org.apache.storm.daemon.supervisor) handleRunning:724, Slot (org.apache.storm.daemon.supervisor) stateMachineStep:218, Slot (org.apache.storm.daemon.supervisor) run:931, Slot (org.apache.storm.daemon.supervisor) With this I can understand that the worker is killed because a blob has changed ( https://github.com/apache/storm/blob/2ba95bbd1c911d4fc6363b1c4b9c4c6d86ac9aae/storm-server/src/main/java/org/apache/storm/daemon/supervisor/Slot.java#L724). In fact, there's a changing blob in the *dynamicState* at that point. I checked the *AsyncLocalizer *which downloads, caches blobs locally, and notifies the Slot state machine of a changing blob. I noticed this: - https://github.com/apache/storm/blob/2ba95bbd1c911d4fc6363b1c4b9c4c6d86ac9aae/storm-server/src/main/java/org/apache/storm/localizer/AsyncLocalizer.java#L339 - https://github.com/apache/storm/blob/2ba95bbd1c911d4fc6363b1c4b9c4c6d86ac9aae/storm-server/src/main/java/org/apache/storm/localizer/AsyncLocalizer.java#L265 - https://github.com/apache/storm/blob/2ba95bbd1c911d4fc6363b1c4b9c4c6d86ac9aae/storm-server/src/main/java/org/apache/storm/localizer/LocallyCachedTopologyBlob.java#L142 - https://github.com/apache/storm/blob/2ba95bbd1c911d4fc6363b1c4b9c4c6d86ac9aae/storm-server/src/main/java/org/apache/storm/localizer/LocallyCachedTopologyBlob.java#L192 Which tell me that (correct me if I'm wrong): - Supervisor tries to update blobs each 30 seconds. - The topology jar blob requires extraction of the resources directory (either from a jar or directly in a classpath URL). It does so in *fetchUnzipToTemp *and it's existence is checked in *isFullyDownloaded*. - The Slot is notified of a changing blob if: - the remote version is different from the local version (the code has changed). - OR the blob is not fully downloaded (the jar exists, and the extracted resources directory exists). Well, I did not have a resources folder under the root of the classpath, and that's why the worker was being restarted each ~30 seconds, as the Slot was being notified of a changing blob everytime *updateBlobs* ran. I created a resources folder (with dummy files) under the root of the classpath and the problem is now solved. However, if I understand correctly, the resources folder is only required for *multilang*. Our topologies do not use *multilang *and this do not happen in Storm 1.1.3 for instance. Am I seeing or doing something wrong and this is an expected behaviour? I am happy to contribute if this is in fact something worth to open an issue and fix. Hope this is the right place for these questions, and thanks in advance for taking your time to look at this. Regards, Diogo
