Yuzhao Chen created STORM-2878:
----------------------------------

             Summary: Supervisor collapse continuously when there is a expired 
assignment for overdue storm
                 Key: STORM-2878
                 URL: https://issues.apache.org/jira/browse/STORM-2878
             Project: Apache Storm
          Issue Type: Bug
          Components: storm-core, storm-server
    Affects Versions: 2.0.0, 1.x
            Reporter: Yuzhao Chen
            Priority: Critical
             Fix For: 2.0.0


For now, when a topology is reassigned or killed for a cluster, supervisor will 
delete 4 files for an overdue storm:
- storm-code
- storm-ser
- storm-jar
- LocalAssignment

Slot.java
static DynamicState cleanupCurrentContainer(DynamicState dynamicState, 
StaticState staticState, MachineState nextState) throws Exception {
        assert(dynamicState.container != null);
        assert(dynamicState.currentAssignment != null);
        assert(dynamicState.container.areAllProcessesDead());
        
        dynamicState.container.cleanUp();
        staticState.localizer.releaseSlotFor(dynamicState.currentAssignment, 
staticState.port);
        DynamicState ret = dynamicState.withCurrentAssignment(null, null);
        if (nextState != null) {
            ret = ret.withState(nextState);
        }
        return ret;
    }

But we do not make a transaction to do this, if an exception occurred during 
deleting storm-code/ser/jar, an overdue local assignment will be left on disk.

Then when supervisor restart from the exception above, the slots will be 
initial and container will be recovered from LocalAssignments, the blob store 
will fetch the files from Nimbus/Master, but will get a KeyNotFoundException, 
and supervisor collapses again.

This will happens continuously and supervisor will never recover until we clean 
up all the local assignments manually.

This is the stack:
2017-12-27 14:15:04.434 o.a.s.l.AsyncLocalizer [INFO] Cleaning up unused 
topologies in /opt/meituan/storm/data/supervisor/stormdist
2017-12-27 14:15:04.434 o.a.s.d.s.AdvancedFSOps [INFO] Deleting path 
/opt/meituan/storm/data/supervisor/stormdist/app_dpsr_realtime_shop_vane_allcates-14-1513685785
2017-12-27 14:15:04.445 o.a.s.d.s.Slot [INFO] STATE EMPTY msInState: 109 -> 
WAITING_FOR_BASIC_LOCALIZATION msInState: 1
2017-12-27 14:15:04.471 o.a.s.d.s.Supervisor [INFO] Starting supervisor with id 
255d3fed-f3ee-4c7e-8a08-b693c9a6a072 at host gq-data-rt48.gq.sankuai.com.
2017-12-27 14:15:04.502 o.a.s.u.Utils [ERROR] An exception happened while 
downloading 
/opt/meituan/storm/data/supervisor/tmp/ca4f8174-59be-40a4-b431-dbc8b697f063/stormjar.jar
 from blob store.
org.apache.storm.generated.KeyNotFoundException: null
        at 
org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26656)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26624)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.generated.Nimbus$beginBlobDownload_result.read(Nimbus.java:26555)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:86) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.generated.Nimbus$Client.recv_beginBlobDownload(Nimbus.java:864)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.generated.Nimbus$Client.beginBlobDownload(Nimbus.java:851) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.blobstore.NimbusBlobStore.getBlob(NimbusBlobStore.java:357) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.utils.Utils.downloadResourcesAsSupervisorAttempt(Utils.java:598)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.utils.Utils.downloadResourcesAsSupervisorImpl(Utils.java:582) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.utils.Utils.downloadResourcesAsSupervisor(Utils.java:574) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.downloadBaseBlobs(AsyncLocalizer.java:123)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:148)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:101)
 ~[storm-core-1.1.2-mt001.jar:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
~[?:1.7.0_76]
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
[?:1.7.0_76]
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
[?:1.7.0_76]
        at java.lang.Thread.run(Thread.java:745) [?:1.7.0_76]
2017-12-27 14:15:04.611 o.a.s.u.Utils [ERROR] An exception happened while 
downloading 
/opt/meituan/storm/data/supervisor/tmp/ca4f8174-59be-40a4-b431-dbc8b697f063/stormjar.jar
 from blob store.
org.apache.storm.generated.KeyNotFoundException: null
        at 
org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26656)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26624)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.generated.Nimbus$beginBlobDownload_result.read(Nimbus.java:26555)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:86) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.generated.Nimbus$Client.recv_beginBlobDownload(Nimbus.java:864)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.generated.Nimbus$Client.beginBlobDownload(Nimbus.java:851) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.blobstore.NimbusBlobStore.getBlob(NimbusBlobStore.java:357) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.utils.Utils.downloadResourcesAsSupervisorAttempt(Utils.java:598)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.utils.Utils.downloadResourcesAsSupervisorImpl(Utils.java:582) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.utils.Utils.downloadResourcesAsSupervisor(Utils.java:574) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.downloadBaseBlobs(AsyncLocalizer.java:123)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:148)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:101)
 ~[storm-core-1.1.2-mt001.jar:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
~[?:1.7.0_76]
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
[?:1.7.0_76]
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
[?:1.7.0_76]
        at java.lang.Thread.run(Thread.java:745) [?:1.7.0_76]
2017-12-27 14:15:04.718 o.a.s.u.Utils [ERROR] An exception happened while 
downloading 
/opt/meituan/storm/data/supervisor/tmp/ca4f8174-59be-40a4-b431-dbc8b697f063/stormcode.ser
 from blob store.
org.apache.storm.generated.KeyNotFoundException: null
        at 
org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26656)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26624)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.generated.Nimbus$beginBlobDownload_result.read(Nimbus.java:26555)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:86) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.generated.Nimbus$Client.recv_beginBlobDownload(Nimbus.java:864)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.generated.Nimbus$Client.beginBlobDownload(Nimbus.java:851) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.blobstore.NimbusBlobStore.getBlob(NimbusBlobStore.java:357) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.utils.Utils.downloadResourcesAsSupervisorAttempt(Utils.java:598)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.utils.Utils.downloadResourcesAsSupervisorImpl(Utils.java:582) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.utils.Utils.downloadResourcesAsSupervisor(Utils.java:574) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.downloadBaseBlobs(AsyncLocalizer.java:124)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:148)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:101)
 ~[storm-core-1.1.2-mt001.jar:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
~[?:1.7.0_76]
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
[?:1.7.0_76]
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
[?:1.7.0_76]
        at java.lang.Thread.run(Thread.java:745) [?:1.7.0_76]
2017-12-27 14:15:04.825 o.a.s.u.Utils [ERROR] An exception happened while 
downloading 
/opt/meituan/storm/data/supervisor/tmp/ca4f8174-59be-40a4-b431-dbc8b697f063/stormcode.ser
 from blob store.
org.apache.storm.generated.KeyNotFoundException: null
        at 
org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26656)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26624)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.generated.Nimbus$beginBlobDownload_result.read(Nimbus.java:26555)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:86) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.generated.Nimbus$Client.recv_beginBlobDownload(Nimbus.java:864)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.generated.Nimbus$Client.beginBlobDownload(Nimbus.java:851) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.blobstore.NimbusBlobStore.getBlob(NimbusBlobStore.java:357) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.utils.Utils.downloadResourcesAsSupervisorAttempt(Utils.java:598)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.utils.Utils.downloadResourcesAsSupervisorImpl(Utils.java:582) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.utils.Utils.downloadResourcesAsSupervisor(Utils.java:574) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.downloadBaseBlobs(AsyncLocalizer.java:124)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:148)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:101)
 ~[storm-core-1.1.2-mt001.jar:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
~[?:1.7.0_76]
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
[?:1.7.0_76]
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
[?:1.7.0_76]
        at java.lang.Thread.run(Thread.java:745) [?:1.7.0_76]
2017-12-27 14:15:04.932 o.a.s.u.Utils [ERROR] An exception happened while 
downloading 
/opt/meituan/storm/data/supervisor/tmp/ca4f8174-59be-40a4-b431-dbc8b697f063/stormconf.ser
 from blob store.
org.apache.storm.generated.KeyNotFoundException: null
        at 
org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26656)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26624)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.generated.Nimbus$beginBlobDownload_result.read(Nimbus.java:26555)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:86) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.generated.Nimbus$Client.recv_beginBlobDownload(Nimbus.java:864)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.generated.Nimbus$Client.beginBlobDownload(Nimbus.java:851) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.blobstore.NimbusBlobStore.getBlob(NimbusBlobStore.java:357) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.utils.Utils.downloadResourcesAsSupervisorAttempt(Utils.java:598)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.utils.Utils.downloadResourcesAsSupervisorImpl(Utils.java:582) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.utils.Utils.downloadResourcesAsSupervisor(Utils.java:574) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.downloadBaseBlobs(AsyncLocalizer.java:125)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:148)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:101)
 ~[storm-core-1.1.2-mt001.jar:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
~[?:1.7.0_76]
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
[?:1.7.0_76]
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
[?:1.7.0_76]
        at java.lang.Thread.run(Thread.java:745) [?:1.7.0_76]
2017-12-27 14:15:05.039 o.a.s.u.Utils [ERROR] An exception happened while 
downloading 
/opt/meituan/storm/data/supervisor/tmp/ca4f8174-59be-40a4-b431-dbc8b697f063/stormconf.ser
 from blob store.
org.apache.storm.generated.KeyNotFoundException: null
        at 
org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26656)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26624)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.generated.Nimbus$beginBlobDownload_result.read(Nimbus.java:26555)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:86) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.generated.Nimbus$Client.recv_beginBlobDownload(Nimbus.java:864)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.generated.Nimbus$Client.beginBlobDownload(Nimbus.java:851) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.blobstore.NimbusBlobStore.getBlob(NimbusBlobStore.java:357) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.utils.Utils.downloadResourcesAsSupervisorAttempt(Utils.java:598)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.utils.Utils.downloadResourcesAsSupervisorImpl(Utils.java:582) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.utils.Utils.downloadResourcesAsSupervisor(Utils.java:574) 
~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.downloadBaseBlobs(AsyncLocalizer.java:125)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:148)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:101)
 ~[storm-core-1.1.2-mt001.jar:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
~[?:1.7.0_76]
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
[?:1.7.0_76]
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
[?:1.7.0_76]
        at java.lang.Thread.run(Thread.java:745) [?:1.7.0_76]
2017-12-27 14:15:05.140 o.a.s.u.Utils [INFO] Could not extract resources from 
/opt/meituan/storm/data/supervisor/tmp/ca4f8174-59be-40a4-b431-dbc8b697f063/stormjar.jar
2017-12-27 14:15:05.142 o.a.s.d.s.Slot [INFO] STATE 
WAITING_FOR_BASIC_LOCALIZATION msInState: 697 -> WAITING_FOR_BLOB_LOCALIZATION 
msInState: 0
2017-12-27 14:15:05.142 o.a.s.l.AsyncLocalizer [WARN] Caught Exception While 
Downloading (rethrowing)... 
java.io.FileNotFoundException: File 
'/opt/meituan/storm/data/supervisor/stormdist/app_dpsr_realtime_shop_vane_allcates-14-1513685785/stormconf.ser'
 does not exist
        at 
org.apache.storm.shade.org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:292)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.shade.org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1815)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.utils.ConfigUtils.readSupervisorStormConfGivenPath(ConfigUtils.java:264)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.utils.ConfigUtils.readSupervisorStormConfImpl(ConfigUtils.java:376)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.utils.ConfigUtils.readSupervisorStormConf(ConfigUtils.java:370)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.call(AsyncLocalizer.java:226)
 ~[storm-core-1.1.2-mt001.jar:?]
        at 
org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.call(AsyncLocalizer.java:213)
 ~[storm-core-1.1.2-mt001.jar:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
~[?:1.7.0_76]
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
[?:1.7.0_76]
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
[?:1.7.0_76]
        at java.lang.Thread.run(Thread.java:745) [?:1.7.0_76]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to