[ 
https://issues.apache.org/jira/browse/STORM-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim updated STORM-2878:
--------------------------------
    Fix Version/s:     (was: 2.0.0)

> Supervisor collapse continuously when there is a expired assignment for 
> overdue storm
> -------------------------------------------------------------------------------------
>
>                 Key: STORM-2878
>                 URL: https://issues.apache.org/jira/browse/STORM-2878
>             Project: Apache Storm
>          Issue Type: Bug
>          Components: storm-core, storm-server
>    Affects Versions: 2.0.0, 1.x
>            Reporter: Yuzhao Chen
>            Priority: Critical
>              Labels: patch
>
> For now, when a topology is reassigned or killed for a cluster, supervisor 
> will delete 4 files for an overdue storm:
> - storm-code
> - storm-ser
> - storm-jar
> - LocalAssignment
> Slot.java
> static DynamicState cleanupCurrentContainer(DynamicState dynamicState, 
> StaticState staticState, MachineState nextState) throws Exception {
>         assert(dynamicState.container != null);
>         assert(dynamicState.currentAssignment != null);
>         assert(dynamicState.container.areAllProcessesDead());
>         
>         dynamicState.container.cleanUp();
>         staticState.localizer.releaseSlotFor(dynamicState.currentAssignment, 
> staticState.port);
>         DynamicState ret = dynamicState.withCurrentAssignment(null, null);
>         if (nextState != null) {
>             ret = ret.withState(nextState);
>         }
>         return ret;
>     }
> But we do not make a transaction to do this, if an exception occurred during 
> deleting storm-code/ser/jar, an overdue local assignment will be left on disk.
> Then when supervisor restart from the exception above, the slots will be 
> initial and container will be recovered from LocalAssignments, the blob store 
> will fetch the files from Nimbus/Master, but will get a KeyNotFoundException, 
> and supervisor collapses again.
> This will happens continuously and supervisor will never recover until we 
> clean up all the local assignments manually.
> This is the stack:
> 2017-12-27 14:15:04.434 o.a.s.l.AsyncLocalizer [INFO] Cleaning up unused 
> topologies in /opt/meituan/storm/data/supervisor/stormdist
> 2017-12-27 14:15:04.434 o.a.s.d.s.AdvancedFSOps [INFO] Deleting path 
> /opt/meituan/storm/data/supervisor/stormdist/app_dpsr_realtime_shop_vane_allcates-14-1513685785
> 2017-12-27 14:15:04.445 o.a.s.d.s.Slot [INFO] STATE EMPTY msInState: 109 -> 
> WAITING_FOR_BASIC_LOCALIZATION msInState: 1
> 2017-12-27 14:15:04.471 o.a.s.d.s.Supervisor [INFO] Starting supervisor with 
> id 255d3fed-f3ee-4c7e-8a08-b693c9a6a072 at host gq-data-rt48.gq.sankuai.com.
> 2017-12-27 14:15:04.502 o.a.s.u.Utils [ERROR] An exception happened while 
> downloading 
> /opt/meituan/storm/data/supervisor/tmp/ca4f8174-59be-40a4-b431-dbc8b697f063/stormjar.jar
>  from blob store.
> org.apache.storm.generated.KeyNotFoundException: null
>       at 
> org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26656)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26624)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.generated.Nimbus$beginBlobDownload_result.read(Nimbus.java:26555)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:86) 
> ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.generated.Nimbus$Client.recv_beginBlobDownload(Nimbus.java:864)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.generated.Nimbus$Client.beginBlobDownload(Nimbus.java:851) 
> ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.blobstore.NimbusBlobStore.getBlob(NimbusBlobStore.java:357) 
> ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.utils.Utils.downloadResourcesAsSupervisorAttempt(Utils.java:598)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.utils.Utils.downloadResourcesAsSupervisorImpl(Utils.java:582)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.utils.Utils.downloadResourcesAsSupervisor(Utils.java:574) 
> ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.downloadBaseBlobs(AsyncLocalizer.java:123)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:148)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:101)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
> ~[?:1.7.0_76]
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [?:1.7.0_76]
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [?:1.7.0_76]
>       at java.lang.Thread.run(Thread.java:745) [?:1.7.0_76]
> 2017-12-27 14:15:04.611 o.a.s.u.Utils [ERROR] An exception happened while 
> downloading 
> /opt/meituan/storm/data/supervisor/tmp/ca4f8174-59be-40a4-b431-dbc8b697f063/stormjar.jar
>  from blob store.
> org.apache.storm.generated.KeyNotFoundException: null
>       at 
> org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26656)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26624)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.generated.Nimbus$beginBlobDownload_result.read(Nimbus.java:26555)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:86) 
> ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.generated.Nimbus$Client.recv_beginBlobDownload(Nimbus.java:864)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.generated.Nimbus$Client.beginBlobDownload(Nimbus.java:851) 
> ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.blobstore.NimbusBlobStore.getBlob(NimbusBlobStore.java:357) 
> ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.utils.Utils.downloadResourcesAsSupervisorAttempt(Utils.java:598)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.utils.Utils.downloadResourcesAsSupervisorImpl(Utils.java:582)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.utils.Utils.downloadResourcesAsSupervisor(Utils.java:574) 
> ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.downloadBaseBlobs(AsyncLocalizer.java:123)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:148)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:101)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
> ~[?:1.7.0_76]
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [?:1.7.0_76]
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [?:1.7.0_76]
>       at java.lang.Thread.run(Thread.java:745) [?:1.7.0_76]
> 2017-12-27 14:15:04.718 o.a.s.u.Utils [ERROR] An exception happened while 
> downloading 
> /opt/meituan/storm/data/supervisor/tmp/ca4f8174-59be-40a4-b431-dbc8b697f063/stormcode.ser
>  from blob store.
> org.apache.storm.generated.KeyNotFoundException: null
>       at 
> org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26656)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26624)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.generated.Nimbus$beginBlobDownload_result.read(Nimbus.java:26555)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:86) 
> ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.generated.Nimbus$Client.recv_beginBlobDownload(Nimbus.java:864)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.generated.Nimbus$Client.beginBlobDownload(Nimbus.java:851) 
> ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.blobstore.NimbusBlobStore.getBlob(NimbusBlobStore.java:357) 
> ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.utils.Utils.downloadResourcesAsSupervisorAttempt(Utils.java:598)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.utils.Utils.downloadResourcesAsSupervisorImpl(Utils.java:582)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.utils.Utils.downloadResourcesAsSupervisor(Utils.java:574) 
> ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.downloadBaseBlobs(AsyncLocalizer.java:124)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:148)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:101)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
> ~[?:1.7.0_76]
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [?:1.7.0_76]
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [?:1.7.0_76]
>       at java.lang.Thread.run(Thread.java:745) [?:1.7.0_76]
> 2017-12-27 14:15:04.825 o.a.s.u.Utils [ERROR] An exception happened while 
> downloading 
> /opt/meituan/storm/data/supervisor/tmp/ca4f8174-59be-40a4-b431-dbc8b697f063/stormcode.ser
>  from blob store.
> org.apache.storm.generated.KeyNotFoundException: null
>       at 
> org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26656)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26624)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.generated.Nimbus$beginBlobDownload_result.read(Nimbus.java:26555)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:86) 
> ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.generated.Nimbus$Client.recv_beginBlobDownload(Nimbus.java:864)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.generated.Nimbus$Client.beginBlobDownload(Nimbus.java:851) 
> ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.blobstore.NimbusBlobStore.getBlob(NimbusBlobStore.java:357) 
> ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.utils.Utils.downloadResourcesAsSupervisorAttempt(Utils.java:598)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.utils.Utils.downloadResourcesAsSupervisorImpl(Utils.java:582)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.utils.Utils.downloadResourcesAsSupervisor(Utils.java:574) 
> ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.downloadBaseBlobs(AsyncLocalizer.java:124)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:148)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:101)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
> ~[?:1.7.0_76]
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [?:1.7.0_76]
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [?:1.7.0_76]
>       at java.lang.Thread.run(Thread.java:745) [?:1.7.0_76]
> 2017-12-27 14:15:04.932 o.a.s.u.Utils [ERROR] An exception happened while 
> downloading 
> /opt/meituan/storm/data/supervisor/tmp/ca4f8174-59be-40a4-b431-dbc8b697f063/stormconf.ser
>  from blob store.
> org.apache.storm.generated.KeyNotFoundException: null
>       at 
> org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26656)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26624)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.generated.Nimbus$beginBlobDownload_result.read(Nimbus.java:26555)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:86) 
> ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.generated.Nimbus$Client.recv_beginBlobDownload(Nimbus.java:864)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.generated.Nimbus$Client.beginBlobDownload(Nimbus.java:851) 
> ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.blobstore.NimbusBlobStore.getBlob(NimbusBlobStore.java:357) 
> ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.utils.Utils.downloadResourcesAsSupervisorAttempt(Utils.java:598)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.utils.Utils.downloadResourcesAsSupervisorImpl(Utils.java:582)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.utils.Utils.downloadResourcesAsSupervisor(Utils.java:574) 
> ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.downloadBaseBlobs(AsyncLocalizer.java:125)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:148)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:101)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
> ~[?:1.7.0_76]
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [?:1.7.0_76]
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [?:1.7.0_76]
>       at java.lang.Thread.run(Thread.java:745) [?:1.7.0_76]
> 2017-12-27 14:15:05.039 o.a.s.u.Utils [ERROR] An exception happened while 
> downloading 
> /opt/meituan/storm/data/supervisor/tmp/ca4f8174-59be-40a4-b431-dbc8b697f063/stormconf.ser
>  from blob store.
> org.apache.storm.generated.KeyNotFoundException: null
>       at 
> org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26656)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26624)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.generated.Nimbus$beginBlobDownload_result.read(Nimbus.java:26555)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:86) 
> ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.generated.Nimbus$Client.recv_beginBlobDownload(Nimbus.java:864)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.generated.Nimbus$Client.beginBlobDownload(Nimbus.java:851) 
> ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.blobstore.NimbusBlobStore.getBlob(NimbusBlobStore.java:357) 
> ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.utils.Utils.downloadResourcesAsSupervisorAttempt(Utils.java:598)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.utils.Utils.downloadResourcesAsSupervisorImpl(Utils.java:582)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.utils.Utils.downloadResourcesAsSupervisor(Utils.java:574) 
> ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.downloadBaseBlobs(AsyncLocalizer.java:125)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:148)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:101)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
> ~[?:1.7.0_76]
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [?:1.7.0_76]
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [?:1.7.0_76]
>       at java.lang.Thread.run(Thread.java:745) [?:1.7.0_76]
> 2017-12-27 14:15:05.140 o.a.s.u.Utils [INFO] Could not extract resources from 
> /opt/meituan/storm/data/supervisor/tmp/ca4f8174-59be-40a4-b431-dbc8b697f063/stormjar.jar
> 2017-12-27 14:15:05.142 o.a.s.d.s.Slot [INFO] STATE 
> WAITING_FOR_BASIC_LOCALIZATION msInState: 697 -> 
> WAITING_FOR_BLOB_LOCALIZATION msInState: 0
> 2017-12-27 14:15:05.142 o.a.s.l.AsyncLocalizer [WARN] Caught Exception While 
> Downloading (rethrowing)... 
> java.io.FileNotFoundException: File 
> '/opt/meituan/storm/data/supervisor/stormdist/app_dpsr_realtime_shop_vane_allcates-14-1513685785/stormconf.ser'
>  does not exist
>       at 
> org.apache.storm.shade.org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:292)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.shade.org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1815)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.utils.ConfigUtils.readSupervisorStormConfGivenPath(ConfigUtils.java:264)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.utils.ConfigUtils.readSupervisorStormConfImpl(ConfigUtils.java:376)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.utils.ConfigUtils.readSupervisorStormConf(ConfigUtils.java:370)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.call(AsyncLocalizer.java:226)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at 
> org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.call(AsyncLocalizer.java:213)
>  ~[storm-core-1.1.2-mt001.jar:?]
>       at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
> ~[?:1.7.0_76]
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [?:1.7.0_76]
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [?:1.7.0_76]
>       at java.lang.Thread.run(Thread.java:745) [?:1.7.0_76]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to