[ https://issues.apache.org/jira/browse/STORM-3701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ethan Li resolved STORM-3701. ----------------------------- Fix Version/s: 2.3.0 Resolution: Fixed Thanks [~li530]. I merged this to master (19e23ae4a44fcf8e74473a48d408a712cde45d64) > Race Condition between cleanup thread and download tasks > -------------------------------------------------------- > > Key: STORM-3701 > URL: https://issues.apache.org/jira/browse/STORM-3701 > Project: Apache Storm > Issue Type: Bug > Components: storm-server > Reporter: Rui Li > Assignee: Rui Li > Priority: Major > Fix For: 2.3.0 > > Time Spent: 2h > Remaining Estimate: 0h > > We captured a issue on our supervisor: > {code} > 2020-06-09 23:30:08.723 o.a.s.l.LocalizedResource AsyncLocalizer Task > Executor - 0 [INFO] completelyRemoveUnusedUser directu for directory > /home/y/var/storm/supervisor/usercache/directu > 2020-06-09 23:30:08.724 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor > - 0 [WARN] Caught Exception While Downloading (rethrowing)... > java.io.FileNotFoundException: File > '/home/y/var/storm/supervisor/stormdist/dg_itp-605-1591745383/stormconf.ser' > does not exist > at > org.apache.storm.shade.org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:297) > ~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y] > at > org.apache.storm.shade.org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1851) > ~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y] > at > org.apache.storm.utils.ConfigUtils.readSupervisorStormConfGivenPath(ConfigUtils.java:316) > ~[storm-client-2.3.0.y.jar:2.3.0.y] > at > org.apache.storm.utils.ConfigUtils.readSupervisorStormConfImpl(ConfigUtils.java:477) > ~[storm-client-2.3.0.y.jar:2.3.0.y] > at > org.apache.storm.utils.ConfigUtils.readSupervisorStormConf(ConfigUtils.java:311) > ~[storm-client-2.3.0.y.jar:2.3.0.y] > at > org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.get(AsyncLocalizer.java:698) > [storm-server-2.3.0.y.jar:2.3.0.y] > at > org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.get(AsyncLocalizer.java:683) > [storm-server-2.3.0.y.jar:2.3.0.y] > at > java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) > [?:1.8.0_242] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [?:1.8.0_242] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [?:1.8.0_242] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > [?:1.8.0_242] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [?:1.8.0_242] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [?:1.8.0_242] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [?:1.8.0_242] > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242] > 2020-06-09 23:30:08.725 o.a.s.d.s.Slot SLOT_6782 [ERROR] > java.io.FileNotFoundException: File > '/home/y/var/storm/supervisor/stormdist/dg_itp-605-1591745383/stormconf.ser' > does not exist > 2020-06-09 23:30:08.725 o.a.s.l.AsyncLocalizer SLOT_6782 [INFO] Port and > assignment info: PortAndAssignmentImpl\{dg_itp-605-1591745383 on 6782} > 2020-06-09 23:30:08.726 o.a.s.l.AsyncLocalizer SLOT_6782 [WARN] Local base > blobs have not been downloaded yet. > java.io.FileNotFoundException: File > '/home/y/var/storm/supervisor/stormdist/dg_itp-605-1591745383/stormconf.ser' > does not exist > at > org.apache.storm.shade.org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:297) > ~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y] > at > org.apache.storm.shade.org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1851) > ~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y] > at > org.apache.storm.utils.ConfigUtils.readSupervisorStormConfGivenPath(ConfigUtils.java:316) > ~[storm-client-2.3.0.y.jar:2.3.0.y] > at > org.apache.storm.utils.ConfigUtils.readSupervisorStormConfImpl(ConfigUtils.java:477) > ~[storm-client-2.3.0.y.jar:2.3.0.y] > at > org.apache.storm.utils.ConfigUtils.readSupervisorStormConf(ConfigUtils.java:311) > ~[storm-client-2.3.0.y.jar:2.3.0.y] > at > org.apache.storm.localizer.AsyncLocalizer.getLocalResources(AsyncLocalizer.java:362) > ~[storm-server-2.3.0.y.jar:2.3.0.y] > at > org.apache.storm.localizer.AsyncLocalizer.releaseSlotFor(AsyncLocalizer.java:472) > [storm-server-2.3.0.y.jar:2.3.0.y] > at > org.apache.storm.daemon.supervisor.Slot.handleWaitingForBlobLocalization(Slot.java:549) > [storm-server-2.3.0.y.jar:2.3.0.y] > at > org.apache.storm.daemon.supervisor.Slot.stateMachineStep(Slot.java:298) > [storm-server-2.3.0.y.jar:2.3.0.y] > at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:1039) > [storm-server-2.3.0.y.jar:2.3.0.y] > {code} > The root issue is the delay at > [https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/localizer/AsyncLocalizer.java#L641] > which will cause the safeTopologyIds information out-of-date. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)