Rui Li created STORM-3701:
-----------------------------
Summary: Race Condition between cleanup thread and download tasks
Key: STORM-3701
URL: https://issues.apache.org/jira/browse/STORM-3701
Project: Apache Storm
Issue Type: Bug
Components: storm-server
Reporter: Rui Li
Assignee: Rui Li
We captured a issue on our supervisor:
2020-06-09 23:30:08.723 o.a.s.l.LocalizedResource AsyncLocalizer Task Executor
- 0 [INFO] completelyRemoveUnusedUser directu for directory
/home/y/var/storm/supervisor/usercache/directu
2020-06-09 23:30:08.724 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 0
[WARN] Caught Exception While Downloading (rethrowing)...
java.io.FileNotFoundException: File
'/home/y/var/storm/supervisor/stormdist/dg_itp-605-1591745383/stormconf.ser'
does not exist
at
org.apache.storm.shade.org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:297)
~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.shade.org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1851)
~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.utils.ConfigUtils.readSupervisorStormConfGivenPath(ConfigUtils.java:316)
~[storm-client-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.utils.ConfigUtils.readSupervisorStormConfImpl(ConfigUtils.java:477)
~[storm-client-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.utils.ConfigUtils.readSupervisorStormConf(ConfigUtils.java:311)
~[storm-client-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.get(AsyncLocalizer.java:698)
[storm-server-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.get(AsyncLocalizer.java:683)
[storm-server-2.3.0.y.jar:2.3.0.y]
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
[?:1.8.0_242]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_242]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_242]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_242]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[?:1.8.0_242]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_242]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_242]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
2020-06-09 23:30:08.725 o.a.s.d.s.Slot SLOT_6782 [ERROR]
java.io.FileNotFoundException: File
'/home/y/var/storm/supervisor/stormdist/dg_itp-605-1591745383/stormconf.ser'
does not exist
2020-06-09 23:30:08.725 o.a.s.l.AsyncLocalizer SLOT_6782 [INFO] Port and
assignment info: PortAndAssignmentImpl\{dg_itp-605-1591745383 on 6782}
2020-06-09 23:30:08.726 o.a.s.l.AsyncLocalizer SLOT_6782 [WARN] Local base
blobs have not been downloaded yet.
java.io.FileNotFoundException: File
'/home/y/var/storm/supervisor/stormdist/dg_itp-605-1591745383/stormconf.ser'
does not exist
at
org.apache.storm.shade.org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:297)
~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.shade.org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1851)
~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.utils.ConfigUtils.readSupervisorStormConfGivenPath(ConfigUtils.java:316)
~[storm-client-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.utils.ConfigUtils.readSupervisorStormConfImpl(ConfigUtils.java:477)
~[storm-client-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.utils.ConfigUtils.readSupervisorStormConf(ConfigUtils.java:311)
~[storm-client-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.localizer.AsyncLocalizer.getLocalResources(AsyncLocalizer.java:362)
~[storm-server-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.localizer.AsyncLocalizer.releaseSlotFor(AsyncLocalizer.java:472)
[storm-server-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.daemon.supervisor.Slot.handleWaitingForBlobLocalization(Slot.java:549)
[storm-server-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.daemon.supervisor.Slot.stateMachineStep(Slot.java:298)
[storm-server-2.3.0.y.jar:2.3.0.y]
at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:1039)
[storm-server-2.3.0.y.jar:2.3.0.y]
The root issue is the delay at
[https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/localizer/AsyncLocalizer.java#L641]
which will cause the safeTopologyIds information out-of-date.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
