[
https://issues.apache.org/jira/browse/STORM-3701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated STORM-3701:
--------------------------
Description:
We captured a issue on our supervisor:
{code}
2020-06-09 23:30:08.723 o.a.s.l.LocalizedResource AsyncLocalizer Task Executor
- 0 [INFO] completelyRemoveUnusedUser directu for directory
/home/y/var/storm/supervisor/usercache/directu
2020-06-09 23:30:08.724 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor -
0 [WARN] Caught Exception While Downloading (rethrowing)...
java.io.FileNotFoundException: File
'/home/y/var/storm/supervisor/stormdist/dg_itp-605-1591745383/stormconf.ser'
does not exist
at
org.apache.storm.shade.org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:297)
~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.shade.org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1851)
~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.utils.ConfigUtils.readSupervisorStormConfGivenPath(ConfigUtils.java:316)
~[storm-client-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.utils.ConfigUtils.readSupervisorStormConfImpl(ConfigUtils.java:477)
~[storm-client-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.utils.ConfigUtils.readSupervisorStormConf(ConfigUtils.java:311)
~[storm-client-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.get(AsyncLocalizer.java:698)
[storm-server-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.get(AsyncLocalizer.java:683)
[storm-server-2.3.0.y.jar:2.3.0.y]
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
[?:1.8.0_242]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_242]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_242]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_242]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[?:1.8.0_242]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_242]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_242]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
2020-06-09 23:30:08.725 o.a.s.d.s.Slot SLOT_6782 [ERROR]
java.io.FileNotFoundException: File
'/home/y/var/storm/supervisor/stormdist/dg_itp-605-1591745383/stormconf.ser'
does not exist
2020-06-09 23:30:08.725 o.a.s.l.AsyncLocalizer SLOT_6782 [INFO] Port and
assignment info: PortAndAssignmentImpl\{dg_itp-605-1591745383 on 6782}
2020-06-09 23:30:08.726 o.a.s.l.AsyncLocalizer SLOT_6782 [WARN] Local base
blobs have not been downloaded yet.
java.io.FileNotFoundException: File
'/home/y/var/storm/supervisor/stormdist/dg_itp-605-1591745383/stormconf.ser'
does not exist
at
org.apache.storm.shade.org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:297)
~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.shade.org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1851)
~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.utils.ConfigUtils.readSupervisorStormConfGivenPath(ConfigUtils.java:316)
~[storm-client-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.utils.ConfigUtils.readSupervisorStormConfImpl(ConfigUtils.java:477)
~[storm-client-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.utils.ConfigUtils.readSupervisorStormConf(ConfigUtils.java:311)
~[storm-client-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.localizer.AsyncLocalizer.getLocalResources(AsyncLocalizer.java:362)
~[storm-server-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.localizer.AsyncLocalizer.releaseSlotFor(AsyncLocalizer.java:472)
[storm-server-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.daemon.supervisor.Slot.handleWaitingForBlobLocalization(Slot.java:549)
[storm-server-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.daemon.supervisor.Slot.stateMachineStep(Slot.java:298)
[storm-server-2.3.0.y.jar:2.3.0.y]
at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:1039)
[storm-server-2.3.0.y.jar:2.3.0.y]
{code}
The root issue is the delay at
[https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/localizer/AsyncLocalizer.java#L641]
which will cause the safeTopologyIds information out-of-date.
was:
We captured a issue on our supervisor:
2020-06-09 23:30:08.723 o.a.s.l.LocalizedResource AsyncLocalizer Task Executor
- 0 [INFO] completelyRemoveUnusedUser directu for directory
/home/y/var/storm/supervisor/usercache/directu
2020-06-09 23:30:08.724 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 0
[WARN] Caught Exception While Downloading (rethrowing)...
java.io.FileNotFoundException: File
'/home/y/var/storm/supervisor/stormdist/dg_itp-605-1591745383/stormconf.ser'
does not exist
at
org.apache.storm.shade.org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:297)
~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.shade.org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1851)
~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.utils.ConfigUtils.readSupervisorStormConfGivenPath(ConfigUtils.java:316)
~[storm-client-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.utils.ConfigUtils.readSupervisorStormConfImpl(ConfigUtils.java:477)
~[storm-client-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.utils.ConfigUtils.readSupervisorStormConf(ConfigUtils.java:311)
~[storm-client-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.get(AsyncLocalizer.java:698)
[storm-server-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.get(AsyncLocalizer.java:683)
[storm-server-2.3.0.y.jar:2.3.0.y]
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
[?:1.8.0_242]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_242]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_242]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_242]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[?:1.8.0_242]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_242]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_242]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
2020-06-09 23:30:08.725 o.a.s.d.s.Slot SLOT_6782 [ERROR]
java.io.FileNotFoundException: File
'/home/y/var/storm/supervisor/stormdist/dg_itp-605-1591745383/stormconf.ser'
does not exist
2020-06-09 23:30:08.725 o.a.s.l.AsyncLocalizer SLOT_6782 [INFO] Port and
assignment info: PortAndAssignmentImpl\{dg_itp-605-1591745383 on 6782}
2020-06-09 23:30:08.726 o.a.s.l.AsyncLocalizer SLOT_6782 [WARN] Local base
blobs have not been downloaded yet.
java.io.FileNotFoundException: File
'/home/y/var/storm/supervisor/stormdist/dg_itp-605-1591745383/stormconf.ser'
does not exist
at
org.apache.storm.shade.org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:297)
~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.shade.org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1851)
~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.utils.ConfigUtils.readSupervisorStormConfGivenPath(ConfigUtils.java:316)
~[storm-client-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.utils.ConfigUtils.readSupervisorStormConfImpl(ConfigUtils.java:477)
~[storm-client-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.utils.ConfigUtils.readSupervisorStormConf(ConfigUtils.java:311)
~[storm-client-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.localizer.AsyncLocalizer.getLocalResources(AsyncLocalizer.java:362)
~[storm-server-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.localizer.AsyncLocalizer.releaseSlotFor(AsyncLocalizer.java:472)
[storm-server-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.daemon.supervisor.Slot.handleWaitingForBlobLocalization(Slot.java:549)
[storm-server-2.3.0.y.jar:2.3.0.y]
at
org.apache.storm.daemon.supervisor.Slot.stateMachineStep(Slot.java:298)
[storm-server-2.3.0.y.jar:2.3.0.y]
at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:1039)
[storm-server-2.3.0.y.jar:2.3.0.y]
The root issue is the delay at
[https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/localizer/AsyncLocalizer.java#L641]
which will cause the safeTopologyIds information out-of-date.
> Race Condition between cleanup thread and download tasks
> --------------------------------------------------------
>
> Key: STORM-3701
> URL: https://issues.apache.org/jira/browse/STORM-3701
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-server
> Reporter: Rui Li
> Assignee: Rui Li
> Priority: Minor
>
> We captured a issue on our supervisor:
> {code}
> 2020-06-09 23:30:08.723 o.a.s.l.LocalizedResource AsyncLocalizer Task
> Executor - 0 [INFO] completelyRemoveUnusedUser directu for directory
> /home/y/var/storm/supervisor/usercache/directu
> 2020-06-09 23:30:08.724 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor
> - 0 [WARN] Caught Exception While Downloading (rethrowing)...
> java.io.FileNotFoundException: File
> '/home/y/var/storm/supervisor/stormdist/dg_itp-605-1591745383/stormconf.ser'
> does not exist
> at
> org.apache.storm.shade.org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:297)
> ~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
> at
> org.apache.storm.shade.org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1851)
> ~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
> at
> org.apache.storm.utils.ConfigUtils.readSupervisorStormConfGivenPath(ConfigUtils.java:316)
> ~[storm-client-2.3.0.y.jar:2.3.0.y]
> at
> org.apache.storm.utils.ConfigUtils.readSupervisorStormConfImpl(ConfigUtils.java:477)
> ~[storm-client-2.3.0.y.jar:2.3.0.y]
> at
> org.apache.storm.utils.ConfigUtils.readSupervisorStormConf(ConfigUtils.java:311)
> ~[storm-client-2.3.0.y.jar:2.3.0.y]
> at
> org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.get(AsyncLocalizer.java:698)
> [storm-server-2.3.0.y.jar:2.3.0.y]
> at
> org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.get(AsyncLocalizer.java:683)
> [storm-server-2.3.0.y.jar:2.3.0.y]
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> [?:1.8.0_242]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_242]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [?:1.8.0_242]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [?:1.8.0_242]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [?:1.8.0_242]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_242]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_242]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
> 2020-06-09 23:30:08.725 o.a.s.d.s.Slot SLOT_6782 [ERROR]
> java.io.FileNotFoundException: File
> '/home/y/var/storm/supervisor/stormdist/dg_itp-605-1591745383/stormconf.ser'
> does not exist
> 2020-06-09 23:30:08.725 o.a.s.l.AsyncLocalizer SLOT_6782 [INFO] Port and
> assignment info: PortAndAssignmentImpl\{dg_itp-605-1591745383 on 6782}
> 2020-06-09 23:30:08.726 o.a.s.l.AsyncLocalizer SLOT_6782 [WARN] Local base
> blobs have not been downloaded yet.
> java.io.FileNotFoundException: File
> '/home/y/var/storm/supervisor/stormdist/dg_itp-605-1591745383/stormconf.ser'
> does not exist
> at
> org.apache.storm.shade.org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:297)
> ~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
> at
> org.apache.storm.shade.org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1851)
> ~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
> at
> org.apache.storm.utils.ConfigUtils.readSupervisorStormConfGivenPath(ConfigUtils.java:316)
> ~[storm-client-2.3.0.y.jar:2.3.0.y]
> at
> org.apache.storm.utils.ConfigUtils.readSupervisorStormConfImpl(ConfigUtils.java:477)
> ~[storm-client-2.3.0.y.jar:2.3.0.y]
> at
> org.apache.storm.utils.ConfigUtils.readSupervisorStormConf(ConfigUtils.java:311)
> ~[storm-client-2.3.0.y.jar:2.3.0.y]
> at
> org.apache.storm.localizer.AsyncLocalizer.getLocalResources(AsyncLocalizer.java:362)
> ~[storm-server-2.3.0.y.jar:2.3.0.y]
> at
> org.apache.storm.localizer.AsyncLocalizer.releaseSlotFor(AsyncLocalizer.java:472)
> [storm-server-2.3.0.y.jar:2.3.0.y]
> at
> org.apache.storm.daemon.supervisor.Slot.handleWaitingForBlobLocalization(Slot.java:549)
> [storm-server-2.3.0.y.jar:2.3.0.y]
> at
> org.apache.storm.daemon.supervisor.Slot.stateMachineStep(Slot.java:298)
> [storm-server-2.3.0.y.jar:2.3.0.y]
> at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:1039)
> [storm-server-2.3.0.y.jar:2.3.0.y]
> {code}
> The root issue is the delay at
> [https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/localizer/AsyncLocalizer.java#L641]
> which will cause the safeTopologyIds information out-of-date.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
