[
https://issues.apache.org/jira/browse/STORM-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16040866#comment-16040866
]
Kristopher Kane commented on STORM-2138:
----------------------------------------
I can confirm this on 1.0.2 but instead of worker failure, I see the Supervisor
shut down after a time. Inspecting the supervisor after it is noticed as down,
I see that the stormconf.ser file is present, with the correct ownership, in
the reported missing file location.
> java.io.FileNotFoundException: stormconf.ser does not exist
> -----------------------------------------------------------
>
> Key: STORM-2138
> URL: https://issues.apache.org/jira/browse/STORM-2138
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
> Affects Versions: 1.0.2
> Reporter: Eddy
>
> We are seeing problems in our storm topology whereby all our workers crash.
> The errors we see are
> 2016-10-07 09:49:33.599 o.a.s.d.supervisor [ERROR] Error on initialization of
> server mk-supervisor
> java.io.FileNotFoundException: File
> '/opt/storm_local/supervisor/stormdist/production_2016_09_13-1-1475831938/stormconf.ser'
> does not exist
> at
> org.apache.storm.shade.org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:292)
> at
> org.apache.storm.shade.org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1815)
> at
> org.apache.storm.config$read_supervisor_storm_conf_given_path.invoke(config.clj:142)
> at
> org.apache.storm.config$read_supervisor_storm_conf.invoke(config.clj:221)
> at
> org.apache.storm.daemon.supervisor$add_blob_references.invoke(supervisor.clj:495)
> at
> org.apache.storm.daemon.supervisor$fn__9307$exec_fn__2466__auto____9308.invoke(supervisor.clj:795)
> at clojure.lang.AFn.applyToHelper(AFn.java:160)
> at clojure.lang.AFn.applyTo(AFn.java:144)
> at clojure.core$apply.invoke(core.clj:630)
> at
> org.apache.storm.daemon.supervisor$fn__9307$mk_supervisor__9352.doInvoke(supervisor.clj:763)
> at clojure.lang.RestFn.invoke(RestFn.java:436)
> at
> org.apache.storm.daemon.supervisor$_launch.invoke(supervisor.clj:1200)
> at
> org.apache.storm.daemon.supervisor$_main.invoke(supervisor.clj:1233)
> at clojure.lang.AFn.applyToHelper(AFn.java:152)
> at clojure.lang.AFn.applyTo(AFn.java:144)
> at org.apache.storm.daemon.supervisor.main(Unknown Source)
> 2016-10-07 09:49:33.608 o.a.s.util [ERROR] Halting process: ("Error on
> initialization")
> java.lang.RuntimeException: ("Error on initialization")
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> at clojure.lang.RestFn.invoke(RestFn.java:423)
> at
> org.apache.storm.daemon.supervisor$fn__9307$mk_supervisor__9352.doInvoke(supervisor.clj:763)
> at clojure.lang.RestFn.invoke(RestFn.java:436)
> at
> org.apache.storm.daemon.supervisor$_launch.invoke(supervisor.clj:1200)
> at
> org.apache.storm.daemon.supervisor$_main.invoke(supervisor.clj:1233)
> at clojure.lang.AFn.applyToHelper(AFn.java:152)
> at clojure.lang.AFn.applyTo(AFn.java:144)
> at org.apache.storm.daemon.supervisor.main(Unknown Source)
> 2016-10-07 09:49:34.668 o.a.s.d.supervisor [INFO] Removing code for storm id
> production_2016_09_13-1-1475831938
> We have looked at https://github.com/apache/storm/pull/418 and
> https://issues.apache.org/jira/browse/STORM-130, which both show the first
> issue as being fixed - however we are still experiencing it in 1.0.2. The
> changes from the fixing commit
> (https://github.com/apache/storm/pull/418/commits/ccd28f8a356f468e66865fa9d9901b0a2628ec74)
> don't seem to be in the current version of the file
> (https://github.com/apache/storm/blob/v1.0.2/storm-core/src/clj/org/apache/storm/daemon/supervisor.clj).
> We get this often when resubmitting a topology, and our only workaround is to
> stop the topology, delete the whole /opt/storm_local directory (which is our
> storm.local.dir) and resubmit the topology. Often, the workers seem to be
> looking for stormconf.ser in the local directory of an old topology that
> isn't even running at the time.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
