[
https://issues.apache.org/jira/browse/STORM-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248377#comment-14248377
]
scooler commented on STORM-582:
-------------------------------
it's also happen in storm-0.9.3
=============================================================
2014-12-16T20:23:33.956+0800 b.s.d.supervisor [INFO] Downloading code for storm
id tv-five-minute-51-1418663725 from
/opt/data/goldmine/storm/local-dir/nimbus/stormdist/tv-five-minute-51-1418663725
2014-12-16T20:23:38.914+0800 b.s.d.supervisor [INFO] Finished downloading code
for storm id tv-five-minute-51-1418663725 from
/opt/data/goldmine/storm/local-dir/nimbus/stormdist/tv-five-minute-51-1418663725
2014-12-16T20:23:38.961+0800 b.s.d.supervisor [INFO] Launching worker with
command: '/usr/local/java/jdk/default/bin/java' '-server' '-Xms1536m'
'-Xmx1536m' '-XX:NewSize=256m' '-XX:MaxNewSize=256m' '-XX:PermSize=128M'
'-XX:+UseConcMarkSweepGC' '-XX:CMSInitiatingOccupancyFraction=70'
'-XX:+PrintGCDetails' '-XX:+PrintGCDateStamps' '-XX:+PrintTenuringDistribution'
'-Xloggc:/opt/data/goldmine/storm/logs/gc.log' '-Djava.awt.headless=true'
'-XX:+HeapDumpOnOutOfMemoryError'
'-XX:HeapDumpPath=/opt/data/goldmine/storm/logs/gc_dump'
'-Djava.library.path=/opt/data/goldmine/storm/local-dir/supervisor/stormdist/tv-five-minute-51-1418663725/resources/Linux-amd64:/opt/data/goldmine/storm/local-dir/supervisor/stormdist/tv-five-minute-51-1418663725/resources:/usr/local/lib:/usr/lib'
'-Dlogfile.name=worker-6702.log'
'-Dstorm.home=/usr/local/goldmine/storm/apache-storm-0.9.3'
'-Dstorm.conf.file=' '-Dstorm.options='
'-Dstorm.log.dir=/opt/data/goldmine/storm/logs'
'-Dlogback.configurationFile=/usr/local/goldmine/storm/apache-storm-0.9.3/logback/cluster.xml'
'-Dstorm.id=tv-five-minute-51-1418663725'
'-Dworker.id=8cb0ef9e-a02a-4a71-8766-f5fd9b62931f' '-Dworker.port=6702' '-cp'
'/usr/local/goldmine/storm/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/goldmine/storm/apache-storm-0.9.3/conf:/opt/data/goldmine/storm/local-dir/supervisor/stormdist/tv-five-minute-51-1418663725/stormjar.jar'
'backtype.storm.daemon.worker' 'tv-five-minute-51-1418663725'
'7d010127-4adb-4f72-8928-136e30d1a7ec' '6702'
'8cb0ef9e-a02a-4a71-8766-f5fd9b62931f'
2014-12-16T20:23:40.466+0800 b.s.d.supervisor [INFO]
8cb0ef9e-a02a-4a71-8766-f5fd9b62931f still hasn't started
2014-12-16T20:23:40.966+0800 b.s.d.supervisor [INFO]
8cb0ef9e-a02a-4a71-8766-f5fd9b62931f still hasn't started
2014-12-16T20:23:41.467+0800 b.s.d.supervisor [INFO]
8cb0ef9e-a02a-4a71-8766-f5fd9b62931f still hasn't started
2014-12-16T20:23:41.967+0800 b.s.d.supervisor [INFO]
8cb0ef9e-a02a-4a71-8766-f5fd9b62931f still hasn't started
2014-12-16T20:23:42.468+0800 b.s.d.supervisor [INFO]
8cb0ef9e-a02a-4a71-8766-f5fd9b62931f still hasn't started
2014-12-16T20:23:42.969+0800 b.s.d.supervisor [INFO]
8cb0ef9e-a02a-4a71-8766-f5fd9b62931f still hasn't started
2014-12-16T20:23:43.469+0800 b.s.d.supervisor [INFO]
8cb0ef9e-a02a-4a71-8766-f5fd9b62931f still hasn't started
2014-12-16T20:23:43.970+0800 b.s.d.supervisor [INFO]
8cb0ef9e-a02a-4a71-8766-f5fd9b62931f still hasn't started
2014-12-16T20:23:44.471+0800 b.s.d.supervisor [INFO]
8cb0ef9e-a02a-4a71-8766-f5fd9b62931f still hasn't started
2014-12-16T20:23:44.972+0800 b.s.d.supervisor [INFO]
8cb0ef9e-a02a-4a71-8766-f5fd9b62931f still hasn't started
2014-12-16T20:39:53.668+0800 b.s.d.supervisor [INFO] Shutting down and clearing
state for id 8cb0ef9e-a02a-4a71-8766-f5fd9b62931f. Current supervisor time:
1418733593. State: :timed-out, Heartbeat:
#backtype.storm.daemon.common.WorkerHeartbeat{:time-secs 1418733561, :storm-id
"tv-five-minute-51-1418663725", :executors #{[395 395] [-1 -1] [113 127]},
:port 6702}
2014-12-16T20:39:53.669+0800 b.s.d.supervisor [INFO] Shutting down
7d010127-4adb-4f72-8928-136e30d1a7ec:8cb0ef9e-a02a-4a71-8766-f5fd9b62931f
2014-12-16T20:39:53.672+0800 b.s.util [INFO] Error when trying to kill 2465.
Process is probably already dead.
2014-12-16T20:39:53.995+0800 b.s.d.supervisor [INFO] Removing code for storm id
tv-five-minute-51-1418663725
2014-12-16T20:39:54.675+0800 b.s.util [INFO] Error when trying to kill 2465.
Process is probably already dead.
2014-12-16T20:39:54.685+0800 b.s.d.supervisor [INFO] Shut down
7d010127-4adb-4f72-8928-136e30d1a7ec:8cb0ef9e-a02a-4a71-8766-f5fd9b62931f
2014-12-16T20:39:54.689+0800 b.s.d.supervisor [INFO] Launching worker with
assignment #backtype.storm.daemon.supervisor.LocalAssignment{:storm-id
"tv-five-minute-51-1418663725", :executors ([395 395] [113 127])} for this
supervisor 7d010127-4adb-4f72-8928-136e30d1a7ec on port 6702 with id
e88d43de-e80b-4c2f-ad5f-a9ae5b4d5e45
2014-12-16T20:39:54.699+0800 b.s.event [ERROR] Error when processing event
java.io.FileNotFoundException: File
'/opt/data/goldmine/storm/local-dir/supervisor/stormdist/tv-five-minute-51-1418663725/stormconf.ser'
does not exist
at org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:299)
~[commons-io-2.4.jar:2.4]
at
org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1763)
~[commons-io-2.4.jar:2.4]
at
backtype.storm.config$read_supervisor_storm_conf.invoke(config.clj:212)
~[storm-core-0.9.3.jar:0.9.3]
at backtype.storm.daemon.supervisor$fn__4256.invoke(supervisor.clj:509)
~[storm-core-0.9.3.jar:0.9.3]
at clojure.lang.MultiFn.invoke(MultiFn.java:241) ~[clojure-1.5.1.jar:na]
at
backtype.storm.daemon.supervisor$sync_processes$iter__4116__4120$fn__4121.invoke(supervisor.clj:285)
~[storm-core-0.9.3.jar:0.9.3]
at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.5.1.jar:na]
at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.5.1.jar:na]
at clojure.lang.RT.seq(RT.java:484) ~[clojure-1.5.1.jar:na]
at clojure.core$seq.invoke(core.clj:133) ~[clojure-1.5.1.jar:na]
at clojure.core$dorun.invoke(core.clj:2780) ~[clojure-1.5.1.jar:na]
at clojure.core$doall.invoke(core.clj:2796) ~[clojure-1.5.1.jar:na]
at
backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:273)
~[storm-core-0.9.3.jar:0.9.3]
at clojure.lang.AFn.applyToHelper(AFn.java:161) [clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na]
at clojure.core$apply.invoke(core.clj:619) ~[clojure-1.5.1.jar:na]
at clojure.core$partial$fn__4190.doInvoke(core.clj:2396)
~[clojure-1.5.1.jar:na]
at clojure.lang.RestFn.invoke(RestFn.java:397) ~[clojure-1.5.1.jar:na]
at backtype.storm.event$event_manager$fn__2467.invoke(event.clj:40)
~[storm-core-0.9.3.jar:0.9.3]
===================================================================================
after removing code for storm id tv-five-minute-51-1418663725
supervisor logs "Launching worker with assignment......",then throw exception
"FileNotFoundException" and supervisor dead
> Nimbus Halt with FileNotFoundException: '../nimbus/../stormconf.ser' dose not
> exists
> ------------------------------------------------------------------------------------
>
> Key: STORM-582
> URL: https://issues.apache.org/jira/browse/STORM-582
> Project: Apache Storm
> Issue Type: Bug
> Affects Versions: 0.9.2-incubating
> Reporter: Jiahong Li
>
> To notice, it is different from STORM-130. It is nimbus to halt. We ran into
> this problem several times, every time it happens is after several days of
> stable running. Here is the stacktrace
> ======================================================
> 2014-11-03 14:30:56 b.s.d.nimbus [INFO] Cleaning inbox ... deleted:
> stormjar-c1c856f0-cf8b-4299-9c20-712f169802b7.jar
> 2014-11-12 19:32:33 b.s.d.nimbus [ERROR] Error when processing event
> java.io.FileNotFoundException: File
> '/tmp/storm-0.9.3/nimbus/stormdist/DNSAnalyse-7-1414992073/stormconf.ser'
> does not exist
> at
> org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:299)
> ~[commons-io-2.4.jar:2.4]
> at
> org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1763)
> ~[commons-io-2.4.jar:2.4]
> at backtype.storm.daemon.nimbus$read_storm_conf.invoke(nimbus.clj:89)
> ~[storm-core-0.9.3-incubating-SNAPSHOT.jar:0.9.3-incubating-SNAPSHOT]
> at
> backtype.storm.daemon.nimbus$read_topology_details.invoke(nimbus.clj:324)
> ~[storm-core-0.9.3-incubating-SNAPSHOT.jar:0.9.3-incubating-SNAPSHOT]
> at
> backtype.storm.daemon.nimbus$mk_assignments$iter__3100__3104$fn__3105.invoke(nimbus.clj:649)
> ~[storm-core-0.9.3-incubating-SNAPSHOT.jar:0.9.3-incubating-SNAPSHOT]
> at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.5.1.jar:na]
> at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.5.1.jar:na]
> at clojure.lang.RT.seq(RT.java:484) ~[clojure-1.5.1.jar:na]
> at clojure.core$seq.invoke(core.clj:133) ~[clojure-1.5.1.jar:na]
> at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30)
> ~[clojure-1.5.1.jar:na]
> at clojure.core.protocols$fn__6026.invoke(protocols.clj:54)
> ~[clojure-1.5.1.jar:na]
> at
> clojure.core.protocols$fn__5979$G__5974__5992.invoke(protocols.clj:13)
> ~[clojure-1.5.1.jar:na]
> at clojure.core$reduce.invoke(core.clj:6177) ~[clojure-1.5.1.jar:na]
> at clojure.core$into.invoke(core.clj:6229) ~[clojure-1.5.1.jar:na]
> at
> backtype.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:648)
> ~[storm-core-0.9.3-incubating-SNAPSHOT.jar:0.9.3-incubating-SNAPSHOT]
> at clojure.lang.RestFn.invoke(RestFn.java:410) ~[clojure-1.5.1.jar:na]
> at
> backtype.storm.daemon.nimbus$fn__3281$exec_fn__1205__auto____3282$fn__3287$fn__3288.invoke(nimbus.clj:907)
> ~[storm-core-0.9.3-incubating-SNAPSHOT.jar:0.9.3-incubating-SNAPSHOT]
> at
> backtype.storm.daemon.nimbus$fn__3281$exec_fn__1205__auto____3282$fn__3287.invoke(nimbus.clj:906)
> ~[storm-core-0.9.3-incubating-SNAPSHOT.jar:0.9.3-incubating-SNAPSHOT]
> at
> backtype.storm.timer$schedule_recurring$this__2169.invoke(timer.clj:99)
> ~[storm-core-0.9.3-incubating-SNAPSHOT.jar:0.9.3-incubating-SNAPSHOT]
> at
> backtype.storm.timer$mk_timer$fn__2152$fn__2153.invoke(timer.clj:50)
> ~[storm-core-0.9.3-incubating-SNAPSHOT.jar:0.9.3-incubating-SNAPSHOT]
> at backtype.storm.timer$mk_timer$fn__2152.invoke(timer.clj:42)
> [storm-core-0.9.3-incubating-SNAPSHOT.jar:0.9.3-incubating-SNAPSHOT]
> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
> at java.lang.Thread.run(Thread.java:679) [na:1.6.0_22]
> 2014-11-12 19:32:33 b.s.util [INFO] Halting process: ("Error when processing
> an event")
> 2014-11-12 19:32:33 b.s.d.nimbus [INFO] Shutting down master
> =====================================================
> To notice, after stable running for 9 days, without printing "Clean up
> {storm-id}" logs, nimbus halt.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
