Hmm cleaning the storm data directory and restarting the supervisor has always worked for me.
On Tue, Mar 10, 2015 at 11:05 AM, Sa Li <[email protected]> wrote: > Unfortunately, it will still come out the error like this : > > 2015-03-09T17:02:51.826-0700 b.s.d.worker [ERROR] Error on initialization > of server mk-worker > java.io.FileNotFoundException: File > '/srv/tmpvar/storm/data/supervisor/stormdist/eventsRawv1-44-1425945533/stormconf.ser' > does not exist > at > org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:299) > ~[commons-io-2.4.jar:2.4] > at > org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1763) > ~[commons-io-2.4.jar:2.4] > at > backtype.storm.config$read_supervisor_storm_conf.invoke(config.clj:212) > ~[storm-core-0.9.3.jar:0.9.3] > at backtype.storm.daemon.worker$worker_data.invoke(worker.clj:182) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.daemon.worker$fn__3743$exec_fn__1108__auto____3744.invoke(worker.clj:363) > ~[storm-core-0.9.3.jar:0.9.3] > at clojure.lang.AFn.applyToHelper(AFn.java:185) > [clojure-1.5.1.jar:na] > at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na] > at clojure.core$apply.invoke(core.clj:617) ~[clojure-1.5.1.jar:na] > at > backtype.storm.daemon.worker$fn__3743$mk_worker__3799.doInvoke(worker.clj:354) > [storm-core-0.9.3.jar:0.9.3] > at clojure.lang.RestFn.invoke(RestFn.java:512) > [clojure-1.5.1.jar:na] > at backtype.storm.daemon.worker$_main.invoke(worker.clj:461) > [storm-core-0.9.3.jar:0.9.3] > at clojure.lang.AFn.applyToHelper(AFn.java:172) > [clojure-1.5.1.jar:na] > at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na] > at backtype.storm.daemon.worker.main(Unknown Source) > [storm-core-0.9.3.jar:0.9.3] > 2015-03-09T17:02:51.835-0700 b.s.util [ERROR] Halting process: ("Error on > initialization") > java.lang.RuntimeException: ("Error on initialization") > at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) > [storm-core-0.9.3.jar:0.9.3] > at clojure.lang.RestFn.invoke(RestFn.java:423) > [clojure-1.5.1.jar:na] > at > backtype.storm.daemon.worker$fn__3743$mk_worker__3799.doInvoke(worker.clj:354) > [storm-core-0.9.3.jar:0.9.3] > at clojure.lang.RestFn.invoke(RestFn.java:512) > [clojure-1.5.1.jar:na] > at backtype.storm.daemon.worker$_main.invoke(worker.clj:461) > [storm-core-0.9.3.jar:0.9.3] > at clojure.lang.AFn.applyToHelper(AFn.java:172) > [clojure-1.5.1.jar:na] > at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na] > at backtype.storm.daemon.worker.main(Unknown Source) > [storm-core-0.9.3.jar:0.9.3] > > > On Fri, Mar 6, 2015 at 3:37 PM, Kosala Dissanayake <[email protected]> > wrote: > >> Yes. Or maybe just rename the directory if you are not comfortable >> deleting stuff. >> >> On Sat, Mar 7, 2015 at 4:26 AM, Sa Li <[email protected]> wrote: >> >>> Thank you Kosala, this is my storm data directory of one supervisor >>> node: >>> >>> [root@complicated-laugh:/srv/tmpvar/storm/data]$ tree >>> . >>> ├── nimbus >>> │ ├── inbox >>> │ └── stormdist >>> │ └── eventsStreamerv1-66-1425605777 >>> │ ├── stormcode.ser >>> │ ├── stormconf.ser >>> │ └── stormjar.jar >>> ├── supervisor >>> │ ├── isupervisor >>> │ │ ├── 1425642310552 >>> │ │ ├── 1425642310552.version >>> │ │ ├── 1425646891479 >>> │ │ ├── 1425646891479.version >>> │ │ ├── 1425652232940 >>> │ │ ├── 1425652232940.version >>> │ │ ├── 1425654754084 >>> │ │ └── 1425654754084.version >>> │ ├── localstate >>> │ │ ├── 1425662664867 >>> │ │ ├── 1425662664867.version >>> │ │ ├── 1425662666276 >>> │ │ ├── 1425662666276.version >>> │ │ ├── 1425662666277 >>> │ │ ├── 1425662666277.version >>> │ │ ├── 1425662667867 >>> │ │ └── 1425662667867.version >>> │ ├── stormdist >>> │ └── tmp >>> └── workers >>> >>> Are you saying I should delete them all (including subdirectories? >>> >>> Thanks >>> >>> AL >>> >>> On Thu, Mar 5, 2015 at 7:41 PM, Kosala Dissanayake <[email protected] >>> > wrote: >>> >>>> Clear your storm data directory and restart your supervisors. >>>> >>>> On Fri, Mar 6, 2015 at 12:40 PM, Sa Li <[email protected]> wrote: >>>> >>>>> My storm is 0.9.3 by the way >>>>> >>>>> On Thu, Mar 5, 2015 at 5:39 PM, Sa Li <[email protected]> wrote: >>>>> >>>>>> Hi, All >>>>>> >>>>>> I am running a trident topo, and having such error, and it is strange >>>>>> that I can't see the error on storm UI, but I see it on worker-6703.log >>>>>> of >>>>>> one node, >>>>>> >>>>>> 2015-03-05T17:29:36.859-0800 b.s.m.n.Client [INFO] failed to send >>>>>> requests to visible-alley/10.100.98.105:6703: >>>>>> java.nio.channels.ClosedChannelException: null >>>>>> at >>>>>> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:405) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> at >>>>>> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> at >>>>>> org.apache.storm.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> at >>>>>> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> at >>>>>> org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> at >>>>>> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> at >>>>>> org.apache.storm.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> at >>>>>> org.apache.storm.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> at >>>>>> org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>> [na:1.7.0_65] >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>> [na:1.7.0_65] >>>>>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] >>>>>> 2015-03-05T17:29:36.859-0800 b.s.m.n.Client [INFO] failed to send >>>>>> requests to visible-alley/10.100.98.105:6703: >>>>>> java.nio.channels.ClosedChannelException: null >>>>>> at >>>>>> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:405) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> at >>>>>> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> at >>>>>> org.apache.storm.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> at >>>>>> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> at >>>>>> org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> at >>>>>> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> at >>>>>> org.apache.storm.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> at >>>>>> org.apache.storm.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> at >>>>>> org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>> [na:1.7.0_65] >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>> [na:1.7.0_65] >>>>>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] >>>>>> >>>>>> >>>>>> and such error on another node: >>>>>> 2015-03-05T17:33:52.892-0800 b.s.d.worker [ERROR] Error on >>>>>> initialization of server mk-worker >>>>>> java.io.FileNotFoundException: File >>>>>> '/srv/tmpvar/storm/data/supervisor/stormdist/eventsStreamerv1-65-1425604313/stormconf.ser' >>>>>> does not exist >>>>>> at >>>>>> org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:299) >>>>>> ~[commons-io-2.4.jar:2.4] >>>>>> at >>>>>> org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1763) >>>>>> ~[commons-io-2.4.jar:2.4] >>>>>> at >>>>>> backtype.storm.config$read_supervisor_storm_conf.invoke(config.clj:212) >>>>>> ~[storm-core-0.9.3.jar:0.9.3] >>>>>> at >>>>>> backtype.storm.daemon.worker$worker_data.invoke(worker.clj:182) >>>>>> ~[storm-core-0.9.3.jar:0.9.3] >>>>>> at >>>>>> backtype.storm.daemon.worker$fn__3743$exec_fn__1108__auto____3744.invoke(worker.clj:363) >>>>>> ~[storm-core-0.9.3.jar:0.9.3] >>>>>> at clojure.lang.AFn.applyToHelper(AFn.java:185) >>>>>> [clojure-1.5.1.jar:na] >>>>>> at clojure.lang.AFn.applyTo(AFn.java:151) >>>>>> [clojure-1.5.1.jar:na] >>>>>> at clojure.core$apply.invoke(core.clj:617) >>>>>> ~[clojure-1.5.1.jar:na] >>>>>> at >>>>>> backtype.storm.daemon.worker$fn__3743$mk_worker__3799.doInvoke(worker.clj:354) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> at clojure.lang.RestFn.invoke(RestFn.java:512) >>>>>> [clojure-1.5.1.jar:na] >>>>>> at backtype.storm.daemon.worker$_main.invoke(worker.clj:461) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> at clojure.lang.AFn.applyToHelper(AFn.java:172) >>>>>> [clojure-1.5.1.jar:na] >>>>>> at clojure.lang.AFn.applyTo(AFn.java:151) >>>>>> [clojure-1.5.1.jar:na] >>>>>> at backtype.storm.daemon.worker.main(Unknown Source) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> 2015-03-05T17:33:52.902-0800 b.s.util [ERROR] Halting process: >>>>>> ("Error on initialization") >>>>>> java.lang.RuntimeException: ("Error on initialization") >>>>>> at >>>>>> backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> at clojure.lang.RestFn.invoke(RestFn.java:423) >>>>>> [clojure-1.5.1.jar:na] >>>>>> at >>>>>> backtype.storm.daemon.worker$fn__3743$mk_worker__3799.doInvoke(worker.clj:354) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> at clojure.lang.RestFn.invoke(RestFn.java:512) >>>>>> [clojure-1.5.1.jar:na] >>>>>> at backtype.storm.daemon.worker$_main.invoke(worker.clj:461) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> at clojure.lang.AFn.applyToHelper(AFn.java:172) >>>>>> [clojure-1.5.1.jar:na] >>>>>> at clojure.lang.AFn.applyTo(AFn.java:151) >>>>>> [clojure-1.5.1.jar:na] >>>>>> at backtype.storm.daemon.worker.main(Unknown Source) >>>>>> [storm-core-0.9.3.jar:0.9.3] >>>>>> >>>>>> This is my yaml, >>>>>> >>>>>> storm.zookeeper.servers: >>>>>> - "10.100.98.100" >>>>>> - "10.100.98.101" >>>>>> - "10.100.98.102" >>>>>> - "10.100.98.103" >>>>>> - "10.100.98.104" >>>>>> - "10.100.98.105" >>>>>> - "10.100.98.106" >>>>>> >>>>>> storm.zookeeper.port: 2181 >>>>>> #nimbus.host: "localhost" >>>>>> nimbus.thrift.port: 6627 >>>>>> ui.port: 8080 >>>>>> nimbus.host: "10.100.98.103" >>>>>> storm.local.dir: "/srv/tmpvar/storm/data" >>>>>> #java.library.path: "/usr/local/lib" >>>>>> java.library.path: "/usr/lib/jvm/java-7-openjdk-amd64" >>>>>> supervisor.slots.ports: >>>>>> - 6700 >>>>>> - 6701 >>>>>> - 6702 >>>>>> - 6703 >>>>>> nimbus.childopts: "-Xmx1024m -Djava.net.preferIPv4Stack=true" >>>>>> ui.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true" >>>>>> supervisor.childopts: "-Djava.net.preferIPv4Stack=true" >>>>>> worker.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true" >>>>>> >>>>>> >>>>>> Do I need to add netty stuff? >>>>>> >>>>>> thanks >>>>>> >>>>>> AL >>>>>> >>>>>> >>>>> >>>> >>> >> >
