Clear your storm data directory and restart your supervisors. On Fri, Mar 6, 2015 at 12:40 PM, Sa Li <[email protected]> wrote:
> My storm is 0.9.3 by the way > > On Thu, Mar 5, 2015 at 5:39 PM, Sa Li <[email protected]> wrote: > >> Hi, All >> >> I am running a trident topo, and having such error, and it is strange >> that I can't see the error on storm UI, but I see it on worker-6703.log of >> one node, >> >> 2015-03-05T17:29:36.859-0800 b.s.m.n.Client [INFO] failed to send >> requests to visible-alley/10.100.98.105:6703: >> java.nio.channels.ClosedChannelException: null >> at >> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:405) >> [storm-core-0.9.3.jar:0.9.3] >> at >> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) >> [storm-core-0.9.3.jar:0.9.3] >> at >> org.apache.storm.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) >> [storm-core-0.9.3.jar:0.9.3] >> at >> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) >> [storm-core-0.9.3.jar:0.9.3] >> at >> org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) >> [storm-core-0.9.3.jar:0.9.3] >> at >> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) >> [storm-core-0.9.3.jar:0.9.3] >> at >> org.apache.storm.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) >> [storm-core-0.9.3.jar:0.9.3] >> at >> org.apache.storm.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) >> [storm-core-0.9.3.jar:0.9.3] >> at >> org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >> [storm-core-0.9.3.jar:0.9.3] >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> [na:1.7.0_65] >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> [na:1.7.0_65] >> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] >> 2015-03-05T17:29:36.859-0800 b.s.m.n.Client [INFO] failed to send >> requests to visible-alley/10.100.98.105:6703: >> java.nio.channels.ClosedChannelException: null >> at >> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:405) >> [storm-core-0.9.3.jar:0.9.3] >> at >> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) >> [storm-core-0.9.3.jar:0.9.3] >> at >> org.apache.storm.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) >> [storm-core-0.9.3.jar:0.9.3] >> at >> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) >> [storm-core-0.9.3.jar:0.9.3] >> at >> org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) >> [storm-core-0.9.3.jar:0.9.3] >> at >> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) >> [storm-core-0.9.3.jar:0.9.3] >> at >> org.apache.storm.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) >> [storm-core-0.9.3.jar:0.9.3] >> at >> org.apache.storm.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) >> [storm-core-0.9.3.jar:0.9.3] >> at >> org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >> [storm-core-0.9.3.jar:0.9.3] >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> [na:1.7.0_65] >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> [na:1.7.0_65] >> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] >> >> >> and such error on another node: >> 2015-03-05T17:33:52.892-0800 b.s.d.worker [ERROR] Error on initialization >> of server mk-worker >> java.io.FileNotFoundException: File >> '/srv/tmpvar/storm/data/supervisor/stormdist/eventsStreamerv1-65-1425604313/stormconf.ser' >> does not exist >> at >> org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:299) >> ~[commons-io-2.4.jar:2.4] >> at >> org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1763) >> ~[commons-io-2.4.jar:2.4] >> at >> backtype.storm.config$read_supervisor_storm_conf.invoke(config.clj:212) >> ~[storm-core-0.9.3.jar:0.9.3] >> at >> backtype.storm.daemon.worker$worker_data.invoke(worker.clj:182) >> ~[storm-core-0.9.3.jar:0.9.3] >> at >> backtype.storm.daemon.worker$fn__3743$exec_fn__1108__auto____3744.invoke(worker.clj:363) >> ~[storm-core-0.9.3.jar:0.9.3] >> at clojure.lang.AFn.applyToHelper(AFn.java:185) >> [clojure-1.5.1.jar:na] >> at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na] >> at clojure.core$apply.invoke(core.clj:617) ~[clojure-1.5.1.jar:na] >> at >> backtype.storm.daemon.worker$fn__3743$mk_worker__3799.doInvoke(worker.clj:354) >> [storm-core-0.9.3.jar:0.9.3] >> at clojure.lang.RestFn.invoke(RestFn.java:512) >> [clojure-1.5.1.jar:na] >> at backtype.storm.daemon.worker$_main.invoke(worker.clj:461) >> [storm-core-0.9.3.jar:0.9.3] >> at clojure.lang.AFn.applyToHelper(AFn.java:172) >> [clojure-1.5.1.jar:na] >> at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na] >> at backtype.storm.daemon.worker.main(Unknown Source) >> [storm-core-0.9.3.jar:0.9.3] >> 2015-03-05T17:33:52.902-0800 b.s.util [ERROR] Halting process: ("Error on >> initialization") >> java.lang.RuntimeException: ("Error on initialization") >> at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) >> [storm-core-0.9.3.jar:0.9.3] >> at clojure.lang.RestFn.invoke(RestFn.java:423) >> [clojure-1.5.1.jar:na] >> at >> backtype.storm.daemon.worker$fn__3743$mk_worker__3799.doInvoke(worker.clj:354) >> [storm-core-0.9.3.jar:0.9.3] >> at clojure.lang.RestFn.invoke(RestFn.java:512) >> [clojure-1.5.1.jar:na] >> at backtype.storm.daemon.worker$_main.invoke(worker.clj:461) >> [storm-core-0.9.3.jar:0.9.3] >> at clojure.lang.AFn.applyToHelper(AFn.java:172) >> [clojure-1.5.1.jar:na] >> at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na] >> at backtype.storm.daemon.worker.main(Unknown Source) >> [storm-core-0.9.3.jar:0.9.3] >> >> This is my yaml, >> >> storm.zookeeper.servers: >> - "10.100.98.100" >> - "10.100.98.101" >> - "10.100.98.102" >> - "10.100.98.103" >> - "10.100.98.104" >> - "10.100.98.105" >> - "10.100.98.106" >> >> storm.zookeeper.port: 2181 >> #nimbus.host: "localhost" >> nimbus.thrift.port: 6627 >> ui.port: 8080 >> nimbus.host: "10.100.98.103" >> storm.local.dir: "/srv/tmpvar/storm/data" >> #java.library.path: "/usr/local/lib" >> java.library.path: "/usr/lib/jvm/java-7-openjdk-amd64" >> supervisor.slots.ports: >> - 6700 >> - 6701 >> - 6702 >> - 6703 >> nimbus.childopts: "-Xmx1024m -Djava.net.preferIPv4Stack=true" >> ui.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true" >> supervisor.childopts: "-Djava.net.preferIPv4Stack=true" >> worker.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true" >> >> >> Do I need to add netty stuff? >> >> thanks >> >> AL >> >> >
