Hello,
Problem:
Topology can be uploaded to storm successfully, and seems it works more
or less fine
BUT i cannot kill it, there are something left always according nimbus
and supervisors logs.
(and in the same time `storm kill` itself completed successfully, UI
shows NO topologies any more)
Moreover, after first kill new topologies uploaded become non-functional.
Was made short investigation of logs.
Looks like a storm process tries to shutdown topology and addresses
topology's files (internal check for working dir failed), does not find
them and crushes.
And it's repeated infinitely.
Second observations:
/opt/storm/workers/ contains only heartbeats/ and pids/ info
looks like topology jars + multilang files was removed somehow
The only way to make the cluster functional again i see in cleaning
/opt/storm/{supervisor,workers,nimbus} and restarting all storm services.
Probably i could clean only directories containing target topology
files, but probably it is not the best solution for topology killing.
I believe there are some magic set of config settings to diminish the
effect. But till now i did not succeed with it.
So any advices on how to fix/override this storm misbehaviour are
appreciated.
----------------------------------------------
MORE TECH DETAILS
*** Servers (VPS) ***
- mem: 4G, disk: 20G, 3CPU*2.5GHz, CentOS6.6
*** Storm ***
- 0.9.3-rc1
- 1x(nimbus + ui) + 3x(supervisor + drpc + zookeeper)
*** Topology ***
8<===========
public static void main(String[] args) throws Exception {
LinearDRPCTopologyBuilder builder = new
LinearDRPCTopologyBuilder("domain.check");
builder.addBolt(new CheckDomainBolt(), 9);
Config conf = new Config();
conf.setDebug(true);
conf.setNumWorkers(6);
StormSubmitter.submitTopology(args[0], conf,
builder.createRemoteTopology());
}
8<===========
with PHP bolt wrapped in ShellBolt
*** Logs ***
*** Nimbus ***
2014-11-12T21:11:14,387+01:00 host.name b.s.d.nimbus INFO Cleaning up
domain-check-1-1415820694
2014-11-12T21:11:24,479+01:00 host.name b.s.d.nimbus INFO Cleaning up
domain-check-1-1415820694
2014-11-12T21:11:34,560+01:00 host.name b.s.d.nimbus INFO Cleaning up
domain-check-1-1415820694
2014-11-12T21:11:34,658+01:00 host.name b.s.cluster WARN Could not
teardown heartbeats for domain-check-1-1415820694 #<RuntimeException
java.lang.RuntimeException:
org.apache.storm.zookeeper.KeeperException$NotEmptyException:
KeeperErrorCode = Directory not empty for
/workerbeats/domain-check-1-1415820694>
2014-11-12T21:11:44,698+01:00 host.name b.s.d.nimbus INFO Cleaning up
domain-check-1-1415820694
2014-11-12T21:11:54,778+01:00 host.name b.s.d.nimbus INFO Cleaning up
domain-check-1-1415820694
*** Supervisor node (this part repeated infinitly the the log as i can
see) ***
2014-11-13T08:20:56,678+01:00 <WORKER1 HOSTNAME> o.a.s.z.ZooKeeper INFO
Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014
09:09 GMT
2014-11-13T08:20:56,681+01:00 <WORKER1 HOSTNAME> o.a.s.z.ZooKeeper INFO
Client environment:host.name=<WORKER1 IP>
2014-11-13T08:20:56,681+01:00 <WORKER1 HOSTNAME> o.a.s.z.ZooKeeper INFO
Client environment:java.version=1.7.0_71
2014-11-13T08:20:56,681+01:00 <WORKER1 HOSTNAME> o.a.s.z.ZooKeeper INFO
Client environment:java.vendor=Oracle Corporation
2014-11-13T08:20:56,681+01:00 <WORKER1 HOSTNAME> o.a.s.z.ZooKeeper INFO
Client
environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71.x86_64/jre
2014-11-13T08:20:56,681+01:00 <WORKER1 HOSTNAME> o.a.s.z.ZooKeeper INFO
Client
environment:java.class.path=/opt/storm-0.9.3/lib/clout-1.0.1.jar:/opt/storm-0.9.3/lib/logback-classic-1.0.13.jar:/opt/storm-0.9.3/lib/disruptor-3.2.1.jar:/opt/storm-0.9.3/lib/json-simple-1.1.jar:/opt/storm-0.9.3/lib/carbonite-1.4.0.jar:/opt/storm-0.9.3/lib/objenesis-1.2.jar:/opt/storm-0.9.3/lib/ring-devel-0.3.11.jar:/opt/storm-0.9.3/lib/commons-logging-1.1.3.jar:/opt/storm-0.9.3/lib/clojure-1.5.1.jar:/opt/storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/opt/storm-0.9.3/lib/core.incubator-0.1.0.jar:/opt/storm-0.9.3/lib/commons-exec-1.1.jar:/opt/storm-0.9.3/lib/compojure-1.1.3.jar:/opt/storm-0.9.3/lib/snakeyaml-1.11.jar:/opt/storm-0.9.3/lib/hiccup-0.3.6.jar:/opt/storm-0.9.3/lib/logback-core-1.0.13.jar:/opt/storm-0.9.3/lib/ring-core-1.1.5.jar:/opt/storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm-0.9.3/lib/asm-4.0.jar:/opt/storm-0.9.3/lib/clj-time-0.4.1.jar:/opt/storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/opt/storm-0.9.3/lib/tools.macro-0.1.0.jar:/opt/storm-0.9.3/lib/commons-io-2.4.jar:/opt/storm-0.9.3/lib/ring-servlet-0.3.11.jar:/opt/storm-0.9.3/lib/kryo-2.21.jar:/opt/storm-0.9.3/lib/jetty-6.1.26.jar:/opt/storm-0.9.3/lib/servlet-api-2.5.jar:/opt/storm-0.9.3/lib/storm-core-0.9.3-rc1.jar:/opt/storm-0.9.3/lib/tools.logging-0.2.3.jar:/opt/storm-0.9.3/lib/jetty-util-6.1.26.jar:/opt/storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/opt/storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/opt/storm-0.9.3/lib/tools.cli-0.2.4.jar:/opt/storm-0.9.3/lib/minlog-1.2.jar:/opt/storm-0.9.3/lib/slf4j-api-1.7.5.jar:/opt/storm-0.9.3/lib/commons-codec-1.6.jar:/opt/storm-0.9.3/lib/jline-2.11.jar:/opt/storm-0.9.3/lib/chill-java-0.3.5.jar:/opt/storm-0.9.3/lib/joda-time-2.0.jar:/opt/storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/opt/storm-0.9.3/lib/commons-lang-2.5.jar:/opt/storm-0.9.3/conf
2014-11-13T08:20:56,681+01:00 <WORKER1 HOSTNAME> o.a.s.z.ZooKeeper INFO
Client environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
2014-11-13T08:20:56,681+01:00 <WORKER1 HOSTNAME> o.a.s.z.ZooKeeper INFO
Client environment:java.io.tmpdir=/tmp
2014-11-13T08:20:56,681+01:00 <WORKER1 HOSTNAME> o.a.s.z.ZooKeeper INFO
Client environment:java.compiler=<NA>
2014-11-13T08:20:56,681+01:00 <WORKER1 HOSTNAME> o.a.s.z.ZooKeeper INFO
Client environment:os.name=Linux
2014-11-13T08:20:56,681+01:00 <WORKER1 HOSTNAME> o.a.s.z.ZooKeeper INFO
Client environment:os.arch=amd64
2014-11-13T08:20:56,681+01:00 <WORKER1 HOSTNAME> o.a.s.z.ZooKeeper INFO
Client environment:os.version=2.6.32-504.el6.x86_64
2014-11-13T08:20:56,681+01:00 <WORKER1 HOSTNAME> o.a.s.z.ZooKeeper INFO
Client environment:user.name=dev
2014-11-13T08:20:56,681+01:00 <WORKER1 HOSTNAME> o.a.s.z.ZooKeeper INFO
Client environment:user.home=/opt/storm
2014-11-13T08:20:56,682+01:00 <WORKER1 HOSTNAME> o.a.s.z.ZooKeeper INFO
Client environment:user.dir=/root
2014-11-13T08:20:56,693+01:00 <WORKER1 HOSTNAME>
o.a.s.z.s.ZooKeeperServer INFO Server
environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2014-11-13T08:20:56,694+01:00 <WORKER1 HOSTNAME>
o.a.s.z.s.ZooKeeperServer INFO Server environment:host.name=<WORKER1 IP>
2014-11-13T08:20:56,694+01:00 <WORKER1 HOSTNAME>
o.a.s.z.s.ZooKeeperServer INFO Server environment:java.version=1.7.0_71
2014-11-13T08:20:56,694+01:00 <WORKER1 HOSTNAME>
o.a.s.z.s.ZooKeeperServer INFO Server environment:java.vendor=Oracle
Corporation
2014-11-13T08:20:56,694+01:00 <WORKER1 HOSTNAME>
o.a.s.z.s.ZooKeeperServer INFO Server
environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71.x86_64/jre
2014-11-13T08:20:56,694+01:00 <WORKER1 HOSTNAME>
o.a.s.z.s.ZooKeeperServer INFO Server
environment:java.class.path=/opt/storm-0.9.3/lib/clout-1.0.1.jar:/opt/storm-0.9.3/lib/logback-classic-1.0.13.jar:/opt/storm-0.9.3/lib/disruptor-3.2.1.jar:/opt/storm-0.9.3/lib/json-simple-1.1.jar:/opt/storm-0.9.3/lib/carbonite-1.4.0.jar:/opt/storm-0.9.3/lib/objenesis-1.2.jar:/opt/storm-0.9.3/lib/ring-devel-0.3.11.jar:/opt/storm-0.9.3/lib/commons-logging-1.1.3.jar:/opt/storm-0.9.3/lib/clojure-1.5.1.jar:/opt/storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/opt/storm-0.9.3/lib/core.incubator-0.1.0.jar:/opt/storm-0.9.3/lib/commons-exec-1.1.jar:/opt/storm-0.9.3/lib/compojure-1.1.3.jar:/opt/storm-0.9.3/lib/snakeyaml-1.11.jar:/opt/storm-0.9.3/lib/hiccup-0.3.6.jar:/opt/storm-0.9.3/lib/logback-core-1.0.13.jar:/opt/storm-0.9.3/lib/ring-core-1.1.5.jar:/opt/storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm-0.9.3/lib/asm-4.0.jar:/opt/storm-0.9.3/lib/clj-time-0.4.1.jar:/opt/storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/opt/storm-0.9.3/lib/tools.macro-0.1.0.jar:/opt/storm-0.9.3/lib/commons-io-2.4.jar:/opt/storm-0.9.3/lib/ring-servlet-0.3.11.jar:/opt/storm-0.9.3/lib/kryo-2.21.jar:/opt/storm-0.9.3/lib/jetty-6.1.26.jar:/opt/storm-0.9.3/lib/servlet-api-2.5.jar:/opt/storm-0.9.3/lib/storm-core-0.9.3-rc1.jar:/opt/storm-0.9.3/lib/tools.logging-0.2.3.jar:/opt/storm-0.9.3/lib/jetty-util-6.1.26.jar:/opt/storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/opt/storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/opt/storm-0.9.3/lib/tools.cli-0.2.4.jar:/opt/storm-0.9.3/lib/minlog-1.2.jar:/opt/storm-0.9.3/lib/slf4j-api-1.7.5.jar:/opt/storm-0.9.3/lib/commons-codec-1.6.jar:/opt/storm-0.9.3/lib/jline-2.11.jar:/opt/storm-0.9.3/lib/chill-java-0.3.5.jar:/opt/storm-0.9.3/lib/joda-time-2.0.jar:/opt/storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/opt/storm-0.9.3/lib/commons-lang-2.5.jar:/opt/storm-0.9.3/conf
2014-11-13T08:20:56,694+01:00 <WORKER1 HOSTNAME>
o.a.s.z.s.ZooKeeperServer INFO Server
environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
2014-11-13T08:20:56,694+01:00 <WORKER1 HOSTNAME>
o.a.s.z.s.ZooKeeperServer INFO Server environment:java.io.tmpdir=/tmp
2014-11-13T08:20:56,694+01:00 <WORKER1 HOSTNAME>
o.a.s.z.s.ZooKeeperServer INFO Server environment:java.compiler=<NA>
2014-11-13T08:20:56,694+01:00 <WORKER1 HOSTNAME>
o.a.s.z.s.ZooKeeperServer INFO Server environment:os.name=Linux
2014-11-13T08:20:56,694+01:00 <WORKER1 HOSTNAME>
o.a.s.z.s.ZooKeeperServer INFO Server environment:os.arch=amd64
2014-11-13T08:20:56,694+01:00 <WORKER1 HOSTNAME>
o.a.s.z.s.ZooKeeperServer INFO Server
environment:os.version=2.6.32-504.el6.x86_64
2014-11-13T08:20:56,694+01:00 <WORKER1 HOSTNAME>
o.a.s.z.s.ZooKeeperServer INFO Server environment:user.name=dev
2014-11-13T08:20:56,694+01:00 <WORKER1 HOSTNAME>
o.a.s.z.s.ZooKeeperServer INFO Server environment:user.home=/opt/storm
2014-11-13T08:20:56,694+01:00 <WORKER1 HOSTNAME>
o.a.s.z.s.ZooKeeperServer INFO Server environment:user.dir=/root
2014-11-13T08:20:57,213+01:00 <WORKER1 HOSTNAME> b.s.d.supervisor INFO
Starting Supervisor with conf {"dev.zookeeper.path"
"/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil,
"topology.builtin.metrics.bucket.size.secs" 60,
"topology.fall.back.on.java.serialization" true,
"topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000,
"topology.skip.missing.kryo.registrations" false,
"storm.messaging.netty.client_worker_threads" 1, "ui.childopts"
"-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign"
true, "topology.trident.batch.emit.interval.millis" 500, "drpc.servers"
["<WORKER1 IP>" "185.27.173.47" "<WORKER3 IP>"],
"storm.messaging.netty.flush.check.interval.ms" 10,
"nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m",
"java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib",
"topology.executor.send.buffer.size" 1024, "storm.local.dir"
"/opt/storm", "storm.messaging.netty.buffer_size" 5242880,
"supervisor.worker.start.timeout.secs" 120,
"topology.enable.message.timeouts" true,
"nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs"
3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate"
"backtype.storm.serialization.DefaultSerializationDelegate",
"topology.worker.shared.thread.pool.size" 4, "nimbus.host"
"185.27.174.45", "storm.messaging.netty.min_wait_ms" 300,
"storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil,
"topology.executor.receive.buffer.size" 1024,
"transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm",
"storm.zookeeper.retry.intervalceiling.millis" 30000,
"supervisor.enable" true, "storm.messaging.netty.server_worker_threads"
1, "storm.zookeeper.servers" ["<WORKER1 IP>" "185.27.173.47" "<WORKER3
IP>"], "transactional.zookeeper.root" "/transactional",
"topology.acker.executors" nil, "topology.transfer.buffer.size" 1024,
"topology.worker.childopts" nil, "drpc.queue.size" 128,
"worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5,
"topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port"
3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts"
"-Xmx768m", "topology.receiver.buffer.size" 8,
"task.heartbeat.frequency.secs" 3, "topology.tasks" nil,
"storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy"
"backtype.storm.spout.SleepSpoutWaitStrategy",
"nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending"
nil, "storm.zookeeper.retry.interval" 1000,
"topology.sleep.spout.wait.strategy.time.ms" 1,
"nimbus.topology.validator"
"backtype.storm.nimbus.DefaultTopologyValidator",
"supervisor.slots.ports" [6700 6701 6702 6703 6704 6705 6706 6707 6708
6709], "topology.environment" nil, "topology.debug" false,
"nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60,
"topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10,
"topology.workers" 1, "supervisor.childopts" "-Xmx256m",
"nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05,
"worker.heartbeat.frequency.secs" 3, "topology.tuple.serializer"
"backtype.storm.serialization.types.ListDelegateSerializer",
"topology.disruptor.wait.strategy"
"com.lmax.disruptor.BlockingWaitStrategy",
"topology.multilang.serializer"
"backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs"
30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory"
"backtype.storm.serialization.DefaultKryoFactory",
"drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1,
"storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count"
1, "storm.thrift.transport"
"backtype.storm.security.auth.SimpleTransportPlugin",
"topology.state.synchronization.timeout.secs" 60,
"supervisor.worker.timeout.secs" 15, "nimbus.file.copy.expiration.secs"
600, "storm.messaging.transport"
"backtype.storm.messaging.netty.Context", "logviewer.appender.name"
"A1", "storm.messaging.netty.max_wait_ms" 1500,
"drpc.request.timeout.secs" 600, "storm.local.hostname" "<WORKER1
HOSTNAME>", "storm.local.mode.zmq" false, "ui.port" 8080,
"nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "distributed",
"topology.max.task.parallelism" nil,
"storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath"
nil}
2014-11-13T08:20:57,244+01:00 <WORKER1 HOSTNAME>
b.s.u.StormBoundedExponentialBackoffRetry INFO The baseSleepTimeMs
[1000] the maxSleepTimeMs [30000] the maxRetries [5]
2014-11-13T08:20:57,325+01:00 <WORKER1 HOSTNAME>
o.a.s.c.f.i.CuratorFrameworkImpl INFO Starting
2014-11-13T08:20:57,328+01:00 <WORKER1 HOSTNAME> o.a.s.z.ZooKeeper INFO
Initiating client connection, connectString=<WORKER1
IP>:2181,185.27.173.47:2181,<WORKER3 IP>:2181 sessionTimeout=20000
watcher=org.apache.storm.curator.ConnectionState@62c97092
2014-11-13T08:20:57,349+01:00 <WORKER1 HOSTNAME> o.a.s.z.ClientCnxn INFO
Opening socket connection to server <WORKER3 IP>/<WORKER3 IP>:2181. Will
not attempt to authenticate using SASL (unknown error)
2014-11-13T08:20:57,356+01:00 <WORKER1 HOSTNAME> o.a.s.z.ClientCnxn INFO
Socket connection established to <WORKER3 IP>/<WORKER3 IP>:2181,
initiating session
2014-11-13T08:20:57,406+01:00 <WORKER1 HOSTNAME> o.a.s.z.ClientCnxn INFO
Session establishment complete on server <WORKER3 IP>/<WORKER3 IP>:2181,
sessionid = 0x349a7ffa2c50018, negotiated timeout = 20000
2014-11-13T08:20:57,409+01:00 <WORKER1 HOSTNAME>
o.a.s.c.f.s.ConnectionStateManager INFO State change: CONNECTED
2014-11-13T08:20:57,412+01:00 <WORKER1 HOSTNAME> b.s.zookeeper INFO
Zookeeper state update: :connected:none
2014-11-13T08:20:58,450+01:00 <WORKER1 HOSTNAME> o.a.s.z.ZooKeeper INFO
Session: 0x349a7ffa2c50018 closed
2014-11-13T08:20:58,450+01:00 <WORKER1 HOSTNAME> o.a.s.z.ClientCnxn INFO
EventThread shut down
2014-11-13T08:20:58,452+01:00 <WORKER1 HOSTNAME>
b.s.u.StormBoundedExponentialBackoffRetry INFO The baseSleepTimeMs
[1000] the maxSleepTimeMs [30000] the maxRetries [5]
2014-11-13T08:20:58,453+01:00 <WORKER1 HOSTNAME>
o.a.s.c.f.i.CuratorFrameworkImpl INFO Starting
2014-11-13T08:20:57,328+01:00 <WORKER1 HOSTNAME> o.a.s.z.ZooKeeper INFO
Initiating client connection, connectString=<WORKER1
IP>:2181,185.27.173.47:2181,<WORKER3 IP>:2181 sessionTimeout=20000
watcher=org.apache.storm.curator.ConnectionState@62c97092
2014-11-13T08:20:57,349+01:00 <WORKER1 HOSTNAME> o.a.s.z.ClientCnxn INFO
Opening socket connection to server <WORKER3 IP>/<WORKER3 IP>:2181. Will
not attempt to authenticate using SASL (unknown error)
2014-11-13T08:20:57,356+01:00 <WORKER1 HOSTNAME> o.a.s.z.ClientCnxn INFO
Socket connection established to <WORKER3 IP>/<WORKER3 IP>:2181,
initiating session
2014-11-13T08:20:57,406+01:00 <WORKER1 HOSTNAME> o.a.s.z.ClientCnxn INFO
Session establishment complete on server <WORKER3 IP>/<WORKER3 IP>:2181,
sessionid = 0x349a7ffa2c50018, negotiated timeout = 20000
2014-11-13T08:20:57,409+01:00 <WORKER1 HOSTNAME>
o.a.s.c.f.s.ConnectionStateManager INFO State change: CONNECTED
2014-11-13T08:20:57,412+01:00 <WORKER1 HOSTNAME> b.s.zookeeper INFO
Zookeeper state update: :connected:none
2014-11-13T08:20:58,450+01:00 <WORKER1 HOSTNAME> o.a.s.z.ZooKeeper INFO
Session: 0x349a7ffa2c50018 closed
2014-11-13T08:20:58,450+01:00 <WORKER1 HOSTNAME> o.a.s.z.ClientCnxn INFO
EventThread shut down
2014-11-13T08:20:58,452+01:00 <WORKER1 HOSTNAME>
b.s.u.StormBoundedExponentialBackoffRetry INFO The baseSleepTimeMs
[1000] the maxSleepTimeMs [30000] the maxRetries [5]
2014-11-13T08:20:58,453+01:00 <WORKER1 HOSTNAME>
o.a.s.c.f.i.CuratorFrameworkImpl INFO Starting
2014-11-13T08:20:58,453+01:00 <WORKER1 HOSTNAME> o.a.s.z.ZooKeeper INFO
Initiating client connection, connectString=<WORKER1
IP>:2181,185.27.173.47:2181,<WORKER3 IP>:2181/storm sessionTimeout=20000
watcher=org.apache.storm.curator.ConnectionState@62b51eb3
2014-11-13T08:20:58,454+01:00 <WORKER1 HOSTNAME> o.a.s.z.ClientCnxn INFO
Opening socket connection to server <WORKER1 IP>/<WORKER1 IP>:2181. Will
not attempt to authenticate using SASL (unknown error)
2014-11-13T08:20:58,455+01:00 <WORKER1 HOSTNAME> o.a.s.z.ClientCnxn INFO
Socket connection established to <WORKER1 IP>/<WORKER1 IP>:2181,
initiating session
2014-11-13T08:20:58,475+01:00 <WORKER1 HOSTNAME> o.a.s.z.ClientCnxn INFO
Session establishment complete on server <WORKER1 IP>/<WORKER1 IP>:2181,
sessionid = 0x149a7ffaa000014, negotiated timeout = 20000
2014-11-13T08:20:58,476+01:00 <WORKER1 HOSTNAME>
o.a.s.c.f.s.ConnectionStateManager INFO State change: CONNECTED
2014-11-13T08:20:58,589+01:00 <WORKER1 HOSTNAME> b.s.d.supervisor INFO
Starting supervisor with id 051ef04a-445e-402f-8010-f49519d363b0 at host
<WORKER1 HOSTNAME>
2014-11-13T08:20:59,358+01:00 <WORKER1 HOSTNAME> b.s.d.supervisor INFO
Shutting down and clearing state for id
79864941-ebae-4624-80bf-f53239440655. Current supervisor time:
1415863259. State: :disallowed, Heartbeat:
#backtype.storm.daemon.common.WorkerHeartbeat{:time-secs 1415863257,
:storm-id "domain-check-1-1415862954", :executors #{[2 2] [8 8] [14 14]
[-1 -1]}, :port 6708}
2014-11-13T08:20:59,358+01:00 <WORKER1 HOSTNAME> b.s.d.supervisor INFO
Shutting down
051ef04a-445e-402f-8010-f49519d363b0:79864941-ebae-4624-80bf-f53239440655
2014-11-13T08:20:59,370+01:00 <WORKER1 HOSTNAME> b.s.event ERROR Error
when processing event
java.io.IOException: . doesn't exist.
at
org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:157) ~[commons-exec-1.1.jar:1.1]
at
org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:147) ~[commons-exec-1.1.jar:1.1]
at backtype.storm.util$exec_command_BANG_.invoke(util.clj:380)
~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at
backtype.storm.util$send_signal_to_process.invoke(util.clj:409)
~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at
backtype.storm.util$kill_process_with_sig_term.invoke(util.clj:420)
~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at
backtype.storm.daemon.supervisor$shutdown_worker.invoke(supervisor.clj:194)
~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at
backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:263)
~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at clojure.lang.AFn.applyToHelper(AFn.java:161)
[clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na]
at clojure.core$apply.invoke(core.clj:619) ~[clojure-1.5.1.jar:na]
at clojure.core$partial$fn__4190.doInvoke(core.clj:2396)
~[clojure-1.5.1.jar:na]
at clojure.lang.RestFn.invoke(RestFn.java:397)
~[clojure-1.5.1.jar:na]
at
backtype.storm.event$event_manager$fn__755.invoke(event.clj:40)
~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
2014-11-13T08:20:59,381+01:00 <WORKER1 HOSTNAME> b.s.util ERROR Halting
process: ("Error when processing an event")
java.lang.RuntimeException: ("Error when processing an event")
at
backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:319)
[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at clojure.lang.RestFn.invoke(RestFn.java:423)
[clojure-1.5.1.jar:na]
at
backtype.storm.event$event_manager$fn__755.invoke(event.clj:48)
[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
2014-11-13T08:20:59,384+01:00 <WORKER1 HOSTNAME> b.s.d.supervisor INFO
Shutting down supervisor 051ef04a-445e-402f-8010-f49519d363b0
2014-11-13T08:20:59,386+01:00 <WORKER1 HOSTNAME> b.s.event INFO Event
manager interrupted
Thank you,
Oleg Tokarev
