[
https://issues.apache.org/jira/browse/STORM-3038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460835#comment-16460835
]
Simon Cooper commented on STORM-3038:
-------------------------------------
In the logs, there's lots of these sort of log files:
{code}392826 [ProcessThread(sid:0 cport:-1):] INFO
o.a.s.s.o.a.z.s.PrepRequestProcessor - Got user-level KeeperException when
processing sessionid:0x163203f116c002f type:delete cxid:0x1 zxid:0x92
txntype:-1 reqpath:n/a Error
Path:/storm/leader-lock/_c_8167ff79-975e-4a40-8b41-eb484e1e931d-latch-0000000000
Error:KeeperErrorCode = NoNode for
/storm/leader-lock/_c_8167ff79-975e-4a40-8b41-eb484e1e931d-latch-0000000000{code}
However, after every retry (every 10 seconds), this is logged:
{code}411061 [timer] INFO o.a.s.d.nimbus - not a leader, skipping assignments
411061 [timer] INFO o.a.s.d.nimbus - not a leader, skipping cleanup{code}
It looks like nimbus isn't correctly recreating the leader latch zookeeper
nodes when ZK connection is lost.
> 'not a leader' exception submitting topologies to LocalCluster
> --------------------------------------------------------------
>
> Key: STORM-3038
> URL: https://issues.apache.org/jira/browse/STORM-3038
> Project: Apache Storm
> Issue Type: Bug
> Affects Versions: 1.2.1
> Reporter: Simon Cooper
> Priority: Major
>
> In our local tests running on our openstack build servers, we get
> intermittent failures due to the following exception being thrown when
> submitting topologies to LocalCluster:
> {code}275354 [main] WARN o.a.s.d.nimbus - Topology submission exception.
> (topology name='AnalyseUpdate') #error {
> :cause not a leader, current leader is NimbusInfo{host='<hostname>',
> port=6627, isLeader=true}
> :via
> [{:type java.lang.RuntimeException
> :message not a leader, current leader is NimbusInfo{host='<hostname>',
> port=6627, isLeader=true}
> :at [org.apache.storm.daemon.nimbus$is_leader doInvoke nimbus.clj 150]}]
> :trace
> [[org.apache.storm.daemon.nimbus$is_leader doInvoke nimbus.clj 150]
> [clojure.lang.RestFn invoke RestFn.java 410]
> [org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__10799
> submitTopologyWithOpts nimbus.clj 1681]
> [org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__10799
> submitTopology nimbus.clj 1774]
> [sun.reflect.GeneratedMethodAccessor299 invoke nil -1]
> [sun.reflect.DelegatingMethodAccessorImpl invoke
> DelegatingMethodAccessorImpl.java 43]
> [java.lang.reflect.Method invoke Method.java 498]
> [clojure.lang.Reflector invokeMatchingMethod Reflector.java 93]
> [clojure.lang.Reflector invokeInstanceMethod Reflector.java 28]
> [org.apache.storm.testing$submit_local_topology invoke testing.clj 310]
> [org.apache.storm.LocalCluster$_submitTopology invoke LocalCluster.clj 49]
> [org.apache.storm.LocalCluster submitTopology nil -1]{code}
> (note that {{isLeader=true}} in the exception message)
> This is despite a retry mechanism we've implemented; this seems to be a
> continual failure, once its hit. This tends to be surrounded by lots of
> zookeeper connection lost/reconnection log message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)