[ 
https://issues.apache.org/jira/browse/STORM-3038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460835#comment-16460835
 ] 

Simon Cooper commented on STORM-3038:
-------------------------------------

In the logs, there's lots of these sort of log files:

{code}392826 [ProcessThread(sid:0 cport:-1):] INFO  
o.a.s.s.o.a.z.s.PrepRequestProcessor - Got user-level KeeperException when 
processing sessionid:0x163203f116c002f type:delete cxid:0x1 zxid:0x92 
txntype:-1 reqpath:n/a Error 
Path:/storm/leader-lock/_c_8167ff79-975e-4a40-8b41-eb484e1e931d-latch-0000000000
 Error:KeeperErrorCode = NoNode for 
/storm/leader-lock/_c_8167ff79-975e-4a40-8b41-eb484e1e931d-latch-0000000000{code}

However, after every retry (every 10 seconds), this is logged:

{code}411061 [timer] INFO  o.a.s.d.nimbus - not a leader, skipping assignments
411061 [timer] INFO  o.a.s.d.nimbus - not a leader, skipping cleanup{code}

It looks like nimbus isn't correctly recreating the leader latch zookeeper 
nodes when ZK connection is lost.

> 'not a leader' exception submitting topologies to LocalCluster
> --------------------------------------------------------------
>
>                 Key: STORM-3038
>                 URL: https://issues.apache.org/jira/browse/STORM-3038
>             Project: Apache Storm
>          Issue Type: Bug
>    Affects Versions: 1.2.1
>            Reporter: Simon Cooper
>            Priority: Major
>
> In our local tests running on our openstack build servers, we get 
> intermittent failures due to the following exception being thrown when 
> submitting topologies to LocalCluster:
> {code}275354 [main] WARN  o.a.s.d.nimbus - Topology submission exception. 
> (topology name='AnalyseUpdate') #error {
>  :cause not a leader, current leader is NimbusInfo{host='<hostname>', 
> port=6627, isLeader=true}
>  :via
>  [{:type java.lang.RuntimeException
>    :message not a leader, current leader is NimbusInfo{host='<hostname>', 
> port=6627, isLeader=true}
>    :at [org.apache.storm.daemon.nimbus$is_leader doInvoke nimbus.clj 150]}]
>  :trace
>  [[org.apache.storm.daemon.nimbus$is_leader doInvoke nimbus.clj 150]
>   [clojure.lang.RestFn invoke RestFn.java 410]
>   [org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__10799 
> submitTopologyWithOpts nimbus.clj 1681]
>   [org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__10799 
> submitTopology nimbus.clj 1774]
>   [sun.reflect.GeneratedMethodAccessor299 invoke nil -1]
>   [sun.reflect.DelegatingMethodAccessorImpl invoke 
> DelegatingMethodAccessorImpl.java 43]
>   [java.lang.reflect.Method invoke Method.java 498]
>   [clojure.lang.Reflector invokeMatchingMethod Reflector.java 93]
>   [clojure.lang.Reflector invokeInstanceMethod Reflector.java 28]
>   [org.apache.storm.testing$submit_local_topology invoke testing.clj 310]
>   [org.apache.storm.LocalCluster$_submitTopology invoke LocalCluster.clj 49]
>   [org.apache.storm.LocalCluster submitTopology nil -1]{code}
> (note that {{isLeader=true}} in the exception message)
> This is despite a retry mechanism we've implemented; this seems to be a 
> continual failure, once its hit. This tends to be surrounded by lots of 
> zookeeper connection lost/reconnection log message.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to