[
https://issues.apache.org/jira/browse/CURATOR-165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209093#comment-14209093
]
zhaogang edited comment on CURATOR-165 at 11/13/14 2:06 AM:
------------------------------------------------------------
This error occured at the time when the client lost connection and reconnected
to the zookeeper again, here is detail logs:
sing socket connection and attempting reconnect
INFO | jvm 1 | 2014/10/30 00:24:38 | 30 Oct 2014 00:24:38 INFO
org.apache.curator.framework.state.ConnectionStateManager - State change:
SUSPENDED
INFO | jvm 1 | 2014/10/30 00:24:38 | 30 Oct 2014 00:24:38 INFO
org.apache.curator.framework.state.ConnectionStateManager - State change:
SUSPENDED
INFO | jvm 1 | 2014/10/30 00:24:39 | 30 Oct 2014 00:24:39 INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
10.67.90.141/10.67.90.141:2181. Will not attempt to authenticate using SASL
(unknown error)
INFO | jvm 1 | 2014/10/30 00:24:39 | 30 Oct 2014 00:24:39 WARN
org.apache.zookeeper.ClientCnxn - Session 0x1489e3e9cce0011 for server null,
unexpected error, closing socket connection and attempting reconnect
INFO | jvm 1 | 2014/10/30 00:24:39 | java.net.ConnectException: Connection
refused
INFO | jvm 1 | 2014/10/30 00:24:39 | at
sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
INFO | jvm 1 | 2014/10/30 00:24:39 | at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
INFO | jvm 1 | 2014/10/30 00:24:39 | at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
INFO | jvm 1 | 2014/10/30 00:24:39 | at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
INFO | jvm 1 | 2014/10/30 00:24:39 | 30 Oct 2014 00:24:39 INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
10.67.90.141/10.67.90.141:2181. Will not attempt to authenticate using SASL
(unknown error)
INFO | jvm 1 | 2014/10/30 00:24:39 | 30 Oct 2014 00:24:39 WARN
org.apache.zookeeper.ClientCnxn - Session 0x2487088d7080011 for server null,
unexpected error, closing socket connection and attempting reconnect
INFO | jvm 1 | 2014/10/30 00:24:39 | java.net.ConnectException: Connection
refused
INFO | jvm 1 | 2014/10/30 00:24:39 | at
sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
INFO | jvm 1 | 2014/10/30 00:24:39 | at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
INFO | jvm 1 | 2014/10/30 00:24:39 | at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
INFO | jvm 1 | 2014/10/30 00:24:39 | at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
INFO | jvm 1 | 2014/10/30 00:24:40 | 30 Oct 2014 00:24:40 INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
10.67.90.142/10.67.90.142:2181. Will not attempt to authenticate using SASL
(unknown error)
INFO | jvm 1 | 2014/10/30 00:24:40 | 30 Oct 2014 00:24:40 INFO
org.apache.zookeeper.ClientCnxn - Socket connection established to
10.67.90.142/10.67.90.142:2181, initiating session
INFO | jvm 1 | 2014/10/30 00:24:40 | 30 Oct 2014 00:24:40 INFO
org.apache.zookeeper.ClientCnxn - Session establishment complete on server
10.67.90.142/10.67.90.142:2181, sessionid = 0x1489e3e9cce0011, negotiated
timeout = 40000
INFO | jvm 1 | 2014/10/30 00:24:40 | 30 Oct 2014 00:24:40 INFO
org.apache.curator.framework.state.ConnectionStateManager - State change:
RECONNECTED
INFO | jvm 1 | 2014/10/30 00:24:40 | 30 Oct 2014 00:24:40 INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
10.67.90.142/10.67.90.142:2181. Will not attempt to authenticate using SASL
(unknown error)
INFO | jvm 1 | 2014/10/30 00:24:40 | 30 Oct 2014 00:24:40 INFO
org.apache.zookeeper.ClientCnxn - Socket connection established to
10.67.90.142/10.67.90.142:2181, initiating session
INFO | jvm 1 | 2014/10/30 00:24:40 | 30 Oct 2014 00:24:40 INFO
org.apache.zookeeper.ClientCnxn - Session establishment complete on server
10.67.90.142/10.67.90.142:2181, sessionid = 0x2487088d7080011, negotiated
timeout = 40000
INFO | jvm 1 | 2014/10/30 00:24:40 | 30 Oct 2014 00:24:40 INFO
org.apache.curator.framework.state.ConnectionStateManager - State change:
RECONNECTED
DEBUG | wrapperp | 2014/10/30 00:24:42 | send a packet PING : ping 00fc2c4f
INFO | jvm 1 | 2014/10/30 00:24:42 | WrapperManager Debug: Received a
packet PING : ping 00fc2c4f
INFO | jvm 1 | 2014/10/30 00:24:42 | WrapperManager Debug: Send a packet
PING : ping 00fc2c4f
DEBUG | wrapperp | 2014/10/30 00:24:42 | read a packet PING : ping 00fc2c4f
DEBUG | wrapperp | 2014/10/30 00:24:46 | send a packet PING : ping 00fc2c78
INFO | jvm 1 | 2014/10/30 00:24:46 | WrapperManager Debug: Received a
packet PING : ping 00fc2c78
INFO | jvm 1 | 2014/10/30 00:24:46 | WrapperManager Debug: Send a packet
PING : ping 00fc2c78
DEBUG | wrapperp | 2014/10/30 00:24:46 | read a packet PING : ping 00fc2c78
INFO | jvm 1 | 2014/10/30 00:24:49 | 30 Oct 2014 00:24:49 ERROR
org.apache.curator.framework.imps.CuratorFrameworkImpl - Background exception
was not retry-able or retry gave up
INFO | jvm 1 | 2014/10/30 00:24:49 | java.lang.IllegalArgumentException:
Path must start with / character
INFO | jvm 1 | 2014/10/30 00:24:49 | at
org.apache.curator.utils.PathUtils.validatePath(PathUtils.java:53)
INFO | jvm 1 | 2014/10/30 00:24:49 | at
org.apache.curator.utils.ZKPaths.getNodeFromPath(ZKPaths.java:56)
INFO | jvm 1 | 2014/10/30 00:24:49 | at
org.apache.curator.framework.recipes.leader.LeaderLatch.checkLeadership(LeaderLatch.java:478)
INFO | jvm 1 | 2014/10/30 00:24:49 | at
org.apache.curator.framework.recipes.leader.LeaderLatch.access$500(LeaderLatch.java:60)
INFO | jvm 1 | 2014/10/30 00:24:49 | at
org.apache.curator.framework.recipes.leader.LeaderLatch$6.processResult(LeaderLatch.java:536)
INFO | jvm 1 | 2014/10/30 00:24:49 | at
org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:730)
INFO | jvm 1 | 2014/10/30 00:24:49 | at
org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:516)
INFO | jvm 1 | 2014/10/30 00:24:49 | at
org.apache.curator.framework.imps.GetChildrenBuilderImpl$2.processResult(GetChildrenBuilderImpl.java:166)
INFO | jvm 1 | 2014/10/30 00:24:49 | at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:590)
INFO | jvm 1 | 2014/10/30 00:24:49 | at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
DEBUG | wrapperp | 2014/10/30 00:24:50 | send a packet PING : ping 00fc2ca1
INFO | jvm 1 | 2014/10/30 00:24:50 | WrapperManager Debug: Received a
packet PING : ping 00fc2ca1
INFO | jvm 1 | 2014/10/30 00:24:50 | WrapperManager Debug: Send a packet
PING : ping 00fc2ca1
DEBUG | wrapperp | 2014/10/30 00:24:50 | read a packet PING : ping 00fc2ca1
DEBUG | wrapperp | 2014/10/30 00:24:54 | send a packet PING : ping 00fc2cca
INFO | jvm 1 | 2014/10/30 00:24:54 | WrapperManager Debug: Received a
packet PING : ping 00fc2cca
INFO | jvm 1 | 2014/10/30 00:24:54 | WrapperManager Debug: Send a packet
PING : ping 00fc2cca
DEBUG | wrapperp | 2014/10/30 00:24:54 | read a packet PING : ping 00fc2cca
DEBUG | wrapperp | 2014/10/30 00:24:58 | send a packet PING : ping 00fc2cf3
INFO | jvm 1 | 2014/10/30 00:24:59 | WrapperManager Debug: Received a
packet PING : ping 00fc2cf3
INFO | jvm 1 | 2014/10/30 00:24:59 | WrapperManager Debug: Send a packet
PING : ping 00fc2cf3
DEBUG | wrapperp | 2014/10/30 00:24:59 | read a packet PING : ping 00fc2cf3
DEBUG | wrapperp | 2014/10/30 00:25:03 | send a packet PING : ping 00fc2d1c
INFO | jvm 1 | 2014/10/30 00:25:03 | WrapperManager Debug: Received a
packet PING : ping 00fc2d1c
INFO | jvm 1 | 2014/10/30 00:25:03 | WrapperManager Debug: Send a packet
PING : ping 00fc2d1c
DEBUG | wrapperp | 2014/10/30 00:25:03 | read a packet PING : ping 00fc2d1c
DEBUG | wrapperp | 2014/10/30 00:25:07 | send a packet PING : ping 00fc2d45
INFO | jvm 1 | 2014/10/30 00:25:07 | WrapperManager Debug: Received a
packet PING : ping 00fc2d45
INFO | jvm 1 | 2014/10/30 00:25:07 | WrapperManager Debug: Send a packet
PING : ping 00fc2d45
was (Author: zhaogang):
This error occured at the time that the when the client lost connection and
reconnected to the zookeeper again, here is detail logs:
sing socket connection and attempting reconnect
INFO | jvm 1 | 2014/10/30 00:24:38 | 30 Oct 2014 00:24:38 INFO
org.apache.curator.framework.state.ConnectionStateManager - State change:
SUSPENDED
INFO | jvm 1 | 2014/10/30 00:24:38 | 30 Oct 2014 00:24:38 INFO
org.apache.curator.framework.state.ConnectionStateManager - State change:
SUSPENDED
INFO | jvm 1 | 2014/10/30 00:24:39 | 30 Oct 2014 00:24:39 INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
10.67.90.141/10.67.90.141:2181. Will not attempt to authenticate using SASL
(unknown error)
INFO | jvm 1 | 2014/10/30 00:24:39 | 30 Oct 2014 00:24:39 WARN
org.apache.zookeeper.ClientCnxn - Session 0x1489e3e9cce0011 for server null,
unexpected error, closing socket connection and attempting reconnect
INFO | jvm 1 | 2014/10/30 00:24:39 | java.net.ConnectException: Connection
refused
INFO | jvm 1 | 2014/10/30 00:24:39 | at
sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
INFO | jvm 1 | 2014/10/30 00:24:39 | at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
INFO | jvm 1 | 2014/10/30 00:24:39 | at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
INFO | jvm 1 | 2014/10/30 00:24:39 | at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
INFO | jvm 1 | 2014/10/30 00:24:39 | 30 Oct 2014 00:24:39 INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
10.67.90.141/10.67.90.141:2181. Will not attempt to authenticate using SASL
(unknown error)
INFO | jvm 1 | 2014/10/30 00:24:39 | 30 Oct 2014 00:24:39 WARN
org.apache.zookeeper.ClientCnxn - Session 0x2487088d7080011 for server null,
unexpected error, closing socket connection and attempting reconnect
INFO | jvm 1 | 2014/10/30 00:24:39 | java.net.ConnectException: Connection
refused
INFO | jvm 1 | 2014/10/30 00:24:39 | at
sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
INFO | jvm 1 | 2014/10/30 00:24:39 | at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
INFO | jvm 1 | 2014/10/30 00:24:39 | at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
INFO | jvm 1 | 2014/10/30 00:24:39 | at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
INFO | jvm 1 | 2014/10/30 00:24:40 | 30 Oct 2014 00:24:40 INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
10.67.90.142/10.67.90.142:2181. Will not attempt to authenticate using SASL
(unknown error)
INFO | jvm 1 | 2014/10/30 00:24:40 | 30 Oct 2014 00:24:40 INFO
org.apache.zookeeper.ClientCnxn - Socket connection established to
10.67.90.142/10.67.90.142:2181, initiating session
INFO | jvm 1 | 2014/10/30 00:24:40 | 30 Oct 2014 00:24:40 INFO
org.apache.zookeeper.ClientCnxn - Session establishment complete on server
10.67.90.142/10.67.90.142:2181, sessionid = 0x1489e3e9cce0011, negotiated
timeout = 40000
INFO | jvm 1 | 2014/10/30 00:24:40 | 30 Oct 2014 00:24:40 INFO
org.apache.curator.framework.state.ConnectionStateManager - State change:
RECONNECTED
INFO | jvm 1 | 2014/10/30 00:24:40 | 30 Oct 2014 00:24:40 INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
10.67.90.142/10.67.90.142:2181. Will not attempt to authenticate using SASL
(unknown error)
INFO | jvm 1 | 2014/10/30 00:24:40 | 30 Oct 2014 00:24:40 INFO
org.apache.zookeeper.ClientCnxn - Socket connection established to
10.67.90.142/10.67.90.142:2181, initiating session
INFO | jvm 1 | 2014/10/30 00:24:40 | 30 Oct 2014 00:24:40 INFO
org.apache.zookeeper.ClientCnxn - Session establishment complete on server
10.67.90.142/10.67.90.142:2181, sessionid = 0x2487088d7080011, negotiated
timeout = 40000
INFO | jvm 1 | 2014/10/30 00:24:40 | 30 Oct 2014 00:24:40 INFO
org.apache.curator.framework.state.ConnectionStateManager - State change:
RECONNECTED
DEBUG | wrapperp | 2014/10/30 00:24:42 | send a packet PING : ping 00fc2c4f
INFO | jvm 1 | 2014/10/30 00:24:42 | WrapperManager Debug: Received a
packet PING : ping 00fc2c4f
INFO | jvm 1 | 2014/10/30 00:24:42 | WrapperManager Debug: Send a packet
PING : ping 00fc2c4f
DEBUG | wrapperp | 2014/10/30 00:24:42 | read a packet PING : ping 00fc2c4f
DEBUG | wrapperp | 2014/10/30 00:24:46 | send a packet PING : ping 00fc2c78
INFO | jvm 1 | 2014/10/30 00:24:46 | WrapperManager Debug: Received a
packet PING : ping 00fc2c78
INFO | jvm 1 | 2014/10/30 00:24:46 | WrapperManager Debug: Send a packet
PING : ping 00fc2c78
DEBUG | wrapperp | 2014/10/30 00:24:46 | read a packet PING : ping 00fc2c78
INFO | jvm 1 | 2014/10/30 00:24:49 | 30 Oct 2014 00:24:49 ERROR
org.apache.curator.framework.imps.CuratorFrameworkImpl - Background exception
was not retry-able or retry gave up
INFO | jvm 1 | 2014/10/30 00:24:49 | java.lang.IllegalArgumentException:
Path must start with / character
INFO | jvm 1 | 2014/10/30 00:24:49 | at
org.apache.curator.utils.PathUtils.validatePath(PathUtils.java:53)
INFO | jvm 1 | 2014/10/30 00:24:49 | at
org.apache.curator.utils.ZKPaths.getNodeFromPath(ZKPaths.java:56)
INFO | jvm 1 | 2014/10/30 00:24:49 | at
org.apache.curator.framework.recipes.leader.LeaderLatch.checkLeadership(LeaderLatch.java:478)
INFO | jvm 1 | 2014/10/30 00:24:49 | at
org.apache.curator.framework.recipes.leader.LeaderLatch.access$500(LeaderLatch.java:60)
INFO | jvm 1 | 2014/10/30 00:24:49 | at
org.apache.curator.framework.recipes.leader.LeaderLatch$6.processResult(LeaderLatch.java:536)
INFO | jvm 1 | 2014/10/30 00:24:49 | at
org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:730)
INFO | jvm 1 | 2014/10/30 00:24:49 | at
org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:516)
INFO | jvm 1 | 2014/10/30 00:24:49 | at
org.apache.curator.framework.imps.GetChildrenBuilderImpl$2.processResult(GetChildrenBuilderImpl.java:166)
INFO | jvm 1 | 2014/10/30 00:24:49 | at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:590)
INFO | jvm 1 | 2014/10/30 00:24:49 | at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
DEBUG | wrapperp | 2014/10/30 00:24:50 | send a packet PING : ping 00fc2ca1
INFO | jvm 1 | 2014/10/30 00:24:50 | WrapperManager Debug: Received a
packet PING : ping 00fc2ca1
INFO | jvm 1 | 2014/10/30 00:24:50 | WrapperManager Debug: Send a packet
PING : ping 00fc2ca1
DEBUG | wrapperp | 2014/10/30 00:24:50 | read a packet PING : ping 00fc2ca1
DEBUG | wrapperp | 2014/10/30 00:24:54 | send a packet PING : ping 00fc2cca
INFO | jvm 1 | 2014/10/30 00:24:54 | WrapperManager Debug: Received a
packet PING : ping 00fc2cca
INFO | jvm 1 | 2014/10/30 00:24:54 | WrapperManager Debug: Send a packet
PING : ping 00fc2cca
DEBUG | wrapperp | 2014/10/30 00:24:54 | read a packet PING : ping 00fc2cca
DEBUG | wrapperp | 2014/10/30 00:24:58 | send a packet PING : ping 00fc2cf3
INFO | jvm 1 | 2014/10/30 00:24:59 | WrapperManager Debug: Received a
packet PING : ping 00fc2cf3
INFO | jvm 1 | 2014/10/30 00:24:59 | WrapperManager Debug: Send a packet
PING : ping 00fc2cf3
DEBUG | wrapperp | 2014/10/30 00:24:59 | read a packet PING : ping 00fc2cf3
DEBUG | wrapperp | 2014/10/30 00:25:03 | send a packet PING : ping 00fc2d1c
INFO | jvm 1 | 2014/10/30 00:25:03 | WrapperManager Debug: Received a
packet PING : ping 00fc2d1c
INFO | jvm 1 | 2014/10/30 00:25:03 | WrapperManager Debug: Send a packet
PING : ping 00fc2d1c
DEBUG | wrapperp | 2014/10/30 00:25:03 | read a packet PING : ping 00fc2d1c
DEBUG | wrapperp | 2014/10/30 00:25:07 | send a packet PING : ping 00fc2d45
INFO | jvm 1 | 2014/10/30 00:25:07 | WrapperManager Debug: Received a
packet PING : ping 00fc2d45
INFO | jvm 1 | 2014/10/30 00:25:07 | WrapperManager Debug: Send a packet
PING : ping 00fc2d45
> LeaderLatch.checkLeadership(LeaderLatch.java:478) will throw a exception,
> then no leader will be elected.
> ---------------------------------------------------------------------------------------------------------
>
> Key: CURATOR-165
> URL: https://issues.apache.org/jira/browse/CURATOR-165
> Project: Apache Curator
> Issue Type: Bug
> Components: Recipes
> Affects Versions: 2.5.0
> Reporter: zhaogang
> Priority: Critical
>
> We use curator for leader election within our 2 clients, sometimes none of
> the 2 clients was a leader.
> We checked the log and found this caused by a exception which was being
> throwed in method LeaderLatch.checkLeadership(), after throwing this
> exception the curator won't do the leader election. Here is error log:
> ERROR org.apache.curator.framework.imps.CuratorFrameworkImpl - Background
> exception was not retry-able or retry gave up
> INFO | jvm 1 | 2014/11/12 00:36:52 | java.lang.IllegalArgumentException:
> Path must start with / character
> INFO | jvm 1 | 2014/11/12 00:36:52 | at
> org.apache.curator.utils.PathUtils.validatePath(PathUtils.java:53)
> INFO | jvm 1 | 2014/11/12 00:36:52 | at
> org.apache.curator.utils.ZKPaths.getNodeFromPath(ZKPaths.java:56)
> INFO | jvm 1 | 2014/11/12 00:36:52 | at
> org.apache.curator.framework.recipes.leader.LeaderLatch.checkLeadership(LeaderLatch.java:478)
> INFO | jvm 1 | 2014/11/12 00:36:52 | at
> org.apache.curator.framework.recipes.leader.LeaderLatch.access$500(LeaderLatch.java:60)
> INFO | jvm 1 | 2014/11/12 00:36:52 | at
> org.apache.curator.framework.recipes.leader.LeaderLatch$6.processResult(LeaderLatch.java:536)
> INFO | jvm 1 | 2014/11/12 00:36:52 | at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:730)
> INFO | jvm 1 | 2014/11/12 00:36:52 | at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:516)
> INFO | jvm 1 | 2014/11/12 00:36:52 | at
> org.apache.curator.framework.imps.GetChildrenBuilderImpl$2.processResult(GetChildrenBuilderImpl.java:166)
> INFO | jvm 1 | 2014/11/12 00:36:52 | at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:590)
> INFO | jvm 1 | 2014/11/12 00:36:52 | at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
