[
https://issues.apache.org/jira/browse/ZOOKEEPER-790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1234#action_1234
]
Flavio Paiva Junqueira commented on ZOOKEEPER-790:
--
Thank you both for all the information. I have been able to reproduce and find
the source of the bug, but I don't have a patch yet. The problem is deeper than
I thought originally. Let me show you what is going on:
I'm including an excerpt of logs from two runs: one good and one bad. The first
run is good (see the excerpt below). I have killed the follower and restarted
it as Vishal suggested. When it comes back, it declares itself as leader, also
as Vishal and Travis observed. However, different from what Vishal and Travis
observed, it drops leadership and follows happily the leader right after.
{noformat}
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:quorump...@654] - LEADING
INFO - [WorkerReceiver Thread:fastleaderelect...@496] - Notification: 3
(n.leader), 0 (n.zxid), 1 (n.round), LEADING (n.state), 3 (n.sid), LOOKING (my
state)
INFO - [WorkerReceiver Thread:fastleaderelect...@496] - Notification: 3
(n.leader), 0 (n.zxid), 1 (n.round), LEADING (n.state), 3 (n.sid), LEADING (my
state)
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:lea...@54] - TCP NoDelay set to: true
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:environm...@97] - Server
environment:zookeeper.version=3.4.0--1, built on 07/15/2010 10:36 GMT
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:environm...@97] - Server
environment:host.name=XX.com
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:environm...@97] - Server
environment:java.version=1.6.0_04
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:environm...@97] - Server
environment:java.vendor=Sun Microsystems Inc.
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:environm...@97] - Server
environment:java.home=/usr/java/jdk1.6.0_04/jre
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:environm...@97] - Server
environment:java.class.path=.X
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:environm...@97] - Server
environment:java.library.path= X
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:environm...@97] - Server
environment:java.io.tmpdir=/tmp
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:environm...@97] - Server
environment:java.compiler=
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:environm...@97] - Server
environment:os.name=Linux
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:environm...@97] - Server
environment:os.arch=amd64
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:environm...@97] - Server
environment:os.version=2.6.18-53.1.21.el5
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:environm...@97] - Server
environment:user.name=X
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:environm...@97] - Server
environment:user.home=X
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:environm...@97] - Server
environment:user.dir=X
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:zookeeperser...@151] - Created
server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 6
datadir /X/zookeeper/version-2 snapdir //zookeeper/version-2
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:files...@82] - Reading snapshot
//zookeeper/version-2/snapshot.100113340
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:files...@82] - Reading snapshot
//zookeeper/version-2/snapshot.100113340
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:filetxnsnap...@208] - Snapshotting:
10011f748
INFO - [SessionTracker:zookeeperser...@315] - Expiring session
0x229d6a9e0ca, timeout of 1ms exceeded
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:lea...@394] - Shutdown called
java.lang.Exception: shutdown Leader! reason: Waiting for a quorum of
followers, only synced with: 2:
at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:394)
at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:317)
at
org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:657)
INFO - [Thread-10:leader$learnercnxaccep...@243] - exception while shutting
down acceptor: java.net.SocketException: Socket closed
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:finalrequestproces...@378] -
shutdown of request processor complete
INFO - [SyncThread:2:syncrequestproces...@151] - SyncRequestProcessor exited!
INFO - [CommitProcessor:2:commitproces...@148] - CommitProcessor exited loop!
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:quorump...@620] - LOOKING
INFO - [QuorumPeer:/0:0:0:0:0:0:0:0:10218:files...@82] - Reading snapshot
/X/zookeeper/version-2/snapshot.10011f748
INFO - [SessionTracker:sessiontrackeri...@158] - SessionTrackerImpl exited
loop!
INFO - [ProcessThread:-1:preprequestproces...@385] - Processed session
termination for sessionid: 0x229d6a9e0ca
ERROR - [ProcessThread:-1:nioservercnxn$factor...@87] - Thread
Thread[ProcessThread:-1,5,main] died
java.lang.NullPointerException