[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043017#comment-13043017
 ] 

Vishal K commented on ZOOKEEPER-335:
------------------------------------

Hi,

I think we are running into this bug where the leader dies after
recording the transaction locally but before sending it out to the followers.

This is causing log divergence, which is causing the client connected
to the old leader (after reboot) to  see different
data. Unfortunately, I don't have the transaction logs to look
into. But based on the ZooKeeper logs I am strongly suspecting that we
are running into this bug. I have given the description below. Please
let me know if you think I am on the wrong track.

We have a 3 node cluster. IP addresses are *.53, *.54 and *.55.

1) 54 is the leader and is rebooted by our test.

2) Node 55 becomes ZK leader
2011-05-30 22:56:45.797 INFO - 
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:QuorumPeer@662] - LEADING

The next transaction it will do will be zxid 0x0x5D00000001. We can see this 
from the zxid below:

2011-05-30 22:56:46.819 INFO - 
[LearnerHandler[myid=2]-/192.168.4.1:33329:Leader@503] - Have quorum of 
supporters; starting up and setting last processed zxid: 399431958528

3) Our client on 55 attempts to create a znode. This is the first
transaction after 55 becomes the leader. The following application
message tells us that is writing to ZK.

2011-05-30 22:56:52.608 INFO - commitTransaction - Committing transaction 21068:

This will translate to ZooKeeper transaction ID 0x5D00000001. Now, at
the same instant 53 was rebooted by the test.

Now 54 and 53 are both down, 55 loses quorum and is rebooted by our watchdog.
2011-05-30 22:56:54.922 WARN - 
[LearnerHandler[myid=2]-/192.168.4.1:33329:LearnerHandler@475] - ******* 
GOODBYE /192.168.4.1:33329 ********
2011-05-30 22:56:55.916 INFO - 
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Leader@396] - Shutdown called
java.lang.Exception: shutdown Leader! reason: Only 0 followers, need 1

At this step, I will assume that 53 logged transaction ID 0x5D00000001
*and* 53 went offline before receiving this transaction from 55. I am
not able to verify this since I don't have the transaction logs. But I
think this assumption is valid and can be inferred from the
description below.


4) Now, 53 and 54 reboot. 53 becomes leader. Since neither 53 and 54
have heard of 55's 0x5D00000001 transaction, 53 also decides to
start doing the transaction from 0x5D00000001.
Corresponding log entry from 53:
2011-05-30 22:58:10.116 INFO - 
[LearnerHandler[myid=1]-/192.168.4.1:52419:Leader@503] - Have quorum of 
supporters; starting up and setting last processed zxid: 399431958528

This is where 55 and rest of the cluster will differ in their
states. Same zxid but different content.

5) 55 reboots and rejoins the cluster. It tells 54 that it has
transaction 0x5D00000001 logged. 54 sends it a diff of transactions
starting from 0x5D00000002.

We can see that from the following entry in 55's log:
2011-05-30 22:58:24.677 WARN - 
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Learner@341] - Got zxid 0x5d00000002 
expected 0x1

Thus, old transaction 0x5D00000001 remains in 55 logs. As a result,
the client connected to 55 sees different data.

- I understand the fix proposed is to perform a null transaction after
leader election. I can work on it, but I am not sure why this is considered as a
complex fix. Mahadev mentioned earlier that the bug fix (for the
current ZAB version) will break backward compatibility. Can someone
explain why?

- In the meantime, as a workaround, we are forcing our client to do a dummy 
transaction
after leader election. We have a few clients so I think this workaround
will work without much overhead.

Comments/suggestions?

Thanks.

> zookeeper servers should commit the new leader txn to their logs.
> -----------------------------------------------------------------
>
>                 Key: ZOOKEEPER-335
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-335
>             Project: ZooKeeper
>          Issue Type: Bug
>          Components: server
>    Affects Versions: 3.1.0
>            Reporter: Mahadev konar
>            Assignee: Mahadev konar
>            Priority: Blocker
>             Fix For: 3.4.0
>
>         Attachments: ZOOKEEPER-790.travis.log.bz2, faultynode-vishal.txt, 
> zk.log.gz, zklogs.tar.gz
>
>
> currently the zookeeper followers do not commit the new leader election. This 
> will cause problems in a failure scenarios with a follower acking to the same 
> leader txn id twice, which might be two different intermittent leaders and 
> allowing them to propose two different txn's of the same zxid.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Reply via email to