[ https://issues.apache.org/jira/browse/ZOOKEEPER-512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Flavio Paiva Junqueira updated ZOOKEEPER-512: --------------------------------------------- Attachment: ZOOKEEPER-512.patch I've found a corner that I was not expecting but was an easy fix. Basically receiveConnection was passing an invalid server identifier to connectOne which was throwing a NPE and was propagating to the Listener. The listener would consequently die, and would stop participating in leader election. The fix is just not to proceed with the connection when that happens. This patch is working fine for my test. It is important to note, though, that if we are overwhelming servers so much (clients are hammering the system and connections are failing), then there will be periods in which there will be no leader. The important invariant to satisfy is that the system converges to a live state once it stabilizes. In my tests, I observe periods with no leader when clients are hammering the servers with requests, but they converge to a leader soon after the clients stop. Of course, if we have no injected faults, the clients requests are executed just fine (there is always a leader). This is the behavior I expect to see. At the same time, although I think it was a good idea to test such an extreme case, I'm still not convinced that this test is realistic. It would be great if we could model the cases this fault injection is trying to emulate to make sure they are really expected cases. Also, I don't see a good way of introducing a unit test for such extreme cases. In fact, I'm not even sure it would make sense to test only leader election under such extreme conditions. > FLE election fails to elect leader > ---------------------------------- > > Key: ZOOKEEPER-512 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-512 > Project: Zookeeper > Issue Type: Bug > Components: quorum, server > Affects Versions: 3.2.0 > Reporter: Patrick Hunt > Assignee: Flavio Paiva Junqueira > Priority: Blocker > Fix For: 3.2.1, 3.3.0 > > Attachments: jst.txt, log3_debug.tar.gz, logs.tar.gz, logs2.tar.gz, > t5_aj.tar.gz, ZOOKEEPER-512.patch, ZOOKEEPER-512.patch, ZOOKEEPER-512.patch, > ZOOKEEPER-512.patch > > > I was doing some fault injection testing of 3.2.1 with ZOOKEEPER-508 patch > applied and noticed that after some time the ensemble failed to re-elect a > leader. > See the attached log files - 5 member ensemble. typically 5 is the leader > Notice that after 16:23:50,525 no quorum is formed, even after 20 minutes > elapses w/no quorum > environment: > I was doing fault injection testing using aspectj. The faults are injected > into socketchannel read/write, I throw exceptions randomly at a 1/200 ratio > (rand.nextFloat() <= .005 => throw IOException > You can see when a fault is injected in the log via: > 2009-08-19 16:57:09,568 - INFO [Thread-74:readrequestfailsintermitten...@38] > - READPACKET FORCED FAIL > vs a read/write that didn't force fail: > 2009-08-19 16:57:09,568 - INFO [Thread-74:readrequestfailsintermitten...@41] > - READPACKET OK > otw standard code/config (straight fle quorum with 5 members) > also see the attached jstack trace. this is for one of the servers. Notice in > particular that the number of sendworkers != the number of recv workers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.