Flavio Paiva Junqueira updated ZOOKEEPER-512:

    Attachment: ZOOKEEPER-512.patch

I've found a corner that I was not expecting but was an easy fix. Basically 
receiveConnection was passing an invalid server identifier to connectOne which 
was throwing a NPE and was propagating to the Listener. The listener would 
consequently die, and would stop participating in leader election. The fix is 
just not to proceed with the connection when that happens.

This patch is working fine for my test. It is important to note, though, that 
if we are overwhelming servers so much (clients are hammering the system and 
connections are failing), then there will be periods in which there will be no 
leader. The important invariant to satisfy is that the system converges to a 
live state once it stabilizes. In my tests, I observe periods with no leader 
when clients are hammering the servers with requests, but they converge to a 
leader soon after the clients stop. Of course, if we have no injected faults, 
the clients requests are executed just fine (there is always a leader). This is 
the behavior I expect to see.

At the same time, although I think it was a good idea to test such an extreme 
case, I'm still not convinced that this test is realistic. It would be great if 
we could model the cases this fault injection is trying to emulate to make sure 
they are really expected cases. 

Also, I don't see a good way of introducing a unit test for such extreme cases. 
In fact, I'm not even sure it would make sense to test only leader election 
under such extreme conditions. 

> FLE election fails to elect leader
> ----------------------------------
>                 Key: ZOOKEEPER-512
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-512
>             Project: Zookeeper
>          Issue Type: Bug
>          Components: quorum, server
>    Affects Versions: 3.2.0
>            Reporter: Patrick Hunt
>            Assignee: Flavio Paiva Junqueira
>            Priority: Blocker
>             Fix For: 3.2.1, 3.3.0
>         Attachments: jst.txt, log3_debug.tar.gz, logs.tar.gz, logs2.tar.gz, 
> t5_aj.tar.gz, ZOOKEEPER-512.patch, ZOOKEEPER-512.patch, ZOOKEEPER-512.patch, 
> ZOOKEEPER-512.patch
> I was doing some fault injection testing of 3.2.1 with ZOOKEEPER-508 patch 
> applied and noticed that after some time the ensemble failed to re-elect a 
> leader.
> See the attached log files - 5 member ensemble. typically 5 is the leader
> Notice that after 16:23:50,525 no quorum is formed, even after 20 minutes 
> elapses w/no quorum
> environment:
> I was doing fault injection testing using aspectj. The faults are injected 
> into socketchannel read/write, I throw exceptions randomly at a 1/200 ratio 
> (rand.nextFloat() <= .005 => throw IOException
> You can see when a fault is injected in the log via:
> 2009-08-19 16:57:09,568 - INFO  [Thread-74:readrequestfailsintermitten...@38] 
> vs a read/write that didn't force fail:
> 2009-08-19 16:57:09,568 - INFO  [Thread-74:readrequestfailsintermitten...@41] 
> otw standard code/config (straight fle quorum with 5 members)
> also see the attached jstack trace. this is for one of the servers. Notice in 
> particular that the number of sendworkers != the number of recv workers.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

Reply via email to