Flavio Paiva Junqueira updated ZOOKEEPER-512:
I've found a corner that I was not expecting but was an easy fix. Basically
receiveConnection was passing an invalid server identifier to connectOne which
was throwing a NPE and was propagating to the Listener. The listener would
consequently die, and would stop participating in leader election. The fix is
just not to proceed with the connection when that happens.
This patch is working fine for my test. It is important to note, though, that
if we are overwhelming servers so much (clients are hammering the system and
connections are failing), then there will be periods in which there will be no
leader. The important invariant to satisfy is that the system converges to a
live state once it stabilizes. In my tests, I observe periods with no leader
when clients are hammering the servers with requests, but they converge to a
leader soon after the clients stop. Of course, if we have no injected faults,
the clients requests are executed just fine (there is always a leader). This is
the behavior I expect to see.
At the same time, although I think it was a good idea to test such an extreme
case, I'm still not convinced that this test is realistic. It would be great if
we could model the cases this fault injection is trying to emulate to make sure
they are really expected cases.
Also, I don't see a good way of introducing a unit test for such extreme cases.
In fact, I'm not even sure it would make sense to test only leader election
under such extreme conditions.
> FLE election fails to elect leader
> Key: ZOOKEEPER-512
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-512
> Project: Zookeeper
> Issue Type: Bug
> Components: quorum, server
> Affects Versions: 3.2.0
> Reporter: Patrick Hunt
> Assignee: Flavio Paiva Junqueira
> Priority: Blocker
> Fix For: 3.2.1, 3.3.0
> Attachments: jst.txt, log3_debug.tar.gz, logs.tar.gz, logs2.tar.gz,
> t5_aj.tar.gz, ZOOKEEPER-512.patch, ZOOKEEPER-512.patch, ZOOKEEPER-512.patch,
> I was doing some fault injection testing of 3.2.1 with ZOOKEEPER-508 patch
> applied and noticed that after some time the ensemble failed to re-elect a
> See the attached log files - 5 member ensemble. typically 5 is the leader
> Notice that after 16:23:50,525 no quorum is formed, even after 20 minutes
> elapses w/no quorum
> I was doing fault injection testing using aspectj. The faults are injected
> into socketchannel read/write, I throw exceptions randomly at a 1/200 ratio
> (rand.nextFloat() <= .005 => throw IOException
> You can see when a fault is injected in the log via:
> 2009-08-19 16:57:09,568 - INFO [Thread-74:readrequestfailsintermitten...@38]
> - READPACKET FORCED FAIL
> vs a read/write that didn't force fail:
> 2009-08-19 16:57:09,568 - INFO [Thread-74:readrequestfailsintermitten...@41]
> - READPACKET OK
> otw standard code/config (straight fle quorum with 5 members)
> also see the attached jstack trace. this is for one of the servers. Notice in
> particular that the number of sendworkers != the number of recv workers.
This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.