We have 3+3 of which 1 floating observer in non target datacenter and
automatic reconfiguring to more observers if we are losing participants.
If the target datacenter blows up this doesn't work, but our main
application will be able to serve customers in a readonly state until
operators switch the non target datacenter to active mode.
On 21 August 2019 20:39:21 Enrico Olivelli <eolive...@gmail.com> wrote:
Il mer 21 ago 2019, 20:27 Cee Tee <c.turks...@gmail.com> ha scritto:
Yes, one side loses quorum and the other remains active. However we
actively control which side that is, because our main application is
active/passive with 2 datacenters. We need Zookeeper to remain active in
the applications active datacenter.
How many zk servers you have? 2 + 3?
If you lose DC #1 you are okay, but if you lose the #2 you cannot have a
quorum of 3, and you cannot simply add another server to #1
Enrico
On 21 August 2019 17:22:00 Alexander Shraer <shra...@gmail.com> wrote:
> That's great! Thanks for sharing.
>
>
>> Added benefit is that we can also control which data center gets the
quorum
>> in case of a network outage between the two.
>
>
> Can you explain how this works? In case of a network outage between two
> DCs, one of them has a quorum of participants and the other doesn't.
> The participants in the smaller set should not be operational at this
time,
> since they can't get quorum. no ?
>
>
>
> Thanks,
> Alex
>
>
> On Wed, Aug 21, 2019 at 7:55 AM Cee Tee <c.turks...@gmail.com> wrote:
>
> We have solved this by implementing a 'zookeeper cluster balancer', it
> calls the admin server api of each zookeeper to get the current status
and
> will issue dynamic reconfigure commands to change dead servers into
> observers so the quorum is not in danger. Once the dead servers
reconnect,
> they take the observer role and are then reconfigured into participants
again.
>
> Added benefit is that we can also control which data center gets the
quorum
> in case of a network outage between the two.
> Regards
> Chris
>
> On 21 August 2019 16:42:37 Alexander Shraer <shra...@gmail.com> wrote:
>
>> Hi,
>>
>> Reconfiguration, as implemented, is not automatic. In your case, when
>> failures happen, this doesn't change the ensemble membership.
>> When 2 of 5 fail, this is still a minority, so everything should work
>> normally, you just won't be able to handle an additional failure. If
you'd
>> like
>> to remove them from the ensemble, you need to issue an explicit
>> reconfiguration command to do so.
>>
>> Please see details in the manual:
>> https://zookeeper.apache.org/doc/r3.5.5/zookeeperReconfig.html
>>
>> Alex
>>
>> On Wed, Aug 21, 2019 at 7:29 AM Gao,Wei <wei....@arcserve.com> wrote:
>>
>>> Hi
>>> I encounter a problem which blocks my development of load balance
using
>>> ZooKeeper 3.5.5.
>>> Actually, I have a ZooKeeper cluster which comprises of five zk
>>> servers. And the dynamic configuration file is as follows:
>>>
>>> server.1=zk1:2888:3888:participant;0.0.0.0:2181
>>> server.2=zk2:2888:3888:participant;0.0.0.0:2181
>>> server.3=zk3:2888:3888:participant;0.0.0.0:2181
>>> server.4=zk4:2888:3888:participant;0.0.0.0:2181
>>> server.5=zk5:2888:3888:participant;0.0.0.0:2181
>>>
>>> The zk cluster can work fine if every member works normally.
However, if
>>> say two of them are suddenly down without previously being notified,
>>> the dynamic configuration file shown above will not be synchronized
>>> dynamically, which leads to the zk cluster fail to work normally.
>>> I think this is a very common case which may happen at any time. If
so,
>>> how can we resolve it?
>>> Really look forward to hearing from you!
>>> Thanks
>>>
