Re: Dynamic starting or stoping of zookeepers in a cluster

[Shawn Heisey]

On 2/24/2021 9:04 AM, DAVID MARTIN NIETO wrote:

If I'm not mistaken the number of zookeepers must be odd. Having 3 zoos on 3 
different machines, if we temporarily lost one of the three machines, we would 
have only two running and it would be an even number.Would it be advisable in 
this case to raise a third party in one of the 2 active machines or with only 
two zookeepers there would be no blockages in their internal votes?


It does not HAVE to be an odd number.  But increasing the total by one 
doesn't add any additional fault tolerance, and exposes an additional 
point of failure.


If you have 3 servers, 2 of them have to be running to maintain quorum. 
 If you have 4 servers, 3 of them have to be running for the cluster to 
be fully operational.


So a 3-server cluster and a 4-server cluster can survive the failure of 
one machine.  This holds true for larger numbers as well -- with 5 
servers or with 6 servers, you can lose two and stay fully operational. 
 Having that extra server that makes the total even is just wasteful.


Thanks,
Shawn

RE: Dynamic starting or stoping of zookeepers in a cluster

[DAVID MARTIN NIETO]

One doubt about it:

In order to have a highly available zookeeper, you must have at least
three separate physical servers for ZK.  Running multiple zookeepers on
one physical machine gains you nothing ... because if the whole machine
fails, you lose all of those zookeepers.  If you have three physical
servers, one can fail with no problems.  If you have five separate
physical servers running ZK, then two of the machines can fail without
taking the cluster down.

If I'm not mistaken the number of zookeepers must be odd. Having 3 zoos on 3 
different machines, if we temporarily lost one of the three machines, we would 
have only two running and it would be an even number.Would it be advisable in 
this case to raise a third party in one of the 2 active machines or with only 
two zookeepers there would be no blockages in their internal votes?

About the dynamic reconfiguration many thanks we've 8.2 but the zoos are in 
3.4.2 version, we're going to test with 3.5 version and the dynamic 
configuration of zookeepers to avoid this problem.

Many thanks.
Kind regards.



De: Joe Lerner 
Enviado: viernes, 19 de febrero de 2021 18:56
Para: solr-user@lucene.apache.org 
Asunto: Re: Dynamic starting or stoping of zookeepers in a cluster

This is solid information. *How about the application, which uses
SOLR/Zookeeper?*

Do we have to follow this guidance, to make the application ZK config aware:

https://zookeeper.apache.org/doc/r3.5.5/zookeeperReconfig.html#ch_reconfig_rebalancing


Or, could we leave it as is, and as long as the ZK Ensemble has the same
IPs?

Thanks!

Joe




--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Dynamic starting or stoping of zookeepers in a cluster

[Joe Lerner]

This is solid information. *How about the application, which uses
SOLR/Zookeeper?*

Do we have to follow this guidance, to make the application ZK config aware:

https://zookeeper.apache.org/doc/r3.5.5/zookeeperReconfig.html#ch_reconfig_rebalancing

  

Or, could we leave it as is, and as long as the ZK Ensemble has the same
IPs?

Thanks!

Joe




--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Dynamic starting or stoping of zookeepers in a cluster

[Shawn Heisey]

On 2/18/2021 8:20 AM, DAVID MARTIN NIETO wrote:

We've a solr cluster with 4 solr servers and 5 zookeepers in HA mode.
We've tested about if our cluster can mantain the service with only the half of 
the cluster, in case of disaster os similar, and we've a problem with the 
zookepers config and its static configuration.

In the start script of the 4 solrs servers there are a list of 5 ip:port of the 5 
zookeepers of the cluster, so when we "lost" the half of machines (we've 2 zoos 
in one machine and 3 on another) in the worst case we lost 3 of these 5 zookeepers. We 
can start a sixth zookeeper (to have 3 with the half of cluster stopped) but to add in 
the solr server we need to stop and restart with a new list of ip:port adding it and 
that's not an automatic or dynamic thing.


In order to have a highly available zookeeper, you must have at least 
three separate physical servers for ZK.  Running multiple zookeepers on 
one physical machine gains you nothing ... because if the whole machine 
fails, you lose all of those zookeepers.  If you have three physical 
servers, one can fail with no problems.  If you have five separate 
physical servers running ZK, then two of the machines can fail without 
taking the cluster down.



¿Somebody knows another configuration or workaround to have a dynamic list of 
zoos and start or stop some of thems without changes in the config and 
start/stop the solr server?


The Zookeeper client was upgraded to 3.5 in Solr 8.2.0.

https://issues.apache.org/jira/browse/SOLR-8346

If you're running at least Solr 8.2.0, and your ZK servers are at least 
version 3.5, then ZK should support dynamic cluster reconfiguration. 
The ZK status page in the admin UI may have some problems after ZK 
undergoes a dynamic reconfiguration, but SolrCloud's core functionality 
should work fine.


Thanks,
Shawn

Dynamic starting or stoping of zookeepers in a cluster

[DAVID MARTIN NIETO]

Hi all,

We've a solr cluster with 4 solr servers and 5 zookeepers in HA mode.
We've tested about if our cluster can mantain the service with only the half of 
the cluster, in case of disaster os similar, and we've a problem with the 
zookepers config and its static configuration.

In the start script of the 4 solrs servers there are a list of 5 ip:port of the 
5 zookeepers of the cluster, so when we "lost" the half of machines (we've 2 
zoos in one machine and 3 on another) in the worst case we lost 3 of these 5 
zookeepers. We can start a sixth zookeeper (to have 3 with the half of cluster 
stopped) but to add in the solr server we need to stop and restart with a new 
list of ip:port adding it and that's not an automatic or dynamic thing.

¿Somebody knows another configuration or workaround to have a dynamic list of 
zoos and start or stop some of thems without changes in the config and 
start/stop the solr server?

Kind regards and thanks a lot.


---
Este mensaje va dirigido únicamente a la(s) persona(s) y/o entidad(es) arriba 
relacionada(s). Puede contener información confidencial o legalmente protegida. 
Si no es usted el destinatario señalado, le rogamos borre del sistema 
inmediatamente el mensaje y sus copias. Asimismo le informamos que cualquier 
copia, divulgación, distribución o uso de los contenidos está prohibida.
---
This message is addressed only to the person (people) and / or entities listed 
above. It may contain confidential or legally protected information. If you are 
not the recipient indicated, please delete the message and its copies 
immediately from the system. We also inform that any copy, disclosure, 
distribution or use of the contents is forbidden
---
Viewnext, S.A. Domicilio Social: Avda. de Burgos 8-A 28036 de Madrid. telf: 
913834060, Fax: 913834090. Reg. M. Madrid: Tomo 3238, Libro:0, Folio: 78, 
Seccion: 8ª, Hoja M-55112, N.I.F.: A-80157746