Re: Zookeeper client with single address pointing to multiple servers
Hi Cee Tee, I've check and this is working for 3.5.x but 3.4.x does not have the node. Is there another way for 3.4.x? On 28/9/19, 1:09 AM, "Cee Tee" wrote: The latest config is available in node /zookeeper/config or via the config command in the cli. If you want to know what servers are actually taking requests you will have to go over them one by one issuing admin api server stats calls. On 27 September 2019 19:04:35 Shawn Heisey wrote: > On 9/27/2019 9:24 AM, Benjamin Reed wrote: >> are you making the assumption that you have a single machine that will >> always be up? that is not a common assumption these days, which is why >> solr might be resistant to such a change. >> >> you can have a single DNS name resolve to multiple IP addresses and >> ZooKeeper client will use all those addresses if you don't like >> specifying a list on all the clients. > > Is there something in the ZK client API that will allow Solr to ask the > ZK client for a list of active servers that it is connected to? > > Currently Solr just parses the zkHost string to obtain a server list for > the "ZK status" portion of our admin UI. This code was written when we > used ZK 3.4.x ... but because we're now using 3.5.x which has dynamic > reconfiguration, the list of active servers can be different than the > zkHost string used when Solr started. > > Thanks, > Shawn
Re: How to scale ZooKeeper to support 10K concurrent connections?
Whether or not you can use local sessions is a critical aspect: https://issues.apache.org/jira/browse/ZOOKEEPER-1147 Patrick On Fri, Sep 27, 2019 at 2:27 PM Michael Han wrote: > >> can launch tens of thousands of calls > > Is it possible for you to quantify this in a form of (read and write) > request per second, and the average request payload if it's OK to disclose? > This information is critical on shaping the best scaling solution. > > Without knowing any of ballpark numbers of your system and workload > characteristics, one immediate experiment you could do is to set up > Observer servers, remove quorum servers from serving client traffic by > redirecting all of your client traffic to observer servers. This will at > least scale your concurrent connections linearly with the number of > observers. This will also scale concurrent requests processing capabilities > for read requests (and to a limited extend, for write requests as well), > but request processing scaling is harder and depends on your workload > characteristics. > > > > On Fri, Sep 27, 2019 at 12:07 PM Yue Shen wrote: > > > Thank you, Jorn. > > > > We don't use Solr. We inherited this architecture from another team, and > we > > don't have time to redesign a new system to scale in 2 months. > > > > As you said, if I were to design it, I would definitely put a queue in > > front of Lambda service, our new design is actually on the way with Kafka > > upfront. However we need to scale it out with the coming holiday > > season before we can roll out the new system, which is just kicked off a > > couple of weeks ago. > > > > At this point, we want to tune ZooKeeper so it can handle 10K concurrent > > calls. Any suggestions? > > > > Thank you, > > Yue > > > > On Fri, Sep 27, 2019 at 10:39 AM Jörn Franke > wrote: > > > > > Put the Solr request on a SQS queue using your 10k instances and have > 10 > > > or so worker working on the queue to put it in Solr. Having 10k > > connections > > > just because lambda creates that many instances does not make sense for > > no > > > database service. > > > > > > > Am 27.09.2019 um 19:01 schrieb Yue Shen : > > > > > > > > Dear ZooKeeper users, > > > > > > > > I have a special use case, in which I use AWS lambda service. > > > > > > > > Inside the lambda service logic, it goes to ZooKeeper to check the > > worker > > > > for the data, if exists, connect to the worker endpoint and send the > > > data. > > > > If the worker isn't assigned, the logic will post a new assignment, > and > > > > wait for it to be assigned to a worker. There is a coordinator to > watch > > > the > > > > new assignment and assign tasks. > > > > > > > > My problem comes with AWS Lambda service, which can launch tens of > > > > thousands of calls. When this happens, I found many calls get > timeout. > > > The > > > > active connections to ZooKeeper plateau around 6500. > > > > > > > > BTW, I run ZooKeeper as 3 node ensemble, run on Quorum. > > > > > > > > How can I scale ZooKeeper to support more concurrent connections? > > > > > > > > Thank you, > > > > Yue > > > > > >
Re: How to scale ZooKeeper to support 10K concurrent connections?
>> can launch tens of thousands of calls Is it possible for you to quantify this in a form of (read and write) request per second, and the average request payload if it's OK to disclose? This information is critical on shaping the best scaling solution. Without knowing any of ballpark numbers of your system and workload characteristics, one immediate experiment you could do is to set up Observer servers, remove quorum servers from serving client traffic by redirecting all of your client traffic to observer servers. This will at least scale your concurrent connections linearly with the number of observers. This will also scale concurrent requests processing capabilities for read requests (and to a limited extend, for write requests as well), but request processing scaling is harder and depends on your workload characteristics. On Fri, Sep 27, 2019 at 12:07 PM Yue Shen wrote: > Thank you, Jorn. > > We don't use Solr. We inherited this architecture from another team, and we > don't have time to redesign a new system to scale in 2 months. > > As you said, if I were to design it, I would definitely put a queue in > front of Lambda service, our new design is actually on the way with Kafka > upfront. However we need to scale it out with the coming holiday > season before we can roll out the new system, which is just kicked off a > couple of weeks ago. > > At this point, we want to tune ZooKeeper so it can handle 10K concurrent > calls. Any suggestions? > > Thank you, > Yue > > On Fri, Sep 27, 2019 at 10:39 AM Jörn Franke wrote: > > > Put the Solr request on a SQS queue using your 10k instances and have 10 > > or so worker working on the queue to put it in Solr. Having 10k > connections > > just because lambda creates that many instances does not make sense for > no > > database service. > > > > > Am 27.09.2019 um 19:01 schrieb Yue Shen : > > > > > > Dear ZooKeeper users, > > > > > > I have a special use case, in which I use AWS lambda service. > > > > > > Inside the lambda service logic, it goes to ZooKeeper to check the > worker > > > for the data, if exists, connect to the worker endpoint and send the > > data. > > > If the worker isn't assigned, the logic will post a new assignment, and > > > wait for it to be assigned to a worker. There is a coordinator to watch > > the > > > new assignment and assign tasks. > > > > > > My problem comes with AWS Lambda service, which can launch tens of > > > thousands of calls. When this happens, I found many calls get timeout. > > The > > > active connections to ZooKeeper plateau around 6500. > > > > > > BTW, I run ZooKeeper as 3 node ensemble, run on Quorum. > > > > > > How can I scale ZooKeeper to support more concurrent connections? > > > > > > Thank you, > > > Yue > > >
Re: How to scale ZooKeeper to support 10K concurrent connections?
Hi, Sorry yes Solr I was in another email. I believe 2 months are time enough to create two SQS queues and corresponding Lambda functions. Doing a denial of service attack on your zookeeper ensemble will not help. If time allows I would try to use Amazon DynamoDb instead of zookeeper as it looks like you are using ZooKeeper in a scenario should not be used. I would probably also not use Kafka it a managed service for the same is available. However, I don’t know your exact business case and those are just ideas. > Am 27.09.2019 um 21:07 schrieb Yue Shen : > > Thank you, Jorn. > > We don't use Solr. We inherited this architecture from another team, and we > don't have time to redesign a new system to scale in 2 months. > > As you said, if I were to design it, I would definitely put a queue in > front of Lambda service, our new design is actually on the way with Kafka > upfront. However we need to scale it out with the coming holiday > season before we can roll out the new system, which is just kicked off a > couple of weeks ago. > > At this point, we want to tune ZooKeeper so it can handle 10K concurrent > calls. Any suggestions? > > Thank you, > Yue > >> On Fri, Sep 27, 2019 at 10:39 AM Jörn Franke wrote: >> >> Put the Solr request on a SQS queue using your 10k instances and have 10 >> or so worker working on the queue to put it in Solr. Having 10k connections >> just because lambda creates that many instances does not make sense for no >> database service. >> Am 27.09.2019 um 19:01 schrieb Yue Shen : >>> >>> Dear ZooKeeper users, >>> >>> I have a special use case, in which I use AWS lambda service. >>> >>> Inside the lambda service logic, it goes to ZooKeeper to check the worker >>> for the data, if exists, connect to the worker endpoint and send the >> data. >>> If the worker isn't assigned, the logic will post a new assignment, and >>> wait for it to be assigned to a worker. There is a coordinator to watch >> the >>> new assignment and assign tasks. >>> >>> My problem comes with AWS Lambda service, which can launch tens of >>> thousands of calls. When this happens, I found many calls get timeout. >> The >>> active connections to ZooKeeper plateau around 6500. >>> >>> BTW, I run ZooKeeper as 3 node ensemble, run on Quorum. >>> >>> How can I scale ZooKeeper to support more concurrent connections? >>> >>> Thank you, >>> Yue >>
Re: How to scale ZooKeeper to support 10K concurrent connections?
Thank you, Enrico. I will check majordodo. Let you know if we can use it for better architecture. Yue On Fri, Sep 27, 2019 at 11:36 AM Enrico Olivelli wrote: > Yue > As Jorn said you should introduce some kind of scalable middleware. > Zookeeper shouldn't be used in the hotpath. > > If you want something simple and based on components from Zookeeper > ecosystem maybe you can give a look to this simple task broker service, > Majordodo (full disclosure, I am one of the maintainers of the project) > > https://github.com/diennea/majordodo > > Enrico > > > Il ven 27 set 2019, 19:39 Jörn Franke ha scritto: > > > Put the Solr request on a SQS queue using your 10k instances and have 10 > > or so worker working on the queue to put it in Solr. Having 10k > connections > > just because lambda creates that many instances does not make sense for > no > > database service. > > > > > Am 27.09.2019 um 19:01 schrieb Yue Shen : > > > > > > Dear ZooKeeper users, > > > > > > I have a special use case, in which I use AWS lambda service. > > > > > > Inside the lambda service logic, it goes to ZooKeeper to check the > worker > > > for the data, if exists, connect to the worker endpoint and send the > > data. > > > If the worker isn't assigned, the logic will post a new assignment, and > > > wait for it to be assigned to a worker. There is a coordinator to watch > > the > > > new assignment and assign tasks. > > > > > > My problem comes with AWS Lambda service, which can launch tens of > > > thousands of calls. When this happens, I found many calls get timeout. > > The > > > active connections to ZooKeeper plateau around 6500. > > > > > > BTW, I run ZooKeeper as 3 node ensemble, run on Quorum. > > > > > > How can I scale ZooKeeper to support more concurrent connections? > > > > > > Thank you, > > > Yue > > >
Re: How to scale ZooKeeper to support 10K concurrent connections?
Thank you, Jorn. We don't use Solr. We inherited this architecture from another team, and we don't have time to redesign a new system to scale in 2 months. As you said, if I were to design it, I would definitely put a queue in front of Lambda service, our new design is actually on the way with Kafka upfront. However we need to scale it out with the coming holiday season before we can roll out the new system, which is just kicked off a couple of weeks ago. At this point, we want to tune ZooKeeper so it can handle 10K concurrent calls. Any suggestions? Thank you, Yue On Fri, Sep 27, 2019 at 10:39 AM Jörn Franke wrote: > Put the Solr request on a SQS queue using your 10k instances and have 10 > or so worker working on the queue to put it in Solr. Having 10k connections > just because lambda creates that many instances does not make sense for no > database service. > > > Am 27.09.2019 um 19:01 schrieb Yue Shen : > > > > Dear ZooKeeper users, > > > > I have a special use case, in which I use AWS lambda service. > > > > Inside the lambda service logic, it goes to ZooKeeper to check the worker > > for the data, if exists, connect to the worker endpoint and send the > data. > > If the worker isn't assigned, the logic will post a new assignment, and > > wait for it to be assigned to a worker. There is a coordinator to watch > the > > new assignment and assign tasks. > > > > My problem comes with AWS Lambda service, which can launch tens of > > thousands of calls. When this happens, I found many calls get timeout. > The > > active connections to ZooKeeper plateau around 6500. > > > > BTW, I run ZooKeeper as 3 node ensemble, run on Quorum. > > > > How can I scale ZooKeeper to support more concurrent connections? > > > > Thank you, > > Yue >
Re: How to scale ZooKeeper to support 10K concurrent connections?
Yue As Jorn said you should introduce some kind of scalable middleware. Zookeeper shouldn't be used in the hotpath. If you want something simple and based on components from Zookeeper ecosystem maybe you can give a look to this simple task broker service, Majordodo (full disclosure, I am one of the maintainers of the project) https://github.com/diennea/majordodo Enrico Il ven 27 set 2019, 19:39 Jörn Franke ha scritto: > Put the Solr request on a SQS queue using your 10k instances and have 10 > or so worker working on the queue to put it in Solr. Having 10k connections > just because lambda creates that many instances does not make sense for no > database service. > > > Am 27.09.2019 um 19:01 schrieb Yue Shen : > > > > Dear ZooKeeper users, > > > > I have a special use case, in which I use AWS lambda service. > > > > Inside the lambda service logic, it goes to ZooKeeper to check the worker > > for the data, if exists, connect to the worker endpoint and send the > data. > > If the worker isn't assigned, the logic will post a new assignment, and > > wait for it to be assigned to a worker. There is a coordinator to watch > the > > new assignment and assign tasks. > > > > My problem comes with AWS Lambda service, which can launch tens of > > thousands of calls. When this happens, I found many calls get timeout. > The > > active connections to ZooKeeper plateau around 6500. > > > > BTW, I run ZooKeeper as 3 node ensemble, run on Quorum. > > > > How can I scale ZooKeeper to support more concurrent connections? > > > > Thank you, > > Yue >
Re: How to scale ZooKeeper to support 10K concurrent connections?
Put the Solr request on a SQS queue using your 10k instances and have 10 or so worker working on the queue to put it in Solr. Having 10k connections just because lambda creates that many instances does not make sense for no database service. > Am 27.09.2019 um 19:01 schrieb Yue Shen : > > Dear ZooKeeper users, > > I have a special use case, in which I use AWS lambda service. > > Inside the lambda service logic, it goes to ZooKeeper to check the worker > for the data, if exists, connect to the worker endpoint and send the data. > If the worker isn't assigned, the logic will post a new assignment, and > wait for it to be assigned to a worker. There is a coordinator to watch the > new assignment and assign tasks. > > My problem comes with AWS Lambda service, which can launch tens of > thousands of calls. When this happens, I found many calls get timeout. The > active connections to ZooKeeper plateau around 6500. > > BTW, I run ZooKeeper as 3 node ensemble, run on Quorum. > > How can I scale ZooKeeper to support more concurrent connections? > > Thank you, > Yue
Re: Zookeeper client with single address pointing to multiple servers
The latest config is available in node /zookeeper/config or via the config command in the cli. If you want to know what servers are actually taking requests you will have to go over them one by one issuing admin api server stats calls. On 27 September 2019 19:04:35 Shawn Heisey wrote: On 9/27/2019 9:24 AM, Benjamin Reed wrote: are you making the assumption that you have a single machine that will always be up? that is not a common assumption these days, which is why solr might be resistant to such a change. you can have a single DNS name resolve to multiple IP addresses and ZooKeeper client will use all those addresses if you don't like specifying a list on all the clients. Is there something in the ZK client API that will allow Solr to ask the ZK client for a list of active servers that it is connected to? Currently Solr just parses the zkHost string to obtain a server list for the "ZK status" portion of our admin UI. This code was written when we used ZK 3.4.x ... but because we're now using 3.5.x which has dynamic reconfiguration, the list of active servers can be different than the zkHost string used when Solr started. Thanks, Shawn
Re: Zookeeper client with single address pointing to multiple servers
On 9/27/2019 9:24 AM, Benjamin Reed wrote: are you making the assumption that you have a single machine that will always be up? that is not a common assumption these days, which is why solr might be resistant to such a change. you can have a single DNS name resolve to multiple IP addresses and ZooKeeper client will use all those addresses if you don't like specifying a list on all the clients. Is there something in the ZK client API that will allow Solr to ask the ZK client for a list of active servers that it is connected to? Currently Solr just parses the zkHost string to obtain a server list for the "ZK status" portion of our admin UI. This code was written when we used ZK 3.4.x ... but because we're now using 3.5.x which has dynamic reconfiguration, the list of active servers can be different than the zkHost string used when Solr started. Thanks, Shawn
How to scale ZooKeeper to support 10K concurrent connections?
Dear ZooKeeper users, I have a special use case, in which I use AWS lambda service. Inside the lambda service logic, it goes to ZooKeeper to check the worker for the data, if exists, connect to the worker endpoint and send the data. If the worker isn't assigned, the logic will post a new assignment, and wait for it to be assigned to a worker. There is a coordinator to watch the new assignment and assign tasks. My problem comes with AWS Lambda service, which can launch tens of thousands of calls. When this happens, I found many calls get timeout. The active connections to ZooKeeper plateau around 6500. BTW, I run ZooKeeper as 3 node ensemble, run on Quorum. How can I scale ZooKeeper to support more concurrent connections? Thank you, Yue
Re: Zookeeper client with single address pointing to multiple servers
Thanks Ben. I'll continue to use the headless service which resolves to multiple IP. On 27/9/19, 11:24 PM, "Benjamin Reed" wrote: are you making the assumption that you have a single machine that will always be up? that is not a common assumption these days, which is why solr might be resistant to such a change. you can have a single DNS name resolve to multiple IP addresses and ZooKeeper client will use all those addresses if you don't like specifying a list on all the clients. ben On Fri, Sep 27, 2019 at 8:19 AM LEE Ween Jiann wrote: > > Thanks, I'm not sure whether Solr would make those change. I will ask them. > > Any reason for this design? > > On 27/9/19, 10:43 PM, "Cee Tee" wrote: > > You could do that as follows: > > 1 Connect to a single always online entrypoint zookeeper of the zookeeper > cluster. > 2 get Data the config node at /zookeeper/config > 3 parse it into a multinode connect string and reconnect using that string. > > > On 27 September 2019 16:33:48 LEE Ween Jiann > wrote: > > > Hi, > > > > From the Zookeeper constructore in JAVA API: > > “To create a ZooKeeper client object, the application needs to pass a > > connection string containing a comma separated list of host:port pairs, > > each corresponding to a ZooKeeper server.” > > > > I see that zookeeper resolves all the IPs from an address and randomly > > picks one. Why would multiple addresses, one for each server be needed? Why > > couldn’t zk client resolve all the servers from a single address? > > > > I’m asking this for helm deployment on Kubernetes as zookeeper is deployed > > with a single headless service that points to multiple server. > > > > >
Re: Zookeeper client with single address pointing to multiple servers
are you making the assumption that you have a single machine that will always be up? that is not a common assumption these days, which is why solr might be resistant to such a change. you can have a single DNS name resolve to multiple IP addresses and ZooKeeper client will use all those addresses if you don't like specifying a list on all the clients. ben On Fri, Sep 27, 2019 at 8:19 AM LEE Ween Jiann wrote: > > Thanks, I'm not sure whether Solr would make those change. I will ask them. > > Any reason for this design? > > On 27/9/19, 10:43 PM, "Cee Tee" wrote: > > You could do that as follows: > > 1 Connect to a single always online entrypoint zookeeper of the zookeeper > cluster. > 2 get Data the config node at /zookeeper/config > 3 parse it into a multinode connect string and reconnect using that > string. > > > On 27 September 2019 16:33:48 LEE Ween Jiann > wrote: > > > Hi, > > > > From the Zookeeper constructore in JAVA API: > > “To create a ZooKeeper client object, the application needs to pass a > > connection string containing a comma separated list of host:port pairs, > > each corresponding to a ZooKeeper server.” > > > > I see that zookeeper resolves all the IPs from an address and randomly > > picks one. Why would multiple addresses, one for each server be needed? > Why > > couldn’t zk client resolve all the servers from a single address? > > > > I’m asking this for helm deployment on Kubernetes as zookeeper is > deployed > > with a single headless service that points to multiple server. > > > > >
Re: Zookeeper client with single address pointing to multiple servers
Thanks, I'm not sure whether Solr would make those change. I will ask them. Any reason for this design? On 27/9/19, 10:43 PM, "Cee Tee" wrote: You could do that as follows: 1 Connect to a single always online entrypoint zookeeper of the zookeeper cluster. 2 get Data the config node at /zookeeper/config 3 parse it into a multinode connect string and reconnect using that string. On 27 September 2019 16:33:48 LEE Ween Jiann wrote: > Hi, > > From the Zookeeper constructore in JAVA API: > “To create a ZooKeeper client object, the application needs to pass a > connection string containing a comma separated list of host:port pairs, > each corresponding to a ZooKeeper server.” > > I see that zookeeper resolves all the IPs from an address and randomly > picks one. Why would multiple addresses, one for each server be needed? Why > couldn’t zk client resolve all the servers from a single address? > > I’m asking this for helm deployment on Kubernetes as zookeeper is deployed > with a single headless service that points to multiple server.
Re: Zookeeper client with single address pointing to multiple servers
You could do that as follows: 1 Connect to a single always online entrypoint zookeeper of the zookeeper cluster. 2 get Data the config node at /zookeeper/config 3 parse it into a multinode connect string and reconnect using that string. On 27 September 2019 16:33:48 LEE Ween Jiann wrote: Hi, From the Zookeeper constructore in JAVA API: “To create a ZooKeeper client object, the application needs to pass a connection string containing a comma separated list of host:port pairs, each corresponding to a ZooKeeper server.” I see that zookeeper resolves all the IPs from an address and randomly picks one. Why would multiple addresses, one for each server be needed? Why couldn’t zk client resolve all the servers from a single address? I’m asking this for helm deployment on Kubernetes as zookeeper is deployed with a single headless service that points to multiple server.
Zookeeper client with single address pointing to multiple servers
Hi, From the Zookeeper constructore in JAVA API: “To create a ZooKeeper client object, the application needs to pass a connection string containing a comma separated list of host:port pairs, each corresponding to a ZooKeeper server.” I see that zookeeper resolves all the IPs from an address and randomly picks one. Why would multiple addresses, one for each server be needed? Why couldn’t zk client resolve all the servers from a single address? I’m asking this for helm deployment on Kubernetes as zookeeper is deployed with a single headless service that points to multiple server.
Re: About ZooKeeper Dynamic Reconfiguration
Hi oo4load, Got it. Thanks a lot! -- Sent from: http://zookeeper-user.578899.n2.nabble.com/
Re: About ZooKeeper Dynamic Reconfiguration
No you have to build a zookeeper cluster manager client using my code. Its a zookeeper client. On 27 September 2019 10:44:51 "Gao,Wei" wrote: *Hi oo4load, How could we integrate this implementation with ZooKeeper 3.5.5? Does it mean we have to mix the implementation code into the already released ZooKeeper 3.5.5 and rebuild it again into another ZooKeeper and re-install it? Thanks.* -- Sent from: http://zookeeper-user.578899.n2.nabble.com/
Re: About ZooKeeper Dynamic Reconfiguration
*Hi oo4load, How could we integrate this implementation with ZooKeeper 3.5.5? Does it mean we have to mix the implementation code into the already released ZooKeeper 3.5.5 and rebuild it again into another ZooKeeper and re-install it? Thanks.* -- Sent from: http://zookeeper-user.578899.n2.nabble.com/
Re: About ZooKeeper Dynamic Reconfiguration
Hi oo4load, Thank you so much for your reply! How I wish I could appreciate your design with actual code! Really look forward to hearing from you. -- Sent from: http://zookeeper-user.578899.n2.nabble.com/
Re: About ZooKeeper Dynamic Reconfiguration
Let me write this from memory. :)
We have the following:
-A running zookeeper cluster with adminserver enabled
-One or more balancer client processes (one per datacenter), of which one
has a master role through some leader election. The master does the work,
the others do nothing.
-In our case, we work with a designated survivor datacenter (has 3
participants and the other non survivor datacenter has 2 participant and 1
observer ) , and the balancer always resides in the designated survivor
datacenter. This is not a requirement, due to above leader election.
-A balancer client configuration file with all predefined Zookeeper servers
(use it for building the client connection string and generating the server
list). Each predefined server under normal condition has a running
Zookeeper in either participant or observer role.
Balancer design:
buildDatacenterAndServerModel(configurationFile) {
enum zookeeperRole PARTICIPANT, OBSERVER, NONE, DOWN
object datacenter has servers
object server has zookeeperRole configuredRole, zookeeperRole activeRole
parse(configurationFile) into (datacenter, servers);
}
shiftMajority(designatedSurvivorDatacenter) {
designatedSurvivorDatacenter.someObserver.dynamicReconfigure(server=PARTICIPANT)
otherDatacenter.someParticipant.dynamicReconfigure(server=OBSERVER)
}
balanceServerRoles() {
if (designatedSurvivorDatacenter.hasMinimumQuorum)
someParticipant.dynamicReconfigure(server=OBSERVER)
if (quorumSize.aboveSafeLimit)
someObserver.dynamicReconfigure(server=PARTICIPANT)
//This is a lot more complicated than 2 simple commands, you need an
algorithm or define several scenarios.
}
main() {
buildDatacenterAndServerModel(configurationFile);
while (iAmLeader) {
parse(zk.getData(/zookeeper/config)) into servers.configuredRole
foreach(server) getServerRole(server:8081/commands/stat) into
servers.activeRole
foreach (server.activeRole=DOWN) dynamicReconfigure(server=OBSERVER) ;
server.setConfiguredRole(OBSERVER)
if(designatedSurvivorDatacenter != datacenter.hasMajority)
shiftMajority(designatedSurvivorDatacenter)
balanceServerRoles()
}
}
Hope this helps. If you need more details, I can check the actual code
coming week.
On Fri, Sep 27, 2019 at 5:06 AM Gao,Wei wrote:
> Hi oo4load,
> Could you please tell me how to implements this to avoid the problem
> above?
> Thanks
>
>
>
> --
> Sent from: http://zookeeper-user.578899.n2.nabble.com/
>
