Re: zookeeper create Znode

[Cee Tee]

Did we miss the question?
Regards
Chris

On 14 May 2020 18:11:18 dong ma  wrote:


hi，I recently encountered a problem. We use zookeeper to manage the status
of some user jobs. For example, when a user job is converted from NEW to
RUNNING state, we will update its `RUNNING` state to the / job / states / $
{jobId} node. At the same time, we will also use PathChildrenCache to
monitor the directory. We correspond to multiple zookeeper clusters to
monitor the status of different clusters. The initialization code is as
follows:

for(Map.Entry zkEntry : zkStringMap.entrySet()) {
CuratorFramework curator = CuratorFrameworkFactory.builder()
.connectString(...)
.sessionTimeoutMs(6)
.connectionTimeoutMs(15000)
.retryPolicy(new ExponentialBackoffRetry(5000, 3))
.namespace("job/states")
.build();
curator.start();

PathChildrenCache cache = new PathChildrenCache(curator, "/", true);

cache.getListenable().addListener(new JobStateListener(zkEntry.getKey(),
stateHandler));

cache.start();
}




if  (node not exists) {

jobStateCurator.create().withMode(CreateMode.EPHEMERAL).forPath(jobPath,
newJobStatus.toString().getBytes());
} else {
jobStateCurator.setData().forPath(jobPath,
newJobStatus.toString().getBytes(charset));
}

if(node

} else {
String finalJobPath = "/final-" + jobId.toString();
LOG.info("Job switch state to final state {}", newJobStatus.toString());

jobStateCurator.create().forPath(finalJobPath,
newJobStatus.toString().getBytes(charset));
int retry = 0;
Stat stat;
while((stat = jobStateCurator.checkExists().forPath(finalJobPath)) == null)
{
LOG.info("check " + finalJobPath + " not exists, retry after 50 ms, times "
+ retry++);
Thread.sleep(50);
}

LOG.info("Success create zk Node in final state for " + jobId.toString() );
jobStateCurator.close();
}

if (!newJobStatus.isGloballyTerminalState()) {

LOG.info("Job switch state to {}", newJobStatus.toString());
if (jobStateCurator.checkExists().forPath(jobPath) == null) {
jobStateCurator.create().withMode(CreateMode.EPHEMERAL).forPath(jobPath,
newJobStatus.toString().getBytes());
} else {
jobStateCurator.setData().forPath(jobPath,
newJobStatus.toString().getBytes(charset));
}
} else {
String finalJobPath = "/final-" + jobId.toString();
LOG.info("Job switch state to final state {}", newJobStatus.toString());

jobStateCurator.create().forPath(finalJobPath,
newJobStatus.toString().getBytes(charset));
int retry = 0;
Stat stat;
while((stat = jobStateCurator.checkExists().forPath(finalJobPath)) == null)
{
LOG.info("check " + finalJobPath + " not exists, retry after 50 ms, times "
+ retry++);
Thread.sleep(50);
}

LOG.info("Success create zk Node in final state for " + jobId.toString() );
jobStateCurator.close();
}

Re: [curator][lock-recipe] implementing locks with just ephemeral nodes

[Cee Tee]

Ah that's correct, we use 3.5 branch.

On 5 February 2020 07:33:47 Rajat Gangwar  wrote:


Hi Chris,

*"The whole tree disappears when the last ephemeral leaf node is removed."*

I think parent node has to be container node for it it work, but container
node  was added post
ZK 3.5.
I am trying to get lock-recipe for ZK 3.4. Did you try to implement with
just ephemeral nodes in version <= 3.4?

Thanks,
Rajat

On Wed, Feb 5, 2020 at 11:32 AM Chris T.  wrote:


I have a type of locking mechanism using ephemerals via Curator
framework, they are created with with creatingParentsIfNeeded. The
whole tree disappears when the last ephemeral leaf node is removed.
Parent nodes have an IP address as value.


Code snip from our class:

CreateModable> localCreateMethod
= createMethod.get();
if (localCreateMethod == null) {
CreateBuilderMain createBuilder = cfClient.create();
CreateModable>
tempCreateMethod = createBuilder.creatingParentContainersIfNeeded();
createMethod.compareAndSet(null, tempCreateMethod);
localCreateMethod = createMethod.get();
}

try {
localCreateMethod.withMode(CreateMode.EPHEMERAL).inBackground(backgroundCallback).forPath(basePath,
getEncryptedBytes(commandNode.getNodeBytes()));
} catch (Exception e) {
log.error("Exception: ", e);
}



[zk: localhost:2181(CONNECTED) 1] get /GROUP_V7POC03/ephemerals/inquirylock
10.194.54.93
[zk: localhost:2181(CONNECTED) 2] stat
/GROUP_V7POC03/ephemerals/inquirylock
cZxid = 0xf20001914e
ctime = Tue Feb 04 11:11:45 CET 2020
mZxid = 0xf200019169
mtime = Tue Feb 04 11:11:48 CET 2020
pZxid = 0xf20001918b
cversion = 2
dataVersion = 3
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 12
numChildren = 2
[zk: localhost:2181(CONNECTED) 3] get /GROUP_V7POC03/ephemerals/inquirylock
10.194.54.93
[zk: localhost:2181(CONNECTED) 4] get /GROUP_V7POC03/ephemerals

[zk: localhost:2181(CONNECTED) 5] stat
/GROUP_V7POC03/ephemerals/inquirylock/lrv1705f.europe.intranet:v7poc03
cZxid = 0xf200019189
ctime = Tue Feb 04 11:11:51 CET 2020
mZxid = 0xf200019189
mtime = Tue Feb 04 11:11:51 CET 2020
pZxid = 0xf200019189
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x10022c4e19c000b
dataLength = 83
numChildren = 0



On 4 February 2020 15:58:39 Jordan Zimmerman 
wrote:

>> can I implement a lock recipe which just
>> tries to create ephemeral node without any persistent nodes.
>
> Ephemeral nodes cannot have any children. Unless you create your lock at
> the root ("/") you'd need some parent persistent node. Is this a big
> problem for you? Unless you have 1000s of unique lock paths there
shouldn't
> be any problem with persistent parent node.
>
> -Jordan

Re: Admin server deadlocks?

[Cee Tee]

Hey that sounds familiar, we do curl to the stats api and they pretty 
consistently start to hang after a while (minutes to hours).
I thought it was something in our environment but I'm happy to read I'm not 
the only one. :)
I used to run a snapshot 3.6 that didn't suffer from this. It appeared 
after 'upgrading' to the official 356.

Regards
Chris

On 20 January 2020 20:54:51 Craig.Condit  wrote:

We have been running Zookeeper 3.5.6 on several clusters for a while now, 
and have noticed (pretty consistently) that the new Admin Server seems to 
stop responding (hangs) after the ZK service has been up and running for a 
while. The stack dumps we have done seem to indicate some sort of lock 
being held, probably by Jetty. This makes sense, as when we see the hang 
start, it happens for all URLs (even the root / 404 not found page).



Has anyone encountered this? I was not able to find a related JIRA.

Re: About ZooKeeper Dynamic Reconfiguration

[Cee Tee]

No problem, I will send you Monday.

On 29 September 2019 04:30:28 "Gao,Wei"  wrote:


Hi oo4load,
 If it is convenient to you, I would like to get the actual code from you
about the zookeeper cluster balancer implementation. My email address is:
 */wei@arcserve.com/*
 Thank you again.



--
Sent from: http://zookeeper-user.578899.n2.nabble.com/

Re: Zookeeper client with single address pointing to multiple servers

[Cee Tee]

You would have to grab the config file from the filesystem, or have some 
process push it into a zookeeper node.

Re: Zookeeper client with single address pointing to multiple servers

[Cee Tee]

The latest config is available in node /zookeeper/config or via the config 
command in the cli. If you want to know what servers are actually taking 
requests you will have to go over them one by one issuing admin api server 
stats calls.


On 27 September 2019 19:04:35 Shawn Heisey  wrote:


On 9/27/2019 9:24 AM, Benjamin Reed wrote:

are you making the assumption that you have a single machine that will
always be up? that is not a common assumption these days, which is why
solr might be resistant to such a change.

you can have a single DNS name resolve to multiple IP addresses and
ZooKeeper client will use all those addresses if you don't like
specifying a list on all the clients.


Is there something in the ZK client API that will allow Solr to ask the
ZK client for a list of active servers that it is connected to?

Currently Solr just parses the zkHost string to obtain a server list for
the "ZK status" portion of our admin UI.  This code was written when we
used ZK 3.4.x ... but because we're now using 3.5.x which has dynamic
reconfiguration, the list of active servers can be different than the
zkHost string used when Solr started.

Thanks,
Shawn

Re: Zookeeper client with single address pointing to multiple servers

[Cee Tee]

You could do that as follows:

1 Connect to a single always online entrypoint zookeeper of the zookeeper 
cluster.

2 get Data the config node at /zookeeper/config
3 parse it into a multinode connect string and reconnect using that string.


On 27 September 2019 16:33:48 LEE Ween Jiann  
wrote:



Hi,

From the Zookeeper constructore in JAVA API:
“To create a ZooKeeper client object, the application needs to pass a 
connection string containing a comma separated list of host:port pairs, 
each corresponding to a ZooKeeper server.”


I see that zookeeper resolves all the IPs from an address and randomly 
picks one. Why would multiple addresses, one for each server be needed? Why 
couldn’t zk client resolve all the servers from a single address?


I’m asking this for helm deployment on Kubernetes as zookeeper is deployed 
with a single headless service that points to multiple server.

Re: About ZooKeeper Dynamic Reconfiguration

[Cee Tee]

No you have to build a zookeeper cluster manager client using my code. Its 
a zookeeper client.


On 27 September 2019 10:44:51 "Gao,Wei"  wrote:


*Hi oo4load,
 How could we integrate this implementation with ZooKeeper 3.5.5? Does it
mean we have to mix the implementation code into the already released
ZooKeeper 3.5.5 and rebuild it again into another ZooKeeper and re-install
it?
 Thanks.*



--
Sent from: http://zookeeper-user.578899.n2.nabble.com/

Re: About ZooKeeper Dynamic Reconfiguration

[Cee Tee]

We have 3+3 of which 1 floating observer in non target datacenter and 
automatic reconfiguring to more observers if we are losing participants.


If the target datacenter blows up this doesn't work, but our main 
application will be able to serve customers in a readonly state until 
operators switch the non target datacenter to active mode.


On 21 August 2019 20:39:21 Enrico Olivelli  wrote:


Il mer 21 ago 2019, 20:27 Cee Tee  ha scritto:



Yes, one side loses quorum and the other remains active. However we
actively control which side that is, because our main application is
active/passive with 2 datacenters. We need Zookeeper to remain active in


the applications active datacenter.




How many zk servers you have? 2 + 3?
If you lose DC #1 you are okay, but if you lose the #2 you cannot have a
quorum of 3, and you cannot simply add another server to #1

Enrico



On 21 August 2019 17:22:00 Alexander Shraer  wrote:
> That's great! Thanks for sharing.
>
>
>> Added benefit is that we can also control which data center gets the
quorum
>> in case of a network outage between the two.
>
>
> Can you explain how this works? In case of a network outage between two
> DCs, one of them has a quorum of participants and the other doesn't.
> The participants in the smaller set should not be operational at this
time,
> since they can't get quorum. no ?
>
>
>
> Thanks,
> Alex
>
>
> On Wed, Aug 21, 2019 at 7:55 AM Cee Tee  wrote:
>
> We have solved this by implementing a 'zookeeper cluster balancer', it
> calls the admin server api of each zookeeper to get the current status
and
> will issue dynamic reconfigure commands to change dead servers into
> observers so the quorum is not in danger. Once the dead servers
reconnect,
> they take the observer role and are then reconfigured into participants
again.
>
> Added benefit is that we can also control which data center gets the
quorum
> in case of a network outage between the two.
> Regards
> Chris
>
> On 21 August 2019 16:42:37 Alexander Shraer  wrote:
>
>> Hi,
>>
>> Reconfiguration, as implemented, is not automatic. In your case, when
>> failures happen, this doesn't change the ensemble membership.
>> When 2 of 5 fail, this is still a minority, so everything should work
>> normally, you just won't be able to handle an additional failure. If
you'd
>> like
>> to remove them from the ensemble, you need to issue an explicit
>> reconfiguration command to do so.
>>
>> Please see details in the manual:
>> https://zookeeper.apache.org/doc/r3.5.5/zookeeperReconfig.html
>>
>> Alex
>>
>> On Wed, Aug 21, 2019 at 7:29 AM Gao,Wei  wrote:
>>
>>> Hi
>>>I encounter a problem which blocks my development of load balance
using
>>> ZooKeeper 3.5.5.
>>>Actually, I have a ZooKeeper cluster which comprises of five zk
>>> servers. And the dynamic configuration file is as follows:
>>>
>>>   server.1=zk1:2888:3888:participant;0.0.0.0:2181
>>>   server.2=zk2:2888:3888:participant;0.0.0.0:2181
>>>   server.3=zk3:2888:3888:participant;0.0.0.0:2181
>>>   server.4=zk4:2888:3888:participant;0.0.0.0:2181
>>>   server.5=zk5:2888:3888:participant;0.0.0.0:2181
>>>
>>>   The zk cluster can work fine if every member works normally.
However, if
>>> say two of them are suddenly down without previously being notified,
>>> the dynamic configuration file shown above will not be synchronized
>>> dynamically, which leads to the zk cluster fail to work normally.
>>>   I think this is a very common case which may happen at any time. If
so,
>>> how can we resolve it?
>>>   Really look forward to hearing from you!
>>> Thanks
>>>

Re: About ZooKeeper Dynamic Reconfiguration

[Cee Tee]

Yes, one side loses quorum and the other remains active. However we 
actively control which side that is, because our main application is 
active/passive with 2 datacenters. We need Zookeeper to remain active in 
the applications active datacenter.


On 21 August 2019 17:22:00 Alexander Shraer  wrote:

That's great! Thanks for sharing.



Added benefit is that we can also control which data center gets the quorum
in case of a network outage between the two.



Can you explain how this works? In case of a network outage between two 
DCs, one of them has a quorum of participants and the other doesn't.
The participants in the smaller set should not be operational at this time, 
since they can't get quorum. no ?




Thanks,
Alex


On Wed, Aug 21, 2019 at 7:55 AM Cee Tee  wrote:

We have solved this by implementing a 'zookeeper cluster balancer', it
calls the admin server api of each zookeeper to get the current status and
will issue dynamic reconfigure commands to change dead servers into
observers so the quorum is not in danger. Once the dead servers reconnect,
they take the observer role and are then reconfigured into participants again.

Added benefit is that we can also control which data center gets the quorum
in case of a network outage between the two.
Regards
Chris

On 21 August 2019 16:42:37 Alexander Shraer  wrote:


Hi,

Reconfiguration, as implemented, is not automatic. In your case, when
failures happen, this doesn't change the ensemble membership.
When 2 of 5 fail, this is still a minority, so everything should work
normally, you just won't be able to handle an additional failure. If you'd
like
to remove them from the ensemble, you need to issue an explicit
reconfiguration command to do so.

Please see details in the manual:
https://zookeeper.apache.org/doc/r3.5.5/zookeeperReconfig.html

Alex

On Wed, Aug 21, 2019 at 7:29 AM Gao,Wei  wrote:


Hi
   I encounter a problem which blocks my development of load balance using
ZooKeeper 3.5.5.
   Actually, I have a ZooKeeper cluster which comprises of five zk
servers. And the dynamic configuration file is as follows:

  server.1=zk1:2888:3888:participant;0.0.0.0:2181
  server.2=zk2:2888:3888:participant;0.0.0.0:2181
  server.3=zk3:2888:3888:participant;0.0.0.0:2181
  server.4=zk4:2888:3888:participant;0.0.0.0:2181
  server.5=zk5:2888:3888:participant;0.0.0.0:2181

  The zk cluster can work fine if every member works normally. However, if
say two of them are suddenly down without previously being notified,
the dynamic configuration file shown above will not be synchronized
dynamically, which leads to the zk cluster fail to work normally.
  I think this is a very common case which may happen at any time. If so,
how can we resolve it?
  Really look forward to hearing from you!
Thanks

Re: About ZooKeeper Dynamic Reconfiguration

[Cee Tee]

We have solved this by implementing a 'zookeeper cluster balancer', it 
calls the admin server api of each zookeeper to get the current status and 
will issue dynamic reconfigure commands to change dead servers into 
observers so the quorum is not in danger. Once the dead servers reconnect, 
they take the observer role and are then reconfigured into participants again.


Added benefit is that we can also control which data center gets the quorum 
in case of a network outage between the two.

Regards
Chris

On 21 August 2019 16:42:37 Alexander Shraer  wrote:


Hi,

Reconfiguration, as implemented, is not automatic. In your case, when
failures happen, this doesn't change the ensemble membership.
When 2 of 5 fail, this is still a minority, so everything should work
normally, you just won't be able to handle an additional failure. If you'd
like
to remove them from the ensemble, you need to issue an explicit
reconfiguration command to do so.

Please see details in the manual:
https://zookeeper.apache.org/doc/r3.5.5/zookeeperReconfig.html

Alex

On Wed, Aug 21, 2019 at 7:29 AM Gao,Wei  wrote:


Hi
   I encounter a problem which blocks my development of load balance using
ZooKeeper 3.5.5.
   Actually, I have a ZooKeeper cluster which comprises of five zk
servers. And the dynamic configuration file is as follows:

  server.1=zk1:2888:3888:participant;0.0.0.0:2181
  server.2=zk2:2888:3888:participant;0.0.0.0:2181
  server.3=zk3:2888:3888:participant;0.0.0.0:2181
  server.4=zk4:2888:3888:participant;0.0.0.0:2181
  server.5=zk5:2888:3888:participant;0.0.0.0:2181

  The zk cluster can work fine if every member works normally. However, if
say two of them are suddenly down without previously being notified,
the dynamic configuration file shown above will not be synchronized
dynamically, which leads to the zk cluster fail to work normally.
  I think this is a very common case which may happen at any time. If so,
how can we resolve it?
  Really look forward to hearing from you!
Thanks

Re: Leader election failing

[Cee Tee]

Quick update:
Apparently the election notifications disappeared somewhere between the
datacenters (firewall) when the sockets were not used for some time. We
fixed this with zookeeper.tcpKeepAlive=true.

Regards,

Chris

On Wed, Aug 8, 2018 at 5:05 PM Andor Molnar 
wrote:

> Some kind of a network split?
>
> It looks like 1-2 and 3-4 were able to communicate each other, but
> connection timed out between these 2 splits. When 5 came back online it
> started with supporters of (1,2) and later 3 and 4 also joined.
>
> There was no such issue the day after.
>
> Which version of ZooKeeper is this? 3.5.something?
>
> Regards,
> Andor
>
>
>
> On Wed, Aug 8, 2018 at 4:52 PM, Chris  wrote:
>
> > Actually i have similar issues on my test and acceptance clusters where
> > leader election fails if the cluster has been running for a couple of
> days.
> > If you stop/start the Zookeepers once they will work fine on further
> > disruptions that day. Not sure yet what the treshold is.
> >
> >
> > On 8 August 2018 4:32:56 pm Camille Fournier  wrote:
> >
> > Hard to say. It looks like about 15 minutes after your first incident
> where
> >> 5 goes down and then comes back up, servers 1 and 2 get socket errors to
> >> their connections with 3, 4, and 6. It's possible if you had waited
> those
> >> 15 minutes, once those errors cleared the quorum would've formed with
> the
> >> other servers. But as for why there were those errors in the first place
> >> it's not clear. Could be a network glitch, or an obscure bug in the
> >> connection logic. Has anyone else ever seen this?
> >> If you see it again, getting a stack trace of the servers when they
> can't
> >> form quorum might be helpful.
> >>
> >> On Wed, Aug 8, 2018 at 11:52 AM Cee Tee  wrote:
> >>
> >> I have a cluster of 5 participants (id 1-5) and 1 observer (id 6).
> >>> 1,2,5 are in datacenter A. 3,4,6 are in datacenter B.
> >>> Yesterday one of the participants (id5, by chance was the leader) was
> >>> rebooted. Although all other servers were online and not suffering from
> >>> networking issues the leader election failed and the cluster remained
> >>> "looking" until the old leader came back online after which it was
> >>> promptly
> >>> elected as leader again.
> >>>
> >>> Today we tried the same exercise on the exact same servers, 5 was still
> >>> leader and was rebooted, and leader election worked fine with 4 as new
> >>> leader.
> >>>
> >>> I have included the logs.  From the logs i see that yesterday 1,2 never
> >>> received new leader proposals from 3,4 and vice versa.
> >>> Today all proposals came through. This is not the first time we've seen
> >>> this type of behavior, where some zookeepers can't seem to find each
> >>> other
> >>> after the leader goes down.
> >>> All servers use dynamic configuration and have the same config node.
> >>>
> >>> How could this be explained? These servers also host a replicated
> >>> database
> >>> cluster and have no history of db replication issues.
> >>>
> >>> Thanks,
> >>> Chris
> >>>
> >>>
> >>>
> >>>
> >
> >
> >
>

Re: Leader election failing

[Cee Tee]

Concluded a test with a 3.4.13 cluster, it shows the same behaviour.

On Mon, Sep 3, 2018 at 4:56 PM Andor Molnar 
wrote:

> Thanks for testing Chris.
>
> So, if I understand you correctly, you're running the latest version from
> branch-3.5. Could we say that this is a 3.5-only problem?
> Have you ever tested the same cluster with 3.4?
>
> Regards,
> Andor
>
>
>
> On Tue, Aug 21, 2018 at 11:29 AM, Cee Tee  wrote:
>
> > I've tested the patch and let it run 6 days. It did not help, result is
> > still the same. (remaining ZKs form islands based on datacenter they are
> > in).
> >
> > I have mitigated it by doing a daily rolling restart.
> >
> > Regards,
> > Chris
> >
> > On Mon, Aug 13, 2018 at 2:06 PM Andor Molnar  >
> > wrote:
> >
> > > Hi Chris,
> > >
> > > Would you mind testing the following patch on your test clusters?
> > > I'm not entirely sure, but the issue might be related.
> > >
> > > https://issues.apache.org/jira/browse/ZOOKEEPER-2930
> > >
> > > Regards,
> > > Andor
> > >
> > >
> > >
> > > On Wed, Aug 8, 2018 at 6:51 PM, Camille Fournier 
> > > wrote:
> > >
> > > > If you have the time and inclination, next time you see this problem
> in
> > > > your test clusters get stack traces and any other diagnostics
> possible
> > > > before restarting. I'm not an expert at network debugging but if you
> > have
> > > > someone who is you might want them to take a look at the connections
> > and
> > > > settings of any switches/firewalls/etc involved, see if there's any
> > > unusual
> > > > configurations or evidence of other long-lived connections failing
> > (even
> > > if
> > > > their services handle the failures more gracefully). Send us the
> stack
> > > > traces also it would be interesting to take a look.
> > > >
> > > > C
> > > >
> > > >
> > > > On Wed, Aug 8, 2018, 11:09 AM Chris  wrote:
> > > >
> > > > > Running 3.5.5
> > > > >
> > > > > I managed to recreate it on acc and test cluster today, failing on
> > > > > shutdown
> > > > > of leader. Both had been running for over a week. After restarting
> > all
> > > > > zookeepers it runs fine no matter how many leader shutdowns i throw
> > at
> > > > it.
> > > > >
> > > > > On 8 August 2018 5:05:34 pm Andor Molnar
>  > >
> > > > > wrote:
> > > > >
> > > > > > Some kind of a network split?
> > > > > >
> > > > > > It looks like 1-2 and 3-4 were able to communicate each other,
> but
> > > > > > connection timed out between these 2 splits. When 5 came back
> > online
> > > it
> > > > > > started with supporters of (1,2) and later 3 and 4 also joined.
> > > > > >
> > > > > > There was no such issue the day after.
> > > > > >
> > > > > > Which version of ZooKeeper is this? 3.5.something?
> > > > > >
> > > > > > Regards,
> > > > > > Andor
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Aug 8, 2018 at 4:52 PM, Chris 
> > wrote:
> > > > > >
> > > > > >> Actually i have similar issues on my test and acceptance
> clusters
> > > > where
> > > > > >> leader election fails if the cluster has been running for a
> couple
> > > of
> > > > > days.
> > > > > >> If you stop/start the Zookeepers once they will work fine on
> > further
> > > > > >> disruptions that day. Not sure yet what the treshold is.
> > > > > >>
> > > > > >>
> > > > > >> On 8 August 2018 4:32:56 pm Camille Fournier <
> cami...@apache.org>
> > > > > wrote:
> > > > > >>
> > > > > >> Hard to say. It looks like about 15 minutes after your first
> > > incident
> > > > > where
> > > > > >>> 5 goes down and then comes back up, servers 1 and 2 get socket
> > > errors
> > > > > to
> > > > > >>> their connections with 3, 4, and 6. It's possible if you had
> > waited
> > &

Re: Leader election failing

[Cee Tee]

I've tested the patch and let it run 6 days. It did not help, result is
still the same. (remaining ZKs form islands based on datacenter they are
in).

I have mitigated it by doing a daily rolling restart.

Regards,
Chris

On Mon, Aug 13, 2018 at 2:06 PM Andor Molnar 
wrote:

> Hi Chris,
>
> Would you mind testing the following patch on your test clusters?
> I'm not entirely sure, but the issue might be related.
>
> https://issues.apache.org/jira/browse/ZOOKEEPER-2930
>
> Regards,
> Andor
>
>
>
> On Wed, Aug 8, 2018 at 6:51 PM, Camille Fournier 
> wrote:
>
> > If you have the time and inclination, next time you see this problem in
> > your test clusters get stack traces and any other diagnostics possible
> > before restarting. I'm not an expert at network debugging but if you have
> > someone who is you might want them to take a look at the connections and
> > settings of any switches/firewalls/etc involved, see if there's any
> unusual
> > configurations or evidence of other long-lived connections failing (even
> if
> > their services handle the failures more gracefully). Send us the stack
> > traces also it would be interesting to take a look.
> >
> > C
> >
> >
> > On Wed, Aug 8, 2018, 11:09 AM Chris  wrote:
> >
> > > Running 3.5.5
> > >
> > > I managed to recreate it on acc and test cluster today, failing on
> > > shutdown
> > > of leader. Both had been running for over a week. After restarting all
> > > zookeepers it runs fine no matter how many leader shutdowns i throw at
> > it.
> > >
> > > On 8 August 2018 5:05:34 pm Andor Molnar 
> > > wrote:
> > >
> > > > Some kind of a network split?
> > > >
> > > > It looks like 1-2 and 3-4 were able to communicate each other, but
> > > > connection timed out between these 2 splits. When 5 came back online
> it
> > > > started with supporters of (1,2) and later 3 and 4 also joined.
> > > >
> > > > There was no such issue the day after.
> > > >
> > > > Which version of ZooKeeper is this? 3.5.something?
> > > >
> > > > Regards,
> > > > Andor
> > > >
> > > >
> > > >
> > > > On Wed, Aug 8, 2018 at 4:52 PM, Chris  wrote:
> > > >
> > > >> Actually i have similar issues on my test and acceptance clusters
> > where
> > > >> leader election fails if the cluster has been running for a couple
> of
> > > days.
> > > >> If you stop/start the Zookeepers once they will work fine on further
> > > >> disruptions that day. Not sure yet what the treshold is.
> > > >>
> > > >>
> > > >> On 8 August 2018 4:32:56 pm Camille Fournier 
> > > wrote:
> > > >>
> > > >> Hard to say. It looks like about 15 minutes after your first
> incident
> > > where
> > > >>> 5 goes down and then comes back up, servers 1 and 2 get socket
> errors
> > > to
> > > >>> their connections with 3, 4, and 6. It's possible if you had waited
> > > those
> > > >>> 15 minutes, once those errors cleared the quorum would've formed
> with
> > > the
> > > >>> other servers. But as for why there were those errors in the first
> > > place
> > > >>> it's not clear. Could be a network glitch, or an obscure bug in the
> > > >>> connection logic. Has anyone else ever seen this?
> > > >>> If you see it again, getting a stack trace of the servers when they
> > > can't
> > > >>> form quorum might be helpful.
> > > >>>
> > > >>> On Wed, Aug 8, 2018 at 11:52 AM Cee Tee 
> > wrote:
> > > >>>
> > > >>> I have a cluster of 5 participants (id 1-5) and 1 observer (id 6).
> > > >>>> 1,2,5 are in datacenter A. 3,4,6 are in datacenter B.
> > > >>>> Yesterday one of the participants (id5, by chance was the leader)
> > was
> > > >>>> rebooted. Although all other servers were online and not suffering
> > > from
> > > >>>> networking issues the leader election failed and the cluster
> > remained
> > > >>>> "looking" until the old leader came back online after which it was
> > > >>>> promptly
> > > >>>> elected as leader again.
> > > >>>>
> > > >>>> Today we tried the same exercise on the exact same servers, 5 was
> > > still
> > > >>>> leader and was rebooted, and leader election worked fine with 4 as
> > new
> > > >>>> leader.
> > > >>>>
> > > >>>> I have included the logs.  From the logs i see that yesterday 1,2
> > > never
> > > >>>> received new leader proposals from 3,4 and vice versa.
> > > >>>> Today all proposals came through. This is not the first time we've
> > > seen
> > > >>>> this type of behavior, where some zookeepers can't seem to find
> each
> > > >>>> other
> > > >>>> after the leader goes down.
> > > >>>> All servers use dynamic configuration and have the same config
> node.
> > > >>>>
> > > >>>> How could this be explained? These servers also host a replicated
> > > >>>> database
> > > >>>> cluster and have no history of db replication issues.
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Chris
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>
> > > >>
> > > >>
> > >
> > >
> > >
> > >
> >
>