Re: Verify zookeeper cluster consistency
iirc Ben (or was it someone else from Facebook) mentioned at a meetup about doing some work in this area. I believe they were hashing the tree but I may be mis-remembering Perhaps one of those folks can shed light. Patrick On Fri, Jul 27, 2018 at 2:04 AM Andor Molnar wrote: > Hi Klearchos, > > Some tool which is constantly querying the znode tree on each of every > participant and tolerates some latency between the inconsistencies, but > reports everything above the threshold could do the trick. > > Personally I'm not aware of anything existing, but feel free to dig github > or create your own one. > > Regards, > Andor > > > > > On Fri, Jul 27, 2018 at 9:16 AM, Chaloulos, Klearchos (Nokia - GR/Athens) < > klearchos.chalou...@nokia.com> wrote: > > > Hello, > > > > I recently ran into a case where my zookeeper servers had inconsistent > > data, that is nodes that existed in one server did not exist in the other > > two. I used 3 servers, with version 3.4.9. Unfortunately I cannot > reproduce > > the issue, as it occurred a long time ago in an environment outside my > > control. > > > > However my question is: Is there a way to detect zookeeper cluster > > inconsistencies? I am not talking about transient inconsistencies that > > exist for, say, a few seconds, but for permanent inconsistencies. In my > > case the inconsistencies persisted for a month! > > > > Best regards, > > > > Klearchos > > > > >
Re: Verify zookeeper cluster consistency
Hi Klearchos, Some tool which is constantly querying the znode tree on each of every participant and tolerates some latency between the inconsistencies, but reports everything above the threshold could do the trick. Personally I'm not aware of anything existing, but feel free to dig github or create your own one. Regards, Andor On Fri, Jul 27, 2018 at 9:16 AM, Chaloulos, Klearchos (Nokia - GR/Athens) < klearchos.chalou...@nokia.com> wrote: > Hello, > > I recently ran into a case where my zookeeper servers had inconsistent > data, that is nodes that existed in one server did not exist in the other > two. I used 3 servers, with version 3.4.9. Unfortunately I cannot reproduce > the issue, as it occurred a long time ago in an environment outside my > control. > > However my question is: Is there a way to detect zookeeper cluster > inconsistencies? I am not talking about transient inconsistencies that > exist for, say, a few seconds, but for permanent inconsistencies. In my > case the inconsistencies persisted for a month! > > Best regards, > > Klearchos > >
Verify zookeeper cluster consistency
Hello, I recently ran into a case where my zookeeper servers had inconsistent data, that is nodes that existed in one server did not exist in the other two. I used 3 servers, with version 3.4.9. Unfortunately I cannot reproduce the issue, as it occurred a long time ago in an environment outside my control. However my question is: Is there a way to detect zookeeper cluster inconsistencies? I am not talking about transient inconsistencies that exist for, say, a few seconds, but for permanent inconsistencies. In my case the inconsistencies persisted for a month! Best regards, Klearchos
