Re: Please help! ZooKeeper 3.7.2 fsync-ing latency issue

[Patrick Hunt]

On Thu, Apr 18, 2024 at 9:15 AM Patrick Hunt  wrote:

> My experience with slow fsyncs is that it's almost always due to
> contention for disk IO. I see that you tuned the snap* sizes down, which is
> reasonable. You might check what ZK activity is happening during this
> period? Perhaps some client is hammering the cluster, have you ruled
> that out?
>
>
Actually one other thing (sorry - it's been a while since I have seen this)
could be GC activity. If something (eg my point about client activity due
to some periodic event...) causes lots of memory pressure, perhaps the GC
is somehow impacting the fsync (or the activity around the fsync). Have you
tried running with GC tracking and see if that's related to the event?

Patrick


> I searched the mail archives, there are other folks reporting this issue,
> you might take a look. I found this one in particular that you might
> checkout:
> https://lists.apache.org/thread/qjrlprmt7pdy63ztvjtvkd0f5zgw5dgk
>
> Patrick
>
> On Thu, Apr 18, 2024 at 3:31 AM Xu Bill  wrote:
>
>> Hello,
>>
>> I have a pretty weird issue of ZooKeeper.
>> Everyday around 17:30, my ZooKeeper throws a warning message in log says
>> "fsync-ing the write ahead log in SyncThread:0 took 36919ms which will
>> adversely effect operation latency.File size is 16777232 bytes.". And this
>> causes my clients connected to ZooKeeper being timed out. I have to restart
>> my clients every day.
>>
>> Though I don't think the size of the txn log file is too big to be
>> handled quickly,
>> still I tried to change parameters to supress the size of txn log. Below
>> is my configuration.
>> preAllocSize=16M
>> snapCount=3
>> snapSizeLimitInKb=32M
>>
>> Even with this configuration, I still got the warnings.
>>
>> I also tried to monitor the IO stats on data disk which the data dir of
>> ZooKeeper is in.
>> But the stats were as the same as usual.
>>
>> Can anybody help give suggestions on how to solve or investigate on this
>> issue?
>> I am using ZooKeeper 3.7.2.
>> The IO stats were tps=122, reading=20.1k/s, writing=2M/s, when the
>> warning was happening.
>>
>> Best regards,
>> Bill
>>
>

Re: Please help! ZooKeeper 3.7.2 fsync-ing latency issue

[Patrick Hunt]

My experience with slow fsyncs is that it's almost always due to contention
for disk IO. I see that you tuned the snap* sizes down, which is
reasonable. You might check what ZK activity is happening during this
period? Perhaps some client is hammering the cluster, have you ruled
that out?

I searched the mail archives, there are other folks reporting this issue,
you might take a look. I found this one in particular that you might
checkout:
https://lists.apache.org/thread/qjrlprmt7pdy63ztvjtvkd0f5zgw5dgk

Patrick

On Thu, Apr 18, 2024 at 3:31 AM Xu Bill  wrote:

> Hello,
>
> I have a pretty weird issue of ZooKeeper.
> Everyday around 17:30, my ZooKeeper throws a warning message in log says
> "fsync-ing the write ahead log in SyncThread:0 took 36919ms which will
> adversely effect operation latency.File size is 16777232 bytes.". And this
> causes my clients connected to ZooKeeper being timed out. I have to restart
> my clients every day.
>
> Though I don't think the size of the txn log file is too big to be handled
> quickly,
> still I tried to change parameters to supress the size of txn log. Below
> is my configuration.
> preAllocSize=16M
> snapCount=3
> snapSizeLimitInKb=32M
>
> Even with this configuration, I still got the warnings.
>
> I also tried to monitor the IO stats on data disk which the data dir of
> ZooKeeper is in.
> But the stats were as the same as usual.
>
> Can anybody help give suggestions on how to solve or investigate on this
> issue?
> I am using ZooKeeper 3.7.2.
> The IO stats were tps=122, reading=20.1k/s, writing=2M/s, when the warning
> was happening.
>
> Best regards,
> Bill
>

Re: Convert Standalone zookeeper to A 3 node Quorum

[Patrick Hunt]

It really depends on your requirements. You should read the admin docs for
insight and examples, start here:
https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_designing

That said, I have a project here which I use for config generation/testing.
YMMV:
https://github.com/phunt/zkconf

Patrick

On Mon, Jun 19, 2023 at 11:32 AM Gaurav Pande  wrote:

> Okay thanks, and could you share what's the standard config for ensemble as
> example? I mean what all parameters it includes.
>
>  Also do I need to add standaloneEnabled=false on these 2 new zk servers
> config file as a part of initial base ensemble config as by default this
> parameter value  is true?  So that Making 2 zk's  run in replicated mode
> from starting (starting one at a time)
>
> Regards,
> Gaurav
>
>
> On Mon, 19 Jun, 2023, 22:03 Patrick Hunt,  wrote:
>
> > On Mon, Jun 19, 2023 at 8:47 AM Gaurav Pande 
> wrote:
> >
> > > Hi Patrick,
> > >
> > > Thanks for guidance here , based on below I only have  presently 1 zk
> > node
> > > so if I provision 2 new VM and install same version of zookeeper on
> them
> > > should I start them as standalone zookeepers first and then make
> changes
> > to
> > > there server config file?
> > >
> > >
> > No, you specifically want to add them as part of the ensemble, one at a
> > time in sequence (see 2).
> >
> >
> > > Also what are the valid zk configs that I would need to add on these 2
> > new
> > > VM's zk server config file?
> > >
> > >
> > You'll need a regular ensemble config, not a standalone.
> >
> > Patrick
> >
> >
> > > Regards,
> > > Gaurav
> > >
> > > On Mon, 19 Jun, 2023, 21:06 Patrick Hunt,  wrote:
> > >
> > > > Two ways to do it come to mind, which I've used in the past:
> > > >
> > > > 1) most straightforward is to "clone" the repos for the two new
> members
> > > of
> > > > the ensemble in their respective configs/datadirs. Just make sure to
> > > update
> > > > the configs appropriately. Including the "myid" for each server. Then
> > > > restart the ensemble and verify.
> > > >
> > > > 2) You can't go from 1->3 servers just by adding two new servers to
> the
> > > > ensemble as they may form a quorum on the "zero" zxid, rather than
> the
> > > zxid
> > > > of the existing member. Rather you would need to go from 1->2, with
> > > quorum,
> > > > and then from 2->3 with quorum. This will ensure that the true state
> of
> > > the
> > > > original quorum is maintained (this is what we implemented for
> Cloudera
> > > > Manager to ensure proper functioning when increasing quorum size).
> > > >
> > > > Good luck,
> > > >
> > > > Patrick
> > > >
> > > >
> > > > On Mon, Jun 19, 2023 at 12:02 AM Gaurav Pande 
> > > > wrote:
> > > >
> > > > > Hi tison,
> > > > >
> > > > > When you say stop services you mean existing standalone Zk service
> > > right?
> > > > > , if that's the case then yes we can. But what's the process ? Also
> > can
> > > > it
> > > > > be done without re-start I didn't know , can you share both
> > > > process/steps?
> > > > >
> > > > > On Mon, 19 Jun, 2023, 10:15 tison,  wrote:
> > > > >
> > > > > > Can you stop the services for reconfig, or you need an online
> > > reconfig?
> > > > > >
> > > > > > Best,
> > > > > > tison.
> > > > > >
> > > > > >
> > > > > > Gaurav Pande  于2023年6月19日周一 11:22写道：
> > > > > >
> > > > > > > Hi Guys,
> > > > > > >
> > > > > > > Any help on this thread please?
> > > > > > >
> > > > > > > Regards,
> > > > > > > Gaurav
> > > > > > >
> > > > > > > On Sun, 18 Jun, 2023, 20:14 Gaurav Pande, <
> gaupand...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hello Guys,
> > > > > > > >
> > > > > > > > Iam new in this space,  I wanted to know process/steps to
> > > convert a
> > > > > > > single
> > > > > > > > Zk node in standalone presently to a 3 node ZK quorum .
> > > > > > > >
> > > > > > > > What steps should I perform to add make my standalone Zk
> node a
> > > > > Quorum
> > > > > > of
> > > > > > > > 3 zk nodes?
> > > > > > > >
> > > > > > > > I am using ZK version 3.5.8 presently.
> > > > > > > >
> > > > > > > > Note - I have single Zookeeper(Standalone) and 3 Kafka
> brokers
> > at
> > > > > this
> > > > > > > > point.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Gaurav
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Convert Standalone zookeeper to A 3 node Quorum

[Patrick Hunt]

On Mon, Jun 19, 2023 at 8:47 AM Gaurav Pande  wrote:

> Hi Patrick,
>
> Thanks for guidance here , based on below I only have  presently 1 zk node
> so if I provision 2 new VM and install same version of zookeeper on them
> should I start them as standalone zookeepers first and then make changes to
> there server config file?
>
>
No, you specifically want to add them as part of the ensemble, one at a
time in sequence (see 2).


> Also what are the valid zk configs that I would need to add on these 2 new
> VM's zk server config file?
>
>
You'll need a regular ensemble config, not a standalone.

Patrick


> Regards,
> Gaurav
>
> On Mon, 19 Jun, 2023, 21:06 Patrick Hunt,  wrote:
>
> > Two ways to do it come to mind, which I've used in the past:
> >
> > 1) most straightforward is to "clone" the repos for the two new members
> of
> > the ensemble in their respective configs/datadirs. Just make sure to
> update
> > the configs appropriately. Including the "myid" for each server. Then
> > restart the ensemble and verify.
> >
> > 2) You can't go from 1->3 servers just by adding two new servers to the
> > ensemble as they may form a quorum on the "zero" zxid, rather than the
> zxid
> > of the existing member. Rather you would need to go from 1->2, with
> quorum,
> > and then from 2->3 with quorum. This will ensure that the true state of
> the
> > original quorum is maintained (this is what we implemented for Cloudera
> > Manager to ensure proper functioning when increasing quorum size).
> >
> > Good luck,
> >
> > Patrick
> >
> >
> > On Mon, Jun 19, 2023 at 12:02 AM Gaurav Pande 
> > wrote:
> >
> > > Hi tison,
> > >
> > > When you say stop services you mean existing standalone Zk service
> right?
> > > , if that's the case then yes we can. But what's the process ? Also can
> > it
> > > be done without re-start I didn't know , can you share both
> > process/steps?
> > >
> > > On Mon, 19 Jun, 2023, 10:15 tison,  wrote:
> > >
> > > > Can you stop the services for reconfig, or you need an online
> reconfig?
> > > >
> > > > Best,
> > > > tison.
> > > >
> > > >
> > > > Gaurav Pande  于2023年6月19日周一 11:22写道：
> > > >
> > > > > Hi Guys,
> > > > >
> > > > > Any help on this thread please?
> > > > >
> > > > > Regards,
> > > > > Gaurav
> > > > >
> > > > > On Sun, 18 Jun, 2023, 20:14 Gaurav Pande, 
> > > wrote:
> > > > >
> > > > > > Hello Guys,
> > > > > >
> > > > > > Iam new in this space,  I wanted to know process/steps to
> convert a
> > > > > single
> > > > > > Zk node in standalone presently to a 3 node ZK quorum .
> > > > > >
> > > > > > What steps should I perform to add make my standalone Zk node a
> > > Quorum
> > > > of
> > > > > > 3 zk nodes?
> > > > > >
> > > > > > I am using ZK version 3.5.8 presently.
> > > > > >
> > > > > > Note - I have single Zookeeper(Standalone) and 3 Kafka brokers at
> > > this
> > > > > > point.
> > > > > >
> > > > > > Regards,
> > > > > > Gaurav
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Convert Standalone zookeeper to A 3 node Quorum

[Patrick Hunt]

Two ways to do it come to mind, which I've used in the past:

1) most straightforward is to "clone" the repos for the two new members of
the ensemble in their respective configs/datadirs. Just make sure to update
the configs appropriately. Including the "myid" for each server. Then
restart the ensemble and verify.

2) You can't go from 1->3 servers just by adding two new servers to the
ensemble as they may form a quorum on the "zero" zxid, rather than the zxid
of the existing member. Rather you would need to go from 1->2, with quorum,
and then from 2->3 with quorum. This will ensure that the true state of the
original quorum is maintained (this is what we implemented for Cloudera
Manager to ensure proper functioning when increasing quorum size).

Good luck,

Patrick


On Mon, Jun 19, 2023 at 12:02 AM Gaurav Pande  wrote:

> Hi tison,
>
> When you say stop services you mean existing standalone Zk service right?
> , if that's the case then yes we can. But what's the process ? Also can it
> be done without re-start I didn't know , can you share both process/steps?
>
> On Mon, 19 Jun, 2023, 10:15 tison,  wrote:
>
> > Can you stop the services for reconfig, or you need an online reconfig?
> >
> > Best,
> > tison.
> >
> >
> > Gaurav Pande  于2023年6月19日周一 11:22写道：
> >
> > > Hi Guys,
> > >
> > > Any help on this thread please?
> > >
> > > Regards,
> > > Gaurav
> > >
> > > On Sun, 18 Jun, 2023, 20:14 Gaurav Pande, 
> wrote:
> > >
> > > > Hello Guys,
> > > >
> > > > Iam new in this space,  I wanted to know process/steps to convert a
> > > single
> > > > Zk node in standalone presently to a 3 node ZK quorum .
> > > >
> > > > What steps should I perform to add make my standalone Zk node a
> Quorum
> > of
> > > > 3 zk nodes?
> > > >
> > > > I am using ZK version 3.5.8 presently.
> > > >
> > > > Note - I have single Zookeeper(Standalone) and 3 Kafka brokers at
> this
> > > > point.
> > > >
> > > > Regards,
> > > > Gaurav
> > > >
> > > >
> > >
> >
>

Re: Impact of Log4J security vulnerability CVE-2021-44228 on zookeeper

[Patrick Hunt]

Hi Brent, there is a discussion going on on the dev list with subject
"Logback" if folks would like to participate.

Regards,

Patrick


On Fri, Dec 17, 2021 at 11:26 PM Brent  wrote:

> I just finished reading through the latest Jira comments and links.  Has
> there been any consensus reached thus far on whether or not ZK is planning
> an upgrade to Log4j 2.16.0 (or at least providing it as an option behind
> slf4j)?
>
> I understand the arguments for/against Log4j 1.x and won't repeat them all
> here.  I'm asking because I know some more cautious organizations are still
> taking action to attempt to mitigate existing ZK installations regardless.
>
> Has anyone made an attempt to see how much effort would be involved in the
> upgrade?  Would you all be open to a pull request?
>
> Thanks for all of your hard work on ZK!
>
> ~Brent
>
> On Mon, Dec 13, 2021 at 8:36 AM Patrick Hunt  wrote:
>
> > This issue is being tracked on ZOOKEEPER-4423.
> >
> > ZK 3.4 does not use log4j 2.x - all versions of zk currently use log4j
> 1.x.
> >
> > Regards,
> >
> > Patrick
> >
> >
> > On Mon, Dec 13, 2021 at 4:02 AM Prasanna kumar <
> > prasannakumarram...@gmail.com> wrote:
> >
> > > Could anyone confirm the same on 3.4 versions?
> > >
> > > On Sun, Dec 12, 2021 at 9:58 AM tison  wrote:
> > >
> > > > Hi Anchal,
> > > >
> > > > I don't speak on behalf of the PMC but it seems ZK just uses log4j
> 1.x,
> > > not
> > > > the affected version.
> > > >
> > > > Best,
> > > > tison.
> > > >
> > > >
> > > > Anchal Sharma2  于2021年12月12日周日 12:19写道：
> > > >
> > > > > Hi All,
> > > > >
> > > > > Any one knows impact of Log4J security vulnerability CVE-2021-44228
> > on
> > > > > zookeeper (version 3.5.8) and mitigation ?I couldn't find any news
> on
> > > > > zookeeper website .
> > > > >
> > > > > Thanks
> > > > > Anchal Sharma
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: Impact of Log4J security vulnerability CVE-2021-44228 on zookeeper

[Patrick Hunt]

This issue is being tracked on ZOOKEEPER-4423.

ZK 3.4 does not use log4j 2.x - all versions of zk currently use log4j 1.x.

Regards,

Patrick


On Mon, Dec 13, 2021 at 4:02 AM Prasanna kumar <
prasannakumarram...@gmail.com> wrote:

> Could anyone confirm the same on 3.4 versions?
>
> On Sun, Dec 12, 2021 at 9:58 AM tison  wrote:
>
> > Hi Anchal,
> >
> > I don't speak on behalf of the PMC but it seems ZK just uses log4j 1.x,
> not
> > the affected version.
> >
> > Best,
> > tison.
> >
> >
> > Anchal Sharma2  于2021年12月12日周日 12:19写道：
> >
> > > Hi All,
> > >
> > > Any one knows impact of Log4J security vulnerability CVE-2021-44228  on
> > > zookeeper (version 3.5.8) and mitigation ?I couldn't find any news on
> > > zookeeper website .
> > >
> > > Thanks
> > > Anchal Sharma
> > >
> > >
> >
>

Re: write performance issue in 3.6.2

[Patrick Hunt]

On Sun, Feb 21, 2021 at 3:28 PM Li Wang  wrote:

> Hi Enrico, Sushant,
>
> I re-run the perf test with the data consistency check feature disabled
> (i.e. -Dzookeeper.digest.enabled=false), the write performance issue of 3.6
> is still there.
>
> With everything exactly the same, the throughput of 3.6 was only 1/2 of 3.4
> and the max latency was more than 8 times.
>
> Any other points or thoughts?
>
>
In the past I've noticed a big impact of GC when doing certain performance
measurements. I assume you are using the same JVM version and GC when
running the two tests? Perhaps our memory footprint has expanded over time.
You should rule out GC by running with gc logging turned on with both
versions and compare the impact.

Regards,

Patrick


> Cheers,
>
> Li
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Sat, Feb 20, 2021 at 9:04 PM Li Wang  wrote:
>
> > Thanks Sushant and Enrico!
> >
> > This is a really good point.  According to the 3.6 documentation, the
> > feature is disabled by default.
> >
> https://zookeeper.apache.org/doc/r3.6.2/zookeeperAdmin.html#ch_administration
> .
> > However, checking the code, the default is enabled.
> >
> > Let me set the zookeeper.digest.enabled to false and see how the write
> > operation performs.
> >
> > Best,
> >
> > Li
> >
> >
> >
> >
> > On Fri, Feb 19, 2021 at 1:32 PM Sushant Mane 
> > wrote:
> >
> >> Hi Li,
> >>
> >> On 3.6.2 consistency checker (adhash based) is enabled by default:
> >>
> >>
> https://github.com/apache/zookeeper/blob/803c7f1a12f85978cb049af5e4ef23bd8b688715/zookeeper-server/src/main/java/org/apache/zookeeper/server/ZooKeeperServer.java#L136
> >> .
> >> It is not present in ZK 3.4.14.
> >>
> >> This feature does have some impact on write performance.
> >>
> >> Thanks,
> >> Sushant
> >>
> >>
> >> On Fri, Feb 19, 2021 at 12:50 PM Enrico Olivelli 
> >> wrote:
> >>
> >> > Li,
> >> > I wonder of we have some new throttling/back pressure mechanisms that
> is
> >> > enabled by default.
> >> >
> >> > Does anyone has some pointer to relevant implementations?
> >> >
> >> >
> >> > Enrico
> >> >
> >> > Il Ven 19 Feb 2021, 19:46 Li Wang  ha scritto:
> >> >
> >> > > Hi,
> >> > >
> >> > > We switched to Netty on both client side and server side and the
> >> > > performance issue is still there.  Anyone has any insights on what
> >> could
> >> > be
> >> > > the cause of higher latency?
> >> > >
> >> > > Thanks,
> >> > >
> >> > > Li
> >> > >
> >> > >
> >> > >
> >> > > On Mon, Feb 15, 2021 at 2:17 PM Li Wang  wrote:
> >> > >
> >> > > > Hi Enrico,
> >> > > >
> >> > > >
> >> > > > Thanks for the reply.
> >> > > >
> >> > > >
> >> > > > 1. We are using NIO based stack, not Netty based yet.
> >> > > >
> >> > > > 2. Yes, here are some metrics on the client side.
> >> > > >
> >> > > >
> >> > > > 3.6: throughput: 7K, failure: 81215228, Avg Latency: 57ms,  Max
> >> Latency
> >> > > 31s
> >> > > >
> >> > > > 3.4: throughput: 15k, failure: 0,  Avg Latency: 30ms,  Max
> Latency:
> >> > 1.6s
> >> > > >
> >> > > >
> >> > > > 3. Yes, the JVM and zoo.cfg config are the exact same
> >> > > >
> >> > > > 10G of Heap
> >> > > >
> >> > > > 13G of Memory
> >> > > >
> >> > > > 5 Participante
> >> > > >
> >> > > > 5 Observere
> >> > > >
> >> > > > Client session timeout: 3000ms
> >> > > >
> >> > > > Server min session time: 4000ms
> >> > > >
> >> > > >
> >> > > >
> >> > > > 4. Yes, there are two types of  WARN logs and many “Expiring
> >> session”
> >> > > > INFO log
> >> > > >
> >> > > >
> >> > > > 2021-02-15 22:04:36,506 [myid:4] - WARN
> >> > > > [NIOWorkerThread-7:NIOServerCnxn@365] - Unexpected exception
> >> > > >
> >> > > > EndOfStreamException: Unable to read additional data from client,
> it
> >> > > > probably closed the socket: address = /100.108.63.116:43366,
> >> session =
> >> > > > 0x400189fee9a000b
> >> > > >
> >> > > > at
> >> > > >
> >> > >
> >> >
> >>
> org.apache.zookeeper.server.NIOServerCnxn.handleFailedRead(NIOServerCnxn.java:164)
> >> > > >
> >> > > > at
> >> > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:327)
> >> > > >
> >> > > > at
> >> > > >
> >> > >
> >> >
> >>
> org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:522)
> >> > > >
> >> > > > at
> >> > > >
> >> > >
> >> >
> >>
> org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
> >> > > >
> >> > > > at
> >> > > >
> >> > >
> >> >
> >>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >> > > >
> >> > > > at
> >> > > >
> >> > >
> >> >
> >>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >> > > >
> >> > > > at java.base/java.lang.Thread.run(Thread.java:834)
> >> > > >
> >> > > >
> >> > > > 2021-02-15 22:05:14,428 [myid:4] - WARN
> >> > > > [SyncThread:4:SyncRequestProcessor@188] - Too busy to snap,
> >> skipping
> >> > > >
> >> > > >
> >> > > > 2021-02-15 22:01:51,427 [myid:4] - INFO
> >> > > > [SessionTracker:ZooKeeperServer@610] - 

Re: ZooKeeper Operator

[Patrick Hunt]

Hm. We occasionally get security reports for things like docker containers
(which we don't directly control). Perhaps we can club the two together, we
should be clear that these are refs and unsupported/unmaintained by Apache
and the PMC.

Patrick

On Mon, Jan 18, 2021 at 10:20 AM Flavio Junqueira  wrote:

> It sounds like a good idea to document it and add relevant pointers, Pat.
>
> -Flavio
>
> > On 18 Jan 2021, at 19:00, Patrick Hunt  wrote:
> >
> > FYI: The awesome operator list has a few including Pravega:
> > https://github.com/operator-framework/awesome-operators
> >
> > I've seen a few more while investigating kubebuilder, operator-sdk (rh)
> and
> > the like:
> > https://github.com/Ghostbaby/zookeeper-operator
> >
> > Perhaps the first thing we might consider is adding a wiki page detailing
> > the available options and insights from the community? Esp if folks are
> > using them. Similar to the client and tools pages:
> > https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZKClientBindings
> > https://cwiki.apache.org/confluence/display/ZOOKEEPER/UsefulTools
> >
> > Patrick
> >
> > On Mon, Jan 18, 2021 at 2:36 AM Enrico Olivelli 
> wrote:
> >
> >> Thanks for sharing!
> >>
> >> We need more support for K8s in the OS community, this is a good step
> >>
> >> Enrico
> >>
> >> Il giorno lun 18 gen 2021 alle ore 11:18 Flavio Junqueira <
> f...@apache.org>
> >> ha scritto:
> >>
> >>> We've been getting questions and sometimes contributions to the
> ZooKeeper
> >>> Kubernetes Operator we originally did for Pravega, so I feel that it
> has
> >>> been useful more broadly. Perhaps this is something that others might
> be
> >>> interested in too, and I thought of mentioning here.
> >>>
> >>> https://github.com/pravega/zookeeper-operator
> >>>
> >>> Thanks,
> >>> -Flavio
> >>
>
>

Re: ZooKeeper Operator

[Patrick Hunt]

FYI: The awesome operator list has a few including Pravega:
https://github.com/operator-framework/awesome-operators

I've seen a few more while investigating kubebuilder, operator-sdk (rh) and
the like:
https://github.com/Ghostbaby/zookeeper-operator

Perhaps the first thing we might consider is adding a wiki page detailing
the available options and insights from the community? Esp if folks are
using them. Similar to the client and tools pages:
https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZKClientBindings
https://cwiki.apache.org/confluence/display/ZOOKEEPER/UsefulTools

Patrick

On Mon, Jan 18, 2021 at 2:36 AM Enrico Olivelli  wrote:

> Thanks for sharing!
>
> We need more support for K8s in the OS community, this is a good step
>
> Enrico
>
> Il giorno lun 18 gen 2021 alle ore 11:18 Flavio Junqueira 
> ha scritto:
>
> > We've been getting questions and sometimes contributions to the ZooKeeper
> > Kubernetes Operator we originally did for Pravega, so I feel that it has
> > been useful more broadly. Perhaps this is something that others might be
> > interested in too, and I thought of mentioning here.
> >
> > https://github.com/pravega/zookeeper-operator
> >
> > Thanks,
> > -Flavio
>

Re: Correct way to check if zookeeper is running

[Patrick Hunt]

I believe it depends which version of netcat you're using, see the
following for background:
https://issues.apache.org/jira/browse/ZOOKEEPER-737

Regards,

Patrick

On Thu, Oct 29, 2020 at 6:18 AM Debraj Manna 
wrote:

> Anyone any thoughts on this one? Same has been asked in stackoverflow also
>
>
> https://stackoverflow.com/questions/64587307/correct-way-to-check-zookeepers-running-mode
>
> On Wed, Oct 28, 2020 at 9:37 PM Debraj Manna 
> wrote:
>
> > Hi
> >
> > Can someone let me know what is the recommended way to check if zookeeper
> > is running via netcat?
> >
> > The documentation
> > 
> > states echo "isro" | nc localhost 2181 .
> >
> > But in few places it is recommended to use -q with netcat
> >
> > https://stackoverflow.com/a/28424409/785523
> >
> > Can someone let me know if -q is needed ?
> >
> >
> >- Zookeeper Version 3.4.13
> >- OS - Ubuntu 18
> >
> >
>

Re: Zookeeper access logs?

[Patrick Hunt]

There is audit capability in newer versions:
https://zookeeper.apache.org/doc/r3.6.1/zookeeperAuditLogs.html

Earlier versions include support for dumping the transaction log (which
goes back a long way):
https://zookeeper.apache.org/doc/r3.6.0/apidocs/zookeeper-server/org/apache/zookeeper/server/LogFormatter.html

For read access you may need to enable logging (log4j) or look at the
connection level metrics.

Patrick

On Tue, Jun 23, 2020 at 3:32 PM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:

> Hi,
>
> Does the zookeeper have access logs, i wanted to see client reads and
> writes information. Recently we had an incident where some of the clients
> started pounding zk with a lot of reads and it was hard to find which
> client was doing that.
>
> Ram
>

Re: zookeeper protocol requirements

[Patrick Hunt]

fwiw this one comes to mind:
http://www.camilletalk.com/whilefalse/2012/12/building-global-highly-available.html

Patrick

On Fri, Jan 3, 2020 at 7:18 AM MATHIEU Christophe <
christophe-f.math...@thalesgroup.com> wrote:

> Hi Zookeeper users,
>
> I need to interconnect some servers of a zookeeper service over a WAN
> network instead of a LAN. This is a campus network using WAN technics such
> as L3 MPLS or L2 MPLS (VPLS, LAN Emulation).
> I tried to find in the zookeeper documentation the network requirements
> related to the server to server protocol in terms of bandwidth and latency
> for instance but I didn't succeed.
> Does someone have the information ?
>
> Thanks
> Rgds
>
> [@@ THALES GROUP INTERNAL @@]
>
>

ZK makes apache 2019 "top 5" projects

[Patrick Hunt]

This is really awesome, check it out:
https://twitter.com/phunt/status/1204966326118141952

Kudos ZooKeeper community on all the hard work and efforts!

Patrick

Re: Does ZK 3.4.14 support Netty 4.1.42.Final?

[Patrick Hunt]

I don't see a patch on that jira and based on the linked thread it seems
like folks were against revving 3.4. If you're interested/motivated perhaps
you can submit a patch? I'm sure @Andor Molnár  won't
mind. ;-)

Also: just remove the netty files from the binary. iirc if you're using NIO
we don't try to load netty and it should just work. I haven't tried this in
quite some time though, we could have added a dependency. I'd suggest
giving it a try.

Patrick


On Mon, Nov 25, 2019 at 10:39 AM Daniel Chan 
wrote:

> Thanks Patrick and Tamas for the information.
>
> Is there any ETA on https://issues.apache.org/jira/browse/ZOOKEEPER-3568?
>
> We are currently running on 3.4.9 server and 3.4.6 client. If moving to
> 3.5.6, should we upgrade the server or client first?
>
> Thanks,
> Daniel
>
> -----Original Message-
> From: Patrick Hunt 
> Sent: Monday, November 25, 2019 9:55 AM
> To: UserZooKeeper 
> Subject: Re: Does ZK 3.4.14 support Netty 4.1.42.Final?
>
> This was discussed relatively recently:
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.apache.org_thread.html_680038b345da49a3d5cb452de5d54d62f14d1df0747690980c218c1a-40-253Cdev.zookeeper.apache.org-253E=DwIBaQ=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE=JE3yjNS4hXa8nS9n2uFCwEqMvv18hzzEnqunUhCoEns=BbAVeHS1OYH8LyYFALpMB3Y_LWoECeuvBs41uJRNkAQ=pRvPNkgqtf35FPguSMVExKsUyE1EYZcI3trC9TpwszQ=
>
> Gist is that while the identified issue didn't affect us directly folks
> should move to 3.5 (or don't use netty in 3.4) given 3.4 is using a version
> of netty that's no longer supported and too difficult to upgrade.
>
> Patrick
>
>
> On Sat, Nov 23, 2019 at 12:36 AM Tamas Penzes  >
> wrote:
>
> > Hi Daniel,
> >
> > I remember that the migration from Netty 3 to 4 wasn't a trivial task,
> > so I would not expect it in any future ZK 3.4 release.
> >
> > But we have ZK 3.5.5 and 3.5.6 and the migration to any of them is not
> > really problematic since they are backward compatible. We have done it
> > for many Hadoop component, without big code changes (if you use
> > Curator, don't forget to use 4.2.0+ and exclude it's own beta ZK).
> >
> > So the best is to try ZK 3.5.6.
> >
> > Regards, Tamaas
> >
> > On Sat, Nov 23, 2019, 00:52 Daniel Chan 
> wrote:
> >
> > > Hi,
> > >
> > > From
> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__mvnrepository.c
> > > om_artifact_org.apache.zookeeper_zookeeper_3.4.14=DwIBaQ=RoP1Yum
> > > CXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE=JE3yjNS4hXa8nS9n2uFCwEqMvv18h
> > > zzEnqunUhCoEns=BbAVeHS1OYH8LyYFALpMB3Y_LWoECeuvBs41uJRNkAQ=PL7JU
> > > eCo6BJ1AJDl7Egx5u7-xSEf3SnaECIWRnvMoGc=
> > ,
> > > Zookeeper depends on Netty 3.10.6.Final.
> > >
> > > However, Netty has CVEs for versions prior to 4.1.42.Final as per
> > >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__nvd.nist.gov_vuln_detail_CVE-2D2019-2D16869=DwIBaQ=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE=JE3yjNS4hXa8nS9n2uFCwEqMvv18hzzEnqunUhCoEns=BbAVeHS1OYH8LyYFALpMB3Y_LWoECeuvBs41uJRNkAQ=K0DkivRX3n0O2CrM65WwY-BsIsqbeTQRjwL6hVTfjFg=
> :
> > > Netty before 4.1.42.Final mishandles whitespace before the colon in
> > > HTTP headers (such as a "Transfer-Encoding : chunked" line), which
> > > leads to
> > HTTP
> > > request smuggling.
> > >
> > > Will Zookeeper (both client and server) work if we use Netty
> > > 4.1.42.Final or above instead?
> > >
> > > Also what jars are needed for the Zookeeper Client?
> > >
> > > Thanks,
> > > Daniel
> > >
> >
>

Re: Does ZK 3.4.14 support Netty 4.1.42.Final?

[Patrick Hunt]

This was discussed relatively recently:
https://lists.apache.org/thread.html/680038b345da49a3d5cb452de5d54d62f14d1df0747690980c218c1a@%3Cdev.zookeeper.apache.org%3E

Gist is that while the identified issue didn't affect us directly folks
should move to 3.5 (or don't use netty in 3.4) given 3.4 is using a version
of netty that's no longer supported and too difficult to upgrade.

Patrick


On Sat, Nov 23, 2019 at 12:36 AM Tamas Penzes 
wrote:

> Hi Daniel,
>
> I remember that the migration from Netty 3 to 4 wasn't a trivial task, so I
> would not expect it in any future ZK 3.4 release.
>
> But we have ZK 3.5.5 and 3.5.6 and the migration to any of them is not
> really problematic since they are backward compatible. We have done it for
> many Hadoop component, without big code changes (if you use Curator, don't
> forget to use 4.2.0+ and exclude it's own beta ZK).
>
> So the best is to try ZK 3.5.6.
>
> Regards, Tamaas
>
> On Sat, Nov 23, 2019, 00:52 Daniel Chan  wrote:
>
> > Hi,
> >
> > From
> > https://mvnrepository.com/artifact/org.apache.zookeeper/zookeeper/3.4.14
> ,
> > Zookeeper depends on Netty 3.10.6.Final.
> >
> > However, Netty has CVEs for versions prior to 4.1.42.Final as per
> > https://nvd.nist.gov/vuln/detail/CVE-2019-16869:
> > Netty before 4.1.42.Final mishandles whitespace before the colon in HTTP
> > headers (such as a "Transfer-Encoding : chunked" line), which leads to
> HTTP
> > request smuggling.
> >
> > Will Zookeeper (both client and server) work if we use Netty 4.1.42.Final
> > or above instead?
> >
> > Also what jars are needed for the Zookeeper Client?
> >
> > Thanks,
> > Daniel
> >
>

Re: Any interest in a gRPC version of ZooKeeper

[Patrick Hunt]

There are quite a few benefits to using grpc imo. It's come up a few times
where I've been part of the discussion - ala we make it b/w compat it would
be a good move imo. Then the question becomes what else do we fix at the
same time? e.g. make version fields 64 bit rather than 32? etc... there are
a bunch of zk4 such jiras that could be addressed at the same time (and
likely in a b/w compat way - ie zk3)

That said, we added things like rest in the past for similar reasons and it
never really took off... Would be a shame to see the same here.

Patrick

On Mon, Nov 18, 2019 at 6:48 AM Jordan Zimmerman 
wrote:

> > That looks like great work. In order to address the issues, why not
> build on top of curator (https://curator.apache.org)?
>
> (Note: I'm the main author of Curator). I'd definitely try to make
> something like Curator for gRPC. I'm not sure exactly what that means at
> this point. But, my main goal is to enable non-JVM clients. We have C,
> python and few others now but they always lag with changes and are hard to
> maintain.
>
> -JZ
>
> > On Nov 18, 2019, at 9:45 AM, Jörn Franke  wrote:
> >
> > That looks like great work. In order to address the issues, why not
> build on top of curator (https://curator.apache.org)?
> >
> > I could support in case question rise with SASL, but I am not sure yet
> if I find the time to actively develop for this unfortunately
> >
> >> Am 18.11.2019 um 15:25 schrieb Jordan Zimmerman <
> jor...@jordanzimmerman.com>:
> >>
> >> Hi Folks,
> >>
> >> I've written a proof of concept implementation of a ServerCnxnFactory
> that implements gRPC. The goal is to make it possible to easily write
> ZooKeeper clients in non-JVM languages. Using the proof of concept I was
> able to write a Golang client easily. What's the interest level of
> something like this? Let's discuss if it's worth pursuing. I'd be willing
> to move this from proof of concept to production but I'll need help (1 or 2
> co-developers).
> >>
> >> If you want to try it, I've pushed the Golang client and some
> instructions here (let me know if you have any issues - I'm a go neophyte).
> Note: "zookeeper/test.go" is the interesting file:
> >>
> >>   https://github.com/Randgalt/zkgrpc <
> https://github.com/Randgalt/zkgrpc>
> >>
> >> Here's the proof of concept on the ZK server side (the interesting
> files are RpcServerCnxn.java, RpcServerCnxnFactory.java,
> RpcZooKeeperServer.java and zookeeper.proto):
> >>
> >>
> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc <
> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc>
>
> >>
> >> Issues:
> >> Writing a client, even with gRPC, will require some work. Sessions have
> to be maintained, watchers have to be maintained, etc.
> >> Currently, Jute is deeply embedded in ZooKeeper. The proof of concept
> has to emulate Jute byte buffers. Ideally, this will be abstracted so that
> only records could be used so that the gRPC connection doesn't have to keep
> marshalling/unmarshalling byte buffers
> >> I don't know enough about the gRPC client/server implementations to
> know if it will meet the needs of ZooKeeper. Anyone have experience here?
> >> I haven't completely thought through how much work it will take to
> write useful clients. As I've shown with the proof of concept simple ZK
> CRUD db operations work well. I need to spend time writing a recipe such as
> Leader Election to see how much work is required.
> >> I'm not sure how things like SASL and reconfig would work with gRPC
> >>
> >> -Jordan
>
>

Re: String inconsistency issue when running ZK with OpenJDK 10 on SKL machines

[Patrick Hunt]

On Mon, Oct 28, 2019 at 12:06 AM Enrico Olivelli 
wrote:

> Fangmin,
>
> Il lun 28 ott 2019, 02:23 Fangmin Lv  ha scritto:
>
> > Hey everyone,
> >
> > (Forgot to add subject in the previous email, resent with clear subject.)
> >
> > I'd like to share some weird inconsistency bugs we saw recently on prod,
> > the root cause and potential fixes of it. It took us around a month to
> > investigate, reproduce and find out the root cause, hopefully the
> > informations here will help people avoid hitting this same potential
> issue.
> >
> > [Trigger conditions and behavior]
> >
> > The inconsistency issue only happened when running ZK with OpenJDK 10 on
> > SKL machines, and it's not because of bugs inside ZK but due to a
> > macro-assembly bug inside JDK.
> >
> > And the behavior of the issues might be:
> >
> > * NONODE returned when getData from a child exist when queried with
> > getChildren, and there is no delete issued
> > * NONODE error returned when try to create a child based on the parent
> node
> > just successfully created, and there is no delete issued
> > * No client is able to acquire the lock even though the previous session
> > who hold the lock already dead
> >
> > [Root cause]
> >
> > The direct cause of the misbehavior above is due to the key/value put
> into
> > the ZooKeeperServer.outstandingChangesForPath HashMap or the
> > DataNode.children HashSet are not visible to the future get or remove,
> > which caused the outstanding changes not visible when leader prepare the
> > following txns, or node being deleted but not removed from
> > DataNode.children.
> >
> > And the 'bad' HashMap/HashSet behavior is not because of concurrency bugs
> > inside ZK, but due to a macro-assembly bug which is used to generate the
> > String.equals intrinsic assembly code in JDK 9 and 10. The bug was
> > introduced in JDK-8144771 when adding AVX-512 instructions support in JDK
> > to optimize the String.equals intrinsic performance with 512 bit vector
> op
> > support. Due to the bug, the String.equals method may return false result
> > when using high band of CPU register (xmm16 - xmm31) with non-empty stack
> > on SKL machines where AVX-512 is available.
> >
> > The macro-assembly bug we hit is in vptest which is used in the
> > string_compare macro assembly code
> > <
> >
> http://hg.openjdk.java.net/jdk/jdk10/file/b09e56145e11/src/hotspot/cpu/x86/macroAssembler_x86.cpp#l4933
> > >.
> > It uses add/sub instruction when saving/resuming register values
> > temporarily from stack, which will affect and distort the ZF (zero flag)
> in
> > FLAGS register from the previous test instruction.
> >
> > For our case, if the key exist in the DataNode.children HashSet, the test
> > instruction result will be zero, ZF bit will be set to 1, if the RSP
> value
> > is not 0 (e.g stack is not empty) after addptr code here, then the ZF bit
> > will be changed to 0, so String.equals compare during removeNode will
> > return false result, and the key won't be removed.
> >
> > There is bug reported in JDK-8207746, the behavior is different, we've
> > confirmed the issue by adding assembly code to log the issue in JDK 10.
> >
> > [Solutions]
> >
> > The possible mitigations are:
> >
> > 1. Disabling the AVX-512 with JVM option -XX:UseAVX=2
> > 2. Using OpenJDK version higher than 10, which has fixed the issue in
> > JDK-8207746
> >
> > Upgrading to OpenJDK 11+ is a better option, since 10 is not well
> > supported, and AVX-512 do helps improving performance.
> >
> > We use JDK 10 due to SSL quorum socket close stall issue mentioned in
> > ZOOKEEPER-3384 ,
> and
> > the SO_LINGER option is not honored in JDK 11. We've unblocked JDK 11 by
> > asynchronously closing the quorum socket, and we're upstreaming that in
> > ZOOKEEPER-3574 .
> >
> > Thanks,
> > Fangmin
> >
>
>
> Thank you for sharing this.
>

10x - thanks!

Patrick


> Do you have any pointer to the jdk11 bugs? Is it solved in 12+?
>
> I am running with jdk11-13 but without ssl, so never seen problems.
>
> Enrico
>
> >
>

Re: Removing Netty support from branch-3.4

[Patrick Hunt]

On Fri, Oct 4, 2019 at 9:14 AM Enrico Olivelli  wrote:

> The release branch 3.4 is frozen and we should cut new releases only for
> important security reasons or other important issues for users that cannot
> upgrade to 3.5.
>
> Given that 3.5 is now the suggested version and the upgrade path is simple
> I think there is no need to put effort into this activity.
>
> Is there any other valid reason for not using 3.4 + Netty in production ?
> We can advise users on the website that Netty 3 is old, and it is suggested
> to move do plain NIO or to ZK 3.5 client.
> Is the Netty dependency flagging us with security risks ?
>
>
We can explain that netty/3.4 whatever we like, the issue is 1) in the near
term we'll deal with reports such as when it's found through automated
means, easier is to just address it directly. 2) eventually there is likely
to be a real issue that can't be explained away, we would need to address
it directly in that case. Once 3.4 is officially "no longer supported" it
would be easier, but atm that's not the case. Perhaps we should document an
EOL for 3.4 to help address and close the loop more generally?

Patrick


> Il giorno ven 4 ott 2019 alle ore 10:52 Andor Molnar  ha
> scritto:
>
> > Hi ZK users / devs,
> >
> > ZooKeeper branch-3.4 is still on Netty 3 which is not maintained by the
> > Netty team anymore. There’s no intention of updating it on our side,
> hence
> > we’re planning to remove it from the codebase completely and ask existing
> > users to upgrade to 3.5, if they still want to use Netty. 3.5 is a much
> > better option anyway in various aspects: Netty 4 performs better, TLS
> > support in both quorum and client communication, etc.
> >
> > The default stack in 3.4 is NIO, so our gut feeling is that the impact on
> > our existing users is low, however the most important effect of this
> change
> > is probably the loss of encrypted client connections.
> >
> > Please share your thoughts about this change and let us know if upgrading
> > to 3.5 is not possible in your use case.
> >
> > Tracking Jira: https://issues.apache.org/jira/browse/ZOOKEEPER-3568
> >
> > Regards,
> > Andor
> >
> >
> >
> >
>

Re: How to scale ZooKeeper to support 10K concurrent connections?

[Patrick Hunt]

Whether or not you can use local sessions is a critical aspect:
https://issues.apache.org/jira/browse/ZOOKEEPER-1147

Patrick

On Fri, Sep 27, 2019 at 2:27 PM Michael Han  wrote:

> >> can launch tens of thousands of calls
>
> Is it possible for you to quantify this in a form of (read and write)
> request per second, and the average request payload if it's OK to disclose?
> This information is critical on shaping the best scaling solution.
>
> Without knowing any of ballpark numbers of your system and workload
> characteristics, one immediate experiment you could do is to set up
> Observer servers, remove quorum servers from serving client traffic by
> redirecting all of your client traffic to observer servers. This will at
> least scale your concurrent connections linearly with the number of
> observers. This will also scale concurrent requests processing capabilities
> for read requests (and to a limited extend, for write requests as well),
> but request processing scaling is harder and depends on your workload
> characteristics.
>
>
>
> On Fri, Sep 27, 2019 at 12:07 PM Yue Shen  wrote:
>
> > Thank you, Jorn.
> >
> > We don't use Solr. We inherited this architecture from another team, and
> we
> > don't have time to redesign a new system to scale in 2 months.
> >
> > As you said, if I were to design it, I would definitely put a queue in
> > front of Lambda service, our new design is actually on the way with Kafka
> > upfront. However we need to scale it out with the coming holiday
> > season before we can roll out the new system, which is just kicked off a
> > couple of weeks ago.
> >
> > At this point, we want to tune ZooKeeper so it can handle 10K concurrent
> > calls. Any suggestions?
> >
> > Thank you,
> > Yue
> >
> > On Fri, Sep 27, 2019 at 10:39 AM Jörn Franke 
> wrote:
> >
> > > Put the Solr request on a SQS queue using your 10k instances and have
> 10
> > > or so worker working on the queue to put it in Solr. Having 10k
> > connections
> > > just because lambda creates that many instances does not make sense for
> > no
> > > database service.
> > >
> > > > Am 27.09.2019 um 19:01 schrieb Yue Shen :
> > > >
> > > > Dear ZooKeeper users,
> > > >
> > > > I have a special use case, in which I use AWS lambda service.
> > > >
> > > > Inside the lambda service logic, it goes to ZooKeeper to check the
> > worker
> > > > for the data, if exists,  connect to the worker endpoint and send the
> > > data.
> > > > If the worker isn't assigned, the logic will post a new assignment,
> and
> > > > wait for it to be assigned to a worker. There is a coordinator to
> watch
> > > the
> > > > new assignment and assign tasks.
> > > >
> > > > My problem comes with AWS Lambda service, which can launch tens of
> > > > thousands of calls. When this happens, I found many calls get
> timeout.
> > > The
> > > > active connections to ZooKeeper plateau around 6500.
> > > >
> > > > BTW, I run ZooKeeper as 3 node ensemble, run on Quorum.
> > > >
> > > > How can I scale ZooKeeper to support more concurrent connections?
> > > >
> > > > Thank you,
> > > > Yue
> > >
> >
>

Re: An Apache Zookeeper Security Vulnerability

[Patrick Hunt]

On Fri, Aug 9, 2019 at 9:34 AM Enrico Olivelli  wrote:

> Those points do not seem a security issue
>
>
Agree. First off the data is not sensitive. Also it's debug level and
logged on the server. See
https://issues.apache.org/jira/browse/ZOOKEEPER-3488 - similar situation
although in this case debug is not the default - user would actively have
to turn this on.

Patrick


>
> Enrico
>
>
> Il ven 9 ago 2019, 17:52 Fu, Xiaoqin  ha scritto:
>
> > Dear developers:
> >  I am a Ph.D. student at Washington State University. I applied
> > dynamic taint analyzer (distTaint) to Apache Zookeeper (version 3.4.11).
> > And then I find a security vulnerability, that exists from 3.4.11-3.4.14
> > and 3.5.5, from tainted paths.
> >
> > Possible information leakage from FileTxnSnapLog to log without LOG
> > control LOG.isDebugEnabled():
> > In org.apache.zookeeper.server.persistence.FileTxnSnapLog, the statement
> > LOG.debug don't have LOG controls:
> > public void processTransaction(TxnHeader hdr,DataTree dt,
> > Map sessions, Record txn)
> > throws KeeperException.NoNodeException {
> > ..
> > if (rc.err != Code.OK.intValue()) {
> > LOG.debug("Ignoring processTxn failure hdr:" + hdr.getType()
> > + ", error: " + rc.err + ", path: " + rc.path);
> > }
> > ..
> > }
> >
> > Sensitive information about hdr type or rc path may be leaked. The
> > conditional statement LOG.isDebugEnabled() should be added:
> > public void processTransaction(TxnHeader hdr,DataTree dt,
> > Map sessions, Record txn)
> > throws KeeperException.NoNodeException {
> > ..
> > if (rc.err != Code.OK.intValue()) {
> > if (LOG.isDebugEnabled())
> > LOG.debug("Ignoring processTxn failure hdr:" + hdr.getType()
> > + ", error: " + rc.err + ", path: " + rc.path);
> > }
> > ..
> > }
> > Please help me confirm it and give it a CVE ID.
> >
> > Thank you very much!
> > Yours sincerely
> > Xiaoqin Fu
> >
> >
>

Re: Issues with using ZooKeeper 3.5.5 together with Solr 8.2.0

[Patrick Hunt]

It sounds to me like a regression. We always had the properties format for
4lw, this (membership:) breaks that. I'd recommend fixing it in the next
3.5/3.6. ie. output the membership on a single line "membership:  \n".
Should be a pretty simple change - anyone interested in taking it on?

Also agree that folks should move off 4lw to the new (better) options, esp
as we plan to deprecate 4lw at some point.

Patrick

On Sun, Aug 4, 2019 at 12:15 PM Enrico Olivelli  wrote:

> Il sab 3 ago 2019, 21:41 Shawn Heisey  ha scritto:
>
> > On 8/2/2019 10:33 AM, Patrick Hunt wrote:
> > > Right, it prints the membership of the quorum, see (for majority case
> > which
> > > is typical):
> > > org.apache.zookeeper.server.quorum.flexible.QuorumMaj#toString
> > >
> >
> https://github.com/apache/zookeeper/blob/faa7cec71fddfb959a7d67923acffdb67d93c953/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/flexible/QuorumMaj.java#L112
> >
> > For our purposes (the Solr project) the output of the "conf" 4lw command
> > is inconsistent, changing when there is a multi-server ensemble.  All of
> > the lines except the "membership: " one use an equals sign as a
> > separator.  Our parsing code fails on that line because there is no
> > equals sign.
> >
> > Whether or not the ZK project should consider this a bug is the question
> > that I am asking.
> >
> > While getting to the bottom of that question, another one arises:  Who
> > are the intended audiences of the "conf" 4lw output?  If one of those
> > audiences is ZK itself, then the output of the command probably will
> > work perfectly for that audience, as ZK uses Java's "properties" API to
> > read its config file, which means that both = and : will work as
> > separators.
> >
> > The current output also works great for a human audience.  Humans are
> > quite flexible.
> >
> > The difficulty is machine-based parsers like the one in Solr, which is
> > very simple and just splits lines on an equal sign.  How much
> > consistency can an audience like this expect?  I would personally say
> > that the way "membership: " is output is a bug.  That line probably
> > should be entirely removed, or the colon could be replaced with an equal
> > sign.  I think that the line only makes sense for a human audience, and
> > that audience probably doesn't really need it.
> >
> > An alternate path:  One statement in the documentation would remove all
> > difficulty, without any code changes in ZK:
> >
> > "The output from the conf 4lw command should be parsed by the Java
> > Properties API for best results."
> >
>
> I think the best option is to switch to the Admin, HTTP + json based, as it
> is possible to integrate better with other automatic tools.
> We are working on docs for 3.6 (expecially the http admin server).
> We also added many new 'commands' to the admin API, which is supposed to be
> the future for the mid/long term
>
> Enrico
>
>
>
> > If that statement is added, then Solr just needs to utilize the
> > Properties API, which is very easy to do, and all is well again.
> >
> > So... I'm thinking we should open an issue in Jira, and then leave it up
> > to the ZK committers whether it's better to change the output or adjust
> > the documentation.  I can supply a patch either way.  What does the
> > community think?
> >
> > Thanks,
> > Shawn
> >
>

Re: Ephemeral nodes not removed

[Patrick Hunt]

The jira you ref'd is the only one that comes to mind. In terms of
troubleshooting - try connecting a client to each of the servers in tern
and see if it's a situation where they have a different view of the world
wrt those znodes. You might also have the client create separate znodes on
each server and ensure that they are consistent. The logs are also
typically a good source of information - check against the session id.

Patrick


On Wed, Jul 31, 2019 at 11:29 PM John Lindwall 
wrote:

> ZooKeeper 3.4.6-1569965
>
> In our environment we seem to have a situation where ephemeral znodes
> are not getting removed after the zookeeper session has been
> terminated.  We can see examples of znodes that were created 3-4 days
> past that still exist, though the zk sessions bound to those znodes
> should no longer exist.
>
> Note that we've had this cluster running to about 4 years and have not
> seen this problem until recently.
>
> 1. I am wondering if there are any known issues that would affect our
> zookeeper version that may cause this behavior?
> 2. Is it possible our servers are simply in a "bad state" and a simple
> reboot might clean things up?
> 3. Any tips on diagnosing this?
>
> We noticed this issue from 2011 but that seems to have been fixed in our
> branch.
>
> 
> https://issues.apache.org/jira/browse/ZOOKEEPER-1208
>
> I also see this issue which it seems was never resolved?
>
> https://issues.apache.org/jira/browse/ZOOKEEPER-3018
>
> Thanks,
> John Lindwall
>
>

Re: Ephemeral znodes not getting removed

[Patrick Hunt]

The jira you ref'd is the only one that comes to mind. In terms of
troubleshooting - try connecting a client to each of the servers in tern
and see if it's a situation where they have a different view of the world
wrt those znodes. You might also have the client create separate znodes on
each server and ensure that they are consistent. The logs are also
typically a good source of information - check against the session id.

Patrick

On Wed, Jul 31, 2019 at 5:54 PM John Lindwall 
wrote:

> ZooKeeper 3.4.6-1569965
>
> In our environment we seem to have a situation where ephemeral znodes
> are not getting removed after the zookeeper session has been
> terminated.  We can see examples of znodes that were created 3-4 days
> past that still exist, though the zk sessions bound to those znodes
> should no longer exist.
>
> Note that we've had this cluster running to about 4 years and have not
> seen this problem until recently.
>
> 1. I am wondering if there are any known issues that would affect our
> zookeeper version that may cause this behavior?
> 2. Is it possible our servers are simply in a "bad state" and a simple
> reboot might clean things up?
> 3. Any tips on diagnosing this?
>
> We noticed this issue from 2011 but that seems to have been fixed in our
> branch.
>
> 
> https://issues.apache.org/jira/browse/ZOOKEEPER-1208
>
> Thanks,
> John Lindwall
>

Re: Issues with using ZooKeeper 3.5.5 together with Solr 8.2.0

[Patrick Hunt]

On Fri, Aug 2, 2019 at 7:12 AM Jan Høydahl  wrote:

> So it turned out that Solr failed parsing the 'conf' response from
> Zookeeper since there was one line that did not conform to the key=value
> syntax:
>
> ...
> quorumPort=2888
> peerType=0
> membership:
> server.1=zoo1:2888:3888:participant;0.0.0.0:2181
> server.2=zoo2:2888:3888:participant;0.0.0.0:2181
> server.3=zoo3:2888:3888:participant;0.0.0.0:2181
> ...
>
> The "membership: " line was the offender here. Can anyone shed some light
> on the purpose of that line? Is it a header for what comes in the lines
> below or is it a key with wrong delimiter (: instead of =)? and empty value?
>
>
Right, it prints the membership of the quorum, see (for majority case which
is typical):
org.apache.zookeeper.server.quorum.flexible.QuorumMaj#toString
https://github.com/apache/zookeeper/blob/faa7cec71fddfb959a7d67923acffdb67d93c953/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/flexible/QuorumMaj.java#L112

Seems like the header, it came in from a change by @Alexander Shraer
 who may have more insight.

Patrick


> We'll fix the parsing in Solr to disregard this line for now.
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> > 31. jul. 2019 kl. 08:34 skrev Zheng Lin Edwin Yeo  >:
> >
> > Sure. I have another thread on this in the Solr mailing list.
> >
> > Regards,
> > Edwin
> >
> > On Wed, 31 Jul 2019 at 14:04, Jörn Franke  wrote:
> >
> >> Could be a Solr issue. Check the Solr mailing list - they can help you
> >> there better on Solr issues.
> >>
> >>> Am 31.07.2019 um 04:27 schrieb Zheng Lin Edwin Yeo <
> edwinye...@gmail.com
> >>> :
> >>>
> >>> Thanks everyone for your reply.
> >>>
> >>> I have tried to put the following in zoo.cfg under ZooKeeper:
> >>> 4lw.commands.whitelist=mntr,conf,ruok
> >>>
> >>> But it is still showing this error.
> >>> *"Errors: - membership: Check 4lq.commands.whitelist setting in
> zookeeper
> >>> configuration file."*
> >>>
> >>> As I am using SolrCloud, the collection config can still be loaded to
> >>> ZooKeeper as per normal. But if I tried to create a collection, I will
> >> get
> >>> the following error:
> >>>
> >>> {
> >>> "responseHeader":{
> >>>   "status":400,
> >>>   "QTime":686},
> >>> "failure":{
> >>>   "192.168.1.2:8983
> >> _solr":"org.apache.solr.client.solrj.SolrServerException:IOException
> >>> occurred when talking to server at: http://192.168.1.2:8983/solr;,
> >>>   "192.168.1.2:8984
> >> _solr":"org.apache.solr.client.solrj.SolrServerException:IOException
> >>> occurred when talking to server at: http://192.168.1.2:8984/solr"},
> >>> "Operation create caused
> >>>
> >>
> exception:":"org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
> >>> Underlying core creation failed while creating collection:
> collection1",
> >>> "exception":{f
> >>>   "msg":"Underlying core creation failed while creating collection:
> >>> collection1",
> >>>   "rspCode":400},
> >>> "error":{
> >>>   "metadata":[
> >>> "error-class","org.apache.solr.common.SolrException",
> >>> "root-error-class","org.apache.solr.common.SolrException"],
> >>>   "msg":"Underlying core creation failed while creating collection:
> >>> collection1",
> >>>   "code":400}}
> >>>
> >>> Is there anything which I may have missed out?
> >>>
> >>> Regards,
> >>> Edwin
> >>>
>  On Wed, 31 Jul 2019 at 00:12, Shawn Heisey 
> wrote:
> 
> > On 7/29/2019 11:45 PM, Enrico Olivelli wrote:
> > Due to potential security risks since ZK 3.5 you have to explicitly
> > whitelist some commands.
> 
>  The 3.5.5 documentation says that "*" can be used to whitelist all
>  commands.
> 
>  But what you just said seems to contradict that.  If your statement is
>  more accurate, then the documentation should be updated to list the
>  commands that are NOT enabled when using a wildcard.
> 
>  There is a SOLR issue to upgrade the client in Solr to 3.5.5:
> 
>  https://issues.apache.org/jira/browse/SOLR-8346
> 
>  A comment was made on this issue saying that the following config is
>  needed when the server is running 3.5.x:
> 
>  4lw.commands.whitelist=mntr,conf,ruok
> 
>  Thanks,
>  Shawn
> 
> >>
>
>

Re: Zookeeper latency calculation

[Patrick Hunt]

High max latency is typically due to gc, slow disk, or issues with vm
oversubscription. Turn on gc logging and verify. I've used strace in the
past to troubleshoot disk issues (fsync - although now a days it's in the
log).

Patrick

On Thu, Jul 18, 2019 at 8:40 AM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:

> I do have a busy system total packet processed by zk is like 60k/s (30k
> send and 30k received) and how about the max latencies? is ti always
> constant ? for me avg latency is 0 but max latency is showing 40s how is it
> possible?
>
>
>
>
> On Wed, Jul 17, 2019 at 8:46 PM Michael Han  wrote:
>
> > Typing send too soon...
> >
> > >> always give avg_latency "0"
> >
> > The latency metrics depends on workloads. Try hit your cluster hard with
> > some artificially generated heavy read / write workloads, you will see
> the
> > number deviates from 0.
> >
> > >> If I recall correctly avg_latency is an int, not float
> >
> > This is now a float on master branch, and the change was made in
> > ZOOKEEPER-2641. I remember this because this actually breaks one of our
> > internal metrics system, where our system expects an int (the old type).
> >
> > On Wed, Jul 17, 2019 at 8:43 PM Michael Han  wrote:
> >
> > > >> always give avg_latency "0"
> > >
> > > The latency metrics depends on workloads.
> > >
> > > On Wed, Jul 17, 2019 at 1:34 AM Enrico Olivelli 
> > > wrote:
> > >
> > >> Il mar 16 lug 2019, 19:05 rammohan ganapavarapu <
> > rammohanga...@gmail.com>
> > >> ha scritto:
> > >>
> > >> > Hi,
> > >> >
> > >> > I am trying to understand how zookeeper latency calculated, mntr
> > command
> > >> > always give avg_latency "0", can some one help how to calculate avg
> > >> request
> > >> > latency in zookeeper?
> > >> >
> > >>
> > >>
> > >> Are you also taking metrics on the client?
> > >> Maybe on the server side the value is biased by very fast ops
> > >>
> > >>
> > >> Enrico
> > >>
> > >> >
> > >> >
> > >> > Thanks,
> > >> > Ram
> > >> >
> > >>
> > >
> >
>

Re: ACL implimentation

[Patrick Hunt]

Hm. While formulating a response I re-discovered this, which I didn't find
earlier:
https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
really we should move the cli shell information in particular into the
"getting started" docs page of ZK.

Here are my notes from the testing session I did:

java -cp build/classes:build/lib/*
org.apache.zookeeper.server.auth.DigestAuthenticationProvider pat:test
pat:test->pat:KDzjoQ5VTKQfcjheJHwntPTKiXc=

zkCli.sh

addauth digest pat:test

create /mynode content digest:pat:KDzjoQ5VTKQfcjheJHwntPTKiXc=:cdrwa

setAcl / digest:pat:KDzjoQ5VTKQfcjheJHwntPTKiXc=:cdrwa

getAcl /foo

setAcl /foo2 digest:pat:KDzjoQ5VTKQfcjheJHwntPTKiXc=:cdrwa



On Mon, Jun 10, 2019 at 2:31 PM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:

> Can you guys share your learnings or exp so that i dont have to go through
> that pain if i want to enable ACL ?
>
> Ram
>
> On Mon, Jun 10, 2019 at 2:00 PM Andor Molnár  wrote:
>
> > Agreed. I had to dig a bunch of Hortonworks / Stackoverflow docs to
> > learn how ACLs work.
> >
> >
> > Andor
> >
> >
> >
> > On 2019. 06. 09. 17:03, Patrick Hunt wrote:
> > > I had to deal with some ACL issues myself recently and noticed the lack
> > of
> > > docs we have, both generally and best practices. I spent a bunch of
> time
> > > when testing the recent ACL changes from Andor just re-learning the
> shell
> > > commands and config necessary to exercise the patches. This would be a
> > > great area for contributions.
> > >
> > > Patrick
> > >
> > > On Fri, Jun 7, 2019 at 8:04 AM rammohan ganapavarapu <
> > > rammohanga...@gmail.com> wrote:
> > >
> > >> Enrico,
> > >>
> > >> Thank you.
> > >>
> > >> Ram
> > >>
> > >> On Fri, Jun 7, 2019 at 5:30 AM Enrico Olivelli 
> > >> wrote:
> > >>
> > >>> Ram
> > >>> Can you describe better your problem ?
> > >>> Usually you are activating auth on clients and then you apply the
> ACLs
> > >> this
> > >>> way all clients will be able to access data.
> > >>>
> > >>> Try our procedure in a staging environment before doing in in
> > production
> > >>>
> > >>> Enrico
> > >>>
> > >>> Il gio 6 giu 2019, 23:56 rammohan ganapavarapu <
> > rammohanga...@gmail.com>
> > >>> ha
> > >>> scritto:
> > >>>
> > >>>> Hi,
> > >>>>
> > >>>> Is there any recommendations or best practices on implementing ACL
> on
> > >>>> existing zookeeper cluster with production data with out downtime?
> > >>>>
> > >>>> Thanks,
> > >>>> Ram
> > >>>>
> >
>

Re: ACL implimentation

[Patrick Hunt]

I had to deal with some ACL issues myself recently and noticed the lack of
docs we have, both generally and best practices. I spent a bunch of time
when testing the recent ACL changes from Andor just re-learning the shell
commands and config necessary to exercise the patches. This would be a
great area for contributions.

Patrick

On Fri, Jun 7, 2019 at 8:04 AM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:

> Enrico,
>
> Thank you.
>
> Ram
>
> On Fri, Jun 7, 2019 at 5:30 AM Enrico Olivelli 
> wrote:
>
> > Ram
> > Can you describe better your problem ?
> > Usually you are activating auth on clients and then you apply the ACLs
> this
> > way all clients will be able to access data.
> >
> > Try our procedure in a staging environment before doing in in production
> >
> > Enrico
> >
> > Il gio 6 giu 2019, 23:56 rammohan ganapavarapu 
> > ha
> > scritto:
> >
> > > Hi,
> > >
> > > Is there any recommendations or best practices on implementing ACL on
> > > existing zookeeper cluster with production data with out downtime?
> > >
> > > Thanks,
> > > Ram
> > >
> >
>

Re: majority of non-failing machines VS quorum

[Patrick Hunt]

2n+1 = ensemble size required to survive n failed zkservers (servers not in
the quorum)

iow: 3 nodes means 1 zkserver can fail and the service is still up. 5 and
you can survive 2 failures.

Patrick

On Wed, May 29, 2019 at 4:43 PM Joel Mestres 
wrote:

> Hello which is the relation between the number of F failling machines that
> the cluster can tolerate and the quorum configuration? F determines the
> posibles numbers of quorums? or F can be greater / smaller than quorum ?
> thanks in advance for your response!
>
> --
>
>
> The information contained in this e-mail may be confidential. It has been
> sent for the sole use of the intended recipient(s). If the reader of this
> message is not an intended recipient, you are hereby notified that any
> unauthorized review, use, disclosure, dissemination, distribution or
> copying of this communication, or any of its contents, is strictly
> prohibited. If you have received it by mistake please let us know by
> e-mail
> immediately and delete it from your system. Many thanks.
>
>
>
> La información
> contenida en este mensaje puede ser confidencial. Ha sido enviada para el
> uso exclusivo del destinatario(s) previsto. Si el lector de este mensaje
> no
> fuera el destinatario previsto, por el presente queda Ud. notificado que
> cualquier lectura, uso, publicación, diseminación, distribución o copiado
> de esta comunicación o su contenido está estrictamente prohibido. En caso
> de que Ud. hubiera recibido este mensaje por error le agradeceremos
> notificarnos por e-mail inmediatamente y eliminarlo de su sistema. Muchas
> gracias.
>
>

Re: Why does ZooKeeper follower shutdown itself when it can not read from leader

[Patrick Hunt]

That was/is the original intent.  ZK was built to "fail fast" when it
didn't know how to handle a particular case, or that case might be error
prone to handle. The expectation is that the parent will restart the ZK
server process when it fails.

Patrick

On Wed, May 22, 2019 at 6:27 PM Qian Zhang  wrote:

> Hi Andor,
>
> I am using ZooKeeper release 3.4.10.
>
> I checked the code, if follower fails to read from leader (e.g., read
> timeout), it will close the socket, see
>
> https://github.com/apache/zookeeper/blob/release-3.4.10/src/java/main/org/apache/zookeeper/server/quorum/Follower.java#L91:L85
> for
> details. And once the socket is close, it will make follower fails to write
> (I guess same socket is used here) which will be treated as an severe
> unrecoverable error, and then shutdown follower, see
>
> https://github.com/apache/zookeeper/blob/release-3.4.10/src/java/main/org/apache/zookeeper/server/quorum/FollowerRequestProcessor.java#L90:L95
>  and
>
> https://github.com/apache/zookeeper/blob/release-3.4.10/src/java/main/org/apache/zookeeper/server/ZooKeeperCriticalThread.java#L48:L51
> .
>
> So it seems shutting down follower when it cannot read from leader is the
> design behavior? Or if my understanding is wrong can you please let me know
> the design behavior in this case? Thanks!
>
>
> Regards,
> Qian Zhang
>
>
> On Wed, May 22, 2019 at 8:52 AM Qian Zhang  wrote:
>
> > Anyone has any ideas?
> >
> > Regards,
> > Qian Zhang
> >
> >
> > On Sun, May 19, 2019 at 6:15 PM Qian Zhang  wrote:
> >
> >> Hi,
> >>
> >> I have a ZooKeeper cluster which has 5 nodes. Today the leader cannot be
> >> connected due to a hardware issue, and then I found the 4 followers just
> >> shutdown, here is the logs:
> >>
> >>> May 18 15:34:28 MD001076 java[29148]: [myid:1] WARN
> >>> [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@89] - Exception when
> >>> following the leader
> >>>   java.net.SocketTimeoutException:
> >>> Read timed out
> >>> at
> >>> java.net.SocketInputStream.socketRead0(Native Method)
> >>> at
> >>> java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> >>> at
> >>> java.net.SocketInputStream.read(SocketInputStream.java:171)
> >>> at
> >>> java.net.SocketInputStream.read(SocketInputStream.java:141)
> >>> at
> >>> java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> >>> at
> >>> java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> >>> at
> >>> java.io.DataInputStream.readInt(DataInputStream.java:387)
> >>> at
> >>> org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> >>> at
> >>>
> org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
> >>> at
> >>>
> org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
> >>> at
> >>> org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:153)
> >>> at
> >>>
> org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:85)
> >>> at
> >>> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:937)
> >>> May 18 15:34:28 MD001076 java[29148]: [myid:1] INFO
> >>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] -
> >>> Accepted socket connectio
> >>> n from /10.249.255.10:42306
> >>> May 18 15:34:28 MD001076 java[29148]: [myid:1] WARN
> >>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@896] -
> >>> Connection request from old cl
> >>> ient /10.249.255.10:42306; will be dropped if server is in r-o mode
> >>> May 18 15:34:28 MD001076 java[29148]: [myid:1] INFO
> >>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@942] -
> >>> Client attempting to establish
> >>>  new session at /10.249.255.10:42306
> >>> May 18 15:34:28 MD001076 java[29148]: [myid:1] ERROR
> >>> [FollowerRequestProcessor:1:ZooKeeperCriticalThread@49] - Severe
> >>> unrecoverable error, from threa
> >>> d : FollowerRequestProcessor:1
> >>>   java.net.SocketException: Socket
> >>> closed
> >>> at
> >>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
> >>> at
> >>> java.net.SocketOutputStream.write(SocketOutputStream.java:155)
> >>> at
> >>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> >>> at
> >>> 

Re: Deprecated CSVInputArchive and XMLInputArchive

[Patrick Hunt]

Seems reasonable to drop them unless someone speaks up.

Patrick

On Thu, May 9, 2019 at 10:11 AM Andor Molnar 
wrote:

> Hi Zili,
>
> I'm surely not the best person to talk about ZooKeeper history, but as far
> as I know these 2 input archives are not actively maintained and I've never
> seen them used in production.
> We probably don't have test coverage for them either, so keeping them in
> the codebase could be questionable.
>
> Regards,
> Andor
>
>
>
>
> On Sat, Apr 13, 2019 at 7:00 PM Zili Chen  wrote:
>
> > Hi,
> >
> > I'm not sure whether user list is a proper place but seems dev list
> > is filled of notifications.
> >
> > During an investigation of the possibility that ZooKeeper support multi
> > serialization frameworks, I found that in jute, CSVInputArchive and
> > XMLInputArchive are never in use. I wonder the story of these
> > implementations and whether they are still valid.
> >
> > Best,
> > tison.
> >
>

Re: Zookeepr quorum and electing a leader.

[Patrick Hunt]

I've never seen this. Are there jira/logs/... that you can point to that
would help identify the version and potentially reproduce?

Patrick

On Mon, Apr 22, 2019 at 10:57 AM Susheel Kumar 
wrote:

> Something similar we have seen during network/VM glitches and making Solr
> nodes go down/in a recovery state.
>
> On Mon, Apr 22, 2019 at 10:38 AM Grzegorz Solecki 
> wrote:
>
> > Having 5 zookeeper server instances in the cluster, is it possible that
> all
> > zookeeper instances will stop responding if one of the zookeeper
> instances
> > stops responding?
> >
> > By stops responding I mean the state in which zookeeper service does not
> > send TCP responses back to other zookeeper instances.
> >
> > Kind regards
> > Greg.
> >
>

Re: Limit on zookeeper ephemeral nodes

[Patrick Hunt]

On Mon, Apr 8, 2019 at 4:45 PM alwin james  wrote:

> Hi Team,
>
> I would like to understand, as part of maintaining a healthy zookeeper
> cluster, does the number of ephemeral nodes important? If so, is there any
> soft/hard limit on the number?
>

No limits outside the heap.


> Or is it more on the amount of data in each node that is important?
>
>
Yes, that will really be your limit.


> What are the other important health checks typically done in production
> systems?
> Any help/pointers is highly appreciated.
>

Take a look at the admin server via jetty, it provides a bunch of metrics
that are good to monitor in production. I believe there has been a bunch of
work recently to add more (I have not been following):
https://zookeeper.apache.org/doc/r3.5.4-beta/zookeeperAdmin.html#sc_zkCommands

Patrick


>
> Regards,
>

Re: openjdk 11 compatibility

[Patrick Hunt]

We've been verifying 3.4 against jdk11 - openjdk11 is not currently an
option available in the apache jenkins instance. However it should be
similar.
https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch34_java11/
but you'll need to go to 3.4.14 according to this
https://issues.apache.org/jira/browse/ZOOKEEPER-3148

Patrick


On Tue, Apr 9, 2019 at 6:32 PM Hari Nakka  wrote:

> ZK Version: 3.4.11
> JDK: 1.8
>
> Thinking of upgrading to openjdk 11. Is it fully tested?
>

Re: Unexpected delay between pings sent from the client to server

[Patrick Hunt]

Try turning on verbose GC logging and look if it's GC. In my experience
that's often the cause. "client application suddenly loses
its Zookeeper session" - might be client GC, or it could be server GC. See
if either of those shed light.

Patrick

On Sat, Apr 6, 2019 at 1:10 PM Norbert Kalmar 
wrote:

> Hi Gelbana,
>
> max_latency tells you the time elapsed between creating the request and
> FinalRequestProcessor processing it. So the cause for being that high could
> be basically anything.
> Turning on debug log for ZooKeeper server could help pinpoint at what point
> the request get stuck for so long.
>
> Regards,
> Norbert
>
> On Wed, Apr 3, 2019 at 3:07 PM Muhammad Gelbana 
> wrote:
>
> > Another couple of things I found:
> >
> > *A couple of Zookeeper client threads are stuck at these stacktraces for
> > ~30 seconds*
> > "pool-2-thread-1-EventThread" #1218 daemon prio=5 os_prio=0
> > tid=0x7ff3f5e23800 nid=0x5cd8 waiting on condition
> [0x7ff3ef803000]
> >java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00018d6d8ed8> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > at
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> > at
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> > at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:504)
> >
> > "pool-2-thread-1-SendThread(72.55.136.25:2181)" #1217 daemon prio=5
> > os_prio=0 tid=0x7ff3f5e23000 nid=0x5cd7 runnable [0x7ff3ef904000]
> >java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x00018d68a730> (a sun.nio.ch.Util$3)
> > - locked <0x00018d68a720> (a
> java.util.Collections$UnmodifiableSet)
> > - locked <0x00018d68a258> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at
> >
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
> > at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1145)
> >
> > *Running the mntr command returned the following stats*
> > zk_version3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on
> > 06/29/2018 04:05 GMT
> > zk_avg_latency0
> > zk_max_latency*17657*
> > zk_min_latency0
> > zk_packets_received1427134
> > zk_packets_sent1596974
> > zk_num_alive_connections64
> > zk_outstanding_requests0
> > zk_server_statefollower
> > zk_znode_count1394
> > zk_watch_count592
> > zk_ephemerals_count192
> > zk_approximate_data_size181257
> > zk_open_file_descriptor_count94
> > zk_max_file_descriptor_count1048576
> > zk_fsync_threshold_exceed_count1
> >
> > I find the *zk_max_latency* extremely hight. I'm wondering what kind of
> > latency is that ? How can I debug the reason for this value ?
> >
> > Thanks,
> > Gelbana
> >
> >
> >
> > On Wed, Apr 3, 2019 at 1:42 PM Muhammad Gelbana 
> > wrote:
> >
> > > I'm trying to debug a problem where our client application suddenly
> loses
> > > its Zookeeper session. I concluded that by looking at the Zookeeper
> > server
> > > logs.
> > >
> > > I increased the logging details for the client and found the following
> > log
> > > messages
> > >
> > >> DEBUG: [07:33:33] [demo | HA | Manager] Got ping response for
> sessionid:
> > >> 0x3000da76fa904b6 after 0ms
> > >> [org.apache.zookeeper.ClientCnxn$SendThread.readResponse]
> > >> DEBUG: [07:34:07] [demo | HA | Manager] Got ping response for
> sessionid:
> > >> 0x3000da76fa904b6 after 0ms
> > >> [org.apache.zookeeper.ClientCnxn$SendThread.readResponse]
> > >> DEBUG: [07:34:40] [demo | HA | Manager] Got ping response for
> sessionid:
> > >> 0x3000da76fa904b6 after 0ms
> > >> [org.apache.zookeeper.ClientCnxn$SendThread.readResponse]
> > >> DEBUG: [07:35:13] [demo | HA | Manager] Got ping response for
> sessionid:
> > >> 0x3000da76fa904b6 after 0ms
> > >> [org.apache.zookeeper.ClientCnxn$SendThread.readResponse]
> > >> DEBUG: [07:35:47] [demo | HA | Manager] Got ping response for
> sessionid:
> > >> 0x3000da76fa904b6 after 0ms
> > >> [org.apache.zookeeper.ClientCnxn$SendThread.readResponse]
> > >> DEBUG: [07:36:20] [demo | HA | Manager] Got ping response for
> sessionid:
> > >> 0x3000da76fa904b6 after 0ms
> > >> [org.apache.zookeeper.ClientCnxn$SendThread.readResponse]
> > >>
> > >
> > > I noticed that the duration between each log message is ~33 seconds
> while
> > > on another environment (my laptop), the duration goes down to ~1
> 

Apache ZooKeeper meetup in Palo Alto March 14th

[Patrick Hunt]

Hi folks. Andor is visiting the USA and he asked if I could facilitate a
meetup the night of March 14th - 530-8pm. This would be in Palo Alto at
Cloudera's HQ - 395 Page Mill Rd, Palo Alto. I will reserve one of our
meeting rooms and order pizza and beer. I plan to keep it low key, a bit of
an "unconference" if you will. A chance for people to catch up face to
face, discuss interesting topics and generally hang out. If you do have
something you'd like to present please let me know.

Regards,

Patrick

Re: Update from 3.4.9 to 3.4.13

[Patrick Hunt]

Review the release notes for each release, they capture the changes.
https://zookeeper.apache.org/releases.html

Given you are upgrading across fix versions we try to only apply fixes (no
features) however sometimes we end up breaking the rules - e.g. if a
security issue comes up.

Regards,

Patrick


On Fri, Dec 14, 2018 at 4:07 PM Karthik K G  wrote:

> Hi Team,
>
> We are trying to upgrade the zookeeper from 3.4.9 to 3.4.13 version, but we
> do not see any major differences in the configurations other than some bug
> fixes, are we having any additional features that we are missing.
>
> Thanks & regards,
> Karthik
>

Re: [Help Wanted] Will zookeeper merge change events?

[Patrick Hunt]

One other thing comes to mind. "service instances restart at the same time"
- have you considered that the issue may be on the ZK server and not on the
client? For example are you co-locating ZK servers with other services
and/or sharing spindles? You might be running into resource contention on
the server, for example slow fsyncs due to disk IO contention can be a real
killer for latency. (that one is easy to check for in the server logs).

Patrick

On Wed, Oct 24, 2018 at 7:40 AM Patrick Hunt  wrote:

> Nothing says that you have to handle client side notifications one-by-one,
> blocking the ZK client notification thread. You can do the aggregation
> yourself if you like, and this can be done very quickly. You can have
> separate threads to process the results (separate from the ZK client
> notification thread). That aspect of the architecture is up to you.
>
> This is some time back, but at the time I was processing 50 thousand
> watches in a similar situation - trigger/herd.
> https://wiki.apache.org/hadoop/ZooKeeper/ServiceLatencyOverview
>
> Regards,
>
> Patrick
>
> On Tue, Oct 23, 2018 at 3:25 PM Michael Han  wrote:
>
>> Hi Jun,
>>
>> >> will it only notify the client of the 100th event or all events from 2
>> -
>> 100 will be notified?
>>
>> All events will be notified. Each watched event will be materialized as a
>> server side response and on client side, each watched event will be
>> processed individually.
>>
>> Depend on how your set watches and the scale of watch (e.g. a single
>> client
>> watch a million znode, or a million clients watch a single znode, or a mix
>> of both), there could be thundering herd effects if multiple watches are
>> triggered.
>>
>>
>> On Tue, Oct 23, 2018 at 4:40 AM Jun Liu  wrote:
>>
>> > Hi,
>> >
>> > Our project, Dubbo[1], an RPC framework, has using Zookeeper as the
>> > service discovery and config centre for a long time. Recently, we
>> received
>> > performance reports from users when a batch of service instances
>> restart at
>> > the same time. One thing I can figure out is that the change of one
>> > instance status will trigger one change event to the Registry Centre -
>> > Zookeeper, so, 100 instances will trigger 100 change events at the same
>> > time.
>> >
>> > AFAIK, Zookeeper client uses a single thread to handle all these events
>> > one by one, if that is the case, will zookeeper merge the following
>> events
>> > and only notify once? For example, if the Zookeeper client is handling
>> the
>> > 1st event, the rest 2-100 events are created, when the 1st event is
>> > finished, will it only notify the client of the 100th event or all
>> events
>> > from 2 - 100 will be notified?
>> >
>> > 1. https://github.com/apache/incubator-dubbo
>> >
>> > Best regards,
>> > Jun from Apache Dubbo (Incubating)
>> >
>> >
>>
>

Re: [Help Wanted] Will zookeeper merge change events?

[Patrick Hunt]

Nothing says that you have to handle client side notifications one-by-one,
blocking the ZK client notification thread. You can do the aggregation
yourself if you like, and this can be done very quickly. You can have
separate threads to process the results (separate from the ZK client
notification thread). That aspect of the architecture is up to you.

This is some time back, but at the time I was processing 50 thousand
watches in a similar situation - trigger/herd.
https://wiki.apache.org/hadoop/ZooKeeper/ServiceLatencyOverview

Regards,

Patrick

On Tue, Oct 23, 2018 at 3:25 PM Michael Han  wrote:

> Hi Jun,
>
> >> will it only notify the client of the 100th event or all events from 2 -
> 100 will be notified?
>
> All events will be notified. Each watched event will be materialized as a
> server side response and on client side, each watched event will be
> processed individually.
>
> Depend on how your set watches and the scale of watch (e.g. a single client
> watch a million znode, or a million clients watch a single znode, or a mix
> of both), there could be thundering herd effects if multiple watches are
> triggered.
>
>
> On Tue, Oct 23, 2018 at 4:40 AM Jun Liu  wrote:
>
> > Hi,
> >
> > Our project, Dubbo[1], an RPC framework, has using Zookeeper as the
> > service discovery and config centre for a long time. Recently, we
> received
> > performance reports from users when a batch of service instances restart
> at
> > the same time. One thing I can figure out is that the change of one
> > instance status will trigger one change event to the Registry Centre -
> > Zookeeper, so, 100 instances will trigger 100 change events at the same
> > time.
> >
> > AFAIK, Zookeeper client uses a single thread to handle all these events
> > one by one, if that is the case, will zookeeper merge the following
> events
> > and only notify once? For example, if the Zookeeper client is handling
> the
> > 1st event, the rest 2-100 events are created, when the 1st event is
> > finished, will it only notify the client of the 100th event or all events
> > from 2 - 100 will be notified?
> >
> > 1. https://github.com/apache/incubator-dubbo
> >
> > Best regards,
> > Jun from Apache Dubbo (Incubating)
> >
> >
>

Re: Healthcheck

[Patrick Hunt]

I created this a few years ago, afaik it still works:
https://github.com/phunt/zk-smoketest

Regards,

Patrick


On Wed, Sep 19, 2018 at 2:05 AM Jens Rantil  wrote:

> Hello,
>
> We are in need for a shell command that we can execute on a specific node
> to make sure that the node has come up and has synced up with the ensemble.
> Is there any such command? Currently we are doing
>
> curl --silent --show-error --fail http://localhost:8080/commands/stat |
> grep -qE '"server_state" : "(follower|leader)"'
>
> but I suspect that that only takes the master election into account and not
> the fact that we've synced up.
>
> Anyone that has a better solution? One idea would to wait for
> `initLimit*tickTime+someDelta` ms and make sure that the same Java system
> process is still running. I also notice that the official Kubernetes Helm
> chart simply is doing an `ruok` 4-letter command for ready and liveness
> check.
>
> Any input appreciated - thanks,
> Jens
> --
> Jens Rantil
> Backend engineer
> Tink AB
>
> Email: jens.ran...@tink.se
> Phone: +46 708 84 18 32
> Web: www.tink.se
>
> Facebook  Linkedin
> <
> http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary
> >
>  Twitter 
>

Re: Zookeeper3.4.12 c client compilation issues on HPUX environment

[Patrick Hunt]

The libtool package typically (e.g. centos and most linux distros based on
similar) has a macro directory under /usr/share/aclocal. Check if macros
came with your libtool package. If not you'll need to hardcode use of
libtool rather than relying on autotools finding/configuring it for you. It
the macros are there it may just be a matter of configuration when running
autoreconf.

Patrick

On Tue, Sep 11, 2018 at 1:52 AM Kiran  wrote:

> Hi,
> I have extracted zookeeper-3.4.12 tar and then ran ./configure
> --without-cppunit.
> Then ran "make" command.
>
> Getting below error with make.But aclocal-1.15 is installed in the
> machine.Running "aclocal --version" gives the location where it is
> installed.
>
> bash-4.3$ make
> CDPATH="${ZSH_VERSION+.}:" && cd . && /bin/sh
> /scratch/zookeeperhp/zookeeper-3.4.12/src/c/missing aclocal-1.15
> /scratch/zookeeperhp/zookeeper-3.4.12/src/c/missing[81]: aclocal-1.15:  not
> found.
> WARNING: 'aclocal-1.15' is missing on your system.
>  You should only need it if you modified 'acinclude.m4' or
>  'configure.ac' or m4 files included by 'configure.ac'.
>  The 'aclocal' program is part of the GNU Automake package:
>  
>  It also requires GNU Autoconf, GNU m4 and Perl in order to run:
>  
>  
>  
> *** Error exit code 127
>
>
> Then based on the documentation of zookeeper tried running "autoreconf
> -if".With this command getting below error.
>
> configure.ac:37: warning: macro 'AM_PATH_CPPUNIT' not found in library
> configure.ac:37: error: possibly undefined macro: AM_PATH_CPPUNIT
>   If this token and others are legitimate, please use m4_pattern_allow.
>   See the Autoconf documentation.
> configure.ac:57: error: possibly undefined macro: AC_PROG_LIBTOOL
> autoreconf: /usr/local/bin/autoconf failed with exit status: 1
>
>
>
> So I have inactivated 'AM_PATH_CPPUNIT' in configure.ac to get rid of this
> error.Now error is as below.But libtool is already installed in the
> machine.
>
> bash-4.3$ autoreconf -if
> configure.ac:57: error: possibly undefined macro: AC_PROG_LIBTOOL
>   If this token and others are legitimate, please use m4_pattern_allow.
>   See the Autoconf documentation.
> autoreconf: /usr/local/bin/autoconf failed with exit status: 1
>
> Please let us know what chanegs need to be done to get libraries for hpux.
>
>
>
> --
> Sent from: http://zookeeper-user.578899.n2.nabble.com/
>

Re: Java 11 OpenJDK/Oracle Java Release Cadence Questions

[Patrick Hunt]

Hi Jeremiah. It's failing consistently on Jenkins, unlikely to officially
support until someone addresses those:
https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper-trunk-java11/

We've been testing with openjdk for quite some time, those are supported.
The docs are ambiguous in that regard:
https://zookeeper.apache.org/doc/r3.4.13/zookeeperAdmin.html#sc_requiredSoftware
however I don't see why Oracle and OpenJDK wouldn't be supported. EOD it's
up to the community.

Patrick

On Tue, Sep 11, 2018 at 10:02 AM Jeremiah Adams 
wrote:

> Hello,
>
>
> Are there any documents available concerning Zookeeper's support for Java
> 11 and documents regarding models supporting Oracle's new licensing and
> release cadences?
>
>
> Thanks.
>
>
> Jeremiah Adams
> Software Engineer
> www.helixeducation.com
> Blog | Twitter<
> https://twitter.com/HelixEducation> | Facebook<
> https://www.facebook.com/HelixEducation> | LinkedIn<
> http://www.linkedin.com/company/3609946>
>

Re: Verify zookeeper cluster consistency

[Patrick Hunt]

iirc Ben (or was it someone else from Facebook) mentioned at a meetup about
doing some work in this area. I believe they were hashing the tree but I
may be mis-remembering Perhaps one of those folks can shed light.

Patrick

On Fri, Jul 27, 2018 at 2:04 AM Andor Molnar 
wrote:

> Hi Klearchos,
>
> Some tool which is constantly querying the znode tree on each of every
> participant and tolerates some latency between the inconsistencies, but
> reports everything above the threshold could do the trick.
>
> Personally I'm not aware of anything existing, but feel free to dig github
> or create your own one.
>
> Regards,
> Andor
>
>
>
>
> On Fri, Jul 27, 2018 at 9:16 AM, Chaloulos, Klearchos (Nokia - GR/Athens) <
> klearchos.chalou...@nokia.com> wrote:
>
> > Hello,
> >
> > I recently ran into a case where my zookeeper servers had inconsistent
> > data, that is nodes that existed in one server did not exist in the other
> > two. I used 3 servers, with version 3.4.9. Unfortunately I cannot
> reproduce
> > the issue, as it occurred a long time ago in an environment outside my
> > control.
> >
> > However my question is: Is there a way to detect zookeeper cluster
> > inconsistencies? I am not talking about transient inconsistencies that
> > exist for, say, a few seconds, but for permanent inconsistencies. In my
> > case the inconsistencies persisted for a month!
> >
> > Best regards,
> >
> > Klearchos
> >
> >
>

Re: Zookeeper memory usage

[Patrick Hunt]

JVM memory usage is limited by your java heap configuration. GC will kick
in to keep it within the bound but the JVM will try to use whatever you
configure as the max.

Patrick

On Tue, Jul 24, 2018 at 4:27 AM Alex Dzyoba  wrote:

> I forgot to mention that I'm using Zookeeper version 3.5.3-beta
> On Tue, Jul 24, 2018 at 2:16 PM Alex Dzyoba  wrote:
> >
> > Hi,
> >
> > Could someone shed some light on why Zookeeper eats that much memory?
> > I have a cluster of 3 with 11 million znodes in it and `mntr` tells me
> > that the approximate data size is 9GB (9019066201 precisely) and the
> > snapshot size on disk is 9.3 GB. But the JVM itself eats a whopping
> > 26GB of RAM.
> >
> > Here is the jstat output for my leader instance:
> >
> >  S0CS1CS0US1U  EC   EUOC OU
> > MC MUCCSC   CCSU   YGC YGCTFGCFGCT GCT
> >  0.0   114688.0  0.0   114688.0 12058624.0 11141120.0 19193856.0
> > 16508964.9 17024.0 16305.7 1920.0 1701.6728   68.993   0
> > 0.000   68.993
> >
> > If we sum usage for all spaces (S0U + S1U + EU + OU + MU + CCSU) it
> > will be 26 GB.
> >
> > Why is there such mismatch between zookeeper approximate data size of
> > 9GB and actual JVM usage (26GB)? Even if we count only G1 old space
> > it's still 16 GB memory usage.
> >
> > I use openjdk 1.8.0_141 with G1GC.
> >
> > Thanks!
>

Re: Trying to figure out what these errors mean

[Patrick Hunt]

On Tue, Jul 24, 2018 at 10:17 AM Martin Cigorraga <
mailing.lists.forwar...@gmail.com> wrote:

> Hi all,
>
> I'm not a DS guys but an Ops guy instead fairly new to Zookeeper and
> Kafka; already spent a ton of time trying to understand what's going
> on here but so far I just scratched the surface of the issue without
> having a complete understanding of it.
>
> The issue:
> We have a cluster of Kafka with a Zookeeper coordinating them all; now
> the Zookeeper box log file (/var/log/zookeeper/zookeeper.out) is being
> fill with this error:
>
> 2018-07-24 08:23:43,178 [myid:3] - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@383] -
> Exception causing close of session 0x317d522019e: Len error
> 1119502
>

Hi Martin.

ZK has a configured limit for message lengths - this is being exceeded for
some reason on the session. Likely Kafka is sending too large a dataset, or
perhaps you have too many child znodes - unfortunately both of these
questions are Kafka specific (acting as a zk client) issues that I don't
have insight on.

See the ZK admin guide for more insight on the config - jute.maxbuffer is
the setting here:
http://zookeeper.apache.org/doc/r3.4.13/zookeeperAdmin.html#Unsafe+Options
Notice you are exceeding that limit "1119502".



> 2018-07-24 08:23:43,178 [myid:3] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1040] -
> Closed socket connection for client /10.20.3.16:40618 which had
> sessionid 0x317d522019e
> 2018-07-24 08:23:43,492 [myid:3] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@215] -
> Accepted socket connection from /10.20.2.24:49410
> 2018-07-24 08:23:43,492 [myid:3] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@931] -
> Client attempting to renew session 0x317d52200b1 at
> /10.20.2.24:49410
> 2018-07-24 08:23:43,493 [myid:3] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@683] -
> Established session 0x317d52200b1 with negotiated timeout 3
> for client /10.20.2.24:49410
>
> //
>
> The IP 10.20.2.24 refers to the box where we run Kafka Manager and
> Kafka Offset Monitor. Looking into the Kafka Offset Monitor logs I can
> see:
>
> 09:06:56 INFO  ClientCnxn:975 - Opening socket connection to server
> ip-10-20-1-6.ec2.internal/10.20.1.6:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 09:06:58 INFO  ClientCnxn:975 - Opening socket connection to server
> ip-10-20-1-6.ec2.internal/10.20.1.6:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 09:06:59 INFO  ClientCnxn:975 - Opening socket connection to server
> ip-10-20-1-6.ec2.internal/10.20.1.6:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 09:07:03 INFO  ClientCnxn:975 - Opening socket connection to server
> ip-10-20-1-6.ec2.internal/10.20.1.6:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 09:07:07 INFO  ClientCnxn:975 - Opening socket connection to server
> ip-10-20-1-6.ec2.internal/10.20.1.6:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 09:07:07 INFO  ClientCnxn:852 - Socket connection established to
> ip-10-20-1-6.ec2.internal/10.20.1.6:2181, initiating session
> 09:07:17 INFO  ClientCnxn:975 - Opening socket connection to server
> ip-10-20-1-6.ec2.internal/10.20.1.6:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 09:07:17 INFO  ClientCnxn:852 - Socket connection established to
> ip-10-20-1-6.ec2.internal/10.20.1.6:2181, initiating session
> 09:07:17 INFO  ClientCnxn:1235 - Session establishment complete on
> server ip-10-20-1-6.ec2.internal/10.20.1.6:2181, sessionid =
> 0x2178e5d0083, negotiated timeout = 3
>
>
Are you using auth? e.g. Kerberos? Basically we attempt to connect using
SASL, if the attempt fails we retry without it. (and as a result no auth)


> With 10.20.1.6 being the Zookeeper box. There's definitely something
> fishy going on here but I can't untanggle what is it.
>
> I'd thankfuly appreciate if you guys can point me into the right
> direction to where can I learn more about these errors.
>

You might check the Kafka docs to see if they talk about the jute.maxbuffer
issue. I'm assuming they see this on larger deployments and must discuss it
somewhere.

Regards,

Patrick



> Thanks!
>

Re: zookeeper c client build error

[Patrick Hunt]

We used to have access to solaris on Apache Jenkins, however that was
removed some time ago. As a result we are no longer able to compile/test
against that OS:
https://builds.apache.org/computer/

You might check some of the open jiras to get more insight, esp those
related to libtool:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20ZOOKEEPER%20AND%20text%20~%20libtool%20and%20resolution%20%3D%20unresolved

>From the looks of what you sent, in particular

libtool: link: /bin/nm -p   ./.libs/libzkst.a ./.libs/libhashtable.a |  |
./libtool[1732]: eval: syntax error at line 1: `|' unexpected

it may just be a matter of updating the makefile to remove those trailing
characters. If you work through this issue please do consider submitting a
patch.

Regards,

Patrick

On Wed, Jun 27, 2018 at 8:17 AM Ashwini Keshavarao 
wrote:

> I am trying to build zookeeper 3.4.12 c client on solaris machine. Below
> are
> the steps that I followed,
>
> 1) unzip/untar the source tarball and cd to the zookeeper-x.x.x/src/c
> directory.
> 2) Do a "./configure [OPTIONS]" to generate the makefile.
>  ./configure --without-cppunit
>
> This step fails with below error,
>
> /bin/sh ./libtool  --tag=CC--mode=link gcc  -Wall -Werror -g -O2
> -D_GNU_SOURCE -no-undefined -version-info 2 -lnsl -lsocket
> -export-symbols-regex
> '(zoo_|zookeeper_|zhandle|Z|format_log_message|log_message|logLevel|deallocate_|allocate_|zerror|is_unrecoverable)'
>
> -o libzookeeper_st.la -rpath /usr/local/lib  libzkst.la libhashtable.la
> libtool: link: /bin/nm -p   ./.libs/libzkst.a ./.libs/libhashtable.a |  |
> /bin/gsed 's/.* //' | sort | uniq > .libs/libzookeeper_st.exp
> ./libtool[1732]: eval: syntax error at line 1: `|' unexpected
> *** Error code 3
> make: Fatal error: Command failed for target `libzookeeper_st.la'
>
> Any idea on the error?
>
> Regards,
> Ashwini.
>
>
>
>
>
>
> --
> Sent from: http://zookeeper-user.578899.n2.nabble.com/
>

Re: Enabling TTL at ZkServer (3.5.4-beta)

[Patrick Hunt]

Are the release notes useful? They talk about this change:
http://zookeeper.apache.org/doc/r3.5.4-beta/releasenotes.html
Notice that documentation has also been updated to reflect what need sto be
done.

Patrick

On Tue, Jun 12, 2018 at 9:00 AM harish lohar  wrote:

> Getting below exception even though i have enabled below in zoo.cfg
> *ttlNodesEnabled=true*
>
> 2018-06-12 10:58:23,876 [myid:2] - INFO  [ProcessThread(sid:2
> cport:-1)::PrepRequestProcessor@878] - Got user-level KeeperException when
> processing sessionid:0x2459ac10002 type:createTtl cxid:0x6
> zxid:0x1003e txntype:-1 reqpath:n/a Error Path:null
> Error:KeeperErrorCode = Unimplemented
>
> Is there anything else needed to be configured ?
>
> Thanks
> Harish
>

Re: starting multiple zookeeper nodes from one command

[Patrick Hunt]

I created this a while ago, I still use it frequently, YMMV:
https://github.com/phunt/zkconf

Patrick

On Fri, Jun 1, 2018 at 9:25 AM Prasanth Mathialagan <
prasanthmathiala...@gmail.com> wrote:

> I don't quite understand what you meant by services. Are you referring to
> running as system service?
>
> On Fri, Jun 1, 2018, 9:17 AM THADC 
> wrote:
>
> > Hi thanks for response. Yes, I could use a script, in fact will start the
> > nodes as services. So, maybe they need to remain separate commands since
> > separate services I guess?
> >
> >
> >
> > --
> > Sent from: http://zookeeper-user.578899.n2.nabble.com/
> >
>

[ANNOUNCE] Apache ZooKeeper 3.5.4-beta

[Patrick Hunt]

The Apache ZooKeeper team is proud to announce Apache ZooKeeper version
3.5.4-beta

ZooKeeper is a high-performance coordination service for distributed
applications. It exposes common services - such as naming,
configuration management, synchronization, and group services - in a
simple interface so you don't have to write them from scratch. You can
use it off-the-shelf to implement consensus, group management, leader
election, and presence protocols. And you can build on it for your
own, specific needs.

For ZooKeeper release details and downloads, visit:
http://zookeeper.apache.org/releases.html

ZooKeeper 3.5.4-beta Release Notes are at:
http://zookeeper.apache.org/doc/r3.5.4-beta/releasenotes.html

We would like to thank the contributors that made the release possible.

Regards,

The ZooKeeper Team

Re: Apache ZooKeeper meetup May 9th in Palo Alto?

[Patrick Hunt]

Sorry, it wasn't recorded. Perhaps the presenters can post their slides.

Regards,

Patrick

On Fri, May 11, 2018 at 10:02 AM, Edward Ribeiro <edward.ribe...@gmail.com>
wrote:

> Oh, I missed the streaming. :(
>
> Was it recorded? The slides are available anywhere?
>
> On Thu, May 10, 2018 at 2:22 PM, Srikanth Viswanathan <
> srikant...@gmail.com>
> wrote:
>
> > Thanks for the presentations! Will the slides be posted somewhere?
> >
> > On Thu, May 10, 2018, 09:36 Prasanth Mathialagan <
> > prasanthmathiala...@gmail.com> wrote:
> >
> > > Gosh!! I missed it!! May be next time :)
> > >
> > > On Thu, May 10, 2018 at 8:24 AM, Andor Molnar <an...@cloudera.com>
> > wrote:
> > >
> > > > Thanks everybody for joining.
> > > > It was great to see you folks in person and talk.
> > > >
> > > > Regards,
> > > > Andor
> > > >
> > > >
> > > >
> > > > On Wed, May 9, 2018 at 9:44 PM, Patrick Hunt <ph...@apache.org>
> wrote:
> > > >
> > > > > On Wed, May 9, 2018 at 9:24 PM, Jeff Widman <j...@jeffwidman.com>
> > > wrote:
> > > > >
> > > > > > Many thanks to Patrick, Andor and the rest of the Cloudera team
> for
> > > > > hosting
> > > > > > us tonight. I appreciated the chance to compare notes with other
> > > users
> > > > > and
> > > > > > also discuss some of where the future of Zookeeper is heading.
> > > > > >
> > > > > >
> > > > > Thanks everyone for attending. It was great to see some old as well
> > as
> > > > new
> > > > > faces.
> > > > >
> > > > >
> > > > > > Cheers,
> > > > > > Jeff
> > > > > >
> > > > > > PS: Apologies to the remote folks for the dead sound at the
> > end--the
> > > > > > hangout connection to the conference room sound system died, and
> > > nobody
> > > > > in
> > > > > > the room knew how to reconnect it.
> > > > > >
> > > > > >
> > > > > Yea, sorry about that. IT did some magic to connect the hangout
> into
> > > the
> > > > > internal audio system of the room and I wasn't able to figure out
> the
> > > > codes
> > > > > they used.
> > > > >
> > > > > Regards,
> > > > >
> > > > > Patrick
> > > > >
> > > > >
> > > > > >
> > > > > > On Fri, Apr 27, 2018 at 6:40 PM, Srikanth Viswanathan <
> > > > > > srikant...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Confirming attendance from Seattle as well. Looking forward to
> > the
> > > > > > > presentations. Particularly excited for the containers talk!
> > > > > > >
> > > > > > > On Thu, Apr 26, 2018, 10:29 Patrick Hunt <ph...@apache.org>
> > wrote:
> > > > > > >
> > > > > > > > Ok, great. I have two speakers already lined up: one on ZK
> and
> > > > > > containers
> > > > > > > > and another on ZK failure modes and recovery. If you haven't
> > > > reached
> > > > > > out
> > > > > > > to
> > > > > > > > me yet and you have something to talk about please LMK asap.
> > > > > > > >
> > > > > > > > I believe we will have video conference available but I'm not
> > > > > entirely
> > > > > > > sure
> > > > > > > > yet - I will try.
> > > > > > > >
> > > > > > > > I'll finalize things and send out a more detailed agenda.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > > Patrick
> > > > > > > >
> > > > > > > > On Thu, Apr 26, 2018 at 1:33 AM, Shivam Goel <
> > > shivam@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Count me in !!
> > > > > > > > >
> > > > > > > > > On Mon, Apr 23, 2018 at 10:08 AM Patrick Hunt <
> > > ph...@apache.org>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi folks. I am interested in hosting a ZooKeeper meetup
> May
> > > 9th
> > > > > in
> > > > > > > > > > Cloudera's Palo Alto offices. It's been a while since we
> > last
> > > > got
> > > > > > > > > together,
> > > > > > > > > > lots of recent changes and some big plans, new additions
> to
> > > the
> > > > > PMC
> > > > > > > and
> > > > > > > > > > committer lists, new contributors.
> > > > > > > > > >
> > > > > > > > > > I was hoping to use the mailing lists to gauge interest.
> > > Please
> > > > > > reply
> > > > > > > > if
> > > > > > > > > > you
> > > > > > > > > > think you would be able to attend or would prefer a
> > different
> > > > > date.
> > > > > > > > Also
> > > > > > > > > > let me know if there's something you would like to
> present
> > to
> > > > the
> > > > > > > > group.
> > > > > > > > > > Food
> > > > > > > > > > and beer will be provided.
> > > > > > > > > >
> > > > > > > > > > Looking forward to hearing from everyone.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > >
> > > > > > > > > > Patrick
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > *Jeff Widman*
> > > > > > jeffwidman.com <http://www.jeffwidman.com/> | 740-WIDMAN-J
> > > (943-6265)
> > > > > > <><
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: removing ZK installation

[Patrick Hunt]

http://bigtop.apache.org/index.html

Patrick

On Tue, May 8, 2018 at 12:55 PM, harish lohar  wrote:

> Could someone please let me know where to get RPM for Centos for Zookeeper.
>
> Thanks
> Harish
>
> On Tue, May 8, 2018 at 1:57 PM, Washko, Daniel 
> wrote:
>
> > Steve, how was zookeeper installed? That should be the method with which
> > you remove it.
> >
> > If you are not sure how it was installed, you can do:
> >
> > rpm -qa |grep zookeeper
> >
> > To determine whether it was installed via an RPM package. If that does
> not
> > unearth a matching RPM then it was probably installed some other way.
> More
> > than likely it could have binary in an archive extracted to, maybe,
> > /opt/zookeeper.
> >
> > If you look at the running zookeeper process it should give you an idea
> of
> > where zookeeper is installed and where the data directory is:
> >
> > ps -ef |grep zookeeper
> >
> > How zookeeper is starting is dependent on which version of Centos you are
> > running. Centos 6 uses upstart and service command. More than likely you
> > will find the zookeeper init script in /etc/init.d. If this is Centos 7
> > then it's systemd. As root you can run systemctl by itself to get a list
> of
> > service scripts. Hit the "/" key and type in zookeeper. It will take you
> to
> > any service script with zookeeper in the name. This will help you
> determine
> > how to stop zookeeper.
> >
> > If neither systemd is showing a zookeeper service nor you see a service
> > script in /etc/init.d (or if service zookeeper stop doesn't work), then
> it
> > would appear that zookeeper was started in some other way, maybe manually
> > without a service or systemd script.
> >
> > You'll want to figure this out because if you have to manually remove
> > zookeeper, instead of using a package manager like RPM, you'll want to
> > disable any startup scripts from running and throwing errors once
> Zookeeper
> > is removed.
> >
> > On 5/8/18, 10:32 AM, "Steph van Schalkwyk" 
> > wrote:
> >
> > Find where it is installed - typically /opt/zookeeper.
> > Also do a which zookeeper to see if it is linked to /usr/bin or some
> > such
> > place.
> > Make sure zookeeper is stopped.
> > Far as I recall, Centos has Upstart, so sudo stop zookeeper and sudo
> > disable zookeeper. Or sudo systemctl stop zookeeper and sudo
> systemctl
> > disable zookeeper.
> > Then cat the /opt/zookeeper/conf/zoo.cfg to see where the data
> > directories
> > and logs are. Delete the data and log directories.
> > Then delete /opt/zookeeper.
> > Steph
> >
> >
> >
> > On Tue, May 8, 2018 at 9:07 AM, Steve Pruitt 
> > wrote:
> >
> > > Hi,
> > >
> > > I need to remove ZooKeeper from a Centos machine.  I tried yum
> > remove to
> > > no avail using instructions I found online.
> > >
> > > Thanks.
> > >
> > > -S
> > >
> > >
> >
> >
> >
>

Re: Confluence wiki contribution.. how?

[Patrick Hunt]

Unfortunately a while back we had to turn off edit perms for general users
due to spam. Send me your Confluence user ID privately and I'll update the
perms so that you can edit.

Regards,

Patrick

On Fri, May 4, 2018 at 12:55 AM, Andor Molnar  wrote:

> Hi Alberto,
>
> Your contribution is much appreciated (...at least your intention).
> Confluence pages are not generated from source, they can be edited
> directly, so you're right that you need write permissions which
> unfortunately I don't know how to get.
>
> More senior people can advise here I believe.
>
> Regards,
> Andor
>
>
>
> On Fri, May 4, 2018 at 9:24 AM Alberto Rodriguez 
> wrote:
>
> > Hi there,
> >
> > Today I was poking around Zookeeper's confluence docs and found out some
> > things that are not updated. I'd like to update them but not sure how to
> do
> > that. I've cloned the project thinking that the documentation was in
> > src/docs but is not there.
> >
> > Anyone could please share with me how to contribute? (I've already got an
> > apache user id, I'm logged into the confluence space but it seems that I
> do
> > not have edit permission)
> >
> > Thank you in advance!
> >
>

Re: ZooKeeper 3.4.11 bug: dataDir and dataLogDir swapped

[Patrick Hunt]

This is a good point Andor. I've updated the release page on the website to
reflect the regression addressed in ZOOKEEPER-2960 and upcoming fix.

Thanks!

Patrick

On Fri, Feb 2, 2018 at 1:07 AM, Andor Molnar  wrote:

> Hi all,
>
> Please be aware that 3.4.11 has a quite unfortunate bug which causes
> ZooKeeper to swap dataDir and dataLogDir parameters. If you configured ZK
> to use separate txn and snapshot folders in these two options and plan to
> upgrade, you'll experience that ZK is trying to load transaction logs from
> snapshot folder and vica versa.
>
> Fix is on the way, 3.4.12 will be released soon and it's recommended to
> postpone upgrading ZooKeeper until that.
>
> *dev*
> I think it'd be useful to add a similar warning message to the Releases
> page too.
>
> Regards,
> Andor
>

Re: Different disk writes per node

[Patrick Hunt]

Hi Eric. I saw your original post but didn't have anything of value to add.
I've never seen this myself. That said "never seen" not in the sense that
it's impossible, just in the sense that I've never looked for such an issue
and have never noticed such a thing. It does seem unusual however.

Unfortunately you didn't provide detail for speculating further. I'm not
very familiar with Exhibitor (know of it - +cc Jordan in case he has
insights) and without more details such as the version of ZK and in
particular the configs of the 5 servers it's hard to come up with much.
Also, are you able to correlate the outlier to a particular ZK role -
perhaps that's the leader? (although I'm not sure why that would matter...)

Patrick

On Wed, Jan 24, 2018 at 10:28 AM, ercliou  wrote:

> Any pointers that could lead to explanation of this behavior?
> Thanks.
>
>
>
> --
> Sent from: http://zookeeper-user.578899.n2.nabble.com/
>

Re: Any technical reasons that the 3.5.x series is still in beta?

[Patrick Hunt]

On Mon, Jan 22, 2018 at 3:47 PM, Jeff Widman  wrote:

> Is there any technical reasons the 3.5.x series is still in beta?
>
> Looking through existing issues, I don't see anything critical that makes
> 3.5.x less stable than the 3.4.x series...
>
> If the answer is "no tech reason, just the social reason that no maintainer
> has had time to push through a full stable release", then I understand that
> perfectly...
>

Mostly. However there is
https://issues.apache.org/jira/browse/ZOOKEEPER-2901 which is a very
serious regression and currently blocking 3.5.4.

Patrick


>
> --
>
> *Jeff Widman*
> jeffwidman.com  | 740-WIDMAN-J (943-6265)
> <><
>

Re: Why are ephemeral nodes written to disk?

[Patrick Hunt]

NP, sorry for the slow response but I've been out on vacation the past few
weeks. ;-)

Regards,

Patrick

On Wed, Jan 17, 2018 at 3:39 PM, Jeff Widman <j...@jeffwidman.com> wrote:

> Thank you. I did not realize sessions could continue even if the ensemble
> was shutdown.
>
> On Jan 17, 2018 3:28 PM, "Patrick Hunt" <ph...@apache.org> wrote:
>
> > On Tue, Jan 9, 2018 at 12:38 PM, Jeff Widman <j...@jeffwidman.com>
> wrote:
> >
> > > Ephemeral nodes only exist for the life of the client session.
> > >
> > > As far as I understand, by definition, a client session ends when the
> > > entire zookeeper ensemble goes down.
> > >
> > > So I would expect that ephemeral nodes are only written to memory, not
> > > disk. The ephemeral nodes would be sync'd across machines as a client
> > > session can span multiple connections if a single zk server fails, but
> > once
> > > the ensemble is down there is no need to recover the ephemeral nodes
> from
> > > disk.
> > >
> > > However, when I looked at a zookeeper ensemble that is 99% ephemeral
> > nodes,
> > > I see a bunch of disk I/O from the zookeeper processes. So it appears
> > that
> > > ephemeral nodes are still written to disk...
> > >
> > > Why is this?
> > >
> >
> > Ephemeral znodes are treated just like persistent znodes in the sense
> that
> > a quorum of nodes need to agree to any change. As such the znode is
> written
> > to the transaction log.
> >
> > "a client session ends when the entire zookeeper ensemble goes down"
> >
> > is not correct. A client session ends either when a client closes it's
> > session explicitly or the ZK quorum leader decides that the session has
> > expired (which is based on the negotiated session timeout). Only while a
> > leader is active can a session be expired (or closed for that matter).
> When
> > you shutdown an ensemble the sessions are maintained. If you were to, for
> > example, shut down an ensemble for an hour and then restart it the
> sessions
> > would still be active. The clock would "reset" when the new leader was
> > elected. If the client session is still active the session would
> continue,
> > any ephemeral znodes would still exist.
> >
> > Patrick
> >
> >
> > >
> > > --
> > >
> > > *Jeff Widman*
> > > jeffwidman.com <http://www.jeffwidman.com/> | 740-WIDMAN-J (943-6265)
> > > <><
> > >
> >
>

Re: Why are ephemeral nodes written to disk?

[Patrick Hunt]

On Tue, Jan 9, 2018 at 12:38 PM, Jeff Widman  wrote:

> Ephemeral nodes only exist for the life of the client session.
>
> As far as I understand, by definition, a client session ends when the
> entire zookeeper ensemble goes down.
>
> So I would expect that ephemeral nodes are only written to memory, not
> disk. The ephemeral nodes would be sync'd across machines as a client
> session can span multiple connections if a single zk server fails, but once
> the ensemble is down there is no need to recover the ephemeral nodes from
> disk.
>
> However, when I looked at a zookeeper ensemble that is 99% ephemeral nodes,
> I see a bunch of disk I/O from the zookeeper processes. So it appears that
> ephemeral nodes are still written to disk...
>
> Why is this?
>

Ephemeral znodes are treated just like persistent znodes in the sense that
a quorum of nodes need to agree to any change. As such the znode is written
to the transaction log.

"a client session ends when the entire zookeeper ensemble goes down"

is not correct. A client session ends either when a client closes it's
session explicitly or the ZK quorum leader decides that the session has
expired (which is based on the negotiated session timeout). Only while a
leader is active can a session be expired (or closed for that matter). When
you shutdown an ensemble the sessions are maintained. If you were to, for
example, shut down an ensemble for an hour and then restart it the sessions
would still be active. The clock would "reset" when the new leader was
elected. If the client session is still active the session would continue,
any ephemeral znodes would still exist.

Patrick


>
> --
>
> *Jeff Widman*
> jeffwidman.com  | 740-WIDMAN-J (943-6265)
> <><
>

Re: Zookeeper session expiration

[Patrick Hunt]

Easy enough to try out. Give it a shot and enter a jira if you find an
issue.

Regards,

Patrick

On Thu, Dec 7, 2017 at 5:47 AM, Jordan Zimmerman <jor...@jordanzimmerman.com
> wrote:

> System.nanoTime() is not affected by clock changes. Really everyone - this
> is simply not an issue in ZooKeeper.
>
> 
> Jordan Zimmerman
>
> > On Dec 7, 2017, at 7:43 AM, Kathryn Hogg <kathryn.h...@oati.net> wrote:
> >
> > I'm pretty new to zookeeper but have a fair amount of experience with
> virtual synchrony going back many years.  Even though time is relative, it
> is possible that if the clock suddenly jumps forward on the server to
> prematurely declare timeouts as expired.  I'm not sure how Zookeeper
> handles that but in Isis, if 2 consecutive calls to gettimeofday had too
> large of a difference, it considered it fishy.
> >
> > Of course, this is why we use ntp with adjtime to avoid clocks going
> backwards or making large jumps forward.
> >
> > -Original Message-
> > From: Patrick Hunt [mailto:ph...@apache.org]
> > Sent: Wednesday, December 06, 2017 5:18 PM
> > To: UserZooKeeper <user@zookeeper.apache.org>
> > Subject: Re: Zookeeper session expiration
> >
> > {External email message: This email is from an external source. Please
> exercise caution prior to opening attachments, clicking on links, or
> providing any sensitive information.}
> >
> > What Jordan said + time use is only in the relative sense, not the
> absolute. Session tracking (expiration) is relative to the start of
> leadership.
> >
> > Patrick
> >
> >> On Mon, Dec 4, 2017 at 12:21 PM, Jordan Zimmerman <
> jor...@jordanzimmerman.com> wrote:
> >>
> >> ZooKeeper, indeed, does not use wall clock time. It uses
> >> System.nanoTime() for most operations. Further, all operations go
> >> through the Leader node so only the Leader's notion of time matters.
> >> The Leader manages the session via a "SessionTracker" instance. The
> code is in SessionTrackerImpl.java.
> >> There is a sessionExpiryQueue which is a kind of priority queue that
> >> returns expired sessions based on System.nanoTime().
> >>
> >> -JZ
> >>
> >>> On Dec 4, 2017, at 12:09 PM, Abraham Fine <af...@apache.org> wrote:
> >>>
> >>> Hello Anthony and Shawn-
> >>>
> >>> To the best of my knowledge ZooKeeper does not use the "wall clock"
> >>> time anywhere. So that should not be the problem.
> >>>
> >>> Please consider enabling debug logging, which should allow you to
> >>> track the "pings".
> >>>
> >>> Thanks,
> >>> Abe
> >>>
> >>>> On Mon, Dec 4, 2017, at 11:51, Anthony Shaya wrote:
> >>>> Thanks Shawn, should I message the developer mailing list for a
> >>>> more definitive answer?
> >>>>
> >>>> Thanks again for the reply.
> >>>>
> >>>> -Original Message-
> >>>> From: Shawn Heisey [mailto:apa...@elyograg.org]
> >>>> Sent: Monday, December 4, 2017 2:49 PM
> >>>> To: user@zookeeper.apache.org
> >>>> Subject: Re: Zookeeper session expiration
> >>>>
> >>>>> On 12/4/2017 8:22 AM, Anthony Shaya wrote:
> >>>>> My question is related to how session expiration works, I noticed
> >>>>> on
> >> many of the client machines the times across these machines were all
> >> off (by anywhere from 1 minute to 20 minutes - which was resolved
> >> after discovery - haven't verified this completely yet). Can this
> >> directly affect session expiration within the zookeeper cluster?
> >>>>>
> >>>>>  *   I read the following in https://na01.safelinks.
> >> protection.outlook.com/?url=https%3A%2F%2Fwiki.apache.org%
> >> 2Fhadoop%2FZooKeeper%2FFAQ=02%7C01%7C%7C6d6643860a4e4a8194c808d53
> >> b50 23ec%7Cc61157e903cb47589165ee7845cb0ca3%7C0%7C0%
> >> 7C636480137750841475=RwGGH19FLeYFmXMrg5GBkSLJ65ANj1
> >> EXkTvwyk6OLd4%3D=0 , "Expirations happens when the cluster
> >> does not hear from the client within the specified session timeout
> period (i.e.
> >> no heartbeat).". So in some case it seems like if the times were wrong
> >> across the machines its possible one of the clients could of
> >> effectively sent a heart beat in the past (not sure about this tbh)
> >> and then the cluster expires the session?
&g

Re: Zookeeper session expiration

[Patrick Hunt]

What Jordan said + time use is only in the relative sense, not the
absolute. Session tracking (expiration) is relative to the start of
leadership.

Patrick

On Mon, Dec 4, 2017 at 12:21 PM, Jordan Zimmerman <
jor...@jordanzimmerman.com> wrote:

> ZooKeeper, indeed, does not use wall clock time. It uses System.nanoTime()
> for most operations. Further, all operations go through the Leader node so
> only the Leader's notion of time matters. The Leader manages the session
> via a "SessionTracker" instance. The code is in SessionTrackerImpl.java.
> There is a sessionExpiryQueue which is a kind of priority queue that
> returns expired sessions based on System.nanoTime().
>
> -JZ
>
> > On Dec 4, 2017, at 12:09 PM, Abraham Fine  wrote:
> >
> > Hello Anthony and Shawn-
> >
> > To the best of my knowledge ZooKeeper does not use the "wall clock" time
> > anywhere. So that should not be the problem.
> >
> > Please consider enabling debug logging, which should allow you to track
> > the "pings".
> >
> > Thanks,
> > Abe
> >
> > On Mon, Dec 4, 2017, at 11:51, Anthony Shaya wrote:
> >> Thanks Shawn, should I message the developer mailing list for a more
> >> definitive answer?
> >>
> >> Thanks again for the reply.
> >>
> >> -Original Message-
> >> From: Shawn Heisey [mailto:apa...@elyograg.org]
> >> Sent: Monday, December 4, 2017 2:49 PM
> >> To: user@zookeeper.apache.org
> >> Subject: Re: Zookeeper session expiration
> >>
> >> On 12/4/2017 8:22 AM, Anthony Shaya wrote:
> >>> My question is related to how session expiration works, I noticed on
> many of the client machines the times across these machines were all off
> (by anywhere from 1 minute to 20 minutes - which was resolved after
> discovery - haven't verified this completely yet). Can this directly affect
> session expiration within the zookeeper cluster?
> >>>
> >>>   *   I read the following in https://na01.safelinks.
> protection.outlook.com/?url=https%3A%2F%2Fwiki.apache.org%
> 2Fhadoop%2FZooKeeper%2FFAQ=02%7C01%7C%7C6d6643860a4e4a8194c808d53b50
> 23ec%7Cc61157e903cb47589165ee7845cb0ca3%7C0%7C0%
> 7C636480137750841475=RwGGH19FLeYFmXMrg5GBkSLJ65ANj1
> EXkTvwyk6OLd4%3D=0 , "Expirations happens when the cluster does
> not hear from the client within the specified session timeout period (i.e.
> no heartbeat).". So in some case it seems like if the times were wrong
> across the machines its possible one of the clients could of effectively
> sent a heart beat in the past (not sure about this tbh) and then the
> cluster expires the session?
> >>
> >> I make these comments without any knowledge of what ZK code actually
> >> does.  I am a member of this list because I'm a representative of the
> >> Apache Solr project, which uses the ZK client in order to maintain a
> >> cluster.
> >>
> >> IMHO, any software which makes actual decisions based on the timestamps
> >> in messages from another system is badly designed.  I would hope that
> the
> >> ZK designers know this, and always make any decisions related to time
> >> using the clock in the local system only.
> >>
> >> If ZK's designers did the right thing, then a session timeout would
> >> indicate that quite literally no heartbeats were received in X seconds,
> >> as measured by the local clock, and the local clock ONLY ... NOT from
> >> timestamp information received from another system.
> >>
> >> Although such a lack of communication could be caused by any number of
> >> things, including network hardware failure, one of the most common
> >> reasons I have seen for problems like this is extreme java garbage
> >> collection pauses in the client software.
> >>
> >> Situations where the heap is a little bit too small can cause a java
> >> program to basically be doing garbage collection constantly, so it
> >> doesn't have much time to do anything else, like send heartbeats to ZK
> >> servers.
> >>
> >> Situations where the heap is HUGE and garbage collection is not well
> >> tuned can lead to pauses of a minute or longer while Java does a massive
> >> full GC.
> >>
> >>>   *   I don't have the zookeeper node log for the above time to see
> what was going on in zookeeper when the cluster determined the session
> expired.
> >>>
> >>>   *   Is there any additional logging I can turn on to troubleshoot zk
> session expiration issues?
> >>
> >> Hopefully your ZK clients also have logging.  Failing that, you could
> >> turn on GC logging for the software with the ZK client (assuming it's a
> >> Java client) and find a program or website that can examine the log and
> >> give you statistics or a graph of GC pauses.
> >>
> >> If there is a problem in software using the client and whatever logging
> >> is available doesn't help you figure out what's wrong, you're generally
> >> going to need to talk to whoever wrote that software for help
> >> troubleshooting it.
> >>
> >> Thanks,
> >> Shawn
> >>
> >>
> >>
> >> This message is intended exclusively for the individual or entity to
> >> which it is 

Re: Hi Zookeeper expert, we encountered a slow attack problem, described as follows, please help to analyze and confirm, thank you very much.

[Patrick Hunt]

PS. If this is an issue for you at the moment you might apply an iptables
based filter to address this until a proper patch can be applied. For
example something like

iptables -A INPUT -p tcp --syn --dport 2181 -m connlimit --connlimit-above
3 -j REJECT

will limit the number of connections from a particular IP address (2181
being the default ZK client port, YMMV)

Regards,

Patrick


On Mon, Nov 27, 2017 at 12:10 PM, Patrick Hunt <ph...@apache.org> wrote:

> Hi zhanggang, thank you for the report. Unfortunately the attachment did
> not come through (probably stripped by the ML daemon).
>
> Iiuc this does sound like a problem that would be good to address. We have
> code in place to limit the number of concurrent sessions that a particular
> IP can hold open at any one time - specifically to address the type of
> issue you describe. However I believe it only addresses session count, not
> tcp connections. This would be a good one to fix - please create a jira and
> if possible submit a patch.
>
> Thanks!
>
> Patrick
>
> On Tue, Nov 21, 2017 at 7:09 AM, zhanggang <zhangga...@huawei.com> wrote:
>
>>
>>
>> Hi Zookeeper expert, we encountered a slow attack problem, described as
>> follows, please help to analyze and confirm, thank you very much.
>>
>>
>>
>> *Problem:*
>>
>> In the client using some method (such as telnet) to establish a tcp
>> connection with server-side zookeeper listening port , but after the
>> establishment of tcp connection, the client does not send any data.
>> However, apache will not disconnect this tcp connection, may lead to the
>> number of connections exhausted , Resulting in DOS
>>
>>
>>
>> *The attack process**：*
>>
>> 1. The client executes the telnet service port
>>
>> Start the client telnet server port 21816,53236,50548, do not exit for a
>> long time:
>>
>> Excuting an order:
>>
>> telnet 3.101.3.119 21816
>>
>> telnet 3.101.3.118 53236
>>
>> telnet 3.101.3.124 50548
>>
>>
>>
>
>

Re: Hi Zookeeper expert, we encountered a slow attack problem, described as follows, please help to analyze and confirm, thank you very much.

[Patrick Hunt]

Hi zhanggang, thank you for the report. Unfortunately the attachment did
not come through (probably stripped by the ML daemon).

Iiuc this does sound like a problem that would be good to address. We have
code in place to limit the number of concurrent sessions that a particular
IP can hold open at any one time - specifically to address the type of
issue you describe. However I believe it only addresses session count, not
tcp connections. This would be a good one to fix - please create a jira and
if possible submit a patch.

Thanks!

Patrick

On Tue, Nov 21, 2017 at 7:09 AM, zhanggang  wrote:

>
>
> Hi Zookeeper expert, we encountered a slow attack problem, described as
> follows, please help to analyze and confirm, thank you very much.
>
>
>
> *Problem:*
>
> In the client using some method (such as telnet) to establish a tcp
> connection with server-side zookeeper listening port , but after the
> establishment of tcp connection, the client does not send any data.
> However, apache will not disconnect this tcp connection, may lead to the
> number of connections exhausted , Resulting in DOS
>
>
>
> *The attack process**：*
>
> 1. The client executes the telnet service port
>
> Start the client telnet server port 21816,53236,50548, do not exit for a
> long time:
>
> Excuting an order:
>
> telnet 3.101.3.119 21816
>
> telnet 3.101.3.118 53236
>
> telnet 3.101.3.124 50548
>
>
>

[ANNOUNCE] Apache ZooKeeper 3.4.11

[Patrick Hunt]

The Apache ZooKeeper team is proud to announce Apache ZooKeeper version
3.4.11

ZooKeeper is a high-performance coordination service for distributed
applications. It exposes common services - such as naming,
configuration management, synchronization, and group services - in a
simple interface so you don't have to write them from scratch. You can
use it off-the-shelf to implement consensus, group management, leader
election, and presence protocols. And you can build on it for your
own, specific needs.

For ZooKeeper release details and downloads, visit:
http://zookeeper.apache.org/releases.html

ZooKeeper 3.4.11 Release Notes are at:
http://zookeeper.apache.org/doc/r3.4.11/releasenotes.html

We would like to thank the contributors that made the release possible.

Regards,

The ZooKeeper Team

Re: Zookeeper quorum goes down for no apparent reason in 3.4.5

[Patrick Hunt]

27.0.0.1:
> 5002
> :FastLeaderElection@815] - New election. My id =  3, proposed
> zxid=0x50006fed5
> 2017-09-21 03:00:03,659 [myid:3] - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@597] - Notification: 1 (message
> format version), 3 (n.leader), 0x50006fed5 (n.zxid), 0xd (n.round), LOOKING
> (n.state), 3 (n.sid), 0x5 (n.peerEpoch) LOOKING (my state)
> 2017-09-21 03:00:03,660 [myid:3] - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@597] - Notification: 1 (message
> format version), 2 (n.leader), 0x50006fec3 (n.zxid), 0xd (n.round), LEADING
> (n.state), 2 (n.sid), 0x6 (n.peerEpoch) LOOKING (my state)
> 2017-09-21 03:00:03,697 [myid:3] - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@597] - Notification: 1 (message
> format version), 2 (n.leader), 0x50006fec3 (n.zxid), 0xd (n.round),
> FOLLOWING (n.state), 1 (n.sid), 0x6 (n.peerEpoch) LOOKING (my state)
> 2017-09-21 03:00:03,697 [myid:3] - INFO  [QuorumPeer[myid=3]/127.0.0.1:
> 5002
> :QuorumPeer@784] - FOLLOWING
> 2017-09-21 03:00:03,697 [myid:3] - INFO  [QuorumPeer[myid=3]/127.0.0.1:
> 5002
> :ZooKeeperServer@162] - Created server with tickTime 2000
> minSessionTimeout
> 4000 maxSessionTimeout 4 datadir /run/zookeeper/version-2 snapdir
> /run/zookeeper/version-2
> 2017-09-21 03:00:03,697 [myid:3] - INFO  [QuorumPeer[myid=3]/127.0.0.1:
> 5002
> :Follower@63] - FOLLOWING - LEADER ELECTION TOOK - 44
> 2017-09-21 03:00:03,700 [myid:3] - WARN  [QuorumPeer[myid=3]/127.0.0.1:
> 5002
> :Learner@338] - Truncating log to get in sync with the leader 0x50006fec3
> 2017-09-21 03:00:03,702 [myid:3] - INFO  [QuorumPeer[myid=3]/127.0.0.1:
> 5002
> :FileSnap@83] - Reading snapshot /run/zookeeper/version-2/snaps
> hot.50006fcf2
> 2017-09-21 03:00:03,707 [myid:3] - WARN  [QuorumPeer[myid=3]/127.0.0.1:
> 5002
> :Learner@374] - Got zxid 0x60001 expected 0x1
> 2017-09-21 03:00:03,708 [myid:3] - INFO  [QuorumPeer[myid=3]/127.0.0.1:
> 5002
> :FileTxnSnapLog@240] - Snapshotting: 0x60086 to
> /run/zookeeper/version-2/snapshot.60086
> 2017-09-21 03:00:03,773 [myid:3] - INFO  [SyncThread:3:FileTxnLog@199] -
> Creating new log file: log.60087
> 2017-09-21 03:00:03,776 [myid:3] - WARN  [QuorumPeer[myid=3]/127.0.0.1:
> 5002
> :Follower@118] - Got zxid 0x60088 expected 0x1
> 2017-09-21 03:00:03,895 [myid:3] - WARN
> [NIOServerCxn.Factory:localhost/127.0.0.1:5002:NIOServerCnxn@357] - caught
> end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x35ea260c1fb0002, likely client has closed socket
> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
> at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServ
> erCnxnFactory.java:208)
> at java.lang.Thread.run(Thread.java:745)
> 2017-09-21 03:00:03,978 [myid:3] - WARN
> [NIOServerCxn.Factory:localhost/127.0.0.1:5002:NIOServerCnxn@362] -
> Exception causing close of session 0x35ea260c1fb0003 due to
> java.io.IOException: Connection reset by peer
> 2017-09-21 03:00:04,000 [myid:3] - INFO
> [SessionTracker:SessionTrackerImpl@162] - SessionTrackerImpl exited loop!
> 2017-09-21 03:00:05,806 [myid:3] - WARN
> [NIOServerCxn.Factory:localhost/127.0.0.1:5002:NIOServerCnxn@357] - caught
> end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x35ea260c1fb0011, likely client has closed socket
> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
> at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServ
> erCnxnFactory.java:208)
> at java.lang.Thread.run(Thread.java:745)
> 2017-09-21 03:00:09,600 [myid:3] - INFO  [Snapshot
> Thread:FileTxnSnapLog@240]
> - Snapshotting: 0x6029c to /run/zookeeper/version-2/snapshot.6029c
> 2017-09-21 03:00:09,606 [myid:3] - INFO  [SyncThread:3:FileTxnLog@199] -
> Creating new log file: log.6029e
> 2017-09-21 03:00:10,399 [myid:3] - WARN
> [NIOServerCxn.Factory:localhost/127.0.0.1:5002:NIOServerCnxn@357] - caught
> end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x35ea260c1fb002f, likely client has closed socket
> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
> at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServ
> erCnxnFactory.java:208)
> at java.lang.Thread.run(Thread.java:745)
>
> Thanks,
> Anand.
>
> On Thu, Oct 5, 2017 at 9:50 AM, Patrick Hunt <ph...@apache.org> wrote:
>
> > Unfortunately I don't see any attached logs, which makes it difficult to
> > provide you with insight. "Not sufficient followers synced" indicates
> that
> > you're losing followers, likely they are falling behind - what is your
> > metric tracking telling your wrt load on the compute and load on the
> > disk/memory/network/etc... also metrics at the ZK level (e.g. are zk
> > latencies increasing?) Check the logs to see if you're seeing "fsync"
> > slowness issues (it's a warning in the server logs). This is a pretty
> > common issue. GC might also be an issue, although that's more rare these
> > days (hard to say w/o knowing your use case, etc...) Again, look to your
> > metrics collection for insight where to start.
> >
> > Patrick
> >
> > On Wed, Oct 4, 2017 at 11:17 AM, Anand Parthasarathy <
> > anpar...@avinetworks.com> wrote:
> >
> > > Hi,
> > >
> > > We have an issue with a 3-node zookeeper ensemble where the quorum goes
> > > down due to no apparent reason every once in a while. Here is what I
> see
> > in
> > > the ZK leader:
> > >
> > > 2017-09-21 03:00:03,648 [myid:3] - INFO  [QuorumPeer[myid=3]/127.0.0.1
> :
> > > 5002:Leader@493] - Shutting down
> > > 2017-09-21 03:00:03,648 [myid:3] - INFO  [QuorumPeer[myid=3]/127.0.0.1
> :
> > > 5002:Leader@499] - Shutdown called
> > > java.lang.Exception: shutdown Leader! reason: Not sufficient followers
> > > synced, only synced with sids: [ 3 ]
> > > at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.
> > java:499)
> > > at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:474)
> > > at org.apache.zookeeper.server.quorum.QuorumPeer.run(
> > > QuorumPeer.java:799)
> > >
> > > I have attached the logs from the 3 nodes around this time. Could you
> > pls.
> > > help understand what the issue could be here. The only thing I see a
> > little
> > > bit ahead of this timestamp is that all of them did a PurgeTask pretty
> > much
> > > at the same time.
> > >
> > > Thanks,
> > > Anand.
> > >
> >
>

Re: Node can be created and listed, but not updated or deleted

[Patrick Hunt]

Could be. What does the environment look like?

Patrick

On Thu, Oct 5, 2017 at 4:55 AM, Stevo Slavić <ssla...@gmail.com> wrote:

> I created same node/path on a different cluster and delete works, so yes,
> it seems length or characters used in the path is not an issue.
>
> What else could it be? Corrupted data?
>
> On Wed, Sep 27, 2017 at 8:29 PM, Patrick Hunt <ph...@apache.org> wrote:
>
> > I spun up a 3.4.10 zk server (standalone) and tried to reproduce the
> issue
> > - the delete works fine for me. Perhaps the length is a red herring (or I
> > had some other diff I didn't notice) and it's something else?
> >
> > zk: localhost:2181(CONNECTED) 12] ls
> > /dd/r-/mmm/ttt/ww.
> > c.h-hhh$qqq.-qqQqq2
> > [ww-dd-k-0_ww-dd-k_
> > nnn_nnn_mmm_m$887960620903477545]
> > [zk: localhost:2181(CONNECTED) 13] ls
> > /dd/r-/mmm/ttt/ww.
> > c.h-hhh$qqq.-qqQqq2/
> > ww-dd-k-0_ww-dd-k_
> > nnn_nnn_mmm_m$887960620903477545
> > []
> > [zk: localhost:2181(CONNECTED) 14] get
> > /dd/r-/mmm/ttt/ww.
> > c.h-hhh$qqq.-qqQqq2/
> > ww-dd-k-0_ww-dd-k_
> > nnn_nnn_mmm_m$887960620903477545
> > foo
> > cZxid = 0x7
> > ctime = Wed Sep 27 11:24:56 PDT 2017
> > mZxid = 0x7
> > mtime = Wed Sep 27 11:24:56 PDT 2017
> > pZxid = 0x7
> > cversion = 0
> > dataVersion = 0
> > aclVersion = 0
> > ephemeralOwner = 0x0
> > dataLength = 3
> > numChildren = 0
> > [zk: localhost:2181(CONNECTED) 15] delete
> > /dd/r-/mmm/ttt/ww.
> > c.h-hhh$qqq.-qqQqq2/
> > ww-dd-k-0_ww-dd-k_
> > nnn_nnn_mmm_m$887960620903477545
> > [zk: localhost:2181(CONNECTED) 16] ls
> > /dd/r-/mmm/ttt/ww.
> > c.h-hhh$qqq.-qqQqq2/
> > ww-dd-k-0_ww-dd-k_
> > nnn_nnn_mmm_m$887960620903477545
> > Node does not exist:
> > /dd/r-/mmm/ttt/ww.
> > c.h-hhh$qqq.-qqQqq2/
> > ww-dd-k-0_ww-dd-k_
> > nnn_nnn_mmm_m$887960620903477545
> > [zk: localhost:2181(CONNECTED) 17] get
> > /dd/r-/mmm/ttt/ww.
> > c.h-hhh$qqq.-qqQqq2/
> > ww-dd-k-0_ww-dd-k_
> > nnn_nnn_mmm_m$887960620903477545
> > Node does not exist:
> > /dd/r-/mmm/ttt/ww.
> > c.h-hhh$qqq.-qqQqq2/
> > ww-dd-k-0_ww-dd-k_
> > nnn_nnn_mmm_m$887960620903477545
> >
> > On Wed, Sep 27, 2017 at 3:09 AM, Stevo Slavić <ssla...@gmail.com> wrote:
> >
> > > In ZK shell:
> > >
> > > ls
> > > /dd/r-/mmm/ttt/ww.
> > > c.h-hhh$qqq.-qqQqq2
> > > [ww-dd-k-0_ww-dd-k_
> > > nnn_nnn_mmm_m$887960620903477545]
> > >
> > > get
> > > /dd/r-/mmm/ttt/ww.
> > > c.h-hhh$qqq.-qqQqq2/
> > > ww-dd-k-0_ww-dd-k_
> > > nnn_nnn_mmm_m$887960620903477545
> > > _
> > > cZxid = 0x432e7
> > > ctime = Fri Sep 22 14:51:22 CEST 2017
> > > mZxid = 0x432e7
> > > mtime = Fri Sep 22 14:51:22 CEST 2017
> > > pZxid = 0x432e7
> > > cversion = 0
> > > dataVersion = 0
> > > aclVersion = 0
> > > ephemeralOwner = 0x0
> > > dataLength = 13
> > > numChildren = 0
> > >
> > > delete
> > > /dd/r-/mmm/ttt/ww.
> > > c.h-hhh$qqq.-qqQqq2/
> > > ww-dd-k-0_ww-dd-k_
> > > nnn_nnn_mmm_m$887960620903477545
> > > Node does not exist:
> > > /dd/r-/mmm/ttt/ww.
> > > c.h-hhh$qqq.-qqQqq2/
> > > ww-dd-k-0_ww-dd-k_
> > &g

[SECURITY] CVE-2017-5637: DOS attack on wchp/wchc four letter words (4lw)

[Patrick Hunt]

CVE-2017-5637: DOS attack on wchp/wchc four letter words (4lw)

Severity: moderate
Vendor:
The Apache Software Foundation
Versions Affected:
ZooKeeper 3.4.0 to 3.4.9
ZooKeeper 3.5.0 to 3.5.2
The unsupported ZooKeeper 1.x through 3.3.x versions may be also affected
Note: The 3.5 branch is still beta at this time.

Description:
Two four letter word commands “wchp/wchc” are CPU intensive and could cause
spike of CPU utilization on ZooKeeper server if abused,
which leads to the server unable to serve legitimate client requests. There
is no known compromise which takes advantage of this vulnerability.

Mitigation:
This affects ZooKeeper ensembles whose client port is publicly accessible,
so it is recommended to protect ZooKeeper ensemble with firewall.
Documentation has also been updated to clarify on this point. In addition,
a patch (ZOOKEEPER-2693) is provided to disable "wchp/wchc” commands
by default.
- ZooKeeper 3.4.x users should upgrade to 3.4.10 or apply the patch.
- ZooKeeper 3.5.x users should upgrade to 3.5.3 or apply the patch.

References
[1] https://issues.apache.org/jira/browse/ZOOKEEPER-2693

Re: Zookeeper quorum goes down for no apparent reason in 3.4.5

[Patrick Hunt]

Unfortunately I don't see any attached logs, which makes it difficult to
provide you with insight. "Not sufficient followers synced" indicates that
you're losing followers, likely they are falling behind - what is your
metric tracking telling your wrt load on the compute and load on the
disk/memory/network/etc... also metrics at the ZK level (e.g. are zk
latencies increasing?) Check the logs to see if you're seeing "fsync"
slowness issues (it's a warning in the server logs). This is a pretty
common issue. GC might also be an issue, although that's more rare these
days (hard to say w/o knowing your use case, etc...) Again, look to your
metrics collection for insight where to start.

Patrick

On Wed, Oct 4, 2017 at 11:17 AM, Anand Parthasarathy <
anpar...@avinetworks.com> wrote:

> Hi,
>
> We have an issue with a 3-node zookeeper ensemble where the quorum goes
> down due to no apparent reason every once in a while. Here is what I see in
> the ZK leader:
>
> 2017-09-21 03:00:03,648 [myid:3] - INFO  [QuorumPeer[myid=3]/127.0.0.1:
> 5002:Leader@493] - Shutting down
> 2017-09-21 03:00:03,648 [myid:3] - INFO  [QuorumPeer[myid=3]/127.0.0.1:
> 5002:Leader@499] - Shutdown called
> java.lang.Exception: shutdown Leader! reason: Not sufficient followers
> synced, only synced with sids: [ 3 ]
> at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:499)
> at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:474)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(
> QuorumPeer.java:799)
>
> I have attached the logs from the 3 nodes around this time. Could you pls.
> help understand what the issue could be here. The only thing I see a little
> bit ahead of this timestamp is that all of them did a PurgeTask pretty much
> at the same time.
>
> Thanks,
> Anand.
>

Re: When zookeeper 3.5 version can go to final stable ?

[Patrick Hunt]

Hi, last update on this was a few days ago, please see the archives.
https://lists.apache.org/thread.html/2d006f5914d5b06777efe378422820
8f889e66f903b561251758e08c@%3Cdev.zookeeper.apache.org%3E

Regards,

Patrick

On Mon, Oct 2, 2017 at 5:03 AM, L. J.W  wrote:

> When zookeeper 3.5 version can go to final stable ? We expect for the local
> session feature and wait for stable version to use in our product
> environment.
> any plan?
>
> L.J.W
>

Re: Node can be created and listed, but not updated or deleted

[Patrick Hunt]

I spun up a 3.4.10 zk server (standalone) and tried to reproduce the issue
- the delete works fine for me. Perhaps the length is a red herring (or I
had some other diff I didn't notice) and it's something else?

zk: localhost:2181(CONNECTED) 12] ls
/dd/r-/mmm/ttt/ww.c.h-hhh$qqq.-qqQqq2
[ww-dd-k-0_ww-dd-k_nnn_nnn_mmm_m$887960620903477545]
[zk: localhost:2181(CONNECTED) 13] ls
/dd/r-/mmm/ttt/ww.c.h-hhh$qqq.-qqQqq2/ww-dd-k-0_ww-dd-k_nnn_nnn_mmm_m$887960620903477545
[]
[zk: localhost:2181(CONNECTED) 14] get
/dd/r-/mmm/ttt/ww.c.h-hhh$qqq.-qqQqq2/ww-dd-k-0_ww-dd-k_nnn_nnn_mmm_m$887960620903477545
foo
cZxid = 0x7
ctime = Wed Sep 27 11:24:56 PDT 2017
mZxid = 0x7
mtime = Wed Sep 27 11:24:56 PDT 2017
pZxid = 0x7
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 3
numChildren = 0
[zk: localhost:2181(CONNECTED) 15] delete
/dd/r-/mmm/ttt/ww.c.h-hhh$qqq.-qqQqq2/ww-dd-k-0_ww-dd-k_nnn_nnn_mmm_m$887960620903477545
[zk: localhost:2181(CONNECTED) 16] ls
/dd/r-/mmm/ttt/ww.c.h-hhh$qqq.-qqQqq2/ww-dd-k-0_ww-dd-k_nnn_nnn_mmm_m$887960620903477545
Node does not exist:
/dd/r-/mmm/ttt/ww.c.h-hhh$qqq.-qqQqq2/ww-dd-k-0_ww-dd-k_nnn_nnn_mmm_m$887960620903477545
[zk: localhost:2181(CONNECTED) 17] get
/dd/r-/mmm/ttt/ww.c.h-hhh$qqq.-qqQqq2/ww-dd-k-0_ww-dd-k_nnn_nnn_mmm_m$887960620903477545
Node does not exist:
/dd/r-/mmm/ttt/ww.c.h-hhh$qqq.-qqQqq2/ww-dd-k-0_ww-dd-k_nnn_nnn_mmm_m$887960620903477545

On Wed, Sep 27, 2017 at 3:09 AM, Stevo Slavić  wrote:

> In ZK shell:
>
> ls
> /dd/r-/mmm/ttt/ww.
> c.h-hhh$qqq.-qqQqq2
> [ww-dd-k-0_ww-dd-k_
> nnn_nnn_mmm_m$887960620903477545]
>
> get
> /dd/r-/mmm/ttt/ww.
> c.h-hhh$qqq.-qqQqq2/
> ww-dd-k-0_ww-dd-k_
> nnn_nnn_mmm_m$887960620903477545
> _
> cZxid = 0x432e7
> ctime = Fri Sep 22 14:51:22 CEST 2017
> mZxid = 0x432e7
> mtime = Fri Sep 22 14:51:22 CEST 2017
> pZxid = 0x432e7
> cversion = 0
> dataVersion = 0
> aclVersion = 0
> ephemeralOwner = 0x0
> dataLength = 13
> numChildren = 0
>
> delete
> /dd/r-/mmm/ttt/ww.
> c.h-hhh$qqq.-qqQqq2/
> ww-dd-k-0_ww-dd-k_
> nnn_nnn_mmm_m$887960620903477545
> Node does not exist:
> /dd/r-/mmm/ttt/ww.
> c.h-hhh$qqq.-qqQqq2/
> ww-dd-k-0_ww-dd-k_
> nnn_nnn_mmm_m$887960620903477545
>
>
> From one of ZK server logs:
> 2017-09-27 11:07:19,654 [myid:3] - INFO  [ProcessThread(sid:3
> cport:-1)::PrepRequestProcessor@648] - Got user-level KeeperException when
> processing sessionid:0x15ec269280a000a type:delete cxid:0x73e
> zxid:0xc0017 txntype:-1 reqpath:n/a Error
> Path:/dd/r-/mmm/ttt/ww.c.h-
> hhh$qqq.-qqQqq2/ww-dd-
> k-0_ww-dd-k_nnn_nnn_mmm_
> m$887960620903477545
> Error:KeeperErrorCode = NoNode for
> /dd/r-/mmm/ttt/ww.
> c.h-hhh$qqq.-qqQqq2/
> ww-dd-k-0_ww-dd-k_
> nnn_nnn_mmm_m$887960620903477545
>
> On Mon, Sep 25, 2017 at 4:36 PM, Jordan Zimmerman <
> jor...@jordanzimmerman.com> wrote:
>
> > There's no internal limit on path lengths. Can you post a sample showing
> > the problem?
> >
> > -Jordan
> >
> > > On Sep 25, 2017, at 3:53 AM, Stevo Slavić  wrote:
> > >
> > > Hello Apache ZooKeeper community,
> > >
> > > While on 3.4.10 I just experienced odd issue with a node that has
> > > relatively lengthy path. It was created and had data stored. In the
> > shell I
> > > could list it and retrieve the data. Problem is that the attempt from
> the
> > > app to update the data as well as attempt from the shell to delete the
> > node
> > > would fail with NoNode.
> > >
> > > Path length (with separators) is 190 

Re: Running Zookeeper as a windows service

[Patrick Hunt]

I haven't had any experience with this myself however googling for
"zookeeper windows service" resulted in a couple of blog posts and a github
repo above the fold. Perhaps one of those will shed light.

Best of luck,

patrick


On Fri, Sep 15, 2017 at 4:41 AM, Giannis Doukas <
giannis.dou...@performance.gr> wrote:

> Hello,
>
> I have installed the zookeeper on Windows server and I run it from cmd.
> But now want to change it and make it run as windows services. What is the
> recommended method to do this?
>
> Thank you in advance,
> Giannis Doukas
>
> This message contains confidential information and is intended only for
> the individual named. If you are not the named addressee you should not
> disseminate, distribute or copy this e-mail. Please notify the sender
> immediately by e-mail if you have received this e-mail by mistake and
> delete this e-mail from your system. E-mail transmission cannot be
> guaranteed to be secure or error-free as information could be intercepted,
> corrupted, lost, destroyed, arrive late or incomplete, or contain viruses.
> The sender therefore does not accept liability for any errors or omissions
> in the contents of this message, which arise as a result of e-mail
> transmission. If verification is required please request a hard-copy
> version.
>
> PERFORMANCE Technologies AE, 4 Evrimedontos Street, 10435, Athens, Greece
>
> 
> ===
>  ?? ??  ? ??? ??? ???  ???
> ??  ?. ?? ??? ? ? ? ?? ?? ?? ??
> ?? ? ?? ???  ?? email. ??? ?? 
> ?? ??? ? ?? ? ?  ?? email  ? ??? ?? ??
> ? ??? ?? ??? ???. ???  ?? ?? ??? ? 
>  ? ? ??? ? ?? ??? ??? ???
> ?? ?? , ?? ??, ?? ??, ?? , ??
> ??? ?? ??? ? ?? ? . ? ?? ??? ?? ?
> ? ?? ??? ?  ? ??? ??? ??? ??? ?
> ?. ?? ? ??  ??? ?? ?? ?? ?? ?.
> PERFORMANCE Technologies AE,  4, 10435, ?, ??
>

Re: Upgrade of Zookeeper and Kafka

[Patrick Hunt]

Kafka is using ZK to manage its coordination. As such it's dependent on ZK.
Which features, etc... it uses may impact compatibility.

Regards,

Patrick

On Wed, Aug 16, 2017 at 1:43 PM, Michael Chen <
yiningchen2...@u.northwestern.edu> wrote:

> Hi Patrick,
>
> Thanks for the reply. I'm still quite new to ZooKeeper, could you explain
> a bit more about why Kafka is involved?
>
> Thank you!
> Michael
>
> > On Aug 16, 2017, at 13:15, Patrick Hunt <ph...@apache.org> wrote:
> >
> > I'm not aware of any issues wrt ZK, but it's probably a better question
> for
> > the Kafka folks (as they are consuming the service).
> >
> > Patrick
> >
> > On Mon, Aug 14, 2017 at 2:57 PM, Carmen Molatch <
> carmen.mola...@jeppesen.com
> >> wrote:
> >
> >> Can I upgrade to Zookeeper 3.4.10 and Kafka 11?  Are there any known
> >> compatability issues?
> >>
> >> Thanks
> >> Carmen
> >>
>

Re: Upgrade of Zookeeper and Kafka

[Patrick Hunt]

I'm not aware of any issues wrt ZK, but it's probably a better question for
the Kafka folks (as they are consuming the service).

Patrick

On Mon, Aug 14, 2017 at 2:57 PM, Carmen Molatch  wrote:

> Can I upgrade to Zookeeper 3.4.10 and Kafka 11?  Are there any known
> compatability issues?
>
> Thanks
> Carmen
>

Re: New PMC Member: Michael Han

[Patrick Hunt]

Kudos Michael, well deserved!

Patrick

On Tue, Jun 27, 2017 at 9:49 AM, Jordan Zimmerman <
jor...@jordanzimmerman.com> wrote:

> Congrats!
>
> > On Jun 27, 2017, at 11:48 AM, Flavio Junqueira  wrote:
> >
> > I'm very happy to announce that the Apache ZooKeeper PMC has voted to
> invite Michael Han to join the PMC and Michael accepted. Michael has done
> outstanding work in the community over the recent past and we felt it was
> time for Michael to deepen his level of engagement by joining the PMC.
> >
> > Please join me in congratulating Michael for his achievement.
> Congratulations, Michael!
> >
> > -Flavio
> >
> >
>
>

Re: Recovering from zxid rollover

[Patrick Hunt]

On Wed, May 24, 2017 at 8:08 AM, Mike Heffner <m...@librato.com> wrote:

> On Tue, May 23, 2017 at 10:21 PM, Patrick Hunt <ph...@apache.org> wrote:
>
> > On Tue, May 23, 2017 at 3:47 PM, Mike Heffner <m...@librato.com> wrote:
> >
> > > Hi,
> > >
> > > I'm curious what the best practices are for handling zxid rollover in a
> > ZK
> > > ensemble. We have a few five-node ZK ensembles (some 3.4.8 and some
> > 3.3.6)
> > > and they periodically rollover their zxid. We see the following in the
> > > system logs on the leader node:
> > >
> > > 2017-05-22 12:54:14,117 [myid:15] - ERROR [ProcessThread(sid:15
> > > cport:-1)::ZooKeeperCriticalThread@49] - Severe unrecoverable error,
> > from
> > > thread : ProcessThread(sid:15 cport:-1):
> > > org.apache.zookeeper.server.RequestProcessor$RequestProcesso
> rException:
> > > zxid lower 32 bits have rolled over, forcing re-election, and therefore
> > new
> > > epoch start
> > >
> > > From my best understanding of the code, this exception will end up
> > causing
> > > the leader to enter shutdown():
> > >
> > > https://github.com/apache/zookeeper/blob/09cd5db55446a4b390f
> > > 82e3548b929f19e33430d/src/java/main/org/apache/zookeeper/
> > > server/ZooKeeperServer.java#L464-L464
> > >
> > > This shuts down the zookeeper instance from servicing requests, but the
> > JVM
> > > is still actually running. What we experience is that while this ZK
> > > instance is still running, the remaining follower nodes can't re-elect
> a
> > > leader (at least within 15 mins) and quorum is offline. Our remediation
> > so
> > > far has been to restart the original leader node, at which point the
> > > cluster recovers.
> > >
> > > The two questions I have are:
> > >
> > > 1. Should the remaining 4 nodes be able to re-elect a leader after zxid
> > > rollover without intervention (restarting)?
> > >
> > >
> > Hi Mike.
> >
> > That is the intent. Originally the epoch would rollover and cause the
> > cluster to hang (similar to what you are reporting), the JIRA is here
> > https://issues.apache.org/jira/browse/ZOOKEEPER-1277
> > However the patch, calling shutdown of the leader, was intended to force
> a
> > re-election before the epoch could rollover.
> >
>
> Should the leader JVM actually exit during this shutdown, thereby allowing
> the init system to restart it?
>
>
iirc it should not be necessary but it's been some time since I looked at
it.


>
> >
> >
> > > 2. If the leader enters shutdown() state after a zxid rollover, is
> there
> > > any scenario where it will return to started? If not, how are others
> > > handling this scenario -- maybe a healthcheck that kills/restarts an
> > > instance that is in shutdown state?
> > >
> > >
> > I have run into very few people who have seen the zxid rollover and
> testing
> > under real conditions is not easily done. We have unit tests but that
> code
> > is just not exercised sufficiently in everyday use. You're not seeing
> > what's intended, please create a JIRA and include any additional details
> > you can (e.g. config, logs)
> >
>
> Sure, I've opened one here:
> https://issues.apache.org/jira/browse/ZOOKEEPER-2791
>
>
> >
> > What I heard people (well really one user, I have personally only seen
> this
> > at one site) were doing prior to 1277 was monitoring the epoch number,
> and
> > when it got close to rolling over (within 10% say) they would force the
> > current leader to restart by restarting the process. The intent of 1277
> was
> > to effectively do this automatically.
> >
>
> We are looking at doing something similar, maybe once a week finding the
> current leader and restarting it. From testing this quickly re-elects a new
> leader and resets the zxid to zero so it should avoid the rollover that
> occurs after a few weeks of uptime.
>
>
Exactly. This is pretty much the same scenario that I've seen in the past,
along with a similar workaround.

You might want to take a look at the work Benedict Jin has done here:
https://issues.apache.org/jira/browse/ZOOKEEPER-2789
Given you are seeing this so frequently it might be something you could
collaborate on with the author of the patch? I have not looked at it in
great detail but it may allow you to run longer w/o seeing the issue. I
have not thought through all the implications though... (including b/w
compat).

Patrick


>
> >
> > Patrick
> >
> >
> > >
> > > Cheers,
> > >
> > > Mike
> > >
> > >
> >
>
> Mike
>

Re: Recovering from zxid rollover

[Patrick Hunt]

On Tue, May 23, 2017 at 3:47 PM, Mike Heffner  wrote:

> Hi,
>
> I'm curious what the best practices are for handling zxid rollover in a ZK
> ensemble. We have a few five-node ZK ensembles (some 3.4.8 and some 3.3.6)
> and they periodically rollover their zxid. We see the following in the
> system logs on the leader node:
>
> 2017-05-22 12:54:14,117 [myid:15] - ERROR [ProcessThread(sid:15
> cport:-1)::ZooKeeperCriticalThread@49] - Severe unrecoverable error, from
> thread : ProcessThread(sid:15 cport:-1):
> org.apache.zookeeper.server.RequestProcessor$RequestProcessorException:
> zxid lower 32 bits have rolled over, forcing re-election, and therefore new
> epoch start
>
> From my best understanding of the code, this exception will end up causing
> the leader to enter shutdown():
>
> https://github.com/apache/zookeeper/blob/09cd5db55446a4b390f
> 82e3548b929f19e33430d/src/java/main/org/apache/zookeeper/
> server/ZooKeeperServer.java#L464-L464
>
> This shuts down the zookeeper instance from servicing requests, but the JVM
> is still actually running. What we experience is that while this ZK
> instance is still running, the remaining follower nodes can't re-elect a
> leader (at least within 15 mins) and quorum is offline. Our remediation so
> far has been to restart the original leader node, at which point the
> cluster recovers.
>
> The two questions I have are:
>
> 1. Should the remaining 4 nodes be able to re-elect a leader after zxid
> rollover without intervention (restarting)?
>
>
Hi Mike.

That is the intent. Originally the epoch would rollover and cause the
cluster to hang (similar to what you are reporting), the JIRA is here
https://issues.apache.org/jira/browse/ZOOKEEPER-1277
However the patch, calling shutdown of the leader, was intended to force a
re-election before the epoch could rollover.


> 2. If the leader enters shutdown() state after a zxid rollover, is there
> any scenario where it will return to started? If not, how are others
> handling this scenario -- maybe a healthcheck that kills/restarts an
> instance that is in shutdown state?
>
>
I have run into very few people who have seen the zxid rollover and testing
under real conditions is not easily done. We have unit tests but that code
is just not exercised sufficiently in everyday use. You're not seeing
what's intended, please create a JIRA and include any additional details
you can (e.g. config, logs)

What I heard people (well really one user, I have personally only seen this
at one site) were doing prior to 1277 was monitoring the epoch number, and
when it got close to rolling over (within 10% say) they would force the
current leader to restart by restarting the process. The intent of 1277 was
to effectively do this automatically.

Patrick


>
> Cheers,
>
> Mike
>
>
> --
>
>   Mike Heffner 
>   Librato, Inc.
>

Re: [ANNOUNCE] Apache ZooKeeper 3.5.3-beta

[Patrick Hunt]

On Thu, Apr 20, 2017 at 9:12 AM, Michael Han <h...@cloudera.com> wrote:

> Just to clarify, 4LW feature is not removed in latest releases (3.4.10 and
> 3.5.3-beta). The feature is still there, it's just disabled by default. You
> can enable the feature if you need it (details in the admin documents).
> Because of the compatibility guarantees provided by ZooKeeper, we wouldn't
> just remove a feature lightly.
>
>
3.5 also adds json support through jetty/http, a significant improvement
over 4lw.

Patrick


> On Thu, Apr 20, 2017 at 6:53 AM, Ben Sherman <bensher...@gmail.com> wrote:
>
> > Thanks for the 4lw warning - I was going to upgrade to 3.4.10 today but
> > didn't expect features to be removed.  It's a shame they are going away,
> > human readable output from a one line script was a nice feature to have
> by
> > default.
> >
> > On Wed, Apr 19, 2017 at 6:02 PM, Michael Han <h...@cloudera.com> wrote:
> >
> > > >> pitfalls coming from 3.4.9 (or .10) to the 3.5.x release?
> > > If coming from 3.4.9, one note is all four letter words except srvr are
> > > disabled by default in 3.5.3 so your devops tool if they depend on 4lw
> > will
> > > stop working (one user already reports this on jira), which is
> expected.
> > In
> > > this case you can either update configuration to enable the subset of
> 4lw
> > > you need, or use modern monitoring primitives provided by ZK (JMX /
> Jetty
> > > admin server). If coming from 3.4.10 then it's fine, since 3.4.10 made
> > same
> > > change to 4lw (disable by default).
> > >
> > > >> make a change to 3.4.x (x>0) in order to maintain backward compat
> > with a
> > > change that we made to 3.5.
> > > Not sure if it's ZOOKEEPER-1633. Basically rolling upgrade would not
> work
> > > from 3.4.x to 3.5.y if x < 6.
> > >
> > > On Wed, Apr 19, 2017 at 5:16 PM, Patrick Hunt <ph...@apache.org>
> wrote:
> > >
> > > > I remember we had to make a change to 3.4.x (x>0) in order to
> maintain
> > > > backward compat with a change that we made to 3.5. I searched but I
> > can't
> > > > remember the specific jira or the specific release, it was some time
> > ago.
> > > > The issue would be that if you try and do a rolling upgrade from
> > 3.4.x-1
> > > to
> > > > 3.5.y it had the potential to fail. Perhaps one of the other
> community
> > > > folks will remember. Other than that I'm not aware of anything. The
> on
> > > disk
> > > > formats are the same and the communication protocols should be b/w
> > > compat.
> > > > I tried running 3.4 client against 3.5.3 during the last release and
> it
> > > > worked ok for me. Not sure if anyone has been testing at the quorum
> > > level.
> > > >
> > > > If anyone does find something (or tests and finds it works) please
> let
> > us
> > > > know so that we can document it.
> > > >
> > > > Patrick
> > > >
> > > > On Wed, Apr 19, 2017 at 4:14 PM, Ben Sherman <bensher...@gmail.com>
> > > wrote:
> > > >
> > > > > Great news, are there any docs written yet or any known pitfalls
> > coming
> > > > > from 3.4.9 (or .10) to the 3.5.x release?
> > > > >
> > > > > On Mon, Apr 17, 2017 at 10:48 AM, Michael Han <h...@cloudera.com>
> > > wrote:
> > > > >
> > > > > > The Apache ZooKeeper team is proud to announce Apache ZooKeeper
> > > version
> > > > > > *3.5.3-beta*.
> > > > > >
> > > > > > ZooKeeper is a high-performance coordination service for
> > distributed
> > > > > > applications. It exposes common services - such as naming,
> > > > > > configuration management, synchronization, and group services -
> in
> > a
> > > > > > simple interface so you don't have to write them from scratch.
> You
> > > can
> > > > > > use it off-the-shelf to implement consensus, group management,
> > leader
> > > > > > election, and presence protocols. And you can build on it for
> your
> > > > > > own, specific needs.
> > > > > >
> > > > > > For ZooKeeper release details and downloads, visit:
> > > > > > https://zookeeper.apache.org/releases.html
> > > > > >
> > > > > > ZooKeeper 3.5.3-beta Release Notes are at:
> > > > > > https://zookeeper.apache.org/doc/r3.5.3-beta/releasenotes.html
> > > > > >
> > > > > > We would like to thank the contributors that made the release
> > > possible.
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > The ZooKeeper Team
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Cheers
> > > Michael.
> > >
> >
>
>
>
> --
> Cheers
> Michael.
>

Re: [ANNOUNCE] Apache ZooKeeper 3.5.3-beta

[Patrick Hunt]

I remember we had to make a change to 3.4.x (x>0) in order to maintain
backward compat with a change that we made to 3.5. I searched but I can't
remember the specific jira or the specific release, it was some time ago.
The issue would be that if you try and do a rolling upgrade from 3.4.x-1 to
3.5.y it had the potential to fail. Perhaps one of the other community
folks will remember. Other than that I'm not aware of anything. The on disk
formats are the same and the communication protocols should be b/w compat.
I tried running 3.4 client against 3.5.3 during the last release and it
worked ok for me. Not sure if anyone has been testing at the quorum level.

If anyone does find something (or tests and finds it works) please let us
know so that we can document it.

Patrick

On Wed, Apr 19, 2017 at 4:14 PM, Ben Sherman  wrote:

> Great news, are there any docs written yet or any known pitfalls coming
> from 3.4.9 (or .10) to the 3.5.x release?
>
> On Mon, Apr 17, 2017 at 10:48 AM, Michael Han  wrote:
>
> > The Apache ZooKeeper team is proud to announce Apache ZooKeeper version
> > *3.5.3-beta*.
> >
> > ZooKeeper is a high-performance coordination service for distributed
> > applications. It exposes common services - such as naming,
> > configuration management, synchronization, and group services - in a
> > simple interface so you don't have to write them from scratch. You can
> > use it off-the-shelf to implement consensus, group management, leader
> > election, and presence protocols. And you can build on it for your
> > own, specific needs.
> >
> > For ZooKeeper release details and downloads, visit:
> > https://zookeeper.apache.org/releases.html
> >
> > ZooKeeper 3.5.3-beta Release Notes are at:
> > https://zookeeper.apache.org/doc/r3.5.3-beta/releasenotes.html
> >
> > We would like to thank the contributors that made the release possible.
> >
> > Regards,
> >
> > The ZooKeeper Team
> >
>

Re: Ideal values for tickTime and syncLimit

[Patrick Hunt]

On Thu, Mar 23, 2017 at 11:14 PM, Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> Hello,
>
> We have a mult dc zk cluster, and all of a sudden we started seeing
> timeouts with observers.
> while we are investigating the possible network issue, we would like to
> understand accepted values for ticket time and synclimit?
>
> The time out is syncLimit * tickTime
>
> currently we have it set to 2 * 6 = 2 minutes. we would like to
> increase the value to 6 minutes.
>
>
2 minutes seems incredibly long to me. Such a setting is saying that you
allow the server to fall 2 minutes behind the quorum. Rather than
investigate increasing the limits I'd recommend you figure out why you're
having such terrible latencies btw dcs (I assume that's the issue, could be
something else, e.g. GC pauses, etc..)


> what is the impact of increasing it?
> what are the acceptable limits?
>
>
Typically determined by the max latency you can allow btw changes in your
cluster and when the clients find out about them. IIRC Camille had done
quite a bit of work with ZK and multi-DC. Perhaps she, or others with this
situation, can provide "real world" insight.

Regards,

Patrick


> Thank you
>

Re: How to modify Client Connection timer

[Patrick Hunt]

On Wed, Mar 22, 2017 at 2:19 PM, Ray Chaudhuri, Shirsha (Nokia -
IN/Bangalore) <shirsha.ray_chaudh...@nokia.com> wrote:

>
>
> Thanks Patrick.
>
> Yes, we did eventually figure out that this timeout value is initialised
> from the Hadoop ZKFC code with the value of "ha.zookeeper.session-timeout.
> ms" or if this is not set, then the default is 5000. Since the client
> Cnxn class sets this timer value to a value _  (session timeout) / (number
> of ZK servers), the timer value for the initial connection to complete is
> one third the value of 5000. Hence the timeout occurring at around 1.6s
> each time.
>
> Which brought us to the question that wouldn't this piece of code
> highlighted from the file zookeeper/ClientCnxn.java _
> public ClientCnxn(String chrootPath, HostProvider hostProvider, int
> sessionTimeout, ZooKeeper zooKeeper,
>
>
> ClientWatchManager watcher, ClientCnxnSocket clientCnxnSocket,
>
>
> long sessionId, byte[] sessionPasswd, boolean canBeReadOnly) {
>
>
> this.zooKeeper = zooKeeper;
>
>
> this.watcher = watcher;
>
>
> this.sessionId = sessionId;
>
>
> this.sessionPasswd = sessionPasswd;
>
>
> this.sessionTimeout = sessionTimeout;
>
>
> this.hostProvider = hostProvider;
>
>
> this.chrootPath = chrootPath;
>
>
>
>
>
> connectTimeout = sessionTimeout / hostProvider.size();
>
>
> readTimeout = sessionTimeout * 2 / 3;
>
>
> readOnly = canBeReadOnly;
>
>
>
>
>
> sendThread = new SendThread(clientCnxnSocket);
>
>
> eventThread = new EventThread();
>
>
> this.clientConfig=zooKeeper.getClientConfig();
>
>
> }
>
>
>
>
> Be leading to trouble if the number of ZK Servers is larger?
>
> This timeout value is initially used when the Client is waiting to connect
> with the server, even before it has negotiated a sessiontimeout value.
>
> So larger the number of ZK Servers, less will be the timer value.
> Shouldn’t we have a lower guard for it and ensure that atleast a min value
> is used for this timer?
>
>
Probably a good idea. I've seen some terrible thundering herd problems that
result from this very basic logic. Some form of exponential backoff would
probably even be better. (configurable?) That said I've never seen the size
of the ensemble be that much of an issue given the other issue.

You might also check the latest trunk, can't remember if someone's been
trying to look at this recently...

Regards,

Patrick


>
>
> Regards
>
> Shirsha
>
>
>
> -Original Message-
> From: Patrick Hunt [mailto:ph...@apache.org]
> Sent: Wednesday, March 22, 2017 11:55 PM
> To: UserZooKeeper <user@zookeeper.apache.org>
> Subject: Re: How to modify Client Connection timer
>
>
>
> You should be able to control that by increasing the session timeout. I'm
> not familiar with the client code you are using however. Additionally the
> ZK client should retry it's connection (as long as you don't close the ZK
>
> object) continually until it is able to reconnect.
>
>
>
> Patrick
>
>
>
> On Mon, Mar 20, 2017 at 3:22 PM, Ray Chaudhuri, Shirsha (Nokia -
>
> IN/Bangalore) <shirsha.ray_chaudh...@nokia.com<mailto:shirsha.ray_
> chaudh...@nokia.com>> wrote:
>
>
>
> > Hi,
>
> >
>
> > We are using ZKFC  as a Zookeeper Client that tries to connect to
>
> > Zookeeper Server at the time of bringup, when we sometimes encounter
>
> > the following issue _
>
> >
>
> > 1.   Client retrieves the ZK Ensemble address (ensemble consists of 3
>
> > nodes)
>
> >
>
> > 2.   Client tries to connect to one of the ZK Server nodes
>
> >
>
> > 3.   However due to other on-going processing (required at the time
> of
>
> > bringup) at the ZK Server Node, it fails to respond in time to the
> Client.
>
> > The response goes after 3 seconds
>
> >
>
> > 4.   The Client side times out by then, typically seen around 1.6 sec
>
> >
>
> > Is there a way this timer at the Client side can be reconfigured to a
>
> > higher value?
>
> >
>
> > Regards
>
> > Shirsha
>
> >
>

Re: How to modify Client Connection timer

[Patrick Hunt]

You should be able to control that by increasing the session timeout. I'm
not familiar with the client code you are using however. Additionally the
ZK client should retry it's connection (as long as you don't close the ZK
object) continually until it is able to reconnect.

Patrick

On Mon, Mar 20, 2017 at 3:22 PM, Ray Chaudhuri, Shirsha (Nokia -
IN/Bangalore)  wrote:

> Hi,
>
> We are using ZKFC  as a Zookeeper Client that tries to connect to
> Zookeeper Server at the time of bringup, when we sometimes encounter the
> following issue _
>
> 1.   Client retrieves the ZK Ensemble address (ensemble consists of 3
> nodes)
>
> 2.   Client tries to connect to one of the ZK Server nodes
>
> 3.   However due to other on-going processing (required at the time of
> bringup) at the ZK Server Node, it fails to respond in time to the Client.
> The response goes after 3 seconds
>
> 4.   The Client side times out by then, typically seen around 1.6 sec
>
> Is there a way this timer at the Client side can be reconfigured to a
> higher value?
>
> Regards
> Shirsha
>

Re: Multiple credentials associated with same principal?

[Patrick Hunt]

On Tue, Feb 14, 2017 at 7:41 AM, Stevo Slavić  wrote:

> Is this natively supported by ZooKeeper or does this require some
> customization?
>
>
Hi Stevo, I've never heard of anyone taking this approach, and I don't
believe it is possible today.

Regards,

Patrick


> On Thu, Feb 2, 2017 at 7:06 PM, Martin Gainty  wrote:
>
> >
> > Stevo-
> > --
> > *From:* Stevo Slavić 
> > *Sent:* Thursday, February 2, 2017 7:38 AM
> > *To:* user@zookeeper.apache.org
> > *Subject:* Re: Multiple credentials associated with same principal?
> >
> > Alternatively, is it possible to set ACL that would grant given
> permissions
> > to any successfully authenticated user?
> >
> > MG>acl group should match subject=GroupOfPeople
> >
> > MG>ZooKeeperServer:
> > MG>best to configure each ACL on server for Group where
> > MG>ServerGroupACL=JAASSubject
> > MG>e,g. GroupACL=JAASSubject=EmergencyRoomNurses
> >
> > MG>ZooKeeperClient:
> > MG>since there can be multiple principals per subject
> > MG>each Principal would represent an individual within the group
> > MG>e.g. JAASPrincipal=NurseCratchett
> >
> >
> > On Wed, Feb 1, 2017 at 1:16 PM, Stevo Slavić  wrote:
> >
> > > Hello Apache ZooKeeper community,
> > >
> > > Is it valid in JAAS config file to associate more than one password per
> > > user, and if so, will ZooKeeper server authenticate user correctly if
> > > provided password matches any of the configured ones?
> > >
> > > Kind regards,
> > > Stevo Slavic.
> > >
> >
>

ZooKeeper DOS exploit published

[Patrick Hunt]

Hi folks. The following exploit was recently published on the web and has
come to our attention, it details a ZooKeeper DOS attack against certain
four letter words (4lw), possible when the client port is exposed to
untrusted actors:

https://webcache.googleusercontent.com/search?q=cache:_CNGIz10PRYJ:https://
www.exploit-db.com/exploits/41277/+=14=en=clnk=us

Typically we address security issues on the security@ private mailing list,
publishing a fixed release before publicly releasing the exploit, however
in this case given the information is publicly available already we decided
there's little point to keeping it on security@ exclusively.
http://zookeeper.apache.org/security.html

A JIRA has been created to track this issue:
https://issues.apache.org/jira/browse/ZOOKEEPER-2693
we expect to include a patch to address in 3.4.10 and 3.5.3.

Patrick

Re: Multiple credentials associated with same principal?

[Patrick Hunt]

Hi Stevo, you might be talking about one of the following variants? (see
the jiras linked to from this jira)
https://issues.apache.org/jira/browse/ZOOKEEPER-1634

Patrick

On Thu, Feb 2, 2017 at 4:38 AM, Stevo Slavić  wrote:

> Alternatively, is it possible to set ACL that would grant given permissions
> to any successfully authenticated user?
>
> On Wed, Feb 1, 2017 at 1:16 PM, Stevo Slavić  wrote:
>
> > Hello Apache ZooKeeper community,
> >
> > Is it valid in JAAS config file to associate more than one password per
> > user, and if so, will ZooKeeper server authenticate user correctly if
> > provided password matches any of the configured ones?
> >
> > Kind regards,
> > Stevo Slavic.
> >
>

Re: Is SSL supported in 3.4.9?

[Patrick Hunt]

Hi Dan. While Netty supports SSL I don't believe anyone worked through the
issues of what needed to change in ZK itself to get it to work until
3.5/ZOOKEEPER-2125. iow Netty was added in 3.4, however Netty+SSL was not
available until 3.5.

Patrick

On Wed, Dec 7, 2016 at 6:44 AM, Dan Langille  wrote:

> Hello,
>
> I'm getting mixed messages from the documentation, and I'm unable to get
> Zookeeper
> to talk on a secureClientPort.
>
> Is SSL supported in Zookeeper 3.4.9?
>
> At https://zookeeper.apache.org/doc/r3.4.9/zookeeperAdmin.html <
> https://zookeeper.apache.org/doc/r3.4.9/zookeeperAdmin.html> I see:
>
> "New in 3.4: Netty is an NIO based client/server communication
> framework Netty framework
> has built in support for encryption (SSL)"
>
> But at https://cwiki.apache.org/confluence/display/ZOOKEEPER/
> ZooKeeper+SSL+User+Guide it
> states that SSL has "been added in ZOOKEEPER-2125", which is 3.5.1, 3.6.0
>
> If someone can confirm that SSL is indeed supported in 3.4.9, I'll keep
> debugging my configuration.
>
> Thank you.
>
> --
> Dan Langille - BSDCan / PGCon
> d...@langille.org
>
>
>

Re: Configurable SSL Context

[Patrick Hunt]

Hi George, I don't see any open JIRAs that mention this, while not an
expert myself it sounds like something that might be useful for others.
Please open a JIRA and submit a patch.

Regards,

Patrick

On Mon, Dec 5, 2016 at 2:23 PM, George Goddard 
wrote:

> Hello,
>
> Are there any plans to make the SSL Context that is being used in the
> NettyServerCnxnFactory (line 360) and X509Util (line 136) configurable?  We
> have been using the 3.5.2-alpha version at work to enable SSL communication
> between the ZooKeeper server and clients but would like to have the
> flexibility to use cipher suites other than TLSv1.
>
> Thanks,
> George

Re: Upgrading Zookeeper

[Patrick Hunt]

See the "backward compatibility" section here:
https://cwiki.apache.org/confluence/display/ZOOKEEPER/ReleaseManagement

The on disk format hasn't changed in some time, although that doesn't
mean you can always downgrade. For example if you use new
functionality based on new transaction types that the old server
version doesn't recognize. Doesn't happen that frequently but it has
happened.

Patrick

On Mon, Sep 12, 2016 at 12:41 AM, Jens Rantil <jens.ran...@tink.se> wrote:
> Somewhat related; If something goes wrong, can one downgrade? Is there a
> policy for that? Is backwards/forward compatability documented somewhere?
>
> Thanks,
> Jens
>
> On Monday, September 12, 2016, Patrick Hunt <ph...@apache.org> wrote:
>
>> Hi Allen. I don't believe there is anything in the docs however there
>> is something in the FAQ:
>>
>> What are the options-process for upgrading ZooKeeper?
>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/FAQ
>>
>> Patrick
>>
>> On Thu, Sep 8, 2016 at 2:58 PM, allen chan <allen.michael.c...@gmail.com
>> <javascript:;>> wrote:
>> > Hello everyone,
>> >
>> > I am reading through the admin guide. I cannot find any section speaking
>> of
>> > upgrades. Is it documented anywhere the procedure for upgrades and
>> anything
>> > to look out for upgrading between versions? I am specifically trying to
>> go
>> > from 3.4.8 to 3.4.9 but curious about overall strategy and information.
>> >
>> > Thanks
>> >
>> > --
>> > Allen Michael Chan
>>
>
>
> --
> Jens Rantil
> Backend engineer
> Tink AB
>
> Email: jens.ran...@tink.se
> Phone: +46 708 84 18 32
> Web: www.tink.se
>
> Facebook <https://www.facebook.com/#!/tink.se> Linkedin
> <http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary>
>  Twitter <https://twitter.com/tink>

Re: Upgrading Zookeeper

[Patrick Hunt]

Hi Allen. I don't believe there is anything in the docs however there
is something in the FAQ:

What are the options-process for upgrading ZooKeeper?
https://cwiki.apache.org/confluence/display/ZOOKEEPER/FAQ

Patrick

On Thu, Sep 8, 2016 at 2:58 PM, allen chan  wrote:
> Hello everyone,
>
> I am reading through the admin guide. I cannot find any section speaking of
> upgrades. Is it documented anywhere the procedure for upgrades and anything
> to look out for upgrading between versions? I am specifically trying to go
> from 3.4.8 to 3.4.9 but curious about overall strategy and information.
>
> Thanks
>
> --
> Allen Michael Chan

Re: log4j configuration for zookeeper with kafka process

[Patrick Hunt]

Hi Mazhar, if you're interested in ZK client logs when running Kafka that's
a question better answered by the Kafka community, there are many ways to
set things up, it's likely controlled by the Kafka configuration.

Regards,

Patrick

On Tue, Sep 6, 2016 at 2:40 AM, Mazhar Shaikh 
wrote:

> Hi All,
>
> How to enable zookeeper logs for kafka process?
>
> kafka provides "log4j.properties" property file which doesn't have
> zookeeper entry.
>
> Can someone please help me with log4j configuration to enable zookeeper
> logs ?
>
>
> Thank you.
>
> Regards,
> Mazhar Shaikh
>

Re: Issue with NettyServerCnxn.java

[Patrick Hunt]

Hi Yuliya - my read is that sendResponse in NIOServerCnxn is logging, then
dropping, any Exceptions encountered during sendResponse. In other words
it's doing best effort response. Not sure if that is "correct", but that's
what it's currently doing in NIO. Surprisingly it's also hiding any
IOExceptions, which is part of the method signature as defined by
ServerCnxn. Some of the calling code is trying to handle IOException in
some cases which is odd... I suspect it was an oversight in ZOOKEEPER-597,
but I'm not sure.

Ben any insight?

Patrick

On Tue, Aug 30, 2016 at 5:15 PM, yuliya Feldman  wrote:

> Hello there,
> We have been extensively testing Netty connection versus NIIO and there
> are some issues that show up I wanted to get community response on.
> In the process of testing https://issues.apache.
> org/jira/browse/ZOOKEEPER-2509 fix we identified that sendResponse()
> method may try to do some operations after close() was invoked - as
> channel.close() in Netty is asynch. and subsequently lead to some NPE.
> NPE itself is not a good thing but the problems aggravates with the fact
> that propagation of NPE will lead to main processing thread exiting and at
> that point ZK server becomes unresponsive - since no requests will be
> processed anymore.
> In NIOServerCnxn.java in sendResponse() it is catching Exception and just
> logs a warning  which was added as part of https://issues.apache.org/
> jira/browse/ZOOKEEPER-597
> I am trying to understand what a behavior should be in case of any
> exception in sendResponse.
> Any insight would be highly appreciated
> Thanks,Yuliya
>
>

Re: Exception using zkCli

[Patrick Hunt]

Take a look at the log4j logs for the client. Notice the operation is
"getChildren", and you mentioned having 34k znodes. A common issues is
"jute.maxbuffer" being exceeded (try googling it) during an operation. ZK
has a limit to the size of message it will send client->server or
server->client. If this is exceeded the connection will be dropped. In your
case it's very likely that the client has requested all of the znodes for a
particular parent, and the response for the operation is exceeding
jute.maxbuffer (which you can increase, see the admin guide).

Patrick

On Fri, Aug 26, 2016 at 3:53 AM, Stefano Salmaso <
stefano.salm...@lastminute.com> wrote:

> Hi all,
> we have a strange behaviour using zkCli.sh shell script.
> If I I try to connect to a zookeeper instance (we have a cluster) all work
> fine.. but using command "ls /" fails!
>
> $ ./zkCli.sh
> /usr/bin/java
> Connecting to localhost:2181
> Welcome to ZooKeeper!
> JLine support is enabled
> [zk: localhost:2181(CONNECTING) 0] ls /
> Exception in thread "main"
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:2255)
> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:2283)
> at org.apache.zookeeper.cli.LsCommand.exec(LsCommand.java:93)
> at
> org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:674)
> at
> org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:577)
> at
> org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:360)
> at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:320)
> at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:280)
>
> Now we have (more or less) 34K znodes. We are using ZK as distributed lock
> (with curator)
>
> Any idea about this?
>
> Thanks
> Stefano
>
> --
> [image: Bravofly Rumbo Group]
>
> Listed on SIX Swiss Exchange
>
> *Our websites*: lastminute.com | bravofly.com | volagratis.com | rumbo.es
> |
> jetcost.com | group.lastminute.com
>
> This e-mail may contain confidential and/or privileged information. If you
> are not the intended recipient (or have received this e-mail in error)
> please notify the sender immediately and destroy this e-mail. Any
> unauthorized copying, disclosure or distribution of the material in this
> e-mail is strictly forbidden and could be a crime.
>

Re: ZK 3.4.6 creating malformed TGT

[Patrick Hunt]

Ok, thanks Irfan!

Patrick

On Wed, Aug 24, 2016 at 2:58 PM, Irfan Hamid <iha...@salesforce.com> wrote:

> Sure. It was not very straightforward. I ended up setting the logging level
> on the client side by adding the -Dlogging.level=DEBUG and
>  -Dsun.security.krb5.debug=true. Additionally I set debug=true in the
> jaas.conf.
>
> With this I got log lines similar to the following on the client:
>
> KrbException: Fail to create credential. (63) - No service creds
> >>> Credentials acquireServiceCreds: no tgt; cannot get creds
> >>> Credentials acquireServiceCreds: no tgt; searching backwards
> msgType is 30
> sname is krbtgt/B.C.NET <http://app.eng.sfdc.net/>
> realm is A.B.C.NET <http://APP.ENG.SFDC.NET>
> cname is zkcli
> crealm is A.B.C.NET <http://APP.ENG.SFDC.NET>
> error Message is Server not found in Kerberos database
> error code is 7
> suSec is 356619
> sTime is Wed Aug 17 18:32:34 GMT 2016 1471458754000
> cTime is Wed Feb 17 16:12:06 GMT 2010 1266423126000
>
>
> The Internet had very little information about all that sname, realm, cname
> and crealm etc. mean. However, a grep for krbtgt on the ZK sources led me
> to Login.java and this code:
>
> Set tickets =
> subject.getPrivateCredentials(KerberosTicket.class);
> for(KerberosTicket ticket: tickets) {
> KerberosPrincipal server = ticket.getServer();
> if (server.getName().equals("krbtgt/" + server.getRealm() +
> "@"
> + server.getRealm())) {
> LOG.debug("Found tgt " + ticket + ".");
> return ticket;
> }
> }
> return null;
>
> Logs didn't indicate that my code had gotten to this point but it did get
> me thinking that this is a problem with my jaas or my krb5.conf since the
> KerberTicket objects are being retrieved from a Java API. The man page on
> krb5.conf gave me the next clues. Playing around with the [domain_realm]
> and default_realm etc. got me to the solution.
>
> Happy to help,
> Irfan.
>
> On Wed, Aug 24, 2016 at 12:37 PM, Patrick Hunt <ph...@apache.org> wrote:
>
> > Hi Irfan, this is very helpful, appreciate you responding with your
> > findings. It definitely will help folks in the future!
> >
> > Could you explain, how did you debug this issue and identify the correct
> > changes? I'm interested to know how you were able to track this one down.
> >
> > Patrick
> >
> >
> > On Thu, Aug 18, 2016 at 9:48 AM, Irfan Hamid <iha...@salesforce.com>
> > wrote:
> >
> > > Thanks Patrick,
> > >
> > > Actually it wasn't a DNS issue but I believe this is something that
> > others
> > > in the community could find useful since the issue had to do with
> default
> > > realms domain-to-realm conversions and krb5.conf issues.
> > >
> > > Our setup has multiple domains (one for prod and one for internal
> testing
> > > environments). I'm not a security expert so my terminology may be off.
> > The
> > > TGT is of the format krbtgt/REALM@REALM. In our krb5.conf we had:
> > >
> > > [domain_realms]
> > > .a.b.c = A.B.C
> > > a.b.c = A.B.C
> > > .b.c = B.C
> > > b.c = B.C
> > >
> > > All our machine names end in b.c, and only some end in a.b.c (the
> client
> > > machines end in b.c). So the first realm that was being selected for
> the
> > > krbtgt was B.C and the second was from [libdefaults] default_realm
> which
> > > was A.B.C. I replaced the standard krb5.conf with a custom one that
> had:
> > >
> > > [domain_realms]
> > > .b.c = A.B.C
> > > b.c = A.B.C
> > >
> > > With this change and pointing to this file via jvm arg
> > > java.security.krb5.conf everything worked fine.
> > >
> > > Thanks,
> > > Irfan.
> > >
> > > On Wed, Aug 17, 2016 at 7:56 PM, Patrick Hunt <ph...@apache.org>
> wrote:
> > >
> > > > I'm not an expert but I believe this indicates an environmental
> issue,
> > > see
> > > > "service ticket not found in the subject" here:
> > > > https://steveloughran.gitbooks.io/kerberos_and_
> > hadoop/content/sections/
> > > > errors.html
> > > >
> > > > Patrick
> > > >
> > > >
> > > > On Tue, Aug 16, 2016 at 12:11 PM, Irfan Hamid <iha...@salesforce.com
> >
> > > > wrote:
> > > >
> > > > 

Re: ZK 3.4.6 creating malformed TGT

[Patrick Hunt]

I'm not an expert but I believe this indicates an environmental issue, see
"service ticket not found in the subject" here:
https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/sections/errors.html

Patrick


On Tue, Aug 16, 2016 at 12:11 PM, Irfan Hamid  wrote:

> Hi,
>
> I'm very certain this is due to an error in my jaas or krb5 conf files but
> I can't seem to figure out where. My jaas.conf looks something like this:
>
> Client {
>   com.sun.security.auth.module.Krb5LoginModule required
>   useKeyTab=true
>   keyTab="/etc/zkcli_app_eng.keytab"
>   storeKey=true
>   useTicketCache=false
>   debug=true
>   principal="zk...@ay.bee.see.net";
> };
>
> But when ZK starts up it sends a TGT request as krbtgt/
> bee.see@ay.bee.see.net, i.e., it shaves off the AY from the realm
> name (
> BEE.SEE.NET is a valid realm in our setup but not the one I want to use).
> I
> see the following log lines:
>  msgType is 30
>  sname is krbtgt/BEE.SEE.NET
>  realm is AY.BEE.SEE.NET
>  cname is zkcli
>  crealm is AY.BEE.SEE.NET
>  error Message is Server not found in Kerberos database
>  error code is 7
>  suSec is 157006
>  sTime is Tue Aug 16 19:00:48 GMT 2016 1471374048000
>  cTime is Fri Sep 30 18:19:26 GMT 2016 1475259566000
>
>
> And a little earlier there was:
> Credentials acquireServiceCreds: main loop: [0] tempService=krbtgt/
> bee.see@ay.bee.see.net
> Realm parseCapaths: no cfg entry
> Realm doInitialParse: cRealm=[AY.BEE.SEE.NET], sRealm=[BEE.SEE.NET]
> Service ticket not found in the subject
> Found ticket for zk...@ay.bee.see.net to go to krbtgt/
> ay.bee.see@ay.bee.see.net expiring on Tue Aug 16 21:00:06 GMT 2016
>
> I looked in the source code and the place I see that might be relevant is
> Login.java where we have Login.getTGT() which tries to obtain a TGT by
> seeing if there's a ticket in the Subject of the form krbtgt/REALM@REALM.
> However, that part doesn't even get called since I don't even see the log
> line at the of the Login thread "TGT refresh thread started".
>
> Any help would be much appreciated.
>
> Thanks,
> Irfan.
>

Re: How to investigate these error codes

[Patrick Hunt]

What version of the c client are you using - multi-threaded or single
threaded? If multi-threaded then the library (incl pthreads) will take care
of handling the periodic heartbeats for you. If single threaded then you
might be starving the event processing - which includes the heartbeat loop.
See the THREADED sections of cli.c for an example.

Patrick

On Mon, Aug 15, 2016 at 12:01 AM, Krizansky, Jan 
wrote:

>  Yes, we're using the C client but we don't seem to have any network issues
> or load issues (in fact the setup is still in development mode so there is
> little to none traffic going through it).
> We have also set fairly high session timeout of 1,800,000 and a tickTime
> of 900,000. Yet we're getting SESSIONEXPIRED error even 2-3 times a minute.
> Are there any investigation steps you could recommend to pinpoint the
> problem?
>
> Thank you,
> Jan
>
> -Original Message-
> From: Flavio Junqueira [mailto:f...@apache.org]
> Sent: Friday, August 12, 2016 6:05 PM
> To: user@zookeeper.apache.org
> Subject: Re: How to investigate these error codes
>
> Hi Jan,
>
> Connection loss means that the client has disconnected from the server it
> was connected to and it will try to connect to another server to avoid
> session expiration.
>
> Session expired means that your session has expired. :-)
>
> Session expiration is important because if you have ephemerals associated
> to that session, they will be gone, so it might trigger some recovery path
> in your application.
>
> You're using the C client? If so, then it is not going to be garbage
> collection on the client side causing your clients to disconnect, which is
> a pretty common cause for applications using the Java client. You may want
> to investigate if you're having some network issues or if perhaps your
> servers are overwhelmed with something. If you're sharing the disk devices
> and other applications are inducing a good number of IOs, then you may end
> up affecting the performance of the server.
>
> -Flavio
>
>
> > On 12 Aug 2016, at 15:39, Krizansky, Jan 
> wrote:
> >
> >  >
> > I'm trying to reach out to you as we couldn't find any satisfying info
> online.
> > We've recently started seeing some errors in our cluster. The prevailing
> one is ZSESSIONEXPIRED but there sometimes is also a ZCONNECTIONLOSS error.
> > We couldn't find any documentation about possible causes of these
> issues. Any recommendation where we should investigate and what might be
> causing these?
> >
> > The ZCONNECTIONLOSS error is fairly rare. But ZSESSIONEXPIRED is very
> common happening on almost every other hit.
> >
> > Thank you,
> >
> > Jan Krizansky
> >
> >
> > NOTICE: This email and any attachments may contain confidential and
> proprietary information of NetSuite Inc. and is for the sole use of the
> intended recipient for the stated purpose. Any improper use or distribution
> is prohibited. If you are not the intended recipient, please notify the
> sender; do not review, copy or distribute; and promptly delete or destroy
> all transmitted information. Please note that all communications and
> information transmitted through this email system may be monitored by
> NetSuite or its agents and that all incoming email is automatically scanned
> by a third party spam and filtering service
> >
> > 
>
>
> NOTICE: This email and any attachments may contain confidential and
> proprietary information of NetSuite Inc. and is for the sole use of the
> intended recipient for the stated purpose. Any improper use or distribution
> is prohibited. If you are not the intended recipient, please notify the
> sender; do not review, copy or distribute; and promptly delete or destroy
> all transmitted information. Please note that all communications and
> information transmitted through this email system may be monitored by
> NetSuite or its agents and that all incoming email is automatically scanned
> by a third party spam and filtering service
>
> 
>

Re: [ANNOUNCE] Chris Nauroth joins the Apache ZooKeeper PMC

[Patrick Hunt]

Kudos Chris!

Patrick

On Sun, Aug 7, 2016 at 11:05 AM, Flavio Junqueira  wrote:

> In recognition of all his contributions to the project, the Apache
> ZooKeeper PMC has invited Chris Nauroth to join the PMC and he has
> accepted. I'd like to take the opportunity to thank Chris for his
> contributions and commitment to the project. Thank you and congratulations
> for joining the PMC, Chris!
>
> -Flavio

Re: Chocolatey packages for ZooKeeper, Kafka?

[Patrick Hunt]

Hi Andrew, if you want to publish somelike like that for ZK (on github say)
we'd be happy to link to it on our wiki "useful tools" page.

Regards,

Patrick

On Thu, Jul 28, 2016 at 6:58 PM, Andrew Pennebaker <
andrew.penneba...@gmail.com> wrote:

> Could we please publish Chocolatey packages for ZooKeeper and Kafka, to
> make it easier for newbies to get started?
>
> https://chocolatey.org/
>
> --
> Cheers,
> Andrew
>

Re: zookeeper SASL issue

[Patrick Hunt]

I recommend you turn on as much debugging as is available, add debug=true
to your Server jaas config, and add "-Dsun.security.krb5.debug=true" when
starting any jvm. That will ensure you get as much detail as the security
logging has available.

Patrick

On Thu, Jul 21, 2016 at 1:12 PM, Michael Han  wrote:

> 'No key to store' implies the keytab file is either not present or is
> invalid. I'd suggest validate the keytab file out of the context of
> zookeeper (for example using kinit) and check if it's valid.
>
> On Thu, Jul 21, 2016 at 10:13 AM, Aneela Saleem 
> wrote:
>
> > Hi,
> >
> > I have configured zookeeper to work with kerberos. I have added following
> > principal.
> > zkcli/hadoop-master@platalyticsrealm
> >
> > Here are contents of my jaas.conf
> >
> > Server {
> >   com.sun.security.auth.module.Krb5LoginModule required
> >   useKeyTab=true
> >   keyTab="/etc/hadoop/conf/zk.keytab"
> >   storeKey=true
> >   useTicketCache=false
> >   principal="zookeeper/hadoop-master@platalyticsrealm";
> > };
> > Client {
> >   com.sun.security.auth.module.Krb5LoginModule required
> >   useKeyTab=true
> >   keyTab="/etc/hadoop/conf/zkcli.keytab"
> >   storeKey=true
> >   useTicketCache=true
> >   principal="zkcli@platalyticsrealm"
> >   debug=true;
> > };
> >
> >
> > When i run zookeeper client using the command
> > *./zkCli.sh -server hadoop-master:2181*
> >
> > it produces following exception
> >
> >  SASL configuration failed: javax.security.auth.login.LoginException: No
> > key to store Will continue connection to Zookeeper server without SASL
> > authentication, if Zookeeper server allows it.
> >
> > Any suggestions
> >
> > Thanks
> >
>
>
>
> --
> Cheers
> Michael.
>

Re: [ANNOUNCE] Apache ZooKeeper 3.5.2-alpha

[Patrick Hunt]

Kudos all, thanks Chris!

Patrick

On Thu, Jul 21, 2016 at 7:46 PM, Flavio Junqueira  wrote:
> Thanks everyone for the push to get 3.5.2-alpha out, and specially to Chris 
> for being the release manager.
>
> -Flavio
>
>> On 21 Jul 2016, at 13:08, Chris Nauroth  wrote:
>>
>> The Apache ZooKeeper team is proud to announce Apache ZooKeeper version
>> 3.5.2-alpha.
>>
>> ZooKeeper is a high-performance coordination service for distributed
>> applications. It exposes common services - such as naming,
>> configuration management, synchronization, and group services - in a
>> simple interface so you don't have to write them from scratch. You can
>> use it off-the-shelf to implement consensus, group management, leader
>> election, and presence protocols. And you can build on it for your
>> own, specific needs.
>>
>> For ZooKeeper release details and downloads, visit:
>> http://zookeeper.apache.org/releases.html
>>
>> ZooKeeper 3.5.2-alpha Release Notes are at:
>> http://zookeeper.apache.org/doc/r3.5.2-alpha/releasenotes.html
>>
>> We would like to thank the contributors that made the release possible.
>>
>> Regards,
>>
>> The ZooKeeper Team
>>
>

Re: zk_outstanding_requests acceptable threshold limit

[Patrick Hunt]

It depends on your load. If it's sustained > 0 for any length of time that
indicates that the server is not able to process requests quickly enough.
Usually that means disk IO problems (that's where I typically see it).

Patrick

On Tue, Jul 12, 2016 at 12:28 PM, rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:

> Any help?
> On Jul 11, 2016 9:20 AM, "rammohan ganapavarapu" 
> wrote:
>
> > Hi,
> >
> > I am trying to setup monitoring for my zk cluster and trying to
> understand
> > what would be the better threshold limit for "zk_outstanding_requests" ?
> >
> > Thanks,
> > Ram
> >
>

Re: etcd3

[Patrick Hunt]

The "ZooKeeper Team" is you -- all of you! :-)

Our project is community based and as such everyone can develop, test,
improve, market, evangelize, etc... the project. If you're using ZK please
feel free to let folks know. If you have ideas or just want to "scratch an
itch" please participate.

At this point on the ycomb forum I already see some great responses such as
"I too have been using ZK for many years now, and it's pretty great.".
Please feel free to comment there, in blog posts (we can host them here at
Apache as well) or anywhere. End of the day we can propone the project all
we want, but nothing beats user testimonials -- and we've got lots of those!

If folks want to put together an "official response" i'm sure we could come
up with something. Please propose, we could host it on the wiki if there's
enough interest.

Patrick

On Thu, Jun 30, 2016 at 4:15 PM, Jordan Zimmerman <
jor...@jordanzimmerman.com> wrote:

> FYI
>
> https://coreos.com/blog/etcd3-a-new-etcd.html
>
> I think the ZooKeeper team needs to have a formal answer for etcd/consul.
> The developer community keeps asking questions like “compare it to ZK” and
> the etcd devs are very public and active with their responses. E.g.
> https://news.ycombinator.com/item?id=12011066
>
> -Jordan
>
> P.S. This gives a chance to plug my TTL PR which is a feature they have
> that ZK doesn’t: https://issues.apache.org/jira/browse/ZOOKEEPER-2169

Re: read under transaction

[Patrick Hunt]

Multi is more of an atomic operation than a "transaction" in the typical
sense. See https://issues.apache.org/jira/browse/ZOOKEEPER-965 for some
background. I don't believe the original use case involved reading multiple
znodes, rather updating multiple.

Patrick

On Mon, Jun 20, 2016 at 2:33 PM, Denis Samoilov  wrote:

> hi,
> I see that there is multi() function to write data under transaction. But
> it accepts only mutation operations. Is it possible to read under
> transaction somehow (so data will be consistent)?
>
> Thank you!
>

Re: Gentoo emerge package for ZooKeeper?

[Patrick Hunt]

I would think that Bigtop would add gentoo support if there's sufficient
demand. Perhaps ask on their user list?

Patrick

On Tue, Jun 28, 2016 at 2:13 AM, Flavio Junqueira  wrote:

> Hi Andrew,
>
> For packaging, we have been delegating to Bigtop, see this jira for some
> context and let us know if that works for you:
>
> https://issues.apache.org/jira/browse/ZOOKEEPER-1604 <
> https://issues.apache.org/jira/browse/ZOOKEEPER-1604>
>
> -Flavio
>
> > On 27 Jun 2016, at 18:47, Andrew Pennebaker 
> wrote:
> >
> > Could ZooKeeper please get an official emerge package, so that it's
> easier
> > for Gentoo users to install?
> >
> > --
> > Cheers,
> > Andrew
>
>