Community over code EU 2024

[Enrico Olivelli]

Hello folks,
I have submitted a talk for COC2024 (formerly known ad ApacheCon) about
Zookeeper and Curator

I hope to meet you in Bratislava in June!

I will be there even if my talk won't be accepted.

More about COC EU here:
https://eu.communityovercode.org/


Cheers
Enrico

CommunityOverCode - Halifax

[Enrico Olivelli]

Hello folks,
Is there anyone at Halifax from our community?

Enrico

Re: [ANNOUNCE] Apache ZooKeeper 3.9.0

[Enrico Olivelli]

Congratulations !

This is great step forward

I hope that people will try out soon the Backup/Restore feature and
that they provide feedback

Enrico

Il giorno ven 4 ago 2023 alle ore 13:24 Andor Molnar
 ha scritto:
>
> The Apache ZooKeeper team is proud to announce Apache ZooKeeper version
> 3.9.0
>
> ZooKeeper is a high-performance coordination service for distributed
> applications. It exposes common services - such as naming,
> configuration management, synchronization, and group services - in a
> simple interface so you don't have to write them from scratch. You can
> use it off-the-shelf to implement consensus, group management, leader
> election, and presence protocols. And you can build on it for your
> own, specific needs.
>
> For ZooKeeper release details and downloads, visit:
> https://zookeeper.apache.org/releases.html
>
> ZooKeeper 3.9.0 Release Notes are at:
> https://zookeeper.apache.org/doc/r3.9.0/releasenotes.html
>
> We would like to thank the contributors that made the release possible.
>
> Regards,
>
> The ZooKeeper Team
>
>

Re: Session Issue

[Enrico Olivelli]

Miao,
Sorry for late reply.

Are you sure that the servers are not running in standalone mode? (Split
brain).

Second thing: your version is very old (3
4.5), it would be better to move to some more recent version.

Enrico

Il Sab 8 Lug 2023, 06:34 wangmiao  ha scritto:

> Hello Team,
>  We are using a zookeeper cluster to serve HBase services, with three
> nodes deployed in the cluster，We found that other clients and regionserver
> services share the same session ID, which caused the regionserver to crash
> Zookeeper version is: 3.4.5
>
>
> Could you please help us to debug this issue?
>
>
> Follower's log
>  2023-07-05 02:25:08,990 INFO org.apache.zookeeper.server.ZooKeeperServer:
> Client attempting to establish new session at /10.11.1.10:51432
>  2023-07-05 02:25:08,991 INFO org.apache.zookeeper.server.ZooKeeperServer:
> Established session 0xff88d1721b6c53bc with negotiated timeout 18 for
> client   /10.11.1.10:51432
>  2023-07-05 02:25:35,880 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Closed socket connection for client /10.11.1.10:51432 which had sessionid
> 0xff88d1721b6c53bc
>
> 0xff88d1721b6c53bc comes from the client 10.11.1.10:51432, 2023-07-05
> 02:25:35880. The server closed it when it thought it timed out
>
>
> master's log
>
>
>  2023-07-05 02:25:35,880 INFO
> org.apache.zookeeper.server.PrepRequestProcessor: Processed session
> termination for sessionid: 0xff88d1721b6c53bc
>  2023-07-05 02:25:35,880 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Closed socket connection for client /10.11.8.18:41402 which had sessionid
> 0xff88d1721b6c53bc
>
> The client displayed in 0xff88d1721b6c53bc is 10.11.8.18:41402,
> 10.11.8.18 is my regionserver service, which was restarted due to a
> session disconnection
> resionserver's log
>  2023-07-05 02:25:35,213 INFO
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL: Slow sync cost: 272
> ms, current pipeline:   
> [DatanodeInfoWithStorage[10.11.8.18:9866,DS-a43ef0f9-d9e8-4fbd-a69b-1f2aec83cb8d,DISK],
> DatanodeInfoWithStorage[10.11.1.11:9866,DS-3a9a9b9d-a405-4a1b-af87-
>  9c0d38eb7fc6,DISK], DatanodeInfoWithStorage[10.11.8.19:9866
> ,DS-8466a61b-419c-44e7-85e2-30d28ff16c0f,DISK]]
>  2023-07-05 02:25:35,881 INFO org.apache.zookeeper.ClientCnxn: Unable to
> read additional data from server sessionid 0xff88d1721b6c53bc, likely
> server has closed socket,   closing socket connection and attempting
> reconnect
>  2023-07-05 02:25:36,573 INFO
> org.apache.hadoop.hbase.regionserver.throttle.PressureAwareThroughputController:
>  a40720868195bb1f851f94e01162801b#cf#compaction#13639 average throughput is
> 29.28   MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active
> operations   remaining, total limit is 69.23 MB/second
>  2023-07-05 02:25:36,591 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server dx-hbaseservice01.dx/10.11.39.10:2181. Will
> not attempt to   authenticate using SASL (unknown error)
>  2023-07-05 02:25:36,592 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established, initiating session, client: /10.11.8.18:29508,
> server: dx-   hbaseservice01.dx/10.11.39.10:2181
>  2023-07-05 02:25:36,597 WARN org.apache.zookeeper.ClientCnxn: Unable to
> reconnect to ZooKeeper service, session 0xff88d1721b6c53bc has expired
>  2023-07-05 02:25:36,597 INFO org.apache.zookeeper.ClientCnxn: EventThread
> shut down for session: 0xff88d1721b6c53bc
> Thanks,
> Miao Wang

Re: How to find current size of a znode

[Enrico Olivelli]

Il Gio 20 Lug 2023, 13:42 Aishwarya Soni  ha
scritto:

> Hi,
>
> Do we need to raise a feature request for this if currently it does not
> exist?
>


Yes, you can open a JIRA (you will have to request a JIRA that we will
approve).

That said, we will need some volunteer to code the patch.
Would you be available?

Enrico


> Regards,
> Aishwarya
>
> On Sat, Jul 15, 2023 at 4:59 PM Aishwarya Soni  >
> wrote:
>
> > We use Apache Storm and Solrcloud along with a few clients that store
> data
> > in the zookeeper. Due to one open bug in solrcloud
> > https://issues.apache.org/jira/browse/SOLR-16415, we see async_ids not
> > being deleted automatically. This is increasing the size of overseer
> znode
> > in zookeeper. Also, storm stores topologies jars (25+ topologies) inside
> > zookeeper which are also heavy in size. These two are the top 2 heavy
> > clients (other than a few other springboot microservices) that increase
> > zookeeper znode size.
> >
> > I want to see how we can get the active size of a specific znode so
> > that we can monitor it and also set the jute.maxbuffer value accordingly.
> >
> > I know zookeeper does not behave well with huge data being stored
> > inside it, but ignoring that fact, how can we get the znode size info?
> >
> > Regards,
> > Aishwarya Soni
> >
> > On Sat, Jul 15, 2023 at 2:24 AM Steph van Schalkwyk <
> > svanschalk...@gmail.com> wrote:
> >
> >> Take a look in the code repo. should be a simple pull.
> >> S
> >>
> >> On Fri, Jul 14, 2023 at 3:23 PM Ruel, Ryan 
> >> wrote:
> >>
> >> > We have an application where the size of individual ZNodes is small (a
> >> few
> >> > KB typically), however our data is distributed in the tree such that
> we
> >> can
> >> > have many sub nodes (10s of thousands, in some cases).
> >> >
> >> > When running the ZK CLI tool to view our data, I was surprised to see
> >> that
> >> > we started to get IOExceptions for exceeding the 1MB jute.maxbuffer.
> >> >
> >> > We've gotten around this by increasing the max buffer size to 10MB,
> but
> >> it
> >> > wasn't clear to me whether the ZNode allowed data size is impacted by
> >> the
> >> > number of sub nodes, or if this buffer size is just reused in various
> >> > places in the client code.
> >> >
> >> > ZK seems to operate just fine with these large numbers of sub nodes,
> >> it's
> >> > just the client tool that was complaining when trying to list sub
> nodes.
> >> >
> >> > /Ryan
> >> >
> >> > On 7/14/23, 3:01 PM, "Steph van Schalkwyk"  >> > > wrote:
> >> >
> >> >
> >> > To your last point - ZK was designed to distribute small packets,
> hence
> >> the
> >> > 1M buffer.
> >> > I've had a client who had a Solr connector that kept on creating new
> >> fields
> >> > from different sources, and the Solr schema quickly grew to 4M. That's
> >> > about the biggest I've seen ZK operate reliably.
> >> >
> >> >
> >> > On Fri, Jul 14, 2023 at 1:09 PM Aishwarya Soni <
> >> aishwarya.ash...@gmail.com
> >> > >
> >> > wrote:
> >> >
> >> >
> >> > > Hi,
> >> > >
> >> > > I want to find what is the current size/memory of a znode, i.e. how
> >> much
> >> > > its utilizing including all its child znodes. I know
> >> > > *zk_approximate_data_size* is the approximate memory consumption for
> >> ALL
> >> > > znodes stored in the ZooKeeper ensemble. But I need to find the
> active
> >> > size
> >> > > of a specific znode out of multiple znodes.
> >> > >
> >> > > How can we get it?
> >> > >
> >> > > Also, what is the safe max value we can assign to jute.maxbuffer? I
> am
> >> > > seeing packet length of 1 GB coming from a couple of clients and it
> is
> >> > > getting errored out with IOException due to jute.maxbuffer set to
> the
> >> > > default value of 1MB.
> >> > >
> >> > > Regards,
> >> > > Aishwarya
> >> > >
> >> >
> >> >
> >> >
> >> >
> >>
> >
>

Re: [ANNOUNCE] Apache ZooKeeper 3.8.2

[Enrico Olivelli]

Congratulations !

Thank you Mate for driving the release

Enrico

Il giorno mar 18 lug 2023 alle ore 17:19 Szalay-Bekő Máté
 ha scritto:
>
> The Apache ZooKeeper team is proud to announce Apache ZooKeeper version
> 3.8.2
>
> ZooKeeper is a high-performance coordination service for distributed
> applications. It exposes common services - such as naming,
> configuration management, synchronization, and group services - in a
> simple interface so you don't have to write them from scratch. You can
> use it off-the-shelf to implement consensus, group management, leader
> election, and presence protocols. And you can build on it for your
> own, specific needs.
>
> Release 3.8.2 is a bugfix release, solving 12 issues, including CVE fixes
> and
> additional test, security and other improvements.
>
> For ZooKeeper release details and downloads, visit:
> https://zookeeper.apache.org/releases.html
>
> ZooKeeper 3.8.2 Release Notes are at:
> https://zookeeper.apache.org/doc/r3.8.2/releasenotes.html
>
> We would like to thank the contributors who made the release possible.
>
> Regards,
> The ZooKeeper Team

Re: Re: Any change from 3.6.3 -> 3.6.4 would cause hostname unresolved issue?

[Enrico Olivelli]

Paolo,

Il Lun 19 Giu 2023, 16:43 Paolo Patierno  ha
scritto:

> Hi all,
> We were able to overcome the binding issue by setting
> quorumListenOnAllIPs=true but from there we are getting a new issue that is
> preventing leader election completion on first start-up.
>
> Getting the log of the current ZooKeeper leader (ID=3) we see the
> following.
> (Starting with ** you can see some additional logs added to
> org.apache.zookeeper.server.quorum.Leader#getDesignatedLeader in order to
> get more information.)
>
> 2023-06-19 12:32:51,990 INFO Have quorum of supporters, sids: [[1, 3],[1,
> 3]]; starting up and setting last processed zxid: 0x1
> (org.apache.zookeeper.server.quorum.Leader)
> [QuorumPeer[myid=3](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)]
> 2023-06-19 12:32:51,990 INFO **
>
> newQVAcksetPair.getQuorumVerifier().getVotingMembers().get(self.getId()).addr
> = my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.default.svc/
> 172.17.0.6:2888 (org.apache.zookeeper.server.quorum.Leader)
> [QuorumPeer[myid=3](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)]
> 2023-06-19 12:32:51,990 INFO ** self.getQuorumAddress() =
>
> my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.default.svc/:2888
> (org.apache.zookeeper.server.quorum.Leader)
> [QuorumPeer[myid=3](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)]
> 2023-06-19 12:32:51,992 INFO ** qs.addr
> my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.default.svc/
> 172.17.0.6:2888, qs.electionAddr
> my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.default.svc/
> 172.17.0.6:3888, qs.clientAddr/127.0.0.1:12181
> (org.apache.zookeeper.server.quorum.QuorumPeer)
> [QuorumPeer[myid=3](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)]
> 2023-06-19 12:32:51,992 DEBUG zookeeper
> (org.apache.zookeeper.common.PathTrie)
> [QuorumPeer[myid=3](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)]
> 2023-06-19 12:32:51,993 WARN Restarting Leader Election
> (org.apache.zookeeper.server.quorum.QuorumPeer)
> [QuorumPeer[myid=3](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)]
>
> So the leader is ZooKeeper with ID=3 and it was ACKed by the ZooKeeper node
> ID=1.
> As you can see we are in the Leader#startZkServer method, and because of
> the reconfiguration enabled, the designatedLeader is processed. The problem
> is that the Leader#getDesignatedLeader is not returning “self” as leader
> but another one (ID=1), because of the difference in the quorum address.
> From the above log, it’s not an actual difference in terms of addresses but
> the self.getQuorumAddress() is returning an  one (even if it’s
> still the same hostname related to ZooKeeper-2 instance). This difference
> causes the allowedToCommit=false, meanwhile the ZooKeeper-2 is still
> reported as leader but it’s not able to commit, so prevents any requests
> and the ZooKeeper ensemble gets stuck.
>
> 2023-06-19 12:32:51,996 WARN Suggested leader: 1
> (org.apache.zookeeper.server.quorum.QuorumPeer)
> [QuorumPeer[myid=3](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)]
> 2023-06-19 12:32:51,996 WARN This leader is not the designated leader, it
> will be initialized with allowedToCommit = false
> (org.apache.zookeeper.server.quorum.Leader)
> [QuorumPeer[myid=3](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)]
>
> The overall issue could be related to DNS problems, with DNS records not
> registered yet during pod initialization (where ZooKeeper is running on
> Kubernetes). But we don’t understand why it’s not able to recover somehow.
>
> Instead of using the quorumListenOnAllIPs=true we also tried a different
> approach by using the 0.0.0.0 address for the binding, so something like:
>
> # Zookeeper nodes configuration
> server.1=0.0.0.0:2888:3888:participant;127.0.0.1:12181
>
> server.2=my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.default.svc:2888:3888:participant;
> 127.0.0.1:12181
>
> server.3=my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.default.svc:2888:3888:participant;
> 127.0.0.1:12181
>
> This way, the self.getQuorumAddress() is not suffering the same problem, it
> doesn’t return an  address but always an actual one. No new
> leader election is needed and everything works fine.
>

This is the release notes page for 3.6.4.
https://zookeeper.apache.org/doc/r3.6.4/releasenotes.html

As you are running on k8s, I guess you are using a statefulset, maybe with
a service with ClusterIP?

Is the readyness probe failing? In that case the dns name should not be
available

What are you using to perform the probe? Are you using the four letters
words API ? (ruok)


Enrico


> On 2023/06/14 06:19:24 Szalay-Bekő Máté wrote:
> > Interesting...
> >
> > I am not familiar with strimzi.io.
> > Quickly checking the release notes, I don't see anything suspicious:
> > https://zookeeper.apache.org/doc/r3.6.4/releasenotes.html
> > Also, QuorumCnxManager was not changed for 2+ years on branch 3.6.
> >
> > Are you use the same java version and zookeeper config for 3.6.3 and
> 3.6.4?
> > Can you share the zookeeper config?
> >
> > Also: zookeeper 3.6 is 

Re: ZK is binding an ephemeral port

[Enrico Olivelli]

Andor

"Log4j 1.2 jmx support not found; jmx disabled."

This means that "log4j" JMX support is disabled.

You wrote in the very first line of your logs
"$ bin/zkServer.sh start-foreground
ZooKeeper JMX enabled by default"


Enrico

Il giorno mar 6 giu 2023 alle ore 16:00 Andor Molnar
 ha scritto:
>
> Looks like we already have a ticket about it:
>
> https://issues.apache.org/jira/browse/ZOOKEEPER-2910
>
> Sure, I'll try to repro on master in a minute.
>
>
>
> On Tue, 2023-06-06 at 21:24 +0800, tison wrote:
> > Can you reproduce this on master? Or only happen on 3.4 and 3.5?
> >
> > Best,
> > tison.
> >
> >
> > Andor Molnar  于2023年6月6日周二 21:16写道：
> >
> > > JMX is disabled by ManagedUtil:
> > >
> > > 2023-06-06 14:56:48,384 [myid:] - INFO  [main:ManagedUtil@47] -
> > > Log4j
> > > 1.2 jmx support not found; jmx disabled.
> > >
> > > Hence I don't see the 9010 tcp port bound. But why would a random
> > > ephemeral port still open?
> > >
> > > Important that this port is random and keeps changing at every
> > > startup.
> > >
> > > Andor
> > >
> > >
> > >
> > > On Tue, 2023-06-06 at 15:08 +0200, Enrico Olivelli wrote:
> > > > Andor,
> > > >
> > > > Il giorno mar 6 giu 2023 alle ore 14:59 Andor Molnar
> > > >  ha scritto:
> > > > > Hi folks,
> > > > >
> > > > > I cannot find an answer for this which annoys the hell out of
> > > > > me.
> > > > > Please help to understand why ZooKeeper is binding a local
> > > > > ephemeral
> > > > > port right after start without any quorum or client socket
> > > > > opened.
> > > > > This
> > > > > is what I see when start a 3.4 or 3.5 server is standalone
> > > > > mode.
> > > > > (Same
> > > > > behaviour is observed with quorum too)
> > > > >
> > > > > $ bin/zkServer.sh start-foreground
> > > > > ZooKeeper JMX enabled by default
> > > > > Using config: /home/andor/git/my-zookeeper/bin/../conf/zoo.cfg
> > > > > 2023-06-06 14:56:48,378 [myid:] - INFO  [main:
> > > > > QuorumPeerConfig@154]
> > > > > -
> > > > > Reading configuration from: /home/andor/git/my-
> > > > > zookeeper/bin/../conf/zoo.cfg
> > > > > ...
> > > > > 2023-06-06 14:56:48,648 [myid:] - INFO  [main:
> > > > > NIOServerCnxnFactory@689]
> > > > > - binding to port 0.0.0.0/0.0.0.0:2181
> > > > >
> > > > > ZooKeeper is running with PID 57126. Admin server is disabled.
> > > > >
> > > > > $ sudo netstat -plnt | grep 57126
> > > > > tcp6   0  0
> > > > > :::2181 :::*LISTEN  571
> > > > > 26/j
> > > > > ava
> > > > > tcp6   0  0
> > > > > :::45669:::*LISTEN  571
> > > > > 26/j
> > > > > ava
> > > > >
> > > > > What is the second line??
> > > >
> > > > It should be JMX
> > > >
> > > > Enrico
> > > >
> > > > > Thanks,
> > > > > Andor
> > > > >
> > > > >
> > > > >
>

Re: ZK is binding an ephemeral port

[Enrico Olivelli]

Andor,

Il giorno mar 6 giu 2023 alle ore 14:59 Andor Molnar
 ha scritto:
>
> Hi folks,
>
> I cannot find an answer for this which annoys the hell out of me.
> Please help to understand why ZooKeeper is binding a local ephemeral
> port right after start without any quorum or client socket opened. This
> is what I see when start a 3.4 or 3.5 server is standalone mode. (Same
> behaviour is observed with quorum too)
>
> $ bin/zkServer.sh start-foreground
> ZooKeeper JMX enabled by default
> Using config: /home/andor/git/my-zookeeper/bin/../conf/zoo.cfg
> 2023-06-06 14:56:48,378 [myid:] - INFO  [main:QuorumPeerConfig@154] -
> Reading configuration from: /home/andor/git/my-
> zookeeper/bin/../conf/zoo.cfg
> ...
> 2023-06-06 14:56:48,648 [myid:] - INFO  [main:NIOServerCnxnFactory@689]
> - binding to port 0.0.0.0/0.0.0.0:2181
>
> ZooKeeper is running with PID 57126. Admin server is disabled.
>
> $ sudo netstat -plnt | grep 57126
> tcp6   0  0
> :::2181 :::*LISTEN  57126/java
> tcp6   0  0
> :::45669:::*LISTEN  57126/java
>
> What is the second line??

It should be JMX

Enrico

>
> Thanks,
> Andor
>
>
>

Re: 3-node Zookeeper ensemble unable to recover if leader fails

[Enrico Olivelli]

Joao,

Il Mar 23 Mag 2023, 16:57 João Silva  ha scritto:

> Hi all,
>
> I've configured a 3-node Kafka (2.13) cluster with Zookeeper (3.6.3), with
> each Zookeeper instance living in the same machine as each Kafka broker
> (Java 11.0.18). Everything worked fine for a long long time.
>
> However, the first time a machine failed (so, both an instance of Zookeeper
> and a Kafka broker), the other 2 were unable to continue working (in this
> case, the leader failed). The 2 Zookeeper instances seemed like they
> couldn't communicate with each other, and were unable to elect a new
> leader. But that doesn't make sense, because they were communicating with
> each other before the failure. Only when the failing machine was booted up
> again, the other 2 machines were able to elect a new leader.
>
> From the logs, I don't get much more information than what I explained
> above. The 2 living machines act like they don't "see" each other, and are
> unable to elect a leader. When the failing machine goes up again, they
> manage to elect a new leader.
>

Are you on some managed environment like k8s?
Are you able to ssh into the nodes and use nc or any other tools to try to
connect to the other machines?


Enrico



> Does anyone can help me shed some light on this problem?
>
> Is there some configuration property I'm missing?
>
> From my internet crawl, I got these 2 articles with problems similar to
> mine, but they don't give a clear answer to why this happened and how to
> fix it:
>
>
> https://stackoverflow.com/questions/54005488/zookeeper-issue-taking-15-minutes-to-recover-if-leader-is-killed
>
>
> https://servicesunavailable.wordpress.com/2014/11/11/zookeeper-leader-election-and-timeouts/
>
> Thanks in advance,
> Joao
>

[ANNOUNCE] Apache ZooKeeper 3.8.1 released

[Enrico Olivelli]

The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 3.8.1

ZooKeeper is a high-performance coordination service for distributed
applications. It exposes common services - such as naming,
configuration management, synchronization, and group services - in a
simple interface so you don't have to write them from scratch. You can
use it off-the-shelf to implement consensus, group management, leader
election, and presence protocols. And you can build on it for your
own, specific needs.

For ZooKeeper release details and downloads, visit:
https://zookeeper.apache.org/releases.html

ZooKeeper 3.8.1 Release Notes are at:
https://zookeeper.apache.org/doc/r3.8.1/releasenotes.html

We would like to thank the contributors that made the release possible.

Regards,

The ZooKeeper Team

Re: [ANNOUNCE] Year 2023 :)

[Enrico Olivelli]

LOL

Enrico

Il giorno sab 31 dic 2022 alle ore 17:26 Szalay-Bekő Máté
 ha scritto:
>
> The Apache ZooKeeper team is proud to announce Year 2023
>
> You may have noticed that the year 2022 also gets EoL very soon. The new
> 2023 version was released successfully and - depending on time zones -
> people started to use it already.
>
> Based on the feedbacks the upgrades seemed to be smooth so far, at least no
> incompatibilities found. Sadly, we were not able to test this version
> before the release. Most likely it will introduce some new issues but
> hopefully also will solve some of the old ones. Let's enjoy it anyway...
>
> Happy new year! ;)

Re: Confusion around client ports and dynamic reconfiguration

[Enrico Olivelli]

Ruel,

Il giorno mer 16 nov 2022 alle ore 16:15 Ruel, Ryan
 ha scritto:
>
> It seems that specifying the SECURE client port in the reconfig command does 
> work, while also keeping the same port defined as "secureClientPort" in 
> zookeeper.conf.
>
> (I thought I had tried this, but may have missed this combination)
>
> In any case, some clarification within the documentation may be helpful!

Would you like to send a PR to add these clarifications?

Thanks
Enrico

>
> /Ryan
>
> On 11/15/22, 10:10 AM, "Ruel, Ryan"  wrote:
>
> In my ZooKeeper setup, I am strictly using TLS for both client and quorum 
> communication.
>
> In zookeeper.conf, I have “secureClientPort=2281” defined, and do not 
> have any “clientPort” option set.
>
> In the 3.8.0 documentation on dynamic reconfiguration 
> (https://urldefense.com/v3/__https://zookeeper.apache.org/doc/r3.8.0/zookeeperReconfig.html__;!!GjvTz_vk!T5lRlM3A1syL82ZMBR5kWiVdxaCcKIlRhDY-6muCLvhez9gyJTYbbhMkWMiee0evVrX0MQc-eYwvZhl1$
>  ), the documentation says that the old “clientPort” configuration option 
> should not be specified, and instead the new server keyword specification 
> should look like this:
>
> server. = ::[:role];[ address>:]**
>
> However, this specification doesn’t consider the secure client port from 
> what I can tell.
>
> In some cases where the server keyword is used, I can just eliminate 
> putting in the client port address and client port, such as within the quorum 
> peer configuration (in zookeeper.conf or within the dynamic configuration 
> file).
>
> In other cases, however, such as using the “reconfig” command in the ZK 
> cli utility, the client port MUST be specified, or a “bad argument” type 
> error is produced.
>
> I of course don’t want to put a dummy port number in the server 
> specification which would then enable insecure communication.
>
> What’s the recommendation for using secure communication only while also 
> using dynamic reconfiguration?
>
> P.S.  Another interesting bit in the documentation is the example:
> server.1=125.23.63.23:2780:2783:participant;2791
> server.2=125.23.63.24:2781:2784:participant;2792
> server.3=125.23.63.25:2782:2785:participant;2793
>
> In what use case would you want to use entirely different ports for each 
> server?  Or is this just a demonstration that this is possible?
>
> /Ryan
>
>
>
>
>
>

[ANNOUNCE] Apache Curator 5.4.0 released

[Enrico Olivelli]

Hello,

The Apache Curator team is pleased to announce the release of version 5.4.0.
Apache  Curator is a Java/JVM client  library for Apache ZooKeeper[1],
a distributed  coordination service.
Apache Curator includes a high-level API framework and utilities to
make using Apache ZooKeeper much easier and more reliable.
It also includes recipes for common use cases and extensions such as
service discovery and a Java 8 asynchronous DSL.
For more details, please visit the project website: https://curator.apache.org/

The download page for Apache Curator is here:
https://curator.apache.org/releases.html

The binary artifacts for Curator are available from Maven Central and
its mirrors.

For general information on Apache Curator, please visit the project website:
https://curator.apache.org


Release Notes - Apache Curator - Version 5.4.0

** Wish
* [CURATOR-596] - Upgrade ZooKeeper to 3.7.1

** Bug
* [CURATOR-578] - EnsembleTracker replace hostname connectString
with wrong ip from zk config
* [CURATOR-597] - Background exception was not retry-able or retry gave up
* [CURATOR-638] - Curator disconnect from zookeeper when IPs change
* [CURATOR-644] - CLONE - Race conditions in LeaderLatch after
reconnecting to ensemble
* [CURATOR-645] - LeaderLatch generates infinite loop with two
LeaderLatch instances competing for the leadership
* [CURATOR-648] - CuratorFramework#blockUntilConnected does now
wait forever if waitTime <= 0
* [CURATOR-649] - Background exception was not retry-able or retry gave up


** Improvement
* [CURATOR-458] - Fix Schema constructor validation logic
* [CURATOR-587] - Use ZooKeeper 3.7+ ZooKeeperServerEmbedded in
order to start TestingServer
* [CURATOR-642] - Upgrade Guava to 31.1
* [CURATOR-646] - Fix RAT failure reports and add a CI task to
prevent regression

** Task
* [CURATOR-524] - Apache is asking us to review our build files
and make sure we're using https for all URLs.
* [CURATOR-653] - Double leader for LeaderLatch


Regards,

The Curator Team

[1] Apache ZooKeeper https://zookeeper.apache.org/

Re: Zookeeper.sync() behavior

[Enrico Olivelli]

Keith

Il Gio 22 Set 2022, 15:03 Keith Turner  ha scritto:

> I have read some documentation about Zookeeper.sync() that makes me
> think there is no need to wait on its callback before reading data,
> but its not explicitly stated in what I have read. I am wondering if
> the following code is a reasonable way to get the latest data for
> '/a/b'?
>
>   ZooKeeper zooKeeper = null;
>
>   AsyncCallback.VoidCallback cb;
>   CompletableFuture returnCode = new CompletableFuture<>();
>   zooKeeper.sync("/a/b", (rc, path, ctx) -> returnCode.complete(rc), null);
>   zooKeeper.getData("/a/b",false, null);
>   if(returnCode.get() != 0) {
> //handle problem with sync
>   }
>
> Thanks
>

I think that generally you have to wait for 'sync' to complete
successfully, then you can call getData()

Otherwise  for instance if the error is ConnectionLoss you haven't really
sent the sync command to the server

I hope that helps

Enrico


> Keith
>

Re: A question about the client side exception

[Enrico Olivelli]

Yong,

Il giorno lun 5 set 2022 alle ore 09:29 Yong Zhang
 ha scritto:
>
> Hi guys,
>
> I saw client has three exceptions: AUTH_FAILED, SESSION_EXPIRED,
> and CONNECTION_LOSS. The question is about connection loss.
> Connection loss has various reason to cause. One scenario we saw is the
> client received the connection loss but the data updated successfully on
> the
> server, then we have a mismatch between the memory and zk server.  So
> our plan is to catch this error and retry it until an explicit error throws
> out. I
> want to know, if there has other exceptions we need to take care?
>
> And I read the code and find there isn’t a timeout exception and looks like
> most exceptions transform to the connection loss exception. Do I understand
> right?

yes
ConnectionLoss and SessionExpired and the most important conditions,
because starting from that error you are not receiving any other
updates from the server.
I don't think there is more to do

Enrico

>
> Look forward to any answer!
>
> Thank you
> Yong

Re: New releases page: endoflife.date

[Enrico Olivelli]

Thanks for the initiative

Enrico

Il giorno mer 27 lug 2022 alle ore 15:17 Kezhu Wang 
ha scritto:
>
> Hi Andor,
>
> Nice work.
>
> The anchor text and links for 3.6, 3.7 and 3.8 are wrong. I guess {{build}}
> does not contribute to the final version.
>
> Best,
> Kezhu
>
> On Wed, Jul 27, 2022 at 6:34 PM Andor Molnar  wrote:
>
> > Hi ZK folks,
> >
> > I'm lettig you know that I've added ZooKeeper to this page:
> >
> > https://endoflife.date/zookeeper
> >
> > It's pretty neat to track releases there. We don't need to manually
> > update it, because it monitors the tags on GitHub mirror page, so it
> > should be all automatic.
> >
> > Hope you like it.
> >
> > Andor
> >
> >
> >
> >

Re: Why do Zookeeper nodes only rejoin the ensemble if they are addressed as 0.0.0.0 in zoo.cfg

[Enrico Olivelli]

Usually the list of servers must be the same across all the ZK nodes, so
you have to set the dns name correctly.
The only change between the ZK servers is usually the contents of the myid
file.

If your machine needs the ZK server to bind on 0.0.0.0 there is a specific
option quorumListenOnAllIPs.

See this page for reference
https://zookeeper.apache.org/doc/r3.4.13/zookeeperAdmin.html#sc_clusterOptions

I hope that helps
Enrico


Il Ven 1 Lug 2022, 21:46 Heller, George A III CTR (USA)
 ha scritto:

> *-- Node1 zoo.cfg --*
>
> *tickTime=2000*
>
> *dataDir=I:/zookeeper/data*
>
> *clientPort=2181*
>
> *4lw.commands.whitelist=mntr,conf,ruok*
>
> *initLimit=5*
>
> *syncLimit=2*
>
> *server.1=0.0.0.0:2888:3888*
>
> *server.2=NODE2DNSNAME.local:2888:3888*
>
> *server.3=NODE3DNSNAME.local:2888:3888*
>
>
>
> *One question I had was why are you defining the server configuration in
> the zoo.cfg instead of zoo.cfg.dynamic file*
>
> * if you are using dynamic configuration introduced in the zookeeper 3.5+
> version? *
>
>
>
> *Regards, Aishwarya Soni*
>
>
>
> I am a newbie to Zookeeper(about 2 months) and I am not using dynamic
> configuration, the server names will never change.
>
> The only reason I am guessing is because I also supply the server names in
> the Solr start command(solr is the client for zookeeper).
>
>
>
> Command is: [solr start -cloud -f -p 8983 -z ” NODE1DNSNAME.local:2182,
> NODE2DNSNAME.local:2181, NODE3DNSNAME.local:2181”] I think zookeeper
> compares the solr start command to the zoo.cfg files and sees differences.
>
>
>
> BTW, I reply like this because I have subscribed to both zookeeper forums,
> but it does not send email to me. The solr forum sends email just fine.
>
>
>
> Thanks,
>
> George
>

Re: Can the leader of a Zookeeper be specifically selected at startup?

[Enrico Olivelli]

George,
really, it should not be a problem which is the leader. it is
automatically chosen.
Each node should be ideally as powerful as the other peers.

why do you need this "preferred leader" ?
I am afraid that you have some flaw in your design

Enrico

Il giorno lun 20 giu 2022 alle ore 05:39 Kezhu Wang 
ha scritto:
>
> Hi,
>
> I think this could be achieved with help from `reconfig`[1]:
> * Configs all nodes with `standaloneEnabled=false`, `reconfigEnabled=true`.
> * Starts node-2 as sole quorum participant.
> * Now node-2 is the leader. You will see "No server failure will be
> tolerated. You need at least 3 servers”.
> * Starts node-1 and node-3 with all quorum.
> * `zkCli.sh config` shows only node-2 for now.
> * `zkCli.sh reconfig -add node-1,node-2` will add both node-1, node-3 to
> quorum.
> * According to `Leader.tryToCommit`[2], node-2 will be the leader due to
> old leadership in old quorum and voter in new quorum.
>
> node-2 is the leader in whole progress.
>
> [1]: https://zookeeper.apache.org/doc/current/zookeeperReconfig.html
> [2]:
> https://github.com/apache/zookeeper/blob/b4f9aab099880ba8ef08eaff697debe6cdeae057/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/Leader.java#L950
>
> Best,
> Kezhu Wang
>
> On June 19, 2022 at 23:00:59, Heller, George A III CTR (USA) (
> george.a.heller2@mail.mil.invalid) wrote:
>
> We have 3 Zookeeper nodes and would like node 2 to always be the leader
> unless node 2 goes down. IF node 2 goes down, then either node 1 or node 3
> would be the leader.
>
>
>
> Can this be done? If so, how would this be done?

Re: Distinguish BadVersion a real exception or false positive for retry

[Enrico Olivelli]

Il giorno gio 16 giu 2022 alle ore 15:35 tison 
ha scritto:
>
> > if you see BadVersionException then the action must not have been
>
> From ZK code it seems that the comparison is about equality.
>
> Is it possible that:
>
> T0: setData(path, data, v0)
> T1: ConnectionLoss, but setData succeeded on the server, and thus version
> changed
> T2: client received ConnectionLoss, retry
> T3: client got BadVersionException, but actually the data is changed as it
> proposed

In any case you cannot rely on the status of the znode in case of
ConnectionLoss,
the fact that the change reached the server or not is not a big deal.
you have to revalidate your local view of the data.

in this case BadVersion and ConnectionLoss mean that you are not up-to-date

Enrico


>
> > you have to read the znode again and compare the version,
>
> Yeah..This is still a best effort check. If there are multiple writers even
> I get the version of znode and it's mismatched I don't know whether it ever
> succeed.
>
> Best,
> tison.
>
>
> Enrico Olivelli  于2022年6月16日周四 21:08写道：
>
> > Tison,
> >
> > Il giorno gio 16 giu 2022 alle ore 15:04 tison 
> > ha scritto:
> > >
> > > Hi ZooKeepers,
> > >
> > > When investigate this issue[1] I notice a possibility that if I write
> > such
> > > a program:
> > >
> > > while (true) {
> > > try {
> > >   zk.setData(path, data, version); // (1)
> > >   break;
> > > } catch(KeeperException.ConnectionLossException e) {
> > >// retry
> > > }
> > > }
> > >
> > > ... then in (1) there can be a case that it throws
> > > KeeperException.BadVersionException but actually the action is successful
> > > on the server side previously (but failed to send a response due to
> > > connection loss).
> >
> > if you see BadVersionException then the action must not have been
> > successful on the server
> > if you see ConnectionLossException then you know nothing about the outcome
> >
> > >
> > > Is this investigation right? If so, it is possibly to distinguish whether
> > > we succeed to apply the op on the server side?
> >
> > you have to read the znode again and compare the version,
> > but please remember that in the meantime (after your read response
> > leaves the server) someone could have change the znode
> >
> > I hope that helps
> >
> > Enrico
> >
> >
> > >
> > > Best,
> > > tison.
> > >
> > > [1] https://github.com/apache/pulsar/issues/13954
> >

Re: Distinguish BadVersion a real exception or false positive for retry

[Enrico Olivelli]

Tison,

Il giorno gio 16 giu 2022 alle ore 15:04 tison 
ha scritto:
>
> Hi ZooKeepers,
>
> When investigate this issue[1] I notice a possibility that if I write such
> a program:
>
> while (true) {
> try {
>   zk.setData(path, data, version); // (1)
>   break;
> } catch(KeeperException.ConnectionLossException e) {
>// retry
> }
> }
>
> ... then in (1) there can be a case that it throws
> KeeperException.BadVersionException but actually the action is successful
> on the server side previously (but failed to send a response due to
> connection loss).

if you see BadVersionException then the action must not have been
successful on the server
if you see ConnectionLossException then you know nothing about the outcome

>
> Is this investigation right? If so, it is possibly to distinguish whether
> we succeed to apply the op on the server side?

you have to read the znode again and compare the version,
but please remember that in the meantime (after your read response
leaves the server) someone could have change the znode

I hope that helps

Enrico


>
> Best,
> tison.
>
> [1] https://github.com/apache/pulsar/issues/13954

Re: Info about Zookeeper 3.8.0

[Enrico Olivelli]

Gello

Il Mar 3 Mag 2022, 17:54 dilip anand  ha scritto:

> Hi,
>
>
>
> Zookeeper 3.8.0 is using what version of log4j and is this "3.8.0" a stable
> release?
>

We are using logback in 3.8.0.
Zookeeper 3.8.0 is released as production ready.

If you are going to upgrade an existing cluster it is always a good
practice to test the upgrade in some staging environment with a
configuration similar to your production  environment

Best regards
Enrico



>
>
> Regards,
>
> Dilip
>
> --
> anand
>

[ANNOUNCE] Apache Curator 5.2.1 released

[Enrico Olivelli]

Hello,

The Apache Curator  team is pleased to announce the  release of version
5.2.1.

Apache  Curator is a Java/JVM client library for Apache ZooKeeper[1], a
distributed coordination service.

Apache Curator includes a  high-level API framework and utilities to  make
using Apache ZooKeeper much easier and more reliable.

It also includes recipes for common use cases and extensions such as
service discovery and a Java 8 asynchronous DSL.

For more details, please visit the project website:
https://curator.apache.org/


The download page for Apache Curator is here:
https://cwiki.apache.org/confluence/display/CURATOR/Releases


The binary artifacts for Curator are available from Maven Central and its
mirrors.

For general information on Apache Curator, please visit the project website:

https://curator.apache.org

Release 5.2.1 is a bugfix release that follows up release 5.2.0, users of
5.1.0 and 5.2.0 are encouraged to upgrade to pick up bug fixes and new
functionality.

Release Notes - Apache Curator - Version 5.2.1

** Bug
* [CURATOR-561] - Race condition preventing reconnection
* [CURATOR-606] - ModeledFrameworkImpl.update() ignores version
for uncompressed data
* [CURATOR-607] - Method getLockPath of InterProcessReadWriteLock
locks is not exposed for use in transactions
* [CURATOR-608] - ZPath resolved state documentation improvements

** Improvement
* [CURATOR-610] - Refactor CountCuratorWatcher in
TestWatcherIdentity.java to improve test logic
* [CURATOR-619] - Replace OutstandingOps with JDK bundled Phaser

** Task
* [CURATOR-625] - Fix "Cannot find a single highest directory for
this project set. First two candidates directories don't share a
common root" build error

Regards,
The Curator Team


[1] Apache ZooKeeper https://zookeeper.apache.org/

[ANNOUNCE] Apache ZooKeeper 3.8.0

[Enrico Olivelli]

The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 3.8.0

ZooKeeper is a high-performance coordination service for distributed
applications. It exposes common services - such as naming,
configuration management, synchronization, and group services - in a
simple interface so you don't have to write them from scratch. You can
use it off-the-shelf to implement consensus, group management, leader
election, and presence protocols. And you can build on it for your
own, specific needs.

For ZooKeeper release details and downloads, visit:
https://zookeeper.apache.org/releases.html

ZooKeeper 3.8.0 Release Notes are at:
https://zookeeper.apache.org/doc/r3.8.0/releasenotes.html

We would like to thank the contributors that made the release possible.

Regards,

The ZooKeeper Team

Re: When will 3.7.0 become a "stable" release

[Enrico Olivelli]

Hello,

Il Ven 25 Feb 2022, 15:20 Scott Guminy  ha scritto:

> 3.7.0 has been out since 27 March, 2021.  When will this release be marked
> as a stable release?  Is it ready for production environments?  Are there
> outstanding bugs that I should wait for?
>

We are going to release 3.8.0 very soon, a VOTE is out.

We should release 3.7.1, the main reason is to get rid of log4j1 that is
flagged by some CVEs.

The main difference between 3.8 and 3.7.1 is that in 3.8 we are shipping
logback and no more log4j1, so you can think to switch directly to 3.8.0.


You can check on JIRA for the list of fixes that went to 3.7.1. I can't
remember anything that is very bad.

Any feedback is really welcome, we are a community and everyone who shares
their experience is for the benefit of everybody

I hope that helps


Thanks
Enrico

Re: Help With zookeeper follower connections

[Enrico Olivelli]

Are you able to verify, using netstat, if those connections are from the
ZooKeeper process?
You can compare the TCP port

Enrico

Il Mer 23 Feb 2022, 14:15 Jason Grammenos
 ha scritto:

> I do have tools that could be opening connections. I have akhq (akhq.io)
> running on two of the hosts [host01, host04 (two nodes for redundancy
> reasons)], connecting to the kafka cluster. I also have telegraph running
> the jolokia2 agent plugin to pull monitoring/stats data from the hosts jmx
> (for both kafka and zookeeper)
>
>
> Jason Grammenos | Operations and Infrastructure Analyst
> Pronouns: he/him
> P: 613.232.7797 x1131
> Toll-free: 866.545.3745 x1131
> jason.gramme...@agilitypr.com
> agilitypr.com
> Learn new PR tips from our free resources.
>
> -----Original Message-
> From: Enrico Olivelli 
> Sent: February 23, 2022 8:08 AM
> To: jason.gramme...@agilitypr.com.invalid
> Cc: UserZooKeeper 
> Subject: Re: Help With zookeeper follower connections
>
> Jason,
> Do you have other tools on those machines that could open client
> connections?
>
> Enrico
>
> Il Mer 23 Feb 2022, 13:30 Jason Grammenos
>  ha scritto:
>
> > Hello,
> >
> >
> >
> > I have a 5 node zookeeper (+ kafka) cluster. I am trying to find out
> > if the inter node connection behaviour is normal.
> >
> > I should have 1 zookeeper leader and 4 followers, and I expect that
> > all 4 followers would open connections exclusively to the leader.
> >
> > So that “echo stat | nc localhost 2181” on the leader should show me 6
> > connections: 4 from the followers and 2 from itself (one via proper
> > ip, one from localhost)
> >
> >
> >
> > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > built on 05/04/2020 15:53 GMT
> >
> > Clients:
> >
> > /host02:48610[1](queued=0,recved=1455831,sent=1455833)
> >
> > /host03:36244[1](queued=0,recved=1459769,sent=1459793)
> >
> > /host05:53680[1](queued=0,recved=484,sent=484)
> >
> > /host01:48978[1](queued=0,recved=226,sent=226)
> >
> > /host04:44810[1](queued=0,recved=52,sent=52)
> >
> > /127.0.0.1:41434[0](queued=0,recved=1,sent=0)
> >
> >
> >
> > But instead I have ended up with a few connections on each node,
> > including the followers (as seen in below output). Some followers are
> > even connecting to themselves? The cluster looks stable, with only 1
> > leader and 4 followers, but I just do not understand this connection
> > behaviour and would like to understand if this is normal behaviour,
> > mis configuration, a bug or something else.
> >
> >
> >
> > user@host01:~$  echo stat | nc
> > localhost 2181
> >
> > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > built on 05/04/2020 15:53 GMT
> >
> > Clients:
> >
> > /host01:50768[1](queued=0,recved=6156,sent=6156)
> >
> > /127.0.0.1:35508[0](queued=0,recved=1,sent=0)
> >
> >
> >
> > Latency min/avg/max: 0/0/3
> >
> > Received: 9855
> >
> > Sent: 9854
> >
> > Connections: 2
> >
> > Outstanding: 0
> >
> > Zxid: 0xb0068
> >
> > Mode: follower
> >
> > Node count: 226
> >
> >
> >
> > user@host02:~$  echo stat | nc
> > localhost 2181
> >
> > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > built on 05/04/2020 15:53 GMT
> >
> > Clients:
> >
> > /127.0.0.1:49624[0](queued=0,recved=1,sent=0)
> >
> >
> >
> > Latency min/avg/max: 0/0/0
> >
> > Received: 3699
> >
> > Sent: 3698
> >
> > Connections: 1
> >
> > Outstanding: 0
> >
> > Zxid: 0xb0068
> >
> > Mode: follower
> >
> > Node count: 226
> >
> >
> >
> > usaer@host03:~$  echo stat |
> > nc localhost 2181
> >
> > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > built on 05/04/2020 15:53 GMT
> >
> > Clients:
> >
> > /host03:50674[1](queued=0,recved=6136,sent=6136)
> >
> > /127.0.0.1:54554[0](queued=0,recved=1,sent=0)
> >
> >
> >
> > Latency min/avg/max: 0/0/22
> >
> > Received: 9795
> >
> > Sent: 9794
> >
> > Connections: 2
> >
> > Outstanding: 0
> >
> > Zxid: 0xb0068
> >
> > Mode: follower
> >
> > Node count: 226
> >
> >
> >
> > user@host04:~$  echo stat | nc
> > localhost 2181
> >
> > Zookeeper ve

Re: Help With zookeeper follower connections

[Enrico Olivelli]

Jason,
Do you have other tools on those machines that could open client
connections?

Enrico

Il Mer 23 Feb 2022, 13:30 Jason Grammenos
 ha scritto:

> Hello,
>
>
>
> I have a 5 node zookeeper (+ kafka) cluster. I am trying to find out if
> the inter node connection behaviour is normal.
>
> I should have 1 zookeeper leader and 4 followers, and I expect that all 4
> followers would open connections exclusively to the leader.
>
> So that “echo stat | nc localhost 2181” on the leader should show me 6
> connections: 4 from the followers and 2 from itself (one via proper ip, one
> from localhost)
>
>
>
> Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built
> on 05/04/2020 15:53 GMT
>
> Clients:
>
> /host02:48610[1](queued=0,recved=1455831,sent=1455833)
>
> /host03:36244[1](queued=0,recved=1459769,sent=1459793)
>
> /host05:53680[1](queued=0,recved=484,sent=484)
>
> /host01:48978[1](queued=0,recved=226,sent=226)
>
> /host04:44810[1](queued=0,recved=52,sent=52)
>
> /127.0.0.1:41434[0](queued=0,recved=1,sent=0)
>
>
>
> But instead I have ended up with a few connections on each node, including
> the followers (as seen in below output). Some followers are even connecting
> to themselves? The cluster looks stable, with only 1 leader and 4
> followers, but I just do not understand this connection behaviour and would
> like to understand if this is normal behaviour, mis configuration, a bug or
> something else.
>
>
>
> user@host01:~$  echo stat | nc
> localhost 2181
>
> Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built
> on 05/04/2020 15:53 GMT
>
> Clients:
>
> /host01:50768[1](queued=0,recved=6156,sent=6156)
>
> /127.0.0.1:35508[0](queued=0,recved=1,sent=0)
>
>
>
> Latency min/avg/max: 0/0/3
>
> Received: 9855
>
> Sent: 9854
>
> Connections: 2
>
> Outstanding: 0
>
> Zxid: 0xb0068
>
> Mode: follower
>
> Node count: 226
>
>
>
> user@host02:~$  echo stat | nc
> localhost 2181
>
> Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built
> on 05/04/2020 15:53 GMT
>
> Clients:
>
> /127.0.0.1:49624[0](queued=0,recved=1,sent=0)
>
>
>
> Latency min/avg/max: 0/0/0
>
> Received: 3699
>
> Sent: 3698
>
> Connections: 1
>
> Outstanding: 0
>
> Zxid: 0xb0068
>
> Mode: follower
>
> Node count: 226
>
>
>
> usaer@host03:~$  echo stat | nc
> localhost 2181
>
> Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built
> on 05/04/2020 15:53 GMT
>
> Clients:
>
> /host03:50674[1](queued=0,recved=6136,sent=6136)
>
> /127.0.0.1:54554[0](queued=0,recved=1,sent=0)
>
>
>
> Latency min/avg/max: 0/0/22
>
> Received: 9795
>
> Sent: 9794
>
> Connections: 2
>
> Outstanding: 0
>
> Zxid: 0xb0068
>
> Mode: follower
>
> Node count: 226
>
>
>
> user@host04:~$  echo stat | nc
> localhost 2181
>
> Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built
> on 05/04/2020 15:53 GMT
>
> Clients:
>
> /host02:42874[1](queued=0,recved=143,sent=143)
>
> /host04:38222[1](queued=0,recved=89,sent=89)
>
> /127.0.0.1:58796[0](queued=0,recved=1,sent=0)
>
> Latency min/avg/max: 0/0/27
>
> Received: 287
>
> Sent: 286
>
> Connections: 3
>
> Outstanding: 0
>
> Zxid: 0xb0056
>
> Mode: follower
>
> Node count: 226
>
>
>
> user@host05:~$  echo stat | nc
> localhost 2181
>
> Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built
> on 05/04/2020 15:53 GMT
>
> Clients:
>
> /127.0.0.1:38680[0](queued=0,recved=1,sent=0)
>
> /host01:41198[1](queued=0,recved=6339,sent=6340)
>
>
>
> Latency min/avg/max: 0/0/8
>
> Received: 10041
>
> Sent: 10041
>
> Connections: 2
>
> Outstanding: 0
>
> Zxid: 0xb0068
>
> Mode: leader
>
> Node count: 226
>
> Proposal sizes last/min/max: 220/36/392
>
>
>
>
>
> Thank you
>
> Jason
>
>
>
> Jason Grammenos | Operations and Infrastructure Analyst
>
> Pronouns: he/him
> P: 613.232.7797 x1131
> Toll-free: 866.545.3745 x1131
> jason.gramme...@agilitypr.com
> agilitypr.com
>
> Learn new PR tips from our free resources
> .
>
>
>

Re: Zookeeper on windows standalone

[Enrico Olivelli]

Il Gio 6 Gen 2022, 19:33 rinilnath r  ha scritto:

> Hi,
>
> Thanks for your reply! Really great to see a hand for help 
>
> Default means my default node , zookeeper is its name.
>
> Basically, I did these
>
> 1. addauth digest zookeeper: adminpass
> 2. setAcl /zookeeper auth: zookeeper: adminpass:cdrwa
>
> Now getAcl /zookeeper, shows correctly the users added.
>
> Problem is when I restart zkserver, again I connect zkcli and all are gone.
>
> getAcl /zookeeper... Will say invalid ACL 3.5.3 or insufficient permission
> in 3.6.3
>
> No clue how to make a persistent set of ACL.
> My security compliance is blocked because of this.
>

Most of the times it is the Java application that sets the acls while
creating the znode.
I can't check your commands now (because I am out)

I hope that someone can give more feedback, otherwise I will try to help
next week


Enrico



>
>
> Thanks and Regards,
> Rinilnath
> Mobile#9786285451
>
> On Thu, 6 Jan, 2022, 22:40 Enrico Olivelli,  wrote:
>
> > Il Gio 6 Gen 2022, 14:45 rinilnath r  ha scritto:
> >
> > > Is this group still valid?
> > >
> >
> > Yes
> > This list is still valid.
> >
> > Probably there are many people still on vacation (like me :) )
> >
> > Can you please share more information?
> >
> > What is it a 'available zookeeper default node'?
> >
> > How are you setting acls?
> >
> >
> >
> > Enrico
> >
> >
> > I am getting ipage reply for my mail
> > >
> > > Thanks and Regards,
> > > Rinilnath
> > > Mobile#9786285451
> > >
> > > On Thu, 6 Jan, 2022, 17:58 rinilnath r,  wrote:
> > >
> > > >
> > > > Hi,
> > > >
> > > > I need to enable ACL in all nodes, as per security compliance.
> > > >
> > > > These are the things I tried
> > > >
> > > > 1. addauth digest with specific user
> > > > 2. SetAcl on available default zookeeper node
> > > >
> > > > But after service restart. The getAcl is saying insufficient
> permission
> > > >
> > > > Zookeeper: 3.6.3
> > > >
> > > > Please help me to do this properly.
> > > >
> > > > Thanks and Regards,
> > > > Rinilnath
> > > > Mobile#9786285451
> > > >
> > >
> >
>

Re: Zookeeper on windows standalone

[Enrico Olivelli]

Il Gio 6 Gen 2022, 14:45 rinilnath r  ha scritto:

> Is this group still valid?
>

Yes
This list is still valid.

Probably there are many people still on vacation (like me :) )

Can you please share more information?

What is it a 'available zookeeper default node'?

How are you setting acls?



Enrico


I am getting ipage reply for my mail
>
> Thanks and Regards,
> Rinilnath
> Mobile#9786285451
>
> On Thu, 6 Jan, 2022, 17:58 rinilnath r,  wrote:
>
> >
> > Hi,
> >
> > I need to enable ACL in all nodes, as per security compliance.
> >
> > These are the things I tried
> >
> > 1. addauth digest with specific user
> > 2. SetAcl on available default zookeeper node
> >
> > But after service restart. The getAcl is saying insufficient permission
> >
> > Zookeeper: 3.6.3
> >
> > Please help me to do this properly.
> >
> > Thanks and Regards,
> > Rinilnath
> > Mobile#9786285451
> >
>

Re: Using Zookeeper as producer-consumer

[Enrico Olivelli]

Hello

Il Mar 28 Dic 2021, 17:26 arvind garg  ha scritto:

> Hi Team,
>
> I am new to zookeeper and thinking to use zookeeper as Producer-Consumer
> for message passing as we do in Kafka. Will it be a good idea to go for
> zookeeper instead of Kafka?
>

No it isn't a good idea.
Zookeeper is meant for coordination and distributed configuration, leader
election and so on...

The most known platforms in this ecosystem are currently Apache Kafka,
Apache Pulsar and Pravega.io.
All of them are based on Zookeeper and the latter are based also on Apache
BookKeeper.

I suggest to take a look and compare those three alternatives

I hope that helps

Enrico




> Thanks
> Arvind Garg
>

Re: Impact of CVE-2021-4104

[Enrico Olivelli]

(Rewriting, my email client changed a few words...sorry)


We are going to switch to LogBack in the next release.
It will hopefully come soon, but we don't have a date.

By the way zookeeper uses slf4j so you could change the log provider by
removing log4j and adding your favourite logging implementation.

In Pulsar for instance we are launching zookeeper and putting log4j2 in the
classpath.

I hope that helps

Enrico

Il Dom 19 Dic 2021, 21:51 Enrico Olivelli  ha scritto:

> We are going to switch to Lombardia in the next release.
> It will hopefully come soon, but we don't have a date.
>
> By the way zookeeper uses slf4j so you could change the log provider by
> removing log4j and adding your favourite logging implementation.
>
> In Pulsar for instance we are launching zookeeper and putting log4j2 in
> the classpath.
>
> I hope that helps
>
> Enrico
>
> Il Dom 19 Dic 2021, 16:39 Jörn Franke  ha scritto:
>
>> Log4j 1.x should  generally not be used anymore. Since it is officially
>> not maintained anymore it is very unlikely that someone will report
>> vulnerabilities on it as they won’t be fixed anyway. Best would be to
>> upgrade to latest log4j 2.17 or later.
>>
>> > Am 18.12.2021 um 23:00 schrieb Rusty Deaton > .invalid>:
>> >
>> > Hi there,
>> >
>> > Given that zookeeper uses log4j 1.2, it appears as though there's a
>> > potentially large CVE,  https://nvd.nist.gov/vuln/detail/CVE-2021-4104
>> .
>> > Is there any official stance on this vulnerability?
>>
>

Re: Impact of CVE-2021-4104

[Enrico Olivelli]

We are going to switch to Lombardia in the next release.
It will hopefully come soon, but we don't have a date.

By the way zookeeper uses slf4j so you could change the log provider by
removing log4j and adding your favourite logging implementation.

In Pulsar for instance we are launching zookeeper and putting log4j2 in the
classpath.

I hope that helps

Enrico

Il Dom 19 Dic 2021, 16:39 Jörn Franke  ha scritto:

> Log4j 1.x should  generally not be used anymore. Since it is officially
> not maintained anymore it is very unlikely that someone will report
> vulnerabilities on it as they won’t be fixed anyway. Best would be to
> upgrade to latest log4j 2.17 or later.
>
> > Am 18.12.2021 um 23:00 schrieb Rusty Deaton  .invalid>:
> >
> > Hi there,
> >
> > Given that zookeeper uses log4j 1.2, it appears as though there's a
> > potentially large CVE,  https://nvd.nist.gov/vuln/detail/CVE-2021-4104 .
> > Is there any official stance on this vulnerability?
>

High heap usage during log replay

[Enrico Olivelli]

Hello,
I am investigating a case in which I have a tx log file of about 4GB, the
ZooKeeper server has 4GB of memory.
The overall database in memory is very small, less than 1 GB.

It looks like that when the ZK server starts and replays the txlog it is
using more than 4GB and it crashes.

The system is on ZK 3.5.x (it is actually Apache Pulsar 2.7.x).

I am not able to reproduce the issue with synthetic data,
has anyone ever observed this behaviour ?


Enrico

Re: Usage Of PEM Files for Keystores and Truststores in Apache zookeeper 3.5.9

[Enrico Olivelli]

Sai,

Il giorno mer 10 nov 2021 alle ore 20:01 sai chandra mouli <
tsaimouli1...@gmail.com> ha scritto:

> Hello,
>
> I am using Apache Zookeeper 3.5.9. My aim is to utilize existing PEM keys
> (private key and signed certificate ) to establish a TLS connection to
> zookeeper.
> I have combined both private key and public certificate into a single file
> and intend to use it as a keystore. Before appending the private key, I
> have converted it into pkcs8 format using *openssl pkcs8 -topk8 ...* . But
> I still kept it (the pkcs8 format private key) encrypted with a password.
> (- BEGIN ENCRYTED PRIVATE KEY- ***). Now, when I try to use
> this private key and its certificate as keystore, the zookeeper is throwing
> an error message
> *"Caused by: org.apache.zookeeper.common.X509Exception$KeyManagerException:
> java.security.spec.InvalidKeySpecException: Inappropriate key
> specification: IOException : DER input, Integer tag error*
> *". *I have mentioned both ssl.keyStore.password and ssl.key.password in
> the config file as a precaution. Still the error persists.
>
> As a next trial, I have removed the encryption and combined the private key
> and its public certificate into a single pem file. Now, when I use this
> file as keystore, I am able to connect to the zookeeper using TLS without
> any issues.
> Is encrypted private key not supported by the zookeeper PEM reader or am I
> missing something here?
>

Unfortunately I don't know, but if you can share the full stacktrace of the
error we can try to understand where the error comes from and follow up
with more details


Enrico


> I would be happy to reply with more details if needed. Hope you can help me
> solve the issue.
>
> Thanks and Regards,
> Sai Chandra Mouli T
>
> P.S: My domain certificate is signed by my own self-signed root CA.
>

Re: Meetup talk about Zookeeper and Pulsar

[Enrico Olivelli]

Damien,

Il Mer 13 Ott 2021, 11:22 Damien Diederen  ha scritto:

>
> Hi Enrico,
>
> > […] I will be talking about Zookeeper and Pulsar
>
> Hope you had a great time!  Do you know if the talk was recorded?  (I
> have found a pointer your slides, but no video, audio, or notes so far.)
>

Here it is
https://youtu.be/PWnO6YQ1HiM

Enrico


>
> https://www.slideshare.net/EnricoOlivelli/apache-bookkeeper-and-apache-zookeeper-for-apache-pulsar
>
> > I hope we could share more talks about Zookeeper here in the mailing
> list.
> > So if you know about events about Zookeeper,  feel free to share here!
>
> Agree!  (And thank you for the link; I hope to also contribute at some
> point.)
>
> Cheers, -D
>
>
>
> Enrico Olivelli  writes:
> > Hello,
> > Tomorrow I will be talking about Zookeeper and Pulsar
> >
> > The time is good for people in Europe and in Asia as it is organised by
> the
> > Pulsar Yahoo! Japan group.
> >
> > It is in English by the way.
> >
> > https://japan-pulsar-user-group.connpass.com/event/222026/
> >
> > I hope we could share more talks about Zookeeper here in the mailing
> list.
> > So if you know about events about Zookeeper,  feel free to share here!
> >
> > Cheers
> > Enrico
>

Meetup talk about Zookeeper and Pulsar

[Enrico Olivelli]

Hello,
Tomorrow I will be talking about Zookeeper and Pulsar

The time is good for people in Europe and in Asia as it is organised by the
Pulsar Yahoo! Japan group.

It is in English by the way.

https://japan-pulsar-user-group.connpass.com/event/222026/

I hope we could share more talks about Zookeeper here in the mailing list.
So if you know about events about Zookeeper,  feel free to share here!

Cheers
Enrico

Re: zookeeper ssl alert “alert bad certificate”

[Enrico Olivelli]

John,

Il giorno sab 14 ago 2021 alle ore 23:19 john mark <
johnmarkcaus...@gmail.com> ha scritto:

> Hi there!
>
> 
>
> I am using Kafka (version 2.3.0) and Zookeeper (version 3.5.5-3) - the
> stable version is 3.6.3.
>
> When I test the SSL of my Zookeeper using this command:
>
> openssl s_client -showcerts -connect 127.0.0.1:2280 -CAfile
> /certs/ca-chain.cert.pem
>
> and I am getting this error:
>
> 140371409225024:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad
> certificate:../ssl/record/rec_layer_s3.c:1543:SSL alert number 42
>


Is the ZooKeeper Java client able to connect ?

>
> But if I will install Zookeeper version 3.5.7 and up and I can add this in
> my zoo.cnf or zookeeper.properties:
>
> ssl.clientAuth=want and I no longer see any SSL errors.
>

So you are saying that with 3.5.7 we are able to see it working by enabling
TLS client auth ?
Do you have errors in the logs of the server ?

>
> Any tips/suggestions on how to fix this SSL error without upgrading (I
> don't want to update at the moment to avoid other conflicts like Kafka
> Cruise Control and others).
>


3.5.7 should be 100% compatible with 3.5.5 so upgrading the ZooKeeper
servers should not be an issue for the clients.

Enrico


>
> Thanks in advance!
>
> Best regards,
>
> John Mark Causing
>

Re: [ANNOUNCE] Apache Curator 5.2.0 released

[Enrico Olivelli]

Great !

Thank you Cameron for driving the release.

This release comes with a few interesting additions like
https://issues.apache.org/jira/browse/CURATOR-584

it would be great to write up something about this release and make more
buzz around Curator

Any volunteers ?

Enrico

Il giorno mar 27 lug 2021 alle ore 00:40 Cameron McKenzie <
cammcken...@apache.org> ha scritto:

> Hello,
>
>
> The Apache Curator  team is pleased to announce the  release of version
> 5.2.0.
>
> Apache  Curator is a Java/JVM client library for Apache ZooKeeper[1], a
> distributed coordination service.
>
> Apache Curator includes a  high-level API framework and utilities to  make
> using Apache ZooKeeper much easier and more reliable.
>
> It also includes recipes for common use cases and extensions such as
> service discovery and a Java 8 asynchronous DSL.
>
> For more details, please visit the project website:
> http://curator.apache.org/
>
>
> The download page for Apache Curator is here:
>
> https://cwiki.apache.org/confluence/display/CURATOR/Releases
>
>
> The binary artifacts for Curator are available from Maven Central and its
> mirrors.
>
>
> For general information on Apache Curator, please visit the project
> website:
>
> https://curator.apache.org
>
> Release 5.2.0 is a feature release that follows up release 5.1.0, users of
> 5.1.0 are encouraged to upgrade to pick up bug fixes and new functionality.
>
> Release Notes - Apache Curator - Version 5.2.0
>
> ** Bug
> * [CURATOR-526] - Error logged for valid config - "Invalid config event
> received: {properties}"
> * [CURATOR-585] - Discovery sample did not check for exceptions
>
> ** New Feature
> * [CURATOR-577] - WebSite should not like to Confluence for downloads
>
> ** Improvement
> * [CURATOR-582] - Migrate to jUnit 5.6
> * [CURATOR-584] - Curator Client Fault Tolerance Extensions
> * [CURATOR-588] - Upgrade ZooKeeper to 3.6.3
> * [CURATOR-590] - Adds possibility to disable parent creation for
> PersistentNode
> * [CURATOR-591] - Update PersistentNode documentation
> * [CURATOR-599] - Hanging indefinitely on some scenarios since
> zookeeper.request.timeout cannot be configured (add support for
> ZKClientConfig)
>
> ** Test
> * [CURATOR-602] - A typo in a variable of TestGroupMember
>
> Regards,
>
>
> The Curator Team
>
>
> [1] Apache ZooKeeper https://zookeeper.apache.org/
>

Re: Zookeeper version 3.6 performance comparison

[Enrico Olivelli]

Il Gio 15 Apr 2021, 04:27 rammohan ganapavarapu 
ha scritto:

> Hi,
>
> Is there any performance/benchmarking comparison available for the
> zookeeper version 3.6 with previous versions? We are planning to upgrade to
> 3.6 from 3.4.* and wanted to make sure we won't see perf drop.
>

Hi
We do not have official benchmarks.

Generally speaking each new major version comes with lots of improvements
and you can expect better performances.

The only thing to care about usually is the introduction of new features
and knobs that can change the performance of your workload but we care much
about not enablingnew featuresby default if those features may affect
performance.

I am sorry if that does not help much

Best regards
Enrico


> Thanks,
> Ram
>

Re: Is there an official ZooKeeper server interface definition?

[Enrico Olivelli]

Tison,
I am not sure I understand what you mean.
If you want to abstract from ZooKeeper you have to create your own set
of interfaces/classes that mimic ZooKeeper API.

The ZooKeeper client API is the main ZooKeeper client class.

If you what to start a ZooKeeper Server you can go with the
ZooKeeperServerEmbedded API available on 3.7.0 or with Curator (but
you already know this :-) )

Enrico

Il giorno gio 25 feb 2021 alle ore 09:30 tison 
ha scritto:
>
> Hi ZooKeepers,
>
> Recently I notice some active development on ClickHouse project[1] which
> said
>
> > Implementation of a server with ZooKeeper interface inside ClickHouse.
>
> I'm curious what is that mean. In other words, how to fake a ZooKeeper
> server with other implementation? What is the important procedure such a
> fake should simulate.
>
> Best,
> tison.
>
> [1] https://github.com/ClickHouse/ClickHouse/issues/17623

Re: write performance issue in 3.6.2

[Enrico Olivelli]

Li,
I wonder of we have some new throttling/back pressure mechanisms that is
enabled by default.

Does anyone has some pointer to relevant implementations?


Enrico

Il Ven 19 Feb 2021, 19:46 Li Wang  ha scritto:

> Hi,
>
> We switched to Netty on both client side and server side and the
> performance issue is still there.  Anyone has any insights on what could be
> the cause of higher latency?
>
> Thanks,
>
> Li
>
>
>
> On Mon, Feb 15, 2021 at 2:17 PM Li Wang  wrote:
>
> > Hi Enrico,
> >
> >
> > Thanks for the reply.
> >
> >
> > 1. We are using NIO based stack, not Netty based yet.
> >
> > 2. Yes, here are some metrics on the client side.
> >
> >
> > 3.6: throughput: 7K, failure: 81215228, Avg Latency: 57ms,  Max Latency
> 31s
> >
> > 3.4: throughput: 15k, failure: 0,  Avg Latency: 30ms,  Max Latency: 1.6s
> >
> >
> > 3. Yes, the JVM and zoo.cfg config are the exact same
> >
> > 10G of Heap
> >
> > 13G of Memory
> >
> > 5 Participante
> >
> > 5 Observere
> >
> > Client session timeout: 3000ms
> >
> > Server min session time: 4000ms
> >
> >
> >
> > 4. Yes, there are two types of  WARN logs and many “Expiring session”
> > INFO log
> >
> >
> > 2021-02-15 22:04:36,506 [myid:4] - WARN
> > [NIOWorkerThread-7:NIOServerCnxn@365] - Unexpected exception
> >
> > EndOfStreamException: Unable to read additional data from client, it
> > probably closed the socket: address = /100.108.63.116:43366, session =
> > 0x400189fee9a000b
> >
> > at
> >
> org.apache.zookeeper.server.NIOServerCnxn.handleFailedRead(NIOServerCnxn.java:164)
> >
> > at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:327)
> >
> > at
> >
> org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:522)
> >
> > at
> >
> org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
> >
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >
> > at java.base/java.lang.Thread.run(Thread.java:834)
> >
> >
> > 2021-02-15 22:05:14,428 [myid:4] - WARN
> > [SyncThread:4:SyncRequestProcessor@188] - Too busy to snap, skipping
> >
> >
> > 2021-02-15 22:01:51,427 [myid:4] - INFO
> > [SessionTracker:ZooKeeperServer@610] - Expiring session
> > 0x400189fee9a001e, timeout of 4000ms exceeded
> >
> >
> >
> > 5. Yes we upgrade both the client and the server to 3.6. Actually, the
> > issue happened with the combinations of
> >
> >
> > 3.4 client and 3.6 server
> >
> > 3.6 client and 3.6 server
> >
> > Please let me know if you need any additional info.
> >
> > Thanks,
> >
> > Li
> >
> >
> >
> >
> > On Mon, Feb 15, 2021 at 1:44 PM Li Wang  wrote:
> >
> >> Hi Enrico,
> >>
> >> Thanks for the reply.
> >>
> >> 1. We are using direct NIO based stack, not Netty based yet.
> >> 2. Yes, on the client side, here are the metrics
> >>
> >> 3.6:
> >>
> >>
> >>
> >>
> >> On Mon, Feb 15, 2021 at 10:44 AM Enrico Olivelli 
> >> wrote:
> >>
> >>> IIRC The main difference is about the switch to Netty 4 and about using
> >>> more DirectMemory. Are you using the Netty based stack?
> >>>
> >>> Apart from that macro difference there have been many many changes
> since
> >>> 3.4.
> >>>
> >>> Do you have some metrics to share?
> >>> Are the  JVM configurations and zoo.cfg configuration equals to each
> >>> other?
> >>>
> >>> Do you see warnings on the server logs?
> >>>
> >>> Did you upgrade both the client and the server or only the server?
> >>>
> >>> Enrico
> >>>
> >>>
> >>> Il Lun 15 Feb 2021, 18:30 Li Wang  ha scritto:
> >>>
> >>> > Hi,
> >>> >
> >>> > We want to upgrade from 3.4.14 to 3.6.2.  During the perform/load
> >>> > comparison test,  it was found that the performance of 3.6 has been
> >>> > significantly degraded compared to 3.4 for the write operation. Under
> >>> the
> >>> > same load, there was a huge number of SessionExpired and
> ConnectionLoss
> >>> > errors in 3.6 while no such errors in 3.4.
> >>> >
> >>> > The load testing is 500 concurrent users with a cluster of 5
> >>> participants
> >>> > and 5 observers. The min session timeout on the server side is
> 4000ms.
> >>> >
> >>> > I wonder if anyone has seen the same issue and has any insights on
> what
> >>> > could be the cause of the performance degradation.
> >>> >
> >>> > Thanks
> >>> >
> >>> > Li
> >>> >
> >>>
> >>
>

Re: write performance issue in 3.6.2

[Enrico Olivelli]

IIRC The main difference is about the switch to Netty 4 and about using
more DirectMemory. Are you using the Netty based stack?

Apart from that macro difference there have been many many changes since
3.4.

Do you have some metrics to share?
Are the  JVM configurations and zoo.cfg configuration equals to each other?

Do you see warnings on the server logs?

Did you upgrade both the client and the server or only the server?

Enrico


Il Lun 15 Feb 2021, 18:30 Li Wang  ha scritto:

> Hi,
>
> We want to upgrade from 3.4.14 to 3.6.2.  During the perform/load
> comparison test,  it was found that the performance of 3.6 has been
> significantly degraded compared to 3.4 for the write operation. Under the
> same load, there was a huge number of SessionExpired and ConnectionLoss
> errors in 3.6 while no such errors in 3.4.
>
> The load testing is 500 concurrent users with a cluster of 5 participants
> and 5 observers. The min session timeout on the server side is 4000ms.
>
> I wonder if anyone has seen the same issue and has any insights on what
> could be the cause of the performance degradation.
>
> Thanks
>
> Li
>

Re: ZooKeeper Server Not Listening on Port 2888

[Enrico Olivelli]

Brian
Thanks for sharing your story.
It is very helpful to the community


Enrico

Il Mar 9 Feb 2021, 00:51 Brian Rickabaugh  ha
scritto:

>   It looks like there were a couple of things going on here:
>
> 1.) After a little more digging, I found that the leader of the ensemble
> was listening on port 2888.  Does only the leader of a ZooKeeper ensemble
> listen on 2888?This seems to be the case but I didn't see it in the
> documentation somewhere.  I may have missed it.
>
> I found that only one of my servers, the leader, was listening on port
> 2888.  When I shut it down, another server in the ensemble started
> listening on port 2888.
>
> 2.) While I did check that iptables was clear off all rules, it appears
> that firewalld did have some rules that were blocking some of the
> connectivity.  When I added the appropriate rules for 2888/tcp and
> 3888/tcp, the ensemble is able to communicate as expected.
>
> I ran in to a red herring while diagnosing the issue.  I wanted to mention
> it here for the record.  I was receiving a
> "java.net.NoRouteToHostException: No route to host" exception before
> shutting down the firewall.  This was a little misleading as I knew there
> was a route between the VMs, they're on the same physical host and I could
> SSH from one to the other.  Tracing the route with traceroute proved it.
> The ncat command yielded the same message, no route, when trying to connect
> to these ports.  It was this fact that pointed me to something going on
> with a firewall somewhere.
>
> At any rate, problem solved.  Thanks for your help, Enrico.
>
> Brian
>
> Quoting Enrico Olivelli :
>
> > With a "grep 2888" you do not see any reference to 2888 ?
> >
> > Can you try with JDK11 ?
> > I am sure that ZK works with JDK15 (I have used it in production) and
> > JDK16-ea
> >
> > Enrico
> >
> > Il giorno lun 8 feb 2021 alle ore 07:44 Brian @ Rickabaugh.Net <
> > br...@rickabaugh.net> ha scritto:
> >
> >> I am using 3 different virtual machines.  One ZooKeeper server on each
> >> VM.  None of the hosts are listening on port 2888 and they each report
> >> the
> >> same error when trying to contact each other.
> >>
> >> I have placed the id file on each of the servers in the data directory.
> >> They each appear to know their individual ids.
> >>
> >> I disabled SELinux and the local firewall on each of the hosts and the
> >> issue remains.
> >>
> >> Brian
> >>
> >> On Feb 8, 2021, at 1:04 AM, Enrico Olivelli 
> wrote:
> >>
> >> Brian
> >>
> >> Il Lun 8 Feb 2021, 03:50 Brian Rickabaugh 
> ha
> >> scritto:
> >>
> >> Hello, everyone.
> >>
> >> I am working on configuring a new set of 3 ZooKeeper servers and I am
> >> finding that none of the servers are listening on port 2888.  There are
> >> no
> >> relevant messages or exceptions logged during start up.
> >>
> >> The servers are listening on port 3888.  The servers do appear to start
> >> up
> >> properly and connect to each other on port 3888.
> >>
> >> After a few seconds, I start seeing exceptions logged along the lines of
> >> the following:
> >>
> >> 2021-02-07 21:30:16,583 [myid:1] - WARN
> >>
> >>
>
> [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@129
> >> ]
> >> - Exception when following the leader
> >> java.io.IOException: Failed connect to box02/10.1.1.202:2888
> >> at
> >>
> >>
>
> org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:298)
> >> at
> >>
> >>
> org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:87)
> >> at
> >> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1458)
> >>
> >> I am using ZooKeeper 3.6.2, Fedora 33 and the following Java runtime.
> >>
> >> Are you using 3 virtual machines/servers or are you running everything
> in
> >> the same machine?
> >>
> >> Did you write the 'id' file in the data directory for each server?
> >>
> >> Enrico
> >>
> >> openjdk version "15.0.1" 2020-10-20
> >> OpenJDK Runtime Environment 20.9 (build 15.0.1+9)
> >> OpenJDK 64-Bit Server VM 20.9 (build 15.0.1+9, mixed mode, sharing)
> >>
> >> My ZooKeeper conf file is as follows:
> >>
> >> -
> >> tickTime=2000
> >> dataDir=/opt/zookeeper-data/
> >> clientPort=2181
> >> initLimit=5
> >> syncLimit=2
> >> server.1=box00:2888:3888
> >> server.2=box01:2888:3888
> >> server.3=box02:2888:3888
> >> -
> >>
> >> Any thoughts or ideas?
> >>
> >> Thanks in advance, Brian
> >>
> >
> >
>

Re: ZooKeeper Server Not Listening on Port 2888

[Enrico Olivelli]

With a "grep 2888" you do not see any reference to 2888 ?

Can you try with JDK11 ?
I am sure that ZK works with JDK15 (I have used it in production) and
JDK16-ea

Enrico

Il giorno lun 8 feb 2021 alle ore 07:44 Brian @ Rickabaugh.Net <
br...@rickabaugh.net> ha scritto:

> I am using 3 different virtual machines.  One ZooKeeper server on each
> VM.  None of the hosts are listening on port 2888 and they each report the
> same error when trying to contact each other.
>
> I have placed the id file on each of the servers in the data directory.
> They each appear to know their individual ids.
>
> I disabled SELinux and the local firewall on each of the hosts and the
> issue remains.
>
> Brian
>
> > On Feb 8, 2021, at 1:04 AM, Enrico Olivelli  wrote:
> >
> > Brian
> >
> > Il Lun 8 Feb 2021, 03:50 Brian Rickabaugh  ha
> > scritto:
> >
> >> Hello, everyone.
> >>
> >> I am working on configuring a new set of 3 ZooKeeper servers and I am
> >> finding that none of the servers are listening on port 2888.  There are
> no
> >> relevant messages or exceptions logged during start up.
> >>
> >> The servers are listening on port 3888.  The servers do appear to start
> up
> >> properly and connect to each other on port 3888.
> >>
> >> After a few seconds, I start seeing exceptions logged along the lines of
> >> the following:
> >>
> >> 2021-02-07 21:30:16,583 [myid:1] - WARN
> >>
> >>
> [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@129
> >> ]
> >> - Exception when following the leader
> >> java.io.IOException: Failed connect to box02/10.1.1.202:2888
> >> at
> >>
> >>
> org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:298)
> >> at
> >>
> org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:87)
> >> at
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1458)
> >>
> >> I am using ZooKeeper 3.6.2, Fedora 33 and the following Java runtime.
> >>
> >
> > Are you using 3 virtual machines/servers or are you running everything in
> > the same machine?
> >
> > Did you write the 'id' file in the data directory for each server?
> >
> > Enrico
> >
> >
> >> openjdk version "15.0.1" 2020-10-20
> >> OpenJDK Runtime Environment 20.9 (build 15.0.1+9)
> >> OpenJDK 64-Bit Server VM 20.9 (build 15.0.1+9, mixed mode, sharing)
> >>
> >> My ZooKeeper conf file is as follows:
> >>
> >> -
> >> tickTime=2000
> >> dataDir=/opt/zookeeper-data/
> >> clientPort=2181
> >> initLimit=5
> >> syncLimit=2
> >> server.1=box00:2888:3888
> >> server.2=box01:2888:3888
> >> server.3=box02:2888:3888
> >> -
> >>
> >> Any thoughts or ideas?
> >>
> >> Thanks in advance, Brian
> >>
>
>

Re: ZooKeeper Server Not Listening on Port 2888

[Enrico Olivelli]

Brian

Il Lun 8 Feb 2021, 03:50 Brian Rickabaugh  ha
scritto:

> Hello, everyone.
>
> I am working on configuring a new set of 3 ZooKeeper servers and I am
> finding that none of the servers are listening on port 2888.  There are no
> relevant messages or exceptions logged during start up.
>
> The servers are listening on port 3888.  The servers do appear to start up
> properly and connect to each other on port 3888.
>
> After a few seconds, I start seeing exceptions logged along the lines of
> the following:
>
> 2021-02-07 21:30:16,583 [myid:1] - WARN
>
> [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@129
> ]
> - Exception when following the leader
> java.io.IOException: Failed connect to box02/10.1.1.202:2888
> at
>
> org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:298)
> at
> org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:87)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1458)
>
> I am using ZooKeeper 3.6.2, Fedora 33 and the following Java runtime.
>

Are you using 3 virtual machines/servers or are you running everything in
the same machine?

Did you write the 'id' file in the data directory for each server?

Enrico


> openjdk version "15.0.1" 2020-10-20
> OpenJDK Runtime Environment 20.9 (build 15.0.1+9)
> OpenJDK 64-Bit Server VM 20.9 (build 15.0.1+9, mixed mode, sharing)
>
> My ZooKeeper conf file is as follows:
>
> -
> tickTime=2000
> dataDir=/opt/zookeeper-data/
> clientPort=2181
> initLimit=5
> syncLimit=2
> server.1=box00:2888:3888
> server.2=box01:2888:3888
> server.3=box02:2888:3888
> -
>
> Any thoughts or ideas?
>
> Thanks in advance, Brian
>

Re: ZooKeeper Operator

[Enrico Olivelli]

Thanks for sharing!

We need more support for K8s in the OS community, this is a good step

Enrico

Il giorno lun 18 gen 2021 alle ore 11:18 Flavio Junqueira 
ha scritto:

> We've been getting questions and sometimes contributions to the ZooKeeper
> Kubernetes Operator we originally did for Pravega, so I feel that it has
> been useful more broadly. Perhaps this is something that others might be
> interested in too, and I thought of mentioning here.
>
> https://github.com/pravega/zookeeper-operator
>
> Thanks,
> -Flavio

Re: The YCSB benchmark tool for zookeeper is now available

[Enrico Olivelli]

Great

Thank you very much Justin


Enrico

Il Lun 14 Dic 2020, 05:36 Justin Ling Mao  ha
scritto:

> Now users can benchmark your zookeeper ensemble with YCSB. Here is the
> ZK-PR(https://github.com/apache/zookeeper/pull/1558). Have fun for it:)

Re: ZooKeeper 3.5.9 release

[Enrico Olivelli]

Rajini,
Sorry for so late reply.

Do you have a way to update to 3.6.2? If you want to upgrade to 3.5.9 the
effort would be very like updating to 3.6.2.

Btw I think it would be worth to cut a new release, I am adding dev@ to my
answer in order to start a discussion.

We should just find a volounteer for driving the release.

Enrico

Il Ven 30 Ott 2020, 18:31 Rajini Sivaram  ha
scritto:

> Hi,
>
> Is there a timeline available for the 3.5.9 bug fix release which would
> contain the fix for https://issues.apache.org/jira/browse/ZOOKEEPER-3829?
>
> Thank you,
>
> Rajini
>

Re: Correlate Ephemeral Owner with connected session

[Enrico Olivelli]

Paul,
do you have a way to upgrade to the latest ZK 3.6.2 ?
many things changed since 3.4.6, it is a pretty old version

The session is declared as "expired" on the server side, and this will in
turn trigger the deletion of the ephemeral nodes, if they aren't deleted
the session is still active from the servers point of view or there is some
kind of bug

Enrico


Il giorno ven 30 ott 2020 alle ore 17:30 Paul Summermatter
 ha scritto:

> RE: ZooKeeper 3.4.6
>
> All,
>
> I'm trying to troubleshoot a problem and could use some guidance
> from the experts on ZK administration. I have a cluster of applications
> that share work and that create ephemeral nodes representing the work in ZK
> expressly so that, if one application fails, the ephemeral nodes should be
> deleted, and the other apps should be able to pick up the work that is now
> not being completed by the failed instance.
>
> Yesterday evening, one application instance suffered from some
> severe memory pressure and had to run multiple stop the world GC cycles.
> The pauses appear to have triggered a SessionExpiredException in
> org.apache.zookeeper.ClientCnxn$SendThread.run (I correlated multiple
> "Pause Full" statements in the GC logs with the ZK session timeout in the
> application logs). After the timeout, the connection was re-established in
> under 1,000ms, but the ephemeral nodes remained in ZooKeeper, leaving them
> as orphans. We've seen this behavior before and have had to delete the
> nodes manually using the zkCli.sh utility.
>
> In an attempt to troubleshoot this issue, I'm trying to correlate
> the ephemeral owner that is listed on a node when you run the 'get' command
> with the ID of an active session. Basically, I'm trying to understand
> whether ZK thinks there is still an active session associated with the
> ephemeral node in the hopes that that might lead to an explanation for why
> the ZK server didn't seem to recognize the session timeout sensed on the
> client that triggered a new connection and would explain why the ephemeral
> nodes were not deleted as they should have been when the connection dropped.
>
> I've tried the various four letter commands on the server to see
> if any of them output anything that looks like the ephemeral owner ID
> without any success. Any suggestions/guidance would be greatly appreciated.
> Note, right now, upgrading is not an option, but I'm certainly open to that
> if there are known issues with ephemeral nodes in 3.4 that are addressed in
> newer versions.
>
> Regards,
> Paul

Re: 32 Bit Zookeeper C-client

[Enrico Olivelli]

Il giorno mer 28 ott 2020 alle ore 16:27 Dipti Mulay 
ha scritto:

> Hi,
>
> I am a newbie to Zookeeper. I want to integrate the Zookeeper C-client
> with my 32 bit code base.
> So basically I need 32 bit Zookeeper C-client libraries.
>
> I have followed the following steps but I am not sure if I am on the right
> path. It would be great if someone could confirm.
>
>
>   1.  Downloaded the Source : Apache Zookeeper 3.6.2. Source Release.
>   2.  The server I am using is Redhat 7 (64 bit)
>   3.  Used the following mvn commands to build:
>  *   mvn clean install -Dskip Tests
>  *   mvn clean -Pfull-build
>  *   mvn install -Pfull-build -Dskip tests
>   4.  After this I do get the libzookeeper_mt.a and libzookeeper_st.a but
> these are 64 bit libraries.
>   5.  After this I change to directory:
> zookeeper-client/zookeeper-client-c/
>   6.  In this directory I run the following commands:
>  *   autoreconf -if
>  *   ./configure  CFLAGS=-m32 LDFLAGS= -m32 –prefix=/mydir/
>  *   make
>  *   make install
>   7.  After these steps I do get the libraries libzookeeper_mt.a and
> libzookeper_st.a in /mydir
>   8.  I can link these libraries with my 32 bit code base and it compiles
> well.
>
> Please let me know if this is the right way to do it ?
>

In my opinion you did it well, we should document it.

Enjoy ZooKeeper :-)

Enrico


>
> Many Thanks
> -Dipti
>
>
>
>

Re: upgrade from 3.4.5 to 3.5.6

[Enrico Olivelli]

Il giorno gio 8 ott 2020 alle ore 14:17 kuldeep singh <
kuldeep.sing...@gmail.com> ha scritto:

> Hi Team,
>
> I am facing one issue in SSL communication between client and zookeeper
> server.
>
> ZK 3.5.6 version
>
> 1. Mi on 3 node
> 2. Applying SSL and 3 nodes cluster is working fine
> 3. Scaled my cluster with 2 nodes and now my cluster have 5 nodes over SSL
>
> but after scaling my SSL is not working between client and ZK server and
> even not able to login using zkCli as well.
>
> Can someone provide the details please why it is happening?
>

Is your client configured to use SSL ?

Enrico



>
> Thanks,
> -
> Kuldeep Singh Budania
> Software Architect
>
>
>
> On Mon, Jul 13, 2020 at 2:19 PM Enrico Olivelli - Diennea
>  wrote:
>
> > It looks like we ported it to 3.5.
> >
> > See the subtask
> > https://issues.apache.org/jira/browse/ZOOKEEPER-2792
> >
> > Enrico
> >
> > Il giorno 13/07/20, 10:37 "kuldeep singh" 
> ha
> > scritto:
> >
> > Hi Team,
> >
> > I appreciate it if I will get a response as soon as possible, as I am
> > stuck
> > at this point.
> >
> > Thanks,
> > -
> > Kuldeep Singh Budania
> >
> >
> >
> > On Mon, Jul 13, 2020 at 11:10 AM kuldeep singh <
> > kuldeep.sing...@gmail.com>
> > wrote:
> >
> > > Hi Team,
> > >
> > > Server to Server communication is not supported in 3.5.6 version
> as
> > per
> > > below JIRA issue?
> > >
> > > https://issues.apache.org/jira/browse/ZOOKEEPER-2639
> > >
> > > Thanks,
> > > -
> > > Kuldeep Singh Budania
> > >
> > >
> > >
> > > On Thu, Jul 2, 2020 at 4:24 PM kuldeep singh <
> > kuldeep.sing...@gmail.com>
> > > wrote:
> > >
> > >> Thanks for the reply.
> > >>
> > >> Now my ZKCli cmd is working fine as we use some our customized
> > >> authentication and we resolve the issue.
> > >>
> > >> Now I am going to implement Server to Server communication.
> > >>
> > >> Thanks,
> > >> -
> > >> Kuldeep Singh Budania
> > >>
> > >>
> > >>
> > >> On Thu, Jul 2, 2020 at 3:53 PM Szalay-Bekő Máté <
> > >> szalay.beko.m...@gmail.com> wrote:
> > >>
> > >>> I think SSL is working for you already... If you managed to start
> > the
> > >>> zkCli.sh and be able to connect to ZooKeeper on the secure port
> > and issue
> > >>> any kind of command (like: " ls / "), then the wire encryption is
> > working
> > >>> and your server/client communication is secured by ZooKeeper.
> > >>>
> > >>> Why you want to run the following command?
> > >>> addauth ztpasswd zooadmin:
> > >>>
> > >>> Do you also want to configure a superDigest user in ZooKeeper?
> > Please
> > >>> note
> > >>> that this command is independent from SSL. If you need to create
> a
> > >>> username-password pair for digest authentication then please use
> > the
> > >>> command in the following way:
> > >>> addauth digest zooadmin:yourSuperSecretPassword
> > >>>
> > >>> Kind regards,
> > >>> Mate
> > >>>
> > >>> On Thu, Jul 2, 2020 at 6:59 AM kuldeep singh <
> > kuldeep.sing...@gmail.com>
> > >>> wrote:
> > >>>
> > >>> > 1. sh zkCli.sh --config /etc/zookeeper -server localhost:2281
> > >>> >
> > >>> > 2. addauth ztpasswd zooadmin:
> > >>> >
> > >>> >
> > >>> > Thanks,
> > >>> > -
> > >>> > Kuldeep Singh Budania
> > >>> >
> > >>> >
> > >>> >
> > >>> > On Thu, Jul 2, 2020 at 9:56 AM kuldeep singh <
> > >>> kuldeep.sing...@gmail.com>
> > >>> > wrote:
> > >>> 

ApacheCon is going to start

[Enrico Olivelli]

Hi ZooKeeper community,

Probably you already know that ApacheCon@Home is going to take place this
week.

Even if it will be a "virtual" conference this time, it will also be a
great chance to meet and see each other.

This is the list of Tracks and a link to Camille's keynote
https://apachecon.com/acah2020/tracks/keynotes.html#W1515
https://apachecon.com/acah2020/tracks/

As you can there are going to be talks about cool projects from people in
our community.

This is the link to the Slack workspace of the event
http://s.apache.org/apachecon-slack

Don't miss this chance of being part of this great ASF community event !

Attending to the event is free
https://hopin.to/events/apachecon-home#schedule

See you at ApacheCon@Home !

Enrico

Re: ZooKeeper Cluster Health Checking

[Enrico Olivelli]

Adrien

Il giorno mer 23 set 2020 alle ore 10:59 adrien ruffie <
adriennolar...@hotmail.fr> ha scritto:

> Hello all,
>
> I have a problem in production ...
>
> We have the following zoo configuration file:
>
> tickTime=4000
> dataDir=/var/lib/zookeeper
>
> dataLogDir=/var/log/zookeeper
>
> initLimit=30
> syncLimit=15
>
> autopurge.snapRetainCount=3
> autopurge.purgeInterval=48
>
> clientPort=2181
> maxClientCnxns=60
>
> server.1=ZOO1:2888:3888
> server.2=ZOO2:2888:3888
> server.3=ZOO3:2888:3888
> server.4=ZOO4:2888:3888
> server.5=ZOO5:2888:3888
>
> We are in zookeeper-3.4.10, but we recently saw, that log and snapshot
> aren't purge ...
> do you know this issue, is a bug, or bad configuration ?
>

Do you see errors in logs ?

Are you using standard Apache distributions?

Enrico


>
> Thank you very much and best regards
>
> Adrien Ruffié
> 
> De : adrien ruffie 
> Envoyé : mercredi 18 juillet 2018 09:01
> À : user@zookeeper.apache.org 
> Objet : RE: ZooKeeper Cluster Health Checking
>
> Ok thank Harish,
>
> I keep the idea !
>
>
> Best regards,
>
>
> Adrien
>
> 
> De : harish lohar 
> Envoyé : mardi 17 juillet 2018 23:13:28
> À : user@zookeeper.apache.org
> Objet : Re: ZooKeeper Cluster Health Checking
>
> We did it via java monitoring app , using zookeeper java api which sends 4
> lw commands to zookeeper and returns the output.
>
>
> Thanks
> Harish
>
> On Tue, Jul 17, 2018 at 2:00 AM adrien ruffie 
> wrote:
>
> > Hi Harish,
> >
> >
> > thank you very much for this advise and explanation !
> >
> > Do you think with just a simple script shell for checking all this
> metrics
> > is enough ? Or would better to do it in a Java with a simple monitoring
> > application?
> >
> >
> > Thank again,
> >
> >
> > Best regards,
> >
> >
> > Adrien
> >
> > 
> > De : harish lohar 
> > Envoyé : mardi 17 juillet 2018 04:13:51
> > À : user@zookeeper.apache.org
> > Objet : Re: ZooKeeper Cluster Health Checking
> >
> > Hi Adrian,
> > Below zookeeper commands are generally used to get health of zookeeper
> > cluster
> > stat
> >
> > Lists brief details for the server and connected clients.
> >
> > usage echo stat | nc server port
> >
> > This gives whether cluster is up /down. If down this will give that
> >
> > Zookeeper instance is currently not serving any request -  which means
> > either the leader election is failing or <= 50% of zookeeper node in
> > cluster are down.
> >
> >
> > mntr
> >
> > *New in 3.4.0:* Outputs a list of variables that could be used for
> > monitoring the health of the cluster.
> >
> > $ echo mntr | nc localhost 2185
> >
> > zk_version  3.4.0
> > zk_avg_latency  0
> > zk_max_latency  0
> > zk_min_latency  0
> > zk_packets_received 70
> > zk_packets_sent 69
> > zk_outstanding_requests 0
> > zk_server_state leader
> > zk_znode_count   4
> > zk_watch_count  0
> > zk_ephemerals_count 0
> > zk_approximate_data_size27
> > zk_followers4   - only exposed by the Leader
> > zk_synced_followers 4   - only exposed by the Leader
> > zk_pending_syncs0   - only exposed by the Leader
> > zk_open_file_descriptor_count 23- only available on Unix platforms
> > zk_max_file_descriptor_count 1024   - only available on Unix platforms
> >
> > The output is compatible with java properties format and the content may
> > change over time (new keys added). Your scripts should expect changes.
> >
> > ATTENTION: Some of the keys are platform specific and some of the keys
> are
> > only exported by the Leader.
> >
> > The output contains multiple lines with the following format:
> >
> >
> > On Mon, Jul 16, 2018 at 10:13 AM adrien ruffie <
> adriennolar...@hotmail.fr>
> > wrote:
> >
> > > Hello all,
> > >
> > >
> > > In my company we have a Zookeeper production cluster.
> > >
> > >
> > > But we don't really know how can we check the health of our cluster...
> > >
> > >
> > > Can we advise us about this topic ?
> > >
> > >
> > > I know this topic may has been cropping up for a while, but I don't
> > really
> > > found any concrete solution.
> > >
> > >
> > > Do you use a monitoring tools ? Which can launch alert ?
> > >
> > > What metrics/properties/any thing which can indicate that our cluster
> > > isn't in good health.
> > >
> > >
> > > Thank you very much and best regards
> > >
> > >
> > > Adrien
> > >
> >
>

Re: Issues building and running Zookeeper Inspector

[Enrico Olivelli - Diennea]

Brent
Please go ahead and send a PR

You can also subscribe to d...@zookeeper.apache.org for discussions related to 
patches.

Thank you very much
Enrico

Il giorno 22/09/20, 19:52 "Brent"  ha scritto:

Hi everyone,

I just filed a Jira related to the Zookeeper Inspector contrib project here:

https://issues.apache.org/jira/browse/ZOOKEEPER-3943

I just wanted to reach out and make sure I'm going about attempting to use
it correctly.  I just ran "mvn clean install -DskipTests" at all levels of
the code tree and then attempted to use both the "zooInspector.sh" script
and invoke the Java directly (with all the CLASSPATH set up properly).

It seems like the core of my issue was that the icons for the UI couldn't
be found (they don't seem to get built into the JAR by default) and
resulted in a bunch of NullPointerExceptions.  I put a proposal to fix this
in the Jira, but wanted to make sure it seems like an acceptable approach
and double-check that I'm not just doing something incorrectly.

If this seems OK and nobody is actively working on this already, I'd be
happy to submit a PR if it would help.

Thanks!




CONFIDENTIALITY & PRIVACY NOTICE
This e-mail (including any attachments) is strictly confidential and may also 
contain privileged information. If you are not the intended recipient you are 
not authorised to read, print, save, process or disclose this message. If you 
have received this message by mistake, please inform the sender immediately and 
destroy this e-mail, its attachments and any copies. Any use, distribution, 
reproduction or disclosure by any person other than the intended recipient is 
strictly prohibited and the person responsible may incur in penalties.
The use of this e-mail is only for professional purposes; there is no guarantee 
that the correspondence towards this e-mail will be read only by the recipient, 
because, under certain circumstances, there may be a need to access this email 
by third subjects belonging to the Company.

Re: Zookeeper on Kubernetes and Presistent Volumes

[Enrico Olivelli]

Remi,
sorry for the late reply.

Your cluster would be able to work if and only if at least X/2 + 1 servers
are up and running and properly connected to the other peers.
any server that is restarted will rejoin from scratch.
It is very dangerous ! you won't be able to recover and probably it will be
very hard to understand what happened (you are going to lose all of your tx
logs)

I don't have experience with Kubernetes and non persistent volumes but as
far as I know it is not supposed to work

Enrico


Il giorno ven 11 set 2020 alle ore 14:21 Remi Serrano 
ha scritto:

> Hello mailing list,
>
> We are assessing running Zookeeper in Kubernetes. There are a bunch of
> examples around and they all use Kubernetes Persistent Volumes.
> For some underlying technical reasons, we would like to avoid the use of
> Kubernetes Persistent Volumes.
> What is the risk to setup a ZK cluster on Kubernetes without persistent
> volume ?
> Sub-question, what happen if a ZK node without persistent get killed and
> rebooted by Kubernetes (ie: without any data) ?
>
> Thanks for your help
>
> Rémi
>
>

Working only (was Re: Zookeeper on Kubernetes and Presistent Volumes)

[Enrico Olivelli]

Rudrani



Il Sab 12 Set 2020, 00:13 Rudrani Angira  ha
scritto:

> Hello everyone
>
> I have two questions to get started on zookeeper project.
>
> How can I build the code and what test cases should I start with.
>

Just run
mvn clean verify
To run the build and tests




I am planning to use eclipse as an IDE for the project. Is anything else
> recommended.
>

Any IDE that supports Maven will work.

Personally I use Apache NetBeans, just open the zookeeper-server  project
and you can start to code and run tests.

Use jdk8 or jdk11+

Enrico

>
> Thank you
> Rudrani
>
> On Fri, Sep 11, 2020 at 5:21 AM Remi Serrano  wrote:
>
> > Hello mailing list,
> >
> > We are assessing running Zookeeper in Kubernetes. There are a bunch of
> > examples around and they all use Kubernetes Persistent Volumes.
> > For some underlying technical reasons, we would like to avoid the use of
> > Kubernetes Persistent Volumes.
> > What is the risk to setup a ZK cluster on Kubernetes without persistent
> > volume ?
> > Sub-question, what happen if a ZK node without persistent get killed and
> > rebooted by Kubernetes (ie: without any data) ?
> >
> > Thanks for your help
> >
> > Rémi
> >
> >
>

[ANNOUNCE] Apache ZooKeeper 3.6.2 released

[Enrico Olivelli]

The Apache ZooKeeper team is proud to announce Apache ZooKeeper version
3.6.2

ZooKeeper is a high-performance coordination service for distributed
applications. It exposes common services - such as naming,
configuration management, synchronization, and group services - in a
simple interface so you don't have to write them from scratch. You can
use it off-the-shelf to implement consensus, group management, leader
election, and presence protocols. And you can build on it for your
own, specific needs.

For ZooKeeper release details and downloads, visit:
https://zookeeper.apache.org/releases.html

ZooKeeper 3.6.2 Release Notes are at:
https://zookeeper.apache.org/doc/r3.6.2/releasenotes.html

We would like to thank the contributors that made the release possible.

Regards,

The ZooKeeper Team

Re: 3.4.8 to 3.5.8 rolling update

[Enrico Olivelli]

James

Il Lun 7 Set 2020, 18:07 James Gover  ha
scritto:

> Hi,
>
> We are upgrading from zookeeper 3.4.8 to 3.5.8 via 3.4.14.
> There are several mentions of going from 3.4.6 to 3.5.8, is going from
> 3.4.14 to 3.5.8 ok.
>
> In terms of Leader-tasks running during the upgrade. The intent is to
> migrate a cluster of 3 zk nodes by:
> * scaling up to 5 nodes (adding two new nodes)
> * one non-leader node at a time; destroying the instance and replacing it
> with a new 3.5.8 node
> * then stoping zk on the leader so another node becomes the leader,
> terminating the instance then adding a 3.5.8 node.
> * removing two (non-leader) node to scale back to 3 nodes
>
> With this process; will leader tasks running on the 3.4.14 leader (using
> curator leaderLatch) fail when this leader is terminated but succeed if run
> after a short delay will run on a new 3.5.8 leader node.
>


Zookeeper itself does not run tasks, it is only the coordination system
that tells which one is the current leader among the peers.

The upgrade of Zookeeper should be seamless for the application  as far as
you keep the Zookeeper quorum up and running, usually by keeping alive
half+1 number of Zookeeper nodes.


I hope that helps
Enrico


> Thanks,
>
> James
>
>
>
> --
> Sent from: http://zookeeper-user.578899.n2.nabble.com/
>

Re: upgrade from 3.4.5 to 3.5.6

[Enrico Olivelli - Diennea]

le?
>>> > >>> > Also can you please send the zkCli command you execute? (you
>>> need to
>>> > >>> > connect to the secure ZooKeeper port, unless portUnification is
>>> > >>> enabled)
>>> > >>> >
>>> > >>> > Kind regards,
>>> > >>> > Mate
>>> > >>> >
>>> > >>> > On Wed, Jul 1, 2020 at 9:48 AM kuldeep singh <
>>> > >>> kuldeep.sing...@gmail.com>
>>> > >>> > wrote:
>>> > >>> >
>>> > >>> >> Hi,
>>> > >>> >>
>>> > >>> >> we have done below changes in java.env file
>>> > >>> >>
>>> > >>> >> export SERVER_JVMFLAGS="
>>> > >>> >>
>>> > >>> >>
>>> > >>>
>>> -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
>>> > >>> >>
>>> > -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks
>>> > >>> >> -Dzookeeper.ssl.keyStore.password=testpass
>>> > >>> >>
>>> > >>>
>>> >
>>> 
-Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks
>>> > >>> >> -Dzookeeper.ssl.trustStore.password=testpass"
>>> > >>> >>
>>> > >>> >>
>>> > >>> >>
>>> > >>> >> export CLIENT_JVMFLAGS="
>>> > >>> >>
>>> > >>> >>
>>> > >>>
>>> -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
>>> > >>> >> -Dzookeeper.client.secure=true
>>> > >>> >>
>>> > -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks
>>> > >>> >> -Dzookeeper.ssl.keyStore.password=testpass
>>> > >>> >>
>>> > >>>
>>> >
>>> 
-Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks
>>> > >>> >> -Dzookeeper.ssl.trustStore.password=testpass"
>>> > >>> >>
>>> > >>> >> I have started the ZK server and it is up without any issue.
>>> > >>> >>
>>> > >>> >> But now when I login to ZkCli then it gives the below error.
>>> > >>> >>
>>> > >>> >> WatchedEvent state:AuthFailed type:None path:null
>>> > >>> >>
>>> > >>> >>
>>> > >>> >> Zookeeper logs :- *2020-07-01 07:38:09,342 - WARN
>>> > >>> >> [nioEventLoopGroup-4-2:ZooKeeperServer@1119] - No
>>> authentication
>>> > >>> provider
>>> > >>> >> for scheme: ztpasswd has x509 ip digest*
>>> > >>> >>
>>> > >>> >> Please help me on this issue
>>> > >>> >>
>>> > >>> >> Thanks,
>>> > >>> >> -
>>> > >>> >> Kuldeep Singh Budania
>>> > >>> >> Software Architect
>>> > >>> >>
>>> > >>> >>
>>> > >>> >> On Wed, Jul 1, 2020 at 12:05 PM kuldeep singh <
>>> > >>> kuldeep.sing...@gmail.com>
>>> > >>> >> wrote:
>>> > >>> >>
>>> > >>> >> > Hi,
>>> > >>> >> >
>>> > >>> >> > My ZK server  is up and running in secure mode, But When I am
>>> > >>> trying to
>>> > >>> >> > connect to the ZK server using ZKCli, it gives the below
>>> error.
>>> > >>> >> >
>>> > >>> >> > WatchedEvent state:AuthFailed type:None path:null
>>> > >>> >> >
>>> > >>> >>

[ANNOUNCE] Apache Curator 5.1.0 released

[Enrico Olivelli]

Hello,


The Apache Curator  team is pleased to announce the  release of version
5.1.0.

Apache  Curator is a Java/JVM client library for Apache ZooKeeper[1], a
distributed coordination service.

Apache Curator includes a  high-level API framework and utilities to  make
using Apache ZooKeeper much easier and more reliable.

It also includes recipes for common use cases and extensions such as
service discovery and a Java 8 asynchronous DSL.

For more details, please visit the project website:
http://curator.apache.org/


The download page for Apache Curator is here:

https://cwiki.apache.org/confluence/display/CURATOR/Releases


The binary artifacts for Curator are available from Maven Central and its
mirrors.


For general information on Apache Curator, please visit the project website:

https://curator.apache.org


Release 5.1.0 is a bugfix release that follows up release 5.0.0, users of
5.0.0 are encouraged to upgrade.


Release Notes:

 [CURATOR-574] - DiscoveryService fatal error on deserializing an empty
byte[] as JSON

 [CURATOR-575] - TestingServer shut down can cause NullPointerException



Regards,


The Curator Team


[1] Apache ZooKeeper https://zookeeper.apache.org/

Problems with Grafana dashboard

[Enrico Olivelli]

Hello,
I am trying to import this Grafana Dashboard that @Justin Ling Mao
  designed for ZooKeeper
https://grafana.com/grafana/dashboards/10465


But I have this problem
react-dom.production.min.js:171 TypeError: Cannot read property 'exports'
of undefined
at e.loadPlugin (DashboardGrid.tsx:166)
at e.componentDidMount (DashboardGrid.tsx:208)
at da (react-dom.production.min.js:213)
at ca (react-dom.production.min.js:205)
at la (react-dom.production.min.js:204)
at Fo (react-dom.production.min.js:200)
at Object.enqueueSetState (react-dom.production.min.js:130)
at r.y.setState (react.production.min.js:13)
at r.i.strategisedSetState (react-sizeme.js:308)
at react-sizeme.js:336

I thought I tested it when Justin first developed it

Does anyone else has the same problem?

Enrico

Re: upgrade from 3.4.5 to 3.5.6

[Enrico Olivelli - Diennea]

I mean in zoo.cfg
Not as a system property

Enrico

Il giorno 25/06/20, 08:19 "Enrico Olivelli - Diennea" 
 ha scritto:

Hi
You have to enable Netty on the server side

Something like:
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory

Hope that helps
Enrico

Il giorno 24/06/20, 19:17 "kuldeep singh"  ha 
scritto:

Hi,

I got below error while setting SSL properties in zkEnv.sh




==

2020-06-24 15:49:35,864 - INFO  [main:QuorumPeerConfig@133] - Reading
configuration from: /etc/zookeeper/zoo.cfg

2020-06-24 15:49:35,874 - INFO  [main:QuorumPeerConfig@385] -
clientPortAddress is 0.0.0.0/0.0.0.0:10181

2020-06-24 15:49:35,874 - INFO  [main:QuorumPeerConfig@399] -
secureClientPortAddress is 0.0.0.0/0.0.0.0:2281

2020-06-24 15:49:35,878 - INFO  [main:X509Util@79] - Setting -D
jdk.tls.rejectClientInitiatedRenegotiation=true to disable 
client-initiated
TLS renegotiation

2020-06-24 15:49:35,897 - INFO  [main:DatadirCleanupManager@78] -
autopurge.snapRetainCount set to 3

2020-06-24 15:49:35,897 - INFO  [main:DatadirCleanupManager@79] -
autopurge.purgeInterval set to 1

2020-06-24 15:49:35,898 - INFO  [
PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task started.

2020-06-24 15:49:35,899 - INFO  [main:ManagedUtil@46] - Log4j found with
jmx enabled.

2020-06-24 15:49:35,903 - INFO  [PurgeTask:FileTxnSnapLog@103] -
zookeeper.snapshot.trust.empty : false

2020-06-24 15:49:35,910 - INFO  [
PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed.

2020-06-24 15:49:35,975 - INFO  [main:QuorumPeerMain@141] - Starting 
quorum
peer

2020-06-24 15:49:35,983 - INFO  [main:ServerCnxnFactory@135] - Using
org.apache.zookeeper.server.NIOServerCnxnFactory as server connection
factory

2020-06-24 15:49:35,986 - INFO  [main:NIOServerCnxnFactory@673] -
Configuring NIO connection handler with 10s sessionless connection 
timeout,
2 selector thread(s), 16 worker threads, and 64 kB direct buffers.

2020-06-24 15:49:35,992 - INFO  [main:NIOServerCnxnFactory@686] - 
binding
to port 0.0.0.0/0.0.0.0:10181

2020-06-24 15:49:35,994 - INFO  [main:ServerCnxnFactory@135] - Using
org.apache.zookeeper.server.NIOServerCnxnFactory as server connection
factory

2020-06-24 15:49:35,995 - ERROR [main:QuorumPeerMain@101] - Unexpected
exception, exiting abnormally

java.lang.UnsupportedOperationException: SSL isn't supported in
NIOServerCnxn

at

org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:644)

at

org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:155)

at

org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123)

at

org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)





I have set the following properties in SERVER_JVMFLAGS in zkEnv.sh file 
 :

"-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory

-Dzookeeper.ssl.keyStore.location=/var/opt/vs/SecureInterface/keystore/CassSpkkeystore.p12
-Dzookeeper.ssl.keyStore.password=EvaiKiO1@123456

-Dzookeeper.ssl.trustStore.location=/var/opt/vs/SecureInterface/keystore/CassSpkTrustStore.jks
-Dzookeeper.ssl.trustStore.password=EvaiKiO1@123456"

Thanks,
-
Kuldeep Singh Budania



On Mon, Jun 22, 2020 at 8:08 PM Jordan Zimmerman 

wrote:

> It's the same as the normal ZooKeeper client:
> 
https://zookeeper.apache.org/doc/r3.6.1/zookeeperAdmin.html#sc_authOptions
> <
> 
https://zookeeper.apache.org/doc/r3.6.1/zookeeperAdmin.html#sc_authOptions
> >
>
> -Jordan
>
> > On Jun 22, 2020, at 5:50 AM, kuldeep singh 

> wrote:
> >
> > Hi Team,
> > How we will do secure communication between the Curator framework 
and
> > zookeeper 3.5.6 ?
> > I didn't get any solution right now.
> > I appreciate it if someone could help me with the same.
> >
> > Thanks,
> > -
> > Kuldeep Singh Budania
> > Software Architect
> >
> >
> > On Fri, Apr 17, 2020 at 4:53 PM Szalay-Bekő Máté <
> szalay.beko.m...@gmail.co

Re: upgrade from 3.4.5 to 3.5.6

[Enrico Olivelli - Diennea]

w result.
> >>>>>
> >>>>>
> >>>>>   1. On 1st node upgrade zookeeper . Traffic was running fine
> >> because
> >>> 2
> >>>>>   nodes are already on old zookeeper.
> >>>>>   2. On 1st node upgrade our application and didn’t find any issue
> >>>>>   3. On 2nd node upgrade zookeeper but got below error and zookeeper
> >>> is
> >>>>>   not taking any requests
> >>>>>   4.
> >>>>>
> >>>>> java.io.EOFException
> >>>>>
> >>>>>at java.io.DataInputStream.readInt(DataInputStream.java:392)
> >>>>>
> >>>>>at
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> 
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:747)
> >>>>>
> >>>>> 2020-03-30 14:19:55,587 - WARN
> >>>>> [RecvWorker:1:QuorumCnxManager$RecvWorker@765] - Interrupting
> >>> SendWorker
> >>>>>
> >>>>> 2020-03-30 14:19:55,588 - ERROR [LearnerHandler-/192.168.44.73:33754
> >>>>> :LearnerHandler@562] - Unexpected exception causing shutdown while
> >>> sock
> >>>>> still open
> >>>>>
> >>>>> java.io.EOFException
> >>>>>
> >>>>>at java.io.DataInputStream.readInt(DataInputStream.java:392)
> >>>>>
> >>>>>at
> >>>>>
> >> org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> >>>>>
> >>>>>at
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> 
org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
> >>>>>
> >>>>>at
> >>>>>
> >>>>
> >>>
> >>
> org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
> >>>>>
> >>>>>at
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> 
org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:476)
> >>>>>
> >>>>> 2020-03-30 14:19:55,588 - WARN
> >>>>> [SendWorker:1:QuorumCnxManager$SendWorker@679] - Interrupted while
> >>>> waiting
> >>>>> for message on queue
> >>>>>
> >>>>> Please let me know is this the known issue or this is different 
issue
> >>>> which
> >>>>> is mention in Apache zookeeper documentation when upgrading from
> >> 3.4.5
> >>> to
> >>>>> 3.5.6
> >>>>>
> >>>>> Thanks,
> >>>>> -
> >>>>> Kuldeep Singh Budania
> >>>>> Software Architect
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Sun, Mar 29, 2020 at 9:06 AM Alexander Shraer 
> >>>>> wrote:
> >>>>>
> >>>>>> +1 to what Mate said (I wrote the quoted instructions).
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Mar 24, 2020 at 7:03 AM Szalay-Bekő Máté <
> >>>>>> szalay.beko.m...@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hi Kuldeep,
> >>>>>>>
> >>>>>>> I just want to provide you some background info about our
> >>>>> documentation.
> >>>>>>> The reason to upgrade to 3.4.6 first is to avoid the following
> >>> error:
> >>>>>>>
> >>>>>>>> 2013-01-30 11:32:10,663 [myid:2] - WARN [localhost/
> >>> 127.0.0.1:2784
> >>>>>>> :QuorumCnxManager@349] - Invalid server id: -65536
> >>>>>>>
> >>&

Re: How to deliberately cause a split brain?

[Enrico Olivelli]

Il giorno ven 19 giu 2020 alle ore 13:40 Tim Ward <
tim.w...@featurespace.co.uk> ha scritto:

> From: Michael Han 
> > There are different cases for split brain and how to test the monitor
> code depends on what signals
> > you are using - but most usually, a split brain case can be created by
> artificially split two quorum
> > out of a single quorum through manual configuration change (e.g. a 7
> quorum servers can be split
> > into a 4 quorum with a 3 quorum - just exclude the servers from each
> config).
>
> Yes, I am coming to the conclusion that one could in effect
>
> (a) build two separate Zookeeper ensembles
> (b) point parts of one's application at one ensemble and the rest at the
> other
>
> and expect the application to misbehave somewhat horribly.
>

I don't know your application, but the application will behave weirdly from
the beginning.
If you use "locks", than they won't work, if you run leader election...you
will have two leaders from the beginning

Usually applications have some other out-of-bound (in respect to ZK)
coordination mechanisms that prevents misbehaviour
like using Compare-And-Set operations and versioning on Database records
for critical state
so hopefully you will detect some problem


>
> And I've come up with some stuff I can monitor to detect all that in
> various ways.
>

Awesome, do you mean a "generic" tool ?


Enrico


>
> *BUT* this detection process would reply on the person who had carefully
> and cunningly maliciously mis-configured Zookeeper at the same time
> *correctly* configuring the monitoring system. How likely is that, I
> wonder? So I'm discussing with my customer how much use this feature is
> really likely to be in the real world ...
>
> Thanks all for the various comments.
>
> Tim Ward
> This message, and any files/attachments transmitted together with it, is
> intended for the use only of the person (or persons) to whom it is
> addressed. It may contain information which is confidential and/or
> protected by legal privilege. Accordingly, any dissemination, distribution,
> copying or use of this message, or any part of it or anything sent together
> with it, other than by intended recipients, may constitute a breach of
> civil or criminal law and is hereby prohibited. Unless otherwise stated,
> any views expressed in this message are those of the person sending it and
> not the sender's employer. No responsibility, legal or otherwise, of
> whatever nature, is accepted as to the accuracy of the contents of this
> message or for the completeness of the message as received. Anyone who is
> not the intended recipient of this message is advised to make no use of it
> and is requested to contact Featurespace Limited as soon as possible. Any
> recipient of this message who has knowledge or suspects that it may have
> been the subject of unauthorised interception or alteration is also
> requested to contact Featurespace Limited.
>

Re: How to deliberately cause a split brain?

[Enrico Olivelli]

I told about using iptables only in order to create a network partition.

If you want a simple split brain just switch every server in the cluster to
standalone mode (comment server.xx lines)
I saw it happening in production, it was so nasty :-(

Enrico

Il giorno gio 18 giu 2020 alle ore 13:50 Jordan Zimmerman <
jor...@jordanzimmerman.com> ha scritto:

> Way back in 2013 I wrote about a scenario whereby ZooKeeper will have two
> leader nodes. I don’t know if it still applies but maybe it will be helpful
> to you:
> http://zookeeper-user.578899.n2.nabble.com/Rolling-config-change-considered-harmful-td7578761.html
>
> 
> Jordan Zimmerman
>
> > On Jun 18, 2020, at 4:07 AM, Tim Ward 
> wrote:
> >
> > From: Szalay-Bekő Máté 
> >> Honestly, I can not really think of any way to get split-brain. If
> someone can, then let us know so that we can fix it ;)
> >
> > This issue arose because, apparently, we did get a split brain once upon
> a time. The hints I've got however are that this happened in a pre-3.5
> version of Zookeeper, involving a (botched?) rescaling from 3 to 7 nodes.
> From which hints I'm guessing that something got fixed in 3.5 so that this
> approach to deliberately provoking a split brain would not work with the
> 3.5.7 we're now using.
> >
> > Tim Ward
> >
> > This message, and any files/attachments transmitted together with it, is
> intended for the use only of the person (or persons) to whom it is
> addressed. It may contain information which is confidential and/or
> protected by legal privilege. Accordingly, any dissemination, distribution,
> copying or use of this message, or any part of it or anything sent together
> with it, other than by intended recipients, may constitute a breach of
> civil or criminal law and is hereby prohibited. Unless otherwise stated,
> any views expressed in this message are those of the person sending it and
> not the sender's employer. No responsibility, legal or otherwise, of
> whatever nature, is accepted as to the accuracy of the contents of this
> message or for the completeness of the message as received. Anyone who is
> not the intended recipient of this message is advised to make no use of it
> and is requested to contact Featurespace Limited as soon as possible. Any
> recipient of this message who has knowledge or suspects that it may have
> been the subject of unauthorised interception or alteration is also
> requested to contact Featurespace Limited.
>

Re: How to deliberately cause a split brain?

[Enrico Olivelli]

Tim
You can just use iptables and simulate that some network paths are not
working properly

Enrico

Il Gio 18 Giu 2020, 10:26 Tim Ward  ha scritto:

> I have been tasked with writing some monitoring code to detect a Zookeeper
> split brain using the monitoring system.
>
>
>
> All well and good, I think I can see how to do that, but what about
> testing? How can I deliberately provoke a Zookeeper ensemble into going
> into a split brain state so that I can test the detection code?
>
>
>
> Scenarios might be along the lines of: the ops people want to increase an
> ensemble size, but get something wrong in the (manual) writing of the
> configuration files or in the (manual) restarting of instances. What
> "something" is likely to work here?
>
>
>
>
>
> *Tim Ward*
>
>
>
>
>
> Principal engineer
>
>
>
>
>
> —
>
>
>
>
>
> *t: *+44 (0) 1223 345 940
>
>
>
>
>
>
>
>
>
> *Broers Building, Hauser Forum*
>
> *21 JJ Thomson Avenue, Cambridge*
>
> *CB3 0FA United Kingdom*
>
>
>
>
>
> featurespace.com * | *Twitter
> * | *LinkedIn
> 
>
>
>
>
>
>
>
>
>
> [image: A screenshot of a cell phone Description automatically generated]
>
>
>
>
> This message, and any files/attachments transmitted together with it, is
> intended for the use only of the person (or persons) to whom it is
> addressed. It may contain information which is confidential and/or
> protected by legal privilege. Accordingly, any dissemination, distribution,
> copying or use of this message, or any part of it or anything sent together
> with it, other than by intended recipients, may constitute a breach of
> civil or criminal law and is hereby prohibited. Unless otherwise stated,
> any views expressed in this message are those of the person sending it and
> not the sender's employer. No responsibility, legal or otherwise, of
> whatever nature, is accepted as to the accuracy of the contents of this
> message or for the completeness of the message as received. Anyone who is
> not the intended recipient of this message is advised to make no use of it
> and is requested to contact Featurespace Limited as soon as possible. Any
> recipient of this message who has knowledge or suspects that it may have
> been the subject of unauthorised interception or alteration is also
> requested to contact Featurespace Limited.
>

Re: [ANNOUNCE] Apache Curator 5.0.0 released

[Enrico Olivelli - Diennea]

Congrats
Enrico

Il giorno 29/05/20, 09:45 "tison"  ha scritto:

Congrats!

Best,
tison.


Szalay-Bekő Máté  于2020年5月29日周五 下午3:14写道：

> congratulations for the Curator community, seems to be a nice release! :)
>
> On Fri, May 29, 2020 at 1:48 AM Cameron McKenzie 
> wrote:
>
> > Hello,
> >
> > The Apache Curator team is pleased to announce the  release of version
> > 5.0.0. Apache  Curator is a Java/JVM client library for Apache
> > ZooKeeper[1], a distributed coordination service. Apache Curator
> includes a
> > high-level API framework and utilities to make using Apache ZooKeeper
> much
> > easier and more reliable. It also includes recipes for common use cases
> and
> >  extensions such as service discovery and a Java 8 asynchronous DSL. For
> > more details, please visit the project website:
> http://curator.apache.org/
> >
> > The download page for Apache Curator is here:
> > https://cwiki.apache.org/confluence/display/CURATOR/Releases
> >
> > The binary artifacts for Curator are available from Maven Central and 
its
> > mirrors.
> >
> > For general information on Apache Curator, please visit the project
> > website:
> > http://curator.apache.org
> >
> > Release Notes - Apache Curator - Version 5.0.0
> >
> > ** Bug
> > * [CURATOR-440] - curator-framework is unable to load in OSGi
> > * [CURATOR-464] - Unable to instantiate client in OSGi
> > * [CURATOR-525] - There is a race condition in Curator which might
> lead
> > to fake SUSPENDED event and ruin CuratorFrameworkImpl inner state
> > * [CURATOR-559] - Inconsistent ZK timeouts
> >
> > ** New Feature
> > * [CURATOR-544] - Implement SessionFailedRetryPolicy
> >
> > ** Improvement
> > * [CURATOR-549] - ZooKeeper 3.6 will add support for Persistent
> > Recursive Watchers - Add Curator support
> > * [CURATOR-558] - ZooKeeper 3.6.0 has many API changes - bring
> Curator
> > up to date
> > * [CURATOR-562] - Remove ConnectionHandlingPolicy
> > * [CURATOR-564] - Changes to retry failed TestingServer starts 
should
> > be applied to TestingCluster
> > * [CURATOR-568] - New option allowing CuratorFramework skip ZK
> ensemble
> > tracking
> >
> > Regards,
> >
> > The Curator Team
> >
> > [1] Apache ZooKeeper https://zookeeper.apache.org/
> >
>




CONFIDENTIALITY & PRIVACY NOTICE
This e-mail (including any attachments) is strictly confidential and may also 
contain privileged information. If you are not the intended recipient you are 
not authorised to read, print, save, process or disclose this message. If you 
have received this message by mistake, please inform the sender immediately and 
destroy this e-mail, its attachments and any copies. Any use, distribution, 
reproduction or disclosure by any person other than the intended recipient is 
strictly prohibited and the person responsible may incur in penalties.
The use of this e-mail is only for professional purposes; there is no guarantee 
that the correspondence towards this e-mail will be read only by the recipient, 
because, under certain circumstances, there may be a need to access this email 
by third subjects belonging to the Company.

Re: Curator 5.0 binary compatibility

[Enrico Olivelli]

Il Mer 27 Mag 2020, 17:38 Jordan Zimmerman  ha
scritto:

> BookKeeper is not using the two recipes then, NodeCache and
> PathChildrenCache. I think we can safely release and document the
> workaround for those that need it.
>

Just to be clear.

I have cited Bookkeeper but my tests where mostly about a few other non OS
projects in my company.

Bookkeeper uses Curator only on the server side, and actually the server
code usually is rebuilt at every release.

The problems I described can arise when Curator is used on clients, like
for Service Discovery or metadata management.


Enrico


> -Jordan
>
> On May 27, 2020, at 9:51 AM, Enrico Olivelli  wrote:
>
> Jordan and Cameron
> I have double checked most of the projects I am aware and finally I came to
> the conclusion that upgrading to 5.0 in all of my cases is safe.
>
> I only add a nuisance on Apache BookKeeper because we are building with
> -Werror and so using deprecated APIs is seen as an error, but it is not a
> binary compatibility issue.
>
> I have also run locally tests of the staged sources.
> I am casting my +1
>
> I think that this discussion as been useful for the community
>
> Thank you
> Enrico
>
> Il giorno mer 27 mag 2020 alle ore 00:55 Jordan Zimmerman <
> jor...@jordanzimmerman.com> ha scritto:
>
> If that comes to pass I'll write a new Tech Note on the wiki on how to
> create a Compatibility JAR.
>
> -JZ
>
> On May 26, 2020, at 5:41 PM, Cameron McKenzie 
>
> wrote:
>
>
> I will release on Sunday if we haven't got any feedback indicating we
> should do otherwise.
>
> On Wed, May 27, 2020 at 6:32 AM Enrico Olivelli 
> <mailto:eolive...@gmail.com >> wrote:
>
>
>
>
> Il giorno mar 26 mag 2020 alle ore 21:53 Jordan Zimmerman <
> jor...@jordanzimmerman.com> ha scritto:
>
> The initial email is almost a week old. I posted on Curator's Twitter
> account too. No response from anyone other than me Cameron and Enrico.
>
> Very
>
> discouraging. Let's give it to the end of the week. If no one responds
>
> I
>
> suggest we release and tell people how to work around it if they need
>
> to.
>
>
>
> Agreed
>
> Enrico
>
>
>
>
> -JZ
>
> On May 25, 2020, at 9:10 AM, Jordan Zimmerman <
>
> jor...@jordanzimmerman.com>
>
> wrote:
>
> If most of the problem is about ListenerContainer, don't we have a way
>
> to
>
> keep it and emulate it using and implementation based on the new API ?
>
>
> Unfortunately not. The issue is that it leaks a Guava class (Function)
>
> in
>
> its API. See here:
>
>
> https://github.com/apache/curator/blob/apache-curator-4.1.1/curator-framework/src/main/java/org/apache/curator/framework/listen/ListenerContainer.java#L87
>
>
> -JZ
>
> On May 25, 2020, at 1:33 AM, Enrico Olivelli 
>
> wrote:
>
>
> Il giorno dom 24 mag 2020 alle ore 23:17 Jordan Zimmerman <
> jor...@jordanzimmerman.com> ha scritto:
>
> Enrico,
>
> It reminds me of the breaking changes in Guava and other widely used
> libraries.
>
>
>
> In fact Guava is terrible for people (like in my company) that deal
>
> with
>
> lots of third party dependencies.
>
>
> The problem for us is that we can never change our APIs if this is the
> case. Note that ListenerContainer has been marked deprecated since
>
> 4.1.1 (
>
>
>
>
> https://github.com/apache/curator/blob/apache-curator-4.1.1/curator-framework/src/main/java/org/apache/curator/framework/listen/ListenerContainer.java
>
> <
>
>
>
> https://github.com/apache/curator/blob/apache-curator-4.1.1/curator-framework/src/main/java/org/apache/curator/framework/listen/ListenerContainer.java
>
>
> ).
>
>
>
> I didn't check the code yet, I am sorry, so maybe I saying something
>
> that
>
> is not doable.
> If most of the problem is about ListenerContainer, don't we have a way
>
> to
>
> keep it and emulate it using and implementation based on the new API ?
>
> as Jordan said, any other comment from the community will be very
> appreciated, maybe we are talking about smoke.
>
> Enrico
>
>
>
> So, we're really left with these options:
>
> Release Curator 5.0 and let the issues fall onto those with
>
> compatibility
>
> problems
> Bundle or refer to a compatibility JAR that is put early in the
>
> CLASSPATH
>
> as I outlined in my test project
> Move Curator 5.0 to a new package so that it can exist in the same JVM
>
> as
>
> earlier versions of Curator.
> Backout the change and mark the APIs as deprecated and push the
>
> problem to
>
> a future version
>
>

Re: Curator 5.0 binary compatibility

[Enrico Olivelli]

Jordan and Cameron
I have double checked most of the projects I am aware and finally I came to
the conclusion that upgrading to 5.0 in all of my cases is safe.

I only add a nuisance on Apache BookKeeper because we are building with
-Werror and so using deprecated APIs is seen as an error, but it is not a
binary compatibility issue.

I have also run locally tests of the staged sources.
I am casting my +1

I think that this discussion as been useful for the community

Thank you
Enrico

Il giorno mer 27 mag 2020 alle ore 00:55 Jordan Zimmerman <
jor...@jordanzimmerman.com> ha scritto:

> If that comes to pass I'll write a new Tech Note on the wiki on how to
> create a Compatibility JAR.
>
> -JZ
>
> > On May 26, 2020, at 5:41 PM, Cameron McKenzie 
> wrote:
> >
> > I will release on Sunday if we haven't got any feedback indicating we
> > should do otherwise.
> >
> > On Wed, May 27, 2020 at 6:32 AM Enrico Olivelli  <mailto:eolive...@gmail.com>> wrote:
> >
> >>
> >>
> >> Il giorno mar 26 mag 2020 alle ore 21:53 Jordan Zimmerman <
> >> jor...@jordanzimmerman.com> ha scritto:
> >>
> >>> The initial email is almost a week old. I posted on Curator's Twitter
> >>> account too. No response from anyone other than me Cameron and Enrico.
> Very
> >>> discouraging. Let's give it to the end of the week. If no one responds
> I
> >>> suggest we release and tell people how to work around it if they need
> to.
> >>>
> >>
> >> Agreed
> >>
> >> Enrico
> >>
> >>
> >>
> >>>
> >>> -JZ
> >>>
> >>> On May 25, 2020, at 9:10 AM, Jordan Zimmerman <
> jor...@jordanzimmerman.com>
> >>> wrote:
> >>>
> >>> If most of the problem is about ListenerContainer, don't we have a way
> to
> >>> keep it and emulate it using and implementation based on the new API ?
> >>>
> >>>
> >>> Unfortunately not. The issue is that it leaks a Guava class (Function)
> in
> >>> its API. See here:
> >>>
> https://github.com/apache/curator/blob/apache-curator-4.1.1/curator-framework/src/main/java/org/apache/curator/framework/listen/ListenerContainer.java#L87
> >>>
> >>> -JZ
> >>>
> >>> On May 25, 2020, at 1:33 AM, Enrico Olivelli 
> wrote:
> >>>
> >>> Il giorno dom 24 mag 2020 alle ore 23:17 Jordan Zimmerman <
> >>> jor...@jordanzimmerman.com> ha scritto:
> >>>
> >>> Enrico,
> >>>
> >>> It reminds me of the breaking changes in Guava and other widely used
> >>> libraries.
> >>>
> >>>
> >>>
> >>> In fact Guava is terrible for people (like in my company) that deal
> with
> >>> lots of third party dependencies.
> >>>
> >>>
> >>> The problem for us is that we can never change our APIs if this is the
> >>> case. Note that ListenerContainer has been marked deprecated since
> 4.1.1 (
> >>>
> >>>
> https://github.com/apache/curator/blob/apache-curator-4.1.1/curator-framework/src/main/java/org/apache/curator/framework/listen/ListenerContainer.java
> >>> <
> >>>
> >>>
> https://github.com/apache/curator/blob/apache-curator-4.1.1/curator-framework/src/main/java/org/apache/curator/framework/listen/ListenerContainer.java
> >>>
> >>> ).
> >>>
> >>>
> >>>
> >>> I didn't check the code yet, I am sorry, so maybe I saying something
> that
> >>> is not doable.
> >>> If most of the problem is about ListenerContainer, don't we have a way
> to
> >>> keep it and emulate it using and implementation based on the new API ?
> >>>
> >>> as Jordan said, any other comment from the community will be very
> >>> appreciated, maybe we are talking about smoke.
> >>>
> >>> Enrico
> >>>
> >>>
> >>>
> >>> So, we're really left with these options:
> >>>
> >>> Release Curator 5.0 and let the issues fall onto those with
> compatibility
> >>> problems
> >>> Bundle or refer to a compatibility JAR that is put early in the
> CLASSPATH
> >>> as I outlined in my test project
> >>> Move Curator 5.0 to a new package so that it can exist in the same JVM
> as
> >>> earlier versions of Curator.
> >>> Backout the change and mark the APIs as 

Re: Curator 5.0 binary compatibility

[Enrico Olivelli]

Il giorno mar 26 mag 2020 alle ore 21:53 Jordan Zimmerman <
jor...@jordanzimmerman.com> ha scritto:

> The initial email is almost a week old. I posted on Curator's Twitter
> account too. No response from anyone other than me Cameron and Enrico. Very
> discouraging. Let's give it to the end of the week. If no one responds I
> suggest we release and tell people how to work around it if they need to.
>

Agreed

Enrico



>
> -JZ
>
> On May 25, 2020, at 9:10 AM, Jordan Zimmerman 
> wrote:
>
> If most of the problem is about ListenerContainer, don't we have a way to
> keep it and emulate it using and implementation based on the new API ?
>
>
> Unfortunately not. The issue is that it leaks a Guava class (Function) in
> its API. See here:
> https://github.com/apache/curator/blob/apache-curator-4.1.1/curator-framework/src/main/java/org/apache/curator/framework/listen/ListenerContainer.java#L87
>
> -JZ
>
> On May 25, 2020, at 1:33 AM, Enrico Olivelli  wrote:
>
> Il giorno dom 24 mag 2020 alle ore 23:17 Jordan Zimmerman <
> jor...@jordanzimmerman.com> ha scritto:
>
> Enrico,
>
> It reminds me of the breaking changes in Guava and other widely used
> libraries.
>
>
>
> In fact Guava is terrible for people (like in my company) that deal with
> lots of third party dependencies.
>
>
> The problem for us is that we can never change our APIs if this is the
> case. Note that ListenerContainer has been marked deprecated since 4.1.1 (
>
> https://github.com/apache/curator/blob/apache-curator-4.1.1/curator-framework/src/main/java/org/apache/curator/framework/listen/ListenerContainer.java
> <
>
> https://github.com/apache/curator/blob/apache-curator-4.1.1/curator-framework/src/main/java/org/apache/curator/framework/listen/ListenerContainer.java
>
> ).
>
>
>
> I didn't check the code yet, I am sorry, so maybe I saying something that
> is not doable.
> If most of the problem is about ListenerContainer, don't we have a way to
> keep it and emulate it using and implementation based on the new API ?
>
> as Jordan said, any other comment from the community will be very
> appreciated, maybe we are talking about smoke.
>
> Enrico
>
>
>
> So, we're really left with these options:
>
> Release Curator 5.0 and let the issues fall onto those with compatibility
> problems
> Bundle or refer to a compatibility JAR that is put early in the CLASSPATH
> as I outlined in my test project
> Move Curator 5.0 to a new package so that it can exist in the same JVM as
> earlier versions of Curator.
> Backout the change and mark the APIs as deprecated and push the problem to
> a future version
>
> -Jordan
>
> On May 24, 2020, at 3:58 PM, Enrico Olivelli 
>
> wrote:
>
>
>
>
> Il Dom 24 Mag 2020, 22:48 Cameron McKenzie 
> <mailto:cammcken...@apache.org >> ha scritto:
>
> Enrico,
> Can you explain your environment that exposes these backwards
> compatibility issues?
>
> Cameron,
> Let's say we have two libraries Foo and Bar that are compiled for
>
> Curator 4.x.
>
>
> I am now using in my Application Baz that use both Foo and Bar. So I
>
> have Curator 4.x on the classpath.
>
> Developers of Foo want to move to Curator 5.x in Foo 2.0, but Bar is
>
> still happy with Curator 4.x.
>
>
> If I want to upgrade Foo to 2.0 I have these chances:
> 1) Curator 5 is compatible with 4.x,so I can simply keep 5 and
>
> everything works
>
> 2) Curator 5 is not compatible with 4.x so I can't have both (this is
>
> current case)
>
> 3) Curator 5 is independent from 4.x and I can keep both of them
>
> The best option for users is 1).
>
> 3) is good anyway, but it needs more work for users that want to migrate.
>
> Option 2) is not good. Users will have to shade/relocate Curator 5 or 4
>
> and Foo 2.0 or Bar.
>
>
> Hope that this explains better the problem
> Enrico
>
>
> I am probably coming from a place of ignorance, but I
> haven't seen new versions of a third party binary being dropped into an
> existing environment without recompiling the application, so I have never
> encountered these binary compatibility issues before. My expectation with
> this release was that if you wanted to pickup the changes in Curator 5.0
> that you would rebuild your application against the new binaries and then
> redeploy the application. Obviously this compilation will break if you
>
> are
>
> using any of the changed APIs, but they are pretty trivial change to fix.
> We could potentially deprecate the existing APIs and add the new ones,
>
> but
>
> this will produce more tech debt to clean up later.
> cheers
>
> On Sat, May 23, 202

Re: Curator 5.0 binary compatibility

[Enrico Olivelli]

Il giorno dom 24 mag 2020 alle ore 23:17 Jordan Zimmerman <
jor...@jordanzimmerman.com> ha scritto:

> Enrico,
>
> It reminds me of the breaking changes in Guava and other widely used
> libraries.


In fact Guava is terrible for people (like in my company) that deal with
lots of third party dependencies.


> The problem for us is that we can never change our APIs if this is the
> case. Note that ListenerContainer has been marked deprecated since 4.1.1 (
> https://github.com/apache/curator/blob/apache-curator-4.1.1/curator-framework/src/main/java/org/apache/curator/framework/listen/ListenerContainer.java
> <
> https://github.com/apache/curator/blob/apache-curator-4.1.1/curator-framework/src/main/java/org/apache/curator/framework/listen/ListenerContainer.java
> >).
>

I didn't check the code yet, I am sorry, so maybe I saying something that
is not doable.
If most of the problem is about ListenerContainer, don't we have a way to
keep it and emulate it using and implementation based on the new API ?

as Jordan said, any other comment from the community will be very
appreciated, maybe we are talking about smoke.

Enrico


>
> So, we're really left with these options:
>
> Release Curator 5.0 and let the issues fall onto those with compatibility
> problems
> Bundle or refer to a compatibility JAR that is put early in the CLASSPATH
> as I outlined in my test project
> Move Curator 5.0 to a new package so that it can exist in the same JVM as
> earlier versions of Curator.
> Backout the change and mark the APIs as deprecated and push the problem to
> a future version
>
> -Jordan
>
> > On May 24, 2020, at 3:58 PM, Enrico Olivelli 
> wrote:
> >
> >
> >
> > Il Dom 24 Mag 2020, 22:48 Cameron McKenzie  <mailto:cammcken...@apache.org>> ha scritto:
> > Enrico,
> > Can you explain your environment that exposes these backwards
> > compatibility issues?
> >
> > Cameron,
> > Let's say we have two libraries Foo and Bar that are compiled for
> Curator 4.x.
> >
> > I am now using in my Application Baz that use both Foo and Bar. So I
> have Curator 4.x on the classpath.
> > Developers of Foo want to move to Curator 5.x in Foo 2.0, but Bar is
> still happy with Curator 4.x.
> >
> > If I want to upgrade Foo to 2.0 I have these chances:
> > 1) Curator 5 is compatible with 4.x,so I can simply keep 5 and
> everything works
> > 2) Curator 5 is not compatible with 4.x so I can't have both (this is
> current case)
> > 3) Curator 5 is independent from 4.x and I can keep both of them
> >
> > The best option for users is 1).
> >
> > 3) is good anyway, but it needs more work for users that want to migrate.
> >
> > Option 2) is not good. Users will have to shade/relocate Curator 5 or 4
> and Foo 2.0 or Bar.
> >
> > Hope that this explains better the problem
> > Enrico
> >
> >
> > I am probably coming from a place of ignorance, but I
> > haven't seen new versions of a third party binary being dropped into an
> > existing environment without recompiling the application, so I have never
> > encountered these binary compatibility issues before. My expectation with
> > this release was that if you wanted to pickup the changes in Curator 5.0
> > that you would rebuild your application against the new binaries and then
> > redeploy the application. Obviously this compilation will break if you
> are
> > using any of the changed APIs, but they are pretty trivial change to fix.
> > We could potentially deprecate the existing APIs and add the new ones,
> but
> > this will produce more tech debt to clean up later.
> > cheers
> >
> > On Sat, May 23, 2020 at 7:40 PM Enrico Olivelli  <mailto:eolive...@gmail.com>> wrote:
> >
> > > I will check you trick ad soon as possible. I am sorry, this is a very
> > > busy week for me and do not have enough cycles. But I think that we
> should
> > > address this problem in order to ease the adoption of the new code and
> APIs.
> > >
> > > Did you evaluate to eventually rollback the breaking changes?
> > >
> > > Another alternative, if we want to let users use both the old and the
> new
> > > APIs is to simply rename all of the packages and start a brand new
> system.
> > > This approach was done in Apache Commons and IIRC it will be done with
> > > Netty5. We also did it with the new Apache Bookkeeper API.
> > >
> > > Pros:
> > > No need to preserve compatibility, we are free to clean up all of the
> tech
> > > debt.
> > > The switch to Curator 5 will be explicit opted in
> > &g

Re: Curator 5.0 binary compatibility

[Enrico Olivelli]

Il Dom 24 Mag 2020, 22:48 Cameron McKenzie  ha
scritto:

> Enrico,
> Can you explain your environment that exposes these backwards
> compatibility issues?


Cameron,
Let's say we have two libraries Foo and Bar that are compiled for Curator
4.x.

I am now using in my Application Baz that use both Foo and Bar. So I have
Curator 4.x on the classpath.
Developers of Foo want to move to Curator 5.x in Foo 2.0, but Bar is still
happy with Curator 4.x.

If I want to upgrade Foo to 2.0 I have these chances:
1) Curator 5 is compatible with 4.x,so I can simply keep 5 and everything
works
2) Curator 5 is not compatible with 4.x so I can't have both (this is
current case)
3) Curator 5 is independent from 4.x and I can keep both of them

The best option for users is 1).

3) is good anyway, but it needs more work for users that want to migrate.

Option 2) is not good. Users will have to shade/relocate Curator 5 or 4 and
Foo 2.0 or Bar.

Hope that this explains better the problem
Enrico


I am probably coming from a place of ignorance, but I
> haven't seen new versions of a third party binary being dropped into an
> existing environment without recompiling the application, so I have never
> encountered these binary compatibility issues before. My expectation with
> this release was that if you wanted to pickup the changes in Curator 5.0
> that you would rebuild your application against the new binaries and then
> redeploy the application. Obviously this compilation will break if you are
> using any of the changed APIs, but they are pretty trivial change to fix.
> We could potentially deprecate the existing APIs and add the new ones, but
> this will produce more tech debt to clean up later.
> cheers
>
> On Sat, May 23, 2020 at 7:40 PM Enrico Olivelli 
> wrote:
>
> > I will check you trick ad soon as possible. I am sorry, this is a very
> > busy week for me and do not have enough cycles. But I think that we
> should
> > address this problem in order to ease the adoption of the new code and
> APIs.
> >
> > Did you evaluate to eventually rollback the breaking changes?
> >
> > Another alternative, if we want to let users use both the old and the new
> > APIs is to simply rename all of the packages and start a brand new
> system.
> > This approach was done in Apache Commons and IIRC it will be done with
> > Netty5. We also did it with the new Apache Bookkeeper API.
> >
> > Pros:
> > No need to preserve compatibility, we are free to clean up all of the
> tech
> > debt.
> > The switch to Curator 5 will be explicit opted in
> >
> > Cons:
> > Cherry picks won't be straightforward.
> >
> > Enrico
> >
> > Il Ven 22 Mag 2020, 23:40 Jordan Zimmerman 
> > ha scritto:
> >
> >> Hi Everyone,
> >>
> >> I've coded a possible solution in the test project. See here:
> >>
> >>
> https://github.com/Randgalt/curator_5_0_test/blob/master/combo/pom.xml#L49
> >>
> >> It uses the Maven dependency plugin to create a small compatibility JAR
> >> that contains the Curator 4.3.0 versions of the classes that have
> changed
> >> in 5.0.0 (i.e. the ones that no longer return ListenerContainer). If
> this
> >> JAR is included in a CLASSPATH before Curator 5.0.0's JARs, these old
> >> classes will take precedence and thus old binaries will continue to
> work.
> >> The curator_5_0_test shows this. run.sh is the previous way with the
> >> error. run-compatibility.sh is with the compatibility JAR.
> >>
> >> Thoughts? Notable, this doesn't change the master code of Curator at
> all.
> >> We could add it to the 5.0 release. I don't think there's an issue with
> >> this "hack". Can anyone think of one? I'd really appreciate people
> testing
> >> with it. Try a build with just Curator 5.0 and then install and include
> >> this curator-5_0-test:combo:1.0-SNAPSHOT early in the CLASSPATH - it
> should
> >> work.
> >>
> >> -Jordan
> >>
> >> On May 21, 2020, at 10:43 AM, Jordan Zimmerman <
> >> jor...@jordanzimmerman.com> wrote:
> >>
> >> Hello All,
> >>
> >> Sorry for the cross-posting but this is important enough to justify it.
> >>
> >> Apache Curator is in the process of releasing version 5.0. We've taken
> >> the opportunity to address some long standing tech debt but this causes
> >> breaking changes. We've detailed the breaks here:
> >> http://curator.apache.org/staging/breaking-changes.html. The Clirr
> >> report shows the exact API changes:
> >> http://curator.apache.org/staging/curator-recipes/cl

Re: Curator 5.0 binary compatibility

[Enrico Olivelli]

I will check you trick ad soon as possible. I am sorry, this is a very busy
week for me and do not have enough cycles. But I think that we should
address this problem in order to ease the adoption of the new code and APIs.

Did you evaluate to eventually rollback the breaking changes?

Another alternative, if we want to let users use both the old and the new
APIs is to simply rename all of the packages and start a brand new system.
This approach was done in Apache Commons and IIRC it will be done with
Netty5. We also did it with the new Apache Bookkeeper API.

Pros:
No need to preserve compatibility, we are free to clean up all of the tech
debt.
The switch to Curator 5 will be explicit opted in

Cons:
Cherry picks won't be straightforward.

Enrico

Il Ven 22 Mag 2020, 23:40 Jordan Zimmerman  ha
scritto:

> Hi Everyone,
>
> I've coded a possible solution in the test project. See here:
>
> https://github.com/Randgalt/curator_5_0_test/blob/master/combo/pom.xml#L49
>
> It uses the Maven dependency plugin to create a small compatibility JAR
> that contains the Curator 4.3.0 versions of the classes that have changed
> in 5.0.0 (i.e. the ones that no longer return ListenerContainer). If this
> JAR is included in a CLASSPATH before Curator 5.0.0's JARs, these old
> classes will take precedence and thus old binaries will continue to work.
> The curator_5_0_test shows this. run.sh is the previous way with the
> error. run-compatibility.sh is with the compatibility JAR.
>
> Thoughts? Notable, this doesn't change the master code of Curator at all.
> We could add it to the 5.0 release. I don't think there's an issue with
> this "hack". Can anyone think of one? I'd really appreciate people testing
> with it. Try a build with just Curator 5.0 and then install and include
> this curator-5_0-test:combo:1.0-SNAPSHOT early in the CLASSPATH - it should
> work.
>
> -Jordan
>
> On May 21, 2020, at 10:43 AM, Jordan Zimmerman 
> wrote:
>
> Hello All,
>
> Sorry for the cross-posting but this is important enough to justify it.
>
> Apache Curator is in the process of releasing version 5.0. We've taken the
> opportunity to address some long standing tech debt but this causes
> breaking changes. We've detailed the breaks here:
> http://curator.apache.org/staging/breaking-changes.html. The Clirr report
> shows the exact API changes:
> http://curator.apache.org/staging/curator-recipes/clirr-report.html. The
> first two of these are the most worrisome. NodeCache's and
> PathChildrenCache's getListenable() methods now have a different return
> type. This has far reaching implications. If a Curator user were to drop in
> Curator 5.0 without any code changes they will get runtime exceptions when
> these methods are called.
>
> I've written a test that shows the problem:
>
> git clone https://github.com/Randgalt/curator_5_0_test.git
> cd curator_5_0_test
> ./run.sh
>
> You will see:
>
> java.lang.NoSuchMethodError:
> org.apache.curator.framework.recipes.cache.PathChildrenCache.getListenable()Lorg/apache/curator/framework/listen/ListenerContainer;
> at binary.Curator50Test.run(Curator50Test.java:26)
> at test.Test.main(Test.java:9)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)
> at java.lang.Thread.run(Thread.java:748)
>
> Enrico Olivelli brought this to our attention. Curator 5.0 is a major
> version bump so breaking changes are implied. But, maybe this is blocker?
> What do people think? If this is a serious enough concern we can come up
> with a workaround.
>
> Please discuss and let's hold off completing the current release until
> this has been fully discussed.
>
> -Jordan
>
>
>

Re: How to EXPORT ZooKeeper data from ZooKeeper v3.5 in Cluster with SSL enabled

[Enrico Olivelli]

Nigam
Usually applications use Zookeeper for coordination, service discovery or
configuration management.
In these case is not useful to port data from dev/qa to/from production.

Or do you need to export data in order to debug a problem?

Btw the only option I know is to write a simple program that visits the
whole tree and dumps it to file.
You won't be able to capture/reconstruct exactly the tree due to the
presence of ephemeral/ttl/container nodes

Maybe others on this list can share tools that I I not aware of.

Hope that helps


Enrico

Il Ven 15 Mag 2020, 20:26 Nigamananda Rout  ha
scritto:

> Hello - I am looking for a programmatic solution so that I can export the
> ZooKeeper (in Cluster with SSL enabled) data to a file.
> I am looking for Export Options, so that I can export PROD Zookeeper data
> to QA/DEV/TEST  ZooKeeper.
> Do we have any option for that. Any suggestions?
>
>
>   *   Using ZooKeeper v3.5
>   *   ZooKeeper in Cluster
>   *   SSL is Enabled
>
> Regards, Nigam...
>
>
>
> This e-mail message and any files transmitted with it may contain
> confidential and proprietary information and are intended solely for the
> use of the individual or entity to which they are addressed. Any
> unauthorized review, use, disclosure or distribution is strictly
> prohibited. If you have received this e-mail in error please notify the
> sender by reply email and destroy all copies of the original message. Thank
> you for your cooperation.
>

Re: [ANNOUNCE] Apache ZooKeeper 3.5.8

[Enrico Olivelli]

Kudos!
Thank you Mate

Enrico

Il Mar 12 Mag 2020, 08:22 Szalay-Bekő Máté  ha
scritto:

> The Apache ZooKeeper team is proud to announce Apache ZooKeeper version
> 3.5.8
>
> ZooKeeper is a high-performance coordination service for distributed
> applications. It exposes common services - such as naming,
> configuration management, synchronization, and group services - in a
> simple interface so you don't have to write them from scratch. You can
> use it off-the-shelf to implement consensus, group management, leader
> election, and presence protocols. And you can build on it for your
> own, specific needs.
>
> For ZooKeeper release details and downloads, visit:
> https://zookeeper.apache.org/releases.html
>
> ZooKeeper 3.5.8 Release Notes are at:
> https://zookeeper.apache.org/doc/r3.5.8/releasenotes.html
> 
>
> We would like to thank the contributors that made the release possible.
>
> Regards,
>
> The ZooKeeper Team
>

Re: ZooKeeper dynamic reconfig issue when Quorum authn/authz is enabled

[Enrico Olivelli]

Il Ven 8 Mag 2020, 17:35 rajsura  ha scritto:

> Hello,
>
> With 'DynamicReconfig' feature in v3.5.7, ideally the servers can be added
> and removed without restarting ZooKeeper service on any of the nodes.
>
> But, with Keberos based quorum authentication/authorization enabled via
> '_HOST' principal check, this is not possible. Because, when you try to add
> a new server, it won't be able to connect to any of the members in the node
> and won't be synced. As all the members reject it based on authorization.
> For it to work, you need to do 'reconfig', then restart leader, the new
> member and rest of the members.
>
> Is this the expected behavior with DynamicReconfig? Or am I missing
> something here.
>

Rajani
It looks like a bug.
Do you have this problem in production or in a test environment?

I am not a user of reconfig, I hope that someone else on this list can give
more help

Enrico



>
>
> --
> Sent from: http://zookeeper-user.578899.n2.nabble.com/
>

Re: ZK not starting during upgrade to use 3.6.1 with SSL communication

[Enrico Olivelli]

Trying to restart from the beginning...

You are upgrading your cluster.
- Before the upgrade were you running SSL client port and SSL Quorum ?
- If you enable only SSL Quorum, does it run ?
- if you enable only client SSL port does it run ?
- is it the problem only when you configure both SSL Quorum and SSL secure
client port ?

Enrico

Il giorno ven 8 mag 2020 alle ore 03:30 blb.dev  ha
scritto:

> Thanks for sharing -
>
> I wasn't using quorumListenOnAllIPs=true, as it's listed as an Unsafe
> Option
> - any reason why you chose to use it?
>
> The bind issue is fixed by setting either clientPort or secureClientPort
> only and only setting it in zoo.cfg or zoo.cfg.dynamic.
>
> However, my quorum still does not start up!  It gives the following stack
> trace:
>
> # bin/zkCli.sh
> /usr/bin/java
> Connecting to localhost:2181
> 2020-05-08 01:25:54,277 [myid:] - INFO  [main:Environment@98] - Client
>
> environment:zookeeper.version=3.6.1--104dcb3e3fb464b30c5186d229e00af9f332524b,
> built on 04/21/2020 15:01 GMT
> 2020-05-08 01:25:54,281 [myid:] - INFO  [main:Environment@98] - Client
> environment:host.name=zoo2
> 2020-05-08 01:25:54,281 [myid:] - INFO  [main:Environment@98] - Client
> environment:java.version=1.8.0_252
> 2020-05-08 01:25:54,284 [myid:] - INFO  [main:Environment@98] - Client
> environment:java.vendor=Oracle Corporation
> 2020-05-08 01:25:54,284 [myid:] - INFO  [main:Environment@98] - Client
>
> environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre
> 2020-05-08 01:25:54,284 [myid:] - INFO  [main:Environment@98] - Client
>
> environment:java.class.path=/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/target/classes:/apache-zookeeper-3.6.1-bin/bin/../build/classes:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/target/lib/*.jar:/apache-zookeeper-3.6.1-bin/bin/../build/lib/*.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-prometheus-metrics-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-jute-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/snappy-java-1.1.7.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/slf4j-log4j12-1.7.25.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/slf4j-api-1.7.25.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_servlet-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_hotspot-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_common-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-native-unix-common-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-native-epoll-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-resolver-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-handler-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-common-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-codec-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-buffer-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/metrics-core-3.2.5.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/log4j-1.2.17.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/json-simple-1.1.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jline-2.11.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-util-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-servlet-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-server-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-security-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-io-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-http-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/javax.servlet-api-3.1.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-databind-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-core-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-annotations-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/commons-lang-2.6.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/commons-cli-1.2.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/audience-annotations-0.5.0.jar:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/src/main/resources/lib/*.jar:/conf:
> 2020-05-08 01:25:54,284 [myid:] - INFO  [main:Environment@98] - Client
>
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> 2020-05-08 01:25:54,285 [myid:] - INFO  [main:Environment@98] - Client
> environment:java.io.tmpdir=/tmp
> 2020-05-08 01:25:54,285 [myid:] - INFO  [main:Environment@98] - Client
> environment:java.compiler=
> 2020-05-08 01:25:54,285 [myid:] - INFO  [main:Environment@98] - Client
> environment:os.name=Linux
> 2020-05-08 01:25:54,285 [myid:] - INFO  [main:Environment@98] - Client
> environment:os.arch=amd64
> 2020-05-08 01:25:54,285 [myid:] - INFO  

Re: ZOOKEEPER-1634: hardening security by teaching server to enforce client authentication

[Enrico Olivelli]

Andor
With ZOOKEEPER-1634 it has been implemented

I think it was a contribution from Twitter

Enrico

Il Mar 5 Mag 2020, 17:51 Andor Molnar  ha scritto:

> Authentication cannot be enforced currently.
> What you can do is to enable SASL on one hand and set up ACLs for znodes
> to be protected on the other hand.
>
> Andor
>
>
>
> > On 2020. May 5., at 17:25, Guilherme Ramos  wrote:
> >
> > Hello.
> > As requested by Enrico Olivelli, I'm trying to use  Zookeeper to 3.6.1
> > using SASL, jaas(with Server defined) and still accepting client
> > connections without credentials.
> >
> > What can be done here? Is there a roadmap to fix this?
> >
> > Thank you.
>
>

Re: ZK not starting during upgrade to use 3.6.1 with SSL communication

[Enrico Olivelli]

from the stacktrace is looks like the error is on Netty and Netty is only
for the communication port to the client,
so the quorum communication should not be affected but this error.

So you have SSL clients and SSL quorum communication ?
With portUnification you r using only one single port that serves both
plain and ssl protocols so you do not need to
start the secureClientPort
if you want only TLS then you can disable port unification and configure
the secureClientPort

I hope that helps

Enrico

Il giorno lun 4 mag 2020 alle ore 20:44 blb.dev  ha
scritto:

> So reading through the section on Upgrading existing non-TLS cluster with
> no
> downtime in the zookeeper admin guide, it says when upgrading from 3.4
> branch I need to start with /sslQuorum=false portUnification=true/.
>
> I tried this and ZK finally came up successfully. However, when I updated
> the config files to /sslQuorum=true portUnification=true/ and did rolling
> restart, then they do not pick up the changes giving the same port bind
> error as noted.
>
> *What do I need to do differently to get a healthy zk when sslQuorum
> communication is enabled? *
>
>
>
> *# cat /logs/zookeeper.out*
> 2020-05-04 18:15:06,565 [myid:] - INFO  [main:QuorumPeerConfig@173] -
> Reading configuration from: /conf/zoo.cfg
> 2020-05-04 18:15:06,584 [myid:] - INFO  [main:QuorumPeerConfig@459] -
> clientPortAddress is 0.0.0.0:2181
> 2020-05-04 18:15:06,584 [myid:] - INFO  [main:QuorumPeerConfig@472] -
> secureClientPortAddress is 0.0.0.0:2281
> 2020-05-04 18:15:06,588 [myid:] - INFO  [main:X509Util@77] - Setting -D
> jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated
> TLS renegotiation
> 2020-05-04 18:15:06,589 [myid:] - INFO  [main:QuorumPeerConfig@479] -
> observerMasterPort is not set
> 2020-05-04 18:15:06,590 [myid:] - INFO  [main:QuorumPeerConfig@496] -
> metricsProvider.className is
> org.apache.zookeeper.metrics.impl.DefaultMetricsProvider
> 2020-05-04 18:15:06,605 [myid:1] - INFO  [main:DatadirCleanupManager@78] -
> autopurge.snapRetainCount set to 10
> 2020-05-04 18:15:06,606 [myid:1] - INFO  [main:DatadirCleanupManager@79] -
> autopurge.purgeInterval set to 24
> 2020-05-04 18:15:06,610 [myid:1] - INFO  [main:ManagedUtil@44] - Log4j 1.2
> jmx support found and enabled.
> 2020-05-04 18:15:06,614 [myid:1] - INFO
> [PurgeTask:DatadirCleanupManager$PurgeTask@139] - Purge task started.
> 2020-05-04 18:15:06,618 [myid:1] - INFO  [PurgeTask:FileTxnSnapLog@124] -
> zookeeper.snapshot.trust.empty : true
> 2020-05-04 18:15:06,629 [myid:1] - INFO  [main:QuorumPeerMain@151] -
> Starting quorum peer
> 2020-05-04 18:15:06,654 [myid:1] - INFO  [PurgeTask:SnapStream@61] -
> zookeeper.snapshot.compression.method = CHECKED
> 2020-05-04 18:15:06,656 [myid:1] - INFO
> [PurgeTask:DatadirCleanupManager$PurgeTask@145] - Purge task completed.
> 2020-05-04 18:15:06,662 [myid:1] - INFO  [main:ServerMetrics@62] -
> ServerMetrics initialized with provider
> org.apache.zookeeper.metrics.impl.DefaultMetricsProvider@1d251891
> 2020-05-04 18:15:06,780 [myid:1] - INFO  [main:NettyServerCnxnFactory@483]
> -
> zookeeper.client.portUnification=true
> 2020-05-04 18:15:06,781 [myid:1] - INFO  [main:NettyServerCnxnFactory@495]
> -
> zookeeper.netty.advancedFlowControl.enabled = false
> 2020-05-04 18:15:06,782 [myid:1] - INFO  [main:NettyServerCnxnFactory@105]
> -
> handshakeThrottlingEnabled = false,
> zookeeper.netty.server.outstandingHandshake.limit = -1
> 2020-05-04 18:15:06,840 [myid:1] - INFO  [main:ServerCnxnFactory@169] -
> Using org.apache.zookeeper.server.NettyServerCnxnFactory as server
> connection factory
> 2020-05-04 18:15:06,841 [myid:1] - WARN  [main:ServerCnxnFactory@309] -
> maxCnxns is not configured, using default value 0.
> 2020-05-04 18:15:06,841 [myid:1] - INFO  [main:NettyServerCnxnFactory@483]
> -
> zookeeper.client.portUnification=true
> 2020-05-04 18:15:06,842 [myid:1] - INFO  [main:NettyServerCnxnFactory@495]
> -
> zookeeper.netty.advancedFlowControl.enabled = false
> 2020-05-04 18:15:06,842 [myid:1] - INFO  [main:NettyServerCnxnFactory@105]
> -
> handshakeThrottlingEnabled = false,
> zookeeper.netty.server.outstandingHandshake.limit = -1
> 2020-05-04 18:15:06,843 [myid:1] - INFO  [main:ServerCnxnFactory@169] -
> Using org.apache.zookeeper.server.NettyServerCnxnFactory as server
> connection factory
> 2020-05-04 18:15:06,843 [myid:1] - WARN  [main:ServerCnxnFactory@309] -
> maxCnxns is not configured, using default value 0.
> 2020-05-04 18:15:06,852 [myid:1] - INFO  [main:QuorumPeer@752] -
> zookeeper.quorumCnxnTimeoutMs=-1
> 2020-05-04 18:15:06,856 [myid:1] - INFO  [main:FileTxnSnapLog@124] -
> zookeeper.snapshot.trust.empty : true
> 2020-05-04 18:15:06,857 [myid:1] - INFO  [main:QuorumPeer@1680] - Local
> sessions disabled
> 2020-05-04 18:15:06,857 [myid:1] - INFO  [main:QuorumPeer@1691] - Local
> session upgrading disabled
> 2020-05-04 18:15:06,857 [myid:1] - INFO  [main:QuorumPeer@1658] - tickTime
> set to 2000
> 2020-05-04 

Re: ZooKeeper config caching issues?

[Enrico Olivelli]

Rajkiran
I am not sure that changing the addresses this way is supported. Apart from
that...
Maybe you can try to enable reconfig and use it to fix the problem.

Otherwise another way would be to add new nodes with the new addresses and
then dismiss the old nodes


Enrico



Il Sab 2 Mag 2020, 10:42 rajsura  ha scritto:

> Latest observation, we noticed that ZooKeeper was complaining about
> dynamic.next file, event though we HAVE NOT ENABLED
> dynamic-reconfiguration.
>
>
> 2020-05-02 01:43:05,870 [myid:21] - ERROR
> [QuorumPeer[myid=21](plain=/0.0.0.0:2181)(secure=disabled):QuorumPeer@1637
> ]
> - Error writing next dynamic config file to disk:
>
>
> And zookeeper user did not have perms to that config directory, so we fixed
> that restarted zookeeper. And then it dumped below dynamic.next, which
> contains the OLD migrated node as a member :O
>
>
> $ sudo cat /opt/zookeeper/conf/zoo.cfg.dynamic.next
> server.17=node1.foo.bar.com:2888:3888:participant;0.0.0.0:2181
> server.19=node2.foo.bar.com:2888:3888:participant;0.0.0.0:2181
> server.20=node3.foo.bar.com:2888:3888:participant;0.0.0.0:2181
> server.21=node4.foo.bar.com:2888:3888:participant;0.0.0.0:2181
> server.22=node5.bar.com:2888:3888:participant;0.0.0.0:2181
>
>
> So, this looks like a bug. And from where is it still fetching this? How do
> we fix it.
> Any lead/help is very much appreciated.
>
> FTR: We haven't enabled dynamic reconfig.
>
> Thanks in advance,
> Rajkiran
>
>
>
> --
> Sent from: http://zookeeper-user.578899.n2.nabble.com/
>

Re: ZK not starting during upgrade to use 3.6.1 with SSL communication

[Enrico Olivelli]

Hi,
>From which version are you upgrading?
Enrico

Il Sab 2 Mag 2020, 01:39 blb.dev  ha scritto:

> I also notice that in the /conf/zoo.cfg file the servers are being added
> out
> of order, but the /data/myid files are all correct server id.
>
> For example, it's showing as:
> server.2=:2888:3888:participant
> server.3=:2888:3888:participant
> server.1=:2888:3888:participant
>
> But I would expect it to be:
> server.1=:2888:3888:participant
> server.2=:2888:3888:participant
> server.3=:2888:3888:participant
>
> Is that an indication of an underlying issue?
>
>
>
> --
> Sent from: http://zookeeper-user.578899.n2.nabble.com/
>

[ANNOUNCE] Apache ZooKeeper 3.6.1

[Enrico Olivelli]

The Apache ZooKeeper team is proud to announce Apache ZooKeeper version
3.6.1

ZooKeeper is a high-performance coordination service for distributed
applications. It exposes common services - such as naming,
configuration management, synchronization, and group services - in a
simple interface so you don't have to write them from scratch. You can
use it off-the-shelf to implement consensus, group management, leader
election, and presence protocols. And you can build on it for your
own, specific needs.

For ZooKeeper release details and downloads, visit:
https://zookeeper.apache.org/releases.html

ZooKeeper 3.6.1 Release Notes are at:
https://zookeeper.apache.org/doc/r3.6.1/releasenotes.html

We would like to thank the contributors that made the release possible.

Regards,

The ZooKeeper Team

Re: Zookeeper-2355

[Enrico Olivelli]

Srikant
It looks like the fix version is 3.5.4 so 3.5.5 should not be affected

Enrico

Il Mar 21 Apr 2020, 18:43 shrikant kalani  ha
scritto:

> Can anyone confirm if below Jira affected 3.5.5 version also ?
>
> https://issues.apache.org/jira/browse/ZOOKEEPER-2355
>
> Thanks
> Srikant Kalani
>
> Sent from my iPhone

Re: Issue starting zookeeper using QuickStart

[Enrico Olivelli - Diennea]

Dave,
are you running on Windows ?
I am not sure ZooKeeper server is able to run on Windows.
Are you using "Git bash" or "cygwin" ?

Which version are you using ?

Enrico

Il giorno 21/04/20, 17:15 "David Cleary"  ha scritto:

Following the QuickStart instructions, I am running into the following 
exception when executing

zkServer start

2020-04-21 09:13:35,005 [myid:] - ERROR [main:ZooKeeperServerMain@69] - 
Invalid arguments, exiting abnormally
java.lang.NumberFormatException: For input string: 
"C:\OpenEdge\WRK\zootest1\zookeeper\bin\..\conf\zoo.cfg"
at 
java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.base/java.lang.Integer.parseInt(Integer.java:652)
at java.base/java.lang.Integer.parseInt(Integer.java:770)
at 
org.apache.zookeeper.server.ServerConfig.parse(ServerConfig.java:78)
at 
org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:109)
at 
org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:67)
at 
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:140)
at 
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:90)
2020-04-21 09:13:35,007 [myid:] - INFO  [main:ZooKeeperServerMain@70] - 
Usage: ZooKeeperServerMain configfile | port datadir [ticktime] [maxcnxns]
Usage: ZooKeeperServerMain configfile | port datadir [ticktime] [maxcnxns]
2020-04-21 09:13:35,012 [myid:] - INFO  [main:ZKAuditProvider@42] - 
ZooKeeper audit is disabled.
2020-04-21 09:13:35,015 [myid:] - ERROR [main:ServiceUtils@42] - Exiting 
JVM with code 2

Code is trying to parse the path to config file for some reason.

clientPortAddress = new InetSocketAddress(Integer.parseInt(args[0]));

I can start the server by not including the start command. However, I have 
no idea how to shut it down.

Any ideas how to get around this issue?

Thanks
Dave




CONFIDENTIALITY & PRIVACY NOTICE
This e-mail (including any attachments) is strictly confidential and may also 
contain privileged information. If you are not the intended recipient you are 
not authorised to read, print, save, process or disclose this message. If you 
have received this message by mistake, please inform the sender immediately and 
destroy this e-mail, its attachments and any copies. Any use, distribution, 
reproduction or disclosure by any person other than the intended recipient is 
strictly prohibited and the person responsible may incur in penalties.
The use of this e-mail is only for professional purposes; there is no guarantee 
that the correspondence towards this e-mail will be read only by the recipient, 
because, under certain circumstances, there may be a need to access this email 
by third subjects belonging to the Company.

Re: reconfiguration error even though reconfig is turned off

[Enrico Olivelli]

This is very annoying, I see.
A possible fix for you is to let the user that executes Zookeeper to write
in the directory.

Can you create a JIRA? It would be better that the daemon does not write to
the directory with code


Enrico

Il Dom 12 Apr 2020, 02:10 allen chan  ha
scritto:

> Hello
>
> Anyone seen this behavior and know how to resolve it?
>
> zookeeper version: zookeeper-3.6.0
> reconfigEnabled=false
>
>
> Upon startup, zk has this complaint even though reconfiguration is
> disabled.
>
> 2020-04-11 17:02:22,393 [myid:1] - ERROR
>
> [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):QuorumPeer@1841
> ]
> - Error writing next dynamic config file to disk
> java.io.FileNotFoundException: /etc/zookeeper/zoo.cfg.dynamic.next.tmp
> (Permission denied)
> at java.io.FileOutputStream.open0(Native Method)
> at java.io.FileOutputStream.open(FileOutputStream.java:270)
> at java.io.FileOutputStream.(FileOutputStream.java:213)
> at java.io.FileOutputStream.(FileOutputStream.java:162)
> at
>
> org.apache.zookeeper.common.AtomicFileOutputStream.(AtomicFileOutputStream.java:59)
> at
>
> org.apache.zookeeper.common.AtomicFileWritingIdiom.(AtomicFileWritingIdiom.java:64)
> at
>
> org.apache.zookeeper.common.AtomicFileWritingIdiom.(AtomicFileWritingIdiom.java:54)
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerConfig.writeDynamicConfig(QuorumPeerConfig.java:568)
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeer.setLastSeenQuorumVerifier(QuorumPeer.java:1838)
> at
> org.apache.zookeeper.server.quorum.Learner.syncWithLeader(Learner.java:679)
> at
> org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:107)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1455)
>
>
> thx
> --
> Allen Michael Chan
>

Re: [ANNOUNCE] New ZooKeeper committer: Mate Szalay-Beko

[Enrico Olivelli]

Kudos Mate!

Welcome aboard

Enrico

Il Ven 3 Apr 2020, 10:46 Norbert Kalmar  ha
scritto:

> Congratulations Máté, well deserved! :)
>
> - Norbert
>
> On Fri, Apr 3, 2020 at 10:42 AM Andor Molnar  wrote:
>
> > The Apache ZooKeeper PMC recently extended committer karma to Mate and he
> > has accepted.
> > Mate has made some great contributions (including C client!) and we are
> > looking forward to even more. :)
> >
> > Congratulations and welcome aboard, Mate!
> >
> >
> >
>

Re: Debian/Ubuntu package of version 3.5?

[Enrico Olivelli - Diennea]

Reynald
You can try to check at BigTop
http://bigtop.apache.org/

As ZooKeeper community we are not providing packaging other than bare tar.gz

Enrico

Il giorno 31/03/20, 08:34 "Reynald Borer"  ha scritto:

Hello everyone,

Is there any Debian/Ubuntu package of zookeeper 3.5 available?

I have tried to find one without success so far.

Thanks and best regards,
Reynald





CONFIDENTIALITY & PRIVACY NOTICE
This e-mail (including any attachments) is strictly confidential and may also 
contain privileged information. If you are not the intended recipient you are 
not authorised to read, print, save, process or disclose this message. If you 
have received this message by mistake, please inform the sender immediately and 
destroy this e-mail, its attachments and any copies. Any use, distribution, 
reproduction or disclosure by any person other than the intended recipient is 
strictly prohibited and the person responsible may incur in penalties.
The use of this e-mail is only for professional purposes; there is no guarantee 
that the correspondence towards this e-mail will be read only by the recipient, 
because, under certain circumstances, there may be a need to access this email 
by third subjects belonging to the Company.

Re: upgrade from 3.4.5 to 3.5.6

[Enrico Olivelli]

Il giorno mar 24 mar 2020 alle ore 11:45 kuldeep singh
 ha scritto:
>
> Hi,
>
> Current Zookeeper version :- 3.4.5
> Upgraded version:- 3.5.6
>
> We are not going with 3.5.7. Our final decision is zookeeper version is
> 3.5.6

I suggest you to move to 3.5.7, 3.5.6 is an older version, see this
problem just as an example:
https://issues.apache.org/jira/browse/ZOOKEEPER-3644

> as per your reply first we need to move latest version of 3.4.x, like below
>
> 3.4.5 -> 3.4.14 -> 3.5.6 (Correct me if I am wrong here)

yes this is ok

>
> But if We are not facing any problem that i have shared you that we have
> set up of 3 node cluster where 2 node are on 3.5.6 version and 1 node on
> 3.4.5, Everything is running fine and didn't get any issue, So what other
> problem we can face if we directly move to 3.5.6
>

Maybe the answer is yes if you are find with having a window of
downtime for ZK server
if peers are not able to talk to each other the system cannot make progress
but when all of the nodes are up and upgraded to 3.5.7 you will be okay.

please test is in some staging environment, non directly in production :-)

Enrico

> Thanks,
> -
> Kuldeep Singh Budania
> Software Architect
>
>
> On Tue, Mar 24, 2020 at 3:58 PM Enrico Olivelli  wrote:
>
> > Hi
> > You have to upgrade to latest 3.4.x Zookeeper then you will upgrade to
> > 3.5.7.
> > All should run well without issues
> >
> >
> > Enrico
> >
> > Il Mar 24 Mar 2020, 10:18 kuldeep singh  ha
> > scritto:
> >
> > > Hi Team,
> > >
> > > We are upgrading zookeeper from 3.4.5 to 3.5.6. I have set up 3 node
> > > cluster where 2 node are on 3.5.6 version and 1 node on 3.4.5.
> > >
> > > Everything is running fine and didn't get any issue on my system.
> > >
> > > but I found something on apache site  that first we need to upgrade on
> > > 3.4.6 than we can upgrade to 3.5.6. So is it mandatory  to go on 3.4.6
> > > first.
> > >
> > > *Upgrading to 3.5.0*
> > >
> > > Upgrading a running ZooKeeper ensemble to 3.5.0 should be done only after
> > > upgrading your ensemble to the 3.4.6 release. Note that this is only
> > > necessary for rolling upgrades (if you're fine with shutting down the
> > > system completely, you don't have to go through 3.4.6). If you attempt a
> > > rolling upgrade without going through 3.4.6 (for example from 3.4.5), you
> > > may get the following error:
> > >
> > > 2013-01-30 11:32:10,663 [myid:2] - INFO [localhost/127.0.0.1:2784
> > > :QuorumCnxManager$Listener@498] - Received connection request /
> > > 127.0.0.1:60876
> > >
> > > 2013-01-30 11:32:10,663 [myid:2] - WARN [localhost/127.0.0.1:2784
> > > :QuorumCnxManager@349] - Invalid server id: -65536
> > >
> > > During a rolling upgrade, each server is taken down in turn and rebooted
> > > with the new 3.5.0 binaries. Before starting the server with 3.5.0
> > > binaries, we highly recommend updating the configuration file so that all
> > > server statements "server.x=..." contain client ports (see the section
> > > Specifying
> > > the client port). As explained earlier you may leave the configuration
> > in a
> > > single file, as well as leave the clientPort/clientPortAddress statements
> > > (although if you specify client ports in the new format, these statements
> > > are now redundant).
> > >
> > > Could you please let me know about this case. Appreciate if respond soon.
> > >
> > > Thanks,
> > > -
> > > Kuldeep Singh Budania
> > >
> >

Re: upgrade from 3.4.5 to 3.5.6

[Enrico Olivelli]

Hi
You have to upgrade to latest 3.4.x Zookeeper then you will upgrade to
3.5.7.
All should run well without issues


Enrico

Il Mar 24 Mar 2020, 10:18 kuldeep singh  ha
scritto:

> Hi Team,
>
> We are upgrading zookeeper from 3.4.5 to 3.5.6. I have set up 3 node
> cluster where 2 node are on 3.5.6 version and 1 node on 3.4.5.
>
> Everything is running fine and didn't get any issue on my system.
>
> but I found something on apache site  that first we need to upgrade on
> 3.4.6 than we can upgrade to 3.5.6. So is it mandatory  to go on 3.4.6
> first.
>
> *Upgrading to 3.5.0*
>
> Upgrading a running ZooKeeper ensemble to 3.5.0 should be done only after
> upgrading your ensemble to the 3.4.6 release. Note that this is only
> necessary for rolling upgrades (if you're fine with shutting down the
> system completely, you don't have to go through 3.4.6). If you attempt a
> rolling upgrade without going through 3.4.6 (for example from 3.4.5), you
> may get the following error:
>
> 2013-01-30 11:32:10,663 [myid:2] - INFO [localhost/127.0.0.1:2784
> :QuorumCnxManager$Listener@498] - Received connection request /
> 127.0.0.1:60876
>
> 2013-01-30 11:32:10,663 [myid:2] - WARN [localhost/127.0.0.1:2784
> :QuorumCnxManager@349] - Invalid server id: -65536
>
> During a rolling upgrade, each server is taken down in turn and rebooted
> with the new 3.5.0 binaries. Before starting the server with 3.5.0
> binaries, we highly recommend updating the configuration file so that all
> server statements "server.x=..." contain client ports (see the section
> Specifying
> the client port). As explained earlier you may leave the configuration in a
> single file, as well as leave the clientPort/clientPortAddress statements
> (although if you specify client ports in the new format, these statements
> are now redundant).
>
> Could you please let me know about this case. Appreciate if respond soon.
>
> Thanks,
> -
> Kuldeep Singh Budania
>

Re: Zookeeper not listening on 2888 and appears nodes are not connecting to each other.

[Enrico Olivelli]

I think this is the problem and it looks like a bug

Exception when following the leader
java.lang.IllegalArgumentException
at
java.util.concurrent.ThreadPoolExecutor.(ThreadPoolExecutor.java:1314)
at
java.util.concurrent.ThreadPoolExecutor.(ThreadPoolExecutor.java:1202)
at java.util.concurrent.Executors.newFixedThreadPool(Executors.
java:89)
at
org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:275)
at
org.apache.zookeeper.server.qu

I can't check the code right know.
But I am sure this is something to address with a bug fix

One question: are nodes reachable with PING each to each other?

In 3.6.0 we have added reachability checks that may interfere with your
setup
We are addressing such kind of issues in 3.6.1 that hopefully will be
released at the beginning of April


Enrico


Enrico


Il Lun 23 Mar 2020, 20:49 rld244  ha scritto:

> rld...@cornell.edu
> Hi,
>
> I'm trying to get a 3 node Zookeeper cluster up and running. I am doing an
> initial install with Ansible from
> https://galaxy.ansible.com/lean_delivery/zookeeper which installs version
> 3.6.0. This gets launched with systemd which is just calling
> bin/zkServer.sh
> start-foreground.
>
> The Ansible role sets the myid's and has a zoo.cfg.dynamic file that is
> called in zoo.cfg. In my efforts to get this to work I've removed the
> reference to the dynamic file.
>
> My zoo.cfg looks like this now.
>
> tickTime=2000
> dataDir=/var/lib/zookeeper
> dataLogDir=/var/log/zookeeper
> initLimit=30
> syncLimit=15
> clientPort=2181
> server.1=0.0.0.0:2888:3888
> server.2=< fqdn of server 2 >:2888:3888
> server.3=< fqdn of server 3 >:2888:3888
> 4lw.commands.whitelist=*
>
> For each server 0.0.0.0 is swapped out in the appropriate place and the
> myid's are correct.
> I upped the initLimit and syncLimit at some point to see if that would make
> a difference.
>
> I've made sure to start the Zookeeper services on each server in the
> correct
> order.
>
> But with all that when I try bin/zkServer.sh status I always get the
> following error
>
> /bin/java
> ZooKeeper JMX enabled by default
> Using config: /opt/zookeeper-3.6.0/bin/../conf/zoo.cfg
> Client port found: 2181. Client address: localhost.
> Error contacting service. It is probably not running.
>
> I think Zookeeper is supposed to be listening on 2888 but am not seeing
> that
> when I try
> netstat -tulpn | grep LISTEN
>
> tcp0  0 0.0.0.0:22  0.0.0.0:*
>  LISTEN
> 1260/sshd
> tcp0  0 127.0.0.1:250.0.0.0:*
>  LISTEN
> 1222/master
> tcp0  0 0.0.0.0:111 0.0.0.0:*
>  LISTEN
> 1/systemd
> tcp6   0  0 :::22   :::*
> LISTEN
> 1260/sshd
> tcp6   0  0 :::8983 :::*
> LISTEN
> 1129/java
> tcp6   0  0 ::1:25  :::*
> LISTEN
> 1222/master
> tcp6   0  0 :::443  :::*
> LISTEN
> 1028/httpd
> tcp6   0  0 :::40093:::*
> LISTEN
> 3354/java
> tcp6   0  0 :::2181 :::*
> LISTEN
> 3354/java
> tcp6   0  0 127.0.0.1:7983  :::*
> LISTEN
> 1129/java
> tcp6   0  0 :::111  :::*
> LISTEN
> 1/systemd
> tcp6   0  0 :::3888 :::*
> LISTEN
> 3354/java
> tcp6   0  0 :::8080 :::*
> LISTEN
> 3354/java
> tcp6   0  0 :::80   :::*
> LISTEN
> 1028/httpd
>
> I'm seeing this in my logs.
>
> 2020-03-23 14:38:38,738 [myid:1] - INFO
>
> [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):ZooKeeperServer@1470
> ]
> - The max bytes for all large requests are set to 104857600
> 2020-03-23 14:38:38,738 [myid:1] - INFO
>
> [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):ZooKeeperServer@1484
> ]
> - The large request threshold is set to -1
> 2020-03-23 14:38:38,738 [myid:1] - INFO
>
> [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):ZooKeeperServer@329
> ]
> - Created server with tickTime 2000 minSessionTimeout 4000
> maxSessionTimeout
> 4 clientPortListenBacklog -1 datadir /var/log/zookeeper/version-2
> snapdir /var/lib/zookeeper/version-2
> 2020-03-23 14:38:38,738 [myid:1] - INFO
>
> [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@75
> ]
> - FOLLOWING - LEADER ELECTION TOOK - 59223 MS
> 2020-03-23 14:38:38,738 [myid:1] - INFO
>
> [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):QuorumPeer@863
> ]
> - Peer state changed: following - discovery
> 2020-03-23 14:38:39,740 [myid:1] - WARN
>
> [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@129
> ]
> - Exception when following the leader
> java.lang.IllegalArgumentException
> at
>
> java.util.concurrent.ThreadPoolExecutor.(ThreadPoolExecutor.java:1314)
> at
>
> java.util.concurrent.ThreadPoolExecutor.(ThreadPoolExecutor.java:1202)
> at
> 

[ANNOUNCE] Apache ZooKeeper 3.6.0 released

[Enrico Olivelli]

The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 3.6.0.

ZooKeeper is a high-performance coordination service for distributed
applications. It exposes common services - such as naming,
configuration management, synchronization, and group services - in a
simple interface so you don't have to write them from scratch. You can
use it off-the-shelf to implement consensus, group management, leader
election, and presence protocols. And you can build on it for your
own, specific needs.

For ZooKeeper release details and downloads, visit:
https://zookeeper.apache.org/releases.html

ZooKeeper 3.6.0 Release Notes are at:
https://zookeeper.apache.org/doc/r3.6.0/releasenotes.html

We would like to thank the contributors that made the release possible.

Regards,

The ZooKeeper Team

Re: [ANNOUNCE] Apache Curator 4.3.0 released

[Enrico Olivelli]

Great
Thank you for driving this release

Enrico

Il Sab 29 Feb 2020, 10:08 tison  ha scritto:

> Congratulations!
>
> Best,
> tison.
>
>
> Cameron McKenzie  于2020年2月29日周六 下午4:59写道：
>
> > Hello,
> >
> > The Apache Curator team is pleased to announce the  release of version
> > 4.3.0. Apache  Curator is a Java/JVM client library for Apache
> > ZooKeeper[1], a distributed coordination service. Apache Curator
> includes a
> > high-level API framework and utilities to make using Apache ZooKeeper
> much
> > easier and more reliable. It also includes recipes for common use cases
> and
> >  extensions such as service discovery and a Java 8 asynchronous DSL. For
> > more details, please visit the project website:
> http://curator.apache.org/
> >
> > The download page for Apache Curator is here:
> > https://cwiki.apache.org/confluence/display/CURATOR/Releases
> >
> > The binary artifacts for Curator are available from Maven Central and its
> > mirrors.
> >
> > For general information on Apache Curator, please visit the project
> > website:
> > http://curator.apache.org
> >
> > Release Notes - Apache Curator - Version 4.3.0
> >
> > ** Bug
> > * [CURATOR-530] - Documentation on InterProcessSemaphoreMutex is
> > misleading
> > * [CURATOR-541] - Infinite loop/repeat in BaseClassForTests
> > * [CURATOR-543] - ZOOKEEPER-1392 broke TestLockACLs
> > * [CURATOR-546] - currentData in ModeledCacheImpl removes ZPath from
> > cache entries
> > * [CURATOR-551] - Curator client always call updateServerList with
> > original serverList value, not the ones updated by EnsembleTracker
> > * [CURATOR-559] - Inconsistent ZK timeouts
> >
> > ** Improvement
> > * [CURATOR-511] - Add toString to ZKPaths PathAndNode
> > * [CURATOR-517] - Remove unused class AdvancedTracerDriver
> > * [CURATOR-533] - Improve CURATOR-505 by making the CircuitBreaker
> > instance shared
> > * [CURATOR-537] - Expose ourPath in LeaderLatch readonly
> > * [CURATOR-547] - Make JAX-RS MessageBodyReader/-Writer impl
> > (JsonServiceInstanceMarshaller) reuse Jackson ObjectMapper
> > * [CURATOR-548] - Bump zookeeper version to 3.5.7
> > * [CURATOR-556] - Fix typo
> > * [CURATOR-560] - Make sure tickTime and minSessionTimeout are set
> >
> > ** Wish
> > * [CURATOR-519] - Curator 4.0.x/4.1.x release using Zookeeper 3.5.5
> >
> > ** Task
> > * [CURATOR-539] - Remove link to obsolete Stack Overflow tag
> >
> > Regards,
> >
> > The Curator Team
> >
> > [1] Apache ZooKeeper https://zookeeper.apache.org/
> >
>

Re: Upgrade guide from 3.4.x to 3.5.x?

[Enrico Olivelli]

If you are upgrading a very short lived cluster (no snapshot on server data
directory) you should upgrade to 3.5.7 (not 3.5.6 or less).
There was an important issue in the upgrade procedure that has been fixed
in 3.5.7


Enrico

Il Sab 15 Feb 2020, 07:27 Szalay-Bekő Máté  ha
scritto:

> An other page on the wiki you can check:
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Upgrade+FAQ
>
> Kind regards,
> Mate
>
> On Sat, Feb 15, 2020, 05:55 Alexander Shraer  wrote:
>
> > Hi, please see “upgrading to 3.5” section here:
> > https://zookeeper.apache.org/doc/r3.5.4-beta/zookeeperReconfig.html
> >
> > On Fri, Feb 14, 2020 at 8:48 PM shrikant kalani <
> shrikantkal...@gmail.com>
> > wrote:
> >
> > > Hi Allen
> > >
> > > We recently upgrade our Zookeeper clusters from 3.4.13 to 3.5.5.
> > >
> > > Yes the rolling upgrade are possible and it is backward compatible
> > meaning
> > > zkclient running on version 3.4.13 can still interact with zkserver
> > 3.5.5.
> > >
> > > Unless you want to leverage dynamic reconfiguration options , the rest
> of
> > > the configuration are very similar. With new version there are other
> > > interesting features like Authentication with Kerberos and TLS , Admin
> UI
> > > which all are optional.
> > >
> > > Thanks
> > > Srikant Kalani
> > > Sent from my iPhone
> > >
> > > > On 15 Feb 2020, at 6:11 AM, allen chan  >
> > > wrote:
> > > >
> > > > Hello
> > > >
> > > > I have been trying to find a guide that describes upgrade process
> from
> > > > 3.4.x to 3.5.x.
> > > > I cannot find anything on the main zookeeper page.
> > > > What i am looking for are breaking changes, configuration changes,
> > > > compatibility matrix, is rolling upgrade ok?
> > > >
> > > > Thanks
> > > > --
> > > > Allen Michael Chan
> > >
> >
>

Re: Admin server deadlocks?

[Enrico Olivelli - Diennea]

Il giorno 12/02/20, 01:21 "gdgenz"  ha scritto:

Andor Molnar-3 wrote
> Enrico, I think I'll step up for RM of 3.5.7 this week. Hope I can find
> some free cycles. Stay tuned.
>
> Andor

We just upgraded several zookeeper clusters (Kafka and ClickHouse) to 3.5.6
and are seeing hangs on the AdminServer, which we are using for some
critical monitoring.  We're not experts at hacking Java builds, so we are
hoping that this fix will be in an official release soon.

On dev@ list we are VOTING 3.5.7, you can test the release


 In the meantime,
can we just update the /lib folder with the new Jetty jars and modify the
java command line accordingly?


Yes


Enrico


Thanks,

Geoff Genz
Principal Engineer, Comcast




--
Sent from: http://zookeeper-user.578899.n2.nabble.com/





CONFIDENTIALITY & PRIVACY NOTICE
This e-mail (including any attachments) is strictly confidential and may also 
contain privileged information. If you are not the intended recipient you are 
not authorised to read, print, save, process or disclose this message. If you 
have received this message by mistake, please inform the sender immediately and 
destroy this e-mail, its attachments and any copies. Any use, distribution, 
reproduction or disclosure by any person other than the intended recipient is 
strictly prohibited and the person responsible may incur in penalties.
The use of this e-mail is only for professional purposes; there is no guarantee 
that the correspondence towards this e-mail will be read only by the recipient, 
because, under certain circumstances, there may be a need to access this email 
by third subjects belonging to the Company.

Re: Zookeeper 3.5.6 supports log4j 2.x?

[Enrico Olivelli]

No, sorry.
But  CVE-2019-17571 does not affect neither Zookeeper  client nor Zookeeper
server.
We have an open ticket about this problem.
Probably we will move to slf4j on 3.7.
In case any help or contribution in this direction will be very appreciated


Enrico



Il Sab 1 Feb 2020, 00:58 Daniel Chan  ha scritto:

> Hi,
>
>
>
> One of the Zookeeper 3.5.6 dependencies is:
>
> log4j > log4j1.2.17
>
>
>
> However, Log4j 1.x had reached end of life according to
> https://logging.apache.org/log4j/1.2/ and also it has a security
> vulnerability:
>
> CVE-2019-17571 has been identified against Log4j 1. Log4j includes a
> SocketServer that accepts serialized log events and deserializes them
> without verifying whether the objects are allowed or not. This can provide
> an attack vector that can be exploited. Since Log4j 1 is no longer
> maintained this issue will not be fixed. Users are urged to upgrade to
> Log4j 2.
>
>
>
> Is there any plan to upgrade to log4j 2.x? or will it work if we just
> replace with log4j 2 jars?
>
>
>
> Thanks,
>
> Daniel
>
>
>

Re: Zookeeper resolving to old host IP addresses

[Enrico Olivelli]

Hi,
Which version of ZK are you using?
Enrico


Il mar 21 gen 2020, 22:51 rammohan ganapavarapu 
ha scritto:

> Hi,
>
> Does zookeeper cache the host IP? if so how long does it cache? I have a
> zk cluster in autoscaling groups and when a new node comes up, other nodes
> still resolving to old IP. Is there any setting to invalidate dns cache for
> zookeeper? or is it jvm dns cache? until other nodes resolves to new IP,
> this node not able to join the cluster.
>
> Thanks,
> Ram
>

Re: Admin server deadlocks?

[Enrico Olivelli]

It looks like we have to release 3.5.7...


Thank you for reporting your issue and the fix
We will follow up on dev@ about the release


Enrico

Il mar 21 gen 2020, 15:49 Craig.Condit  ha scritto:

> I don’t have stack traces handy, but was able to eliminate the problem in
> our environment by updating Jetty to 9.4.24 (ZOOKEEPER-3638 backport).
>
> Craig
>
> From: "Chris T." 
> Reply-To: "user@zookeeper.apache.org" 
> Date: Tuesday, January 21, 2020 at 5:09 AM
> To: "user@zookeeper.apache.org" 
> Subject: Re: Admin server deadlocks?
>
> Yes, attached.
>
> On Tue, Jan 21, 2020 at 6:19 AM Enrico Olivelli  <mailto:eolive...@gmail.com>> wrote:
> Do you have a dump of the stack traces if the JVM during the problem?
>
>
> Enrico
>
> Il lun 20 gen 2020, 21:06 Cee Tee  c.turks...@gmail.com>> ha scritto:
>
> > Hey that sounds familiar, we do curl to the stats api and they pretty
> > consistently start to hang after a while (minutes to hours).
> > I thought it was something in our environment but I'm happy to read I'm
> > not
> > the only one. :)
> > I used to run a snapshot 3.6 that didn't suffer from this. It appeared
> > after 'upgrading' to the official 356.
> > Regards
> > Chris
> >
> > On 20 January 2020 20:54:51 Craig.Condit  <mailto:craig.con...@target.com>> wrote:
> >
> > > We have been running Zookeeper 3.5.6 on several clusters for a while
> > now,
> > > and have noticed (pretty consistently) that the new Admin Server seems
> > to
> > > stop responding (hangs) after the ZK service has been up and running
> for
> > a
> > > while. The stack dumps we have done seem to indicate some sort of lock
> > > being held, probably by Jetty. This makes sense, as when we see the
> hang
> > > start, it happens for all URLs (even the root / 404 not found page).
> > >
> > >
> > > Has anyone encountered this? I was not able to find a related JIRA.
> >
> >
> >
> >
>

Re: Admin server deadlocks?

[Enrico Olivelli]

Do you have a dump of the stack traces if the JVM during the problem?


Enrico

Il lun 20 gen 2020, 21:06 Cee Tee  ha scritto:

> Hey that sounds familiar, we do curl to the stats api and they pretty
> consistently start to hang after a while (minutes to hours).
> I thought it was something in our environment but I'm happy to read I'm
> not
> the only one. :)
> I used to run a snapshot 3.6 that didn't suffer from this. It appeared
> after 'upgrading' to the official 356.
> Regards
> Chris
>
> On 20 January 2020 20:54:51 Craig.Condit  wrote:
>
> > We have been running Zookeeper 3.5.6 on several clusters for a while
> now,
> > and have noticed (pretty consistently) that the new Admin Server seems
> to
> > stop responding (hangs) after the ZK service has been up and running for
> a
> > while. The stack dumps we have done seem to indicate some sort of lock
> > being held, probably by Jetty. This makes sense, as when we see the hang
> > start, it happens for all URLs (even the root / 404 not found page).
> >
> >
> > Has anyone encountered this? I was not able to find a related JIRA.
>
>
>
>

Re: ZooKeeper in secure mode

[Enrico Olivelli - Diennea]

Praveen
In order to use Netty it is better for you to use 3.5.6 that contains Netty 4, 
ZooKeeper 3.4.x uses the deprecated Netty 3. For TSL, and it is known to have 
security flaws and it is no more maintained

Btw your problem looks like there is a missing class and it is weird

Enrico

Il giorno 16/01/20, 10:25 "Praveen Kumar K S"  ha 
scritto:

Hello,

I'm looking for help on enabling authentication in zookeeper. Please note
below approach I have tried.

1. I followed

https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide
2. I'm deploying zookeeper as single node using docker
3. Zookeeper version is 3.4.13
4. Below are some important environmental variables in zookeeper container


CLIENT_JVMFLAGS=-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
-Dzookeeper.client.secure=true
-Dzookeeper.ssl.keyStore.location=/opt/vault/zookeeper/ssl/KeyStore.jks
-Dzookeeper.ssl.keyStore.password=XX@123
-Dzookeeper.ssl.trustStore.location=/opt/vault/zookeeper/ssl/truststore.jks
-Dzookeeper.ssl.trustStore.password=XX@123


SERVER_JVMFLAGS=-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
-Dzookeeper.ssl.keyStore.location=/opt/vault/zookeeper/ssl/KeyStore.jks
-Dzookeeper.ssl.keyStore.password=XX@123
-Dzookeeper.ssl.trustStore.location=/opt/vault/zookeeper/ssl/truststore.jks
-Dzookeeper.ssl.trustStore.password=XX@123


zookeeper.serverCnxnFactory="org.apache.zookeeper.server.NettyServerCnxnFactory"

5. Below is conf file
server.1=0.0.0.0:2888:3888
secureClientPort=2281
initLimit=5
syncLimit=2
tickTime=2000
clientPort=2181
clientPortAddress=zookeeper
dataLogDir=/opt/vault/zookeeper/logs
dataDir=/opt/vault/zookeeper/data

6. Zookeeper is healthy
7. I tried connecting to Zookeeper server from my machine using zkCli.sh.
But getting below error

2020-01-16 14:21:27,798 [myid:] - INFO  [main:ZooKeeper@442] - Initiating
client connection, connectString=zookeeper:2281 sessionTimeout=3
watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@531d72ca
Exception in thread "main" java.io.IOException: Couldn't instantiate
org.apache.zookeeper.ClientCnxnSocketNetty
at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1851)
at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:453)
at org.apache.zookeeper.ZooKeeperMain.connectToZK(ZooKeeperMain.java:283)
at org.apache.zookeeper.ZooKeeperMain.(ZooKeeperMain.java:297)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:290)
Caused by: java.lang.ClassNotFoundException:
org.apache.zookeeper.ClientCnxnSocketNetty
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1848)
... 4 more

8.Zookeeper is working fine on 2181
9.I tried to connect Kafka to Zookeeper on port 2281. Getting below error

[2020-01-16 09:12:07,477] INFO Initiating client connection,
connectString=zookeeper:2281 sessionTimeout=6000
watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@5c33f1a9
(org.apache.zookeeper.ZooKeeper)
[2020-01-16 09:12:07,488] INFO [ZooKeeperClient] Waiting until connected.
(kafka.zookeeper.ZooKeeperClient)
[2020-01-16 09:12:07,489] INFO Opening socket connection to server
zookeeper/172.16.13.2:2281. Will not attempt to authenticate using SASL
(unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-01-16 09:12:07,493] INFO Socket error occurred: zookeeper/
172.16.13.2:2281: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-01-16 09:12:08,599] INFO Opening socket connection to server
zookeeper/172.16.13.2:2281. Will not attempt to authenticate using SASL
(unknown error) (org.apache.zookeeper.ClientCnxn)

Please help and advice.

Regards,
Praveen Kumar K S
+91-9986855625





CONFIDENTIALITY & PRIVACY NOTICE
This e-mail (including any attachments) is strictly confidential and may also 
contain privileged information. If you are not the intended recipient you are 
not authorised to read, print, save, process or disclose this message. If you 
have received this message by mistake, please inform the sender immediately and 
destroy this e-mail, its attachments and any copies. Any use, distribution, 
reproduction or disclosure by any person other than the intended recipient is 
strictly prohibited and the person responsible may incur in penalties.
The use of this e-mail is only for professional purposes; there is no guarantee 
that 

Re: Zookeeper and curator SASL authentication

[Enrico Olivelli - Diennea]

Yes, they are system properties

You can take this guide (about Kafka) as example
https://docs.confluent.io/current/kafka/authentication_sasl/authentication_sasl_gssapi.html



Il giorno 15/01/20, 13:17 "Arpit Jain"  ha scritto:

I have not passed those parameters. Is this something I need to set in
Zookeeper (zoo.cfg) ?

On Wed, Jan 15, 2020 at 12:12 PM Enrico Olivelli - Diennea <
enrico.olive...@diennea.com> wrote:

> Usually with SASL auth you are using:
> kerberos.removeHostFromPrincipal=true
> kerberos.removeRealmFromPrincipal=true
>
> is this the case for you ?
>
> Enrico
>
> Il giorno 15/01/20, 13:01 "Arpit Jain"  ha
> scritto:
>
> I have asked in Curator mailing list as well but not much help. I am
> able
> to set ACL with sasl scheme by using zkCli.sh client in Zookeeper
> server.
> The idea is to use Curator to set the ACLs so that only my client
> application can access its Znodes.
>
>
> On Wed, Jan 15, 2020 at 9:21 AM Szalay-Bekő Máté <
> szalay.beko.m...@gmail.com>
> wrote:
>
> > I am not sure what is wrong with the code... I am not familiar with
> > Curator. I can try to google / reproduce this and see what is wrong,
> but it
> > will take a while for me. So first I would ask the others, maybe
> there is
> > someone who knows both ZooKeeper SASL and Curator and can help you
> more in
> > this mailing list. If noone replies, then I will try to setup a 
dummy
> > project with Curator to test this.
> >
> > Did you also ask around the Curator mailing list maybe? Would it
> help if I
> > send you code about setting the ACLs using plain ZooKeeper (and no
> Curator)?
> >
> > On Tue, Jan 14, 2020 at 2:48 PM Arpit Jain 
> wrote:
> >
> >> Thanks for the clarification.
> >> I am able to authenticate client with Zookeeper. However, when I
> started
> >> to set ACLs with the same client, I get error messages. This is how
> I am
> >> creating curator client for setting ACLs
> >>
> >> CuratorFrameworkFactory.Builder builder =
> >>
> >> CuratorFrameworkFactory.builder().connectString(
> >> coordinatorHosts).retryPolicy(retryPolicy)
> >>
> >> .connectionTimeoutMs(coordinatorConnectionTimeout
> >> ).sessionTimeoutMs(coordinatorSessionTimeout);
> >>
> >> final CuratorFramework curatorFramework =
> >>
> >> builder.authorization("sasl", "zkclient/
> z...@example.com"
> >> .getBytes()).aclProvider(new ACLProvider() {
> >>
> >> @Override
> >>
> >> public List getDefaultAcl() {
> >>
> >> return ZooDefs.Ids.CREATOR_ALL_ACL;
> >>
> >> }
> >>
> >>
> >> @Override
> >>
> >> public List getAclForPath(String path) {
> >>
> >> return ZooDefs.Ids.CREATOR_ALL_ACL;
> >>
> >> }
> >>
> >> }).build();
> >>
> >>
> >>  I see below logs in Zookeeper node:
> >>
> >>
> >>
> >>
> >>
> >> *2020-01-14 13:27:53,174 [myid:1] - INFO
> >>  [NIOWorkerThread-3:SaslServerCallbackHandler@120] - Successfully
> >> authenticated client: authenticationID=zkclient/z...@example.com
> >> ;  authorizationID=zkclient/z...@example.com
> >> .2020-01-14 13:27:53,175 [myid:1] - INFO
> >>  [NIOWorkerThread-3:SaslServerCallbackHandler@136] - Setting
> authorizedID:
> >> zkclient/z...@example.com 2020-01-14 13:27:53,175
> >> [myid:1] - INFO  [NIOWorkerThread-3:ZooKeeperServer@1170] - adding
> SASL
> >> authorization for authorizationID: zkclient/z...@example.com
> >> 2020-01-14 13:27:53,182 [myid:1] - INFO
> >>  [NIOWorkerThread-7:ZooKeeperServer@1095] - got auth packet
> >>