Re: Zookeeper resolving to old host IP addresses
Yep, give it a try.
https://cr.openjdk.java.net/~iris/se/11/latestSpec/api/java.base/java/net/doc-files/net-properties.html
I don't have personal experience with these settings.
Andor
On Wed, 2020-01-22 at 10:37 -0800, rammohan ganapavarapu wrote:
> Hi Andor,
>
> On OS side the hostname resolves to new IP, so it could be JVM is
> the one
> caching. Any setting son jvm to invalidate cache? In some other posts
> i did
> see some one recommending these but not sure it it works.
>
> -Dnetworkaddress.cache.ttl=0
>
> -Dnetworkaddress.cache.negative.ttl=0
>
> Ram
>
> On Wed, Jan 22, 2020 at 2:03 AM Andor Molnar
> wrote:
>
> > Hi Ram,
> >
> > As far as I can see from the code, ZooKeeper uses the standard Java
> > calls
> > getByName() and getAllByName() every time it’s trying to connect to
> > a
> > server.
> >
> >
> > // zookeeper.ipReachableTimeout is not defined
> > if (ipReachableTimeout <= 0) {
> > address = InetAddress.getByName(this.hostname);
> > } else {
> > address = getReachableAddress(this.hostname,
> > ipReachableTimeout);
> > }
> >
> >
> > ZK doesn’t (and definitely should not) cache IP addresses. It’s
> > either the
> > cache of JVM or your DNS server.
> >
> > Dynamic reconfig is available in 3.5.x versions which are already
> > stable
> > now and I think with that you don’t need to reuse existing
> > hostnames.
> > Instead use reconfig commands to properly remove old nodes and add
> > new
> > ones. Sounds like more cumbersome, but maybe more reliable.
> >
> > Andor
> >
> >
> >
> >
> > > On 2020. Jan 21., at 23:14, rammohan ganapavarapu <
> > rammohanga...@gmail.com> wrote:
> > > But still happening for me, is there any config on zookeeper side
> > > to make
> > > this fix to work?
> > >
> > > Ram
> > >
> > > On Tue, Jan 21, 2020 at 2:12 PM Michael Han
> > > wrote:
> > >
> > > > Could be ZOOKEEPER-1506, though this should be fixed already in
> > > > 3.4.14.
> > > >
> > > > On Tue, Jan 21, 2020 at 2:01 PM rammohan ganapavarapu <
> > > > rammohanga...@gmail.com> wrote:
> > > >
> > > > > Hi Enrico,
> > > > >
> > > > > I see same with both 3.4.5 and 3.4.14
> > > > >
> > > > > Ram
> > > > >
> > > > > On Tue, Jan 21, 2020 at 1:53 PM Enrico Olivelli <
> > > > > eolive...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > > Which version of ZK are you using?
> > > > > > Enrico
> > > > > >
> > > > > >
> > > > > > Il mar 21 gen 2020, 22:51 rammohan ganapavarapu <
> > > > rammohanga...@gmail.com
> > > > > > ha scritto:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > Does zookeeper cache the host IP? if so how long does it
> > > > > > > cache? I
> > > > have
> > > > > a
> > > > > > > zk cluster in autoscaling groups and when a new node
> > > > > > > comes up, other
> > > > > > nodes
> > > > > > > still resolving to old IP. Is there any setting to
> > > > > > > invalidate dns
> > > > cache
> > > > > > for
> > > > > > > zookeeper? or is it jvm dns cache? until other nodes
> > > > > > > resolves to new
> > > > > IP,
> > > > > > > this node not able to join the cluster.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Ram
> > > > > > >
Re: ZooKeeper Meetup @ Facebook is 1 Week Away! (January 29th 2020)
Mohamed, Could you please answer the question about whether there will be an NDA required at the door? On Wed, Jan 22, 2020 at 12:51 PM Mohamed Jeelani wrote: > ZooKeeper Meetup @ Facebook is just 1 week away! (January 29th 2020) - > Please RSVP if you haven’t yet at - https://zkmeetup2020.splashthat.com/ > > We have an exciting Agenda with some great speakers lined up for the > evening – > > 4.30pm – 5:00pm – Registration/Welcome/Meet & Greets > > 5:00pm – 5:50pm – ZooKeeper @ Facebook (Hear from our engineers on what > we’ve been upto including insights into upcoming features like performance > improvements, demand control and running zookeeper in containers) > 5.50pm – 6:05pm - Upgrading to ZooKeeper 3.5 in Cloudera, Mate > Szalay-Beko, staff software engineer in CLoudera will be joining us > remotely from Budapest > 6:05pm – 6:25pm - Zookeeper Security in Salesforce - Fiaz Hossain (Eng > Lead for Security & Site Reliability CTO) & Suhas Dantkales (Lead Software > Engineer) from Salesforce > 6:25pm – 6:45pm – Zookeeper @ Twitter – Michael Han, Software engineer on > Twitter's platform team > 6:45pm – 7:00pm – ZooKeeper Performance by San Jose State University > > 7:00pm – 8:00pm – Happy Hour/Networking > > Be early so that you don’t miss out on some cool ZK@FB Swag! We are > looking forward to ‘meeting-up’ with you! > > Please RSVP at - https://zkmeetup2020.splashthat.com/ > >
ZooKeeper Meetup @ Facebook is 1 Week Away! (January 29th 2020)
ZooKeeper Meetup @ Facebook is just 1 week away! (January 29th 2020) - Please RSVP if you haven’t yet at - https://zkmeetup2020.splashthat.com/ We have an exciting Agenda with some great speakers lined up for the evening – 4.30pm – 5:00pm – Registration/Welcome/Meet & Greets 5:00pm – 5:50pm – ZooKeeper @ Facebook (Hear from our engineers on what we’ve been upto including insights into upcoming features like performance improvements, demand control and running zookeeper in containers) 5.50pm – 6:05pm - Upgrading to ZooKeeper 3.5 in Cloudera, Mate Szalay-Beko, staff software engineer in CLoudera will be joining us remotely from Budapest 6:05pm – 6:25pm - Zookeeper Security in Salesforce - Fiaz Hossain (Eng Lead for Security & Site Reliability CTO) & Suhas Dantkales (Lead Software Engineer) from Salesforce 6:25pm – 6:45pm – Zookeeper @ Twitter – Michael Han, Software engineer on Twitter's platform team 6:45pm – 7:00pm – ZooKeeper Performance by San Jose State University 7:00pm – 8:00pm – Happy Hour/Networking Be early so that you don’t miss out on some cool ZK@FB Swag! We are looking forward to ‘meeting-up’ with you! Please RSVP at - https://zkmeetup2020.splashthat.com/
Re: [ANNOUNCE] Enrico Olivelli new ZooKeeper PMC Member
Congratulations Enrico! __ On 1/21/20, 1:41 PM, "Flavio Junqueira" wrote: I'm pleased to announce that Enrico Olivelli recently became the newest ZooKeeper PMC member. Enrico has contributed immensely to this community; he became a ZooKeeper committer in May 2019 and now he joins the PMC. Join me in congratulating him on the achievement. Congrats, Enrico! -Flavio on behalf of the Apache ZooKeeper PMC
Reading snapshot
Hi, What is the purpose of reading snapshot from disk while zk processes starting up? i have a snapshot of ~2gb and it is taking time to read that snapshot, hence cluster recovery time. Is it to get the last zxid of that server? ```2020-01-21 23:11:24,510 [myid:106] - INFO [QuorumPeer[myid=106]/0:0:0:0:0:0:0:0:2181:QuorumPeer@967] - OBSERVING 2020-01-21 23:11:24,510 [myid:106] - INFO [QuorumPeer[myid=106]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@174] - Created server with tickTime 6 minSessionTimeout 12 maxSessionTimeout 120 datadir /opt/data/zookeeper/logs/version-2 snapdir /opt/data/zookeeper/data/version-2 2020-01-21 23:11:24,510 [myid:106] - INFO [QuorumPeer[myid=106]/0:0:0:0:0:0:0:0:2181:ObserverZooKeeperServer@64] - syncEnabled =true 2020-01-21 23:11:24,511 [myid:106] - INFO [QuorumPeer[myid=106]/0:0:0:0:0:0:0:0:2181:QuorumPeer$QuorumServer@185] - Resolved hostname: rzk100ea to address: rzk100ea/192.168.10.78 2020-01-21 23:11:24,512 [myid:106] - INFO [QuorumPeer[myid=106]/0:0:0:0:0:0:0:0:2181:Observer@65] - Observing rzk100ea/192.168.10.78:2888 2020-01-21 23:11:24,711 [myid:106] - INFO [QuorumPeer[myid=106]/0:0:0:0:0:0:0:0:2181:FileSnap@86] - Reading snapshot /opt/data/zookeeper/data/version-2/snapshot.5a 2020-01-21 23:12:31,954 [myid:106] - INFO [QuorumPeer[myid=106]/0:0:0:0:0:0:0:0:2181:Learner@336] - Getting a snapshot from leader 0x5b 2020-01-21 23:18:32,497 [myid:106] - INFO [QuorumPeer[myid=106]/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@301] - Snapshotting: 0x5b to /opt/data/zookeeper/data/version-2/snapshot.5b``` Thanks, Ram
Re: Zookeeper resolving to old host IP addresses
Hi Andor,
On OS side the hostname resolves to new IP, so it could be JVM is the one
caching. Any setting son jvm to invalidate cache? In some other posts i did
see some one recommending these but not sure it it works.
-Dnetworkaddress.cache.ttl=0
-Dnetworkaddress.cache.negative.ttl=0
Ram
On Wed, Jan 22, 2020 at 2:03 AM Andor Molnar wrote:
> Hi Ram,
>
> As far as I can see from the code, ZooKeeper uses the standard Java calls
> getByName() and getAllByName() every time it’s trying to connect to a
> server.
>
>
> // zookeeper.ipReachableTimeout is not defined
> if (ipReachableTimeout <= 0) {
> address = InetAddress.getByName(this.hostname);
> } else {
> address = getReachableAddress(this.hostname, ipReachableTimeout);
> }
>
>
> ZK doesn’t (and definitely should not) cache IP addresses. It’s either the
> cache of JVM or your DNS server.
>
> Dynamic reconfig is available in 3.5.x versions which are already stable
> now and I think with that you don’t need to reuse existing hostnames.
> Instead use reconfig commands to properly remove old nodes and add new
> ones. Sounds like more cumbersome, but maybe more reliable.
>
> Andor
>
>
>
>
> > On 2020. Jan 21., at 23:14, rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
> >
> > But still happening for me, is there any config on zookeeper side to make
> > this fix to work?
> >
> > Ram
> >
> > On Tue, Jan 21, 2020 at 2:12 PM Michael Han wrote:
> >
> >> Could be ZOOKEEPER-1506, though this should be fixed already in 3.4.14.
> >>
> >> On Tue, Jan 21, 2020 at 2:01 PM rammohan ganapavarapu <
> >> rammohanga...@gmail.com> wrote:
> >>
> >>> Hi Enrico,
> >>>
> >>> I see same with both 3.4.5 and 3.4.14
> >>>
> >>> Ram
> >>>
> >>> On Tue, Jan 21, 2020 at 1:53 PM Enrico Olivelli
> >>> wrote:
> >>>
> Hi,
> Which version of ZK are you using?
> Enrico
>
>
> Il mar 21 gen 2020, 22:51 rammohan ganapavarapu <
> >> rammohanga...@gmail.com
>
> ha scritto:
>
> > Hi,
> >
> > Does zookeeper cache the host IP? if so how long does it cache? I
> >> have
> >>> a
> > zk cluster in autoscaling groups and when a new node comes up, other
> nodes
> > still resolving to old IP. Is there any setting to invalidate dns
> >> cache
> for
> > zookeeper? or is it jvm dns cache? until other nodes resolves to new
> >>> IP,
> > this node not able to join the cluster.
> >
> > Thanks,
> > Ram
> >
>
> >>>
> >>
>
>
Re: Zookeeper resolving to old host IP addresses
Andor,
Thank you, i will look into dynamic reconfig option.
Ram
On Wed, Jan 22, 2020 at 2:03 AM Andor Molnar wrote:
> Hi Ram,
>
> As far as I can see from the code, ZooKeeper uses the standard Java calls
> getByName() and getAllByName() every time it’s trying to connect to a
> server.
>
>
> // zookeeper.ipReachableTimeout is not defined
> if (ipReachableTimeout <= 0) {
> address = InetAddress.getByName(this.hostname);
> } else {
> address = getReachableAddress(this.hostname, ipReachableTimeout);
> }
>
>
> ZK doesn’t (and definitely should not) cache IP addresses. It’s either the
> cache of JVM or your DNS server.
>
> Dynamic reconfig is available in 3.5.x versions which are already stable
> now and I think with that you don’t need to reuse existing hostnames.
> Instead use reconfig commands to properly remove old nodes and add new
> ones. Sounds like more cumbersome, but maybe more reliable.
>
> Andor
>
>
>
>
> > On 2020. Jan 21., at 23:14, rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
> >
> > But still happening for me, is there any config on zookeeper side to make
> > this fix to work?
> >
> > Ram
> >
> > On Tue, Jan 21, 2020 at 2:12 PM Michael Han wrote:
> >
> >> Could be ZOOKEEPER-1506, though this should be fixed already in 3.4.14.
> >>
> >> On Tue, Jan 21, 2020 at 2:01 PM rammohan ganapavarapu <
> >> rammohanga...@gmail.com> wrote:
> >>
> >>> Hi Enrico,
> >>>
> >>> I see same with both 3.4.5 and 3.4.14
> >>>
> >>> Ram
> >>>
> >>> On Tue, Jan 21, 2020 at 1:53 PM Enrico Olivelli
> >>> wrote:
> >>>
> Hi,
> Which version of ZK are you using?
> Enrico
>
>
> Il mar 21 gen 2020, 22:51 rammohan ganapavarapu <
> >> rammohanga...@gmail.com
>
> ha scritto:
>
> > Hi,
> >
> > Does zookeeper cache the host IP? if so how long does it cache? I
> >> have
> >>> a
> > zk cluster in autoscaling groups and when a new node comes up, other
> nodes
> > still resolving to old IP. Is there any setting to invalidate dns
> >> cache
> for
> > zookeeper? or is it jvm dns cache? until other nodes resolves to new
> >>> IP,
> > this node not able to join the cluster.
> >
> > Thanks,
> > Ram
> >
>
> >>>
> >>
>
>
Re: [ANNOUNCE] Enrico Olivelli new ZooKeeper PMC Member
Congrats Enrico! On Tue, Jan 21, 2020 at 10:40 PM Flavio Junqueira wrote: > I'm pleased to announce that Enrico Olivelli recently became the newest > ZooKeeper PMC member. Enrico has contributed immensely to this community; > he became a ZooKeeper committer in May 2019 and now he joins the PMC. > > Join me in congratulating him on the achievement. Congrats, Enrico! > > -Flavio on behalf of the Apache ZooKeeper PMC
[ZOOKEEPER-2164] This ZooKeeper instance is not currently serving requests
Hello, I have a kubernetes cluster with 3 zookeeper nodes (as stateful set) that, for cost-saving purposes, it downscaled every evening and upscaled in the morning. Since the upgrade to 3.5.6 (previously I was using using zookeeper shipped with Kafka 2.3.0 archive) the nodes are experiencing issues with establishing the quorum. I believe it's related to the JIRA ticket https://issues.apache.org/jira/browse/ZOOKEEPER-2164. However, although it seems to me as a quite a serious bug, the ticket is stale. Have there been any other steps performed to fix that issue or are there any workarounds? Currently, 2 out of 3 nodes have established a quorum and second node is always returning "This ZooKeeper instance is not currently serving requests" although there are no errors in the logs, only some repeated lines about FastLeaderElection: [2020-01-22 11:38:46,076] INFO Have smaller server identifier, so dropping the connection: (3, 2) (org.apache.zookeeper.server.quorum.QuorumCnxManager) [2020-01-22 11:38:46,076] INFO Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) (org.apache.zookeeper.server.quorum.FastLeaderElection) [2020-01-22 11:38:46,078] INFO Notification: 2 (message format version), 1 (n.leader), 0x10010 (n.zxid), 0x1 (n.round), LEADING (n.state), 1 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)0 (n.config version) (org.apache.zookeeper.server.quorum.FastLeaderElection) I understand that scaling zookeeper to 1 node and then scaling up step by step should allow them to establish correct quorum of 3 but I don't want to have to do things like this every morning. Also, as far as I understand the issue correctly, if I were now to perform a rolling update of the zookeeper pods (which happens in reverse order) the pods wouldn't establish any quorum again. Thanks, Jan
Re: Zookeeper resolving to old host IP addresses
Hi Ram,
As far as I can see from the code, ZooKeeper uses the standard Java calls
getByName() and getAllByName() every time it’s trying to connect to a server.
// zookeeper.ipReachableTimeout is not defined
if (ipReachableTimeout <= 0) {
address = InetAddress.getByName(this.hostname);
} else {
address = getReachableAddress(this.hostname, ipReachableTimeout);
}
ZK doesn’t (and definitely should not) cache IP addresses. It’s either the
cache of JVM or your DNS server.
Dynamic reconfig is available in 3.5.x versions which are already stable now
and I think with that you don’t need to reuse existing hostnames. Instead use
reconfig commands to properly remove old nodes and add new ones. Sounds like
more cumbersome, but maybe more reliable.
Andor
> On 2020. Jan 21., at 23:14, rammohan ganapavarapu
> wrote:
>
> But still happening for me, is there any config on zookeeper side to make
> this fix to work?
>
> Ram
>
> On Tue, Jan 21, 2020 at 2:12 PM Michael Han wrote:
>
>> Could be ZOOKEEPER-1506, though this should be fixed already in 3.4.14.
>>
>> On Tue, Jan 21, 2020 at 2:01 PM rammohan ganapavarapu <
>> rammohanga...@gmail.com> wrote:
>>
>>> Hi Enrico,
>>>
>>> I see same with both 3.4.5 and 3.4.14
>>>
>>> Ram
>>>
>>> On Tue, Jan 21, 2020 at 1:53 PM Enrico Olivelli
>>> wrote:
>>>
Hi,
Which version of ZK are you using?
Enrico
Il mar 21 gen 2020, 22:51 rammohan ganapavarapu <
>> rammohanga...@gmail.com
ha scritto:
> Hi,
>
> Does zookeeper cache the host IP? if so how long does it cache? I
>> have
>>> a
> zk cluster in autoscaling groups and when a new node comes up, other
nodes
> still resolving to old IP. Is there any setting to invalidate dns
>> cache
for
> zookeeper? or is it jvm dns cache? until other nodes resolves to new
>>> IP,
> this node not able to join the cluster.
>
> Thanks,
> Ram
>
>>>
>>
Re: [ANNOUNCE] Enrico Olivelli new ZooKeeper PMC Member
Congratulations Enrico, well earned! :) Regards, Norbert On Tue, Jan 21, 2020 at 11:15 PM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Congratulations Enrico!! > > On Tue, Jan 21, 2020 at 1:41 PM Flavio Junqueira wrote: > > > I'm pleased to announce that Enrico Olivelli recently became the newest > > ZooKeeper PMC member. Enrico has contributed immensely to this community; > > he became a ZooKeeper committer in May 2019 and now he joins the PMC. > > > > Join me in congratulating him on the achievement. Congrats, Enrico! > > > > -Flavio on behalf of the Apache ZooKeeper PMC >
Re: [ANNOUNCE] Enrico Olivelli new ZooKeeper PMC Member
Hi Enrico, Repeating myself, but again: Congratulations! :-) Andor > On 2020. Jan 22., at 6:45, Szalay-Bekő Máté > wrote: > > Congratulations! :) > > On Wed, Jan 22, 2020, 06:36 David Mollitor wrote: > >> You've been a great help to me. Well deserved! >> >> On Wed, Jan 22, 2020, 12:09 AM Mohammad arshad >> >> wrote: >> >>> Congratulations Enrico! >>> >>> >>> -Original Message- >>> From: rammohan ganapavarapu [mailto:rammohanga...@gmail.com] >>> Sent: Wednesday, January 22, 2020 3:45 AM >>> To: user@zookeeper.apache.org >>> Cc: DevZooKeeper >>> Subject: Re: [ANNOUNCE] Enrico Olivelli new ZooKeeper PMC Member >>> >>> Congratulations Enrico!! >>> >>> On Tue, Jan 21, 2020 at 1:41 PM Flavio Junqueira wrote: >>> I'm pleased to announce that Enrico Olivelli recently became the newest ZooKeeper PMC member. Enrico has contributed immensely to this community; he became a ZooKeeper committer in May 2019 and now he joins >>> the PMC. Join me in congratulating him on the achievement. Congrats, Enrico! -Flavio on behalf of the Apache ZooKeeper PMC >>> >>
