Re: Spread Zookeeper nodes from one datacenter to two datacenters
I don’t think you need a third data Center. You can still go with 2 DC with 3 and 2 ZK nodes. A cluster with 5 nodes. You can keep 1 node in each dc as observer node. This will make sure only 3 nodes are participating in leader election process and hence a quorum of 3 will work. On Mon, 24 Jul 2023 at 2:58 PM, wrote: > Hi guys > > Today we have just one datacenter with a few NiFi clusters, so we use a > dedicated 3-node zookeeper cluster in that datacenter. We are now planning > to expand to another datacenter, so we would like to split the NiFi nodes > as well as zookeeper nodes to the two datacenters. However 2 zookeeper > nodes is not a good quorum number, so we had the idea to do the following > regarding zookeeper: > >- Datacenter 1: 2 zookeeper nodes >- Datacenter 2: 2 zookeeper nodes >- Location 3 (another small DC): 1 zookeper node -> no NiFis > > All locations are connected via dark fiber, however the third location is > bit more far away from the others (everything withing 100km). Now, as we > anyway split the NiFi clusters over the two datacenters. Shall we limit the > NiFi zookeeper client (state-management.xml) to the zookeeper nodes located > within the same datacenter? Any comments to our design idea? What’s the > best way to configure zookeeper clients in a way that local (same > datacenter) zookeepers are preferred? > > Any other ideas how we should configure this related to zookeeper? Shall > we use just one zookeeper per location and distribute the load over all 3 > nodes/datacenters evenly? This would then cause load between the > datacenters under normal circumstances… > > Cheers Josef > > > > >
Re: Read performance of 3.4.6 vs 3.8.0 according to zookeeper-benchmark
Hi Will, I remember there was a discussion in the past that starting 3.6 the performance is reduced because of Prometheus metrics endpoint enabled. May be you can try disabling metrics. Recently we compared performance between 3.6.2 and 3.8.0 and found 3.8.0 performing way better than 3.6.2. There was improvement in both read and updates. Thanks Srikant Kalani On Sun, 28 Aug 2022 at 5:37 PM, tison wrote: > Hi Will, > > Thanks for reporting this case! Could you run the benchmark on 3.7.1 also? > It can help us bisect the point of change. > > Best, > tison. > > > Will Now 于2022年8月28日周日 05:56写道： > > > Prior to my server upgrade I'm doing some benchmarking of zookeeper > server > > 3.4.6 and 3.8.0 using zookeeper-benchmark > > https://github.com/brownsys/zookeeper-benchmark > > > > Methodology: For each server version I extracted it anew from the tar > file > > onto my linux box, tweaked zoo.cfg to run a single node. On 3.8.0 I > enabled > > 4 letter words. I run it with a fresh (empty) dataDir. Using default > > settings for zookeeper-benchmark. (Contrary to best practices) I am > > running the server and test client on the same node :-/ > > > > I perform 3 runs and show the READ average below. I am focussing just on > > READ results here; results are in reads/sec, so bigger numbers are > better: > > > > 27,052: 3.4.6 > > 16,805: 3.8.0, digest.enabled=true: > > 16,682: 3.8.0: digest.enabled=false > > > > I saw similar results on a windows box and re-ran on linux for additional > > confirmation. I find it hard to believe that 3.8.0 is 40% slower on > reads! > > > > A while back I performed similar comparisons of 3.4.6 vs 3.6.x and I got > > slow results in 3.6.x initially, but disabling digest.enabled fixed it > and > > the two versions were then comparable. In 3.8.0 I am seeing poor results > > with or without digest enabled. > > > > Any thoughts? Are there some new settings I need to tweak? Is my > > methodology flawed? > > > > Thanks! > > >
Re: write performance issue in 3.6.2
Hi Andor Thanks for your reply. We are planning to perform one more round of stress testing and then I would be able to provide the details logs needed for any troubleshooting. Other details are provided against each question. - which version of Zookeeper is being used, 3.6.2 at server side and 3.6.1 at client side - how many nodes are you running in the ZK cluster, 3 nodes cluster - what is the server configuration? any custom setting is in place? Server runs with standard configuration. We have 30G memory allocated to each jvm. The number of znodes in cluster at anytime ranges between 2million to 4 million. - what is the hardware and software setup? on-prem or cloud? instance type? CPU, memory, disk properties, operating system, etc. It’s on prem, running on Rhel 7. The bare metal host has 48 cores and 378G memory shared among different services. We are using SSD drives. - network characteristics Can you provide more details what I should provide here ? - how many clients are connected and what are they doing? share the relevant source code of your client or the command that you’re running, Around 120 client connections on each node. - 3.6 has advanced monitoring capabilities, setup Prometheus and share screenshots of relevant metrics We have prometheus and Grafana up n running. Any specific metric we should be looking for ? So far what we have noticed is latency spikes up when we see the issue. - server and client logs, debug enabled if possible, Will try to provide from our next testing. - security settings: TLS, Kerberos, etc. TLS enabled in quorum as well as for client connections. - ...anything else which could be important Thanks Srikant Kalani On Fri, 23 Apr 2021 at 5:25 PM, Andor Molnar wrote: > Hi folks, > > As previously mentioned the community won’t be able to help if you don’t > share more information about your scenario. We need to see the following: > > - which version of Zookeeper is being used, > - how many nodes are you running in the ZK cluster, > - what is the server configuration? any custom setting is in place? > - what is the hardware and software setup? on-prem or cloud? instance > type? CPU, memory, disk properties, operating system, etc. > - network characteristics > - how many clients are connected and what are they doing? share the > relevant source code of your client or the command that you’re running, > - 3.6 has advanced monitoring capabilities, setup Prometheus and share > screenshots of relevant metrics > - server and client logs, debug enabled if possible, > - security settings: TLS, Kerberos, etc. > - ...anything else which could be important > > In a nutshell, either you have to share information about your production > system or provide a reproduction setup. Performance issues are pretty hard > to resolve, because of the so many moving parts. The community is willing > to help, but you need to share information to be successful. > > shrikant, > ZK 3.6 has throttling for both client connections and requests. Request > throttling can be disabled and it’s disabled by default, but connection > throttling is not. From the log messages we can tell which throttling is in > effect for your scenario. > > Regards, > Andor > > > > > On 2021. Apr 21., at 5:25, shrikant kalani > wrote: > > > > Hello Everyone, > > > > We are also using zookeeper 3.6.2 with ssl turned on both sides. We > > observed the same behaviour where under high write load the ZK server > > starts expiring the session. There are no jvm related issues. During high > > load the max latency increases significantly. > > > > Also the session expiration message is not accurate. We do have session > > expiration set to 40 sec but ZK server disconnects the client within 10 > sec. > > > > Also the logs prints throttling the request but ZK documentation says > > throttling is disabled by default. Can someone check the code once to see > > if it is enabled or disabled. I am not a developer and hence not familiar > > with java code. > > > > Thanks > > Srikant Kalani > > > > On Wed, 21 Apr 2021 at 11:03 AM, Michael Han wrote: > > > >> What is the workload looking like? Is it pure write, or mixed read > write? > >> > >> A couple of ideas to move this forward: > >> * Publish the performance benchmark so the community can help. > >> * Bisect git commit and find the bad commit that caused the regression. > >> * Use the fine grained metrics introduced in 3.6 (e.g per processor > stage > >> metrics) to measure where time spends during writes. We might have to > add > >> these metrics on 3.4 to get a fair comparison. > >> > >> For the throttling - the RequestThrottler i
Re: write performance issue in 3.6.2
Hello Everyone, We are also using zookeeper 3.6.2 with ssl turned on both sides. We observed the same behaviour where under high write load the ZK server starts expiring the session. There are no jvm related issues. During high load the max latency increases significantly. Also the session expiration message is not accurate. We do have session expiration set to 40 sec but ZK server disconnects the client within 10 sec. Also the logs prints throttling the request but ZK documentation says throttling is disabled by default. Can someone check the code once to see if it is enabled or disabled. I am not a developer and hence not familiar with java code. Thanks Srikant Kalani On Wed, 21 Apr 2021 at 11:03 AM, Michael Han wrote: > What is the workload looking like? Is it pure write, or mixed read write? > > A couple of ideas to move this forward: > * Publish the performance benchmark so the community can help. > * Bisect git commit and find the bad commit that caused the regression. > * Use the fine grained metrics introduced in 3.6 (e.g per processor stage > metrics) to measure where time spends during writes. We might have to add > these metrics on 3.4 to get a fair comparison. > > For the throttling - the RequestThrottler introduced in 3.6 does introduce > latency, but should not impact throughput this much. > > On Thu, Mar 11, 2021 at 11:46 AM Li Wang wrote: > > > The CPU usage of both server and client are normal (< 50%) during the > test. > > > > Based on the investigation, the server is too busy with the load. > > > > The issue doesn't exist in 3.4.14. I wonder why there is a significant > > write performance degradation from 3.4.14 to 3.6.2 and how we can address > > the issue. > > > > Best, > > > > Li > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 11, 2021 at 11:25 AM Andor Molnar wrote: > > > > > What is the CPU usage of both server and client during the test? > > > > > > Looks like server is dropping the clients because either the server or > > > both are too busy to deal with the load. > > > This log line is also concerning: "Too busy to snap, skipping” > > > > > > If that’s the case I believe you'll have to profile the server process > to > > > figure out where the perf bottleneck is. > > > > > > Andor > > > > > > > > > > > > > > > > On 2021. Feb 22., at 5:31, Li Wang wrote: > > > > > > > > Thanks, Patrick. > > > > > > > > Yes, we are using the same JVM version and GC configurations when > > > > running the two tests. I have checked the GC metrics and also the > heap > > > dump > > > > of the 3.6, the GC pause and the memory usage look okay. > > > > > > > > Best, > > > > > > > > Li > > > > > > > > On Sun, Feb 21, 2021 at 3:34 PM Patrick Hunt > wrote: > > > > > > > >> On Sun, Feb 21, 2021 at 3:28 PM Li Wang wrote: > > > >> > > > >>> Hi Enrico, Sushant, > > > >>> > > > >>> I re-run the perf test with the data consistency check feature > > disabled > > > >>> (i.e. -Dzookeeper.digest.enabled=false), the write performance > issue > > of > > > >> 3.6 > > > >>> is still there. > > > >>> > > > >>> With everything exactly the same, the throughput of 3.6 was only > 1/2 > > of > > > >> 3.4 > > > >>> and the max latency was more than 8 times. > > > >>> > > > >>> Any other points or thoughts? > > > >>> > > > >>> > > > >> In the past I've noticed a big impact of GC when doing certain > > > performance > > > >> measurements. I assume you are using the same JVM version and GC > when > > > >> running the two tests? Perhaps our memory footprint has expanded > over > > > time. > > > >> You should rule out GC by running with gc logging turned on with > both > > > >> versions and compare the impact. > > > >> > > > >> Regards, > > > >> > > > >> Patrick > > > >> > > > >> > > > >>> Cheers, > > > >>> > > > >>> Li > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> On Sat, Feb 20, 2021 at 9:04 PM Li Wang wrote: > > > >>> > > > Thanks Sushant and Enrico! > > > > > > This is a really good point. According to the 3.6 documentation, > > the > > > feature is disabled by default. > > > > > > >>> > > > >> > > > > > > https://zookeeper.apache.org/doc/r3.6.2/zookeeperAdmin.html#ch_administration > > > >>> . > > > However, checking the code, the default is enabled. > > > > > > Let me set the zookeeper.digest.enabled to false and see how the > > write > > > operation performs. > > > > > > Best, > > > > > > Li > > > > > > > > > > > > > > > On Fri, Feb 19, 2021 at 1:32 PM Sushant Mane < > > sushantma...@gmail.com> > > > wrote: > > > > > > > Hi Li, > > > > > > > > On 3.6.2 consistency checker (adhash based) is enabled by > default: > > > > > > > > > > > >>> > > > >> > > > > > >
Re: ZK client session expired after implementing SSL
Scott, We are not yet able to fix this issue. Our findings suggest that we have one housekeeping application which cleanups old znodes ( we are yet not using ttl znodes) every hour. This application reads thousands of znodes using get children command which is causing throttling for other client applications and results in client disconnects. Interestingly application works fine with NIO server cxn factory in the same ZK version. So we believe the issue is at Netty level. I am waiting for someone from ZK dev team who can help us to understand this netty behaviour. There might be some netty limitations. Thanks Srikant Kalani On Thu, 30 Jul 2020 at 9:08 PM, Scott Guminy wrote: > Srikant, > > Did you ever resolve this issue? I might be seeing something similar. I'm > also on 3.5.5 with quorum SSL enabled. > > On Mon, Jul 13, 2020 at 10:42 PM shrikant kalani > > wrote: > > > When I turned on Debug logs I can see the requests are getting throttled > > and even PING request from client is throttled. Later event buffer was > full > > and it discarded the Autoread.enable message. > > > > Is there a way to avoid discarding of message or increase the throttling > > limit ? > > > > The cluster memory is well under controlled. > > > > Thanks > > Srikant Kalani > > > > On Mon, 13 Jul 2020 at 11:21 PM, shrikant kalani < > shrikantkal...@gmail.com > > > > > wrote: > > > > > Adding one more email list > > > > > > On Mon, 13 Jul 2020 at 10:49 PM, shrikant kalani < > > shrikantkal...@gmail.com> > > > wrote: > > > > > >> Hi > > >> > > >> We are seeing a very uncommon behaviour. We implemented SSL for quorum > > >> communication in version 3.5.5. After the change we are seeing ZK > client > > >> applications are frequently getting crashed with session expired > > message. > > >> While there are no signs of GC in the application, it is hard to > figure > > out > > >> why ZK server is disconnecting those application. > > >> > > >> Any thoughts ? > > >> > > >> Thanks > > >> Srikant Kalani > > >> > > > > > >
Re: Zookeeper session expiration
Currently our production is running with 3.5.5 and it will take time to move to 3.6.1. When I dig more into this it seems to be related to Netty protocol and it’s limitation. The system is stable when I fail back to NIO and without SSL. As soon as I turned on Netty we are seeing sessions getting throttled which in turn sometimes throttles the ping request too from clients. I believe we should get an option to configure Netty in such a way that ping commands are never throttled. Thanks Srikant Kalani On Mon, 20 Jul 2020 at 7:02 PM, Szalay-Bekő Máté wrote: > Hello, > > can you reproduce the problem with the latest 3.5 version? I mean 3.5.8. > There were a few bugfixes recently that can help. e.g.: > https://issues.apache.org/jira/browse/ZOOKEEPER-3756 > Also you can try to increase some timeout parameters, see > > https://zookeeper.apache.org/doc/r3.5.8/zookeeperAdmin.html#sc_configuration > (like minSessionTimeout, maxSessionTimeout, syncLimit) > > Kind regards, > Mate > > On Mon, Jul 13, 2020 at 5:19 PM Srikant Kalani > wrote: > > > I am facing a similar issue in my application. > > > > Zookeeper Server Version 3.5.5 > > > > I implemented SSL ( server to server ) in quorum communication. > > > > After that ZK client frequently receives session timeouts. > > > > When I turned off SSL then application is behaving normally and there are > > no > > timeouts. > > > > Any thoughts ? > > > > Thanks > > Srikant Kalani > > > > > > > > -- > > Sent from: http://zookeeper-user.578899.n2.nabble.com/ > > >
Re: ZK client session expired after implementing SSL
When I turned on Debug logs I can see the requests are getting throttled and even PING request from client is throttled. Later event buffer was full and it discarded the Autoread.enable message. Is there a way to avoid discarding of message or increase the throttling limit ? The cluster memory is well under controlled. Thanks Srikant Kalani On Mon, 13 Jul 2020 at 11:21 PM, shrikant kalani wrote: > Adding one more email list > > On Mon, 13 Jul 2020 at 10:49 PM, shrikant kalani > wrote: > >> Hi >> >> We are seeing a very uncommon behaviour. We implemented SSL for quorum >> communication in version 3.5.5. After the change we are seeing ZK client >> applications are frequently getting crashed with session expired message. >> While there are no signs of GC in the application, it is hard to figure out >> why ZK server is disconnecting those application. >> >> Any thoughts ? >> >> Thanks >> Srikant Kalani >> >
Re: ZK client session expired after implementing SSL
Adding one more email list On Mon, 13 Jul 2020 at 10:49 PM, shrikant kalani wrote: > Hi > > We are seeing a very uncommon behaviour. We implemented SSL for quorum > communication in version 3.5.5. After the change we are seeing ZK client > applications are frequently getting crashed with session expired message. > While there are no signs of GC in the application, it is hard to figure out > why ZK server is disconnecting those application. > > Any thoughts ? > > Thanks > Srikant Kalani >
ZK client session expired after implementing SSL
Hi We are seeing a very uncommon behaviour. We implemented SSL for quorum communication in version 3.5.5. After the change we are seeing ZK client applications are frequently getting crashed with session expired message. While there are no signs of GC in the application, it is hard to figure out why ZK server is disconnecting those application. Any thoughts ? Thanks Srikant Kalani
Encrypted Keystore support
Hi I am running a zookeeper cluster with SSL turned on. The client connecting to the cluster has encrypted keystore , is there a way we can use encrypted keystore ? The property zookeeper.ssl.keystore expects a plain keystore Thanks Srikant Kalani
Follower sync with Leader
Hi Users, We are seeing an issue in our cluster where one follower which loads an old snapshot doesn’t sync with leader. Can you tell me do follower read txlog file while coming up or they are dependent on leader to provide the difference ? If they are dependent on leaders how the difference is calculated ? Zxid is last transaction in txlog or snapshot ? Thanks Srikant Kalani Sent from my iPhone
Zookeeper-2355
Can anyone confirm if below Jira affected 3.5.5 version also ? https://issues.apache.org/jira/browse/ZOOKEEPER-2355 Thanks Srikant Kalani Sent from my iPhone
Follower Data not in sync after restart
Hello Everyone, We are using Zookeeper 3.5.5 in our environment. We don’t take frequent snapshots as our snap count value is 10M. When we restart our cluster, follower node tries to read the snapshot from the disk which is too old while the latest transaction are in txlog. While the leader node after reading snapshot takes a new snapshot from txlogfile. Followers tries to get the difference from leader node but somehow I don’t see updated znode in follower instance which is causing issues in our application. Can anyone suggest how I can Ensure that leader should send the complete snapshot to follower or is there a way each instance can take a snapshot while the instance is stopped so that next time when they start they should read the latest snapshot file. Or each follower should load the last txlog ? The immediate resolution we are following is to remove the data dir for follower and start the instance. Doing this way it takes complete copy from leader. Thanks Srikant Kalani Sent from my iPhone
Re: Upgrade guide from 3.4.x to 3.5.x?
Hi Allen We recently upgrade our Zookeeper clusters from 3.4.13 to 3.5.5. Yes the rolling upgrade are possible and it is backward compatible meaning zkclient running on version 3.4.13 can still interact with zkserver 3.5.5. Unless you want to leverage dynamic reconfiguration options , the rest of the configuration are very similar. With new version there are other interesting features like Authentication with Kerberos and TLS , Admin UI which all are optional. Thanks Srikant Kalani Sent from my iPhone > On 15 Feb 2020, at 6:11 AM, allen chan wrote: > > Hello > > I have been trying to find a guide that describes upgrade process from > 3.4.x to 3.5.x. > I cannot find anything on the main zookeeper page. > What i am looking for are breaking changes, configuration changes, > compatibility matrix, is rolling upgrade ok? > > Thanks > -- > Allen Michael Chan
Application thread hung while acquiring Zookeeper lock
Hi Everyone , Our application is getting hung while acquiring Zookeeper lock. The below thread is never released which cause application outage as it never acquires a lock on a node. We recently upgraded our Zookeeper server to 3.5.5 but the client is still using 3.4.13 version. Anyone aware of any issues or potential bug ? acquireLock(ZKLockService.java:34) - waiting to lock thanks Srikant Kalani Sent from my iPhone
RE: Authorisation in Zookeeper
Enrico , Do you have some examples to show. Right now my user is authenticated based on host level certs. How should I add the scheme and then add authorisation rule ? Thanks Srikant Kalani Sent from Mail for Windows 10 From: Enrico Olivelli Sent: 11 January 2020 21:05 To: UserZooKeeper Subject: Re: Authorisation in Zookeeper Il giorno sab 11 gen 2020 alle ore 09:31 shrikant kalani < shrikantkal...@gmail.com> ha scritto: > > My system account means a client process running with unix user id. > > I want user A to have full access while all other users should only read > data from znodes. > Yes ACLs are your way to go Enrico > > Thanks > Srikant Kalani > Sent from my iPhone > > > On 11 Jan 2020, at 2:20 PM, Enrico Olivelli wrote: > > > > Srikant > > > > Il sab 11 gen 2020, 03:48 shrikant kalani ha > > scritto: > > > >> Hi Zookeeper Users > >> > >> I have implemented TLS authentication in my cluster. Right now the > >> authentication is done based on host name (X509). > >> > >> Now I want to implement authorisation based on user I’d like only my > >> system account should be able to read write data to znodes. > >> > > > > Can you define 'my system account'? > > Is your goal that only authenticated users are able to access data? > > > > > > Enrico > > > >> > >> How I can do that ? Is ACLs is the only solution ? > >> > >> Thanks > >> Srikant Kalani > >> > >> Sent from my iPhone >
Re: Authorisation in Zookeeper
My system account means a client process running with unix user id. I want user A to have full access while all other users should only read data from znodes. Thanks Srikant Kalani Sent from my iPhone > On 11 Jan 2020, at 2:20 PM, Enrico Olivelli wrote: > > Srikant > > Il sab 11 gen 2020, 03:48 shrikant kalani ha > scritto: > >> Hi Zookeeper Users >> >> I have implemented TLS authentication in my cluster. Right now the >> authentication is done based on host name (X509). >> >> Now I want to implement authorisation based on user I’d like only my >> system account should be able to read write data to znodes. >> > > Can you define 'my system account'? > Is your goal that only authenticated users are able to access data? > > > Enrico > >> >> How I can do that ? Is ACLs is the only solution ? >> >> Thanks >> Srikant Kalani >> >> Sent from my iPhone
Authorisation in Zookeeper
Hi Zookeeper Users I have implemented TLS authentication in my cluster. Right now the authentication is done based on host name (X509). Now I want to implement authorisation based on user I’d like only my system account should be able to read write data to znodes. How I can do that ? Is ACLs is the only solution ? Thanks Srikant Kalani Sent from my iPhone
Re: User Interface for Zookeeper/Kafka administration?
For Kafka you can use Kafka Manager and for Zookeeper i am not sure how much Zookeeper Admin UI in version 3.5 helps you. Thanks Srikant Kalani Sent from my iPhone > On 6 Jan 2020, at 9:11 PM, Andor Molnar wrote: > > Hi, > > There’s no such User Interface built-in for ZooKeeper and I’m not sure about > Kafka. Hadoop companies like Cloudera and MapR creates proprietary software > that are able to “visualize” clusters in one way or the other. > > I’m not aware of such open source projects. > > Andor > > > >> On 2019. Dec 23., at 10:31, Sunil CHAUDHARI >> wrote: >> >> Hi, >> I have setup 3 nodes zookeepr and 3 Brokers Kafka cluster on Linux servers. >> Is there any user interface available where I can see complete picture of my >> both kafka and zookeeper nodes? >> Or I have only command line interface ☹☹, which is quite time consuming and >> present very selective data based on commands. >> >> >> Thanks >> Sunil. >> >> CONFIDENTIAL NOTE: >> The information contained in this email is intended only for the use of the >> individual or entity named above and may contain information that is >> privileged, confidential and exempt from disclosure under applicable law. If >> the reader of this message is not the intended recipient, you are hereby >> notified that any dissemination, distribution or copying of this >> communication is strictly prohibited. If you have received this message in >> error, please immediately notify the sender and delete the mail. Thank you. >
Re: Zookeeper server and client authentication
Enrico,
Is 3.6 going to be available soon ? Within 1 month ?
Thanks
Srikant Kalani
Sent from my iPhone
> On 30 Dec 2019, at 9:23 PM, Enrico Olivelli wrote:
>
> If you try to use wrong credentials, corrupted keytab...you won't be able
> to read/write.
> Connection maybe is allowed
>
> Enrico
>
> Il lun 30 dic 2019, 14:19 Arpit Jain ha scritto:
>
>> Just to confirm the settings I have in my environment:
>>
>> 1. On ZK side, my JAAS file looks like this:
>> Server {
>> com.sun.security.auth.module.Krb5LoginModule required
>> useKeyTab=true
>> keyTab="/conf/zoo1.keytab"
>> storeKey=true
>> useTicketCache=false
>> principal="zookeeper/z...@example.com";
>> };
>> The principal "*zookeeper/z...@example.com "* has been
>> created in Kerberos server running locally. I am able to start ZK with this
>> principal and I can see ticket exchange between ZK and Kerberos for this
>> principal.
>>
>> 2. On client (Curator) side, JAAS file looks like below. Principal
>> "*zkcli...@example.com
>> "* is present in Kerberos server. The curator is
>> able
>> to connect properly to ZK (with or without principal) even though SASL is
>> enabled. May be I should use ZK 3.6 as you pointed out to enforce
>> authentication.
>> Client {
>> com.sun.security.auth.module.Krb5LoginModule required
>> useKeyTab=true
>> keyTab="/tmp/zkclient.keytab"
>> storeKey=true
>> useTicketCache=false
>> principal="zkcli...@example.com";
>> };
>>
>> Just want to make sure my settings are correct.
>>
>> Thanks
>>
>>> On Mon, Dec 30, 2019 at 12:47 PM Enrico Olivelli
>>> wrote:
>>>
>>> Arpit,
>>> Up to 3.5.x you can only leverage auth only in conjunction with ACLs.
>>>
>>> I hope we are able to release 3.6.0 within a couple of weeks.
>>>
>>> If you have time you can build from branch-3.6 and run the server
>> enabling
>>> that feature tha you are pointing to.
>>> It is a server side change only so you can use 3.5 in your application
>>>
>>>
>>> Enrico
>>>
>>> Il lun 30 dic 2019, 13:23 shrikant kalani ha
>>> scritto:
>>>
>>>> Couple of things which you can check -
>>>> 1) if your Zookeeper server is not running with Zookeeper I’d then you
>>>> need to set Zookeeper.sasl.client.username
>>>> 2) set java.security.auth.login.config
>>>>
>>>> And I also faced the same issue that there is no strict enforcement to
>>>> allow only authenticated client. Unless someone is aware of the way I
>>> doubt
>>>> we may need to wait for 3.6
>>>>
>>>> Thanks
>>>> Srikant
>>>>
>>>> Sent from my iPhone
>>>>
>>>>> On 30 Dec 2019, at 8:11 PM, Arpit Jain
>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I have configured Zookeeper 3.5.5 to use SASL authentication using
>>>>> Kerberos. I am able to authenticate ZK with Kerberos server but I
>> don't
>>>> see
>>>>> any authentication happening between Zookeeper client (curator) and
>> ZK
>>>>> server. I have put the following setting in zoo.cfg and followed this
>>>> guide
>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
>>>>> .
>>>>>
>>>>>
>>>>
>>>
>> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
>>>>> requireClientAuthScheme=sasl
>>>>>
>>>>> What additional setting I need to provide so that only authenticated
>>>>> clients (for which principals are present in Kerberos server) can
>>> connect
>>>>> to ZK server ?
>>>>> I also found this link
>>>>> https://github.com/apache/zookeeper/pull/118/commits which
>>>>> mentions that it will be strict only from ZK 3.6 onwards and
>> currently
>>> ZK
>>>>> does not enforce it even if we have the configuration.
>>>>>
>>>>> Thanks
>>>>
>>>
>>
Re: Zookeeper server and client authentication
Couple of things which you can check - 1) if your Zookeeper server is not running with Zookeeper I’d then you need to set Zookeeper.sasl.client.username 2) set java.security.auth.login.config And I also faced the same issue that there is no strict enforcement to allow only authenticated client. Unless someone is aware of the way I doubt we may need to wait for 3.6 Thanks Srikant Sent from my iPhone > On 30 Dec 2019, at 8:11 PM, Arpit Jain wrote: > > Hi, > > I have configured Zookeeper 3.5.5 to use SASL authentication using > Kerberos. I am able to authenticate ZK with Kerberos server but I don't see > any authentication happening between Zookeeper client (curator) and ZK > server. I have put the following setting in zoo.cfg and followed this guide > https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication > . > > authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider > requireClientAuthScheme=sasl > > What additional setting I need to provide so that only authenticated > clients (for which principals are present in Kerberos server) can connect > to ZK server ? > I also found this link > https://github.com/apache/zookeeper/pull/118/commits which > mentions that it will be strict only from ZK 3.6 onwards and currently ZK > does not enforce it even if we have the configuration. > > Thanks
Re: Do Zookeeper 3.5.5 supports MTLS ?
Thanks Enrico. I am able to successfully implement TLS in the cluster. Is there a way I can provide access to znodes only for a particular system I’d. I tried Digest scheme authentication but that is not working as expected. Can you confirm the password provided in Digest Auth is some random password or actual password. Sent from my iPhone > On 21 Dec 2019, at 8:22 PM, Enrico Olivelli wrote: > > Yes it does > > Check > http://zookeeper.apache.org/doc/r3.5.6/zookeeperAdmin.html#sc_authOptions > > Hope that helps > Enrico > > Il sab 21 dic 2019, 09:44 shrikant kalani ha > scritto: > >> Yes Mutual Auth and TLS. >> >> Sent from my iPhone >> >>>> On 21 Dec 2019, at 3:51 PM, Enrico Olivelli wrote: >>> >>> Srikant >>> What do you mean with MTLS? Mutual auth and TLS? >>> >>> Enrico >>> >>> Il ven 20 dic 2019, 09:49 shrikant kalani ha >>> scritto: >>> >>>> Hi >>>> >>>> Can someone help me in configuring Zookeeper with MTLS ? >>>> >>>> Thanks >>>> Srikant Kalani >>>> >>>> Sent from my iPhone >>>> >>
Re: Do Zookeeper 3.5.5 supports MTLS ?
Yes Mutual Auth and TLS. Sent from my iPhone > On 21 Dec 2019, at 3:51 PM, Enrico Olivelli wrote: > > Srikant > What do you mean with MTLS? Mutual auth and TLS? > > Enrico > > Il ven 20 dic 2019, 09:49 shrikant kalani ha > scritto: > >> Hi >> >> Can someone help me in configuring Zookeeper with MTLS ? >> >> Thanks >> Srikant Kalani >> >> Sent from my iPhone >>
Do Zookeeper 3.5.5 supports MTLS ?
Hi Can someone help me in configuring Zookeeper with MTLS ? Thanks Srikant Kalani Sent from my iPhone
