I would say if in doubt add a safety. (a config parameter to turn it off). Cost is almost zero and worst case it will just give us peace of mind. ;-)
Patrick On Mon, Mar 21, 2016 at 9:41 PM, Alexander Shraer <[email protected]> wrote: > ok, thanks for the suggestion, I'll look into it. For reconfig I think its > pretty clear that its an admin > functionality. I just always imagined that its controlled via acls, but I > understand > the concerns now. > > getConfig returns the dynamic config (list of all servers with all ports > and quorum system if defined) > and has an option to filter that info and just return the server connection > string (server and client port only). > > > On Mon, Mar 21, 2016 at 9:32 PM, Patrick Hunt <[email protected]> wrote: > >> On Mon, Mar 21, 2016 at 9:14 PM, Alexander Shraer <[email protected]> >> wrote: >> > I don't think that getConfig should be an admin functionality. It is >> > essential for client-side re-balancing >> > that we implemented (all clients shoudl be able to detect configuration >> > changes via getConfig). It could >> > be hidden somewhat by defining higher-level re-balancing >> > policies (ZOOKEEPER-2016) >> > but there hasn't been enough progress on that. Perhaps instead getConfig >> > should be controlled >> > by a separate flag ? >> > >> >> I believe that the Hadoop community has something we could use: >> >> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/InterfaceClassification.html >> (whether through annotations or just documenting it in the API javadoc) >> >> e.g. we could list getConfig as public/unstable for example and still >> ship it as GA. That would mark it as something that could change re >> API policy. >> >> Is the entire config exposed through getConfig? If so then we might >> want to enable/disable it with a flag similar to reconfig. Might be >> safer to just do that if we're not sure. >> >> >> Re classification - we could do the same thing with reconfig, but I >> think that would be a mistake. If we feel strongly where it should >> live long term we should just move it now. >> >> Patrick >> >> > >> > On Mon, Mar 21, 2016 at 9:04 PM, Patrick Hunt <[email protected]> wrote: >> > >> >> On Mon, Mar 21, 2016 at 8:52 PM, Alexander Shraer <[email protected]> >> >> wrote: >> >> > Hi Patrick, Flavio, >> >> > >> >> > Since there seems to be consensus on this, I can add this flag, unless >> >> > someone else wants to. I assume that getConfig should still work >> >> regardless >> >> > of the flag ? is there a security concern with clients knowing the >> list >> >> of >> >> > servers? >> >> > >> >> >> >> We've always hidden that detail from users. We don't even expose which >> >> server you're connected to today. I remember Ben (and perhaps Flavio?) >> >> highlighting this as important to maintain although I'm not super >> >> familiar with the specifics on why. It made sense to me though from >> >> the perspective that we don't want clients to make assumptions that >> >> probably shouldn't. >> >> >> >> My thinking is that we should 1) add a config option to enable >> >> reconfig (off by default), 2) move reconfig specific functionality >> >> from ZooKeeper.java (including getconfig) into an "admin" package, >> >> within say a class ZooKeeperAdmin, 3) document/test use of ACLs for >> >> when folks do want to enable reconfig and are also worried about auth. >> >> (e.g. turn on kerb) >> >> >> >> Again, I don't see any of this as a quality issue personally. As such >> >> I don't see why any of this (1-3) should hold up a 3.5.2-alpha if we >> >> were interested in doing such a release. Adjusting the API should be >> >> done before we move to "beta" though. Although that seems like a >> >> pretty mechanical (eclipse/idea) type refactoring? >> >> >> >> Patrick >> >> >> >> > Cheers, >> >> > Alex >> >> > On Mar 21, 2016 8:34 PM, "Patrick Hunt" <[email protected]> wrote: >> >> > >> >> >> On Thu, Mar 17, 2016 at 4:08 PM, Flavio Junqueira <[email protected]> >> >> wrote: >> >> >> > I gotta say that I'm not super excited about this option, but for >> some >> >> >> reason I ended up carrying the flag. To recap, I just raised this >> option >> >> >> because it seems that there are folks interested in features in 3.5 >> like >> >> >> SSL and not necessarily in reconfiguration. SSL is important and to >> take >> >> >> Kafka as an example, it sucks that we can't have a whole set up using >> >> SSL. >> >> >> For ZK, the real questions are: >> >> >> > >> >> >> > 1- how fast can we make 3.5 stable? >> >> >> > 2- would it be faster if we have a way of disabling >> reconfiguration? >> >> >> > 3- would enough users care about a stable 3.5 that has >> reconfiguration >> >> >> disabled? >> >> >> > >> >> >> > It is taking a long time to get 3.5 to beta. There has been some >> good >> >> >> activity around 3.5.2 release, which is a great step, but it is >> unclear >> >> >> when 3.5.3 is going to come and if we will be able to make 3.5 beta >> >> then. >> >> >> Frankly, disabling reconfiguration sounds undesirable because it is >> an >> >> >> important feature, but I currently don't use it in production, so >> from a >> >> >> practical point of view, I can go both ways. Whether we go through >> the >> >> >> trouble of doing 2 depends on users interested in that option and >> folks >> >> >> willing to implement it. >> >> >> > >> >> >> > To answer your question, Powell, my pseudo-proposal is kind of a >> funny >> >> >> option because once the feature is stable, then we wouldn't need a >> >> switch >> >> >> any longer, so there is not need of a deprecation path, we just start >> >> >> ignoring it from the first beta release. Until it is beta, I'd say >> that >> >> >> default is disabled. >> >> >> >> >> >> I would argue that we need this even when it does become stable. To >> me >> >> >> this is not a quality issue so much as it is an auth issue. We want >> to >> >> >> make it simple for folks to run a vanilla/old ZK cluster and not >> worry >> >> >> about the security implications of having reconfig enabled. >> >> >> >> >> >> Patrick >> >> >> >> >> >> > >> >> >> > -Flavio >> >> >> > >> >> >> >> On 17 Mar 2016, at 17:44, powell molleti >> <[email protected] >> >> > >> >> >> wrote: >> >> >> >> >> >> >> >> Hi Flavio, >> >> >> >> >> >> >> >> Generally do config options and command line args come under the >> same >> >> >> SLA as API?. I was assuming as such hence my question. Perhaps if the >> >> >> expectation is that this config option is temporary from get go then >> >> may be >> >> >> it is ok. The default for re-config support will be enabled or >> >> disabled?. >> >> >> >> >> >> >> >> I am just thinking from provisioning point of view when people >> >> generate >> >> >> config options etc. >> >> >> >> >> >> >> >> Thanks >> >> >> >> Powell. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Thursday, March 17, 2016 12:12 AM, Flavio Junqueira < >> >> [email protected]> >> >> >> wrote: >> >> >> >> Hi Powell, >> >> >> >> >> >> >> >> I was thinking config file and system property server side. What's >> >> your >> >> >> concern with compatibility? The API itself wouldn't change, but the >> >> config >> >> >> option wouldn't exist in previous versions and would not exist >> either in >> >> >> later stable versions of 3.5. >> >> >> >> >> >> >> >> -Flavio >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>> On 16 Mar 2016, at 22:08, powell molleti >> >> <[email protected]> >> >> >> wrote: >> >> >> >>> >> >> >> >>> Will this option be supplied via config file/args/API?. Will this >> >> >> option be a temporary thing i.e what about its compatibility?. >> >> >> >>> >> >> >> >>> thanks >> >> >> >>> Powell. >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> On Wednesday, March 16, 2016 2:46 PM, Flavio Junqueira < >> >> [email protected]> >> >> >> wrote: >> >> >> >>> The main issue to sort out is stability of the API. There is a >> >> >> security concern around the fact that clients can freely reconfigure >> the >> >> >> ensemble. If we follow the plan that Pat proposed some time ago: >> >> >> >>> >> >> >> >>> >> >> >> >> >> >> https://mail-archives.apache.org/mod_mbox/zookeeper-dev/201407.mbox/%3CCANLc_9KG6-Dhm=wwfuwzniogk70pg+ihmhpigyfjdslf9-e...@mail.gmail.com%3E >> >> >> < >> >> >> >> >> >> https://mail-archives.apache.org/mod_mbox/zookeeper-dev/201407.mbox/%3CCANLc_9KG6-Dhm=wwfuwzniogk70pg+ihmhpigyfjdslf9-e...@mail.gmail.com%3E >> >> >> > >> >> >> >>> >> >> >> >>> Locking the API is the main step to move it to beta. Sorting out >> >> bugs >> >> >> is definitely necessary, but it isn't the main thing that is keeping >> >> 3.5 in >> >> >> alpha. >> >> >> >>> >> >> >> >>> About making it experimental, I was raising the option of having >> a >> >> >> switch that disables the API calls, not the code. The reason why that >> >> could >> >> >> work is that anyone using 3.5 who uses the "experimental" API must >> >> explicit >> >> >> turn on the switch and enable the calls. If they do it, they need to >> be >> >> >> aware that the API can change. >> >> >> >>> >> >> >> >>> I must say that I haven't really looked closely into doing it, >> and >> >> I'm >> >> >> not even entirely convinced that this is a good idea, but since Jason >> >> >> raised the point, I'm exploring options. >> >> >> >>> >> >> >> >>> -Flavio >> >> >> >>> >> >> >> >>> >> >> >> >>>> On 16 Mar 2016, at 20:59, Alexander Shraer <[email protected]> >> >> wrote: >> >> >> >>>> >> >> >> >>>> Looking at the list of ~50 blocker and critical bugs in >> ZooKeeper, >> >> >> only 3-4 >> >> >> >>>> are related to reconfig. Given this, and the fact that it is >> run in >> >> >> >>>> production since 2012 in multiple companies, I don't think its >> more >> >> >> >>>> unstable than any other part of ZooKeeper. >> >> >> >>>> >> >> >> >>>> There are multiple reconfig-related bugs that turned out really >> >> >> difficult >> >> >> >>>> to debug without access to the actual system or at least to the >> >> Hudson >> >> >> >>>> machines where some tests are failing. In the past, Michi and I >> >> >> physically >> >> >> >>>> went to Hortonworks to debug one such issue, but this is of >> course >> >> >> not a >> >> >> >>>> good method, and we weren't able to arrange such a visit again. >> >> >> >>>> >> >> >> >>>> Regarding making it optional - the reconfig logic has several >> >> >> different >> >> >> >>>> parts, some would be really difficult to disable using a >> >> configuration >> >> >> >>>> parameter. But the actual dynamic expansion of the cluster is >> >> >> triggered by >> >> >> >>>> the reconfig command, so it should not affect users who don't >> >> invoke >> >> >> it. >> >> >> >>>> >> >> >> >>>> On Wed, Mar 16, 2016 at 1:09 PM, Flavio P JUNQUEIRA < >> >> [email protected]> >> >> >> wrote: >> >> >> >>>> >> >> >> >>>>> I suppose we could give it a try. How do other folks feel about >> >> it? >> >> >> >>>>> >> >> >> >>>>> -Flavio >> >> >> >>>>> On 16 Mar 2016 19:52, "Jason Rosenberg" <[email protected]> >> wrote: >> >> >> >>>>> >> >> >> >>>>>> So, you could enable the dynamic reconfiguration feature >> behind a >> >> >> config >> >> >> >>>>>> option, and document that it should only be enabled >> >> experimentally, >> >> >> use >> >> >> >>>>> at >> >> >> >>>>>> your own risk. Keep it off by default. Allow only static >> >> config by >> >> >> >>>>>> default, until it's stable? >> >> >> >>>>>> >> >> >> >>>>>> Jason >> >> >> >>>>>> >> >> >> >>>>>> On Wed, Mar 16, 2016 at 3:34 PM, Flavio Junqueira < >> >> [email protected]> >> >> >> >>>>> wrote: >> >> >> >>>>>> >> >> >> >>>>>>> Hi Jason, >> >> >> >>>>>>> >> >> >> >>>>>>> The consumer in Kafka is pretty independent from the core >> >> >> (brokers), >> >> >> >>>>>>> that's how that project manages to make such a separation. We >> >> don't >> >> >> >>>>> have >> >> >> >>>>>>> the same with ZooKeeper as the feature we are talking about >> is >> >> >> part of >> >> >> >>>>>> the >> >> >> >>>>>>> server and the only way I see of doing what you say is to >> turn >> >> off >> >> >> >>>>>>> features. More specifically, we'd need to disable the >> reconfig >> >> API >> >> >> and >> >> >> >>>>> do >> >> >> >>>>>>> not allow any change to the configuration, even though the >> code >> >> is >> >> >> >>>>> there. >> >> >> >>>>>>> >> >> >> >>>>>>> Reconfig here refers to the ability of changing the >> >> configuration >> >> >> of an >> >> >> >>>>>>> ensemble (e.g., changing the set of servers). >> >> >> >>>>>>> >> >> >> >>>>>>> -Flavio >> >> >> >>>>>>> >> >> >> >>>>>>>> On 16 Mar 2016, at 19:14, Jason Rosenberg <[email protected] >> > >> >> >> wrote: >> >> >> >>>>>>>> >> >> >> >>>>>>>> So, it would seem sensible to me to have a release where all >> >> >> features >> >> >> >>>>>> are >> >> >> >>>>>>>> stable, except where noted. E.g. mark certain features as >> only >> >> >> >>>>> 'alpha >> >> >> >>>>>>>> quality', e.g. the 're-config feature'. (I assume it's >> >> possible >> >> >> to >> >> >> >>>>>>> happily >> >> >> >>>>>>>> use 3.5.X without exercising the unstable re-config bits?). >> >> >> >>>>>>>> >> >> >> >>>>>>>> There's precedent for doing this sort of thing in other >> >> projects, >> >> >> >>>>> e.g. >> >> >> >>>>>> in >> >> >> >>>>>>>> Kafka, they've had several release where a new "Consumer >> API" >> >> is >> >> >> >>>>>> shipped >> >> >> >>>>>>>> that is available for beta-testing, but you can still just >> use >> >> the >> >> >> >>>>>> older >> >> >> >>>>>>>> stable consumer api, etc. >> >> >> >>>>>>>> >> >> >> >>>>>>>> Jason >> >> >> >>>>>>>> >> >> >> >>>>>>>> On Wed, Mar 16, 2016 at 2:01 PM, powell molleti >> >> >> >>>>>>> <[email protected] >> >> >> >>>>>>>>> wrote: >> >> >> >>>>>>>> >> >> >> >>>>>>>>> Hi Doug, >> >> >> >>>>>>>>> Is 3.5 being an alpha release preventing you from using >> it?. >> >> Or >> >> >> have >> >> >> >>>>>> you >> >> >> >>>>>>>>> run into issues with it?. In general perhaps ZK 3.5 being >> >> >> labeled as >> >> >> >>>>>>> alpha >> >> >> >>>>>>>>> might not be fair, since it is far more stable then what >> most >> >> >> people >> >> >> >>>>>>>>> associate an alpha release to be. >> >> >> >>>>>>>>> Perhaps if you do not use re-config feature may be it will >> >> just >> >> >> work >> >> >> >>>>>> for >> >> >> >>>>>>>>> you?. >> >> >> >>>>>>>>> There are many examples of 3.5.X being used in productions >> >> from >> >> >> my >> >> >> >>>>>>> limited >> >> >> >>>>>>>>> knowledge. >> >> >> >>>>>>>>> ThanksPowell. >> >> >> >>>>>>>>> >> >> >> >>>>>>>>> On Wednesday, March 16, 2016 2:44 AM, Flavio Junqueira < >> >> >> >>>>>>> [email protected]> >> >> >> >>>>>>>>> wrote: >> >> >> >>>>>>>>> >> >> >> >>>>>>>>> >> >> >> >>>>>>>>> None of us expected the reconfig changes to take this long >> to >> >> >> >>>>>> stabilize. >> >> >> >>>>>>>>> Until we get there, I don't think we can do anything else >> with >> >> >> 3.5. >> >> >> >>>>>> The >> >> >> >>>>>>>>> best bet we have is to work harder to bring 3.5 into a >> stable >> >> >> rather >> >> >> >>>>>>> than >> >> >> >>>>>>>>> trying to work around it. >> >> >> >>>>>>>>> >> >> >> >>>>>>>>> There are lots of people interested in seeing 3.5 stable, >> and >> >> if >> >> >> we >> >> >> >>>>>> get >> >> >> >>>>>>>>> everyone to contribute more patches and code reviews, we >> >> should >> >> >> be >> >> >> >>>>>> able >> >> >> >>>>>>> to >> >> >> >>>>>>>>> do it sooner. After all, it is a community based effort, so >> >> the >> >> >> >>>>>>> community >> >> >> >>>>>>>>> shouldn't rely on just 2-3 people doing the work. >> >> >> >>>>>>>>> >> >> >> >>>>>>>>> -Flavio >> >> >> >>>>>>>>> >> >> >> >>>>>>>>>> On 15 Mar 2016, at 17:28, Chris Nauroth < >> >> >> [email protected]> >> >> >> >>>>>>>>> wrote: >> >> >> >>>>>>>>>> >> >> >> >>>>>>>>>> Doug, I forgot to respond to your question about 3.4. >> Since >> >> >> 3.4 is >> >> >> >>>>>> the >> >> >> >>>>>>>>>> stable maintenance line, we are very conservative about >> >> >> >>>>> back-porting >> >> >> >>>>>> to >> >> >> >>>>>>>>>> it. Our policy is to limit back-ports to critical bug >> fixes >> >> and >> >> >> >>>>> not >> >> >> >>>>>>>>>> introduce any new features in the 3.4 line. This is a >> >> matter of >> >> >> >>>>>>> managing >> >> >> >>>>>>>>>> risk. >> >> >> >>>>>>>>>> >> >> >> >>>>>>>>>> Jason, your question about release cadence is a fair >> one. If >> >> >> it's >> >> >> >>>>>> any >> >> >> >>>>>>>>>> consolation, we are now taking the approach of trying to >> >> limit >> >> >> the >> >> >> >>>>>>> scope >> >> >> >>>>>>>>>> of anything new introduced in 3.5 too. That would allow >> us >> >> to >> >> >> >>>>> focus >> >> >> >>>>>> on >> >> >> >>>>>>>>>> stabilization: resolving blocker bugs and freezing public >> >> >> APIs. I >> >> >> >>>>>>> think >> >> >> >>>>>>>>>> this will help us accelerate the releases into beta and >> >> eventual >> >> >> >>>>> GA. >> >> >> >>>>>>>>>> >> >> >> >>>>>>>>>> I am new to ZooKeeper release management, so I'd like to >> hear >> >> >> >>>>>> thoughts >> >> >> >>>>>>>>>> from more experienced committers and PMC members about >> your >> >> >> >>>>> proposal >> >> >> >>>>>> to >> >> >> >>>>>>>>>> try to cut a stable release for a limited subset of what >> is >> >> in >> >> >> >>>>>>> branch-3.5 >> >> >> >>>>>>>>>> now. My instinct is that it would be challenging to >> >> cherry-pick >> >> >> >>>>> out >> >> >> >>>>>>>>>> pieces of branch-3.5 piecemeal at this point. This would >> >> become >> >> >> >>>>>>> another >> >> >> >>>>>>>>>> release line for an already resource-constrained volunteer >> >> >> staff to >> >> >> >>>>>>>>>> manage. I'd prefer to dedicate those limited resources to >> >> >> overall >> >> >> >>>>>> 3.5 >> >> >> >>>>>>>>>> stabilization. Also, a 3.5 release in which certain >> features >> >> >> >>>>>>> "vanished" >> >> >> >>>>>>>>>> because of not meeting some stability criteria would be >> >> >> >>>>> undesirable. >> >> >> >>>>>>>>>> >> >> >> >>>>>>>>>> --Chris Nauroth >> >> >> >>>>>>>>>> >> >> >> >>>>>>>>>> >> >> >> >>>>>>>>>> >> >> >> >>>>>>>>>> >> >> >> >>>>>>>>>> On 3/15/16, 10:12 AM, "Jason Rosenberg" <[email protected] >> > >> >> >> wrote: >> >> >> >>>>>>>>>> >> >> >> >>>>>>>>>>> Chris, >> >> >> >>>>>>>>>>> >> >> >> >>>>>>>>>>> Can you say whether some parts of 3.5.X are more stable >> than >> >> >> >>>>> others >> >> >> >>>>>>>>> (e.g. >> >> >> >>>>>>>>>>> if we don't care about certain new features, is it >> >> relatively >> >> >> >>>>>> stable)? >> >> >> >>>>>>>>>>> Would it be possible to cut out a version that only has >> the >> >> >> bits >> >> >> >>>>> we >> >> >> >>>>>>>>> think >> >> >> >>>>>>>>>>> are stable (and release that)? >> >> >> >>>>>>>>>>> >> >> >> >>>>>>>>>>> From that timeline, and the historic release cadence, it >> >> would >> >> >> >>>>> seem >> >> >> >>>>>> to >> >> >> >>>>>>>>> be >> >> >> >>>>>>>>>>> a >> >> >> >>>>>>>>>>> years away before we get to the stable release? >> >> >> >>>>>>>>>>> >> >> >> >>>>>>>>>>> Thanks, >> >> >> >>>>>>>>>>> >> >> >> >>>>>>>>>>> Jason >> >> >> >>>>>>>>>>> >> >> >> >>>>>>>>>>> On Tue, Mar 15, 2016 at 1:06 PM, Chris Nauroth < >> >> >> >>>>>>>>> [email protected]> >> >> >> >>>>>>>>>>> wrote: >> >> >> >>>>>>>>>>> >> >> >> >>>>>>>>>>>> Hello Doug, >> >> >> >>>>>>>>>>>> >> >> >> >>>>>>>>>>>> Thanks for your interest in the SSL feature! >> >> >> >>>>>>>>>>>> >> >> >> >>>>>>>>>>>> At this point, I think we're still pretty far away from >> >> >> >>>>> declaring a >> >> >> >>>>>>>>>>>> stable >> >> >> >>>>>>>>>>>> release in the 3.5 line. I don't think we're close >> enough >> >> >> that >> >> >> >>>>>>> anyone >> >> >> >>>>>>>>>>>> can >> >> >> >>>>>>>>>>>> offer a reliable ETA. This is an earlier thread that >> >> >> describes >> >> >> >>>>> the >> >> >> >>>>>>>>>>>> high-level strategy for release planning in the 3.5 >> line: >> >> >> >>>>>>>>>>>> >> >> >> >>>>>>>>>>>> https://s.apache.org/ADK1 >> >> >> >>>>>>>>>>>> >> >> >> >>>>>>>>>>>> The next step is a 3.5.2-alpha release. We're working >> on >> >> >> >>>>>> resolving a >> >> >> >>>>>>>>>>>> few >> >> >> >>>>>>>>>>>> more blockers before we produce a release candidate. >> >> >> Hopefully >> >> >> >>>>>> that >> >> >> >>>>>>>>>>>> will >> >> >> >>>>>>>>>>>> get done in the next few weeks. >> >> >> >>>>>>>>>>>> >> >> >> >>>>>>>>>>>> --Chris Nauroth >> >> >> >>>>>>>>>>>> >> >> >> >>>>>>>>>>>> >> >> >> >>>>>>>>>>>> >> >> >> >>>>>>>>>>>> >> >> >> >>>>>>>>>>>> On 3/15/16, 9:39 AM, "Doug" <[email protected]> >> wrote: >> >> >> >>>>>>>>>>>> >> >> >> >>>>>>>>>>>>> I know it's only been a few months, but I was >> wondering if >> >> >> there >> >> >> >>>>>>> was a >> >> >> >>>>>>>>>>>>> ballpark release date for a stable version of 3.5.1. >> Or is >> >> >> there >> >> >> >>>>>> any >> >> >> >>>>>>>>>>>>> chance >> >> >> >>>>>>>>>>>>> the SSL feature would be added to 3.4.8? Just another >> >> person >> >> >> >>>>>> looking >> >> >> >>>>>>>>> to >> >> >> >>>>>>>>>>>>> have >> >> >> >>>>>>>>>>>>> that feature in a stable version. Thanks for all you >> do! >> >> :) >> >> >> >>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>> -- >> >> >> >>>>>>>>>>>>> View this message in context: >> >> >> >>>>>>>>>>>>> >> >> >> >>>>>>>>>>>> >> >> >> >>>>>>>>>>>> >> >> >> >>>>>>>>> >> >> >> >>>>>>> >> >> >> >>>>>> >> >> >> >>>>> >> >> >> >> >> >> http://zookeeper-user.578899.n2.nabble.com/Zookeeper-with-SSL-release-dat >> >> >> >>>>>>>>>>>> e >> >> >> >>>>>>>>>>>>> -tp7581744p7582136.html >> >> >> >>>>>>>>>>>>> Sent from the zookeeper-user mailing list archive at >> >> >> Nabble.com. >> >> >> >>>>>>>>>>>>> >> >> >> >>>>>>>>>>>> >> >> >> >>>>>>>>>>>> >> >> >> >>>>>>>>>> >> >> >> >>>>>>>>> >> >> >> >>>>>>>>> >> >> >> >>>>>>>>> >> >> >> >>>>>>>>> >> >> >> >>>>>>> >> >> >> >>>>>>> >> >> >> >>>>>> >> >> >> >>>>> >> >> >> > >> >> >> >> >> >>
