Looks like to me. I marked it as such. Patrick
On Mon, Jan 9, 2012 at 6:49 PM, Neha Narkhede <neha.narkh...@gmail.com> wrote: > Patrick, > > Looks like https://issues.apache.org/jira/browse/ZOOKEEPER-1356 is a > duplicate of 338 ? If yes, then I'll mark it to reflect the same. > > Thanks, > Neha > > On Mon, Jan 9, 2012 at 5:36 PM, Patrick Hunt <ph...@apache.org> wrote: >> dup of https://issues.apache.org/jira/browse/ZOOKEEPER-338 ? >> >> Patrick >> >> On Mon, Jan 9, 2012 at 3:17 PM, Ted Dunning <ted.dunn...@gmail.com> wrote: >>> Neha >>> >>> Filing a jira is a great way to further the discussion. >>> >>> Sent from my iPhone >>> >>> On Jan 9, 2012, at 15:33, Neha Narkhede <neha.narkh...@gmail.com> wrote: >>> >>>>>> If you just have machine names in a list that you pass in, then yes, we >>>> could re-resolve on every reconnect and you could just re-alias that name >>>> to a new IP. But you'll have to put in logic that will do that but not >>>> break people using DNS RR. >>>> >>>> Having a list of machine names that can be changed to point to new IPs >>>> seems reasonable too. To be able to do the upgrade without having to >>>> restart all clients, besides turning off DNS caching in the JVM, we >>>> still have to solve the problem of zookeeper client caching the IPs in >>>> code. Having 2 levels of DNS caching, one in the JVM and one in code >>>> (which cannot be turned off) doesn't look like a good idea. Unless I'm >>>> missing the purpose of such IP caching in zookeeper ? >>>> >>>>>> I realize that moving machines is difficult when you have lots of >>>>>> clients. >>>> I'm a bit surprised your admins can't maintain machine IP addresses on a >>>> machine move given a cluster of that complexity, though >>>> >>>> Its not like it can't be done, it definitely has quite some >>>> operational overhead. We are trying to brainstorm various approaches >>>> and come up with one that will involve the least overhead on such >>>> upgrades going forward. >>>> >>>> Having said that, seems like re-resolving host names in reconnect >>>> doesn't look like a bad idea, provided it doesn't break the DNS RR use >>>> case. If that sounds good, can I go ahead a file a JIRA for this ? >>>> >>>> Thanks, >>>> Neha >>>> >>>> On Mon, Jan 9, 2012 at 11:04 AM, Camille Fournier <cami...@apache.org> >>>> wrote: >>>>> We don't shuffle IPs after the initial resolution of IP addresses. >>>>> >>>>> In DNS RR, you resolve to a list of IPs, shuffle these, and then we round >>>>> robin through them trying to connect. If you re-resolve on every >>>>> round-robin, you have to put in logic to know which ones have changed and >>>>> somehow maintain that shuffle order or you aren't doing a fair back end >>>>> round robin, which people using the ZK client against DNS RR are relying >>>>> on >>>>> today. >>>>> >>>>> If you just have machine names in a list that you pass in, then yes, we >>>>> could re-resolve on every reconnect and you could just re-alias that name >>>>> to a new IP. But you'll have to put in logic that will do that but not >>>>> break people using DNS RR. >>>>> >>>>> I realize that moving machines is difficult when you have lots of clients. >>>>> I'm a bit surprised your admins can't maintain machine IP addresses on a >>>>> machine move given a cluster of that complexity, though. I also think that >>>>> if we're going to be putting special cases like this in we might just want >>>>> to go all the way to a pluggable reconnection scheme, but maybe that is >>>>> too >>>>> aggressive. >>>>> >>>>> C >>>>> >>>>> On Mon, Jan 9, 2012 at 1:51 PM, Neha Narkhede >>>>> <neha.narkh...@gmail.com>wrote: >>>>> >>>>>> Maybe I didn't express myself clearly. When I said DNS RR, I meant its >>>>>> simplest implementation which resolves a hostname to multiple IPs. >>>>>> >>>>>> Whatever method you use to map host names to IPs, the problem is that >>>>>> the zookeeper client code will always cache the IPs. So to be able to >>>>>> swap out a machine, all clients would have to be restarted, which if >>>>>> you have 100s of clients, is a major pain. If you want to move the >>>>>> entire cluster to new machines, this becomes even harder. >>>>>> >>>>>> I don't see why re-resolving host names to IPs in the reconnect logic >>>>>> is a problem for zookeeper, since you shuffle the list of IPs anyways. >>>>>> >>>>>> Thanks, >>>>>> Neha >>>>>> >>>>>> >>>>>> On Mon, Jan 9, 2012 at 10:31 AM, Camille Fournier <cami...@apache.org> >>>>>> wrote: >>>>>>> You can't sensibly round robin within the client code if you re-resolve >>>>>> on >>>>>>> every reconnect, if you're using dns rr. If that's your goal you'd want >>>>>>> a >>>>>>> list of dns alias names and re-resolve each hostname when you hit it on >>>>>>> reconnect. But that will break people using dns rr. >>>>>>> You can look into writing a pluggable reconnect logic into the zk >>>>>>> client, >>>>>>> that's what would be required to do this but at the end of the day >>>>>>> you'll >>>>>>> have to give your users special clients to make that work. >>>>>>> >>>>>>> C >>>>>>> On Jan 9, 2012 1:16 PM, "Neha Narkhede" <neha.narkh...@gmail.com> >>>>>> wrote: >>>>>>> >>>>>>>> I was reading through the client code and saw that zookeeper client >>>>>>>> caches the server IPs during startup and maintains it for the rest of >>>>>>>> its lifetime. If we go with the DNS RR approach or a load balancer >>>>>>>> approach, and later swap out a server with a new one ( with a new IP >>>>>>>> ), all clients would have to be restarted to be able to "forget" the >>>>>>>> old IP and see the new one. That doesn't look like a clean approach to >>>>>>>> such upgrades. One way of getting around this problem, is adding the >>>>>>>> resolution of host names to IPs in the "reconnect" logic in addition >>>>>>>> to the constructor. So when such upgrades happen and the client >>>>>>>> reconnects, it will see the new list of IPs, and wouldn't require to >>>>>>>> be restarted. >>>>>>>> >>>>>>>> Does this approach sound good or am I missing something here ? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Neha >>>>>>>> >>>>>>>> On Wed, Dec 21, 2011 at 7:21 PM, Camille Fournier <cami...@apache.org> >>>>>>>> wrote: >>>>>>>>> DNS RR is good. I had good experiences using that for my client >>>>>>>>> configs for exactly the reasons you are listing. >>>>>>>>> >>>>>>>>> On Wed, Dec 21, 2011 at 8:43 PM, Neha Narkhede < >>>>>> neha.narkh...@gmail.com> >>>>>>>> wrote: >>>>>>>>>> Thanks for the responses! >>>>>>>>>> >>>>>>>>>>>> How are your clients configured to find the zks now? >>>>>>>>>> >>>>>>>>>> Our clients currently use the list of hostnames and ports that >>>>>>>>>> comprise the zookeeper cluster. For example, >>>>>>>>>> zoo1:port1,zoo2:port2,zoo3:port3 >>>>>>>>>> >>>>>>>>>>>>> - switch DNS, >>>>>>>>>>> - wait for caches to die, >>>>>>>>>> >>>>>>>>>> This is something we thought about however, if I understand it >>>>>>>>>> correctly, doesn't JVM cache DNS entries forever until it is >>>>>> restarted >>>>>>>>>> ? We haven't specifically turned DNS caching off on our clients. So >>>>>>>>>> this solution would require us to restart the clients to see the new >>>>>>>>>> list of zookeeper hosts. >>>>>>>>>> >>>>>>>>>> Another thought is to use DNS RR and have the client zk url have one >>>>>>>>>> name that resolves to and returns a list of IPs to the zookeeper >>>>>>>>>> client. This has the advantage of being able to perform hardware >>>>>>>>>> migration without changing the client connection url, in the future. >>>>>>>>>> Do people have thoughts about using a DNS RR ? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Neha >>>>>>>>>> >>>>>>>>>> On Tue, Dec 20, 2011 at 1:06 PM, Ted Dunning <ted.dunn...@gmail.com> >>>>>>>> wrote: >>>>>>>>>>> In particular, aren't you using DNS names? If you are, then you can >>>>>>>>>>> >>>>>>>>>>> - expand the quorum with the new hardware on new IP addresses, >>>>>>>>>>> - switch DNS, >>>>>>>>>>> - wait for caches to die, >>>>>>>>>>> - restart applications without reconfig or otherwise force new >>>>>>>> connections, >>>>>>>>>>> - decrease quorum size again >>>>>>>>>>> >>>>>>>>>>> On Tue, Dec 20, 2011 at 12:26 PM, Camille Fournier < >>>>>> cami...@apache.org >>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> How are your clients configured to find the zks now? How many >>>>>> clients >>>>>>>> do >>>>>>>>>>>> you have? >>>>>>>>>>>> >>>>>>>>>>>> From my phone >>>>>>>>>>>> On Dec 20, 2011 3:14 PM, "Neha Narkhede" <neha.narkh...@gmail.com> >>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> As part of upgrading to Zookeeper 3.3.4, we also have to migrate >>>>>> our >>>>>>>>>>>>> zookeeper cluster to new hardware. I'm trying to figure out the >>>>>> best >>>>>>>>>>>>> strategy to achieve that with no downtime. >>>>>>>>>>>>> Here are some possible solutions I see at the moment, I could >>>>>> have >>>>>>>>>>>>> missed a few though - >>>>>>>>>>>>> >>>>>>>>>>>>> 1. Swap each machine out with a new machine, but with the same >>>>>>>> host/IP. >>>>>>>>>>>>> >>>>>>>>>>>>> Pros: No client side config needs to be changed. >>>>>>>>>>>>> Cons: Relatively tedious task for Operations >>>>>>>>>>>>> >>>>>>>>>>>>> 2. Add new machines, with different host/IPs to the existing >>>>>>>> cluster, >>>>>>>>>>>>> and remove the older machines, taking care to maintain the >>>>>> quorum at >>>>>>>>>>>>> all times >>>>>>>>>>>>> >>>>>>>>>>>>> Pros: Easier for Operations >>>>>>>>>>>>> Cons: Client side configs need to be changed and clients need to >>>>>> be >>>>>>>>>>>>> restarted/bounced. Another problem is having a large quorum for >>>>>>>>>>>>> sometime (potentially 9 nodes). >>>>>>>>>>>>> >>>>>>>>>>>>> 3. Hide the new cluster behind either a Hardware load balancer >>>>>> or a >>>>>>>>>>>>> DNS server resolving to all host ips. >>>>>>>>>>>>> >>>>>>>>>>>>> Pros: Makes it easier to move hardware around in the future >>>>>>>>>>>>> Cons: Possible timeout issues with load balancers messing with >>>>>>>>>>>>> zookeeper functionality or performance >>>>>>>>>>>>> >>>>>>>>>>>>> Read this and found it helpful - >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>> >>>>>> http://apache.markmail.org/message/44tbj53q2jufplru?q=load+balancer+list:org%2Eapache%2Ehadoop%2Ezookeeper-user&page=1 >>>>>>>>>>>>> But would like to hear from the authors and the users who might >>>>>> have >>>>>>>>>>>>> tried this in a real production setup. >>>>>>>>>>>>> >>>>>>>>>>>>> I'm very interested in finding a long term solution for masking >>>>>> the >>>>>>>>>>>>> zookeeper host names. Any inputs here are appreciated ! >>>>>>>>>>>>> >>>>>>>>>>>>> In addition to this, it will also be great to know what people >>>>>> think >>>>>>>>>>>>> about options 1 and 2, as a solution for hardware changes in >>>>>>>>>>>>> Zookeeper. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Neha >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>> >>>>>>
