Re: Multi-master (HA)
On 5/9/2014 3:01 PM, John Wobus wrote: ...if anyone has specific thoughts on how to make this sort of thing easier in BIND -- even just at the level of "boy, it irritates me that I can't make BIND do " -- such comments will fall on welcoming ears. I agree that it would be nice if effort were made into making flipping masters straight-forward, i.e., not require a change to every zone declaration and not force the operator to deal with zone files that suddenly need to switch between binary and ascii. (There may be good ways to do this now that I'm unaware of.) Where is the line drawn these days between DNS management protocols and provisioning protocols? Because, I've long thought the idea of feeding a config (i.e. the contents of a named.conf file) to a "named" instance via "rndc" would be an easy and secure way of quickly reconfiguring it to a different role (e.g. from master to slave, or _vice_versa_, for a whole bunch of views/zones in one fell swoop). Since the config is in a very regular, structured format, I'm sure some sort of encoding and/or compression could be employed to make the actual data transfer size fairly compact. The only big gotcha that comes to mind here is if the named.conf is segmented via include files with different access privileges (e.g. not letting key definitions be world-readable), that segmentation/protection would need to be preserved on the receiving side. - Kevin ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Multi-master (HA)
...if anyone has specific thoughts on how to make this sort of thing easier in BIND -- even just at the level of "boy, it irritates me that I can't make BIND do " -- such comments will fall on welcoming ears. I agree that it would be nice if effort were made into making flipping masters straight-forward, i.e., not require a change to every zone declaration and not force the operator to deal with zone files that suddenly need to switch between binary and ascii. (There may be good ways to do this now that I'm unaware of.) (I've wondered why bind doesn't simply write an ascii copy of the zone file in addition to the binary copy.) Running multiple dynamic-dns masters would be absolutely fantastic except of course when it didn't work. Seems like a reason to have multiple masters is to handle the case where some are unreachable, in which case keeping them in synch becomes interesting. If the main point is to eliminate single points of failure, a "three masters with quorum" system might serve the purpose. I like the idea of configuring zone information in a zone, and think it would be fun to be on the team brainstorming how to guard against sneaky config attacks. John Wobus Cornell University IT ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Multi-master (HA)
On 2014-05-08 07:13, Barry S. Finkel wrote: On 2014-05-07 15:06, Lawrence K. Chen, P.Eng. wrote: OTOH, the idea of multi-master is intriguing.the only down side I see, is hat I >> have one really powerful server for my current master(Sun Fire X4170)and my >> other servers are weak leftoversjust passed EOL last year. >> And, have all the servers doing full DNSSEC signing could be >> interesting. It also raises the question of how does the outside world cope with all the servers >> having identical zones...signed on slightly different times, etc. >> (especially since I'm using unix timestamp for zone serialavoids >> issues of multiple admins incrementing serial without >> noticing others and/or collisions with DNSSEC's >> incrementing of serials.) Dave Warren replied: I wouldn't expect any real issues here, Windows DNS has done multimaster DNS since Windows 2000. In the case of Windows, dynamic updates (via client or GUI) can be done at any location, the serial numbers are incremented automatically, but the zones and servers may vary from each other for a brief period of time. So for example, DC1 and DC2 may start with serial 100, DC1 will receive 2 changes and be up to 102, DC2 will give 5 different changes and be up to 105. When Active Directory synchronization happens outside of DNS, the two sides merge changes together, and set the serial to the higher of the two plus one, so the serial would be 106. To the outside world, records can appear/disappear for a brief period while the servers drift out of sync, similar to what could happen in a BIND configuration without notifies as resolvers hit the two DNS servers round-robin. The only thing that causes issues is if you use DNS to create a non-Active Directory slave. BIND will throw errors because it will see serial 100, 101, 102, then get a notify from the second server about 101. However, the slave will still sync up once the AD servers sync to 106. The fix here is to configure BIND to only slave off of one master or the other, not both. While there might be other factors involved in turning BIND into a true multi-master solution, I wouldn't expect zones drifting out of sync or having minor differences to be a big factor since it happens in the wild already. As I have written before, see MS article 282826. If one is going to slave an MS AD DNS server, one has to choose ONLY ONE AD DNS Server as a master. As I see it, there is no way that AD can choose a zone serial number from among all of the AD DNS Servers. Assuming that a zone has the same contents and same serial number, say n, on all Domain Controllers. Then, one Windows machine sends a DDNS update for the zone to DC1 at the same time that another Windows machine sends a different DDNS update for that zone to DC2. Now, each DC has serial number n+1 and different contents. When AD synchronizes the zone contents and serial number under the covers, what serial number can it choose? It can't choose n+1, as that serial number has already been used. It can't choose n+2, as it does not know if another DDNS for the same zone has arrived before the synchronization has taken place. n+2 works fine, the situation is no worse off than it was with two servers each at n+1 and being slightly out of sync. At the n+2 step, the zones are closer to being in sync then they were. The logic that MS DNS uses is to always set the serial number to the highest seen anywhere +1 and it works very well internally. Even if you don't follow the advice in 282826, it actually works surprisingly well; as AD syncs up (which tends to happen very quickly for DNS servers in the same site, slower with intra-site replication), the changes merge together, the serial increments and BIND gets the latest zone. You obviously have to use AXFR rather than IFXR, and you have to accept that newly added records will appear and disappear from the BIND zone when/if BIND flips between AD masters, but the effects are understandable and manageable. (I'm not advocating slaving off of multiple AD masters, I agree completely with 282826 -- it's a dumb idea. But I've seen it done and if you ignore BIND's logs and understand that newly added records need to propagate before they will exist reliably, it works well in production) But again, the point of this isn't "how to integrate MS DNS and BIND", it's "What happens, in the real world, if multi-master authoritative servers were to serve ever-so-slightly-different-versions of the same zone with the same serial", and the answer is that this is already battle-tested in the real world and it works very well, outside of slaves which aren't aware of this design or aren't part of the multi-master configuration. IIRC, 282826 says that if a DC is not used as a master for a BIND slave, then its zone serial number is not important. Indeed -- And that's my point, the situation where slightly different versions of the same zone are being served b
Re: Multi-master (HA)
I wouldn't say we migrated in that direction due to anything other then lack of good options. What BIND is missing is the concept of an update master. Augment BIND with the following: * Each master is aware of the other masters. * One master is defined as an update master (rndc control?) * Each master knows all the configurations necessary to act as a slave to the update master * Each master knows all the configurations necessary to be the update master. With the above, it would become relatively trivial to simply issue a directive and have the servers change their roles. If the update master is isolated, the directive must be able to be accepted at one of the other masters so that it can become the update master. When the isolation ends, the update master must realize it's new state and demote itself cleanly. I am doing this manually by having the zone configurations hold the masters list as well as update policies. To convert, the only lines that get changed are the "type", "masters" and "update-policy" stanzas. They get (un)commented as appropriate and then bind reloaded. The one trick I had to pay attention to is that when making the update master a slave master, I needed to touch all the zone files to prevent bind from immediately expiring them. It is also necessary to issue rndc refresh commands to the new slave to force it to perform SOA checks against the new update master. Otherwise, in the case of isolation, it won't bother to update it's zones until the next refresh cycle ends. -- John On 5/8/2014 7:32 AM, Tony Finch wrote: A few thoughts... The DNS protocol is already pretty good at replicating zone data - see for instance John Wingenbach's message in which he describes how their deployment gradually converged on a fairly standard architecture :-) I think multi-master makes most sense if the primary master uses DNS UPDATE for zone edits (and use raw file format), to minimize the differences between the primary and the secondaries. You probably want to ensure update forwarding is allowed, so that update clients do not have to worry so much about finding the current primary master. When a secondary takes over as primary it will need to update the SOA MNAME to point to itself so updates go to the right place. Most of the problem is actually one of remote configuration management: promoting a secondary to a primary is not all that different from setting up the secondary in the first place or making other co-ordinated changes. For instance it would be nice to be able to set up a zone once on the primary and have it automatically provisioned on the secondaries. I like Phil Mayers' zone-template idea, which might make it easier to flip from secondary to primary, as well as reducing the size and ensuring the consistency of large configs. Metazones are a tempting idea but the details get yucky the more of BIND's features you want to support. Also I am rather wary about the idea of putting secrets in a DNS zone; if you have an out-of-band way of distributing them it makes sense to use the same channel for the rest of the configuration. (http://ci.nii.ac.jp/naid/110007502948 - Vixie's metazones paper.) Tony. ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Multi-master (HA)
On 2014-05-07 15:06, Lawrence K. Chen, P.Eng. wrote: OTOH, the idea of multi-master is intriguing.the only down side I see, is hat I >> have one really powerful server for my current master(Sun Fire X4170)and my >> other servers are weak leftoversjust passed EOL last year. >> And, have all the servers doing full DNSSEC signing could be >> interesting. It also raises the question of how does the outside world cope with all the servers >> having identical zones...signed on slightly different times, etc. >> (especially since I'm using unix timestamp for zone serialavoids >> issues of multiple admins incrementing serial without >> noticing others and/or collisions with DNSSEC's >> incrementing of serials.) Dave Warren replied: I wouldn't expect any real issues here, Windows DNS has done multimaster DNS since Windows 2000. In the case of Windows, dynamic updates (via client or GUI) can be done at any location, the serial numbers are incremented automatically, but the zones and servers may vary from each other for a brief period of time. So for example, DC1 and DC2 may start with serial 100, DC1 will receive 2 changes and be up to 102, DC2 will give 5 different changes and be up to 105. When Active Directory synchronization happens outside of DNS, the two sides merge changes together, and set the serial to the higher of the two plus one, so the serial would be 106. To the outside world, records can appear/disappear for a brief period while the servers drift out of sync, similar to what could happen in a BIND configuration without notifies as resolvers hit the two DNS servers round-robin. The only thing that causes issues is if you use DNS to create a non-Active Directory slave. BIND will throw errors because it will see serial 100, 101, 102, then get a notify from the second server about 101. However, the slave will still sync up once the AD servers sync to 106. The fix here is to configure BIND to only slave off of one master or the other, not both. While there might be other factors involved in turning BIND into a true multi-master solution, I wouldn't expect zones drifting out of sync or having minor differences to be a big factor since it happens in the wild already. As I have written before, see MS article 282826. If one is going to slave an MS AD DNS server, one has to choose ONLY ONE AD DNS Server as a master. As I see it, there is no way that AD can choose a zone serial number from among all of the AD DNS Servers. Assuming that a zone has the same contents and same serial number, say n, on all Domain Controllers. Then, one Windows machine sends a DDNS update for the zone to DC1 at the same time that another Windows machine sends a different DDNS update for that zone to DC2. Now, each DC has serial number n+1 and different contents. When AD synchronizes the zone contents and serial number under the covers, what serial number can it choose? It can't choose n+1, as that serial number has already been used. It can't choose n+2, as it does not know if another DDNS for the same zone has arrived before the synchronization has taken place. IIRC, 282826 says that if a DC is not used as a master for a BIND slave, then its zone serial number is not important. Another problem that I saw when I had BIND servers slaving AD zones was this - during patching of the DCs, the zone serial number might decrease. In most cases, after the DC patching was complete, the serial number reverted to the proper value. I was not allowed to open a trouble ticket with MS to determine why the zone serial on the DC was decreasing. The Windows support group did not see this as a problem. It might not have been a problem, as I saw many times where the zone serial number changed in an AD zone when the zone contents had not changed. This just meant more unnecessary zone transfers from the master to the slave. --Barry Finkel ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Multi-master (HA)
A few thoughts... The DNS protocol is already pretty good at replicating zone data - see for instance John Wingenbach's message in which he describes how their deployment gradually converged on a fairly standard architecture :-) I think multi-master makes most sense if the primary master uses DNS UPDATE for zone edits (and use raw file format), to minimize the differences between the primary and the secondaries. You probably want to ensure update forwarding is allowed, so that update clients do not have to worry so much about finding the current primary master. When a secondary takes over as primary it will need to update the SOA MNAME to point to itself so updates go to the right place. Most of the problem is actually one of remote configuration management: promoting a secondary to a primary is not all that different from setting up the secondary in the first place or making other co-ordinated changes. For instance it would be nice to be able to set up a zone once on the primary and have it automatically provisioned on the secondaries. I like Phil Mayers' zone-template idea, which might make it easier to flip from secondary to primary, as well as reducing the size and ensuring the consistency of large configs. Metazones are a tempting idea but the details get yucky the more of BIND's features you want to support. Also I am rather wary about the idea of putting secrets in a DNS zone; if you have an out-of-band way of distributing them it makes sense to use the same channel for the rest of the configuration. (http://ci.nii.ac.jp/naid/110007502948 - Vixie's metazones paper.) Tony. -- f.anthony.n.finchhttp://dotat.at/ Viking, North Utsire: Easterly or southeasterly 4 or 5, occasionally 6 in south Viking. Slight or moderate. Rain or showers. Good, occasionally poor. ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Multi-master (HA)
On 2014-05-07 15:06, Lawrence K. Chen, P.Eng. wrote: OTOH, the idea of multi-master is intriguing.the only down side I see, is hat I have one really powerful server for my current master(Sun Fire X4170)and my other servers are weak leftoversjust passed EOL last year. And, have all the servers doing full DNSSEC signing could be interesting. It also raises the question of how does the outside world cope with all the servers having identical zones...signed on slightly different times, etc. (especially since I'm using unix timestamp for zone serialavoids issues of multiple admins incrementing serial without noticing others and/or collisions with DNSSEC's incrementing of serials.) I wouldn't expect any real issues here, Windows DNS has done multimaster DNS since Windows 2000. In the case of Windows, dynamic updates (via client or GUI) can be done at any location, the serial numbers are incremented automatically, but the zones and servers may vary from each other for a brief period of time. So for example, DC1 and DC2 may start with serial 100, DC1 will receive 2 changes and be up to 102, DC2 will give 5 different changes and be up to 105. When Active Directory synchronization happens outside of DNS, the two sides merge changes together, and set the serial to the higher of the two plus one, so the serial would be 106. To the outside world, records can appear/disappear for a brief period while the servers drift out of sync, similar to what could happen in a BIND configuration without notifies as resolvers hit the two DNS servers round-robin. The only thing that causes issues is if you use DNS to create a non-Active Directory slave. BIND will throw errors because it will see serial 100, 101, 102, then get a notify from the second server about 101. However, the slave will still sync up once the AD servers sync to 106. The fix here is to configure BIND to only slave off of one master or the other, not both. While there might be other factors involved in turning BIND into a true multi-master solution, I wouldn't expect zones drifting out of sync or having minor differences to be a big factor since it happens in the wild already. -- Dave Warren http://www.hireahit.com/ http://ca.linkedin.com/in/davejwarren ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Multi-master (HA)
Well, we use two masters in different locations, w/o DLZ. Files for signed zones are being generated from databases and uploaded to servers. What we need here - is propagating of DDNS plus periodical synchronizing of zones, journals etc. Regarding zone templates - I'm using it with NSD4 and I'm totally happy. Actually I don't have words to emphasize how I love those templates! 2014-05-08 2:06 GMT+04:00 Lawrence K. Chen, P.Eng. : > > > On 05/06/14 13:39, Evan Hunt wrote: >> On Tue, May 06, 2014 at 06:20:11PM +, Baird, Josh wrote: >>> Hi, >>> >>> For those of you who operate at multiple sites or datacenters, are you >>> doing any HA for your BIND masters? Ideally, we would have a master in >>> each datacenter; maybe not an active one, but one that is standing by in >>> case your primary master becomes unavailable. >>> >>> Do you have multiple "active" masters and list them as master in each of >>> your slave's zone definitions? This seems like it could get rather >>> messy. One thought is to use a technology like VMWare SRM which will >>> spin up a master/virtual machine automatically in a second datacenter if >>> your primary master goes down. This coupled with Layer2 connectivity >>> between your sites could make things fairly simple. The >>> standby/secondary master would retain the same IP address as your >>> primary, so everything should just *work*. >>> >>> What are others doing? Any thoughts, ideas or advice is much >>> appreciated. >> >> Thank you for bringing this up. As it happens, high-availability/ >> multi-master support in BIND is something we've been seriously considering >> for a future release. There's been a lot of internal discussion of use >> cases, requirements, and possible design approaches. >> >> I don't want to influence the conversation here by saying too much about >> the ideas we've had so far, but I wanted to say: if anyone has specific >> thoughts on how to make this sort of thing easier in BIND -- even just at >> the level of "boy, it irritates me that I can't make BIND do " -- >> such comments will fall on welcoming ears. >> > > I hadn't thought of doing multi-master...but the issue of promoting a slave to > master for DR had come up. At the time the problem was DNSSEC. Its one thing > for the slave to become master, its another when it needs to change entries in > the zone file to redirect key web-services to DR instances. (at the time, it > was create two signed zone files each time...and secure transfer the second > one out of bandbut no DR web servers were ever setup, so both were > identical files and eventually got scrapped. The issue of raw vs text on > secondaries came up after abandonment. But, DR comes up now and > then...recently its using DNS appliances and cloud... > > OTOH, the idea of multi-master is intriguing.the only down side I see, is > that I have one really powerful server for my current master(Sun Fire > X4170)and my other servers are weak leftoversjust passed EOL last > year. And, have all the servers doing full DNSSEC signing could be > interesting. > > It also raises the question of how does the outside world cope with all the > servers having identical zones...signed on slightly different times, etc. > (especially since I'm using unix timestamp for zone serialavoids issues of > multiple admins incrementing serial without noticing others and/or collisions > with DNSSEC's incrementing of serials.) > > But, it shouldn't be too hard to implement since, our nameservers are managed > by CFEngine. And, it makes possible for all my name servers to have both > internal and external views. Instead of having to have separate external > slaves and internal slaves. (and other issues that I'm still working through > with having thisnamely my recursive caching servers hitting external > slaves instead of internal slaves...) > > Things have gotten more complicated since we started allowing vanity internal > namesbefore it was one subdomain that only existed on internal, and > everybody had to put their host in there, as -host..ksu.edu > but then certain VIPs wanted host..ksu.edu to work even though its a > 10.x.x.x address. > > It would also mean one of our satellite campuses that refuses to use our > caching servers (and even sent our server that was providing the service for > their campus back, which they had firewalled their users from using while it > was there)...can have their own caching servers work without needing to > understand that our whois record doesn't list our stealth/internal > nameservers...which is why they can't resolve any internal services and need > to track down somebody to give them the 10.x.x.x IP and having their users use > that, etc. > > Wonder if they know about the change in forwarding on my caching resolvers to > AD? > > -- > Who: Lawrence K. Chen, P.Eng. - W0LKC - Sr. Unix Systems Administrator > For: Enterprise Server Technologies (EST) -- & SafeZone Ally > ___
Re: Multi-master (HA)
On 05/06/14 13:39, Evan Hunt wrote: > On Tue, May 06, 2014 at 06:20:11PM +, Baird, Josh wrote: >> Hi, >> >> For those of you who operate at multiple sites or datacenters, are you >> doing any HA for your BIND masters? Ideally, we would have a master in >> each datacenter; maybe not an active one, but one that is standing by in >> case your primary master becomes unavailable. >> >> Do you have multiple "active" masters and list them as master in each of >> your slave's zone definitions? This seems like it could get rather >> messy. One thought is to use a technology like VMWare SRM which will >> spin up a master/virtual machine automatically in a second datacenter if >> your primary master goes down. This coupled with Layer2 connectivity >> between your sites could make things fairly simple. The >> standby/secondary master would retain the same IP address as your >> primary, so everything should just *work*. >> >> What are others doing? Any thoughts, ideas or advice is much >> appreciated. > > Thank you for bringing this up. As it happens, high-availability/ > multi-master support in BIND is something we've been seriously considering > for a future release. There's been a lot of internal discussion of use > cases, requirements, and possible design approaches. > > I don't want to influence the conversation here by saying too much about > the ideas we've had so far, but I wanted to say: if anyone has specific > thoughts on how to make this sort of thing easier in BIND -- even just at > the level of "boy, it irritates me that I can't make BIND do " -- > such comments will fall on welcoming ears. > I hadn't thought of doing multi-master...but the issue of promoting a slave to master for DR had come up. At the time the problem was DNSSEC. Its one thing for the slave to become master, its another when it needs to change entries in the zone file to redirect key web-services to DR instances. (at the time, it was create two signed zone files each time...and secure transfer the second one out of bandbut no DR web servers were ever setup, so both were identical files and eventually got scrapped. The issue of raw vs text on secondaries came up after abandonment. But, DR comes up now and then...recently its using DNS appliances and cloud... OTOH, the idea of multi-master is intriguing.the only down side I see, is that I have one really powerful server for my current master(Sun Fire X4170)and my other servers are weak leftoversjust passed EOL last year. And, have all the servers doing full DNSSEC signing could be interesting. It also raises the question of how does the outside world cope with all the servers having identical zones...signed on slightly different times, etc. (especially since I'm using unix timestamp for zone serialavoids issues of multiple admins incrementing serial without noticing others and/or collisions with DNSSEC's incrementing of serials.) But, it shouldn't be too hard to implement since, our nameservers are managed by CFEngine. And, it makes possible for all my name servers to have both internal and external views. Instead of having to have separate external slaves and internal slaves. (and other issues that I'm still working through with having thisnamely my recursive caching servers hitting external slaves instead of internal slaves...) Things have gotten more complicated since we started allowing vanity internal namesbefore it was one subdomain that only existed on internal, and everybody had to put their host in there, as -host..ksu.edu but then certain VIPs wanted host..ksu.edu to work even though its a 10.x.x.x address. It would also mean one of our satellite campuses that refuses to use our caching servers (and even sent our server that was providing the service for their campus back, which they had firewalled their users from using while it was there)...can have their own caching servers work without needing to understand that our whois record doesn't list our stealth/internal nameservers...which is why they can't resolve any internal services and need to track down somebody to give them the 10.x.x.x IP and having their users use that, etc. Wonder if they know about the change in forwarding on my caching resolvers to AD? -- Who: Lawrence K. Chen, P.Eng. - W0LKC - Sr. Unix Systems Administrator For: Enterprise Server Technologies (EST) -- & SafeZone Ally ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Multi-master (HA)
I run bind multi master on 5 different site. my solution is bind-dlz with galeraDB backed. we are very satisfied by this configuration and works flawlessy until now. Rick On 5/7/14 8:11 PM, John Wingenbach wrote: I run a multi-master environment. We have 3 data centers which are considered to be able to run even though the rest are down. Initially, we ran our masters with the same exact configurations on each. One of the data centers was administratively defined as being the 'update master'. From there, any changes were first done locally and then rsync'd to each of the other data centers. Once in place, rndc reload was executed to pick up the changes on all of the masters. However, with the dawning of DNSSEC, that became problematic. Later we moved to dynamic updates and simply sent the update commands to each master separately. That worked but still resulted in issues with resyncing the zones after one of the data centers was out of communication. Now we have moved to one 'update master' and the rest being slave masters. When we want to change the update master, we have scripts which make the needed mods in the zone configurations and then restart named. It's not the prettiest method but it does provide the single point of update, automatic recovery if one of the datacenters is not reachable and full support of DNSSEC. There is no issue with zone file format as the zones are kept in text format and upon conversion to slave, we touch each of the files to prevent the new slave from expiring the zones immediately. -- John On 5/6/2014 2:20 PM, Baird, Josh wrote: Hi, For those of you who operate at multiple sites or datacenters, are you doing any HA for your BIND masters? Ideally, we would have a master in each datacenter; maybe not an active one, but one that is standing by in case your primary master becomes unavailable. Do you have multiple "active" masters and list them as master in each of your slave's zone definitions? This seems like it could get rather messy. One thought is to use a technology like VMWare SRM which will spin up a master/virtual machine automatically in a second datacenter if your primary master goes down. This coupled with Layer2 connectivity between your sites could make things fairly simple. The standby/secondary master would retain the same IP address as your primary, so everything should just *work*. What are others doing? Any thoughts, ideas or advice is much appreciated. Thanks, Josh ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Multi-master (HA)
I run a multi-master environment. We have 3 data centers which are considered to be able to run even though the rest are down. Initially, we ran our masters with the same exact configurations on each. One of the data centers was administratively defined as being the 'update master'. From there, any changes were first done locally and then rsync'd to each of the other data centers. Once in place, rndc reload was executed to pick up the changes on all of the masters. However, with the dawning of DNSSEC, that became problematic. Later we moved to dynamic updates and simply sent the update commands to each master separately. That worked but still resulted in issues with resyncing the zones after one of the data centers was out of communication. Now we have moved to one 'update master' and the rest being slave masters. When we want to change the update master, we have scripts which make the needed mods in the zone configurations and then restart named. It's not the prettiest method but it does provide the single point of update, automatic recovery if one of the datacenters is not reachable and full support of DNSSEC. There is no issue with zone file format as the zones are kept in text format and upon conversion to slave, we touch each of the files to prevent the new slave from expiring the zones immediately. -- John On 5/6/2014 2:20 PM, Baird, Josh wrote: Hi, For those of you who operate at multiple sites or datacenters, are you doing any HA for your BIND masters? Ideally, we would have a master in each datacenter; maybe not an active one, but one that is standing by in case your primary master becomes unavailable. Do you have multiple "active" masters and list them as master in each of your slave's zone definitions? This seems like it could get rather messy. One thought is to use a technology like VMWare SRM which will spin up a master/virtual machine automatically in a second datacenter if your primary master goes down. This coupled with Layer2 connectivity between your sites could make things fairly simple. The standby/secondary master would retain the same IP address as your primary, so everything should just *work*. What are others doing? Any thoughts, ideas or advice is much appreciated. Thanks, Josh ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Multi-master (HA)
On 5/6/14, 3:41 PM, Phil Mayers wrote: > I could expand on the idea if people don't think it's too insane ;o) I > suspect it has all kinds of caveats I haven't thought of, however. The concept of a "meta-zone" containing config data is something that I discussed with a number of ISC staff over the last 5 years and would LOVE to see additional discussion of. signature.asc Description: OpenPGP digital signature ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Multi-master (HA)
On 06/05/2014 19:39, Evan Hunt wrote:
I don't want to influence the conversation here by saying too much about
the ideas we've had so far, but I wanted to say: if anyone has specific
thoughts on how to make this sort of thing easier in BIND -- even just at
the level of "boy, it irritates me that I can't make BIND do " --
such comments will fall on welcoming ears.
If you want a bind slave to be capable of becoming a master with no
dependencies - other than a valid copy of the zone data and the
existing, on-box configs - you basically munge the config file. It might
be nice if that were easier, especially for large files with lots of
zones. Maybe there should be:
zone-template "foo" {
masters ...
filename "zones/$name/file";
key-directory "keys/$name";
};
zone "bar" {
zone-template foo;
};
An extreme version of that is: it might be nice if the config *were* a
zone, updatable locally via nsupdate, slaved from the "primary" during
normal operations. Suitable design of the schema for that zone could
make it a very useful feature generally (think "subscribe to
service-set"), and would avoid the need for everyone who does this to
hand-roll rsync/perl scripts to manage bind.
I could expand on the idea if people don't think it's too insane ;o) I
suspect it has all kinds of caveats I haven't thought of, however.
Obviously for sites with SQL DBs driving zone & contents, and already
replicated out to or near your offsite slaves then you're basically
there already. But there are reasons why you might not want to do that;
in particular it might leak more information that the minimum DNS zone
contents, to a machine you have less control over.
At a more basic level, the difference in default file format
necessitates either conversion during change from slave to master - I
assert this is undesirable as this is very likely happening in a hurry,
possibly even with junior staff or semi-automated systems doing the work
- or planning ahead to ensure the format is consistent everywhere (as we
did). Something to note at very least. Maybe this could be made a bit
more robust by detecting on-disk format and going ahead, rather than
renaming it out of the way or failing to start?
DNSSEC keys are a tricky one; we tar/gpg/scp to the offsite slave.
Obviously not an option with an HSM.
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe
from this list
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: Multi-master (HA)
On Tue, May 06, 2014 at 06:20:11PM +, Baird, Josh wrote: > Hi, > > For those of you who operate at multiple sites or datacenters, are you > doing any HA for your BIND masters? Ideally, we would have a master in > each datacenter; maybe not an active one, but one that is standing by in > case your primary master becomes unavailable. > > Do you have multiple "active" masters and list them as master in each of > your slave's zone definitions? This seems like it could get rather > messy. One thought is to use a technology like VMWare SRM which will > spin up a master/virtual machine automatically in a second datacenter if > your primary master goes down. This coupled with Layer2 connectivity > between your sites could make things fairly simple. The > standby/secondary master would retain the same IP address as your > primary, so everything should just *work*. > > What are others doing? Any thoughts, ideas or advice is much > appreciated. Thank you for bringing this up. As it happens, high-availability/ multi-master support in BIND is something we've been seriously considering for a future release. There's been a lot of internal discussion of use cases, requirements, and possible design approaches. I don't want to influence the conversation here by saying too much about the ideas we've had so far, but I wanted to say: if anyone has specific thoughts on how to make this sort of thing easier in BIND -- even just at the level of "boy, it irritates me that I can't make BIND do " -- such comments will fall on welcoming ears. -- Evan Hunt -- e...@isc.org Internet Systems Consortium, Inc. ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Multi-master (HA)
Josh, we use multiple masters across multiple hosts, with mysql as a backend for the zone data. Each DNS server is a master and has it’s own local mysql DB. Each mysql database is then kept in ‘sync’ using mysql replication over a VPN link from a single (private) admin host. The single admin host (i.e. master mysql database) sits on a cluster framework, so it is HA. By doing things this way, if we do have any problems with our primary admin cluster, all the other DNS servers continue to serve clients without interruption. If there is a big problem with the admin cluster, it doesn’t take long to repoint the replication to another system or even just run manual mysql updates on the databases, if it really came down to it. From my experience, systems often need to resolve hosts to boot cleanly (ntp springs to mind), so having the DNS daemon itself in a cluster/HA control, often means it will only be started once the main OS has started, which is often a wee bit too late. Hope this is of some use… If you do go down the ‘putting named under cluster control’ route, just check that your local host doesn’t need it before cluster starts it up :-) I’ve seen that bite a number of my customers before... cheers marty On 6 May 2014, at 19:20, Baird, Josh wrote: > Hi, > > For those of you who operate at multiple sites or datacenters, are you doing > any HA for your BIND masters? Ideally, we would have a master in each > datacenter; maybe not an active one, but one that is standing by in case your > primary master becomes unavailable. > > Do you have multiple "active" masters and list them as master in each of your > slave's zone definitions? This seems like it could get rather messy. One > thought is to use a technology like VMWare SRM which will spin up a > master/virtual machine automatically in a second datacenter if your primary > master goes down. This coupled with Layer2 connectivity between your sites > could make things fairly simple. The standby/secondary master would retain > the same IP address as your primary, so everything should just *work*. > > What are others doing? Any thoughts, ideas or advice is much appreciated. > > Thanks, > > Josh > > ___ > Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe > from this list > > bind-users mailing list > bind-users@lists.isc.org > https://lists.isc.org/mailman/listinfo/bind-users - Marty Lee e: ma...@maui-systems.co.uk Technical Directorv: +44 845 869 2661 Maui Systems Ltd f: +44 871 433 8922 Scotland, UK w: http://www.maui-systems.co.uk signature.asc Description: Message signed with OpenPGP using GPGMail ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Multi-master (HA)
Hi, For those of you who operate at multiple sites or datacenters, are you doing any HA for your BIND masters? Ideally, we would have a master in each datacenter; maybe not an active one, but one that is standing by in case your primary master becomes unavailable. Do you have multiple "active" masters and list them as master in each of your slave's zone definitions? This seems like it could get rather messy. One thought is to use a technology like VMWare SRM which will spin up a master/virtual machine automatically in a second datacenter if your primary master goes down. This coupled with Layer2 connectivity between your sites could make things fairly simple. The standby/secondary master would retain the same IP address as your primary, so everything should just *work*. What are others doing? Any thoughts, ideas or advice is much appreciated. Thanks, Josh ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
