Re: [ZWeb] DNS still fishy?
On 10/12/06, Jens Vagelpohl [EMAIL PROTECTED] wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 12 Oct 2006, at 08:57, Justizin wrote: Anyway, everything except these hosts need to be removed from the rotation: ns1.zoneedit.com ns7.zoneedit.com ns.qutang.net ns*.zope.com Then I suggest you do that and end the current confusion in regards to which server does what (and which server even has the correct data). (a) I don't control the actual registrar records (b) Yes, these were listed in the zone itself as the NS, but noone should be doing lookups via these servers, because ZoneEdit is not authoritative for the NS records of this zone, the registrar is. I've removed them, but I politely request that you stop being an asshole unless you want to wear this hat yourself. I'm sick, I was stranded in the middle of nowhere when this change took place, and I was rushed. It's all of our fault. Don't make me come over there. I'd love to see more backups once they have copies of the zone. If you want to grab a copy of the zone, you'll have to transfer manually from ns1.zoneedit.com or ns7.zoneedit.com, from one of these IP addresses: No you don't. Setting a machine up as a slave, in that terrible bind- centric world, will cause it to pull the data automatically. ZoneEdit apparently does not run BIND, or at least does not send NOTIFY requests. I don't know what you want me to do. Three nameservers is fine for now. Eight would be far better. I still don't understand why we would need that many... but I don't want to discuss this any further. Matter of fact, since zoneedit does not support NOTIFY it is probably a bad thing to even have my server on the list. I suggest you limit the official servers to the ones you mentioned, the zoneedit/qutang/zope.com hosts until NOTIFY is working. jens You don't understand because you're an idiot, Jens, and you've never guaranteed 100% uptime. I was basically shut up by your whining when I tried to explain all of the precautions we should take in order to avoid what happened to zope.org this week. I won't respond to demands that I rush ever again. -- Justizin, Independent Interactivity Architect ACM SIGGRAPH SysMgr, Reporter http://www.siggraph.org/ ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] DNS still fishy?
Justizin wrote: I'd love to see more backups once they have copies of the zone. Why? zope.org has happily lived off two nameservers for years and years... All of a sudden, we need to have more backups, the upshot of which has been people in europe getting served bad dns from ns.qutang.net :-( What's wrong with just having ns1.zoneedit.com and ns7.zoneedit.com (could we also use ns(2-6).zoneedit.com?) and be done with it? Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] DNS still fishy?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 12 Oct 2006, at 09:15, Justizin wrote: (a) I don't control the actual registrar records (b) Yes, these were listed in the zone itself as the NS, but noone should be doing lookups via these servers, because ZoneEdit is not authoritative for the NS records of this zone, the registrar is. To stay strictly on technical issues, I think you're constantly implying that the DNS servers for the zope.org zone that are listed by the registrar are not the same as the DNS servers the zone data itself contains. Can you explain why this discrepancy exists, or why it makes sense? I'd love to see more backups once they have copies of the zone. If you want to grab a copy of the zone, you'll have to transfer manually from ns1.zoneedit.com or ns7.zoneedit.com, from one of these IP addresses: No you don't. Setting a machine up as a slave, in that terrible bind- centric world, will cause it to pull the data automatically. ZoneEdit apparently does not run BIND, or at least does not send NOTIFY requests. I don't know what you want me to do. Nothing. I am describing the situation where you have a bind slave and you are configuring a slave zone for the first time. At that moment you don't have to manually pull the zone data, bind will magically fetch it. This was a hint for people who might want to set up a slave. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (Darwin) iD8DBQFFLkJnRAx5nvEhZLIRApZWAKCdD4MxCtrJuZ+ezihcYnnC+KugmQCghgEC bAxQ9hjKbWdXHVdz5nuTzT8= =0e5C -END PGP SIGNATURE- ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] DNS still fishy?
Jens Vagelpohl wrote: It makes sense to have name servers in different physical locations and on different networks in case one provider runs into trouble. The point of contention is the number of slaves. Right, which brings me back to my other point: why, when 2 server have been fine for about a decade, do we need to change now? Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] DNS still fishy?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 12 Oct 2006, at 09:20, Chris Withers wrote: Justizin wrote: I'd love to see more backups once they have copies of the zone. Why? zope.org has happily lived off two nameservers for years and years... All of a sudden, we need to have more backups, the upshot of which has been people in europe getting served bad dns from ns.qutang.net :-( What's wrong with just having ns1.zoneedit.com and ns7.zoneedit.com (could we also use ns(2-6).zoneedit.com?) and be done with it? It makes sense to have name servers in different physical locations and on different networks in case one provider runs into trouble. The point of contention is the number of slaves. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (Darwin) iD8DBQFFLkLoRAx5nvEhZLIRArSuAKC1xDSZzd+Y4elgChwKb8i9INCerACfZMBZ wdI8SlUIRqp+QWM6Wbj7wqw= =zPH2 -END PGP SIGNATURE- ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] DNS still fishy?
On 10/12/06, Chris Withers [EMAIL PROTECTED] wrote: Justizin wrote: I'd love to see more backups once they have copies of the zone. Why? zope.org has happily lived off two nameservers for years and years... All of a sudden, we need to have more backups, the upshot of which has been people in europe getting served bad dns from ns.qutang.net :-( This is a logical fallacy. Services were not unavailable because we have more than two nameservers, services were unavailable because we rushed. ns.qutang.net did not serve any bad dns that ns*.zoneedit.com were not serving. The errors were in ZoneEdit's copy of the Zone. I was thinking just now over a smoke about someone I used to work with at Rackspace, the datacenter engineer. Bob was a member of the NASA Challenge Safety Team. He personally recommended against launching the Challenger, which exploded, killing some astronauts. I learned from working with him that you should never tell someone with more experience to be less cautious. What's wrong with just having ns1.zoneedit.com and ns7.zoneedit.com (could we also use ns(2-6).zoneedit.com?) and be done with it? We can only use the nameservers that zoneedit allocates us. Yanno, people used to pay $75 per half hour for this expertise. -- Justizin, Independent Interactivity Architect ACM SIGGRAPH SysMgr, Reporter http://www.siggraph.org/ ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] DNS still fishy?
Just a couple of notes here. Although zoneedit has been running fine for me for years without a single problem, obviously it would be nice with some backup. Preferably something with another ISP and located on like another continent or something. Two of these backups would be even better. But honestly, compare the likelyhood that all three of these would fail at one time, together with the increasing likelyhood than one server of them is misconfigured and starts disturbing the usage for a minor part of the users, then we will quickly realize that the more backups and failsafes we have the larger the likelyhood that something of this will go wrong. 8 servers seems to be to be a complete overkill, and it will only cause problems. I will change my mind on this the time all zone-edit servers stop working at the same time as two of the backups fail. Don't overcomplicate things. It just makes them fail. ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] DNS still fishy?
On 10/12/06, Jens Vagelpohl [EMAIL PROTECTED] wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 12 Oct 2006, at 09:15, Justizin wrote: (a) I don't control the actual registrar records (b) Yes, these were listed in the zone itself as the NS, but noone should be doing lookups via these servers, because ZoneEdit is not authoritative for the NS records of this zone, the registrar is. To stay strictly on technical issues, I think you're constantly implying that the DNS servers for the zope.org zone that are listed by the registrar are not the same as the DNS servers the zone data itself contains. Can you explain why this discrepancy exists, or why it makes sense? I prepared a copy of the zone in ZoneEdit with small changes to reflect the plans for a new configuration, including new nameservers. I pulled the zone into ns.qutang.net early last week and sent out an e-mail which, surely, was just lost in the white noise. oh well. so, because we wanted to start modifying the zone really soon, i told rob page to change the registrar to point at: ns1.zoneedit.com ns7.zoneedit.com ns.qutang.net These nameservers all had the same data, including the same incorrect records. FWIW, three records with the same IP address went sour: www.zope.org cvs.zope.org zope.org This is curious, because I recall making an effort to individually copy each record from the zone file that Rob sent me, to avoid just this sort of mistake. whatever, these records pointed at .1 instead of .171 Nothing. I am describing the situation where you have a bind slave and you are configuring a slave zone for the first time. At that moment you don't have to manually pull the zone data, bind will magically fetch it. This was a hint for people who might want to set up a slave. Handy. I am writing a how-to for making djbdns comply with both ends of the NOTIFY chain. There are a bunch of tools for this, very simple djb-ish stuff, but nothing is part of the package. If someone running BIND wants to pull from zoneedit and send the rest of us NOTIFY requests when a change is detected, we can pretty much do that now. I should be set up to respond to NOTIFY. I have to add something into the tinydns-data chain which enacts changes to live configuration so that it spurs a NOTIFY to slaves. -- Justizin, Independent Interactivity Architect ACM SIGGRAPH SysMgr, Reporter http://www.siggraph.org/ ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] DNS still fishy?
On 10/12/06, Lennart Regebro [EMAIL PROTECTED] wrote: Just a couple of notes here. Although zoneedit has been running fine for me for years without a single problem, obviously it would be nice with some backup. Preferably something with another ISP and located on like another continent or something. Two of these backups would be even better. But honestly, compare the likelyhood that all three of these would fail at one time, together with the increasing likelyhood than one server of them is misconfigured and starts disturbing the usage for a minor part of the users, then we will quickly realize that the more backups and failsafes we have the larger the likelyhood that something of this will go wrong. the worst that happens is that some changes fail to propogate. changes to DNS should always be approached with the assumption that this will happen. What's worse is for there to be no copy of a zone available. It should never be necessary for an A record to change immediately, because this cannot be relied upon. The best defense to this is, however, to set TTLs at 300s, or 5 minutes, about a week in advance. 8 servers seems to be to be a complete overkill, and it will only cause problems. I will change my mind on this the time all zone-edit servers stop working at the same time as two of the backups fail. It could cause problems, and that's why we aren't really using eight servers right now, but it should not cause problems. It is a challenge, also, that our DNS is not hosted in the same location as the website. So, it's possible that DNS will be unreachable when an outage occurs, i.e. a fibre being cut in the middle of the ocean, and this outage may not actually affect our site. I bet ten bucks if we rely entirely on zoneedit's nameservers that this will happen once for at least twelve hours for some significant region of the world within the next year. Don't overcomplicate things. It just makes them fail. This assumption really has nothing to do with what happened this week. What happened this week was either: (a) a typo (b) an erroneously truncated string If there were only two nameservers, they would have pointed at the wrong IP, and the site would have been perceptually unavailable for a few hours to two days for various people. If there were eight, the same would happen, for about the same time frame. So, if you want to only use two nameservers, that's okay with me. Remember to wake me up when the zone is unreachable for someone and we want to run more. :) I always assume, if anything, that some machines, network connections, disk drives, etc.. will invariably fail, and that you can never have too many if they are available. I like the idea of a group of zope community members collectively providing DNS service. Maybe we should even talk about running multiple copies of the flat content in different places. If my site goes down, esp if one of my machines fail, I much prefer to feel comfortable that I can reach zope.org than rely on the possibility that i might have copies of recent releases in another location. if i'm going to keep copies of the releases around for myself, might as well mirror them, eh? While having a set of servers configured by various people sounds as if it would be overcomplicated, with proper planning and coordination, we should be able to keep it simple. When making changes to DNS, always assume that for 48 hours there will be between a 90-10 and 10-90 split between people who have your new records and people who have old records. When changing nameservers, double or triple this, because some people will have cached records from the old nameserver *and* more recently cached NS records, so they may continue querying the old nameserver until the cached NS record itself expires. When something critical like svn/cvs or the main website need to be changed, again, it is necessary to drop the TTL, on the entire zone, even, to something really short like 300s about a week in advance. This ensures that everyone in the world has a copy of the zone which says: no copy of this zone and no records in this zone are good for longer than five minutes.. Just before a switch is made, you can proxy the old front-end apache server to the new host explicitly, and then update records. for five or ten minutes some people's requests will be slow because they are possibly doubling-back across the internet, but at least they can't really tell what's going on, just that for a few minutes it is a 'little bit slow'. -- Justizin, Independent Interactivity Architect ACM SIGGRAPH SysMgr, Reporter http://www.siggraph.org/ ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] DNS still fishy?
On 10/12/06, Lennart Regebro [EMAIL PROTECTED] wrote: On 10/12/06, Justizin [EMAIL PROTECTED] wrote: It could cause problems, and that's why we aren't really using eight servers right now, but it should not cause problems. Servers should not fail. This should not cause problems. But in reality, it will. Servers failing will not cause problems, the only real risk would be tampering. The reason for having many servers is to protect against failure. It is a challenge, also, that our DNS is not hosted in the same location as the website. So, it's possible that DNS will be unreachable when an outage occurs, i.e. a fibre being cut in the middle of the ocean, and this outage may not actually affect our site. Which is why one or two backups on another continent is nice to have. Three or more is best. Don't overcomplicate things. It just makes them fail. This assumption really has nothing to do with what happened this week. I'm not convinced. Then take over, Lennart. I do not care. You don't have to be convinced. Explain to me how this problem is related to the outage, which was as simple as this: records served by three of five nameservers were incorrect. the other two were zope.com nameservers, and they don't delegate to zoneedit afaik. So, if you want to only use two nameservers, that's okay with me. Please respons to what I write, and argue against what I argue, instead of making up arguments against things I have never said. I, explicitly in my last mail, said that one or two backups on other continents would be necssary, but that the previously mentioned *eight* backups would cause more problems than they solve. You said you don't understand why we don't just use zoneedit. What makes four servers less failure prone than eight, so long as they all agree that zoneedit is in charge. If you don't agree with this, you are welcome to explain to me why. But do NOT argue against me by implying that I have said something stupid, which I never said. Oh whatever. Look, I'm sick of this conversation. I did a better job than anyone else in the conversation would have, and problems happened because we spent a week on something that we should have spent 2-4 weeks on. We learned something. I think the real issue is that we ran into a problem, which I tried hard to avoid, and people are still arguing that I am proposing to take too many precautions. -- Justizin, Independent Interactivity Architect ACM SIGGRAPH SysMgr, Reporter http://www.siggraph.org/ ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] DNS still fishy?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 12 Oct 2006, at 10:05, Lennart Regebro wrote: But honestly, compare the likelyhood that all three of these would fail at one time, together with the increasing likelyhood than one server of them is misconfigured and starts disturbing the usage for a minor part of the users, then we will quickly realize that the more backups and failsafes we have the larger the likelyhood that something of this will go wrong. 8 servers seems to be to be a complete overkill, and it will only cause problems. I will change my mind on this the time all zone-edit servers stop working at the same time as two of the backups fail. Don't overcomplicate things. It just makes them fail. Exactly. We are not building a carrier-grade solution here because, as the programmer idiom goes, it is YAGNI (you ain't gonna need it). Keeping a carrier-grade solution running correctly is always more effort than keeping the simple solution up. There's a diminishing return between upkeep/effort/maintenance/script-writing and oops, DNS is gone for an hour. I seriously don't see the added value. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (Darwin) iD8DBQFFLmpZRAx5nvEhZLIRAt/JAKCtd4n0eXB+40oC9taJu9NXjzpsjQCgrxpt EWr/MZcXHi7iMWqNkKNYdiU= =OHbm -END PGP SIGNATURE- ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] DNS still fishy?
On 10/12/06, Justizin [EMAIL PROTECTED] wrote: Servers failing will not cause problems, the only real risk would be tampering. I was unclear, sorry. What I ment to say is that things go wrong. Your statement this should not cause problems, is equivalent to servers will not fail and my point then was that in that case we can run with one server and be done with it. The reality is that servers fail. The reality is also that complex setups cause problems, no matter that they shouldn't. The reason for having many servers is to protect against failure. With increasing number of servers you get better protection against failure. But the increasing protection you get gets less and less with each server. At the same time, configuration weirdness and other stuff is likely to INCREASE the error rate the more backups you have, because of Murphys law and other stuff. At one point, this increase in problems will overwhelm the increase in protection. I would also like to claim that this crossover point is nowehere near the previously mentioned number of eight servers, but rather closer two have one or two backups on another continent. Some maths: Say that a server fails one day per month in average (which is way more than we really will have). One backup server located on anotehr continent then means that we will statistically have DNS outage only one day in 900. Thats one day every three years. Two backups located on different continents will give us a failure rate of one day per 27000 days. That's one day every seventy-fifth year. How would five-six increasing backup servers in any reasonable way actually increase that realiability? It wouldn't, because for every server you add, you increase the risk of something going wrong. That's probably not an exponential risk, but I'm pretty sure somebody somewhere will fuck something up more often than every seventy-fifth year, so I don't actually think that having more than two backups on different continents is gonna increase realiability. Three or more is best. If you talk about total number of DNS servers, then I agree. Two at zoneedit, one or two more somewhere else. Then take over, Lennart. I do not care. Oh, you do care, because you get angry- You said you don't understand why we don't just use zoneedit. No. I have never said anything like that. Please read what I say, and answer that. I have been discussing politics on the internet for 15 years, and one thing I have learned is to completely stop any discussion when you get accused of an opinion you don't have because constructive discussion have at that point failed. Please read my emails, and answer they things I said, not the things I did not say. What makes four servers less failure prone than eight, so long as they all agree that zoneedit is in charge. I think that is a pretty obvious question. The more things you have the more things will fail. Look, I'm sick of this conversation. I did a better job than anyone else in the conversation would have, and problems happened because we spent a week on something that we should have spent 2-4 weeks on. We learned something. That is quite possible. I am not claiming you did a bad job. I have never said I would do a better job. I don't complain, whine or say you are stupid. I'm say one simple thing: Having eight servers is overkill and cause more problems than it solves. Please discuss this instead of trying to make this be about some sort of personal issue. It is not. You are a professional. I am a professional. Lets please all behave like it. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.nuxeo.org/ ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
Re: [ZWeb] DNS still fishy?
FYI, there's a problem with your host Justizin: server ns1.zoneedit.com Default server: ns1.zoneedit.com Address: 207.234.248.200#53 cvs.zope.org Server: ns1.zoneedit.com Address:207.234.248.200#53 Name: cvs.zope.org Address: 63.240.213.173 server ns.qutang.net Default server: ns.qutang.net Address: 70.84.6.50#53 cvs.zope.org Server: ns.qutang.net Address:70.84.6.50#53 Name: cvs.zope.org Address: 63.240.213.171 In my opinion, the registrar should only have zoneedit.com servers in it for the time being. Andrew On 10/12/06 11:02 AM, Justizin [EMAIL PROTECTED] wrote: On 10/12/06, Lennart Regebro [EMAIL PROTECTED] wrote: Just a couple of notes here. Although zoneedit has been running fine for me for years without a single problem, obviously it would be nice with some backup. Preferably something with another ISP and located on like another continent or something. Two of these backups would be even better. But honestly, compare the likelyhood that all three of these would fail at one time, together with the increasing likelyhood than one server of them is misconfigured and starts disturbing the usage for a minor part of the users, then we will quickly realize that the more backups and failsafes we have the larger the likelyhood that something of this will go wrong. the worst that happens is that some changes fail to propogate. changes to DNS should always be approached with the assumption that this will happen. What's worse is for there to be no copy of a zone available. It should never be necessary for an A record to change immediately, because this cannot be relied upon. The best defense to this is, however, to set TTLs at 300s, or 5 minutes, about a week in advance. 8 servers seems to be to be a complete overkill, and it will only cause problems. I will change my mind on this the time all zone-edit servers stop working at the same time as two of the backups fail. It could cause problems, and that's why we aren't really using eight servers right now, but it should not cause problems. It is a challenge, also, that our DNS is not hosted in the same location as the website. So, it's possible that DNS will be unreachable when an outage occurs, i.e. a fibre being cut in the middle of the ocean, and this outage may not actually affect our site. I bet ten bucks if we rely entirely on zoneedit's nameservers that this will happen once for at least twelve hours for some significant region of the world within the next year. Don't overcomplicate things. It just makes them fail. This assumption really has nothing to do with what happened this week. What happened this week was either: (a) a typo (b) an erroneously truncated string If there were only two nameservers, they would have pointed at the wrong IP, and the site would have been perceptually unavailable for a few hours to two days for various people. If there were eight, the same would happen, for about the same time frame. So, if you want to only use two nameservers, that's okay with me. Remember to wake me up when the zone is unreachable for someone and we want to run more. :) I always assume, if anything, that some machines, network connections, disk drives, etc.. will invariably fail, and that you can never have too many if they are available. I like the idea of a group of zope community members collectively providing DNS service. Maybe we should even talk about running multiple copies of the flat content in different places. If my site goes down, esp if one of my machines fail, I much prefer to feel comfortable that I can reach zope.org than rely on the possibility that i might have copies of recent releases in another location. if i'm going to keep copies of the releases around for myself, might as well mirror them, eh? While having a set of servers configured by various people sounds as if it would be overcomplicated, with proper planning and coordination, we should be able to keep it simple. When making changes to DNS, always assume that for 48 hours there will be between a 90-10 and 10-90 split between people who have your new records and people who have old records. When changing nameservers, double or triple this, because some people will have cached records from the old nameserver *and* more recently cached NS records, so they may continue querying the old nameserver until the cached NS record itself expires. When something critical like svn/cvs or the main website need to be changed, again, it is necessary to drop the TTL, on the entire zone, even, to something really short like 300s about a week in advance. This ensures that everyone in the world has a copy of the zone which says: no copy of this zone and no records in this zone are good for longer than five minutes.. Just before a switch is made, you can proxy the old front-end apache server to the new host explicitly, and then
Re: [ZWeb] DNS still fishy?
Can we have only zoneedit as the registered nameservers? 3 out of the 5 listed name servers at the registrar are wrong. We need this fixed ASAP. Andrew ___ Zope-web maillist - Zope-web@zope.org http://mail.zope.org/mailman/listinfo/zope-web
