Re: DNS problems to RoadRunner - tcp vs udp
Jon Kibler [EMAIL PROTECTED] writes: Okay, I stand corrected. I was approaching this from a security perspective only, and apparently based on incorrect information. It always puzzles me when people say things like that - it's as if they've lost sight of the *whole point* of security being to protect services, data, etc. and ensure that they continue uninterrupted and uncorrupted. If one is approaching problems from a security perspective only with no concern to breaking services... wouldn't the even more secure solution involve just reaching for the power switch? But this leaves me with a couple of questions: Various hardening documents for Cisco routers specify the best practices are to only allow 53/tcp connections to/from secondary name servers. Plus, from all I can tell, Cisco's 'ip inspect dns' CBAC appears to only handle UDP data connections and anything TCP would be denied. From what you are saying, the hardening recommendations are wrong and that CBAC may break some DNS responses. Is this correct? I bet if you look in these same hardening documents you'll find suggestions for static bogon filters (which become stale over time), filtering all ICMP (breaks PMTUD) and all sorts of other jack moves. Also, other than That's what the RFCs call for, why use TCP for data exchange instead of larger UDP packets? Well, the inheritance is from the default IP maximum datagram size of 576 bytes, RFC879, which a host must observe absent specific knowledge that the far end can do better. In the vast majority of cases (assuming your resolver and cacheing nameserver won't puke) I suspect there would be no problem sending a somewhat-bigger-than-this-size DNS reply... right up till you go over 1500 bytes for your datagram, at which point you're back to square one. With cryptologically signed replies containing a lot of records and other data, bigger than 1500 bytes is certainly not outside the realm of possibility. Trying to fragment and reassemble DNS queries is a step away from goodness. With TCP, you have a virtual stream service rather than a datagram service, and in exchange for the overhead of setting up and tearing down the connection, one gets the ability to do transactions that are much larger. ---Rob
RE: [NANOG] Introducing latency for testing?
It's not free, but at a recent trade show I did see what appeared to be an affordable unit from Apposite Technologies (apposite-tech.com). And there's always PacketStorm. Frank -Original Message- From: Mike Lyon [mailto:[EMAIL PROTECTED] Sent: Friday, May 02, 2008 3:13 PM To: NANOG Subject: [NANOG] Introducing latency for testing? So I want to mimic some latency in a test network for DB replication. I am wondering what other's have used for this? Obviously, the best way to would be to actually have one box across the US or across the globe to actually test against but what if you don't have that? Are there any GPL software router solutions that would allow you to tweak the latency in between the two test boxes? Thanks in advance. -Mike ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: .255 addresses still not usable after all these years?
On Jun 14, 2008, at 12:26 AM, Mike Lewinski wrote: David Hubbard wrote: I remember back in the day of old hardware and operating systems we'd intentionally avoid using .255 IP addresses for anything even when the netmask on our side would have made it fine, so I just thought I'd try it out for kicks today. From two of four ISP's it worked fine, from Verizon FIOS and Road Runner commercial, it didn't. So I guess that old problem still lingers? The TCP/IP stack in Windows XP is broken in this regard, possibly in Vista as well, though I've yet to have the displeasure of finding out. I have a router with a .255 loopback IP on it. My Windows XP hosts cannot SSH to it. The specific error that Putty throws is Network error: Cannot assign requested address. At least if I ever need to completely protect a device from access by Windows users, I have a good option :) Mike From what I recall, Microsoft's stack was based on the only free one they could afford back in the Trumpet/Winsock days, namely BSD's. It is either dependent on how the stack is integrated, or it simply implies that BSD's stack is(was) also broken (I'd tend to doubt that). Also, Vista's stack was supposed to have been re-developed from scratch, never checked it. Greg VILLAIN
Re: DNS problems to RoadRunner - tcp vs udp
Not to toss flammables onto the pyre. BUT there is a large difference from what the RFC's allow and common practice. In our shop TCP is blocked to all but authoratative secondaries as TCP is sinply too easy to DoS a DNS server with. We simply don't need a few thousand drones clogging the TCP connection table all trying to do zone transfers ( yes it happened and logs show drones are still trying ) For a long time there has been a effective practice of UDP == resolution requests TCP == zone transfers It would have been better if a separate port had been defined for zone transfers as that would obviate the need for a application layer gateway to allow TCP transfers so that zone transfers can be blocked and resolution requests allowed for now all TCP is blocked. Now just because someone has a bright idea they drag out a 20 y/o RFC and say SEE, SEE you must allow this because the RFC says so all the while ignoring the 20 years of operational discipline that RFC was written when the internet was like the quad at college everyone knew one and other and we were all working towards a common goal of interoperability and open systems , These days the net is more like a seedy waterfront after midnight where criminal gangs are waiting to ambush the unwary and consequently networks need to be operated from that standpoint. At the University networking level it is extremely difficult as we need to maintain a open network as much as possible but protect our infrastructure services so that they have 5 nines of availability back in the day a few small hosts would serve DNS nicely and we did not have people trying to take them down and/or infecting local hosts and attempting DHCP starvation attacks. And no we are not at the 5 nines level but we are working on it. - Scott Randy Bush wrote: If my server responded to TCP queries from anyone other than a secondary server, I would be VERY concerned. you may want to read the specs randy
Re: DNS problems to RoadRunner - tcp vs udp
Scott McGrath wrote: [..] For a long time there has been a effective practice of UDP == resolution requests TCP == zone transfers WRONG. TCP is there as a fallback when the answer of the question is too large. Zone transfer you can limit in your software. If you can't configure your dns servers properly then don't run DNS. Also note that botnets have much more effective ways of taking you out. And sometimes domains actually require TCP because there are too many records for a label eg http://stupid.domain.name/node/651 If you are thus blocking TCP for DNS resolution you suddenly where blocking google and thus for some people The Internet. Also see: http://homepages.tesco.net/J.deBoynePollard/FGA/dns-edns0-and-firewalls.html (Which was the second hit for google(EDNS0) after a link to RFC2671) Greets, Jeroen signature.asc Description: OpenPGP digital signature
Re: DNS problems to RoadRunner - tcp vs udp
Jon Kibler writes: Also, other than That's what the RFCs call for, why use TCP for data exchange instead of larger UDP packets? TCP is more robust for large (Path MTU) data transfers, and less prone to spoofing. A few months ago I sent a message to SwiNOG (like NANOG only less North American and more Swiss) about this topic, trying to explain some of the tradeoffs: http://www.mail-archive.com/[EMAIL PROTECTED]/msg02612.html Mostly I think that people approaching this from a security perspective only often forget that by fencing in the(ir idea of the) current status quo, they often prevent beneficial evolution of protocols as well, contributing to the Internet's ossification. -- Simon.
Re: DNS problems to RoadRunner - tcp vs udp
Scott McGrath wrote: There is no call for insults on this list Insults? Where? If you feel insulted by any of the comments made on this list by people, then you probably are indeed on the wrong list. But that is just me. - Rather thought this list was about techincal discussions affecting all of us and keeping DNS alive for the majority of our customers certainly qualifies. [..blabber about DNS attacks over TCP..] If I where a botnet herder and I had to take out your site and I was going to pick TCP for some magical reason then I would not care about your DNS servers, I would just hit your webservers, hard. I mean just the 'index.html' (http://www.harvard.edu/) is 24Kb, that is excluding pictures and there is bound to be larger data there which you are going to send and the bots only have to say ACK to once in a while. Multiply that by say a small botnet of 1M hosts, each just requests that 24Kb file. You will have a million flows and won't have any way to rate limit that or control it. Your link was already full trying to send it back to the clients and next to that your server was probably not able to process it in the first place. Simple, effective, nothing you can do about it, except get way and way more hardware. If somebody wants to take you out, they will take you out. Just get one other box with 10GE (not too hard to do) or just get a million of them with a little bit of connectivity (which is quite easy apparently)... We/I am more than aware of the DNS mechanisms and WHY there are there trouble is NO DNS server can handle directed TCP attacks even the root servers crumbled under directed botnet activity and we have taken the decision to accept some collateral damage in order to keep services available. The root servers crumbled wow, I must have missed somebody taking out all the 13 separate and then individually anycasted root servers. Which btw only do UDP as currently '.' is still small enough. $ dig @a.root-servers.net. . NS +tcp [..] ;; Query time: 95 msec ;; SERVER: 2001:503:ba3e::2:30#53(2001:503:ba3e::2:30) ;; WHEN: Sat Jun 14 23:45:52 2008 ;; MSG SIZE rcvd: 604 That is only 1 packet to 1 packet, still only 500 bytes. While your little webserver would generate 24kb for that same sequence. We are a well connected university network with multi-gigabit ingress and egress with 10G on Abilene so we try to protect the internet from attacks originating within our borders AND we really feel the full wrath of botnets as we do not have a relatively slow WAN link to buffer the effects. The whole point generally of botnets is just the Denial of Service (DoS), if that is because your link is full or the upstreams link is full or because the service can't service clients anymore. But clearly, as you are blocking TCP-DNS you are DoSing yourself already, so the botherders win. Also note that Abilene internally might be 10G and in quite some places even 40G, but you still have to hand it off to the rest of the world and those will count as those 'slow WAN' links that you think everybody else on this planet is behind. (Hint: 10GE is kinda the minimum for most reasonably sized ISP's) Yes - we are blocking TCP too many problems with drone armies and we started about a year ago when our DNS servers became unresponsive for no apparent reason. Investigation showed TCP flows of hundreds of megabits/sec and connection table overflows from tens of thousands of bots all trying to simultaneously do zone transfers and failing tried active denial systems and shunning with limited effectiveness. How is a failed AXFR going to generate a lot of traffic, unless they are repeating themselves over and over and over again? Thus effectively just packeting you? Also, are you talking about Recursive or Authoritive DNS servers here? Where those bots on your network, or where they remote? We are well aware of the host based mechanisms to control zone information, Trouble is with TCP if you can open the connection you can DoS so we don't allow the connection to be opened and this is enforced at the network level where we can drop at wire speed. Do you mean that the hosts which do TCP are allowed to do transfers or not? As in the latter case they can't generate big answers, they just get 1 packet back and then end then FIN. Note also, that if they are simply trying to overload your hosts, UDP is much more effective in doing that already and you have that hole wide open apparently otherwise you wouldn't have DNS. Open to better ideas but if you look at the domain in my email address you will see we are a target for hostile activity just so someone can 'make their bones'. It probably has nothing to do with the domain name, it more likely has something to do with certain services that are available or provided on your network. Also recall we have a comittment to openess so we would like to make TCP services available but until we
Bandcon Transport Services..
Anyone here use Bandcon's transport services? Positive/Negative experiences, any feedback would be helpful.. Thanks ck
Re: DNS problems to RoadRunner - tcp vs udp
On Sat, 14 Jun 2008, Scott McGrath wrote: Also recall we have a comittment to openess so we would like to make TCP services available but until we have effective DNS DoS mitigation which can work with 10Gb links It's not going to happen. I feel your pain, but I think there may be a slight mis-analysis of the situation. However I may be mistaken, given the lack of details. The 10Gb really doesn't have much to do with tcp-state-table problems. Any network with a large user population probably should have separate DNS servers for their authoritative zones answering the Internet at-large and their recursive resolvers serving their user population. DNS recursive resolvers may not need to answer unsolicited queries from the Internet at large. It may make sense to keep those servers behind stateful packet gateways, and only allow both UDP and TCP responses from the Internet to UDP and TCP queries made by the local, authorized users. Because you don't know what Answer all the other DNS servers may give, including a Truncated answer, recursive resolvers must be able to use TCP to send queries to the Internet at large, and receive TCP queries from its local, authorized user population. If your own local users are DOSing your own DNS recursive resolvers, hopefully that's your own problem. A DNS authoritative server may only need to answer unsolicited UDP queries from the Internet at large. Because DNS clients (stub, resolvers) must send a query as UDP first, and may use TCP if the Answer has the truncated bit set, an authoritative name server which knows all its answers will always fit in the minimum DNS Answer and never sets the truncated bit shouldn't get a TCP DNS query. RFC1112 says DNS servers should answer unsolicited TCP DNS queries anyway, but its not a MUST and it may rate limit its TCP answers. Given those constraints, it may make sense for DNS authoritative servers to limit TCP, either with an ACL or rate-limit the TCP/SYNs. But its only a medium term solution. DNS answers are growing. Someday those DNS authoritative servers probaly will need to send a large DNS Answer. But that is under the control of the local DNS administrator. So hopefully he or she will know when the DNS server breaks, and will fix it then. Also, modern TCP/IP stacks and modern name server implementations don't have as many tcp-state-table issues as they did at the beginning of the decade. Any DOS attack based on TCP would disrupt HTTP/Web servers just as much as TCP/DNS servers. So many of the same mitigation techniques (and attacks) for Web servers may be applicable to DNS servers. So briefly 1. Separate your authoritative and recursive name servers 2. Recursive name servers should only get replies to their own DNS queries from the Internet, they can use both UDP and TCP 3. Recursive name servers should only get queries from their own user population, they can use both UDP and TCP 4. Authoritative servers may only need to answer UDP queries from the Internet, if they never truncates its Answers. But the DNS administrator should plan what to do when its Answers get too large. Most DNS servers don't provide good alerts to DNS administrators doing stupid things, like sending big DNS answers while blocking TCP. I tried to capture some of these ideas in some ACLs http://www.donelan.com/dnsacl.html
Re: DNS problems to RoadRunner - tcp vs udp
Sean Donelan wrote: 1. Separate your authoritative and recursive name servers 2. Recursive name servers should only get replies to their own DNS queries from the Internet, they can use both UDP and TCP We've just completed a project to separate our authoritative and recursive servers and I have a couple notes... 1) For the recursive-only, we're using a combination of BIND's query-source address a.b.c.d and listen-on e.f.g.h in the hopes of providing some additional measure of protection against cache poisoning. The listen-on IPs are ACL'd at the borders so non-clients cannot get ANY packets to them. The query-source address itself doesn't appear in the listen-on list either and won't respond to queries. I know this isn't foolproof, but it probably raises the bar slightly against off-net poisoning attempts. 2) The biggest drawback to separation after years of service is that customers have come to expect their DNS changes are propagated instantly when they are on-net. This turns out to be more of an annoyance to us than our customers, since our zone is probably the most frequently updated. 3) I've gone so far as to remove the root hint zone from our auth-only boxes, again out of paranoia (recursion no does the trick, this is just an extra bit of insurance against someone flipping that bit due to a lack of understanding of the architecture). There is one third party we have to use an 'also-notify' by IP address in this case for their zone. Mike
Re: DNS problems to RoadRunner - tcp vs udp
On 15/06/2008, at 12:45 PM, Mike Lewinski wrote: 2) The biggest drawback to separation after years of service is that customers have come to expect their DNS changes are propagated instantly when they are on-net. This turns out to be more of an annoyance to us than our customers, since our zone is probably the most frequently updated. If you're running bind for your recursive boxes, `rndc -s server flushname zone' run against each recursive box should do the trick for bind 9.3.0 upwards. It will flush the cache for that zone only. There was a bug where it wouldn't flush negative caches, but that might be fixed. YMMV, etc. Usual common sense warnings apply. -- Nathan Ward
