RE: Router / Protocol Problem
Good morning everyone. I just wanted to say thanks for all the help. I did discover the problem this morning and I should be hit with a herring. I upgraded the IOS on the router with the issue to match the other router and the problem was still there. So I tested and noticed the following line in the logs, since I was on console it popped up right in front of me. Sep 7 06:50:20.697 EST: %SEC-6-IPACCESSLOGP: list 166 denied tcp 69.50.222.8(25) - 69.4.74.14(2421), 4 packets What is this I thought? What is my ACL 166 doing this? I thought I tested removing all access-lists from interfaces with the original problem came up. Apparently not. Here is my ACL 166, the first line is what was being matched. Apparently some how this connection is being matched via NBAR for good old Code Red. access-list 166 deny ip any any dscp 1 log access-list 166 deny tcp any any eq sunrpc access-list 166 deny tcp any any eq 135 access-list 166 deny tcp any any eq 137 access-list 166 deny tcp any any eq 138 access-list 166 deny tcp any any eq 139 access-list 166 deny tcp any any eq 445 access-list 166 deny tcp any any eq 5554 access-list 166 deny tcp any any eq 9996 access-list 166 deny tcp any any eq 1025 access-list 166 deny udp any any eq 1434 access-list 166 deny udp any any eq 135 access-list 166 deny udp any any eq netbios-ns access-list 166 deny udp any any eq netbios-dgm access-list 166 deny udp any any eq netbios-ss access-list 166 deny udp any any eq 445 access-list 166 deny icmp any any redirect access-list 166 deny ip 127.0.0.0 0.255.255.255 any access-list 166 deny ip 10.0.0.0 0.255.255.255 any access-list 166 deny ip 172.16.0.0 0.15.255.255 any access-list 166 deny ip 192.168.0.0 0.0.255.255 any access-list 166 permit ip any any class-map match-any http-hacks match protocol http url *default.ida* match protocol http url *cmd.exe* match protocol http url *root.exe* policy-map mark-inbound-http-hacks class http-hacks set ip dscp 1 I have always had this on my FE0/0 as an outbound ACL, well atleast since Code Red came about: ip access-group 166 out. Now I have two questions. Is that not a good idea to have this on FE0/0 out? Second, why the heck would a smtp connection be matched via my http-hacks class-map? Thanks again everyone, Mike -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Rodney Dunn Sent: Wednesday, September 06, 2006 8:45 PM To: Christopher L. Morrow Cc: Rodney Dunn; Mike Walter; Hank Nussbacher; Justin M. Streiner; nanog@merit.edu Subject: Re: Router / Protocol Problem Then that proves it's not a local router problem then. :) On Wed, Sep 06, 2006 at 07:49:26PM +, Christopher L. Morrow wrote: On Wed, 6 Sep 2006, Rodney Dunn wrote: Get a sniffer trace. Packets on the wire prove what's going on. provided the packets get back to him, it seems his problem is traffic getting back to him :( so probably no packets will be on the wire (none in question atleast)...
RE: Router / Protocol Problem
At 07:27 AM 07-09-06 -0400, Mike Walter wrote: Best moved to cisco-nsp. -Hank Nussbacher http://www.interall.co.il Good morning everyone. I just wanted to say thanks for all the help. I did discover the problem this morning and I should be hit with a herring. I upgraded the IOS on the router with the issue to match the other router and the problem was still there. So I tested and noticed the following line in the logs, since I was on console it popped up right in front of me. Sep 7 06:50:20.697 EST: %SEC-6-IPACCESSLOGP: list 166 denied tcp 69.50.222.8(25) - 69.4.74.14(2421), 4 packets What is this I thought? What is my ACL 166 doing this? I thought I tested removing all access-lists from interfaces with the original problem came up. Apparently not. Here is my ACL 166, the first line is what was being matched. Apparently some how this connection is being matched via NBAR for good old Code Red. access-list 166 deny ip any any dscp 1 log access-list 166 deny tcp any any eq sunrpc access-list 166 deny tcp any any eq 135 access-list 166 deny tcp any any eq 137 access-list 166 deny tcp any any eq 138 access-list 166 deny tcp any any eq 139 access-list 166 deny tcp any any eq 445 access-list 166 deny tcp any any eq 5554 access-list 166 deny tcp any any eq 9996 access-list 166 deny tcp any any eq 1025 access-list 166 deny udp any any eq 1434 access-list 166 deny udp any any eq 135 access-list 166 deny udp any any eq netbios-ns access-list 166 deny udp any any eq netbios-dgm access-list 166 deny udp any any eq netbios-ss access-list 166 deny udp any any eq 445 access-list 166 deny icmp any any redirect access-list 166 deny ip 127.0.0.0 0.255.255.255 any access-list 166 deny ip 10.0.0.0 0.255.255.255 any access-list 166 deny ip 172.16.0.0 0.15.255.255 any access-list 166 deny ip 192.168.0.0 0.0.255.255 any access-list 166 permit ip any any class-map match-any http-hacks match protocol http url *default.ida* match protocol http url *cmd.exe* match protocol http url *root.exe* policy-map mark-inbound-http-hacks class http-hacks set ip dscp 1 I have always had this on my FE0/0 as an outbound ACL, well atleast since Code Red came about: ip access-group 166 out. Now I have two questions. Is that not a good idea to have this on FE0/0 out? Second, why the heck would a smtp connection be matched via my http-hacks class-map? Thanks again everyone, Mike -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Rodney Dunn Sent: Wednesday, September 06, 2006 8:45 PM To: Christopher L. Morrow Cc: Rodney Dunn; Mike Walter; Hank Nussbacher; Justin M. Streiner; nanog@merit.edu Subject: Re: Router / Protocol Problem Then that proves it's not a local router problem then. :) On Wed, Sep 06, 2006 at 07:49:26PM +, Christopher L. Morrow wrote: On Wed, 6 Sep 2006, Rodney Dunn wrote: Get a sniffer trace. Packets on the wire prove what's going on. provided the packets get back to him, it seems his problem is traffic getting back to him :( so probably no packets will be on the wire (none in question atleast)... +++ This Mail Was Scanned By Mail-seCure System at the Tel-Aviv University CC.
RE: Router / Protocol Problem
Apparently some how this connection is being matched via NBAR for good old Code Red. Best moved to cisco-nsp. What!? Network operator discovers that measures taken to mitigate an old network security measure, long past their sell-by date, are now causing random grief. Seems to me like bang on topic for NANOG. What other such temporary mitigating measures are still in place long after the danger has passed. Note, that Code RED was a both an application vulnerability and a network DDoS. Even though there are likely still many hosts running the vulnerable application, the number is not sufficient to cause another massive DDoD and measures taken to protect against this particular peculiar DDoS, really don't have a good technical reason to remain in place. This is probably also another instance of the well-known ops problem: We know how to get stuff deployed but we can't undeploy stuff because we are too busy deploying other stuff. --Michael Dillon
Re: Router / Protocol Problem
On Thu, 7 Sep 2006 07:27:16 -0400 Mike Walter [EMAIL PROTECTED] wrote: Sep 7 06:50:20.697 EST: %SEC-6-IPACCESSLOGP: list 166 denied tcp 69.50.222.8(25) - 69.4.74.14(2421), 4 packets [...] I'm not very familiar with NBAR or how to use it for CodeRed, but this first rule: access-list 166 deny ip any any dscp 1 log Seems dubious. So I'm not not sure what sets the codepoint to 01 by default, but apparently CodeRed does? Nevertheless, this seems like a very weak basis for determining whether something is malicious. access-list 166 deny tcp any any eq 5554 access-list 166 deny tcp any any eq 9996 access-list 166 deny tcp any any eq 1025 access-list 166 deny udp any any eq 1434 You may realize this, but I bet some of the rules above I bet are matching on the occasional legitimate packets. Particular the last four rules above. In fact, I bet the rule that matches on TCP destination port 1025 probably has a lot of falsepositives. I'm not sure what you're trying to do with some of them, but if it is to stop some sort of worm, presumably you know that it will also stop applications that happen to choose those source ports. Windows hosts and apps will probably match the 1025 rule fairly frequently, UDP and NTP will match the UDP rule occasionally and various things will the others more or less frequently depending on what traverses your net. Now I have two questions. Is that not a good idea to have this on FE0/0 out? Second, why the heck would a smtp connection be matched via my http-hacks class-map? You don't show the interface config, but my guess is that the SMTP- looking packet may have originally had a codepoint of 1 and didn't really have anything to do with your policy-map. John
Re: Router / Protocol Problem
[EMAIL PROTECTED] writes: Network operator discovers that measures taken to mitigate an old network security measure, long past their sell-by date, are now causing random grief. Seems to me like bang on topic for NANOG. Agreed. Rare that people do haircuts on router configs; they're tedious and can not be delegated to an intern or someone else who doesn't have historical context. I just cut a config by half by removing unused ACLs, and even that is fairly painful. What other such temporary mitigating measures are still in place long after the danger has passed. (?) It's been almost nine and a half years and was a short-lived problem, but I'll betcha that an announcement from AS 7007 will have reachability problems to a measurable fraction of the Internet. That would make a kind of cool experiment. Vinny, you listening? ---Rob
Re: Router / Protocol Problem
Hi John, John Kristoff wrote: On Thu, 7 Sep 2006 07:27:16 -0400 Mike Walter [EMAIL PROTECTED] wrote: Sep 7 06:50:20.697 EST: %SEC-6-IPACCESSLOGP: list 166 denied tcp 69.50.222.8(25) - 69.4.74.14(2421), 4 packets [...] I'm not very familiar with NBAR or how to use it for CodeRed, but this first rule: access-list 166 deny ip any any dscp 1 log Seems dubious. So I'm not not sure what sets the codepoint to 01 by default, but apparently CodeRed does? Nevertheless, this seems like a very weak basis for determining whether something is malicious. It's his NBAR config lower down that sets the dscp value: class-map match-any http-hacks match protocol http url *default.ida* match protocol http url *cmd.exe* match protocol http url *root.exe* policy-map mark-inbound-http-hacks class http-hacks set ip dscp 1 So, there's probably two things that could happen here: One, NBAR is incorrectly identifying the SMTP traffic as code red, or two, the SMTP traffic is already marked with dscp 1. If you've using these values internally in your own network then they should be reset on all externally received traffic. Sam
Re: Router / Protocol Problem
Yeah. Don't want any operational stuff here. Need to get back to who's got a free 300-baud dialup in Antwerp. Hank Nussbacher wrote: At 07:27 AM 07-09-06 -0400, Mike Walter wrote: Best moved to cisco-nsp. -Hank Nussbacher http://www.interall.co.il Good morning everyone. I just wanted to say thanks for all the help. I did discover the problem this morning and I should be hit with a herring. I upgraded the IOS on the router with the issue to match the other router and the problem was still there. So I tested and noticed the following line in the logs, since I was on console it popped up right in front of me. Sep 7 06:50:20.697 EST: %SEC-6-IPACCESSLOGP: list 166 denied tcp 69.50.222.8(25) - 69.4.74.14(2421), 4 packets What is this I thought? What is my ACL 166 doing this? I thought I tested removing all access-lists from interfaces with the original problem came up. Apparently not. Here is my ACL 166, the first line is what was being matched. Apparently some how this connection is being matched via NBAR for good old Code Red. access-list 166 deny ip any any dscp 1 log access-list 166 deny tcp any any eq sunrpc access-list 166 deny tcp any any eq 135 access-list 166 deny tcp any any eq 137 access-list 166 deny tcp any any eq 138 access-list 166 deny tcp any any eq 139 access-list 166 deny tcp any any eq 445 access-list 166 deny tcp any any eq 5554 access-list 166 deny tcp any any eq 9996 access-list 166 deny tcp any any eq 1025 access-list 166 deny udp any any eq 1434 access-list 166 deny udp any any eq 135 access-list 166 deny udp any any eq netbios-ns access-list 166 deny udp any any eq netbios-dgm access-list 166 deny udp any any eq netbios-ss access-list 166 deny udp any any eq 445 access-list 166 deny icmp any any redirect access-list 166 deny ip 127.0.0.0 0.255.255.255 any access-list 166 deny ip 10.0.0.0 0.255.255.255 any access-list 166 deny ip 172.16.0.0 0.15.255.255 any access-list 166 deny ip 192.168.0.0 0.0.255.255 any access-list 166 permit ip any any class-map match-any http-hacks match protocol http url *default.ida* match protocol http url *cmd.exe* match protocol http url *root.exe* policy-map mark-inbound-http-hacks class http-hacks set ip dscp 1 I have always had this on my FE0/0 as an outbound ACL, well atleast since Code Red came about: ip access-group 166 out. Now I have two questions. Is that not a good idea to have this on FE0/0 out? Second, why the heck would a smtp connection be matched via my http-hacks class-map? Thanks again everyone, Mike -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Rodney Dunn Sent: Wednesday, September 06, 2006 8:45 PM To: Christopher L. Morrow Cc: Rodney Dunn; Mike Walter; Hank Nussbacher; Justin M. Streiner; nanog@merit.edu Subject: Re: Router / Protocol Problem Then that proves it's not a local router problem then. :) On Wed, Sep 06, 2006 at 07:49:26PM +, Christopher L. Morrow wrote: On Wed, 6 Sep 2006, Rodney Dunn wrote: Get a sniffer trace. Packets on the wire prove what's going on. provided the packets get back to him, it seems his problem is traffic getting back to him :( so probably no packets will be on the wire (none in question atleast)... +++ This Mail Was Scanned By Mail-seCure System at the Tel-Aviv University CC. -- Requiescas in pace o email Ex turpi causa non oritur actio http://members.cox.net/larrysheldon/
Re: comast email issues, who else has them?
On Thu, 7 Sep 2006, Christopher L. Morrow wrote: Perhaps some of the comcast folks reading might take a better/harder look at their customer service tickets and do a 'better' job (note I'm not even half of a comcast customer so I'm not sure that there even IS a problem...) on this issue? I am a Comcast customer. I'm typing this message over a Comcast line, although you won't find anything in this message header which shows that. No X-Originating-IP, no received line, nothing mandatory showing where I am today. Most ISPs support customer applications because they believe the more services customers use from the provider, the less likely customers will switch providers, not because its mandatory to have those applications. But there is no requirement to use your ISP's mail server or any other application from your ISP. Likewise there is no requirement for a ISP to offer any E-mail or Usenet, or FTP, or legal music downloads, or any other application to its customers. There isn't even a requirement for it to have any customer service. Few of the large free Email providers have any easy way to talk to any human about mail problems. So you don't even get the satisfaction of yelling at a first level tech about your frustrations. In general blaming the first level tech for something isn't going to get anyone anywhere near a solution. Perhaps Sean's actually saying: The right tool is to use another provider? even though Steven's thought is that the 'other provider' is in the same boat of clue :( Mail Service Providers have a wide variety of policies, service levels, delivery performance, etc. Searching Google for free mail provider returns over 500 hits. Yahoo directory lists over 100 free mail providers. There is hardly a monopoly or duopoly of mail service providers, although the big four or five mail providers sometimes try to throw their weight around. Yeah, I know he was joking but what would a E-Mail Neutrality Law do to mail providers? I happen to think the problem is with the bulk mail forwarding services that don't pre-filter mail. But that's just my opinion. I choose not to use unfiltered bulk mail forwarding services so I don't have those problems.
RE: Router / Protocol Problem
On Sep 6, 2006, at 9:04 AM, Mike Walter wrote: Recently with no changes to my network, I have been having problems connecting to certain websites and mail servers. I am always able to ping the sites and trace route without error. If I telnet to port 80 or port 25 it does not connect. If I login to my router and telnet sourcing my each of Internet Providers ports, I am able to get to the sites. I have talked with all the providers and none Check a packet dump and see if your affected boxes are sending SWE (SYN with ECN enabled) instead of plain SYN packets. Some firewalls (at least default m0n0wall and older PIX) reject and dump ECN syn packets while allowing pure syns through. If your affected peer has you running through some weird filter that's dropping SWE packets, this would cause symptoms exactly as you're seeing - ping is fine, traceroute is fine, but TCP sessions never complete the handshake (as the receiving side never got the first SYN). - J
Multiple BGP Routes in FIB
Hi,
There is an interesting discussion going on in the IDR WG and i am
cross posting a mail on Nanog to hear from the operators, if what is
described below, a common practise followed by them:
I don't think its correct to advertise one while using both for
forwarding.
NOTE: I am assuming that the routes share the same path length but have
different AS Paths (as mentioned by you earlier in this mail)
I think this is being done by many providers.
Consider two paths for nlri X
as_path 1 {x y z} next_hop n1
as_path 2 {m n z} next_hop n2
Are you suggesting that providers are installing ecmp routes for X with
next-hops n1 and n2, while advertising only one of the paths to their IBGP
peers?
Yes.
Do providers really do this? Would they install multiple BGP Paths
with different AS Paths (but same length) in their FIB, and yet
advertise only one?
Is the the right thing to do?
Glen
Re: comast email issues, who else has them?
Thus spake Sean Donelan [EMAIL PROTECTED] But there is no requirement to use your ISP's mail server or any other application from your ISP. Likewise there is no requirement for a ISP to offer any E-mail or Usenet, or FTP, or legal music downloads, or any other application to its customers. There isn't even a requirement for it to have any customer service. Few of the large free Email providers have any easy way to talk to any human about mail problems. So you don't even get the satisfaction of yelling at a first level tech about your frustrations. However, the reality is that a significant fraction of users will use their ISP's email service, if one is provided. They'll tolerate minor failures because changing your email address and distributing it to everyone is such a hassle. More and more folks are wising up to this and switching to Yahoo mail or Gmail so they don't have to do it ever again, but OTOH those services are better-run than most ISP mail systems. I happen to think the problem is with the bulk mail forwarding services that don't pre-filter mail. But that's just my opinion. I choose not to use unfiltered bulk mail forwarding services so I don't have those problems. That's not the problem, because I'm not using a bulk mail forwarding service. It's just a single vanity domain hosted by a single Linux box with a half-dozen accounts. And I read the mail _on that box_. There is nothing complicated going on here; we're talking stuff people were doing just fine in the 1980s. I can get email from and send email to anyone on the planet reliably except Comcast customers, which, unfortunately, includes several family members and friends. And even that worked for years; it just broke a few months ago. The real killer is it's broken in both directions; I can't come up with any legitimate reason for that. Inbound (to comcast), I could blame on spam filters, but not outbound. S Stephen Sprunk God does not play dice. --Albert Einstein CCIE #3723 God is an inveterate gambler, and He throws the K5SSSdice at every possible opportunity. --Stephen Hawking
Re: comast email issues, who else has them?
On Sep 6, 2006, at 5:11 PM, Christopher L. Morrow wrote: On Wed, 6 Sep 2006, Stephen Sprunk wrote: Because Comcast's tools are broken and when other mail admins or even their own customers call them on it, they're not even competent enough to understand the complaint and refuse to escalate? I hate to say this, and get involved in the melee, but... Perhaps the problem is that for an average customer service employee there are 1000 calls about something meaningless and not-wrong and only 1 call about something truly wrong? So escalating every problem that seems even half baked isn't an option? Agreed. While working at a small ISP many years ago I used to make it a point to take a few first level support calls a week -- it gives you a new appreciation for the tech support people and helps you understand what really bothers your customers. I also used to get some of the other NEs to take a few calls a week -- understanding the pain it caused (and making customers into real people) cut down on the more intrusive testing[1]. It can also provide you with much entertainment -- for example, I used to get calls asking things like Can I get the Internet in my house?. A few times I asked Depends, how big is your house?, but no one ever got it... Or the little old lady who would call up every few days and say Dearie, the internet is broken again, can you please reboot it?... Warren [1] Where testing means Eh, lets just reload it and see if the problem goes away...
Re: comast email issues, who else has them?
Christopher L. Morrow wroteth on 9/6/2006 5:11 PM: On Wed, 6 Sep 2006, Stephen Sprunk wrote: Because Comcast's tools are broken and when other mail admins or even their own customers call them on it, they're not even competent enough to understand the complaint and refuse to escalate? I hate to say this, and get involved in the melee, but... Perhaps the problem is that for an average customer service employee there are 1000 calls about something meaningless and not-wrong and only 1 call about something truly wrong? So escalating every problem that seems even half baked isn't an option? You're probably right. However, if someone called my place of employment (a small local ISP) and complained followed by quite a few others, I would at least escalate the issue so someone higher than me can check out logs, connectivity, etc.. things I don't have access too to make sure there isn't a problem. What is unfortunate is the fact that this generally doesn't happen. You get lots of calls and Tier I does the obvious and it works and works on those others that call that the issue must be them and it's case closed and nothing gets escalated. It's even worse of the problem gets seemingly solved and the customer doesn't call back for quite a while.. gives the appearance all is well even though it truly is not. Perhaps some of the comcast folks reading might take a better/harder look at their customer service tickets and do a 'better' job (note I'm not even half of a comcast customer so I'm not sure that there even IS a problem...) on this issue? Most ISP's could do a better job. The last ISP I worked at utilized RT for their support. I think a strong ticketing system and using that ticketing system to it's full potential would go a long way in getting things solved faster as well as being able to see trends that could then get escalated without lots of pissed off people having to call and bitch whine and moan before escalation happens. You could easily see an issue with a properly setup ticketing system such as RT. In general blaming the first level tech for something isn't going to get anyone anywhere near a solution. Perhaps Sean's actually saying: The right tool is to use another provider? even though Steven's thought is that the 'other provider' is in the same boat of clue :( ... good point. It may not even be the techs fault on any tier level. It might be company policy, unfortunately.
TCP receive window set to 0; DoS or not?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 New listener, first-time caller. I've been seeing some systems that stop serving pages, and I also see the Linux Treason Uncloaked! kernel messages that indicate a remote system reduced its rcv win from 1 to 0... is there a non-malicious explanation for this, aside from a remote host running out of socket buffers? Seems to happen too often for that to be the case, and my googling has shown that it may be outside of spec. Certainly the warning is clear enough... - -- The whole point of the Internet is that different kinds of computers can interoperate. Every time you see a web site that only supports certain browsers or operating systems, they clearly don't get it. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFAI+QPlSPhv5tocwRAg5hAKCBNvX4U8hAmtfnGImRug5t1IUoBACfbXlS kNJe5BAexBENqtsb1TULL3I= =2Cit -END PGP SIGNATURE-
Re: Router / Protocol Problem
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Seems dubious. So I'm not not sure what sets the codepoint to 01 by default, but apparently CodeRed does? Nevertheless, this seems like a very weak basis for determining whether something is malicious. There is an elegant solution; administrators should set the evil bit on any malicious packets seeking egress; http://www.faqs.org/rfcs/rfc3514.html Quoting: 0x0 If the bit is set to 0, the packet has no evil intent. Hosts, network elements, etc., SHOULD assume that the packet is harmless, and SHOULD NOT take any defensive measures. (We note that this part of the spec is already implemented by many common desktop operating systems.) 0x1 If the bit is set to 1, the packet has evil intent. Secure systems SHOULD try to defend themselves against such packets. Insecure systems MAY chose to crash, be penetrated, etc. And now for something completely different... - -- The whole point of the Internet is that different kinds of computers can interoperate. Every time you see a web site that only supports certain browsers or operating systems, they clearly don't get it. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFAI/WPlSPhv5tocwRAnhrAJ40WgDRn+9fSPXa5U4qZGRRGRbjowCfbBxI AaDLCfYgGF1MjcieyDvuuME= =pibC -END PGP SIGNATURE-
Re: TCP receive window set to 0; DoS or not?
I've been seeing some systems that stop serving pages, and I also see the Linux Treason Uncloaked! kernel messages that indicate a remote system reduced its rcv win from 1 to 0... is there a non-malicious explanation for this, aside from a remote host running out of socket buffers? Seems to happen too often for that to be the case, and my googling has shown that it may be outside of spec. Certainly the warning is clear enough... I've seen this, quite a bit, on some heavy traffic web clusters. Some impolite web browsers will shrink the TCP window to kill the socket connection instead of a proper fin/reset. - billn
Re: TCP receive window set to 0; DoS or not?
On Thu, 7 Sep 2006, Joshua Brewer wrote: What about when we're seeing this on port 25? Sand worms. In all seriousness, your guess is as good as mine, at that point. If memory serves, the platforms we saw this on most, with web browsers, were mobile devices. What kind of volume are you seeing this in? - billn
Re: TCP receive window set to 0; DoS or not?
On Thu, Sep 07, 2006 at 03:04:58PM -0700, [EMAIL PROTECTED] wrote:
I've been seeing some systems that stop serving pages, and I also see
the Linux Treason Uncloaked! kernel messages that indicate a remote
system reduced its rcv win from 1 to 0... is there a non-malicious
explanation for this, aside from a remote host running out of socket
buffers? Seems to happen too often for that to be the case, and
my googling has shown that it may be outside of spec. Certainly
the warning is clear enough...
I've seen this, quite a bit, on some heavy traffic web clusters. Some
impolite web browsers will shrink the TCP window to kill the socket
connection instead of a proper fin/reset.
Advertising a window of 0 is a perfectly valid way of telling the other
side that you are temporarily out of resoruces, and would like them to
stop sending you data. This can be caused by any number of things, from a
completely bogged down box, to an application which isn't read()ing off
its socket buffer (thus for all intents and purposes the kernel is out of
resources to buffer any more data for that socket). It doesn't kill the
TCP session, it just throttles it back. The sender then goes into problem
the zero window mode, waiting for this condition to go away. It is
described in RFC 1122 section 4.2.2.17:
Probing of zero (offered) windows MUST be supported.
A TCP MAY keep its offered receive window closed
indefinitely. As long as the receiving TCP continues to
send acknowledgments in response to the probe segments, the
sending TCP MUST allow the connection to stay open.
etc etc etc
Looking at the Linux code which calls the error message (tcp_timer.c
tcp_retransmit_timer()), the condition which triggers it is:
if (!tp-snd_wnd !sock_flag(sk, SOCK_DEAD)
!((1 sk-sk_state) (TCPF_SYN_SENT | TCPF_SYN_RECV))) {
/* Receiver dastardly shrinks window. Our retransmits
* become zero probes, but we should not timeout this
* connection. If the socket is an orphan, time it out,
* we cannot allow such beasts to hang infinitely.
*/
It looks like it is just detecting this condition and changing its
behavior in accordance with the RFC. Since the actual print of the message
is wrapped in #ifdef TCP_DEBUG, it probably isn't intended to be displayed
to end users at all. As for the cute Treason Uncloaked message, thats
what you get for running an OS written by/for 14 year olds. :)
Or at least thats make 15 minute take on it, having not touched Linux
(gleefully) in many, many years.
--
Richard A Steenbergen [EMAIL PROTECTED] http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: TCP receive window set to 0; DoS or not?
On Thu, 7 Sep 2006, Richard A Steenbergen wrote: Advertising a window of 0 is a perfectly valid way of telling the other side that you are temporarily out of resoruces, and would like them to stop sending you data. This can be caused by any number of things, from a This makes sense when taken in combination with our earlier assessment that it was coming from mobile devices, like PDAs or smart phones, limited on both CPU and optimized to save power. - billn
Re: comast email issues, who else has them?
On Thu, 7 Sep 2006, S. Ryan wrote: Christopher L. Morrow wroteth on 9/6/2006 5:11 PM: something truly wrong? So escalating every problem that seems even half baked isn't an option? You're probably right. However, if someone called my place of employment (a small local ISP) and complained followed by quite a few others, I would at least escalate the issue so someone higher than me can check out logs, connectivity, etc.. things I don't have access too to make sure there isn't a problem. What is unfortunate is the fact that this generally doesn't happen. You get lots of calls and Tier I does the obvious and it works and works on I think you are doing 2 things here: 1) assuming that there is a local and flat (all in one place and within earshot) tier1 2) someone is correlating issues across all tier1 tickets in 'real time' (or near realtime even) those others that call that the issue must be them and it's case closed and nothing gets escalated. It's even worse of the problem gets seemingly solved and the customer doesn't call back for quite a while.. gives the appearance all is well even though it truly is not. Ask a credit card company about the number of sub 10$ fraudulent charges they get on a monthly basis across their customer base, they do nothing to stop it... in fact they don't track it (in most cases) because it's not federally reportable :( Unless someone has a ticketing system that tracks problems and allows you to correlate the events in near-realtime for 'problem caused by' there is no want to know when there is a mass problem :( Or atleast it's much harder to do that correlation :( Perhaps some of the comcast folks reading might take a better/harder look at their customer service tickets and do a 'better' job (note I'm not even half of a comcast customer so I'm not sure that there even IS a problem...) on this issue? Most ISP's could do a better job. The last ISP I worked at utilized RT for their support. I think a strong ticketing system and using that ticketing system to it's full potential would go a long way in getting yes, agree, see above. things solved faster as well as being able to see trends that could then get escalated without lots of pissed off people having to call and bitch whine and moan before escalation happens. You could easily see an issue with a properly setup ticketing system such as RT. .. and someone actually monitoring that ticketting system :) don't forget the 'monitor the system part' because I would guarantee that comcast has some form of ticketting system (just using them since they are in the example/email here...) In general blaming the first level tech for something isn't going to get anyone anywhere near a solution. Perhaps Sean's actually saying: The right tool is to use another provider? even though Steven's thought is that the 'other provider' is in the same boat of clue :( ... good point. It may not even be the techs fault on any tier level. It might be company policy, unfortunately. yes :( try getting support for mci phone service apparently they stopped providing it a while ago :(
Re: TCP receive window set to 0; DoS or not?
On Thu, 7 Sep 2006 [EMAIL PROTECTED] wrote: On Thu, 7 Sep 2006, Joshua Brewer wrote: What about when we're seeing this on port 25? Sand worms. In all seriousness, your guess is as good as mine, at that point. If memory serves, the platforms we saw this on most, with web browsers, were mobile devices. What kind of volume are you seeing this in? I see this on web, ftp, rsync as well... so perhaps it's just impolite people? :)
Re: TCP receive window set to 0; DoS or not?
On Thu, 7 Sep 2006 19:24:02 -0400, Richard A Steenbergen [EMAIL PROTECTED] wrote: Advertising a window of 0 is a perfectly valid way of telling the other side that you are temporarily out of resoruces, and would like them to stop sending you data. This can be caused by any number of things, from a completely bogged down box, to an application which isn't read()ing off its socket buffer (thus for all intents and purposes the kernel is out of resources to buffer any more data for that socket). It doesn't kill the TCP session, it just throttles it back. The sender then goes into problem the zero window mode, waiting for this condition to go away. It is described in RFC 1122 section 4.2.2.17: Probing of zero (offered) windows MUST be supported. A TCP MAY keep its offered receive window closed indefinitely. As long as the receiving TCP continues to send acknowledgments in response to the probe segments, the sending TCP MUST allow the connection to stay open. etc etc etc Advertising a zero window is perfectly proper, Probing one is mandatory, per RFC 793: The sending TCP must be prepared to accept from the user and send at least one octet of new data even if the send window is zero. The sending TCP must regularly retransmit to the receiving TCP even when the window is zero. Two minutes is recommended for the retransmission interval when the window is zero. This retransmission is essential to guarantee that when either TCP has a zero window the re-opening of the window will be reliably reported to the other. But closing an open window is a bad idea: The mechanisms provided allow a TCP to advertise a large window and to subsequently advertise a much smaller window without having accepted that much data. This, so called shrinking the window, is strongly discouraged. The robustness principle dictates that TCPs will not shrink the window themselves, but will be prepared for such behavior on the part of other TCPs. --Steven M. Bellovin, http://www.cs.columbia.edu/~smb
Re: TCP receive window set to 0; DoS or not?
Christopher L. Morrow wrote: I see this on web, ftp, rsync as well... so perhaps it's just impolite people? :) Who knows. My DNS servers get a few of those per day. David Smith MVN.net
Re: comast email issues, who else has them?
On 9/6/06, Stephen Sprunk [EMAIL PROTECTED] wrote: Telling half my family members they have to go get Gmail so they can email the other half of my family members is ridiculous. Too bad Comcast has a monopoly (or, where a duopoly, the competition is just as incompetent) so they have no incentive to fix it. I'm tired of this duopoly fiction. Cable modem providers are usually de facto monopolies, though some of them may use PPPoE or other methods of supporting wholesale service, but DSL is only a monopoly at the copper layer, not at the DSLAM layer in more than half of the US or at the IP layer in almost all of the US. Beyond that, there are a number of service providers that will accept VPN tunnels of various sorts so you can get service access independent of your ISP's policies. That doesn't mean, of course, that it isn't too much hassle for your relatives to want to do that just to avoid the limitations of Comcast, but they've got lots of options if they want them.
