Re: [NANOG] Microsoft.com PMTUD black hole?
On 5/7/08, Tomas L. Byrnes <[EMAIL PROTECTED]> wrote: > I'm not sure what the issue is here. > > Just about every modern firewall I've used has an option to enable PMTU > on interfaces, while blocking all other ICMP. > > Is MS not running something manufactured in the last 10 years at their > perimeter? Unless things have changed drastically since we parted ways, it's a simple ACL applied on all edge interfaces. It should be possible for them to modify it to allow the list of ICMP subtypes listed at http://www.cymru.com/Documents/icmp-messages.html It would *certainly* make troubleshooting easier for the poor folks at Microsoft, since one side effect of the edge filter being set that way meant we couldn't traceroute outside the network; the port unreachable messages never made it back, so everything outside the edge routers was all just stars. Of course, that was in a former lifetime, so it's entirely possible and probable things have changed considerably since then. ^_^;; Matt (speaking only for myself, not for my current employer, and most certainly not for my previous employer who I'm still somewhat bitter at, not having gotten any of my hardware back yet...) ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole? (working with Microsoft on issues)
* [EMAIL PROTECTED] (Janet Sullivan) [Thu 08 May 2008, 23:35 CEST]: >1) Yes, Microsoft blocks ICMP for the most part, which will break Path >MTU Discovery. This is a known issue. If you run into it, its most >likely because the servers you are trying to talk to in MS-land don't >have black hole router detection turned on. I find it hilarious that one part of the company had to come up with a hack to work around the inability of another part of the company to understand how TCP/IP works -- Niels. -- ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole? (working with Microsoft on issues)
I thought I'd post a few constructive comments on this thread. (Full disclosure: I am an ex-Microsoft employee. I do not speak for the company, I'm just trying to help out the network community.) 1) Yes, Microsoft blocks ICMP for the most part, which will break Path MTU Discovery. This is a known issue. If you run into it, its most likely because the servers you are trying to talk to in MS-land don't have black hole router detection turned on. 2) Instead of trying to get all the various ACLs and firewalls in Microsoft fixed to allow PMTUD, you are more likely to experience joy if you can contact the server owners. Ask if they have black hole router detection turned on, and if not, if they can do so. 3) So how do you get in contact with the server owners or MSN's networking people? [EMAIL PROTECTED] is your best bet. That's the email address monitored by the basic Tier 1 "Service Operations Center". They cut tickets, follow scripts, and do very basic front line work. They probably won't be able to fix the problem for you, but they CAN get you in touch with the right people. 4) FINDING the right people can be a challenge, even internally. Microsoft is a very big company, and its far from centralized. Be specific in what URLs and IPs you are having trouble with, and be prepared to bounce around a bit. The people who run microsoft.com's servers aren't the same group that does hotmail, etc. Have patience, and try to get ticket numbers for tracking at much as possible. 5) Try to give a realistic estimate of how many users are being impacted by the problem. Your problem will be triaged as it moves through various groups, and yes, the response time may not be what you want. Your problem is one fire among many, and there aren't enough firefighters. 6) Be nice. Seriously. People love to hate Microsoft, and sometimes take it out on the poor overworked geeks who are trying to actually make things better. Every vulnerability, BSOD, or Vista delay is not the fault of the network or systems engineer you get in touch with. ;-) ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
On Wed, 7 May 2008, Michael Sinatra wrote: > Nathan Anderson/FSR wrote: >> Here is a brief update on the situation: >> >> I have been in contact with someone at Microsoft's service operations >> center, who has confirmed for me that MS does in fact block _all_ ICMP >> at the edge of their network, that they are aware that this will in fact >> break PMTUD, and that they have no current plans to change this practice >> which they have implemented in the interest of security. > > Although the need for your previous apology has already been questioned > in this forum, the confirmation that they block not only certain ICMP > types, but all ICMP, further vacates the need for any apology for > criticizing this behavior in a pubic forum. It is disheartening for > those of us who use and support MSFT's products to learn that their > understanding of security lacks even the basic nuance to know not to > block an entire--critical--portion of the Internet Protocol. Perhaps > they should also block _all_ TCP and UDP as well, and then we can move on. > > I agree with Iljitsch that it happens frequently, but I think I am > justified in expecting more than that from Microsoft. Anything less > would be unprofessional. I wonder if MS knows about: ICMP Packet Filtering v1.2 from 2003: http://www.cymru.com/Documents/icmp-messages.html Only been around 5 years or so. Hopefully MS people reading this email will take note, read the entire page and implement what everyone else has been doing for a number of years. -Hank ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
-- [EMAIL PROTECTED] wrote: --- First of all I would like to thank everyone for their support and concern. We certainly have a lot of things to "fix" at Microsoft. In fact, I can tell you that we have several brand new positions open (working on my team and for teams near mine) and could use more hands at the tiller. My apologies to the moderators for posting a help wanted but I figure since folks are expressing concerns with Microsoft networking they should have the opportunity to come over and help. --- Not support. Concern about M$ on s many levels. Your apologies should go to the other 10,000 folks that had to endure your improper email: http://www.nanog.org/listfaq.html#nocjobs BOSD Network Engineer 231413 BOSD Senior Network Engineer231414 heh, you mean BSOD engineer? ;-) scott ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
First of all I would like to thank everyone for their support and concern. We certainly have a lot of things to "fix" at Microsoft. In fact, I can tell you that we have several brand new positions open (working on my team and for teams near mine) and could use more hands at the tiller. My apologies to the moderators for posting a help wanted but I figure since folks are expressing concerns with Microsoft networking they should have the opportunity to come over and help. We have an INCREDIBLE amount of interesting work ahead of us. I can't really speak to it directly but I can tell you that this is a really good place to be if you think big. We have a very interesting playbook for the next few years. Keep in mind that this is one of the few spots where thinking big can actually result in action. The current positions... Principal Network Engineer 204091 Senior Network Engineer 185014 SR PM 220793 IT/Ops PM 2 220797 Network Engineer 3 226032 Group Manager, Core Engineering 227621 Senior Network Engineer 229347 BOSD Network Engineer 231413 BOSD Senior Network Engineer231414 If you think you have what it takes go check out these positions at http://www.microsoft.com/careers and apply (the numbers to the right are the job codes). Microsoft is an incredible place to work if you truly enjoy what you do. To be honest I don't read as much NANOG as I did a number of years ago. I had a couple friends point this thread out to me. I hope you are all doing well. Regards, Blaine ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
On Wed, 7 May 2008, Deepak Jain wrote: > > I know of a tool that a major financial institution uses when certifying > your networks security -- that scrapes the version number from your > ESTMP banner to decide whether you comply or not (and other banners). > (Rather than actually testing for a specific vulnerability). Simply > blocking all of these packets from their test host gives you a high > passing score; possibly a perfect one. [Irony and humor aside...] Cisco PIX/ASA firewalls in SMTP fuxup mode are so incredibly broken. Possibly the worst SMTP implementation ever. Tony. -- f.anthony.n.finch <[EMAIL PROTECTED]> http://dotat.at/ FISHER GERMAN BIGHT: VARIABLE 3, BUT EASTERLY 4 OR 5 IN SOUTH GERMAN BIGHT. SLIGHT. FOG PATCHES. MODERATE OR GOOD, OCCASIONALLY VERY POOR. ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
On 8 mei 2008, at 9:53, Joel Jaeggli wrote: > Oddly enough there is a draft on the subject of icmp filtering > recomendations is making the rounds. > http://tools.ietf.org/wg/opsec/draft-gont-opsec-icmp-filtering-00.txt > The opsec working group ([EMAIL PROTECTED]) and the authors would > appreciate feedback from operators on the subject. Speaking as someone who isn't interested in reading an explanation of what happens when the message is filtered for every ICMP message known to man, I find this a completely useless document: I can't find the recommendations. Either they're there but impossible to find by looking at the table of contents or searching for "recommend", or they're not there in which case the title is EXTREMELY misleading. Also: 2.1.1.5.4. Operational/interoperability impact if blocked Filtering this error message breaks the Path-MTU Discovery mechansim described in [RFC1191]. This is completely insufficient because it doesn't mention that 99% of all TCP traffic on today's internet uses PMTUD and filtering these messages leads to broken connectivity towards destinations that have an MTU lower than the source (lower than 1500 in practice). Please spell check and five levels of numbering is considered bad style. ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
Bjørn Mork wrote: > Iljitsch van Beijnum <[EMAIL PROTECTED]> writes: > After all, Microsoft must have a reason to block all icmp. Or? > >> However, in that case the only workable course of action would be TO >> DISABLE PATH MTU DISCOVERY! >> >> You can't have your cake and eat it too. > > But maybe the death of icmp is worth some sort of ceremony? Cake or > not. Oddly enough there is a draft on the subject of icmp filtering recomendations is making the rounds. http://tools.ietf.org/wg/opsec/draft-gont-opsec-icmp-filtering-00.txt The opsec working group ([EMAIL PROTECTED]) and the authors would appreciate feedback from operators on the subject. thanks joelja > Bjørn > > ___ > NANOG mailing list > NANOG@nanog.org > http://mailman.nanog.org/mailman/listinfo/nanog > ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
Iljitsch van Beijnum <[EMAIL PROTECTED]> writes: > Now Microsoft is also the company that built the OS that could be > crashed by a maliciously crafted fragmented IP packet, so maybe > there's something to this security policy. (One hopes that this bug > and others like it are now fixed.) Although the fact that Microsoft block all icmp makes me wonder which unfixed icmp related security holes they know about... I am not saying that there are any such holes in current Windows versions, but I will certainly not use a Windows server in an environment where I could receive icmp after learning that Microsoft themselves don't trust Windows' icmp handling. After all, Microsoft must have a reason to block all icmp. Or? > However, in that case the only workable course of action would be TO > DISABLE PATH MTU DISCOVERY! > > You can't have your cake and eat it too. But maybe the death of icmp is worth some sort of ceremony? Cake or not. Bjørn ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
On 7-May-2008, at 17:07:06, Deepak Jain wrote: > Many non-SP IT folks think they understand TCP, grudgingly accept > UDP for DNS from external sources and think everything else is > bollocks. Many *might* have a fit if they saw Microsoft accepting > ICMPs because that seems inconsistent with their knowledge of turn- > the-knob network security. To their view, their Linksys/Netgear/ > whathaveyou COTS firewalls block everything too. > > I don't think I'm exaggerating here. No, you are not. I have seen the same from "firewall engineers" at large companies, people who, supposedly, have done "network security" for years. Even after showing them numerous Web sites detailing current best practices, especially Rob Thomas's fine site, these folks would not change their practices. Some days it is hard to not give in to the "I give up" feelings. ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
Nathan Anderson/FSR wrote: > Nevertheless, the person I have been in contact with is naturally not > the final decision-maker on this issue and is going to continue to pass > the issue on up the chain of command for me. So although this issue is > not over and I do not have a final verdict from MS yet, I felt that, > given that I don't know how much time to expect to pass between now and > when that final verdict is rendered, it would be appropriate to let > everybody here know what I have learned thus far. Hopefully public > dissemination of this information factoid will prevent others in a > position similar to mine from having to helplessly beat their heads into > their keyboards. Let's also not ignore the generally overworked IT administrator at any small or medium sized enterprise. He/she may not be (as many folks I've run into are) of the mistaken impression that ICMP *is* bad and leaves you vulnerable to all sorts of things like SMURF. There are even tools out there that "test" your vulnerability by "pinging" you and do other investigations. I know of a tool that a major financial institution uses when certifying your networks security -- that scrapes the version number from your ESTMP banner to decide whether you comply or not (and other banners). (Rather than actually testing for a specific vulnerability). Simply blocking all of these packets from their test host gives you a high passing score; possibly a perfect one. [Irony and humor aside...] Many non-SP IT folks think they understand TCP, grudgingly accept UDP for DNS from external sources and think everything else is bollocks. Many *might* have a fit if they saw Microsoft accepting ICMPs because that seems inconsistent with their knowledge of turn-the-knob network security. To their view, their Linksys/Netgear/whathaveyou COTS firewalls block everything too. I don't think I'm exaggerating here. Just a thought, not saying its a good one or whose fault it is... Deepak Jain AiNET ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
Tomas L. Byrnes wrote: > As far as who Iljitsch is, everyone misspeaks from time to time. Even > those of us who have been at this for nearly 3 decades. I was simply LOLing at the fact that you found it necessary to give him a link to the NetHeaven article is all. ;-) -- Nathan Anderson First Step Internet, LLC [EMAIL PROTECTED] ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
Iljitsch van Beijnum wrote: > The problem is in the direction from M$ to you, so you can't fix that > from your end. I wonder if they've installed SP3 on their servers... Ah, you are right. I re-read the section on black-hole detection in http://technet.microsoft.com/en-us/library/bb878081.aspx more closely this time, and found that, yes, it only helps if the host trying to send the large packets has the feature enabled: "When PMTU black hole router detection is enabled, TCP tries to send segments with the DF flag set to 0 after several retransmissions of a segment are not acknowledged. If a segment with the DF flag set to 0 is acknowledged, the MSS is decreased and the DF flag is set to 1 in subsequent segments on the connection. Enabling PMTU black hole detection increases the maximum number of retransmissions that are performed for a given segment, and therefore has an effect on overall performance." I for some reason interpreted the advertisement of the black hole detection feature as being a help to clients impacted by the inability of the server to perform PMTUD. -- Nathan Anderson First Step Internet, LLC [EMAIL PROTECTED] ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
Sorry if I misunderstood what you were saying. I thought you said that they couldn't have their cake and eat it too, as in protect against Ping of death, AND do PMTUD. As for those flames, well, that was a long time ago, in a valley not too far from here ;-). Can we have the 'net before the endless September back? > -Original Message- > From: Iljitsch van Beijnum [mailto:[EMAIL PROTECTED] > Sent: Wednesday, May 07, 2008 2:40 PM > To: Tomas L. Byrnes > Cc: [EMAIL PROTECTED] > Subject: Re: [NANOG] Microsoft.com PMTUD black hole? > > On 7 mei 2008, at 23:20, Tomas L. Byrnes wrote: > > > I was responding to his post that blocking or disabling > PMTUD was the > > way to avoid the ping of death, which is False, nothing > more, nothing > > less. > > I never said that disabling PMTUD will get rid of the ping of > death, what I said was that if your system is susceptible to > a ping of death you may be tempted to filter ICMP but if you > do that then you need to disable PMTUD because PMTUD + ICMP > filtering = breakage. > > > As far as who Iljitsch is, everyone misspeaks from time to > time. Even > > those of us who have been at this for nearly 3 decades. > > After making the jump to academia I often feel a bit long in > the tooth between all these students. But considering that > (apparently) some people have been posting flames on NANOG > for 30 years makes me feel young in comparison. :-) > ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
On 7 mei 2008, at 23:20, Tomas L. Byrnes wrote: > I was responding to his post that blocking or disabling PMTUD was the > way to avoid the ping of death, which is False, nothing more, nothing > less. I never said that disabling PMTUD will get rid of the ping of death, what I said was that if your system is susceptible to a ping of death you may be tempted to filter ICMP but if you do that then you need to disable PMTUD because PMTUD + ICMP filtering = breakage. > As far as who Iljitsch is, everyone misspeaks from time to time. Even > those of us who have been at this for nearly 3 decades. After making the jump to academia I often feel a bit long in the tooth between all these students. But considering that (apparently) some people have been posting flames on NANOG for 30 years makes me feel young in comparison. :-) ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
I was responding to his post that blocking or disabling PMTUD was the way to avoid the ping of death, which is False, nothing more, nothing less. As far as who Iljitsch is, everyone misspeaks from time to time. Even those of us who have been at this for nearly 3 decades. > -Original Message- > From: Nathan Anderson/FSR [mailto:[EMAIL PROTECTED] > Sent: Wednesday, May 07, 2008 2:08 PM > To: [EMAIL PROTECTED] > Subject: Re: [NANOG] Microsoft.com PMTUD black hole? > > Tomas L. Byrnes wrote: > > > The remedy you have below is NOT the only one, and is, in fact, a > > non-sequitur in this case. > > How so? Iljitsch is suggesting that ICMP blockers originate > packets without DF set if they are going to block the ICMP > messages that PMTUD needs in order to work in the first > place. That's what (I think) he means by "disabling path MTU > discovery." > > > The network-level solution to ping of death is to BLOCK fragmented > > packets, and the way to ensure this doesn't self-deny-service is to > > perform PMTUD and Black-Hole Router discovery. > > Which end are you talking about here, the servers or the > client? If the servers, how do you expect them to do PMTUD > if they _can't hear the ICMP messages_? > > Also, for some reason, as I pointed out before, XP black hole > router discovery doesn't seem to be working for me for > whatever reason. Does anybody have any clue why that might > be the case? > > -- > Nathan Anderson > First Step Internet, LLC > [EMAIL PROTECTED] > > ___ > NANOG mailing list > NANOG@nanog.org > http://mailman.nanog.org/mailman/listinfo/nanog > ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
On 7 mei 2008, at 23:08, Nathan Anderson/FSR wrote: > How so? Iljitsch is suggesting that ICMP blockers originate packets > without DF set if they are going to block the ICMP messages that PMTUD > needs in order to work in the first place. That's what (I think) he > means by "disabling path MTU discovery." Yes. > Also, for some reason, as I pointed out before, XP black hole router > discovery doesn't seem to be working for me for whatever reason. Does > anybody have any clue why that might be the case? The problem is in the direction from M$ to you, so you can't fix that from your end. I wonder if they've installed SP3 on their servers... ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
Kevin Oberman wrote: >> I agree with Iljitsch that it happens frequently, but I think I am >> justified in expecting more than that from Microsoft. Anything less >> would be unprofessional. > > And you would consider an organization that threatens someone who > complains publicly about its obvious incompetence "professional"? Absolutely not. That was actually the point of my statement, although I admit that it wasn't clear. > Many of Microsoft's people are highly professional, but the corporation, > as a whole, has been found to be large scale law breakers on two > continents and frequently incapable of even the most basic of technical > operations. I'm afraid that I don't see them as at all "professional". I > quit expecting any such behavior from them over a decade ago, probably > closer to two. And mentioning security and Microsoft is inviting bad > jokes and shudders. > > * Speaking only for myself * Agreed. michael ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
Tomas L. Byrnes wrote: > Some Edumacation on the topic is here: You do know who it is that you are responding to, right? :) http://www.oreillynet.com/pub/au/970 -- Nathan Anderson First Step Internet, LLC [EMAIL PROTECTED] ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
Tomas L. Byrnes wrote: > The remedy you have below is NOT the only one, and is, in fact, a > non-sequitur in this case. How so? Iljitsch is suggesting that ICMP blockers originate packets without DF set if they are going to block the ICMP messages that PMTUD needs in order to work in the first place. That's what (I think) he means by "disabling path MTU discovery." > The network-level solution to ping of death is to BLOCK fragmented > packets, and the way to ensure this doesn't self-deny-service is to > perform PMTUD and Black-Hole Router discovery. Which end are you talking about here, the servers or the client? If the servers, how do you expect them to do PMTUD if they _can't hear the ICMP messages_? Also, for some reason, as I pointed out before, XP black hole router discovery doesn't seem to be working for me for whatever reason. Does anybody have any clue why that might be the case? -- Nathan Anderson First Step Internet, LLC [EMAIL PROTECTED] ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
Tomas L. Byrnes wrote: > I'm not sure what the issue is here. > > Just about every modern firewall I've used has an option to enable PMTU > on interfaces, while blocking all other ICMP. > > Is MS not running something manufactured in the last 10 years at their > perimeter? Not sure, but you actually entered in here to a subthread of the original conversation, this one about other possible ways of dealing with black hole "ICMP-munchers" in a pre-emptive fashion. I had a brainstorm that I thought would be workable, which is what we were discussing here. Apparently, it turns out my idea was no good. ;-) The original discussion about MS blocking ICMP to their own servers, which is the discussion it sounds like you are looking for, is over that-a-way... *points* -- Nathan Anderson First Step Internet, LLC [EMAIL PROTECTED] ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
Some Edumacation on the topic is here: http://www.netheaven.com/pmtu.html > -Original Message- > From: Iljitsch van Beijnum [mailto:[EMAIL PROTECTED] > Sent: Wednesday, May 07, 2008 1:35 PM > To: Michael Sinatra > Cc: [EMAIL PROTECTED] > Subject: Re: [NANOG] Microsoft.com PMTUD black hole? > > On 7 mei 2008, at 21:46, Michael Sinatra wrote: > > >> MS does in fact block _all_ ICMP > >> at the edge of their network, that they are aware that > this will in > >> fact break PMTUD, and that they have no current plans to > change this > >> practice which they have implemented in the interest of security. > > > Perhaps > > they should also block _all_ TCP and UDP as well, and then > we can move > > on. > > > I agree with Iljitsch that it happens frequently, but I think I am > > justified in expecting more than that from Microsoft. > Anything less > > would be unprofessional. > > Right. > > Now Microsoft is also the company that built the OS that > could be crashed by a maliciously crafted fragmented IP > packet, so maybe there's something to this security policy. > (One hopes that this bug and others like it are now fixed.) > > However, in that case the only workable course of action > would be TO DISABLE PATH MTU DISCOVERY! > > You can't have your cake and eat it too. > > ___ > NANOG mailing list > NANOG@nanog.org > http://mailman.nanog.org/mailman/listinfo/nanog > ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
The remedy you have below is NOT the only one, and is, in fact, a non-sequitur in this case. PMTUD uses the DF (for Don't_Fragment) bit, and works by getting an ICMP Fragmentation needed response from the hop on the path where the packet is too large, not a fragmentation and forward, so the union of PMTUD packets and fragmented ones is 0. The network-level solution to ping of death is to BLOCK fragmented packets, and the way to ensure this doesn't self-deny-service is to perform PMTUD and Black-Hole Router discovery. > -Original Message- > From: Iljitsch van Beijnum [mailto:[EMAIL PROTECTED] > Sent: Wednesday, May 07, 2008 1:35 PM > To: Michael Sinatra > Cc: [EMAIL PROTECTED] > Subject: Re: [NANOG] Microsoft.com PMTUD black hole? > > On 7 mei 2008, at 21:46, Michael Sinatra wrote: > > >> MS does in fact block _all_ ICMP > >> at the edge of their network, that they are aware that > this will in > >> fact break PMTUD, and that they have no current plans to > change this > >> practice which they have implemented in the interest of security. > > > Perhaps > > they should also block _all_ TCP and UDP as well, and then > we can move > > on. > > > I agree with Iljitsch that it happens frequently, but I think I am > > justified in expecting more than that from Microsoft. > Anything less > > would be unprofessional. > > Right. > > Now Microsoft is also the company that built the OS that > could be crashed by a maliciously crafted fragmented IP > packet, so maybe there's something to this security policy. > (One hopes that this bug and others like it are now fixed.) > > However, in that case the only workable course of action > would be TO DISABLE PATH MTU DISCOVERY! > > You can't have your cake and eat it too. > > ___ > NANOG mailing list > NANOG@nanog.org > http://mailman.nanog.org/mailman/listinfo/nanog > ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
On 7 mei 2008, at 21:46, Michael Sinatra wrote: >> MS does in fact block _all_ ICMP >> at the edge of their network, that they are aware that this will in >> fact >> break PMTUD, and that they have no current plans to change this >> practice >> which they have implemented in the interest of security. > Perhaps > they should also block _all_ TCP and UDP as well, and then we can > move on. > I agree with Iljitsch that it happens frequently, but I think I am > justified in expecting more than that from Microsoft. Anything less > would be unprofessional. Right. Now Microsoft is also the company that built the OS that could be crashed by a maliciously crafted fragmented IP packet, so maybe there's something to this security policy. (One hopes that this bug and others like it are now fixed.) However, in that case the only workable course of action would be TO DISABLE PATH MTU DISCOVERY! You can't have your cake and eat it too. ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
Nathan Anderson/FSR wrote: > Here is a brief update on the situation: > > I have been in contact with someone at Microsoft's service operations > center, who has confirmed for me that MS does in fact block _all_ ICMP > at the edge of their network, that they are aware that this will in fact > break PMTUD, and that they have no current plans to change this practice > which they have implemented in the interest of security. Although the need for your previous apology has already been questioned in this forum, the confirmation that they block not only certain ICMP types, but all ICMP, further vacates the need for any apology for criticizing this behavior in a pubic forum. It is disheartening for those of us who use and support MSFT's products to learn that their understanding of security lacks even the basic nuance to know not to block an entire--critical--portion of the Internet Protocol. Perhaps they should also block _all_ TCP and UDP as well, and then we can move on. I agree with Iljitsch that it happens frequently, but I think I am justified in expecting more than that from Microsoft. Anything less would be unprofessional. *Speaking for myself only, of course!* michael ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
I'm not sure what the issue is here. Just about every modern firewall I've used has an option to enable PMTU on interfaces, while blocking all other ICMP. Is MS not running something manufactured in the last 10 years at their perimeter? > -Original Message- > From: Nathan Anderson/FSR [mailto:[EMAIL PROTECTED] > Sent: Wednesday, May 07, 2008 12:39 PM > To: [EMAIL PROTECTED] > Cc: [EMAIL PROTECTED] > Subject: Re: [NANOG] Microsoft.com PMTUD black hole? > > [EMAIL PROTECTED] wrote: > > > The usual case where you get screwed over is when the > router trying to > > toss the ICMP FRAG NEEDED is *behind* the ICMP-munching > firewall. And > > in case (2), you still can't assume that path MTU == local MTU, > > because your local MTU is likely 1500, and the fragging > router often > > trying to stuff your 1500 byte packet down an PPPoE tunnel > that's got an MTU of 1492 > > Yes, but my point was precisely that one OR the other side (server OR > client) is going to NOT have the ICMP-munching firewall in > between itself and the "RITM" as I have affectionately been > calling it (although it is definitely possible that there are > two ICMP-munchers on either side of the RITM). > > And case #2 is exactly what is occurring right now _anyway_: > hosts assume that path MTU == local MTU even if there is > already an active PMTU cache entry from a recent earlier > communication with the remote host. So I don't see how > making that assumption _after_ making an honest attempt at > actively determining whether or not it is actually the case > is any more broken than they way things are already being done. > > The problem is that, as I realized at the end of the message > you quoted, there are potentially multiple paths between the > same two hosts, and the path that the packet takes in one > direction is not guaranteed to be the same path that the > packet takes in the opposite direction. > > -- > Nathan Anderson > First Step Internet, LLC > [EMAIL PROTECTED] > > ___ > NANOG mailing list > NANOG@nanog.org > http://mailman.nanog.org/mailman/listinfo/nanog > ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
[EMAIL PROTECTED] wrote: > The usual case where you get screwed over is when the router trying to toss > the ICMP FRAG NEEDED is *behind* the ICMP-munching firewall. And in case (2), > you still can't assume that path MTU == local MTU, because your local MTU is > likely 1500, and the fragging router often trying to stuff your 1500 byte > packet down an PPPoE tunnel that's got an MTU of 1492 Yes, but my point was precisely that one OR the other side (server OR client) is going to NOT have the ICMP-munching firewall in between itself and the "RITM" as I have affectionately been calling it (although it is definitely possible that there are two ICMP-munchers on either side of the RITM). And case #2 is exactly what is occurring right now _anyway_: hosts assume that path MTU == local MTU even if there is already an active PMTU cache entry from a recent earlier communication with the remote host. So I don't see how making that assumption _after_ making an honest attempt at actively determining whether or not it is actually the case is any more broken than they way things are already being done. The problem is that, as I realized at the end of the message you quoted, there are potentially multiple paths between the same two hosts, and the path that the packet takes in one direction is not guaranteed to be the same path that the packet takes in the opposite direction. -- Nathan Anderson First Step Internet, LLC [EMAIL PROTECTED] ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
Here is a brief update on the situation: I have been in contact with someone at Microsoft's service operations center, who has confirmed for me that MS does in fact block _all_ ICMP at the edge of their network, that they are aware that this will in fact break PMTUD, and that they have no current plans to change this practice which they have implemented in the interest of security. Nevertheless, the person I have been in contact with is naturally not the final decision-maker on this issue and is going to continue to pass the issue on up the chain of command for me. So although this issue is not over and I do not have a final verdict from MS yet, I felt that, given that I don't know how much time to expect to pass between now and when that final verdict is rendered, it would be appropriate to let everybody here know what I have learned thus far. Hopefully public dissemination of this information factoid will prevent others in a position similar to mine from having to helplessly beat their heads into their keyboards. I, naturally, voiced my strong objection over this security policy, and attempted to make a reasoned argument with the contact I have over there. We will see what comes of this. Some have asked me to post copies of my private communication with my Microsoft contact here. I don't think it is appropriate for me to post copies of private communication without the other party's consent, so I will have to decline unless he first gives me said consent. Others have asked for valid contact information for the Microsoft NOC, since the ARIN records for their 207.46.0.0/16 do not appear to be up to date. I eventually found a working e-mail address from somebody off-list who pointed to the WHOIS lookup from TUCOWS for microsoft.comosoft.com (which I'm still not clear on what exactly this is...). The e-mail address that was gleaned from this lookup was [EMAIL PROTECTED], which goes to the Microsoft Corporate Domains Team. They, in turn, forwarded my message on to [EMAIL PROTECTED], which generated a ticket # for me and is, as I understand it, the e-mail address I was looking for in the first place (leads to their network/system people). I hope this is helpful to others. Regards, -- Nathan Anderson First Step Internet, LLC [EMAIL PROTECTED] ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
Iljitsch van Beijnum wrote: > No. This would add significant delay because you'd have to give the > other side enough time to respond to the large packet (also sending a > large packet on something like GPRS/EDGE is a waste of bandwidth and > battery power) while if there is ICMP filtering, there won't be a > response, which is exactly the reason why we're in this bind in the > first place I admit the idea needs tweaking (at best), and it was just a stray thought :-), but 1) even if there is ICMP filtering happening way at the other end, I (the TCP initiator) will still get a response from the router in the middle (RITM) that is reducing the total path MTU if I try to send a packet through it larger than the actual path MTU, and 2) if I don't get a response to my single large packet (either from a RITM or the other end) in a timely fashion (less than a second?), then the client/initiator may just assume that path MTU == local MTU and will set its MSS accordingly (which is no different than what is happening now), until it has a reason to think differently. Also, if there is already something in the local PMTU cache for a single host address, I'm not sure I follow why it would be a bad idea for the TCP initiator to consult that cache when preparing the SYN. Although, on second thought, I suppose it is possible (and, in more than a few cases, likely) that in instances of route path asymmetry, the PMTU of the path from the initiator to the server may be different than the PMTU of the path back from the server to the client. Hmmm. Okay, scratch that idea then. :-P -- Nathan Anderson First Step Internet, LLC [EMAIL PROTECTED] ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
Thus spake "Nathan Anderson/FSR" <[EMAIL PROTECTED]> > A member of Microsoft's GNS network escalations team saw my > postings on NANOG about this issue and took offense at my use > of this forum to raise this issue with them, and criticized me as > being unprofessional and lacking in business acumen. First, it's "unprofessional and lacking in business acumen" for someone to criticize their customers to their face. As one manager taught me, "The customer may not always be right, but they're never wrong." Second, it's their own damn fault for not maintaining their contact information properly in public databases. If the only option they leave you is to post to NANOG, because they don't respond to (or even accept) direct requests to the listed contacts, then that's what you have to do. Many companies are guilty of the latter, and we all get the benefit of seeing the state of their customer service for reference when making future buying decisions. Very few are arrogant enough to do the former, though. S Stephen Sprunk "God does not play dice." --Albert Einstein CCIE #3723 "God is an inveterate gambler, and He throws the K5SSSdice at every possible opportunity." --Stephen Hawking ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
Glen Turner wrote: > Amazing. A fine case study of a person in customer contact undoing the > work of millions of dollars in PR. Whatever you say about Steve Ballmer > he's a great sales person at heart. He must despair at some of his staff. > The rest of us however, despair at having to support their crap. Patrick Giagnocavo [EMAIL PROTECTED] ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
On Tue, May 06, 2008 at 06:12:42PM -0700, Nathan Anderson/FSR wrote: > A member of Microsoft's GNS network escalations team saw my postings on > NANOG about this issue and took offense at my use of this forum to raise > this issue with them, and criticized me as being unprofessional and > lacking in business acumen. This is a typical Microsoft reaction: blame the messenger for their own incompetence, laziness, stupidity, and greed. I think you should post their assinine message so that it can receive the public ridicule it surely deserves. ---Rsk ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
Iljitsch van Beijnum <[EMAIL PROTECTED]> writes: > Many years ago I had occasion to terminate dial-up service over L2TP > from modem pools operated by a service provider who shall remain > nameless to protect the guilty. This service had the unfortunate > tendency to drap all packets larger than 576 bytes. So we needed to > negotiate a 576-byte MTU over PPP. > > We then got many complaints from users who dialed in using ISDN > routers (yes this was a while ago) because of broken path MTU > discovery. The behavior that Microsoft exhibits was EXTREMELY common > in those days, and I have no reason to assume it's any less common > today. (I also see it regularly with IPv6.) What I did was clear the > DF bit on packets going out to the L2TP virtual interfaces so the > packets could be fragmented. Right. I once stumbled across a SOHO-router doing just that. I never understood why, but now you've given at least one explanation how it could appear to be a good idea. I can also provide the reason why we found it to be an extremely bad idea at the time: Some (most? all?) systems won't set both the DF flag and the identification field at the same time. If you clear the DF flag without changing the identification field, you might end up with fragmented packets that are impossible to reassemble. Which was why I stumbled across the DF-clearing SOHO-router in the first place. The random problems it generated were extremely difficult to debug, and when we started we truly believed that we had a problem with a layer 4 load balancing switch. Note: There are solutions that will both clear the DF flag and generate a new id. E.g. http://www.openbsd.org/faq/pf/scrub.html This is the proper way to clear DF, if you must. Never just clear it. Bjørn ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
On 07/05/2008, at 4:42 PM, Glen Turner wrote: > Amazing. A fine case study of a person in customer contact undoing the > work of millions of dollars in PR. I wouldn't worry too much about it, Glen. My observation is that the millions of dollars in PR isn't working very well either :-) - mark -- Mark Newton Email: [EMAIL PROTECTED] (W) Network Engineer Email: [EMAIL PROTECTED] (H) Internode Systems Pty Ltd Desk: +61-8-82282999 "Network Man" - Anagram of "Mark Newton" Mobile: +61-416-202-223 ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
Nathan Anderson/FSR wrote: > A member of Microsoft's GNS network escalations team saw my postings on > NANOG about this issue and took offense at my use of this forum to raise > this issue with them, and criticized me as being unprofessional and > lacking in business acumen. Hang on a tick. Aren't you one of their customers.. > As I pointed out in my post earlier today timestamped at 2:29PM, I was > using an XP SP3 host to perform my tests with... ...why yes, you are. I can't think of any other supplier that would be so unprofessional and so lacking in business acumen as to say that their customer was UALIBI. Amazing. A fine case study of a person in customer contact undoing the work of millions of dollars in PR. Whatever you say about Steve Ballmer he's a great sales person at heart. He must despair at some of his staff. -- Glen Turner ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
> A member of Microsoft's GNS network escalations team saw my postings on > NANOG about this issue and took offense at my use of this forum to raise > this issue with them, and criticized me as being unprofessional and > lacking in business acumen. they try that intimidation every time a vulnerability or bug is revealed. laugh and post their overly-aggressive message on a public web site. randy ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
On 6 mei 2008, at 23:29, Nathan Anderson/FSR wrote: > Now, although that makes sense, in order to avoid issues like the > one we > are facing with Microsoft, would it not make _more_ sense for the > stack > to look at the PMTU cache first, and then adjust its own MSS just for > connections to that one host? Maybe even send out an MTU - 40 ICMP > packet to the host that we want to build a TCP connection with FIRST > to > get an ICMP type 3 code 4 response from the router in-between with the > smaller MTU? No. This would add significant delay because you'd have to give the other side enough time to respond to the large packet (also sending a large packet on something like GPRS/EDGE is a waste of bandwidth and battery power) while if there is ICMP filtering, there won't be a response, which is exactly the reason why we're in this bind in the first place (along with the stupid idea that DF should be set for ALL packets rather than just once in a while). And adjusting the MSS based on ephemeral information is the wrong thing to do in the first place. The path MTU can vary. Once you've advertised a small MSS you can never increase it. It is incredibly unprofessional that people enable PMTUD, then break it and require the rest of the world to implement workarounds. Either use PMTUD properly by accepting the ICMP messages or turn PMTUD off. ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
On 6 mei 2008, at 23:48, [EMAIL PROTECTED] wrote: >> A more common approach is to rewrite the MSS option in all TCP SYNs >> with a smaller value so there won't be TCP segments large enough to >> trigger the problem. AFAIK, all boxes that do PPPoE do this. > And just the other day, you were saying: >> Very few people out there use an MTU significantly below 1500 >> bytes. A >> 1500-byte MTU will give you an _average_ packet size of ~1000 on >> long- >> lived TCP flows because there is one tiny ACK for every two full size >> data segments. Right. Why is that noteworthy? I have a lot more to say about MTU issues in this draft about negotating MTUs between two hosts/routers on a subnet so jumboframes can be deployed without manual configuration: http://www.ietf.org/internet-drafts/draft-van-beijnum-multi-mtu-02.txt > Apparently, there's a *reason* why RFC1122, section 3.3.3 says: > It is generally desirable to avoid local fragmentation and to > choose EMTU_S low enough to avoid fragmentation in any gateway > along the path. In the absence of actual knowledge of the > minimum MTU along the path, the IP layer SHOULD use > EMTU_S <= 576 whenever the destination address is not on a > connected network, and otherwise use the connected network's > MTU. Tell it to Microsoft and their ICMP-filtering friends... ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
Tomas L. Byrnes wrote: > Interestingly, Windows XP, Sp3, released today, describes changes in > PMTUD behavior. > > Black Hole Router detection is now on by default: As I pointed out in my post earlier today timestamped at 2:29PM, I was using an XP SP3 host to perform my tests with, and it made no difference. I also used BBR's DrTCP application to make sure that black hole router detection was, in fact, enabled on my XP box before commencing my packet captures. I cannot explain why it made no difference, but at the same time I don't know enough about how WinNT's black hole router detection works to begin speculating at this point. I do plan on looking into it, however. -- Nathan Anderson First Step Internet, LLC [EMAIL PROTECTED] ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
All, A member of Microsoft's GNS network escalations team saw my postings on NANOG about this issue and took offense at my use of this forum to raise this issue with them, and criticized me as being unprofessional and lacking in business acumen. Therefore, I would like to publicly apologize for my actions here. It was not my intention to "humiliate" Microsoft into compliance but rather to find a means of effective contact with them since none was to be found before today. However, I recognize that I did step over the line, especially with regards to one comment I made in an earlier post about "giving Microsoft too much credit." I apologize for this and retract this, and ask their forgiveness. As I promised, I will not be posting any more to this list regarding this issue unless it is to report the final verdict that I receive from my now-open ticket with Microsoft (thanks to this list, I found an effective contact), or to discuss the mechanics of PMTUD in general. Regards, -- Nathan Anderson First Step Internet, LLC [EMAIL PROTECTED] ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
On May 6, 2008, at 8:51 PM, Tomas L. Byrnes wrote: > Interestingly, Windows XP, Sp3, released today, describes changes in > PMTUD behavior. > > Black Hole Router detection is now on by default: > > http://download.microsoft.com/download/6/8/7/687484ed-8174-496d-8db9-f02 > b40c12982/Overview%20of%20Windows%20XP%20Service%20Pack%203.pdf > <http://download.microsoft.com/download/6/8/7/687484ed-8174-496d-8db9-f02b40c12982/Overview%20of%20Windows%20XP%20Service%20Pack%203.pdf > or http://tinyurl.com/323xb Regards > >> -Original Message- >> From: Robert Bonomi [mailto:[EMAIL PROTECTED] >> Sent: Tuesday, May 06, 2008 3:54 PM >> To: [EMAIL PROTECTED] >> Subject: Re: [NANOG] Microsoft.com PMTUD black hole? >> >> ` >> >>> Date: Tue, 06 May 2008 14:29:03 -0700 >>> From: Nathan Anderson/FSR <[EMAIL PROTECTED]> >>> Subject: Re: [NANOG] Microsoft.com PMTUD black hole? >>> >>> >>> Now, although that makes sense, in order to avoid issues >> like the one >>> we are facing with Microsoft, would it not make _more_ >> sense for the >>> stack to look at the PMTU cache first, and then adjust its own MSS >>> just for connections to that one host? >> >> This _is_ Microsoft we're talking about, remember. 'sense' >> and 'Microsoft' >> are, at a =minimum= orthogonal to each other -- and may not >> even inhabit the same address-space. >> >> As for standards, it is official Microsoft policy to "embrace >> and extend", >> not to implement in a way compatible with the rest of the >> world. *sigh* >> >> I -don't- believe the rumor that "PMTUD/Vista Ultimate" sends >> incrementally increasing-size packets, and uses the first one >> that -doesn't- get through >> as the size limit. >> >> >> >> ___ >> NANOG mailing list >> NANOG@nanog.org >> http://mailman.nanog.org/mailman/listinfo/nanog >> > > ___ > NANOG mailing list > NANOG@nanog.org > http://mailman.nanog.org/mailman/listinfo/nanog ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
Interestingly, Windows XP, Sp3, released today, describes changes in PMTUD behavior. Black Hole Router detection is now on by default: http://download.microsoft.com/download/6/8/7/687484ed-8174-496d-8db9-f02 b40c12982/Overview%20of%20Windows%20XP%20Service%20Pack%203.pdf > -Original Message- > From: Robert Bonomi [mailto:[EMAIL PROTECTED] > Sent: Tuesday, May 06, 2008 3:54 PM > To: [EMAIL PROTECTED] > Subject: Re: [NANOG] Microsoft.com PMTUD black hole? > > ` > > > Date: Tue, 06 May 2008 14:29:03 -0700 > > From: Nathan Anderson/FSR <[EMAIL PROTECTED]> > > Subject: Re: [NANOG] Microsoft.com PMTUD black hole? > > > > > > Now, although that makes sense, in order to avoid issues > like the one > > we are facing with Microsoft, would it not make _more_ > sense for the > > stack to look at the PMTU cache first, and then adjust its own MSS > > just for connections to that one host? > > This _is_ Microsoft we're talking about, remember. 'sense' > and 'Microsoft' > are, at a =minimum= orthogonal to each other -- and may not > even inhabit the same address-space. > > As for standards, it is official Microsoft policy to "embrace > and extend", > not to implement in a way compatible with the rest of the > world. *sigh* > > I -don't- believe the rumor that "PMTUD/Vista Ultimate" sends > incrementally increasing-size packets, and uses the first one > that -doesn't- get through > as the size limit. > > > > ___ > NANOG mailing list > NANOG@nanog.org > http://mailman.nanog.org/mailman/listinfo/nanog > ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
` > Date: Tue, 06 May 2008 14:29:03 -0700 > From: Nathan Anderson/FSR <[EMAIL PROTECTED]> > Subject: Re: [NANOG] Microsoft.com PMTUD black hole? > > > Now, although that makes sense, in order to avoid issues like the one we > are facing with Microsoft, would it not make _more_ sense for the stack > to look at the PMTU cache first, and then adjust its own MSS just for > connections to that one host? This _is_ Microsoft we're talking about, remember. 'sense' and 'Microsoft' are, at a =minimum= orthogonal to each other -- and may not even inhabit the same address-space. As for standards, it is official Microsoft policy to "embrace and extend", not to implement in a way compatible with the rest of the world. *sigh* I -don't- believe the rumor that "PMTUD/Vista Ultimate" sends incrementally increasing-size packets, and uses the first one that -doesn't- get through as the size limit. ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
Nathan Anderson/FSR wrote: [...] > connections to that one host? Maybe even send out an MTU - 40 ICMP :s/40/sized. Brain fart. -- Nathan Anderson First Step Internet, LLC [EMAIL PROTECTED] ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
Iljitsch van Beijnum wrote: > A more common approach is to rewrite the MSS option in all TCP SYNs [snip] Yeah, we do this now, but the software that we have been using for PPPoE termination as well as for a huge portion of our clients (MikroTik RouterOS) doesn't do it correctly in my estimation when you flip on the automatic "change-tcp-mss" option...it rewrites the MSS in ALL SYNs passing through it, either coming OR going. This has the effect of breaking communication with other hosts that actually have a SMALLER MSS than our PPPoE customers since our client will get a SYN+ACK from the remote host that we have rewritten to reflect a larger MSS than the remote host is capable of dealing with. Because MikroTik rewrote both the SYNs generated by us as well as received by us, our customer's host is now under the impression that the lowest MSS between the two hosts matches its own. At least that's the best theory I've come up with. We can write (and have written) custom IP manglers on the MikroTik boxes that only touch SYNs generated by our clients, and only when the MSS is larger than a certain value (in order to honor MSSes even lower than that allowed by their PPPoE gateway). But it's a PITA to deal with. I'd just rather everyone follow protocol. :-P Although we can't always expect everyone to do it by the book, I don't think it is too much to ask that those who operate sizable networks that nearly everyone is required to interact with on a daily basis (read: Microsoft) act responsibly. > All of this even went so far that the IETF came up with RFC 4821, > which will do path MTU discovery by correlating lost packets with > packet sizes to determine the path MTU rather than depend on ICMP > messages. What's funny is that I ran my tests from a Windows XP host with the recently-released Service Pack 3 installed, which is supposed to activate Microsoft's "PMTUD Black Hole Router Detection" by default (available pre-SP3 but apparently not turned on without a registry change). I haven't read up on exactly how it's supposed to work, but I think the basic idea is that if the TCP connection is negotiated properly but it doesn't get a response beyond that, it will try lower and lower MSSes until it does. However it works (or doesn't as the case may be), it didn't make a lick of difference. I waited and waited for content to be delivered to me until eventually Microsoft's end sent me a TCP RST. While I was poking at this, though, I had a thought...most IP stacks I believe keep a path MTU cache of some sort. I know Windows does: if I send an ICMP packet with DF set that is larger than the PPPoE gateway can handle, I get something similar to the following: C:\Documents and Settings\nathana>ping 64.126.160.1 -f -l 1472 Pinging 64.126.160.1 with 1472 bytes of data: Reply from 64.126.142.249: Packet needs to be fragmented but DF set. Packet needs to be fragmented but DF set. Packet needs to be fragmented but DF set. Packet needs to be fragmented but DF set. [...] Next time that I try the same thing, Windows doesn't even bother trying to send the packet. It looks at its PMTU table for that IP, and already KNOWS it is too big: C:\Documents and Settings\nathana>ping 64.126.160.1 -f -l 1472 Pinging 64.126.160.1 with 1472 bytes of data: Packet needs to be fragmented but DF set. Packet needs to be fragmented but DF set. [...] However, even when trying this with www.msnbc.msn.com, and with the MSNBC entry in its PMTU cache (and its IP set statically in my 'hosts' file so that Akamai/MS round-robin DNS doesn't screw with me during the test), when I tried to build a TCP connection to MSNBC from this same host, Windows told the remote host it had a 1460 MSS. Now, although that makes sense, in order to avoid issues like the one we are facing with Microsoft, would it not make _more_ sense for the stack to look at the PMTU cache first, and then adjust its own MSS just for connections to that one host? Maybe even send out an MTU - 40 ICMP packet to the host that we want to build a TCP connection with FIRST to get an ICMP type 3 code 4 response from the router in-between with the smaller MTU? That would put the burden of PMTUD on the host requesting the TCP session rather than on the one responding, but if hosts were "smarter" like this it seems to me it might smooth out some of these issues. The remote end could be "broken" with respect to PMTUD but it wouldn't matter. Thoughts? -- Nathan Anderson First Step Internet, LLC [EMAIL PROTECTED] ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
Brandon Butterworth wrote: > I used to see it a lot when hosting on windows was popular and people > realised they needed a firewall or decided to add a load balancer > but broke PMTUD by leaving it enabled on the servers. Yeah, but this is Microsoft's OWN server farm we are talking about here, not some small podunk IIS-based hosting provider. ...well, you may be right. I am probably giving MS too much credit here. On another note, someone pointed out to me off-list that I apparently tyop'd "hostmaster" when I sent the e-mail to MS. I have since re-sent it to the properly-spelled address and again promptly received a "User unknown" bounceback. -- Nathan Anderson First Step Internet, LLC [EMAIL PROTECTED] ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
On 6 mei 2008, at 21:58, Brandon Butterworth wrote: >> Has anyone else here seen problems with microsoft/msn/hotmail/ >> live.com >> sites not performing PMTUD correctly? > I used to see it a lot when hosting on windows was popular and people > realised they needed a firewall or decided to add a load balancer > but broke PMTUD by leaving it enabled on the servers. > I've not heard of it for some time so those people got > a clue or moved to something else (or everyone worked around them) Many years ago I had occasion to terminate dial-up service over L2TP from modem pools operated by a service provider who shall remain nameless to protect the guilty. This service had the unfortunate tendency to drap all packets larger than 576 bytes. So we needed to negotiate a 576-byte MTU over PPP. We then got many complaints from users who dialed in using ISDN routers (yes this was a while ago) because of broken path MTU discovery. The behavior that Microsoft exhibits was EXTREMELY common in those days, and I have no reason to assume it's any less common today. (I also see it regularly with IPv6.) What I did was clear the DF bit on packets going out to the L2TP virtual interfaces so the packets could be fragmented. A more common approach is to rewrite the MSS option in all TCP SYNs with a smaller value so there won't be TCP segments large enough to trigger the problem. AFAIK, all boxes that do PPPoE do this. All of this even went so far that the IETF came up with RFC 4821, which will do path MTU discovery by correlating lost packets with packet sizes to determine the path MTU rather than depend on ICMP messages. ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Re: [NANOG] Microsoft.com PMTUD black hole?
> Has anyone else here seen problems with microsoft/msn/hotmail/live.com > sites not performing PMTUD correctly? I used to see it a lot when hosting on windows was popular and people realised they needed a firewall or decided to add a load balancer but broke PMTUD by leaving it enabled on the servers. I've not heard of it for some time so those people got a clue or moved to something else (or everyone worked around them) brandon ___ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
[NANOG] Microsoft.com PMTUD black hole?
Hello, Has anyone else here seen problems with microsoft/msn/hotmail/live.com sites not performing PMTUD correctly? We have, for a while now, had people on our network complain of poor microsoft.com reachability, and discovered we can work around the issue by changing MSS on all TCP SYN as they go out of our network. I recently watched the whole conversation between msn.com and a host on our network (with the MSS rewrite disabled), and if I'm reading it right, we are following PMTUD protocol correctly by sending back ICMP type 3 code 4, but all Microsoft hosts seem to ignore this and continue to send packets back to our host with an MSS that is too large. I hope I'm wrong and that it is we who are doing something stupid, but after cruising Google for a while, I found a multitude of other complaints from people connected to other ISPs specifically about not being able to reach Microsoft web sites. It seems crazy that MS could have PMTUD broken for so long with nobody ever raising a complaint to them directly, though, which makes me wonder if there is another answer here that I'm missing. I sent the following message to a couple of addresses that I gleaned from ARIN WHOIS for the IP block in question and threw hostmaster in there just in case it went somewhere, but [EMAIL PROTECTED] appears to be defunct. I have yet to receive acknowledgment of receipt from the other address. Are there any microsoft.com admins that hang out here that can comment on this or get in touch with me, or is there perhaps someone on here with connections to the Microsoft NOC? (BTW, I stripped the referenced libpcap attachment off of this message to the list just so that I wouldn't accidentally incur the wrath of NANOG...if y'all want to see it, I'm happy to post it.) Thanks, -- Nathan Anderson First Step Internet, LLC [EMAIL PROTECTED] Original Message Subject: Microsoft/MSN/Live!/Hotmail behind blackhole router? Date: Thu, 01 May 2008 19:00:46 -0700 From: Nathan Anderson/FSR <[EMAIL PROTECTED]> To: [EMAIL PROTECTED], [EMAIL PROTECTED], [EMAIL PROTECTED] To microsoft.com NOC admins: I work for a regional ISP in the inland pacific northwest. May of our customers' connections have MTUs of less than 1500, and we get routine complaints from them that they have trouble reaching web sites that are under your administration. Usually we can fix the problem by "mangling" the TCP SYNs originating from our customers and headed to the world to reflect a lower value; however, we would rather not have to do that. The fact that we are REQUIRED to do this in order for your sites to be reachable by our customers strongly suggests that either the servers that respond to HTTP requests sent to www.microsoft/msn/hotmail/live.com are behind routers that are blocking ALL ICMP traffic sent their way -- even ICMP type 3 code 4 (packet too large, DF set), which is necessary in order for Path MTU Discovery to work -- or the servers themselves are not listening to the ICMP messages that we are sending their way when our routers are forced to drop a packet sent by you which is too large to be forwarded to a customer of ours. I set up a test connection "on the bench" so to speak, and had our router capture a copy of the conversation between our test client and www.msnbc.msn.com and forward that conversation encapsulated in TZSP to the same test client over a different interface. The capture clearly shows our test client establishing the TCP connection with MSNBC (SYN/SYN+ACK/ACK), and then goes on to show MSNBC send ethernet MTU-sized packets our way that an intermediate router of ours drops and responds with "packet too big, DF set." Despite this, MSNBC continues to retrasmit the original packet with the same payload and the same size back to us. We continue to respond "packet too big, DF set," but the MSNBC server never seems to get the message (literally). We see the same behavior with all sites across the board contained within the 207.46.0.0/16 space, regardless of actual hostname/FQDN. We also find this ironic considering that Microsoft published a Technet article a few years back on black hole routers and the problems they pose, found at http://technet.microsoft.com/en-us/library/bb878081.aspx (which we can't read/access unless we are mangling the MSS). We would appreciate it if Microsoft NOC admins would please look into the matter and take the appropriate corrective action: allowing ICMP type 3 code 4 messages through your routers/firewalls, and making sure that your servers respond to them appropriately as defined in RFC 1191. I have attached the capture we made of the conversation to this e-mail message in libpcap format for your analysis. The test client itself had a 1500 MTU to a desktop router, which in turn had an MTU of 1492 on its uplink to us. I am available to answer any additional clarifying questions you may have. Thank you for your time and attention to this matter. Regar
