Re: [lisp] Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Mon, Jul 18, 2011 at 12:15 PM, Noel Chiappa j...@mercury.lcs.mit.edu wrote: Let me make sure I understand your point here. You don't seem to be disagreeing with the assertion that for most sites (even things like very large universities, etc), their 'working set' (of nodes they communicate) with will be much smaller than the network as a whole? Why would you assume this to be true if LISP also promises to make multi-homing end-sites cheaper and easier, and independent of the ISP's willingness to provide BGP without extra cost? You see, if every SOHO network and power user can suddenly become multi-homed without spending a great deal of money on a powerful router and ISP services which support BGP, many of these networks will do so. The working sets of a scaled-up, LISP future will make the BGP DFZ of today look small. So only the very largest content providers (YouTube, etc) will have 'working sets' which include a fairly large share of the entire Internet? No, any end-site of interest to a DoS attacker must be able to deal with a working set which includes the entire Internet. The reason for this is obvious: it will be the best way to attack a LISP infrastructure, and it will not be difficult for attackers to send packets where each packet's source address appears to be from a different mapping system entry. Some people have commented that LISP hopes to prevent source address spoofing through technical means that have not been fully explored. This is a good goal but it must require the ETR doing address validation to look-up state from the mapping system. It will have the same cache churn problem as an ITR subject to a reflection attack (or an outbound DoS flow meant to disable that ITR.) So there is no practical means of doing source address validation on ETRs (under DoS.) Even if you did that, the ITR must still be subject to the occasional large flow of outbound traffic from a compromised host (dorm machine, open wireless, hacked server, etc.) which is intended to disable the ITR. I have previously commented that such sites have lots of specialized infrastructure to handle their traffic loads - do you think it will be infeasible for them to have specialized LISP infrastructure too? (Leaving aside for a moment what that infrastruture would look like - it's not necessarily separate hardware, it might be integrated into existing boxes on the periphery of their site.) Again, every content shop will need to have that specialized infrastructure. Every site that someone might have a motive to launch a DoS attack against must be able to withstand at least trivial DoS. If you think only the super-huge sites will have a large working set, you are again ignoring DoS attacks. The same is true of ISP subscriber access platforms. If my ISP's BRAS effectively goes down regularly, I won't keep that ISP service very long, I'll change to a competitor. The more subscribers on one BRAS, the more likely it will receive frequent DoS attacks. So in reality, the common cache size needed to achieve a high hit rate really does not matter, unless you wish to ignore DoS (which you seem to want to do very badly.) -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts _ NANOG mailing list NANOG@nanog.org https://mailman.nanog.org/mailman/listinfo/nanog
Re: NDP DoS attack (was Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?))
On Jul 15, 2011, at 10:24 AM, Jimmy Hess wrote: In most cases if you have a DoS attack coming from the same Layer-2 network that a router is attached to, it would mean there was already a serious security incident that occured to give the attacker that special point to attack fr This scenario is quite common in physical/virtual co-lo IDCs, FYI - customer A attacking customer B, both within the same subnet, in many cases. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com The basis of optimism is sheer terror. -- Oscar Wilde _ NANOG mailing list NANOG@nanog.org https://mailman.nanog.org/mailman/listinfo/nanog
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
We all make mistakes in not questioning our own positions, from time to time. You, Jeff, seem to be making that very same mistake. Please keep these points in mind: * Rome wasn't built in a day. The current system didn't come ready-made pre-built with all the bells and whistles you are used to. It grew slowly over time, as we learned what works, what doesn't, and what was missing. Any system that attempts to deal with locator/id separation will assuredly not be built in a day, either. * While you have stated a problem relating to a security consideration – specifically that there is a potential reflection attack that could cause cache thrashing, the solution may not be what you expect. * Yes, you were asked. Even so... Novelty isn't something worth arguing over, except in patent battles. Usefulness is only worth arguing over marginally more. Deployment (or lack thereof) speaks for itself. LISP or ILNP or what-have-you either will or won't be deployed over the long run. * Never is a very long time. Many uses of never have been used relating to the Internet. It is the corollary to Imminent Death of the 'Net: film @ 11. I still have the NANOG tee-shirt with Robert Metcalfe, someone with considerably more notoriety, eating his hat. Eliot _ NANOG mailing list NANOG@nanog.org https://mailman.nanog.org/mailman/listinfo/nanog
Re: NDP DoS attack (was Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?))
On Mon, Jul 11, 2011 at 8:17 PM, Karl Auer ka...@biplane.com.au wrote: RFC3756 IPv6 Neighbor Discovery (ND) Trust Models and Threats In this attack, the attacking node begins fabricating addresses with the subnet prefix and continuously sending packets to them. The last hop router is obligated to resolve these addresses by sending neighbor solicitation packets. A legitimate host attempting to enter the network may not be able to obtain Neighbor Discovery service from the last hop router as it will be already busy with sending other solicitations. Hi Karl, My off-the-cuff naive solution to this problem would be to discard the oldest incomplete solicitation to fit the new one and, upon receiving an apparently unsolicited response to a discarded solicitation, restart the process flagging that particular query non-discardable. That would be an implementation change, not a protocol change. I would expect to occasionally lose a packet due to the discard while the router was under attack with the accordingly minimal impact. I would also expect to see a multicast flood on the LAN of about the same data rate as the inbound attack packets. Where does this naive approach break down? On Fri, Jul 15, 2011 at 12:13 AM, Fernando Gont ferna...@gont.com.ar wrote: On 07/15/2011 12:24 AM, Jimmy Hess wrote: A similarly hazardous situation exists with IPv4, and it is basically unheard of for IPv4's Layer 2/ARP security weaknesses to be exploited to create a DoS condition, even though they can be (very easily), IMO, the situation is different, in that the typical IPv4 subnet size eliminate some of the attack vectors. Hi Fernando, Not at a practical level. The reason it's unheard of for IPv4 is that if you're a hacker with an ability to generate arbitrary packets on a LAN, DOSing the adjacent router by overwhelming its ARP cache is one of the least interesting things you can do... and one of the easiest to get busted at. It isn't necessary (or possible) to solve every conceivable *local* DOS attack. And frankly remote saturation-bomb attacks are out of bounds too. The concern Karl presented was that it was possible to remotely disable an IPv6 LAN with tailored traffic much less than that network's inbound capacity. Solve that issue with IPv6 ND and we're done. Regards, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004 _ NANOG mailing list NANOG@nanog.org https://mailman.nanog.org/mailman/listinfo/nanog
Re: NDP DoS attack (was Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?))
On Sun, Jul 17, 2011 at 11:42 AM, William Herrin b...@herrin.us wrote: My off-the-cuff naive solution to this problem would be to discard the oldest incomplete solicitation to fit the new one and, upon receiving an apparently unsolicited response to a discarded solicitation, restart the process flagging that particular query non-discardable. Do you mean to write, flagging that ND entry non-discardable? Once the ND entry is in place, it should not be purged for quite some time (configurable is a plus), on the order of minutes or hours. Making them permanent would, however, cause the ND table to eventually become full when foolish things like frequent source address changes for privacy are in use, many clients are churning in and out of the LAN, etc. Where does this naive approach break down? It breaks down because the control-plane can't handle the relatively small number of punts which must be generated in order to send ND solicits, and without the ability to install incomplete entries into the data-plane, those punts cannot be policed without, by design, discarding some good punts along with the bad punts resulting from DoS traffic. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts _ NANOG mailing list NANOG@nanog.org https://mailman.nanog.org/mailman/listinfo/nanog
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Sun, Jul 17, 2011 at 11:07 AM, Eliot Lear l...@cisco.com wrote: We all make mistakes in not questioning our own positions, from time to time. You, Jeff, seem to be making that very same mistake. Rome wasn't built in a day. The current system didn't come ready-made pre-built with all the bells and whistles you are used to. It grew slowly over time, as we learned what works, what doesn't, and what was missing. Any system that attempts to deal with locator/id separation will assuredly not be built in a day, either. LISP work has been going on for a long time to still not have any useful discussion on a designed-in, trivial DoS which will affect any ITR and make the work being done to allow ETRs to validate source addresses (or even do loose uRPF) into a DoS vector for ETRs as well. While you have stated a problem relating to a security consideration – specifically that there is a potential reflection attack that could cause cache thrashing, the solution may not be what you expect. I agree, a solution might be available. One has not been presented yet. In my earliest postings to the IETF LISP list, the ones which received zero replies, I suggest a way to significantly improve the cache churn DoS problem. It is not novel, as Darrel Lewis informed me, which means that even already-available research has not been applied to LISP in this area, and the Mapping Service protocol ties the hands of implementors so they *cannot* apply such techniques while still conforming to the specifications. Yes, you were asked. Even so... Novelty isn't something worth arguing over, except in patent battles. Really? Novelty, by definition, advances the state of the art. You may not think it's very important to inform people that LISP is based on essentially the same flow-caching scheme used in the 1990s, but I do. Never is a very long time. Many uses of never have been used relating to the Internet. It is the corollary to Imminent Death of the 'Net: film @ 11. I still have the NANOG tee-shirt with Robert Metcalfe, someone with considerably more notoriety, eating his hat. And yet, I am quite comfortable with the statement that LISP can never scale up to meet the demands of the Internet. Perhaps with fundamental changes to its design, and its advocates giving up some of their current assumptions, some progress could be made. In its current form, though, LISP will never be a useful tool to scale the Internet, and in fact, it cannot meet the demands of today's Internet. Unless, of course, you pretend that the ability to DoS any router with a trivial amount of traffic is not worthy of concern. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts _ NANOG mailing list NANOG@nanog.org https://mailman.nanog.org/mailman/listinfo/nanog
Re: NDP DoS attack (was Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?))
On Jul 17, 2011, at 10:35 AM, Jeff Wheeler wrote: On Sun, Jul 17, 2011 at 11:42 AM, William Herrin b...@herrin.us wrote: My off-the-cuff naive solution to this problem would be to discard the oldest incomplete solicitation to fit the new one and, upon receiving an apparently unsolicited response to a discarded solicitation, restart the process flagging that particular query non-discardable. Do you mean to write, flagging that ND entry non-discardable? Once the ND entry is in place, it should not be purged for quite some time (configurable is a plus), on the order of minutes or hours. Making them permanent would, however, cause the ND table to eventually become full when foolish things like frequent source address changes for privacy are in use, many clients are churning in and out of the LAN, etc. I believe it was obvious in the context he means flagging the incomplete ND entry as one that is not to be discarded early for pruning of potentially DOS-related ND entries. Basically an ND entry would have the following states and timers: Flagbits: I Incomplete (1 = ND entry is not complete) D Discardable (1 = Incomplete entry is result of incoming packet for unverified neighbor) Timers: A Age -- Counts up from time ND entry was created (most likely synthetic and calculated by storing T in the ND entry and using Tnow - Tentry). The system would have a two timer policies: 1 for Incomplete Timeout and 1 for Complete Timeout. (TI and TC) At A=TI, an incomplete entry would be discarded regardless of the D flag. At A=TC, a complete entry would be discarded regardless of the D flag. When a packet arrives for a host which does not exist in the ND table, a new entry with flags I and D would be created. An ND request would be generated as normal. When a new ND table entry is required and the table is full, the oldest entry with both I and D flags (max(A)) would be replaced with the new entry. When an ND response is received matching an entry with the I flag set, the I and D flags would be cleared and the entry would be filled in with the appropriate data. When an ND response is received not matching an entry with the I flag set, a new entry with the I flag and no D flag would be created. In this way, you cannot overflow the neighbor table in a way that creates significant disruption unless there are actually too many neighbors, in which case, it's bad network design and not DOS. Where does this naive approach break down? It breaks down because the control-plane can't handle the relatively small number of punts which must be generated in order to send ND solicits, and without the ability to install incomplete entries into the data-plane, those punts cannot be policed without, by design, discarding some good punts along with the bad punts resulting from DoS traffic. I think most of this punting could be handled at the line card level. Is there any reason that the ND process can't be moved into line-card level silicon as described above? Sure, that doesn't solve the problem on current hardware, but, it moves it from design problem to implementation issue, which IMHO is a step in the right direction. Owen _ NANOG mailing list NANOG@nanog.org https://mailman.nanog.org/mailman/listinfo/nanog
Re: NDP DoS attack (was Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?))
On Sun, Jul 17, 2011 at 3:40 PM, Owen DeLong o...@delong.com wrote: Basically an ND entry would have the following states and timers: I've discussed what you have described with some colleagues in the past. The idea has merit and I would certainly not complain if vendors included it (as a knob) on their boxes. The downfalls of this approach are that they still don't ensure the discovery of new neighbors (rather than ever seen neighbors) during DoS, and you make the local DoS a bit more complex by needing to establish more rules for purging these semi-permanent entries. I think most of this punting could be handled at the line card level. Is there any reason that the ND process can't be moved into line-card level silicon as described above? You could implement ND solicit in the data-plane (and remove punts entirely) in even some current chips, to say nothing of future ones. Whether or not that is a good idea, well, keep in mind that the ND solicits would then be mcasted to the LAN at a potentially unlimited rate. That is not necessarily a problem unless the L2 implementation is not too good with respect to multicast. For example, in some switches (mostly those that are routers that can switch) the L2 mcast has surprising caveats, such as using up a lot of fabric capacity for whatever replication scheme has been chosen. Of course, you also hope NDP on all the connected hosts works right. I believe some Juniper customers noticed a pretty big problem with JUNOS NDP implementation when deploying boxes using the DE-CIX addressing scheme, and in a situation like that, the ingress router for the attack could be crippled by spurious responses from the other mis-behaving hosts on the LAN, essentially like smurf except without sending any garbage back out to the Internet. What you definitely don't want to do is assume this fixes the local DoS, because it doesn't. I would like for you to keep in mind that a host on the LAN, misconfigured to do something like local proxy-arp, or otherwise responding to all ND solicits, would accidentally DoS the LAN's gateway. I do not think we should assume that the local DoS won't happen, or is fixable with a whack-a-mole method. Sure, that doesn't solve the problem on current hardware, but, it moves it from design problem to implementation issue, which IMHO is a step in the right direction. Well, it already is a design problem that implementations can largely work-around. Vendors just aren't doing it. :-/ -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts _ NANOG mailing list NANOG@nanog.org https://mailman.nanog.org/mailman/listinfo/nanog
Re: NDP DoS attack (was Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?))
On Sun, Jul 17, 2011 at 1:35 PM, Jeff Wheeler j...@inconcepts.biz wrote: On Sun, Jul 17, 2011 at 11:42 AM, William Herrin b...@herrin.us wrote: My off-the-cuff naive solution to this problem would be to discard the oldest incomplete solicitation to fit the new one and, upon receiving an apparently unsolicited response to a discarded solicitation, restart the process flagging that particular query non-discardable. Do you mean to write, flagging that ND entry non-discardable? Hi Jeff, I meant flagging the new incomplete solicitation ineligible for previous sentence's early load-based discard. When you get a response to a solicitation you no longer remember making, solicit again and don't forget about it until the normal protocol timeouts this time. Where does this naive approach break down? It breaks down because the control-plane can't handle the relatively small number of punts which must be generated in order to send ND solicits, and without the ability to install incomplete entries into the data-plane, those punts cannot be policed without, by design, discarding some good punts along with the bad punts resulting from DoS traffic. Let me try to restate what you've said to make sure I understand. When the data plane knows what ARP or ND is underway, it can guard against overwhelming the control plane by discarding excessive traffic for the same yet-unresolved destination while allowing packets for new destinations on the lan through to the control plane. Without that knowledge, it can only have one queue causing the data plane to discard packets which would initiate neighbor discovery prior to the control plane seeing them, preventing any solicitation or implementing the logic I described. This doesn't sound particularly hard to me. Most CPE has the control and data planes on the same silicon. A guard at the data plane is pointless in the first place. Just punt the packet up and move on. On the bigger iron where the planes are on running on different chips, you can move the initial ND solicitation packet into the data plane. After failing to find an incomplete ND, generate and send the ND solicitation and THEN make the decision whether to punt to the control plane or discard. If you discard, the control plane will find out about the good ones when the response comes back. This means you could multiply a unicast flood into a multicast flood but you'll have to pump out several orders of magnitude more packets than with the original problem before it causes me grief. Still, you've sold me on part of the claim: A /64 is inherently vulnerable to a remote DOS attack that a /120 is not. Now sell me on the other part: How does this require effort on the attacker's part that's enough smaller than the general form flood his link attack that I should care about it beyond poking my vendors to see if they've reasonably covered the high-load corner cases? I see how the original attack could kill a lan with a relatively small number of packets. With the naive solution, it seems to require something a lot closer to a steady flood. Regards, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004 _ NANOG mailing list NANOG@nanog.org https://mailman.nanog.org/mailman/listinfo/nanog
Re: NDP DoS attack (was Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?))
On Jul 17, 2011, at 1:17 PM, Jeff Wheeler wrote: On Sun, Jul 17, 2011 at 3:40 PM, Owen DeLong o...@delong.com wrote: Basically an ND entry would have the following states and timers: I've discussed what you have described with some colleagues in the past. The idea has merit and I would certainly not complain if vendors included it (as a knob) on their boxes. The downfalls of this approach are that they still don't ensure the discovery of new neighbors (rather than ever seen neighbors) during DoS, and you make the local DoS a bit more complex by needing to establish more rules for purging these semi-permanent entries. Sure they do... Just not necessarily on the first attempt. There are no semi-permanent entries. In fact, it doesn't make any entry more permanent than today's state. The D flag just makes entries more readily discardable than today's entries. So you have some misconceptions about how it would work in practice, I think. Under DOS, the first packet that arrives for a known host generates the standard ND request sent to the host, but, the Incomplete ND table entry is created with the D flag set. If the host responds before the ND table entry is discarded, all functions as normal. If the entry is discarded before the host responds, then the response from the host creates a new incomplete entry without the D flag set. This entry will live for the normal time that an incomplete ND entry would be kept (not eligible for early discard) and the retry packet from the originating host would then generate a new ND request and the response should arrive before the normal incomplete ND timer expires. At that point a normal complete entry is created and things continue to function. So, what happens under this scenario is that you have a small chance that you need to wait for an initial connection retry on an unseen host, but, you can easily discard incomplete ND entries for which no response has yet been received. Further, since you're only discarding the oldest one entry each time you need to create a new entry in a full table, this would only start discarding things when an actual table overflow is occurring whether from DOS or other cause. If it's another cause, I don't think this makes life any worse. If it's DOS, then, it should be relatively rare that a responsive host is the oldest ND table entry that would get discarded, no? I think most of this punting could be handled at the line card level. Is there any reason that the ND process can't be moved into line-card level silicon as described above? You could implement ND solicit in the data-plane (and remove punts entirely) in even some current chips, to say nothing of future ones. Whether or not that is a good idea, well, keep in mind that the ND solicits would then be mcasted to the LAN at a potentially unlimited rate. There's no reason it would have to be an unlimited rate, but, I think that would probably be acceptable in most cases anyway. That is not necessarily a problem unless the L2 implementation is not too good with respect to multicast. For example, in some switches (mostly those that are routers that can switch) the L2 mcast has surprising caveats, such as using up a lot of fabric capacity for whatever replication scheme has been chosen. If your L2 implementation sucks on Mcast in IPv6, you're kind of in a bad way anyway. Of course, you also hope NDP on all the connected hosts works right. I believe some Juniper customers noticed a pretty big problem with JUNOS NDP implementation when deploying boxes using the DE-CIX addressing scheme, and in a situation like that, the ingress router for the attack could be crippled by spurious responses from the other mis-behaving hosts on the LAN, essentially like smurf except without sending any garbage back out to the Internet. I think the bad NDP implementations on the hosts will get sorted fairly quickly anyway. Since all a spurious hosts would do is create a new incomplete entry without the D flag set the FIRST time it sends an unsolicited ND response, I'm not sure how that would really cripple the ingress router. Care to explain that? What you definitely don't want to do is assume this fixes the local DoS, because it doesn't. I would like for you to keep in mind that a host on the LAN, misconfigured to do something like local proxy-arp, or otherwise responding to all ND solicits, would accidentally DoS the LAN's gateway. I do not think we should assume that the local DoS won't happen, or is fixable with a whack-a-mole method. I consider local DOS to be a corner case unique to universities and very poorly run colos. We've already had that discussion and IIRC agreed to disagree. Sure, that doesn't solve the problem on current hardware, but, it moves it from design problem to implementation issue, which IMHO is a step in the right direction. Well, it already is a design problem that implementations can largely work-around.
Re: NDP DoS attack (was Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?))
On Jul 17, 2011, at 1:32 PM, William Herrin wrote: On Sun, Jul 17, 2011 at 1:35 PM, Jeff Wheeler j...@inconcepts.biz wrote: On Sun, Jul 17, 2011 at 11:42 AM, William Herrin b...@herrin.us wrote: My off-the-cuff naive solution to this problem would be to discard the oldest incomplete solicitation to fit the new one and, upon receiving an apparently unsolicited response to a discarded solicitation, restart the process flagging that particular query non-discardable. Do you mean to write, flagging that ND entry non-discardable? Hi Jeff, I meant flagging the new incomplete solicitation ineligible for previous sentence's early load-based discard. When you get a response to a solicitation you no longer remember making, solicit again and don't forget about it until the normal protocol timeouts this time. If you're going to solicit again rather than wait for a new packet, what's the point of not just installing a complete entry? After all, if someone sends you a spurious response, they'll likely also be able to respond to the solicit, so, you don't really protect anything by sending the solicit. I figured you stick the ineligible incomplete entry in the table and wait for the retransmit of the original packet. Where does this naive approach break down? It breaks down because the control-plane can't handle the relatively small number of punts which must be generated in order to send ND solicits, and without the ability to install incomplete entries into the data-plane, those punts cannot be policed without, by design, discarding some good punts along with the bad punts resulting from DoS traffic. Let me try to restate what you've said to make sure I understand. When the data plane knows what ARP or ND is underway, it can guard against overwhelming the control plane by discarding excessive traffic for the same yet-unresolved destination while allowing packets for new destinations on the lan through to the control plane. Without that knowledge, it can only have one queue causing the data plane to discard packets which would initiate neighbor discovery prior to the control plane seeing them, preventing any solicitation or implementing the logic I described. This doesn't sound particularly hard to me. Most CPE has the control and data planes on the same silicon. A guard at the data plane is pointless in the first place. Just punt the packet up and move on. I think Jeff's focus here is on the kinds of core and TOR switches that are mostly used in datacenters, not so much the CPE end of the world. Still, you've sold me on part of the claim: A /64 is inherently vulnerable to a remote DOS attack that a /120 is not. More accurately, the larger your single subnet address space, the more vulnerable you are to table overflow attacks. A /120 is exactly as vulnerable as an IPv4 /24. A /96 is exactly as vulnerable as an IPv4 /0. With bigger address spaces come new challenges. In the real world, I think this is less of an issue because: a. While the attack surface is large, the benefits of carrying out such an attack are relatively small. b. It's a relatively easy attack to spot, identify, quench, and likely trace. Owen _ NANOG mailing list NANOG@nanog.org https://mailman.nanog.org/mailman/listinfo/nanog
Re: NDP DoS attack (was Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?))
On Jul 14, 2011, at 8:24 PM, Jimmy Hess wrote: On Thu, Jul 14, 2011 at 9:35 PM, Jared Mauch ja...@puck.nether.net wrote: On Jul 14, 2011, at 10:06 PM, Fernando Gont ferna...@gont.com.ar wrote: Anyone on a layer-2 network can do something interesting like flood all f's and kill the lan. Trying to keep the majority of thoughts here for layer-3 originated attacks, even if the target is a layer2 item. - Jared In most cases if you have a DoS attack coming from the same Layer-2 network that a router is attached to, it would mean there was already a serious security incident that occured to give the attacker that special point to attack from. That's one possibility. The other likely possibility is that you are a University. Owen
Re: NDP DoS attack (was Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?))
On Thu, 14 Jul 2011 23:13:03 PDT, Owen DeLong said: On Jul 14, 2011, at 8:24 PM, Jimmy Hess wrote: In most cases if you have a DoS attack coming from the same Layer-2 network that a router is attached to, it would mean there was already a serious security incident that occured to give the attacker that special point to attack from. That's one possibility. The other likely possibility is that you are a University. Nope. Unless you want to add or you are a cable provider, or you are a DSL provider, or you are a to that. (Hint - what percent of students launch DoS attacks that cut themselves off from the net? Compare to what percent of non-student machines out on cable and DSL are botted or pwned) Even if you're a university with resident students, if said students are on the same Layer-2 as anything you actually care about, you have a serious security incident. Student manages to DoS the router out of the dorm and strands 3 floors of dorm without internet is just as interesting as Joe Sixpack manages to DoS the router at the cable head end and strands 3 blocks of Comcast customers without internet, for the *exact same reasons*. If the student is able to play more level-2 games than Joe Sixpack can, you misdesigned your network. pgpiIx2FrZzn7.pgp Description: PGP signature
Re: NDP DoS attack (was Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?))
On Thu, Jul 14, 2011 at 9:47 PM, Owen DeLong o...@delong.com wrote: Very true. This is where Mr. Wheeler's arguments depart from reality. He's right in that the problem can't be truly fixed without some very complicated code added to lots of devices, but, it can be mitigated relatively easily and mitigation really is good enough for most real world purposes. ok,I'll bite, what's the solution?
Re: NDP DoS attack (was Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?))
On 07/11/2011 09:17 PM, Karl Auer wrote: I realise this is not specific implementations as you requested, but it seems to me that the problem is generic enough not to require that. The attack is made possible by the design of the protocol, not any failing of specific implementations. Specific implementations need to describe what they've done about it (mitigation or prevention). Vulnerability to this specific issues has a great deal to do with the implementation. After all, whenever there's a data structure that can potentially grow out of bounds (or hit a limit), it becomes a resource management issue. In this particular case, if the implementation enforces a limit on the number of entries in the INCOMPLETE state, then only nodes that have never communicated with the outside world could be affected by this attack. And if those entries that are in the INCOMPLETE state are pruned periodically (e.g. in a round-robin fashion), chances are that even those new hosts would be able to get into the neighbor cache and hence remain unaffected by this attack. Thanks, -- Fernando Gont e-mail: ferna...@gont.com.ar || fg...@acm.org PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
Re: NDP DoS attack (was Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?))
On Thu, Jul 14, 2011 at 7:29 PM, Fernando Gont ferna...@gont.com.ar wrote: On 07/11/2011 09:17 PM, Karl Auer wrote: Vulnerability to this specific issues has a great deal to do with the implementation. After all, whenever there's a data structure that can Yes In this particular case, if the implementation enforces a limit on the number of entries in the INCOMPLETE state, then only nodes that have never communicated with the outside world could be affected by this attack. And if those entries that are in the INCOMPLETE state are pruned periodically (e.g. in a round-robin fashion), chances are that Not only that but it's possible to differentiate _how_ an entry is added to the table when the table reaches a high water mark it's possible to drop the packet that was attempting to cause a NDP discover, fail to add the INCOMPLETE entry to the table, but _still_ send the outgoing NDP neighbor solicitation, and complete the entry or whitelist the destination if the neighbor advertises itself. That is: if the destination is good, the neighbor will respond to the NDP solicit, even though the neighbor doesn't have an entry in the table. So a small number of packets are lost at initial setup, due to the attack, but further packets are unaffected, So long as the attack does not overwhelm router CPU, and so long as the INCOMPLETE entry high water mark is at a low enough level, so there is still ample space in the table. Even more sophisticated strategies may be available. It should be possible to mitigate this, so long as the attack does not actually originate from a neighbor on the same subnet as a router IP interface on an IPv6 subnet with sufficient number of IPs. even those new hosts would be able to get into the neighbor cache and hence remain unaffected by this attack. Thanks, -- -JH
Re: NDP DoS attack (was Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?))
On Jul 14, 2011, at 6:24 PM, Jimmy Hess wrote: On Thu, Jul 14, 2011 at 7:29 PM, Fernando Gont ferna...@gont.com.ar wrote: On 07/11/2011 09:17 PM, Karl Auer wrote: Vulnerability to this specific issues has a great deal to do with the implementation. After all, whenever there's a data structure that can Yes In this particular case, if the implementation enforces a limit on the number of entries in the INCOMPLETE state, then only nodes that have never communicated with the outside world could be affected by this attack. And if those entries that are in the INCOMPLETE state are pruned periodically (e.g. in a round-robin fashion), chances are that Not only that but it's possible to differentiate _how_ an entry is added to the table when the table reaches a high water mark it's possible to drop the packet that was attempting to cause a NDP discover, fail to add the INCOMPLETE entry to the table, but _still_ send the outgoing NDP neighbor solicitation, and complete the entry or whitelist the destination if the neighbor advertises itself. That is: if the destination is good, the neighbor will respond to the NDP solicit, even though the neighbor doesn't have an entry in the table. So a small number of packets are lost at initial setup, due to the attack, but further packets are unaffected, So long as the attack does not overwhelm router CPU, and so long as the INCOMPLETE entry high water mark is at a low enough level, so there is still ample space in the table. Even more sophisticated strategies may be available. It should be possible to mitigate this, so long as the attack does not actually originate from a neighbor on the same subnet as a router IP interface on an IPv6 subnet with sufficient number of IPs. even those new hosts would be able to get into the neighbor cache and hence remain unaffected by this attack. Thanks, -- -JH Very true. This is where Mr. Wheeler's arguments depart from reality. He's right in that the problem can't be truly fixed without some very complicated code added to lots of devices, but, it can be mitigated relatively easily and mitigation really is good enough for most real world purposes. Owen
Re: NDP DoS attack (was Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?))
On 07/14/2011 10:24 PM, Jimmy Hess wrote: In this particular case, if the implementation enforces a limit on the number of entries in the INCOMPLETE state, then only nodes that have never communicated with the outside world could be affected by this attack. And if those entries that are in the INCOMPLETE state are pruned periodically (e.g. in a round-robin fashion), chances are that Not only that but it's possible to differentiate _how_ an entry is added to the table when the table reaches a high water mark it's possible to drop the packet that was attempting to cause a NDP discover, fail to add the INCOMPLETE entry to the table, but _still_ send the outgoing NDP neighbor solicitation, and complete the entry or whitelist the destination if the neighbor advertises itself. Agreed. I should double-check whether there's room in the current specifications to do this -- however, whether the specs allow this or not is irrelevant. At the point you're being hit with a DoS, you better do something about it (particularly when it's possible, as in this case!) That is: if the destination is good, the neighbor will respond to the NDP solicit, even though the neighbor doesn't have an entry in the table. Modulo that when the high water mark has not been hit, the router should probably *not* create ND cache entries in response to this gratuitous ND advertisements, since otherwise it would open the door to a DoS from local attackers. It should be possible to mitigate this, so long as the attack does not actually originate from a neighbor on the same subnet as a router IP interface on an IPv6 subnet with sufficient number of IPs. Well, unless there's some layer-2 anti-spoofing mitigation in place, with /64 subnets the local attacker typically *will* have enough addresses. Thanks, -- Fernando Gont e-mail: ferna...@gont.com.ar || fg...@acm.org PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
Re: NDP DoS attack (was Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?))
On Jul 14, 2011, at 10:06 PM, Fernando Gont ferna...@gont.com.ar wrote: It should be possible to mitigate this, so long as the attack does not actually originate from a neighbor on the same subnet as a router IP interface on an IPv6 subnet with sufficient number of IPs. Well, unless there's some layer-2 anti-spoofing mitigation in place, with /64 subnets the local attacker typically *will* have enough addresses. Solving a local attack is something I consider different in scope than the current draft being discussed in 6man, v6ops, ipv6@ etc... Anyone on a layer-2 network can do something interesting like flood all f's and kill the lan. Trying to keep the majority of thoughts here for layer-3 originated attacks, even if the target is a layer2 item. - Jared
Re: NDP DoS attack (was Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?))
On 07/14/2011 11:35 PM, Jared Mauch wrote: Well, unless there's some layer-2 anti-spoofing mitigation in place, with /64 subnets the local attacker typically *will* have enough addresses. Solving a local attack Well, I was talking about not *introducing* ;-) one. is something I consider different in scope than the current draft being discussed in 6man, v6ops, ipv6@ etc... Which I-D are you referring to? Thanks, -- Fernando Gont e-mail: ferna...@gont.com.ar || fg...@acm.org PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
Re: NDP DoS attack (was Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?))
http://tools.ietf.org/html/draft-gashinsky-v6nd-enhance-00 Sent from my iThing On Jul 14, 2011, at 10:57 PM, Fernando Gont ferna...@gont.com.ar wrote: On 07/14/2011 11:35 PM, Jared Mauch wrote: Well, unless there's some layer-2 anti-spoofing mitigation in place, with /64 subnets the local attacker typically *will* have enough addresses. Solving a local attack Well, I was talking about not *introducing* ;-) one. is something I consider different in scope than the current draft being discussed in 6man, v6ops, ipv6@ etc... Which I-D are you referring to? Thanks, -- Fernando Gont e-mail: ferna...@gont.com.ar || fg...@acm.org PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
Re: NDP DoS attack (was Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?))
On Thu, Jul 14, 2011 at 9:35 PM, Jared Mauch ja...@puck.nether.net wrote: On Jul 14, 2011, at 10:06 PM, Fernando Gont ferna...@gont.com.ar wrote: Anyone on a layer-2 network can do something interesting like flood all f's and kill the lan. Trying to keep the majority of thoughts here for layer-3 originated attacks, even if the target is a layer2 item. - Jared In most cases if you have a DoS attack coming from the same Layer-2 network that a router is attached to, it would mean there was already a serious security incident that occured to give the attacker that special point to attack from. A similarly hazardous situation exists with IPv4, and it is basically unheard of for IPv4's Layer 2/ARP security weaknesses to be exploited to create a DoS condition, even though they can be (very easily), IPv4 Layer 2 DoS conditions are often due to a malfunction or error than intended attack; more likely, IPv6 Layer 2 security weaknesses will be used to intercept traffic for snooping, or quietly subvert network policy. LAN DoS conditions are noticed quickly, and usually result in physical unplugging of the attacking (or malfunctioning) node. Methods can be designed to protect against spoofed NDP flooding on the LAN that do not require the router's involvement. For IPv4 switched networks there is a technology referred to as 'Dynamic ARP Inspection'. Untrusted IPv6 LAN environments will need to implement SEND or some form of 'Dynamic ND inspection' plus RA-guard. If it comes down to solving a remote DoS issue at the cost of creating a LAN DoS issue that comes down to 'hosts on the LAN having to spoof' I would say that's easily well worth it. -- -JH
Re: NDP DoS attack (was Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?))
On 07/15/2011 12:24 AM, Jimmy Hess wrote: A similarly hazardous situation exists with IPv4, and it is basically unheard of for IPv4's Layer 2/ARP security weaknesses to be exploited to create a DoS condition, even though they can be (very easily), IMO, the situation is different, in that the typical IPv4 subnet size eliminate some of the attack vectors. For example, it would be virtually impossible for an ARP cache to grow without bounds, and consume all kernel memory, because the typical IPv4 subnet size imposes a limit on the number of entries. That is *not* the case with v6. IPv4 Layer 2 DoS conditions are often due to a malfunction or error than intended attack; more likely, IPv6 Layer 2 security weaknesses will be used to intercept traffic for snooping, or quietly subvert network policy. LAN DoS conditions are noticed quickly, and usually result in physical unplugging of the attacking (or malfunctioning) node. Assuming the admin of the possibly-ipv6-enabled-by-default router is IPv6 aware, etc. Methods can be designed to protect against spoofed NDP flooding on the LAN that do not require the router's involvement. Which ones? For IPv4 switched networks there is a technology referred to as 'Dynamic ARP Inspection'. Untrusted IPv6 LAN environments will need to implement SEND or some form of 'Dynamic ND inspection' plus RA-guard. Good luck with deploying SEND. OTOH, forget about current implementations of RA-Guard: * http://tools.ietf.org/id/draft-gont-v6ops-ra-guard-evasion-01.txt * http://tools.ietf.org/id/draft-gont-6man-nd-extension-headers-01.txt If it comes down to solving a remote DoS issue at the cost of creating a LAN DoS issue that comes down to 'hosts on the LAN having to spoof' I would say that's easily well worth it. You *can* fix the remote DoS issue, *without* introducing the locally-exploitable one. That's the point. Thanks, -- Fernando Gont e-mail: ferna...@gont.com.ar || fg...@acm.org PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
to send out email to say check your router vendor web site for fixed images. The better fix would be to get them to also add support for draft-andrews-v6ops-6to4-router-option-02.txt which greys out the checkbox when 0.0.0.0 is sent as a response to the option. =20 Remember operators are in the position to alleviate lots of the 6to4 issues themselves. =20 Blocking over IPv4 transport is just silly. It's just as likely = =3D that your record is destined for an end-host that has native IPv6 =3D connectivity with an intermediate resolver that desn't have IPv6 as it is that =3D you're sending that to a 6to4 host. Further, there's no reason to believe = the 6to4 host won't attempt to resolve via IPv6, so, it doesn't really =3D= help anyway. =3D20 Real network operators have a relatively low BS threshold, they = have customers to support and businesses to run, and they don't have =3D thumb wrestle these people who don't actually have any skin in the game. =3D20 I agree, but, it's not hard to run 6to4 relays and running them does = =3D much more to alleviate the problems with 6to4 than anything you proposed above. Indeed, what you proposed above will likely create more =3D customer issues rather than reduce them. =3D20 Owen =3D20 Cameron =3D20 =3D20 Ron =3D20 =3D20 -Original Message- From: Leo Bicknell [mailto:bickn...@ufp.org] Sent: Monday, July 11, 2011 3:35 PM To: nanog@nanog.org Subject: Re: Anybody can participate in the IETF (Was: Why is IPv6 = =3D broken?) =3D20 In a message written on Sun, Jul 10, 2011 at 06:16:09PM +0200, =3D Jeroen Massar wrote: Eh ANYBODY, including you, can sign up to the IETF mailing =3D lists and participate there, just like a couple of folks from NANOG are = =3D already doing. =3D20 The way the IETF and the operator community interact is badly =3D broken. =3D20 The IETF does not want operators in many steps of the process. If = =3D you try to bring up operational concerns in early protocol = development =3D for example you'll often get a we'll look at that later response, =3D= which in many cases is right. Sometimes you just have to play with =3D= something before you worry about the operational details. It also = does =3D not help that many operational types are not hardcore programmers, = and =3D can't play in the sandbox during the major development cycles. =3D20 =3D20 =3D20 =3D20 =3D20 =3D20 =3D20 =20 =20 --=20 Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org =20 -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Jul 12, 2011, at 10:59 PM, Mark Andrews wrote: I didn't claim it would work with existing CPE equipment. Declaring 6to4 historic won't work with existing CPE equipment either. If the hosts behind it stop using 2002::/16 addresses as a product of a software update which seems rather more likely (also there some evidence for that), it will. that said yes one assumption is that you have to continue to support it. snip It is really hard to justify the expansion and deployment of new relays = when in fact tunneled traffic can be observed to be on the decline = (possibly because devices particularly hosts that do receive regular = updates receive tweaks to their address selection algorithm). = http://asert.arbornetworks.com/2011/04/six-months-six-providers-and-ipv6/ Which may or may not be a short term dip. correlation is not causation but... http://arstechnica.com/apple/news/2010/11/apple-fixes-broken-ipv6-by-breaking-it-some-more.ars We are yet to see much in the way of IPv6 only content. When that appears, which it will, the tunneled traffic will go up unless ISPs have deployed native IPv6 to all customers. Are you willing to bet on which will happen first? I'm willing to bet that subpar experience due to auto-tunneling is considered a liability for content providers. This whole area is in a state of flux. What would have been much better would have been to encourage CPE vendors to release images which address some of the known issues. Just adding a check box saying enable 6to4 and for ISP to send out email to say check your router vendor web site for fixed images. The better fix would be to get them to also add support for draft-andrews-v6ops-6to4-router-option-02.txt which greys out the checkbox when 0.0.0.0 is sent as a response to the option. =20 Remember operators are in the position to alleviate lots of the 6to4 issues themselves. =20 Blocking over IPv4 transport is just silly. It's just as likely = =3D that your record is destined for an end-host that has native IPv6 =3D connectivity with an intermediate resolver that desn't have IPv6 as it is that =3D you're sending that to a 6to4 host. Further, there's no reason to believe = the 6to4 host won't attempt to resolve via IPv6, so, it doesn't really =3D= help anyway. =3D20 Real network operators have a relatively low BS threshold, they = have customers to support and businesses to run, and they don't have =3D thumb wrestle these people who don't actually have any skin in the game. =3D20 I agree, but, it's not hard to run 6to4 relays and running them does = =3D much more to alleviate the problems with 6to4 than anything you proposed above. Indeed, what you proposed above will likely create more =3D customer issues rather than reduce them. =3D20 Owen =3D20 Cameron =3D20 =3D20 Ron =3D20 =3D20 -Original Message- From: Leo Bicknell [mailto:bickn...@ufp.org] Sent: Monday, July 11, 2011 3:35 PM To: nanog@nanog.org Subject: Re: Anybody can participate in the IETF (Was: Why is IPv6 = =3D broken?) =3D20 In a message written on Sun, Jul 10, 2011 at 06:16:09PM +0200, =3D Jeroen Massar wrote: Eh ANYBODY, including you, can sign up to the IETF mailing =3D lists and participate there, just like a couple of folks from NANOG are = =3D already doing. =3D20 The way the IETF and the operator community interact is badly =3D broken. =3D20 The IETF does not want operators in many steps of the process. If = =3D you try to bring up operational concerns in early protocol = development =3D for example you'll often get a we'll look at that later response, =3D= which in many cases is right. Sometimes you just have to play with =3D= something before you worry about the operational details. It also = does =3D not help that many operational types are not hardcore programmers, = and =3D can't play in the sandbox during the major development cycles. =3D20 =3D20 =3D20 =3D20 =3D20 =3D20 =3D20 =20 =20 --=20 Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org =20 -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
Jeff, On Jul 12, 2011, at 20:13 , Jeff Wheeler wrote: On Tue, Jul 12, 2011 at 11:42 AM, Leo Bicknell bickn...@ufp.org wrote: I'll pick on LISP as an example, since many operators are at least aware of it. Some operators have said we need a locator and identifier split. Interesting feedback. The IETF has gone off and started playing in the sandbox, trying to figure out how to make that go. As an operator (who understands how most things work in very great detail), Granted. You are the real world expert. Now can you stop repeating this in each email and move on? I found the LISP folks very much uninterested in my concerns about if LISP can ever be made to scale up to Internet-scale, with respect to a specific DDoS vector. This is completely false. Several people gave credit to you about the existence of the threat you pointed out. I also think that an explosion of small, multi-homed SOHO networks would be a disaster, because we might have 3 million FIB instead of 360k FIB after a few years. These things are directly related to each-other, too. So I emailed some LISP gurus off-list and discussed my concern. I was encouraged to post to the LISP IETF list, which I did. To my great surprise, not one single person was interested in my problem. This is again false. We had mail exchange both privately and on the mailinglist. We proposed to you text to be added to the threats draft but you did not like it. We are asking to propose text but we have no answer from you on this point. If you think it is a small problem, well, you should try going back to late-1990s flow-cache routing in your data-center networks and see what happens when you get DDoS. I am sure most of us remember some of those painful experiences. Now there is a LISP threats draft which the working group mandates they produce, discussing various security problems. The current paper is a laundry list of what if scenarios, like, what if a malicious person could fill the LISP control-plane with garbage. BGP has the same issue, if some bad guy had enable on a big enough network that their peers/transits don't filter their routes, they could do a lot of damage before they were stopped. So you are saying that BGP can be victim of similar attacks/problem still... if you are reading this email it means that the Internet is still running... This sometimes happens even by accident, for example, some poor guy accidentally announcing 12/9 and giving ATT a really bad day. What it doesn't contain is anything relevant to the special-case DDoS that all LISP sites would be vulnerable to, due to the IMO bad flow-cache management system that is specified. If you still think that LISP is using a flow-cache you should have a second read to the set of drafts. I am having a very great deal of trouble getting the authors of the threats document to even understand what the problem is, For the third time: this is false. We got the problem, we were asking for more specific information in order to quantify the risk. We asked you help to state the problem and explained to you where the solution should be addressed. But you seem to be stuck on the operator vs. researcher discussion, which IMHO is just pointless. because as one of them put it, he is just a researcher. I am sure he and his colleagues are very smart guys, but they clearly do not remember our 1990s pains. That is the not an operator problem. It is understandable. Others who have been around long enough simply dismiss this problem, because they believe the unparalleled benefits of LISP for mobility and multi-homing SOHO sites must greatly out-weigh the fact that, well, if you are a content provider and you receive a DDoS, your site will be down and there isn't a damn thing you can do about it, other than spec routers that have way, way more FIB than the number of possible routes, again due to the bad caching scheme. The above is what I think is the ego-invested problem, where certain pretty smart, well-intentioned people have a lot of time, and professional credibility, invested in making LISP work. I'm sure it isn't pleasing for these guys to defend their project against my argument that it may never be able to reach Internet-scale, and that they have missed what I claim is a show-stopping problem with an easy way to improve it through several years of development. Especially since I am a guy who did not ever participate in the IETF before, someone they don't know from a random guy on the street. I am glad that this NANOG discussion has got some of these LISP folks to pay more attention to my argument, and my suggested improvement (I am not only bashing their project; I have positive input, too.) Simply posting to their mailing list once and emailing a few draft authors did not cause any movement at all. Evidently it does get attention, though, to jump up and down on a different list.
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
Hello Jeff, On 13 Jul 2011, at 10:08, Luigi Iannone wrote: Jeff, On Jul 12, 2011, at 20:13 , Jeff Wheeler wrote: On Tue, Jul 12, 2011 at 11:42 AM, Leo Bicknell bickn...@ufp.org wrote: I'll pick on LISP as an example, since many operators are at least aware of it. Some operators have said we need a locator and identifier split. Interesting feedback. The IETF has gone off and started playing in the sandbox, trying to figure out how to make that go. As an operator (who understands how most things work in very great detail), Granted. You are the real world expert. Now can you stop repeating this in each email and move on? I found the LISP folks very much uninterested in my concerns about if LISP can ever be made to scale up to Internet-scale, with respect to a specific DDoS vector. This is completely false. Several people gave credit to you about the existence of the threat you pointed out. Definitely, since 2007 we are working on LISP and always keep scalability in mind. All our researches are always constrained by the scalability. What you pointed out is very interesting and we are working on it. Nevertheless, and I am just repeating myself now, the problem you expose is an operational problem that can also have security implications. This is why we proposed you several time to expose the problem at next IETF such that we can all discuss the problem and propose solutions to be added in the main specs. The cache management problem is really interesting and we could even imagine a draft like implementation guidelines that would explain how to implement the caches in a smart way. The solution to the problem you are pointing-out can also be proposed in the deployment draft as it could be alleviate with the help of filters. I also think that an explosion of small, multi-homed SOHO networks would be a disaster, because we might have 3 million FIB instead of 360k FIB after a few years. These things are directly related to each-other, too. So I emailed some LISP gurus off-list and discussed my concern. I was encouraged to post to the LISP IETF list, which I did. To my great surprise, not one single person was interested in my problem. This is again false. We had mail exchange both privately and on the mailinglist. We proposed to you text to be added to the threats draft but you did not like it. We are asking to propose text but we have no answer from you on this point. Among the people that are interesting by the problem, you have the threats authors. You received personal replies from me on Jul 6th @2:08pm and Jul 7th @1:13 am. In our SVN, the draft is already updated with your comments and you are in the acks. The version will be submitted just after the IETF with the comments that will come from the presentation we will make during IETF81. I understand that you would like to have the comment addresses immediately when you are thinking about them but sometime it is nice to take a few days to think about the problem in order to problem a more robust description and thus build more sustainable ways to a solution. If you think it is a small problem, well, you should try going back to late-1990s flow-cache routing in your data-center networks and see what happens when you get DDoS. I am sure most of us remember some of those painful experiences. Now there is a LISP threats draft which the working group mandates they produce, discussing various security problems. The current paper is a laundry list of what if scenarios, like, what if a malicious person could fill the LISP control-plane with garbage. BGP has the same issue, if some bad guy had enable on a big enough network that their peers/transits don't filter their routes, they could do a lot of damage before they were stopped. So you are saying that BGP can be victim of similar attacks/problem still... if you are reading this email it means that the Internet is still running... Some people like Randy Bush try to find solutions for BGP not to be destroyed in presence of malicious guys. For BGP as for LISP, things take time just become the problem is definitely more complex as we always have to make tradeoffs between various contradictory elements. This sometimes happens even by accident, for example, some poor guy accidentally announcing 12/9 and giving ATT a really bad day. What it doesn't contain is anything relevant to the special-case DDoS that all LISP sites would be vulnerable to, due to the IMO bad flow-cache management system that is specified. If you still think that LISP is using a flow-cache you should have a second read to the set of drafts. Depends you definition of flow. If a flow is defined by the destination prefix, then yes, LISP has a flow-cache ;-) I am having a very great deal of trouble getting the authors of the threats document to even understand what the problem is, For the third time: this is
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
Luigi, you have mis-understood quite a bit of the content of my message. I'm not sure if this is of any further interest to NANOG readers, but as it is basically what seems to go on a lot, from my observations of IETF list activity, I'll copy my reply to the list as you have done. On Wed, Jul 13, 2011 at 4:08 AM, Luigi Iannone lu...@net.t-labs.tu-berlin.de wrote: Granted. You are the real world expert. Now can you stop repeating this in each email and move on? No. This is a point that needs to be not only made, but driven home. You do not understand how routers work, which is why you are having such difficulty understanding the severity of this problem. The lisp-threats work you have done is basically all control-plane / signalling issues, and no data-plane issues. This is not a coincidence; it is because your knowledge of the control-plane side is good and of the data-plane is weak. This is completely false. Several people gave credit to you about the existence of the threat you pointed out. Really? In April, when I posted a serious problem, and received no replies? Now, the original folks who I discussed this with, before ever posting to the IETF LISP list, are finally seeking clarification, because apparently there may have been some confusion in April, possibly leading to their total dismissal of this as a practical concern. This is again false. We had mail exchange both privately and on the mailinglist. We proposed to you text to be added to the threats draft but you did not like it. We are asking to propose text but we have no answer from you on this point. Actually, you classified this as an implementation concern, which is false. You have said yourself that this is why you believe it deserves just one sentence, if that, in the lisp-threats draft. This is not an implementation-specific concern, it is a design flaw in the MS negative response scheme, which emerges to produce a trivial DoS threat if LISP ever scales up. Now there is a LISP threats draft which the working group mandates they produce, discussing various security problems. The current paper is a laundry list of what if scenarios, like, what if a malicious person could fill the LISP control-plane with garbage. BGP has the So you are saying that BGP can be victim of similar attacks/problem still... if you are reading this email it means that the Internet is still running... This is where I believe you are mis-reading my message. Your threats draft covers legitimate concerns which also exist in the current system that is widely deployed, which is largely, BGP plus big FIB. What you don't cover, at all, is an IMO critical new threat that emerges in the data-plane from the design of the MS protocol. If you still think that LISP is using a flow-cache you should have a second read to the set of drafts. This language may appear unclear if you haven't read it in the context of my other postings. LISP routing most certainly is a flow-cache, however, the definition of flow is different. Some platforms and routing schemes see a flow as a layer-3 destination /32 or similar (some 90s routers), others more granular (firewalls, where flows are usually layer-4 and often stateful), and with LISP, the flow the address space routed from your ITR to a remote ETR, which may cover a large amount of address space and many smaller flows. The LISP drafts also refer to these flows as tunnels, but that language could easily be confused to mean much more permanent, static tunnels, or MPLS-like tunnels which are signaled throughout the network of P routers. So there are clear semantic issues of importance when talking about LISP, and all these terms must be read in the correct context. For the third time: this is false. We got the problem, we were asking for more specific information in order to quantify the risk. We asked you help You haven't got it, or you would already understand the risk very well. It is not my intention to fault you and your colleagues for failing to understand this; but to demonstrate clearly that the right kind of expertise is absolutely not being applied to LISP, and there is a huge and possibly intractable threat that was completely overlooked when producing what is meant to be an authoritative document on currently-known threats to LISP. to state the problem and explained to you where the solution should be addressed. But you seem to be stuck on the operator vs. researcher discussion, which IMHO is just pointless. Substantially all operators are stuck there. They should participate more. Let me now ask a simple question: why are you so strongly against LISP? No new work has been done to address the problem of scaling up the number of locators or multi-homed end-sites. However, the *claims* being made by LISP advocates is that the caching scheme you have, which is not novel, does solve this problem. It does not. It cannot as there has been no novel work on this. It is very
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
Jeff, on one point we agree, there is value in continuing this thread. I've tried to bring the discussion back to the technical issues, but I failed. Personally, I find your emails aggressive and close to offensive in some sentences. Differently from you, in my replies (all of them public) I never judged your competences. For me this thread is closed. Have a nice day Luigi On Jul 13, 2011, at 11:21 , Jeff Wheeler wrote: Luigi, you have mis-understood quite a bit of the content of my message. I'm not sure if this is of any further interest to NANOG readers, but as it is basically what seems to go on a lot, from my observations of IETF list activity, I'll copy my reply to the list as you have done. On Wed, Jul 13, 2011 at 4:08 AM, Luigi Iannone lu...@net.t-labs.tu-berlin.de wrote: Granted. You are the real world expert. Now can you stop repeating this in each email and move on? No. This is a point that needs to be not only made, but driven home. You do not understand how routers work, which is why you are having such difficulty understanding the severity of this problem. The lisp-threats work you have done is basically all control-plane / signalling issues, and no data-plane issues. This is not a coincidence; it is because your knowledge of the control-plane side is good and of the data-plane is weak. This is completely false. Several people gave credit to you about the existence of the threat you pointed out. Really? In April, when I posted a serious problem, and received no replies? Now, the original folks who I discussed this with, before ever posting to the IETF LISP list, are finally seeking clarification, because apparently there may have been some confusion in April, possibly leading to their total dismissal of this as a practical concern. This is again false. We had mail exchange both privately and on the mailinglist. We proposed to you text to be added to the threats draft but you did not like it. We are asking to propose text but we have no answer from you on this point. Actually, you classified this as an implementation concern, which is false. You have said yourself that this is why you believe it deserves just one sentence, if that, in the lisp-threats draft. This is not an implementation-specific concern, it is a design flaw in the MS negative response scheme, which emerges to produce a trivial DoS threat if LISP ever scales up. Now there is a LISP threats draft which the working group mandates they produce, discussing various security problems. The current paper is a laundry list of what if scenarios, like, what if a malicious person could fill the LISP control-plane with garbage. BGP has the So you are saying that BGP can be victim of similar attacks/problem still... if you are reading this email it means that the Internet is still running... This is where I believe you are mis-reading my message. Your threats draft covers legitimate concerns which also exist in the current system that is widely deployed, which is largely, BGP plus big FIB. What you don't cover, at all, is an IMO critical new threat that emerges in the data-plane from the design of the MS protocol. If you still think that LISP is using a flow-cache you should have a second read to the set of drafts. This language may appear unclear if you haven't read it in the context of my other postings. LISP routing most certainly is a flow-cache, however, the definition of flow is different. Some platforms and routing schemes see a flow as a layer-3 destination /32 or similar (some 90s routers), others more granular (firewalls, where flows are usually layer-4 and often stateful), and with LISP, the flow the address space routed from your ITR to a remote ETR, which may cover a large amount of address space and many smaller flows. The LISP drafts also refer to these flows as tunnels, but that language could easily be confused to mean much more permanent, static tunnels, or MPLS-like tunnels which are signaled throughout the network of P routers. So there are clear semantic issues of importance when talking about LISP, and all these terms must be read in the correct context. For the third time: this is false. We got the problem, we were asking for more specific information in order to quantify the risk. We asked you help You haven't got it, or you would already understand the risk very well. It is not my intention to fault you and your colleagues for failing to understand this; but to demonstrate clearly that the right kind of expertise is absolutely not being applied to LISP, and there is a huge and possibly intractable threat that was completely overlooked when producing what is meant to be an authoritative document on currently-known threats to LISP. to state the problem and explained to you where the solution should be addressed. But you seem to be stuck on the operator vs. researcher discussion, which
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Jul 13, 2011, at 13:03 , Luigi Iannone wrote: Jeff, on one point we agree, there is value in continuing this thread. There is _no_ value. my mistake... Luigi I've tried to bring the discussion back to the technical issues, but I failed. Personally, I find your emails aggressive and close to offensive in some sentences. Differently from you, in my replies (all of them public) I never judged your competences. For me this thread is closed. Have a nice day Luigi On Jul 13, 2011, at 11:21 , Jeff Wheeler wrote: Luigi, you have mis-understood quite a bit of the content of my message. I'm not sure if this is of any further interest to NANOG readers, but as it is basically what seems to go on a lot, from my observations of IETF list activity, I'll copy my reply to the list as you have done. On Wed, Jul 13, 2011 at 4:08 AM, Luigi Iannone lu...@net.t-labs.tu-berlin.de wrote: Granted. You are the real world expert. Now can you stop repeating this in each email and move on? No. This is a point that needs to be not only made, but driven home. You do not understand how routers work, which is why you are having such difficulty understanding the severity of this problem. The lisp-threats work you have done is basically all control-plane / signalling issues, and no data-plane issues. This is not a coincidence; it is because your knowledge of the control-plane side is good and of the data-plane is weak. This is completely false. Several people gave credit to you about the existence of the threat you pointed out. Really? In April, when I posted a serious problem, and received no replies? Now, the original folks who I discussed this with, before ever posting to the IETF LISP list, are finally seeking clarification, because apparently there may have been some confusion in April, possibly leading to their total dismissal of this as a practical concern. This is again false. We had mail exchange both privately and on the mailinglist. We proposed to you text to be added to the threats draft but you did not like it. We are asking to propose text but we have no answer from you on this point. Actually, you classified this as an implementation concern, which is false. You have said yourself that this is why you believe it deserves just one sentence, if that, in the lisp-threats draft. This is not an implementation-specific concern, it is a design flaw in the MS negative response scheme, which emerges to produce a trivial DoS threat if LISP ever scales up. Now there is a LISP threats draft which the working group mandates they produce, discussing various security problems. The current paper is a laundry list of what if scenarios, like, what if a malicious person could fill the LISP control-plane with garbage. BGP has the So you are saying that BGP can be victim of similar attacks/problem still... if you are reading this email it means that the Internet is still running... This is where I believe you are mis-reading my message. Your threats draft covers legitimate concerns which also exist in the current system that is widely deployed, which is largely, BGP plus big FIB. What you don't cover, at all, is an IMO critical new threat that emerges in the data-plane from the design of the MS protocol. If you still think that LISP is using a flow-cache you should have a second read to the set of drafts. This language may appear unclear if you haven't read it in the context of my other postings. LISP routing most certainly is a flow-cache, however, the definition of flow is different. Some platforms and routing schemes see a flow as a layer-3 destination /32 or similar (some 90s routers), others more granular (firewalls, where flows are usually layer-4 and often stateful), and with LISP, the flow the address space routed from your ITR to a remote ETR, which may cover a large amount of address space and many smaller flows. The LISP drafts also refer to these flows as tunnels, but that language could easily be confused to mean much more permanent, static tunnels, or MPLS-like tunnels which are signaled throughout the network of P routers. So there are clear semantic issues of importance when talking about LISP, and all these terms must be read in the correct context. For the third time: this is false. We got the problem, we were asking for more specific information in order to quantify the risk. We asked you help You haven't got it, or you would already understand the risk very well. It is not my intention to fault you and your colleagues for failing to understand this; but to demonstrate clearly that the right kind of expertise is absolutely not being applied to LISP, and there is a huge and possibly intractable threat that was completely overlooked when producing what is meant to be an authoritative document on currently-known threats to LISP. to state the problem and explained to you
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
In message 430fff20-43ed-45bb-846d-fee8769fc...@bogus.com, Joel Jaeggli write s: On Jul 12, 2011, at 10:59 PM, Mark Andrews wrote: =20 I didn't claim it would work with existing CPE equipment. Declaring 6to4 historic won't work with existing CPE equipment either. If the hosts behind it stop using 2002::/16 addresses as a product of a = software update which seems rather more likely (also there some evidence = for that), it will. that said yes one assumption is that you have to = continue to support it. When you switch the source address preference from 2002::/16 to IPv4 you loose insight into which machines have 2002::/16 addresses still without explict testing. snip It is really hard to justify the expansion and deployment of new = relays =3D when in fact tunneled traffic can be observed to be on the decline =3D (possibly because devices particularly hosts that do receive regular = =3D updates receive tweaks to their address selection algorithm). =3D = http://asert.arbornetworks.com/2011/04/six-months-six-providers-and-ipv6/ =20 Which may or may not be a short term dip. correlation is not causation but... = http://arstechnica.com/apple/news/2010/11/apple-fixes-broken-ipv6-by-break= ing-it-some-more.ars We are yet to see much in the way of IPv6 only content. When that appears, which it will, the = tunneled traffic will go up unless ISPs have deployed native IPv6 to all = customers. Are you willing to bet on which will happen first? I'm willing to bet that subpar experience due to auto-tunneling is = considered a liability for content providers. This whole area is in a state of flux. -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org
RE: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
Leo, Maybe we can fix this by: a) bringing together larger groups of clueful operators in the IETF b) deciding which issues interest them c) showing up and being vocal as a group in protocol developing working groups To some degree, we already do this in the IETF OPS area, but judging by your comments, we don't do it nearly enough. Comments? Ron -Original Message- From: Leo Bicknell [mailto:bickn...@ufp.org] Sent: Monday, July 11, 2011 3:35 PM To: nanog@nanog.org Subject: Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?) In a message written on Sun, Jul 10, 2011 at 06:16:09PM +0200, Jeroen Massar wrote: Eh ANYBODY, including you, can sign up to the IETF mailing lists and participate there, just like a couple of folks from NANOG are already doing. The way the IETF and the operator community interact is badly broken. The IETF does not want operators in many steps of the process. If you try to bring up operational concerns in early protocol development for example you'll often get a we'll look at that later response, which in many cases is right. Sometimes you just have to play with something before you worry about the operational details. It also does not help that many operational types are not hardcore programmers, and can't play in the sandbox during the major development cycles.
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
In a message written on Tue, Jul 12, 2011 at 11:28:58AM -0400, Ronald Bonica wrote: Maybe we can fix this by: a) bringing together larger groups of clueful operators in the IETF b) deciding which issues interest them c) showing up and being vocal as a group in protocol developing working groups To some degree, we already do this in the IETF OPS area, but judging by your comments, we don't do it nearly enough. I don't think it's that simple, sadly. I'll no doubt get flamed by the 5 people on NANOG that also participate in the IETF in a regular basis, but the reality is most operators don't want to sit through multi-year protocol devopment work, or have much of anything to do with pie in the sky ideas. The IETF can, should, and does do both of those things today. Where the friction occurs is there is no good place to loop the operators back in, so they are often kicked out, discouraged, or just uninterested on the front end (we're going to go play with new ideas kids!) and then not brought back in (it's ready for deployment, wait, why are no operators interested). So it's not that individual issues are of interest to operators (outside of the IETF OPS area, which is a special case), it's that the process needs work. I'll pick on LISP as an example, since many operators are at least aware of it. Some operators have said we need a locator and identifier split. Interesting feedback. The IETF has gone off and started playing in the sandbox, trying to figure out how to make that go. Several years of coding have occured, a bunch of proof of concept testing is going on. Even many of the operators who wanted such a spit are not really interested in following the details of the work right now. Of course, if you are, you can, I'm not advocating any exclusions. But there is no roadmap in the IETF process now for LISP that says We've got this 90% baked, we need to circulate a draft to the NANOG mailing list, request operator comments, and actively solicit operators to participate in the expanded test network. We need that mechanism to tell folks hey, it's real enough your operational feedback is now useful and come test our new idea. Today the IETF just finishes their work, tosses it over the wall and hopes for the best. Generally it's not 100%, and vendors make propretary changes to the standards slowly over time to meet the needs of operators. It would be far better if there was at least one round of ask the operators and incorproate feedback before it went over the wall, and in paricular before working groups disbanded. In short, make it easy for the operators to participate at the right time in the process. It will be better for everyone! -- Leo Bicknell - bickn...@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Tue, Jul 12, 2011 at 8:28 AM, Ronald Bonica rbon...@juniper.net wrote: Leo, Maybe we can fix this by: a) bringing together larger groups of clueful operators in the IETF b) deciding which issues interest them c) showing up and being vocal as a group in protocol developing working groups To some degree, we already do this in the IETF OPS area, but judging by your comments, we don't do it nearly enough. Comments? There may be an OPS area, but it is not listened to. Witness the latest debacle with the attempt at trying to make 6to4 historic. Various non-practicing entities were able to derail what network operators largely supported. Since the IETF failed to make progress operators will do other things to stop 6to4 ( i have heard no over IPv4 transport, blackhole 6to4 anycast, decom relay routers...) Real network operators have a relatively low BS threshold, they have customers to support and businesses to run, and they don't have thumb wrestle these people who don't actually have any skin in the game. Cameron Ron -Original Message- From: Leo Bicknell [mailto:bickn...@ufp.org] Sent: Monday, July 11, 2011 3:35 PM To: nanog@nanog.org Subject: Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?) In a message written on Sun, Jul 10, 2011 at 06:16:09PM +0200, Jeroen Massar wrote: Eh ANYBODY, including you, can sign up to the IETF mailing lists and participate there, just like a couple of folks from NANOG are already doing. The way the IETF and the operator community interact is badly broken. The IETF does not want operators in many steps of the process. If you try to bring up operational concerns in early protocol development for example you'll often get a we'll look at that later response, which in many cases is right. Sometimes you just have to play with something before you worry about the operational details. It also does not help that many operational types are not hardcore programmers, and can't play in the sandbox during the major development cycles.
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Tue, Jul 12, 2011 at 8:42 AM, Leo Bicknell bickn...@ufp.org wrote: In a message written on Tue, Jul 12, 2011 at 11:28:58AM -0400, Ronald Bonica wrote: Maybe we can fix this by: a) bringing together larger groups of clueful operators in the IETF b) deciding which issues interest them c) showing up and being vocal as a group in protocol developing working groups To some degree, we already do this in the IETF OPS area, but judging by your comments, we don't do it nearly enough. I don't think it's that simple, sadly. I'll no doubt get flamed by the 5 people on NANOG that also participate in the IETF in a regular basis, but the reality is most operators don't want to sit through multi-year protocol devopment work, or have much of anything to do with pie in the sky ideas. The IETF can, should, and does do both of those things today. Where the friction occurs is there is no good place to loop the operators back in, so they are often kicked out, discouraged, or just uninterested on the front end (we're going to go play with new ideas kids!) and then not brought back in (it's ready for deployment, wait, why are no operators interested). So it's not that individual issues are of interest to operators (outside of the IETF OPS area, which is a special case), it's that the process needs work. I'll pick on LISP as an example, since many operators are at least aware of it. Some operators have said we need a locator and identifier split. Interesting feedback. The IETF has gone off and started playing in the sandbox, trying to figure out how to make that go. Several years of coding have occured, a bunch of proof of concept testing is going on. Even many of the operators who wanted such a spit are not really interested in following the details of the work right now. Of course, if you are, you can, I'm not advocating any exclusions. W.R.T. to LISP, in defense of the IETF or the IRTF, i do not believe the IETF has told the world that LISP is the best fit for the Internet or solves any specific problem well. The IETF has never said the Internet Architecture is going to LISP, and it likely will not / cannot. My expectation is that LISP will go away as quickly as it came. Cameron But there is no roadmap in the IETF process now for LISP that says We've got this 90% baked, we need to circulate a draft to the NANOG mailing list, request operator comments, and actively solicit operators to participate in the expanded test network. We need that mechanism to tell folks hey, it's real enough your operational feedback is now useful and come test our new idea. Today the IETF just finishes their work, tosses it over the wall and hopes for the best. Generally it's not 100%, and vendors make propretary changes to the standards slowly over time to meet the needs of operators. It would be far better if there was at least one round of ask the operators and incorproate feedback before it went over the wall, and in paricular before working groups disbanded. In short, make it easy for the operators to participate at the right time in the process. It will be better for everyone! -- Leo Bicknell - bickn...@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On 10 Jul 2011, at 21:22, Owen DeLong wrote: On Jul 10, 2011, at 12:23 PM, William Herrin wrote: Consider, for example, RFC 3484. That's the one that determines how an IPv6 capable host selects which of a group of candidate IPv4 and IPv6 addresses for a particular host name gets priority. How is a server's address priority NOT an issue that should be managed at an operations level by individual server administrators? Yet the working group which produced it came up with a static prioritization that is the root cause of a significant portion of the IPv6 deployment headaches we face. 3484 specifies a static default. By definition, defaults in absence of operator configuration kind of have to be static. Having a reasonable and expected set of defaults documented in an RFC provides a known quantity for what operators can/should expect from hosts they have not configured. I see nothing wrong with RFC 3484 other than I would agree that the choices made were suboptimal. Mostly that was based on optimism and a lack of experience available at the time of writing. There is another RFC and there are APIs and most operating systems have configuration mechanisms where an operator CAN set that to something other than the 3484 defaults. There is a DHCPv6 option to configure host policy described in http://tools.ietf.org/html/draft-ietf-6man-addr-select-opt-01, which is hopefully approaching a WGLC at IETF81. RFC3484 was originally published in 2003, which is a long time ago. And of course it turned out that, with wider operational experience and feedback from the operator community, there were some issues uncovered and some omissions. Perhaps some of those might have been foreseen, but I highly doubt all of them would have. Many of the issues were captured in RFC5220, which led to the work on RFC3484-bis, which is also close to publication. Now, perhaps the DHCPv6 option and the 3484-bis drafts could be posted to the NANOG list at an appropriate time, for example when the docs hit WGLC. But there are a lot of WGLCs out there and the question is then whether the NANOG list, or some special NANOG list for IETF WGLCs, would want all those notifications. As a co-author of the DHCPv6 and 3484-bis drafts, I am quite happy to post about these to NANOG as they approach last call. There are three open issues on 3484-bis that some feedback would be welcomed on. There should be an operations callout as well -- a section where proposed operations defaults (as well as statics for which a solid case can be made for an operations tunable) are extracted from the thick of it and offered for operator scrutiny prior to publication of the RFC. I think this would be a good idea, actually. It would probably be more effective to propose it to IETF than to NANOG, however. Whether it's a separate section in the draft, or a recommendation to post to operators communities (which is more than just NANOG of course), the question as mentioned above is how that's done in a way to get the attention of appropriate operators without drowning them in notifications. Tim
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Tue, Jul 12, 2011 at 11:42 AM, Leo Bicknell bickn...@ufp.org wrote: I'll pick on LISP as an example, since many operators are at least aware of it. Some operators have said we need a locator and identifier split. Interesting feedback. The IETF has gone off and started playing in the sandbox, trying to figure out how to make that go. As an operator (who understands how most things work in very great detail), I found the LISP folks very much uninterested in my concerns about if LISP can ever be made to scale up to Internet-scale, with respect to a specific DDoS vector. I also think that an explosion of small, multi-homed SOHO networks would be a disaster, because we might have 3 million FIB instead of 360k FIB after a few years. These things are directly related to each-other, too. So I emailed some LISP gurus off-list and discussed my concern. I was encouraged to post to the LISP IETF list, which I did. To my great surprise, not one single person was interested in my problem. If you think it is a small problem, well, you should try going back to late-1990s flow-cache routing in your data-center networks and see what happens when you get DDoS. I am sure most of us remember some of those painful experiences. Now there is a LISP threats draft which the working group mandates they produce, discussing various security problems. The current paper is a laundry list of what if scenarios, like, what if a malicious person could fill the LISP control-plane with garbage. BGP has the same issue, if some bad guy had enable on a big enough network that their peers/transits don't filter their routes, they could do a lot of damage before they were stopped. This sometimes happens even by accident, for example, some poor guy accidentally announcing 12/9 and giving ATT a really bad day. What it doesn't contain is anything relevant to the special-case DDoS that all LISP sites would be vulnerable to, due to the IMO bad flow-cache management system that is specified. I am having a very great deal of trouble getting the authors of the threats document to even understand what the problem is, because as one of them put it, he is just a researcher. I am sure he and his colleagues are very smart guys, but they clearly do not remember our 1990s pains. That is the not an operator problem. It is understandable. Others who have been around long enough simply dismiss this problem, because they believe the unparalleled benefits of LISP for mobility and multi-homing SOHO sites must greatly out-weigh the fact that, well, if you are a content provider and you receive a DDoS, your site will be down and there isn't a damn thing you can do about it, other than spec routers that have way, way more FIB than the number of possible routes, again due to the bad caching scheme. The above is what I think is the ego-invested problem, where certain pretty smart, well-intentioned people have a lot of time, and professional credibility, invested in making LISP work. I'm sure it isn't pleasing for these guys to defend their project against my argument that it may never be able to reach Internet-scale, and that they have missed what I claim is a show-stopping problem with an easy way to improve it through several years of development. Especially since I am a guy who did not ever participate in the IETF before, someone they don't know from a random guy on the street. I am glad that this NANOG discussion has got some of these LISP folks to pay more attention to my argument, and my suggested improvement (I am not only bashing their project; I have positive input, too.) Simply posting to their mailing list once and emailing a few draft authors did not cause any movement at all. Evidently it does get attention, though, to jump up and down on a different list. Go figure! If operators don't provide input and *perspective* to things like LISP, we will end up with bad results. How many of us are amazed that we still do not have 32:32 bits BGP communities to go along with 32 bit ASNs, for signalling requests to transit providers without collision with other networks' community schemes? It is a pretty stupid situation, and yet here we are, with 32 bit ASN for years, and if you want to do advertisement control with 32 bit ASNs used, you are either mapping your 32 bit neighbors to special numbers, or your community scheme can overlap with others. That BGP community problem is pretty tiny compared to, what if people really started rolling out something new and clever like LISP, but in a half-baked, broken way that takes us back to 1990s era of small DDoS taking out whole data-center aggregation router. A lot of us think IPv6 is over-baked and broken, and probably this is why it has taken such a very long time to get anywhere with it. But ultimately, it is our fault for not participating. I am reversing my own behavior and providing input to some WGs I care about, in what time I have to do so. More operators should do the same.
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On 07/12/2011 08:43, Cameron Byrne wrote: Witness the latest debacle with the attempt at trying to make 6to4 historic. Various non-practicing entities were able to derail what network operators largely supported. Since the IETF failed to make progress operators will do other things to stop 6to4 ( i have heard no over IPv4 transport, blackhole 6to4 anycast, decom relay routers...) FYI, my understanding (and I'm sure Ron will correct me if I'm wrong) is that what's actually happening is that the IESG is pushing that draft forward knowing that it's going to be appealed. The appeal process will then sort itself out, and we'll have a result. The fact that one person can stall the end result through the appeals process is both a strength and occasionally a burden of the way the IETF does its work. Doug -- Nothin' ever doesn't change, but nothin' changes much. -- OK Go Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/
RE: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
-Original Message- From: Leo Bicknell [mailto:bickn...@ufp.org] Sent: Tuesday, July 12, 2011 11:42 AM To: Ronald Bonica Cc: nanog@nanog.org Subject: Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?) [snip] But there is no roadmap in the IETF process now for LISP that says We've got this 90% baked, we need to circulate a draft to the NANOG mailing list, request operator comments, and actively solicit operators to participate in the expanded test network. We need that mechanism to tell folks hey, it's real enough your operational feedback is now useful and come test our new idea. Leo, We need to fix this problem. Without the feedback loop that you describe, the IETF will never know whether they are producing useful stuff or nonsense. How does the following sound as a solution: Let's say we set up an new IETF mailing list, primarily for the use of operators. When an operator sees a draft that might be of interest to the operational community, he creates a new thread on the list, copying the draft authors and WG chairs. (The authors and chairs can decide whether to add the WG to the thread). The OPS AD will consider thread contents when evaluating the draft. Ron
RE: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
Cameron, Please stay tuned. While 6-to-4-historic is on hold, it is far from being dead. Expect more discussion in Quebec and on the mailing list. I doubt if there will be any final decision before Quebec. Ron -Original Message- From: Cameron Byrne [mailto:cb.li...@gmail.com] Sent: Tuesday, July 12, 2011 11:44 AM To: Ronald Bonica Cc: Leo Bicknell; nanog@nanog.org Subject: Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?) On Tue, Jul 12, 2011 at 8:28 AM, Ronald Bonica rbon...@juniper.net wrote: Leo, Maybe we can fix this by: a) bringing together larger groups of clueful operators in the IETF b) deciding which issues interest them c) showing up and being vocal as a group in protocol developing working groups To some degree, we already do this in the IETF OPS area, but judging by your comments, we don't do it nearly enough. Comments? There may be an OPS area, but it is not listened to. Witness the latest debacle with the attempt at trying to make 6to4 historic. Various non-practicing entities were able to derail what network operators largely supported. Since the IETF failed to make progress operators will do other things to stop 6to4 ( i have heard no over IPv4 transport, blackhole 6to4 anycast, decom relay routers...) Real network operators have a relatively low BS threshold, they have customers to support and businesses to run, and they don't have thumb wrestle these people who don't actually have any skin in the game. Cameron Ron -Original Message- From: Leo Bicknell [mailto:bickn...@ufp.org] Sent: Monday, July 11, 2011 3:35 PM To: nanog@nanog.org Subject: Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?) In a message written on Sun, Jul 10, 2011 at 06:16:09PM +0200, Jeroen Massar wrote: Eh ANYBODY, including you, can sign up to the IETF mailing lists and participate there, just like a couple of folks from NANOG are already doing. The way the IETF and the operator community interact is badly broken. The IETF does not want operators in many steps of the process. If you try to bring up operational concerns in early protocol development for example you'll often get a we'll look at that later response, which in many cases is right. Sometimes you just have to play with something before you worry about the operational details. It also does not help that many operational types are not hardcore programmers, and can't play in the sandbox during the major development cycles.
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
Leo Bicknell wrote: In short, make it easy for the operators to participate at the right time in the process. It will be better for everyone! Unfortunately, where you want to be inserted into the process is when everybody has said their piece 80-dozen times and are tired and just want to get on with life. So it doesn't matter whether you're an operator or the IESG -- you're not going to make many friends at that point telling them they got it wrong. On the other hand, is it really too much to ask operators -- especially big ones with a vested interest in not having the IETF throw crap over the wall for them to debug -- to *hire* a liaison whose job is to monitor a swath of working groups, bofs, etc, and participate the entire way through? I imagine they'd be pretty popular amongst clueful vendors, and would give you a leg up knowing what's good and what's just sales-drek. Mike
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Jul 12, 2011, at 12:40 PM, Michael Thomas wrote: Leo Bicknell wrote: In short, make it easy for the operators to participate at the right time in the process. It will be better for everyone! Unfortunately, where you want to be inserted into the process is when everybody has said their piece 80-dozen times and are tired and just want to get on with life. So it doesn't matter whether you're an operator or the IESG -- you're not going to make many friends at that point telling them they got it wrong. On the other hand, is it really too much to ask operators -- especially big ones with a vested interest in not having the IETF throw crap over the wall for them to debug -- to *hire* a liaison whose job is to monitor a swath of working groups, bofs, etc, and participate the entire way through? I imagine they'd be pretty popular amongst clueful vendors, and would give you a leg up knowing what's good and what's just sales-drek. By definition if crap has been thrown of the wall and you're trying to deploy it, that means: * you have a commercial or other compelling reason to run it. * someone has implemented it. the bar to make something relevant on those two points is much higher, than the one that involves submitting an internet draft. getting something through draft to publication via a working group is itself a rather involved process. Plenty of crap is thrown over the wall which you will never use, because the marketplace doesn't care, nobody built it, nobody has that problem it turns out, it turned out to be too hard or it was actually a dumb idea. in the market place for idea this seems normal and healthy. Mike
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Jul 12, 2011, at 12:53 PM, Owen DeLong wrote: On Jul 12, 2011, at 8:43 AM, Cameron Byrne wrote: On Tue, Jul 12, 2011 at 8:28 AM, Ronald Bonica rbon...@juniper.net wrote: Leo, Maybe we can fix this by: a) bringing together larger groups of clueful operators in the IETF b) deciding which issues interest them c) showing up and being vocal as a group in protocol developing working groups To some degree, we already do this in the IETF OPS area, but judging by your comments, we don't do it nearly enough. Comments? There may be an OPS area, but it is not listened to. Witness the latest debacle with the attempt at trying to make 6to4 historic. Various non-practicing entities were able to derail what network operators largely supported. Since the IETF failed to make progress operators will do other things to stop 6to4 ( i have heard no over IPv4 transport, blackhole 6to4 anycast, decom relay routers...) Those are all REALLY bad ideas. Speaking as an operator, the best thing you can do to alleviate the problems with 6to4 is operate more, not less 6to4 relays. Unless of course the large providers get their shared transition space in which case all 6to4 behind it will break in a really ugly way, pretty much exactly like in the mobile operator in question. The goal of 6to4 to historic was not to encourage the outcome described, it was to take having 6to4 as a default method of any kind off the table going into the future. If mature adults want to use it great, but conformance tests shouldn't require it, CPE shouldn't it on just because what they think they have a is a public IP with not filtering and hosts shouldn't use it unless told to do so.. Blocking over IPv4 transport is just silly. It's just as likely that your record is destined for an end-host that has native IPv6 connectivity with an intermediate resolver that desn't have IPv6 as it is that you're sending that to a 6to4 host. Further, there's no reason to believe the 6to4 host won't attempt to resolve via IPv6, so, it doesn't really help anyway. Real network operators have a relatively low BS threshold, they have customers to support and businesses to run, and they don't have thumb wrestle these people who don't actually have any skin in the game. I agree, but, it's not hard to run 6to4 relays and running them does much more to alleviate the problems with 6to4 than anything you proposed above. Indeed, what you proposed above will likely create more customer issues rather than reduce them. Owen Cameron Ron -Original Message- From: Leo Bicknell [mailto:bickn...@ufp.org] Sent: Monday, July 11, 2011 3:35 PM To: nanog@nanog.org Subject: Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?) In a message written on Sun, Jul 10, 2011 at 06:16:09PM +0200, Jeroen Massar wrote: Eh ANYBODY, including you, can sign up to the IETF mailing lists and participate there, just like a couple of folks from NANOG are already doing. The way the IETF and the operator community interact is badly broken. The IETF does not want operators in many steps of the process. If you try to bring up operational concerns in early protocol development for example you'll often get a we'll look at that later response, which in many cases is right. Sometimes you just have to play with something before you worry about the operational details. It also does not help that many operational types are not hardcore programmers, and can't play in the sandbox during the major development cycles.
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Jul 11, 2011, at 7:19 PM, Jeff Wheeler wrote: Again, this is only hard to understand (or accept) if you don't know how your routers work. * why do you think there is an ARP and ND table? * why do you think there are policers to protect the CPU from excessive ARP/ND punts or traffic? * do you even know the limit of your boxes' ARP / ND tables? Do you realize that limit is a tiny fraction of one /64? * do you understand what happens when your ARP/ND policers are reached? * did you think about the impact on neighboring routers and protocol next-hops, not just servers? * did you every try to deploy a /16 on a flat LAN with a lot of hosts and see what happens? Doesn't work too well. A v6 /64 is 281 trillion times bigger than a v4 /16. There's no big leap of logic here as to why one rogue machine could break your LAN. FYI, in case you're interested in these topics, the IETF working group ARMD was chartered to explore address resolution scale. I'm one of the co-chairs. It's in the Operations Area, and we'd love to have more operators involved - if you're willing to contribute, your input will help set the direction. (If operators don't contribute, it will be just another vendor-led circle... well, you know the score.) For details please see http://tools.ietf.org/wg/armd/charters. Cheers, -Benson
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
In message 56e0fb8f-bb53-4db0-829b-39dfbab48...@bogus.com, Joel Jaeggli write s: On Jul 12, 2011, at 12:53 PM, Owen DeLong wrote: =20 On Jul 12, 2011, at 8:43 AM, Cameron Byrne wrote: =20 On Tue, Jul 12, 2011 at 8:28 AM, Ronald Bonica rbon...@juniper.net = wrote: Leo, =20 Maybe we can fix this by: =20 a) bringing together larger groups of clueful operators in the IETF b) deciding which issues interest them c) showing up and being vocal as a group in protocol developing = working groups =20 To some degree, we already do this in the IETF OPS area, but judging = by your comments, we don't do it nearly enough. =20 Comments? =20 =20 There may be an OPS area, but it is not listened to. =20 Witness the latest debacle with the attempt at trying to make 6to4 = historic. =20 Various non-practicing entities were able to derail what network operators largely supported. Since the IETF failed to make progress operators will do other things to stop 6to4 ( i have heard no over IPv4 transport, blackhole 6to4 anycast, decom relay routers...) =20 Those are all REALLY bad ideas. Speaking as an operator, the best = thing you can do to alleviate the problems with 6to4 is operate more, not less = 6to4 relays. Unless of course the large providers get their shared transition space = in which case all 6to4 behind it will break in a really ugly way, pretty = much exactly like in the mobile operator in question.=20 And would deploying draft-andrews-v6ops-6to4-router-option-02.txt and/or adding router reachability tests have addressed this issue? The goal of 6to4 to historic was not to encourage the outcome described, = it was to take having 6to4 as a default method of any kind off the table = going into the future. If mature adults want to use it great, but = conformance tests shouldn't require it, CPE shouldn't it on just because = what they think they have a is a public IP with not filtering and hosts = shouldn't use it unless told to do so.. But that is *not* what the draft did. Making the protocol historic did LOTS more than that. I think there was universal consensus that 6to4 should be off by default. There was this nuke 6to4 from orbit attitude which did nothing to help with already deployed/shipped boxes. 6to4 historic is actually harmful for dealing with the existing problems as it tells vendors not to include 6to4 support in future products which means operators won't have boxes with fixes to other problems to alleviate the problems cause but the currently deployed customer boxes. What would have been much better would have been to encourage CPE vendors to release images which address some of the known issues. Just adding a check box saying enable 6to4 and for ISP to send out email to say check your router vendor web site for fixed images. The better fix would be to get them to also add support for draft-andrews-v6ops-6to4-router-option-02.txt which greys out the checkbox when 0.0.0.0 is sent as a response to the option. Remember operators are in the position to alleviate lots of the 6to4 issues themselves. Blocking over IPv4 transport is just silly. It's just as likely = that your record is destined for an end-host that has native IPv6 = connectivity with an intermediate resolver that desn't have IPv6 as it is that = you're sending that to a 6to4 host. Further, there's no reason to believe the 6to4 host won't attempt to resolve via IPv6, so, it doesn't really = help anyway. =20 Real network operators have a relatively low BS threshold, they have customers to support and businesses to run, and they don't have = thumb wrestle these people who don't actually have any skin in the game. =20 I agree, but, it's not hard to run 6to4 relays and running them does = much more to alleviate the problems with 6to4 than anything you proposed above. Indeed, what you proposed above will likely create more = customer issues rather than reduce them. =20 Owen =20 Cameron =20 =20 Ron =20 =20 -Original Message- From: Leo Bicknell [mailto:bickn...@ufp.org] Sent: Monday, July 11, 2011 3:35 PM To: nanog@nanog.org Subject: Re: Anybody can participate in the IETF (Was: Why is IPv6 = broken?) =20 In a message written on Sun, Jul 10, 2011 at 06:16:09PM +0200, = Jeroen Massar wrote: Eh ANYBODY, including you, can sign up to the IETF mailing = lists and participate there, just like a couple of folks from NANOG are = already doing. =20 The way the IETF and the operator community interact is badly = broken. =20 The IETF does not want operators in many steps of the process. If = you try to bring up operational concerns in early protocol development = for example you'll often get a we'll look at that later response, = which in many cases is right. Sometimes you just have to play with = something before you worry about the operational details. It also does
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Jul 12, 2011, at 2:21 PM, Joel Jaeggli wrote: On Jul 12, 2011, at 12:53 PM, Owen DeLong wrote: On Jul 12, 2011, at 8:43 AM, Cameron Byrne wrote: On Tue, Jul 12, 2011 at 8:28 AM, Ronald Bonica rbon...@juniper.net wrote: Leo, Maybe we can fix this by: a) bringing together larger groups of clueful operators in the IETF b) deciding which issues interest them c) showing up and being vocal as a group in protocol developing working groups To some degree, we already do this in the IETF OPS area, but judging by your comments, we don't do it nearly enough. Comments? There may be an OPS area, but it is not listened to. Witness the latest debacle with the attempt at trying to make 6to4 historic. Various non-practicing entities were able to derail what network operators largely supported. Since the IETF failed to make progress operators will do other things to stop 6to4 ( i have heard no over IPv4 transport, blackhole 6to4 anycast, decom relay routers...) Those are all REALLY bad ideas. Speaking as an operator, the best thing you can do to alleviate the problems with 6to4 is operate more, not less 6to4 relays. Unless of course the large providers get their shared transition space in which case all 6to4 behind it will break in a really ugly way, pretty much exactly like in the mobile operator in question. Actually, if those same providers run 6to4 gateways/routers on their networks in that shared transition space with public IPv6 addresses on the exterior, it would not break at all. As I said, the resolution to the 6to4 problems described is to run MORE, not less 6to4 gateways. The goal of 6to4 to historic was not to encourage the outcome described, it was to take having 6to4 as a default method of any kind off the table going into the future. If mature adults want to use it great, but conformance tests shouldn't require it, CPE shouldn't it on just because what they think they have a is a public IP with not filtering and hosts shouldn't use it unless told to do so.. I have no problem with saying 6to4 should not be enabled by default. However, that doesn't change the fact that the best way to resolve things given current shipping software and hardware is to deploy 6to4 gateways in the appropriate places. Owen Blocking over IPv4 transport is just silly. It's just as likely that your record is destined for an end-host that has native IPv6 connectivity with an intermediate resolver that desn't have IPv6 as it is that you're sending that to a 6to4 host. Further, there's no reason to believe the 6to4 host won't attempt to resolve via IPv6, so, it doesn't really help anyway. Real network operators have a relatively low BS threshold, they have customers to support and businesses to run, and they don't have thumb wrestle these people who don't actually have any skin in the game. I agree, but, it's not hard to run 6to4 relays and running them does much more to alleviate the problems with 6to4 than anything you proposed above. Indeed, what you proposed above will likely create more customer issues rather than reduce them. Owen Cameron Ron -Original Message- From: Leo Bicknell [mailto:bickn...@ufp.org] Sent: Monday, July 11, 2011 3:35 PM To: nanog@nanog.org Subject: Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?) In a message written on Sun, Jul 10, 2011 at 06:16:09PM +0200, Jeroen Massar wrote: Eh ANYBODY, including you, can sign up to the IETF mailing lists and participate there, just like a couple of folks from NANOG are already doing. The way the IETF and the operator community interact is badly broken. The IETF does not want operators in many steps of the process. If you try to bring up operational concerns in early protocol development for example you'll often get a we'll look at that later response, which in many cases is right. Sometimes you just have to play with something before you worry about the operational details. It also does not help that many operational types are not hardcore programmers, and can't play in the sandbox during the major development cycles.
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Jul 12, 2011 6:42 PM, Mark Andrews ma...@isc.org wrote:
In message 56e0fb8f-bb53-4db0-829b-39dfbab48...@bogus.com, Joel Jaeggli
write
s:
On Jul 12, 2011, at 12:53 PM, Owen DeLong wrote:
=20
On Jul 12, 2011, at 8:43 AM, Cameron Byrne wrote:
=20
On Tue, Jul 12, 2011 at 8:28 AM, Ronald Bonica rbon...@juniper.net
=
wrote:
Leo,
=20
Maybe we can fix this by:
=20
a) bringing together larger groups of clueful operators in the IETF
b) deciding which issues interest them
c) showing up and being vocal as a group in protocol developing =
working groups
=20
To some degree, we already do this in the IETF OPS area, but judging
=
by your comments, we don't do it nearly enough.
=20
Comments?
=20
=20
There may be an OPS area, but it is not listened to.
=20
Witness the latest debacle with the attempt at trying to make 6to4 =
historic.
=20
Various non-practicing entities were able to derail what network
operators largely supported. Since the IETF failed to make progress
operators will do other things to stop 6to4 ( i have heard no
over IPv4 transport, blackhole 6to4 anycast, decom relay routers...)
=20
Those are all REALLY bad ideas. Speaking as an operator, the best =
thing you
can do to alleviate the problems with 6to4 is operate more, not less =
6to4
relays.
Unless of course the large providers get their shared transition space =
in which case all 6to4 behind it will break in a really ugly way, pretty
=
much exactly like in the mobile operator in question.=20
And would deploying draft-andrews-v6ops-6to4-router-option-02.txt and/or
adding router reachability tests have addressed this issue?
The goal of 6to4 to historic was not to encourage the outcome described,
=
it was to take having 6to4 as a default method of any kind off the table
=
going into the future. If mature adults want to use it great, but =
conformance tests shouldn't require it, CPE shouldn't it on just because
=
what they think they have a is a public IP with not filtering and hosts
=
shouldn't use it unless told to do so..
But that is *not* what the draft did. Making the protocol historic
did LOTS more than that. I think there was universal consensus
that 6to4 should be off by default.
There was this nuke 6to4 from orbit attitude which did nothing to
help with already deployed/shipped boxes. 6to4 historic is actually
harmful for dealing with the existing problems as it tells vendors
not to include 6to4 support in future products which means operators
won't have boxes with fixes to other problems to alleviate the
problems cause but the currently deployed customer boxes.
What would have been much better would have been to encourage CPE
vendors to release images which address some of the known issues.
Just adding a check box saying enable 6to4 and for ISP to send
out email to say check your router vendor web site for fixed
images. The better fix would be to get them to also add support
for draft-andrews-v6ops-6to4-router-option-02.txt which greys out
the checkbox when 0.0.0.0 is sent as a response to the option.
Remember operators are in the position to alleviate lots of the
6to4 issues themselves.
But they will not. If there is not a revenue forecast, there is no project.
That said, CGN is moving forward as a keep the lights on initiative as
is real native v6.
I don't care to rehash this yet again with no progress.
Cb.
Blocking over IPv4 transport is just silly. It's just as likely =
that your
record is destined for an end-host that has native IPv6 =
connectivity
with an intermediate resolver that desn't have IPv6 as it is that =
you're
sending that to a 6to4 host. Further, there's no reason to believe the
6to4 host won't attempt to resolve via IPv6, so, it doesn't really =
help
anyway.
=20
Real network operators have a relatively low BS threshold, they have
customers to support and businesses to run, and they don't have =
thumb
wrestle these people who don't actually have any skin in the game.
=20
I agree, but, it's not hard to run 6to4 relays and running them does =
much
more to alleviate the problems with 6to4 than anything you proposed
above. Indeed, what you proposed above will likely create more =
customer
issues rather than reduce them.
=20
Owen
=20
Cameron
=20
=20
Ron
=20
=20
-Original Message-
From: Leo Bicknell [mailto:bickn...@ufp.org]
Sent: Monday, July 11, 2011 3:35 PM
To: nanog@nanog.org
Subject: Re: Anybody can participate in the IETF (Was: Why is IPv6 =
broken?)
=20
In a message written on Sun, Jul 10, 2011 at 06:16:09PM +0200, =
Jeroen Massar wrote:
Eh ANYBODY, including you, can sign up to the IETF mailing =
lists
and participate there, just like a couple of folks from NANOG are =
already doing.
=20
The way the IETF
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Jul 12, 2011, at 7:20 PM, Owen DeLong wrote: On Jul 12, 2011, at 2:21 PM, Joel Jaeggli wrote: On Jul 12, 2011, at 12:53 PM, Owen DeLong wrote: On Jul 12, 2011, at 8:43 AM, Cameron Byrne wrote: On Tue, Jul 12, 2011 at 8:28 AM, Ronald Bonica rbon...@juniper.net wrote: Leo, Maybe we can fix this by: a) bringing together larger groups of clueful operators in the IETF b) deciding which issues interest them c) showing up and being vocal as a group in protocol developing working groups To some degree, we already do this in the IETF OPS area, but judging by your comments, we don't do it nearly enough. Comments? There may be an OPS area, but it is not listened to. Witness the latest debacle with the attempt at trying to make 6to4 historic. Various non-practicing entities were able to derail what network operators largely supported. Since the IETF failed to make progress operators will do other things to stop 6to4 ( i have heard no over IPv4 transport, blackhole 6to4 anycast, decom relay routers...) Those are all REALLY bad ideas. Speaking as an operator, the best thing you can do to alleviate the problems with 6to4 is operate more, not less 6to4 relays. Unless of course the large providers get their shared transition space in which case all 6to4 behind it will break in a really ugly way, pretty much exactly like in the mobile operator in question. Actually, if those same providers run 6to4 gateways/routers on their networks in that shared transition space with public IPv6 addresses on the exterior, it would not break at all. arin 2011-5 specfically cites numbering cpe in space as the justification for deployment. the cpe therefore have to be natted and you are implying that you'll be natting the 6to4, overall I'd put that in the less desirable category as far as violating expectations go... As I said, the resolution to the 6to4 problems described is to run MORE, not less 6to4 gateways. Are you advocating draft-kuarsingh-v6ops-6to4-provider-managed-tunnel? http://tools.ietf.org/html/draft-kuarsingh-v6ops-6to4-provider-managed-tunnel-02 The goal of 6to4 to historic was not to encourage the outcome described, it was to take having 6to4 as a default method of any kind off the table going into the future. If mature adults want to use it great, but conformance tests shouldn't require it, CPE shouldn't it on just because what they think they have a is a public IP with not filtering and hosts shouldn't use it unless told to do so.. I have no problem with saying 6to4 should not be enabled by default. However, that doesn't change the fact that the best way to resolve things given current shipping software and hardware is to deploy 6to4 gateways in the appropriate places. and we have http://tools.ietf.org/html/draft-ietf-v6ops-6to4-advisory-02 The fact of the matter is more 6to4 relays is only an anodyne as is rejiggering the address selection priority, the pain may go down it won't go away. Owen Blocking over IPv4 transport is just silly. It's just as likely that your record is destined for an end-host that has native IPv6 connectivity with an intermediate resolver that desn't have IPv6 as it is that you're sending that to a 6to4 host. Further, there's no reason to believe the 6to4 host won't attempt to resolve via IPv6, so, it doesn't really help anyway. Real network operators have a relatively low BS threshold, they have customers to support and businesses to run, and they don't have thumb wrestle these people who don't actually have any skin in the game. I agree, but, it's not hard to run 6to4 relays and running them does much more to alleviate the problems with 6to4 than anything you proposed above. Indeed, what you proposed above will likely create more customer issues rather than reduce them. Owen Cameron Ron -Original Message- From: Leo Bicknell [mailto:bickn...@ufp.org] Sent: Monday, July 11, 2011 3:35 PM To: nanog@nanog.org Subject: Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?) In a message written on Sun, Jul 10, 2011 at 06:16:09PM +0200, Jeroen Massar wrote: Eh ANYBODY, including you, can sign up to the IETF mailing lists and participate there, just like a couple of folks from NANOG are already doing. The way the IETF and the operator community interact is badly broken. The IETF does not want operators in many steps of the process. If you try to bring up operational concerns in early protocol development for example you'll often get a we'll look at that later response, which in many cases is right. Sometimes you just have to play with something before you worry about the operational details. It also does not help that many operational types are not hardcore programmers, and can't play in the sandbox during the major development cycles.
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
and running them does = much more to alleviate the problems with 6to4 than anything you proposed above. Indeed, what you proposed above will likely create more = customer issues rather than reduce them. =20 Owen =20 Cameron =20 =20 Ron =20 =20 -Original Message- From: Leo Bicknell [mailto:bickn...@ufp.org] Sent: Monday, July 11, 2011 3:35 PM To: nanog@nanog.org Subject: Re: Anybody can participate in the IETF (Was: Why is IPv6 = broken?) =20 In a message written on Sun, Jul 10, 2011 at 06:16:09PM +0200, = Jeroen Massar wrote: Eh ANYBODY, including you, can sign up to the IETF mailing = lists and participate there, just like a couple of folks from NANOG are = already doing. =20 The way the IETF and the operator community interact is badly = broken. =20 The IETF does not want operators in many steps of the process. If = you try to bring up operational concerns in early protocol development = for example you'll often get a we'll look at that later response, = which in many cases is right. Sometimes you just have to play with = something before you worry about the operational details. It also does = not help that many operational types are not hardcore programmers, and = can't play in the sandbox during the major development cycles. =20 =20 =20 =20 =20 =20 =20 -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Sun, Jul 10, 2011 at 4:22 PM, Owen DeLong o...@delong.com wrote: On Jul 10, 2011, at 12:23 PM, William Herrin wrote: Consider, for example, RFC 3484. That's the one that determines how an IPv6 capable host selects which of a group of candidate IPv4 and IPv6 addresses for a particular host name gets priority. How is a server's address priority NOT an issue that should be managed at an operations level by individual server administrators? Yet the working group which produced it came up with a static prioritization that is the root cause of a significant portion of the IPv6 deployment headaches we face. 3484 specifies a static default. By definition, defaults in absence of operator configuration kind of have to be static. Having a reasonable and expected set of defaults documented in an RFC provides a known quantity for what operators can/should expect from hosts they have not configured. I see nothing wrong with RFC 3484 other than I would agree that the choices made were suboptimal. Mostly that was based on optimism and a lack of experience available at the time of writing. Hi Owen, A more optimal answer would have been to make records more like MX or SRV records -- with explicit priorities the clients are encouraged to follow. I wasn't there but I'd be willing to bet there was a lonely voice in the room saying, hey, this should be controlled by the sysadmin. A lonely voice that got shouted down. Today's RFC candidates are required to call out IANA considerations and security considerations in special sections. They do so because each of these areas has landmines that the majority of working groups are ill equipped to consider on their own. There should be an operations callout as well -- a section where proposed operations defaults (as well as statics for which a solid case can be made for an operations tunable) are extracted from the thick of it and offered for operator scrutiny prior to publication of the RFC. I think this would be a good idea, actually. It would probably be more effective to propose it to IETF than to NANOG, however. If the complaint is that the IETF doesn't adequately listen to the operations folk, then I think it makes sense to consult the operations folks early and often on potential fixes. If folks here think it would help, -that- is when I'll it to the IETF. Regards, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Jul 10, 2011, at 11:57 PM, William Herrin wrote: On Sun, Jul 10, 2011 at 4:22 PM, Owen DeLong o...@delong.com wrote: On Jul 10, 2011, at 12:23 PM, William Herrin wrote: Consider, for example, RFC 3484. That's the one that determines how an IPv6 capable host selects which of a group of candidate IPv4 and IPv6 addresses for a particular host name gets priority. How is a server's address priority NOT an issue that should be managed at an operations level by individual server administrators? Yet the working group which produced it came up with a static prioritization that is the root cause of a significant portion of the IPv6 deployment headaches we face. 3484 specifies a static default. By definition, defaults in absence of operator configuration kind of have to be static. Having a reasonable and expected set of defaults documented in an RFC provides a known quantity for what operators can/should expect from hosts they have not configured. I see nothing wrong with RFC 3484 other than I would agree that the choices made were suboptimal. Mostly that was based on optimism and a lack of experience available at the time of writing. Hi Owen, A more optimal answer would have been to make records more like MX or SRV records -- with explicit priorities the clients are encouraged to follow. I wasn't there but I'd be willing to bet there was a lonely voice in the room saying, hey, this should be controlled by the sysadmin. A lonely voice that got shouted down. Give me a break... multiple implementations have chosen to tweak the algorithm independently and at various times. It's just an rfc, not the gospel according to richard draves. Acknowledgments The author would like to acknowledge the contributions of the IPng Working Group, particularly Marc Blanchet, Brian Carpenter, Matt Crawford, Alain Durand, Steve Deering, Robert Elz, Jun-ichiro itojun Hagino, Tony Hain, M.T. Hollinger, JINMEI Tatuya, Thomas Narten, Erik Nordmark, Ken Powell, Markku Savela, Pekka Savola, Hesham Soliman, Dave Thaler, Mauro Tortonesi, Ole Troan, and Stig Venaas. In addition, the anonymous IESG reviewers had many great comments and suggestions for clarification. Today's RFC candidates are required to call out IANA considerations and security considerations in special sections. They do so because each of these areas has landmines that the majority of working groups are ill equipped to consider on their own. There should be an operations callout as well -- a section where proposed operations defaults (as well as statics for which a solid case can be made for an operations tunable) are extracted from the thick of it and offered for operator scrutiny prior to publication of the RFC. I think this would be a good idea, actually. It would probably be more effective to propose it to IETF than to NANOG, however. If the complaint is that the IETF doesn't adequately listen to the operations folk, then I think it makes sense to consult the operations folks early and often on potential fixes. If folks here think it would help, -that- is when I'll it to the IETF. Regards, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: Why is IPv6 broken?
On Sun, 2011-07-10 at 10:14 -0400, Jeff Wheeler wrote: Cogent's policy of requiring a new contract, and from what I am still being told by some European customers, new money, from customers in exchange for provisioning IPv6 on existing circuits, means a simple technical project gets caught up in the complexities of budgeting and contract execution. Can we have IPv6 transit? Yes, please turn up a session to.. That was asking Cogent for IPv6 dual-stack on our existing IPv4 transit. I'm not saying it's any good, but it certainly didn't cost extra. Tom
Re: Why is IPv6 broken?
On 11/07/2011 08:25, Tom Hill wrote: I'm not saying it's any good, but it certainly didn't cost extra. Several people mentioned this to Jeff on IRC a short time ago, so it's not clear why he chose to suggest that ipv6 users in Europe were being fleeced by Cogent for a set-up fee. Perhaps it has happened, but it appears not to be their policy. Of course, if you actually want a full ipv6 table, you will need to go elsewhere. Nick
Re: Why is IPv6 broken?
On Mon, Jul 11, 2011 at 3:25 AM, Tom Hill t...@ninjabadger.net wrote: On Sun, 2011-07-10 at 10:14 -0400, Jeff Wheeler wrote: Cogent's policy of requiring a new contract, and from what I am still being told by some European customers, new money, from customers in exchange for provisioning IPv6 on existing circuits, means a simple technical project gets caught up in the complexities of budgeting and contract execution. Can we have IPv6 transit? Yes, please turn up a session to.. That was asking Cogent for IPv6 dual-stack on our existing IPv4 transit. I continue to hear different. In my first-hand experience just about three weeks ago, I was told by Cogent that I need to execute a new contract to get IPv6 added to an existing IPv4 circuit (U.S. customer.) This turned a simple pilot project with only a few I.T. folks involved into, well, I'm still waiting on this new contract to be executed. I'm not surprised. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Mon, 11 Jul 2011, William Herrin wrote: If the complaint is that the IETF doesn't adequately listen to the operations folk, then I think it makes sense to consult the operations folks early and often on potential fixes. If folks here think it would help, -that- is when I'll it to the IETF. I started participating in the IETF 1-2 years ago. Coming from Fidonet background, the threshold of entry felt very low, as long as you make any kind of sense, people will discuss with you there and it doesn't matter who you are. You don't even have to go to the meetings (I've only been to a single one). I encourage everybody to participate, at least to subscribe to the WG mailing lists and keep a look out for the draft announcements and give feedback to those. If we in the ISP business don't do this, the show will be run by the vendors and academics (as is the case right now). They're saying come to us, you're saying come to us, and as long as both do this the rate of communication is going to be limited. What is needed is more people with operational backgrounds. For instance, I pitched the idea that ended up as a draft, dunno what will come of it: http://www.ietf.org/mail-archive/web/isis-wg/current/msg02556.html This has purely operational background and the puritans didn't like it (they didn't even understand why one would want to do it like that), but after a while I feel I received some traction and it might actually end up as a protocol enhancement that will help some ISPs in their daily work. Even something like your IGP isn't done, and can be enhanced even if it takes time. -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: Why is IPv6 broken?
On Mon, 2011-07-11 at 04:50 -0400, Jeff Wheeler wrote: Can we have IPv6 transit? Yes, please turn up a session to.. That was asking Cogent for IPv6 dual-stack on our existing IPv4 transit. I continue to hear different. In my first-hand experience just about three weeks ago, I was told by Cogent that I need to execute a new contract to get IPv6 added to an existing IPv4 circuit (U.S. customer.) This turned a simple pilot project with only a few I.T. folks involved into, well, I'm still waiting on this new contract to be executed. I'm not surprised. In fairness, we have a small commit. If you're talking multi-gigabit+, then perhaps they could be a little more concerned about the amount of IPv6 traffic that you might start pushing, leading to delay tactics and/or a required contract change to protect themselves. (Not that it's likely much to be concerned about. But then, I don't know who your customer is. ;)) Or the more likely reality that one hand doesn't talk to the other and everyone's getting varying answers/actions from Cogent, depending on whom they speak with. Tom
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
Sent from my iPad On Jul 11, 2011, at 2:57, William Herrin b...@herrin.us wrote: On Sun, Jul 10, 2011 at 4:22 PM, Owen DeLong o...@delong.com wrote: On Jul 10, 2011, at 12:23 PM, William Herrin wrote: Consider, for example, RFC 3484. That's the one that determines how an IPv6 capable host selects which of a group of candidate IPv4 and IPv6 addresses for a particular host name gets priority. How is a server's address priority NOT an issue that should be managed at an operations level by individual server administrators? Yet the working group which produced it came up with a static prioritization that is the root cause of a significant portion of the IPv6 deployment headaches we face. 3484 specifies a static default. By definition, defaults in absence of operator configuration kind of have to be static. Having a reasonable and expected set of defaults documented in an RFC provides a known quantity for what operators can/should expect from hosts they have not configured. I see nothing wrong with RFC 3484 other than I would agree that the choices made were suboptimal. Mostly that was based on optimism and a lack of experience available at the time of writing. Hi Owen, A more optimal answer would have been to make records more like MX or SRV records -- with explicit priorities the clients are encouraged to follow. I wasn't there but I'd be willing to bet there was a lonely voice in the room saying, hey, this should be controlled by the sysadmin. A lonely voice that got shouted down. Uh, right, because the average system administrator wants the remote host telling his systems which address to prefer? Besides, that would have been DESTINATION address selection, not source address selection which isn't what we're talking about. I wasn't there either, but, it _IS_ controlled by the sysadmin. There are defaults in case the sysadmin is asleep at the switch (RFC 3484) and there are handles and knobs for the sysadmin to tune if he wants (the other RFC that I referred you to). Today's RFC candidates are required to call out IANA considerations and security considerations in special sections. They do so because each of these areas has landmines that the majority of working groups are ill equipped to consider on their own. There should be an operations callout as well -- a section where proposed operations defaults (as well as statics for which a solid case can be made for an operations tunable) are extracted from the thick of it and offered for operator scrutiny prior to publication of the RFC. I think this would be a good idea, actually. It would probably be more effective to propose it to IETF than to NANOG, however. If the complaint is that the IETF doesn't adequately listen to the operations folk, then I think it makes sense to consult the operations folks early and often on potential fixes. If folks here think it would help, -that- is when I'll it to the IETF. I think it would help. Hopefully others will express similar sentiment. Owen
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Mon, Jul 11, 2011 at 3:08 AM, Joel Jaeggli joe...@bogus.com wrote: On Jul 10, 2011, at 11:57 PM, William Herrin wrote: A more optimal answer would have been to make records more like MX or SRV records -- with explicit priorities the clients are encouraged to follow. I wasn't there but I'd be willing to bet there was a lonely voice in the room saying, hey, this should be controlled by the sysadmin. A lonely voice that got shouted down. Give me a break... multiple implementations have chosen to tweak the algorithm independently and at various times. It's just an rfc, not the gospel according to richard draves. Acknowledgments The author would like to acknowledge the contributions of the IPng Working Group, particularly Marc Blanchet, Brian Carpenter, Matt Crawford, Alain Durand, Steve Deering, Robert Elz, Jun-ichiro itojun Hagino, Tony Hain, M.T. Hollinger, JINMEI Tatuya, Thomas Narten, Erik Nordmark, Ken Powell, Markku Savela, Pekka Savola, Hesham Soliman, Dave Thaler, Mauro Tortonesi, Ole Troan, and Stig Venaas. In addition, the anonymous IESG reviewers had many great comments and suggestions for clarification. Joel, I am giving you a break. Instead of calling this list of folks to the carpet over a failure of imagination that by the time we've ubiquitously deployed IPv6 will have been the root cause of billions if not tens of billions of dollars in needless industry expense, I'm trying to move the discussion past the errors and focus on ways to help the next team of smart, selfless and dedicated individuals avoid sullying their results with a similar mistake. Denial keeps the discussion focused on the errors. You don't want that and neither do I. Today's RFC candidates are required to call out IANA considerations and security considerations in special sections. They do so because each of these areas has landmines that the majority of working groups are ill equipped to consider on their own. There should be an operations callout as well -- a section where proposed operations defaults (as well as statics for which a solid case can be made for an operations tunable) are extracted from the thick of it and offered for operator scrutiny prior to publication of the RFC. Do you find this adjustment objectionable? Do you have other fresh ideas to float? Something better than the tired refrain about operators not showing up? 'Cause I have to tell you: Several years ago I picked a working group and I showed up. And I faced and lost the argument against the persistent certainty on the workability of ridiculous deployment scenarios by folks who never managed any system larger than a software development lab. And I stopped participating in the group about a year ago as the core of participants who hadn't given up wandered off into la la land. Regards, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Jul 11, 2011, at 8:13 AM, William Herrin wrote: Today's RFC candidates are required to call out IANA considerations and security considerations in special sections. They do so because each of these areas has landmines that the majority of working groups are ill equipped to consider on their own. There should be an operations callout as well -- a section where proposed operations defaults (as well as statics for which a solid case can be made for an operations tunable) are extracted from the thick of it and offered for operator scrutiny prior to publication of the RFC. Do you find this adjustment objectionable? Do you have other fresh ideas to float? Something better than the tired refrain about operators not showing up? The operations area has a directorate. It reviews basically every draft in front of the IESG. I'm on it. Am I not an operator? Do I think that adding yet another required section to an internet draft is going to increase it's quality? No I do not. 'Cause I have to tell you: Several years ago I picked a working group and I showed up. And I faced and lost the argument against the persistent certainty on the workability of ridiculous deployment scenarios by folks who never managed any system larger than a software development lab. And I stopped participating in the group about a year ago as the core of participants who hadn't given up wandered off into la la land. Regards, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Mon, Jul 11, 2011 at 11:20 AM, Joel Jaeggli joe...@bogus.com wrote: On Jul 11, 2011, at 8:13 AM, William Herrin wrote: Today's RFC candidates are required to call out IANA considerations and security considerations in special sections. They do so because each of these areas has landmines that the majority of working groups are ill equipped to consider on their own. There should be an operations callout as well -- a section where proposed operations defaults (as well as statics for which a solid case can be made for an operations tunable) are extracted from the thick of it and offered for operator scrutiny prior to publication of the RFC. Do you find this adjustment objectionable? Do you have other fresh ideas to float? Something better than the tired refrain about operators not showing up? The operations area has a directorate. It reviews basically every draft in front of the IESG. I'm on it. Am I not an operator? Well, you work at Zynga, a company which makes facebook games. Before that you worked at Nokia, company which makes phones but doesn't run phone networks. Before that it was Check Point Software, a company which makes firewalls but doesn't run networks. And before that it was the University of Oregon. Do you believe any of those roles offered you the perspective you'd gain working for an ISP, telco or MSO? Are you not an operator? Sure, why not. It's a big tent. Are you well qualified to represent operator interests before the IETF? You really haven't been speaking to the issues I had to deal with when I led an ISP and you've expressed little respect for the validity of issues I face now. But you do show up. Regards, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
\
I have found my input on the LISP list completely ignored because, as
you suggest, my concerns are real-world and don't have any impact on
someone's pet project. LISP as it stands today can never work on the
Internet, and regardless of the fine reputations of the people at
Cisco and other organizations who are working on it, they are either
furthering it only because they would rather work on a pet project
than something useful to customers, or because they truly cannot
understand its deep, insurmountable design flaws at Internet-scale.
You would generally hope that someone saying, LISP can't work at
Internet-scale because anyone will be able to trivially DoS any LISP
ITR ('router' for simplicity), but here is a way you can improve it,
well, that remark, input, and person should be taken quite seriously,
their input examined, and other assumptions about the way LISP is
supposed to work ought to be questioned. None of this has happened.
Jeff I've spend many hours working through the issues you brought up (indeed
cache management, population, and security are three of my focus areas in LISP,
and something we considered when we started this), have been socializing them
with the LISP team, and can personally say that I take your comments very
seriously. Or testing group in house as well as on the LISP beta network have
been working through these issues. Also, we've had an email thread going on
about this for, by my count, 3-4 replies back and forth.
While I appreciate your opinions above, I have to say that I disagree with
them, and also with the conclusions you draw.
-Darrel
P.S. oh and Randy Bush is pretty damn smart. :-)
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On 7/10/11 6:29 PM, Randy Bush wrote: The IETF is run by volunteers. They volunteer because they find designing protocols to be fun. For the most part, operators are not entertained by designing network protocols. So, for the most part they don't partiticpate. Randy Bush, Editorial zone: Into the Future with the Internet Vendor Task Force: a very curmudgeonly view, or testing spaghetti, ACM SIGCOMM Computer Communication Review Volume 35 Issue 5, October 2005. http://archive.psg.com/051000.ccr-ivtf.html I agree with Randy. Well, that's no surprise, I usually agree with Randy. But I didn't know/remember that he'd managed to get his rant published! Good work But the problem has been pretty apparent since circa 1991. I remember calls for an Internet Operator's Task Force (IOTF) to parallel IETF sometime in '92 or '93. Folks have asked me from time to time why I stopped participating in the IETF a decade or so ago. My usual tongue-in-cheek reply is, it's more important to use the protocols we already have before we build more. (CF. nukes.) IPv6 was certainly a part of it (as was security). As I remind folks from time to time, I'm the guy that originally registered v6 with IANA. But PIPE-SIP-SIPP was a much simpler, shorter, cleaner extension using 64-bit addresses. My proposal used the upper 32-bits extending the then 16-bit BGP ASN, making addresses match topology, shrinking the routing tables Although I *do* find designing protocols to be fun, these days I only post Experimental drafts. There are committees (dysfunctional working groups) where the chair cannot get his own drafts through the process in under 4 years. It took about 7 years to publish the group negotiation extension to SSH, many years after it was shipping. It's no wonder that operators don't want to participate.
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Mon, Jul 11, 2011 at 11:20 AM, Joel Jaeggli joe...@bogus.com wrote: On Jul 11, 2011, at 8:13 AM, William Herrin wrote: Today's RFC candidates are required to call out IANA considerations and security considerations in special sections. They do so because each of these areas has landmines that the majority of working groups are ill equipped to consider on their own. There should be an operations callout as well -- a section where proposed operations defaults (as well as statics for which a solid case can be made for an operations tunable) are extracted from the thick of it and offered for operator scrutiny prior to publication of the RFC. Do you find this adjustment objectionable? Do I think that adding yet another required section to an internet draft is going to increase it's quality? No I do not. Joel, You may be right. Calling out IANA considerations doesn't seem to have made the IETF any smarter on the shared ISP IPv4 space. And I have no idea if calling out security implications has helped reduce security-related design flaws. On the other hand, calling out ops issues in RFCs is a modest reform that at worst shouldn't hurt anything. That beats my next best idea: asking the ops area to schedule its meetings with the various NOG meetings instead of with the rest of the IETF so that the attendance is ops who dabble in development instead of developers who dabble in ops. You disagree? What are your thoughts on fixing the problem? Regards, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
In a message written on Sun, Jul 10, 2011 at 06:16:09PM +0200, Jeroen Massar wrote: Eh ANYBODY, including you, can sign up to the IETF mailing lists and participate there, just like a couple of folks from NANOG are already doing. The way the IETF and the operator community interact is badly broken. The IETF does not want operators in many steps of the process. If you try to bring up operational concerns in early protocol development for example you'll often get a we'll look at that later response, which in many cases is right. Sometimes you just have to play with something before you worry about the operational details. It also does not help that many operational types are not hardcore programmers, and can't play in the sandbox during the major development cycles. But this shuts out operators and discourages them from participating when they are needed, which is at the idea phase and towards the end of development. If the IETF really wanted to get useful operator impact, they would slightly modify their process. On the front end there would be a more clear way for operational types to add to the To-Do list stuff we really need to make the Internet work better. Then, some sausage would be made largely without operator involvement (but hey, if you want to participate no exclusions), and then when developmen is about 80-90% done there would be an operational testing and comment period. Operators would be actively brought back in the process to test some small scale deployments and provide feedback of operational concerns that might lead to some tweaks, and then boom, out the door it goes. I suspect this would both increase operator participation by a few orders of magnitude, and also keep the operators from annoying the developers so much when they are in trying things out mode. -- Leo Bicknell - bickn...@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ pgpGFY3IVVPMN.pgp Description: PGP signature
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Mon, Jul 11, 2011 at 3:18 PM, William Herrin b...@herrin.us wrote: On the other hand, calling out ops issues in RFCs is a modest reform that at worst shouldn't hurt anything. That beats my next best idea: I think if this were done, some guy like me would spend endless hours arguing with others about what should and should not be documented in this proposed section, without it actually benefiting the process or the improving the underlying protocol function / specification. Let me give you an example: BGP Messages, which are up to 4KB, need to be expanded to support future features like as-path signing. Randy Bush proposes to extend them to 65,535 octets, the maximum size without significantly changing the message header. This raises a few concerns which I label as operational, for example, off-by-one bugs in code can fail to be detected by a neighboring BGP speaker in some circumstances, because an age-old (since BGP 1) idiot check in the protocol is being silently removed. If you ask me, that is operational and belongs in such a section. I'm sure others will disagree. So we would have a bunch of arguing over whether or not to call this out specifically. Another person believes that expanding the message will affect some vendors' custom TCP stacks, due to window size considerations. I might think that is a developer problem and the affected vendors should fix their crappy TCP implementations, but it might produce unusual stalling problems, etc. which operators have to troubleshoot. Is that an operational issue? Should it be documented? There can be many operational concerns when creating or modifying a protocol specification, and every person won't agree on what belongs and what doesn't. However, I do not think the requirement to document them will improve the process or the protocols. It will only add work. Besides, you want IETF people who are claimed not to understand operational problems to figure them out and document them in the RFCs? I do not think this will be helpful. More hands-on operators participating in their process is what is needed. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Mon, Jul 11, 2011 at 3:35 PM, Leo Bicknell bickn...@ufp.org wrote: The IETF does not want operators in many steps of the process. If you try to bring up operational concerns in early protocol development for example you'll often get a we'll look at that later response, which in many cases is right. Sometimes you just have to play with something before you worry about the operational details. It also I really don't understand why that is right / good. People get personally invested in their project / spec, and not only that, vendor people get their company's time and money invested in proof-of-concept. The longer something goes on with what may be serious design flaws, the harder it is to get them fixed, simply because of momentum. Wouldn't it be nice if we could change the way that next-header works in IPv6 now? Or get rid of SLAAC and erase the RFCs recommending /80 and /64 from history? -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Jul 11, 2011, at 12:18 PM, William Herrin wrote: On Mon, Jul 11, 2011 at 11:20 AM, Joel Jaeggli joe...@bogus.com wrote: On Jul 11, 2011, at 8:13 AM, William Herrin wrote: Today's RFC candidates are required to call out IANA considerations and security considerations in special sections. They do so because each of these areas has landmines that the majority of working groups are ill equipped to consider on their own. There should be an operations callout as well -- a section where proposed operations defaults (as well as statics for which a solid case can be made for an operations tunable) are extracted from the thick of it and offered for operator scrutiny prior to publication of the RFC. Do you find this adjustment objectionable? Do I think that adding yet another required section to an internet draft is going to increase it's quality? No I do not. Joel, You may be right. Calling out IANA considerations doesn't seem to have made the IETF any smarter on the shared ISP IPv4 space. And I have no idea if calling out security implications has helped reduce security-related design flaws. On the other hand, calling out ops issues in RFCs is a modest reform that at worst shouldn't hurt anything. To my mind, one of the really good criterion for an operational considerations document is some actual experience running it. That beats my next best idea: asking the ops area to schedule its meetings with the various NOG meetings instead of with the rest of the IETF so that the attendance is ops who dabble in development instead of developers who dabble in ops. The OPS area works on OPS and Management. Protocol development of the sort you're describing generally occurs in working-groups chartered in the Internet or Routing areas... At least one of the ops chairs are on this list attends nanog regularly etc. Participants, especially those that actually do the work are the important part as far as I'm concerned. Rough consensus is an ugly an imperfect business, it should be recognized that not everyone is going to come away from every exchange with what they want. You disagree? What are your thoughts on fixing the problem? I'm not sure that we agree on the dimensions of the problem. on the question of ipv6 is broken: * You're going to have to cope with what you have and can squeeze out of vendors in the near term. implmentors don't change that fast. * People have to show up with the problem statement and stick around to do the work * the outcomes are not always pretty. I hope that my time is productively employed. http://tools.ietf.org/html/draft-gashinsky-v6nd-enhance-00 Regards, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Jul 11, 2011, at 12:57 PM, Jeff Wheeler wrote: On Mon, Jul 11, 2011 at 3:35 PM, Leo Bicknell bickn...@ufp.org wrote: The IETF does not want operators in many steps of the process. If you try to bring up operational concerns in early protocol development for example you'll often get a we'll look at that later response, which in many cases is right. Sometimes you just have to play with something before you worry about the operational details. It also I really don't understand why that is right / good. People get personally invested in their project / spec, and not only that, vendor people get their company's time and money invested in proof-of-concept. The longer something goes on with what may be serious design flaws, the harder it is to get them fixed, simply because of momentum. Wouldn't it be nice if we could change the way that next-header works in IPv6 now? Or get rid of SLAAC and erase the RFCs recommending /80 and /64 from history? No... I like SLAAC and find it useful in a number of places. What's wrong with /64? Yes, we need better DOS protection in switches and routers to accommodate some of the realities of those decisions, but, that's not to say that SLAAC or /64s are bad. They're fine ideas with proper protections. I'm not sure about the /80 reference as I haven't encountered that recommendation outside of some perverse ideas about point-to-point links. Owen
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Mon, Jul 11, 2011 at 3:41 PM, Jeff Wheeler j...@inconcepts.biz wrote: On Mon, Jul 11, 2011 at 3:18 PM, William Herrin b...@herrin.us wrote: On the other hand, calling out ops issues in RFCs is a modest reform that at worst shouldn't hurt anything. That beats my next best idea: I think if this were done, some guy like me would spend endless hours arguing with others about what should and should not be documented in this proposed section, without it actually benefiting the process or the improving the underlying protocol function / specification. Let me give you an example: BGP Messages, which are up to 4KB, need to be expanded to support future features like as-path signing. Randy Bush proposes to extend them to 65,535 octets, the maximum size without significantly changing the message header. This raises a few concerns which I label as operational, for example, off-by-one bugs in code can fail to be detected by a neighboring BGP speaker in some circumstances, because an age-old (since BGP 1) idiot check in the protocol is being silently removed. If you ask me, that is operational and belongs in such a section. Hi Jeff, Thanks for your thoughtful response. Question: It seems to me like figuring out what is or isn't a security issue to be called out has exactly the same pitfalls. How do you deal with it? Besides, you want IETF people who are claimed not to understand operational problems to figure them out and document them in the RFCs? I do not think this will be helpful. More hands-on operators participating in their process is what is needed. You're an IETF person trying to figure out what is or isn't an operations issue so that you can call it out. How might you go about figuring that out? Personally, I might ask a few ops: Lend me your ear for three minutes to tell you about what I'm working on. Now that that I've given you the pitch, is this something you'd like to control in a configuration or is it something you want to -just work-? Control = operations issue. Just work = not an operations issue. Regards, Bill -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
You disagree? What are your thoughts on fixing the problem? I'm not sure that we agree on the dimensions of the problem. on the question of ipv6 is broken: * You're going to have to cope with what you have and can squeeze out of vendors in the near term. implmentors don't change that fast. * People have to show up with the problem statement and stick around to do the work * the outcomes are not always pretty. I don't think that has anything to do with the problem Bill is trying to address. While it is the topic that started this thread, the problem that I think Bill is trying to address and which I agree needs to be addressed is that IETF standards are developed with what has become increasingly obvious as insufficient operator input. Yes, operators are partially to blame in that decisions are made by those that show up and operators have a hard time showing up to the IETF process for a variety of reasons that are mostly related to the realities of running day to day operations and not realy something the IETF can easily address. However, part of the problem also relates to ways in which the IETF is particularly difficult for operators to credibly participate. (the amount of ego and religion in some of the working groups, the need for a thick skin if you want to make a statement that goes counter to the current dogma, the time-consuming nature of meaningful participation, etc.). I don't pretend to have answers to all of these problems, but, I think there first needs to be recognition and consensus that the lack of operator input into the IETF is becoming increasingly problematic and is impacting the ability to deploy what is developed by the IETF. Owen
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
Once upon a time, there was only the IETF, then NOGs came and standards became sloppy
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Mon, Jul 11, 2011 at 5:12 PM, Owen DeLong o...@delong.com wrote: No... I like SLAAC and find it useful in a number of places. What's wrong with /64? Yes, we need better DOS protection in switches and routers See my slides http://inconcepts.biz/~jsw/IPv6_NDP_Exhaustion.pdf for why no vendor's implementation is effective DOS protection today and how much complexity is involved in doing it correctly, which requires not only knobs on routers, but also on layer-2 access switches, which is not easy to implement. It's a whole lot smarter to just configure a smaller network when that is practical. In fact, that advice should be the standard. I really don't understand why we need SLAAC. I believe it is a relic of a mindset when a DHCP client might have been hard to implement cost-effectively in a really light-weight client device (coffee pot? wrist-watch?) Or when running a DHCP server was some big undertaking that couldn't be made not only obvious, but transparent, to SOHO users buying any $99 CPE. I do understand why SLAAC needs /64. Okay, so configure /64 on those networks where SLAAC is utilized. Otherwise, do something else. Pretty simple! Again, please see my slides. to accommodate some of the realities of those decisions, but, that's not to say that SLAAC or /64s are bad. They're fine ideas with proper protections. The proper protections are kinda hard to do if you have relatively dumb layer-2 access switches. It is a lot harder than RA Guard, and we aren't ever likely to see that feature on a large base of installed legacy switches, like Cisco 2950. Replacing those will be expensive. We can't replace them yet anyway because similar switches (price) today still do not have RA Guard, let alone any knobs to defend against neighbor table churn, etc. I'm not sure if they ever will have the later. I'm not sure about the /80 reference as I haven't encountered that recommendation outside of some perverse ideas about point-to-point links. This is because you didn't follow IPv6 progress until somewhat recently, and you are not aware that the original suggestion for prefix length was 80 bits, leaving just 48 bits for the host portion of the address. This was later revised. It helps to know a bit of the history that got us to where we are now. It was originally hoped, by some, that we may not even need NDP because the layer-2 adjacency would always be encoded in the end of the layer-3 address. Some people still think vendors may get us to that point with configuration knobs. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Mon, Jul 11, 2011 at 3:35 PM, Leo Bicknell bickn...@ufp.org wrote: If the IETF really wanted to get useful operator impact, they would slightly modify their process. On the front end there would be a more clear way for operational types to add to the To-Do list stuff we really need to make the Internet work better. Hi Leo, That's an interesting idea, but how does it work? As it stands, I can join a working group mailing list and submit an I-D any time I feel like it. There is almost zero barrier to entry. And I can take it to any step short of the final publication track through the simple expedient of working on it myself. How does the to-do list differ from this? Does it provide some kind of push counter to the folks who hum against publication? How's it work? Then, some sausage would be made largely without operator involvement (but hey, if you want to participate no exclusions), and then when developmen is about 80-90% done there would be an operational testing and comment period. That's another interesting idea. Would you mind gaming it out for me? Use RFC 3484. You have I-D-v6-address-selection 90% ready for publication as RFC 3484. Now what? In their prescience, the operator feedback is going to be with multiple addresses on a server representing various Internet paths with various reliabilities, we need a way to communicate to the client which addresses to prefer based on our expert knowledge of the reliability of our local network. What elicits that feedback? What do the authors of I-D-v6-address-selection do with the feedback prior to publication? Thanks, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Mon, Jul 11, 2011 at 5:10 PM, Joel Jaeggli joe...@bogus.com wrote: On Jul 11, 2011, at 12:18 PM, William Herrin wrote: On Mon, Jul 11, 2011 at 11:20 AM, Joel Jaeggli joe...@bogus.com wrote: On Jul 11, 2011, at 8:13 AM, William Herrin wrote: Today's RFC candidates are required to call out IANA considerations and security considerations in special sections. They do so because each of these areas has landmines that the majority of working groups are ill equipped to consider on their own. There should be an operations callout as well -- a section where proposed operations defaults (as well as statics for which a solid case can be made for an operations tunable) are extracted from the thick of it and offered for operator scrutiny prior to publication of the RFC. Do you find this adjustment objectionable? Do I think that adding yet another required section to an internet draft is going to increase it's quality? No I do not. Joel, You may be right. Calling out IANA considerations doesn't seem to have made the IETF any smarter on the shared ISP IPv4 space. And I have no idea if calling out security implications has helped reduce security-related design flaws. On the other hand, calling out ops issues in RFCs is a modest reform that at worst shouldn't hurt anything. To my mind, one of the really good criterion for an operational considerations document is some actual experience running it. Hi Joel, I'm not looking for anything that sophisticated. I just want a list of These are the things that can be tuned at an operations level (plus the defaults) and these are the things that can't be tuned but someone in the discussion thought a reasonable person might want them to be. The idea is that an ops guy should be able to read the three-paragraph intro, jump to the list of tunables and then be able to offer feedback along the lines of, Whoa! Of course X should be tunable, are you kidding? Here's a rough description of where I want to configure it. and I'm never going to alter Y. You can make it configurable, but I'd just as soon deal with everybody having the same value of Y. Heck, make it multiple choice. 1 is no chance I'll ever want to change this value and 5 is I'll want to set this value case by case. That beats my next best idea: asking the ops area to schedule its meetings with the various NOG meetings instead of with the rest of the IETF so that the attendance is ops who dabble in development instead of developers who dabble in ops. The OPS area works on OPS and Management. Protocol development of the sort you're describing generally occurs in working-groups chartered in the Internet or Routing areas... A moment ago you said, the ops area reviews basically every draft in front of the IESG. Participants, especially those that actually do the work are the important part as far as I'm concerned. I don't disagree. But producing flawed standards does nobody any good, least of all the folks who poured their hearts and souls into making them. Rough consensus is an ugly an imperfect business, it should be recognized that not everyone is going to come away from every exchange with what they want. Which if you were talking about a rough consensus of operations folk addressing operations issues would be just fine. This is basically what happens at the address registries like ARIN and it more or less works. That's broken in the IETF. The ops folks aren't there for the consensus checks. As a consequence, ops issues are being decided not with -rough- consensus but with -false- consensus. False consensus falls apart when you try to bring the excluded folks back to the party, as you must in the operators' case with any standard the IETF produces. You disagree? What are your thoughts on fixing the problem? I'm not sure that we agree on the dimensions of the problem. on the question of ipv6 is broken: * You're going to have to cope with what you have and can squeeze out of vendors in the near term. implmentors don't change that fast. * People have to show up with the problem statement and stick around to do the work * the outcomes are not always pretty. V6 poses some difficult challenges and you're right that in the short term we're basically stuck with what is and have to make the best of it. But V6 isn't my focus in this thread. The ops are sufficiently irate at this point that they'll keep pounding on the WG's and the vendors until fixes happen. My focus in this thread is this: how do we help the next teams avoid the discourtesy and the smackdown that the v6 teams are getting for not adequately recognizing the ops' issues. These guys should have been heroes but instead they screwed the pooch and everybody's paying for it. How do we fix the systemic problems so that next time they are heroes? Regards, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web:
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Mon, 11 Jul 2011, William Herrin wrote: On Mon, Jul 11, 2011 at 5:10 PM, Joel Jaeggli joe...@bogus.com wrote: On Jul 11, 2011, at 12:18 PM, William Herrin wrote: snip My focus in this thread is this: how do we help the next teams avoid the discourtesy and the smackdown that the v6 teams are getting for not adequately recognizing the ops' issues. These guys should have been heroes but instead they screwed the pooch and everybody's paying for it. How do we fix the systemic problems so that next time they are heroes? get to the party early: http://trac.tools.ietf.org/bof/trac/ and stay late... lots of new work inbound. - Lucy Regards, Bill Herrin
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Jul 11, 2011, at 3:37 PM, William Herrin wrote: On Mon, Jul 11, 2011 at 5:10 PM, Joel Jaeggli joe...@bogus.com wrote: On Jul 11, 2011, at 12:18 PM, William Herrin wrote: On Mon, Jul 11, 2011 at 11:20 AM, Joel Jaeggli joe...@bogus.com wrote: On Jul 11, 2011, at 8:13 AM, William Herrin wrote: Today's RFC candidates are required to call out IANA considerations and security considerations in special sections. They do so because each of these areas has landmines that the majority of working groups are ill equipped to consider on their own. There should be an operations callout as well -- a section where proposed operations defaults (as well as statics for which a solid case can be made for an operations tunable) are extracted from the thick of it and offered for operator scrutiny prior to publication of the RFC. Do you find this adjustment objectionable? Do I think that adding yet another required section to an internet draft is going to increase it's quality? No I do not. Joel, You may be right. Calling out IANA considerations doesn't seem to have made the IETF any smarter on the shared ISP IPv4 space. And I have no idea if calling out security implications has helped reduce security-related design flaws. On the other hand, calling out ops issues in RFCs is a modest reform that at worst shouldn't hurt anything. To my mind, one of the really good criterion for an operational considerations document is some actual experience running it. Hi Joel, I'm not looking for anything that sophisticated. I just want a list of These are the things that can be tuned at an operations level (plus the defaults) and these are the things that can't be tuned but someone in the discussion thought a reasonable person might want them to be. The idea is that an ops guy should be able to read the three-paragraph intro, jump to the list of tunables and then be able to offer feedback along the lines of, Whoa! Of course X should be tunable, are you kidding? Here's a rough description of where I want to configure it. and I'm never going to alter Y. You can make it configurable, but I'd just as soon deal with everybody having the same value of Y. Heck, make it multiple choice. 1 is no chance I'll ever want to change this value and 5 is I'll want to set this value case by case. That beats my next best idea: asking the ops area to schedule its meetings with the various NOG meetings instead of with the rest of the IETF so that the attendance is ops who dabble in development instead of developers who dabble in ops. The OPS area works on OPS and Management. Protocol development of the sort you're describing generally occurs in working-groups chartered in the Internet or Routing areas... A moment ago you said, the ops area reviews basically every draft in front of the IESG. I said there is an ops directorate that reviews basically every draft in front of the iesg... much like their are genart reviews, security reviews iana reviews etc. The principle work on a draft by the time that occurs is already done unless the iesg returns the draft to the work group. SNIP Participants, especially those that actually do the work are the important part as far as I'm concerned. My focus in this thread is this: how do we help the next teams avoid the discourtesy and the smackdown that the v6 teams are getting for not adequately recognizing the ops' issues. These guys should have been heroes but instead they screwed the pooch and everybody's paying for it. How do we fix the systemic problems so that next time they are heroes? IPNG was a long time ago. things have changed and will continue to change because standards are written by humans and cemented with consensus which is imperment and changable. Rational changes, Requirements change and things should evolve, that's not failure it's healthy. Regards, Bill Herrin
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Mon, Jul 11, 2011 at 5:03 PM, Jeff Wheeler j...@inconcepts.biz wrote: On Mon, Jul 11, 2011 at 5:12 PM, Owen DeLong o...@delong.com wrote: No... I like SLAAC and find it useful in a number of places. What's wrong with /64? Yes, we need better DOS protection in switches and routers See my slides http://inconcepts.biz/~jsw/IPv6_NDP_Exhaustion.pdf for why no vendor's implementation is effective DOS protection today and how much complexity is involved in doing it correctly, which requires [snip] If every vendor's implementation is vulnerable to a NDP Exhaustion vulnerability, how come the behavior of specific routers has not been documented specifically? If zero devices are not vulnerable, you came to this conclusion because you tested every single implementation against IPv6 NDP DoS, or? How come there are no security advisories. What's the CWE or CVE number for this vulnerability? I'm not denying the that NDP overflow might be a DoS issue for all IPv6 routers, but I haven't seen any specific documentation from vendors or security researchers about specific DoS conditions that can be caused by NDP overflow on particular devices It would be useful to at least have the risk properly described, in terms of what kind of DoS condition could arise on specific implementations. Regards, -- -JH
NDP DoS attack (was Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?))
On Mon, 2011-07-11 at 18:48 -0500, Jimmy Hess wrote: It would be useful to at least have the risk properly described, in terms of what kind of DoS condition could arise on specific implementations. RFC3756 IPv6 Neighbor Discovery (ND) Trust Models and Threats Section 4.3.2 In this attack, the attacking node begins fabricating addresses with the subnet prefix and continuously sending packets to them. The last hop router is obligated to resolve these addresses by sending neighbor solicitation packets. A legitimate host attempting to enter the network may not be able to obtain Neighbor Discovery service from the last hop router as it will be already busy with sending other solicitations. This DoS attack is different from the others in that the attacker may be off-link. The resource being attacked in this case is the conceptual neighbor cache, which will be filled with attempts to resolve IPv6 addresses having a valid prefix but invalid suffix. This is a DoS attack. The above RFC and RFC3971 (SEND) both have good descriptions of a BUNCH of possible attacks. RFC3971 is a bit dismissive IMHO of this particular attack. I realise this is not specific implementations as you requested, but it seems to me that the problem is generic enough not to require that. The attack is made possible by the design of the protocol, not any failing of specific implementations. Specific implementations need to describe what they've done about it (mitigation or prevention). Regards, K. -- ~~~ Karl Auer (ka...@biplane.com.au) +61-2-64957160 (h) http://www.biplane.com.au/kauer/ +61-428-957160 (mob) GPG fingerprint: DA41 51B1 1481 16E1 F7E2 B2E9 3007 14ED 5736 F687 Old fingerprint: B386 7819 B227 2961 8301 C5A9 2EBC 754B CD97 0156 signature.asc Description: This is a digitally signed message part
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Mon, Jul 11, 2011 at 7:48 PM, Jimmy Hess mysi...@gmail.com wrote: If every vendor's implementation is vulnerable to a NDP Exhaustion vulnerability, how come the behavior of specific routers has not been documented specifically? Well, I am in the business of knowing the behavior of kit being considered by my clients for their applications. Every box breaks when tested, period. I imagine you have tested zero, thus you have no data of your own to go on. No vendors are rushing to spend money on independent testing laboratories to produce reports about this, because they pretty much all know their boxes will break (or are not even aware of the potential problem, in the case of a few scary vendors.) If zero devices are not vulnerable, you came to this conclusion because you tested every single implementation against IPv6 NDP DoS, or? Although I have tested many routers to verify my thinking, if you actually read the slides and understand how routers work, you too will know that every router is vulnerable. If you don't know, you don't understand how routers work. It's that simple. How come there are no security advisories. What's the CWE or CVE number for this vulnerability? Again, no one is interested in this problem yet because vendors really don't want their customers to demand more knobs. Cisco is the only vendor who has done anything at all. If you read about their knob, you immediately realize that it is a knob to control the failure mode of the box, not to fix anything. Why? It can't be fixed without not using /64 (or similar) or going to the extreme lengths I outline in those slides. It would be useful to at least have the risk properly described, in terms of what kind of DoS condition could arise on specific implementations. Let's take 6500/SUP720 for example. On this platform, a policer is shared between the need to resolve ARP entries and ND table entries. If you attack a dual-stack SUP720 box it will break not only IPv6 neighbor resolution, but also IPv4 neighbor resolution. This is pretty much the worst-case scenario because not only will your IPv6 break, which may annoy customers but not be a disaster; it will also break mission-critical IPv4. That's bad. Routing-protocol adjacencies can be affected, disabling not just some hosts downstream of the box, but also its upstream connectivity. It doesn't get any worse than that. You are right to question my statements. I'm not an independent lab doing professional tests and showing the environment and conditions of how you can reproduce the results. I'm just a guy helping my clients decide what kit to buy, and how they should configure their networks. The only reason I have bothered to produce slides is because we are at a point where we have end-customers questioning our reluctance to provision /64 networks for mixed-use data-center LANs, and until vendors actually do something to address this, or the standard changes, I need to increase awareness of this problem so I am not forced to deploy a broken design on my own networks the way a lot of other clueless people are. Again, this is only hard to understand (or accept) if you don't know how your routers work. * why do you think there is an ARP and ND table? * why do you think there are policers to protect the CPU from excessive ARP/ND punts or traffic? * do you even know the limit of your boxes' ARP / ND tables? Do you realize that limit is a tiny fraction of one /64? * do you understand what happens when your ARP/ND policers are reached? * did you think about the impact on neighboring routers and protocol next-hops, not just servers? * did you every try to deploy a /16 on a flat LAN with a lot of hosts and see what happens? Doesn't work too well. A v6 /64 is 281 trillion times bigger than a v4 /16. There's no big leap of logic here as to why one rogue machine could break your LAN. There is no router which is not vulnerable to this. If you don't believe me, read the Cisco documentation on their knob limiting ND entries per interface, after which there may be service impact on that interface. That's the best anyone is doing right now. Of course, vendors understand that we, as customers, can configure a subnet smaller than /64. They are leaving us open to link-local issues right now even with a smaller global subnet size, but at least that cannot be exploited from the Internet. And as it happens, exactly the same features / knobs are needed to fix both problems with /64, and with link-local neighbor learning. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On 07/10/2011 12:45, Owen DeLong wrote: On Jul 10, 2011, at 9:16 AM, Jeroen Massar wrote: On 2011-07-10 17:56 , David Miller wrote: [..] +1 The lack of will on the part of the IETF to attract input from and involve operators in their processes (which I would posit is a critical element in the process). Discussing how the IETF should fix itself is both fruitless, and off topic for this list. However ... While this is true, there are a couple of factors that make it more difficult than it would appear on the surface. Number one: Participating effectively in IETF is a rather time-consuming process. While a lot of engineers and developers may have IETF effort as a primary part of their job function and/or get their employer to let them spend time on it, operators are often too busy keeping what they already have running and it can be _VERY_ difficult to get management to support the idea of investing time in things like IETF which are not seen by management as having direct operational impact. NANOG is about the limit of their vision on such things and even that is not well supported in a lot of organizations. Number two: While anyone can participate, approaching IETF as an operator requires a rather thick skin, or, at least it did the last couple of times I attempted to participate. I've watched a few times where operators were shouted down by purists and religion over basic real-world operational concerns. It seems to be a relatively routine practice and does not lead to operators wanting to come back to an environment where they feel unwelcome. What you're saying is absolutely right (unfortunately), but the answer is that operators need to suck it up and get involved. The problem will not fix itself if we don't. The good news is that in many areas (at least, the areas that I participate in) there is starting to be a lot more sympathy toward operational concerns/realities, and real progress is being made. Yes, it's slow, arduous, and often frustrating. (How's that for a sales pitch?) But there is literally no other solution to improving the situation that for the people that care to get involved in helping to fix it. For those interested in IPv6 I highly recommend subscribing to the the 6man and v6ops lists, listen in on the conversations for a while, and then chime in when you feel comfortable. Treat those on the list with the same courtesy and respect that you'd like to be treated with, and way more often than not it will bear fruit. hth, Doug -- Nothin' ever doesn't change, but nothin' changes much. -- OK Go Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
Well, you work at Zynga, a company which makes facebook games. Before that you worked at Nokia, company which makes phones but doesn't run phone networks. Before that it was Check Point Software, a company which makes firewalls but doesn't run networks. And before that it was the University of Oregon. down to the ad homina pretty quickly. congrats. fyi, what joel does/did at those companies is/was build and run networks and data centers. next ad hominem attack? randy
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
My focus in this thread is this: how do we help the next teams avoid the discourtesy and the smackdown that the v6 teams are getting for not adequately recognizing the ops' issues. These guys should have been heroes but instead they screwed the pooch and everybody's paying for it. How do we fix the systemic problems so that next time they are heroes? get to the party early: http://trac.tools.ietf.org/bof/trac/ and stay late... lots of new work inbound. aka, suit up and show up. talk is cheap. randy
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
I said there is an ops directorate that reviews basically every draft in front of the iesg. and this directorate is a group of actual operators? randy
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Jul 10, 2011, at 12:16 PM, Jeroen Massar wrote: On 2011-07-10 17:56 , David Miller wrote: [..] +1 The lack of will on the part of the IETF to attract input from and involve operators in their processes (which I would posit is a critical element in the process). Eh ANYBODY, including you, can sign up to the IETF mailing lists and participate there, just like a couple of folks from NANOG are already doing. You are on NANOG out of your own free will, the same applies to the IETF. If you don't participate here your voice is not heard either, just like at the IETF. Peeking at the i...@ietf.org member list, I don't see your name there. You can signup here: https://www.ietf.org/mailman/listinfo/ipv6 Thanks, Jeroen. For IPv6 functionality, I'd suggest i...@ietf.org (https://www.ietf.org/mailman/listinfo/ipv6). For IPv6 operational issues, I'd suggest v6...@ietf.org (https://www.ietf.org/mailman/listinfo/v6ops). For security-related issues, you might also look into op...@ietf.org (https://www.ietf.org/mailman/listinfo/opsec). On Jul 10, 2011, at 3:45 PM, Owen DeLong wrote: Number two: While anyone can participate, approaching IETF as an operator requires a rather thick skin, or, at least it did the last couple of times I attempted to participate. I've watched a few times where operators were shouted down by purists and religion over basic real-world operational concerns. That goes both ways. I periodically see dismissive statements about the IETF on operational lists, and dismissive statements about operators on IETF lists. I would classify David's comment as dismissive, the kind of comment that causes IETF folks to not participate in operational meetings or lists, and the kind of comment cited by operational folks such as you as reasons to leave IETF meetings and lists. Such comments tend to come from a small set of individuals on each side. If such comments bother you, feel free to block the in-duh-viduals that send them. Personally, I try to listen to them; they are often telling me something I need to hear but don't want to.
Re: Why is IPv6 broken?
On Sat, Jul 9, 2011 at 5:25 PM, Bob Network network...@hotmail.com wrote: Why is IPv6 broken? You should have titled your thread, my own personal rant about Hurricane Electric's IPv6 strategy. You may also have left out the dodgy explanation of peering policies and technicalities, since these issues have been remarkably static since about 1996. The names of the networks change, but the song remains the same. This is not a novel subject on this mailing list. In fact, there have been a number of threads discussing HE's practices lately. If you are so interested in them, I suggest you review the list archive. There are quite a few serious, unresolved technical problems with IPv6 adoption besides a few networks playing chicken with their collective customer-bases. The lack of will on the part of vendors and operators to participate in the IETF process, and make necessary and/or beneficial changes to the IPv6 standards, has left us in a situation where IPv6 implementation produces networks which are vulnerable to trivial DoS attacks and network intrusions. The lack of will on the part of access providers to insist on functioning IPv6 support on CPE and BRAS platforms has even mid-sized ISPs facing nine-figure (as in, hundred-million-dollars) expenses to forklift-upgrade their access networks and end-user equipment, at a time when IPv6 seems to be the only way to continue growing the Internet. The lack of will on the part of major transit networks, including Savvis, to deploy IPv6 capabilities to their customers, means that customers caught in multi-year contracts may have no option for native connectivity. Cogent's policy of requiring a new contract, and from what I am still being told by some European customers, new money, from customers in exchange for provisioning IPv6 on existing circuits, means a simple technical project gets caught up in the complexities of budgeting and contract execution. If you believe that the most serious problem facing IPv6 adoption is that HE / Level3 / Cogent don't carry a full table, you are living in a fantasy world. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: Why is IPv6 broken?
On 7/10/2011 10:14 AM, Jeff Wheeler wrote: On Sat, Jul 9, 2011 at 5:25 PM, Bob Networknetwork...@hotmail.com wrote: Why is IPv6 broken? You should have titled your thread, my own personal rant about Hurricane Electric's IPv6 strategy. You may also have left out the dodgy explanation of peering policies and technicalities, since these issues have been remarkably static since about 1996. The names of the networks change, but the song remains the same. This is not a novel subject on this mailing list. In fact, there have been a number of threads discussing HE's practices lately. If you are so interested in them, I suggest you review the list archive. There are quite a few serious, unresolved technical problems with IPv6 adoption besides a few networks playing chicken with their collective customer-bases. The lack of will on the part of vendors and operators to participate in the IETF process, and make necessary and/or beneficial changes to the IPv6 standards, has left us in a situation where IPv6 implementation produces networks which are vulnerable to trivial DoS attacks and network intrusions. The lack of will on the part of access providers to insist on functioning IPv6 support on CPE and BRAS platforms has even mid-sized ISPs facing nine-figure (as in, hundred-million-dollars) expenses to forklift-upgrade their access networks and end-user equipment, at a time when IPv6 seems to be the only way to continue growing the Internet. The lack of will on the part of major transit networks, including Savvis, to deploy IPv6 capabilities to their customers, means that customers caught in multi-year contracts may have no option for native connectivity. Cogent's policy of requiring a new contract, and from what I am still being told by some European customers, new money, from customers in exchange for provisioning IPv6 on existing circuits, means a simple technical project gets caught up in the complexities of budgeting and contract execution. +1 The lack of will on the part of the IETF to attract input from and involve operators in their processes (which I would posit is a critical element in the process). And the lack of will/fore site on the part of the IETF to respond to input from operators that they have received. If fingers can be pointed at both sides, i.e. operators and IETF, then both sides are to blame. The IETF only has value if they are publishing standards that work properly in the real world. If the implementers of these standards say that they are broken, then the IETF has failed to provide value. If you believe that the most serious problem facing IPv6 adoption is that HE / Level3 / Cogent don't carry a full table, you are living in a fantasy world. +1 -DMM
Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On 2011-07-10 17:56 , David Miller wrote: [..] +1 The lack of will on the part of the IETF to attract input from and involve operators in their processes (which I would posit is a critical element in the process). Eh ANYBODY, including you, can sign up to the IETF mailing lists and participate there, just like a couple of folks from NANOG are already doing. You are on NANOG out of your own free will, the same applies to the IETF. If you don't participate here your voice is not heard either, just like at the IETF. Peeking at the i...@ietf.org member list, I don't see your name there. You can signup here: https://www.ietf.org/mailman/listinfo/ipv6 Greets, Jeroen
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On 7/10/2011 12:16 PM, Jeroen Massar wrote: On 2011-07-10 17:56 , David Miller wrote: [..] +1 The lack of will on the part of the IETF to attract input from and involve operators in their processes (which I would posit is a critical element in the process). Eh ANYBODY, including you, can sign up to the IETF mailing lists and participate there, just like a couple of folks from NANOG are already doing. You are on NANOG out of your own free will, the same applies to the IETF. If you don't participate here your voice is not heard either, just like at the IETF. True, anyone can participate in the IETF processes. However, if key players do not participate, then something is broken. I will take my lumps for not participating. My point was - If fingers can be pointed at both sides, i.e. operators and IETF, then both sides are to blame. In the corporate world, if I were contemplating changing the framework of a system, then I would need to get buy in / agreement from the stakeholders of that system. If I was going to change the framework behind an HR system, then the HR managers and HR systems experts would all have to agree to the change. If I changed the framework and broke all of the HR systems and then told my boss that I scheduled a meeting and nobody from HR showed up and therefore I used that as agreement in their absence, then I would get fired. Yes, I understand that corporate environments are very different from the IETF environment, but there are perhaps some lessons to learn from the corporate environment. Most RFCs operate within a meritocracy. A standard can be proposed for Example Protocol v10 and if nobody likes it outside of the IETF, then it is not implemented by anyone and it eventually dies on the vine. IPv6 is different in that it is the underpinning of every other protocol/standard that will exist on or operate over the internet for the next 20-30 years (probably) We had 10+ years of IPv6 not being implemented by anyone (seriously), yet it didn't die on the vine. Perhaps the process for Example Protocol v10 and the process for IPv6 should be different - given the fundamental difference in their scope. No, we can't change the past. Those who do not learn from history are doomed to repeat it. - Santayana. I would say that many variables that got us to where we are today - which is out of IPv4 addresses and faced with only IPv6, which many believe is fundamentally flawed, as our only way forward - holds some lessons to be learned... but perhaps this is just me - and if so, I apologize for the noise. Peeking at the i...@ietf.org member list, I don't see your name there. You can signup here: https://www.ietf.org/mailman/listinfo/ipv6 Absolutely true, fixed. Greets, Jeroen -DMM
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Sun, Jul 10, 2011 at 1:41 PM, David Miller dmil...@tiggee.com wrote: On 7/10/2011 12:16 PM, Jeroen Massar wrote: You are on NANOG out of your own free will, the same applies to the IETF. If you don't participate here your voice is not heard either, just like at the IETF. True, anyone can participate in the IETF processes. However, if key players do not participate, then something is broken. I will take my lumps for not participating. My point was - If fingers can be pointed at both sides, i.e. operators and IETF, then both sides are to blame. Hi David, This is a process problem, not an individual problem. The IETF is run by volunteers. They volunteer because they find designing protocols to be fun. For the most part, operators are not entertained by designing network protocols. So, for the most part they don't partiticpate. This is not going to change. And it also isn't the problem -- people who enjoy the work tend to do better work. The problem is that the IETF routinely exceeds the scope of designing network protocols. Participants in the working groups take what are fundamentally operations issues unto themselves. They do so knowing they lack adequate participation by network operators. And the process that leads to RFCs offers inadequate checks and balances to mitigate that behavior. Consider, for example, RFC 3484. That's the one that determines how an IPv6 capable host selects which of a group of candidate IPv4 and IPv6 addresses for a particular host name gets priority. How is a server's address priority NOT an issue that should be managed at an operations level by individual server administrators? Yet the working group which produced it came up with a static prioritization that is the root cause of a significant portion of the IPv6 deployment headaches we face. I don't know the whole solution to this problem, but I'm pretty sure I know the first step. Today's RFC candidates are required to call out IANA considerations and security considerations in special sections. They do so because each of these areas has landmines that the majority of working groups are ill equipped to consider on their own. There should be an operations callout as well -- a section where proposed operations defaults (as well as statics for which a solid case can be made for an operations tunable) are extracted from the thick of it and offered for operator scrutiny prior to publication of the RFC. Food for thought. Regards, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: Why is IPv6 broken?
so... how much of the heavy lifting are you personally willing to do and how much are you depending/expecting others to do on your behalf? public whining that the v6 network does not mirror the v4 network is not productive and is not news. of course ymmv. /bill
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Jul 10, 2011, at 9:16 AM, Jeroen Massar wrote: On 2011-07-10 17:56 , David Miller wrote: [..] +1 The lack of will on the part of the IETF to attract input from and involve operators in their processes (which I would posit is a critical element in the process). Eh ANYBODY, including you, can sign up to the IETF mailing lists and participate there, just like a couple of folks from NANOG are already doing. You are on NANOG out of your own free will, the same applies to the IETF. If you don't participate here your voice is not heard either, just like at the IETF. Peeking at the i...@ietf.org member list, I don't see your name there. You can signup here: https://www.ietf.org/mailman/listinfo/ipv6 Greets, Jeroen While this is true, there are a couple of factors that make it more difficult than it would appear on the surface. Number one: Participating effectively in IETF is a rather time-consuming process. While a lot of engineers and developers may have IETF effort as a primary part of their job function and/or get their employer to let them spend time on it, operators are often too busy keeping what they already have running and it can be _VERY_ difficult to get management to support the idea of investing time in things like IETF which are not seen by management as having direct operational impact. NANOG is about the limit of their vision on such things and even that is not well supported in a lot of organizations. Number two: While anyone can participate, approaching IETF as an operator requires a rather thick skin, or, at least it did the last couple of times I attempted to participate. I've watched a few times where operators were shouted down by purists and religion over basic real-world operational concerns. It seems to be a relatively routine practice and does not lead to operators wanting to come back to an environment where they feel unwelcome. Owen
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On 07/10/2011 12:45 PM, Owen DeLong wrote: While this is true, there are a couple of factors that make it more difficult than it would appear on the surface. Number one: Participating effectively in IETF is a rather time-consuming process. While a lot of engineers and developers may have IETF effort as a primary part of their job function and/or get their employer to let them spend time on it, operators are often too busy keeping what they already have running and it can be _VERY_ difficult to get management to support the idea of investing time in things like IETF which are not seen by management as having direct operational impact. NANOG is about the limit of their vision on such things and even that is not well supported in a lot of organizations. Vendors make up the vast bulk of attendance at ietf. And vendors are there for one reason: to make stuff that you'll be paying for. So you pay for it at ietf time, or you pay for it at deployment time. Either way, you'll be paying. Number two: While anyone can participate, approaching IETF as an operator requires a rather thick skin, or, at least it did the last couple of times I attempted to participate. I've watched a few times where operators were shouted down by purists and religion over basic real-world operational concerns. It seems to be a relatively routine practice and does not lead to operators wanting to come back to an environment where they feel unwelcome. If you're trying to imply that operators get singled out, that's not been my experience. You definitely need to have a thick skin given egos and there's definitely a large pool of professional ietf finger waggers, but their holier than thou attitude is spread to all in their path, from what I've seen. I won't speak for every working group, but the ones i've been involved with have been pretty receptive to operator input. Mike
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Jul 10, 2011, at 12:23 PM, William Herrin wrote: On Sun, Jul 10, 2011 at 1:41 PM, David Miller dmil...@tiggee.com wrote: On 7/10/2011 12:16 PM, Jeroen Massar wrote: You are on NANOG out of your own free will, the same applies to the IETF. If you don't participate here your voice is not heard either, just like at the IETF. True, anyone can participate in the IETF processes. However, if key players do not participate, then something is broken. I will take my lumps for not participating. My point was - If fingers can be pointed at both sides, i.e. operators and IETF, then both sides are to blame. Hi David, This is a process problem, not an individual problem. The IETF is run by volunteers. They volunteer because they find designing protocols to be fun. For the most part, operators are not entertained by designing network protocols. So, for the most part they don't partiticpate. This is not going to change. And it also isn't the problem -- people who enjoy the work tend to do better work. The problem is that the IETF routinely exceeds the scope of designing network protocols. Participants in the working groups take what are fundamentally operations issues unto themselves. They do so knowing they lack adequate participation by network operators. And the process that leads to RFCs offers inadequate checks and balances to mitigate that behavior. Consider, for example, RFC 3484. That's the one that determines how an IPv6 capable host selects which of a group of candidate IPv4 and IPv6 addresses for a particular host name gets priority. How is a server's address priority NOT an issue that should be managed at an operations level by individual server administrators? Yet the working group which produced it came up with a static prioritization that is the root cause of a significant portion of the IPv6 deployment headaches we face. 3484 specifies a static default. By definition, defaults in absence of operator configuration kind of have to be static. Having a reasonable and expected set of defaults documented in an RFC provides a known quantity for what operators can/should expect from hosts they have not configured. I see nothing wrong with RFC 3484 other than I would agree that the choices made were suboptimal. Mostly that was based on optimism and a lack of experience available at the time of writing. There is another RFC and there are APIs and most operating systems have configuration mechanisms where an operator CAN set that to something other than the 3484 defaults. I don't know the whole solution to this problem, but I'm pretty sure I know the first step. I don't know what you had in mind, but, reading RFC 5014 would be my suggestion as a good starting point. Today's RFC candidates are required to call out IANA considerations and security considerations in special sections. They do so because each of these areas has landmines that the majority of working groups are ill equipped to consider on their own. There should be an operations callout as well -- a section where proposed operations defaults (as well as statics for which a solid case can be made for an operations tunable) are extracted from the thick of it and offered for operator scrutiny prior to publication of the RFC. I think this would be a good idea, actually. It would probably be more effective to propose it to IETF than to NANOG, however. Owen
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
On Sun, Jul 10, 2011 at 3:45 PM, Owen DeLong o...@delong.com wrote:
Number two: While anyone can participate, approaching IETF as an
operator requires a rather thick skin, or, at least it did the last couple
of times I attempted to participate. I've watched a few times where
I am subscribed to the IDR (BGP, etc.) and LISP lists. These are
populated with different people and cover entirely different topics.
My opinion is the following:
* The IDR list is welcoming of operators, but whether or not your
opinion is listened to or included in the process, I do not know.
Randy Bush, alone, posts more on this list than the sum of all
operators who post in the time I've been reading. I think Randy's
influence is 100% negative, and it concerns me deeply that one
individual has the potential to do so much damage to essential
protocols like BGP. Also, the priorities of this list are pretty
fucked. Inaction within this working group is the reason we still
don't have expanded BGP communities for 32 bit ASNs. The reason for
this is operators aren't participating. The people on the list or the
current participants of the WG should not be blamed. My gripe about
Randy Bush having the potential to do huge damage would not exist if
there were enough people on the list who understand what they're doing
to offer counter-arguments.
operators were shouted down by purists and religion over basic
real-world operational concerns. It seems to be a relatively routine
practice and does not lead to operators wanting to come back to
an environment where they feel unwelcome.
I have found my input on the LISP list completely ignored because, as
you suggest, my concerns are real-world and don't have any impact on
someone's pet project. LISP as it stands today can never work on the
Internet, and regardless of the fine reputations of the people at
Cisco and other organizations who are working on it, they are either
furthering it only because they would rather work on a pet project
than something useful to customers, or because they truly cannot
understand its deep, insurmountable design flaws at Internet-scale.
You would generally hope that someone saying, LISP can't work at
Internet-scale because anyone will be able to trivially DoS any LISP
ITR ('router' for simplicity), but here is a way you can improve it,
well, that remark, input, and person should be taken quite seriously,
their input examined, and other assumptions about the way LISP is
supposed to work ought to be questioned. None of this has happened.
LISP is a pet project to get some people their Ph.D.s and keep some
old guard vendor folks from jumping ship to another company. It is a
shame that the IETF is manipulated to legitimize that kind of thing.
Then again, I could be wrong. Randy Bush could be a genius and LISP
could revolutionize mobility.
--
Jeff S Wheeler j...@inconcepts.biz
Sr Network Operator / Innovative Network Concepts
Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)
The IETF is run by volunteers. They volunteer because they find designing protocols to be fun. For the most part, operators are not entertained by designing network protocols. So, for the most part they don't partiticpate. Randy Bush, Editorial zone: Into the Future with the Internet Vendor Task Force: a very curmudgeonly view, or testing spaghetti, ACM SIGCOMM Computer Communication Review Volume 35 Issue 5, October 2005. http://archive.psg.com/051000.ccr-ivtf.html
Why is IPv6 broken?
Why is IPv6 broken? It's broken, first and foremost, because not all network providers who claim to be tier 1 are tier 1. Even worse, some of these providers run 6to4 relays or providers to home users. A user has no choice which provider is running their 6to4 relay...so, they might end up using a relay that is run by a provider who doesn't peer with their intended destination. I don't think the IETF saw that one coming. But the result is to make 6to4 even more broken. Now, I know some people want 6to4 to die, but while it still exists in some form, user experience is worse than it could be. The temporary fix is for any provider to run their own 6to4 relay for their own customers (assuming that they themselves have full connectivity). Right now, unless you buy transit from multiple tier 1s, and do so with carefully chosen tier ones, you have only part of the IPv6 internet. Many tier 1s are unsuitable even as backup connections, since you still want your backup connection to have access to the whole internet! Good tier 2 providers might be an excellent choice, sine good providers have already done this leg work and can monitor their providers for compliance. A few myths... Routing table size has nothing to do with completeness of routes. Google may be one route, through aggregation. And SmallCo may advertise a large route through one provider, and, due to traffic engineering, a smaller route through a second one - in many cases, anyone that had the large route would be able to contact SmallCo, even without the smaller route being present. So routing table size doesn't work. In addition, some providers aggregate their routing tables to reduce routing load and such. Others intentionally don't or deaggregate it intentionally so that they can brag about having bigger routing tables. What you need to ask is: How many /64s can you get to from your network, and how many of these /64s are reachable from at least one other major provider (you don't care about internal-only networks, after all)? They can give you that information, but many won't want to. It's also not about technical people not getting along. It's about business players trying to make money, but not just that either. It's also about ensuring that providers don't end up assuming more than their share of costs for a link. Just because you have a common peering point doesn't mean that turning peering on would reduce your costs. In some cases it may increase costs tremendously, particularly on your long haul backbone links, because the other party would like to take advantage of an attitude of trust on the internet. That's why we end up with peering policies and contracts. What is the issue? Let's take Hurricane. This is no different than other providers...basically, they want to say, We shouldn't need to pay for IPv6 transit from anyone. This is what Cogent said on IPv4 a few years ago. Google used to say this too for IPv6, not sure if they are still saying it. Basically, We know we're big enough that you won't want to screw your users by not peering with us. A small network couldn't do this tactic - a 100 node network who said to the IPv4 tier 1s: Hey, I'm in the Podunk Internet Exchange, so are you, so I'm going to peer from you so I don't have to buy any bandwidth for my web server (placed in the Podunk exchange). Sure, they would like to - it would save a ton of money if their site got lots of hits. I mean, who wouldn't want free connectivity? In IPv6, we're going through what we settled years ago in IPv4 - who has to pay who to connect. After all, even free peering connections have a cost in manpower, debugging, traffic engineering, documentation, etc. Some players who aren't getting free interconnection to tier 1s in IPv4 want to get it in IPv6. So they've worked to attract lots of users, and done so under the guise of We like IPv6 and want to promote it. Others have not bothered with trying to attract the users, but have said, We're too big for you to not want to give us connectivity for free, since it would piss off your users if you don't (Google did this at one point in the past, may still be doing it). The Google example is basically trying to use a monopoly position to force business decisions. Now, HE, Google, and others would want you to think, Hey, IPv6 is all new, and these $#@! other providers just want to make a buck on something they have no right to. Well, perhaps. But what they aren't saying is, We can turn on BGP for IPv6 on our existing connections to other providers, with no cost to us, and actually have full connectivity. The issue isn't about cost today - nobody is charging extra for IPv6 in addition to IPv4 on a pipe where you already buy IPv4 bandwidth. And Google and HE already buy IPv4 bandwidth. What they are thinking of is the future, 15 years from now, when there is no IPv4 - in that future, IPv6 isn't insignificant
Re: Why is IPv6 broken?
deBunk Where did you get all this from ? There is not even one single reference to a URL, not to be rude but how long did it take you to write this theory ? As for It's broken, first and foremost... They may be a Tier 1 provider of other services and also happen to offer IPv6 at which they are only a Tier 2 or 3 but using the marketing gimics of theyre original Tier 1 status to get acknowledgement. I stopped reading shortly after 'I think' the second paragraph and scanned the rest for URLs that might have made this clear and to the point but did not find any. Heresay. /deBunk On Sat, Jul 09, 2011 at 03:25:27PM -0600, Bob Network wrote: Why is IPv6 broken? It's broken, first and foremost, because not all network providers who claim to be tier 1 are tier 1. Even worse, some of these providers run 6to4 relays or providers to home users. A user has no choice which provider is running their 6to4 relay...so, they might end up using a relay that is run by a provider who doesn't peer with their intended destination. I don't think the IETF saw that one coming. But the result is to make 6to4 even more broken. Now, I know some people want 6to4 to die, but while it still exists in some form, user experience is worse than it could be. The temporary fix is for any provider to run their own 6to4 relay for their own customers (assuming that they themselves have full connectivity). Right now, unless you buy transit from multiple tier 1s, and do so with carefully chosen tier ones, you have only part of the IPv6 internet. Many tier 1s are unsuitable even as backup connections, since you still want your backup connection to have access to the whole internet! Good tier 2 providers might be an excellent choice, sine good providers have already done this leg work and can monitor their providers for compliance. A few myths... Routing table size has nothing to do with completeness of routes. Google may be one route, through aggregation. And SmallCo may advertise a large route through one provider, and, due to traffic engineering, a smaller route through a second one - in many cases, anyone that had the large route would be able to contact SmallCo, even without the smaller route being present. So routing table size doesn't work. In addition, some providers aggregate their routing tables to reduce routing load and such. Others intentionally don't or deaggregate it intentionally so that they can brag about having bigger routing tables. What you need to ask is: How many /64s can you get to from your network, and how many of these /64s are reachable from at least one other major provider (you don't care about internal-only networks, after all)? They can give you that information, but many won't want to. It's also not about technical people not getting along. It's about business players trying to make money, but not just that either. It's also about ensuring that providers don't end up assuming more than their share of costs for a link. Just because you have a common peering point doesn't mean that turning peering on would reduce your costs. In some cases it may increase costs tremendously, particularly on your long haul backbone links, because the other party would like to take advantage of an attitude of trust on the internet. That's why we end up with peering policies and contracts. What is the issue? Let's take Hurricane. This is no different than other providers...basically, they want to say, We shouldn't need to pay for IPv6 transit from anyone. This is what Cogent said on IPv4 a few years ago. Google used to say this too for IPv6, not sure if they are still saying it. Basically, We know we're big enough that you won't want to screw your users by not peering with us. A small network couldn't do this tactic - a 100 node network who said to the IPv4 tier 1s: Hey, I'm in the Podunk Internet Exchange, so are you, so I'm going to peer from you so I don't have to buy any bandwidth for my web server (placed in the Podunk exchange). Sure, they would like to - it would save a ton of money if their site got lots of hits. I mean, who wouldn't want free connectivity? In IPv6, we're going through what we settled years ago in IPv4 - who has to pay who to connect. After all, even free peering connections have a cost in manpower, debugging, traffic engineering, documentation, etc. Some players who aren't getting free interconnection to tier 1s in IPv4 want to get it in IPv6. So they've worked to attract lots of users, and done so under the guise of We like IPv6 and want to promote it. Others have not bothered with trying to attract the users, but have said, We're too big for you to not want to give us connectivity for free, since it would piss off your users if you don't (Google did this at one point in the past, may still be doing it). The Google example is basically trying
