Re: BCP38 For BGP Customers
- Original Message - > From: "Joel Halpern" > To: "Brian Turnbow" > Cc: nanog@nanog.org > Sent: Tuesday, November 8, 2022 10:03:20 AM > Subject: Re: BCP38 For BGP Customers > There is work a tthe IETF on an addon to RPKI called ASPA. There is a > draft that describes how the combiantion of ASPA and RPKI can be used to > help with DDOS prevention. > > There is also a working group at the IETF called SAVNET that is looking > at what technological additions can be made to address the shortcomings > in BCP 38. In fairness, there is distinct disagreement as to what those > shortcomings are, and whether the ideas being presented can help. Input > from more operators would be great. (For completeness, I am a co-chair > of that working group.) Wait; people are actually trying to implement BCP38, still? :-} Cheers, -- jra > On 11/8/2022 9:39 AM, Brian Turnbow via NANOG wrote: >>> This may not exist yet, but what about a uRPF-like feature that uses RPKI, >>> IRR, >>> etc. instead of current BGP feed? >> >> There is rfc8704 that extends urpf >> But I do not know of any commercial available solutions -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
Re: BCP38 For BGP Customers
On Tue, Nov 8, 2022 at 9:08 PM Grant Taylor via NANOG wrote: > This thread has made me wonder if there isn't a need for a 3rd type of > uRPF or comparable filtering wherein the incoming interface is a viable > route in the RIB even if it's not the best route in the FIB. Hi Grant, Two problems here: 1. uRPF reuses the existing FIB. Either the next hop matches the source (strict) or some entry in the FIB matches the source (loose). The hardware "fast path" to implement the per-packet FIB is already there; all it takes is a second lookup. Unless someone comes up with a really clever idea, that's basically all the filtering you can squeeze out of the FIB. To do something more, you'd have to implement an additional "fast path" in the router that can act on every packet. That's kinda expensive. 2. BGP is a distance-vector protocol, not a link-state protocol. The significance for filtering is that the BGP RIB on a given router does not describe the complete state of the network. It only knows about itself and its immediate neighbors. That's not algorithmically guaranteed to be enough knowledge about the network to determine that a packet with a given source address can't reasonably come from a particular interface. Consider the following scenario: A - B - C - D - E - A | F B receives a packet from A to F via C. Is the packet legitimate? B can't know because the only route to A in B's RIB is the direct connection to A. C didn't pass E's route to A back to B because it was a longer, unpreferred path. C might not even have received the route from D. So even if you go to the expense of building a fast path that can consider what's in the RIB instead of just the FIB, you don't have the information you need in the RIB either. Regards, Bill Herrin -- For hire. https://bill.herrin.us/resume/
Re: BCP38 For BGP Customers
On 11/8/22 2:01 PM, Matthew Petach wrote: You're thinking about it from the upstream perspective, where a route could be accepted but depreferenced and thus not actively used. Think about it from the downstream network's perspective, though. If you're my upstream, and I don't want to use your link for inbound traffic, but I'd like to be able to send out some traffic over the link, how can I advertise the prefix to you in a way that would both ensure that you have it in your table locally, so that uRPF is happy, but also to ensure no packets actually make *use* of that routing table entry? Sure, you could tag the routes with 'no-export', but that only prevents the prefix from propagating outward, it doesn't prevent traffic on that router from using the routing table entry. You can try adjusting your MED, and hope the upstream doesn't squash the MED back to a default value it applies to all its customers. For the most part, you're up against a wall. You don't know how your upstream's route selection process is stacked with respect to routes you advertise, so you have no certainty that if you announce a prefix to them, it won't potentially be used to carry all your inbound traffic. Interesting points Matt. Hence why I asked. :-) The only way to be sure that you won't take inbound traffic on a link is to not advertise the prefix at all across that link. I agree that's the only way to be absolutely certain. I would naively wonder if AS Path pre-pends would help mitigate this. However, based on the RIB vs FIB sub-threads in this larger thread, I'm beginning to doubt that such will work with uRPF because the route would be depreffed and thus not in the FIB thereby would be filtered by uRPF. As Mr. Bush might say, "I recommend all my competitors implement this in their networks." *nod*nod* Thank you for the understandable explanation Matt. -- Grant. . . . unix || die smime.p7s Description: S/MIME Cryptographic Signature
Re: BCP38 For BGP Customers
On 11/8/22 1:01 PM, William Herrin wrote: Hi Grant, Hi Bill, Two words: asymmetric routing. ACK Useful automated reverse path filtering can ONLY be used when there is exactly ONE valid path to which and from which packets can be received. This is where strict mode uRPF actually works. This seems to be predicated on /strict/ uRPF enforcement. As for loose mode, it's basically useless in a BCP38 discussion. Loose mode only filters bogons. It doesn't prevent impersonation of any IP address currently routed in the system and doesn't do anything at all on a router with a default route. Okay. I didn't see how /loose/ uRPF could do any good save for the DFZ or other situation where there isn't a default route. This thread has made me wonder if there isn't a need for a 3rd type of uRPF or comparable filtering wherein the incoming interface is a viable route in the RIB even if it's not the best route in the FIB. Thank you for the explanation Bill. -- Grant. . . . unix || die smime.p7s Description: S/MIME Cryptographic Signature
Re: Geo-IP Sling.com and/or Dish Network Contact.
Yes, the IP space is updated, not sure what Sling is using, but this customer will be redirected to YoutubeTV which doesn't seem to have the same geolocation issues. There's a significant [growing] issue here as if I'm several months into the ARIN allocation and can not utilize the space, I expect a number of providers are facing similar issues. - Jared On Mon, Nov 07, 2022 at 02:57:03PM -0500, Josh Luthman wrote: > Did you update the block with those services listed in the link? They > usually update ~weekly and of course there's a delay to the providers as > well. > > On Fri, Nov 4, 2022 at 5:35 PM Jared Mauch wrote: > > > Anyone figure this out? Have a new block that works with everything else > > it seems but them and about to tell this customer to switch from their > > service. > > > > Or if someone knows why 23.138.114.0/24 would be geolocated outside > > US/Michigan would be great to know. > > > > Thanks! > > > > Sent via RFC1925 compliant device > > > > On May 11, 2022, at 10:35 AM, Josh Luthman > > wrote: > > > > > > Dish/Sling isn't on here but check this list: > > > > https://thebrotherswisp.com/index.php/geo-and-vpn/ > > > > On Tue, May 10, 2022 at 5:18 PM Nicholas Warren > > wrote: > > > >> Does anyone know how to get ahold of Sling.com or Dish to update location > >> information on IPv4 addresses? > >> > >> I don’t know if meta discussion is allowed on-list, but maybe geolocation > >> contacts could be listed on the community site? > >> > >> - Nich Warren > >> > > -- Jared Mauch | pgp key available via finger from ja...@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
Re: BCP38 For BGP Customers
On Tue, Nov 8, 2022 at 5:28 AM Douglas Fischer wrote: > Another important point to note is that you MUST NOT drop everything else > that doesn't match this Prefix-List. > But put a bandwidth and PPS control on what doesn't match the prefix-list, > and block what exceeds. > Among other reasons, it is very common that Exceeded TTL responses come with > source IPs for private use, or IP blocks that are for public use but not > announced to the DFZ. Hi Douglas, TTL exceeded messages are not important. They're useful to diagnostics but nothing breaks if they're lost. There's no need to broadly allow misconfigured customers to originate packets from RFC1918 space nor to allow them to originate ICMP type 11 (time exceeded) messages from RFC1918 space. You should not do so. The packets you're worried about here are fragmentation needed: ICMP type 3 code 4. Fragmentation needed messages are used for Path MTU discovery. When PMTUD breaks, TCP fails. Path MTU discovery is the one place in the core TCP/IP protocol stack where the end-to-end principle was abandoned. TCP requires the ability to receive this notification from any midpoint node in order to shrink packets too large for that midpoint node's next hop. It's the most broken part of the TCP/IP design. Unfortunately, your solution of allowing ICMP type 3 messages from RFC1918 space is dysfunctional. Even if you accept the packets (and tailor your acceptance so that it only applies to ICMP type 3 messages with a source address in RFC1918 space) the packets are highly likely to be discarded elsewhere in the path. In the past, I've suggested that vendors implement a feature allowing destination unreachables generated from a privately addressed interface to be originated from a different, user-configured public IP address. So far I haven't seen any takers. Regards, Bill Herrin -- For hire. https://bill.herrin.us/resume/
Re: BCP38 For BGP Customers
On Mon, Nov 07, 2022 at 02:47:57PM -0500, Tom Beecher wrote: > > > > Are you taking the stance of "if you don't send us the prefix, then > > we don't accept the traffic"? > > > > If you were one of my upstreams, and you implemented that, you would very > quickly no longer be one of my upstreams. Yes, I suffer from having two upstreams that each have a shared transit supplier. They are most likely to only have a single best path on their network and i can observe in the flow data it's not the one I expect it to be. I'm not sure how that provider (3356) would make it happen. I can tell you that the uRPF that 7018 does made me not able to utilize one of the providers for outbound traffic because they never opened the proper ticket for routing that IP space until I had side-escalated to some people that could help me after several months. Thankfully it's not a lot of bits but was still annoying to diagnose and triage. - Jared > On Mon, Nov 7, 2022 at 2:22 PM Charles Rumford via NANOG > wrote: > > > Hello - > > > > I'm are currently working on getting BCP38 filtering in place for our BGP > > customers. My current plan is to use the Juniper uRPF feature to filter > > out > > spoofed traffic based on the routing table. The mentality would be: "If > > you > > don't send us the prefix, then we don't accept the traffic". This has > > raised > > some issues amongst our network engineers regarding multi-homed customers. > > > > One of the issues raised was if a multi-homed BGP customer revoked a > > prefix from > > one of their peerings, but continued sending us traffic on the link then > > we > > would drop the traffic. > > > > I would like to hear what others are doing for BCP38 deployments for BGP > > customers. Are you taking the stance of "if you don't send us the prefix, > > then > > we don't accept the traffic"? Are you putting in some kind of fall back > > filter > > in based on something like IRR data? > > > > Thanks! > > > > -- > > Charles Rumford (he/his/him) > > Network Engineer | Deft > > 1-312-268-9342 | charl...@deft.com > > deft.com > > -- Jared Mauch | pgp key available via finger from ja...@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
Re: Random Early Detect and streaming video
Sorry, everyone, my initial reply was only to Saku so I'm replying again for visibility to the list. On Tue, 8 Nov 2022 at 02:57, Saku Ytti wrote: > Hey, > > > On Mon, 7 Nov 2022 at 21:58, Graham Johnston > wrote: > > > > I've been involved in service provider networks, small retail ISPs, for > 20+ years now. Largely though, we've never needed complex QoS, as at > $OLD_DAY_JOB, we had been consistently positioned to avoid regular link > congestion by having sufficient capacity. In the few instances when we've > had link congestion, egress priority queuing met our needs. > > What does 'egress priority queueing' mean? Do you mean 'send all X, > before any Y, send all Y before any Z'? If this, then this must have > been quite some time now, as since traffic managers were implemented > in hardware ages ago, this hasn't been available. And the only thing > that has been available has been 'X has guaranteed rate X1, Y has Y1 > and Z has Z1' and love it or hate it, that's the QoS tool industry has > decided you need. > Yeah, I'm sure I didn't use all of the features, we did have to set a bandwidth-share value and possibly a bit more. I guess as I look at my past it was more a case of not having to perform any rate limiting on the parts of the network that I'm thinking about, and long term familiarity with that platform as compared to the new environment which I'm less familiar with, and is a different platform, Juniper to be specific. > > > combine that with the buffering and we should adjust the drop profile to > kick in at a higher percentage. Today we use 70% to start triggering the > drop behavior, but my head tells me it should be higher. The reason I am > saying this is that we are dropping packets ahead of full link congestion, > yes that is what RED was designed to do, but I surmise that we are making > this application worse than is actually intended. > > I wager almost no one knows what their RED curve is, and different > vendors have different default curves which is then the curve almost > everyone uses. Some use a RED curve such that everything is basically > tail drop (Juniper, 0% drop at 96% fill and 100% drop at 98% fill). > Some are linear. Some allow defining just two points, some allow > defining 64 points. And almost no one has any idea what their curve > is, i.e. mostly it doesn't matter. If it usually mattered, we'd all > know what the curve is and why. As practical example Juniper has > basically > Overall, with my current concern being drops before they seem to be necessary, combined with you comments about Juniper which I take to be the behavior of default drop profile, I feel more confident that our current drop profile behavior is just more aggressive than it needs to be. > > In your case, I assume you have at least two points with 0% drop at > 69% fill, then a linear curve from 70% to 100% fill with 1% to 100% > drop. It doesn't seem outright wrong to me. You have 2-3 goals here, > to avoid synchronising TCP flows so that you have steady fill, instead > of wave-like behaviour and to reduce queueing delay for packets not > dropped, which would experience as long a delay as there is queue if > tail dropped. You could have a 3rd possible goal, if you map more than > 1 class of packets into the same queue you can still give them > different curves, so during congestion in a single queue can show two > different behaviours depending on packet. > So what is the problem you're trying to fix? Can you measure it? > As mentioned above, my problem/supposition is that we drop too much before it's necessary and impact the customer experience in a way that isn't needed. While I can't directly measure the customer experience, I can measure drop rate versus bandwidth. If my supposition is correct, that a drop profile that drops later (at a higher utilization rate), we'd see less dropped packets, and possibly a higher utilization rate. While this whole configuration policy is in place to reduce utilization, we operate these links with a hard cap, thus I'd like to use as much of it as possible. What may have changed is that in the past these links were functionally operated at their capacity, rather than right now where we are slightly below capacity. > > I suspect in a modern high speed network with massive amounts of flows > the wave-like synchronisation is not a problem. If you can't measure > it or If your only goal is to reduce queueing delay because you have > 'strategic' congestion, perhaps instead of worrying about RED, use > tail only and reduce queue size to something that is tolerable 1ms-5ms > max? > On many levels, it does seem like what I want is tail drop rather than RED. > -- > ++ytti > Thanks for your response, Saku. I also am a user of Oxidized, thanks for that as well.
Re: BCP38 For BGP Customers
On Tue, Nov 8, 2022 at 8:44 AM Grant Taylor via NANOG wrote: > [...] > > I don't understand why you would want to allow packets that couldn't > return the same path. > > As for asymmetrically routed packets, I would still expect a return path > to exist, even if it's not utilized. > > Grant, You're thinking about it from the upstream perspective, where a route could be accepted but depreferenced and thus not actively used. Think about it from the downstream network's perspective, though. If you're my upstream, and I don't want to use your link for inbound traffic, but I'd like to be able to send out some traffic over the link, how can I advertise the prefix to you in a way that would both ensure that you have it in your table locally, so that uRPF is happy, but also to ensure no packets actually make *use* of that routing table entry? Sure, you could tag the routes with 'no-export', but that only prevents the prefix from propagating outward, it doesn't prevent traffic on that router from using the routing table entry. You can try adjusting your MED, and hope the upstream doesn't squash the MED back to a default value it applies to all its customers. For the most part, you're up against a wall. You don't know how your upstream's route selection process is stacked with respect to routes you advertise, so you have no certainty that if you announce a prefix to them, it won't potentially be used to carry all your inbound traffic. The only way to be sure that you won't take inbound traffic on a link is to not advertise the prefix at all across that link. Why might this be necessary, you ask? Imagine you've got links of different sizes on your network. You have a 1G link to provider A You have a 1G link to provider B You have a 10G link to cheap provider C You have a 10G link into a peering exchange Somewhere beyond provider A, someone decides they don't like one of your customers, and sends a 5Gbps DDoS flow at you. If you continue advertising that prefix to provider A, everyone going through provider A will suffer, including all their customers. You have plenty of capacity to take the inbound flow through the exchange point and through cheap provider C. So, you stop advertising the prefix under attack to provider A and provider B, to ensure the traffic doesn't saturate your 1G links. Inbound traffic happily shifts to the exchange point port and provider C's port. Life is good. Oh no! Provider A has implemented uRPF, and now all their customers are unhappy because they can't reach your websites on that prefix, because the *return* traffic is still flowing out the 1G link directly to provider A. Trying to implement a source-based routing filter to redirect all traffic *coming* from the prefix under attack destined to ISP A to instead go through ISP C is a pain in the butt. So, you grudgingly stop accepting routes from ISP A, as that's the only way to make the uRPF pain stop. Now, none of your traffic is flowing out the 1G link to ISP A; their customers are happy again, because they can reach websites on your prefix that is under attack (via ISP C, or the exchange point). At the end of the month, you look at your contracts, and realize that you had to spend a chunk of your limited engineering resources working around the upstream uRPF filter, and ultimately ended up not sending traffic across a link you were paying for. When renewal time comes, you decide it's not worth the headache to pay for a link to ISP A that you can't reliably use, and that requires manual intervention to work around whenever creative routing solutions are necessary. You don't renew your contract with ISP A. As Mr. Bush might say, "I recommend all my competitors implement this in their networks." Thanks! Matt
Re: BCP38 For BGP Customers
On Tue, Nov 8, 2022 at 12:29 PM Mike Hammett wrote: >> "Reverse path filtering literally says don't accept a packet from >> somewhere that isn't currently the next hop for that packet's source >> address." > > FIB or RIB? > > I knew of uRPF as available over an interface, per the routing table, not > best path available. Or is that implementation dependent? FIB. The RIB is never consulted on a per-packet basis. It's not built to be fast enough to consult on a per-packet basis. Regards, Bill Herrin -- For hire. https://bill.herrin.us/resume/
Re: BCP38 For BGP Customers
" Reverse path filtering literally says don't accept a packet from somewhere that isn't currently the next hop for that packet's source address." FIB or RIB? I knew of uRPF as available over an interface, per the routing table, not best path available. Or is that implementation dependent? - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "William Herrin" To: "Grant Taylor" Cc: nanog@nanog.org Sent: Tuesday, November 8, 2022 2:01:49 PM Subject: Re: BCP38 For BGP Customers On Tue, Nov 8, 2022 at 8:40 AM Grant Taylor via NANOG wrote: > Maybe it's the lack of caffeine, but would someone please remind / > enlighten me as to why uRPF is a bad idea on downstream interfaces? Hi Grant, Two words: asymmetric routing. If the downstream network is architected in such a way that there's more than one path in and out of their network then there's no way to guarantee that any particular router believes the forward path to that network goes to a particular next hop. It can currently map to any next hop that goes in the direction of one of the valid paths. That routing is completely independent of how next hops are selected in the other direction. Packets can travel in one path and out another. Reverse path filtering literally says don't accept a packet from somewhere that isn't currently the next hop for that packet's source address. When it's possible for the forward route to point a different direction than the one from which valid packets are received, that is the wrong thing to do. In fact, it breaks the network. Useful automated reverse path filtering can ONLY be used when there is exactly ONE valid path to which and from which packets can be received. This is where strict mode uRPF actually works. > N.B. Maybe I'm thinking more a varient of uRPF wherein if I have a route > to the source from the interface in question. Thus not uRPF-strict > (active route) nor uRPF-loose (route on any interface). Even if a particular router happens to have RIB entries for all the valid paths to a host (a sketchy proposition at best), only one such entry will be stored in the FIB where uRPF looks to make its filtering decision. As for loose mode, it's basically useless in a BCP38 discussion. Loose mode only filters bogons. It doesn't prevent impersonation of any IP address currently routed in the system and doesn't do anything at all on a router with a default route. Regards, Bill Herrin -- For hire. https://bill.herrin.us/resume/
Re: BCP38 For BGP Customers
On Tue, Nov 8, 2022 at 8:40 AM Grant Taylor via NANOG wrote: > Maybe it's the lack of caffeine, but would someone please remind / > enlighten me as to why uRPF is a bad idea on downstream interfaces? Hi Grant, Two words: asymmetric routing. If the downstream network is architected in such a way that there's more than one path in and out of their network then there's no way to guarantee that any particular router believes the forward path to that network goes to a particular next hop. It can currently map to any next hop that goes in the direction of one of the valid paths. That routing is completely independent of how next hops are selected in the other direction. Packets can travel in one path and out another. Reverse path filtering literally says don't accept a packet from somewhere that isn't currently the next hop for that packet's source address. When it's possible for the forward route to point a different direction than the one from which valid packets are received, that is the wrong thing to do. In fact, it breaks the network. Useful automated reverse path filtering can ONLY be used when there is exactly ONE valid path to which and from which packets can be received. This is where strict mode uRPF actually works. > N.B. Maybe I'm thinking more a varient of uRPF wherein if I have a route > to the source from the interface in question. Thus not uRPF-strict > (active route) nor uRPF-loose (route on any interface). Even if a particular router happens to have RIB entries for all the valid paths to a host (a sketchy proposition at best), only one such entry will be stored in the FIB where uRPF looks to make its filtering decision. As for loose mode, it's basically useless in a BCP38 discussion. Loose mode only filters bogons. It doesn't prevent impersonation of any IP address currently routed in the system and doesn't do anything at all on a router with a default route. Regards, Bill Herrin -- For hire. https://bill.herrin.us/resume/
Spoofer Report for NANOG for Oct 2022
In response to feedback from operational security communities, CAIDA's source address validation measurement project (https://spoofer.caida.org) is automatically generating monthly reports of ASes originating prefixes in BGP for systems from which we received packets with a spoofed source address. We are publishing these reports to network and security operations lists in order to ensure this information reaches operational contacts in these ASes. This report summarises tests conducted within usa, can. Inferred improvements during Oct 2022: ASNName Fixed-By 33696 NEXTARRAY-ASN-01 2022-10-02 3994862022-10-05 397086 LAYER-HOST-HOUSTON 2022-10-22 Further information for the inferred remediation is available at: https://spoofer.caida.org/remedy.php Source Address Validation issues inferred during Oct 2022: ASNName First-Spoofed Last-Spoofed 19230 NANOG 2016-06-13 2022-10-03 209CENTURYLINK-US-LEGACY-QWEST 2016-08-16 2022-10-29 6128 CABLE-NET-1 2016-09-03 2022-10-03 20412 CLARITY-TELECOM 2016-09-30 2022-10-31 271BCNET 2016-10-24 2022-10-27 30036 MEDIACOM-ENTERPRISE-BUSINESS 2016-11-16 2022-10-12 22898 ATLINK2016-12-16 2022-10-14 23089 HOTWIRE-COMMUNICATIONS2017-09-30 2022-10-04 33452 RW2018-09-19 2022-10-27 14031 SCXY 2018-10-18 2022-10-26 20278 NEXEON2019-03-05 2022-10-07 21804 ACCESS-SK 2019-06-09 2022-10-26 398836 NP-NETWORKS 2021-03-12 2022-10-30 394414 E2WS 2022-05-08 2022-10-31 400517 2022-10-03 2022-10-03 400328 2022-10-14 2022-10-14 Further information for these tests where we received spoofed packets is available at: https://spoofer.caida.org/recent_tests.php?country_include=usa,can_block=1 Please send any feedback or suggestions to spoofer-i...@caida.org
Re: [EXTERNAL] Re: BCP38 For BGP Customers
The Internet Draft is at: https://datatracker.ietf.org/doc/html/draft-sriram-sidrops-bar-sav-01 Some slides that will be used to present thematerial on Friday are at:https://datatracker.ietf.org/meeting/115/materials/slides-115-savnet-lowering-improper-block-and-improper-admit-for-sav-the-bar-sav-approach On 11/8/2022 12:17 PM, Compton, Rich A wrote: Hi Joel, can you please point us to the IETF draft document that describes how a "combination of ASPA and RPKI can be used to help with DDoS prevention". I was not able to find it. Thanks! -Rich On 11/8/22, 8:05 AM, "NANOG on behalf of Joel Halpern"j...@joelhalpern.com> wrote: CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. There is work a tthe IETF on an addon to RPKI called ASPA. There is a draft that describes how the combiantion of ASPA and RPKI can be used to help with DDOS prevention. There is also a working group at the IETF called SAVNET that is looking at what technological additions can be made to address the shortcomings in BCP 38. In fairness, there is distinct disagreement as to what those shortcomings are, and whether the ideas being presented can help. Input from more operators would be great. (For completeness, I am a co-chair of that working group.) Yours, Joel On 11/8/2022 9:39 AM, Brian Turnbow via NANOG wrote: > Hi Mike > > > >> This may not exist yet, but what about a uRPF-like feature that uses RPKI, IRR, etc. instead of current BGP feed? > > There is rfc8704 that extends urpf > But I do not know of any commercial available solutions > > > Brian E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
Re: [EXTERNAL] Re: BCP38 For BGP Customers
Hi Joel, can you please point us to the IETF draft document that describes how a "combination of ASPA and RPKI can be used to help with DDoS prevention". I was not able to find it. Thanks! -Rich On 11/8/22, 8:05 AM, "NANOG on behalf of Joel Halpern" wrote: CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. There is work a tthe IETF on an addon to RPKI called ASPA. There is a draft that describes how the combiantion of ASPA and RPKI can be used to help with DDOS prevention. There is also a working group at the IETF called SAVNET that is looking at what technological additions can be made to address the shortcomings in BCP 38. In fairness, there is distinct disagreement as to what those shortcomings are, and whether the ideas being presented can help. Input from more operators would be great. (For completeness, I am a co-chair of that working group.) Yours, Joel On 11/8/2022 9:39 AM, Brian Turnbow via NANOG wrote: > Hi Mike > > > >> This may not exist yet, but what about a uRPF-like feature that uses RPKI, IRR, etc. instead of current BGP feed? > > There is rfc8704 that extends urpf > But I do not know of any commercial available solutions > > > Brian E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
Re: BCP38 For BGP Customers
On 11/8/22 6:28 AM, Douglas Fischer wrote: I also have this concern about Spoofing coming from Downstreams. +1 And after a lot of struggle I can say that using uRPF in strict mode per interface doing FIB lookup is not a good idea! Maybe it's the lack of caffeine, but would someone please remind / enlighten me as to why uRPF is a bad idea on downstream interfaces? N.B. Maybe I'm thinking more a varient of uRPF wherein if I have a route to the source from the interface in question. Thus not uRPF-strict (active route) nor uRPF-loose (route on any interface). I've spent a lot of time wrestling with this issue, and the measurement that matters most, which are support tickets, doesn't show good results. Will you please share a high level of what the technical problems are? The best option I've found so far? It is to use the same Prefix-List that you use to release the routes accepted in the BGP session in a Filter Policy applied in the interface with the Downstream. Another important point to note is that you MUST NOT drop everything else that doesn't match this Prefix-List. Please clarify if there are routes that would return the packets that would be dropped back to your router & downstream client. I don't understand why you would want to allow packets that couldn't return the same path. As for asymmetrically routed packets, I would still expect a return path to exist, even if it's not utilized. (Yes, I know it's hard to accept that...) #headScratching But put a bandwidth and PPS control on what doesn't match the prefix-list, and block what exceeds. And why do it this way? Among other reasons, it is very common that Exceeded TTL responses come with source IPs for private use, or IP blocks that are for public use but not announced to the DFZ. I'll argue -- possibly from a place of ignorance -- that there should not be any packets on the Internet at large (default free zone) to or from IPs not part of the Internet (DFZ). I view the TTL Exceeded packets as errant in their non-globally-routed source address and as such should be dropped early / often. Both "be liberal in what you accept and conservative in what you send" (Jon Postel) and "Brown M" (Van Halen) come to mind, as does the newer industry phrase "fail-fast". -- Grant. . . . unix || die smime.p7s Description: S/MIME Cryptographic Signature
Re: BCP38 For BGP Customers
There is work a tthe IETF on an addon to RPKI called ASPA. There is a draft that describes how the combiantion of ASPA and RPKI can be used to help with DDOS prevention. There is also a working group at the IETF called SAVNET that is looking at what technological additions can be made to address the shortcomings in BCP 38. In fairness, there is distinct disagreement as to what those shortcomings are, and whether the ideas being presented can help. Input from more operators would be great. (For completeness, I am a co-chair of that working group.) Yours, Joel On 11/8/2022 9:39 AM, Brian Turnbow via NANOG wrote: Hi Mike This may not exist yet, but what about a uRPF-like feature that uses RPKI, IRR, etc. instead of current BGP feed? There is rfc8704 that extends urpf But I do not know of any commercial available solutions Brian
RE: BCP38 For BGP Customers
Hi Mike > This may not exist yet, but what about a uRPF-like feature that uses RPKI, > IRR, etc. instead of current BGP feed? There is rfc8704 that extends urpf But I do not know of any commercial available solutions Brian
Re: BCP38 For BGP Customers
I also have this concern about Spoofing coming from Downstreams. And after a lot of struggle I can say that using uRPF in strict mode per interface doing FIB lookup is not a good idea! And I feel sad to have to say that. I've spent a lot of time wrestling with this issue, and the measurement that matters most, which are support tickets, doesn't show good results. The best option I've found so far? It is to use the same Prefix-List that you use to release the routes accepted in the BGP session in a Filter Policy applied in the interface with the Downstream. Another important point to note is that you MUST NOT drop everything else that doesn't match this Prefix-List. (Yes, I know it's hard to accept that...) But put a bandwidth and PPS control on what doesn't match the prefix-list, and block what exceeds. And why do it this way? Among other reasons, it is very common that Exceeded TTL responses come with source IPs for private use, or IP blocks that are for public use but not announced to the DFZ. With this method of a Policy-Filter based on the same Prefix-List as BGP and a rate-limit that doesn't match the prefix-list you won't block 100% of the spoofed packets that might come from a downstream, but it certainly will. block an eventual DDoS vector coming from this interface. It is worth remembering that your neighborhood router with Downstream has to support this type of filtering in high capacity. And it is almost certain that only hardware based filtering scenarios will handle this. Em seg., 7 de nov. de 2022 às 16:23, Charles Rumford via NANOG < nanog@nanog.org> escreveu: > Hello - > > I'm are currently working on getting BCP38 filtering in place for our BGP > customers. My current plan is to use the Juniper uRPF feature to filter > out > spoofed traffic based on the routing table. The mentality would be: "If > you > don't send us the prefix, then we don't accept the traffic". This has > raised > some issues amongst our network engineers regarding multi-homed customers. > > One of the issues raised was if a multi-homed BGP customer revoked a > prefix from > one of their peerings, but continued sending us traffic on the link then > we > would drop the traffic. > > I would like to hear what others are doing for BCP38 deployments for BGP > customers. Are you taking the stance of "if you don't send us the prefix, > then > we don't accept the traffic"? Are you putting in some kind of fall back > filter > in based on something like IRR data? > > Thanks! > > -- > Charles Rumford (he/his/him) > Network Engineer | Deft > 1-312-268-9342 | charl...@deft.com > deft.com > -- Douglas Fernando Fischer Engº de Controle e Automação
Re: Random Early Detect and streaming video
> > What does 'egress priority queueing' mean? Do you mean 'send all X, > before any Y, send all Y before any Z'? If this, then this must have > been quite some time now, as since traffic managers were implemented > in hardware ages ago, this hasn't been available. > As you'll probably remember, I'm just an academic who tries to keep up with the times. What I can add is that in the latest MEF CECP (Carrier Ethernet Certified Professional) courseware (Blueprint D), egress bandwidth profiles that are based on token buckets are still actively advocated as a means of applying QoS. Cheers, Etienne On Tue, Nov 8, 2022 at 10:00 AM Saku Ytti wrote: > Hey, > > > On Mon, 7 Nov 2022 at 21:58, Graham Johnston > wrote: > > > > I've been involved in service provider networks, small retail ISPs, for > 20+ years now. Largely though, we've never needed complex QoS, as at > $OLD_DAY_JOB, we had been consistently positioned to avoid regular link > congestion by having sufficient capacity. In the few instances when we've > had link congestion, egress priority queuing met our needs. > > What does 'egress priority queueing' mean? Do you mean 'send all X, > before any Y, send all Y before any Z'? If this, then this must have > been quite some time now, as since traffic managers were implemented > in hardware ages ago, this hasn't been available. And the only thing > that has been available has been 'X has guaranteed rate X1, Y has Y1 > and Z has Z1' and love it or hate it, that's the QoS tool industry has > decided you need. > > > combine that with the buffering and we should adjust the drop profile to > kick in at a higher percentage. Today we use 70% to start triggering the > drop behavior, but my head tells me it should be higher. The reason I am > saying this is that we are dropping packets ahead of full link congestion, > yes that is what RED was designed to do, but I surmise that we are making > this application worse than is actually intended. > > I wager almost no one knows what their RED curve is, and different > vendors have different default curves which is then the curve almost > everyone uses. Some use a RED curve such that everything is basically > tail drop (Juniper, 0% drop at 96% fill and 100% drop at 98% fill). > Some are linear. Some allow defining just two points, some allow > defining 64 points. And almost no one has any idea what their curve > is, i.e. mostly it doesn't matter. If it usually mattered, we'd all > know what the curve is and why. As practical example Juniper has > basically > > In your case, I assume you have at least two points with 0% drop at > 69% fill, then a linear curve from 70% to 100% fill with 1% to 100% > drop. It doesn't seem outright wrong to me. You have 2-3 goals here, > to avoid synchronising TCP flows so that you have steady fill, instead > of wave-like behaviour and to reduce queueing delay for packets not > dropped, which would experience as long a delay as there is queue if > tail dropped. You could have a 3rd possible goal, if you map more than > 1 class of packets into the same queue you can still give them > different curves, so during congestion in a single queue can show two > different behaviours depending on packet. > So what is the problem you're trying to fix? Can you measure it? > > I suspect in a modern high speed network with massive amounts of flows > the wave-like synchronisation is not a problem. If you can't measure > it or If your only goal is to reduce queueing delay because you have > 'strategic' congestion, perhaps instead of worrying about RED, use > tail only and reduce queue size to something that is tolerable 1ms-5ms > max? > > -- > ++ytti > -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale
Re: Random Early Detect and streaming video
Hey, On Mon, 7 Nov 2022 at 21:58, Graham Johnston wrote: > I've been involved in service provider networks, small retail ISPs, for 20+ > years now. Largely though, we've never needed complex QoS, as at > $OLD_DAY_JOB, we had been consistently positioned to avoid regular link > congestion by having sufficient capacity. In the few instances when we've > had link congestion, egress priority queuing met our needs. What does 'egress priority queueing' mean? Do you mean 'send all X, before any Y, send all Y before any Z'? If this, then this must have been quite some time now, as since traffic managers were implemented in hardware ages ago, this hasn't been available. And the only thing that has been available has been 'X has guaranteed rate X1, Y has Y1 and Z has Z1' and love it or hate it, that's the QoS tool industry has decided you need. > combine that with the buffering and we should adjust the drop profile to kick > in at a higher percentage. Today we use 70% to start triggering the drop > behavior, but my head tells me it should be higher. The reason I am saying > this is that we are dropping packets ahead of full link congestion, yes that > is what RED was designed to do, but I surmise that we are making this > application worse than is actually intended. I wager almost no one knows what their RED curve is, and different vendors have different default curves which is then the curve almost everyone uses. Some use a RED curve such that everything is basically tail drop (Juniper, 0% drop at 96% fill and 100% drop at 98% fill). Some are linear. Some allow defining just two points, some allow defining 64 points. And almost no one has any idea what their curve is, i.e. mostly it doesn't matter. If it usually mattered, we'd all know what the curve is and why. As practical example Juniper has basically In your case, I assume you have at least two points with 0% drop at 69% fill, then a linear curve from 70% to 100% fill with 1% to 100% drop. It doesn't seem outright wrong to me. You have 2-3 goals here, to avoid synchronising TCP flows so that you have steady fill, instead of wave-like behaviour and to reduce queueing delay for packets not dropped, which would experience as long a delay as there is queue if tail dropped. You could have a 3rd possible goal, if you map more than 1 class of packets into the same queue you can still give them different curves, so during congestion in a single queue can show two different behaviours depending on packet. So what is the problem you're trying to fix? Can you measure it? I suspect in a modern high speed network with massive amounts of flows the wave-like synchronisation is not a problem. If you can't measure it or If your only goal is to reduce queueing delay because you have 'strategic' congestion, perhaps instead of worrying about RED, use tail only and reduce queue size to something that is tolerable 1ms-5ms max? -- ++ytti
