Re: Route leak in Bangladesh
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 2/Jul/15 19:48, Hugo Slabbert wrote: Jeebuz; you accept /128s? Perhaps le 24 le 48? Yes, that was a typo - /48 :-). Mark. -BEGIN PGP SIGNATURE- iQIcBAEBCAAGBQJVlixgAAoJEGcZuYTeKm+GLRQP/2HjuqFJW+pzhuH9qSbltl1D hz//dUVzJnGn2dE96lJGa+EcgZ096ax5sjSoNm/Ea9/cXJkauWN+oRp1H1DoNfvL Mp8RRcE9iVaavh5SdEBEEyKeqYHUjvyXmZWbCiVH1rebmTqUN0xHTne6AT3PrwhT rkyK+dN5tBKSrg5WWcsAaYvRLNOL8R93YTA8HjeNXzkp9zk29oPcrlGf2us/jPcq Qp1IIoOLBndeDT8A9rV5ubQNzPUEJJEoHGb5ESsPv8pMt26CaYExG1PPNhhNKyvD kIZTYD0L6lp42Ai5C+Jmv7hpL8ATH8m6dGxwzj47afbwYxbk76QsZKNPz3pFsOVV TDLMK07+aSlIozzQ9I9qFYjNA2jeYCwHcQsedcKCydYdzXMOvVMZW6HL/Ai0gBlf TRkOZMGyzerWk+hVn/YJLqFgfWonnUSTzX0ZWh0duGFEk7pVyvZdGPyJkF9J8UBq yJ38Zj+PvhPj0ZEt3LHShwUbMjcXAZ/4198HvQw+J2yFAAwm/ucHuAGL+Q/2LbWF D8V1EHF3eWBKo9SvvbYS1dNYHKxf8b7vdGgxIZqyMNUZpvjfuuXk3m9DwFQpjOnc //73PLWQtWb1udf1zdXkEQIwwAkF4UjBqgg1rW+CTkvFXXKLgLenVrnE9LbcI9UK QUDfEhw8o4LKTOHyWJah =GT1h -END PGP SIGNATURE-
Re: Route leak in Bangladesh
On Wed 2015-Jul-01 17:02:13 +0200, Mark Tinka mark.ti...@seacom.mu wrote: On 1/Jul/15 16:54, Nick Hilliard wrote: you probably want to ignore more rpsl constructs and depend solely on as-sets, aut-nums and route/route6 objects. RPSL is not going to live up to your expectations. Honestly, I'm ambivalent about using the IRR data for prefix-list generation (even without RPSL), also because of how much junk there is in there, and also how redundant some of it really is, e.g., someone creating a /32 (IPv4) route object and yet we only accept up to a /24 (IPv4) on the actual eBGP session, e.t.c. What I'm more focused is how we can continue to scale our current system, which is much more strict, focuses on deploying customer aggregates + le 24 le 128, instead of enumerating all possible de-aggregates that have been registered in the IRR (helps keep the configuration file small and manageable, without sacrificing reachability). And then see how we can add RPKI into the mix to make things even simpler, if at all. Orphaned/crufty data and braindead moves (e.g. someone including a large upstream's AS-SET in their own or something) aside, the deagg issue should be handled by the tools, no? We do automated prefix gen from IRR data, and I know IRRPT will aggregate the route records before spitting out the prefix list. I would have assumed that other tools do the same? Options are available to define your max prefix length and then build your filters off of that, e.g. aggregated route object upto /max prefix len. You still face issues with people people registering discontiguous blocks (e.g. network has a.b.0.0/22 but creates records for a.b.0.0/23 and a.b.3.0/24, but leaves out a.b.2.0/24), but there isn't much we can do about that without human interaction to verify the actual allocation. Also: focuses on deploying customer aggregates + le 24 le 128... Jeebuz; you accept /128s? Perhaps le 24 le 48? Mark. -- Hugo signature.asc Description: Digital signature
Re: Route leak in Bangladesh
On 1/Jul/15 17:11, Nick Hilliard wrote: The source code is available on github.com/inex. Lots of IXPs use it in production. Thanks, Nick. I'll have a bit of a sniff... Mark.
Re: Route leak in Bangladesh
On Wed, Jul 01, 2015 at 08:25:06AM +0200, Mark Tinka wrote: On 30/Jun/15 17:09, Job Snijders wrote: If you are a network providing transit to the leak originator mentioned in the above paragraph, I believe a prefix based filter could have made a big difference. And therein lies the secret sauce. Given that we've had an incident like this twice in the past month, I'm seriously concerned about the network operations of top-tier providers. I'll say we certainly try hard to mitigate these issues. It's hard because while platitudes on this list don't cause IOS devices to not send a full routing table by default (for example). I would like to see others participate in the dialog with vendors so we don't seem to be quite an outlier with wow, you have really large configs. The vendors haven't quite kept pace with the increase in density proportional to the number of configuration lines and it sure feels like we are the only people pushing them to improve. This combined with the number of devices that are doing kinky routing to 'optmize' a network make it more likely that something will cause damage. rfc1925 2.(9)a applies. - Jared -- Jared Mauch | pgp key available via finger from ja...@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
Re: Route leak in Bangladesh
On 01/07/2015 15:57, Mark Tinka wrote: Remember some high-end Cisco routers only have 2MB of NVRAM. This could get tested with a large prefix-list configuration. Junos may not have much of a space issue since the configuration is stored on the compact flash or HDD. Not at all. Even C6500 could store startup-config on external CF which could be 2G. Trie compilation or process will be very OS-dependent, and how the vendor has chosen to optimize that operation. Naah, trie compilation is simple, particularly with a line oriented configuration like IOS (one of the worse offenders). Once the config is syntax-checked, a regexp will split it out trivially and the binary form of the data can be compiled. Even on Junos, that sort of config will be handled by lex/yacc, which is highly optimised. Insertion / deletion of data in a patricia trie is ridiculously fast and there are a couple of bsd licensed implementations out there. Nick
Re: Route leak in Bangladesh
On 1/Jul/15 17:04, Nick Hilliard wrote: Naah, trie compilation is simple, particularly with a line oriented configuration like IOS (one of the worse offenders). Once the config is syntax-checked, a regexp will split it out trivially and the binary form of the data can be compiled. Even on Junos, that sort of config will be handled by lex/yacc, which is highly optimised. Insertion / deletion of data in a patricia trie is ridiculously fast and there are a couple of bsd licensed implementations out there. My experience around this was mostly when Cisco began introducing Turbo ACL's. There seemed to be a few problems around that time, but it was such a long time ago that I barely remember the details. That said, I'm not quite sure if there are specific issues Jared and others are facing around this in their networks. From my side, none that we have witnessed. But then again, our configurations are significantly smaller because we do not take data from the IRR. Mark.
Re: Route leak in Bangladesh
That they do. Thanks for a great system, BTW! - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: Nick Hilliard n...@foobar.org To: Mark Tinka mark.ti...@seacom.mu, Jared Mauch ja...@puck.nether.net Cc: North American Network Operators' Group nanog@nanog.org Sent: Wednesday, July 1, 2015 10:11:13 AM Subject: Re: Route leak in Bangladesh On 01/07/2015 16:02, Mark Tinka wrote: Honestly, I'm ambivalent about using the IRR data for prefix-list generation (even without RPSL), also because of how much junk there is in there, and also how redundant some of it really is, e.g., someone creating a /32 (IPv4) route object and yet we only accept up to a /24 (IPv4) on the actual eBGP session, e.t.c. We went through this a couple of years ago at INEX and ended up with a provisioning system which allows the operator to only allow specific IRRDB source: entries, customisable per customer. You're right that it would be foolish to accept any IRRDB source because a lot of them are complete trash. Otherwise, it works incredibly well for us and has stopped innumerable prefix leaks and other silly misconfigs. The source code is available on github.com/inex. Lots of IXPs use it in production. Nick
Re: Route leak in Bangladesh
On 01/07/2015 16:02, Mark Tinka wrote: Honestly, I'm ambivalent about using the IRR data for prefix-list generation (even without RPSL), also because of how much junk there is in there, and also how redundant some of it really is, e.g., someone creating a /32 (IPv4) route object and yet we only accept up to a /24 (IPv4) on the actual eBGP session, e.t.c. We went through this a couple of years ago at INEX and ended up with a provisioning system which allows the operator to only allow specific IRRDB source: entries, customisable per customer. You're right that it would be foolish to accept any IRRDB source because a lot of them are complete trash. Otherwise, it works incredibly well for us and has stopped innumerable prefix leaks and other silly misconfigs. The source code is available on github.com/inex. Lots of IXPs use it in production. Nick
Re: Route leak in Bangladesh
On 01/07/2015 15:12, Jared Mauch wrote: I would like to see others participate in the dialog with vendors so we don't seem to be quite an outlier with wow, you have really large configs. The vendors haven't quite kept pace with the increase in density proportional to the number of configuration lines and it sure feels like we are the only people pushing them to improve. This is a strange sort of thing really. There's no reason that a compiled prefix list of 250k entries should take up much RAM in a trie structure; there's no reason that a competently written parser shouldn't be able to handle 20 megs of prefix lists / sets in a trivial amount of time and there's no reason that writing a 20 meg configuration file should take long to write to disk / flash / etc. BIRD handles this in ultraquick time. Even recent versions of Quagga can now suck + parse 10 megs of prefix filters in a second or two and write them out in less. But Junos / IOS / XR puke horribly. What gives? Nick
Re: Route leak in Bangladesh
On 01/07/2015 15:51, Mark Tinka wrote: I found RPSL complicated a few years ago, and sort of put that on the back-burner. you probably want to ignore more rpsl constructs and depend solely on as-sets, aut-nums and route/route6 objects. RPSL is not going to live up to your expectations. Nick
Re: Route leak in Bangladesh
On 01/07/2015 17:03, Joe Abley wrote: The idea of configuring this stuff from the IRR is great in terms of distributing the ops cycles in the right places, but it doesn't help with verifying that the end result isn't insane, as I think you and Mike have described on this list over the past couple of days. that doesn't invalidate it as being part of a critical mechanism for filtering ebgp. Implemented well, it will catch 99% of problems. maxprefixes with no autorecover catches 75% of the rest. Between these two mechanisms, that's pretty good. Nick
Re: Route leak in Bangladesh
On 1/Jul/15 16:54, Nick Hilliard wrote: you probably want to ignore more rpsl constructs and depend solely on as-sets, aut-nums and route/route6 objects. RPSL is not going to live up to your expectations. Honestly, I'm ambivalent about using the IRR data for prefix-list generation (even without RPSL), also because of how much junk there is in there, and also how redundant some of it really is, e.g., someone creating a /32 (IPv4) route object and yet we only accept up to a /24 (IPv4) on the actual eBGP session, e.t.c. What I'm more focused is how we can continue to scale our current system, which is much more strict, focuses on deploying customer aggregates + le 24 le 128, instead of enumerating all possible de-aggregates that have been registered in the IRR (helps keep the configuration file small and manageable, without sacrificing reachability). And then see how we can add RPKI into the mix to make things even simpler, if at all. Mark.
Re: Route leak in Bangladesh
On Wed, Jul 01, 2015 at 03:54:16PM +0100, Nick Hilliard wrote: On 01/07/2015 15:51, Mark Tinka wrote: I found RPSL complicated a few years ago, and sort of put that on the back-burner. you probably want to ignore more rpsl constructs and depend solely on as-sets, aut-nums and route/route6 objects. RPSL is not going to live up to your expectations. Yes, like any technology there are lots of knobs that one could use but are not recommended. You just need these few objects and life will be simple. - Jared -- Jared Mauch | pgp key available via finger from ja...@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
Re: Route leak in Bangladesh
On 1/Jul/15 16:52, Nick Hilliard wrote: This is a strange sort of thing really. There's no reason that a compiled prefix list of 250k entries should take up much RAM in a trie structure; there's no reason that a competently written parser shouldn't be able to handle 20 megs of prefix lists / sets in a trivial amount of time and there's no reason that writing a 20 meg configuration file should take long to write to disk / flash / etc. BIRD handles this in ultraquick time. Even recent versions of Quagga can now suck + parse 10 megs of prefix filters in a second or two and write them out in less. But Junos / IOS / XR puke horribly. What gives? Nick, I think the concerns are two-fold: 1. The time it takes to process the trie. 2. How much physical space there is to support the configuration. Remember some high-end Cisco routers only have 2MB of NVRAM. This could get tested with a large prefix-list configuration. Junos may not have much of a space issue since the configuration is stored on the compact flash or HDD. Trie compilation or process will be very OS-dependent, and how the vendor has chosen to optimize that operation. Mark.
Re: Route leak in Bangladesh
On 1/Jul/15 16:12, Jared Mauch wrote: I would like to see others participate in the dialog with vendors so we don't seem to be quite an outlier with wow, you have really large configs. The vendors haven't quite kept pace with the increase in density proportional to the number of configuration lines and it sure feels like we are the only people pushing them to improve. I'm happy to help beat up the vendors (it's a favorite pass time of mine), but I'll be honest, we don't particularly have this problem in our network because - well, besides being a reasonably young operation - we do the opposite where we request customers to provide their prefix data, and we upload that into the network, together with strict AS_PATH lists (and max-prefix to boot). I found RPSL complicated a few years ago, and sort of put that on the back-burner. I have looked at it less in recent times because I'm putting quite a bit of work into RPKI (vendor implementation issues have been slowing me down, though, but we're looking better compared to just twelve months ago). So perhaps if there are other operators on this list using RPSL to build reasonably large configuration files that are testing the limits of router code, I echo Jared's plea to beat up your vendors and make this a priority, before we all get taken offline for the 3rd time in one year by the next boo-boo. And for anyone running Brocade, extra points if we can get them to implement RPKI as well :-)... Mark.
Re: Route leak in Bangladesh
On 1 Jul 2015, at 11:03, Jared Mauch wrote: On Wed, Jul 01, 2015 at 03:54:16PM +0100, Nick Hilliard wrote: On 01/07/2015 15:51, Mark Tinka wrote: I found RPSL complicated a few years ago, and sort of put that on the back-burner. you probably want to ignore more rpsl constructs and depend solely on as-sets, aut-nums and route/route6 objects. RPSL is not going to live up to your expectations. Yes, like any technology there are lots of knobs that one could use but are not recommended. You just need these few objects and life will be simple. ... as long as the people responsible for maintaining those as-sets don't get confused and include lots of other inappropriate as-sets by reference, right? The idea of configuring this stuff from the IRR is great in terms of distributing the ops cycles in the right places, but it doesn't help with verifying that the end result isn't insane, as I think you and Mike have described on this list over the past couple of days. Joe
Re: Route leak in Bangladesh
On 30/Jun/15 17:04, Nick Hilliard wrote: plus: - fully automate ingress prefix management - use maxprefixes with manual reenable on all ebgp sessions Yes, good point - forgot about that one; default max-prefix for all eBGP sessions. Mark.
Re: Route leak in Bangladesh
On 30/Jun/15 17:09, Job Snijders wrote: If you are a network providing transit to the leak originator mentioned in the above paragraph, I believe a prefix based filter could have made a big difference. And therein lies the secret sauce. Given that we've had an incident like this twice in the past month, I'm seriously concerned about the network operations of top-tier providers. Mark.
Re: Route leak in Bangladesh
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 30/Jun/15 16:53, Sandra Murphy wrote: That sort of AS_PATH filtering would not have helped in this case. The AS originated the routes, it did not propagate an upstream route. So an AS_PATH filter to just its own AS would have passed these routes. You would need origin validation on your outbound routes. Job suggested prefix filters on outbound routes. (If you are doing prefix filters on your inbound customer links, it might be excessive caution to also prefix filter customers prefixes on outbound links? Or is it: you can never be too careful, belt-and-suspenders, measure twice, etc?) Assuming you're running the same hardware/software across your backbone, correct prefix filters on inbound sessions to downstreams should be just fine. If those break, it's likely whatever broke them would also break them on egress to your upstreams and peers. The problem with egress prefix filters to upstreams and peers is that it's simply not scalable. Assuming you have a uniform routing policy where neighbors are all treated as eBGP sessions, then there is no real difference between upstreams, peers and other customers. Imagine having to build outbound prefix filters across your entire backbone for a uniform eBGP routing policy. Mark. -BEGIN PGP SIGNATURE- iQIcBAEBCAAGBQJVk4b7AAoJEGcZuYTeKm+GjPkP/1vEnL7mh0alWw+p6xCScUyH NxTYOYg1eBYUWQnGIWc+UTfZzKyr/LYbNyBF2Msf1aeNBOEb6kIY2geHUIGhOAZv DYIzggbvwWvd3X92aV76m3Nm8+z6nkDxnhYWgfefcXMofNTgHhQNKgsFp0efdDhA Mru60Cwi87apBLwY9wKYGqDtIgncKjLj92GfggimD7iwidvHZBXpKLIvPBE8sg9p aA/P9QqV2ZpVwoTtZy1Kb7B0FBogQFhPJX9RbWQUm0WwCuqMc8C7SibQMoF6hN0k rTuex7F4iPxTdvAcex/rRzIrQnDArIrMGkdOq3liQ8RG5d93Rara4NA9BgT6+ja/ idQ88lXjlBwzEEBh6k44Kg9Q686K503PK+hR8WrvETfojN8C4uD4WhUuqh3m2EPW UwJiZ8YD8oWQhLYpjdq/Rl7ozwu2ogi/ko69XuImi7f8OWscHD6QURoC0hONgLqF Rq7UgNcnOekUbTA+eP7ANFwKXNO+o9tomZ1tpmZqhNF5LLvazQFETcpEO2huQiON 2apxUiLWp3o8qCYKlvfUvREeF7fXaosgjXviWkjbdZc0v6hNjpd+M2uFPTz9CDgx PF9R+MzCu9C+gcfZRv4veY/ZFMxNxTNhOxppx69uyTG9+XCRXb5evjoV3VZPi/Qx RPUZQ1Ekzl0gAE7D4US6 =VZEA -END PGP SIGNATURE-
Route leak in Bangladesh
We have just received alert from bgpmon that AS58587 Fiber @ Home Limited has hijacked most of our (AS43996) prefixes and Hurricane Electric gladly accepted them. Anybody see their prefixes hijacked as well? -- Grzegorz Janoszka
Re: Route leak in Bangladesh
be nice if some technical details were included
Re: Route leak in Bangladesh
At 10:27 30/06/2015 +0200, Grzegorz Janoszka wrote: We have just received alert from bgpmon that AS58587 Fiber @ Home Limited has hijacked most of our (AS43996) prefixes and Hurricane Electric gladly accepted them. Anybody see their prefixes hijacked as well? Welcome to the party :-) Not only you. -Hank -- Grzegorz Janoszka
Re: Route leak in Bangladesh
A friend in AS58587 confirmed that this was caused by a configuration error - it seems like related to redistribution, and they already fixed that. - Matsuzaki Yoshinobu m...@iij.ad.jp - IIJ/AS2497 INOC-DBA: 2497*629 In message 559252e9.6030...@janoszka.pl Date: Tue, 30 Jun 2015 10:27:21 +0200 Grzegorz Janoszka grzeg...@janoszka.pl wrote We have just received alert from bgpmon that AS58587 Fiber @ Home Limited has hijacked most of our (AS43996) prefixes and Hurricane Electric gladly accepted them. Anybody see their prefixes hijacked as well? -- Grzegorz Janoszka
Re: Route leak in Bangladesh
Randy Bush ra...@psg.com wrote A friend in AS58587 confirmed that this was caused by a configuration error - it seems like related to redistribution, and they already fixed that. 7007 all over again. do not redistribute bgp into igp. do not redistribute igp into bgp. I also suggested them to implement BGP community based route filtering in their outbound policy. Any other suggestions or thoughts to prevent such incidents in general? - Matsuzaki Yoshinobu m...@iij.ad.jp - IIJ/AS2497 INOC-DBA: 2497*629
Re: Route leak in Bangladesh
On 30/Jun/15 15:22, Matsuzaki Yoshinobu wrote: I also suggested them to implement BGP community based route filtering in their outbound policy. Any other suggestions or thoughts to prevent such incidents in general? - Get your downstreams to create route objects before you turn them up. - Get your provisioning teams to validate the prefixes being provided by your downstreams. - Use both prefix- and AS_PATH-based filters for your downstreams. - Use BGP communities (as you've stated). - No exceptions. Mark.
Re: Route leak in Bangladesh
On Tue, Jun 30, 2015 at 10:22:38PM +0900, Matsuzaki Yoshinobu wrote: Randy Bush ra...@psg.com wrote A friend in AS58587 confirmed that this was caused by a configuration error - it seems like related to redistribution, and they already fixed that. 7007 all over again. do not redistribute bgp into igp. do not redistribute igp into bgp. I also suggested them to implement BGP community based route filtering in their outbound policy. Any other suggestions or thoughts to prevent such incidents in general? In addition to the BGP community scheme, outbound as-path filters could help. Most network's list of transit providers is fairly static, it would be easiy with as-path filters to prevent announcing upstream routes to other upstreams or peering partners. Kind regards, Job
Re: Route leak in Bangladesh
At 18:03 30/06/2015 +0900, Randy Bush wrote: be nice if some technical details were included Your prefix: xx.104.150.0/24: Prefix Description: Update time: 2015-06-30 07:39 (UTC) Detected by #peers: 8 Detected prefix: xx.104.150.0/24 Announced by: AS58587 (FIBERATHOME-BD Fiber @ Home Limited,BD) Upstream AS: AS6939 (HURRICANE - Hurricane Electric, Inc.,US) ASpath: 25152 6939 58587 and another: On Tuesday June 30th 2015 at 7:37 UTC we detected a Origin AS Change event for your prefix (23.44.244.0/22 Akamai) The detected prefix: 23.44.244.0/22, was announced by AS58587 (FIBERATHOME-BD Fiber @ Home Limited,BD) Alert description: Origin AS Change Detected Prefix: 23.44.244.0/22 Detected Origin AS: 58587 Expected Origin AS: 1680 Same Aspath of 6939 58587 -Hank
Re: Route leak in Bangladesh
On 30 Jun 2015, at 9:41, Job Snijders wrote: In addition to the BGP community scheme, outbound as-path filters could help. I agree, but possibly not in the case of a redistribution loop. (We don't know that's what happened, exactly, but I thought it was worth mentioning.) Joe
Re: Route leak in Bangladesh
On Tue, 30 Jun 2015, Matsuzaki Yoshinobu wrote: Randy Bush ra...@psg.com wrote A friend in AS58587 confirmed that this was caused by a configuration error - it seems like related to redistribution, and they already fixed that. 7007 all over again. do not redistribute bgp into igp. do not redistribute igp into bgp. I also suggested them to implement BGP community based route filtering in their outbound policy. Any other suggestions or thoughts to prevent such incidents in general? At a minimum, AS-PATH filtering of outgoing routes to just your ASN(s) and your downstream customer ASNs. Whether this is done manually, built using AS-SETs from your route registry of choice, or through some other automated means is another story. jms
Re: Route leak in Bangladesh
On Jun 30, 2015, at 10:39 AM, Justin M. Streiner strei...@cluebyfour.org wrote: On Tue, 30 Jun 2015, Matsuzaki Yoshinobu wrote: Randy Bush ra...@psg.com wrote A friend in AS58587 confirmed that this was caused by a configuration error - it seems like related to redistribution, and they already fixed that. 7007 all over again. do not redistribute bgp into igp. do not redistribute igp into bgp. I also suggested them to implement BGP community based route filtering in their outbound policy. Any other suggestions or thoughts to prevent such incidents in general? At a minimum, AS-PATH filtering of outgoing routes to just your ASN(s) and your downstream customer ASNs. Whether this is done manually, built using AS-SETs from your route registry of choice, or through some other automated means is another story. That sort of AS_PATH filtering would not have helped in this case. The AS originated the routes, it did not propagate an upstream route. So an AS_PATH filter to just its own AS would have passed these routes. You would need origin validation on your outbound routes. Job suggested prefix filters on outbound routes. (If you are doing prefix filters on your inbound customer links, it might be excessive caution to also prefix filter customers prefixes on outbound links? Or is it: you can never be too careful, belt-and-suspenders, measure twice, etc?) --Sandy signature.asc Description: Message signed with OpenPGP using GPGMail
Re: Route leak in Bangladesh
On Tue, Jun 30, 2015 at 09:44:12AM -0400, Joe Abley wrote: On 30 Jun 2015, at 9:41, Job Snijders wrote: In addition to the BGP community scheme, outbound as-path filters could help. I agree, but possibly not in the case of a redistribution loop. (We don't know that's what happened, exactly, but I thought it was worth mentioning.) Joe, you are right. In this specific situation, for a small to medium sized network, it might be prudent to apply an outbound prefix-filter on all transit peering sessions and thus only allowing prefixes which actually belong to downstream customers and the network itself. One could generate that prefix-list based on the network's AS-SET. I wouldn't deploy /only/ outbound prefix-filters. This method should be viewed as complementary to other methods such as the already mentioned a BGP community scheme. Kind regards, Job
Re: Route leak in Bangladesh
On 30/06/2015 14:29, Mark Tinka wrote: - Get your downstreams to create route objects before you turn them up. - Get your provisioning teams to validate the prefixes being provided by your downstreams. - Use both prefix- and AS_PATH-based filters for your downstreams. - Use BGP communities (as you've stated). - No exceptions. plus: - fully automate ingress prefix management - use maxprefixes with manual reenable on all ebgp sessions I've been caught with fully automated IRR based per-session prefix filtering where the customer put the IXP AS macro into their AS macro. When the customer did a 7007 on this, we accepted everything that they announced back to us, oy vey. So you need both. Nick
Re: Route leak in Bangladesh
On 30/06/2015 17:09, Job Snijders wrote: If you were the network causing a leak of this type, prefix filters on inbound facing your customers might not have prevented this. If you are a network providing transit to the leak originator mentioned in the above paragraph, I believe a prefix based filter could have made a big difference. We seem to be assuming that this leak occurred within the context of a customer-provider BGP relationship. But what if this is not the case? What if this was a peering session - perhaps via a route server at an exchange point. max-pref on a session with a route server is an extremely blunt (and potentially ineffective) tool for the job. In some regions the use to route servers and the lack of clue about anything BGP beyond one session to the route server (and one session to transit) is scary. We place our faith in the IXP operator, that they know best, while there may be no evidence that they do... ;-) -- Graham Beneke
Re: Route leak in Bangladesh
On 30/Jun/15 16:24, Job Snijders wrote: In this specific situation, for a small to medium sized network, it might be prudent to apply an outbound prefix-filter on all transit peering sessions and thus only allowing prefixes which actually belong to downstream customers and the network itself. I say that regardless of size, deploy the ultimate solution as the network is only bound to grow. It's harder for folk to undo old habits as they become more entrenched. Mark.
Re: Route leak in Bangladesh
Some more data from BGPmon.net: This affected close to 28,000 prefixes from 4,477 unique Autonomous systems. The hijacks were originated by AS58587 and propagated via AS45796 (15,002 prefixes) and AS6939 (25,841). The AS45796 paths were only seen via one of our peers, while the AS6939 path had a more significant visibility. The event started at 07:37 UTC and lasted for a few minutes. Cheers Andree .-- My secret spy satellite informs me that at 2015-06-30 3:26 AM Matsuzaki Yoshinobu wrote: A friend in AS58587 confirmed that this was caused by a configuration error - it seems like related to redistribution, and they already fixed that. - Matsuzaki Yoshinobu m...@iij.ad.jp - IIJ/AS2497 INOC-DBA: 2497*629 In message 559252e9.6030...@janoszka.pl Date: Tue, 30 Jun 2015 10:27:21 +0200 Grzegorz Janoszka grzeg...@janoszka.pl wrote We have just received alert from bgpmon that AS58587 Fiber @ Home Limited has hijacked most of our (AS43996) prefixes and Hurricane Electric gladly accepted them. Anybody see their prefixes hijacked as well? -- Grzegorz Janoszka
Re: Route leak in Bangladesh
On Tue, Jun 30, 2015 at 04:38:48PM +0200, Mark Tinka wrote: On 30/Jun/15 16:24, Job Snijders wrote: In this specific situation, for a small to medium sized network, it might be prudent to apply an outbound prefix-filter on all transit peering sessions and thus only allowing prefixes which actually belong to downstream customers and the network itself. I say that regardless of size, deploy the ultimate solution as the network is only bound to grow. It's harder for folk to undo old habits as they become more entrenched. Nothing is ever regardless of anything :-) I would forsee issues if i'd try to add an eleven megabyte prefix-list on all devices in the network. Kind regards, Job
Re: Route leak in Bangladesh
On Tue, Jun 30, 2015 at 10:53:45AM -0400, Sandra Murphy wrote: That sort of AS_PATH filtering would not have helped in this case. The AS originated the routes, it did not propagate an upstream route. So an AS_PATH filter to just its own AS would have passed these routes. You would need origin validation on your outbound routes. Job suggested prefix filters on outbound routes. (If you are doing prefix filters on your inbound customer links, it might be excessive caution to also prefix filter customers prefixes on outbound links? Or is it: you can never be too careful, belt-and-suspenders, measure twice, etc?) I wouldn't consider it to be excessive caution to bring more safeguards to the game, you never know when diarrhea will strike. If you were the network causing a leak of this type, prefix filters on inbound facing your customers might not have prevented this. If you are a network providing transit to the leak originator mentioned in the above paragraph, I believe a prefix based filter could have made a big difference. Kind regards, Job
Re: Route leak in Bangladesh
On Jun 30, 2015, at 9:41 AM, Job Snijders j...@instituut.net wrote: In addition to the BGP community scheme, outbound as-path filters could help. Most network's list of transit providers is fairly static, it would be easiy with as-path filters to prevent announcing upstream routes to other upstreams or peering partners. Except that this was not a route leak per se. The announced routes AS_PATH shows them as originated by the offending AS, not *propagated* by the offending AS. So the outbound AS_PATH did not retain the upstream portion of the path. --Sandy signature.asc Description: Message signed with OpenPGP using GPGMail
Re: Route leak in Bangladesh
On Tue, 30 Jun 2015, Sandra Murphy wrote: On Jun 30, 2015, at 10:39 AM, Justin M. Streiner strei...@cluebyfour.org wrote: At a minimum, AS-PATH filtering of outgoing routes to just your ASN(s) and your downstream customer ASNs. Whether this is done manually, built using AS-SETs from your route registry of choice, or through some other automated means is another story. That sort of AS_PATH filtering would not have helped in this case. The AS originated the routes, it did not propagate an upstream route. I didn't realise they offending AS was originating those routes, rather than propagating the existing ones. So an AS_PATH filter to just its own AS would have passed these routes. That's why I suggested it as a minimum precaution. When I worked in the service provider world, we did prefix + AS-PATH filtering + max-prefix, which was pretty effective in keeping BGP-borne madness down to a dull roar. Would that stop everything? No, but it did help a lot. I still work in a BGP-speaking organization - just not one that has downstream BGP-speaking customers at this point. You would need origin validation on your outbound routes. Job suggested prefix filters on outbound routes. (If you are doing prefix filters on your inbound customer links, it might be excessive caution to also prefix filter customers prefixes on outbound links? Or is it: you can never be too careful, belt-and-suspenders, measure twice, etc?) It depends on how much automation can be done to update the necessary filters and AS-PATH ACLs, and how much you trust both the automation method and the data source for those filters. jms
Re: Route leak in Bangladesh
I have sent this to a contact at another Bangladeshi ISP that should be able to reach the right person for this ASAP. On 30-Jun-2015, at 1:57 pm, Grzegorz Janoszka grzeg...@janoszka.pl wrote: We have just received alert from bgpmon that AS58587 Fiber @ Home Limited has hijacked most of our (AS43996) prefixes and Hurricane Electric gladly accepted them. Anybody see their prefixes hijacked as well?
Re: Route leak in Bangladesh
A friend in AS58587 confirmed that this was caused by a configuration error - it seems like related to redistribution, and they already fixed that. 7007 all over again. do not redistribute bgp into igp. do not redistribute igp into bgp. randy
RE: Route leak in Bangladesh
Yes, we have seen one of our prefixes hikacked. We contacted to Fiberathome and they told us the issue has been solved. Greetings. Ferran. -Mensaje original- De: NANOG [mailto:nanog-boun...@nanog.org] En nombre de Grzegorz Janoszka Enviado el: martes, 30 de junio de 2015 10:27 Para: nanog@nanog.org Asunto: Route leak in Bangladesh We have just received alert from bgpmon that AS58587 Fiber @ Home Limited has hijacked most of our (AS43996) prefixes and Hurricane Electric gladly accepted them. Anybody see their prefixes hijacked as well? -- Grzegorz Janoszka
