Re: Advertisement of Equinix Chicago IX Subnet

[Job Snijders]

On Wed, Mar 27, 2019 at 09:36:20PM +, Graham Johnston wrote:
> This afternoon at around 12:17 central time today we began learning
> the subnet for the Equinix IX in Chicago via a transit provider; we
> are on the IX as well. The subnet in question is 208.115.136.0/23.
> Using stat.ripe.net I can see that this subnet is also being learned
> by others, see the snip below. On our network this caused a nasty
> routing loop until we figured out what was wrong. My current best
> understanding is that because the route was learned via eBGP it
> trumped the OSPF learned route. As soon as I filtered the
> advertisement from my transit provider everything returned to normal.
> What am I doing that isn’t best practices that would have prevented
> this?

There is two pieces to help prevent this type of failure:

1/ Equinix should have created a RPKI ROA for 208.115.136.0/23, with an
   Origin ASN of 0 or one of their own ASNs, and a Max Length of 23.

2/ You should implement RPKI based BGP Origin Validation in your network
   and honor those ROAs.

Kind regards,

Job

Re: Advertisement of Equinix Chicago IX Subnet

[Job Snijders]

On Thu, Mar 28, 2019 at 02:59:43PM +0100, Niels Bakker wrote:
> * christopher.morrell.na...@gmail.com (Christopher Morrell) [Thu 28 Mar 2019, 
> 14:35 CET]:
> > I've been bit by this in the past at two different exchanges. I too
> > have a policy applied to deny IXP LANs from upstreams and peers. It
> > would be nice if there was a list of all IXP LANs somewhere that we
> > could generically add to all upstream and peers.
> 
> I like Nick Hilliard's posted solution much better than creating
> static bogon lists that people will eventually forget about.

IXPs can use RPKI ROAs to signal to the world what their intentions are!
IXPs could either create a ROA with an Origin ASN of '0' to suggest to
the world that the peering lan prefix should never be visible in the
DFZ, or they can specify their own services ASN and simply not announce
the prefix. In either case IXPs should carefully specify the Max Length
value to be the same as the Prefix Length value of the peering lan
prefix.

Kind regards,

Job

Re: Advertisement of Equinix Chicago IX Subnet

[Niels Bakker]

* christopher.morrell.na...@gmail.com (Christopher Morrell) [Thu 28 Mar 2019, 
14:35 CET]:
I've been bit by this in the past at two different exchanges.  I too 
have a policy applied to deny IXP LANs from upstreams and peers.  It 
would be nice if there was a list of all IXP LANs somewhere that we 
could generically add to all upstream and peers.


I like Nick Hilliard's posted solution much better than creating 
static bogon lists that people will eventually forget about.



-- Niels.

Re: Advertisement of Equinix Chicago IX Subnet

[Christopher Morrell]

I've been bit by this in the past at two different exchanges.  I too have a
policy applied to deny IXP LANs from upstreams and peers.  It would be nice
if there was a list of all IXP LANs somewhere that we could generically add
to all upstream and peers.


On Thu, Mar 28, 2019 at 9:11 AM Eric Dugas  wrote:

> I have a policy applied to my upstreams and peers to deny the IXP's LANs
> were connected to. I don't think of any reason to learn these routes from
> someone else's network.
>
> On Wed, Mar 27, 2019 at 7:44 PM Cummings, Chris 
> wrote:
>
>> Not too sure about your topology, but I’ve had something similar bite me,
>> so we typically put a prefix list inbound to deny receiving our internal
>> prefixes from our peers. This probably doesn’t work as well if your network
>> is less “eyeballish” than ours, however.
>>
>> /chris
>>
>>
>>
>> On Wed, Mar 27, 2019 at 4:37 PM -0500, "Graham Johnston" <
>> johnst...@westmancom.com> wrote:
>>
>> This afternoon at around 12:17 central time today we began learning the
>>> subnet for the Equinix IX in Chicago via a transit provider; we are on the
>>> IX as well. The subnet in question is 208.115.136.0/23. Using
>>> stat.ripe.net
>>> 
>>> I can see that this subnet is also being learned by others, see the snip
>>> below. On our network this caused a nasty routing loop until we figured out
>>> what was wrong. My current best understanding is that because the route was
>>> learned via eBGP it trumped the OSPF learned route. As soon as I filtered
>>> the advertisement from my transit provider everything returned to normal.
>>> What am I doing that isn’t best practices that would have prevented this?
>>>
>>>
>>>
>>> Thanks,
>>>
>>> graham
>>>
>>>
>>>
>>>
>>>
>>> RIPE Info
>>>
>>> *1* RRCs see *1* peers announcing *208.115.136.0/23
>>> * originated by *AS32703*
>>> 
>>>
>>> · ▼RRC00 in *Amsterdam, Netherlands* sees *1* ASN orginating 
>>> *208.115.136.0/23
>>> *.AS32703
>>>
>>> o▼*AS32703
>>> *
>>>  is
>>> seen as the origin by *1* peer.192.102.254.1
>>>
>>> §  ▼*192.102.254.1
>>> *
>>>  is
>>> announcing route *AS395152*
>>> 
>>>  *AS63297*
>>> 
>>>  *AS6327*
>>> 
>>>  *AS36280*
>>> 
>>> *AS32703*
>>> 
>>> .
>>>
>>> §  Origin: IGP
>>>
>>> §  Next Hop: 192.102.254.1
>>>
>>> §  Peer: 192.102.254.1
>>>
>>> §  Community: 63297:1000
>>>
>>> §  AS Path: 395152 63297 6327 36280 32703
>>>
>>> §  Last Updated: 2019-03-27T17:17:19
>>>
>>>
>>>
>>>
>>>
>>> Route-views
>>>
>>> route-views.chicago.routeviews.org
>>> >
>>> show ip bgp 208.115.136.0
>>>
>>> BGP routing table entry for 208.115.136.0/23
>>>
>>> Paths: (1 available, best #1, table Default-IP-Routing-Table)
>>>
>>>   Not advertised to any peer
>>>
>>>   32709 32703
>>>
>>> 208.115.136.134 from 208.115.136.134 (63.134.128.248)
>>>
>>>   Origin IGP, localpref 100, valid, external, best
>>>
>>>   AddPath ID: RX 0, TX 64414249
>>>
>>>   Last update: Wed Mar 27 17:16:09 2019
>>>
>>

Re: Advertisement of Equinix Chicago IX Subnet

[Eric Dugas]

I have a policy applied to my upstreams and peers to deny the IXP's LANs
were connected to. I don't think of any reason to learn these routes from
someone else's network.

On Wed, Mar 27, 2019 at 7:44 PM Cummings, Chris  wrote:

> Not too sure about your topology, but I’ve had something similar bite me,
> so we typically put a prefix list inbound to deny receiving our internal
> prefixes from our peers. This probably doesn’t work as well if your network
> is less “eyeballish” than ours, however.
>
> /chris
>
>
>
> On Wed, Mar 27, 2019 at 4:37 PM -0500, "Graham Johnston" <
> johnst...@westmancom.com> wrote:
>
> This afternoon at around 12:17 central time today we began learning the
>> subnet for the Equinix IX in Chicago via a transit provider; we are on the
>> IX as well. The subnet in question is 208.115.136.0/23. Using
>> stat.ripe.net
>> 
>> I can see that this subnet is also being learned by others, see the snip
>> below. On our network this caused a nasty routing loop until we figured out
>> what was wrong. My current best understanding is that because the route was
>> learned via eBGP it trumped the OSPF learned route. As soon as I filtered
>> the advertisement from my transit provider everything returned to normal.
>> What am I doing that isn’t best practices that would have prevented this?
>>
>>
>>
>> Thanks,
>>
>> graham
>>
>>
>>
>>
>>
>> RIPE Info
>>
>> *1* RRCs see *1* peers announcing *208.115.136.0/23
>> * originated by *AS32703*
>> 
>>
>> · ▼RRC00 in *Amsterdam, Netherlands* sees *1* ASN orginating 
>> *208.115.136.0/23
>> *.AS32703
>>
>> o▼*AS32703
>> *
>>  is
>> seen as the origin by *1* peer.192.102.254.1
>>
>> §  ▼*192.102.254.1
>> *
>>  is
>> announcing route *AS395152*
>> 
>>  *AS63297*
>> 
>>  *AS6327*
>> 
>>  *AS36280*
>> 
>> *AS32703*
>> 
>> .
>>
>> §  Origin: IGP
>>
>> §  Next Hop: 192.102.254.1
>>
>> §  Peer: 192.102.254.1
>>
>> §  Community: 63297:1000
>>
>> §  AS Path: 395152 63297 6327 36280 32703
>>
>> §  Last Updated: 2019-03-27T17:17:19
>>
>>
>>
>>
>>
>> Route-views
>>
>> route-views.chicago.routeviews.org
>> >
>> show ip bgp 208.115.136.0
>>
>> BGP routing table entry for 208.115.136.0/23
>>
>> Paths: (1 available, best #1, table Default-IP-Routing-Table)
>>
>>   Not advertised to any peer
>>
>>   32709 32703
>>
>> 208.115.136.134 from 208.115.136.134 (63.134.128.248)
>>
>>   Origin IGP, localpref 100, valid, external, best
>>
>>   AddPath ID: RX 0, TX 64414249
>>
>>   Last update: Wed Mar 27 17:16:09 2019
>>
>

Re: Advertisement of Equinix Chicago IX Subnet

[Cummings, Chris]

Not too sure about your topology, but I’ve had something similar bite me, so we 
typically put a prefix list inbound to deny receiving our internal prefixes 
from our peers. This probably doesn’t work as well if your network is less 
“eyeballish” than ours, however.

/chris



On Wed, Mar 27, 2019 at 4:37 PM -0500, "Graham Johnston" 
mailto:johnst...@westmancom.com>> wrote:

This afternoon at around 12:17 central time today we began learning the subnet 
for the Equinix IX in Chicago via a transit provider; we are on the IX as well. 
The subnet in question is 208.115.136.0/23. Using 
stat.ripe.net
 I can see that this subnet is also being learned by others, see the snip 
below. On our network this caused a nasty routing loop until we figured out 
what was wrong. My current best understanding is that because the route was 
learned via eBGP it trumped the OSPF learned route. As soon as I filtered the 
advertisement from my transit provider everything returned to normal. What am I 
doing that isn’t best practices that would have prevented this?

Thanks,
graham


RIPE Info
1 RRCs see 1 peers announcing 208.115.136.0/23 originated by 
AS32703

· ▼RRC00 in Amsterdam, Netherlands sees 1 ASN orginating 
208.115.136.0/23.AS32703

o
▼AS32703
 is seen as the origin by 1 peer.192.102.254.1

§  
▼192.102.254.1
 is announcing route 
AS395152
 
AS63297
 
AS6327
 
AS36280AS32703.

§  Origin: IGP

§  Next Hop: 192.102.254.1

§  Peer: 192.102.254.1

§  Community: 63297:1000

§  AS Path: 395152 63297 6327 36280 32703

§  Last Updated: 2019-03-27T17:17:19


Route-views
route-views.chicago.routeviews.org>
 show ip bgp 208.115.136.0
BGP routing table entry for 208.115.136.0/23
Paths: (1 available, best #1, table Default-IP-Routing-Table)
  Not advertised to any peer
  32709 32703
208.115.136.134 from 208.115.136.134 (63.134.128.248)
  Origin IGP, localpref 100, valid, external, best
  AddPath ID: RX 0, TX 64414249
  Last update: Wed Mar 27 17:16:09 2019

RE: Advertisement of Equinix Chicago IX Subnet

[Graham Johnston]

Thank you Nick.

Graham Johnston
Manager, Network Services
Westman Communications Group
1906 Park Avenue | Brandon, MB | R7B 0R9
204-717-2829 |    
johnst...@westmancom.com



        

-Original Message-
From: Nick Hilliard  
Sent: March 27, 2019 4:50 PM
To: Graham Johnston 
Cc: nanog@nanog.org
Subject: Re: Advertisement of Equinix Chicago IX Subnet

CAUTION: This email is from an external source. Do not click links or open 
attachments unless you recognize the sender and know the content is safe.

Graham Johnston wrote on 27/03/2019 21:36:
> What am I doing that isn't best practices that would have prevented this?

you're setting the next-hop of the prefixes learned at the IXP to be
your own IP address from the IXP subnet (i.e. 208.115.136.0/23).

When your routers learn this address from an external source, that is
preferred to your internal OSPF route.  Ergo your IX traffic is sent out
via transit.

There are two things you should do:

1. change the bgp distance for ebgp to be higher than all your IGPs.  On
a cisco router, you would use something like:

router bgp xxx
  address-family ipv4
   distance bgp 200 200 200
  address-family ipv6
   distance bgp 200 200 200

2. use next-hop-self on internal ibgp sessions to ensure that when you
redistribute the eBGP routes learned from your IX towards the internals
of your network, the next-hop address is set to be the loopback address
of your peering router.  I.e. you remove the requirement for your
internal network to know anything about the IXP address range.

Nick

Re: Advertisement of Equinix Chicago IX Subnet

[Nick Hilliard]

Graham Johnston wrote on 27/03/2019 21:36:

What am I doing that isn’t best practices that would have prevented this?


you're setting the next-hop of the prefixes learned at the IXP to be  
your own IP address from the IXP subnet (i.e. 208.115.136.0/23).


When your routers learn this address from an external source, that is  
preferred to your internal OSPF route.  Ergo your IX traffic is sent out  
via transit.


There are two things you should do:

1. change the bgp distance for ebgp to be higher than all your IGPs.  On  
a cisco router, you would use something like:


router bgp xxx
 address-family ipv4
  distance bgp 200 200 200
 address-family ipv6
  distance bgp 200 200 200

2. use next-hop-self on internal ibgp sessions to ensure that when you  
redistribute the eBGP routes learned from your IX towards the internals  
of your network, the next-hop address is set to be the loopback address  
of your peering router.  I.e. you remove the requirement for your  
internal network to know anything about the IXP address range.


Nick

Advertisement of Equinix Chicago IX Subnet

[Graham Johnston]

This afternoon at around 12:17 central time today we began learning the subnet 
for the Equinix IX in Chicago via a transit provider; we are on the IX as well. 
The subnet in question is 208.115.136.0/23. Using stat.ripe.net I can see that 
this subnet is also being learned by others, see the snip below. On our network 
this caused a nasty routing loop until we figured out what was wrong. My 
current best understanding is that because the route was learned via eBGP it 
trumped the OSPF learned route. As soon as I filtered the advertisement from my 
transit provider everything returned to normal. What am I doing that isn’t best 
practices that would have prevented this?

Thanks,
graham


RIPE Info
1 RRCs see 1 peers announcing 208.115.136.0/23 originated by 
AS32703

* ▼RRC00 in Amsterdam, Netherlands sees 1 ASN orginating 
208.115.136.0/23.AS32703

o▼AS32703 is seen as the origin by 1 
peer.192.102.254.1

§  ▼192.102.254.1 is announcing route 
AS395152 AS63297 
AS6327 
AS36280AS32703.

§  Origin: IGP

§  Next Hop: 192.102.254.1

§  Peer: 192.102.254.1

§  Community: 63297:1000

§  AS Path: 395152 63297 6327 36280 32703

§  Last Updated: 2019-03-27T17:17:19


Route-views
route-views.chicago.routeviews.org> show ip bgp 208.115.136.0
BGP routing table entry for 208.115.136.0/23
Paths: (1 available, best #1, table Default-IP-Routing-Table)
  Not advertised to any peer
  32709 32703
208.115.136.134 from 208.115.136.134 (63.134.128.248)
  Origin IGP, localpref 100, valid, external, best
  AddPath ID: RX 0, TX 64414249
  Last update: Wed Mar 27 17:16:09 2019