Re: [c-nsp] Default Route recalculated every 60 seconds.
Just thought I would update this thread, the carrier ended up labbing this for us and found it did not occur in the lab. The only difference was the IOS on the CE router and only receiving a default, not a partial table and default. I still couldn't believe it, so we updated our router to 16.09.05 and the default is stable. We were originally on asr1001x-universalk9.16.06.04.SPA.bin. I will take a packet capture (on the router) again and check if the default update is coming every sixty seconds. Brad From: Gert Doering Sent: Tuesday, April 21, 2020 7:10 AM To: Bradley Ordner Cc: Gert Doering; James Bensley; cisco-nsp@puck.nether.net Subject: Re: [c-nsp] Default Route recalculated every 60 seconds. Hi, On Mon, Apr 20, 2020 at 08:54:27PM +, Bradley Ordner wrote: > Thanks Gert, I will now ask them to do packet capture on their side and see > if they are advertising this default to any other customer every 60 seconds. > > Something else I noticed, we only accept routes less than or equal to /18. I > noticed that many updates come in, for different prefixes. I can???t see how > the Internet could be that unstable unless there is something wrong with > their network. Wonder what is the norm when seeing so many prefixes change. The Internet is huge - 70.000 networks(!) connected together. Things are rebuilt and changed all over the place all the time, and links and devices fail and get repaired all over the time. So yes, there's a constant stream of BGP updates. Google for Geoff Huston. He's done a number of very good presentation on the dynamics of BGP updates over time. gert -- "If was one thing all people took for granted, was conviction that if you feed honest figures into a computer, honest figures come out. Never doubted it myself till I met a computer with a sense of humor." Robert A. Heinlein, The Moon is a Harsh Mistress Gert Doering - Munich, Germany g...@greenie.muc.de ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Default Route recalculated every 60 seconds.
Hi, On Mon, Apr 20, 2020 at 08:54:27PM +, Bradley Ordner wrote: > Thanks Gert, I will now ask them to do packet capture on their side and see > if they are advertising this default to any other customer every 60 seconds. > > Something else I noticed, we only accept routes less than or equal to /18. I > noticed that many updates come in, for different prefixes. I can???t see how > the Internet could be that unstable unless there is something wrong with > their network. Wonder what is the norm when seeing so many prefixes change. The Internet is huge - 70.000 networks(!) connected together. Things are rebuilt and changed all over the place all the time, and links and devices fail and get repaired all over the time. So yes, there's a constant stream of BGP updates. Google for Geoff Huston. He's done a number of very good presentation on the dynamics of BGP updates over time. gert -- "If was one thing all people took for granted, was conviction that if you feed honest figures into a computer, honest figures come out. Never doubted it myself till I met a computer with a sense of humor." Robert A. Heinlein, The Moon is a Harsh Mistress Gert Doering - Munich, Germany g...@greenie.muc.de signature.asc Description: PGP signature ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Default Route recalculated every 60 seconds.
Thanks Gert, I will now ask them to do packet capture on their side and see if they are advertising this default to any other customer every 60 seconds. Something else I noticed, we only accept routes less than or equal to /18. I noticed that many updates come in, for different prefixes. I can’t see how the Internet could be that unstable unless there is something wrong with their network. Wonder what is the norm when seeing so many prefixes change. Brad Sent from my iPhone > On 20 Apr 2020, at 7:55 pm, Gert Doering wrote: > > Hi, > >> On Mon, Apr 20, 2020 at 09:36:55AM +, Bradley Ordner wrote: >> They have told me they have no other issues with other customers and same >> config, but this could be a bug between different IOS versions because I am >> running IOS-XE and they may be running XR as they have a ASR9K. > > Strictly speaking, there is no "issue", except that the counter for > "how old is the route?" on your side is being reset every minute. > > Packet forwarding works, routing is stable, no CPU churn. > > WRT "bugs between different IOS versions" - please read what I wrote > before: frequent reannouncements of a single route *can not* be triggered > by anything on your side. There is nothing in the BGP protocol which > would enable this. (If it happens for *all* routes, it could be a > soft reconfig request going awry, but this not what you see) > > gert > > -- > "If was one thing all people took for granted, was conviction that if you > feed honest figures into a computer, honest figures come out. Never doubted > it myself till I met a computer with a sense of humor." > Robert A. Heinlein, The Moon is a Harsh Mistress > > Gert Doering - Munich, Germany g...@greenie.muc.de ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Default Route recalculated every 60 seconds.
Hi, On Mon, Apr 20, 2020 at 09:36:55AM +, Bradley Ordner wrote: > They have told me they have no other issues with other customers and same > config, but this could be a bug between different IOS versions because I am > running IOS-XE and they may be running XR as they have a ASR9K. Strictly speaking, there is no "issue", except that the counter for "how old is the route?" on your side is being reset every minute. Packet forwarding works, routing is stable, no CPU churn. WRT "bugs between different IOS versions" - please read what I wrote before: frequent reannouncements of a single route *can not* be triggered by anything on your side. There is nothing in the BGP protocol which would enable this. (If it happens for *all* routes, it could be a soft reconfig request going awry, but this not what you see) gert -- "If was one thing all people took for granted, was conviction that if you feed honest figures into a computer, honest figures come out. Never doubted it myself till I met a computer with a sense of humor." Robert A. Heinlein, The Moon is a Harsh Mistress Gert Doering - Munich, Germany g...@greenie.muc.de signature.asc Description: PGP signature ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Default Route recalculated every 60 seconds.
Hi James, Interesting you mention this, as someone else on Cisco Community Page was interested in the IOS version. Possibly once this design is finished the default may come from somewhere else, but I don't think I will be there when it happens. They have told me they have no other issues with other customers and same config, but this could be a bug between different IOS versions because I am running IOS-XE and they may be running XR as they have a ASR9K. I am going to ask the question and see what happens. Thanks for the info. Brad From: James Bensley Sent: Monday, 20 April 2020 6:30 PM To: Bradley Ordner ; cisco-nsp@puck.nether.net Subject: Re: [c-nsp] Default Route recalculated every 60 seconds. On Sat, 18 Apr 2020 at 07:11, Bradley Ordner wrote: > > I am about to leave an Enterprise environment due to Pre Covid-19 redundancy > and I just need to find the root cause of this issue before I leave. > > We recently built a Layer 2 Circuit over a Providers SDN Backbone so we could > get a 2Gb Internet link. We peered with this neighbor and filtered a partial > table, so we get about 3 routes. For some reason, every 30 seconds the > default route uptime resets to 00:00 in the routing table. I spoke with > Carrier, they made a few changes and one was the BGP advertisement timer. It > is now set to 60 seconds and now the default route resets every 60 seconds. > > The carrier, keeps blaming my side so I opened a Cisco TAC case and they > haven't got around to looking at it yet, probably because it really sounds > like it is the carrier side. I took some packet captures and indeed every 60 > seconds an update with the default is sent. Our router constantly accepts > this, recalculates and enters it into the routing table. > > I can't seem to figure out if this is some type of bug or not. The router has > been rebooted and is due for IOS upgrade shortly, but wanted to see if anyone > has seen this or point me in the right direction. > > Thanks > > Brad Hi Brad, Do you know if your provider uses IOS-XR on their PE? There was a bug a couple of years ago in XR (I've searched on cisco.com but can't find the BugID right now) in which XR was re-advertising the default route every 30 or 60 seconds. We had it, and if I recall correctly it wasn't being withdrawn, just a new BGP UPDATE was sent to supersede the existing route, so as other posters have said, packet capture the BGP TCP packets with your provider or use some "debug bgp" commands to see whats really going on. Also, maybe reconsider if need them to advertise a default route if you're getting a partial table from them. Cheers, James. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Default Route recalculated every 60 seconds.
On Sat, 18 Apr 2020 at 07:11, Bradley Ordner wrote: > > I am about to leave an Enterprise environment due to Pre Covid-19 redundancy > and I just need to find the root cause of this issue before I leave. > > We recently built a Layer 2 Circuit over a Providers SDN Backbone so we could > get a 2Gb Internet link. We peered with this neighbor and filtered a partial > table, so we get about 3 routes. For some reason, every 30 seconds the > default route uptime resets to 00:00 in the routing table. I spoke with > Carrier, they made a few changes and one was the BGP advertisement timer. It > is now set to 60 seconds and now the default route resets every 60 seconds. > > The carrier, keeps blaming my side so I opened a Cisco TAC case and they > haven't got around to looking at it yet, probably because it really sounds > like it is the carrier side. I took some packet captures and indeed every 60 > seconds an update with the default is sent. Our router constantly accepts > this, recalculates and enters it into the routing table. > > I can't seem to figure out if this is some type of bug or not. The router has > been rebooted and is due for IOS upgrade shortly, but wanted to see if anyone > has seen this or point me in the right direction. > > Thanks > > Brad Hi Brad, Do you know if your provider uses IOS-XR on their PE? There was a bug a couple of years ago in XR (I've searched on cisco.com but can't find the BugID right now) in which XR was re-advertising the default route every 30 or 60 seconds. We had it, and if I recall correctly it wasn't being withdrawn, just a new BGP UPDATE was sent to supersede the existing route, so as other posters have said, packet capture the BGP TCP packets with your provider or use some "debug bgp" commands to see whats really going on. Also, maybe reconsider if need them to advertise a default route if you're getting a partial table from them. Cheers, James. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Default Route recalculated every 60 seconds.
Thanks Robert & Gert. I will go back to carrier as I have checked both changing attributes and my route to peer. I have tested this in GNS3 with different IOS as well, and everything i stable. Brad From: Robert Raszuk Sent: Saturday, 18 April 2020 8:26 PM To: Bradley Ordner Cc: cisco-nsp@puck.nether.net Subject: Re: [c-nsp] Default Route recalculated every 60 seconds. Hi Bradley, >From my cisco days I recall that you should not be seeing RIB being updated >over and over with the same route even if BGP keeps sending you implicit >withdraws in the form of new BGP UPDATEs. Of course I will not tell you if the >above is still identical today on all XE, NX & XR :) However the above only works if your next hop is stable. If you are recursing your routes over BGP (double recursion) then yes you will see this churn going on into RIB. But there is simple fix/test - just set a static route matching next hop of received BGP prefixes towards your eBGP peer with interface and IP address and see if it helps. Cheers, R. On Sat, Apr 18, 2020 at 8:10 AM Bradley Ordner mailto:bradin...@hotmail.com>> wrote: I am about to leave an Enterprise environment due to Pre Covid-19 redundancy and I just need to find the root cause of this issue before I leave. We recently built a Layer 2 Circuit over a Providers SDN Backbone so we could get a 2Gb Internet link. We peered with this neighbor and filtered a partial table, so we get about 3 routes. For some reason, every 30 seconds the default route uptime resets to 00:00 in the routing table. I spoke with Carrier, they made a few changes and one was the BGP advertisement timer. It is now set to 60 seconds and now the default route resets every 60 seconds. The carrier, keeps blaming my side so I opened a Cisco TAC case and they haven't got around to looking at it yet, probably because it really sounds like it is the carrier side. I took some packet captures and indeed every 60 seconds an update with the default is sent. Our router constantly accepts this, recalculates and enters it into the routing table. I can't seem to figure out if this is some type of bug or not. The router has been rebooted and is due for IOS upgrade shortly, but wanted to see if anyone has seen this or point me in the right direction. Thanks Brad ___ cisco-nsp mailing list cisco-nsp@puck.nether.net<mailto:cisco-nsp@puck.nether.net> https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Default Route recalculated every 60 seconds.
Hi Bradley, >From my cisco days I recall that you should not be seeing RIB being updated over and over with the same route even if BGP keeps sending you implicit withdraws in the form of new BGP UPDATEs. Of course I will not tell you if the above is still identical today on all XE, NX & XR :) However the above only works if your next hop is stable. If you are recursing your routes over BGP (double recursion) then yes you will see this churn going on into RIB. But there is simple fix/test - just set a static route matching next hop of received BGP prefixes towards your eBGP peer with interface and IP address and see if it helps. Cheers, R. On Sat, Apr 18, 2020 at 8:10 AM Bradley Ordner wrote: > I am about to leave an Enterprise environment due to Pre Covid-19 > redundancy and I just need to find the root cause of this issue before I > leave. > > We recently built a Layer 2 Circuit over a Providers SDN Backbone so we > could get a 2Gb Internet link. We peered with this neighbor and filtered a > partial table, so we get about 3 routes. For some reason, every 30 > seconds the default route uptime resets to 00:00 in the routing table. I > spoke with Carrier, they made a few changes and one was the BGP > advertisement timer. It is now set to 60 seconds and now the default route > resets every 60 seconds. > > The carrier, keeps blaming my side so I opened a Cisco TAC case and they > haven't got around to looking at it yet, probably because it really sounds > like it is the carrier side. I took some packet captures and indeed every > 60 seconds an update with the default is sent. Our router constantly > accepts this, recalculates and enters it into the routing table. > > I can't seem to figure out if this is some type of bug or not. The router > has been rebooted and is due for IOS upgrade shortly, but wanted to see if > anyone has seen this or point me in the right direction. > > Thanks > > Brad > > > ___ > cisco-nsp mailing list cisco-nsp@puck.nether.net > https://puck.nether.net/mailman/listinfo/cisco-nsp > archive at http://puck.nether.net/pipermail/cisco-nsp/ > ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Default Route recalculated every 60 seconds.
Hi, On Sat, Apr 18, 2020 at 06:06:49AM +, Bradley Ordner wrote: > The carrier, keeps blaming my side so I opened a Cisco TAC case > and they haven't got around to looking at it yet, probably because > it really sounds like it is the carrier side. I took some packet > captures and indeed every 60 seconds an update with the default is > sent. Our router constantly accepts this, recalculates and enters > it into the routing table. If you can see the default route being announced by the carrier, it's (very obviously so) not your router who is causing this (if you see an outgoing route refresh request, and then an update of *everything*, things would be different). There is no mechanism in BGP to tell the other end "of all routes you sent me, please refresh just the default route". Do not let carriers bullshit you into believing "it's all your fault". Now for "why is it sending an update" - it might be attributes changing (due to something oscillating in the carrier network), or just a dumb implementation on their side. Try looking at the attributes for two consecutive updates (metric, communities, ...) and see if something changes. (OTOH, except for academic interest, I would not worry too much about it either - one route recalculation every 60 seconds which leads to the same result "out to that carrier" will not negatively affect your network) gert -- "If was one thing all people took for granted, was conviction that if you feed honest figures into a computer, honest figures come out. Never doubted it myself till I met a computer with a sense of humor." Robert A. Heinlein, The Moon is a Harsh Mistress Gert Doering - Munich, Germany g...@greenie.muc.de signature.asc Description: PGP signature ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
[c-nsp] Default Route recalculated every 60 seconds.
I am about to leave an Enterprise environment due to Pre Covid-19 redundancy and I just need to find the root cause of this issue before I leave. We recently built a Layer 2 Circuit over a Providers SDN Backbone so we could get a 2Gb Internet link. We peered with this neighbor and filtered a partial table, so we get about 3 routes. For some reason, every 30 seconds the default route uptime resets to 00:00 in the routing table. I spoke with Carrier, they made a few changes and one was the BGP advertisement timer. It is now set to 60 seconds and now the default route resets every 60 seconds. The carrier, keeps blaming my side so I opened a Cisco TAC case and they haven't got around to looking at it yet, probably because it really sounds like it is the carrier side. I took some packet captures and indeed every 60 seconds an update with the default is sent. Our router constantly accepts this, recalculates and enters it into the routing table. I can't seem to figure out if this is some type of bug or not. The router has been rebooted and is due for IOS upgrade shortly, but wanted to see if anyone has seen this or point me in the right direction. Thanks Brad ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
