Re: [c-nsp] Default Route recalculated every 60 seconds.

[Bradley Ordner]

Just thought I would update this thread, the carrier ended up labbing this for 
us and found it did not occur in the lab. The only difference was the IOS on 
the CE router and only receiving a default, not a partial table and default.

I still couldn't believe it, so we updated our router to 16.09.05 and the 
default is stable.

We were originally on asr1001x-universalk9.16.06.04.SPA.bin.

I will take a packet capture (on the router) again and check if the default 
update is coming every sixty seconds.

Brad



From: Gert Doering
Sent: Tuesday, April 21, 2020 7:10 AM
To: Bradley Ordner
Cc: Gert Doering; James Bensley; cisco-nsp@puck.nether.net
Subject: Re: [c-nsp] Default Route recalculated every 60 seconds.

Hi,

On Mon, Apr 20, 2020 at 08:54:27PM +, Bradley Ordner wrote:
> Thanks Gert, I will now ask them to do packet capture on their side and see 
> if they are advertising this default to any other customer every 60 seconds.
>
> Something else I noticed, we only accept routes less than or equal to /18. I 
> noticed that many updates come in, for different prefixes. I can???t see how 
> the Internet could be that unstable unless there is something wrong with 
> their network. Wonder what is the norm when seeing so many prefixes change.

The Internet is huge - 70.000 networks(!) connected together.  Things
are rebuilt and changed all over the place all the time, and links and
devices fail and get repaired all over the time.

So yes, there's a constant stream of BGP updates.

Google for Geoff Huston.  He's done a number of very good presentation
on the dynamics of BGP updates over time.

gert
--
"If was one thing all people took for granted, was conviction that if you
 feed honest figures into a computer, honest figures come out. Never doubted
 it myself till I met a computer with a sense of humor."
 Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany g...@greenie.muc.de
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] Default Route recalculated every 60 seconds.

[Bradley Ordner]

Thanks Gert, I will now ask them to do packet capture on their side and see if 
they are advertising this default to any other customer every 60 seconds. 

Something else I noticed, we only accept routes less than or equal to /18. I 
noticed that many updates come in, for different prefixes. I can’t see how the 
Internet could be that unstable unless there is something wrong with their 
network. Wonder what is the norm when seeing so many prefixes change. 

Brad




Sent from my iPhone

> On 20 Apr 2020, at 7:55 pm, Gert Doering  wrote:
> 
> Hi,
> 
>> On Mon, Apr 20, 2020 at 09:36:55AM +0000, Bradley Ordner wrote:
>> They have told me they have no other issues with other customers and same 
>> config, but this could be a bug between different IOS versions because I am 
>> running IOS-XE and they may be running XR as they have a ASR9K.
> 
> Strictly speaking, there is no "issue", except that the counter for
> "how old is the route?" on your side is being reset every minute.
> 
> Packet forwarding works, routing is stable, no CPU churn.
> 
> WRT "bugs between different IOS versions" - please read what I wrote
> before: frequent reannouncements of a single route *can not* be triggered
> by anything on your side.  There is nothing in the BGP protocol which 
> would enable this.  (If it happens for *all* routes, it could be a 
> soft reconfig request going awry, but this not what you see)
> 
> gert
> 
> -- 
> "If was one thing all people took for granted, was conviction that if you 
> feed honest figures into a computer, honest figures come out. Never doubted 
> it myself till I met a computer with a sense of humor."
> Robert A. Heinlein, The Moon is a Harsh Mistress
> 
> Gert Doering - Munich, Germany g...@greenie.muc.de
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] Default Route recalculated every 60 seconds.

[Bradley Ordner]

Hi James,

Interesting you mention this, as someone else on Cisco Community Page was 
interested in the IOS version. Possibly once this design is finished the 
default may come from somewhere else, but I don't think I will be there when it 
happens.

They have told me they have no other issues with other customers and same 
config, but this could be a bug between different IOS versions because I am 
running IOS-XE and they may be running XR as they have a ASR9K.

I am going to ask the question and see what happens. Thanks for the info.

Brad




From: James Bensley 
Sent: Monday, 20 April 2020 6:30 PM
To: Bradley Ordner ; cisco-nsp@puck.nether.net 

Subject: Re: [c-nsp] Default Route recalculated every 60 seconds.

On Sat, 18 Apr 2020 at 07:11, Bradley Ordner  wrote:
>
> I am about to leave an Enterprise environment due to Pre Covid-19 redundancy 
> and I just need to find the root cause of this issue before I leave.
>
> We recently built a Layer 2 Circuit over a Providers SDN Backbone so we could 
> get a 2Gb Internet link. We peered with this neighbor and filtered a partial 
> table, so we get about 3 routes. For some reason, every 30 seconds the 
> default route uptime resets to 00:00 in the routing table. I spoke with 
> Carrier, they made a few changes and one was the BGP advertisement timer. It 
> is now set to 60 seconds and now the default route resets every 60 seconds.
>
> The carrier, keeps blaming my side so I opened a Cisco TAC case and they 
> haven't got around to looking at it yet, probably because it really sounds 
> like it is the carrier side. I took some packet captures and indeed every 60 
> seconds an update with the default is sent. Our router constantly accepts 
> this, recalculates and enters it into the routing table.
>
> I can't seem to figure out if this is some type of bug or not. The router has 
> been rebooted and is due for IOS upgrade shortly, but wanted to see if anyone 
> has seen this or point me in the right direction.
>
> Thanks
>
> Brad

Hi Brad,

Do you know if your provider uses IOS-XR on their PE? There was a bug
a couple of years ago in XR (I've searched on cisco.com but can't find
the BugID right now) in which XR was re-advertising the default route
every 30 or 60 seconds. We had it, and if I recall correctly it wasn't
being withdrawn, just a new BGP UPDATE was sent to supersede the
existing route, so as other posters have said, packet capture the BGP
TCP packets with your provider or use some "debug bgp" commands to see
whats really going on.

Also, maybe reconsider if need them to advertise a default route if
you're getting a partial table from them.

Cheers,
James.
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] Default Route recalculated every 60 seconds.

[Bradley Ordner]

Thanks Robert & Gert. I will go back to carrier as I have checked both changing 
attributes and my route to peer.

I have tested this in GNS3 with different IOS as well, and everything i stable.

Brad


From: Robert Raszuk 
Sent: Saturday, 18 April 2020 8:26 PM
To: Bradley Ordner 
Cc: cisco-nsp@puck.nether.net 
Subject: Re: [c-nsp] Default Route recalculated every 60 seconds.

Hi Bradley,

>From my cisco days I recall that you should not be seeing RIB being updated 
>over and over with the same route even if BGP keeps sending you implicit 
>withdraws in the form of new BGP UPDATEs. Of course I will not tell you if the 
>above is still identical today on all XE, NX & XR :)

However the above only works if your next hop is stable. If you are recursing 
your routes over BGP (double recursion) then yes you will see this churn going 
on into RIB.

But there is simple fix/test - just set a static route matching next hop of 
received BGP prefixes towards your eBGP peer with interface and IP address and 
see if it helps.

Cheers,
R.



On Sat, Apr 18, 2020 at 8:10 AM Bradley Ordner 
mailto:bradin...@hotmail.com>> wrote:
I am about to leave an Enterprise environment due to Pre Covid-19 redundancy 
and I just need to find the root cause of this issue before I leave.

We recently built a Layer 2 Circuit over a Providers SDN Backbone so we could 
get a 2Gb Internet link. We peered with this neighbor and filtered a partial 
table, so we get about 3 routes. For some reason, every 30 seconds the 
default route uptime resets to 00:00 in the routing table. I spoke with 
Carrier, they made a few changes and one was the BGP advertisement timer. It is 
now set to 60 seconds and now the default route resets every 60 seconds.

The carrier, keeps blaming my side so I opened a Cisco TAC case and they 
haven't got around to looking at it yet, probably because it really sounds like 
it is the carrier side. I took some packet captures and indeed every 60 seconds 
an update with the default is sent. Our router constantly accepts this, 
recalculates and enters it into the routing table.

I can't seem to figure out if this is some type of bug or not. The router has 
been rebooted and is due for IOS upgrade shortly, but wanted to see if anyone 
has seen this or point me in the right direction.

Thanks

Brad


___
cisco-nsp mailing list  
cisco-nsp@puck.nether.net<mailto:cisco-nsp@puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[c-nsp] Default Route recalculated every 60 seconds.

[Bradley Ordner]

I am about to leave an Enterprise environment due to Pre Covid-19 redundancy 
and I just need to find the root cause of this issue before I leave.

We recently built a Layer 2 Circuit over a Providers SDN Backbone so we could 
get a 2Gb Internet link. We peered with this neighbor and filtered a partial 
table, so we get about 3 routes. For some reason, every 30 seconds the 
default route uptime resets to 00:00 in the routing table. I spoke with 
Carrier, they made a few changes and one was the BGP advertisement timer. It is 
now set to 60 seconds and now the default route resets every 60 seconds.

The carrier, keeps blaming my side so I opened a Cisco TAC case and they 
haven't got around to looking at it yet, probably because it really sounds like 
it is the carrier side. I took some packet captures and indeed every 60 seconds 
an update with the default is sent. Our router constantly accepts this, 
recalculates and enters it into the routing table.

I can't seem to figure out if this is some type of bug or not. The router has 
been rebooted and is due for IOS upgrade shortly, but wanted to see if anyone 
has seen this or point me in the right direction.

Thanks

Brad


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] Nexus 5k ISSU

[Bradley Ordner]

I have done this on the 7K and I don’t trust it anymore. I had OSPF adjacencies 
go down when the supervisor failed over. 

We plan for outage now, we only have one per DC :( and do it manually. 

Even running the ISSU commands to see If the device was ready failed sometimes. 

What I would suggest, which we tried as well to no effect is to reboot the 
supervisors or what ever the 5k brains are called one by one before trying 
ISSU. That way it’s fresh.

Brad Ordner



Sent from my iPhone

> On 2 Nov 2019, at 9:19 am, harbor235  wrote:
> 
> Hi everyone,
> 
> What are your experiences with Nexus5K ISSU and VPCs.  Do you see service
> interruptions? ISSU is never quite ISSU. During role changes and/or VPCs
> reforming I see short duration losses. Is this standard?
> 
> 
> Mike
> ___
> cisco-nsp mailing list  cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] Cisco ASR RIB Failure ?

[Bradley Ordner]

You should have the command ‘show ip bgp rib-failure’ which will point you in 
the right direction. 

Brad Ordner 


Sent from my iPhone

> On 5 Dec 2018, at 6:05 pm, Lukas Tribus  wrote:
> 
> Hi,
> 
> 
>> On Wed, 5 Dec 2018 at 07:58, Olivier CALVANO  wrote:
>> 
>> Hi
>> 
>> On all of my router, i have :
>> 
>> ASR1002.BLD1#sh ip bgp 172.16.0.1
>> BGP routing table entry for  172.16.0.1/32, version 1184149
>> Paths: (2 available, best #1, table default, not advertised to EBGP peer,
>> RIB-failure(17))
>> 
>> 
>> how can I do to find the problem of "RIB-failure(17)"
> 
> show ip route 172.16.0.1
> 
> You probably have a route with a lower administrative distance, than
> EBGP (20), for example a static route.
> 
> 
> 
> lukas
> ___
> cisco-nsp mailing list  cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[c-nsp] Qos Statistics on the 7K

[Bradley Ordner]

Hi,

This may have been asked before, even on Cisco Support Community I have an 
answer but it doesn't seem to be working for me.

We have a Layer 3 port with a QoS policy for marking traffic inbound. I have 
added the 'statistics per-entry' command in our ACL but I do not see any hits. 
When checking the policy and queueing, I see traffic being matched.

We are only marking inbound on this port, is it not supported or do I have a 
bug? I am on version - 7.2(0)D1(1)

Match: access-group QOSACL- BLAH
46082768 packets
  set dscp 56

Thanks

Brad Ordner

___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] Nexus 7700 sup2e Upgrade

[Bradley Ordner]

Hi, 

I’m interested in how you go, we are planning a cold boot upgrade from 7.2.0(1) 
(D1) to 8.2. We seem to have some BFD and Mac sec bugs. 

We attempted an ISSU update a few months back to a later 7 code and it failed. 
We only have one 7K chassis per DC so can’t risk another crash. 

Do you need to update any EPLDs? the only new feature we want to use is OTV 
loopbacks as join interfaces.

Brad

Sent from my iPhone

> On 3 Aug 2018, at 4:26 pm, Tristan Gulyas  wrote:
> 
> Hi all,
> 
> We're testing 8.2 for the same reason (security fixes).  So far, so good!  
> Multi-campus LAN, 7706s for distribution, border and core routing.
> 
> Tristan
> 
>> On 1 Aug 2018, at 6:12 am, Charles Spurgeon  
>> wrote:
>> 
>> * Nick Griffin  [2018-07-30 10:25:21 -0500]:
>>> Looking to upgrade some 7ks from 6.2.12 to something 7.2 or 7.3 to
>> support the peering of layer 3 devices across vpc port
>> channels. Looking to see what code versions others are using that
>> have proven to be stable.
>> 
>> We've been running 7.3 but are moving to 8.2 to resolve the vulns
>> announced here:
>> https://tools.cisco.com/security/center/viewErp.x?alertId=ERP-67770
>> 
>> Some of the vulns listed were fixed as of 7.3(2)D1(2) and some aren't
>> fixed until 8.1/8.2 code.
>> 
>> 7.3(2)D1(2) has been stable for us for 198 days of runtime on a pair
>> of 7710s with sup2e in a data center with VPC (traditional multi-tier
>> design).
>> 
>> Currently we have been smoke testing 8.2 in the lab to replace the 7.3
>> code on the next maintenance window.
>> 
>> -Charles
>> ___
>> cisco-nsp mailing list  cisco-nsp@puck.nether.net
>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> archive at http://puck.nether.net/pipermail/cisco-nsp/
> 
> ___
> cisco-nsp mailing list  cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] Nexus 7k Upgrade Path

[Bradley Ordner]

I’ll be honest I was pretty confident after reading the config guide on the 
website. This was going to happen this weekend but I wanted more time to 
prepare. 

The config guide does say ‘one command does it all’ which sounded great! I 
wasn’t aware of some of the issues people have seen such as config missing. 

It’s impossible to take this thing out of service as well so I need a good 
outage to perform this work at two data centres. Device has been online for 
almost two years as well. 

My only small concern was Cisco’s recommended and stable release is 6.x. I 
thought they would have had some thing closer to 8.

Brad Ordner 

Sent from my iPhone

> On 24 Feb 2018, at 10:17 am, Hunter Fuller  wrote:
> 
> On Fri, Feb 23, 2018 at 8:06 AM Justin M. Streiner 
> wrote:
> 
>> Vendors also sometimes conflate "ISSU" and "hitless", or their
>> documentation doesn't always make it clear that an ISSU carries the
>> potential of outages.
> 
> 
> For what it is worth - there is a NX-OS command for checking whether an
> ISSU will be hitless: "show install all impact ?" will show you what you
> need to know.
> 
> We don't run much Nexus stuff, but we did upgrade our Nexus 7010 from
> version 4.something all the way to 7.2 with only ISSU. We had to do some
> careful planning, and some ISSU did fail, but the failure and rollback was
> just as hitless as the successes, and it told us what needed to be
> corrected for the future.
> 
> So far so good, with this strategy. I am very surprised to hear people
> talking about their problems with the ISSU process. I could not be happier
> with it.
> 
> # show system uptime
> System start time:  Sat Dec 20 17:54:34 2014
> System uptime:  1161 days, 4 hours, 36 minutes, 22 seconds
> 
> -- 
> 
> --
> Hunter Fuller
> Network Engineer
> VBH Annex B-5
> +1 256 824 5331
> 
> Office of Information Technology
> The University of Alabama in Huntsville
> Systems and Infrastructure
> ___
> cisco-nsp mailing list  cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] Nexus 7k Upgrade Path

[Bradley Ordner]

Thanks, I’ll take this all onboard. The reason for the upgrade is we wanted to 
start using Loopbacks for OTV and finish a DR design that seems to have not 
been completed. 



Sent from my iPhone

> On 23 Feb 2018, at 5:58 pm, Pete Templin <peteli...@templin.org> wrote:
> 
> I would even go so far as to:
> 
> load system/kickstart files
> isolate the box (shutdown all ports)
> power-cycle the box, let it boot into the new code
> perform EPLD updates on all cards
> run the ISSU command to ensure all of the little microcode thingies (PSUs, 
> fans, etc.) are covered
> unisolate the box
> 
>> On 2/22/18 11:44 PM, Pavel Skovajsa wrote:
>> Definitely not a stupid question. While the double ISSU would work we
>> generally would not do it for big jumps like that.
>> 
>> The problem is that the whole procedure tended to be buggy so we are too
>> afraid. Not speaking about crazy bugs we ran into half year later because
>> "triggered by previous issu upgrade" and we needed to reload anyway.
>> 
>> So - our recomendation for jumps like this - load the system a kickstart
>> files and reload the box? Ideally power cyclethere were fw bugs that
>> needed hard reboot to fix...
>> 
>> -pavel
>> 
>> 
>> 
>> Dňa 23. 2. 2018 7:34 používateľ "Justin M. Streiner" <
>> strei...@cluebyfour.org> napísal:
>> 
>> On Fri, 23 Feb 2018, Bradley Ordner wrote:
>> 
>> We have a Nexus 7K with two SUP2Es. We need to get to software version
>>> 8.1(2). It says that you can't double hop to a software version without an
>>> outage. Although I have found the following -
>>> ISSU from 7.2(0)D1(1) to 7.3(1)D1(1) then to 8.1(2).
>>> We currently are on 7.2(0)D1(1) according to the doco I should be able to
>>> upgrade as each version can ISSU to the next?
>>> Has anyone performed this before?
>>> I have posted this on Cisco Support Community, with no response so either
>>> it is a stupid question or no one has done it before.
>>> 
>> I haven't had to do a double-hop upgrade in a while, but my past experience
>> with ISSUs on the Nexus 7K has been mixed. Sometimes the 7K ecosystem
>> benefits from a full reboot.  Also, keep in mind that if any of the EPLDs
>> on your linecards need to be upgraded, the affected linecards will have to
>> take some amount of outage.  How much of an impact such an outage would
>> have depends entirely on your network design.
>> 
>> jms
>> 
>> ___
>> cisco-nsp mailing list  cisco-nsp@puck.nether.net
>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>> ___
>> cisco-nsp mailing list  cisco-nsp@puck.nether.net
>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> archive at http://puck.nether.net/pipermail/cisco-nsp/
> 
> ___
> cisco-nsp mailing list  cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[c-nsp] Nexus 7k Upgrade Path

[Bradley Ordner]

Hi,


Only been on the list for a few months but found it very informative. I had a 
question regarding the Nexus 7K ISSU upgrades.


We have a Nexus 7K with two SUP2Es. We need to get to software version 8.1(2). 
It says that you can't double hop to a software version without an outage. 
Although I have found the following -


ISSU from 7.2(0)D1(1) to 7.3(1)D1(1) then to 8.1(2).



We currently are on 7.2(0)D1(1) according to the doco I should be able to 
upgrade as each version can ISSU to the next?



Has anyone performed this before?


I have posted this on Cisco Support Community, with no response so either it is 
a stupid question or no one has done it before.


Thanks


Brad Ordner

___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/