Re: [c-nsp] FIB scale on ASR9001
Hi,
On Thu, Nov 11, 2021 at 03:02:35PM +0100, Lukas Tribus wrote:
> > We're on 6.5.3 ("nothing interesting in newer versions, and still
> > supported").
>
> When I tested RTR on IOS-XR I hit some strange bugs in the RTR client,
> specifically the RTR session would hang in certain scenarios (router
> restart, RTR server reboot, etc), requiring manual intervention with a
> "clear bgp rpki-server 1.2.3.4". It was *a lot* better in more recent
> releases, but 6.5.3 was not part of those better releases.
Yeah. That bug hits here as well, every now and then one of the two
sessions (or both) get stuck. We have a script that walks all our XR
boxes and verifies that they all see the same amount of prefixes on
both sessions, across all boxes... and then alerts, and we go "clear".
Not exactly displaying best of software quality, but more of a minor
nuisance (for us).
gert
--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress
Gert Doering - Munich, Germany g...@greenie.muc.de
signature.asc
Description: PGP signature
___
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FIB scale on ASR9001
On Thu, 11 Nov 2021 at 15:01, Mark Tinka wrote: > > > > On 11/11/21 15:43, Lukas Tribus wrote: > > > For ROV to work reliably it needs to be able to reconsider previously > > rejected invalids, so I would not recommend disabling > > soft-reconfiguartion inbound, instead I'd suggest to set it to always. > > If my router vendor is not automatically applying BGP policy based on > RPKI state, it shouldn't matter what ended up in RIB if I have not set > an explicit policy to act on RPKI state. > > So if a previously-Invalid route now becomes a VRP, and is then good to > go for export toward a neighbor based on existing policy that matches, > why can we not leverage Route Refresh to only cater for that change? I believe with the amount of RAM we have on those boxes nowadays, keeping a copy of everything should be a non-issue. On the other hand, leveraging route-refresh changes your EBGP behaviour, which can trigger remote and local bugs, or, as in your case, trigger humans with most likely a little over-dramatic monitoring. I won't trust other peoples BGP routers and implementations more than I absolutely have to and I don't think my time is well spent arguing with other people about their underdimensioned control plane CPU, oversensitive CPU load monitoring or troubleshooting corner cases in their BGP implementation that trigger bugs in route refresh code. And then the need to explain in a RFO why your network heavily uses route-refresh which triggered that remote bug in the first place, while your competitor didn't change anything in their BGP configuration in the last decade, so "they didn't have any issue with this, only your network has issues''. Those are all rabbit holes that I will gladly trade for a little bit of RAM usage in a heartbeat. Lukas ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FIB scale on ASR9001
On 11/11/21 16:26, Lukas Tribus wrote: Still, the monitoring script makes sense. Yep - lots of monitoring is needed. I'm monitoring some odd behaviour where Fort will, after a few days, grow its delta in the number of VRP's vs. rpki-client. Probably one of the best reasons to run different validators. Mark. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FIB scale on ASR9001
On Thu, 11 Nov 2021 at 15:12, Mark Tinka wrote: > > > > On 11/11/21 16:02, Lukas Tribus wrote: > > > When I tested RTR on IOS-XR I hit some strange bugs in the RTR client, > > specifically the RTR session would hang in certain scenarios (router > > restart, RTR server reboot, etc), requiring manual intervention with a > > "clear bgp rpki-server 1.2.3.4". It was *a lot* better in more recent > > releases, but 6.5.3 was not part of those better releases. I still did > > not have a lot of trust in XR for this reason, so I wrote a NETCONF > > script that would make some sanity checks on the RTR session state on > > XR boxes (nagios friendly): > > > > https://gist.github.com/lukastribus/695c9e780d118755271519d4d3cb54f3 > > > > (minimum number of absolute v4 and v6 VRPs on each RTR server, maximum > > v4/v6 VRP deviation between the RTR servers, RTR servers not in > > "connected" state). > > And now, with the code you are currently running for IOS XR? I haven't been able to reproduce the hung RTR session issue due to reboot/reloads in newer code. 6.7.3 (32-bit)/7.1.3 (64-bit) should be fine. I don't recall if 6.6.X was good or bad, but 6.5 certainly was. Still, the monitoring script makes sense. Lukas ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FIB scale on ASR9001
On 11/11/21 16:02, Lukas Tribus wrote: When I tested RTR on IOS-XR I hit some strange bugs in the RTR client, specifically the RTR session would hang in certain scenarios (router restart, RTR server reboot, etc), requiring manual intervention with a "clear bgp rpki-server 1.2.3.4". It was *a lot* better in more recent releases, but 6.5.3 was not part of those better releases. I still did not have a lot of trust in XR for this reason, so I wrote a NETCONF script that would make some sanity checks on the RTR session state on XR boxes (nagios friendly): https://gist.github.com/lukastribus/695c9e780d118755271519d4d3cb54f3 (minimum number of absolute v4 and v6 VRPs on each RTR server, maximum v4/v6 VRP deviation between the RTR servers, RTR servers not in "connected" state). And now, with the code you are currently running for IOS XR? Mark. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FIB scale on ASR9001
Hello Gert,
On Thu, 11 Nov 2021 at 08:18, Gert Doering wrote:
>
> Hi,
>
> On Thu, Nov 11, 2021 at 08:27:44AM +0200, Mark Tinka wrote:
> > We have nothing against the forwarding performance of the ASR9001. It's
> > the control/management plane that seems to be slowing down (at least for
> > us, anyway) with newer code and a growing Internet DFZ.
>
> "newer code" might be the key issue here - what version are you on?
>
> Our 9001 have been extremely well-behaved in regards to BGP performance
> and robustness. Not as fast as the RSP440, but still well up to the
> job.
>
> We're on 6.5.3 ("nothing interesting in newer versions, and still
> supported").
When I tested RTR on IOS-XR I hit some strange bugs in the RTR client,
specifically the RTR session would hang in certain scenarios (router
restart, RTR server reboot, etc), requiring manual intervention with a
"clear bgp rpki-server 1.2.3.4". It was *a lot* better in more recent
releases, but 6.5.3 was not part of those better releases. I still did
not have a lot of trust in XR for this reason, so I wrote a NETCONF
script that would make some sanity checks on the RTR session state on
XR boxes (nagios friendly):
https://gist.github.com/lukastribus/695c9e780d118755271519d4d3cb54f3
(minimum number of absolute v4 and v6 VRPs on each RTR server, maximum
v4/v6 VRP deviation between the RTR servers, RTR servers not in
"connected" state).
cheers,
lukas
___
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FIB scale on ASR9001
On 11/11/21 15:43, Lukas Tribus wrote: For ROV to work reliably it needs to be able to reconsider previously rejected invalids, so I would not recommend disabling soft-reconfiguartion inbound, instead I'd suggest to set it to always. If my router vendor is not automatically applying BGP policy based on RPKI state, it shouldn't matter what ended up in RIB if I have not set an explicit policy to act on RPKI state. So if a previously-Invalid route now becomes a VRP, and is then good to go for export toward a neighbor based on existing policy that matches, why can we not leverage Route Refresh to only cater for that change? I'm wondering if we can carry a loose relationship between the VRP database, and RIB state. Of course it would be better to store ROV-rejects separately, instead of rejecting them. Better yet: add a new drop statement in RPL, which makes the route not eligible for path selection, but doesn't remove it from memory - this way the operator has full flexibility. I still don't get why we need to worry about Invalid routes, if the operator is doing ROV to drop them. As soon as they become Valid, the RTR update will tell us that and we can allow for incremental changes for what we ask our neighbors to accept. I can't believe I'm saying this, but a famous CPE vendor from Latvia actually supports this (routing filter action: reject vs discard) [1], maybe the XR BGP team could steal some ideas from them ;) :-). I don't like to depend on optional or not commonly used BGP features and code paths in other people's networks for changes of any kind on my end. In Juniper's case, their default to keep a copy of Adj-RIB-In had the unintended side-effect of making ROV deployment less exciting. However, I still feel not being able to leverage Route Refresh and avoid this clumsiness with ROV is below par for what we can design. After all, that was the whole point of Route Refresh... Memory consumption was negligible for my use-case, iirc it was 200 - 300 MB for 30 - 40 peers, one of which was transit (so full table - this was about a year ago). I believe we had this conversation before, and you mentioned that this could be a DoS vector, which is true but all the solutions that we can possibly suggest simply implement a "selective" soft-reconfig inbound always" anyway, so the DoS vector needs to be addressed in a different way, in my opinion. Well, we had several peers disable sessions with us because their CPU was being impacted by all the refresh messages we were sending. So we have seen it become a DoS vector toward others, and then by extension, toward us when we have to lose shorter paths to those peers due to the session tear-down. Mark. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FIB scale on ASR9001
Hello, On Thu, 11 Nov 2021 at 10:22, Saku Ytti wrote: > > On Thu, 11 Nov 2021 at 10:19, Mark Tinka wrote: > > > Thanks for the clue, Saku. Hopefully someone here has the energy to ask > > Cisco to update their documentation, to make this a recommendation. I > > can't be asked :-). > > I think it should just be a config error. You're not just cucking > yourself, but your peers and customers. So it shouldn't be a choice > you can make. > > We can also imagine improvements > 1) by default keep all RPKI rejects, and have 'soft-inbound never' > optionally to turn that off When enabling ROV on the ASR9001, I set everything to "soft-reconfiguration inbound always", because I couldn't imagine how ROV would work in a reliable way otherwise. I didn't think people would complain about periodic route-refreshes, I just didn't trust that it would work reliably on huge internet exchanges with many peers and questionable BGP implementations on the other side. I wanted my EBGP behaviour to remain *exactly* the same, just as before-ROV. For ROV to work reliably it needs to be able to reconsider previously rejected invalids, so I would not recommend disabling soft-reconfiguartion inbound, instead I'd suggest to set it to always. Of course it would be better to store ROV-rejects separately, instead of rejecting them. Better yet: add a new drop statement in RPL, which makes the route not eligible for path selection, but doesn't remove it from memory - this way the operator has full flexibility. I can't believe I'm saying this, but a famous CPE vendor from Latvia actually supports this (routing filter action: reject vs discard) [1], maybe the XR BGP team could steal some ideas from them ;) I don't like to depend on optional or not commonly used BGP features and code paths in other people's networks for changes of any kind on my end. Memory consumption was negligible for my use-case, iirc it was 200 - 300 MB for 30 - 40 peers, one of which was transit (so full table - this was about a year ago). I believe we had this conversation before, and you mentioned that this could be a DoS vector, which is true but all the solutions that we can possibly suggest simply implement a "selective" soft-reconfig inbound always" anyway, so the DoS vector needs to be addressed in a different way, in my opinion. cheers, lukas [1] https://wiki.mikrotik.com/wiki/Manual:Routing/Routing_filters ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FIB scale on ASR9001
On 11/11/21 11:22, Saku Ytti wrote: I think it should just be a config error. You're not just cucking yourself, but your peers and customers. So it shouldn't be a choice you can make. I don't disagree, especially as there are likely several other operators working this way, and not knowing it because the neighbor either hasn't complained, or isn't detecting for Route Refresh noise. However, the documentation should still be updated for folk running old code earlier than the new code which would have this improvement. We can also imagine improvements 1) by default keep all RPKI rejects, and have 'soft-inbound never' optionally to turn that off Similar to how Junos does it, but specifically for RPKI. That would make sense. Of course, if someone already uses 'soft-reconfiguration inbound' for historical reasons, then keeping it as they enable ROV works out for them anyway. 2) have 1 bit per neighbor indicating policy had rpki rejects and 2 bits for validation database update iindicating database become less/more permissive IFF database became more permissive and neighbor has rpki rejects and we have soft-inbound never, then refresh Reasonable. Mark. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FIB scale on ASR9001
On Thu, 11 Nov 2021 at 10:19, Mark Tinka wrote: > Thanks for the clue, Saku. Hopefully someone here has the energy to ask > Cisco to update their documentation, to make this a recommendation. I > can't be asked :-). I think it should just be a config error. You're not just cucking yourself, but your peers and customers. So it shouldn't be a choice you can make. We can also imagine improvements 1) by default keep all RPKI rejects, and have 'soft-inbound never' optionally to turn that off 2) have 1 bit per neighbor indicating policy had rpki rejects and 2 bits for validation database update iindicating database become less/more permissive IFF database became more permissive and neighbor has rpki rejects and we have soft-inbound never, then refresh -- ++ytti ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FIB scale on ASR9001
On 11/11/21 10:26, Gert Doering wrote: I've been extremely underwhelmed with the quality of IOS XR documentation in many respects, ROV being one of them. (And ROV on IOS XE is full of different eels... as if a totally different company has implemented it, without ever speaking to someone who has done this before, or understand what it's supposed to do) I and the usual friends you know of have been fighting with Cisco to fix ROV's default behaviour in IOS XE since 2015. They refuse to listen. Personally, I gave up. We don't run RPKI on our CSR1000v route reflectors, so I'm not bothered. We don't run RPKI on the few ASR1000's that are in the network, and those are also being replaced by the MX204. We are also working on replacing our ASR920's with something better. We ran RPKI on those for all of 3 days until we turned it off. The box has bigger problems, like insufficient FIB, that we are already dealing with. We're moving towards Arista 7280R, which is not without its own set of surprises... We use this box for customer Layer 2 aggregation in the data centre. Very happy with it. As far as its maturity goes as an IP/MPLS box goes, I cannot tell you. But, knowing you, I'll keep my eyes on arista-nsp open wide open :-). It's an awfully quiet list - haven't got anything new on there since February of this year. Mark. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FIB scale on ASR9001
On 11/11/21 10:10, Saku Ytti wrote: Correct. Add 'keep none' to junos, and you'll have the same issue. Since we started running ROV in IOS XE in 2014, we would have hit this issue then and allowed for it if the BGP best path evaluation process in IOS XE did not do strange things by default, which I believe have not yet been fixed. So we turned it off then. We always used Juniper (MX80 included) for peering back then, so didn't run into this given Junos' default policy. We ran the ASR9001 for peering/transit for a long time before coming up against this, but it makes sense that the complaints only picked up in the last 2 years when ROV was on the rise, globally. Thanks for the clue, Saku. Hopefully someone here has the energy to ask Cisco to update their documentation, to make this a recommendation. I can't be asked :-). Mark. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FIB scale on ASR9001
Hi, On Thu, Nov 11, 2021 at 10:14:16AM +0200, Mark Tinka wrote: > I have read a lot of Cisco documentation on configuring ROV on IOS XE > and IOS XR, and unless something has been recently updated, this is not > one of the explicit recommendations in that documentation. I can see how > easy it is to overlook (or if you do have 'soft-reconfiguration inbound' > configured, how that significance can also be overlooked). I've been extremely underwhelmed with the quality of IOS XR documentation in many respects, ROV being one of them. (And ROV on IOS XE is full of different eels... as if a totally different company has implemented it, without ever speaking to someone who has done this before, or understand what it's supposed to do) > All that notwithstanding, the move to 100Gbps peering/transit + faster > CPU's on the MX204 is still a worthy reason to move away from the > ASR9001, for us anyway :-). Yeah, not argueing that decision, just curious about the problems you saw. We're moving towards Arista 7280R, which is not without its own set of surprises... gert -- "If was one thing all people took for granted, was conviction that if you feed honest figures into a computer, honest figures come out. Never doubted it myself till I met a computer with a sense of humor." Robert A. Heinlein, The Moon is a Harsh Mistress Gert Doering - Munich, Germany g...@greenie.muc.de signature.asc Description: PGP signature ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FIB scale on ASR9001
On 11/11/21 09:51, Gert Doering wrote: I seem to remember it's related to RPKI ROV (every time the RPKI server sends new data the BGP implementation asks for a refresh to re-validate the incoming feed). Or at least there is something in the back of my head that says "I have heard someone talk about this unintended side-effect of RPKI ROV, together with a BGP setup without 'soft in always'". Might there be a correlation in your environment? Yes, this makes sense. We, generally, did not run "soft-reconfiguration inbound" since the IOS days, to save on RAM. Also, Route Refresh was the gold standard. I'm not sure the RAM issue is a big deal anymore, considering how large it is in routers these days... But I can see how this creates an undesired side effect with ROV, which then puts pressure on Route Refresh. I have to say, we did not consider it; which hardly surprises me since TAC didn't either. I have read a lot of Cisco documentation on configuring ROV on IOS XE and IOS XR, and unless something has been recently updated, this is not one of the explicit recommendations in that documentation. I can see how easy it is to overlook (or if you do have 'soft-reconfiguration inbound' configured, how that significance can also be overlooked). All that notwithstanding, the move to 100Gbps peering/transit + faster CPU's on the MX204 is still a worthy reason to move away from the ASR9001, for us anyway :-). Mark. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FIB scale on ASR9001
On Thu, 11 Nov 2021 at 10:08, Mark Tinka wrote: > And Junos defaults to always maintaining a copy of Adj-RIB-In, which is > why we don't have that issue there, non? Correct. Add 'keep none' to junos, and you'll have the same issue. -- ++ytti ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FIB scale on ASR9001
On 11/11/21 09:51, Saku Ytti wrote: When you receive an RTR update, you change your ingress policy, and you don't know what is the correct result without re-evaluating. I'm with you now - makes sense. And Junos defaults to always maintaining a copy of Adj-RIB-In, which is why we don't have that issue there, non? Mark. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FIB scale on ASR9001
Hi, On Thu, Nov 11, 2021 at 09:26:12AM +0200, Mark Tinka wrote: > We are currently running 6.7.1, mainly to fix some LDPv6 issues. > > However, we started seeing this "too many BGP refresh messages" issue as > far back as 6.4.2. I seem to remember it's related to RPKI ROV (every time the RPKI server sends new data the BGP implementation asks for a refresh to re-validate the incoming feed). Or at least there is something in the back of my head that says "I have heard someone talk about this unintended side-effect of RPKI ROV, together with a BGP setup without 'soft in always'". Might there be a correlation in your environment? gert -- "If was one thing all people took for granted, was conviction that if you feed honest figures into a computer, honest figures come out. Never doubted it myself till I met a computer with a sense of humor." Robert A. Heinlein, The Moon is a Harsh Mistress Gert Doering - Munich, Germany g...@greenie.muc.de signature.asc Description: PGP signature ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FIB scale on ASR9001
On Thu, 11 Nov 2021 at 09:50, Mark Tinka wrote: > What's the thought-process here? When you receive an RTR update, you change your ingress policy, and you don't know what is the correct result without re-evaluating. -- ++ytti ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FIB scale on ASR9001
On 11/11/21 09:42, Saku Ytti wrote: Did you run RPKI? Yes. Did you have softinbound disabled? Yes. This would cause a refresh on every RTR update. Basically a misconfiguration, if you run RPKI you must keep policy rejected routes. What's the thought-process here? Mark. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
