Re: [uknof] Full table routers
. [...] I guess that I am then limited to how fast BIRD or FRR will run on a single core. You can also try OpenBGPD ;-) -- Willy Manga @ongolaboy https://ongola.blogspot.com/ OpenPGP_signature Description: OpenPGP digital signature
Re: [uknof] Full table routers
Andy, Running on a VM is better for us in the long term. We run all of our firewalls on VMs, so we need to get a robust and performant 10G traffic path through multiple VMs. We use ESXi now which does not give us as much control or visibility at the network layer. So the plan is to move to Ubuntu, KVM and OVS. This should give us more options. I guess that I am then limited to how fast BIRD or FRR will run on a single core. Thanks John From: Andy Davidson Sent: Thursday, June 29, 2023 3:52 PM To: John P Bourke Cc: uknof@lists.uknof.org.uk Subject: Re: [uknof] Full table routers Hi, John Feels like quite a fragile implementation once you separate control from forwarding. Will's suggestion to not use a full table or Tim's suggestion to use PC routers feels way more robust. You can use these Aristas in applications in your network that don't need full table, and you can do 10s of Mpps on a pc router (see Pim from IPng's presentation https://www.swinog.ch/wp-content/uploads/2021/12/Pim-van-Pelt-IPng-Networks-Evolution-of-DPDK-Controlplanes.pdf ) Andy From: uknof mailto:uknof-boun...@lists.uknof.org.uk>> on behalf of John P Bourke mailto:john.bou...@mobileinternet.com>> Date: Wednesday, 28 June 2023 at 21:25 To: Tim Bray mailto:t...@kooky.org>>, uknof@lists.uknof.org.uk<mailto:uknof@lists.uknof.org.uk> mailto:uknof@lists.uknof.org.uk>> Subject: Re: [uknof] Full table routers Hi I may have "an" answer. I think the Americans call this a "Hail Mary Pass". I have a bunch Arista 7150s, which are EOL and a disappointment. But I found this. https://research.kudelskisecurity.com/2015/10/01/hacking-arista-appliances-for-fun-and-profit/#comments The Arista runs a full Centos 7.6. You strip out the Arista BGP process and BIRD (or FRR I guess) and you have a route server. I say route server, because by pulling the Arista BGP process you have no interaction with the RIB. Thanks John BTW - Not dissing Arista. The 7150 is a bit of a unicorn in their portfolio, using a chipset from Intel which they bought from a startup, which Intel then dropped so Arista understandably did not put a lot of effort into beyond the High Frequency Trading use cases that this low latency switch is good for. From: Tim Bray mailto:t...@kooky.org>> Sent: Wednesday, June 28, 2023 6:56 PM To: uknof@lists.uknof.org.uk<mailto:uknof@lists.uknof.org.uk> Subject: Re: [uknof] Full table routers On 28/06/2023 10:27, John P Bourke wrote: Any recommendations for full table routers. We don't need more than 10G. I used Debian + FRR on HP proliants. With startech Nics with intel chipset. Unusual, but did the trick. Help that there was a whole stack of the same hardware running services in the same place.They take a while to boot, but you can make it faster and I think the newer variants are better. Software wise, takes a bit of getting used to. Sometimes conflict between FRR and what Debian wants to do for network setup. Also you can use CAKE :) Also run any scripts or monitoring you want onboard (like counting the BFD flaps per hour to watch the problems that go away and come back very quickly) See also distributions that bundle FRR more specifically for networking rather than a general distribution. -- Tim Bray Huddersfield, GB t...@kooky.org<mailto:t...@kooky.org> +44 7966479015
Re: [uknof] Full table routers
Hi, John Feels like quite a fragile implementation once you separate control from forwarding. Will’s suggestion to not use a full table or Tim’s suggestion to use PC routers feels way more robust. You can use these Aristas in applications in your network that don’t need full table, and you can do 10s of Mpps on a pc router (see Pim from IPng’s presentation https://www.swinog.ch/wp-content/uploads/2021/12/Pim-van-Pelt-IPng-Networks-Evolution-of-DPDK-Controlplanes.pdf ) Andy From: uknof on behalf of John P Bourke Date: Wednesday, 28 June 2023 at 21:25 To: Tim Bray , uknof@lists.uknof.org.uk Subject: Re: [uknof] Full table routers Hi I may have “an” answer. I think the Americans call this a “Hail Mary Pass”. I have a bunch Arista 7150s, which are EOL and a disappointment. But I found this. https://research.kudelskisecurity.com/2015/10/01/hacking-arista-appliances-for-fun-and-profit/#comments The Arista runs a full Centos 7.6. You strip out the Arista BGP process and BIRD (or FRR I guess) and you have a route server. I say route server, because by pulling the Arista BGP process you have no interaction with the RIB. Thanks John BTW – Not dissing Arista. The 7150 is a bit of a unicorn in their portfolio, using a chipset from Intel which they bought from a startup, which Intel then dropped so Arista understandably did not put a lot of effort into beyond the High Frequency Trading use cases that this low latency switch is good for. From: Tim Bray Sent: Wednesday, June 28, 2023 6:56 PM To: uknof@lists.uknof.org.uk Subject: Re: [uknof] Full table routers On 28/06/2023 10:27, John P Bourke wrote: Any recommendations for full table routers. We don’t need more than 10G. I used Debian + FRR on HP proliants. With startech Nics with intel chipset. Unusual, but did the trick. Help that there was a whole stack of the same hardware running services in the same place.They take a while to boot, but you can make it faster and I think the newer variants are better. Software wise, takes a bit of getting used to. Sometimes conflict between FRR and what Debian wants to do for network setup. Also you can use CAKE :) Also run any scripts or monitoring you want onboard (like counting the BFD flaps per hour to watch the problems that go away and come back very quickly) See also distributions that bundle FRR more specifically for networking rather than a general distribution. -- Tim Bray Huddersfield, GB t...@kooky.org<mailto:t...@kooky.org> +44 7966479015
Re: [uknof] Full table routers
Will “In fact it may not have enough RAM and CPU to effectively deal with a modern full table” Its impressive to be able to get to a Linux shell from the Arista command prompt, but as you say, under the bonnet there may not be much vroom vroom. Not sure what “good” is in terms of timing for ingesting the full table. Thanks John From: Will Hargrave Sent: Wednesday, June 28, 2023 9:46 PM To: John P Bourke Cc: uknof@lists.uknof.org.uk Subject: Re: [uknof] Full table routers Hi John, Why not simply accept fewer routes (plus a default) into the existing Arista EOS BGP and so the hardware FIB? Then you can actually take advantage of the hardware forwarding. With this setup you’re using the relatively slow control plane (the Intel FM6000 was released a decade ago and I can’t imagine Arista paired it with a super-fast SoC…) to route and that won’t work very quickly at all. In fact it may not have enough RAM and CPU to effectively deal with a modern full table, it would be better to just use a modern 1U server for this. Will On 28 Jun 2023, at 21:21, John P Bourke wrote: Hi I may have “an” answer. I think the Americans call this a “Hail Mary Pass”. I have a bunch Arista 7150s, which are EOL and a disappointment. But I found this. https://research.kudelskisecurity.com/2015/10/01/hacking-arista-appliances-for-fun-and-profit/#comments The Arista runs a full Centos 7.6. You strip out the Arista BGP process and BIRD (or FRR I guess) and you have a route server. I say route server, because by pulling the Arista BGP process you have no interaction with the RIB. Thanks John BTW – Not dissing Arista. The 7150 is a bit of a unicorn in their portfolio, using a chipset from Intel which they bought from a startup, which Intel then dropped so Arista understandably did not put a lot of effort into beyond the High Frequency Trading use cases that this low latency switch is good for. From: Tim Bray mailto:t...@kooky.org>> Sent: Wednesday, June 28, 2023 6:56 PM To: uknof@lists.uknof.org.uk<mailto:uknof@lists.uknof.org.uk> Subject: Re: [uknof] Full table routers On 28/06/2023 10:27, John P Bourke wrote: Any recommendations for full table routers. We don’t need more than 10G. I used Debian + FRR on HP proliants. With startech Nics with intel chipset. Unusual, but did the trick. Help that there was a whole stack of the same hardware running services in the same place.They take a while to boot, but you can make it faster and I think the newer variants are better. Software wise, takes a bit of getting used to. Sometimes conflict between FRR and what Debian wants to do for network setup. Also you can use CAKE :) Also run any scripts or monitoring you want onboard (like counting the BFD flaps per hour to watch the problems that go away and come back very quickly) See also distributions that bundle FRR more specifically for networking rather than a general distribution. -- Tim Bray Huddersfield, GB t...@kooky.org<mailto:t...@kooky.org> +44 7966479015
Re: [uknof] Full table routers
John This is a good suggestion. We consciously opted for Arista 7050 switches over full-table routers a good few years ago (2016) and made up the difference with selective route download. EOS has a really cool feature where a prefix list can be set to a HTTP endpoint so we have internal anycast endpoints whereby each switch can dynamically update its prefixlist, and those are managed by sflow analysis. IIRC the busiest edge has about 5k prefixes installed (plus default) and it has worked flawlessly with newly relevant routes being installed in a few minutes. We've moved on and up a few generations hardware wise now but the basic operation is still the same. cheers Simon photo ( https://simwood.com ) Simon Woodhead CEO, Simwood *Phone* +44 330 122 3021 ( tel:+44+330+122+3021 ) | *Mobile* +44 7976 238 487 ( tel:+44+7976+238+487 ) *Website* simwood.com ( https://simwood.com ) | *Email* simon.woodh...@simwood.com ( simon.woodh...@simwood.com ) linkedin ( https://www.linkedin.com/in/simonwoodhead/ ) twitter ( https://twitter.com/simwoodesms ) IMPORTANT: The contents of this email and any attachments are confidential. They are intended for the named recipient(s) only. If you have received this email by mistake, please notify the sender immediately and do not disclose the contents to anyone or make copies thereof. On Wed, Jun 28, 2023 at 21:46:08, Will Hargrave < w...@harg.net > wrote: > > > > Hi John, > > > > Why not simply accept fewer routes (plus a default) into the existing > Arista EOS BGP and so the hardware FIB? Then you can actually take > advantage of the hardware forwarding. > > > > With this setup you’re using the relatively slow control plane (the Intel > FM6000 was released a decade ago and I can’t imagine Arista paired it with > a super-fast SoC…) to route and that won’t work very quickly at all. In > fact it may not have enough RAM and CPU to effectively deal with a modern > full table, it would be better to just use a modern 1U server for this. > > > > Will > > > > On 28 Jun 2023, at 21:21, John P Bourke wrote: > > > >> >> >> Hi >> >> >> >> >> >> >> >> I may have “an” answer. I think the Americans call this a “Hail Mary >> Pass”. >> >> >> >> >> >> >> >> I have a bunch Arista 7150s, which are EOL and a disappointment. But I >> found this. >> >> >> >> >> >> >> >> https:/ / research. kudelskisecurity. com/ 2015/ 10/ 01/ >> hacking-arista-appliances-for-fun-and-profit/ >> #comments ( >> https://research.kudelskisecurity.com/2015/10/01/hacking-arista-appliances-for-fun-and-profit/#comments >> ) >> >> >> >> >> >> >> >> The Arista runs a full Centos 7.6. You strip out the Arista BGP process >> and BIRD (or FRR I guess) and you have a route server. I say route >> server, because by pulling the Arista BGP process you have no interaction >> with the RIB. >> >> >> >> >> >> >> >> Thanks >> >> >> >> >> >> >> >> John >> >> >> >> >> >> >> >> BTW – Not dissing Arista. The 7150 is a bit of a unicorn in their >> portfolio, using a chipset from Intel which they bought from a startup, >> which Intel then dropped so Arista understandably did not put a lot of >> effort into beyond the High Frequency Trading use cases that this low >> latency switch is good for. >> >> >> >> >> >> >> >> >> >> >> >> *From:* Tim Bray < tim@ kooky. org ( t...@kooky.org ) > >> *Sent:* Wednesday, June 28, 2023 6:56 PM >> *To:* uknof@ lists. uknof. org. uk ( uknof@lists.uknof.org.uk ) >> *Subject:* Re: [uknof] Full table routers >> >> >> >> >> >> >> >> >> On 28/06/2023 10:27, John P Bourke wrote: >> >> >> >>> >>> >>> Any recommendations for full table routers. We don’t need more than 10G. >>> >>> >> >> >> >> I used Debian + FRR on HP proliants. With startech Nics with intel >> chipset. Unusual, but did the trick. Help that there was a whole >> stack of the same hardware running services in the same place. They >> take a while to boot, but you can make it faster and I think the newer >> variants are better. >> >> >> >> >> >> >> >> Software wise, takes a bit of getting used to. Sometimes conflict >> between FRR and what Debian wants to do for network setup. Also you >> can use CAKE :) Also run any scripts or monitoring you want onboard >> (like counting the BFD flaps per hour to watch the problems that go away >> and come back very quickly) >> >> >> >> See also distributions that bundle FRR more specifically for networking >> rather than a general distribution. >> >> -- Tim Bray Huddersfield, GB tim@ kooky. org ( t...@kooky.org ) +44 >> 7966479015 >> > >
Re: [uknof] Full table routers
Hi John, Why not simply accept fewer routes (plus a default) into the existing Arista EOS BGP and so the hardware FIB? Then you can actually take advantage of the hardware forwarding. With this setup you’re using the relatively slow control plane (the Intel FM6000 was released a decade ago and I can’t imagine Arista paired it with a super-fast SoC…) to route and that won’t work very quickly at all. In fact it may not have enough RAM and CPU to effectively deal with a modern full table, it would be better to just use a modern 1U server for this. Will On 28 Jun 2023, at 21:21, John P Bourke wrote: Hi I may have “an” answer. I think the Americans call this a “Hail Mary Pass”. I have a bunch Arista 7150s, which are EOL and a disappointment. But I found this. https://research.kudelskisecurity.com/2015/10/01/hacking-arista-appliances-for-fun-and-profit/#comments The Arista runs a full Centos 7.6. You strip out the Arista BGP process and BIRD (or FRR I guess) and you have a route server. I say route server, because by pulling the Arista BGP process you have no interaction with the RIB. Thanks John BTW – Not dissing Arista. The 7150 is a bit of a unicorn in their portfolio, using a chipset from Intel which they bought from a startup, which Intel then dropped so Arista understandably did not put a lot of effort into beyond the High Frequency Trading use cases that this low latency switch is good for. From: Tim Bray Sent: Wednesday, June 28, 2023 6:56 PM To: uknof@lists.uknof.org.uk Subject: Re: [uknof] Full table routers On 28/06/2023 10:27, John P Bourke wrote: Any recommendations for full table routers. We don’t need more than 10G. I used Debian + FRR on HP proliants. With startech Nics with intel chipset.Unusual, but did the trick. Help that there was a whole stack of the same hardware running services in the same place. They take a while to boot, but you can make it faster and I think the newer variants are better. Software wise, takes a bit of getting used to. Sometimes conflict between FRR and what Debian wants to do for network setup. Also you can use CAKE :) Also run any scripts or monitoring you want onboard (like counting the BFD flaps per hour to watch the problems that go away and come back very quickly) See also distributions that bundle FRR more specifically for networking rather than a general distribution. -- Tim Bray Huddersfield, GB t...@kooky.org<mailto:t...@kooky.org> +44 7966479015
Re: [uknof] Full table routers
On Wed, 28 Jun 2023 at 11:27, John P Bourke wrote: > Any recommendations for full table routers. We don’t need more than 10G. Mikrotik CCR2116[1] or CCR2216[2]? [1]: https://mikrotik.com/product/ccr2116_12g_4splus [2]: https://mikrotik.com/product/ccr2216_1g_12xs_2xq
Re: [uknof] Full table routers
Hi I may have “an” answer. I think the Americans call this a “Hail Mary Pass”. I have a bunch Arista 7150s, which are EOL and a disappointment. But I found this. https://research.kudelskisecurity.com/2015/10/01/hacking-arista-appliances-for-fun-and-profit/#comments The Arista runs a full Centos 7.6. You strip out the Arista BGP process and BIRD (or FRR I guess) and you have a route server. I say route server, because by pulling the Arista BGP process you have no interaction with the RIB. Thanks John BTW – Not dissing Arista. The 7150 is a bit of a unicorn in their portfolio, using a chipset from Intel which they bought from a startup, which Intel then dropped so Arista understandably did not put a lot of effort into beyond the High Frequency Trading use cases that this low latency switch is good for. From: Tim Bray Sent: Wednesday, June 28, 2023 6:56 PM To: uknof@lists.uknof.org.uk Subject: Re: [uknof] Full table routers On 28/06/2023 10:27, John P Bourke wrote: Any recommendations for full table routers. We don’t need more than 10G. I used Debian + FRR on HP proliants. With startech Nics with intel chipset. Unusual, but did the trick. Help that there was a whole stack of the same hardware running services in the same place.They take a while to boot, but you can make it faster and I think the newer variants are better. Software wise, takes a bit of getting used to. Sometimes conflict between FRR and what Debian wants to do for network setup. Also you can use CAKE :) Also run any scripts or monitoring you want onboard (like counting the BFD flaps per hour to watch the problems that go away and come back very quickly) See also distributions that bundle FRR more specifically for networking rather than a general distribution. -- Tim Bray Huddersfield, GB t...@kooky.org<mailto:t...@kooky.org> +44 7966479015
Re: [uknof] Full table routers
--- Begin Message --- On 28/06/2023 10:27, John P Bourke wrote: Any recommendations for full table routers. We don’t need more than 10G. I used Debian + FRR on HP proliants. With startech Nics with intel chipset. Unusual, but did the trick. Help that there was a whole stack of the same hardware running services in the same place. They take a while to boot, but you can make it faster and I think the newer variants are better. Software wise, takes a bit of getting used to. Sometimes conflict between FRR and what Debian wants to do for network setup. Also you can use CAKE :) Also run any scripts or monitoring you want onboard (like counting the BFD flaps per hour to watch the problems that go away and come back very quickly) See also distributions that bundle FRR more specifically for networking rather than a general distribution. -- Tim Bray Huddersfield, GB t...@kooky.org +44 7966479015 --- End Message ---
[uknof] Full table routers
Hi Any recommendations for full table routers. We don't need more than 10G. Thanks John
