Re: BGP Enabled transit in Chicago (River North) and equipment recommendation
They are obviously not running full tables on their 3640. I'd imagine a raspberry pi would have more BGP capability and throughput than a 3640, though I don't recommend doing that even as a joke. But an ERR would be fine if they're expecting nothing more than a slightly faster 3640 with maybe some extra features. On 9/3/19 3:54 PM, Florian Brandstetter via NANOG wrote: Ubiquiti's EdgeRouter Lite is equipped with 512 MiB of DDR2 memory, of which after startup, roughly 491 MiB can be utilized. 119 MiB of the remaining memory are allocated by the base of the router already, which leaves you with a remainder of 372 MiB memory. Memory usage depends on the architecture for objects, for example there's a large difference between x86 and x86_64, since on x86_64, the compiler will generally use 64bit boundaries to be faster; the ERL runs on a MIPS64 architecture, which will have a similar trade-off. To get to the point, let's have a quick look at the components using memory: bgpd, zebra, kernel. Roughly 180 MiB of memory are required to keep a single full table in bgpd alone, leaving you with 192 MiB of free memory. Accounting further, zebra will eat at least another 100 MiB for exporting the BGP RIB to the Kernel (FIB), leaving you with 100 MiB. At this point, you have a mere 92 MiB left for fitting the routes into the kernel, and to leave room for RX buffers on sockets. I don't see full tables happening from a memory perspective on the EdgeRouter Lite, you would want to look at something with at least 2 GiB of memory to keep the whole system running smoothly, and when using Quagga and Zebra, that's still aimed rather low. FRRouting at this point uses 2 GiB for 4 full tables on an x86 system, without any magic attached. Having kept it unmentioned, the EdgeRouter Lite has a dual-core with 500 MHz, and surely your BGP updates processing isn't offloaded, hence you will pretty quickly kill the whole router when you flood it with a full table, unless you set very low queue sizes, which isn't really reliable though since you generally want BGP to converge fast - not after a period of 15 minutes with the CPU sitting on 100%. You might want to install something like OpenWRT (which I don't know the possibility of on an ERL), and run BIRD if you're tied to a low memory footprint, however, in a base vendor-generic setup of the ERL, it's beyond my understanding why one would even suggest running a full table on it. Sent from Mailspring
Re: BGP Enabled transit in Chicago (River North) and equipment recommendation
Ubiquiti's EdgeRouter Lite is equipped with 512 MiB of DDR2 memory, of which after startup, roughly 491 MiB can be utilized. 119 MiB of the remaining memory are allocated by the base of the router already, which leaves you with a remainder of 372 MiB memory. Memory usage depends on the architecture for objects, for example there's a large difference between x86 and x86_64, since on x86_64, the compiler will generally use 64bit boundaries to be faster; the ERL runs on a MIPS64 architecture, which will have a similar trade-off. To get to the point, let's have a quick look at the components using memory: bgpd, zebra, kernel. Roughly 180 MiB of memory are required to keep a single full table in bgpd alone, leaving you with 192 MiB of free memory. Accounting further, zebra will eat at least another 100 MiB for exporting the BGP RIB to the Kernel (FIB), leaving you with 100 MiB. At this point, you have a mere 92 MiB left for fitting the routes into the kernel, and to leave room for RX buff ers on sockets. I don't see full tables happening from a memory perspective on the EdgeRouter Lite, you would want to look at something with at least 2 GiB of memory to keep the whole system running smoothly, and when using Quagga and Zebra, that's still aimed rather low. FRRouting at this point uses 2 GiB for 4 full tables on an x86 system, without any magic attached. Having kept it unmentioned, the EdgeRouter Lite has a dual-core with 500 MHz, and surely your BGP updates processing isn't offloaded, hence you will pretty quickly kill the whole router when you flood it with a full table, unless you set very low queue sizes, which isn't really reliable though since you generally want BGP to converge fast - not after a period of 15 minutes with the CPU sitting on 100%. You might want to install something like OpenWRT (which I don't know the possibility of on an ERL), and run BIRD if you're tied to a low memory footprint, however, in a base vendor-generic setup of the ERL, it's beyond my understanding why one would even suggest running a full table on it.
Re: BGP Enabled transit in Chicago (River North) and equipment recommendation
On 9/3/2019 1:54 PM, Florian Brandstetter wrote: I don't see full tables happening from a memory perspective on the EdgeRouter Lite, you would want to look at something with at least 2 GiB of memory to keep the whole system running smoothly, and when using Quagga and Zebra, that's still aimed rather low. FRRouting at this point uses 2 GiB for 4 full tables on an x86 system, without any magic attached. I'm going based on actual tests done when the ERL came out in 2012. While its cores are anemic by today's standards, it handled multiple full tables fine back then. Since then, the ERs have stopped using Quagga/Zebra and now use ZebOS (which is also used in several other commercial products). It's a bit faster and smaller. In today's world, its tight, but it will still work. You might want to install something like OpenWRT (which I don't know the possibility of on an ERL), and run BIRD if you're tied to a low memory footprint, however, in a base vendor-generic setup of the ERL, it's beyond my understanding why one would even suggest running a full table on it. As I pointed out, its the dirt cheap option. Better option is the ER4 and later which is better suited to the increased table size. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org/ http://www.ahbl.org
Re: BGP Enabled transit in Chicago (River North) and equipment recommendation
I will note that Comcast will do BGP on their enterprise fiber circuits. Comcast DIA (which they call EDI) comes in increments of 1M up to 10M, then 10M up to 100M, etc. So you could get 10M or 80M (not sure if "10MB/Sec" means 10Mbps or 80MBps) and do BGP over that, if it's available. RCN is likely similar but I'm not as familiar with their offerings. On Tue, Sep 3, 2019 at 3:35 PM Brielle wrote: > On 9/3/2019 12:19 PM, Matt Harris wrote: > > But even the higher-end Ubiquiti EdgeRouter series products can handle > > full tables if you understand and accept their limitations in doing so > > if budget is a huge concern but you still need to take full tables. > > As long as you stick with the 1.10.10 series of firmware, the EdgeRouter > series is pretty stable. I run an EdgeRouter Infinity (8 x 10G SFP+) at > home with both IPv4 and IPv6 BGP feeds on a CenturyLink Fiber+ connection. > > You can get the original EdgeRouter Lite for sub $100 and it will have > no problem taking a full feed for both v4 and v6... However, better > choice is the EdgeRouter 4, which is a bit newer and much faster. > > There's also the ER6P if you need something with more ports, or even the > ER12/12P if you want something with a built in network switch. Just > avoid the budget oriented ERX and ER10X which lack the RAM. > > All of the ERs are low power consumption and Linux based with a > Vyatta/VyOS/Juniper-like CLI, so pretty easy to work with. > > (I do Ubiquiti deployments, so can answer a lot of questions anyone > might have). > > -- > Brielle Bruns > The Summit Open Source Development Group > http://www.sosdg.org/ http://www.ahbl.org >
Re: BGP Enabled transit in Chicago (River North) and equipment recommendation
On 9/3/2019 12:19 PM, Matt Harris wrote: But even the higher-end Ubiquiti EdgeRouter series products can handle full tables if you understand and accept their limitations in doing so if budget is a huge concern but you still need to take full tables. As long as you stick with the 1.10.10 series of firmware, the EdgeRouter series is pretty stable. I run an EdgeRouter Infinity (8 x 10G SFP+) at home with both IPv4 and IPv6 BGP feeds on a CenturyLink Fiber+ connection. You can get the original EdgeRouter Lite for sub $100 and it will have no problem taking a full feed for both v4 and v6... However, better choice is the EdgeRouter 4, which is a bit newer and much faster. There's also the ER6P if you need something with more ports, or even the ER12/12P if you want something with a built in network switch. Just avoid the budget oriented ERX and ER10X which lack the RAM. All of the ERs are low power consumption and Linux based with a Vyatta/VyOS/Juniper-like CLI, so pretty easy to work with. (I do Ubiquiti deployments, so can answer a lot of questions anyone might have). -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org/ http://www.ahbl.org
Re: BGP Enabled transit in Chicago (River North) and equipment recommendation
I’ve had BGP from comcast business in River North before, not sure what their minimum bandwidth is for that. Tunnels may be simplest at that bandwidth level. > On Sep 3, 2019, at 12:52 PM, Florian Brandstetter via NANOG > wrote: > > Might be worth to consider running a software router on that scale with > perhaps some cheap quad-port GbE PCIe NICs. BIRD would be the BGP daemon to > go, or FRRouting if you want an integrated shell. Hardware routers for 100 > Mbit egress seem a bit overpowered, however, as scaleable you want to go, > some Ubiquiti routers might be a cheap option. > > As for transit, have you considered a redundant tunnel-based solution > instead? You can run that transparently on top of your RCN connection, with > negligible costs for your commit and no additional connection fees. > > On Sep. 3 2019, at 6:17 pm, ADNS NetBSD List Subscriber > wrote: > I have a need for a BGP enabled connection in the River North section of > Chicago. We have a small number of IP blocks that we want to use. Currently, > we have some equipment at 350 E. Cermak (Steadfast Networks) and are looking > at downsizing and bringing stuff > in-house. Our bandwidth requirements are miniscule (10MB/Sec is fine). > > I know RCN offers business cable-modem service but probably not BGP. > > Also, we’d like to ditch our 3640 router in favor of a smaller “desktop” size > router, but none of them seem to do BGP (not surprising). Any > recommendations on hardware would be welcome as well >
Re: BGP Enabled transit in Chicago (River North) and equipment recommendation
On Tue, Sep 3, 2019 at 12:44 PM ADNS NetBSD List Subscriber wrote: > > Also, we’d like to ditch our 3640 router in favor of a smaller “desktop” > size router, but none of them seem to do BGP (not surprising). Any > recommendations on hardware would be welcome as well > I can think of lots of smaller, even desktop sized routers that have no problem doing BGP assuming you're only taking a single transit circuit and only accepting a default route from your upstream provider. Ubiquiti's tiniest EdgeRouters which cost around $100 will do BGP just fine under those circumstances if you're not pushing tons of traffic or pps. If you want something a little nicer/fancier, you can get a Juniper SRX (probably a 300, 320, or 345 depending on how much bandwidth you're going to use). If you want to run multiple transits and take full tables, then at that point I'd recommend going a bit bigger: maybe a Juniper MX or a Cisco ASR. But even the higher-end Ubiquiti EdgeRouter series products can handle full tables if you understand and accept their limitations in doing so if budget is a huge concern but you still need to take full tables. Take care, Matt
Re: BGP Enabled transit in Chicago (River North) and equipment recommendation
Might be worth to consider running a software router on that scale with perhaps some cheap quad-port GbE PCIe NICs. BIRD would be the BGP daemon to go, or FRRouting if you want an integrated shell. Hardware routers for 100 Mbit egress seem a bit overpowered, however, as scaleable you want to go, some Ubiquiti routers might be a cheap option. As for transit, have you considered a redundant tunnel-based solution instead? You can run that transparently on top of your RCN connection, with negligible costs for your commit and no additional connection fees. On Sep. 3 2019, at 6:17 pm, ADNS NetBSD List Subscriber wrote: > I have a need for a BGP enabled connection in the River North section of > Chicago. We have a small number of IP blocks that we want to use. Currently, > we have some equipment at 350 E. Cermak (Steadfast Networks) and are looking > at downsizing and bringing stuff > in-house. Our bandwidth requirements are miniscule (10MB/Sec is fine). > > I know RCN offers business cable-modem service but probably not BGP. > > Also, we’d like to ditch our 3640 router in favor of a smaller “desktop” size > router, but none of them seem to do BGP (not surprising). Any recommendations > on hardware would be welcome as well > > >
BGP Enabled transit in Chicago (River North) and equipment recommendation
I have a need for a BGP enabled connection in the River North section of Chicago. We have a small number of IP blocks that we want to use. Currently, we have some equipment at 350 E. Cermak (Steadfast Networks) and are looking at downsizing and bringing stuff in-house. Our bandwidth requirements are miniscule (10MB/Sec is fine). I know RCN offers business cable-modem service but probably not BGP. Also, we’d like to ditch our 3640 router in favor of a smaller “desktop” size router, but none of them seem to do BGP (not surprising). Any recommendations on hardware would be welcome as well
