Mikrotik Cloud Core Router and BGP real life experiences?
Hi, looking at the specs of Mikrotik Cloud Core Routers it seems to be to good to be true [1] having so much bang for the bucks. So virtually all smaller ISPs would drop their CISCO gear for Mikrotik Routerboards. We are using a handful of Mikrotik boxes, but on a much lower network level (splitting networks; low end router behind ADSL modem, ...). We're happy with them. So I am asking for real life experience and not lab values with Mikrotik Cloud Core Routers and BGP. How good can they handle full tables and a bunch of peering sessions? How good does the box react when adding filters (during attacks)? Reloading the table? etc. etc. I am looking for _real_ _life_ values compared to a CISCO NPE-G2. Please tell me/us from your first hand experience. Thanks! greetings, Martin [1] If something sounds too good to be true, it probably is.
Re: Mikrotik Cloud Core Router and BGP real life experiences?
I am going to be deploying 4 as edge routers in the next few weeks, each will have 1 or 2 full tables plus partial IX tables. So I should have some empirical info soon. They will be doing eBGP to upstreams and iBGP/OSPF internally. I went with the 16gb RAM models. However these boxes are basically Linux running on top of tilera CPUs, in terms of throughput as long as everything stays on the fastpath they have no issues doing wire speed on all ports, however the moment you add a firewall rule or the like they drop to 1.5gbps. On 27/12/2013, at 9:47 pm, Martin Hotze m.ho...@hotze.com wrote: Hi, looking at the specs of Mikrotik Cloud Core Routers it seems to be to good to be true [1] having so much bang for the bucks. So virtually all smaller ISPs would drop their CISCO gear for Mikrotik Routerboards. We are using a handful of Mikrotik boxes, but on a much lower network level (splitting networks; low end router behind ADSL modem, ...). We're happy with them. So I am asking for real life experience and not lab values with Mikrotik Cloud Core Routers and BGP. How good can they handle full tables and a bunch of peering sessions? How good does the box react when adding filters (during attacks)? Reloading the table? etc. etc. I am looking for _real_ _life_ values compared to a CISCO NPE-G2. Please tell me/us from your first hand experience. Thanks! greetings, Martin [1] If something sounds too good to be true, it probably is.
RE: Mikrotik Cloud Core Router and BGP real life experiences?
Thanks, estimated traffic levels are at about half a gig, but at least 50 megs of UDP (VoIP) in both directions. one thing is that I haven't found a solution for redundant power supply. #m -Original Message- From: Geraint Jones [mailto:gera...@koding.com] Sent: Friday, December 27, 2013 10:03 AM To: Martin Hotze Cc: nanog@nanog.org Subject: Re: Mikrotik Cloud Core Router and BGP real life experiences? I am going to be deploying 4 as edge routers in the next few weeks, each will have 1 or 2 full tables plus partial IX tables. So I should have some empirical info soon. They will be doing eBGP to upstreams and iBGP/OSPF internally. I went with the 16gb RAM models. However these boxes are basically Linux running on top of tilera CPUs, in terms of throughput as long as everything stays on the fastpath they have no issues doing wire speed on all ports, however the moment you add a firewall rule or the like they drop to 1.5gbps. On 27/12/2013, at 9:47 pm, Martin Hotze m.ho...@hotze.com wrote: (...)
Re: The Making of a Router
On Fri, Dec 27, 2013 at 1:33 AM, valdis.kletni...@vt.edu wrote: On Thu, 26 Dec 2013 11:16:53 -0800, Seth Mattinen said: On 12/26/13, 9:24, Andrew D Kirch wrote: If he can afford a 10G link... he should be buying real gear... I mean, look, I've got plenty of infrastructure horror stories, but lets not cobble together our own 10gbit solutions, please? At least get one of the new microtik CCR's with a 10gig sfp+? They're only a kilobuck... If you can't afford that I suggest you can't afford to be an ISP. Unless all the money is going into the 10 gig link. If you've sunk so much into the 10G link (or anything else, for that matter) that you don't have a kilobuck to spare, you're probably undercapitalized to be an ISP. I have issue with this line of thought. Granted, a router is built with custom ASICs and most network people understand IOS. However, this is where the benefit of a multi-thousand buck router ends. Most have limited RAM, so this limits the size of your policies and how many routes can be stored and the likes. With a computer with multi 10s or 100s of gigs of RAM, this really isn't an issue. Routers also have slow-ish processors (which is fine for pure routing since they are custom chips but) if you want to do packet inspection, this can slow things down quite a bit. You could argue that this is the same with iptables or pf. However, if you just offload the packets and analyze generally boring packets with snort or bro or whatever, packets flow as fast as they would without analysis. If you have multiple VPNs, this can start to slow down a router whereas a computer can generally keep up. ... And then there's the money issue. Sure, if you're buying a gig+ link, you should be able to afford a fully spec'd out router. However, (in my experience) people don't order equipment with all features enabled and when you find you need a feature, you have to put in a request to buy it and then it takes a month (if you're lucky) for it to be approved. This isn't the case if you use ipt/pf - if te feature is there, it's there - use it. And if a security flaw is found in a router, it might be fixed in the next month... or not. With Linux/BSD, it'll be fixed within a few days (at the most). And, if your support has expired on a router or the router is EOL, you're screwed. I think in the near future, processing packets with GPUs will become a real thing which will make doing massive real time deep packet inspection at 10G+ a real thing. Granted, your network people knowing IOS when they're hired is a big win for just ordering Cisco. But, I don't see that as a show stopper. Stating the scope of what a box is supposed to be used for and not putting endless crap on it might be another win for an actual router. However, this is a people/business thing and not a technical issue. Also, I'm approaching this as more of a question of the best tool for the job vs pure economics - a server is generally going to be cheaper, but I generally find a server nicer/easier to configure than a router.
Re: Mikrotik Cloud Core Router and BGP real life experiences?
On 27/12/2013, at 10:13 pm, Martin Hotze m.ho...@hotze.com wrote: Thanks, estimated traffic levels are at about half a gig, but at least 50 megs of UDP (VoIP) in both directions. one thing is that I haven't found a solution for redundant power supply. Buy 2 :) #m -Original Message- From: Geraint Jones [mailto:gera...@koding.com] Sent: Friday, December 27, 2013 10:03 AM To: Martin Hotze Cc: nanog@nanog.org Subject: Re: Mikrotik Cloud Core Router and BGP real life experiences? I am going to be deploying 4 as edge routers in the next few weeks, each will have 1 or 2 full tables plus partial IX tables. So I should have some empirical info soon. They will be doing eBGP to upstreams and iBGP/OSPF internally. I went with the 16gb RAM models. However these boxes are basically Linux running on top of tilera CPUs, in terms of throughput as long as everything stays on the fastpath they have no issues doing wire speed on all ports, however the moment you add a firewall rule or the like they drop to 1.5gbps. On 27/12/2013, at 9:47 pm, Martin Hotze m.ho...@hotze.com wrote: (...)
Re: The Making of a Router
In talking about RAMBOOT I also realized the instructions are out of date
on the website.
The ramboot boot target script was updated since I created the initial
page to generate the correct fstab, and enable the root account, set a
hostname, etc. so you can actually use the OS until you create a new image.
I extracted the script from the initird to make it easier to grab:
http://ramboot.org/download/RAMBOOT/RAMBOOT-pre0.2/SYSLINUX/initrd/scripts/ramboot
Essentially, by adding a new ramboot script to
/usr/share/initramfs-tools/scripts along side nfs and local it
creates a new boot= target (since the init script looks for
scripts/${BOOT}.
As mentioned on the website, the ramboot process needs a more complete
version of busybox (for tar archive support) and the mke2fs tool added to
/usr/lib/initramfs-tools/bin/ so they will be available to the initrd.
Once you configure networking (see the INSTALL/setup_network script) you
can do an apt-get update and apt-get the packages you need from Ubuntu
12.04 LTS.
Example starting point:
apt-get install sudo
apt-get install nano
apt-get install ssh
apt-get install vlan
apt-get install bridge-utils
On Thu, Dec 26, 2013 at 8:27 PM, Ray Soucy r...@maine.edu wrote:
The basic idea of RAMBOOT is typical in Embedded Linux development.
Linux makes use of multi-stage boot process. One of the stages involves
using an initial ramdisk (initrd) to provide a base root filesystem which
can be used to locate and mount the system root, then continue the boot
process from there.
For an in-memory OS, instead of the boot process mounting a pre-loaded
storage device with the expected root filesystem (e.g. your installed HDD),
you modify it to:
1) Create and format a ramdisk.
2) Create the expected directory structure and system files for the root
filesystem on that ramdisk.
The root filesystem includes the /dev directory and appropriate device
nodes, the basic Linux filesystem and your init program.
The easy way to do that is just to have a TAR archive that you extract to
the ramdisk on boot, better yet use compression on it (e.g. tar.gz) so that
the archive can be read from storage (e.g. USB flash) more quickly.
Today, the initramfs in Linux handles a lot more than simply mounting the
storage device. It performs hardware discovery and loads appropriate
modules. As such the Debian project has a dynamic build system for
initramfs that is run to build the initrd when a new kernel package is
installed, it's called initramfs-tools.
You can manually build your own initramfs using the examples on the
RAMBOOT website, but the point of RAMBOOT is to make building an in-memory
OS quick and simple.
RAMBOOT instead adds configuration to initramfs-tools so that each time a
new initrd is generated, it includes the code needed for RAMBOOT.
The RAMBOOT setup adds handling of a new boot target called ramboot to
the kernel arguments. This allows the same kernel to be used for a normal
installation and remain unaffected, but when you add the argument
boot=ramboot as a kernel option to the bootloader, it triggers the
RAMBOOT process described above.
Having a common kernel between your development environment and embedded
environment makes it much easier to test and verify functionality.
The other part of RAMBOOT is that it makes use of Ubuntu Core. Ubuntu
Core is a stripped down minimal (and they really do mean minimal) root
filesystem for Embedded Linux development. It includes apt-get, though, so
you can install all the packages you need from Ubuntu on the running system.
RAMBOOT then has a development script to make a new root filesystem
archive with the packages you've installed as a baseline. This allows for
you to boot a RAMBOOT system, install your desired packages and change
system configuration files as desired, then build a persistent image of
that install that will be used for future boots.
I also have the start of a script to remove unused kernel modules, and
other files (internationalization for example) which add to the OS
footprint.
You could build the root filesystem on your own (and compile all the
necessary packages) but using Ubuntu Core provides a solid base and allows
for the rapid addition of packages from the giant Ubuntu repository.
Lastly, I make use of SYSLINUX as a bootloader because my goal was to use
a USB stick as the bootflash on an Atom box. Unfortunately, the Atom BIOS
will only boot a USB device if it has a DOS boot partition, so GRUB was a
no-go. The upside is that since the USB uses SYSLINUX and is DOS
formatted, it's easily mounted in Windows or Mac OS X, allowing you to copy
new images or configuration to it easily.
For the boot device I make use of the on-board vertical USB socket on the
system board (typical for most system boards these days) and a low-profile
USB stick. I find the Verbatim Store 'n' Go 8GB USB stick ideally suited
for this as it's less than a quarter-inch
RE: Mikrotik Cloud Core Router and BGP real life experiences?
On 27/12/2013, at 10:13 pm, Martin Hotze m.ho...@hotze.com wrote: Thanks, estimated traffic levels are at about half a gig, but at least 50 megs of UDP (VoIP) in both directions. one thing is that I haven't found a solution for redundant power supply. Buy 2 :) on 3am I only want to read the notification and know what to do first in the morning. And not jump out and bring the spare into production. #m
Re: The Making of a Router
On the topic of building a software router for an ISP, has anyone tried it using OpenFlow? The idea is to have a Linux server run BGP and a hardware switch to move the packets. The switch would be programmed by the Linux server using the OpenFlow protocol. I am looking at the HP 5400 zl switches as the hardware platform and RouteFlow https://sites.google.com/site/routeflow/ to program the BGP rules. One issue is that the HP switch will only allow a limited amount of rules to be processed in hardware (about 4096 rules I believe). Will this be enough to cover most of the traffic of a FTTH ISP on the fast path? Regards, Baldur
Re: The Making of a Router
On Fri, Dec 27, 2013 at 3:05 PM, Baldur Norddahl baldur.nordd...@gmail.comwrote: On the topic of building a software router for an ISP, has anyone tried it using OpenFlow? The idea is to have a Linux server run BGP and a hardware switch to move the packets. The switch would be programmed by the Linux server using the OpenFlow protocol. I am looking at the HP 5400 zl switches as the hardware platform and RouteFlow https://sites.google.com/site/routeflow/ to program the BGP rules. One issue is that the HP switch will only allow a limited amount of rules to be processed in hardware (about 4096 rules I believe). Will this be enough to cover most of the traffic of a FTTH ISP on the fast path? You want to use the switch for what ? To connect last-mile customers ? For L3 aggregation ? You want to run the switch as an edge router with limited BGP ? What's the exact use case you are thinking about ? Eugeniu
Re: The Making of a Router
You could look into Noviflow! F. Sent from my mobile device. Apologies for any typo. On Dec 27, 2013, at 8:05, Baldur Norddahl baldur.nordd...@gmail.com wrote: On the topic of building a software router for an ISP, has anyone tried it using OpenFlow? The idea is to have a Linux server run BGP and a hardware switch to move the packets. The switch would be programmed by the Linux server using the OpenFlow protocol. I am looking at the HP 5400 zl switches as the hardware platform and RouteFlow https://sites.google.com/site/routeflow/ to program the BGP rules. One issue is that the HP switch will only allow a limited amount of rules to be processed in hardware (about 4096 rules I believe). Will this be enough to cover most of the traffic of a FTTH ISP on the fast path? Regards, Baldur
Re: Mikrotik Cloud Core Router and BGP real life experiences?
On 12/27/2013 05:59 AM, Martin Hotze wrote: On 27/12/2013, at 10:13 pm, Martin Hotze m.ho...@hotze.com wrote: Thanks, estimated traffic levels are at about half a gig, but at least 50 megs of UDP (VoIP) in both directions. one thing is that I haven't found a solution for redundant power supply. Buy 2 :) on 3am I only want to read the notification and know what to do first in the morning. And not jump out and bring the spare into production. #m You set them both up configure the spare for fail-over.
Re: Mikrotik Cloud Core Router and BGP real life experiences?
My real world experience with these is that they suck. Plain and simple. Don't waste your time. On Dec 27, 2013 3:49 AM, Martin Hotze m.ho...@hotze.com wrote: Hi, looking at the specs of Mikrotik Cloud Core Routers it seems to be to good to be true [1] having so much bang for the bucks. So virtually all smaller ISPs would drop their CISCO gear for Mikrotik Routerboards. We are using a handful of Mikrotik boxes, but on a much lower network level (splitting networks; low end router behind ADSL modem, ...). We're happy with them. So I am asking for real life experience and not lab values with Mikrotik Cloud Core Routers and BGP. How good can they handle full tables and a bunch of peering sessions? How good does the box react when adding filters (during attacks)? Reloading the table? etc. etc. I am looking for _real_ _life_ values compared to a CISCO NPE-G2. Please tell me/us from your first hand experience. Thanks! greetings, Martin [1] If something sounds too good to be true, it probably is.
RE: Mikrotik Cloud Core Router and BGP real life experiences?
My real world experience with these is that they suck. Plain and simple. Don't waste your time. Would you mind elaborating what you were trying to accomplish and what failed? Thank you. Ray -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
Re: Mikrotik Cloud Core Router and BGP real life experiences?
People who tested say they don't forward more than 500Mbps per port. 2013/12/27 matt kelly mjke...@gmail.com My real world experience with these is that they suck. Plain and simple. Don't waste your time. On Dec 27, 2013 3:49 AM, Martin Hotze m.ho...@hotze.com wrote: Hi, looking at the specs of Mikrotik Cloud Core Routers it seems to be to good to be true [1] having so much bang for the bucks. So virtually all smaller ISPs would drop their CISCO gear for Mikrotik Routerboards. We are using a handful of Mikrotik boxes, but on a much lower network level (splitting networks; low end router behind ADSL modem, ...). We're happy with them. So I am asking for real life experience and not lab values with Mikrotik Cloud Core Routers and BGP. How good can they handle full tables and a bunch of peering sessions? How good does the box react when adding filters (during attacks)? Reloading the table? etc. etc. I am looking for _real_ _life_ values compared to a CISCO NPE-G2. Please tell me/us from your first hand experience. Thanks! greetings, Martin [1] If something sounds too good to be true, it probably is. -- Eduardo Schoedler
[SPAM]RE: [SPAM]Re: Mikrotik Cloud Core Router and BGP real life experiences?
Guess I should chime in here. As far as the CCR, I know several customers running in excess of 1 gig of traffic though them, one has 16 BGP sessions, several of those are full tables, and the rest are on an peering exchange. There are other units, like the ones we supply, that does more than 20 gig in real word usages. They are very capable devices, but depending on how many features you enable, of course that will affect their overall abilities. This would be real word, and yes, I work with 1000's of ISPs across North America, many between 100-10gig of traffic, cable companies, DSL providers, and WISPs, and many of these ONLY use MikroTik. As another person said, grab two and configure so that you split your load up, we have done that in areas where redundancy is important. Seeing the Dual 10GigE model with 8 GigE ports costs $1,249 or so, hard to beat them in price, and add two or more to get your redundancy. Dennis Burgess, Mikrotik Certified Trainer Author of Learn RouterOS- Second Edition Link Technologies, Inc -- Mikrotik WISP Support Services Office: 314-735-0270 Website: http://www.linktechs.net - Skype: linktechs -- Create Wireless Coverage's with www.towercoverage.com - 900Mhz - LTE - 3G - 3.65 - TV Whitespace -Original Message- From: Eduardo Schoedler [mailto:lis...@esds.com.br] Sent: Friday, December 27, 2013 8:10 AM To: NANOG list Subject: [SPAM]Re: Mikrotik Cloud Core Router and BGP real life experiences? People who tested say they don't forward more than 500Mbps per port. 2013/12/27 matt kelly mjke...@gmail.com My real world experience with these is that they suck. Plain and simple. Don't waste your time. On Dec 27, 2013 3:49 AM, Martin Hotze m.ho...@hotze.com wrote: Hi, looking at the specs of Mikrotik Cloud Core Routers it seems to be to good to be true [1] having so much bang for the bucks. So virtually all smaller ISPs would drop their CISCO gear for Mikrotik Routerboards. We are using a handful of Mikrotik boxes, but on a much lower network level (splitting networks; low end router behind ADSL modem, ...). We're happy with them. So I am asking for real life experience and not lab values with Mikrotik Cloud Core Routers and BGP. How good can they handle full tables and a bunch of peering sessions? How good does the box react when adding filters (during attacks)? Reloading the table? etc. etc. I am looking for _real_ _life_ values compared to a CISCO NPE-G2. Please tell me/us from your first hand experience. Thanks! greetings, Martin [1] If something sounds too good to be true, it probably is. -- Eduardo Schoedler
RE: Mikrotik Cloud Core Router and BGP real life experiences?
They can not handle a full routing table. The load balancing doesn't work. They can not properly reassemble fragmented packets, and therefore drop all but the first piece. They can not reliably handle traffic loads over maybe 200 Mbps, we needed 4-6 Gbps capacity. They can not hold a gre tunnel connection. On Dec 27, 2013 9:07 AM, Raymond Burkholder r...@oneunified.net wrote: My real world experience with these is that they suck. Plain and simple. Don't waste your time. Would you mind elaborating what you were trying to accomplish and what failed? Thank you. Ray -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
RE: Mikrotik Cloud Core Router and BGP real life experiences?
I too am curious... We've used them for a few months as edge devices and most (if not all) *knock on wood* of the issues we've had have been fixed by RouterOS updates, configuration changes (lots of chefs in the kitchen), or were circuit/carrier related. While I would never compare them apples-to-apples to Cisco, Juniper, etc devices... they have, in our experience, proven to be good inexpensive routers with a few quirks here and there. From: Raymond Burkholder [r...@oneunified.net] Sent: Friday, December 27, 2013 9:05 AM To: 'NANOG list' Subject: RE: Mikrotik Cloud Core Router and BGP real life experiences? My real world experience with these is that they suck. Plain and simple. Don't waste your time. Would you mind elaborating what you were trying to accomplish and what failed? Thank you. Ray -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
[SPAM]RE: [SPAM]RE: Mikrotik Cloud Core Router and BGP real life experiences?
We have many with full routing tables. Load balancing, works fine, I have one site with 8 DSL lines doing balancing across them. We typically don't use a GRE tunnel, but OpenVPN or IPSEC work great. Dennis Burgess, Mikrotik Certified Trainer Author of Learn RouterOS- Second Edition Link Technologies, Inc -- Mikrotik WISP Support Services Office: 314-735-0270 Website: http://www.linktechs.net - Skype: linktechs -- Create Wireless Coverage's with www.towercoverage.com - 900Mhz - LTE - 3G - 3.65 - TV Whitespace -Original Message- From: matt kelly [mailto:mjke...@gmail.com] Sent: Friday, December 27, 2013 8:41 AM To: Raymond Burkholder Cc: NANOG list Subject: [SPAM]RE: Mikrotik Cloud Core Router and BGP real life experiences? They can not handle a full routing table. The load balancing doesn't work. They can not properly reassemble fragmented packets, and therefore drop all but the first piece. They can not reliably handle traffic loads over maybe 200 Mbps, we needed 4-6 Gbps capacity. They can not hold a gre tunnel connection. On Dec 27, 2013 9:07 AM, Raymond Burkholder r...@oneunified.net wrote: My real world experience with these is that they suck. Plain and simple. Don't waste your time. Would you mind elaborating what you were trying to accomplish and what failed? Thank you. Ray -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
Re: The Making of a Router
I need a solution for everything except the last-mile customers. The customers are connected to a Zhone PON switch. From there they will arrive at our core switch as Q-in-Q vlans, one vlan per customer. I need a router that will do two full routing tables for our uplinks, a number of partial routing tables for our IX peers, IPv6 support, IPv4 proxy arp support and the ability to handle a large number of Q-in-Q vlans. And of course I will need two for redundancy. The uplinks, the links to edge switches and many of the IX peers are all 10 Gbit/s links. IPv4 proxy arp is especially important given the state of IPv4 exhaustion. Being a new ISP in the RIPE region, we only got 1024 IPs. When we run out of that initial assignment, we have to buy IP-addresses at a steep price. Therefore we can not afford to give each home a full IPv4 subnet. They will have to share the subnet with multiple other customers. This is achieved through proxy arp on the switch. We are an upstart and just buying the fancy Juniper switch times two would burn half of my seed capital. Like Nick Cameo I have seriously considered going with a Linux solution. I know I can build it. I just don't know if I can make it stable enough or make it perform good enough. I am looking into an OpenFlow solution as a middle ground. It allows me to buy cheaper switches/routers. The servers will do the thinking but the actual work of moving packets is still done in hardware on the switches. OpenFlow supports controller fail over, so I will not go down with just one server crash. Poor performance on the servers will not affect customer traffic directly. Regards, Baldur On Fri, Dec 27, 2013 at 2:11 PM, Eugeniu Patrascu eu...@imacandi.netwrote: On Fri, Dec 27, 2013 at 3:05 PM, Baldur Norddahl baldur.nordd...@gmail.com wrote: On the topic of building a software router for an ISP, has anyone tried it using OpenFlow? The idea is to have a Linux server run BGP and a hardware switch to move the packets. The switch would be programmed by the Linux server using the OpenFlow protocol. I am looking at the HP 5400 zl switches as the hardware platform and RouteFlow https://sites.google.com/site/routeflow/ to program the BGP rules. One issue is that the HP switch will only allow a limited amount of rules to be processed in hardware (about 4096 rules I believe). Will this be enough to cover most of the traffic of a FTTH ISP on the fast path? You want to use the switch for what ? To connect last-mile customers ? For L3 aggregation ? You want to run the switch as an edge router with limited BGP ? What's the exact use case you are thinking about ? Eugeniu
Re: The Making of a Router
On Dec 27, 2013 10:08 AM, Baldur Norddahl baldur.nordd...@gmail.com wrote: We are an upstart and just buying the fancy Juniper switch times two would burn half of my seed capital. Then you didn't ask for nearly enough capital.
Re: Mikrotik Cloud Core Router and BGP real life experiences?
FYI... Mikrotik Cloud Core routers are nice, however one has to keep something in mind when deploying them... Only One Core (of the CPU) is dedicated to each port / process. So this is good so as to contain what happens on a single port from taxing the whole CPU.. But not so good when you need more cpu power than a single core for that port. Also, BGP process will only use one core. While these units make for great 'customer facing' edge routers, with plenty of power and the ability to keep issues contained... The X-86 based (Core2Duo/i5/i7) Mikrotik are more suitable (Processing power wise) for running multiple full BGP tables peering. Regards Good Luck. Faisal Imtiaz Snappy Internet Telecom - Original Message - From: Geraint Jones gera...@koding.com To: Martin Hotze m.ho...@hotze.com Cc: nanog@nanog.org Sent: Friday, December 27, 2013 4:02:45 AM Subject: Re: Mikrotik Cloud Core Router and BGP real life experiences? I am going to be deploying 4 as edge routers in the next few weeks, each will have 1 or 2 full tables plus partial IX tables. So I should have some empirical info soon. They will be doing eBGP to upstreams and iBGP/OSPF internally. I went with the 16gb RAM models. However these boxes are basically Linux running on top of tilera CPUs, in terms of throughput as long as everything stays on the fastpath they have no issues doing wire speed on all ports, however the moment you add a firewall rule or the like they drop to 1.5gbps. On 27/12/2013, at 9:47 pm, Martin Hotze m.ho...@hotze.com wrote: Hi, looking at the specs of Mikrotik Cloud Core Routers it seems to be to good to be true [1] having so much bang for the bucks. So virtually all smaller ISPs would drop their CISCO gear for Mikrotik Routerboards. We are using a handful of Mikrotik boxes, but on a much lower network level (splitting networks; low end router behind ADSL modem, ...). We're happy with them. So I am asking for real life experience and not lab values with Mikrotik Cloud Core Routers and BGP. How good can they handle full tables and a bunch of peering sessions? How good does the box react when adding filters (during attacks)? Reloading the table? etc. etc. I am looking for _real_ _life_ values compared to a CISCO NPE-G2. Please tell me/us from your first hand experience. Thanks! greetings, Martin [1] If something sounds too good to be true, it probably is.
Re: The Making of a Router
On gio, 2013-12-26 at 11:33 -0500, Nick Cameo wrote: Hello Everyone, We are looking to put together a 2u server with a few PCIe 3 x8 (recommendations appreciated). The router will take a voip transcoding line card, and will act as an edge router for a telecom company. For things like BGP (Quagga, Zebra, all that lovely stuff!!!), static routes, and firewall capabilities we are thinking gentoo linux stripped for sure however, what about the BSDs? FreeBSD or OpenBSD. Any comments, feedback, does, and don'ts are much appreciated. Kind Regards, Nick. I would definitely consider Alpine Linux [1], which is designed exactly for that purpose. Small footprint, run-from-ram, config from on flash, security oriented. KR, - leo [1] http://alpinelinux.org/about
Re: The Making of a Router
On Thu, 26 Dec 2013, Andrew D Kirch wrote: If he can afford a 10G link... he should be buying real gear... I mean, look, I've got plenty of infrastructure horror stories, but lets not cobble together our own 10gbit solutions, please? At least get one of the new microtik CCR's with a 10gig sfp+? They're only a kilobuck... If you can't afford that I suggest you can't afford to be an ISP. +1 Build-your-own routers are perfectly OK for a lab environment if you want to tinker with something, but I absolutely would not put an all-in-one box that I built myself in production. You end up combining some of the downsides of a hardware-based router with some of the downsides of a server (new attack vectors, another device that needs to be backed up, patched, and monitored, possibly getting a new collection of devices and drivers to play nicely with each other, etc). Doing this also requires all of the people in your on-call rotation to be experienced sysadmins / server ops, in addition to being experiences network engineers / NOC ops. There are a lot of occasions with a server where 'just reboot it' can make a problem much worse. Route servers running Linux or *BSD are another story. There are many situations where they can be extremely useful, but they are not all-in-one route server/RADIUS/VPN termination/web server/user shell boxes. jms
Re: Mikrotik Cloud Core Router and BGP real life experiences?
Exactly what Faisal Said. The BGP process appears to be single threaded at the moment. So taking on full BGP tables can be a bit slow compared to a decent X86 box. But in terms of raw forwarding power they are pretty monstrous. We replaced a few Maxxwave 6 port Atom's with the CCR. ~400Mb/s and ~40K pps aggregate across all ports. CPU load went from ~25% to ~0-2%. These are in a configuration where they have little or no firewall/nat/queue rules. And in most cases are running MPLS. We've not had any issues with stability so far either (Knock on wood). Nick Olsen Network Operations (855) FLSPEED x106 From: Faisal Imtiaz fai...@snappytelecom.net Sent: Friday, December 27, 2013 10:33 AM To: Geraint Jones gera...@koding.com Cc: nanog@nanog.org, Martin Hotze m.ho...@hotze.com Subject: Re: Mikrotik Cloud Core Router and BGP real life experiences? FYI... Mikrotik Cloud Core routers are nice, however one has to keep something in mind when deploying them... Only One Core (of the CPU) is dedicated to each port / process. So this is good so as to contain what happens on a single port from taxing the whole CPU.. But not so good when you need more cpu power than a single core for that port. Also, BGP process will only use one core. While these units make for great 'customer facing' edge routers, with plenty of power and the ability to keep issues contained... The X-86 based (Core2Duo/i5/i7) Mikrotik are more suitable (Processing power wise) for running multiple full BGP tables peering. Regards Good Luck. Faisal Imtiaz Snappy Internet Telecom - Original Message - From: Geraint Jones gera...@koding.com To: Martin Hotze m.ho...@hotze.com Cc: nanog@nanog.org Sent: Friday, December 27, 2013 4:02:45 AM Subject: Re: Mikrotik Cloud Core Router and BGP real life experiences? I am going to be deploying 4 as edge routers in the next few weeks, each will have 1 or 2 full tables plus partial IX tables. So I should have some empirical info soon. They will be doing eBGP to upstreams and iBGP/OSPF internally. I went with the 16gb RAM models. However these boxes are basically Linux running on top of tilera CPUs, in terms of throughput as long as everything stays on the fastpath they have no issues doing wire speed on all ports, however the moment you add a firewall rule or the like they drop to 1.5gbps. On 27/12/2013, at 9:47 pm, Martin Hotze m.ho...@hotze.com wrote: Hi, looking at the specs of Mikrotik Cloud Core Routers it seems to be to good to be true [1] having so much bang for the bucks. So virtually all smaller ISPs would drop their CISCO gear for Mikrotik Routerboards. We are using a handful of Mikrotik boxes, but on a much lower network level (splitting networks; low end router behind ADSL modem, ...). We're happy with them. So I am asking for real life experience and not lab values with Mikrotik Cloud Core Routers and BGP. How good can they handle full tables and a bunch of peering sessions? How good does the box react when adding filters (during attacks)? Reloading the table? etc. etc. I am looking for _real_ _life_ values compared to a CISCO NPE-G2. Please tell me/us from your first hand experience. Thanks! greetings, Martin [1] If something sounds too good to be true, it probably is.
Re: Mikrotik Cloud Core Router and BGP real life experiences?
PPPoE Server is single thread too. 2013/12/27 Nick Olsen n...@flhsi.com Exactly what Faisal Said. The BGP process appears to be single threaded at the moment. So taking on full BGP tables can be a bit slow compared to a decent X86 box. But in terms of raw forwarding power they are pretty monstrous. We replaced a few Maxxwave 6 port Atom's with the CCR. ~400Mb/s and ~40K pps aggregate across all ports. CPU load went from ~25% to ~0-2%. These are in a configuration where they have little or no firewall/nat/queue rules. And in most cases are running MPLS. We've not had any issues with stability so far either (Knock on wood). Nick Olsen Network Operations (855) FLSPEED x106 From: Faisal Imtiaz fai...@snappytelecom.net Sent: Friday, December 27, 2013 10:33 AM To: Geraint Jones gera...@koding.com Cc: nanog@nanog.org, Martin Hotze m.ho...@hotze.com Subject: Re: Mikrotik Cloud Core Router and BGP real life experiences? FYI... Mikrotik Cloud Core routers are nice, however one has to keep something in mind when deploying them... Only One Core (of the CPU) is dedicated to each port / process. So this is good so as to contain what happens on a single port from taxing the whole CPU.. But not so good when you need more cpu power than a single core for that port. Also, BGP process will only use one core. While these units make for great 'customer facing' edge routers, with plenty of power and the ability to keep issues contained... The X-86 based (Core2Duo/i5/i7) Mikrotik are more suitable (Processing power wise) for running multiple full BGP tables peering. Regards Good Luck. Faisal Imtiaz Snappy Internet Telecom - Original Message - From: Geraint Jones gera...@koding.com To: Martin Hotze m.ho...@hotze.com Cc: nanog@nanog.org Sent: Friday, December 27, 2013 4:02:45 AM Subject: Re: Mikrotik Cloud Core Router and BGP real life experiences? I am going to be deploying 4 as edge routers in the next few weeks, each will have 1 or 2 full tables plus partial IX tables. So I should have some empirical info soon. They will be doing eBGP to upstreams and iBGP/OSPF internally. I went with the 16gb RAM models. However these boxes are basically Linux running on top of tilera CPUs, in terms of throughput as long as everything stays on the fastpath they have no issues doing wire speed on all ports, however the moment you add a firewall rule or the like they drop to 1.5gbps. On 27/12/2013, at 9:47 pm, Martin Hotze m.ho...@hotze.com wrote: Hi, looking at the specs of Mikrotik Cloud Core Routers it seems to be to good to be true [1] having so much bang for the bucks. So virtually all smaller ISPs would drop their CISCO gear for Mikrotik Routerboards. We are using a handful of Mikrotik boxes, but on a much lower network level (splitting networks; low end router behind ADSL modem, ...). We're happy with them. So I am asking for real life experience and not lab values with Mikrotik Cloud Core Routers and BGP. How good can they handle full tables and a bunch of peering sessions? How good does the box react when adding filters (during attacks)? Reloading the table? etc. etc. I am looking for _real_ _life_ values compared to a CISCO NPE-G2. Please tell me/us from your first hand experience. Thanks! greetings, Martin [1] If something sounds too good to be true, it probably is. -- Eduardo Schoedler
Re: Mikrotik Cloud Core Router and BGP real life experiences?
On 12/27/13 6:40 AM, matt kelly wrote: They can not handle a full routing table. The load balancing doesn't work. They can not properly reassemble fragmented packets, and therefore drop all but the first piece. They can not reliably handle traffic loads over maybe 200 Mbps, we needed 4-6 Gbps capacity. They can not hold a gre tunnel connection. Can't say anything about MicroTik specifically, but I've used Linux as a routing platform for many years, off and on, and took a reasonably close look at performance about a year ago, in the previous job, using relatively high-end, but pre-Sandy Bridge, generic hardware. We were looking to support ca. 8 x 10 GbE ports with several full tables, and the usual suspects wanted the usual 6-figure amounts for boxes that could do that (the issue being the full routes -- 8 x 10 GbE with minimal routing is a triviality these days). Routing table size was completely not an issue in our environment; we were looking at a number of concurrent flows in the high-5 to low-6-digit range, and since Linux uses a route cache, it was that number, rather than the number of full tables we carried, that was important. Doing store-and-forward packet processing, as opposed to cut-through switching, took about 5 microseconds per packet, and consumed about that much CPU time. The added latency was not an issue for us; but at 5 us, that's 200Kpps per CPU. With 1500-byte packets, that's about 2.4 Gb/s total throughput; but with 40-byte packets, it's only 64 Mb/s (!). But that's per CPU. Our box had 24 CPUs (if you count a hyperthreaded pair as 2), and this work is eminently parallelizable. So a theoretical upper bound on throughput with this box would have been 4.8 Mpps -- 57.6 Gb/s with 1500-byte packets, 1.5 Gb/s with 40-byte packets. The Linux network stack (plus RSS on the NICs) seemed to do quite a good job of input-side parallelism - but we saw a lot of lock contention on the output side. At that point, we abandoned the project, as it was incidental to the organization's mission. I think that with a little more work, we could have gotten within, say, a factor of 2 of the limits above, which would have been good enough for us (though surely not for everybody). Incrementally faster hardware would have incrementally better performance. OpenFlow, which marries cheap, fast, and dumb ASICs with cheap, slower, and infinitely flexible generic CPU and RAM, seemed, and still seems, like the clearly right approach. At the time, it didn't seem ready for prime time, either in the selection of OpenFlow-capable routers or in the software stack. I imagine there's been some progress made since. Whether the market will allow it to flourish is another question. Below a certain maximum throughput, routing with generic boxes is actually pretty easy. Today, I'd say that maximum is roughly in the low-single-gigabit range. Higher is possible, but gets progressively harder to get right (and it's not a firm bound, anyway, as it depends on traffic mix and other requirements). Whether it's worth doing really depends on your goals and skill. Most people will probably prefer a canned solution from a vendor. People who grow and eat their own food surely eat better, and more cheaply, than those who buy at the supermarket; but it's not for everybody. Jim Shankland
Megapath/Covad DNS broken?
Hello, Seems like some Megapath/Covad DNS servers are non-responsive: 64.105.172.26 64.105.172.27 We have dozens of end-users down that utilize these servers. Anyone from MP/Covad alive this morning?? Thanks, -Bobby
RE: Help me make sense of these traceroutes please
Thanks to everyone who responded off list and on. Sam Moats On 2013-12-26 11:21, Josephson, Marcus wrote: Start at slide 50: This is documented further by the following Nanog presentation. http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N47_Sun.pdf -Marcus -Original Message- From: Jimmy Hess [mailto:mysi...@gmail.com] Sent: Wednesday, December 25, 2013 10:28 AM To: Martin Hotze Cc: nanog@nanog.org Subject: Re: Help me make sense of these traceroutes please On Wed, Dec 25, 2013 at 8:03 AM, Martin Hotze m.ho...@hotze.com wrote: On 2013-12-25 00:16, Sam Moats wrote: ... You are likely seeing the effects of asymmetric routing. . .. or the effect of passing traffic through NSA infrastructure. Ah... NSA. That's probably it. So much for my theory of a Router virtual chassis straddling the atlantic. or the extra kinetic energy carried by the overseas-bound packet took longer for the router to absorb and rebound with an ICMP. But in all seriousness --- what is probably happening here, is the result of extra hops that don't show up in traceroute. MPLS tunnels could well fit the bill. Other things to consider when latency seems sensitive to destination IP --- are preceding device in the traceroute might also have multiple links to the same device; with one link congested and some form of IP-based load sharing, that happens to be the toward-overseas link. SCNR, #m -- -JH
Re: Mikrotik Cloud Core Router and BGP real life experiences?
The issues I see are because of routers versions. The Cloud core routers are a fairly new platform. As such, the software isn¹t as stable as it should be. The OS is up to version 6.7. There were some betas before 6.0 was released. However, almost every version that has been released addresses issues with the cloud core. The cloud cores only run Version 6. We did se BGP issues early on accepting more than one full routing table. We saw other issues but they were fixed with subsequent OS software releases. Justin -- Justin Wilson j...@mtin.net MTCNA CCNA MTCRE MTCWE - COMTRAIN Aol Yahoo IM: j2sw http://www.mtin.net/blog xISP News http://www.zigwireless.com High Speed Internet Options http://www.thebrotherswisp.com The Brothers Wisp -Original Message- From: Jim Shankland na...@shankland.org Date: Friday, December 27, 2013 at 11:26 AM To: nanog@nanog.org Subject: Re: Mikrotik Cloud Core Router and BGP real life experiences? On 12/27/13 6:40 AM, matt kelly wrote: They can not handle a full routing table. The load balancing doesn't work. They can not properly reassemble fragmented packets, and therefore drop all but the first piece. They can not reliably handle traffic loads over maybe 200 Mbps, we needed 4-6 Gbps capacity. They can not hold a gre tunnel connection. Can't say anything about MicroTik specifically, but I've used Linux as a routing platform for many years, off and on, and took a reasonably close look at performance about a year ago, in the previous job, using relatively high-end, but pre-Sandy Bridge, generic hardware. We were looking to support ca. 8 x 10 GbE ports with several full tables, and the usual suspects wanted the usual 6-figure amounts for boxes that could do that (the issue being the full routes -- 8 x 10 GbE with minimal routing is a triviality these days). Routing table size was completely not an issue in our environment; we were looking at a number of concurrent flows in the high-5 to low-6-digit range, and since Linux uses a route cache, it was that number, rather than the number of full tables we carried, that was important. Doing store-and-forward packet processing, as opposed to cut-through switching, took about 5 microseconds per packet, and consumed about that much CPU time. The added latency was not an issue for us; but at 5 us, that's 200Kpps per CPU. With 1500-byte packets, that's about 2.4 Gb/s total throughput; but with 40-byte packets, it's only 64 Mb/s (!). But that's per CPU. Our box had 24 CPUs (if you count a hyperthreaded pair as 2), and this work is eminently parallelizable. So a theoretical upper bound on throughput with this box would have been 4.8 Mpps -- 57.6 Gb/s with 1500-byte packets, 1.5 Gb/s with 40-byte packets. The Linux network stack (plus RSS on the NICs) seemed to do quite a good job of input-side parallelism - but we saw a lot of lock contention on the output side. At that point, we abandoned the project, as it was incidental to the organization's mission. I think that with a little more work, we could have gotten within, say, a factor of 2 of the limits above, which would have been good enough for us (though surely not for everybody). Incrementally faster hardware would have incrementally better performance. OpenFlow, which marries cheap, fast, and dumb ASICs with cheap, slower, and infinitely flexible generic CPU and RAM, seemed, and still seems, like the clearly right approach. At the time, it didn't seem ready for prime time, either in the selection of OpenFlow-capable routers or in the software stack. I imagine there's been some progress made since. Whether the market will allow it to flourish is another question. Below a certain maximum throughput, routing with generic boxes is actually pretty easy. Today, I'd say that maximum is roughly in the low-single-gigabit range. Higher is possible, but gets progressively harder to get right (and it's not a firm bound, anyway, as it depends on traffic mix and other requirements). Whether it's worth doing really depends on your goals and skill. Most people will probably prefer a canned solution from a vendor. People who grow and eat their own food surely eat better, and more cheaply, than those who buy at the supermarket; but it's not for everybody. Jim Shankland
Weekly Routing Table Report
This is an automated weekly mailing describing the state of the Internet Routing Table as seen from APNIC's router in Japan. The posting is sent to APOPS, NANOG, AfNOG, AusNOG, SANOG, PacNOG, LacNOG, TRNOG, CaribNOG and the RIPE Routing Working Group. Daily listings are sent to bgp-st...@lists.apnic.net For historical data, please see http://thyme.rand.apnic.net. If you have any comments please contact Philip Smith pfsi...@gmail.com. Routing Table Report 04:00 +10GMT Sat 28 Dec, 2013 Report Website: http://thyme.rand.apnic.net Detailed Analysis: http://thyme.rand.apnic.net/current/ Analysis Summary BGP routing table entries examined: 476821 Prefixes after maximum aggregation: 190476 Deaggregation factor: 2.50 Unique aggregates announced to Internet: 236639 Total ASes present in the Internet Routing Table: 45824 Prefixes per ASN: 10.41 Origin-only ASes present in the Internet Routing Table: 35504 Origin ASes announcing only one prefix: 16271 Transit ASes present in the Internet Routing Table:5971 Transit-only ASes present in the Internet Routing Table:176 Average AS path length visible in the Internet Routing Table: 4.6 Max AS path length visible: 53 Max AS path prepend of ASN ( 50404) 51 Prefixes from unregistered ASNs in the Routing Table: 2660 Unregistered ASNs in the Routing Table: 629 Number of 32-bit ASNs allocated by the RIRs: 5609 Number of 32-bit ASNs visible in the Routing Table:4349 Prefixes from 32-bit ASNs in the Routing Table: 13678 Number of bogon 32-bit ASNs visible in the Routing Table: 1 Special use prefixes present in the Routing Table:0 Prefixes being announced from unallocated address space: 1520 Number of addresses announced to Internet: 2661350716 Equivalent to 158 /8s, 160 /16s and 253 /24s Percentage of available address space announced: 71.9 Percentage of allocated address space announced: 71.9 Percentage of available address space allocated: 100.0 Percentage of address space in use by end-sites: 95.4 Total number of prefixes smaller than registry allocations: 166420 APNIC Region Analysis Summary - Prefixes being announced by APNIC Region ASes: 113329 Total APNIC prefixes after maximum aggregation: 34207 APNIC Deaggregation factor:3.31 Prefixes being announced from the APNIC address blocks: 115615 Unique aggregates announced from the APNIC address blocks:48424 APNIC Region origin ASes present in the Internet Routing Table:4874 APNIC Prefixes per ASN: 23.72 APNIC Region origin ASes announcing only one prefix: 1207 APNIC Region transit ASes present in the Internet Routing Table:836 Average APNIC Region AS path length visible:4.6 Max APNIC Region AS path length visible: 28 Number of APNIC region 32-bit ASNs visible in the Routing Table:800 Number of APNIC addresses announced to Internet: 729291968 Equivalent to 43 /8s, 120 /16s and 28 /24s Percentage of available APNIC address space announced: 85.2 APNIC AS Blocks4608-4864, 7467-7722, 9216-10239, 17408-18431 (pre-ERX allocations) 23552-24575, 37888-38911, 45056-46079, 55296-56319, 58368-59391, 63488-63999, 131072-133631 APNIC Address Blocks 1/8, 14/8, 27/8, 36/8, 39/8, 42/8, 43/8, 49/8, 58/8, 59/8, 60/8, 61/8, 101/8, 103/8, 106/8, 110/8, 111/8, 112/8, 113/8, 114/8, 115/8, 116/8, 117/8, 118/8, 119/8, 120/8, 121/8, 122/8, 123/8, 124/8, 125/8, 126/8, 133/8, 150/8, 153/8, 163/8, 171/8, 175/8, 180/8, 182/8, 183/8, 202/8, 203/8, 210/8, 211/8, 218/8, 219/8, 220/8, 221/8, 222/8, 223/8, ARIN Region Analysis Summary Prefixes being announced by ARIN Region ASes:163452 Total ARIN prefixes after maximum aggregation:81927 ARIN Deaggregation factor: 2.00 Prefixes being announced from the ARIN address blocks: 164176 Unique aggregates announced from the ARIN address blocks: 75931 ARIN Region origin ASes present in the Internet Routing Table:15934 ARIN
Re: Mikrotik Cloud Core Router and BGP real life experiences?
On 12/27/13, 10:01, Justin Wilson wrote: The issues I see are because of routers versions. The Cloud core routers are a fairly new platform. As such, the software isn¹t as stable as it should be. The OS is up to version 6.7. There were some betas before 6.0 was released. However, almost every version that has been released addresses issues with the cloud core. The cloud cores only run Version 6. Unless my knowledge is out of date, the one thing RouterOS has that others in the same scope lack is a full MPLS stack that's not experimental. ~Seth
Re: Mikrotik Cloud Core Router and BGP real life experiences?
nanog-requ...@nanog.org said: We replaced a few Maxxwave 6 port Atom's with the CCR. ~400Mb/s and ~40K pps aggregate across all ports. CPU load went from ~25% to ~0-2%. These are in a configuration where they have little or no firewall/nat/queue rules. And in most cases are running MPLS. How much CPU does it take to implement BCP-38? -- These are my opinions. I hate spam.
AOL Postmaster
Can a AOL Postmaster hit me off-list please J Dennis Burgess, Mikrotik Certified Trainer Author of Learn RouterOS- Second Edition http://www.wlan1.com/product_p/mikrotik%20book-2.htm Link Technologies, Inc -- Mikrotik WISP Support Services Office: 314-735-0270 tel:314-735-0270 Website: http://www.linktechs.net http://www.linktechs.net/ - Skype: linktechs skype:linktechs?call -- Create Wireless Coverage's with www.towercoverage.com http://www.towercoverage.com/ - 900Mhz - LTE - 3G - 3.65 - TV Whitespace
Re: The Making of a Router
On Fri, Dec 27, 2013 at 4:18 PM, Jon Sands fohdee...@gmail.com wrote: On Dec 27, 2013 10:08 AM, Baldur Norddahl baldur.nordd...@gmail.com wrote: We are an upstart and just buying the fancy Juniper switch times two would burn half of my seed capital. Then you didn't ask for nearly enough capital. Another told Nick Cameo that if he can afford a 10G link, he can afford Juniper. You could not be more wrong. The 10G uplink goes for $0 in initial fee and less than $4k / month with unlimited traffic. The Juniper gear is $100k up front for two routers able to handle the 10G links. What I get from you guys is that in your opinion it is not possible to set up a small ISP without spending a ton on Juniper or Cisco. I am not buying that. Even if I did not have a clear limit on my capital, I would be looking at avoiding paying that kind of money, because in the end the money comes out of my own pocket. Everybody have critical services running on servers. DHCP, DNS, Radius and so on are all on servers and you will be down if these services are down. What is with the knee jerk reaction for suggesting that the BGP daemon could also be run on a server? There seems to be many advantages of doing it this way, and not all of them are related to cost. Regards, Baldur
Re: Mikrotik Cloud Core Router and BGP real life experiences?
Depends. This router isn't running BCP-38 as it's run at our borders. Looking at the specs. Firewall rules do take it out of fastpath. Which means it's going to take a decent performance hit on paper. I'm not sure if their auto method of enabling BCP-38 IE. the IPSettingsRP Filter method would accomplish the same outcome, Without taking the router out of fastpath. I assume it works the same as using firewall rules. Just Behind the scenes. That being said, Real world testing. Running the traffic levels I mentioned before. I put a single simple firewall rule on the router. Which effectively took it out of fastpath. And also enabled connection tracking. I saw no noticeable change in CPU load. Nick Olsen Network Operations (855) FLSPEED x106 From: Hal Murray hmur...@megapathdsl.net Sent: Friday, December 27, 2013 2:38 PM To: nanog@nanog.org Cc: Hal Murray hmur...@megapathdsl.net Subject: Re: Mikrotik Cloud Core Router and BGP real life experiences? nanog-requ...@nanog.org said: We replaced a few Maxxwave 6 port Atom's with the CCR. ~400Mb/s and ~40K pps aggregate across all ports. CPU load went from ~25% to ~0-2%. These are in a configuration where they have little or no firewall/nat/queue rules. And in most cases are running MPLS. How much CPU does it take to implement BCP-38? -- These are my opinions. I hate spam.
Re: The Making of a Router
On Fri, Dec 27, 2013 at 10:00 PM, Baldur Norddahl baldur.nordd...@gmail.com wrote: On Fri, Dec 27, 2013 at 4:18 PM, Jon Sands fohdee...@gmail.com wrote: On Dec 27, 2013 10:08 AM, Baldur Norddahl baldur.nordd...@gmail.com wrote: We are an upstart and just buying the fancy Juniper switch times two would burn half of my seed capital. Then you didn't ask for nearly enough capital. Another told Nick Cameo that if he can afford a 10G link, he can afford Juniper. You could not be more wrong. The 10G uplink goes for $0 in initial fee and less than $4k / month with unlimited traffic. The Juniper gear is $100k up front for two routers able to handle the 10G links. What you should understand is not the fact that a 10G interface is expensive, but what you can do with that interface tends to get very expensive. If you want to move traffic from one interface to another, you can achieve this today with two physical interfaces on a Linux box. How many PPS ? Well, that's another story. You then want shaping, Q-in-Q and other stuff which consume a lot of resources even on dedicated hardware. What I get from you guys is that in your opinion it is not possible to set up a small ISP without spending a ton on Juniper or Cisco. I am not buying that. Even if I did not have a clear limit on my capital, I would be looking at avoiding paying that kind of money, because in the end the money comes out of my own pocket. You can build your ISP without getting big routers but you need to cut back a little bit on your expectations about what you can in terms of features: - Do pool NAT for your users if they accept this. You can easily squeeze a lot of users on a single IP address. Downside is that if one of them does something bad, that IP might get blackholed on some providers and the rest will suffer. Also, you might want to take into consideration regulatory requirements like to know what users used what port to what destination for a certain number of months (in Europe regulations vary, but the smallest period is 6 months). - If you give them VoIP/IPTV then assign a VLAN for VOIP and another for IPTV and run it to all your users to their STBs and make use of IGMP snooping for Multicast traffic on all your switches - You can run full table BGP with Quagga on Linux (it worked for me when the DFZ was at around 270k prefixes, I assume it will work with 480k prefixes today) - also, do you really need full tables ?. Your IGP, if you don't run anything fancy should be a few tens of routes, that can be achieved with modest L3 switches that do 64/128 routes in hardware. Everybody have critical services running on servers. DHCP, DNS, Radius and so on are all on servers and you will be down if these services are down. What is with the knee jerk reaction for suggesting that the BGP daemon could also be run on a server? There seems to be many advantages of doing it this way, and not all of them are related to cost. For the sake of a good night sleep, you would want to separate all the services on different physical machines for redundancy/availability and load sharing. Once you grow, you can move to more powerful and dedicated hardware for your networking needs. Eugeniu
Re: The Making of a Router
Well said Baldur For those who are movie buffs.. here is the snippet that visually summaries.. http://www.youtube.com/watch?v=dEkOT3IngMQ As to the knee jerk reaction to a server doing routing such folks tend to forget that Routers are purpose built serversand most of the Internet Routing was originally done by servers (or main frames or minis) etc. :) Faisal Imtiaz Snappy Internet Telecom 7266 SW 48 Street Miami, FL 33155 Tel: 305 663 5518 x 232 Help-desk: (305)663-5518 Option 2 or Email: supp...@snappytelecom.net - Original Message - What I get from you guys is that in your opinion it is not possible to set up a small ISP without spending a ton on Juniper or Cisco. I am not buying that. Even if I did not have a clear limit on my capital, I would be looking at avoiding paying that kind of money, because in the end the money comes out of my own pocket. Everybody have critical services running on servers. DHCP, DNS, Radius and so on are all on servers and you will be down if these services are down. What is with the knee jerk reaction for suggesting that the BGP daemon could also be run on a server? There seems to be many advantages of doing it this way, and not all of them are related to cost. Regards, Baldur
Re: Mikrotik Cloud Core Router and BGP real life experiences?
On 27. des. 2013 17:26, Jim Shankland wrote: snip Routing table size was completely not an issue in our environment; we were looking at a number of concurrent flows in the high-5 to low-6-digit range, and since Linux uses a route cache, it was that number, rather than the number of full tables we carried, that was important. snip FYI, Linux no longer has a routing cache, so any performance numbers with the cache in place is void on modern kernels. It was deemed too fragile, handled mixed traffic badly, and was way easy to DoS. It wasnt simply just ripped out of course, the full lookups was made way faster and a bunch of scalability issues got plugged in the process. All in all, in PPS, Linux should now handle mixed traffic much better, but less diverse traffic patterns might be a little slower than before. However, all in all, much more consistent and predictable. Not everything is peachy though, there are still some cases that sucked last I checked. Running tons of tunnels beeing one. Multicast rx was severely gimped for a while after the removal, but that got fixed.
Re: The Making of a Router
I Am not trying to be inflammatory.. Why does everything has to be measured in Extremes ? e.g. If someone says I need a 10g interface, why is it automatically assumed that the router is going to be running @ Full Line Rate ? I think we often confuse the performance of Software and Hardware as being monolithic, which is far from the truth. In today fast moving world, we don't even bother to look under the hood to see what is it that is being taxed.. the Software ? or Hardware ? or process of how we are doing what we want to get accomplished ? While it is easy to talk about the extremes, we often forget to point out to others that the 'Middle' is a huge place. Regards Faisal Imtiaz Snappy Internet Telecom - Original Message - From: Eugeniu Patrascu eu...@imacandi.net To: Baldur Norddahl baldur.nordd...@gmail.com Cc: nanog@nanog.org Sent: Friday, December 27, 2013 3:23:11 PM Subject: Re: The Making of a Router On Fri, Dec 27, 2013 at 10:00 PM, Baldur Norddahl baldur.nordd...@gmail.com wrote: On Fri, Dec 27, 2013 at 4:18 PM, Jon Sands fohdee...@gmail.com wrote: On Dec 27, 2013 10:08 AM, Baldur Norddahl baldur.nordd...@gmail.com wrote: We are an upstart and just buying the fancy Juniper switch times two would burn half of my seed capital. Then you didn't ask for nearly enough capital. Another told Nick Cameo that if he can afford a 10G link, he can afford Juniper. You could not be more wrong. The 10G uplink goes for $0 in initial fee and less than $4k / month with unlimited traffic. The Juniper gear is $100k up front for two routers able to handle the 10G links. What you should understand is not the fact that a 10G interface is expensive, but what you can do with that interface tends to get very expensive. If you want to move traffic from one interface to another, you can achieve this today with two physical interfaces on a Linux box. How many PPS ? Well, that's another story. You then want shaping, Q-in-Q and other stuff which consume a lot of resources even on dedicated hardware. What I get from you guys is that in your opinion it is not possible to set up a small ISP without spending a ton on Juniper or Cisco. I am not buying that. Even if I did not have a clear limit on my capital, I would be looking at avoiding paying that kind of money, because in the end the money comes out of my own pocket. You can build your ISP without getting big routers but you need to cut back a little bit on your expectations about what you can in terms of features: - Do pool NAT for your users if they accept this. You can easily squeeze a lot of users on a single IP address. Downside is that if one of them does something bad, that IP might get blackholed on some providers and the rest will suffer. Also, you might want to take into consideration regulatory requirements like to know what users used what port to what destination for a certain number of months (in Europe regulations vary, but the smallest period is 6 months). - If you give them VoIP/IPTV then assign a VLAN for VOIP and another for IPTV and run it to all your users to their STBs and make use of IGMP snooping for Multicast traffic on all your switches - You can run full table BGP with Quagga on Linux (it worked for me when the DFZ was at around 270k prefixes, I assume it will work with 480k prefixes today) - also, do you really need full tables ?. Your IGP, if you don't run anything fancy should be a few tens of routes, that can be achieved with modest L3 switches that do 64/128 routes in hardware. Everybody have critical services running on servers. DHCP, DNS, Radius and so on are all on servers and you will be down if these services are down. What is with the knee jerk reaction for suggesting that the BGP daemon could also be run on a server? There seems to be many advantages of doing it this way, and not all of them are related to cost. For the sake of a good night sleep, you would want to separate all the services on different physical machines for redundancy/availability and load sharing. Once you grow, you can move to more powerful and dedicated hardware for your networking needs. Eugeniu
Re: Mikrotik Cloud Core Router and BGP real life experiences?
How about SMP Affinity in CCR? System Resources IRQ. 2013/12/27 Andre Tomt andre-na...@tomt.net On 27. des. 2013 17:26, Jim Shankland wrote: snip Routing table size was completely not an issue in our environment; we were looking at a number of concurrent flows in the high-5 to low-6-digit range, and since Linux uses a route cache, it was that number, rather than the number of full tables we carried, that was important. snip FYI, Linux no longer has a routing cache, so any performance numbers with the cache in place is void on modern kernels. It was deemed too fragile, handled mixed traffic badly, and was way easy to DoS. It wasnt simply just ripped out of course, the full lookups was made way faster and a bunch of scalability issues got plugged in the process. All in all, in PPS, Linux should now handle mixed traffic much better, but less diverse traffic patterns might be a little slower than before. However, all in all, much more consistent and predictable. Not everything is peachy though, there are still some cases that sucked last I checked. Running tons of tunnels beeing one. Multicast rx was severely gimped for a while after the removal, but that got fixed. -- Eduardo Schoedler
Re: The Making of a Router
On Dec 27, 2013, at 3:37 PM, Faisal Imtiaz fai...@snappytelecom.net wrote: e.g. If someone says I need a 10g interface, why is it automatically assumed that the router is going to be running @ Full Line Rate ? Those of us with experience know that when “something bad(tm)” happens, those features and “expensive silicon” start to show some ROI. Is it a full trade-off? Depends on the risks of your business and exposure. You can get some inexpensive hardware to do fairly fancy features these days. That can be very good, but caries that risk. Make sure you evaluate it carefully. - Jared
Re: The Making of a Router
On Fri, 27 Dec 2013, Baldur Norddahl wrote: Everybody have critical services running on servers. DHCP, DNS, Radius and so on are all on servers and you will be down if these services are down. What is with the knee jerk reaction for suggesting that the BGP daemon could also be run on a server? There seems to be many advantages of doing it this way, and not all of them are related to cost. If you want to use servers as routers, that's your choice. I think what most people in the thread have been saying is not to use one server (or even a pair of servers) for everything. It's one thing if server XYZ goes down and some web services are offline. It's another thing entirely if that same server goes down and your entire business is offline. jms
Re: The Making of a Router
On Fri, Dec 27, 2013 at 10:18:47AM -0500, Jon Sands wrote: On Dec 27, 2013 10:08 AM, Baldur Norddahl baldur.nordd...@gmail.com wrote: We are an upstart and just buying the fancy Juniper switch times two would burn half of my seed capital. Then you didn't ask for nearly enough capital. There *is* a world outside of Silly Valley, you know... a world where money doesn't flow like a mighty cascade from the benevolent wallets of vulture capitalists, into the waiting arms of every crackpot with an elevator pitch. - Matt
Re: The Making of a Router
Fair point.. but in real life, isn't that true for everything... I say the same be familiar(honest awareness) with the limits (limitations) and capabilities of your specific solution, be it a 'dyi' or a commercial solution, before pushing it to the limit. Unless of course, you have factored in the ability to deal with the consequences. Most 'DYI' solutions, make the non-techy bean counters very nervous, and seeing a major 'name brand' label for some odd reason makes them real comfortable, ir-respective of the capabilities or function of either solution. If you have to answer to the bean counters, then this is a very valid point to be considered. :) Faisal Imtiaz Snappy Internet Telecom 7266 SW 48 Street Miami, FL 33155 Tel: 305 663 5518 x 232 Help-desk: (305)663-5518 Option 2 or Email: supp...@snappytelecom.net - Original Message - From: Jared Mauch ja...@puck.nether.net To: Faisal Imtiaz fai...@snappytelecom.net Cc: Eugeniu Patrascu eu...@imacandi.net, North American Network Operators' Group nanog@nanog.org Sent: Friday, December 27, 2013 4:04:12 PM Subject: Re: The Making of a Router On Dec 27, 2013, at 3:37 PM, Faisal Imtiaz fai...@snappytelecom.net wrote: e.g. If someone says I need a 10g interface, why is it automatically assumed that the router is going to be running @ Full Line Rate ? Those of us with experience know that when “something bad(tm)” happens, those features and “expensive silicon” start to show some ROI. Is it a full trade-off? Depends on the risks of your business and exposure. You can get some inexpensive hardware to do fairly fancy features these days. That can be very good, but caries that risk. Make sure you evaluate it carefully. - Jared
Re: The Making of a Router
On Fri, 27 Dec 2013, Faisal Imtiaz wrote: Most 'DYI' solutions, make the non-techy bean counters very nervous, and seeing a major 'name brand' label for some odd reason makes them real comfortable, ir-respective of the capabilities or function of either solution. For a lot of organizations, one of the values of big dedicated hardware for the network is in the support contracts that are purchased with the hardware. If something breaks, there is someone to call to work with and resolve the situation. In a DIY setup, you're usually on your own. As others have noted, that's fine, if you understand and accept that risk. jms
Re: The Making of a Router
On Fri, Dec 27, 2013 at 6:48 PM, Justin M. Streiner strei...@cluebyfour.org wrote: If you want to use servers as routers, that's your choice. I think what most people in the thread have been saying is not to use one server (or even a pair of servers) for everything. It's one thing if server XYZ goes down and some web services are offline. It's another thing entirely if that same server goes down and your entire business is offline. You need at least two servers. Much of the functional separation can be achieved by using virtualisation: Two virtual servers per function, but only two physical servers. It is a lot cheaper to just get two beefy servers than to invest in expensive 10G routers. Later on it is easy to move the VMs to more servers if needed. The exception might be any function that moves packets. It is hard to get the necessary performance from dedicated hardware. Trying to do it from a VM will not help. In that case I would dedicate two physical machines to the purpose. However I am actually trying to avoid having the server move packets - I want to use an OpenFlow switch to do the brunt work. Running a BGP daemon on a VM to remote control an OpenFlow switch should not be a problem. Regards, Baldur
Re: Vyatta to VyOS
This is great. I've been using Vyatta for a long while, but the constant bugs and the lack of turnaround on fixing them was really sad. With my lovely sales rep Erica calling me up every now and again trying to sell more licenses, in the way only soulless sales drones can. Extract money from the customer, disregard all complaints and requests for fixing of simple bugs. I was asking for the puppet/chef modules for Vyatta devices for 2 years and finally ended up using their unsupported CLI api to do part of it myself. That kind of automation is a huge chunk of SDN. I'm curious if VyOS will support vPlane, which under Broadcom they seem to have finally shipped. http://www.brocade.com/downloads/documents/at_a_glance/brocade-vyatta-5600vrouter-aag.pdf http://dpdk.org/ On 12/23/13, 3:49 PM, Zach Underwood wrote: Can I just say this is why I love FOSS. I will be testing VyOS on some of my vyatta routes after the holidays. On Mon, Dec 23, 2013 at 4:37 PM, Josh Hoppes josh.hop...@gmail.com wrote: Ubiquiti has been contributing to VyOS, so I'm assuming it is the version they are using as the upstream for their code. On Mon, Dec 23, 2013 at 1:18 PM, Nolan Rollo nro...@kw-corp.com wrote: I wonder how Ubiquiti Networks is going to react to this since their EdgeMax Routers run a fork of the Vyatta code (EdgeOS). http://community.ubnt.com/t5/EdgeMAX/Vyatta-Community-Edition-dead/m-p/591487/highlight/true#M16059 It looks like there is a post in the form where a UBNT Employee said that they were working directly with the VyOS guys. In this case I wonder what other commercial vendor is going to jump on the open source bandwagon. -Original Message- From: Scott Howard [mailto:sc...@doc.net.au] Sent: Monday, December 23, 2013 1:45 PM To: Ray Soucy Cc: NANOG Subject: Re: Vyatta to VyOS Who wants to tell them that it's really 2013? News 22 Dec *2012* Version 1.0.0 (hydrogen) released. Scott On Mon, Dec 23, 2013 at 7:18 AM, Ray Soucy r...@maine.edu wrote: Many here might be interested, In response to Brocade not giving the community edition of Vyatta much attention recently, some of the more active community members have created a fork of the GPL code used in Vyatta. It's called VyOS, and yesterday they released 1.0. http://vyos.net/ I've been playing with the development builds and it seems to be every bit as stable as the Vyatta releases. Will be interesting to see how the project unfolds :-) -- Ray Patrick Soucy Network Engineer University of Maine System T: 207-561-3526 F: 207-561-3531 MaineREN, Maine's Research and Education Network www.maineren.net
The Cidr Report
This report has been generated at Fri Dec 27 21:13:40 2013 AEST.
The report analyses the BGP Routing Table of AS2.0 router
and generates a report on aggregation potential within the table.
Check http://www.cidr-report.org/2.0 for a current version of this report.
Recent Table History
Date PrefixesCIDR Agg
20-12-13486354 275059
21-12-13487165 275707
22-12-13487402 275799
23-12-13487254 275762
24-12-13487144 275938
25-12-13487469 276125
26-12-13487843 275972
27-12-13487889 276061
AS Summary
45972 Number of ASes in routing system
18831 Number of ASes announcing only one prefix
4201 Largest number of prefixes announced by an AS
AS7029 : WINDSTREAM - Windstream Communications Inc
118702848 Largest address span announced by an AS (/32s)
AS4134 : CHINANET-BACKBONE No.31,Jin-rong Street
Aggregation Summary
The algorithm used in this report proposes aggregation only
when there is a precise match using the AS path, so as
to preserve traffic transit policies. Aggregation is also
proposed across non-advertised address space ('holes').
--- 27Dec13 ---
ASnumNetsNow NetsAggr NetGain % Gain Description
Table 487538 276069 21146943.4% All ASes
AS28573 3430 92 333897.3% NET Serviços de Comunicação
S.A.
AS6389 3027 56 297198.1% BELLSOUTH-NET-BLK -
BellSouth.net Inc.
AS17974 2731 189 254293.1% TELKOMNET-AS2-AP PT
Telekomunikasi Indonesia
AS7029 4201 1734 246758.7% WINDSTREAM - Windstream
Communications Inc
AS4766 2946 961 198567.4% KIXS-AS-KR Korea Telecom
AS36998 18196 181399.7% SDN-MOBITEL
AS18881 1789 31 175898.3% Global Village Telecom
AS1785 2146 390 175681.8% AS-PAETEC-NET - PaeTec
Communications, Inc.
AS10620 2691 1080 161159.9% Telmex Colombia S.A.
AS18566 2049 565 148472.4% MEGAPATH5-US - MegaPath
Corporation
AS4323 2942 1513 142948.6% TWTC - tw telecom holdings,
inc.
AS7303 1743 461 128273.6% Telecom Argentina S.A.
AS4755 1810 596 121467.1% TATACOMM-AS TATA
Communications formerly VSNL
is Leading ISP
AS22773 2322 1159 116350.1% ASN-CXA-ALL-CCI-22773-RDC -
Cox Communications Inc.
AS7552 1254 178 107685.8% VIETEL-AS-AP Viettel
Corporation
AS22561 1252 225 102782.0% AS22561 - CenturyTel Internet
Holdings, Inc.
AS9829 1559 660 89957.7% BSNL-NIB National Internet
Backbone
AS7545 2122 1307 81538.4% TPG-INTERNET-AP TPG Telecom
Limited
AS18101 990 186 80481.2% RELIANCE-COMMUNICATIONS-IN
Reliance Communications
Ltd.DAKC MUMBAI
AS35908 906 107 79988.2% VPLSNET - Krypt Technologies
AS4808 1138 384 75466.3% CHINA169-BJ CNCGROUP IP
network China169 Beijing
Province Network
AS701 1507 778 72948.4% UUNET - MCI Communications
Services, Inc. d/b/a Verizon
Business
AS24560 1098 374 72465.9% AIRTELBROADBAND-AS-AP Bharti
Airtel Ltd., Telemedia
Services
AS8151 1381 659 72252.3% Uninet S.A. de C.V.
AS6983 1295 580 71555.2% ITCDELTA - ITC^Deltacom
AS13977 855 143 71283.3% CTELCO - FAIRPOINT
COMMUNICATIONS, INC.
AS4788 884 206 67876.7% TMNET-AS-AP TM Net, Internet
Service Provider
AS4780 1004 326 67867.5% SEEDNET Digital United Inc.
AS855733 57 67692.2% CANET-ASN-4 - Bell Aliant
BGP Update Report
BGP Update Report Interval: 19-Dec-13 -to- 26-Dec-13 (7 days) Observation Point: BGP Peering with AS131072 TOP 20 Unstable Origin AS Rank ASNUpds % Upds/PfxAS-Name 1 - AS982958402 3.2% 62.1 -- BSNL-NIB National Internet Backbone 2 - AS840251576 2.8% 25.5 -- CORBINA-AS OJSC Vimpelcom 3 - AS35819 36302 2.0% 69.7 -- MOBILY-AS Etihad Etisalat Company (Mobily) 4 - AS755229624 1.6% 23.5 -- VIETEL-AS-AP Viettel Corporation 5 - AS15003 24811 1.4% 91.9 -- NOBIS-TECH - Nobis Technology Group, LLC 6 - AS13118 23152 1.3% 609.3 -- ASN-YARTELECOM OJSC Rostelecom 7 - AS453822752 1.2% 43.3 -- ERX-CERNET-BKB China Education and Research Network Center 8 - AS45899 21089 1.1% 64.9 -- VNPT-AS-VN VNPT Corp 9 - AS37004 18982 1.0% 759.3 -- SUBURBAN-AS 10 - AS381618867 1.0% 72.0 -- COLOMBIA TELECOMUNICACIONES S.A. ESP 11 - AS41691 18540 1.0% 545.3 -- SUMTEL-AS-RIPE Summa Telecom LLC 12 - AS37492 17705 1.0% 281.0 -- ORANGE-TN 13 - AS477516838 0.9% 311.8 -- GLOBE-TELECOM-AS Globe Telecoms 14 - AS911616828 0.9% 65.2 -- GOLDENLINES-ASN 012 Smile Communications Ltd. 15 - AS59217 12846 0.7% 12846.0 -- AZONNELIMITED-AS-AP Azonne Limited 16 - AS760212788 0.7% 82.5 -- SPT-AS-VN Saigon Postel Corporation 17 - AS13188 12675 0.7% 14.4 -- BANKINFORM-AS TOV Bank-Inform 18 - AS31148 11882 0.6% 11.8 -- FREENET-AS Freenet Ltd. 19 - AS36998 11577 0.6% 6.4 -- SDN-MOBITEL 20 - AS662910408 0.6%1486.9 -- NOAA-AS - NOAA TOP 20 Unstable Origin AS (Updates per announced prefix) Rank ASNUpds % Upds/PfxAS-Name 1 - AS59217 12846 0.7% 12846.0 -- AZONNELIMITED-AS-AP Azonne Limited 2 - AS544658993 0.5%8993.0 -- QPM-AS-1 - QuickPlay Media Inc. 3 - AS7202 8654 0.5%4327.0 -- FAMU - Florida A M University 4 - AS194063794 0.2%3794.0 -- TWRS-MA - Towerstream I, Inc. 5 - AS227473087 0.2%3087.0 -- TCIS - TulsaConnect, LLC 6 - AS304374469 0.2%2234.5 -- GE-MS003 - General Electric Company 7 - AS603454421 0.2%2210.5 -- NBITI-AS Nahjol Balagheh International Research Institution 8 - AS624312146 0.1%2146.0 -- NCSC-IE-AS National Cyber Security Centre 9 - AS4847 9493 0.5%1898.6 -- CNIX-AP China Networks Inter-Exchange 10 - AS284771821 0.1%1821.0 -- UNIVERSIDAD AUTONOMA DEL ESTADO DE MORELOS 11 - AS4771 1810 0.1%1810.0 -- NZTELECOM Telecom New Zealand Ltd. 12 - AS322446723 0.4%1680.8 -- LIQUID-WEB-INC - Liquid Web, Inc. 13 - AS314591578 0.1%1578.0 -- BBC-RD-AS BBC 14 - AS411891561 0.1%1561.0 -- ICONCEPT-AS Faroese Telecom 15 - AS2818 4668 0.2%1556.0 -- BBC BBC Internet Services, UK 16 - AS662910408 0.6%1486.9 -- NOAA-AS - NOAA 17 - AS1880 6969 0.4%1161.5 -- STUPI Svensk Teleutveckling Produktinnovation, STUPI AB 18 - AS7741 1113 0.1%1113.0 -- CIBC-WORLD-MARKETS - CIBC World Markets 19 - AS39250 996 0.1% 996.0 -- ASN-HYPERGRID HyperGrid s.r.l. 20 - AS62445 898 0.1% 898.0 -- ADA-VOICE-AS ADA VOICE SRL TOP 20 Unstable Prefixes Rank Prefix Upds % Origin AS -- AS Name 1 - 109.161.64.0/20 22988 1.2% AS13118 -- ASN-YARTELECOM OJSC Rostelecom 2 - 103.243.164.0/22 12846 0.7% AS59217 -- AZONNELIMITED-AS-AP Azonne Limited 3 - 85.249.160.0/20 10886 0.6% AS41691 -- SUMTEL-AS-RIPE Summa Telecom LLC 4 - 192.58.232.0/24 10360 0.5% AS6629 -- NOAA-AS - NOAA 5 - 115.170.128.0/17 9474 0.5% AS4847 -- CNIX-AP China Networks Inter-Exchange 6 - 206.152.15.0/248993 0.5% AS54465 -- QPM-AS-1 - QuickPlay Media Inc. 7 - 193.95.93.0/24 8565 0.4% AS37492 -- ORANGE-TN 8 - 193.95.92.0/24 8546 0.4% AS37492 -- ORANGE-TN 11 - 89.221.206.0/247390 0.4% AS41691 -- SUMTEL-AS-RIPE Summa Telecom LLC 12 - 67.210.190.0/236203 0.3% AS11976 -- FIDN - Fidelity Communication International Inc. 13 - 199.187.118.0/24 4907 0.2% AS11054 -- LIVEPERSON LivePerson, Inc 14 - 165.156.25.0/244467 0.2% AS30437 -- GE-MS003 - General Electric Company 16 - 168.223.206.0/23 4377 0.2% AS7202 -- FAMU - Florida A M University 17 - 168.223.200.0/22 4277 0.2% AS7202 -- FAMU - Florida A M University 18 - 192.108.213.0/24 3979 0.2% AS1880 -- STUPI Svensk Teleutveckling Produktinnovation, STUPI AB 19 - 203.8.161.0/24 3942 0.2% AS4802 -- ASN-IINET iiNet Limited 20 - 67.210.188.0/233932 0.2% AS11976 -- FIDN - Fidelity Communication International Inc. Details at
Re: Mikrotik Cloud Core Router and BGP real life experiences?
Regards Alexander Alexander Neilson Neilson Productions Ltd alexan...@neilson.net.nz 021 329 681 On 28/12/2013, at 5:06 am, Eduardo Schoedler lis...@esds.com.br wrote: PPPoE Server is single thread too. PPP package is getting a multicore upgrade in 6.8 or 6.9 release. May introduce bugs but they are working to Multi core all the processes properly. 2013/12/27 Nick Olsen n...@flhsi.com Exactly what Faisal Said. The BGP process appears to be single threaded at the moment. So taking on full BGP tables can be a bit slow compared to a decent X86 box. But in terms of raw forwarding power they are pretty monstrous. We replaced a few Maxxwave 6 port Atom's with the CCR. ~400Mb/s and ~40K pps aggregate across all ports. CPU load went from ~25% to ~0-2%. These are in a configuration where they have little or no firewall/nat/queue rules. And in most cases are running MPLS. We've not had any issues with stability so far either (Knock on wood). Nick Olsen Network Operations (855) FLSPEED x106 From: Faisal Imtiaz fai...@snappytelecom.net Sent: Friday, December 27, 2013 10:33 AM To: Geraint Jones gera...@koding.com Cc: nanog@nanog.org, Martin Hotze m.ho...@hotze.com Subject: Re: Mikrotik Cloud Core Router and BGP real life experiences? FYI... Mikrotik Cloud Core routers are nice, however one has to keep something in mind when deploying them... Only One Core (of the CPU) is dedicated to each port / process. So this is good so as to contain what happens on a single port from taxing the whole CPU.. But not so good when you need more cpu power than a single core for that port. Also, BGP process will only use one core. While these units make for great 'customer facing' edge routers, with plenty of power and the ability to keep issues contained... The X-86 based (Core2Duo/i5/i7) Mikrotik are more suitable (Processing power wise) for running multiple full BGP tables peering. Regards Good Luck. Faisal Imtiaz Snappy Internet Telecom - Original Message - From: Geraint Jones gera...@koding.com To: Martin Hotze m.ho...@hotze.com Cc: nanog@nanog.org Sent: Friday, December 27, 2013 4:02:45 AM Subject: Re: Mikrotik Cloud Core Router and BGP real life experiences? I am going to be deploying 4 as edge routers in the next few weeks, each will have 1 or 2 full tables plus partial IX tables. So I should have some empirical info soon. They will be doing eBGP to upstreams and iBGP/OSPF internally. I went with the 16gb RAM models. However these boxes are basically Linux running on top of tilera CPUs, in terms of throughput as long as everything stays on the fastpath they have no issues doing wire speed on all ports, however the moment you add a firewall rule or the like they drop to 1.5gbps. On 27/12/2013, at 9:47 pm, Martin Hotze m.ho...@hotze.com wrote: Hi, looking at the specs of Mikrotik Cloud Core Routers it seems to be to good to be true [1] having so much bang for the bucks. So virtually all smaller ISPs would drop their CISCO gear for Mikrotik Routerboards. We are using a handful of Mikrotik boxes, but on a much lower network level (splitting networks; low end router behind ADSL modem, ...). We're happy with them. So I am asking for real life experience and not lab values with Mikrotik Cloud Core Routers and BGP. How good can they handle full tables and a bunch of peering sessions? How good does the box react when adding filters (during attacks)? Reloading the table? etc. etc. I am looking for _real_ _life_ values compared to a CISCO NPE-G2. Please tell me/us from your first hand experience. Thanks! greetings, Martin [1] If something sounds too good to be true, it probably is. -- Eduardo Schoedler smime.p7s Description: S/MIME cryptographic signature
Re: The Making of a Router
On 12/27/2013 4:23 PM, Matt Palmer wrote: There *is* a world outside of Silly Valley, you know... a world where money doesn't flow like a mighty cascade from the benevolent wallets of vulture capitalists, into the waiting arms of every crackpot with an elevator pitch. - Matt Yes, and in that world, one should probably not start up a FTTH ISP when one has not even budgeted for a router, among a thousand other things. And if you must, you should probably figure out your cost breakdown beforehand, not after. Baldur, you mention $200k total to move 10gb with Juniper (which seems insanely off to me). Look into Brocades CER line, you can move 4x 10gbe per chassis for under 12k. -- Jon Sands
Re: The Making of a Router
It seems to be a pretty hot button issue, but I feel that modern hardware is more than capable of pushing packets. The old wisdom of only hardware can do it efficiently is starting to prove untrue. 10G might still be a challenge (I haven't tested), but 1G is not even close to being an issue. Depending on the target for your deployment, it might make sense to whitebox a router or firewall instead of spending 20K on it. Especially if you're working with any kind of scale. TL;DR I think the backlash against anything but big iron routing is becoming an old way of thinking. On Fri, Dec 27, 2013 at 6:56 PM, Jon Sands fohdee...@gmail.com wrote: On 12/27/2013 4:23 PM, Matt Palmer wrote: There *is* a world outside of Silly Valley, you know... a world where money doesn't flow like a mighty cascade from the benevolent wallets of vulture capitalists, into the waiting arms of every crackpot with an elevator pitch. - Matt Yes, and in that world, one should probably not start up a FTTH ISP when one has not even budgeted for a router, among a thousand other things. And if you must, you should probably figure out your cost breakdown beforehand, not after. Baldur, you mention $200k total to move 10gb with Juniper (which seems insanely off to me). Look into Brocades CER line, you can move 4x 10gbe per chassis for under 12k. -- Jon Sands -- Ray Patrick Soucy Network Engineer University of Maine System T: 207-561-3526 F: 207-561-3531 MaineREN, Maine's Research and Education Network www.maineren.net
Re: The Making of a Router
On 28 Dec 2013, at 00:13 , Ray Soucy r...@maine.edu wrote: It seems to be a pretty hot button issue, but I feel that modern hardware is more than capable of pushing packets. The old wisdom of only hardware can do it efficiently is starting to prove untrue. 10G might still be a challenge (I haven't tested), Read the publications from http://routebricks.org/ (and check the year) I thought there was a presentation about this (or similar) a few years ago but maybe I just remember the “software router” thread from 5 years ago. /bz — Bjoern A. Zeeb ? ??? ??? ??: '??? ??? ?? ??? ?? ?? ??? ??? ??? ? ? ?? ?? ? ', ? ?, ??? ? ?? ?, ?.???
Re: The Making of a Router
On Sat, Dec 28, 2013 at 12:56 AM, Jon Sands fohdee...@gmail.com wrote: Yes, and in that world, one should probably not start up a FTTH ISP when one has not even budgeted for a router, among a thousand other things. And if you must, you should probably figure out your cost breakdown beforehand, not after. Baldur, you mention $200k total to move 10gb with Juniper (which seems insanely off to me). Look into Brocades CER line, you can move 4x 10gbe per chassis for under 12k. I was saying $100k for two Juniper routers total. Perhaps we could get back on track, instead of trying to second guess what we did or did not budget for. You have absolute no information about our business plans. The Brocade BR-CER-2024F-4X-RT-AC - Brocade NetIron CER 2024F-4X goes for about $21k and we need two of them. That is enough to buy a full year of unlimited 10G internet. And even then, we would be short on 10G ports. It is not that we could not bring that money if that was the only way to do it. It is just that I have so many other things that I could spend that money on, that would further our business plans so much more. I can not even say if the Juniper or the Brocade will actually solve my problem. I need it to route to ten of thousands of VLANS (Q-in-Q), both with IPv4 and IPv6. It needs to act as IPv6 router on every VLAN, and very few devices seems to like having that many IP-addresses assigned. It also needs to do VRRP and proxy arp for every VLAN. The advantage of a software solution is that I can test it all before buying. Also to some limited degree, I am able to fix shortcomings myself. Regards, Baldur
Re: The Making of a Router
On 12/27/2013 8:18 PM, Baldur Norddahl wrote: Brocade NetIron CER 2024F-4X goes for about $21k As one last aside, if you're paying 21k, you're paying a little more than twice too much. Call Brocade and get yourself a real quote. I think peoples main point here is that any handful of thousand dollars you think you will be saving by going PC based, is going to dissapear the very first downtime you experience with zero support. Also, all the features you mention needing are standard features that any modern router should have no problem with. It's been a while but if I remember right you can terminate and route up to 4096 qinq based ve's on a single CER. If you're network topology has you terminating ten thousand qinq networks on a single box, I would take a step back and examine my network topology (if it were me, anyway) -- Jon Sands
Re: The Making of a Router
On a side note, Q-in-Q support has been added to the recent 3.10 Linux kernel, configured using the ip command. It will be popping up in distributions soon [tm]. Another interesting addition is IPv6 NAT (transparent redirect, prefix translation, etc). On Fri, Dec 27, 2013 at 8:18 PM, Baldur Norddahl baldur.nordd...@gmail.comwrote: On Sat, Dec 28, 2013 at 12:56 AM, Jon Sands fohdee...@gmail.com wrote: Yes, and in that world, one should probably not start up a FTTH ISP when one has not even budgeted for a router, among a thousand other things. And if you must, you should probably figure out your cost breakdown beforehand, not after. Baldur, you mention $200k total to move 10gb with Juniper (which seems insanely off to me). Look into Brocades CER line, you can move 4x 10gbe per chassis for under 12k. I was saying $100k for two Juniper routers total. Perhaps we could get back on track, instead of trying to second guess what we did or did not budget for. You have absolute no information about our business plans. The Brocade BR-CER-2024F-4X-RT-AC - Brocade NetIron CER 2024F-4X goes for about $21k and we need two of them. That is enough to buy a full year of unlimited 10G internet. And even then, we would be short on 10G ports. It is not that we could not bring that money if that was the only way to do it. It is just that I have so many other things that I could spend that money on, that would further our business plans so much more. I can not even say if the Juniper or the Brocade will actually solve my problem. I need it to route to ten of thousands of VLANS (Q-in-Q), both with IPv4 and IPv6. It needs to act as IPv6 router on every VLAN, and very few devices seems to like having that many IP-addresses assigned. It also needs to do VRRP and proxy arp for every VLAN. The advantage of a software solution is that I can test it all before buying. Also to some limited degree, I am able to fix shortcomings myself. Regards, Baldur -- Ray Patrick Soucy Network Engineer University of Maine System T: 207-561-3526 F: 207-561-3531 MaineREN, Maine's Research and Education Network www.maineren.net
Re: The Making of a Router
On Fri, 27 Dec 2013, Baldur Norddahl wrote: Another told Nick Cameo that if he can afford a 10G link, he can afford Juniper. You could not be more wrong. The 10G uplink goes for $0 in initial fee and less than $4k / month with unlimited traffic. The Juniper gear is $100k up front for two routers able to handle the 10G links. What I get from you guys is that in your opinion it is not possible to set up a small ISP without spending a ton on Juniper or Cisco. I am not buying that. Small ISPs don't need 10gb transit connections. Even if you did want cisco gear capable of handling 10gb transit, it can be done on the secondary market for a fraction of that $100k figure. -- Jon Lewis, MCP :) | I route | therefore you are _ http://www.lewis.org/~jlewis/pgp for PGP public key_
Re: The Making of a Router
On Sat, Dec 28, 2013 at 3:14 AM, Jon Lewis jle...@lewis.org wrote: On Fri, 27 Dec 2013, Baldur Norddahl wrote: Another told Nick Cameo that if he can afford a 10G link, he can afford Juniper. You could not be more wrong. The 10G uplink goes for $0 in initial fee and less than $4k / month with unlimited traffic. The Juniper gear is $100k up front for two routers able to handle the 10G links. What I get from you guys is that in your opinion it is not possible to set up a small ISP without spending a ton on Juniper or Cisco. I am not buying that. Small ISPs don't need 10gb transit connections. Even if you did want cisco gear capable of handling 10gb transit, it can be done on the secondary market for a fraction of that $100k figure. We are a small ISP that sells 1000/1000 service to our customers. It seems wise to have larger pipe upstream than what you are selling... Regards, Baldur
Re: The Making of a Router
Interested on where you are buying transit at $1750/mo for full 10G ports ($0.175/meg)? On Fri, Dec 27, 2013 at 8:18 PM, Baldur Norddahl baldur.nordd...@gmail.comwrote: On Sat, Dec 28, 2013 at 12:56 AM, Jon Sands fohdee...@gmail.com wrote: Yes, and in that world, one should probably not start up a FTTH ISP when one has not even budgeted for a router, among a thousand other things. And if you must, you should probably figure out your cost breakdown beforehand, not after. Baldur, you mention $200k total to move 10gb with Juniper (which seems insanely off to me). Look into Brocades CER line, you can move 4x 10gbe per chassis for under 12k. I was saying $100k for two Juniper routers total. Perhaps we could get back on track, instead of trying to second guess what we did or did not budget for. You have absolute no information about our business plans. The Brocade BR-CER-2024F-4X-RT-AC - Brocade NetIron CER 2024F-4X goes for about $21k and we need two of them. *That is enough to buy a full year of unlimited 10G internet. And even then, we would be short on 10G ports.* It is not that we could not bring that money if that was the only way to do it. It is just that I have so many other things that I could spend that money on, that would further our business plans so much more. I can not even say if the Juniper or the Brocade will actually solve my problem. I need it to route to ten of thousands of VLANS (Q-in-Q), both with IPv4 and IPv6. It needs to act as IPv6 router on every VLAN, and very few devices seems to like having that many IP-addresses assigned. It also needs to do VRRP and proxy arp for every VLAN. The advantage of a software solution is that I can test it all before buying. Also to some limited degree, I am able to fix shortcomings myself. Regards, Baldur
Re: The Making of a Router
On Fri, 27 Dec 2013 07:23:36 -0500 (EST), Justin M. Streiner strei...@cluebyfour.org said: You end up combining some of the downsides of a hardware-based router with some of the downsides of a server (new attack vectors, another device that needs to be backed up, patched, and monitored... Might be a good idea to back up, patch and monitor your routers too... Just sayin'
Re: The Making of a Router
On Sat, Dec 28, 2013 at 3:50 AM, Brian Loveland br...@aereo.com wrote: Interested on where you are buying transit at $1750/mo for full 10G ports ($0.175/meg)? I did not that claim that. I said two times $21k divided by 12 = $3500 per month. Try he.net. Regards, Baldur
Re: The Making of a Router
clearly you have a deep understanding of what you are doing, the market, what costs and capabilities are, and where to get what you need. now please remind me of what it was you were asking. randy
Re: The Making of a Router
On Sat, Dec 28, 2013 at 4:10 AM, Randy Bush ra...@psg.com wrote: clearly you have a deep understanding of what you are doing, the market, what costs and capabilities are, and where to get what you need. now please remind me of what it was you were asking. randy I asked if anyone here has experience using OpenFlow in an ISP setting. Clearly the answer is no. Regards, Baldur
Re: The Making of a Router
On Fri, 27 Dec 2013 20:37:52 +, Faisal Imtiaz said: e.g. If someone says I need a 10g interface, why is it automatically assumed that the router is going to be running @ Full Line Rate ? It may not be full line rate - but it's a pretty sure bet that you plan to run it at a fairly high percentage duty cycle, either right now or in the fairly near future. If you weren't, you'd have bought a 1G interface instead. pgpUA3IO_WzY7.pgp Description: PGP signature
Re: The Making of a Router
On Sat, 28 Dec 2013 02:18:55 +0100, Baldur Norddahl said: I was saying $100k for two Juniper routers total. Right. And the point that others are trying to make clear is that if that $100K is half your capitalization, you have $200K - and that's nowhere near enought to cover all the stuff you're going to hit starting an ISP. (Hint - what's your projected burn rate for the first two months of business?) pgpnekkPKiKba.pgp Description: PGP signature
Re: The Making of a Router
Right. And the point that others are trying to make clear is that if that $100K is half your capitalization, you have $200K - and that's nowhere near enought to cover all the stuff you're going to hit starting an ISP. (Hint - what's your projected burn rate for the first two months of business?) not to worry. growth is not going to be an issue doing openflow due to today's tcam limits.
Re: The Making of a Router
On Fri, 27 Dec 2013 23:06:26 -0500, Randy Bush said: not to worry. growth is not going to be an issue doing openflow due to today's tcam limits. I said burn rate, not growth rate, Randy.. .:) pgppkYVsYy42z.pgp Description: PGP signature
Re: The Making of a Router
I propose cage fighting at the next NANOG summit. Sent from my Mobile Device. Original message From: Randy Bush ra...@psg.com Date: 12/27/2013 7:07 PM (GMT-09:00) To: valdis.kletni...@vt.edu Cc: North American Network Operators' Group nanog@nanog.org Subject: Re: The Making of a Router Right. And the point that others are trying to make clear is that if that $100K is half your capitalization, you have $200K - and that's nowhere near enought to cover all the stuff you're going to hit starting an ISP. (Hint - what's your projected burn rate for the first two months of business?) not to worry. growth is not going to be an issue doing openflow due to today's tcam limits.
Re: The Making of a Router
This has gotten a bit ridiculous. I was hoping someone could give technical insight into why this is good or not and not just buy a box branded as a router because I said so or your business will fail. I'm all for hearing about the business theory of running an ISP (not my background or day job) but didn't think that's what the OP was asking about (and it didn't seem they were taking business suggestions very well anyway). This thread started cool and about 10 posts in, started sucking. Warren Bailey wbai...@satelliteintelligencegroup.com wrote: I propose cage fighting at the next NANOG summit. Sent from my Mobile Device. Original message From: Randy Bush ra...@psg.com Date: 12/27/2013 7:07 PM (GMT-09:00) To: valdis.kletni...@vt.edu Cc: North American Network Operators' Group nanog@nanog.org Subject: Re: The Making of a Router Right. And the point that others are trying to make clear is that if that $100K is half your capitalization, you have $200K - and that's nowhere near enought to cover all the stuff you're going to hit starting an ISP. (Hint - what's your projected burn rate for the first two months of business?) not to worry. growth is not going to be an issue doing openflow due to today's tcam limits.
Re: The Making of a Router
On Sat, 28 Dec 2013, William Waites wrote: On Fri, 27 Dec 2013 07:23:36 -0500 (EST), Justin M. Streiner strei...@cluebyfour.org said: You end up combining some of the downsides of a hardware-based router with some of the downsides of a server (new attack vectors, another device that needs to be backed up, patched, and monitored... Might be a good idea to back up, patch and monitor your routers too... Just sayin' Yes, a given. jms
Re: The Making of a Router
Warren Bailey viahttp://support.google.com/mail/bin/answer.py?hl=enanswer=1311182ctx=mail nanog.org : I propose cage fighting at the next NANOG summit. Reminds me of some of the BOFs in 2000ish. Anyway, Ray's TL;DR I think the backlash against anything but big iron routing is becoming an old way of thinking. should send a message to CJ that for other than Tier 1 providers, a lot of people are looking for something else that pencils out better.. -- Joe Hamelin, W7COM, Tulalip, WA, 360-474-7474
Re: The Making of a Router
At the very least, could we fight about something worthwhile? I¹m all for a good fight, and I¹m the first one to trigger the nuclear explosion.. But the subject matter of this peepee competition is tiring. I query the nanog gods frequently, sometimes you get useful feedback and sometimes you get a bunch of haters trying to piss on your parade. There is probably some validity on both sides, but a Friday night email fight should be reserved for those email blacklist douchebags.. Arguing over DIY routers being shittier than radical COTS equipment can be done off list. It¹s supposed to be Christmas.. Have a coke and a smile and STFU if you aren¹t going to add some value. I can¹t wait for this thread to fan back up on Monday morning when the normal(er) people read this. The first rule of fight club is.. You don¹t talk about fight club. On 12/27/13, 8:58 PM, Shawn Wilson ag4ve...@gmail.com wrote: This has gotten a bit ridiculous. I was hoping someone could give technical insight into why this is good or not and not just buy a box branded as a router because I said so or your business will fail. I'm all for hearing about the business theory of running an ISP (not my background or day job) but didn't think that's what the OP was asking about (and it didn't seem they were taking business suggestions very well anyway). This thread started cool and about 10 posts in, started sucking. Warren Bailey wbai...@satelliteintelligencegroup.com wrote: I propose cage fighting at the next NANOG summit. Sent from my Mobile Device. Original message From: Randy Bush ra...@psg.com Date: 12/27/2013 7:07 PM (GMT-09:00) To: valdis.kletni...@vt.edu Cc: North American Network Operators' Group nanog@nanog.org Subject: Re: The Making of a Router Right. And the point that others are trying to make clear is that if that $100K is half your capitalization, you have $200K - and that's nowhere near enought to cover all the stuff you're going to hit starting an ISP. (Hint - what's your projected burn rate for the first two months of business?) not to worry. growth is not going to be an issue doing openflow due to today's tcam limits.
Re: The Making of a Router
There has been a lot of conversation lately regarding 10Gbps+ routing without higher cost devices such as the junipers. I have been looking into a few options myself, below are my opinions so far. What are your recommendations, real life experiences and ideas? -Mikrotik Cloud Core Router The Mikrotik CCR might have 2 SFP+ ports but with any ACLs, etc fast path is disabled, this already limits the functionality a lot. The BGP calculations only happen on a single core which provides very slow performance for full routing tables. RouterOS is very unstable and had a large number of bugs even with version 6. I have had issues using them even on some small test environments, would not recommend this hardware for nearly any setup. -Linux Based Software Routing Quagga is great for BGP with the correct CPUs and configurations. Vyatta or VyOS provides a stable and simple configuration method for Quagga. The issues with all of the options currently available is forwarding plane performance, you are only looking at 1Gbps+ at line rate. Most providers will have to deal with DDOS attacks at one point or another and would not recommend taking the chance. If you are only looking at 1Gbps or less worth of traffic this is a great option. DDOS attacks information from just the Arbor Networks hardware. http://www.digitalattackmap.com/ Userspace processing of the forwarding plane will help a lot to overcome this issue. There are a few different solutions out there but the most common is Intel DPDK. Some of you would know about the Intel DPDK from the upcoming brocade vRouter 5600 which supports 10Gbps line rate per core. I can see Intel DPDK being used for other solutions such as DDOS filtering as currently you require specialised hardware such as Arbor Networks or NSFOCUS. It would be much cheaper if you could do some filtering from x86 hardware at line rate. http://blog.lukego.com/blog/2013/01/04/kernel-bypass-networking/ Brocade vRouter 5600 might be an option when it is released depending on price. As you still need to get all the hardware required and make sure you do your research regarding the chipsets, etc. Most Intel SFP+ NIC will handle around 9MPPS but has great support for drivers. Solarflare have some nice NICs that can handle 16MPPS but I can see a lot more reviews for different manufacturers coming out after the vRouter release. Hopefully VyOS or some other open source project can integrate Intel DPDK. -OpenFlow OpenFlow is a great method for really high PPS but the major limiting factor is the flow entries and flow mods. I personally like this architecture as it allows the control plane to run on X86 and the Data Plane to run on specialised hardware. For providers with 1 IP transit provider and a few peering IX most OpenFlow hardware will support enough flow entries. The issue is supporting providers with a reasonable number of full routing tables; I think summarization will help a decent amount to lower the flow entries required. NoviSwitch 1248 supports 1 million flow entries which is a reasonable number for smaller providers. I have only started to get my hand dirty with OpenFlow and would like to know if anyone is using it in production for routing? What OpenFlow controller are you using? E.g. RouteFlow https://sites.google.com/site/routeflow/ -Brocade CER The older model CER devices had a lot of issues/bugs but the newer models such as BR-CER-2024C-4X-RT-AC seem to be a lot more stable. There are reviews on webhostingtalk with people pushing more than 30Gbps on the newer models without issue. Based on other people’s comments such as Jon Sands the units should be around 10K each new which makes the units cost affective for a lot of implementations. If you are lucky enough to find one second hand you would only be looking around $5-6K. The 2024C-4X-RT has 4 SFP+ ports which is alright but would really like to see some larger options. Currently a lot of people just create a port channel with all 4 ports to a SFP+ switch which allow them to connect more ports up but need to be careful about overprovisioning. -Layer 3 SFP+ Switch Great for providers with only one uplink as they just use a default route but most providers require more than one uplink. There are lot of cheap options out there even the junipers are not that costly. Regards, Steven. Date: Fri, 27 Dec 2013 21:34:00 -0500 (EST) From: Justin M. Streiner strei...@cluebyfour.org To: William Waites wwai...@tardis.ed.ac.uk Cc: nanog@nanog.org Subject: Re: The Making of a Router Message-ID: pine.lnx.4.64.1312272133090.22...@whammy.cluebyfour.org Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed On Sat, 28 Dec 2013, William Waites wrote: On Fri, 27 Dec 2013 07:23:36 -0500 (EST), Justin M. Streiner strei...@cluebyfour.org said: You end up combining some of the downsides of a hardware-based router with some of the downsides of a server (new attack vectors, another device that needs to be backed up, patched,
Re: The Making of a Router
Hello Baldur, Your design regarding proxy arp for every VLAN might hit some issues. If you look at the nanog history you will find people having issues with proxy arp for large number of VLANs, what is your requirement for proxy arp? Doing something at the access switch will most likely be better for you such as PVLAN or Brocade IP follow ve statement. If you are planning to put clients on the same subnet what are you planning to put in place to limit client stealing each other’s IPs? Only a few Brocade devices support the ARP ACLs rules which are a really nice feature, IP Source Guard works reasonable if using a DHCP server otherwise you need to specify the MAC address. Some other brand switches support filtering the ARP packets per access port. Regards, Steven. Date: Sat, 28 Dec 2013 02:18:55 +0100 From: Baldur Norddahl baldur.nordd...@gmail.com To: nanog@nanog.org nanog@nanog.org Subject: Re: The Making of a Router Message-ID: capkb-7c2+pebvp+wwyx0s3dlwqmy_hdpbzgqipvq_sfj_3u...@mail.gmail.com Content-Type: text/plain; charset=ISO-8859-1 On Sat, Dec 28, 2013 at 12:56 AM, Jon Sands fohdee...@gmail.com wrote: Yes, and in that world, one should probably not start up a FTTH ISP when one has not even budgeted for a router, among a thousand other things. And if you must, you should probably figure out your cost breakdown beforehand, not after. Baldur, you mention $200k total to move 10gb with Juniper (which seems insanely off to me). Look into Brocades CER line, you can move 4x 10gbe per chassis for under 12k. I was saying $100k for two Juniper routers total. Perhaps we could get back on track, instead of trying to second guess what we did or did not budget for. You have absolute no information about our business plans. The Brocade BR-CER-2024F-4X-RT-AC - Brocade NetIron CER 2024F-4X goes for about $21k and we need two of them. That is enough to buy a full year of unlimited 10G internet. And even then, we would be short on 10G ports. It is not that we could not bring that money if that was the only way to do it. It is just that I have so many other things that I could spend that money on, that would further our business plans so much more. I can not even say if the Juniper or the Brocade will actually solve my problem. I need it to route to ten of thousands of VLANS (Q-in-Q), both with IPv4 and IPv6. It needs to act as IPv6 router on every VLAN, and very few devices seems to like having that many IP-addresses assigned. It also needs to do VRRP and proxy arp for every VLAN. The advantage of a software solution is that I can test it all before buying. Also to some limited degree, I am able to fix shortcomings myself. Regards, Baldur
