Re: The Making of a Router

[Ray Soucy]

> for i in /proc/sys/net/ipv4/conf/*/arp_announce; do echo 2 > $i;done

+1 setting arp_announce in Linux is essential if being used as a router
with more than one subnet.

I would also recommend setting arp_ignore.  For Linux-based routers, I've
found the following settings to be optimal:

echo 1 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/all/arp_ignore

On a side note, this underscores what a lot of people on-list are saying:

If you don't understand the internals of a Linux system, for example,
"rolling your own" will bite you.

It's also pretty rare to find a network engineer who is also a Linux
system-level developer, so finding and maintaining that talent can often be
a challenge.

Many make a leap and go on to assert that because of this software-based
systems can never be viable, which I disagree with.  After all, the latest
OS offerings from Cisco run a Linux kernel.  Nearly all the Ciena DWDM and
ME gear I run is built on Linux.  These companies aren't doing quite as
much with hardware acceleration as they would lead you to believe.

I think Intel DPDK will be a disruptive technology for networking.

At the end of the day, I'm pretty anxious to see the days of over-priced
routers driving up network service costs go away.




On Sun, Dec 29, 2013 at 4:10 AM, Laurent GUERBY  wrote:
>
> On Sun, 2013-12-29 at 03:31 +0100, Baldur Norddahl wrote:
> > (...)
> > The users each have a unique VLAN (Q-in-Q). The question is, what do I
put
> > on those VLANs, if I do not want to put a full IPv4 subnet on each?
> >
> > My own answer to that is to have the users share a larger subnet, for
> > example I could have a full class C sized subnet shared between 253
> > users/VLANs.
> >
> > To allow these users to communicate with each other, and so they can
> > communicate with the default gateway IP, I will need proxy arp. And in a
> > non-OpenFlow solution, also the associated security functions such as
> > DHCP-snooping to prevent hijacking of IP addresses.
> >
> > Which devices can solve this task?
>
> Hi Baldur,
>
> Assuming you manage 1.1.1.0/24 and 2001:db8:0::/48 and
> have a Linux box on both ends you can get rid of
> IPv4 and v6 interco subnets and arp proxy the following way:
>
> 1/ on the gateway
> ip addr add 1.1.1.0/32 dev lo
>
> for all client VLAN "NN" on eth0 :
> ip -6 addr add fe80::1/64 dev eth0.NN
> ip -6 route add 2001:db8:0:NN00::/56 via fe80::1:NN dev eth0.NN
>
> 2/ on user CPE number "NN" CPE WAN interface being eth0 :
> ip addr add 1.1.1.NN/32 dev eth0
> ip route add 1.1.1.0/32 dev eth0
> ip route add default via 1.1.1.0
> ip -6 addr add fe80::1:NN/64 dev eth0
> ip -6 route add default via fe80::1 dev eth0
> # ip -6 addr add  2001:db8:0:NN00::1/56 dev eth0 # optional
>
> Note: NN in hex for IPv6
>
> The trick in IPv4 is that linux by default will answer to ARP requests
> for "1.1.1.0" on all interfaces even if the adress is on the loopback.
> And in IPv6 use static link local on both ends. You can replace
> "1.1.1.0" by any IPv4, but since ".0" are rarely assigned to end users
> it doesn't waste anything and keep traceroute with public IPv4.
>
> The nice thing of this setup is that it "virtualizes" the routing from
> the client point of view: you can split/balance your clients on multiple
> physical gateways and not change a line to the client configuration
> while it's being moved, you just have to configure your IGP between
> gateways to properly distribute internal routes.
>
> We (AS197422 / tetaneutral.net) use this for virtual machines too (with
> "tapNN" interfaces from KVM instead of "eth0.NN"): it allows us to move
> virtual machines around physical machines without user reconfiguration,
> not waste any IPv4 and avoid all issues with shared L2 (rogue RA/ARP
> spoofing/whatever) since there's no shared L2 anymore between user VM.
> It also allows us to not pre split our IPv4 space in a fixed scheme,
> we manage only /32 so no waste at all.
>
> Of course you still have work to do on PPS tuning.
>
> Sincerely,
>
> Laurent GUERBY
> AS197422 http://tetaneutral.net peering http://as197422.net
>
> PS: minimum settings on a Linux router
> echo 1 > /proc/sys/net/ipv4/ip_forward
> for i in /proc/sys/net/ipv6/conf/*; do for j in autoconf accept_ra; do
echo 0 > $i/$j; done;done
> echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
> echo 65536 > /proc/sys/net/ipv6/route/max_size
> for i in /proc/sys/net/ipv4/conf/*/arp_announce; do echo 2 > $i;done
>
> PPS: we also like to give /56 to our users in IPv6, it makes a nice /24
> IPv4 <=> /48 IPv6 correspondance (256 users).
>
>
>



--
Ray Patrick Soucy
Network Engineer
University of Maine System

T: 207-561-3526
F: 207-561-3531

MaineREN, Maine's Research and Education Network
www.maineren.net

Re: The Making of a Router

[Laurent GUERBY]

On Sun, 2013-12-29 at 03:31 +0100, Baldur Norddahl wrote:
> (...)
> The users each have a unique VLAN (Q-in-Q). The question is, what do I put
> on those VLANs, if I do not want to put a full IPv4 subnet on each?
> 
> My own answer to that is to have the users share a larger subnet, for
> example I could have a full class C sized subnet shared between 253
> users/VLANs.
> 
> To allow these users to communicate with each other, and so they can
> communicate with the default gateway IP, I will need proxy arp. And in a
> non-OpenFlow solution, also the associated security functions such as
> DHCP-snooping to prevent hijacking of IP addresses.
> 
> Which devices can solve this task?

Hi Baldur,

Assuming you manage 1.1.1.0/24 and 2001:db8:0::/48 and
have a Linux box on both ends you can get rid of
IPv4 and v6 interco subnets and arp proxy the following way:

1/ on the gateway
ip addr add 1.1.1.0/32 dev lo

for all client VLAN "NN" on eth0 :
ip -6 addr add fe80::1/64 dev eth0.NN
ip -6 route add 2001:db8:0:NN00::/56 via fe80::1:NN dev eth0.NN

2/ on user CPE number "NN" CPE WAN interface being eth0 :
ip addr add 1.1.1.NN/32 dev eth0
ip route add 1.1.1.0/32 dev eth0
ip route add default via 1.1.1.0
ip -6 addr add fe80::1:NN/64 dev eth0
ip -6 route add default via fe80::1 dev eth0
# ip -6 addr add  2001:db8:0:NN00::1/56 dev eth0 # optional

Note: NN in hex for IPv6

The trick in IPv4 is that linux by default will answer to ARP requests
for "1.1.1.0" on all interfaces even if the adress is on the loopback.
And in IPv6 use static link local on both ends. You can replace
"1.1.1.0" by any IPv4, but since ".0" are rarely assigned to end users
it doesn't waste anything and keep traceroute with public IPv4.

The nice thing of this setup is that it "virtualizes" the routing from
the client point of view: you can split/balance your clients on multiple
physical gateways and not change a line to the client configuration
while it's being moved, you just have to configure your IGP between
gateways to properly distribute internal routes.

We (AS197422 / tetaneutral.net) use this for virtual machines too (with
"tapNN" interfaces from KVM instead of "eth0.NN"): it allows us to move
virtual machines around physical machines without user reconfiguration,
not waste any IPv4 and avoid all issues with shared L2 (rogue RA/ARP
spoofing/whatever) since there's no shared L2 anymore between user VM.
It also allows us to not pre split our IPv4 space in a fixed scheme,
we manage only /32 so no waste at all.

Of course you still have work to do on PPS tuning.

Sincerely,

Laurent GUERBY
AS197422 http://tetaneutral.net peering http://as197422.net

PS: minimum settings on a Linux router
echo 1 > /proc/sys/net/ipv4/ip_forward
for i in /proc/sys/net/ipv6/conf/*; do for j in autoconf accept_ra; do echo 0 > 
$i/$j; done;done
echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
echo 65536 > /proc/sys/net/ipv6/route/max_size
for i in /proc/sys/net/ipv4/conf/*/arp_announce; do echo 2 > $i;done

PPS: we also like to give /56 to our users in IPv6, it makes a nice /24
IPv4 <=> /48 IPv6 correspondance (256 users).

Re: The Making of a Router

[Baldur Norddahl]

On Sat, Dec 28, 2013 at 8:09 AM, sten rulz  wrote:

> Hello Baldur,
>
> Your design regarding proxy arp for every VLAN might hit some issues. If
> you look at the nanog history you will find people having issues with proxy
> arp for large number of VLANs, what is your requirement for proxy arp?
> Doing something at the access switch will most likely be better for you
> such as PVLAN or Brocade IP follow ve statement. If you are planning to put
> clients on the same subnet what are you planning to put in place to limit
> client stealing each other’s IPs? Only a few Brocade devices support the
> ARP ACLs rules which are a really nice feature, IP Source Guard works
> reasonable if using a DHCP server otherwise you need to specify the MAC
> address. Some other brand switches support filtering the ARP packets per
> access port.
>

This is a complex question that depends entirely on the capabilities of the
equipment I can get. I was considering an OpenFlow solution, where this is
easy: I would make rules that would only forward traffic with correct
source IP from each VLAN. If the user tries something funny, nothing
happens and his traffic is just dropped.

But I am bit let down on the capabilities of current OpenFlow switches.
Most only support OpenFlow 1.0 which is simply not good enough. That has no
IPv6 support, which naturally is a requirement. I know about the HP
offerings, but they only support 4k rules in hardware, which is a far cry
from being enough. There is NoviFlow who are still working on getting me a
quote. If they can give me a competitive price I might still consider
OpenFlow.

The problem is this: A conventional approach assigns a full IPv4 subnet to
each user. This uses a minimum of 4 addresses of each user. I currently
have to pay somewhere between $10 and $20 for each address and this will
only become more expensive in the future.

The users each have a unique VLAN (Q-in-Q). The question is, what do I put
on those VLANs, if I do not want to put a full IPv4 subnet on each?

My own answer to that is to have the users share a larger subnet, for
example I could have a full class C sized subnet shared between 253
users/VLANs.

To allow these users to communicate with each other, and so they can
communicate with the default gateway IP, I will need proxy arp. And in a
non-OpenFlow solution, also the associated security functions such as
DHCP-snooping to prevent hijacking of IP addresses.

Which devices can solve this task?

To me the work seems quite simple. For outbound packets, check that the
source IP matches the expected IP on the VLAN, then forward the packet
according to the routing table. For inbound packets, lookup the destination
IP and find the correct VLAN, then push the VLAN tag on the packet and
forward it using the normal MAC lookup. For ARP packets, lookup the
destination VLAN from the destination IP, change the VLAN tag and forward
the packet.

There is no reason a device should not be able to handle a large number of
rules such as the above. The NoviSwitch will do it. However it appears that
a lot of devices are quite limited in this regard. I could buy a
router/switch for every few thousand users and split the work between them.
Split the cost on many users, so the extra cost would probably not be
prohibitive. This is the do the work at the edge solution, although I would
be hosting the equipment in the same rack as the core router. But why fill
a rack with equipment, to do simple dummy work, that should be manageable
by a single device?

Regards,

Baldur

Re: The Making of a Router

[Matt Palmer]

On Sat, Dec 28, 2013 at 08:53:53AM -0600, Chris Adams wrote:
> There is a significant value in "just plug it in and it works", and if
> you don't figure your time investment (both up-front and on-going) into
> the cost, you are greatly fooling yourself.

What ISP-grade router are you using that is plug-and-play?  So far, all the
ones I've used have required some quite considerable configuration and
testing to verify they do everything I need, and that's *after* I've spent a
long time reading lies^Wfine marketing documentation trying to decide which
combination of features best suits my needs, and then trying to guess what
other features my business might need during the service lifetime of the
device.

- Matt

-- 
I can only guess that the designer of the things had a major Toilet Duck
habit and had managed to score a couple of industrial-sized bottles of the
stuff the night before.
-- Tanuki

Re: The Making of a Router

[Matt Palmer]

On Fri, Dec 27, 2013 at 08:47:25PM -0500, Jon Sands wrote:
> On 12/27/2013 8:18 PM, Baldur Norddahl wrote:
> >Brocade NetIron CER 2024F-4X goes for
> >about $21k
> 
> As one last aside, if you're paying 21k, you're paying a little more
> than twice too much. Call Brocade and get yourself a real quote.

s/a real quote/a referral to a reseller who takes a week to even return your
call, let alone get a quote to you/

Been there, done that, got the T-shirt, about three months ago, and it was
for quite a chunk more kit than $21k list.  If they're so slack about
getting around to taking my money, how sharp are they going to be when I
need to cost them money (ask for support)?

> I think peoples main point here is that any handful of thousand dollars
> you think you will be saving by going PC based, is going to dissapear the
> very first downtime you experience with zero support.

So that ~$10k device comes with unlimited 24x7 30 minute response time
support, which will get me in touch directly with the people who built and
maintain the system I'm running, and with SLA penalties large enough to
completely cover the additional costs I incur due to the vendor's failure to
perform?  That's quite a deal.

- Matt

-- 
A byte walks into a bar and orders a pint. Bartender asks him "What's
wrong?" The byte says "Parity error." Bartender nods and says "Yeah, I
thought you looked a bit off."

Re: The Making of a Router

[Randy Bush]

> Pretty much what everyone else said. I'm a huge linux person, almost
> everything I use is linux, run full Myth set up etc, but I wouldn't
> use it for a high PPS situation like this. It's just asking for
> suffering later, at the worst possible times.

to paraphrase MO from many years ago (as i do not have a direct quote),
do not use home appliances as production routers.

there are niches where routeflow/vandervecken are usable.  there are
niches where quagga/bird/openbgp are useful.

but not core/backbone of any non-trivial network.  they might be usable
for _slightly_ more than trivial if your business model values your time
at zero.

randy

Re: The Making of a Router

[Blake Dunlap]

Pretty much what everyone else said. I'm a huge linux person, almost
everything I use is linux, run full Myth set up etc, but I wouldn't use it
for a high PPS situation like this. It's just asking for suffering later,
at the worst possible times.

-Blake


On Sat, Dec 28, 2013 at 9:45 AM, Shawn Wilson  wrote:

>
>
> Chris Adams  wrote:
> >Once upon a time, Shawn Wilson  said:
> >> I was hoping someone could give technical insight into why this is
> >good or not and not just "buy a box branded as a router because I said
> >so or your business will fail". I'm all for hearing about the business
> >theory of running an ISP (not my background or day job) but didn't
> >think that's what the OP was asking about (and it didn't seem they were
> >taking business suggestions very well anyway).
> >
> >There's been some technical insight here I would say.  I'm a big Linux,
> >Open Source, and Free Software advocate, and I'll use Linux-based
> >systems for routing/firewalling small stuff, but for high speed/PPS,
> >get
> >a router with a hardware forwarding system (I like Juniper myself).
> >
> >You can build a decently-fast Linux (or *BSD) system, but you'll need
> >to
> >spend a good bit of time carefully choosing motherboards, cards, etc.
> >to
> >maximize packet handling, possibly buying multiple of each to find the
> >best working combination.  Make sure you buy a full set of spares once
> >you find a working combination (because in the PC industry, six months
> >is a lifetime).  Then you have to build your OS install, tweaking the
> >setup, network stack, etc.
> >
> >After that, you have to stay on top of updates and such (so plan for
> >more reboots); while on a hardware-forwarding router you can mostly
> >partition off the control plane, on a Linux/*BSD system, the base OS is
> >the forwarding plane.  Also, if something breaks, falls over under an
> >attack, etc., you're generally going to be on your own to figure it
> >out.
> >Maybe you can Google the answer (and hope it isn't "that'll be fixed in
> >kernel 3..  Not saying that doesn't happen with
> >router vendors (quoting RFCs at router engineers is "fun"), but it is
> >IMHO less often.
> >
> >The question becomes: what is your time worth?  You could spend
> >hundreds
> >of hours going from the start to your satisfactory in-service router,
> >and have a potentially higher upkeep cost.  Can you hire somebody with
> >all the same Linux/*BSD knowlege as yourself, so you are not on-call
> >for
> >your home-built router around the clock?
> >
> >I've used Linux on all my computers for almost 20 years, I develop on
> >Linux, and contribute to a Linux distribution.  However, when I want to
> >record TV to watch later, I plug in a TiVo, not build a MythTV box.
> >There is a significant value in "just plug it in and it works", and if
> >you don't figure your time investment (both up-front and on-going) into
> >the cost, you are greatly fooling yourself.
>
> I agree with all of this to some degree. IDK whether cost of ownership on
> a hardware router or a desktop is more or less - I jus haven't done the
> research. We use them at work and at home I have Cisco and Linksys gear
> (plus Linux doing some things the router could like DHCP) - go figure.
>
> I agree that some network cards and boards work better than others (and am
> partial to the Intel Pro cards - though I'm unsure if they're still the
> best). I would also hesitate to route that much traffic with a PC. Though,
> I have no technical reason for this bias.
>
> If you have hardware in production, you really should have a spare -
> whether we're talking servers, HDDs, batteries, or routers. Ie, that
> comment is not unique to servers. I also don't think warranty has any
> bearing on this - I've seen servers stay down for over a day because (both
> HP and Dell for their respective hardware) screwed up and the company
> didn't budget for a spare board and I've seen a third of a network be taken
> out because multiple switch ports just died. How much would a spare switch
> have cost compared to 50 people not online?
>
> At any rate, I'm interested in this because I've worked in both
> environments and haven't seen a large difference between the two approaches
> (never worked at an ISP or high bandwidth web environment though). I do
> like the PC router approach because it allows more versatility wrt dumping
> packets (no need to dig out that 10mbit dumb hub and throttle the whole
> network), I can run snort or do simple packet inspection with iptables
> (some routers can do this but most can't or require a license). So I'm
> sorta leaning to the PC router as being better - maybe not cheaper but
> better.
>
>

Re: The Making of a Router

[Shawn Wilson]

Chris Adams  wrote:
>Once upon a time, Shawn Wilson  said:
>> I was hoping someone could give technical insight into why this is
>good or not and not just "buy a box branded as a router because I said
>so or your business will fail". I'm all for hearing about the business
>theory of running an ISP (not my background or day job) but didn't
>think that's what the OP was asking about (and it didn't seem they were
>taking business suggestions very well anyway).
>
>There's been some technical insight here I would say.  I'm a big Linux,
>Open Source, and Free Software advocate, and I'll use Linux-based
>systems for routing/firewalling small stuff, but for high speed/PPS,
>get
>a router with a hardware forwarding system (I like Juniper myself).
>
>You can build a decently-fast Linux (or *BSD) system, but you'll need
>to
>spend a good bit of time carefully choosing motherboards, cards, etc.
>to
>maximize packet handling, possibly buying multiple of each to find the
>best working combination.  Make sure you buy a full set of spares once
>you find a working combination (because in the PC industry, six months
>is a lifetime).  Then you have to build your OS install, tweaking the
>setup, network stack, etc.
>
>After that, you have to stay on top of updates and such (so plan for
>more reboots); while on a hardware-forwarding router you can mostly
>partition off the control plane, on a Linux/*BSD system, the base OS is
>the forwarding plane.  Also, if something breaks, falls over under an
>attack, etc., you're generally going to be on your own to figure it
>out.
>Maybe you can Google the answer (and hope it isn't "that'll be fixed in
>kernel 3..  Not saying that doesn't happen with
>router vendors (quoting RFCs at router engineers is "fun"), but it is
>IMHO less often.
>
>The question becomes: what is your time worth?  You could spend
>hundreds
>of hours going from the start to your satisfactory in-service router,
>and have a potentially higher upkeep cost.  Can you hire somebody with
>all the same Linux/*BSD knowlege as yourself, so you are not on-call
>for
>your home-built router around the clock?
>
>I've used Linux on all my computers for almost 20 years, I develop on
>Linux, and contribute to a Linux distribution.  However, when I want to
>record TV to watch later, I plug in a TiVo, not build a MythTV box.
>There is a significant value in "just plug it in and it works", and if
>you don't figure your time investment (both up-front and on-going) into
>the cost, you are greatly fooling yourself.

I agree with all of this to some degree. IDK whether cost of ownership on a 
hardware router or a desktop is more or less - I jus haven't done the research. 
We use them at work and at home I have Cisco and Linksys gear (plus Linux doing 
some things the router could like DHCP) - go figure.

I agree that some network cards and boards work better than others (and am 
partial to the Intel Pro cards - though I'm unsure if they're still the best). 
I would also hesitate to route that much traffic with a PC. Though, I have no 
technical reason for this bias. 

If you have hardware in production, you really should have a spare - whether 
we're talking servers, HDDs, batteries, or routers. Ie, that comment is not 
unique to servers. I also don't think warranty has any bearing on this - I've 
seen servers stay down for over a day because (both HP and Dell for their 
respective hardware) screwed up and the company didn't budget for a spare board 
and I've seen a third of a network be taken out because multiple switch ports 
just died. How much would a spare switch have cost compared to 50 people not 
online?

At any rate, I'm interested in this because I've worked in both environments 
and haven't seen a large difference between the two approaches (never worked at 
an ISP or high bandwidth web environment though). I do like the PC router 
approach because it allows more versatility wrt dumping packets (no need to dig 
out that 10mbit dumb hub and throttle the whole network), I can run snort or do 
simple packet inspection with iptables (some routers can do this but most can't 
or require a license). So I'm sorta leaning to the PC router as being better - 
maybe not cheaper but better. 

Re: The Making of a Router

[Chris Adams]

Once upon a time, Shawn Wilson  said:
> I was hoping someone could give technical insight into why this is good or 
> not and not just "buy a box branded as a router because I said so or your 
> business will fail". I'm all for hearing about the business theory of running 
> an ISP (not my background or day job) but didn't think that's what the OP was 
> asking about (and it didn't seem they were taking business suggestions very 
> well anyway).

There's been some technical insight here I would say.  I'm a big Linux,
Open Source, and Free Software advocate, and I'll use Linux-based
systems for routing/firewalling small stuff, but for high speed/PPS, get
a router with a hardware forwarding system (I like Juniper myself).

You can build a decently-fast Linux (or *BSD) system, but you'll need to
spend a good bit of time carefully choosing motherboards, cards, etc. to
maximize packet handling, possibly buying multiple of each to find the
best working combination.  Make sure you buy a full set of spares once
you find a working combination (because in the PC industry, six months
is a lifetime).  Then you have to build your OS install, tweaking the
setup, network stack, etc.

After that, you have to stay on top of updates and such (so plan for
more reboots); while on a hardware-forwarding router you can mostly
partition off the control plane, on a Linux/*BSD system, the base OS is
the forwarding plane.  Also, if something breaks, falls over under an
attack, etc., you're generally going to be on your own to figure it out.
Maybe you can Google the answer (and hope it isn't "that'll be fixed in
kernel 3..  Not saying that doesn't happen with
router vendors (quoting RFCs at router engineers is "fun"), but it is
IMHO less often.

The question becomes: what is your time worth?  You could spend hundreds
of hours going from the start to your satisfactory in-service router,
and have a potentially higher upkeep cost.  Can you hire somebody with
all the same Linux/*BSD knowlege as yourself, so you are not on-call for
your home-built router around the clock?

I've used Linux on all my computers for almost 20 years, I develop on
Linux, and contribute to a Linux distribution.  However, when I want to
record TV to watch later, I plug in a TiVo, not build a MythTV box.
There is a significant value in "just plug it in and it works", and if
you don't figure your time investment (both up-front and on-going) into
the cost, you are greatly fooling yourself.

-- 
Chris Adams 

Re: The Making of a Router

[Miquel van Smoorenburg]

In article 
 you 
write:
>It seems to be a pretty "hot button" issue, but I feel that modern hardware
>is more than capable of pushing packets.  The old wisdom of "only hardware
>can do it efficiently" is starting to prove untrue.  10G might still be a
>challenge (I haven't tested), but 1G is not even close to being an issue.
> Depending on the target for your deployment, it might make sense to
>whitebox a router or firewall instead of spending 20K on it.  Especially if
>you're working with any kind of scale.

Yes well, but also remember that bandwidth is not everything. Packets
per second is. And if you're going to provide internet connectivity
to endusers, some of them /will/ get hit with DDOS attacks. With
a hardware router you can survive that as long as the DDOS is not
consuming all your bandwidth. A software router being bombarded
with a few gigabits of 64 byte packets .. not so much.

This is also the reason btw that you should look into shaping the
outgoing bandwidth to each enduser, to prevent one of them being
DDOSsed filling up the entire link he/she is on.

Mike.

Re: The Making of a Router

[sten rulz]

Hello Baldur,

Your design regarding proxy arp for every VLAN might hit some issues. If
you look at the nanog history you will find people having issues with proxy
arp for large number of VLANs, what is your requirement for proxy arp?
Doing something at the access switch will most likely be better for you
such as PVLAN or Brocade IP follow ve statement. If you are planning to put
clients on the same subnet what are you planning to put in place to limit
client stealing each other’s IPs? Only a few Brocade devices support the
ARP ACLs rules which are a really nice feature, IP Source Guard works
reasonable if using a DHCP server otherwise you need to specify the MAC
address. Some other brand switches support filtering the ARP packets per
access port.

Regards,
Steven.

Date: Sat, 28 Dec 2013 02:18:55 +0100
From: Baldur Norddahl 
To: "nanog@nanog.org" 
Subject: Re: The Making of a Router
Message-ID:

Content-Type: text/plain; charset=ISO-8859-1

On Sat, Dec 28, 2013 at 12:56 AM, Jon Sands  wrote:

> Yes, and in that world, one should probably not start up a FTTH ISP when
> one has not even budgeted for a router, among a thousand other things. And
> if you must, you should probably figure out your cost breakdown
beforehand,
> not after. Baldur, you mention $200k total to move 10gb with Juniper
(which
> seems insanely off to me). Look into Brocades CER line, you can move 4x
> 10gbe per chassis for under 12k.
>

I was saying $100k for two Juniper routers total.

Perhaps we could get back on track, instead of trying to second guess what
we did or did not budget for. You have absolute no information about our
business plans.

The Brocade BR-CER-2024F-4X-RT-AC - Brocade NetIron CER 2024F-4X goes for
about $21k and we need two of them. That is enough to buy a full year of
unlimited 10G internet. And even then, we would be short on 10G ports.

It is not that we could not bring that money if that was the only way to do
it. It is just that I have so many other things that I could spend that
money on, that would further our business plans so much more.

I can not even say if the Juniper or the Brocade will actually solve my
problem. I need it to route to ten of thousands of VLANS (Q-in-Q), both
with IPv4 and IPv6. It needs to act as IPv6 router on every VLAN, and very
few devices seems to like having that many IP-addresses assigned. It also
needs to do VRRP and proxy arp for every VLAN.

The advantage of a software solution is that I can test it all before
buying. Also to some limited degree, I am able to fix shortcomings myself.

Regards,

Baldur

Re: The Making of a Router

[sten rulz]

There has been a lot of conversation lately regarding 10Gbps+ routing
without higher cost devices such as the junipers. I have been looking into
a few options myself, below are my opinions so far. What are your
recommendations, real life experiences and ideas?

-Mikrotik Cloud Core Router
The Mikrotik CCR might have 2 SFP+ ports but with any ACLs, etc fast path
is disabled, this already limits the functionality a lot. The BGP
calculations only happen on a single core which provides very slow
performance for full routing tables. RouterOS is very unstable and had a
large number of bugs even with version 6. I have had issues using them even
on some small test environments, would not recommend this hardware for
nearly any setup.
-Linux Based Software Routing
Quagga is great for BGP with the correct CPUs and configurations. Vyatta or
VyOS provides a stable and simple configuration method for Quagga. The
issues with all of the options currently available is forwarding plane
performance, you are only looking at 1Gbps+ at line rate. Most providers
will have to deal with DDOS attacks at one point or another and would not
recommend taking the chance. If you are only looking at 1Gbps or less worth
of traffic this is a great option.
DDOS attacks information from just the Arbor Networks hardware.
http://www.digitalattackmap.com/
Userspace processing of the forwarding plane will help a lot to overcome
this issue. There are a few different solutions out there but the most
common is Intel DPDK. Some of you would know about the Intel DPDK from the
upcoming brocade vRouter 5600 which supports 10Gbps line rate per core. I
can see Intel DPDK being used for other solutions such as DDOS filtering as
currently you require specialised hardware such as Arbor Networks or
NSFOCUS. It would be much cheaper if you could do some filtering from x86
hardware at line rate.
http://blog.lukego.com/blog/2013/01/04/kernel-bypass-networking/
Brocade vRouter 5600 might be an option when it is released depending on
price. As you still need to get all the hardware required and make sure you
do your research regarding the chipsets, etc. Most Intel SFP+ NIC will
handle around 9MPPS but has great support for drivers. Solarflare have some
nice NICs that can handle 16MPPS but I can see a lot more reviews for
different manufacturers coming out after the vRouter release. Hopefully
VyOS or some other open source project can integrate Intel DPDK.
-OpenFlow
OpenFlow is a great method for really high PPS but the major limiting
factor is the flow entries and flow mods. I personally like this
architecture as it allows the control plane to run on X86 and the Data
Plane to run on specialised hardware. For providers with 1 IP transit
provider and a few peering IX most OpenFlow hardware will support enough
flow entries. The issue is supporting providers with a reasonable number of
full routing tables; I think summarization will help a decent amount to
lower the flow entries required. NoviSwitch 1248 supports 1 million flow
entries which is a reasonable number for smaller providers. I have only
started to get my hand dirty with OpenFlow and would like to know if anyone
is using it in production for routing? What OpenFlow controller are you
using? E.g. RouteFlow
https://sites.google.com/site/routeflow/
-Brocade CER
The older model CER devices had a lot of issues/bugs but the newer models
such as BR-CER-2024C-4X-RT-AC seem to be a lot more stable. There are
reviews on webhostingtalk with people pushing more than 30Gbps on the newer
models without issue. Based on other people’s comments such as Jon Sands
the units should be around 10K each new which makes the units cost
affective for a lot of implementations. If you are lucky enough to find one
second hand you would only be looking around $5-6K. The 2024C-4X-RT has 4
SFP+ ports which is alright but would really like to see some larger
options. Currently a lot of people just create a port channel with all 4
ports to a SFP+ switch which allow them to connect more ports up but need
to be careful about overprovisioning.
-Layer 3 SFP+ Switch
Great for providers with only one uplink as they just use a default route
but most providers require more than one uplink. There are lot of cheap
options out there even the junipers are not that costly.

Regards,
Steven.

Date: Fri, 27 Dec 2013 21:34:00 -0500 (EST)
From: "Justin M. Streiner" 
To: William Waites 
Cc: nanog@nanog.org
Subject: Re: The Making of a Router
Message-ID: 
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed

On Sat, 28 Dec 2013, William Waites wrote:

> On Fri, 27 Dec 2013 07:23:36 -0500 (EST), "Justin M. Streiner" <
strei...@cluebyfour.org> said:
>
>> You end up combining some of the downsides of a hardware-based
>> router with some of the downsides of a server (new attack
>> vectors, another device that needs to be backed up, patched, and
>> monitored...
>

Re: The Making of a Router

[Warren Bailey]

At the very least, could we fight about something worthwhile? I¹m all for
a good fight, and I¹m the first one to trigger the nuclear explosion.. But
the subject matter of this peepee competition is tiring. I query the nanog
gods frequently, sometimes you get useful feedback and sometimes you get a
bunch of haters trying to piss on your parade. There is probably some
validity on both sides, but a Friday night email fight should be reserved
for those email blacklist douchebags.. Arguing over DIY routers being
shittier than radical COTS equipment can be done off list. It¹s supposed
to be Christmas.. Have a coke and a smile and STFU if you aren¹t going to
add some value.

I can¹t wait for this thread to fan back up on Monday morning when the
normal(er) people read this.

The first rule of fight club is.. You don¹t talk about fight club.

On 12/27/13, 8:58 PM, "Shawn Wilson"  wrote:

>This has gotten a bit ridiculous.
>
>I was hoping someone could give technical insight into why this is good
>or not and not just "buy a box branded as a router because I said so or
>your business will fail". I'm all for hearing about the business theory
>of running an ISP (not my background or day job) but didn't think that's
>what the OP was asking about (and it didn't seem they were taking
>business suggestions very well anyway).
>
>This thread started cool and about 10 posts in, started sucking.
>
>Warren Bailey  wrote:
>>I propose cage fighting at the next NANOG summit.
>>
>>
>>Sent from my Mobile Device.
>>
>>
>> Original message 
>>From: Randy Bush 
>>Date: 12/27/2013 7:07 PM (GMT-09:00)
>>To: valdis.kletni...@vt.edu
>>Cc: North American Network Operators' Group 
>>Subject: Re: The Making of a Router
>>
>>
>>> Right.  And the point that others are trying to make clear is that if
>>> that $100K is half your capitalization, you have $200K - and that's
>>> nowhere near enought to cover all the stuff you're going to hit
>>> starting an ISP.  (Hint - what's your projected burn rate for the
>>> first two months of business?)
>>
>>not to worry.  growth is not going to be an issue doing openflow due to
>>today's tcam limits.
>

Re: The Making of a Router

[Joe Hamelin]

>Warren Bailey 
>via
 nanog.org :
>I propose cage fighting at the next NANOG summit.

Reminds me of some of the BOFs in 2000ish.

Anyway, Ray's "TL;DR I think the backlash against anything but big iron
routing is becoming an old way of thinking." should send a message to C&J
that for other than "Tier 1" providers, a lot of people are looking for
something else that pencils out better..


--
Joe Hamelin, W7COM, Tulalip, WA, 360-474-7474

Re: The Making of a Router

[Justin M. Streiner]

On Sat, 28 Dec 2013, William Waites wrote:


On Fri, 27 Dec 2013 07:23:36 -0500 (EST), "Justin M. Streiner" 
 said:

   > You end up combining some of the downsides of a hardware-based
   > router with some of the downsides of a server (new attack
   > vectors, another device that needs to be backed up, patched, and
   > monitored...

Might be a good idea to back up, patch and monitor your routers
too... Just sayin'


Yes, a given.

jms

Re: The Making of a Router

[Shawn Wilson]

This has gotten a bit ridiculous. 

I was hoping someone could give technical insight into why this is good or not 
and not just "buy a box branded as a router because I said so or your business 
will fail". I'm all for hearing about the business theory of running an ISP 
(not my background or day job) but didn't think that's what the OP was asking 
about (and it didn't seem they were taking business suggestions very well 
anyway).

This thread started cool and about 10 posts in, started sucking. 

Warren Bailey  wrote:
>I propose cage fighting at the next NANOG summit.
>
>
>Sent from my Mobile Device.
>
>
> Original message 
>From: Randy Bush 
>Date: 12/27/2013 7:07 PM (GMT-09:00)
>To: valdis.kletni...@vt.edu
>Cc: North American Network Operators' Group 
>Subject: Re: The Making of a Router
>
>
>> Right.  And the point that others are trying to make clear is that if
>> that $100K is half your capitalization, you have $200K - and that's
>> nowhere near enought to cover all the stuff you're going to hit
>> starting an ISP.  (Hint - what's your projected burn rate for the
>> first two months of business?)
>
>not to worry.  growth is not going to be an issue doing openflow due to
>today's tcam limits.

Re: The Making of a Router

[Warren Bailey]

I propose cage fighting at the next NANOG summit.


Sent from my Mobile Device.


 Original message 
From: Randy Bush 
Date: 12/27/2013 7:07 PM (GMT-09:00)
To: valdis.kletni...@vt.edu
Cc: North American Network Operators' Group 
Subject: Re: The Making of a Router


> Right.  And the point that others are trying to make clear is that if
> that $100K is half your capitalization, you have $200K - and that's
> nowhere near enought to cover all the stuff you're going to hit
> starting an ISP.  (Hint - what's your projected burn rate for the
> first two months of business?)

not to worry.  growth is not going to be an issue doing openflow due to
today's tcam limits.

Re: The Making of a Router

[Valdis . Kletnieks]

On Fri, 27 Dec 2013 23:06:26 -0500, Randy Bush said:

> not to worry.  growth is not going to be an issue doing openflow due to
> today's tcam limits.

I said burn rate, not growth rate, Randy.. .:)


pgppkYVsYy42z.pgp
Description: PGP signature

Re: The Making of a Router

[Randy Bush]

> Right.  And the point that others are trying to make clear is that if
> that $100K is half your capitalization, you have $200K - and that's
> nowhere near enought to cover all the stuff you're going to hit
> starting an ISP.  (Hint - what's your projected burn rate for the
> first two months of business?)

not to worry.  growth is not going to be an issue doing openflow due to
today's tcam limits.

Re: The Making of a Router

[Valdis . Kletnieks]

On Sat, 28 Dec 2013 02:18:55 +0100, Baldur Norddahl said:

> I was saying $100k for two Juniper routers total.

Right.  And the point that others are trying to make clear is that if
that $100K is half your capitalization, you have $200K - and that's nowhere
near enought to cover all the stuff you're going to hit starting an ISP.
(Hint - what's your projected burn rate for the first two months of business?)


pgpnekkPKiKba.pgp
Description: PGP signature

Re: The Making of a Router

[Valdis . Kletnieks]

On Fri, 27 Dec 2013 20:37:52 +, Faisal Imtiaz said:

> e.g. If someone says I need a 10g interface, why is it automatically assumed
> that the router is going to be running @ Full Line Rate ?

It may not be full line rate - but it's a pretty sure bet that you plan
to run it at a fairly high percentage duty cycle, either right now or in the
fairly near future.

If you weren't, you'd have bought a 1G interface instead.


pgpUA3IO_WzY7.pgp
Description: PGP signature

Re: The Making of a Router

[Baldur Norddahl]

On Sat, Dec 28, 2013 at 4:10 AM, Randy Bush  wrote:

> clearly you have a deep understanding of what you are doing, the market,
> what costs and capabilities are, and where to get what you need.  now
> please remind me of what it was you were asking.
>
> randy
>

I asked if anyone here has experience using OpenFlow in an ISP setting.
Clearly the answer is no.

Regards,

Baldur

Re: The Making of a Router

[Randy Bush]

clearly you have a deep understanding of what you are doing, the market,
what costs and capabilities are, and where to get what you need.  now
please remind me of what it was you were asking.

randy

Re: The Making of a Router

[Baldur Norddahl]

On Sat, Dec 28, 2013 at 3:50 AM, Brian Loveland  wrote:

> Interested on where you are buying transit at $1750/mo for full 10G ports
> ($0.175/meg)?
>
>
I did not that claim that. I said two times $21k divided by 12 = $3500 per
month. Try he.net.

Regards,

Baldur

Re: The Making of a Router

[William Waites]

On Fri, 27 Dec 2013 07:23:36 -0500 (EST), "Justin M. Streiner" 
 said:

> You end up combining some of the downsides of a hardware-based
> router with some of the downsides of a server (new attack
> vectors, another device that needs to be backed up, patched, and
> monitored...

Might be a good idea to back up, patch and monitor your routers
too... Just sayin'

Re: The Making of a Router

[Brian Loveland]

Interested on where you are buying transit at $1750/mo for full 10G ports
($0.175/meg)?

On Fri, Dec 27, 2013 at 8:18 PM, Baldur Norddahl
wrote:

> On Sat, Dec 28, 2013 at 12:56 AM, Jon Sands  wrote:
>
> > Yes, and in that world, one should probably not start up a FTTH ISP when
> > one has not even budgeted for a router, among a thousand other things.
> And
> > if you must, you should probably figure out your cost breakdown
> beforehand,
> > not after. Baldur, you mention $200k total to move 10gb with Juniper
> (which
> > seems insanely off to me). Look into Brocades CER line, you can move 4x
> > 10gbe per chassis for under 12k.
> >
>
> I was saying $100k for two Juniper routers total.
>
> Perhaps we could get back on track, instead of trying to second guess what
> we did or did not budget for. You have absolute no information about our
> business plans.
>
> The Brocade BR-CER-2024F-4X-RT-AC - Brocade NetIron CER 2024F-4X goes for
> about $21k and we need two of them.
>
> *That is enough to buy a full year of unlimited 10G internet. And even
> then, we would be short on 10G ports.*
> It is not that we could not bring that money if that was the only way to do
> it. It is just that I have so many other things that I could spend that
> money on, that would further our business plans so much more.
>
> I can not even say if the Juniper or the Brocade will actually solve my
> problem. I need it to route to ten of thousands of VLANS (Q-in-Q), both
> with IPv4 and IPv6. It needs to act as IPv6 router on every VLAN, and very
> few devices seems to like having that many IP-addresses assigned. It also
> needs to do VRRP and proxy arp for every VLAN.
>
> The advantage of a software solution is that I can test it all before
> buying. Also to some limited degree, I am able to fix shortcomings myself.
>
> Regards,
>
> Baldur
>

Re: The Making of a Router

[Baldur Norddahl]

On Sat, Dec 28, 2013 at 3:14 AM, Jon Lewis  wrote:

> On Fri, 27 Dec 2013, Baldur Norddahl wrote:
>
>  Another told Nick Cameo that if he can afford a 10G link, he can afford
>> Juniper. You could not be more wrong. The 10G uplink goes for $0 in
>> initial
>> fee and less than $4k / month with unlimited traffic. The Juniper gear is
>> $100k up front for two routers able to handle the 10G links.
>>
>> What I get from you guys is that in your opinion it is not possible to set
>> up a small ISP without spending a ton on Juniper or Cisco. I am not buying
>> that.
>>
>
> Small ISPs don't need 10gb transit connections.  Even if you did want
> cisco gear capable of handling 10gb transit, it can be done on the
> secondary market for a fraction of that $100k figure.
>

We are a small ISP that sells 1000/1000 service to our customers. It seems
wise to have larger pipe upstream than what you are selling...

Regards,

Baldur

Re: The Making of a Router

[Jon Lewis]

On Fri, 27 Dec 2013, Baldur Norddahl wrote:


Another told Nick Cameo that if he can afford a 10G link, he can afford
Juniper. You could not be more wrong. The 10G uplink goes for $0 in initial
fee and less than $4k / month with unlimited traffic. The Juniper gear is
$100k up front for two routers able to handle the 10G links.

What I get from you guys is that in your opinion it is not possible to set
up a small ISP without spending a ton on Juniper or Cisco. I am not buying
that.


Small ISPs don't need 10gb transit connections.  Even if you did want 
cisco gear capable of handling 10gb transit, it can be done on the 
secondary market for a fraction of that $100k figure.


--
 Jon Lewis, MCP :)   |  I route
 |  therefore you are
_ http://www.lewis.org/~jlewis/pgp for PGP public key_

Re: The Making of a Router

[Ray Soucy]

On a side note, Q-in-Q support has been added to the recent 3.10 Linux
kernel, configured using the "ip" command.  It will be popping up in
distributions "soon [tm]".  Another interesting addition is IPv6 NAT
(transparent redirect, prefix translation, etc).


On Fri, Dec 27, 2013 at 8:18 PM, Baldur Norddahl
wrote:

> On Sat, Dec 28, 2013 at 12:56 AM, Jon Sands  wrote:
>
> > Yes, and in that world, one should probably not start up a FTTH ISP when
> > one has not even budgeted for a router, among a thousand other things.
> And
> > if you must, you should probably figure out your cost breakdown
> beforehand,
> > not after. Baldur, you mention $200k total to move 10gb with Juniper
> (which
> > seems insanely off to me). Look into Brocades CER line, you can move 4x
> > 10gbe per chassis for under 12k.
> >
>
> I was saying $100k for two Juniper routers total.
>
> Perhaps we could get back on track, instead of trying to second guess what
> we did or did not budget for. You have absolute no information about our
> business plans.
>
> The Brocade BR-CER-2024F-4X-RT-AC - Brocade NetIron CER 2024F-4X goes for
> about $21k and we need two of them. That is enough to buy a full year of
> unlimited 10G internet. And even then, we would be short on 10G ports.
>
> It is not that we could not bring that money if that was the only way to do
> it. It is just that I have so many other things that I could spend that
> money on, that would further our business plans so much more.
>
> I can not even say if the Juniper or the Brocade will actually solve my
> problem. I need it to route to ten of thousands of VLANS (Q-in-Q), both
> with IPv4 and IPv6. It needs to act as IPv6 router on every VLAN, and very
> few devices seems to like having that many IP-addresses assigned. It also
> needs to do VRRP and proxy arp for every VLAN.
>
> The advantage of a software solution is that I can test it all before
> buying. Also to some limited degree, I am able to fix shortcomings myself.
>
> Regards,
>
> Baldur
>



-- 
Ray Patrick Soucy
Network Engineer
University of Maine System

T: 207-561-3526
F: 207-561-3531

MaineREN, Maine's Research and Education Network
www.maineren.net

Re: The Making of a Router

[Jon Sands]

On 12/27/2013 8:18 PM, Baldur Norddahl wrote:

Brocade NetIron CER 2024F-4X goes for
about $21k


As one last aside, if you're paying 21k, you're paying a little more 
than twice too much. Call Brocade and get yourself a real quote. I think 
peoples main point here is that any handful of thousand dollars you 
think you will be saving by going PC based, is going to dissapear the 
very first downtime you experience with zero support. Also, all the 
features you mention needing are standard features that any modern 
router should have no problem with. It's been a while but if I remember 
right you can terminate and route up to 4096 qinq based ve's on a single 
CER. If you're network topology has you terminating ten thousand qinq 
networks on a single box, I would take a step back and examine my 
network topology (if it were me, anyway)


--
Jon Sands

Re: The Making of a Router

[Baldur Norddahl]

On Sat, Dec 28, 2013 at 12:56 AM, Jon Sands  wrote:

> Yes, and in that world, one should probably not start up a FTTH ISP when
> one has not even budgeted for a router, among a thousand other things. And
> if you must, you should probably figure out your cost breakdown beforehand,
> not after. Baldur, you mention $200k total to move 10gb with Juniper (which
> seems insanely off to me). Look into Brocades CER line, you can move 4x
> 10gbe per chassis for under 12k.
>

I was saying $100k for two Juniper routers total.

Perhaps we could get back on track, instead of trying to second guess what
we did or did not budget for. You have absolute no information about our
business plans.

The Brocade BR-CER-2024F-4X-RT-AC - Brocade NetIron CER 2024F-4X goes for
about $21k and we need two of them. That is enough to buy a full year of
unlimited 10G internet. And even then, we would be short on 10G ports.

It is not that we could not bring that money if that was the only way to do
it. It is just that I have so many other things that I could spend that
money on, that would further our business plans so much more.

I can not even say if the Juniper or the Brocade will actually solve my
problem. I need it to route to ten of thousands of VLANS (Q-in-Q), both
with IPv4 and IPv6. It needs to act as IPv6 router on every VLAN, and very
few devices seems to like having that many IP-addresses assigned. It also
needs to do VRRP and proxy arp for every VLAN.

The advantage of a software solution is that I can test it all before
buying. Also to some limited degree, I am able to fix shortcomings myself.

Regards,

Baldur

Re: The Making of a Router

[Bjoern A. Zeeb]

On 28 Dec 2013, at 00:13 , Ray Soucy  wrote:

> It seems to be a pretty "hot button" issue, but I feel that modern hardware
> is more than capable of pushing packets.  The old wisdom of "only hardware
> can do it efficiently" is starting to prove untrue.  10G might still be a
> challenge (I haven't tested),


Read the publications from http://routebricks.org/  (and check the year)

I thought there was a presentation about this (or similar) a few years ago
but maybe I just remember the “software router” thread from 5 years ago.

/bz

— 
Bjoern A. Zeeb ? ??? ??? ??:
'??? ???  ??  ??? ?? ?? ??? ??? ??? ? ? 
?? ?? ? ',  ? ?, "??? ? ?? ?", ?.???

Re: The Making of a Router

[Ray Soucy]

It seems to be a pretty "hot button" issue, but I feel that modern hardware
is more than capable of pushing packets.  The old wisdom of "only hardware
can do it efficiently" is starting to prove untrue.  10G might still be a
challenge (I haven't tested), but 1G is not even close to being an issue.
 Depending on the target for your deployment, it might make sense to
whitebox a router or firewall instead of spending 20K on it.  Especially if
you're working with any kind of scale.

TL;DR I think the backlash against anything but big iron routing is
becoming an old way of thinking.



On Fri, Dec 27, 2013 at 6:56 PM, Jon Sands  wrote:

> On 12/27/2013 4:23 PM, Matt Palmer wrote:
>
>> There *is* a world outside of Silly Valley, you know... a world where
>> money doesn't flow like a mighty cascade from the benevolent wallets of
>> vulture capitalists, into the waiting arms of every crackpot with an
>> elevator pitch. - Matt
>>
>
> Yes, and in that world, one should probably not start up a FTTH ISP when
> one has not even budgeted for a router, among a thousand other things. And
> if you must, you should probably figure out your cost breakdown beforehand,
> not after. Baldur, you mention $200k total to move 10gb with Juniper (which
> seems insanely off to me). Look into Brocades CER line, you can move 4x
> 10gbe per chassis for under 12k.
>
> --
> Jon Sands
>



-- 
Ray Patrick Soucy
Network Engineer
University of Maine System

T: 207-561-3526
F: 207-561-3531

MaineREN, Maine's Research and Education Network
www.maineren.net

Re: The Making of a Router

[Jon Sands]

On 12/27/2013 4:23 PM, Matt Palmer wrote:
There *is* a world outside of Silly Valley, you know... a world where 
money doesn't flow like a mighty cascade from the benevolent wallets 
of vulture capitalists, into the waiting arms of every crackpot with 
an elevator pitch. - Matt 


Yes, and in that world, one should probably not start up a FTTH ISP when 
one has not even budgeted for a router, among a thousand other things. 
And if you must, you should probably figure out your cost breakdown 
beforehand, not after. Baldur, you mention $200k total to move 10gb with 
Juniper (which seems insanely off to me). Look into Brocades CER line, 
you can move 4x 10gbe per chassis for under 12k.


--
Jon Sands

Re: The Making of a Router

[Baldur Norddahl]

On Fri, Dec 27, 2013 at 6:48 PM, Justin M. Streiner  wrote:

> If you want to use servers as routers, that's your choice.  I think what
> most people in the thread have been saying is not to use one server (or
> even a pair of servers) for everything.  It's one thing if server XYZ goes
> down and some web services are offline.  It's another thing entirely if
> that same server goes down and your entire business is offline.
>

You need at least two servers. Much of the functional separation can be
achieved by using virtualisation: Two virtual servers per function, but
only two physical servers. It is a lot cheaper to just get two beefy
servers than to invest in expensive 10G routers. Later on it is easy to
move the VMs to more servers if needed.

The exception might be any function that moves packets. It is hard to get
the necessary performance from dedicated hardware. Trying to do it from a
VM will not help. In that case I would dedicate two physical machines to
the purpose. However I am actually trying to avoid having the server move
packets - I want to use an OpenFlow switch to do the brunt work. Running a
BGP daemon on a VM to remote control an OpenFlow switch should not be a
problem.

Regards,

Baldur

Re: The Making of a Router

[Justin M. Streiner]

On Fri, 27 Dec 2013, Faisal Imtiaz wrote:

Most 'DYI' solutions, make the non-techy bean counters very nervous, 
and seeing a major 'name brand' label for some odd reason makes them 
real comfortable, ir-respective of the capabilities or function of 
either solution.


For a lot of organizations, one of the values of big dedicated hardware 
for the network is in the support contracts that are purchased with the 
hardware.  If something breaks, there is someone to call to work with and 
resolve the situation.  In a DIY setup, you're usually on your own.  As 
others have noted, that's fine, if you understand and accept that risk.


jms

Re: The Making of a Router

[Faisal Imtiaz]

Fair point.. but in real life, isn't that true for everything...

I say the same  be familiar(honest awareness) with the limits (limitations) 
and capabilities of your specific solution, be it a 'dyi' or a commercial 
solution, before pushing it to the limit.

Unless of course, you have factored in the ability to deal with the 
consequences.

Most 'DYI' solutions, make the non-techy bean counters very nervous, and seeing 
a major 'name brand' label for some odd reason makes them real comfortable, 
ir-respective of the capabilities or function of either solution.

If you have to answer to the bean counters, then this is a very valid point to 
be considered.


:)


Faisal Imtiaz
Snappy Internet & Telecom
7266 SW 48 Street
Miami, FL 33155
Tel: 305 663 5518 x 232

Help-desk: (305)663-5518 Option 2 or Email: supp...@snappytelecom.net 

- Original Message -
> From: "Jared Mauch" 
> To: "Faisal Imtiaz" 
> Cc: "Eugeniu Patrascu" , "North American Network 
> Operators' Group" 
> Sent: Friday, December 27, 2013 4:04:12 PM
> Subject: Re: The Making of a Router
> 
> 
> On Dec 27, 2013, at 3:37 PM, Faisal Imtiaz  wrote:
> 
> > e.g. If someone says I need a 10g interface, why is it automatically
> > assumed that the router is going to be running @ Full Line Rate ?
> 
> Those of us with experience know that when “something bad(tm)” happens, those
> features and “expensive silicon” start to show some ROI.  Is it a full
> trade-off?  Depends on the risks of your business and exposure.
> 
> You can get some inexpensive hardware to do fairly fancy features these days.
> That can be very good, but caries that risk.  Make sure you evaluate it
> carefully.
> 
> - Jared

Re: The Making of a Router

[Matt Palmer]

On Fri, Dec 27, 2013 at 10:18:47AM -0500, Jon Sands wrote:
> On Dec 27, 2013 10:08 AM, "Baldur Norddahl" 
> wrote:
> 
> > We are an upstart and just buying the fancy Juniper switch times two
> would burn half of my seed capital.
> 
> Then you didn't ask for nearly enough capital.

There *is* a world outside of Silly Valley, you know... a world where money
doesn't flow like a mighty cascade from the benevolent wallets of vulture
capitalists, into the waiting arms of every crackpot with an elevator pitch.

- Matt

Re: The Making of a Router

[Justin M. Streiner]

On Fri, 27 Dec 2013, Baldur Norddahl wrote:


Everybody have critical services running on servers. DHCP, DNS, Radius and
so on are all on servers and you will be down if these services are down.
What is with the knee jerk reaction for suggesting that the BGP daemon
could also be run on a server? There seems to be many advantages of doing
it this way, and not all of them are related to cost.


If you want to use servers as routers, that's your choice.  I think what 
most people in the thread have been saying is not to use one server (or 
even a pair of servers) for everything.  It's one thing if server XYZ goes 
down and some web services are offline.  It's another thing entirely if 
that same server goes down and your entire business is offline.


jms

Re: The Making of a Router

[Jared Mauch]

On Dec 27, 2013, at 3:37 PM, Faisal Imtiaz  wrote:

> e.g. If someone says I need a 10g interface, why is it automatically assumed 
> that the router is going to be running @ Full Line Rate ?

Those of us with experience know that when “something bad(tm)” happens, those 
features and “expensive silicon” start to show some ROI.  Is it a full 
trade-off?  Depends on the risks of your business and exposure.

You can get some inexpensive hardware to do fairly fancy features these days.  
That can be very good, but caries that risk.  Make sure you evaluate it 
carefully.

- Jared

Re: The Making of a Router

[Faisal Imtiaz]

I Am not trying to be inflammatory..

Why does everything has to be measured in Extremes ?
e.g. If someone says I need a 10g interface, why is it automatically assumed 
that the router is going to be running @ Full Line Rate ?

I think we often confuse the performance of "Software" and Hardware as being 
monolithic, which is far from the truth.

In today fast moving world, we don't even bother to look under the hood to see 
what is it that is being taxed.. the Software ? or Hardware ? or process of how 
we are doing what we want to get accomplished  ?  

While it is easy to talk about the extremes, we often forget to point out to 
others that the 'Middle' is a huge place.

Regards

Faisal Imtiaz
Snappy Internet & Telecom

- Original Message -
> From: "Eugeniu Patrascu" 
> To: "Baldur Norddahl" 
> Cc: nanog@nanog.org
> Sent: Friday, December 27, 2013 3:23:11 PM
> Subject: Re: The Making of a Router
> 
> On Fri, Dec 27, 2013 at 10:00 PM, Baldur Norddahl  > wrote:
> 
> > On Fri, Dec 27, 2013 at 4:18 PM, Jon Sands  wrote:
> >
> > > On Dec 27, 2013 10:08 AM, "Baldur Norddahl" 
> > > wrote:
> > >
> > > > We are an upstart and just buying the fancy Juniper switch times two
> > > would burn half of my seed capital.
> > >
> > > Then you didn't ask for nearly enough capital.
> > >
> >
> > Another told Nick Cameo that if he can afford a 10G link, he can afford
> > Juniper. You could not be more wrong. The 10G uplink goes for $0 in initial
> > fee and less than $4k / month with unlimited traffic. The Juniper gear is
> > $100k up front for two routers able to handle the 10G links.
> >
> >
> What you should understand is not the fact that a 10G interface is
> expensive, but what you can do with that interface tends to get very
> expensive.
> If you want to move traffic from one interface to another, you can achieve
> this today with two physical interfaces on a Linux box. How many PPS ?
> Well, that's another story. You then want shaping, Q-in-Q and other stuff
> which consume a lot of resources even on dedicated hardware.
> 
> 
> > What I get from you guys is that in your opinion it is not possible to set
> > up a small ISP without spending a ton on Juniper or Cisco. I am not buying
> > that. Even if I did not have a clear limit on my capital, I would be
> > looking at avoiding paying that kind of money, because in the end the money
> > comes out of my own pocket.
> >
> >
> You can build your ISP without getting big routers but you need to cut back
> a little bit on your expectations about what you can in terms of features:
> - Do pool NAT for your users if they accept this. You can easily squeeze a
> lot of users on a single IP address. Downside is that if one of them does
> something bad, that IP might get blackholed on some providers and the rest
> will suffer. Also, you might want to take into consideration regulatory
> requirements like to know what users used what port to what destination for
> a certain number of months (in Europe regulations vary, but the smallest
> period is 6 months).
> - If you give them VoIP/IPTV then assign a VLAN for VOIP and another for
> IPTV and run it to all your users to their STBs and make use of IGMP
> snooping for Multicast traffic on all your switches
> - You can run full table BGP with Quagga on Linux (it worked for me when
> the DFZ was at around 270k prefixes, I assume it will work with 480k
> prefixes today) - also, do you really need full tables ?. Your IGP, if you
> don't run anything fancy should be a few tens of routes, that can be
> achieved with modest L3 switches that do 64/128 routes in hardware.
> 
> 
> > Everybody have critical services running on servers. DHCP, DNS, Radius and
> > so on are all on servers and you will be down if these services are down.
> > What is with the knee jerk reaction for suggesting that the BGP daemon
> > could also be run on a server? There seems to be many advantages of doing
> > it this way, and not all of them are related to cost.
> >
> >
> For the sake of a good night sleep, you would want to separate all the
> services on different physical machines for redundancy/availability and
> load sharing.
> 
> Once you grow, you can move to more powerful and dedicated hardware for
> your networking needs.
> 
> Eugeniu
> 

Re: The Making of a Router

[Faisal Imtiaz]

Well said Baldur

For those who are movie buffs.. here is the snippet that visually summaries..

http://www.youtube.com/watch?v=dEkOT3IngMQ


As to the knee jerk reaction to a server doing routing such folks tend to 
forget that 
Routers are purpose built serversand most of the Internet Routing was 
originally done by servers (or main frames or minis) etc.

:)


Faisal Imtiaz
Snappy Internet & Telecom
7266 SW 48 Street
Miami, FL 33155
Tel: 305 663 5518 x 232

Help-desk: (305)663-5518 Option 2 or Email: supp...@snappytelecom.net 

- Original Message -

> What I get from you guys is that in your opinion it is not possible to set
> up a small ISP without spending a ton on Juniper or Cisco. I am not buying
> that. Even if I did not have a clear limit on my capital, I would be
> looking at avoiding paying that kind of money, because in the end the money
> comes out of my own pocket.
> 
> Everybody have critical services running on servers. DHCP, DNS, Radius and
> so on are all on servers and you will be down if these services are down.
> What is with the knee jerk reaction for suggesting that the BGP daemon
> could also be run on a server? There seems to be many advantages of doing
> it this way, and not all of them are related to cost.
> 
> Regards,
> 
> Baldur
> 

Re: The Making of a Router

[Eugeniu Patrascu]

On Fri, Dec 27, 2013 at 10:00 PM, Baldur Norddahl  wrote:

> On Fri, Dec 27, 2013 at 4:18 PM, Jon Sands  wrote:
>
> > On Dec 27, 2013 10:08 AM, "Baldur Norddahl" 
> > wrote:
> >
> > > We are an upstart and just buying the fancy Juniper switch times two
> > would burn half of my seed capital.
> >
> > Then you didn't ask for nearly enough capital.
> >
>
> Another told Nick Cameo that if he can afford a 10G link, he can afford
> Juniper. You could not be more wrong. The 10G uplink goes for $0 in initial
> fee and less than $4k / month with unlimited traffic. The Juniper gear is
> $100k up front for two routers able to handle the 10G links.
>
>
What you should understand is not the fact that a 10G interface is
expensive, but what you can do with that interface tends to get very
expensive.
If you want to move traffic from one interface to another, you can achieve
this today with two physical interfaces on a Linux box. How many PPS ?
Well, that's another story. You then want shaping, Q-in-Q and other stuff
which consume a lot of resources even on dedicated hardware.


> What I get from you guys is that in your opinion it is not possible to set
> up a small ISP without spending a ton on Juniper or Cisco. I am not buying
> that. Even if I did not have a clear limit on my capital, I would be
> looking at avoiding paying that kind of money, because in the end the money
> comes out of my own pocket.
>
>
You can build your ISP without getting big routers but you need to cut back
a little bit on your expectations about what you can in terms of features:
- Do pool NAT for your users if they accept this. You can easily squeeze a
lot of users on a single IP address. Downside is that if one of them does
something bad, that IP might get blackholed on some providers and the rest
will suffer. Also, you might want to take into consideration regulatory
requirements like to know what users used what port to what destination for
a certain number of months (in Europe regulations vary, but the smallest
period is 6 months).
- If you give them VoIP/IPTV then assign a VLAN for VOIP and another for
IPTV and run it to all your users to their STBs and make use of IGMP
snooping for Multicast traffic on all your switches
- You can run full table BGP with Quagga on Linux (it worked for me when
the DFZ was at around 270k prefixes, I assume it will work with 480k
prefixes today) - also, do you really need full tables ?. Your IGP, if you
don't run anything fancy should be a few tens of routes, that can be
achieved with modest L3 switches that do 64/128 routes in hardware.


> Everybody have critical services running on servers. DHCP, DNS, Radius and
> so on are all on servers and you will be down if these services are down.
> What is with the knee jerk reaction for suggesting that the BGP daemon
> could also be run on a server? There seems to be many advantages of doing
> it this way, and not all of them are related to cost.
>
>
For the sake of a good night sleep, you would want to separate all the
services on different physical machines for redundancy/availability and
load sharing.

Once you grow, you can move to more powerful and dedicated hardware for
your networking needs.

Eugeniu

Re: The Making of a Router

[Baldur Norddahl]

On Fri, Dec 27, 2013 at 4:18 PM, Jon Sands  wrote:

> On Dec 27, 2013 10:08 AM, "Baldur Norddahl" 
> wrote:
>
> > We are an upstart and just buying the fancy Juniper switch times two
> would burn half of my seed capital.
>
> Then you didn't ask for nearly enough capital.
>

Another told Nick Cameo that if he can afford a 10G link, he can afford
Juniper. You could not be more wrong. The 10G uplink goes for $0 in initial
fee and less than $4k / month with unlimited traffic. The Juniper gear is
$100k up front for two routers able to handle the 10G links.

What I get from you guys is that in your opinion it is not possible to set
up a small ISP without spending a ton on Juniper or Cisco. I am not buying
that. Even if I did not have a clear limit on my capital, I would be
looking at avoiding paying that kind of money, because in the end the money
comes out of my own pocket.

Everybody have critical services running on servers. DHCP, DNS, Radius and
so on are all on servers and you will be down if these services are down.
What is with the knee jerk reaction for suggesting that the BGP daemon
could also be run on a server? There seems to be many advantages of doing
it this way, and not all of them are related to cost.

Regards,

Baldur

Re: The Making of a Router

[Justin M. Streiner]

On Thu, 26 Dec 2013, Andrew D Kirch wrote:

If he can afford a 10G link... he should be buying real gear...  I mean, 
look, I've got plenty of infrastructure horror stories, but lets not cobble 
together our own 10gbit solutions, please?  At least get one of the new 
microtik CCR's with a 10gig sfp+?  They're only a kilobuck... If you can't 
afford that I suggest you can't afford to be an ISP.


+1

Build-your-own routers are perfectly OK for a lab environment if you want 
to tinker with something, but I absolutely would not put an all-in-one box 
that I built myself in production.  You end up combining some of the 
downsides of a hardware-based router with some of the downsides of a 
server (new attack vectors, another device that needs to be backed up, 
patched, and monitored, possibly getting a new collection of devices and 
drivers to play nicely with each other, etc).


Doing this also requires all of the people in your on-call rotation to be 
experienced sysadmins / server ops, in addition to being experiences 
network engineers / NOC ops.  There are a lot of occasions with a server 
where 'just reboot it' can make a problem much worse.


Route servers running Linux or *BSD are another story.  There are many 
situations where they can be extremely useful, but they are not all-in-one 
route server/RADIUS/VPN termination/web server/user shell boxes.


jms

Re: The Making of a Router

[Leonardo Arena]

On gio, 2013-12-26 at 11:33 -0500, Nick Cameo wrote:
> Hello Everyone,
> 
> We are looking to put together a 2u server with a few PCIe 3 x8
> (recommendations appreciated). The router will take a voip transcoding
> line card, and will act as an edge router for a telecom company.
> 
> For things like BGP (Quagga, Zebra, all that lovely stuff!!!), static
> routes, and firewall capabilities we are thinking gentoo linux
> stripped for sure however, what about the BSDs? FreeBSD or OpenBSD.
> Any comments, feedback, does, and don'ts are much appreciated.
> 
> Kind Regards,
> 
> Nick.
> 

I would definitely consider Alpine Linux [1], which is designed exactly
for that purpose. Small footprint, run-from-ram, config from on flash,
security oriented.

KR,

- leo

[1] http://alpinelinux.org/about

Re: The Making of a Router

[Jon Sands]

On Dec 27, 2013 10:08 AM, "Baldur Norddahl" 
wrote:

> We are an upstart and just buying the fancy Juniper switch times two
would burn half of my seed capital.

Then you didn't ask for nearly enough capital.

Re: The Making of a Router

[Baldur Norddahl]

I need a solution for everything except the last-mile customers. The
customers are connected to a Zhone PON switch. From there they will arrive
at our core switch as Q-in-Q vlans, one vlan per customer. I need a router
that will do two full routing tables for our uplinks, a number of partial
routing tables for our IX peers,  IPv6 support, IPv4 proxy arp support and
the ability to handle a large number of Q-in-Q vlans. And of course I will
need two for redundancy. The uplinks, the links to edge switches and many
of the IX peers are all 10 Gbit/s links.

IPv4 proxy arp is especially important given the state of IPv4 exhaustion.
Being a new ISP in the RIPE region, we only got 1024 IPs. When we run out
of that initial assignment, we have to buy IP-addresses at a steep price.
Therefore we can not afford to give each home a full IPv4 subnet. They will
have to share the subnet with multiple other customers. This is achieved
through proxy arp on the switch.

We are an upstart and just buying the fancy Juniper switch times two would
burn half of my seed capital.

Like Nick Cameo I have seriously considered going with a Linux solution. I
know I can build it. I just don't know if I can make it stable enough or
make it perform good enough.

I am looking into an OpenFlow solution as a middle ground. It allows me to
buy cheaper switches/routers. The servers will do the "thinking" but the
actual work of moving packets is still done in hardware on the switches.
OpenFlow supports controller fail over, so I will not go down with just one
server crash. Poor performance on the servers will not affect customer
traffic directly.

Regards,

Baldur





On Fri, Dec 27, 2013 at 2:11 PM, Eugeniu Patrascu wrote:

> On Fri, Dec 27, 2013 at 3:05 PM, Baldur Norddahl <
> baldur.nordd...@gmail.com> wrote:
>
>> On the topic of building a software router for an ISP, has anyone tried it
>> using OpenFlow? The idea is to have a Linux server run BGP and a hardware
>> switch to move the packets. The switch would be programmed by the Linux
>> server using the OpenFlow protocol.
>>
>> I am looking at the HP 5400 zl switches as the hardware platform and
>> RouteFlow https://sites.google.com/site/routeflow/ to program the BGP
>> rules.
>>
>> One issue is that the HP switch will only allow a limited amount of rules
>> to be processed in hardware (about 4096 rules I believe). Will this be
>> enough to cover most of the traffic of a FTTH ISP on the fast path?
>>
>
> You want to use the switch for what ? To connect last-mile customers ? For
> L3 aggregation ? You want to run the switch as an edge router with limited
> BGP ? What's the exact use case you are thinking about ?
>
> Eugeniu
>

Re: The Making of a Router

[Francois Menard]

You could look into Noviflow!

F.

Sent from my mobile device. Apologies for any typo.

> On Dec 27, 2013, at 8:05, Baldur Norddahl  wrote:
> 
> On the topic of building a software router for an ISP, has anyone tried it
> using OpenFlow? The idea is to have a Linux server run BGP and a hardware
> switch to move the packets. The switch would be programmed by the Linux
> server using the OpenFlow protocol.
> 
> I am looking at the HP 5400 zl switches as the hardware platform and
> RouteFlow https://sites.google.com/site/routeflow/ to program the BGP rules.
> 
> One issue is that the HP switch will only allow a limited amount of rules
> to be processed in hardware (about 4096 rules I believe). Will this be
> enough to cover most of the traffic of a FTTH ISP on the fast path?
> 
> Regards,
> 
> Baldur

Re: The Making of a Router

[Eugeniu Patrascu]

On Fri, Dec 27, 2013 at 3:05 PM, Baldur Norddahl
wrote:

> On the topic of building a software router for an ISP, has anyone tried it
> using OpenFlow? The idea is to have a Linux server run BGP and a hardware
> switch to move the packets. The switch would be programmed by the Linux
> server using the OpenFlow protocol.
>
> I am looking at the HP 5400 zl switches as the hardware platform and
> RouteFlow https://sites.google.com/site/routeflow/ to program the BGP
> rules.
>
> One issue is that the HP switch will only allow a limited amount of rules
> to be processed in hardware (about 4096 rules I believe). Will this be
> enough to cover most of the traffic of a FTTH ISP on the fast path?
>

You want to use the switch for what ? To connect last-mile customers ? For
L3 aggregation ? You want to run the switch as an edge router with limited
BGP ? What's the exact use case you are thinking about ?

Eugeniu

Re: The Making of a Router

[Baldur Norddahl]

On the topic of building a software router for an ISP, has anyone tried it
using OpenFlow? The idea is to have a Linux server run BGP and a hardware
switch to move the packets. The switch would be programmed by the Linux
server using the OpenFlow protocol.

I am looking at the HP 5400 zl switches as the hardware platform and
RouteFlow https://sites.google.com/site/routeflow/ to program the BGP rules.

One issue is that the HP switch will only allow a limited amount of rules
to be processed in hardware (about 4096 rules I believe). Will this be
enough to cover most of the traffic of a FTTH ISP on the fast path?

Regards,

Baldur

Re: The Making of a Router

[Ray Soucy]

In talking about RAMBOOT I also realized the instructions are out of date
on the website.

The "ramboot" boot target script was updated since I created the initial
page to generate the correct fstab, and enable the root account, set a
hostname, etc. so you can actually use the OS until you create a new image.

I extracted the script from the initird to make it easier to grab:

http://ramboot.org/download/RAMBOOT/RAMBOOT-pre0.2/SYSLINUX/initrd/scripts/ramboot

Essentially, by adding a new "ramboot" script to
"/usr/share/initramfs-tools/scripts" along side "nfs" and "local" it
creates a new "boot=" target (since the init script looks for
"scripts/${BOOT}".

As mentioned on the website, the ramboot process needs a more complete
version of busybox (for tar archive support) and the mke2fs tool added to
"/usr/lib/initramfs-tools/bin/" so they will be available to the initrd.

Once you configure networking (see the "INSTALL/setup_network" script) you
can do an apt-get update and apt-get the packages you need from Ubuntu
12.04 LTS.

Example starting point:

apt-get install sudo
apt-get install nano
apt-get install ssh
apt-get install vlan
apt-get install bridge-utils



On Thu, Dec 26, 2013 at 8:27 PM, Ray Soucy  wrote:

> The basic idea of RAMBOOT is typical in Embedded Linux development.
>
> Linux makes use of multi-stage boot process.  One of the stages involves
> using an initial ramdisk (initrd) to provide a base root filesystem which
> can be used to locate and mount the system root, then continue the boot
> process from there.
>
> For an in-memory OS, instead of the boot process mounting a pre-loaded
> storage device with the expected root filesystem (e.g. your installed HDD),
> you modify it to:
>
> 1) Create and format a ramdisk.
> 2) Create the expected directory structure and system files for the root
> filesystem on that ramdisk.
>
> The root filesystem includes the /dev directory and appropriate device
> nodes, the basic Linux filesystem and your init program.
>
> The easy way to do that is just to have a TAR archive that you extract to
> the ramdisk on boot, better yet use compression on it (e.g. tar.gz) so that
> the archive can be read from storage (e.g. USB flash) more quickly.
>
> Today, the initramfs in Linux handles a lot more than simply mounting the
> storage device.  It performs hardware discovery and loads appropriate
> modules.  As such the Debian project has a dynamic build system for
> initramfs that is run to build the initrd when a new kernel package is
> installed, it's called "initramfs-tools".
>
> You can manually build your own initramfs using the examples on the
> RAMBOOT website, but the point of RAMBOOT is to make building an in-memory
> OS quick and simple.
>
> RAMBOOT instead adds configuration to initramfs-tools so that each time a
> new initrd is generated, it includes the code needed for RAMBOOT.
>
> The RAMBOOT setup adds handling of a new boot target called "ramboot" to
> the kernel arguments.  This allows the same kernel to be used for a normal
> installation and remain unaffected, but when you add the argument
> "boot=ramboot" as a kernel option to the bootloader, it triggers the
> RAMBOOT process described above.
>
> Having a common kernel between your development environment and embedded
> environment makes it much easier to test and verify functionality.
>
> The other part of RAMBOOT is that it makes use of "Ubuntu Core".  Ubuntu
> Core is a stripped down minimal (and they really do mean minimal) root
> filesystem for Embedded Linux development.  It includes apt-get, though, so
> you can install all the packages you need from Ubuntu on the running system.
>
> RAMBOOT then has a development script to make a new root filesystem
> archive with the packages you've installed as a baseline.  This allows for
> you to boot a RAMBOOT system, install your desired packages and change
> system configuration files as desired, then build a persistent image of
> that install that will be used for future boots.
>
> I also have the start of a script to remove unused kernel modules, and
> other files (internationalization for example) which add to the OS
> footprint.
>
> You could build the root filesystem on your own (and compile all the
> necessary packages) but using Ubuntu Core provides a solid base and allows
> for the rapid addition of packages from the giant Ubuntu repository.
>
> Lastly, I make use of SYSLINUX as a bootloader because my goal was to use
> a USB stick as the bootflash on an Atom box.  Unfortunately, the Atom BIOS
> will only boot a USB device if it has a DOS boot partition, so GRUB was a
> no-go.  The upside is that since the USB uses SYSLINUX and is DOS
> formatted, it's easily mounted in Windows or Mac OS X, allowing you to copy
> new images or configuration to it easily.
>
> For the boot device I make use of the on-board vertical USB socket on the
> system board (typical for most system boards these days) and a low-profile
> USB stick.  I find the Verbatim "S

Re: The Making of a Router

[shawn wilson]

On Fri, Dec 27, 2013 at 1:33 AM,   wrote:
> On Thu, 26 Dec 2013 11:16:53 -0800, Seth Mattinen said:
>> On 12/26/13, 9:24, Andrew D Kirch wrote:
>> >
>> > If he can afford a 10G link... he should be buying real gear...  I mean,
>> > look, I've got plenty of infrastructure horror stories, but lets not
>> > cobble together our own 10gbit solutions, please?  At least get one of
>> > the new microtik CCR's with a 10gig sfp+?  They're only a kilobuck... If
>> > you can't afford that I suggest you can't afford to be an ISP.
>>
>>
>> Unless all the money is going into the 10 gig link.
>
> If you've sunk so much into the 10G link (or anything else, for that matter)
> that you don't have a kilobuck to spare, you're probably undercapitalized to 
> be
> an ISP.
>

I have issue with this line of thought. Granted, a router is built
with custom ASICs and most network people understand IOS. However,
this is where the benefit of a multi-thousand buck router ends. Most
have limited RAM, so this limits the size of your policies and how
many routes can be stored and the likes. With a computer with multi
10s or 100s of gigs of RAM, this really isn't an issue. Routers also
have slow-ish processors (which is fine for pure routing since they
are custom chips but) if you want to do packet inspection, this can
slow things down quite a bit. You could argue that this is the same
with iptables or pf. However, if you just offload the packets and
analyze generally boring packets with snort or bro or whatever,
packets flow as fast as they would without analysis. If you have
multiple VPNs, this can start to slow down a router whereas a computer
can generally keep up.

... And then there's the money issue. Sure, if you're buying a gig+
link, you should be able to afford a fully spec'd out router. However,
(in my experience) people don't order equipment with all features
enabled and when you find you need a feature, you have to put in a
request to buy it and then it takes a month (if you're lucky) for it
to be approved. This isn't the case if you use ipt/pf - if te feature
is there, it's there - use it.

And if a security flaw is found in a router, it might be fixed in the
next month... or not. With Linux/BSD, it'll be fixed within a few days
(at the most). And, if your support has expired on a router or the
router is EOL, you're screwed.

I think in the near future, processing packets with GPUs will become a
real thing which will make doing massive real time deep packet
inspection at 10G+ a real thing.

Granted, your network people knowing IOS when they're hired is a big
win for just ordering Cisco. But, I don't see that as a show stopper.
Stating the scope of what a box is supposed to be used for and not
putting endless crap on it might be another win for an actual router.
However, this is a people/business thing and not a technical issue.

Also, I'm approaching this as more of a question of the best tool for
the job vs pure economics - a server is generally going to be cheaper,
but I generally find a server nicer/easier to configure than a router.

Re: The Making of a Router

[Valdis . Kletnieks]

On Thu, 26 Dec 2013 11:16:53 -0800, Seth Mattinen said:
> On 12/26/13, 9:24, Andrew D Kirch wrote:
> >
> > If he can afford a 10G link... he should be buying real gear...  I mean,
> > look, I've got plenty of infrastructure horror stories, but lets not
> > cobble together our own 10gbit solutions, please?  At least get one of
> > the new microtik CCR's with a 10gig sfp+?  They're only a kilobuck... If
> > you can't afford that I suggest you can't afford to be an ISP.
>
>
> Unless all the money is going into the 10 gig link.

If you've sunk so much into the 10G link (or anything else, for that matter)
that you don't have a kilobuck to spare, you're probably undercapitalized to be
an ISP.



pgp38ehiqud4d.pgp
Description: PGP signature

Re: The Making of a Router

[Ray Soucy]

Chipsets and drivers matter a lot in the 1G+ range.

I've had pretty good luck with the Intel stuff because they offload a lot
in hardware and make open drivers available to the community.


On Thu, Dec 26, 2013 at 7:48 PM, Olivier Cochard-Labbé
wrote:

> Le 26 déc. 2013 22:02, "Nick Cameo"  a écrit :
> >
> > Any benchmarks of freebsd vs openbsd vs present day linux kern?
> >
> Hi,
>
> Here are my own benchs using smallest packet size (sorry no Linux):
> http://dev.bsdrp.net/benchs/BSD.network.performance.TenGig.png
>
> My conclusion: building a line-rate gigabit router (or a few rules ipfw
> firewall) is possible on commodity server without problem with FreeBSD.
> Building a 10gigabit router (this mean routing about 14Mpps) will be more
> complex in present day.
> Note: The packet generator used was the high-perf netmap pkg-gen, allowing
> me to generate about 13Mpps on this same hardware (under FreeBSD), but I'm
> not aware of forwarding tools that use netmap: There are only packet
> generator and capture tools available.
>



-- 
Ray Patrick Soucy
Network Engineer
University of Maine System

T: 207-561-3526
F: 207-561-3531

MaineREN, Maine's Research and Education Network
www.maineren.net

Re: The Making of a Router

[Ray Soucy]

The basic idea of RAMBOOT is typical in Embedded Linux development.

Linux makes use of multi-stage boot process.  One of the stages involves
using an initial ramdisk (initrd) to provide a base root filesystem which
can be used to locate and mount the system root, then continue the boot
process from there.

For an in-memory OS, instead of the boot process mounting a pre-loaded
storage device with the expected root filesystem (e.g. your installed HDD),
you modify it to:

1) Create and format a ramdisk.
2) Create the expected directory structure and system files for the root
filesystem on that ramdisk.

The root filesystem includes the /dev directory and appropriate device
nodes, the basic Linux filesystem and your init program.

The easy way to do that is just to have a TAR archive that you extract to
the ramdisk on boot, better yet use compression on it (e.g. tar.gz) so that
the archive can be read from storage (e.g. USB flash) more quickly.

Today, the initramfs in Linux handles a lot more than simply mounting the
storage device.  It performs hardware discovery and loads appropriate
modules.  As such the Debian project has a dynamic build system for
initramfs that is run to build the initrd when a new kernel package is
installed, it's called "initramfs-tools".

You can manually build your own initramfs using the examples on the RAMBOOT
website, but the point of RAMBOOT is to make building an in-memory OS quick
and simple.

RAMBOOT instead adds configuration to initramfs-tools so that each time a
new initrd is generated, it includes the code needed for RAMBOOT.

The RAMBOOT setup adds handling of a new boot target called "ramboot" to
the kernel arguments.  This allows the same kernel to be used for a normal
installation and remain unaffected, but when you add the argument
"boot=ramboot" as a kernel option to the bootloader, it triggers the
RAMBOOT process described above.

Having a common kernel between your development environment and embedded
environment makes it much easier to test and verify functionality.

The other part of RAMBOOT is that it makes use of "Ubuntu Core".  Ubuntu
Core is a stripped down minimal (and they really do mean minimal) root
filesystem for Embedded Linux development.  It includes apt-get, though, so
you can install all the packages you need from Ubuntu on the running system.

RAMBOOT then has a development script to make a new root filesystem archive
with the packages you've installed as a baseline.  This allows for you to
boot a RAMBOOT system, install your desired packages and change system
configuration files as desired, then build a persistent image of that
install that will be used for future boots.

I also have the start of a script to remove unused kernel modules, and
other files (internationalization for example) which add to the OS
footprint.

You could build the root filesystem on your own (and compile all the
necessary packages) but using Ubuntu Core provides a solid base and allows
for the rapid addition of packages from the giant Ubuntu repository.

Lastly, I make use of SYSLINUX as a bootloader because my goal was to use a
USB stick as the bootflash on an Atom box.  Unfortunately, the Atom BIOS
will only boot a USB device if it has a DOS boot partition, so GRUB was a
no-go.  The upside is that since the USB uses SYSLINUX and is DOS
formatted, it's easily mounted in Windows or Mac OS X, allowing you to copy
new images or configuration to it easily.

For the boot device I make use of the on-board vertical USB socket on the
system board (typical for most system boards these days) and a low-profile
USB stick.  I find the Verbatim "Store 'n' Go" 8GB USB stick ideally suited
for this as it's less than a quarter-inch high after the USB adapter.

RAMBOOT as a project is in the very early stages, so you should be
comfortable with Linux before you build a system on it.  And I really feel
it's more of an example than anything at this point.

There are several advantages though:

The most common point of failure on a Linux system is the storage device
(either HDD or SSD).
The biggest bottleneck in system performance is storage IO.
Using a ramdisk eliminates both these concerns (in fact, even an Atom
system has surprisingly great performance when run using a ramdisk).

The result is that you get a very reliable, high-performance system.

The other benefit to RAMBOOT is that the root filesystem is NOT persistent.
 This means that like a Cisco device, every boot of the system brings you
to a known working state OS-wise.  There are hundreds of system files in a
Linux system; any one of which being modified could cause problems.  For
both security and availability concerns a lot of effort is invested in
detecting changes to system files and avoiding them.  With RAMBOOT the
problem is easily avoided.

A minimal system can fit within a 512MB ramdisk.  But with RAM being so
cheap these days, I think even reserving up to 2GB of RAM for a ramdisk
would be fine (e.g. for a 4-8GB system).

Re: The Making of a Router

[Olivier Cochard-Labbé]

Le 26 déc. 2013 22:02, "Nick Cameo"  a écrit :
>
> Any benchmarks of freebsd vs openbsd vs present day linux kern?
>
Hi,

Here are my own benchs using smallest packet size (sorry no Linux):
http://dev.bsdrp.net/benchs/BSD.network.performance.TenGig.png

My conclusion: building a line-rate gigabit router (or a few rules ipfw
firewall) is possible on commodity server without problem with FreeBSD.
Building a 10gigabit router (this mean routing about 14Mpps) will be more
complex in present day.
Note: The packet generator used was the high-perf netmap pkg-gen, allowing
me to generate about 13Mpps on this same hardware (under FreeBSD), but I'm
not aware of forwarding tools that use netmap: There are only packet
generator and capture tools available.

Re: The Making of a Router

[Jay Ashworth]

- Original Message -
> From: "Matt Palmer" 

> I've got plenty of horror stories of vendor support. Perhaps we can get
> together some day and have a story-off. 

Just subject-tag it so we can archive it, ok guys?

Cheers,
-- jr 'whacky weekend' a
-- 
Jay R. Ashworth  Baylink   j...@baylink.com
Designer The Things I Think   RFC 2100
Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII
St Petersburg FL USA   #natog  +1 727 647 1274

Re: The Making of a Router

[Valdis . Kletnieks]

On Thu, 26 Dec 2013 11:33:13 -0500, Nick Cameo said:
> Hello Everyone,
>
> We are looking to put together a 2u server with a few PCIe 3 x8
> (recommendations appreciated). The router will take a voip transcoding
> line card, and will act as an edge router for a telecom company.

Two things you want to do:

1) Split this into multiple boxes if you can.  That makes maintaining
one component a lot easier, especially when you get to point 2, which is...

2) Redundancy/failover.  Sure, it may be more expensive, but the first
time your HA failover changes a 2AM "Holy Crap" into an "Oh, bother" it
will be worth the price of admission


pgp9bla2DfM_J.pgp
Description: PGP signature

Re: The Making of a Router

[Nick Cameo]

>> Unless they deem that it's "outside of scope".  Or they can't get anyone to
>> you inside of SLA[1].  Or they send someone incompetent.  Or it's a problem
>> that's never happened before.

Amen!

>> *Everything* is a nightmare to support.  A DIY project just means that
>> you're betting you're smarter than whoever the vendor sends to fix their
>> thing.  Maybe it's a good bet, maybe it isn't.

Amen Again!


>> I'm sure you've got plenty of horror stories of DIY project support; I've
>> got plenty of horror stories of vendor support.  Perhaps we can get together
>> some day and have a story-off.  

Nightmares come in colours of green, teal, and purple!

Re: The Making of a Router

[Matt Palmer]

On Thu, Dec 26, 2013 at 05:21:11PM +, Warren Bailey wrote:
> Not to mention the fact that this "router" will require support. The build
> before buy people are silly.  Let the smart router guys do their thing and
> use their box accordingly.  When it breaks call to inform them it broke
> and they will fix it.

Unless they deem that it's "outside of scope".  Or they can't get anyone to
you inside of SLA[1].  Or they send someone incompetent.  Or it's a problem
that's never happened before.

> Diy projects are a nightmare to support.

*Everything* is a nightmare to support.  A DIY project just means that
you're betting you're smarter than whoever the vendor sends to fix their
thing.  Maybe it's a good bet, maybe it isn't.

I'm sure you've got plenty of horror stories of DIY project support; I've
got plenty of horror stories of vendor support.  Perhaps we can get together
some day and have a story-off.  

- Matt

[1] So you might get some SLA credits at some point in the future.  So what? 
It won't even cover your SLA payouts to your customers, let alone the lost
business and reputation.

Re: The Making of a Router

[Nick Cameo]

Oh my bad. I did not mean it like that at all! I am more that capable
of putting it
together using gentoo instead of debian (a little pedagogy goes a long way). And
if he would like, he can post the ISO on his webstie alongside the
different distro.
This is what I was leaning too...

Please don't be offended.

N.

Re: The Making of a Router

[Seth Mattinen]

On 12/26/13, 13:22, Nick Cameo wrote:

Ummm, if it's ok with the community, can you kindly elaborate:). I am
not too fond of Debian since my horrible experience with Squeeze Desktop.
I would maybe like to try this using the combination of SSD, in memory, and
Gentoo?



Not to sound rude, but if someone gives you a how-to but you don't like 
it (since making a router and a desktop environment are totally the same 
thing), you are welcome to come up with your own based on what you like 
instead of telling them to give you new instructions to suit your 
preferences.


~Seth

Re: The Making of a Router

[Nick Cameo]

> One of the biggest advantages is the low cost of hardware allows you to
> maintain spare systems, reducing the time to service restoration in the
> event of failure.  Dependability-wise, I feel that whitebox Linux systems
> are pretty much at Cisco levels these days, especially if running
> in-memory.

With your guidance, I can put together a gentoo environment that will
run in memory
tailored for this purpose, and would be obliged to share it with the
community if
anyone else is interested.

N.

Re: The Making of a Router

[Nick Cameo]

Inline response exist,

On 12/26/13, Ray Soucy  wrote:
> You can build using commodity hardware and get pretty good results.
>
> I've had really good luck with Supermicro whitebox hardware, and
> Intel-based network cards.  The "Hot Lava Systems" cards have a nice
> selection for a decent price if you're looking for SFP and SFP+ cards that
> use Intel chipsets.

I like the supermicro as well however we have a couple of IBM x3250
with 2 pcie v3
x8 that are begging for a intel network card.

> There might be some benefits in going with something like FreeBSD, but I
> find that Linux has a lot more eyeballs on it making it much easier to
> develop for, troubleshoot, and support.  There are a few options if you
> want to go the Linux route.

This is very important to consider. I would be speculating, or even
worse, expecting
the same type of community support from the BSD verse that I have been
getting from the linux community.

>
> Option 1: Roll your own OS.  This takes quite a bit of effort, but if you
> have the tallant to do it you can generally get exactly what you want.

If Free/OpenBSD is ruled out, I could crack open the LFS project. You only
have to do it once right? Or maybe just reach out to the gentoo community
for a stripped version, and build outwards.

> The biggest point of failure I've experienced with Linux-based routers on
> whitebox hardware has been HDD failure.  Other than that, the 100+ units
> I've had deployed over the past 3+ years have been pretty much flawless.
>

SSD

> Thankfully, they currently run an in-memory OS, so a disk failure only
> affects logging.
> If you want to build your own OS, I'll shamelessly plug a side project of
> mine: RAMBOOT
>
> http://ramboot.org/
>
> RAMBOOT makes use of the Ubuntu Core rootfs, and a modified boot process
> (added into initramfs tools, so kernel updates generate the right kernel
> automatically).  Essentially, I use a kernel ramdisk instead of an HDD for
> the root filesystem and "/" is mounted on "/dev/ram1".
>
> The bootflash can be removed while the system is running as it's only
> mounted to save system configuration or update the OS.
>
> I haven't polished it up much, but there is enough there to get going
> pretty quickly.

Ummm, if it's ok with the community, can you kindly elaborate :). I am
not too fond of Debian since my horrible experience with Squeeze Desktop.
I would maybe like to try this using the combination of SSD, in memory, and
Gentoo?

>
> You'll also want to pay attention to the settings you use for the kernel.
>  Linux is tuned as a desktop or server, not a router, so there are some
> basics you should take care of (like disabling ICMP redirects, increasing
> the ARP table size, etc).

Totally strip it as much as possible. If anyone has a Gentoo stripped kernel
config that they would like to share, please do :).

>
> I have some examples in: http://soucy.org/xorp/xorp-1.7-pre/TUNING
> or http://soucy.org/tmp/netfilter.txt (more recent, but includes firewall
> examples).

Will definitely look into all your sites.

>
> Also a note of caution.  I would stick with a longterm release of Linux.
>  I've had good experience with 2.6.32, and 3.10.  I'm eager to use some of
> the post-3.10 features, though, so I'm anxious for the next longterm branch
> to be locked in.
>

We are comfy with 3.4 right now...


> One of the biggest advantages is the low cost of hardware allows you to
> maintain spare systems, reducing the time to service restoration in the
> event of failure.  Dependability-wise, I feel that whitebox Linux systems
> are pretty much at Cisco levels these days, especially if running
> in-memory.

Really interested with the "in-memory", however, I would love to implement
it using gentoo as mentioned above.


Kind Regards,

N.

Re: The Making of a Router

[Thomas York]

On 12/26/13 11:33 AM, "Nick Cameo"  wrote:

>Hello Everyone,
>
>We are looking to put together a 2u server with a few PCIe 3 x8
>(recommendations appreciated). The router will take a voip transcoding
>line card, and will act as an edge router for a telecom company.
>
>For things like BGP (Quagga, Zebra, all that lovely stuff!!!), static
>routes, and firewall capabilities we are thinking gentoo linux
>stripped for sure however, what about the BSDs? FreeBSD or OpenBSD.
>Any comments, feedback, does, and don'ts are much appreciated.
>
>Kind Regards,
>
>Nick.
>



Depends on how skilled you are at maintaining Linux vs BSD, honestly.
Personally, I've accomplished something similar with great performance in
the past on Linux. I ran Debian 7 + latest compiled Quagga + latest
compiled Libreswan + Shorewall. If you're going to have a lot of different
people changing the rules, I would go with Shorewall. The syntax is
brain-dead simple, even though you're stuck with the network stack
limitations of Linux. A lot of my issues with doing this in Linux have to
do with distro's loading a bunch of net filter helpers by default, which
can be a major pain in the ass (I'm looking at you, SIP and SNMP modules).
I had to do a lot of tweaking to the conn track tables to make them large
enough to handle lots of traffic, but obviously YMMV.

Have you tried labbing BSD vs Linux to see which you like better? I'd
probably do that before throwing it in to production.
--
Thomas York
ExactTarget, a salesforce.com company 
Network Engineer
ty...@exacttarget.com
Office: (317) 832-4384
Mobile: (317) 660-5426

Re: The Making of a Router

[Nick Cameo]

On 12/26/13, Alessandro Ratti  wrote:
> if you want build by yourself I will suggest gentoo and/or freebsd with
> bird (http://bird.network.cz/) for routing stuff (maybe with 10G nics).

Hello Alessandro,

Any benchmarks of freebsd vs openbsd vs present day linux kern?

Re: The Making of a Router

[Nick Cameo]

On 12/26/13, Faisal Imtiaz  wrote:
> I am a believer of not having to re-invent the wheel...
>
> Having said that.. have you looked at 'purpose built appliances'  e.g.
>
> http://www.lannerinc.com/
> http://us.axiomtek.com/
>
> If you are looking for a full router
> Consider such as these...
>http://www.linktechs.net/
>http://www.maxxwave.com/
>
> and there are a few others but the concept is the same
>
> Personally, I am not a believer in making a single device be the do all /
> end all of everything..
> While one can do everything on a big server .. however breaking things out
> e.g. voip trans-coding and routing make maintenance, availability, and
> ability to create redundancy much more practical.
>

Point taken! Transcoding tasks abstracted out :).

N.

Re: The Making of a Router

[Nick Cameo]

> Have you tried labbing BSD vs Linux to see which you like better? I'd
> probably do that before throwing it in to production.
> --


Great advice Thomas! I will be creating a BSD virtual machine to get a
feel however, with linux I can think broad scale and forcast better.
With BSD, I am concerned with making choices that will shoot me in the
foot.

In essence, which BSD operating (Free vs OpenBSD) system is more
widely supported, and proven successful for this task. I am talking
about everything from hardware compatibility to software support for
pfsense and openbgpd from the community. Finally, performance
comparison to a present day Linux Kernel.

PS conntrack optimization for IPTables. Been there done that... I am
strong with Gentoo and am pretty sure can put together a rock solid
machine, just don't want to turn a blind eye though yeah?

N.

Re: The Making of a Router

[Seth Mattinen]

On 12/26/13, 11:28, Faisal Imtiaz wrote:

Ahh.. so you are the one who bought those two from Ebay !
I was watching, but got to them rather late.

If you are the one who got them.. you got a great deal.

These have Mikrotik ROS license with them, you can do BGP/ OSPF etc. with them
If you want to reload them with other OS, all you got to do is pull out the 
FlashCard, and install anything else you want.

The Core2 Duo Model with Mikrotik ROS ver 5 will handle about 1G to 1.5G of 
traffic with much trouble.
With ROS ver 6 .. they will do about 25-30% more.

The i5 & i7 versions will handle 3-5Gig of traffic easily... and these do 
support 10G SFP+ Intel nic's with ROS 6.x



Nope, that wasn't me. I got mine probably a year ago. I pulled the flash 
and tried Vyatta on it, but Vyatta turned out to be a buggy unpleasant 
experience. RouterOS had an IPv6 OSPF bug where it ignores some LSAs 
that made it a showstopper.


~Seth

Re: The Making of a Router

[Faisal Imtiaz]

Ahh.. so you are the one who bought those two from Ebay !
I was watching, but got to them rather late.

If you are the one who got them.. you got a great deal.

These have Mikrotik ROS license with them, you can do BGP/ OSPF etc. with them 
If you want to reload them with other OS, all you got to do is pull out the 
FlashCard, and install anything else you want.

The Core2 Duo Model with Mikrotik ROS ver 5 will handle about 1G to 1.5G of 
traffic with much trouble.
With ROS ver 6 .. they will do about 25-30% more.

The i5 & i7 versions will handle 3-5Gig of traffic easily... and these do 
support 10G SFP+ Intel nic's with ROS 6.x

Regards.

Faisal Imtiaz
Snappy Internet & Telecom
7266 SW 48 Street
Miami, FL 33155
Tel: 305 663 5518 x 232

Help-desk: (305)663-5518 Option 2 or Email: supp...@snappytelecom.net 

- Original Message -
> From: "Seth Mattinen" 
> To: nanog@nanog.org
> Sent: Thursday, December 26, 2013 2:18:14 PM
> Subject: Re: The Making of a Router
> 
> On 12/26/13, 8:46, Faisal Imtiaz wrote:
> > I am a believer of not having to re-invent the wheel...
> >
> > Having said that.. have you looked at 'purpose built appliances'  e.g.
> >
> > http://www.lannerinc.com/
> > http://us.axiomtek.com/
> >
> > If you are looking for a full router
> > Consider such as these...
> > http://www.linktechs.net/
> > http://www.maxxwave.com/
> >
> 
> 
> I bought two Maxxwaves with the Core2 Duo processor and it's an Axiomtek
> NA-822 inside. It's a nifty platform, I'm probably going to put BIRD on
> it and make some BGP RRs one of these fine days. I'm too OCD to suffer a
> "server" in the routing/switching bays as the only thing with ports on
> the back.
> 
> I wouldn't imagine trying to route 10 gig with it though, or sharing any
> functions (like VOIP) on it.
> 
> ~Seth
> 
> 

Re: The Making of a Router

[Seth Mattinen]

On 12/26/13, 8:46, Faisal Imtiaz wrote:

I am a believer of not having to re-invent the wheel...

Having said that.. have you looked at 'purpose built appliances'  e.g.

http://www.lannerinc.com/
http://us.axiomtek.com/

If you are looking for a full router
Consider such as these...
http://www.linktechs.net/
http://www.maxxwave.com/




I bought two Maxxwaves with the Core2 Duo processor and it's an Axiomtek 
NA-822 inside. It's a nifty platform, I'm probably going to put BIRD on 
it and make some BGP RRs one of these fine days. I'm too OCD to suffer a 
"server" in the routing/switching bays as the only thing with ports on 
the back.


I wouldn't imagine trying to route 10 gig with it though, or sharing any 
functions (like VOIP) on it.


~Seth

Re: The Making of a Router

[Seth Mattinen]

On 12/26/13, 9:24, Andrew D Kirch wrote:


If he can afford a 10G link... he should be buying real gear...  I mean,
look, I've got plenty of infrastructure horror stories, but lets not
cobble together our own 10gbit solutions, please?  At least get one of
the new microtik CCR's with a 10gig sfp+?  They're only a kilobuck... If
you can't afford that I suggest you can't afford to be an ISP.



Unless all the money is going into the 10 gig link.

~Seth

Re: The Making of a Router

[Eduardo Schoedler]

For router with Freebsd+BIRD/Quagga, I suggest BSDRP.http://bsdrp.net



2013/12/26 Ray Soucy 

> You can build using commodity hardware and get pretty good results.
>
> I've had really good luck with Supermicro whitebox hardware, and
> Intel-based network cards.  The "Hot Lava Systems" cards have a nice
> selection for a decent price if you're looking for SFP and SFP+ cards that
> use Intel chipsets.
>
> There might be some benefits in going with something like FreeBSD, but I
> find that Linux has a lot more eyeballs on it making it much easier to
> develop for, troubleshoot, and support.  There are a few options if you
> want to go the Linux route.
>
> Option 1: Roll your own OS.  This takes quite a bit of effort, but if you
> have the tallant to do it you can generally get exactly what you want.
>
> Option 2: Use an established distribution.
>
> Vyatta doesn't seem to be doing much with its FOSS release "Vyatta Core"
> anymore, but the community has forked the GPL parts into "VyOS".  I've been
> watching them pretty closely and helping out where I can; I think the
> project is going to win over a lot of people over the next few years.
>
> http://www.vyatta.org/
> http://www.vyos.net/
>
> The biggest point of failure I've experienced with Linux-based routers on
> whitebox hardware has been HDD failure.  Other than that, the 100+ units
> I've had deployed over the past 3+ years have been pretty much flawless.
>
> Thankfully, they currently run an in-memory OS, so a disk failure only
> affects logging.
>
> If you want to build your own OS, I'll shamelessly plug a side project of
> mine: RAMBOOT
>
> http://ramboot.org/
>
> RAMBOOT makes use of the Ubuntu Core rootfs, and a modified boot process
> (added into initramfs tools, so kernel updates generate the right kernel
> automatically).  Essentially, I use a kernel ramdisk instead of an HDD for
> the root filesystem and "/" is mounted on "/dev/ram1".
>
> The bootflash can be removed while the system is running as it's only
> mounted to save system configuration or update the OS.
>
> I haven't polished it up much, but there is enough there to get going
> pretty quickly.
>
> You'll also want to pay attention to the settings you use for the kernel.
>  Linux is tuned as a desktop or server, not a router, so there are some
> basics you should take care of (like disabling ICMP redirects, increasing
> the ARP table size, etc).
>
> I have some examples in: http://soucy.org/xorp/xorp-1.7-pre/TUNING
> or http://soucy.org/tmp/netfilter.txt (more recent, but includes firewall
> examples).
>
> Also a note of caution.  I would stick with a longterm release of Linux.
>  I've had good experience with 2.6.32, and 3.10.  I'm eager to use some of
> the post-3.10 features, though, so I'm anxious for the next longterm branch
> to be locked in.
>
> If running a proxy server of any kind, you'll want to adjust
> TCP_TIMEWAIT_LEN in the header file and re-compile the kernel, else you'll
> run into ephemeral port exhaustion before you touch the limits of the CPU.
>  I recommend 15 seconds (the default in Linux is 60).
>
> Routing-engine -wise.  I currently have a large XORP 1.6 deployment because
> I have a need for multicast routing (PIM-SM), but XORP is very touchy and
> takes quite a bit of operational experience to avoid problems.  Quagga has
> much more active development and eyeballs.  BIRD is also very interesting.
>  I like the model of BIRD a lot (more of a traditional daemon than trying
> to be a Cisco or Juniper clone).  It doesn't seem to be as far along as
> Quagga though.
>
> One of the biggest advantages is the low cost of hardware allows you to
> maintain spare systems, reducing the time to service restoration in the
> event of failure.  Dependability-wise, I feel that whitebox Linux systems
> are pretty much at Cisco levels these days, especially if running
> in-memory.
>
>
>
>
> On Thu, Dec 26, 2013 at 1:07 PM, jim deleskie  wrote:
>
> > I've recently pushed a "large" BSD box to a load of over 300, for more
> then
> > an hour, while under test,  some things slowed a little, but she kept on
> > working!
> >
> > -jim
> >
> >
> > On Thu, Dec 26, 2013 at 1:59 PM, Shawn Wilson 
> wrote:
> >
> > > Totally agree that a routing box should be standalone for tons of
> > reasons.
> > > Even separating network routing and call routing.
> > >
> > > It used to be that BSD's network stack was much better than Linux's
> under
> > > load. I'm not sure if this is still the case - I've never been put in
> the
> > > situation where the Linux kernel was at its limits. FWIW
> > >
> > > Jared Mauch  wrote:
> > > >Have to agree on the below. I've seen too many devices be so
> integrated
> > > >they do no task well, and can't be rebooted to troubleshoot due to
> > > >everyone using them.
> > > >
> > > >Jared Mauch
> > > >
> > > >> On Dec 26, 2013, at 10:55 AM, Andrew D Kirch 
> > > >wrote:
> > > >>
> > > >> Don't put all this in one box.
> > >
> > >
> > >
> >
>
>
>
> --
> Ray Patrick Soucy

Re: The Making of a Router

[Ray Soucy]

You can build using commodity hardware and get pretty good results.

I've had really good luck with Supermicro whitebox hardware, and
Intel-based network cards.  The "Hot Lava Systems" cards have a nice
selection for a decent price if you're looking for SFP and SFP+ cards that
use Intel chipsets.

There might be some benefits in going with something like FreeBSD, but I
find that Linux has a lot more eyeballs on it making it much easier to
develop for, troubleshoot, and support.  There are a few options if you
want to go the Linux route.

Option 1: Roll your own OS.  This takes quite a bit of effort, but if you
have the tallant to do it you can generally get exactly what you want.

Option 2: Use an established distribution.

Vyatta doesn't seem to be doing much with its FOSS release "Vyatta Core"
anymore, but the community has forked the GPL parts into "VyOS".  I've been
watching them pretty closely and helping out where I can; I think the
project is going to win over a lot of people over the next few years.

http://www.vyatta.org/
http://www.vyos.net/

The biggest point of failure I've experienced with Linux-based routers on
whitebox hardware has been HDD failure.  Other than that, the 100+ units
I've had deployed over the past 3+ years have been pretty much flawless.

Thankfully, they currently run an in-memory OS, so a disk failure only
affects logging.

If you want to build your own OS, I'll shamelessly plug a side project of
mine: RAMBOOT

http://ramboot.org/

RAMBOOT makes use of the Ubuntu Core rootfs, and a modified boot process
(added into initramfs tools, so kernel updates generate the right kernel
automatically).  Essentially, I use a kernel ramdisk instead of an HDD for
the root filesystem and "/" is mounted on "/dev/ram1".

The bootflash can be removed while the system is running as it's only
mounted to save system configuration or update the OS.

I haven't polished it up much, but there is enough there to get going
pretty quickly.

You'll also want to pay attention to the settings you use for the kernel.
 Linux is tuned as a desktop or server, not a router, so there are some
basics you should take care of (like disabling ICMP redirects, increasing
the ARP table size, etc).

I have some examples in: http://soucy.org/xorp/xorp-1.7-pre/TUNING
or http://soucy.org/tmp/netfilter.txt (more recent, but includes firewall
examples).

Also a note of caution.  I would stick with a longterm release of Linux.
 I've had good experience with 2.6.32, and 3.10.  I'm eager to use some of
the post-3.10 features, though, so I'm anxious for the next longterm branch
to be locked in.

If running a proxy server of any kind, you'll want to adjust
TCP_TIMEWAIT_LEN in the header file and re-compile the kernel, else you'll
run into ephemeral port exhaustion before you touch the limits of the CPU.
 I recommend 15 seconds (the default in Linux is 60).

Routing-engine -wise.  I currently have a large XORP 1.6 deployment because
I have a need for multicast routing (PIM-SM), but XORP is very touchy and
takes quite a bit of operational experience to avoid problems.  Quagga has
much more active development and eyeballs.  BIRD is also very interesting.
 I like the model of BIRD a lot (more of a traditional daemon than trying
to be a Cisco or Juniper clone).  It doesn't seem to be as far along as
Quagga though.

One of the biggest advantages is the low cost of hardware allows you to
maintain spare systems, reducing the time to service restoration in the
event of failure.  Dependability-wise, I feel that whitebox Linux systems
are pretty much at Cisco levels these days, especially if running in-memory.




On Thu, Dec 26, 2013 at 1:07 PM, jim deleskie  wrote:

> I've recently pushed a "large" BSD box to a load of over 300, for more then
> an hour, while under test,  some things slowed a little, but she kept on
> working!
>
> -jim
>
>
> On Thu, Dec 26, 2013 at 1:59 PM, Shawn Wilson  wrote:
>
> > Totally agree that a routing box should be standalone for tons of
> reasons.
> > Even separating network routing and call routing.
> >
> > It used to be that BSD's network stack was much better than Linux's under
> > load. I'm not sure if this is still the case - I've never been put in the
> > situation where the Linux kernel was at its limits. FWIW
> >
> > Jared Mauch  wrote:
> > >Have to agree on the below. I've seen too many devices be so integrated
> > >they do no task well, and can't be rebooted to troubleshoot due to
> > >everyone using them.
> > >
> > >Jared Mauch
> > >
> > >> On Dec 26, 2013, at 10:55 AM, Andrew D Kirch 
> > >wrote:
> > >>
> > >> Don't put all this in one box.
> >
> >
> >
>



-- 
Ray Patrick Soucy
Network Engineer
University of Maine System

T: 207-561-3526
F: 207-561-3531

MaineREN, Maine's Research and Education Network
www.maineren.net

Re: The Making of a Router

[jim deleskie]

I've recently pushed a "large" BSD box to a load of over 300, for more then
an hour, while under test,  some things slowed a little, but she kept on
working!

-jim


On Thu, Dec 26, 2013 at 1:59 PM, Shawn Wilson  wrote:

> Totally agree that a routing box should be standalone for tons of reasons.
> Even separating network routing and call routing.
>
> It used to be that BSD's network stack was much better than Linux's under
> load. I'm not sure if this is still the case - I've never been put in the
> situation where the Linux kernel was at its limits. FWIW
>
> Jared Mauch  wrote:
> >Have to agree on the below. I've seen too many devices be so integrated
> >they do no task well, and can't be rebooted to troubleshoot due to
> >everyone using them.
> >
> >Jared Mauch
> >
> >> On Dec 26, 2013, at 10:55 AM, Andrew D Kirch 
> >wrote:
> >>
> >> Don't put all this in one box.
>
>
>

Re: The Making of a Router

[Shawn Wilson]

Totally agree that a routing box should be standalone for tons of reasons. Even 
separating network routing and call routing.

It used to be that BSD's network stack was much better than Linux's under load. 
I'm not sure if this is still the case - I've never been put in the situation 
where the Linux kernel was at its limits. FWIW

Jared Mauch  wrote:
>Have to agree on the below. I've seen too many devices be so integrated
>they do no task well, and can't be rebooted to troubleshoot due to
>everyone using them. 
>
>Jared Mauch
>
>> On Dec 26, 2013, at 10:55 AM, Andrew D Kirch 
>wrote:
>> 
>> Don't put all this in one box.

Re: The Making of a Router

[Andrew D Kirch]

On 12/26/2013 12:05 PM, Alessandro Ratti wrote:

  (maybe with 10G nics).


If he can afford a 10G link... he should be buying real gear...  I mean, 
look, I've got plenty of infrastructure horror stories, but lets not 
cobble together our own 10gbit solutions, please?  At least get one of 
the new microtik CCR's with a 10gig sfp+?  They're only a kilobuck... If 
you can't afford that I suggest you can't afford to be an ISP.


Andrew

Re: The Making of a Router

[Warren Bailey]

Not to mention the fact that this "router" will require support. The build 
before buy people are silly. Let the smart router guys do their thing and use 
their box accordingly. When it breaks call to inform them it broke and they 
will fix it. Diy projects are a nightmare to support.


Sent from my Mobile Device.


 Original message 
From: Eric Clark 
Date: 12/26/2013 8:00 AM (GMT-09:00)
To: Faisal Imtiaz 
Cc: nanog@nanog.org
Subject: Re: The Making of a Router


I also wonder about re-inventing the wheel. The router part is easy, you could 
even do that with a windows box (that's a joke).

Obviously capital cost is part of it, but the man hours involved in doing what 
you're talking about, especially since you are talking about a telco 
whatever you come up with has to be pretty darn reliable...

Certainly would be interested in a little more information about the use case.


Eric

On Dec 26, 2013, at 8:46 AM, Faisal Imtiaz  wrote:

> I am a believer of not having to re-invent the wheel...
>
> Having said that.. have you looked at 'purpose built appliances'  e.g.
>
> http://www.lannerinc.com/
> http://us.axiomtek.com/
>
> If you are looking for a full router
> Consider such as these...
>   http://www.linktechs.net/
>   http://www.maxxwave.com/
>
> and there are a few others but the concept is the same
>
> Personally, I am not a believer in making a single device be the do all / end 
> all of everything..
> While one can do everything on a big server .. however breaking things out 
> e.g. voip trans-coding and routing make maintenance, availability, and 
> ability to create redundancy much more practical.
>
>
> Regards
>
> Faisal Imtiaz
> Snappy Internet & Telecom
>
>
> - Original Message -
>> From: "Nick Cameo" 
>> To: nanog@nanog.org
>> Sent: Thursday, December 26, 2013 11:33:13 AM
>> Subject: The Making of a Router
>>
>> Hello Everyone,
>>
>> We are looking to put together a 2u server with a few PCIe 3 x8
>> (recommendations appreciated). The router will take a voip transcoding
>> line card, and will act as an edge router for a telecom company.
>>
>> For things like BGP (Quagga, Zebra, all that lovely stuff!!!), static
>> routes, and firewall capabilities we are thinking gentoo linux
>> stripped for sure however, what about the BSDs? FreeBSD or OpenBSD.
>> Any comments, feedback, does, and don'ts are much appreciated.
>>
>> Kind Regards,
>>
>> Nick.
>>
>>
>

Re: The Making of a Router

[Alessandro Ratti]

if you want build by yourself I will suggest gentoo and/or freebsd with
bird (http://bird.network.cz/) for routing stuff (maybe with 10G nics).

Don't put all this in one box.


+1


On Thu, Dec 26, 2013 at 5:33 PM, Nick Cameo  wrote:

> Hello Everyone,
>
> We are looking to put together a 2u server with a few PCIe 3 x8
> (recommendations appreciated). The router will take a voip transcoding
> line card, and will act as an edge router for a telecom company.
>
> For things like BGP (Quagga, Zebra, all that lovely stuff!!!), static
> routes, and firewall capabilities we are thinking gentoo linux
> stripped for sure however, what about the BSDs? FreeBSD or OpenBSD.
> Any comments, feedback, does, and don'ts are much appreciated.
>
> Kind Regards,
>
> Nick.
>
>


-- 
Ciao,

Alessandro

Re: The Making of a Router

[Jared Mauch]

Have to agree on the below. I've seen too many devices be so integrated they do 
no task well, and can't be rebooted to troubleshoot due to everyone using them. 

Jared Mauch

> On Dec 26, 2013, at 10:55 AM, Andrew D Kirch  wrote:
> 
> Don't put all this in one box.

Re: The Making of a Router

[Jared Mauch]

Look at the ubnt edgemax devices

Jared Mauch

> On Dec 26, 2013, at 10:46 AM, Faisal Imtiaz  wrote:
> 
> I am a believer of not having to re-invent the wheel...
> 
> Having said that.. have you looked at 'purpose built appliances'  e.g. 
> 
> http://www.lannerinc.com/
> http://us.axiomtek.com/
> 
> If you are looking for a full router
> Consider such as these...
>   http://www.linktechs.net/
>   http://www.maxxwave.com/
> 
> and there are a few others but the concept is the same
> 
> Personally, I am not a believer in making a single device be the do all / end 
> all of everything..
> While one can do everything on a big server .. however breaking things out 
> e.g. voip trans-coding and routing make maintenance, availability, and 
> ability to create redundancy much more practical.
> 
> 
> Regards 
> 
> Faisal Imtiaz
> Snappy Internet & Telecom
> 
> 
> - Original Message -
>> From: "Nick Cameo" 
>> To: nanog@nanog.org
>> Sent: Thursday, December 26, 2013 11:33:13 AM
>> Subject: The Making of a Router
>> 
>> Hello Everyone,
>> 
>> We are looking to put together a 2u server with a few PCIe 3 x8
>> (recommendations appreciated). The router will take a voip transcoding
>> line card, and will act as an edge router for a telecom company.
>> 
>> For things like BGP (Quagga, Zebra, all that lovely stuff!!!), static
>> routes, and firewall capabilities we are thinking gentoo linux
>> stripped for sure however, what about the BSDs? FreeBSD or OpenBSD.
>> Any comments, feedback, does, and don'ts are much appreciated.
>> 
>> Kind Regards,
>> 
>> Nick.
>> 
>> 

Re: The Making of a Router

[Eric Clark]

I also wonder about re-inventing the wheel. The router part is easy, you could 
even do that with a windows box (that's a joke).

Obviously capital cost is part of it, but the man hours involved in doing what 
you're talking about, especially since you are talking about a telco 
whatever you come up with has to be pretty darn reliable...

Certainly would be interested in a little more information about the use case.


Eric

On Dec 26, 2013, at 8:46 AM, Faisal Imtiaz  wrote:

> I am a believer of not having to re-invent the wheel...
> 
> Having said that.. have you looked at 'purpose built appliances'  e.g. 
> 
> http://www.lannerinc.com/
> http://us.axiomtek.com/
> 
> If you are looking for a full router
> Consider such as these...
>   http://www.linktechs.net/
>   http://www.maxxwave.com/
> 
> and there are a few others but the concept is the same
> 
> Personally, I am not a believer in making a single device be the do all / end 
> all of everything..
> While one can do everything on a big server .. however breaking things out 
> e.g. voip trans-coding and routing make maintenance, availability, and 
> ability to create redundancy much more practical.
> 
> 
> Regards 
> 
> Faisal Imtiaz
> Snappy Internet & Telecom
> 
> 
> - Original Message -
>> From: "Nick Cameo" 
>> To: nanog@nanog.org
>> Sent: Thursday, December 26, 2013 11:33:13 AM
>> Subject: The Making of a Router
>> 
>> Hello Everyone,
>> 
>> We are looking to put together a 2u server with a few PCIe 3 x8
>> (recommendations appreciated). The router will take a voip transcoding
>> line card, and will act as an edge router for a telecom company.
>> 
>> For things like BGP (Quagga, Zebra, all that lovely stuff!!!), static
>> routes, and firewall capabilities we are thinking gentoo linux
>> stripped for sure however, what about the BSDs? FreeBSD or OpenBSD.
>> Any comments, feedback, does, and don'ts are much appreciated.
>> 
>> Kind Regards,
>> 
>> Nick.
>> 
>> 
> 

Re: The Making of a Router

[Andrew D Kirch]

If you're trying to do this cheaply, I'd recommend an appropriate sized 
Mikrotik router, and perhaps something running digium's transcoding 
hardware/Asterisk, or some Adtran hardware.


Don't put all this in one box.

Andrew

On 12/26/2013 11:33 AM, Nick Cameo wrote:

Hello Everyone,

We are looking to put together a 2u server with a few PCIe 3 x8
(recommendations appreciated). The router will take a voip transcoding
line card, and will act as an edge router for a telecom company.

For things like BGP (Quagga, Zebra, all that lovely stuff!!!), static
routes, and firewall capabilities we are thinking gentoo linux
stripped for sure however, what about the BSDs? FreeBSD or OpenBSD.
Any comments, feedback, does, and don'ts are much appreciated.

Kind Regards,

Nick.

Re: The Making of a Router

[Faisal Imtiaz]

I am a believer of not having to re-invent the wheel...

Having said that.. have you looked at 'purpose built appliances'  e.g. 

http://www.lannerinc.com/
http://us.axiomtek.com/

If you are looking for a full router
Consider such as these...
   http://www.linktechs.net/
   http://www.maxxwave.com/

and there are a few others but the concept is the same

Personally, I am not a believer in making a single device be the do all / end 
all of everything..
While one can do everything on a big server .. however breaking things out e.g. 
voip trans-coding and routing make maintenance, availability, and ability to 
create redundancy much more practical.
 

Regards 

Faisal Imtiaz
Snappy Internet & Telecom


- Original Message -
> From: "Nick Cameo" 
> To: nanog@nanog.org
> Sent: Thursday, December 26, 2013 11:33:13 AM
> Subject: The Making of a Router
> 
> Hello Everyone,
> 
> We are looking to put together a 2u server with a few PCIe 3 x8
> (recommendations appreciated). The router will take a voip transcoding
> line card, and will act as an edge router for a telecom company.
> 
> For things like BGP (Quagga, Zebra, all that lovely stuff!!!), static
> routes, and firewall capabilities we are thinking gentoo linux
> stripped for sure however, what about the BSDs? FreeBSD or OpenBSD.
> Any comments, feedback, does, and don'ts are much appreciated.
> 
> Kind Regards,
> 
> Nick.
> 
> 

The Making of a Router

[Nick Cameo]

Hello Everyone,

We are looking to put together a 2u server with a few PCIe 3 x8
(recommendations appreciated). The router will take a voip transcoding
line card, and will act as an edge router for a telecom company.

For things like BGP (Quagga, Zebra, all that lovely stuff!!!), static
routes, and firewall capabilities we are thinking gentoo linux
stripped for sure however, what about the BSDs? FreeBSD or OpenBSD.
Any comments, feedback, does, and don'ts are much appreciated.

Kind Regards,

Nick.