Re: AW: AW: AW: /27 the new /24

[Mike Hammett]

I know of literally hundreds of ISPs using them in the US and I'm sure that 
number is in the thousands. After hearing complaints from larger networks of 
their larger gear... it's the same shit everyone else deals with. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


- Original Message -

From: "Jérôme Nicolle"  
To: nanog@nanog.org 
Sent: Friday, October 9, 2015 7:10:31 AM 
Subject: Re: AW: AW: AW: /27 the new /24 

Hello James, 

Le 05/10/2015 06:45, James Jun a écrit : 
> I'm not aware of any carrier-grade network that operates on these things. 

With the availability of a 80Gbps model and upcoming updates to the 
routing process (in RouterOS 7), chances are these boxes will drag much 
more attention in the next 2-3 years. 

Still, it looks like their products are widely used in developping 
countries where cheap hardware and flexible / low power requirements 
(you'd run a CCR1009 off a car battery for a solid 2 weeks - no 
regulator needed) makes them the only viable choice. 

I know of at least a dozen ISP running these as well, here in western 
Europe. It solved many space and power issues in dense carrier hotels, 
and is a cheap and efficient way out of a 6500/7600 (in sub 20Gbps 
scenarios) based network. 

I wouldn't sell transit or provide criticial services with a 
mikrotik-based network just yet, mostly for the lack of enough personnal 
confidence and experience with them, but havin endured nights of 
debugging with poor quality code in recent major player's routers, I 
doubt they're as misfits as you suggest. 

Best regards, 

-- 
Jérôme Nicolle 

Re: Mikrotik in the DFZ (Was Re: AW: AW: /27 the new /24)

[Jérôme Nicolle]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hello William,

Le 03/10/2015 10:23, William Waites a écrit :
> I wish it were possible today to run different software on their 
> larger boxes. If some like-minded small providers wanted to get 
> together with us to fund a FreeBSD port to the CCR routers that
> would be great. Please contact me off-list if you are interested in
> this, I'll coordinate.

One of my contacts has worked on a similar path, and we encountered
many issues that makes it quite difficult.

Here is what I gathered, while I never had no access to the
NDA-covered material.

The Tilera architecture and its SDK is a great way to start such
project but Mikrotik didn't just use an off-the-shelf chip as
recommended, they also made slight changes to how the network
interfaces operates, and didn't provide any documentation.

It's not as easy as swapping a driver and rebuild a kernel, more like
changing how the programmable logic in Tilera's interface blocks
dispatch frames among the core's interconnexion grid.

Also, the cores are not fully compliant with MIPS specifications,
aren't combined as an SMP assembly at all (rather a NUMA grid with
added glue logic) and you can't even load the first instruction at 0x0
without using Tilera's own proprietary init code to allocate
ressources, initialize cores and setup the multiple "containers" (kind
of hardware virtualization).

So it's not quite about porting an OS than it has to do with tight
coupling of proprietary control code, bare-metal and FPGA logic, and a
specific data-plane implementation.

Still, the attempts have gone as far as booting a cutom linux kernel
spanned among a single CPU instance made of all 36 cores, but it has
no access to the network interfaces on the mikrotik board.

It does, however, work flawlessly on Tilera's developpment boards and
appliances, though neither the Linux kernel's data plane or
DPDK-derived code are yet able to take advantage of the specificities
of Tilera's architecture.

Nevertheless, if you want do get deeper and have enough motivation to
get past the technical difficulties, I'd gladly try to help into
bringing an alternative OS to these box.

Best regards,

- -- 
Jérôme Nicolle
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2
Comment: GPGTools - https://gpgtools.org

iEYEARECAAYFAlYXtDUACgkQbt+nwQamihu3eACdFRkX/yXrEJHJHm9F7HD0ClV4
2ikAnjy6a7KRheMlKTfFRaccfuYQInfc
=qRKm
-END PGP SIGNATURE-

Re: AW: AW: AW: /27 the new /24

[Jérôme Nicolle]

Hello James,

Le 05/10/2015 06:45, James Jun a écrit :
> I'm not aware of any carrier-grade network that operates on these things.

With the availability of a 80Gbps model and upcoming updates to the
routing process (in RouterOS 7), chances are these boxes will drag much
more attention in the next 2-3 years.

Still, it looks like their products are widely used in developping
countries where cheap hardware and flexible / low power requirements
(you'd run a CCR1009 off a car battery for a solid 2 weeks - no
regulator needed) makes them the only viable choice.

I know of at least a dozen ISP running these as well, here in western
Europe. It solved many space and power issues in dense carrier hotels,
and is a cheap and efficient way out of a 6500/7600 (in sub 20Gbps
scenarios) based network.

I wouldn't sell transit or provide criticial services with a
mikrotik-based network just yet, mostly for the lack of enough personnal
confidence and experience with them, but havin endured nights of
debugging with poor quality code in recent major player's routers, I
doubt they're as misfits as you suggest.

Best regards,

-- 
Jérôme Nicolle

Re: AW: AW: AW: /27 the new /24

[James Jun]

On Sat, Oct 03, 2015 at 08:10:36AM -0500, Mike Hammett wrote:
> 
> People keep thinking I want Level 3 to replace a loaded 6500 with a CCR and 
> that's simply not what I'm saying at all. The point of rattling off the 
> newer\smaller hardware was to say that if the site doesn't require 40G\100G, 
> doesn't have the revenue to support an MX480, etc. you should put in a 
> smaller\cheaper box. 
> Cost is a non-issue at that point because the smaller gear that's all you 
> need will have far less operational cost. Someone thought a particular POP 
> was going to be a big hit... and wasn't. 

In an SP environment, there is an escalating operating cost and network 
complexity to having small full-featured routers (ie. MX80, ASR9001, CER2k, 
etc) at every data center, POP or anywhere you need to terminate customers.  
The reality is that small routers (even if you were to use ghetto routers) have 
poor economics in port density.  It's feasible for a startup ISP to spam MX80 
or equivalent anytime they need more ports, but there comes a point where 
plopping a big chassis is the way to go.

At my place, we started with MX80s to cheap out on router ports anytime we had 
to light a data center.  That only got us far and we ended up having to migrate 
to ASR 9010s and start phasing out small routers.  The increasing complexity of 
having dozens of small routers and managing LSP mesh to remainder of the 
network is ugly.  Moreover, full-table BGP routers are also the places where 
you exercise edge policy with complex routing policies.  Even with automation, 
managing dozens of those in a region that could have been served by only 2 
routers is annoying.  It's easier to haul IP customers to fewer, but more 
reliable large-chassis platforms and use packet-optical network to get to the 
customer premise.

Between the above and the lack of control-plane redundancy on most small 
routers, there are operational complexities & economic realities to keep in 
mind; it's not strictly about whether a site requires 40G/100G.  


> On the flip side, if there are 200 ports of customers chances are you need 
> the big interfaces that aren't on the old boxes. You have the bigger revenue. 
> Heck, the new big boxes probably still use less power than the old big boxes 
> anyway. 

The idea has its merits, however in practice, it isn't feasible.  People don't 
put in line cards into their router with expectation that they need to be 
replaced 2 years down the road because FIB TCAM ran out.  Even if you have the 
revenue to justify new line cards, constant migration of customer interfaces 
means disruptive maintenance for that customer.  We'd prefer IP network to be 
as reliable as dial-tone, if possible.

The global routing table is approaching 600k today.  Lot of line cards in 
installed base today only handle ~1.0/1.3 to ~1.8 million IPv4.  When you start 
replacing those line cards (and mind you, a 24x10GE line card has a list price 
running into $300k range), the next logical level is 4 million IPv4.  With all 
the deaggs with /24s, just how long of time are we going to have with /27 
explosion before 4 million FIB runs out of space?

I can see /25 being contemplated, but the cost of moving to /27 just isn't 
worth it at the moment.


> 
> What I learned from this thread: Once you mention MT\UBNT routers, people 
> assume you're using a MT\UBNT hammer everywhere. 

I'm not aware of any carrier-grade network that operates on these things.


Best,
james

Re: AW: /27 the new /24

[Randy Bush]

> It's not too far off though. One way of looking at it is, for each
> extra bit we allow, we potentially double the table size.

that is math.  in reality

table size is proportional to

multihoming + traffic engineering

randy

Re: Mikrotik in the DFZ (Was Re: AW: AW: /27 the new /24)

[Mike Hammett]

Sure MT has issues, but so does everyone. As someone that has used them for 10+ 
years, the past six months has seen a bit of a re-awakening over there. You can 
see this in the time to completion of many feature requests, bug fixes, new 
features, etc. I'm not sure they're going to do everything everyone is after, 
but they certainly have shown a huge increase in willingness to go the right 
direction. 

Of course it's easy for someone running big iron to scoff at the lack of 
feature X or feature Y. To that I say, what are the capabilities of your $200 
router? Your $2k router? I haven't priced out new low-end gear from the big 
iron vendors, but I can't imagine at what price point you need to be at to have 
a multi-gig capable VPLS router. For Mikrotik you're in the $200 - $1k range, 
depending on what you mean by "multi-gig". One thing I miss as I start to use 
more non-Mikrotik hardware... Torch. I wish everything had Torch. Put Packet 
Sniffer in the list of things I'd like to see everywhere. I don't want port 
mirror as who's to say I have something to mirror to everywhere that can also 
capture? Put a few basic filters and drop the PCAP right on the damn box. Now 
obviously with something running BSD you could code up whatever you'd like or 
have an array of open-source packages to work with , but that wouldn't have the 
nice feature integration of a router OS. 

I have no problem running Mikrotik in the DFZ. Mine pull down full tables in 30 
- 35 seconds, can handle somewhere in the 30 - 60 gb range when firewall rules 
are applied and so on. They'd cost under $1,500 new, but I got mine put 
together for a fraction of that. They're so cheap you can run two. Run two and 
now you have the advantage of being able to do maintenance without downtime. 
It's a little kludgey, but can get get the job done at a price point the others 
can't. Maybe with newer CCRs and ROS7 I could drop the need for the x86 boxes. 
We'll see. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


- Original Message -

From: "William Waites"  
To: j...@anexia.at 
Cc: na...@ics-il.net, nanog@nanog.org 
Sent: Saturday, October 3, 2015 3:23:49 AM 
Subject: Mikrotik in the DFZ (Was Re: AW: AW: /27 the new /24) 

On Fri, 2 Oct 2015 23:11:47 +, Jürgen Jaritsch  said: 

> Regarding the words "I have a small router which handles 
> multiple full tables ...": push and pull a few full tables at 
> the same time and you'll see what's happening: the CCRs are 
> SLOW. And why? Because the software is not as good as it could 
> be: the BGP daemon uses only one core of a 36(?) core CPU. 

To expand on this, the problem is worse than being single-threaded. I 
had one of these in the lab and fed it 2x full tables. Sure it wasn't 
the fastest at accepting them but then I noticed that even in steady 
state one of the CPUs was pegged. What was happening -- and this was 
confirmed by Mikrotik -- was that it was recalculating the *entire* 
FIB for each update. The general background noise of announce / 
withdraw messages means it is doing this all the time. Any churn and 
it would have a very hard time. 

There are other serious bugs such as not doing recursive next hop 
lookup for IPv6 (it does for IPv4). This makes them unuseable as BGP 
routers even for partial tables with most non-trivial iBGP 
topologies. All of which may be fixed one day in version 7 of their 
operating system, which will inevitably have many bugs as any software 
project .0 release will, so we'll have to wait for 7.x for it to be 
reasonably safe to use. 

That said, we use a lot of Mikrotik kit for our rural 
networks. They're weird and quirky but you can't beat them on price, 
port density and power consumption. With 16 ports and 36 cores surely 
they should be capable of pushing several Gbps of traffic with a few 
full tables. 

I wish it were possible today to run different software on their 
larger boxes. If some like-minded small providers wanted to get 
together with us to fund a FreeBSD port to the CCR routers that would 
be great. Please contact me off-list if you are interested in this, 
I'll coordinate. 

As it is we don't let them anywhere near the DFZ, that's done with PCs 
running FreeBSD and BIRD which can easily do the job but is still an 
order of magnitude more expensive (and an order of magnitude less 
expensive than what you need if you want 10s of Gbps). 

-w 

-- 
William Waites  | School of Informatics 
http://tardis.ed.ac.uk/~wwaites/ | University of Edinburgh 
https://hubs.net.uk/ | HUBS AS60241 

The University of Edinburgh is a charitable body, registered in 
Scotland, with registration number SC005336. 

Re: AW: /27 the new /24

[Baldur Norddahl]

Except we might very well reach 1+ million routes soon without accepting
longer prefixes than /24. Also route updates is a concern - do I really
need to be informed every time someone on the other end of the world resets
a link?

On 3 October 2015 at 12:57, William Waites  wrote:

> On Sat, 3 Oct 2015 12:42:01 +0200, Baldur Norddahl <
> baldur.nordd...@gmail.com> said:
>
> > 2 million routes will not be enough if we go full /27. This is
> > not a scalable solution. Something else is needed to provide
> > multihoming for small networks (LISP?).
>
> It's not too far off though. One way of looking at it is, for each
> extra bit we allow, we potentially double the table size. So with 500k
> routes and a /24 limit now, we might expect 4 million with /27. Not
> exactly because it depends strongly on the distribution of prefix
> lengths, but probably not a bad guess.
>
> Also there are optimisations that I wonder if the vendors are doing to
> preserve TCAM such as aggregating adjacent networks with the same next
> hop into the supernet. That would mitigate the impact of wanton
> deaggregation at least and the algorithm doesn't look too hard. Do the
> big iron vendors do this?
>
> -w
>
> --
> William Waites   |  School of Informatics
>http://tardis.ed.ac.uk/~wwaites/   | University of Edinburgh
>  https://hubs.net.uk/ |  HUBS AS60241
>
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>

Re: AW: AW: AW: /27 the new /24

[Mike Hammett]

I don't think we are talking different things, though I think we are talking in 
circles and thus the thread probably needs to die. 


People keep thinking I want Level 3 to replace a loaded 6500 with a CCR and 
that's simply not what I'm saying at all. The point of rattling off the 
newer\smaller hardware was to say that if the site doesn't require 40G\100G, 
doesn't have the revenue to support an MX480, etc. you should put in a 
smaller\cheaper box. Cost is a non-issue at that point because the smaller gear 
that's all you need will have far less operational cost. Someone thought a 
particular POP was going to be a big hit... and wasn't. On the flip side, if 
there are 200 ports of customers chances are you need the big interfaces that 
aren't on the old boxes. You have the bigger revenue. Heck, the new big boxes 
probably still use less power than the old big boxes anyway. 



What I learned from this thread: Once you mention MT\UBNT routers, people 
assume you're using a MT\UBNT hammer everywhere. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


- Original Message -

From: "Jürgen Jaritsch"  
To: "Mike Hammett"  
Cc: "NANOG"  
Sent: Saturday, October 3, 2015 6:06:59 AM 
Subject: AW: AW: AW: /27 the new /24 

Hi Mike, 

> but the boxes that have been there for 10 years have more than paid for 
> themselves (unless they're a shitty business). 

No question about that! But why should they throw them away if they can still 
print $$$ with these boxes? They have to change nothing till the global routing 
table reaches at least 768k ... so let's say this will happen in 12-18 months. 
They have enough time to prepare, migrate, etc ... and while all the side 
stories are happening they are still able to print $$$ with the "old shit". 

> What I was saying is that my little business with meager means (and revenues) 
> can afford a box to do it. 

This is definitely a question about sizing. Replacing a box with ~200 connected 
customers (only at this box!) is way more complex and this is nothing 
unrealistic. 

> If their business hasn't boomed, maybe it's time to replace that old 6500 
> with a 4500x or a QFX-5100 or an x670 or whatever. 

4500x => no MPLS features 

QFX-5100 => very nice box (I'm a big fan) but complicate (and expensive!) 
licensing. 

Extreme x670 => nice box too - we also use this. But it's simply too small and 
the BGP configuration on these boxes is horrible. It's also not possible to 
provide Ethernet over MPLS with LACP BPDU forwarding ... too less features. 
Nice for aggregation and POP interconnect. 

All three models are new and shiny but they can't replace a 6500/7600. Too less 
port density and too less features (people are still using SDH. You need SDH in 
an 6500/7600? Simply install the required line card ...). If you really plan to 
replace a 6509 or even a 6513 you have to go with something like Juniper 
MX480/960 (I'm in love ... :D) or Cisco Nexus 7k/9k. 

One thing that will more and more happen: physical separation. There will be 
boxes with 10G/40G/100G only and boxes with 100M/1G only. Why? It's easier for 
vendors to remove old compatibility requirements (like electrical interfaces). 
So what we did in the past 3 years (replacing old boxes with new boxes with 
1G/10G interfaces) was useless - we'll get our "old shit" back in place and 
bring them up and running. Of course: the "old shit" will be reduced to do 
aggregation layer or to something like "multihop instance" to transport the 
customers access port to the "real big and powerful router". Solving this with 
Layer2 extensions (like VLANs) is not practicable because you'll ran into other 
problems (like STP instances, etc). Probably it makes sense to solve it with 
Layer2VPN (Ethernet over MPLS, etc) to transport the physical interface to a 
virtual interface. 

Lots of things to think about :(. 


> Your decreased power bill alone will pay it off. If it has boomed, then ten 
> years of revenues should get you whatever the bigger Ciscos are or an MX or 
> whatever the bigger Extremes are. 

Power is no argument. You get power starting at 0,10 Eur /kWh. Another 0,10 Eur 
/ kWh for cooling and we talk about 0,20 Eur / kWh => Cisco 6513 (configured 
with 11 line cards + 2x SUP) with 2x 6kW PSU uses 3,8kW. 3,8kW * 24 hours * 30 
days = 2.736 kWh per month. 2.736 * 0,20 Eur = 547,2 Eur per month for power 
consumption + cooling. If you have a good sales engineer you earn the revenue 
for this "side cost" with 1 customer :). Realistic calculation is: 10 customers 
are required to earn the money for the footprint. 


> Don't whine about my choices in gear I mentioned. I was just thr

AW: AW: AW: /27 the new /24

[Jürgen Jaritsch]

rk & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300
Telefax: +43-5-0556-500

E-Mail: jjarit...@anexia-it.com 
Web: http://www.anexia-it.com 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

-Ursprüngliche Nachricht-
Von: NANOG [mailto:nanog-boun...@nanog.org] Im Auftrag von Mike Hammett
Gesendet: Samstag, 03. Oktober 2015 02:52
Cc: NANOG 
Betreff: Re: AW: AW: /27 the new /24

I don't expect carriers to be running UBNT\Mikrotik, but the boxes that have 
been there for 10 years have more than paid for themselves (unless they're a 
shitty business). It's time to rip and replace with whatever is appropriate for 
that site. No, I obviously don't think I'm going to change anyone's opinion on 
the matter (at least not anyone that matters in one of these networks). What I 
was saying is that my little business with meager means (and revenues) can 
afford a box to do it. They can too. 



I don't doubt their situation sucks... but either you fix it or you don't. Time 
and the rest of the Internet won't wait for them. 


If their business hasn't boomed, maybe it's time to replace that old 6500 with 
a 4500x or a QFX-5100 or an x670 or whatever. Your decreased power bill alone 
will pay it off. If it has boomed, then ten years of revenues should get you 
whatever the bigger Ciscos are or an MX or whatever the bigger Extremes are. 

Don't whine about my choices in gear I mentioned. I was just throwing things 
out there. Old big, new small if no money or old big new big if money. 


BTW: ROS 7 won't have multi-threaded BGP, but will be optimized to handle full 
table imports in a significantly reduced time. Oh, and I'm not sure that you 
couldn't do at least three nines with MT\UBNT. Well, no experience with the 
EdgeRouters yet. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


- Original Message -

From: "Jürgen Jaritsch"  
To: "Mike Hammett"  
Cc: "NANOG"  
Sent: Friday, October 2, 2015 6:11:47 PM 
Subject: AW: AW: /27 the new /24 

Hi Mike, 

sorry, this was probably sent to quick ... let me please explain my POV of your 
statement: 

I want to concentrate my detailed answer only to the backbone situation which 
is often handled by the 6500/7600 - I guess all of us know that the 6500/7600 
has a ton of additional features ... 


6-7 years in the past carriers (and/or big ISPs) had only n*1G backbone 
capacities built with platforms that only had n*100M interfaces another 3-5 
years before. Their only invest in these 3-5 years was to add the Gig line 
cards, install some software updates and add new fibre optics (GBICs). Chassis, 
cabling, management interfaces etc could be remain in the cabinet - they only 
had to replace ONE line card (let's say for a few thousand bucks) and with this 
invest they were able to scale up their capacities. Of course: at some point 
they also had to replace the SUPs, PSUs, FANs, etc. But the invest in the 
surrounding stuff is nothing compared with completely new machines. 

So what all these companies did was buying a machine with an basic 
configuration and since 10(!) years they are able to expand this machines with 
(more or less) small and cheap upgrades. 

In backbone situations the 6500/7600 are definitely at the end of the resources 
the platform can provide. Most of the carriers (and of course also the bigger 
ISPs) had a real chance to evaluate a new model/vendor to ran future networks 
(with possibly also a very good scale-up path and scaling- and 
upgrade-options). Most of the before mentioned are already in an migration 
process (let's take a look at Seabone ... they are migration from Cisco to a 
mix of Juniper and Huawei). 

Summary: there are strict limitations within the Cisco 6500/7600 platform and 
these limitations forces the big players to move this boxes out (or move them 
into other parts of their network). The limitation with 1Mio routes is not a 
secret and the admins of these boxes decide what they want to use (e.g. 768k 
routes for IPv4 unicast and 256k routes for MPLS+VRF, etc). If the global 
routing table reaches the 768k mark (I guess this will happen in the next 
12-18months) most of the boxes will crash again (as it happened in Aug 2014). 


Regarding the words "I have a small router which handles multiple full tables 
...": push and pull a few full tables at the same time and you'll see what's 
happening: the CCRs are SLOW. And why? Because the software is not as good as 
it could be: the BGP daemon uses only one core of a 36(?) core CPU. Same 
problem in the past with the EoIP daemon (not sure if they fixed it on the CCRs 
- they fixed it 

Re: AW: /27 the new /24

[William Waites]

On Sat, 3 Oct 2015 12:42:01 +0200, Baldur Norddahl  
said:

> 2 million routes will not be enough if we go full /27. This is
> not a scalable solution. Something else is needed to provide
> multihoming for small networks (LISP?).

It's not too far off though. One way of looking at it is, for each
extra bit we allow, we potentially double the table size. So with 500k
routes and a /24 limit now, we might expect 4 million with /27. Not
exactly because it depends strongly on the distribution of prefix
lengths, but probably not a bad guess.

Also there are optimisations that I wonder if the vendors are doing to
preserve TCAM such as aggregating adjacent networks with the same next
hop into the supernet. That would mitigate the impact of wanton
deaggregation at least and the algorithm doesn't look too hard. Do the
big iron vendors do this?

-w

--
William Waites   |  School of Informatics
   http://tardis.ed.ac.uk/~wwaites/   | University of Edinburgh
 https://hubs.net.uk/ |  HUBS AS60241

The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.


pgpQpvBY0HsM0.pgp
Description: PGP signature

Re: AW: /27 the new /24

[Baldur Norddahl]

2 million routes will not be enough if we go full /27. This is not a
scalable solution. Something else is needed to provide multihoming for
small networks (LISP?).

Regards,

Baldur


On 3 October 2015 at 11:03, Youssef Bengelloun-Zahr  wrote:

> Hi,
>
> FYI, newer linecard models from BROCADE can hold 2 million routes.
> Probably others can do that now too.
>
> Disclaimer : I'm not working for them or defending them, just setting an
> information straight.
>
> My 2 cents.
>
>
>
> > Le 3 oct. 2015 à 10:33, Jürgen Jaritsch  a écrit :
> >
> > As mentioned before: even the new SUP2T from Cisco is limited to 1Mio
> routes ...
> >
> > There are MANY other vendors with the same limitations: Juniper,
> Brocade, etc
> >
> > And the solt equipment is not the 99USD trash from the super market at
> the corner ...
> >
> >
> > Jürgen Jaritsch
> > Head of Network & Infrastructure
> >
> > ANEXIA Internetdienstleistungs GmbH
> >
> > Telefon: +43-5-0556-300
> > Telefax: +43-5-0556-500
> >
> > E-Mail: j...@anexia.at
> > Web: http://www.anexia.at
> >
> > Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
> > Geschäftsführer: Alexander Windbichler
> > Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT
> U63216601
> >
> >
> > -Original Message-
> > From: Max Tulyev [max...@netassist.ua]
> > Received: Samstag, 03 Okt. 2015, 9:11
> > To: nanog@nanog.org [nanog@nanog.org]
> > Subject: Re: AW: /27 the new /24
> >
> > Which routers? DIR-300 with OpenWRT/Quagga? :)
> >
> > I think all above-the-trash level routers supports >1M routes, isn't it?
> >
> >> On 02.10.15 17:45, Jürgen Jaritsch wrote:
> >> Hi,
> >>
> >> this would at least help to get rid of many old routing engines around
> the world :) ... or people would keep their "learn nothing smaller than
> /24" filters in place. Also an option - but not for companies who act as an
> IP transit provider.
> >>
> >>
> >> best regards
> >>
> >> Jürgen Jaritsch
> >> Head of Network & Infrastructure
> >>
> >> ANEXIA Internetdienstleistungs GmbH
> >>
> >> Telefon: +43-5-0556-300
> >> Telefax: +43-5-0556-500
> >>
> >> E-Mail: jjarit...@anexia-it.com
> >> Web: http://www.anexia-it.com
> >>
> >> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
> >> Geschäftsführer: Alexander Windbichler
> >> Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT
> U63216601
> >>
> >>
> >> -Ursprüngliche Nachricht-
> >> Von: NANOG [mailto:nanog-boun...@nanog.org] Im Auftrag von Justin
> Wilson - MTIN
> >> Gesendet: Freitag, 02. Oktober 2015 16:32
> >> An: NANOG
> >> Betreff: /27 the new /24
> >>
> >> I was in a discussion the other day and several Tier2 providers were
> talking about the idea of adjusting their BGP filters to accept prefixes
> smaller than a /24.  A few were saying they thought about going down to as
> small as a /27.  This was mainly due to more networks coming online and not
> having even a /24 of IPv4 space.  The first argument is against this is the
> potential bloat the global routing table could have.  Many folks have
> worked hard for years to summarize and such. others were saying they would
> do a /26 or bigger.
> >>
> >> However, what do we do about the new networks which want to do BGP but
> only can get small allocations from someone (either a RIR or one of their
> upstreams)?
> >>
> >> Just throwing that out there. Seems like an interesting discussion.
> >>
> >>
> >> Justin Wilson
> >> j...@mtin.net
> >>
> >> ---
> >> http://www.mtin.net Owner/CEO
> >> xISP Solutions- Consulting – Data Centers - Bandwidth
> >>
> >> http://www.midwest-ix.com  COO/Chairman
> >> Internet Exchange - Peering - Distributed Fabric
> >
>

AW: AW: /27 the new /24

[Jürgen Jaritsch]

Hi Mike,

it's not a bureaucracy problem ... if you're a big player and you have to 
decide about a 2-3 Mio invest to upgrade only a few of your POPs (and let's say 
you have hundreds of POPs) it will be hard to find the "right" decision.

Some questions  these decision makers have to think about:

#) What are the future plans for this POP?
#) How upgradeable / expandable is the new equipment?
#) Does our engineers know everything they need to run & debug & fix this new 
equipment?
#) TOC incl support contract over the complete lifetime?
#) Product life cycle? (Is it outdated in two years??)
#) Will we keep spare parts onsite or nearby?
#) How long needs the vendor to deliver everything I need?
#) Is it compatible with all the already installed equipment?
#) Migration plan to move existing customers to the new equipment?

There are a ton of additional questions ... but I guess I pointed out some of 
the most important. Big players can't only calculate the price of the equipment 
- most of the time all the surrounding stuff (installation, new cabinets, 
migrations, training of engineers, etc) is producing 0,5x to 1x of the 
equipment costs. To get some easy numbers: take the discounted price (no one 
pays list prices ...) of an equipment and take this price x2 => that will be a 
realistic number to get the box onsite, up and running.

It's not all the time something simple like a router with 20 patch cords :(.

Best regards

Jürgen Jaritsch
Head of Network & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300
Telefax: +43-5-0556-500

E-Mail: jjarit...@anexia-it.com 
Web: http://www.anexia-it.com 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

-Ursprüngliche Nachricht-
Von: NANOG [mailto:nanog-boun...@nanog.org] Im Auftrag von Mike Hammett
Gesendet: Samstag, 03. Oktober 2015 04:53
Cc: NANOG 
Betreff: Re: AW: /27 the new /24

A better truth may be that I have no idea about bureaucracies... which I'll 
happily admit to. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


- Original Message -

From: "Jürgen Jaritsch"  
To: "Mike Hammett" , "NANOG"  
Sent: Friday, October 2, 2015 2:25:10 PM 
Subject: AW: /27 the new /24 

> Stop using old shit. 

Sorry, but the truth is: you have no idea about how earning revenue works and 
you obviously also have no idea about carrier grade networks. 




Jürgen Jaritsch 
Head of Network & Infrastructure 

ANEXIA Internetdienstleistungs GmbH 

Telefon: +43-5-0556-300 
Telefax: +43-5-0556-500 

E-Mail: jjarit...@anexia-it.com 
Web: http://www.anexia-it.com 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt 
Geschäftsführer: Alexander Windbichler 
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 

-Ursprüngliche Nachricht- 
Von: NANOG [mailto:nanog-boun...@nanog.org] Im Auftrag von Mike Hammett 
Gesendet: Freitag, 02. Oktober 2015 20:38 
An: NANOG  
Betreff: Re: /27 the new /24 

Chances are the revenue passing scales to some degree as well. Small business 
with small bandwidth needs buys small and has small revenue. Big business with 
big bandwidth needs buys big and has big revenue to support big router. 

I can think of no reason why ten years goes by and you haven't had a need to 
throw out the old network for new. If your business hasn't scaled with the 
times, then you need to get rid of your Cat 6500 and get something more power, 
space, heat, etc. efficient. 


I saw someone replace a stack of Mikrotik CCRs with a pair of old Cisco 
routers. I don't know what they were at the moment, but they had GBICs, so they 
weren't exactly new. Each router had two 2500w power supplies. They'll be worse 
in every way (other than *possibly* BGP convergence). The old setup consumed at 
most 300 watts. The new setup requires $500/month in power... and is worse. 

Stop using old shit. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


- Original Message - 

From: "William Herrin"  
To: "Mike Hammett"  
Cc: "NANOG"  
Sent: Friday, October 2, 2015 1:09:16 PM 
Subject: Re: /27 the new /24 

On Fri, Oct 2, 2015 at 11:50 AM, Mike Hammett  wrote: 
> How many routers out there have this limitation? A $100 router 
> I bought ten years ago could manage many full tables. If 
> someone's network can't match that today, should I really have 
> any pity for them? 

Hi Mike, 

The technology doesn't work the way you think it does. Or more 
precisely, it only works the way you think it does on small (cheap)

AW: AW: /27 the new /24

[Jürgen Jaritsch]

Hi,

yeah, of course there are newer models ... I mentioned the older ones (from the 
past 3-5 years). There are also Cisco routers available that are able to handle 
more than 1 Mio routes - of course also from Juniper.




Jürgen Jaritsch
Head of Network & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300
Telefax: +43-5-0556-500

E-Mail: jjarit...@anexia-it.com 
Web: http://www.anexia-it.com 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

-Ursprüngliche Nachricht-
Von: Youssef Bengelloun-Zahr [mailto:yous...@720.fr] 
Gesendet: Samstag, 03. Oktober 2015 11:03
An: Jürgen Jaritsch 
Cc: nanog@nanog.org; max...@netassist.ua
Betreff: Re: AW: /27 the new /24

Hi,

FYI, newer linecard models from BROCADE can hold 2 million routes. Probably 
others can do that now too.

Disclaimer : I'm not working for them or defending them, just setting an 
information straight.

My 2 cents.



> Le 3 oct. 2015 à 10:33, Jürgen Jaritsch  a écrit :
> 
> As mentioned before: even the new SUP2T from Cisco is limited to 1Mio routes 
> ...
> 
> There are MANY other vendors with the same limitations: Juniper, Brocade, etc
> 
> And the solt equipment is not the 99USD trash from the super market at the 
> corner ...
> 
> 
> Jürgen Jaritsch
> Head of Network & Infrastructure
> 
> ANEXIA Internetdienstleistungs GmbH
> 
> Telefon: +43-5-0556-300
> Telefax: +43-5-0556-500
> 
> E-Mail: j...@anexia.at
> Web: http://www.anexia.at
> 
> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
> Geschäftsführer: Alexander Windbichler
> Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
> 
> 
> -Original Message-
> From: Max Tulyev [max...@netassist.ua]
> Received: Samstag, 03 Okt. 2015, 9:11
> To: nanog@nanog.org [nanog@nanog.org]
> Subject: Re: AW: /27 the new /24
> 
> Which routers? DIR-300 with OpenWRT/Quagga? :)
> 
> I think all above-the-trash level routers supports >1M routes, isn't it?
> 
>> On 02.10.15 17:45, Jürgen Jaritsch wrote:
>> Hi,
>> 
>> this would at least help to get rid of many old routing engines around the 
>> world :) ... or people would keep their "learn nothing smaller than /24" 
>> filters in place. Also an option - but not for companies who act as an IP 
>> transit provider.
>> 
>> 
>> best regards
>> 
>> Jürgen Jaritsch
>> Head of Network & Infrastructure
>> 
>> ANEXIA Internetdienstleistungs GmbH
>> 
>> Telefon: +43-5-0556-300
>> Telefax: +43-5-0556-500
>> 
>> E-Mail: jjarit...@anexia-it.com
>> Web: http://www.anexia-it.com
>> 
>> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
>> Geschäftsführer: Alexander Windbichler
>> Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
>> 
>> 
>> -Ursprüngliche Nachricht-
>> Von: NANOG [mailto:nanog-boun...@nanog.org] Im Auftrag von Justin Wilson - 
>> MTIN
>> Gesendet: Freitag, 02. Oktober 2015 16:32
>> An: NANOG
>> Betreff: /27 the new /24
>> 
>> I was in a discussion the other day and several Tier2 providers were talking 
>> about the idea of adjusting their BGP filters to accept prefixes smaller 
>> than a /24.  A few were saying they thought about going down to as small as 
>> a /27.  This was mainly due to more networks coming online and not having 
>> even a /24 of IPv4 space.  The first argument is against this is the 
>> potential bloat the global routing table could have.  Many folks have worked 
>> hard for years to summarize and such. others were saying they would do a /26 
>> or bigger.
>> 
>> However, what do we do about the new networks which want to do BGP but only 
>> can get small allocations from someone (either a RIR or one of their 
>> upstreams)?
>> 
>> Just throwing that out there. Seems like an interesting discussion.
>> 
>> 
>> Justin Wilson
>> j...@mtin.net
>> 
>> ---
>> http://www.mtin.net Owner/CEO
>> xISP Solutions- Consulting – Data Centers - Bandwidth
>> 
>> http://www.midwest-ix.com  COO/Chairman
>> Internet Exchange - Peering - Distributed Fabric
> 

Re: AW: /27 the new /24

[Youssef Bengelloun-Zahr]

Hi,

FYI, newer linecard models from BROCADE can hold 2 million routes. Probably 
others can do that now too.

Disclaimer : I'm not working for them or defending them, just setting an 
information straight.

My 2 cents.



> Le 3 oct. 2015 à 10:33, Jürgen Jaritsch  a écrit :
> 
> As mentioned before: even the new SUP2T from Cisco is limited to 1Mio routes 
> ...
> 
> There are MANY other vendors with the same limitations: Juniper, Brocade, etc
> 
> And the solt equipment is not the 99USD trash from the super market at the 
> corner ...
> 
> 
> Jürgen Jaritsch
> Head of Network & Infrastructure
> 
> ANEXIA Internetdienstleistungs GmbH
> 
> Telefon: +43-5-0556-300
> Telefax: +43-5-0556-500
> 
> E-Mail: j...@anexia.at
> Web: http://www.anexia.at
> 
> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
> Geschäftsführer: Alexander Windbichler
> Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
> 
> 
> -Original Message-
> From: Max Tulyev [max...@netassist.ua]
> Received: Samstag, 03 Okt. 2015, 9:11
> To: nanog@nanog.org [nanog@nanog.org]
> Subject: Re: AW: /27 the new /24
> 
> Which routers? DIR-300 with OpenWRT/Quagga? :)
> 
> I think all above-the-trash level routers supports >1M routes, isn't it?
> 
>> On 02.10.15 17:45, Jürgen Jaritsch wrote:
>> Hi,
>> 
>> this would at least help to get rid of many old routing engines around the 
>> world :) ... or people would keep their "learn nothing smaller than /24" 
>> filters in place. Also an option - but not for companies who act as an IP 
>> transit provider.
>> 
>> 
>> best regards
>> 
>> Jürgen Jaritsch
>> Head of Network & Infrastructure
>> 
>> ANEXIA Internetdienstleistungs GmbH
>> 
>> Telefon: +43-5-0556-300
>> Telefax: +43-5-0556-500
>> 
>> E-Mail: jjarit...@anexia-it.com
>> Web: http://www.anexia-it.com
>> 
>> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
>> Geschäftsführer: Alexander Windbichler
>> Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
>> 
>> 
>> -Ursprüngliche Nachricht-
>> Von: NANOG [mailto:nanog-boun...@nanog.org] Im Auftrag von Justin Wilson - 
>> MTIN
>> Gesendet: Freitag, 02. Oktober 2015 16:32
>> An: NANOG
>> Betreff: /27 the new /24
>> 
>> I was in a discussion the other day and several Tier2 providers were talking 
>> about the idea of adjusting their BGP filters to accept prefixes smaller 
>> than a /24.  A few were saying they thought about going down to as small as 
>> a /27.  This was mainly due to more networks coming online and not having 
>> even a /24 of IPv4 space.  The first argument is against this is the 
>> potential bloat the global routing table could have.  Many folks have worked 
>> hard for years to summarize and such. others were saying they would do a /26 
>> or bigger.
>> 
>> However, what do we do about the new networks which want to do BGP but only 
>> can get small allocations from someone (either a RIR or one of their 
>> upstreams)?
>> 
>> Just throwing that out there. Seems like an interesting discussion.
>> 
>> 
>> Justin Wilson
>> j...@mtin.net
>> 
>> ---
>> http://www.mtin.net Owner/CEO
>> xISP Solutions- Consulting – Data Centers - Bandwidth
>> 
>> http://www.midwest-ix.com  COO/Chairman
>> Internet Exchange - Peering - Distributed Fabric
> 

RE: AW: /27 the new /24

[Jürgen Jaritsch]

As mentioned before: even the new SUP2T from Cisco is limited to 1Mio routes ...

There are MANY other vendors with the same limitations: Juniper, Brocade, etc

And the solt equipment is not the 99USD trash from the super market at the 
corner ...


Jürgen Jaritsch
Head of Network & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300
Telefax: +43-5-0556-500

E-Mail: j...@anexia.at
Web: http://www.anexia.at

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601


-Original Message-
From: Max Tulyev [max...@netassist.ua]
Received: Samstag, 03 Okt. 2015, 9:11
To: nanog@nanog.org [nanog@nanog.org]
Subject: Re: AW: /27 the new /24

Which routers? DIR-300 with OpenWRT/Quagga? :)

I think all above-the-trash level routers supports >1M routes, isn't it?

On 02.10.15 17:45, Jürgen Jaritsch wrote:
> Hi,
>
> this would at least help to get rid of many old routing engines around the 
> world :) ... or people would keep their "learn nothing smaller than /24" 
> filters in place. Also an option - but not for companies who act as an IP 
> transit provider.
>
>
> best regards
>
> Jürgen Jaritsch
> Head of Network & Infrastructure
>
> ANEXIA Internetdienstleistungs GmbH
>
> Telefon: +43-5-0556-300
> Telefax: +43-5-0556-500
>
> E-Mail: jjarit...@anexia-it.com
> Web: http://www.anexia-it.com
>
> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
> Geschäftsführer: Alexander Windbichler
> Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
>
>
> -Ursprüngliche Nachricht-
> Von: NANOG [mailto:nanog-boun...@nanog.org] Im Auftrag von Justin Wilson - 
> MTIN
> Gesendet: Freitag, 02. Oktober 2015 16:32
> An: NANOG
> Betreff: /27 the new /24
>
> I was in a discussion the other day and several Tier2 providers were talking 
> about the idea of adjusting their BGP filters to accept prefixes smaller than 
> a /24.  A few were saying they thought about going down to as small as a /27. 
>  This was mainly due to more networks coming online and not having even a /24 
> of IPv4 space.  The first argument is against this is the potential bloat the 
> global routing table could have.  Many folks have worked hard for years to 
> summarize and such. others were saying they would do a /26 or bigger.
>
> However, what do we do about the new networks which want to do BGP but only 
> can get small allocations from someone (either a RIR or one of their 
> upstreams)?
>
> Just throwing that out there. Seems like an interesting discussion.
>
>
> Justin Wilson
> j...@mtin.net
>
> ---
> http://www.mtin.net Owner/CEO
> xISP Solutions- Consulting – Data Centers - Bandwidth
>
> http://www.midwest-ix.com  COO/Chairman
> Internet Exchange - Peering - Distributed Fabric
>

Mikrotik in the DFZ (Was Re: AW: AW: /27 the new /24)

[William Waites]

On Fri, 2 Oct 2015 23:11:47 +, Jürgen Jaritsch  said:

> Regarding the words "I have a small router which handles
> multiple full tables ...": push and pull a few full tables at
> the same time and you'll see what's happening: the CCRs are
> SLOW. And why? Because the software is not as good as it could
> be: the BGP daemon uses only one core of a 36(?) core CPU.

To expand on this, the problem is worse than being single-threaded. I
had one of these in the lab and fed it 2x full tables. Sure it wasn't
the fastest at accepting them but then I noticed that even in steady
state one of the CPUs was pegged. What was happening -- and this was
confirmed by Mikrotik -- was that it was recalculating the *entire*
FIB for each update. The general background noise of announce /
withdraw messages means it is doing this all the time. Any churn and
it would have a very hard time.

There are other serious bugs such as not doing recursive next hop
lookup for IPv6 (it does for IPv4). This makes them unuseable as BGP
routers even for partial tables with most non-trivial iBGP
topologies. All of which may be fixed one day in version 7 of their
operating system, which will inevitably have many bugs as any software
project .0 release will, so we'll have to wait for 7.x for it to be
reasonably safe to use.

That said, we use a lot of Mikrotik kit for our rural
networks. They're weird and quirky but you can't beat them on price,
port density and power consumption. With 16 ports and 36 cores surely 
they should be capable of pushing several Gbps of traffic with a few
full tables.

I wish it were possible today to run different software on their
larger boxes. If some like-minded small providers wanted to get
together with us to fund a FreeBSD port to the CCR routers that would
be great. Please contact me off-list if you are interested in this,
I'll coordinate.

As it is we don't let them anywhere near the DFZ, that's done with PCs
running FreeBSD and BIRD which can easily do the job but is still an
order of magnitude more expensive (and an order of magnitude less
expensive than what you need if you want 10s of Gbps).

-w

--
William Waites   |  School of Informatics
   http://tardis.ed.ac.uk/~wwaites/   | University of Edinburgh
 https://hubs.net.uk/ |  HUBS AS60241

The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.


pgpkqJVGALdoZ.pgp
Description: PGP signature

Re: AW: /27 the new /24

[Max Tulyev]

Which routers? DIR-300 with OpenWRT/Quagga? :)

I think all above-the-trash level routers supports >1M routes, isn't it?

On 02.10.15 17:45, Jürgen Jaritsch wrote:
> Hi,
> 
> this would at least help to get rid of many old routing engines around the 
> world :) ... or people would keep their "learn nothing smaller than /24" 
> filters in place. Also an option - but not for companies who act as an IP 
> transit provider.
> 
> 
> best regards
> 
> Jürgen Jaritsch
> Head of Network & Infrastructure
> 
> ANEXIA Internetdienstleistungs GmbH
> 
> Telefon: +43-5-0556-300
> Telefax: +43-5-0556-500
> 
> E-Mail: jjarit...@anexia-it.com 
> Web: http://www.anexia-it.com 
> 
> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
> Geschäftsführer: Alexander Windbichler
> Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
> 
> 
> -Ursprüngliche Nachricht-
> Von: NANOG [mailto:nanog-boun...@nanog.org] Im Auftrag von Justin Wilson - 
> MTIN
> Gesendet: Freitag, 02. Oktober 2015 16:32
> An: NANOG
> Betreff: /27 the new /24
> 
> I was in a discussion the other day and several Tier2 providers were talking 
> about the idea of adjusting their BGP filters to accept prefixes smaller than 
> a /24.  A few were saying they thought about going down to as small as a /27. 
>  This was mainly due to more networks coming online and not having even a /24 
> of IPv4 space.  The first argument is against this is the potential bloat the 
> global routing table could have.  Many folks have worked hard for years to 
> summarize and such. others were saying they would do a /26 or bigger.  
> 
> However, what do we do about the new networks which want to do BGP but only 
> can get small allocations from someone (either a RIR or one of their 
> upstreams)?
> 
> Just throwing that out there. Seems like an interesting discussion.
> 
> 
> Justin Wilson
> j...@mtin.net
> 
> ---
> http://www.mtin.net Owner/CEO
> xISP Solutions- Consulting – Data Centers - Bandwidth
> 
> http://www.midwest-ix.com  COO/Chairman
> Internet Exchange - Peering - Distributed Fabric
> 

Re: AW: /27 the new /24

[Mike Hammett]

A better truth may be that I have no idea about bureaucracies... which I'll 
happily admit to. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


- Original Message -

From: "Jürgen Jaritsch"  
To: "Mike Hammett" , "NANOG"  
Sent: Friday, October 2, 2015 2:25:10 PM 
Subject: AW: /27 the new /24 

> Stop using old shit. 

Sorry, but the truth is: you have no idea about how earning revenue works and 
you obviously also have no idea about carrier grade networks. 




Jürgen Jaritsch 
Head of Network & Infrastructure 

ANEXIA Internetdienstleistungs GmbH 

Telefon: +43-5-0556-300 
Telefax: +43-5-0556-500 

E-Mail: jjarit...@anexia-it.com 
Web: http://www.anexia-it.com 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt 
Geschäftsführer: Alexander Windbichler 
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 

-Ursprüngliche Nachricht- 
Von: NANOG [mailto:nanog-boun...@nanog.org] Im Auftrag von Mike Hammett 
Gesendet: Freitag, 02. Oktober 2015 20:38 
An: NANOG  
Betreff: Re: /27 the new /24 

Chances are the revenue passing scales to some degree as well. Small business 
with small bandwidth needs buys small and has small revenue. Big business with 
big bandwidth needs buys big and has big revenue to support big router. 

I can think of no reason why ten years goes by and you haven't had a need to 
throw out the old network for new. If your business hasn't scaled with the 
times, then you need to get rid of your Cat 6500 and get something more power, 
space, heat, etc. efficient. 


I saw someone replace a stack of Mikrotik CCRs with a pair of old Cisco 
routers. I don't know what they were at the moment, but they had GBICs, so they 
weren't exactly new. Each router had two 2500w power supplies. They'll be worse 
in every way (other than *possibly* BGP convergence). The old setup consumed at 
most 300 watts. The new setup requires $500/month in power... and is worse. 

Stop using old shit. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


- Original Message - 

From: "William Herrin"  
To: "Mike Hammett"  
Cc: "NANOG"  
Sent: Friday, October 2, 2015 1:09:16 PM 
Subject: Re: /27 the new /24 

On Fri, Oct 2, 2015 at 11:50 AM, Mike Hammett  wrote: 
> How many routers out there have this limitation? A $100 router 
> I bought ten years ago could manage many full tables. If 
> someone's network can't match that today, should I really have 
> any pity for them? 

Hi Mike, 

The technology doesn't work the way you think it does. Or more 
precisely, it only works the way you think it does on small (cheap) 
end-user routers. Those routers do everything in software on a 
general-purpose CPU using radix tries for the forwarding table (FIB). 
They don't have to (and can't) handle both high data rates and large 
routing tables at the same time. 

For a better understanding how the big iron works, check out 
https://www.pagiamtzis.com/cam/camintro/ . You'll occasionally see 
folks here talk about TCAM. This stands for Ternary Content 
Addressable Memory. It's a special circuit, different from DRAM and 
SRAM, used by most (but not all) big iron routers. The TCAM permits an 
O(1) route lookup instead of an O(log n) lookup. The architectural 
differences which balloon from there move the router cost from your 
$100 router into the hundreds of thousands of dollars. 

Your BGP advertisement doesn't just have to be carried on your $100 
router. It also has to be carried on the half-million-dollar routers. 
That makes it expensive. 

Though out of date, this paper should help you better understand the 
systemic cost of a BGP route advertisement: 
http://bill.herrin.us/network/bgpcost.html 

Regards, 
Bill Herrin 




-- 
William Herrin  her...@dirtside.com b...@herrin.us 
Owner, Dirtside Systems . Web: <http://www.dirtside.com/> 

Re: AW: /27 the new /24

[Randy Bush]

> From: "Jürgen Jaritsch"  
> To: "Mike Hammett" , "NANOG"  
>> Stop using old shit. 
> Sorry, but the truth is: you have no idea about how earning revenue
> works and you obviously also have no idea about carrier grade
> networks.

bingo!

Re: AW: AW: /27 the new /24

[Mike Hammett]

I don't expect carriers to be running UBNT\Mikrotik, but the boxes that have 
been there for 10 years have more than paid for themselves (unless they're a 
shitty business). It's time to rip and replace with whatever is appropriate for 
that site. No, I obviously don't think I'm going to change anyone's opinion on 
the matter (at least not anyone that matters in one of these networks). What I 
was saying is that my little business with meager means (and revenues) can 
afford a box to do it. They can too. 



I don't doubt their situation sucks... but either you fix it or you don't. Time 
and the rest of the Internet won't wait for them. 


If their business hasn't boomed, maybe it's time to replace that old 6500 with 
a 4500x or a QFX-5100 or an x670 or whatever. Your decreased power bill alone 
will pay it off. If it has boomed, then ten years of revenues should get you 
whatever the bigger Ciscos are or an MX or whatever the bigger Extremes are. 

Don't whine about my choices in gear I mentioned. I was just throwing things 
out there. Old big, new small if no money or old big new big if money. 


BTW: ROS 7 won't have multi-threaded BGP, but will be optimized to handle full 
table imports in a significantly reduced time. Oh, and I'm not sure that you 
couldn't do at least three nines with MT\UBNT. Well, no experience with the 
EdgeRouters yet. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


- Original Message -

From: "Jürgen Jaritsch"  
To: "Mike Hammett"  
Cc: "NANOG"  
Sent: Friday, October 2, 2015 6:11:47 PM 
Subject: AW: AW: /27 the new /24 

Hi Mike, 

sorry, this was probably sent to quick ... let me please explain my POV of your 
statement: 

I want to concentrate my detailed answer only to the backbone situation which 
is often handled by the 6500/7600 - I guess all of us know that the 6500/7600 
has a ton of additional features ... 


6-7 years in the past carriers (and/or big ISPs) had only n*1G backbone 
capacities built with platforms that only had n*100M interfaces another 3-5 
years before. Their only invest in these 3-5 years was to add the Gig line 
cards, install some software updates and add new fibre optics (GBICs). Chassis, 
cabling, management interfaces etc could be remain in the cabinet - they only 
had to replace ONE line card (let's say for a few thousand bucks) and with this 
invest they were able to scale up their capacities. Of course: at some point 
they also had to replace the SUPs, PSUs, FANs, etc. But the invest in the 
surrounding stuff is nothing compared with completely new machines. 

So what all these companies did was buying a machine with an basic 
configuration and since 10(!) years they are able to expand this machines with 
(more or less) small and cheap upgrades. 

In backbone situations the 6500/7600 are definitely at the end of the resources 
the platform can provide. Most of the carriers (and of course also the bigger 
ISPs) had a real chance to evaluate a new model/vendor to ran future networks 
(with possibly also a very good scale-up path and scaling- and 
upgrade-options). Most of the before mentioned are already in an migration 
process (let's take a look at Seabone ... they are migration from Cisco to a 
mix of Juniper and Huawei). 

Summary: there are strict limitations within the Cisco 6500/7600 platform and 
these limitations forces the big players to move this boxes out (or move them 
into other parts of their network). The limitation with 1Mio routes is not a 
secret and the admins of these boxes decide what they want to use (e.g. 768k 
routes for IPv4 unicast and 256k routes for MPLS+VRF, etc). If the global 
routing table reaches the 768k mark (I guess this will happen in the next 
12-18months) most of the boxes will crash again (as it happened in Aug 2014). 


Regarding the words "I have a small router which handles multiple full tables 
...": push and pull a few full tables at the same time and you'll see what's 
happening: the CCRs are SLOW. And why? Because the software is not as good as 
it could be: the BGP daemon uses only one core of a 36(?) core CPU. Same 
problem in the past with the EoIP daemon (not sure if they fixed it on the CCRs 
- they fixed it on x86). 

Routerboards are nice and cool and to be honest: I'm a big fan of this stuff 
(also Ubiquiti). But with this boxes you're not able to ran a stable enterprise 
class carrier network with >99,5% uptime. And that’s thei MAIN reason why "the 
old shit" is still online :). 

Hopefully my words explained my hard "you know nothing" blabla ? 

Best regards 


Jürgen Jaritsch 
Head of Network & Infrastructure 

ANEXIA Internetdienstleistungs GmbH 

Telefon: +43-5-0556-300 
Telefax: +43-5-0556-500 

E-Mail: jjarit...@an

Re: AW: AW: /27 the new /24

[Mel Beckman]

Well said, Jürgen!

-mel via cell

> On Oct 2, 2015, at 4:13 PM, Jürgen Jaritsch  wrote:
> 
> Hi Mike,
> 
> sorry, this was probably sent to quick ... let me please explain my POV of 
> your statement:
> 
> I want to concentrate my detailed answer only to the backbone situation which 
> is often handled by the 6500/7600 - I guess all of us know that the 6500/7600 
> has a ton of additional features ...
> 
> 
> 6-7 years in the past carriers (and/or big ISPs) had only n*1G backbone 
> capacities built with platforms that only had n*100M interfaces another 3-5 
> years before. Their only invest in these 3-5 years was to add the Gig line 
> cards, install some software updates and add new fibre optics (GBICs). 
> Chassis, cabling, management interfaces etc could be remain in the cabinet - 
> they only had to replace ONE line card (let's say for a few thousand bucks) 
> and with this invest they were able to scale up their capacities. Of course: 
> at some point they also had to replace the SUPs, PSUs, FANs, etc. But the 
> invest in the surrounding stuff is nothing compared with completely new 
> machines.
> 
> So what all these companies did was buying a machine with an basic 
> configuration and since 10(!) years they are able to expand this machines 
> with (more or less) small and cheap upgrades. 
> 
> In backbone situations the 6500/7600 are definitely at the end of the 
> resources the platform can provide. Most of the carriers (and of course also 
> the bigger ISPs) had a real chance to evaluate a new model/vendor to ran 
> future networks (with possibly also a very good scale-up path and scaling- 
> and upgrade-options). Most of the before mentioned are already in an 
> migration process (let's take a look at Seabone ... they are migration from 
> Cisco to a mix of Juniper and Huawei).
> 
> Summary: there are strict limitations within the Cisco 6500/7600 platform and 
> these limitations forces the big players to move this boxes out (or move them 
> into other parts of their network). The limitation with 1Mio routes is not a 
> secret and the admins of these boxes decide what they want to use (e.g. 768k 
> routes for IPv4 unicast and 256k routes for MPLS+VRF, etc). If the global 
> routing table reaches the 768k mark (I guess this will happen in the next 
> 12-18months) most of the boxes will crash again (as it happened in Aug 2014). 
> 
> 
> Regarding the words "I have a small router which handles multiple full tables 
> ...": push and pull a few full tables at the same time and you'll see what's 
> happening: the CCRs are SLOW. And why? Because the software is not as good as 
> it could be: the BGP daemon uses only one core of a 36(?) core CPU. Same 
> problem in the past with the EoIP daemon (not sure if they fixed it on the 
> CCRs - they fixed it on x86).
> 
> Routerboards are nice and cool and to be honest: I'm a big fan of this stuff 
> (also Ubiquiti). But with this boxes you're not able to ran a stable 
> enterprise class carrier network with >99,5% uptime. And that’s thei MAIN 
> reason why "the old shit" is still online :).
> 
> Hopefully my words explained my hard "you know nothing" blabla ?
> 
> Best regards
> 
> 
> Jürgen Jaritsch
> Head of Network & Infrastructure
> 
> ANEXIA Internetdienstleistungs GmbH
> 
> Telefon: +43-5-0556-300
> Telefax: +43-5-0556-500
> 
> E-Mail: jjarit...@anexia-it.com 
> Web: http://www.anexia-it.com 
> 
> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
> Geschäftsführer: Alexander Windbichler
> Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
> 
> -Ursprüngliche Nachricht-
> Von: NANOG [mailto:nanog-boun...@nanog.org] Im Auftrag von Mike Hammett
> Gesendet: Freitag, 02. Oktober 2015 21:33
> Cc: NANOG 
> Betreff: Re: AW: /27 the new /24
> 
> Hrm. 
> 
> 
> 
> 
> - 
> Mike Hammett 
> Intelligent Computing Solutions 
> http://www.ics-il.com 
> 
> 
> 
> Midwest Internet Exchange 
> http://www.midwest-ix.com 
> 
> 
> - Original Message -
> 
> From: "Jürgen Jaritsch"  
> To: "Mike Hammett" , "NANOG"  
> Sent: Friday, October 2, 2015 2:25:10 PM 
> Subject: AW: /27 the new /24 
> 
>> Stop using old shit.
> 
> Sorry, but the truth is: you have no idea about how earning revenue works and 
> you obviously also have no idea about carrier grade networks. 
> 
> 
> 
> 
> Jürgen Jaritsch 
> Head of Network & Infrastructure 
> 
> ANEXIA Internetdienstleistungs GmbH 
> 
> Telefon: +43-5-0556-300 
> Telefax: +43-5-0556-500 
> 
> E-Mail: jjarit.

AW: AW: /27 the new /24

[Jürgen Jaritsch]

Hi Mike,

sorry, this was probably sent to quick ... let me please explain my POV of your 
statement:

I want to concentrate my detailed answer only to the backbone situation which 
is often handled by the 6500/7600 - I guess all of us know that the 6500/7600 
has a ton of additional features ...


6-7 years in the past carriers (and/or big ISPs) had only n*1G backbone 
capacities built with platforms that only had n*100M interfaces another 3-5 
years before. Their only invest in these 3-5 years was to add the Gig line 
cards, install some software updates and add new fibre optics (GBICs). Chassis, 
cabling, management interfaces etc could be remain in the cabinet - they only 
had to replace ONE line card (let's say for a few thousand bucks) and with this 
invest they were able to scale up their capacities. Of course: at some point 
they also had to replace the SUPs, PSUs, FANs, etc. But the invest in the 
surrounding stuff is nothing compared with completely new machines.

So what all these companies did was buying a machine with an basic 
configuration and since 10(!) years they are able to expand this machines with 
(more or less) small and cheap upgrades. 

In backbone situations the 6500/7600 are definitely at the end of the resources 
the platform can provide. Most of the carriers (and of course also the bigger 
ISPs) had a real chance to evaluate a new model/vendor to ran future networks 
(with possibly also a very good scale-up path and scaling- and 
upgrade-options). Most of the before mentioned are already in an migration 
process (let's take a look at Seabone ... they are migration from Cisco to a 
mix of Juniper and Huawei).

Summary: there are strict limitations within the Cisco 6500/7600 platform and 
these limitations forces the big players to move this boxes out (or move them 
into other parts of their network). The limitation with 1Mio routes is not a 
secret and the admins of these boxes decide what they want to use (e.g. 768k 
routes for IPv4 unicast and 256k routes for MPLS+VRF, etc). If the global 
routing table reaches the 768k mark (I guess this will happen in the next 
12-18months) most of the boxes will crash again (as it happened in Aug 2014). 


Regarding the words "I have a small router which handles multiple full tables 
...": push and pull a few full tables at the same time and you'll see what's 
happening: the CCRs are SLOW. And why? Because the software is not as good as 
it could be: the BGP daemon uses only one core of a 36(?) core CPU. Same 
problem in the past with the EoIP daemon (not sure if they fixed it on the CCRs 
- they fixed it on x86).

Routerboards are nice and cool and to be honest: I'm a big fan of this stuff 
(also Ubiquiti). But with this boxes you're not able to ran a stable enterprise 
class carrier network with >99,5% uptime. And that’s thei MAIN reason why "the 
old shit" is still online :).

Hopefully my words explained my hard "you know nothing" blabla ?

Best regards


Jürgen Jaritsch
Head of Network & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300
Telefax: +43-5-0556-500

E-Mail: jjarit...@anexia-it.com 
Web: http://www.anexia-it.com 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

-Ursprüngliche Nachricht-
Von: NANOG [mailto:nanog-boun...@nanog.org] Im Auftrag von Mike Hammett
Gesendet: Freitag, 02. Oktober 2015 21:33
Cc: NANOG 
Betreff: Re: AW: /27 the new /24

Hrm. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


- Original Message -

From: "Jürgen Jaritsch"  
To: "Mike Hammett" , "NANOG"  
Sent: Friday, October 2, 2015 2:25:10 PM 
Subject: AW: /27 the new /24 

> Stop using old shit. 

Sorry, but the truth is: you have no idea about how earning revenue works and 
you obviously also have no idea about carrier grade networks. 




Jürgen Jaritsch 
Head of Network & Infrastructure 

ANEXIA Internetdienstleistungs GmbH 

Telefon: +43-5-0556-300 
Telefax: +43-5-0556-500 

E-Mail: jjarit...@anexia-it.com 
Web: http://www.anexia-it.com 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt 
Geschäftsführer: Alexander Windbichler 
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 

-Ursprüngliche Nachricht- 
Von: NANOG [mailto:nanog-boun...@nanog.org] Im Auftrag von Mike Hammett 
Gesendet: Freitag, 02. Oktober 2015 20:38 
An: NANOG  
Betreff: Re: /27 the new /24 

Chances are the revenue passing scales to some degree as well. Small business 
with small bandwidth needs buys small and has small revenue. Big business with 
big bandwidth needs buys big and has big revenue to support big router. 

I can think of no reason 

Re: AW: /27 the new /24

[Mike Hammett]

Hrm. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


- Original Message -

From: "Jürgen Jaritsch"  
To: "Mike Hammett" , "NANOG"  
Sent: Friday, October 2, 2015 2:25:10 PM 
Subject: AW: /27 the new /24 

> Stop using old shit. 

Sorry, but the truth is: you have no idea about how earning revenue works and 
you obviously also have no idea about carrier grade networks. 




Jürgen Jaritsch 
Head of Network & Infrastructure 

ANEXIA Internetdienstleistungs GmbH 

Telefon: +43-5-0556-300 
Telefax: +43-5-0556-500 

E-Mail: jjarit...@anexia-it.com 
Web: http://www.anexia-it.com 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt 
Geschäftsführer: Alexander Windbichler 
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 

-Ursprüngliche Nachricht- 
Von: NANOG [mailto:nanog-boun...@nanog.org] Im Auftrag von Mike Hammett 
Gesendet: Freitag, 02. Oktober 2015 20:38 
An: NANOG  
Betreff: Re: /27 the new /24 

Chances are the revenue passing scales to some degree as well. Small business 
with small bandwidth needs buys small and has small revenue. Big business with 
big bandwidth needs buys big and has big revenue to support big router. 

I can think of no reason why ten years goes by and you haven't had a need to 
throw out the old network for new. If your business hasn't scaled with the 
times, then you need to get rid of your Cat 6500 and get something more power, 
space, heat, etc. efficient. 


I saw someone replace a stack of Mikrotik CCRs with a pair of old Cisco 
routers. I don't know what they were at the moment, but they had GBICs, so they 
weren't exactly new. Each router had two 2500w power supplies. They'll be worse 
in every way (other than *possibly* BGP convergence). The old setup consumed at 
most 300 watts. The new setup requires $500/month in power... and is worse. 

Stop using old shit. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


- Original Message - 

From: "William Herrin"  
To: "Mike Hammett"  
Cc: "NANOG"  
Sent: Friday, October 2, 2015 1:09:16 PM 
Subject: Re: /27 the new /24 

On Fri, Oct 2, 2015 at 11:50 AM, Mike Hammett  wrote: 
> How many routers out there have this limitation? A $100 router 
> I bought ten years ago could manage many full tables. If 
> someone's network can't match that today, should I really have 
> any pity for them? 

Hi Mike, 

The technology doesn't work the way you think it does. Or more 
precisely, it only works the way you think it does on small (cheap) 
end-user routers. Those routers do everything in software on a 
general-purpose CPU using radix tries for the forwarding table (FIB). 
They don't have to (and can't) handle both high data rates and large 
routing tables at the same time. 

For a better understanding how the big iron works, check out 
https://www.pagiamtzis.com/cam/camintro/ . You'll occasionally see 
folks here talk about TCAM. This stands for Ternary Content 
Addressable Memory. It's a special circuit, different from DRAM and 
SRAM, used by most (but not all) big iron routers. The TCAM permits an 
O(1) route lookup instead of an O(log n) lookup. The architectural 
differences which balloon from there move the router cost from your 
$100 router into the hundreds of thousands of dollars. 

Your BGP advertisement doesn't just have to be carried on your $100 
router. It also has to be carried on the half-million-dollar routers. 
That makes it expensive. 

Though out of date, this paper should help you better understand the 
systemic cost of a BGP route advertisement: 
http://bill.herrin.us/network/bgpcost.html 

Regards, 
Bill Herrin 




-- 
William Herrin  her...@dirtside.com b...@herrin.us 
Owner, Dirtside Systems . Web: <http://www.dirtside.com/> 

AW: /27 the new /24

[Jürgen Jaritsch]

> Stop using old shit.

Sorry, but the truth is: you have no idea about how earning revenue works and 
you obviously also have no idea about carrier grade networks.




Jürgen Jaritsch
Head of Network & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300
Telefax: +43-5-0556-500

E-Mail: jjarit...@anexia-it.com 
Web: http://www.anexia-it.com 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

-Ursprüngliche Nachricht-
Von: NANOG [mailto:nanog-boun...@nanog.org] Im Auftrag von Mike Hammett
Gesendet: Freitag, 02. Oktober 2015 20:38
An: NANOG 
Betreff: Re: /27 the new /24

Chances are the revenue passing scales to some degree as well. Small business 
with small bandwidth needs buys small and has small revenue. Big business with 
big bandwidth needs buys big and has big revenue to support big router. 

I can think of no reason why ten years goes by and you haven't had a need to 
throw out the old network for new. If your business hasn't scaled with the 
times, then you need to get rid of your Cat 6500 and get something more power, 
space, heat, etc. efficient. 


I saw someone replace a stack of Mikrotik CCRs with a pair of old Cisco 
routers. I don't know what they were at the moment, but they had GBICs, so they 
weren't exactly new. Each router had two 2500w power supplies. They'll be worse 
in every way (other than *possibly* BGP convergence). The old setup consumed at 
most 300 watts. The new setup requires $500/month in power... and is worse. 

Stop using old shit. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


- Original Message -

From: "William Herrin"  
To: "Mike Hammett"  
Cc: "NANOG"  
Sent: Friday, October 2, 2015 1:09:16 PM 
Subject: Re: /27 the new /24 

On Fri, Oct 2, 2015 at 11:50 AM, Mike Hammett  wrote: 
> How many routers out there have this limitation? A $100 router 
> I bought ten years ago could manage many full tables. If 
> someone's network can't match that today, should I really have 
> any pity for them? 

Hi Mike, 

The technology doesn't work the way you think it does. Or more 
precisely, it only works the way you think it does on small (cheap) 
end-user routers. Those routers do everything in software on a 
general-purpose CPU using radix tries for the forwarding table (FIB). 
They don't have to (and can't) handle both high data rates and large 
routing tables at the same time. 

For a better understanding how the big iron works, check out 
https://www.pagiamtzis.com/cam/camintro/ . You'll occasionally see 
folks here talk about TCAM. This stands for Ternary Content 
Addressable Memory. It's a special circuit, different from DRAM and 
SRAM, used by most (but not all) big iron routers. The TCAM permits an 
O(1) route lookup instead of an O(log n) lookup. The architectural 
differences which balloon from there move the router cost from your 
$100 router into the hundreds of thousands of dollars. 

Your BGP advertisement doesn't just have to be carried on your $100 
router. It also has to be carried on the half-million-dollar routers. 
That makes it expensive. 

Though out of date, this paper should help you better understand the 
systemic cost of a BGP route advertisement: 
http://bill.herrin.us/network/bgpcost.html 

Regards, 
Bill Herrin 




-- 
William Herrin  her...@dirtside.com b...@herrin.us 
Owner, Dirtside Systems . Web:  

AW: /27 the new /24

[Jürgen Jaritsch]

Welcome to the real world ...

Cisco SUP720-3BXL
Cisco RSP720-3BXL

and even the new and shiny SUP2T only supports 1 Mio routes (dicvided to IPv4 
MPLS, IPv4 VRF, IPv4 global routes, etc).

I guess this is still the truth: there are at least a few ten thousand of these 
devices running big parts of the internet. Take a look at some big players 
network - e.g. Level3. Their customer access routers in Slovakia, Austria and 
Germany are still based on the Cisco 6500/7600 platform.

Of course there are many other vendors and platforms available which do NOT 
have this limitations. But there are also at least a ton of vendors on the 
market with exactly the same limitation :(.


best regards


Jürgen Jaritsch
Head of Network & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300
Telefax: +43-5-0556-500

E-Mail: jjarit...@anexia-it.com 
Web: http://www.anexia-it.com 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601


-Ursprüngliche Nachricht-
Von: NANOG [mailto:nanog-boun...@nanog.org] Im Auftrag von Mike Hammett
Gesendet: Freitag, 02. Oktober 2015 17:51
Cc: NANOG
Betreff: Re: /27 the new /24

How many routers out there have this limitation? A $100 router I bought ten 
years ago could manage many full tables. If someone's network can't match that 
today, should I really have any pity for them? 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


- Original Message -

From: "Matthew Kaufman"  
To: "Mike Hammett"  
Cc: "NANOG"  
Sent: Friday, October 2, 2015 10:48:29 AM 
Subject: Re: /27 the new /24 

Cheaper than buying everyone TCAM 

Matthew Kaufman 

(Sent from my iPhone) 

> On Oct 2, 2015, at 8:32 AM, Mike Hammett  wrote: 
> 
> Much m ore than I'm willing to spend. ;-) 
> 
> 
> 
> 
> - 
> Mike Hammett 
> Intelligent Computing Solutions 
> http://www.ics-il.com 
> 
> 
> 
> Midwest Internet Exchange 
> http://www.midwest-ix.com 
> 
> 
> - Original Message - 
> 
> From: "Matthew Kaufman"  
> To: "Justin Wilson - MTIN"  
> Cc: "NANOG"  
> Sent: Friday, October 2, 2015 9:48:33 AM 
> Subject: Re: /27 the new /24 
> 
> A /24 isn't that expensive yet... 
> 
> Matthew Kaufman 
> 
> (Sent from my iPhone) 
> 
>> On Oct 2, 2015, at 7:32 AM, Justin Wilson - MTIN  wrote: 
>> 
>> I was in a discussion the other day and several Tier2 providers were talking 
>> about the idea of adjusting their BGP filters to accept prefixes smaller 
>> than a /24. A few were saying they thought about going down to as small as a 
>> /27. This was mainly due to more networks coming online and not having even 
>> a /24 of IPv4 space. The first argument is against this is the potential 
>> bloat the global routing table could have. Many folks have worked hard for 
>> years to summarize and such. others were saying they would do a /26 or 
>> bigger. 
>> 
>> However, what do we do about the new networks which want to do BGP but only 
>> can get small allocations from someone (either a RIR or one of their 
>> upstreams)? 
>> 
>> Just throwing that out there. Seems like an interesting discussion. 
>> 
>> 
>> Justin Wilson 
>> j...@mtin.net 
>> 
>> --- 
>> http://www.mtin.net Owner/CEO 
>> xISP Solutions- Consulting – Data Centers - Bandwidth 
>> 
>> http://www.midwest-ix.com COO/Chairman 
>> Internet Exchange - Peering - Distributed Fabric 
> 

AW: /27 the new /24

[Jürgen Jaritsch]

Hi,

this would at least help to get rid of many old routing engines around the 
world :) ... or people would keep their "learn nothing smaller than /24" 
filters in place. Also an option - but not for companies who act as an IP 
transit provider.


best regards

Jürgen Jaritsch
Head of Network & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300
Telefax: +43-5-0556-500

E-Mail: jjarit...@anexia-it.com 
Web: http://www.anexia-it.com 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601


-Ursprüngliche Nachricht-
Von: NANOG [mailto:nanog-boun...@nanog.org] Im Auftrag von Justin Wilson - MTIN
Gesendet: Freitag, 02. Oktober 2015 16:32
An: NANOG
Betreff: /27 the new /24

I was in a discussion the other day and several Tier2 providers were talking 
about the idea of adjusting their BGP filters to accept prefixes smaller than a 
/24.  A few were saying they thought about going down to as small as a /27.  
This was mainly due to more networks coming online and not having even a /24 of 
IPv4 space.  The first argument is against this is the potential bloat the 
global routing table could have.  Many folks have worked hard for years to 
summarize and such. others were saying they would do a /26 or bigger.  

However, what do we do about the new networks which want to do BGP but only can 
get small allocations from someone (either a RIR or one of their upstreams)?

Just throwing that out there. Seems like an interesting discussion.


Justin Wilson
j...@mtin.net

---
http://www.mtin.net Owner/CEO
xISP Solutions- Consulting – Data Centers - Bandwidth

http://www.midwest-ix.com  COO/Chairman
Internet Exchange - Peering - Distributed Fabric