Re: MPLS in the campus Network?

[Mark Tinka]

On 24/Oct/16 22:13, Wayne Bouchard wrote:

> If the reason for L2 transport is purely customer driven and purely
> ptp, then a L2 VPN solution would be better than directly transporting
> the frames. If you don't have to bridge it directly, don't. Keep the
> core at layer 3 wherever possible. L2 can be very hard to debug when
> there are issues.

Not sure I understand - are you advocating for EoMPLS (l2vpn) over
EoDWDM more often than not, if possible?.

Mark.

Re: MPLS in the campus Network?

[Wayne Bouchard]

If the reason for L2 transport is purely customer driven and purely
ptp, then a L2 VPN solution would be better than directly transporting
the frames. If you don't have to bridge it directly, don't. Keep the
core at layer 3 wherever possible. L2 can be very hard to debug when
there are issues.

On Thu, Oct 20, 2016 at 06:58:51PM +0200, Mark Tinka wrote:
> 
> 
> On 20/Oct/16 18:45, Roland Dobbins wrote:
> 
> >
> > Sure - but it's probably worth revisiting the origins of those
> > requirements, and whether there are better alternatives.
> 
> Indeed.
> 
> What we've seen is customers who prefer to manage their own IP layer,
> and just need transport. These types of customers tend to be split
> between EoDWDM and EoMPLS preferences. Whatever the case, their primary
> requirement is control of their IP domain.
> 
> What we're not seeing anymore is l3vpn requirements, particularly on the
> back of on-premise IT infrastructure moving into the cloud. We see this
> driving a lot of regular IP growth.
> 
> Mark.

---
Wayne Bouchard
w...@typo.org
Network Dude
http://www.typo.org/~web/

Re: MPLS in the campus Network?

[Mark Tinka]

On 22/Oct/16 23:59, Marian Ďurkovič wrote:

>
> The question here is, whether MPLS is the *optimal* solution for campus needs.
>
> The same functionality could be obviously achived by multiple technologies,
> and while MPLS is well supported on high-end SP routers, various limitations
> appear when people try to use it on commodity ASICs which typically empower
> today's ethernet switches - one of them being e.g. limited ability to
> effectively load-balance traffic over multiple parallel links.
>
> Yes, in theory we could build all campus LANs using high-end SP routers, but
> when 100GE backbone is desired (which is often the case in EDU/NREN sector), 
> the costs of such solution jump to unacceptable heights.
>
> Thus we looked for another technology, which doesn't have the usual L2 
> problems
> and is able to provide services we need (including L2 extensions to remote
> campuses) at reasonable costs and with enough simplicity. 
>
> To avoid typical L2 problems, you clearly need a solution based on L3 routing.
> And TRILL is exactly that - although it maintains L2 interface to the outside
> world, internally it performs dynamic L3 routing by IS-IS protocol with all
> safety belts like TTL check, RPF check etc. 
>
> IMHO, TRILL is much better fit for campus needs, since it was specifically
> designed for this networking space - and our 6-months production fully 
> confirms
> that view (of course, YMMV).

I don't consider the ASR920 or CES2000 to be particularly high-end
routers, but YMMV.

True, merchant silicon presents a number of data plane challenges that
may, at first, seem non-trivial or completely go unnoticed. That is why
we stay away from the ACX5000, for example. I expect improvements to
come with newer-generation ASIC's/NP's, but that tests one's patience.

But, like I said, I have not run TRILL myself, so I'm not going to tell
you that it's not an ideal technology for this use-case. All I'll say is
that MPLS is not limited to high-end platforms, even when custom silicon
is involved.

Mark.

Re: MPLS in the campus Network?

[Marian Ďurkovič]

On Sat, 22 Oct 2016 21:29:22 +0200, Mark Tinka wrote 
> On 21/Oct/16 19:02, Javier Solis wrote:
> > With that said, what are the best options to be able to cost effectively
> > scale without using vlans and maintaining a routed core? What technology 
> > would someone suggest (mpls, vxlan,etc) to be the best possible solution?
  
> IME, MPLS is a good use-case here. If you are going to use the same /24 (or
> whatever prefix applies to you) across multiple locations, you will need some
> kind of overlay. Be it IP-in-IP, GRE, MPLS (l2vpn's or l3vpn's) or plain old >
Ethernet, you will need something.
>
> MPLS makes a lot of sense to me. It's native in hardware, upper-layer
> agnostic, mature, and reasonably affordable even at low scale.

The question here is, whether MPLS is the *optimal* solution for campus needs.

The same functionality could be obviously achived by multiple technologies,
and while MPLS is well supported on high-end SP routers, various limitations
appear when people try to use it on commodity ASICs which typically empower
today's ethernet switches - one of them being e.g. limited ability to
effectively load-balance traffic over multiple parallel links.

Yes, in theory we could build all campus LANs using high-end SP routers, but
when 100GE backbone is desired (which is often the case in EDU/NREN sector), 
the costs of such solution jump to unacceptable heights.

Thus we looked for another technology, which doesn't have the usual L2 problems
and is able to provide services we need (including L2 extensions to remote
campuses) at reasonable costs and with enough simplicity. 

To avoid typical L2 problems, you clearly need a solution based on L3 routing.
And TRILL is exactly that - although it maintains L2 interface to the outside
world, internally it performs dynamic L3 routing by IS-IS protocol with all
safety belts like TTL check, RPF check etc. 

IMHO, TRILL is much better fit for campus needs, since it was specifically
designed for this networking space - and our 6-months production fully confirms
that view (of course, YMMV).


   With kind regards,

   M.

Re: MPLS in the campus Network?

[Mark Tinka]

On 21/Oct/16 19:02, Javier Solis wrote:

> With that said, what are the best options to be able to cost
> effectively scale without using vlans and maintaining a routed core?
> What technology would someone suggest (mpls, vxlan,etc) to be the best
> possible solution?
>

IME, MPLS is a good use-case here. If you are going to use the same /24
(or whatever prefix applies to you) across multiple locations, you will
need some kind of overlay. Be it IP-in-IP, GRE, MPLS (l2vpn's or
l3vpn's) or plain old Ethernet, you will need something.

MPLS makes a lot of sense to me. It's native in hardware, upper-layer
agnostic, mature, and reasonably affordable even at low scale. While I
would not say MPLS is simple from a "complexity in your network"
standpoint, it does provide a notable amount of simplicity when you're
looking for an overlay that is transparent to all manner of Layer 2 and
Layer 3 applications that a packet-based network needs to transport.

Mark.

Re: MPLS in the campus Network?

[David Bass]

This is exactly what we are recommending and building for our customers in that 
space. Most of the time the university network acts as a provider, so to me it 
only makes sense to use that type of tech.  The biggest problem then is 
support, which could be something they are unwilling or unable to overcome. 

> On Oct 21, 2016, at 1:45 PM, Leo Bicknell  wrote:
> 
> In a message written on Fri, Oct 21, 2016 at 12:02:24PM -0500, Javier Solis 
> wrote:
>> In a campus network the challenge becomes extending subnets across your
>> core. You may have a college that started in one building with their own
>> /24, but now have offices and labs in other buildings. They want to stay on
>> the same network, but that's not feasible with the routed core setup
>> without some other technology overlay. We end up not being able to extend
>> the L2 like we did in the past and today we modify router ACL's to allow
>> communications. If you already have hundreds of vlans spanned across the
>> network, it's hard to get a campus to migrate to the routed core. I think
>> this may be one of Marks challenge, correct me if I'm wrong please.
> 
> FWIW, if I had to solve the "college across buildings with common
> access control" problem I would create MPLS L3 VPN's, one subnet
> per building (where it is a VLAN inside of a building), with a
> "firewall in the cloud" somewhere to get between VLAN's with all
> of the policy in one place.
> 
> No risk of the L2 across buildings mess, including broadcast and
> multicast issues at L2.  All tidy L3 routing.  Can use a real
> firewall between L3 VPN instances to get real policy tools (AV, URL
> Filtering, Malware detection, etc) rather than router ACL's.  Scales
> to huge sizes because it's all L3 based.
> 
> Combine with 802.1x port authentication and NAC, and in theory every
> L3 VPN could be in every building, with each port dynamically assigning
> the VLAN based on the user's login!  Imagine never manually configuring
> them again.  Write a script that makes all the colleges (20? 40? 60?)
> appear in every building all attached to their own MPLS VPN's, and
> then the NAC handles port assignment.
> 
> -- 
> Leo Bicknell - bickn...@ufp.org
> PGP keys at http://www.ufp.org/~bicknell/

Re: MPLS in the campus Network?

[James R Cutler]

On Oct 21, 2016, at 4:18 PM, Youssef Ghorbal  wrote, 
in part:
> 
> Until people start complaining they can no more auto discover their
> Time Capsule left in the other building whereas their colleagues in
> the other building can etc etc. All fancy discover protocols breaks
> without L2 continuity !


Minor Correction:  Correctly configured*, an Airport Extreme basestation (Time 
Capsule or not) does not require L2 connectivity to discover. In fact, Wide 
Area is used for discovery of many services not necessarily reachable by L2 
connectivity. Apple’s Back to My Mac service is one example.

*Apple’s "Back to My Mac” Wide Area Bonjour is enabled on an Airport 
basestation by entering appropriate Apple ID and password data in the Base 
Station tab as accessed by Airport Utility.


James R. Cutler
james.cut...@consultant.com
PGP keys at http://pgp.mit.edu

Re: MPLS in the campus Network?

[Youssef Ghorbal]

> FWIW, if I had to solve the "college across buildings with common
> access control" problem I would create MPLS L3 VPN's, one subnet
> per building (where it is a VLAN inside of a building), with a
> "firewall in the cloud" somewhere to get between VLAN's with all
> of the policy in one place.
>
> No risk of the L2 across buildings mess, including broadcast and
> multicast issues at L2.  All tidy L3 routing.  Can use a real
> firewall between L3 VPN instances to get real policy tools (AV, URL
> Filtering, Malware detection, etc) rather than router ACL's.  Scales
> to huge sizes because it's all L3 based.

Until people start complaining they can no more auto discover their
Time Capsule left in the other building whereas their colleagues in
the other building can etc etc. All fancy discover protocols breaks
without L2 continuity !
Welcome to the campus network nightmare :)
For now, there is no perfect solution ! either you cope with L2 hell
or users inconvenience (and yes people tend to think that the campus
network is expected to work as their home network)

I've also stumbled upon some "Building Automation and Control
Networks" (BACnet/IP for instance) where each building has some
automats that all needs to be in the same network segment.

> Combine with 802.1x port authentication and NAC, and in theory every
> L3 VPN could be in every building, with each port dynamically assigning
> the VLAN based on the user's login!  Imagine never manually configuring
> them again.  Write a script that makes all the colleges (20? 40? 60?)
> appear in every building all attached to their own MPLS VPN's, and
> then the NAC handles port assignment.

Here again, it's perfect until you start coping with old stuff, all
fancy new ethernet capable "things" or scientific/industrial
equipments. The "802.1x what ? it's plug'n play man !" attitude.

(my experience is with research institutes/academy kind of campuses)

Youssef Ghorbal

Re: MPLS in the campus Network?

[Leo Bicknell]

In a message written on Fri, Oct 21, 2016 at 12:02:24PM -0500, Javier Solis 
wrote:
> In a campus network the challenge becomes extending subnets across your
> core. You may have a college that started in one building with their own
> /24, but now have offices and labs in other buildings. They want to stay on
> the same network, but that's not feasible with the routed core setup
> without some other technology overlay. We end up not being able to extend
> the L2 like we did in the past and today we modify router ACL's to allow
> communications. If you already have hundreds of vlans spanned across the
> network, it's hard to get a campus to migrate to the routed core. I think
> this may be one of Marks challenge, correct me if I'm wrong please.

FWIW, if I had to solve the "college across buildings with common
access control" problem I would create MPLS L3 VPN's, one subnet
per building (where it is a VLAN inside of a building), with a
"firewall in the cloud" somewhere to get between VLAN's with all
of the policy in one place.

No risk of the L2 across buildings mess, including broadcast and
multicast issues at L2.  All tidy L3 routing.  Can use a real
firewall between L3 VPN instances to get real policy tools (AV, URL
Filtering, Malware detection, etc) rather than router ACL's.  Scales
to huge sizes because it's all L3 based.

Combine with 802.1x port authentication and NAC, and in theory every
L3 VPN could be in every building, with each port dynamically assigning
the VLAN based on the user's login!  Imagine never manually configuring
them again.  Write a script that makes all the colleges (20? 40? 60?)
appear in every building all attached to their own MPLS VPN's, and
then the NAC handles port assignment.

-- 
Leo Bicknell - bickn...@ufp.org
PGP keys at http://www.ufp.org/~bicknell/


pgpdxSVr3MRkH.pgp
Description: PGP signature

Re: MPLS in the campus Network?

[Javier Solis]

Our campus started off with L2 vlans spanning through the core, but we
migrated to routing in the core and moved our many spanning tree/broadcast
domains to the edge of buildings fronted by redundant routing with ecmp to
a redundant core utilizing ospf.

In a campus network the challenge becomes extending subnets across your
core. You may have a college that started in one building with their own
/24, but now have offices and labs in other buildings. They want to stay on
the same network, but that's not feasible with the routed core setup
without some other technology overlay. We end up not being able to extend
the L2 like we did in the past and today we modify router ACL's to allow
communications. If you already have hundreds of vlans spanned across the
network, it's hard to get a campus to migrate to the routed core. I think
this may be one of Marks challenge, correct me if I'm wrong please.

With that said, what are the best options to be able to cost effectively
scale without using vlans and maintaining a routed core? What technology
would someone suggest (mpls, vxlan,etc) to be the best possible solution?

Thank you to the participants in the discussion. I always enjoy reading
comments posted.

-Javier

On Oct 21, 2016 11:46 AM, "Mark Tinka"  wrote:

>
>
> On 21/Oct/16 16:19, Marian Ďurkovič wrote:
>
> >
> > Much easier to setup, operate & maintain than MPLS and obviously much
> > lower cost. Based on 6-months production experience, my recommendation
> > would be to stay away from MPLS in the campus.
>
> I'd be curious to hear what MPLS-specific issues you faced in the 6
> months you had to operate such a network.
>
> Been running IP/MPLS Core, Edge and Access networks for over 15 years,
> and apart from bugs which affect any protocol or feature implementation,
> I can't say it has been a nightmare to operate to the point of not
> recommending it.
>
> I have far fewer words to say about STP, although - I'll admit - I've
> never run TRILL.
>
> Mark.
>

Re: MPLS in the campus Network?

[Mark Tinka]

On 21/Oct/16 16:19, Marian Ďurkovič wrote:

>
> Much easier to setup, operate & maintain than MPLS and obviously much
> lower cost. Based on 6-months production experience, my recommendation
> would be to stay away from MPLS in the campus.

I'd be curious to hear what MPLS-specific issues you faced in the 6
months you had to operate such a network.

Been running IP/MPLS Core, Edge and Access networks for over 15 years,
and apart from bugs which affect any protocol or feature implementation,
I can't say it has been a nightmare to operate to the point of not
recommending it.

I have far fewer words to say about STP, although - I'll admit - I've
never run TRILL.

Mark.

Re: MPLS in the campus Network?

[Marian Ďurkovič]

> Compared to MPLS, a L2 solution with 100 Gb/s interfaces between
> core switches and a 10G connection for each buildings looks so much
> cheaper. But we worry about future trouble using Trill, SPB, or other
> technologies, not only the "open" ones, but specifically the proprietary
> ones based on central controller and lots of magic (some colleagues feel
> the debug nightmare are garanteed).
> 
> If you had to make such a choice recently, did you choose an MPLS design
> even at lower speed ?

A year ago we built NREN backbone using TRILL instead of MPLS.
40 POPs, no central controller, RFC standardized TRILL protocol
i.e. L2 routing using IS-IS, no STP.

See my recent presentation at  http://md.bts.sk/sanet-100g-2.pdf 
for more details.

Much easier to setup, operate & maintain than MPLS and obviously much
lower cost. Based on 6-months production experience, my recommendation
would be to stay away from MPLS in the campus.


  With kind regards,

 M.

Re: MPLS in the campus Network?

[Nick Hilliard]

Mark Tinka wrote:
> Not sure what gear you're using now, but you'll get full routing and
> MPLS features on the platforms such as the Cisco ASR920. I'd have
> recommended the Cisco ME3600X, but they just announced EoS/EoL last
> night, which means that while you can still order it until October 2017,
> the ASR920 will be cheaper and is the future of of Metro-E from Cisco in
> this segment.

the ME3600x is mostly fine for this sort of thing, with the exception of
its control plane policing mechanism which is badly limited in a number
of important ways, not least the fact that its traffic categorisation
buckets are non-configurable and not what you might necessarily want for
customer edge service.

Nick

Re: MPLS in the campus Network?

[Mark Tinka]

On 20/Oct/16 18:45, Roland Dobbins wrote:

>
> Sure - but it's probably worth revisiting the origins of those
> requirements, and whether there are better alternatives.

Indeed.

What we've seen is customers who prefer to manage their own IP layer,
and just need transport. These types of customers tend to be split
between EoDWDM and EoMPLS preferences. Whatever the case, their primary
requirement is control of their IP domain.

What we're not seeing anymore is l3vpn requirements, particularly on the
back of on-premise IT infrastructure moving into the cloud. We see this
driving a lot of regular IP growth.

Mark.

Re: MPLS in the campus Network?

[Roland Dobbins]

On 20 Oct 2016, at 23:32, Mark Tinka wrote:


Some requirements call for Ethernet transport as opposed to IP.


Sure - but it's probably worth revisiting the origins of those 
requirements, and whether there are better alternatives.


---
Roland Dobbins 

Re: MPLS in the campus Network?

[Mark Tinka]

On 20/Oct/16 18:29, Jason Lixfeld wrote:

> Likely not at the PE, true, but he did say Internet access, so I err’d on the 
> side of assuming DFZ, somewhere.  If that assumption is true, FIB resources 
> for the SP interconnect nodes and filtering towards the PEs, absolutely..

I assumed 0/0 + ::/0 to upstreams, but if it had to be the DFZ, you can
still hold a full table in RAM but install just default and some select
entries into FIB.

This would only be necessary if there are downstreams that require the
full table. However, if it's for internal use, and a certain amount of
DFZ-related traffic engineering is required, the OP could make the case
for just having the border routers as the boxes that can handle a full
feed. This could be just one or two routers.

That would leave the majority of the network running cheaper routers
with smaller FIB's.

But without really knowing the OP's Internet routing design needs, it
could go either way.

Mark.

Re: MPLS in the campus Network?

[Mark Tinka]

On 20/Oct/16 18:24, Roland Dobbins wrote:

>
> And I'd definitely recommend figuring out why that's being done so
> broadly today, and working to reduce its prevalence and scope, moving
> forward.

Some requirements call for Ethernet transport as opposed to IP.

I don't know the details of the OP's user requirements, but from our
side, we have as many customers asking for IP services as they do Ethernet.

Mark.

Re: MPLS in the campus Network?

[Jason Lixfeld]

> On Oct 20, 2016, at 12:23 PM, Mark Tinka  wrote:
> 
> 
> 
> On 20/Oct/16 17:12, Jason Lixfeld wrote:
> 
>> 
>> It’s only more expensive the more big vendor products you use.  Sometimes 
>> you need to (i.e.: Boxes with big RIB/FIBs for DFZ, or deep buffers), but 
>> more and more, people are looking to OCP/White Box Switches [1][2].
> 
> It doesn't sound like the OP needs massive FIB space, so he could implement 
> FIB filtering and run the smaller boxes that have all the features but lack 
> the FIB real estate of the larger routers/switches.
> 
> This is what we do for our Metro-E Access networks.
> 
> Mark.

Likely not at the PE, true, but he did say Internet access, so I err’d on the 
side of assuming DFZ, somewhere.  If that assumption is true, FIB resources for 
the SP interconnect nodes and filtering towards the PEs, absolutely.

Re: MPLS in the campus Network?

[Roland Dobbins]

On 20 Oct 2016, at 23:17, Mark Tinka wrote:


especially looking at how much Layer 2 traffic you're hauling around.


And I'd definitely recommend figuring out why that's being done so 
broadly today, and working to reduce its prevalence and scope, moving 
forward.


---
Roland Dobbins 

Re: MPLS in the campus Network?

[Mark Tinka]

On 20/Oct/16 17:12, Jason Lixfeld wrote:

>
> It’s only more expensive the more big vendor products you use.  Sometimes you 
> need to (i.e.: Boxes with big RIB/FIBs for DFZ, or deep buffers), but more 
> and more, people are looking to OCP/White Box Switches [1][2].

It doesn't sound like the OP needs massive FIB space, so he could
implement FIB filtering and run the smaller boxes that have all the
features but lack the FIB real estate of the larger routers/switches.

This is what we do for our Metro-E Access networks.

Mark.

Re: MPLS in the campus Network?

[Mark Tinka]

On 20/Oct/16 17:05, Leo Bicknell wrote:

> I would challenge your port cost assumption for "routers".  For
> instance the Arista 7280 could deliver can be had with 48 10GE SFP+
> ports with full Internet routing capabilities.  If you're used
> to Cisco or Juniper, it is worth looking further afield these days.

We're currently looking at Arista's development in the IP/MPLS space.
While we are not yet ready to spend money on them for that, we think the
future is bright. So we are working very closely with them to get their
IP and MPLS code to where it would make sense for us. I'm hopeful we
shall be investing in Arista for routing in the very near future.

But we love them for switching, and will be activating some of their
platforms for that use-case very soon.

Mark.


signature.asc
Description: OpenPGP digital signature

Re: MPLS in the campus Network?

[Mark Tinka]

On 20/Oct/16 15:43, steven brock wrote:

>
> If you had to make such a choice recently, did you choose an MPLS design
> even at lower speed ?
> How would you convince your management that MPLS is the best solution for
> your campus network ? How would you justify the cost or speed difference ?

IP/MPLS would be my recommendation.

From an operational perspective, running it in your Core and Access
backbone removes any need for STP (and all the associated headache).

Not sure what gear you're using now, but you'll get full routing and
MPLS features on the platforms such as the Cisco ASR920. I'd have
recommended the Cisco ME3600X, but they just announced EoS/EoL last
night, which means that while you can still order it until October 2017,
the ASR920 will be cheaper and is the future of of Metro-E from Cisco in
this segment.

Brocade's CES 2000 platform would also be a good choice here.

Juniper ACX5000 presents some challenges re: the use of that Broadcom
chip, although I know a number of operators that have had the courage to
deploy it for this use-case.

Ultimately, whatever vendor you choose, the guaranteed way to not get
that 3AM call is to run IP/MPLS for your Core and Access backbones,
especially looking at how much Layer 2 traffic you're hauling around.

Mark.

Re: MPLS in the campus Network?

[Jason Lixfeld]

Hi,

> On Oct 20, 2016, at 9:43 AM, steven brock  wrote:
> 
> Compared to MPLS, a L2 solution with 100 Gb/s interfaces between
> core switches and a 10G connection for each buildings looks so much
> cheaper. But we worry about future trouble using Trill, SPB, or other
> technologies, not only the "open" ones, but specifically the proprietary
> ones based on central controller and lots of magic (some colleagues feel
> the debug nightmare are garanteed).

From my perspective, in this day and age, no service provider or campus should 
really be using any sort of layer 2 protection mechanism in their backbone, if 
they can help it.

> If you had to make such a choice recently, did you choose an MPLS design
> even at lower speed ?

Yup.  5 or so years ago, and never looked back.  Actually, this was in 
conjunction with upgrading our 1G backbone to a 10G backbone, so it was an 
upgrade for us in all senses of the word.

> How would you convince your management that MPLS is the best solution for
> your campus network ?

You already did:


> We are not satisfied with the current backbone design ; we had our share
> of problems in the past:
> - high CPU load on the core switches due to multiple instances of spanning
> tree slowly converging when a topology change happens (somehow fixed
> with a few instances of MSTP)
> - spanning tree interoperability problems and spurious port blocking
> (fixed by BPDU filtering)
> - loops at the edge and broadcast/multicast storms (fixed with traffic
> limits and port blocking based on threshhold)
> - some small switches at the edge are overloaded with large numbers of
> MAC addresses (fixed with reducing broadcast domain size and subnetting)
> 
> This architecture doesn't feel very solid.
> Even if the service provisionning seems easy from an operational point
> of view (create a VLAN and it is immediately available at any point of the
> L2 backbone), we feel the configuration is not always consistent.
> We have to rely on scripts pushing configuration elements and human
> discipline (and lots of duct-tape, especially for QoS and VRFs).



> How would you justify the cost or speed difference ?

It’s only more expensive the more big vendor products you use.  Sometimes you 
need to (i.e.: Boxes with big RIB/FIBs for DFZ, or deep buffers), but more and 
more, people are looking to OCP/White Box Switches [1][2].

For example, assuming a BCM Trident II based board with 48 SFP+ cages and 6 
QSFP+ cages, you get a line-rate, MPLS capable 10G port for $65.  Or, if you’re 
like me and hate the idea of breakout cables, you’re at about $100/SFP+ cage, 
at which points the QSPF+ cages are pretty much free.

Software wise, there are lots of vendors.  One that I like is IPInfusion’s 
OcNOS[3] codebase.  They are putting a lot of resources into building a service 
provider feature set (full-blown MPLS/VPLS/EVPN, etc.) for OCP switches.  There 
are others, but last time I looked a couple of years ago, they were less 
focused on MPLS and more focused on SDN: Cumulus Networks[4], PICA8[5], Big 
Switch Networks[6].

> Thanks for your insights!

[1] 
https://www.linx.net/communications/press-releases/lon2-revolutionary-development
[2] 
http://www.ipinfusion.com/about/press/london-internet-exchange-use-ip-infusion’s-ocnos™-network-operating-system-new-london-in
[3] http://www.ipinfusion.com/products/ocnos
[4] https://cumulusnetworks.com
[5] http://www.pica8.com
[6] http://www.bigswitch.com

Re: MPLS in the campus Network?

[Leo Bicknell]

From what you describe I do think you have many options, including
more than just the ones you laid out.  When you're under 10km and
own your own fiber the possibilities are virtually limitless.

First off, you don't want to be running spanning tree across a
campus.  While I don't think you need to elminate it completely as
some in the industry are pressing, doing it at the scale you describe
is probably a world of hurt.

I would challenge your port cost assumption for "routers".  For
instance the Arista 7280 could deliver can be had with 48 10GE SFP+
ports with full Internet routing capabilities.  If you're used
to Cisco or Juniper, it is worth looking further afield these days.

I would also challenge that there is one way to do the job.  It may
be easier to build a couple of networks.  Perhaps a router based one
to deliver IP services, and a separate "Metro Ethernet" network to
deliver L2 VLAN transport.  It may sound crazy that buying two
boxes is chepaer than one, but it can be depending on the exact
scale and port count.  Heck, depending on your port count doing
passive DWDM to interconnect switches in each office may be cheaper
than encapsulating in MPLS.  A lot of it also depends on your 
monitoring requirements, or lack of.

In a message written on Thu, Oct 20, 2016 at 03:43:26PM +0200, steven brock 
wrote:
> How would you convince your management that MPLS is the best solution for
> your campus network ? How would you justify the cost or speed difference ?

Well, cost and speed are two prime considerations, but there are other
important considerations.

Vendors support platforms and features based on the customer base.
If you buy a box everyone does MPLS on, and then use it for TRILL,
you'll be in a world of hurt.  Particularly if you want long, stable
life ride with the crowd.  Use a platform many others are using for
the same job.

-- 
Leo Bicknell - bickn...@ufp.org
PGP keys at http://www.ufp.org/~bicknell/


pgpALqdUJoKza.pgp
Description: PGP signature

MPLS in the campus Network?

[steven brock]

Dear NANOG members,

We operate a campus network reaching more than 100 buildings on 5 campuses.
We also operate a regional backbone and the interconnexion to our NREN.
The current architecture is made of a L2 backbone and a few routers.
Most of the buildings are connected with a 1 Gb/s link using our own
optical fiber
(only a few building are connected at 10 Gb/s).
In a smaller number of buildings (a few dozens), we also operate the
internal network, made of ethernet switches (in a multi-vendor environment).
In each building, we provide at least an edge switch, marking the boundary
between
us and the customer, where we deliver the different services on ethernet
ports.

The services we currently offer:
- L2 interconnections (400 vlans are present in 2 buildings or more;
only a few VLANs are present in more than 30 buildings)
- IPv4 et IPv6 routing (hundreds of subnets) and Internet access,
- specific interconnections (ex: terminating a VPN to the customer,
say a national private infrastructure delivered by the NREN through
MPLS L2/L3VPN and stitched to the customer network using a specific VLAN)
- routing isolation using routing instances (~ VRF Lite) : only 5
instances, but we could have more,
- routing and filtering using open source firewalls running on servers
in our DCs (less than 15 platforms, as most customer operate their
own firewall),
- user authentication,
- shared VPN platform allowing direct access for an identified user into
the customer network (based on radius attribute) - this platform uses
VLANs to interconnect to the rest of the network,
- wireless LAN, also allowing direct access for an identified user into
the customer network ; the platform is a centralized controller, and
it uses VLAN to interconnect to the rest of the network.
(those last two services could use just a VLAN or a dedicated subnet
delivered on a port of the edge switch which is then connected to the
customer firewall)

We are not satisfied with the current backbone design ; we had our share
of problems in the past:
- high CPU load on the core switches due to multiple instances of spanning
tree slowly converging when a topology change happens (somehow fixed
with a few instances of MSTP)
- spanning tree interoperability problems and spurious port blocking
(fixed by BPDU filtering)
- loops at the edge and broadcast/multicast storms (fixed with traffic
limits and port blocking based on threshhold)
- some small switches at the edge are overloaded with large numbers of
MAC addresses (fixed with reducing broadcast domain size and subnetting)

This architecture doesn't feel very solid.
Even if the service provisionning seems easy from an operational point
of view (create a VLAN and it is immediately available at any point of the
L2 backbone), we feel the configuration is not always consistent.
We have to rely on scripts pushing configuration elements and human
discipline (and lots of duct-tape, especially for QoS and VRFs).

We are re-designing our network architecture.
We have enough fiber to imagine many ways to link the core network
devices.
We find MPLS has its merit as a platform, to bring all the network services
we currently provide (L2, L3 VPN, VPLS, and soon EVPN)
However, we also want to upgrade the infrastructure to allow future growth
of the traffic. Some labs, especially in physics, could need more than 10
Gb/s
in the coming years. Our cycles of evolution are long (we keep a backbone
technology
for 8 years). MPLS is definitely not cheap considering the price of a 10G
or 100G
router interface.

Compared to MPLS, a L2 solution with 100 Gb/s interfaces between
core switches and a 10G connection for each buildings looks so much
cheaper. But we worry about future trouble using Trill, SPB, or other
technologies, not only the "open" ones, but specifically the proprietary
ones based on central controller and lots of magic (some colleagues feel
the debug nightmare are garanteed).

If you had to make such a choice recently, did you choose an MPLS design
even at lower speed ?
How would you convince your management that MPLS is the best solution for
your campus network ? How would you justify the cost or speed difference ?

Thanks for your insights!