Re: [uknof] 'White Box' switching and OS - Any experiences worth sharing?

[James Bensley]

On 17 February 2018 at 10:11, Neil J. McRae  wrote:
> Deployed no but just a matter of time - I doubt there are huge latency 
> benefits that you don’t already see from the current plethora of optimised DC 
> switches and code that’s available and I’m totally unconvinced that low 
> buffer solutions are right for anything outside of the DC with today’s OTT 
> steaming needs mixed with a lot of the access technology in cable, DSL and 
> FTTP.
>
> Our requirement is only to run code we know we need to run- simplifies 
> security risks and stupid bugs in things we don’t use and allows focused 
> testing and telemetry in the end to end service.
>

Hi Neil,

Yeah I agree, I'd like to have only the exact features we need
running. Also we would be able to test the code in house against our
own requirements/standards.

I'm also keen on the idea of being able to add counters to anything we
want, something the traditional vendors are either reluctant or slow
to do. This also opens the possibility of "temporary" counters, if
they create a performance hit, we may only need them when
troubleshooting.

Cheers,
James.

Re: [uknof] 'White Box' switching and OS - Any experiences worth sharing?

[Nick Hilliard]

Richard Halfpenny wrote:
> but running the ports at 1G on Trident-II is a real no-no.

running the ports at 1G on Trident-II is a no-no if there is egress
pressure on the 1G ports.  If the traffic bursts on the 1G port are
predominantly ingress, then that may not be a major issue.  Most
organisations will have traffic profiles which are substantially more
heavily loaded one way or the other, so this isn't as clear-cut as
"don't ever do this".  Monitoring egress drops is always a sensible
thing to do.

Also the trident chipset will only operate in cut-thru mode if all ports
on the device are configured to be the same speed, which will impact on
buffer utilisation.

Nick

Re: [uknof] 'White Box' switching and OS - Any experiences worth sharing?

[Richard Halfpenny]

On 15 February 2018 at 20:55, David Farrell  wrote:

> Hi UKNOF,
>
> Just wondering if anyone on list had looked at this and had any
> experiences they'd care to share.
>
> I'm just starting to evaluate the options out there, for various reasons I
> need NOS with a 'traditional' CLI that network engineers would be familiar
> with as well as all the API bells and whistles. I've been leaning towards
> IP Infusion's OcNOS, and have seen coverage of both LINX and BT evaluating
> that. The context here is looking to transition from a old-school STP based
> DC LAN towards EVPN w/ VXLAN and taking a look around to see what's
> available alongside Juniper/Cisco offerings... Any experiences, positive or
> negative would be be useful. I've been doing some research but thought I'd
> ask the clueful here.
>
> After some time out in the wilderness, I look forward to seeing folks at
> UKNOF 40 :)
>
> David.
>


We're doing 10G FTTP in various metros (through CityFibre dark/passive)
using 48-port Trident-II pizzaboxes fully loaded with FlexOptix BiDi's for
aggregation.  Cumulus Linux on the control plane.   You **really** need to
be aware of the buffer limitations on whatever platform you deploy.  We've
moved the service ratelimiting down to CPE where we have a reasonable
amount of buffer to play with but running the ports at 1G on Trident-II is
a real no-no.

Rich.
Network Engineering Manager
Exa Networks Ltd :: AS30740
richard.halfpe...@exa.net.uk

Re: [uknof] 'White Box' switching and OS - Any experiences worth sharing?

[Neil J. McRae]

Try and make it happen without having to learn the CLI - at the end of it 
you’ll be so much more satisfied.

Sent from my iPhone

On 17 Feb 2018, at 11:40, David Farrell 
> wrote:


On Thu, 15 Feb 2018 at 20:55, David Farrell 
> wrote:
Hi UKNOF,

Just wondering if anyone on list had looked at this and had any experiences 
they'd care to share.

I'm just starting to evaluate the options out there, for various reasons I need 
NOS with a 'traditional' CLI that network engineers would be familiar with as 
well as all the API bells and whistles. I've been leaning towards IP Infusion's 
OcNOS, and have seen coverage of both LINX and BT evaluating that. The context 
here is looking to transition from a old-school STP based DC LAN towards EVPN 
w/ VXLAN and taking a look around to see what's available alongside 
Juniper/Cisco offerings... Any experiences, positive or negative would be be 
useful. I've been doing some research but thought I'd ask the clueful here.

After some time out in the wilderness, I look forward to seeing folks at UKNOF 
40 :)

David.

Folks,

Appreciate all the replies both on and off list, really helpful stuff!

David.

Re: [uknof] 'White Box' switching and OS - Any experiences worth sharing?

[Tom Hill]

On 17/02/18 09:45, James Bensley wrote:
>> On 17 February 2018 at 01:15, Aftab Siddiqui
>>  wrote:
>> And they also have Cisco like CLI wrapper with ‘?’ help
>> 
> It has a standard Linux CLI (by which I mean BASH-like), the "ip"
> command suite has been extended so support all the extra features
> they have built into Cumulus Linux like VXLAN, EVPN, MPLS etc so
> server guys interact with the switch CLI as if were a server and/or
> edit plain text config files in /etc.

If you're coming at this from a Cisco background, almost certainly you'd
find the 'net' command (it's called 'NCLU' in Cumulus parlance) to be
the best route in. It will provide tab completion, and commit/abort
functionality. It works remarkably well.

The good news is that it'll also give you nice, repeatable templates for
whatever automation you decide to achieve. This will save you from those
'server guys' editing text files thinking it's identical to Debian
config (hint: it isn't, it's much improved.)

-- 
Tom Hill
Network Manager

Bytemark Limited
http://www.bytemark.co.uk/
tel. +44 1904 890 890



signature.asc
Description: OpenPGP digital signature

Re: [uknof] 'White Box' switching and OS - Any experiences worth sharing?

[David Farrell]

On Thu, 15 Feb 2018 at 20:55, David Farrell  wrote:

> Hi UKNOF,
>
> Just wondering if anyone on list had looked at this and had any
> experiences they'd care to share.
>
> I'm just starting to evaluate the options out there, for various reasons I
> need NOS with a 'traditional' CLI that network engineers would be familiar
> with as well as all the API bells and whistles. I've been leaning towards
> IP Infusion's OcNOS, and have seen coverage of both LINX and BT evaluating
> that. The context here is looking to transition from a old-school STP based
> DC LAN towards EVPN w/ VXLAN and taking a look around to see what's
> available alongside Juniper/Cisco offerings... Any experiences, positive or
> negative would be be useful. I've been doing some research but thought I'd
> ask the clueful here.
>
> After some time out in the wilderness, I look forward to seeing folks at
> UKNOF 40 :)
>
> David.
>

Folks,

Appreciate all the replies both on and off list, really helpful stuff!

David.

Re: [uknof] 'White Box' switching and OS - Any experiences worth sharing?

[Will Hargrave]

On 15 Feb 2018, at 20:55, David Farrell wrote:

I'm just starting to evaluate the options out there, for various 
reasons I
need NOS with a 'traditional' CLI that network engineers would be 
familiar
with as well as all the API bells and whistles. I've been leaning 
towards
IP Infusion's OcNOS, and have seen coverage of both LINX and BT 
evaluating
that. The context here is looking to transition from a old-school STP 
based

DC LAN towards EVPN w/ VXLAN and taking a look around to see what's
available alongside Juniper/Cisco offerings... Any experiences, 
positive or
negative would be be useful. I've been doing some research but thought 
I'd

ask the clueful here.


You should definitely investigate Arista. It isn’t a whitebox 
solution, however has an improved ‘traditional’ CLI and is really an 
industry leader in DC networking.


We’ve been running Arista (w/IP core and VXLAN) at LONAP since the 
launch of the 100GE product in 2016 with great results. In our case, the 
cost per port is above the cheaper low buffer trident/tomahawk solutions 
- it is only recently that the whitebox vendors have started to add 
larger-buffer products.


https://people.ucsc.edu/~warner/buffer.html is a good resource comparing 
various hardware.


After some time out in the wilderness, I look forward to seeing folks 
at

UKNOF 40 :)


See you there! :)

Re: [uknof] 'White Box' switching and OS - Any experiences worth sharing?

[Neil J. McRae]

Deployed no but just a matter of time - I doubt there are huge latency benefits 
that you don’t already see from the current plethora of optimised DC switches 
and code that’s available and I’m totally unconvinced that low buffer solutions 
are right for anything outside of the DC with today’s OTT steaming needs mixed 
with a lot of the access technology in cable, DSL and FTTP. 

Our requirement is only to run code we know we need to run- simplifies security 
risks and stupid bugs in things we don’t use and allows focused testing and 
telemetry in the end to end service. 

Sent from my iPhone

> On 17 Feb 2018, at 10:04, James Bensley  wrote:
> 
>> On 17 February 2018 at 09:43, Neil J. McRae  wrote:
>> However, you might want to look at P4 capable platforms which are looking
>> very good for focused thin OS networking.
> 
> Are you using P4 at all Neil?
> 
> I'd be very keen to hear from anyone that is, what their experience has been.
> 
> I've downloaded the BMv2 target [1] and started playing around with
> code that would run on x86 to test. However, when it came to real
> hardware testing Barefoot Networks seem to have gone from one ASIC
> which supports P4, from when I first looked into P4, to multiple ASICs
> [2] but they don't seem to be in stock anywhere?!
> 
> If anyone has some P4 hardware, even in the lab, I'd love to hear
> about your experiences. I've only tested it inside a VM which is
> really more just letter one learn the syntax and architecture. I
> wanted to test the impact of simplifying the forwarding pipeline to
> the bare minimum to reduce latency, advanced/dynamic port buffers, and
> iOAM. What have you managed to achieve with it that you couldn't
> before?
> 
> Cheers,
> James.
> 
> [1] https://github.com/p4lang/behavioral-model
> [2] https://barefootnetworks.com/products/brief-tofino/
> 

Re: [uknof] 'White Box' switching and OS - Any experiences worth sharing?

[James Bensley]

On 17 February 2018 at 09:43, Neil J. McRae  wrote:
> However, you might want to look at P4 capable platforms which are looking
> very good for focused thin OS networking.

Are you using P4 at all Neil?

I'd be very keen to hear from anyone that is, what their experience has been.

I've downloaded the BMv2 target [1] and started playing around with
code that would run on x86 to test. However, when it came to real
hardware testing Barefoot Networks seem to have gone from one ASIC
which supports P4, from when I first looked into P4, to multiple ASICs
[2] but they don't seem to be in stock anywhere?!

If anyone has some P4 hardware, even in the lab, I'd love to hear
about your experiences. I've only tested it inside a VM which is
really more just letter one learn the syntax and architecture. I
wanted to test the impact of simplifying the forwarding pipeline to
the bare minimum to reduce latency, advanced/dynamic port buffers, and
iOAM. What have you managed to achieve with it that you couldn't
before?

Cheers,
James.

[1] https://github.com/p4lang/behavioral-model
[2] https://barefootnetworks.com/products/brief-tofino/

Re: [uknof] 'White Box' switching and OS - Any experiences worth sharing?

[James Bensley]

On 17 February 2018 at 01:15, Aftab Siddiqui  wrote:
> And they also have Cisco like CLI wrapper with ‘?’ help
>
> On Sat, 17 Feb 2018 at 4:27 am, Sascha Luck [ml]  wrote:
>>
>> Hi David,
>>
>> have a look at Cumulus Linux. I've played with this on VMs and
>> Mellanox switches and VXLAN/EVPN with unnumbered eBGP is working.
>> The've been good with implementing asked-for features too, like
>> "q-in-vxlan" (I've not tested that yet)

I was also going to mention Cumulus. I'm also not using it in
production and evaluating the VM version right now. As already
mentioned it supports EVPN with VXLAN. It has a standard Linux CLI (by
which I mean BASH-like), the "ip" command suite has been extended so
support all the extra features they have built into Cumulus Linux like
VXLAN, EVPN, MPLS etc so server guys interact with the switch CLI as
if were a server and/or edit plain text config files in /etc. It helps
to break down the barrier that some of our server guys have, where the
Cisco or Junos CLI is very alien for them. So although that's not an
API it does mean that your switch and server CLI is "the same", and
that you can use your Linux orchestration tool du jour like
Ansible/Salt/Puppet etc. to manage switches and servers alike.

There was a great preso at NetDev 2.2, definitely worth a watch if you
want to quickly get up to sped with EVPN in Cumulus:
https://www.netdevconf.org/2.2/session.html?prabhu-linuxbridge-tutorial

Cheers,
James.

Re: [uknof] 'White Box' switching and OS - Any experiences worth sharing?

[Neil J. McRae]

If you want proper routing, there really is only one choice that’s up to it 
which is IP Infusion. Cumulus is great in certain use cases (Which they are 
completely transparent about BTW).

We have a box doing just about everything features wise.

However, you might want to look at P4 capable platforms which are looking very 
good for focused thin OS networking.

Cheers,
Neil.


On 17/02/2018, 01:22, "uknof on behalf of Aftab Siddiqui" 
 on 
behalf of aftab.siddi...@gmail.com> wrote:

And they also have Cisco like CLI wrapper with ‘?’ help
On Sat, 17 Feb 2018 at 4:27 am, Sascha Luck [ml] 
> wrote:
Hi David,

have a look at Cumulus Linux. I've played with this on VMs and
Mellanox switches and VXLAN/EVPN with unnumbered eBGP is working.
The've been good with implementing asked-for features too, like
"q-in-vxlan" (I've not tested that yet)

cheers,
Sascha

On Thu, Feb 15, 2018 at 08:55:23PM +, David Farrell wrote:
>Just wondering if anyone on list had looked at this and had any experiences
>they'd care to share.
>
>I'm just starting to evaluate the options out there, for various reasons I
>need NOS with a 'traditional' CLI that network engineers would be familiar
>with as well as all the API bells and whistles. I've been leaning towards
>IP Infusion's OcNOS, and have seen coverage of both LINX and BT evaluating
>that. The context here is looking to transition from a old-school STP based
>DC LAN towards EVPN w/ VXLAN and taking a look around to see what's
>available alongside Juniper/Cisco offerings... Any experiences, positive or
>negative would be be useful. I've been doing some research but thought I'd
>ask the clueful here.


>
>After some time out in the wilderness, I look forward to seeing folks at
>UKNOF 40 :)
>
>David.
--
Best Wishes,

Aftab A. Siddiqui

Re: [uknof] 'White Box' switching and OS - Any experiences worth sharing?

[Aftab Siddiqui]

And they also have Cisco like CLI wrapper with ‘?’ help
On Sat, 17 Feb 2018 at 4:27 am, Sascha Luck [ml]  wrote:

> Hi David,
>
> have a look at Cumulus Linux. I've played with this on VMs and
> Mellanox switches and VXLAN/EVPN with unnumbered eBGP is working.
> The've been good with implementing asked-for features too, like
> "q-in-vxlan" (I've not tested that yet)
>
> cheers,
> Sascha
>
> On Thu, Feb 15, 2018 at 08:55:23PM +, David Farrell wrote:
> >Just wondering if anyone on list had looked at this and had any
> experiences
> >they'd care to share.
> >
> >I'm just starting to evaluate the options out there, for various reasons I
> >need NOS with a 'traditional' CLI that network engineers would be familiar
> >with as well as all the API bells and whistles. I've been leaning towards
> >IP Infusion's OcNOS, and have seen coverage of both LINX and BT evaluating
> >that. The context here is looking to transition from a old-school STP
> based
> >DC LAN towards EVPN w/ VXLAN and taking a look around to see what's
> >available alongside Juniper/Cisco offerings... Any experiences, positive
> or
> >negative would be be useful. I've been doing some research but thought I'd
> >ask the clueful here.
>
>
> >
> >After some time out in the wilderness, I look forward to seeing folks at
> >UKNOF 40 :)
> >
> >David.
>
> --
Best Wishes,

Aftab A. Siddiqui

Re: [uknof] 'White Box' switching and OS - Any experiences worth sharing?

[Sascha Luck [ml]]

Hi David,

have a look at Cumulus Linux. I've played with this on VMs and
Mellanox switches and VXLAN/EVPN with unnumbered eBGP is working.
The've been good with implementing asked-for features too, like
"q-in-vxlan" (I've not tested that yet)

cheers,
Sascha

On Thu, Feb 15, 2018 at 08:55:23PM +, David Farrell wrote:

Just wondering if anyone on list had looked at this and had any experiences
they'd care to share.

I'm just starting to evaluate the options out there, for various reasons I
need NOS with a 'traditional' CLI that network engineers would be familiar
with as well as all the API bells and whistles. I've been leaning towards
IP Infusion's OcNOS, and have seen coverage of both LINX and BT evaluating
that. The context here is looking to transition from a old-school STP based
DC LAN towards EVPN w/ VXLAN and taking a look around to see what's
available alongside Juniper/Cisco offerings... Any experiences, positive or
negative would be be useful. I've been doing some research but thought I'd
ask the clueful here.





After some time out in the wilderness, I look forward to seeing folks at
UKNOF 40 :)

David.