Re: DPDK and energy efficiency

[Robert Bays]

To the nanog community, I’m sorry to have dragged this conversation out 
further.  I'm only responding to this because there are a significant number of 
open source projects and commercial products that use DPDK, or similar 
userspace network environment in their implementations.  The statements in this 
thread incorrectly cast them, because they use DPDK, as inefficient.  But the 
reality is they have all been designed from day one not to unnecessarily 
consume power.  Please ask your open source dev and/or vendor of choice to 
verify.  But please don’t rely on the information in this thread to make 
decisions about what you deploy in your network.

> On Feb 23, 2021, at 11:44 PM, Etienne-Victor Depasquale  
> wrote:
> 
> Hello Robert,
> 
> Your statement that DPDK “keeps utilization at 100% regardless of packet 
> activity” is just not correct.  You further pre-suppose "widespread DPDK's 
> core operating inefficiency” without any data to backup the operating 
> inefficacy assertion.  
> 
> This statement is incorrect. 
> I have provided references (please see earlier e-mails) that investigate the 
> operation of DPDK. 
> These references are items of peer-reviewed research that investigate a 
> perceived problem with deployment of DPDK. 
> If the power consumption incurred while running DPDK were a corner case, 
> then there would be little to no research value in investigating such 
> behavior.

Your references don’t take into account the code that this community would 
actually deploy; open source implementations like DANOS, FD.io <http://fd.io/>, 
or OVS.  They don’t audit any commercial products that implement userspace 
stacks.  None of your references say that DPDK is inherently inefficient.  The 
closest they come is to say that tight polling is inefficient.  But tight 
polling, even in the earliest days of DPDK, was never meant to be a design 
pattern that was actually deployed into production.  I was there for those 
early conversations.

> Please don’t mislead the community into believing that DPDK == power bad
> I have to object to this statement. It does seem to imply malice, or, at 
> best, amateurish behaviour, whether you intended it or not.
> 

Object all you want.  You are misleading people with your comments.  And in the 
process you are denigrating a large swath of OSS projects and commercial 
products that use DPDK.  Your survey questions are leading and provide a false 
dichotomy.  And when you post the results here, they will be archived forever 
to continue to spread misinformation, unfortunately.  

> Everything following is informational.  Stop here if so inclined.
>  Please stop delving into the detail of DPDK's facilities without regard for 
> your logical omission: 
> that whether the facilities are available or not, DPDK's deployment profile 
> (meaning: how it's being used in general), as indicated by the references 
> I've provided,
> are leading to high power inefficiency on cores partitioned to the data plane.
> 

I’ve been writing network appliance code for over 20 years.  I designed network 
architectures for years before that.  I have 10s of thousands of DPDK based 
appliances in production at this moment across multiple different use cases. I 
work with companies that have 100s of thousands of units in production that 
leverage userspace runtimes.  I do think I understand DPDK’s deployment profile 
better than you.  That’s what I have been trying to tell you.  People don’t 
write inefficient DPDK code to put into production.  We’re not dumb.  We’ve 
been thinking about power consumption from day one.  DPDK was never supposed to 
be just a tight loop poll.  You were always supposed to put in the very minimal 
extra work to modulate power consumption.  

> The takeaway is that DPDK (and similar) doesn’t guarantee runaway power bills.
> Of course it doesn't. 
> Even the second question of that bare-bones survey tried to communicate this 
> much.
> 
> If you have questions, I’d be happy to discuss off line
> I would be happy to answer your objections in detail off line too.
> Just let me know.
> 

Unfortunately, you don’t seem to be receptive to the numerous people 
contradicting your assertions.  So I’m out.  I’ll let my comments stand here.

> Cheers,
> 
> Etienne
> 
> 
> On Wed, Feb 24, 2021 at 12:12 AM Robert Bays  <mailto:rob...@gdk.org>> wrote:
> Hi Etienne,
> 
> Your statement that DPDK “keeps utilization at 100% regardless of packet 
> activity” is just not correct.  You further pre-suppose "widespread DPDK's 
> core operating inefficiency” without any data to backup the operating 
> inefficacy assertion.  Your statements, taken at face value, lead people to 
> believe that if a project uses DPDK it’s going to increase their power costs. 
>  And that’s just not the c

Re: DPDK and energy efficiency

[Robert Bays]

Hi Etienne,

Your statement that DPDK “keeps utilization at 100% regardless of packet 
activity” is just not correct.  You further pre-suppose "widespread DPDK's core 
operating inefficiency” without any data to backup the operating inefficacy 
assertion.  Your statements, taken at face value, lead people to believe that 
if a project uses DPDK it’s going to increase their power costs.  And that’s 
just not the case.  Please don’t mislead the community into believing that DPDK 
== power bad.

Everything following is informational.  Stop here if so inclined.

DPDK does not dictate CPU utilization or power consumption, the application 
leveraging DPDK does.  It’s the application that decides how to poll packets.  
If an application implements DPDK using only a tight polling loop, then it will 
keep CPU cores that are running DPDK threads at 100%.  But only the most simple 
and/or bespoke (think trading) applications are implemented this way.  You 
don’t need tight polling all the time to get the performance gains provided by 
DPDK or similar environments.  The vast majority of applications that this 
audience would actually install in their networks do not do tight polling all 
the time and therefore don’t consume 100% of the CPU all the time.   An 
interesting, helpful research effort you could lead would be to survey the 
ecosystem to catalog those applications that do fall into the power hungry 
category and help them to change their code.  

Intel DPDK application development guidelines don’t pre-suppose tight polling 
all the time and offer at least two methods for optimizing power against 
throughput.  The older method is to use adaptive polling; increasing the 
polling frequency as traffic load increases.  This keeps cpu utilization low 
when packet load is light and increases it as traffic levels warrant.  The 
second method is to use P-states and/or C-states to put the processor into 
lower power modes when traffic loads are lighter.  We have found that adaptive 
polling works better across a larger pool of hardware types, and therefore that 
is what DANOS uses, amongst other things.  

Further, performance and power consumption are dictated by a multivariate set 
of application decisions including: design patterns such as single thread run 
to completion models vs. passing mbufs between multiple threads, buffer sizes 
and cache management algorithms, combining and/or separating tx/rx threads, 
binding threads to specific lcores, reserved cores for DPDK threads, 
hyperthreading, kernel schedulers, hypervisor schedulers, interface drivers, 
etc.  All of these are application specific, not DPDK generic.  Well written 
applications that leverage DPDK provide knobs for the user to tune these 
settings for their specific environment and use case.  None of this unique to 
DPDK.  Solution designs were cribbed from previous technologies.

The takeaway is that DPDK (and similar) doesn’t guarantee runaway power bills.  
Power consumption is dictated by the application.  Look for well behaved 
applications and everything will be alright.

If you have questions, I’d be happy to discuss off line.

Thanks,
Robert.


> On Feb 22, 2021, at 11:27 PM, Etienne-Victor Depasquale  
> wrote:
> 
> Sorry, last line should have been:
> "intended to get an impression of how widespread ***knowledge of*** DPDK's 
> core operating inefficiency is",
> not:
> "intended to get an impression of how widespread DPDK's core operating 
> inefficiency is"
> 
> On Tue, Feb 23, 2021 at 8:22 AM Etienne-Victor Depasquale  
> wrote:
> Beyond RX/TX CPU affinity, in DANOS you can further tune power consumption by 
> changing the adaptive polling rate.  It doesn’t, per the survey, "keep 
> utilization at 100% regardless of packet activity.” 
> Robert, you seem to be conflating DPDK 
> with DANOS' power control algorithms that modulate DPDK's default behaviour.
> 
> Let me know what you think; otherwise, I'm pretty confident that DPDK does:
> "keep utilization at 100% regardless of packet activity.”   
> 
> Keep in mind that this is a bare-bones survey intended for busy, 
> knowledgeable people (the ones you'd find on NANOG) -
> not a detailed breakdown of modes of operation of DPDK or DANOS.
> DPDK has been designed for fast I/O that's unencumbered by the trappings of 
> general-purpose OSes, 
> and that's the impression that needs to be forefront.
> Power control, as well as any other dimensions of modulation, 
> are detailed modes of operation that are well beyond the scope of a 
> bare-bones 2-question survey 
> intended to get an impression of how widespread DPDK's core operating 
> inefficiency is.
> 
> Cheers,
> 
> Etienne 
> 
> On Mon, Feb 22, 2021 at 10:20 PM Robert Bays  wrote:
> Beyond RX/TX CPU affinity, in DANOS you can further tune power consumption by 
> changing the ada

Re: DPDK and energy efficiency

[Robert Bays]

Beyond RX/TX CPU affinity, in DANOS you can further tune power consumption by 
changing the adaptive polling rate.  It doesn’t, per the survey, "keep 
utilization at 100% regardless of packet activity.”  Adaptive polling changes 
in DPDK optimize for tradeoffs between power consumption, latency/jitter and 
drops during throughput ramp up periods.  Ideally your DPDK implementation has 
an algorithm that tries to automatically optimize based on current traffic 
patterns.

In DANOS refer to the “system default dataplane power-profile” config command 
tree for adaptive polling settings.  Interface RX/TX affinity is configured on 
a per interface basis under the “interfaces dataplane” config command tree.

-robert


> On Feb 22, 2021, at 11:46 AM, Jared Geiger  wrote:
> 
> DANOS lets you specify how many dataplane cores you use versus control plane 
> cores. So if you put a 16 core host in to handle 2GB of traffic, you can 
> adjust the dataplane worker cores as needed. Control plane cores don't stay 
> at 100% utilization. 
> 
> I use that technique plus DANOS runs on VMware (not oversubscribed) which 
> allows me to use the hardware for other VMs. NICS are attached to the VM via 
> PCI Passthrough which helps eliminate the overhead to the VMware hypervisor 
> itself.
> 
> I have an 8 core VM with 4 cores set to dataplane and 4 to control plane. The 
> 4 control plane cores are typically idle only processing BGP route updates, 
> SNMP, logs, etc.
> 
> ~Jared
> 
> On Sun, Feb 21, 2021 at 11:30 PM Etienne-Victor Depasquale  
> wrote:
> Hello folks,
> 
> I've just followed a thread regarding use of CGNAT and noted a suggestion 
> (regarding DANOS) that includes use of DPDK.
> 
> As I'm interested in the breadth of adoption of DPDK, and as I'm a researcher 
> into energy and power efficiency, I'd love to hear your feedback on your use 
> of power consumption control by DPDK.
> 
> I've drawn up a bare-bones, 2-question survey at this link: 
> 
> https://www.surveymonkey.com/r/J886DPY. 
> 
> Responses have been set to anonymous.
> 
> Cheers,
> 
> Etienne
> 
> -- 
> Ing. Etienne-Victor Depasquale
> Assistant Lecturer
> Department of Communications & Computer Engineering
> Faculty of Information & Communication Technology
> University of Malta
> Web. https://www.um.edu.mt/profile/etiennedepasquale

Re: cheap MPLS router recommendations

[Robert Bays]

> On Oct 26, 2020, at 11:51 AM, Eric Kuhnke  wrote:
> 
> If we're talking about whitebox router and ipifusion, what we're really 
> talking about is vyatta/vyOS and the linux foundation DANOS stuff on an 
> ordinary x86-64 server that has a weird shape.
> 

Maybe tangential, but to be clear, VyOS is not equivalent to DANOS/Vyatta.  And 
DANOS is a subset of Vyatta.

VyOS is a fork of the second generation Vyatta code which used the Linux kernel 
as the packet forwarding plane.  Control plane integration is based on a 
proprietary scripting language with bash shell interpretation.

The current generation Vyatta is for the most part a completely different 
system.  Some of the changes at a very high level:  Linux kernel based 
forwarding has been replaced with bespoke DPDK based software packet forwarding 
to improve features and performance.  Merchant silicon forwarding support was 
also added to be able to support hybrid software/hardware forwarding paths on 
the same box.  The control plane is a completely re-written event driven system 
using yang as the modeling language to improve features, ease of use, 
operation, and security.  The Vyatta routing protocol stack is proprietary.

DANOS is the open source version of current generation Vyatta, which makes up 
the bulk of the Vyatta code since AT released it to LF.  Active Vyatta 
development on the OSS is done upstream in github.  The major differences 
between Vyatta and DANOS are mostly due to licensing restrictions.  FRR in 
DANOS replaces the proprietary routing protocol stack in Vyatta.  Also, the 
merchant silicon integration in the DANOS forwarding plane currently only 
supports a limited number of Broadcom DNX based systems using OpenNSL rather 
than a proprietary SDK.

From a hardware perspective, Vyatta/DANOS supports hybrid software/hardware 
forwarding environments.  So you could install it on standard x86 system that 
looks like a server to enable software forwarding.  In this environment it has 
been tested to 100(s)Gbps depending on the hardware.  Or you could install it 
as a VNF in the cloud.  Or you could install on a merchant silicon based 
whitebox switch and get hardware based forwarding at the full Tb capacity of 
the merchant silicon chip.  There are install guides at danosproject.org 
 for those use cases.

There are also vendors that produce hybrid systems with large enough punt paths 
between the x86 SOC and the merchant silicon to support true hybrid 
environments where some features/forwarding are handled by the merchant silicon 
and some by the SOC.  One example, in a CPE device you may want local traffic 
routing and filtering services handled by the merchant silicon while IPsec is 
handled in software using DPDK on the x86 SOC.  

Vyatta/DANOS is deployed in production at scale in all of these footprints.  So 
it’s not just for weirdly shaped servers anymore.

-r

Re: AT released DANOS code to Linux Foundation

[Robert Bays]

For the open source version we replaced our proprietary routing protocol stack 
with FRR.  

Since the AT acquisition we have also added support for a few merchant 
silicon platforms in a hybrid software/hardware forwarding plane.  ONIE images 
are available from the same link.

Cheers,
Robert.


> On Nov 18, 2019, at 2:24 PM, Jared Geiger  wrote:
> 
> DANOS is using FRR in the opensource version at least.
> 
> On Mon, Nov 18, 2019 at 1:15 PM Mike Hammett  > wrote:
> Chances are, if there was a decision to be made, UBNT made the wrong choice.
> 
> That said, I've heard a lot of good about ZebOS.  *shrugs*
> 
> 
> 
> -
> Mike Hammett
> Intelligent Computing Solutions 
>   
>  
>  
> 
> Midwest Internet Exchange 
>   
>  
> 
> The Brothers WISP 
>   
> 
> From: "Rubens Kuhl" mailto:rube...@gmail.com>>
> To: "Nanog" mailto:nanog@nanog.org>>
> Sent: Monday, November 18, 2019 3:10:39 PM
> Subject: Re: AT released DANOS code to Linux Foundation
> 
> 
> 
> On Mon, Nov 18, 2019 at 5:55 PM Brielle  > wrote:
> On 11/18/2019 1:31 PM, Jared Geiger wrote:
> > This past Friday, the code for DANOS was released as open source to the 
> > Linux Foundation and published at https://github.com/danos 
> > 
> 
> This is pretty awesome news.
> 
>  From what I'm reading, it looks like the commercial support options 
> will be able to use ZebOS as the routing engine instead of quagga? 
> EdgeOS has been using it for a while, and was a huge step up in terms of 
> stability and functionality.
> 
> 
> Curiously, at the same time EdgeOS replaced Quagga with ZebOS I started 
> reading more complaints and more people dropping UBNT altogether in the L3 
> world. 
> So I wonder if it was a good decision or not... 
> 
> 
> Rubens
>  
> 

Re: Small full BGP table capable router with low power consumption

[Robert Bays]

> On Dec 5, 2017, at 8:59 AM, Eric Kuhnke  wrote:
> 
> It is worth mentioning for those who have not seen a Ubiquiti "edgrouter"
> in person yet, or worked with one, where their operating system came
> from...  When Vyatta was acquired by Brocade, the core Vyatta team jumped
> ship and were hired directly by Ubiquiti.

Not really.  There were two developers that quit Vyatta and subsequently went 
to Ubiquiti.  And that happened long before the Brocade acquisition.  The core 
Vyatta team is still going strong, working on the Vyatta NOS.

-robert

Re: scaling linux-based router hardware recommendations

[Robert Bays]

I was trying not to pitch my company on list, but the performance numbers I 
quoted are on the Vyatta/Brocade vRouter which is commercially available.  
Other vendors also also have publicly available performance numbers that are 
interesting.


 On Jan 28, 2015, at 5:02 AM, Paul S. cont...@winterei.se wrote:
 
 That's the problem though.
 
 Everyone has presentations for the most part, very few actual tools that 
 end users can just use exist.
 
 On 1/28/2015 午後 08:02, Robert Bays wrote:
 On Jan 27, 2015, at 8:31 AM, Jim Shankland na...@shankland.org wrote:
 
 My expertise, such as it ever was, is a bit stale at this point, and my
 figures might be a little off. But I think the general principle
 applies: think about the minimum number of x86 instructions, and the
 minimum number of main memory accesses, to inspect a packet header, do a
 routing table lookup, and enqueue the packet on an outbound interface. I
 can't see that ever getting reduced to the point where a generic server
 can handle 40-byte packets at line rate (for that matter, line rate is
 increasing a lot faster than speed of generic server these days).
 Using DPDK it’s possible to do everything stated and achieve 10Gbps line 
 rate at 64byte packets on multiple interfaces simultaneously.  Add ACLs to 
 the test setup and you can reach significant portions of 10Gbps at 64byte 
 packets and full line rate at 128bytes.
 
 Check out Venky Venkatesan’s presentation at the last DPDK Summit for 
 interesting information on pps/CPU cycles and some of the things that can be 
 done to optimize forwarding in a generic processor environment.
 
 http://www.slideshare.net/jstleger/6-dpdk-summit-2014-intel-presentation-venky-venkatesan
 
 
 

Re: scaling linux-based router hardware recommendations

[Robert Bays]

 On Jan 27, 2015, at 8:31 AM, Jim Shankland na...@shankland.org wrote:
 
 My expertise, such as it ever was, is a bit stale at this point, and my 
 figures might be a little off. But I think the general principle 
 applies: think about the minimum number of x86 instructions, and the 
 minimum number of main memory accesses, to inspect a packet header, do a 
 routing table lookup, and enqueue the packet on an outbound interface. I 
 can't see that ever getting reduced to the point where a generic server 
 can handle 40-byte packets at line rate (for that matter, line rate is 
 increasing a lot faster than speed of generic server these days).

Using DPDK it’s possible to do everything stated and achieve 10Gbps line rate 
at 64byte packets on multiple interfaces simultaneously.  Add ACLs to the test 
setup and you can reach significant portions of 10Gbps at 64byte packets and 
full line rate at 128bytes.

Check out Venky Venkatesan’s presentation at the last DPDK Summit for 
interesting information on pps/CPU cycles and some of the things that can be 
done to optimize forwarding in a generic processor environment.

http://www.slideshare.net/jstleger/6-dpdk-summit-2014-intel-presentation-venky-venkatesan

Re: Any recommended router. They are reliable and have good support.

[Robert Bays]

On Nov 22, 2011, at 9:09 AM, James Jones wrote:

 On Tue, Nov 22, 2011 at 10:43 AM, lorddoskias lorddosk...@gmail.com wrote:
 
 On 11/22/2011 3:38 PM, Deric Kwok wrote:
 
 Hi
 
 Can I know any selection of Linux routers except cisco / juniper?
 
 They are reliable and have  good support provided
 
 We would like to get one for testing.
 
 Thank you
 
 
 
 http://www.vyatta.com/ might be worth checking.
 
 
 If you are not afraid of the command line check xorp it is what vyatta is
 based on.

Vyatta uses Quagga for the protocol stack.  We switched away from XORP a few 
years ago.

Cheers,
Robert.

Re: Vyatta as a BRAS

[Robert Bays]

On 7/13/10 10:56 AM, Dobbins, Roland wrote:
 
 On Jul 14, 2010, at 12:39 AM, khatfi...@socllc.net
 khatfi...@socllc.net wrote:
 
 I haven't done real world testing with Vyatta but we consistently
 pass 750KPPS+ without the slightest hiccup on our FreeBSD routing
 systems.
 
 750kpps packeting the box itself?
 
 Also, note that kpps is a small amount of traffic, compared to what
 even very small botnets can dish out.

I work for Vyatta.  We regularly see 700+kpps per core using a Nehalem
class cpu with higher rates possible in tuned systems.  On a multi-core
system this translates to a fairly high level of throughput.  To echo an
earlier post, Linux can comfortably handle gigabit.

It wasn't too long ago that this wasn't the case.  The growth in the
number of cores available to the end user, the introduction of
multi-queue nics, the move away from the FSB architecture towards QPI,
ever faster PCIe...  The technology is directionally trending towards
faster, more consistent network throughputs whether your Linux host is
acting as a router, firewall, web server, or whatever.  There are
activities taking place on the software front as well to increase speed
and consistency in the realms of forwarding and firewall, including
technologies that separate the control and forwarding planes.  There is
still headroom available in commodity compute to scale further.

I will be the first to admit that Vyatta won't work for everyone.  We
still have a lot of work to do for our system to fit seamlessly in some
environments.  But, the bet that we have made is that commodity compute
coupled with the amazing OSS dev community can keep pace with a good
portion of the networking worlds needs.  So far, that bet looks like a
good one.

To discount all software routing running on general purpose processors
as being antiquated seems to me to be premature, especially given the
various vendors interests as more functionality migrates into the cloud.
 As that happens commodity components in the cloud fabric will
necessarily need to behave more like network appliances.

Cheers,
Robert.