Re: Anyone heard about DPDK?

[Aleksandar Lazic]

Hi all.

Wow so much feedback, thanks all for the answers ;-)

Am 12.02.2019 um 15:23 schrieb Alexandre Cassen:
> There has been a lot of applications/stack built around DPDK last few years.
> Mostly because people found it easy to code stuff around DPDK and are so happy
> to display perf graph about their DPDK application vs plain Linux Kernel 
> stack.

Would you like to share such a comparison?

> My intention here would be to warn a little bit about this collective 
> enthusiasm
> around DPDK. Integrating DPDK is easy and mostly fun (even if you have to 
> learn
> and dig into their rte lib and mbuf related), but most of people are 
> completely
> blind about security ! Ok Linux kernel and netdev is slow in respect of NIC
> available nowadays (10G, 40G and multiple 100G on core-networks), but using
> Linux TCP/IP stack you will benefit the hardcore hacking task done during last
> 30years by Linux netdev core guys ! this long process mostly fix and solve
> hardcore issues and for some : security issues. And you will certainly not be
> protected by a 'super fast' self proclaimed performance soft. Mostly because
> these applications are mostly features oriented than security or protocol
> full-picture, and are using this 'super fast, best of ever' argument to 
> enforce
> people mind to adopt.

When I take a look into the doc then I see some security informations.

https://doc.dpdk.org/guides/prog_guide/rte_security.html

How does such a application handle the security topic?

> The way DPDK is working in polling mode is certainly not the best at all. DPDK
> is PCI 'stealing' NIC from kernel to handle/manage itself in userspace by
> forcing active loop (100% CPU polling) to handle descriptors and convert to
> mbuf. latter you can 'forward' mbuf to Linux kernel by using KNI netdevice to
> use Linux Kernel machinery as a slow-path for complicated/not_focused
> packet-flow (most application are using KNI for ARP,DHCP,...). But most of the
> time application are implementing 'minimal' adjacent network features to make 
> it
> work in its networking environment : and here is the problem: you are focused 
> on
> perf and because of it you are making shortcut about considering potential
> threats... a prediction could be to see large number of network security holes
> opened, and specially an old bunch of security holes making a fun revival (a 
> lot
> of fun with TCP)

So this means that a application can be used with DPDK when it uses the
KNI (=Kernel NIC Interface) right?

https://doc.dpdk.org/guides/prog_guide/kernel_nic_interface.html

How much "slower" is the way via KNI?

> In contrast recent Linux Kernel introduced XDP and eBPF machinery that are
> certainly much more future proof than DPDK. First consideration in XDP design 
> is
> : you only TAP in data/packet you are interested in and not making an hold-up 
> on
> whole traffic. So XDP is for fast path but only for protocol or workflow
> identified. You program and attach an eBPF program to a specific NIC, if there
> is no match then packet simply continue its journey into Linux Kernel stack.
> 
> XDP is a response from kernel netdev community to address DPDK users. The fact
> that DPDK introduced and extended PMP to support AF_XDP is certainly a sign 
> that
> XDP is going/doing into the right direction.

Sounds a interesting future for the linux kernel.

When we take a look into the container and cloud world, does this DPDK makes any
sense? I mean when I run a container on AWS/Google/Azure I'm normally so far
from any Hardware that this high traffic possibility isn't available for the
container, right?

To the list members:
Maybe it's offtopic from the HAProxy list so please apologize for all the noise.

> regs,
> Alexandre

Regards
Aleks

> On 12/02/2019 14:04, Federico Iezzi wrote:
>> Nowadays most VNF (virtual network function) in the telco operators are built
>> around DPDK. Not demos, most 5G will be like that. 4G is migrating as we 
>> speak
>> on this new architecture.
>> There isn't any TCP stack built-it but the libraries can be used to build 
>> one.
>> VPP has integrated DPDK in this way.
>>
>> Linux network stack is not designed to managed millions of packets per 
>> second,
>> DPDK bypass it completely offloading everything in userspace. The beauty is
>> that also the physical nic drivers are in userspace using specific DPDK
>> drivers. Linux networking stack works in interrupt mode, DPDK is in polling
>> mode, basically with a while true.
>>
>>  From F5 at the dpdk summit as a relevant reference to what HAProxy does.
>> https://dpdksummitnorthamerica2018.sched.com/event/IhiF/dpdk-on-f5-big-ip-virtual-adcs-brent-blood-f5-networks
>>
>> https://www.youtube.com/watch?v=6zu81p3oTeo
>>
>> Regards,
>> Federico
>>
>> On Tue, 12 Feb 2019 at 11:08, Julien Laffaye > > wrote:
>>
>>     Something like http://seastar.io/ or https://fd.io/ ? :)
>>
>>     On Mon, Feb 11, 2019 at 11:25 AM Baptiste >     

Re: Anyone heard about DPDK?

[Alexandre Cassen]

There has been a lot of applications/stack built around DPDK last few 
years. Mostly because people found it easy to code stuff around DPDK and 
are so happy to display perf graph about their DPDK application vs plain 
Linux Kernel stack.


My intention here would be to warn a little bit about this collective 
enthusiasm around DPDK. Integrating DPDK is easy and mostly fun (even if 
you have to learn and dig into their rte lib and mbuf related), but most 
of people are completely blind about security ! Ok Linux kernel and 
netdev is slow in respect of NIC available nowadays (10G, 40G and 
multiple 100G on core-networks), but using Linux TCP/IP stack you will 
benefit the hardcore hacking task done during last 30years by Linux 
netdev core guys ! this long process mostly fix and solve hardcore 
issues and for some : security issues. And you will certainly not be 
protected by a 'super fast' self proclaimed performance soft. Mostly 
because these applications are mostly features oriented than security or 
protocol full-picture, and are using this 'super fast, best of ever' 
argument to enforce people mind to adopt.


The way DPDK is working in polling mode is certainly not the best at 
all. DPDK is PCI 'stealing' NIC from kernel to handle/manage itself in 
userspace by forcing active loop (100% CPU polling) to handle 
descriptors and convert to mbuf. latter you can 'forward' mbuf to Linux 
kernel by using KNI netdevice to use Linux Kernel machinery as a 
slow-path for complicated/not_focused packet-flow (most application are 
using KNI for ARP,DHCP,...). But most of the time application are 
implementing 'minimal' adjacent network features to make it work in its 
networking environment : and here is the problem: you are focused on 
perf and because of it you are making shortcut about considering 
potential threats... a prediction could be to see large number of 
network security holes opened, and specially an old bunch of security 
holes making a fun revival (a lot of fun with TCP)


In contrast recent Linux Kernel introduced XDP and eBPF machinery that 
are certainly much more future proof than DPDK. First consideration in 
XDP design is : you only TAP in data/packet you are interested in and 
not making an hold-up on whole traffic. So XDP is for fast path but only 
for protocol or workflow identified. You program and attach an eBPF 
program to a specific NIC, if there is no match then packet simply 
continue its journey into Linux Kernel stack.


XDP is a response from kernel netdev community to address DPDK users. 
The fact that DPDK introduced and extended PMP to support AF_XDP is 
certainly a sign that XDP is going/doing into the right direction.


regs,
Alexandre



On 12/02/2019 14:04, Federico Iezzi wrote:
Nowadays most VNF (virtual network function) in the telco operators are 
built around DPDK. Not demos, most 5G will be like that. 4G is migrating 
as we speak on this new architecture.
There isn't any TCP stack built-it but the libraries can be used to 
build one. VPP has integrated DPDK in this way.


Linux network stack is not designed to managed millions of packets per 
second, DPDK bypass it completely offloading everything in userspace. 
The beauty is that also the physical nic drivers are in userspace using 
specific DPDK drivers. Linux networking stack works in interrupt mode, 
DPDK is in polling mode, basically with a while true.


 From F5 at the dpdk summit as a relevant reference to what HAProxy does.
https://dpdksummitnorthamerica2018.sched.com/event/IhiF/dpdk-on-f5-big-ip-virtual-adcs-brent-blood-f5-networks
https://www.youtube.com/watch?v=6zu81p3oTeo

Regards,
Federico

On Tue, 12 Feb 2019 at 11:08, Julien Laffaye > wrote:


Something like http://seastar.io/ or https://fd.io/ ? :)

On Mon, Feb 11, 2019 at 11:25 AM Baptiste mailto:bed...@gmail.com>> wrote:

Hi,

HAProxy requires a TCP stack below it. DPDK itself is not enough.

Baptiste

Re: Anyone heard about DPDK?

[Federico Iezzi]

Nowadays most VNF (virtual network function) in the telco operators are
built around DPDK. Not demos, most 5G will be like that. 4G is migrating as
we speak on this new architecture.
There isn't any TCP stack built-it but the libraries can be used to build
one. VPP has integrated DPDK in this way.

Linux network stack is not designed to managed millions of packets per
second, DPDK bypass it completely offloading everything in userspace. The
beauty is that also the physical nic drivers are in userspace using
specific DPDK drivers. Linux networking stack works in interrupt mode, DPDK
is in polling mode, basically with a while true.

>From F5 at the dpdk summit as a relevant reference to what HAProxy does.
https://dpdksummitnorthamerica2018.sched.com/event/IhiF/dpdk-on-f5-big-ip-virtual-adcs-brent-blood-f5-networks
https://www.youtube.com/watch?v=6zu81p3oTeo

Regards,
Federico

On Tue, 12 Feb 2019 at 11:08, Julien Laffaye  wrote:

> Something like http://seastar.io/ or https://fd.io/ ? :)
>
> On Mon, Feb 11, 2019 at 11:25 AM Baptiste  wrote:
>
>> Hi,
>>
>> HAProxy requires a TCP stack below it. DPDK itself is not enough.
>>
>> Baptiste
>>
>>>

Re: Anyone heard about DPDK?

[Julien Laffaye]

Something like http://seastar.io/ or https://fd.io/ ? :)

On Mon, Feb 11, 2019 at 11:25 AM Baptiste  wrote:

> Hi,
>
> HAProxy requires a TCP stack below it. DPDK itself is not enough.
>
> Baptiste
>
>>

Re: Anyone heard about DPDK?

[Baptiste]

Hi,

HAProxy requires a TCP stack below it. DPDK itself is not enough.

Baptiste

>

Re: Anyone heard about DPDK?

[Igor Cicimov]

On Mon, 11 Feb 2019 1:49 am Bruno Henc  Hi,
>
>
> Another good explanation on what DPDK does is available here:
>
>
> https://learning.oreilly.com/videos/oscon-2017/9781491976227/9781491976227-video306685
>
> https://wiki.fd.io/images/1/1d/40_Gbps_IPsec_on_commodity_hardware.pdf
>
>
>
> On 2/10/19 12:21 PM, Aleksandar Lazic wrote:
> > Am 10.02.2019 um 12:06 schrieb Lukas Tribus:
> >> On Sun, 10 Feb 2019 at 10:48, Aleksandar Lazic 
> wrote:
> >>> Hi.
> >>>
> >>> I have seen this in some twitter posts and asked me if it's something
> useable for a Loadbalancer like HAProxy ?
> >>>
> >>> https://www.dpdk.org/
> >>>
> >>> To be honest it looks like a virtual NIC, but I'm not sure.
> >> See:
> >> https://www.mail-archive.com/haproxy@formilux.org/msg26748.html
> > 8-O Sorry I have forgotten that Question.
> > Sorry the noise and thanks for your patience.
> >
> >> lukas
> > Greetings
> > Aleks
> >
>

Acording to this:

DPDK allows the host to process packets faster by bypassing the Linux
kernel. Instead, interactions with the NIC are performed using drivers and
the DPDK libraries.

It might help network performance. Source
https://docs.paloaltonetworks.com/vm-series/8-0/vm-series-deployment/set-up-the-vm-series-firewall-on-kvm/performance-tuning-of-the-vm-series-for-kvm/integrate-open-vswitch-with-dpdk.html

It is discussed in context of KVM and OvS so not sure if useful in other
cases.

>

Re: Anyone heard about DPDK?

[Bruno Henc]

Hi,


Another good explanation on what DPDK does is available here:

https://learning.oreilly.com/videos/oscon-2017/9781491976227/9781491976227-video306685

https://wiki.fd.io/images/1/1d/40_Gbps_IPsec_on_commodity_hardware.pdf



On 2/10/19 12:21 PM, Aleksandar Lazic wrote:

Am 10.02.2019 um 12:06 schrieb Lukas Tribus:

On Sun, 10 Feb 2019 at 10:48, Aleksandar Lazic  wrote:

Hi.

I have seen this in some twitter posts and asked me if it's something useable 
for a Loadbalancer like HAProxy ?

https://www.dpdk.org/

To be honest it looks like a virtual NIC, but I'm not sure.

See:
https://www.mail-archive.com/haproxy@formilux.org/msg26748.html

8-O Sorry I have forgotten that Question.
Sorry the noise and thanks for your patience.


lukas

Greetings
Aleks

Re: Anyone heard about DPDK?

[Aleksandar Lazic]

Am 10.02.2019 um 12:06 schrieb Lukas Tribus:
> On Sun, 10 Feb 2019 at 10:48, Aleksandar Lazic  wrote:
>>
>> Hi.
>>
>> I have seen this in some twitter posts and asked me if it's something 
>> useable for a Loadbalancer like HAProxy ?
>>
>> https://www.dpdk.org/
>>
>> To be honest it looks like a virtual NIC, but I'm not sure.
> 
> See:
> https://www.mail-archive.com/haproxy@formilux.org/msg26748.html

8-O Sorry I have forgotten that Question.
Sorry the noise and thanks for your patience.

> lukas

Greetings
Aleks

Re: Anyone heard about DPDK?

[Lukas Tribus]

On Sun, 10 Feb 2019 at 10:48, Aleksandar Lazic  wrote:
>
> Hi.
>
> I have seen this in some twitter posts and asked me if it's something useable 
> for a Loadbalancer like HAProxy ?
>
> https://www.dpdk.org/
>
> To be honest it looks like a virtual NIC, but I'm not sure.

See:
https://www.mail-archive.com/haproxy@formilux.org/msg26748.html


lukas

Anyone heard about DPDK?

[Aleksandar Lazic]

Hi.

I have seen this in some twitter posts and asked me if it's something useable 
for a Loadbalancer like HAProxy ?
 
https://www.dpdk.org/

To be honest it looks like a virtual NIC, but I'm not sure.

Regards
Aleks

Re: Does anyone heard about DPDK

[Aleksandar Lazic]

Hi Willy,

Willy Tarreau wrote on 17.07.2017:

> Hi,

> On Mon, Jul 17, 2017 at 11:34:21AM -0700, Jerry Scharf wrote:
>> The only thing I know that uses DPDK is ostineto, a network traffic
>> generator tool. I think it is going to be a while before any of this gets
>> integrated into the kernel for general networking. My understanding is that
>> this is for people who really want to squeeze all they can out of a 10/40G
>> interface.

> We do use DPDK at haproxy technologies for our traffic generation tools.
> But while it's useful to process packets, it really is not when you have
> to deal with userland code requiring a full-fledged TCP stack.

> A DPDK-based TCP stack would have to be completely callback-oriented,
> which is a very different way to deal with events than what is done with
> syscalls. While this model would provide a larger scalability, it also
> comes with some difficulties when you have to deal with some inevitable
> locking for example. It also doesn't make it as easy to batch processing
> to benefit from the code being hot in the instruction cache. But nowadays
> haproxy uses some callbacks so at least "only" the polling+fd+tcp+udp
> parts would have to be rewritten. SSL would have to be dropped, just like
> UNIX sockets. And we wouldn't benefit from some of the goodies we currently
> have in the stacks provided by the operating systems, like delayed ACKs,
> fastopen and so on.

> The real good stuff about DPDK is that it allows to build multi-10/100G
> switches and routers from affordable hardware when you know exactly what
> you want to achieve. You must not have to use high throughput between the
> system's stack and the devices however. This is perfect to route traffic
> between VMs for example.

Thanks for the detail information.
Now I understand why it's not that useful for haproxy or any other 
user land daemon.

> Willy

-- 
Best Regards
Aleks

Re: Does anyone heard about DPDK

[Willy Tarreau]

Hi,

On Mon, Jul 17, 2017 at 11:34:21AM -0700, Jerry Scharf wrote:
> The only thing I know that uses DPDK is ostineto, a network traffic
> generator tool. I think it is going to be a while before any of this gets
> integrated into the kernel for general networking. My understanding is that
> this is for people who really want to squeeze all they can out of a 10/40G
> interface.

We do use DPDK at haproxy technologies for our traffic generation tools.
But while it's useful to process packets, it really is not when you have
to deal with userland code requiring a full-fledged TCP stack.

A DPDK-based TCP stack would have to be completely callback-oriented,
which is a very different way to deal with events than what is done with
syscalls. While this model would provide a larger scalability, it also
comes with some difficulties when you have to deal with some inevitable
locking for example. It also doesn't make it as easy to batch processing
to benefit from the code being hot in the instruction cache. But nowadays
haproxy uses some callbacks so at least "only" the polling+fd+tcp+udp
parts would have to be rewritten. SSL would have to be dropped, just like
UNIX sockets. And we wouldn't benefit from some of the goodies we currently
have in the stacks provided by the operating systems, like delayed ACKs,
fastopen and so on.

The real good stuff about DPDK is that it allows to build multi-10/100G
switches and routers from affordable hardware when you know exactly what
you want to achieve. You must not have to use high throughput between the
system's stack and the devices however. This is perfect to route traffic
between VMs for example.

Willy

Re: Does anyone heard about DPDK

[Jerry Scharf]

The only thing I know that uses DPDK is ostineto, a network traffic 
generator tool. I think it is going to be a while before any of this 
gets integrated into the kernel for general networking. My understanding 
is that this is for people who really want to squeeze all they can out 
of a 10/40G interface.


Until it's a general part of the kernel networking, I won't get any 
benefit at the IP level.


jerry

On 7/17/17 11:05 AM, Lukas Tribus wrote:

Hello,


Am 15.07.2017 um 14:18 schrieb Andrew Smalley:

On 15 July 2017 at 10:32, Aleksandar Lazic  wrote:

Hi,

Network acceleration with DPDK
https://lwn.net/Articles/725254/

--
Best Regards
Aleks

I believe eBPF + XDP is more interesting at this point, but I guess it
all depends on what you are trying to achieve.

Anyway, other than some cheaper way to drop connections from
particular IPs, I fail to see how haproxy would work in conjunction
with such lower level stacks.


Lukas





--
Soundhound Devops
"What could possibly go wrong?"

Re: Does anyone heard about DPDK

[Lukas Tribus]

Hello,


Am 15.07.2017 um 14:18 schrieb Andrew Smalley:
> On 15 July 2017 at 10:32, Aleksandar Lazic  wrote:
>> Hi,
>>
>> Network acceleration with DPDK
>> https://lwn.net/Articles/725254/
>>
>> --
>> Best Regards
>> Aleks

I believe eBPF + XDP is more interesting at this point, but I guess it
all depends on what you are trying to achieve.

Anyway, other than some cheaper way to drop connections from
particular IPs, I fail to see how haproxy would work in conjunction
with such lower level stacks.


Lukas

Re: Does anyone heard about DPDK

[Andrew Smalley]

HI Aleksandar

I've only ever seen Intel's DPDK being used really with OpenVSwitch
and am not sure how it would help haproxy (Not that I am the best
person to say if its good for haproxy)

Andrew Smalley

Loadbalancer.org Ltd.

www.loadbalancer.org
+1 888 867 9504 / +44 (0)330 380 1064
asmal...@loadbalancer.org

Leave a Review | Deployment Guides | Blog


On 15 July 2017 at 10:32, Aleksandar Lazic  wrote:
> Hi,
>
> Network acceleration with DPDK
> https://lwn.net/Articles/725254/
>
> --
> Best Regards
> Aleks
>
>

Does anyone heard about DPDK

[Aleksandar Lazic]

Hi,

Network acceleration with DPDK
https://lwn.net/Articles/725254/

-- 
Best Regards
Aleks