Re: [vpp-dev] zero memcpy interface (alternative to memif) to pass packets from VPP to another trusted DPDK application
Thank you Damjan. Really appreciate it. I got the idea. Let me look into the snort plugin code and experiment and get back to you should I have any questions. - PK Das On Tue, May 24, 2022 at 8:51 AM Damjan Marion wrote: > > On 24.05.2022., at 14:28, PRANAB DAS wrote: > > Hi Damjan, > > The snort plugin could be simple. But I am not familiar with VPP > memory/buffer management as such and I am finding it not so easy to > navigate. > And looking at the physmem.c/h code, it appears the VPP uses heap memory > (mmap) and has its own heap/buffer management system. > > > VPP uses mmap system call (same like DPDK) to map shared memory, but that > is not heap memory. > > VPP buffer memory is shared memory (typically hugepage backed). There is > one block of shared memory per NUMA, and each block have own FD. > VPP can pass that FD to remote app over UDS and remote app can simply map > that shared memory, Both memif and snort plugins do exactly that. > > Main complexity is creating another shared memory region which holds > enqueue and dequeue rings: In snort plugin you can see how that shared > memory is orgaised by looking at src/plugins/snort/daq_vpp.h > > > Could you clarify? And in the snort plugin exposes/shares the mmap memory > region to the external program (snort). Is there any documentation that > provides an overview of VPP buffer management and how VPP memory/buffers > can be shared with external trusted applications? > > > No, unfortunately there is no documentation, but there is running code > (snort plugin) which does exactly that. > If you have any questions, ask here and i will try to help. > > > Thank you > > - PK Das > > > > > On Tue, May 24, 2022 at 7:51 AM Damjan Marion wrote: > >> >> VPP is not DPDK application, VPP features doesnt maintain DPDK metadata, >> in many cases we use native drivers so DPDK is not even loaded so you >> cannot use rte_ring unless you want to write complete translation layer >> which will likely be significantly slower that using native way which snort >> plugin uses. >> >> Why do you think zero copy interface in snort plugin is not simple? >> >> — >> Damjan >> >> On 24.05.2022., at 13:34, PRANAB DAS wrote: >> >> >> Thank you very much for your response Benoit! >> I am wondering if there is another option e.g. rte-ring of DPDK or if >> that option is not feasible in VPP? >> Could you comment? We are looking for a service-chaining application >> similar to snort but would like to have a much simpler zero-copy interface. >> >> Thank you, >> >> - P K DAS >> >> On Tue, May 24, 2022 at 4:22 AM Benoit Ganne (bganne) >> wrote: >> >>> It all depends on what you want to do. Usually, we try to avoid sharing >>> buffers read/write between multiple process, it makes debugging much harder >>> - especially buffer leaks or use-after-free... >>> Here is an example to share VPP buffers read-only with snort so that >>> snort can inspect traffic and gives a verdict back without any copy: >>> https://git.fd.io/vpp/tree/src/plugins/snort >>> >>> Best >>> ben >>> >>> > -Original Message- >>> > From: vpp-dev@lists.fd.io On Behalf Of PRANAB >>> DAS >>> > Sent: Monday, May 23, 2022 20:11 >>> > To: vpp-dev@lists.fd.io >>> > Subject: Re: [vpp-dev] zero memcpy interface (alternative to memif) to >>> > pass packets from VPP to another trusted DPDK application >>> > >>> > Hi >>> > >>> > Could some one comment on zero-copy alternative to memif interface? Any >>> > ideas, comments are welcome. >>> > >>> > Thank you >>> > >>> > - Pranab K Das >>> >> >> >> >> > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#21443): https://lists.fd.io/g/vpp-dev/message/21443 Mute This Topic: https://lists.fd.io/mt/91252298/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] zero memcpy interface (alternative to memif) to pass packets from VPP to another trusted DPDK application
> On 24.05.2022., at 14:28, PRANAB DAS wrote: > > Hi Damjan, > > The snort plugin could be simple. But I am not familiar with VPP > memory/buffer management as such and I am finding it not so easy to navigate. > And looking at the physmem.c/h code, it appears the VPP uses heap memory > (mmap) and has its own heap/buffer management system. VPP uses mmap system call (same like DPDK) to map shared memory, but that is not heap memory. VPP buffer memory is shared memory (typically hugepage backed). There is one block of shared memory per NUMA, and each block have own FD. VPP can pass that FD to remote app over UDS and remote app can simply map that shared memory, Both memif and snort plugins do exactly that. Main complexity is creating another shared memory region which holds enqueue and dequeue rings: In snort plugin you can see how that shared memory is orgaised by looking at src/plugins/snort/daq_vpp.h > Could you clarify? And in the snort plugin exposes/shares the mmap memory > region to the external program (snort). Is there any documentation that > provides an overview of VPP buffer management and how VPP memory/buffers can > be shared with external trusted applications? No, unfortunately there is no documentation, but there is running code (snort plugin) which does exactly that. If you have any questions, ask here and i will try to help. > > Thank you > > - PK Das > > > > > On Tue, May 24, 2022 at 7:51 AM Damjan Marion <mailto:dmar...@me.com>> wrote: > > VPP is not DPDK application, VPP features doesnt maintain DPDK metadata, in > many cases we use native drivers so DPDK is not even loaded so you cannot use > rte_ring unless you want to write complete translation layer which will > likely be significantly slower that using native way which snort plugin uses. > > Why do you think zero copy interface in snort plugin is not simple? > > — > Damjan > >> On 24.05.2022., at 13:34, PRANAB DAS > <mailto:pkdas.bos...@gmail.com>> wrote: >> >> >> Thank you very much for your response Benoit! >> I am wondering if there is another option e.g. rte-ring of DPDK or if that >> option is not feasible in VPP? >> Could you comment? We are looking for a service-chaining application similar >> to snort but would like to have a much simpler zero-copy interface. >> >> Thank you, >> >> - P K DAS >> >> On Tue, May 24, 2022 at 4:22 AM Benoit Ganne (bganne) > <mailto:bga...@cisco.com>> wrote: >> It all depends on what you want to do. Usually, we try to avoid sharing >> buffers read/write between multiple process, it makes debugging much harder >> - especially buffer leaks or use-after-free... >> Here is an example to share VPP buffers read-only with snort so that snort >> can inspect traffic and gives a verdict back without any copy: >> https://git.fd.io/vpp/tree/src/plugins/snort >> <https://git.fd.io/vpp/tree/src/plugins/snort> >> >> Best >> ben >> >> > -Original Message- >> > From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> >> > mailto:vpp-dev@lists.fd.io>> On Behalf Of PRANAB DAS >> > Sent: Monday, May 23, 2022 20:11 >> > To: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> >> > Subject: Re: [vpp-dev] zero memcpy interface (alternative to memif) to >> > pass packets from VPP to another trusted DPDK application >> > >> > Hi >> > >> > Could some one comment on zero-copy alternative to memif interface? Any >> > ideas, comments are welcome. >> > >> > Thank you >> > >> > - Pranab K Das >> >> >> -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#21442): https://lists.fd.io/g/vpp-dev/message/21442 Mute This Topic: https://lists.fd.io/mt/91252298/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] zero memcpy interface (alternative to memif) to pass packets from VPP to another trusted DPDK application
Hi Damjan, The snort plugin could be simple. But I am not familiar with VPP memory/buffer management as such and I am finding it not so easy to navigate. And looking at the physmem.c/h code, it appears the VPP uses heap memory (mmap) and has its own heap/buffer management system. Could you clarify? And in the snort plugin exposes/shares the mmap memory region to the external program (snort). Is there any documentation that provides an overview of VPP buffer management and how VPP memory/buffers can be shared with external trusted applications? Thank you - PK Das On Tue, May 24, 2022 at 7:51 AM Damjan Marion wrote: > > VPP is not DPDK application, VPP features doesnt maintain DPDK metadata, > in many cases we use native drivers so DPDK is not even loaded so you > cannot use rte_ring unless you want to write complete translation layer > which will likely be significantly slower that using native way which snort > plugin uses. > > Why do you think zero copy interface in snort plugin is not simple? > > — > Damjan > > On 24.05.2022., at 13:34, PRANAB DAS wrote: > > > Thank you very much for your response Benoit! > I am wondering if there is another option e.g. rte-ring of DPDK or if that > option is not feasible in VPP? > Could you comment? We are looking for a service-chaining application > similar to snort but would like to have a much simpler zero-copy interface. > > Thank you, > > - P K DAS > > On Tue, May 24, 2022 at 4:22 AM Benoit Ganne (bganne) > wrote: > >> It all depends on what you want to do. Usually, we try to avoid sharing >> buffers read/write between multiple process, it makes debugging much harder >> - especially buffer leaks or use-after-free... >> Here is an example to share VPP buffers read-only with snort so that >> snort can inspect traffic and gives a verdict back without any copy: >> https://git.fd.io/vpp/tree/src/plugins/snort >> >> Best >> ben >> >> > -Original Message----- >> > From: vpp-dev@lists.fd.io On Behalf Of PRANAB DAS >> > Sent: Monday, May 23, 2022 20:11 >> > To: vpp-dev@lists.fd.io >> > Subject: Re: [vpp-dev] zero memcpy interface (alternative to memif) to >> > pass packets from VPP to another trusted DPDK application >> > >> > Hi >> > >> > Could some one comment on zero-copy alternative to memif interface? Any >> > ideas, comments are welcome. >> > >> > Thank you >> > >> > - Pranab K Das >> > > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#21441): https://lists.fd.io/g/vpp-dev/message/21441 Mute This Topic: https://lists.fd.io/mt/91252298/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] zero memcpy interface (alternative to memif) to pass packets from VPP to another trusted DPDK application
VPP is not DPDK application, VPP features doesnt maintain DPDK metadata, in many cases we use native drivers so DPDK is not even loaded so you cannot use rte_ring unless you want to write complete translation layer which will likely be significantly slower that using native way which snort plugin uses. Why do you think zero copy interface in snort plugin is not simple? — Damjan > On 24.05.2022., at 13:34, PRANAB DAS wrote: > > > Thank you very much for your response Benoit! > I am wondering if there is another option e.g. rte-ring of DPDK or if that > option is not feasible in VPP? > Could you comment? We are looking for a service-chaining application similar > to snort but would like to have a much simpler zero-copy interface. > > Thank you, > > - P K DAS > >> On Tue, May 24, 2022 at 4:22 AM Benoit Ganne (bganne) >> wrote: >> It all depends on what you want to do. Usually, we try to avoid sharing >> buffers read/write between multiple process, it makes debugging much harder >> - especially buffer leaks or use-after-free... >> Here is an example to share VPP buffers read-only with snort so that snort >> can inspect traffic and gives a verdict back without any copy: >> https://git.fd.io/vpp/tree/src/plugins/snort >> >> Best >> ben >> >> > -Original Message- >> > From: vpp-dev@lists.fd.io On Behalf Of PRANAB DAS >> > Sent: Monday, May 23, 2022 20:11 >> > To: vpp-dev@lists.fd.io >> > Subject: Re: [vpp-dev] zero memcpy interface (alternative to memif) to >> > pass packets from VPP to another trusted DPDK application >> > >> > Hi >> > >> > Could some one comment on zero-copy alternative to memif interface? Any >> > ideas, comments are welcome. >> > >> > Thank you >> > >> > - Pranab K Das > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#21440): https://lists.fd.io/g/vpp-dev/message/21440 Mute This Topic: https://lists.fd.io/mt/91252298/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] zero memcpy interface (alternative to memif) to pass packets from VPP to another trusted DPDK application
Thank you very much for your response Benoit! I am wondering if there is another option e.g. rte-ring of DPDK or if that option is not feasible in VPP? Could you comment? We are looking for a service-chaining application similar to snort but would like to have a much simpler zero-copy interface. Thank you, - P K DAS On Tue, May 24, 2022 at 4:22 AM Benoit Ganne (bganne) wrote: > It all depends on what you want to do. Usually, we try to avoid sharing > buffers read/write between multiple process, it makes debugging much harder > - especially buffer leaks or use-after-free... > Here is an example to share VPP buffers read-only with snort so that snort > can inspect traffic and gives a verdict back without any copy: > https://git.fd.io/vpp/tree/src/plugins/snort > > Best > ben > > > -Original Message- > > From: vpp-dev@lists.fd.io On Behalf Of PRANAB DAS > > Sent: Monday, May 23, 2022 20:11 > > To: vpp-dev@lists.fd.io > > Subject: Re: [vpp-dev] zero memcpy interface (alternative to memif) to > > pass packets from VPP to another trusted DPDK application > > > > Hi > > > > Could some one comment on zero-copy alternative to memif interface? Any > > ideas, comments are welcome. > > > > Thank you > > > > - Pranab K Das > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#21438): https://lists.fd.io/g/vpp-dev/message/21438 Mute This Topic: https://lists.fd.io/mt/91252298/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] zero memcpy interface (alternative to memif) to pass packets from VPP to another trusted DPDK application
Hi Could some one comment on zero-copy alternative to memif interface? Any ideas, comments are welcome. Thank you - Pranab K Das -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#21432): https://lists.fd.io/g/vpp-dev/message/21432 Mute This Topic: https://lists.fd.io/mt/91252298/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] zero memcpy interface (alternative to memif) to pass packets from VPP to another trusted DPDK application
Hello all, Is there any alternative to memif, that would allow packets from VPP to be passed to another DPDK application process without any memcpy? The DPDK supports multi-process where the primary process and secondary processes can share the same mempool/bufpool (disabling ASLR is required). Since VPP includes/ DPDK and launches as primary DPDK process (in eal-init), would it not be possible to put the vlib_buffer packets on a rte_ring which would allow the secondary DPDK processes to read/write the packet/vlib_buffer payload (assuming vlib header is untouched). I know this topic has been addressed or discussed in the past. I believe there are use-cases where there are two datapath containers one running VPP and the other a "trusted" datapath engine (DPDK or non-DPDK) and requires a high performance zero copy packet or memory interface between them. The memif performs memcpy on tx and rx to vlib buffer. Is there any way we can avoid such memcpy and share packets between two datapath containers/processes. DPDK supports external buffer/memory management and I believe VPP uses its own buffer/memory management and does not use DPDK default huge-pages mempool. Could you clarify. Can VPP/VLIB support external buffer management? E.g. exporting BSD mbufs to VLIB buffer and then to DPDK mbuf. Could VPP buffer management and memif experts (Damjan, Neale, Dave B, Benoit et al) and anybody else could provide some guidance in this regard? Do you foresee the need for a high-performance zero copy alternative to memif when the processes are trusted and can share memory? If so, what directions can one proceed? Really appreciate your help. Thank you very much - Pranab K Das -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#21426): https://lists.fd.io/g/vpp-dev/message/21426 Mute This Topic: https://lists.fd.io/mt/91252298/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
