Fiona and Keith, thanks for your help.
I've tried to set buf_addr/buf_iova of a mbuf on my own, but I guess
that I miss something because transferring such mbufs seems not to work.
Here are some more details:
I need to transfer video data with an UDP/RTP based protocol over
ethernet. First I create a rte_mempool holding the video frames (e.g.
5MB per frame). Then I create a rte_mempool for mbufs holding the
UDP/RTP header and another rte_mempool holding mbufs with no data to
store the addresses of the video frame segments:
struct rte_mempool* frame_pool = rte_mempool_create("FramePool", 32,
5*1024*1024, ...);
struct rte_mempool* hdr_pool = rte_pktmbuf_pool_create("HeaderPool",
1024 - 1, MBUF_CACHE_SIZE, 0, 256, SOCKET_ID_ANY);
struct rte_mempool* pay_pool = rte_pktmbuf_pool_create("PayloadPool",
1024 - 1, MBUF_CACHE_SIZE, 0, 0, SOCKET_ID_ANY);
void* framebuf;
rte_mempool_get(frame_pool, &framebuf);
Then I prepare the header mbuf (RTP header here not used):
struct rte_mbuf* mhdr = rte_pktmbuf_alloc(hdr_pool);
struct ether_hdr* eth_hdr = rte_pktmbuf_mtod(mhdr, struct ether_hdr *);
...
struct ipv4_hdr* ip_hdr = (struct ipv4_hdr*)(eth_hdr + 1);
...
struct udp_hdr* udp_hdr = (struct udp_hdr*)(ip_hdr + 1);
...
int payloadSize = 1024;
mhdr->nb_segs = 2;
mhdr->data_len = sizeof(struct ether_hdr) + sizeof(struct ipv4_hdr) +
sizeof(struct udp_hdr);
mhdr->pkt_len = m->data_len + payloadSize;
// don't know if the following lines is really that important
mhdr->ol_flags = 0;
mhdr->ol_flags |= PKT_TX_IPV4;
mhdr->ol_flags |= PKT_TX_IP_CKSUM;
mhdr->vlan_tci = 0;
mhdr->vlan_tci_outer = 0;
mhdr->l2_len = sizeof(struct ether_hdr);
mhdr->l3_len = sizeof(struct ipv4_hdr);
mhdr->l4_len = 0;
Then I prepared the payload mbuf and linked it to the header mbuf:
struct rte_mbuf* mpay = rte_pktmbuf_alloc(pay_pool);
mhdr->next = mpay;
mpay->buf_addr = framebuf;
mpay->buf_iova = rte_mempool_virt2iova(framebuf);
mpay->nb_segs = 1;
mpay->data_len = payloadSize;
mpay->pkt_len = m_hdr->pkt_len;
Not really sure if I filled all required mbuf members correctly. At
least when trying to transfer the mbuf using rte_eth_tx_burst(), it
doesn't work (rte_eth_tx_burst() returns no error but no data is
transfered by the NIC).
Important remarks: when I create the payload mbufs with sizes != 0, and
don't touch the buf_addr/buf_iova members, then the transfer works! I'm
using the Mellanox mlx5 PMD.
Hope to get some hints whats wrong.
Thanks
Sofia
On 09/10/2018 11:46 PM, Trahe, Fiona wrote:
-----Original Message-----
From: users [mailto:[email protected]] On Behalf Of Wiles, Keith
Sent: Monday, September 10, 2018 1:52 PM
To: Sofia Baran <[email protected]>
Cc: [email protected]
Subject: Re: [dpdk-users] dpdk and bulk data (video/audio)
On Sep 10, 2018, at 6:28 AM, Sofia Baran <[email protected]> wrote:
Hi All,
I want/try to us DPDK for transferring larger amount of data, e.g. video frames
which usually are
stored within memory buffers with sizes of several MB (remark: by using huges
pages, these buffers
could be physically contiguous).
When looking at the DPDK documentation, library APIs and examples, I can't find
a way/hint how to
transfer larger buffers using DPDK without copying the video buffer fragments
to the payload sections
of the mbufs - which results in high CPU loads.
Within the ip_fragmentation example indirect mbufs are used, pointing to the
payload section of a
direct mbuf (holding the header). But in my understanding the maximum size of a
mbuf payload is 65KB
(uint16_t)!?
It is true that mbufs only hold (64K - 1). The concept of mbufs is normally an
ethernet packet and they
are limited to 64K.
You can create a small mbuf (128 bytes) then set offset/data in the mbuf to
point to the video buffer
only if you can find the physical memory address for the data. The mbuf
normally holds the physical
address of the mbuf->data not the attached buffer in this case. This of course
means you have to
manage the mbuf internal structure members yourself and be very careful you do
not rearrange the
mbuf members as that can cause a performance problem.
But the 64k-1 limit still applies, unless I'm misunderstanding.
A way to get around this is to use chained mbufs.
So create lots of small mbufs, each 128bytes, holding no payload, just the mbuf
struct.
Chain them together with each buf_iova/buf_addr pointing to the next 64k-1
segment of the payload.
You'll need ~17 mbufs per MB, is this an acceptable overhead?
You can also consider using compressdev API to compress the data before transmitting.
However will have the same problem - and can use the same solution - for
passing the data to the PMD to be compressed/decompressed.
I'm pretty new to DPDK so maybe I missed something. I hope that someone can
provide me some hits
how to avoid copying the entire payload.
Thanks
Sofia Baran
Regards,
Keith
