dpdk-devbind.py fail on AWS c5n.metal instance running 'Amazon Linux 2 Kernel 5.10'

Using AWS I have, * c5n.metal instance * running 64-bit x86 Amazon Linux 2 AMI (HVM) - Kernel 5.10 ( ami-0ed9277fb7eb570c9) * DPDK 22.03-rc0 * I've installed the ena kernel fix in https://github.com/amzn/amzn-drivers/tree/master/userspace/dpdk/enav2-vfio-patch I add a second NIC to the instance

solved: dpdk-devbind.py fail on AWS c5n.metal instance running 'Amazon Linux 2 Kernel 5.10

An earlier email sent today to this distribution incorrectly claimed that dpdk-devbind.py was not working on Amazon Linux 2 Kernel 5.10. This was my error. IOMMU was not turned on. I thought it default ON; it does not even for c5n.metal instances. The correct steps are: * follow

Concept clarification on NICS with multiple queues

My NIC has 32 RX and 32 TX queues. Now given this API, static uint16_t rte_eth_tx_burst ( uint16_t port_id, uint16_t queue_id, struct rte_mbuf ** tx_pkts, uint16_t nb_pkts ) I gather the most natural approach for efficient packet transmissions

DPDK support for UDP RPCs with congestion handle, packet loss

The paper: https://www.usenix.org/system/files/nsdi19-kalia.pdf Is a UDP only library built on top of DPDK delivering (or direct to libverbs for Mellanox) that, "handles packet loss, congestion, and background request execution. In microbenchmarks, one CPU core can handle up to 10 million small

If or how one gets an IP address associated with a vfio-pci bound NIC

I'm trying to use DPDK on AWS i3.metal instances. I have the code built with AWS vfio-patches. In order to be logged into the machine on one NIC while having a free ENA NIC for DPDK, I attached a second NIC. ./dpdk-devbind.py is able to see the second NIC, and bind to it. *All that's working

RSS Hash suport in DPDK on Amazon ENA NICs broken?

On AWS EC2 I configure: - c5n.metal instance - Stock AWS AMI Ubuntu 20.04 LTS ($ cat /proc/version Linux version 5.4.0-1045-aws) - Apply the vfio pci driver from https://github.com/amzn/am zn-drivers/tree/master/userspace/dpdk/enav2-vfio-patch - Using DPDK code from head of main branch. I built

Re: If or how one gets an IP address associated with a vfio-pci bound NIC

> > David wrote: > >I don't think you understand the intent behind the DPDK framework. > > Thank you. Indeed I stand corrected. Associating an IP address with the DPDK NIC works from a bad frame on the problem. I'd appreciate one additional bit of information if possible. Once the DPDK NIC is

drivers/common/cnxk warning

In recent versions of DPDK: ../drivers/common/cnxk/roc_irq.c:53:7: note: limit is 9223372036854775807 bytes, but argument is 18446744073709551548 [2920/3102] Linking target app/dpdk-testpmd. ../drivers/common/cnxk/roc_irq.c: In function ‘irq_config’: ../drivers/common/cnxk/roc_irq.c:53:7:

(AWS ENA NIC) queued v. out packets reported by any of the DPDK stats functions

On DPDK head commit (917229) I have a simple program creating two TXQs each sending 15000 IPV4 UDP packets. As far as I can see there are no errors anymore. I have looked. I burst TX them in blocks of 15; rte_eth_tx_burst always returns 15. The lcores are launched with rte_eal_mp_remote_launch,

Introduction to sending UDP packets through DPDK with code

I've asked and received several responses on DPDK in this forum. And along the way I've noticed others asking the same questions: - what's an lcore - memory pools - creating and sending packets - etc You may then find this link helpful: https://github.com/rodgarrison/reinvent which covers all

Re: [dpdk-users] DPDK RX TCP checksum failed

https://github.com/rodgarrison/reinvent demonstrates ether/UDP chec-kum offload running on AWS which uses virtio. I don't think TCP offload is substantially different. Look here:

Preliminary AWS ENA Benchmark Results on DPDK

https://github.com/rodgarrison/reinvent#benchmarks I show two c5n.metal boxes with AWS ENA NICs running under DPDK in the same data center running 1 TXQ and 1 RXQ transceiving 72-byte UDP packets (32-byte payload): - transmit around 874,410 packets/sec/queue If the NIC's TXQ ring is not full

Seeking H/W hosting companies not using virtualized NICs

I recently sent an email to this user group on benchmarking AWS ENA NICs on DPDK. Being a large cloud provider, ENA NICs' are virtual. Anybody know an alternate hosting company that offers vanilla Intel Xeon HW with regular run-of-the-mill-plugged-into-the-box-PCI-bus NICs? Non-virtualized HW is

Mellanox/DPDK links

Where is the latest (most current) downlink for: DPDK-MLNX: Mellanox's customization of DPDK? MLNX_DPDK_Quick_Start_Guide? I cannot get a list of available versions Nvidia's download page and the only versions I can find:

mlx5_common: No Verbs device matches PCI device 0000:01:00.1

I built the current version of DPDK directly from dpdk.org after I installed the current OFED Mellanox driver set: * MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64.iso with ./install --dpdk I am using a Mellanox Technologies MT27710 Family [ConnectX-4 Lx] which is Ethernet only; there is no IB

mbuf memory layout

I create a mempool thusly with 0 bytes private size and 1024 bytes data room size: rte_mempool *pool = rte_pktmbuf_pool_create("test", 1024, 0, 0, 1024, 0); from which I allocate 1 mbuf. which of the following three memory layouts best describes mbufs I allocate from this pool? I can't tell if

Fwd: mbuf memory layout

Darn it - typo: the pointer to the start of the data room in a mbuf was incorrectly described earlier. I should have written: rte_pktmbuf_mtod(mbuf, ); // provide a pointer to the start of the data room. From: fwefew 4t4tg <7532ya...@gmail.com> Date: Tue, Feb 1, 2022 at 1:16 PM S

Re: allocating a mempool w/ rte_pktmbuf_pool_create()

stay in the pool and will never be used. The access to the per-lcore table is of course faster than the multi-producer/consumer pool. The cache can be disabled if the cache_size argument is set to 0; it can be useful to avoid losing objects in cache. On Sat, Jan 29, 2022 at 9:29 PM fwefew 4t4tg <

Re: allocating a mempool w/ rte_pktmbuf_pool_create()

2022-01-29 18:46 (UTC-0500), fwefew 4t4tg: > [...] > > 1. Does cache_size include or exclude data_room_size? > > 2. Does cache_size include or exclude sizeof(struct rtre_mbuf)? > > 3. Does cache size include or exclude RTE_PKTMBUF_HEADROOM? > > Cache size is mea

allocating a mempool w/ rte_pktmbuf_pool_create()

The API rte_pktmbuf_pool_create() https://doc.dpdk.org/api/rte__mbuf_8h.html#a593921f13307803b94bbb4e0932db962 at first glance seems minimal, complete. It's not. It's really not. The doc's link to `rte_mempool_create()` helps a little but not much. It lacks at least the following important

Question: DPDK thread model

I completed code to initialize an AWS ENA adapter with RX, TX queues. With this work in hand, DPDK creates one thread pinned to the right core as per the --lcores argument. So far so good. The DPDK documentation and example code is fairly clear here. What's not as clear is how RX packets are

re: ports and queues

Which is the preferred relationship? * N receive queues share one (TCP/UDP) port whence all TX queues send to this port or * Each transmit queue chooses a TCP/UDP port where, on the RX side, each receive queue listens on its own port for that queue. I'm betting it's the second case. However,

Re: mlx5_common: No Verbs device matches PCI device 0000:01:00.1

e a single port (mlx5_bond_0). You should only probe the > first PCI BDF - 01:00.0, not the 2nd one. > > By the way, the --dpdk installation flag should not be necessary, it is an > old flag keps for Mellanox OFED builds lower than 5.x. > > Regards, > Erez > > On Tue, 5 Apr 2022 a

Re: mlx5_common: No Verbs device matches PCI device 0000:01:00.1

wrote: > I assume your tree assumes there's a ConnectX-3 device installed, while > the kernel driver doesnt support it for quite a while > I would suggest re-compiling while excluding mlx4 PMD support. > > Thanks, > Erez > > On Tue, 5 Apr 2022 at 23:54, fwefew 4t4tg <7532ya..

Re: need definitions / references

ire. This seems to have some connection to tx_rs_thresh. On Thu, Mar 17, 2022 at 12:48 PM Stephen Hemminger < step...@networkplumber.org> wrote: > On Thu, 17 Mar 2022 12:05:33 -0400 > fwefew 4t4tg <7532ya...@gmail.com> wrote: > > > DPDK provides but does not define (that I can see

need definitions / references

DPDK provides but does not define (that I can see at least) these TXQ config fields. What are these fields what do they do? https://doc.dpdk.org/api/structrte__eth__thresh.html uint8_t pthresh uint8_t hthresh uint8_t wthresh And here: What is the RS bit?

Hosted DPDK development using Mellanox NICs

This is a follow-up to earlier questions I posted to this group on alternatives to AWS for DPDK development. >From https://news.ycombinator.com/ I was turned on to Equinix from which bare metal boxes are offered. I found they have better hardware for this purpose at less than half the cost. The

Re: Is there a way to instruct NIC/DPDK which RXQ to deliver a packet to not RSS?

s/prog_guide/rte_flow.html Thank you! That was not on my radar for some reason. Much appreciated. On Sun, Dec 18, 2022 at 5:14 PM Stephen Hemminger < step...@networkplumber.org> wrote: > On Sun, 18 Dec 2022 16:41:13 -0500 > fwefew 4t4tg <7532ya...@gmail.com> wrote: > > >

DPDK and DMA

Consider a valid DPDK TXQ with its mempool of rte_mbufs. Application code will allocate a mbuf from the pool and prepare it with headers, data, and so on. When the mbuf(s) are enqueued to the NIC with rte_eth_tx_burst() does DPDK DMA the memory into the NIC? Is this an optimization worth

Re: DPDK and DMA

with it. On Wed, Jan 11, 2023 at 6:26 AM Dmitry Kozlyuk wrote: > 2023-01-08 16:05 (UTC-0500), fwefew 4t4tg: > > Consider a valid DPDK TXQ with its mempool of rte_mbufs. Application code > > will allocate a mbuf from the pool and prepare it with headers, data, and > > so on. &g

Is there a way to instruct NIC/DPDK which RXQ to deliver a packet to not RSS?

I am aware of RSS; I coded it and have gotten it to work between machines. However, the elapsed time per packet increases from <100ns/packet RSS-OFF to 700-800ns RSS-ON in my test setup. In my scenario I have hardcoded routing between TXQs on client machines and RXQs on the server machines.

Re: Using rdtsc to timestamp RTT of packets

, and answers the issue of invariance/constancy clearly. On Mon, Mar 6, 2023 at 11:56 AM Stephen Hemminger < step...@networkplumber.org> wrote: > On Sun, 5 Mar 2023 20:01:15 -0500 > fwefew 4t4tg <7532ya...@gmail.com> wrote: > > > I think rdtsc does all this. But then I read

Callback or hook into completion queues?

Once I call rte_eth_tx_burst() (Mellanox Connect5-LX) is there a way to inspect or get a callback when transmitted packets go into the NIC's completion queue? This is related to the earlier question on timestamping with rdtsc(). Ideally I'd take the timestamp as soon (close) to the time the

Using rdtsc to timestamp RTT of packets

I convinced myself that a viable way to measure timestamps between a request packet and its response packet can be the difference between two Intel rdtsc calls The restrictions to valid use include: - RTT (time difference) must be calculated on the same CORE - fencing instructions (lfence)

DPDK and full TCP-IP stack

https://stackoverflow.com/questions/65841190/does-dpdk-provide-a-native-tcp-ip-network-stack-implemetation points out there's no native TCP-IP in DPDK stack until v20.11. We're at 23.03. Was this completed?

Re: SRIOV

If you have configured *SR-IOV on baremetal* to make virtual NICs and have used *TC to mirror packets* between two virtual NICs, please contact me at 7532ya...@gmail.com. I need some assistance. We can talk off-line.

--lcores: what it does and does not do

I've made a DPDK test program that does the following on the same machine and for the same NIC. This is a test; it does nothing practical: * Creates 1 RX queue then reads and pretty prints contents in a loop * Creates 1 TX queue then sends packets to a hardcoded IP address in a loop When I run

Re: --lcores: what it does and does not do

@1,(1)@2" which is passed to 'rte_eal_init'. On Sat, Mar 30, 2024 at 12:52 PM Stephen Hemminger < step...@networkplumber.org> wrote: > On Sat, 30 Mar 2024 12:06:15 -0400 > fwefew 4t4tg <7532ya...@gmail.com> wrote: > > > I've made a DPDK test program that does th