Hi Marcus, Thanks for the suggestions. I have actually been monitoring /proc/net/udp during the runs and the buffers stay at 0 until the start time passes, leading me to believe the only buffering of samples is happening on the FPGA. Should I see these buffers fill up to 50 MB (or anything non-zero)?
Also, the kernel buffers have been resized at the command line and rechecked so I know at least that part was done correctly. I am also going to try this with different devices and fpga images Thanks, Patrick From: Marcus D. Leech <patchvonbr...@gmail.com> Sent: Tuesday, July 30, 2024 8:09 PM To: usrp-users@lists.ettus.com Subject: [USRP-users] Re: X440: UDP network buffer not utilized? *** WARNING: This message originates from outside the STR organization. *** *** Please take appropriate care in opening any links or attachments. *** On 30/07/2024 20:00, Panuski, Patrick via USRP-users wrote: Hi USRP users, I’m doing a timed transmission and I’m tracking how many samples are buffered before the transmission starts. I do this by calling send() with tx_metadata timestamps in the future and accumulating the return value from send() until it starts returning 0. Send() returns 0 until the start time occurs, after which the send() call starts returning a non-zero number again indicating samples are successfully being sent from the host. Based on this accumulated number, the total number of buffered samples is only 63,488 (=248 kB of data) despite having the linux kernel network buffer sizes (net.core.wmem_max and wmem_default settings) set to 50 MB. Is this expected? How can I increase the number of buffered samples between the host and device? Latency is not a concern. Should I be adjusting the samples per buffer or using timeout=0 here? Background: I’m using an X440 and my end goal is to get at least 4 channels coherently transmitting (i.e. stable phase relationship for an entire burst/session) at 50 MSps or greater on each channel for runs on the order of minutes. The problem I’m running into is that during transmission at these sample rates, it is very likely that one of the channels underflows, and even if it’s just a single underflow, it breaks the phase relationship with the other channels for the rest of the run since I’m using multithreaded streamers in a custom C++ program. The real core of my problem is trying to figure out why I’m underflowing at all. My hardware setup seems more than capable (see below); CPU usage per active core stays below 50% or so, the network traffic doesn’t and shouldn’t come anywhere close to the 4x 10GbE capacity, and the file reads from SSD are staying far ahead of the sender. Based on my testing, I’m virtually certain that a large-ish buffer between my UHD application and the X440 would solve all underflow issues, but it seems right now the buffer is only 248 kB, or about 1 ms @ 50 MSps. I have occasionally gotten a full transmission to complete without a single underflow. The onset of the underflow seems to randomly happen, which is another indicator that a larger buffer will help smooth these inconsistencies. Setup info: 1. UHD 4.6 on host and device, FPGA is running a customized image (X4_400 stock image but with the RAM replay block replaced with DDC/DUC) 2. Host Hardware – AMD threadripper, cores set to performance, Intel E810-CQDA2 NIC configured to split 1x 100GbE port into 4x logical 10GbE ports 3. Host Software – Ubuntu 20.04, running a C++ program that spawns 2 threads for each transmit channel; 1 “producer” for reading a file into a series of very large buffers in memory, and the other “consumer” for moving a pointer to the correct point in those buffers and calling send(). The producer threads start early and I have proven that they never fall behind the consumers. I’ve increased the net.core.wmem_max and wmem_default values to 50 MB, enabled tx pause frames on the NIC, maxed out the tx/rx descriptors on the NIC, and followed all the other tuning tips and tricks. I am not running DPDK as I didn’t think it would be necessary at these sample rates, although I could be wrong about that. Any help here would be much appreciated, thanks! Patrick _______________________________________________ USRP-users mailing list -- usrp-users@lists.ettus.com<mailto:usrp-users@lists.ettus.com> To unsubscribe send an email to usrp-users-le...@lists.ettus.com<mailto:usrp-users-le...@lists.ettus.com> You can look at /proc/net/udp when you have things running to check queue sizes, etc. Keep in mind that simply changing /etc/sysctl.conf will not do *anything* until the next reboot.
_______________________________________________ USRP-users mailing list -- usrp-users@lists.ettus.com To unsubscribe send an email to usrp-users-le...@lists.ettus.com
