I tried Brian’s suggestion to rebuild UHD and the FPGA off of the commits he suggested (thanks Brian). However, with this combination I am getting significantly more underruns than I did previously, even with the benchmark rate program. Here’s the output of benchmark_rate that I got originally with UHD 4.0.0.rfnoc-devel-788-g1f8463cc.
./benchmark_rate --rx_rate 100e6 --tx_rate 100e6 --channels="0,1" Benchmark rate summary: Num received samples: 2016651900 Num dropped samples: 0 Num overruns detected: 0 Num transmitted samples: 2005972016 Num sequence errors (Tx): 0 Num sequence errors (Rx): 0 Num underruns detected: 562 Num late commands: 0 Num timeouts (Tx): 0 Num timeouts (Rx): 0 And now I get this with UHD 3.14.0.HEAD-31-g98057752. Benchmark rate summary: Num received samples: 2001309816 Num dropped samples: 0 Num overruns detected: 0 Num transmitted samples: 1841996424 Num sequence errors (Tx): 0 Num sequence errors (Rx): 0 Num underruns detected: 353655 Num late commands: 0 Num timeouts (Tx): 0 Num timeouts (Rx): 0 One difference I did notice between these two versions of UHD is the maximum samples per packet returned from the get_max_num_samps() function for both the Rx and Tx streams. With the version from the rfnoc-devel branch, I get 1996 samples for both the Rx and Tx streams. But, the UHD 3.14.0 version gives 1996 samples for the Rx stream but only 996 samples for the Tx stream. I’m not sure if this is causing the additional underruns. In any case, was a change made to limit the number of transmit samples per packet? Are there additional network configurations that I need to make to increase the maximum samples per packet for the Tx stream or to limit the underruns with these versions of UHD and the FPGA firmware? BTW, setting “spp” in the transmit stream args does not allow more than the 996 samples per packet. Thanks, Alan From: Brian Padalino <[email protected]> Sent: Tuesday, August 28, 2018 8:57 PM To: Alan Conrad <[email protected]> Cc: [email protected] Subject: Re: [USRP-users] Transmit Thread Stuck Receiving Tx Flow Control Packets On Tue, Aug 28, 2018 at 4:02 PM Alan Conrad via USRP-users <[email protected]<mailto:[email protected]>> wrote: Hi All, I’ve been working on an application that requires two receive streams and two transmit streams, written using the C++ API. I have run into a problem when transmitting packets and I am hoping that someone has seen something similar and/or may be able to shed some light on this. My application is streaming two receive and two transmit channels, each at 100 Msps over dual 10GigE interfaces (NIC is Intel X520-DA2). I have two receive threads, each calling recv() on separate receive streams, and two transmit threads each calling send(), also on separate transmit streams. Each receive thread copies samples into a large circular buffer. Each transmit thread reads samples from the buffer to be sent in the send() call. So, each receive thread is paired with a transmit thread through a shared circular buffer with some mutex locking to prevent simultaneous access to shared circular buffer memory. I did read in the UHD manual that recv() is not thread safe. I assumed that this meant that recv() is not thread safe when called on the same rx_streamer from two different threads but would be ok when called on different rx_streamers. If this is not the case, please let me know. On to my problem… After running for several minutes, one of the transmit threads will get stuck in the send() call. Using strace to monitor the system calls it appears that the thread is in a loop continuously calling the poll() and recvfrom() system calls from within the UHD API. Here’s the output of strace attached to one of the transmit threads after this has occurred. These are the only two system calls that get logged for the transmit thread once this problem occurs. 11:19:04.564078 poll([{fd=62, events=POLLIN}], 1, 100) = 0 (Timeout) 11:19:04.664276 recvfrom(62, 0x5619724e90c0, 1472, MSG_DONTWAIT, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) 11:19:04.664381 poll([{fd=62, events=POLLIN}], 1, 100) = 0 (Timeout) 11:19:04.764600 recvfrom(62, 0x5619724e90c0, 1472, MSG_DONTWAIT, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) 11:19:04.764699 poll([{fd=62, events=POLLIN}], 1, 100) = 0 (Timeout) 11:19:04.864906 recvfrom(62, 0x5619724e90c0, 1472, MSG_DONTWAIT, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) This partial stack trace shows that the transmit thread is stuck in the while loop in the tx_flow_ctrl() function. I think this is happening due to missed or missing TX flow control packets. #0 0x00007fdb8fe4fbf9 in __GI___poll (fds=fds@entry=0x7fdb167fb510, nfds=nfds@entry=1, timeout=timeout@entry=100) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007fdb9186de45 in poll (__timeout=100, __nfds=1, __fds=0x7fdb167fb510) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 uhd::transport::wait_for_recv_ready (timeout=0.10000000000000001, sock_fd=<optimized out>) at /home/aconrad/rfnoc/src/uhd/host/lib/transport/udp_common.hpp:59 #3 udp_zero_copy_asio_mrb::get_new (index=@0x55726266f6e8: 28, timeout=<optimized out>, this=<optimized out>) at /home/aconrad/rfnoc/src/uhd/host/lib/transport/udp_zero_copy.cpp:79 #4 udp_zero_copy_asio_impl::get_recv_buff (this=0x55726266f670, timeout=<optimized out>) at /home/aconrad/rfnoc/src/uhd/host/lib/transport/udp_zero_copy.cpp:226 #5 0x00007fdb915d48cc in tx_flow_ctrl (fc_cache=..., async_xport=..., endian_conv=0x7fdb915df600 <uhd::ntohx<unsigned int>(unsigned int)>, unpack=0x7fdb918b1090 <uhd::transport::vrt::chdr::if_hdr_unpack_be(unsigned int const*, uhd::transport::vrt::if_packet_info_t&)>) at /home/aconrad/rfnoc/src/uhd/host/lib/usrp/device3/device3_io_impl.cpp:345 The poll() and recvfrom() calls are in the udp_zero_copy_asio_mrb::get_new() function in udp_zero_copy.cpp. Has anyone seen this problem before or have any suggestions on what else to look at to further debug this problem? I have not yet used Wireshark to see what’s happening on the wire, but I’m planning to do that. Also note that, if I run a single transmit/receive pair (instead of two) I don’t see this problem and everything works as I expect. My hardware is an X310 with the XG firmware and dual SBX-120 daughterboards. Here are the software versions I’m using, as displayed by the UHD API when the application starts. [00:00:00.000049] Creating the usrp device with: addr=192.168.30.2,second_addr=192.168.40.2... [INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_4.0.0.rfnoc-devel-788-g1f8463cc The host is a Dell PowerEdge R420 with 24 CPU cores and 24 GB ram. I think the clock speed is a little lower than recommended at 2.7 GHz but thought that I could distribute the work load across the various cores to account for that. Also, I have followed the instructions to setup dual 10 GigE interfaces for the X310 here, https://kb.ettus.com/Using_Dual_10_Gigabit_Ethernet_on_the_USRP_X300/X310<https://protect-us.mimecast.com/s/MKjlC31KyLFp6Lm5cgYuvL?domain=kb.ettus.com>. Any help is appreciated. I think you're hitting this: https://github.com/EttusResearch/uhd/issues/203 Which is the same thing that I hit. I tracked it down to something happening in the FPGA with the DMA FIFO. I rebuilt my FPGA and UHD off the following commits, which switch over to byte based flow control: UHD commit 98057752006b5c567ed331c5b14e3b8a281b83b9 FPGA commit c7015a9a57a77c0e312f0c56e461ac479cf7f1e9 And the problem disappeared for the time being. The infinite loop still exists as a potential issue, but it seemed whatever was causing the lockup in the DMA FIFO disappeared or at least couldn't be reproduced. Give that a shot and see if it works for you, or if you can still reproduce it? We never got to the root cause of the problem. Brian
_______________________________________________ USRP-users mailing list [email protected] http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
