Thanks to those who helped.
The problem has disappeared. I hesitate to say “resolved” because we don’t know
exactly what the problem was. We knew there was a problem with our
configuration, but not what the problem was – just hoping that someone with
more experience could tell us where to look since we only followed instructions
provided by ettus on line. I will detail our findings for the event that it is
helpful to someone else.
It seems to have been related to the FPGA image since that was the last thing
we changed before behaviour returned to expected (ie 25MSPS transfer across a
1GbE link without any drop outs).
The set up is:
* X310 with a1GbE SFP+ module in socket 0, and a generic but ‘intel
compatible’ 10GbE SFP+ module in socket 1. The IP addresses have been altered
to not cause conflicts on our LAN.
* The 1GbE connection connects to the LAN
* The 10GbE connection connects to a 1GbE connection on the Xavier
* nVidia AGX Xavier with a 2x 10GbE PCIe card inserted. The same generic
10GbE SFP+ modules do not work in the PCIe card, so we have a 1GbE SFP+ in one
of the sockets.
* The on board 1GbE connects to the LAN
* The 1GbE on the PCIe card is connected to the 10GbE module on the X310
* The xavier has UHD 4.0.0 + gnu radio 3.8 + gr-ettus (RFNoC) 4.0
installed which is the setup described in the workshop for RFNoC 4
(https://kb.ettus.com/images/e/e9/rfnoc4_workshop_slides_2020_part_2.pdf)
The X310 can be ‘seen’ by the usrp-probe software either across the LAN or
directly but always seems to go via the direct link first (reports the direct
IP address when connecting and no loss of data if either LAN cable is unplugged
while running). Works similarly via both routes.
We setup a simple USRP source to QT frequency sink flow. Any time the sample
rate was set above 2MSPS the letter ‘D’ (drop out = sequence numbers not in
sequence) was output repeatedly to the console. This was the only ‘error’.
Coincidentally the 2MSPS rate would be the largest sample rate we would expect
to receive reliably if the ethernet negotiation had resolved to a 100Base-TX
link. However the interface card (on the xavier) reported a 1000mb/s link.
When the X310 arrived this setup would not work until we updated the FPGA image
to one compatible with uhd 4.0.0. This was done following the instructions here
: https://files.ettus.com/manual/page_usrp_x3x0.html#x3x0_flash
We tried all possible combinations of ethernet interfaces with the same
results. At some point we stopped using gnuradio-companion in favour of the
benchmark_rate application but the results were the same – dropouts above
2MSPS. We tried a different computer (windows PC with ubuntu VM) but the same
results. We then tried a newer version of UHD (4.1.3 I think). In this case the
‘XG’ image was programmed. The results were very similar. The last test was to
be a direct link between the on-board ethernet of the xavier (rather than via
the PCI ethernet interfaces) and the 1GbE module on the X310. The xavier only
had the 4.0.0 uhd software installed so it was necessary to replace the FPGA
image for uhd 4.0.0. Initially this was done with the ‘XG’ image
(unintentionally) but then we changed back to the ‘HG’ image.
After this we found that the benchmark_rate script would work up to 16MSPS
without dropouts, and above 16MSPS it would crash (reset) the Xavier. Thinking
this a vast improvement over what we were getting (16MSPS vs 2MSPS) we replaced
the cables to the configuration described above. Coincidentally we discovered
that gnuradio-companion will now work at 25MSPS without dropouts. So we can get
25MSPS transfer across a 1GbE link from a 10GbE SFP+ module in slot 1 of the
X310 to a 1GbE SFP+ module on a PCIe card on the Xavier platform.
What changed? No idea. But its working, so there’s no plan to investigate
further, unless performance is inadequate when we change to a 10GbE SFP+ module
on the xavier.
Kind regards
Jonathan Pratt
From: Rob Kossler <[email protected]>
Sent: Thursday, 2 December 2021 1:23 AM
To: Marcus D. Leech <[email protected]>
Cc: [email protected]
Subject: [USRP-users] Re: USRP streaming data performance
WARNING: This message has originated from an untrusted source. Be mindful of
attachments and embedded links.
Perhaps post your actual command line and errors? Are you running 1 or 2
channels?
It is my understanding (not necessarily correct) that a "D" is an error
generated on the CPU side (not an error message from the FPGA). So, I think
that the FPGA does not know anything bad is happening as it sends all of the
sequential packets. The CPU verifies the sequence index in each received
packet and generates the "D" error when it detects an out-of-sequence packet
(indicating that packets got dropped or swapped along the way). This differs
from an "O" error which comes from the FPGA (radio block) when the radio has
data ready to stuff in a FIFO but the FIFO is full because a downstream
component (likely UHD running on the CPU) is not consuming fast enough and
backpressures the stream.
On Tue, Nov 30, 2021 at 7:19 PM Marcus D. Leech
<[email protected]<mailto:[email protected]>> wrote:
On 2021-11-30 19:14, Jonathan Pratt wrote:
Have looked into the benchmark rate utility (uhd/host/build/examples/) which
shows dropouts at any sample rate above 2MSPS. Have also run the same thing
from a PC laptop running ubuntu 20.04 in a virtual machine with the same
gnuradio (3.8), same uhd (4.0.0) and same gr-ettus (RFNoC 4). It also flags
dropouts consistently at any sample rate above 2MSPS. The fpga image for the
X310 was downloaded and installed according to the instructions in the hardware
manual (https://files.ettus.com/manual/page_usrp_x3x0.html). It appears to us
that the issue lies with something to do with the X310 or software that is
communicating with it, or the fpga image.
Is there any setup item we can change to get net traffic without dropouts? Will
there be a problem if we don’t drain data from the other receives at the same
time since there are four of them?
Thanks
Jonathan Pratt
LOTS of people on this list *routinely* stream data out of their X310s even
over 1Gbit links at MUCH MUCH higher sample rates. What kind of 1Gbit or
10Gbit interface do you
have? Are you using the SFP+ ports or the RJ-45 port?
What kind of computer is this on? We generally DO NOT recommend virtual
machine implementations because the performance tends to suffer, PARTICULARLY
the Network and USB performance.
From: Marcus D. Leech <[email protected]><mailto:[email protected]>
Sent: Tuesday, 30 November 2021 10:35 AM
To: [email protected]<mailto:[email protected]>
Subject: [USRP-users] Re: USRP streaming data performance
WARNING: This message has originated from an untrusted source. Be mindful of
attachments and embedded links.
On 2021-11-29 18:22, Jonathan Pratt wrote:
We are looking to develop a standalone sdr platform connecting an nVidia Jetson
AGX Xavier to a USRP X310. The X310 has two dual receiver boards installed but
we are only trying to stream data from one core at this time.
The Xavier is an octacore ARM platform with all cores enabled and running close
to 1.5GHz.
The connection between the two devices is via ethernet running at 1Gbit. The
xavier has a x16 PCIe interface connector and we are using a NIC with 1Gbps SFP
module – we intend to run the link at 10Gbit in the future. The onboard 1Gbps
ethernet is connected to our LAN
The Xavier is running ubuntu 18.04 for arm. This is the development platform
that nVidia provides. uhd 4.0.0, gnuradio 3.8 and RfNoC 4 have been installed
on the Xavier.
The Xavier is given a simple flow to run under gnuradio-companion which
consists of a USRP Source connected directly to the QT GUI Frequency sink (or
Null Sink)
The network buffers and mtu on the xavier are increased to at least those
recommended.
The result we get is a whole lot of ‘D’s output if we increase the sample rate
beyond 2MSPS. The same result is found if we run the flow from the command
line (without the gui).
For comparison we have done a similar setup with a USRP E312 – connected to a
xavier, a ubuntu 20 virtual machine and directly to a computer running ubuntu
20.04. In each case when we increase the sample rate beyond 2MSPS we get ‘O’s
output.
Can anyone please indicate what setup is required to achieve the 25MSPS that
should be possible across this link?
Thanks in advance
_______________________________________________
USRP-users mailing list --
[email protected]<mailto:[email protected]>
To unsubscribe send an email to
[email protected]<mailto:[email protected]>
You might want to run "benchmark_rate" from the UHD examples code to eliminate
GR entirely at first--just to get a feel for what your machine is capable of.
I'll note that the "network mode" in E312 (where it streams to a regular PC)
has *considerable* performance constraints, and achieving even 2Msps is a bit
of a
miracle.
The X310, on the other hand, is Niagara Falls. Any streaming-performance
issues are your host. The FPGA on the X310 can stream to the xGIGe interfaces
as fast as
physics allows, pretty much.
For a "order of magnitude" benchmark, I can achieve 10Msps from a B2xx into an
Odroid XU4Q with 8-bit samples (there's a USB bandwidth issue there). I can
even
"do stuff" at 10Msps, including several different radio astronomy signal
processing chains. I would expect your Xavier to be similar.
_______________________________________________
USRP-users mailing list --
[email protected]<mailto:[email protected]>
To unsubscribe send an email to
[email protected]<mailto:[email protected]>
_______________________________________________
USRP-users mailing list --
[email protected]<mailto:[email protected]>
To unsubscribe send an email to
[email protected]<mailto:[email protected]>
_______________________________________________
USRP-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
