Carlos, On the host side if you're using a single streamer, it will work to time-align samples from every USRP as they come in. The reason you see 1002 received packets without a timestamp match is because 1000 was the threshold for maximum number of alignment failures [1]. This can be changed or increased with [2]:
set_alignment_failure_threshold(1000); But I doubt this will solve the issue for you. Likely just delay it. >From the errors you included, I'd say the real problem is: > *ERROR] [X300] 192.168.100.2 <http://192.168.100.2/>: x300 fw > communication failure #1EnvironmentError: IOError: x300 fw poke32 - reply > timed out* > There is either a backup or a lack of samples coming from the radio. Do you see flow control errors leading up to this failure? I'd say that pulling pieces out of this system would be a good way to start troubleshooting: - Drop the sample rate (what is it, by the way?) and see if there's a threshold where things start working [3] - Keep the original sample rate and try removing a radio or two from the system. Does this help things? It sounds like this is something you can reproduce independent of your DSP blocks, which helps simplify things a bit. If you could go a bit further and find a simple example (i.e. take one of the GNURadio UHD shipping examples and expand it to use 3x USRPs) and get it to reproduce the behavior, then I'd be happy to set something up on my end try to reproduce the behavior you're seeing. Sam [1] http://lists.ettus.com/pipermail/usrp-users_lists.ettus.com/2017-August/053986.html [2] http://lists.ettus.com/pipermail/usrp-users_lists.ettus.com/2014-January/036210.html [3] http://lists.ettus.com/pipermail/usrp-users_lists.ettus.com/2014-November/039561.html On Thu, Oct 17, 2019 at 3:46 PM Carlos Bocanegra via USRP-users < usrp-users@lists.ettus.com> wrote: > Hello community, > > I am working on a gnuradio application that synchronously retrieves data > from a usrp_source, does some DSP on each of the streams and selects the > signal to send on a usrp_sink based on some metric computation. The sources > and sinks represent 3 USRP X310 devices, each with 2 UBX-160 > daughterboards, and are synchronized using an Octoclock-G. Thus, creating a > 6x1 MISO system. The USRP synchronization is done using the Python API, as > well as the flowgraph. The DSP blocks are coded in C++. The flowgraph would > be something like: > > *self.connect((usrp_source, 0), my_dsp_0, (my_switch, 0))* > *self.connect((usrp_source, 1), my_dsp_1, (my_switch, 1))* > *self.connect((usrp_source, 2), my_dsp_2, (my_switch, 2))* > *self.connect((usrp_source, 3), my_dsp_3, (my_switch, 3))* > *self.connect(my_switch, another_dsp_0, (usrp_sink, 0))* > > The application runs for a while until UHD outputs a message saying that > the poke32 has timed out on one of the USRP. After that, a misalignment > error on the receiver streams is shown and the green lights on the USRP go > off. > > We are under the impression that the DSP blocks working independently may > be requesting samples from the source at different times, causing the > misalignment. Could this be the case? Is there an issue with my network > connection? > > I have tried to get rid of the misalignment error by increasing the > threshold in the file > *{uhd_prefix}/host/lib/transport/super_recv_packet_handler.hpp, *through > the function *set_alignment_failure_threshold.* That got rid of the > misalignment error, but the failure coming from the poke function is still > there. > > Interestingly, the exact same behavior arises when I just connect > file_sinks to the outputs from the usrp_source and a simple analog signal > to the usrp_sink. Below the flowgraph: > > *self.connect((usrp_source, 0), file_sink_0)* > self.connect((usrp_source, 1), file_sink_1) > self.connect((usrp_source, 2), file_sink_2) > self.connect((usrp_source, 3), file_sink_3) > *self.connect(analog_sig_c, (usrp_sink, 0))* > > I'd appreciate any help on this since I'm running out of ideas here. > Please, find below more details. > > *The error:* > > * [ERROR] [X300] 192.168.100.2 <http://192.168.100.2>: x300 fw > communication failure #1EnvironmentError: IOError: x300 fw poke32 - reply > timed out* > > * [ERROR] [X300] 192.168.100.2 <http://192.168.100.2>: x300 fw > communication failure #1EnvironmentError: IOError: x300 fw poke32 - reply > timed out* > > * [ERROR] [X300] 192.168.100.2 <http://192.168.100.2>: x300 fw > communication failure #3EnvironmentError: IOError: x300 fw poke32 - reply > timed out* > [ERROR] [UHD] An unexpected exception was caught in a task loop.The task > loop will now exit, things may not work.EnvironmentError: IOError: > 192.168.100.2: x300 fw communication failure #3 > > > > * [ERROR] [STREAMER] The receive packet handler failed to time-align > packets.1002 received packets were processed by the handler.However, a > timestamp match could not be determined.* > > *Here are some details of my network configuration:* > - UHD built from source, tag v3.14.1.1. > - Gnuradio built from source, tag 3.7.13.5. > - X310 USRP, each equipped with two UBX-160 daughterboards. > - The communication is over Ethernet using the 1Gb port on the X310. > - Each USRP is connected to a dedicated NIC. > - The MTU is set to 1500 for the all interfaces. > - The USRPs are synchronized using an Octoclock-G and the parameters tuned > in the Python script. The procedure is similar to the one followed in > benchmark_rate (uhd), for the usrp_source and usrp_sink. > - The NIC ring buffers are set to the maximum for both TX and RX, to 4096, > using ethtool. > - The OS ring buffers are set as suggested bu Ettus, using the following > commands: > > > > *>> sudo sysctl -w net.core.rmem_max=33554432 >> sudo sysctl -w > net.core.wmem_max=33554432 >> sudo sysctl -w > net.core.rmem_default=33554432 >> sudo sysctl -w > net.core.wmem_default=33554432* > > Best, > Carlos > _______________________________________________ > USRP-users mailing list > USRP-users@lists.ettus.com > http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com >
_______________________________________________ USRP-users mailing list USRP-users@lists.ettus.com http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
