Hi Rob, Thanks again for helping :)
> Gesendet: Mittwoch, 06. Mai 2020 um 10:11 Uhr > Von: "Rob Kossler" <[email protected]> > An: "Lukas Haase" <[email protected]> > Cc: "[email protected]" <[email protected]> > Betreff: Re: [USRP-users] Unpredictable delays in loopback configuration > > > After lot of debugging I have some progress: > > 1.) Having multiple channels is not enough since it only sets > > _stream_now=FALSE but not _start_time. Consequently both Source+Sink use > > get_time_now()+0.1 which may be different > I think that you need to call get_time_now()+0.1 just once (not > separately for source and sink) and use the result for both. What I meant is that gr-uhd calls them (in this default configuration). What happens is: 1.) usrp_source_impl::start() calls get_time_now()+0.1 2.) usrp_sink_impl::start() calls get_time_now()+0.15 Both are called automatically. This can (and must) be overwritten by explicitely calling set_start_time on each. > > 2.) set_unknown_pps only sets the the time to zero at next pps > I agree. To make sure: Do you know if the flowgraph starts before the time is set on the next pps? For example, if I have [Signal Source] --> [USRP Sink], does [Signal Source] start producing samples already which will be buffered (and potentially overflow) or will all of this start once time is acquired? > > 4.) set_start_time *has* to be called on *both* USRP Source/Sink. This > > can't be done via grc and requires py edits. > set_start_time is a gnuradio and/or gr-ettus function that has no > equivalent in UHD. So, I don't really have experience with this one. Yes. Basically they just set state variables: void usrp_sink_impl::set_start_time(const ::uhd::time_spec_t& time) { _start_time = time; _start_time_set = true; _stream_now = false; } Those variables are used by the usrp_source_impl::start() and usrp_sink_impl::start() mentioned above. For example, within usrp_sink_impl::start(): _tx_stream = _dev->get_tx_stream(_stream_args); _metadata.start_of_burst = true; _metadata.end_of_burst = false; // Bursty tx will need to send a tx_time to activate time spec _metadata.has_time_spec = !_stream_now && pmt::is_null(_length_tag_key); _nitems_to_send = 0; if (_start_time_set) { _start_time_set = false; // cleared for next run _metadata.time_spec = _start_time; } else { _metadata.time_spec = get_time_now() + ::uhd::time_spec_t(0.15); } _tx_stream->send(gr_vector_const_void_star(_nchan), 0, _metadata, 1.0); And for usrp_source_impl::start(): _rx_stream = _dev->get_rx_stream(_stream_args); static const double reasonable_delay = 0.1; // order of magnitude over RTT ::uhd::stream_cmd_t stream_cmd(::uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS); stream_cmd.stream_now = _stream_now; if (_start_time_set) { _start_time_set = false; // cleared for next run stream_cmd.time_spec = _start_time; } else { stream_cmd.time_spec = get_time_now() + ::uhd::time_spec_t(reasonable_delay); } this->issue_stream_cmd(stream_cmd); > > Now comes the part that I don't understand. When I use: > > > > self.uhd_usrp_sink_0.set_start_time(uhd.time_spec(1.0)) > > self.uhd_usrp_source_0.set_start_time(uhd.time_spec(1.0)) > > > > I always get a string of "L" and the flowgraph is stuck. No matter which > > number I put here. After long time I figured out that the start time for > > USRP Sink has to be at least 100ms later. This works: > > > > self.uhd_usrp_sink_0.set_start_time(uhd.time_spec(1.1)) > > self.uhd_usrp_source_0.set_start_time(uhd.time_spec(1.0)) > > > > 1.0001 does not work and neither does 1.001, 1.01 etc. But any number > > larger than 1.1 does. > > > > WHY is this the case? > This sounds like a bug or quirk in gr-ettus. In my own software (and > in the Ettus UHD examples), setting a common start time for Tx and Rx > is typical usage. I did some more debugging and found out that this occurs when I have one signal that I send and receive back with through the USRP and then somehow combine again with the original signal. Combine can be plotting them in the same plot. Simple example (both USRP Source+Sink have set_start_time(1.2) set): +---------+ +---------------+ | | | Cosine Source |---+-------------------------------------->| QT GUI | +---------------+ | +-----------+ +-------------+ | Freq | `--->| USRP Sink | | USRP Source |---->| Sink | +-----------+ +-------------+ | | +---------+ Instead of "QT GUI Freq Sink" we could have Multiply, Add. etc. If I break the top connection it works! The other "fix" is what I described above: When I let the USRP Sink start *later* (set_start_time(1.3) it works too. I am not exactly sure why and how and what I should let this time difference be. There must be a more elegant way to solve this? > > Also, why do we need to wait so long (100ms does not work, I get "WARN: > > USRP Source Block caught rx error code: 2" and "U")? What exactly has to > > happen within this one second and what are the cases when this one second > > may not be enough? > > Again, this is likely a question better answered by a > gnuradio/gr-ettus person. Probably. Unfortunately I never had luck to receive response (neither here nor in the gnuradio mailing list, nor on github bug report). > I typically use delays as short as 10-50 ms > between get_time_now() and start of streaming. How are you exactly determining these times? How long do they need to be? I assume this time must be long enough for this command to reach the USRP? (i.e., total delays of OS (python -> gr-uhd -> uhd) -> Driver/kernel (TCP/IP+Ethernet) -> Link -> FPGA processing) I do not understand WHY we need this exactly. > For Tx streaming, you > just need to make sure that the streaming data arrives at the radio > prior to the motherboard clock exceeding the time indicated in the > time stamp of the first streaming sample or else you get "Late (L)" > messages. Ok. Let me attempt to answer my previous question: At t0, set_unknown_pps() will set the time on the mainboard to zero exactly at the next pps (even if no external PPS is connected as in my case). The request goes through python, UHD, IP Stack, Ethernet, cable to FPGA and once it arrives there USRP clock starts at zero. Call this time t1. On the host, some time t2 > t0 (but not necessarily t2 > t1 ??) after set_unknown_pps(), usrp_sink_impl::start() will be executed which sends an empty packet with .has_time_spec = true .time_spec = uhd.time_spec_t(X) Again, this request takes some time to proagate to USRP and lands there at t3. Now X has to be larger than t3-t1 ? That means it includes two packets to the USRP and an unknown processing delay on the host computer between set_unknown_pps and _tx_stream->send() ? The communication to the USRP would be more straight forward to determine (e.g. ping divided by two). The processing delay on the host computer is more vague and depends on CPU load but I guess can still be reasonably well bounded with a second or so? > For Rx streaming, the radio block will start sending > streaming samples once the motherboard clock reaches the indicated > time stamp. I see. So similarly with the code snippets above, t0 and t1 will be the same. At some time t2' (t2' > t0, but not necessarily t2' > t1?), usrp_source_impl::start() will be executed which calls issue_stream_cmd() with the given delay X. This command reaches the USRP at t3'. > I don't think that there is any warning if you tell the > Rx streamer to start streaming at say time 10.123 when in fact the > current motherboard time is already 12.2. I think it will just start > streaming immediately with no warning (but I'm not 100% sure on this > case). According to this, there would be no requirement on X, correct? However, when X is too small for RX streaming, I get WARN: USRP Source Block caught rx error code: 2 So this is probably the case when X < (t3'-t1) ? BTW, can the L message also be produced by RX (USRP Source)? > The "Underrun (U)" warning indicates that the radio has already > started streaming and has new samples to send (to put into the send > FIFO) but the send FIFO is FULL (the samples are not being consumed > fast enough downstream and so the radio has no place to put the newest > samples). > > > > I am using USRP X310+UBX+gnuradio for an application where my transmitted > > > signal (TX) is reflected and received (RX). > > > For my tests I use the simple loopback config (TX -> 30dB Attn -> RX), > > > transmit a pulse and plot both on the same plot (triggered in the TX > > > pulse). > > > > > > The response comes around a whopping 38ms later! > How do you measure this delay? Does this include software delays, > 10Gb delays, etc? I plotted the transmitted pulse and the received pulse on the same plot. Since the samples are time-discrete on the host, they only include analog delays. But after all your explanations, I understand where this comes from: In the default config, stream start for RX is at 0.1 and for TX 0.15. That makes already 50ms difference. This is in addition to the difference due to the separate get_time_now calls made by gr-uhd and since my pulse repeats every 100ms, the 38ms may actually be the 62ms reply from the previous pulse. > > > I added a delay block before plotting the TX signal but the delay is > > > still random each time I restart the flowgraph (residual time delays up > > > to 700us). > > > > > > I wonder if I can achieve sample-accurate alignment that is consistent > > > across flowgraph runs (maybe even USRP power cycles). > Assuming that I understand what you mean, this is possible - even with > USRP power cycles. When I simultaneously transmit and receive using > timed samples, my receive signal always starts at the same sample > within the Tx waveform (not sample 1, but that can be calibrated). This this part is fixed and understood now. Thanks, Lukas _______________________________________________ USRP-users mailing list [email protected] http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
