Hello all
I'm working on a project to use 16 N200s in a multi USRP configuration for
a radar project. They are synchronized using 3 Octoclocks (one master with
a GPSDO, and two slaves that drive the 16 N200s). I've been getting many
errors that I was hoping someone could help with. I have attached a sample
program that mimics how the USRPs are used. The basic goal here is to
sample a determined number of samples while transmitting some spaced pulses
(a "pulse sequence") and then repeat. It would be beneficial to have as
little delay as possible in between loops. I've been running into some
problems however.
BASIC INFO
We have 16 N200s + 1 spare
3 Octoclocks (1 has gps and it drives the other two)
LFTX and LFRX daughterboards in each N200
linux; GNU C++ version 4.8.5; Boost_105400;
UHD_003.010.002.000-0-gbd6e21dc
Intel Corporation Ethernet Controller 10-Gigabit X540-AT2
Devices are all connected via 3 daisy chained Netgear XS708E switches
Ideal sampling rates: 5-10MHz TX, 5-10MHz RX depending on
hardware/software limitations.
One of the issues I sometimes get after several minutes of runtime (or
several thousand pulse sequences) is the overflow(O) with the out of
sequence flag set in the rx metadata. I also get sequence errors(S) on the
tx side too. It seems to happen more often and faster with higher sampling
rate. I've gathered from other mailing list posts that this is likely a
network configuration issue. Can someone recommend a known working computer
build and network configuration that can handle the amount of USRPs and
data we are attempting to use?
Another major issue I eventually get during runtime is a slurry of lates(L)
on TX and then a LATE in the RX metadata. I've tried increasing the time in
the future that the TX/RX should start (from when the stream commands are
called) and I've tried to minimize the number of operations happening
between that calculation and when TX/RX start, but the lates still
eventually happen. I've tried to time profile what I can in my code and it
seems I should really only need about ~0.5ms of delay, but even at 3-10ms
of delay I have issues. I feel like 10ms of time should be more that plenty
of time for the host to issue stream commands. I don't seem to get lates if
I have test applications that individually test TX or RX, but when I put
them together using threads, I can't seem to find a way to eliminate the
lates. Any ideas on how I should set up what I'm trying to do here?
Here is the compiler line to make the test program:
g++ -o test_rx_tx -std=c++11 -Wall test_rx_tx.cpp -lboost_system -luhd
If I can explain anything in further detail please let me know.
Thank you for your time.
--
-Keith Kotyk
#include <uhd/utils/thread_priority.hpp>
#include <uhd/utils/safe_main.hpp>
#include <uhd/utils/static.hpp>
#include <uhd/usrp/multi_usrp.hpp>
#include <cstdlib>
#include <vector>
#include <string>
#include <iostream>
#include <complex>
#include <string>
#include <thread>
#define ADDR std::string("addr0=192.168.10.100, addr1=192.168.10.101, addr2=192.168.10.102, addr3=192.168.10.103, addr4=192.168.10.104, addr5=192.168.10.105, addr6=192.168.10.106, addr7=192.168.10.107, addr8=192.168.10.108, addr9=192.168.10.109, addr10=192.168.10.110, addr11=192.168.10.111, addr12=192.168.10.112, addr13=192.168.10.113, addr14=192.168.10.114, addr15=192.168.10.115")
#define CLKSRC std::string("external")
#define TIMESRC std::string("external")
#define RXSUBDEV std::string("A:A A:B")
#define TXSUBDEV std::string("A:A")
#define TXCHAN {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}
#define RXCHAN {0,1,2,3,4,5,6,7,8,10,12,14,16,18,20,22,24,26,28,30}
#define TXRATE 5e6
#define RXRATE 5e6
#define TXFREQ 10e6
#define RXFREQ 10e6
#define DELAY 3e-3
#define PULSETIMES {0.0, 13500e-6, 18000e-6, 30000e-6, 33000e-6, 39000e-6, 40500e-6}
#define SAMPSPERCHAN 320600
std::vector<std::complex<float>> make_ramped_pulse(double tx_rate){
auto amp = 1.0/sqrt(2.0);
auto pulse_len = 300.0e-6;
int tr_start_pad = 60, tr_end_pad = 60;
int num_samps_per_antenna = std::ceil(pulse_len * tx_rate) + tr_start_pad + tr_end_pad;
std::vector<double> tx_freqs = {1e6};
auto default_v = std::complex<float>(0.0,0.0);
std::vector<std::complex<float>> samples(num_samps_per_antenna,default_v);
for (auto j=tr_start_pad; j< num_samps_per_antenna - tr_end_pad; j++) {
auto nco_point = std::complex<float>(0.0,0.0);
for (auto freq : tx_freqs) {
auto sampling_freq = 2 * M_PI * freq/tx_rate;
auto radians = fmod(sampling_freq * j, 2 * M_PI);
auto I = amp * cos(radians);
auto Q = amp * sin(radians);
nco_point += std::complex<float>(I,Q);
}
samples[j] = nco_point;
}
auto ramp_size = int(10e-6 * tx_rate);
for (auto j=tr_start_pad,k=0; j<tr_start_pad+ramp_size; j++,k++){
auto a = ((k)*1.0)/ramp_size;
samples[j] *= std::complex<float>(a,0);
}
for (auto j=num_samps_per_antenna-tr_end_pad - 1, k=0;
j>num_samps_per_antenna-tr_end_pad-1-ramp_size;
j--,k++)
{
auto a = ((k)*1.0)/ramp_size;
samples[j] *= std::complex<float>(a,0);
}
return samples;
}
int UHD_SAFE_MAIN(int argc, char *argv[]) {
auto usrp_d = uhd::usrp::multi_usrp::make(ADDR);
usrp_d->set_clock_source(CLKSRC);
usrp_d->set_time_source(TIMESRC);
usrp_d->set_time_unknown_pps(uhd::time_spec_t(0.0));
usrp_d->set_gpio_attr("RXA", "CTRL", 0xFFFF, 0b11111111);
usrp_d->set_gpio_attr("RXA", "DDR", 0xFFFF, 0b11111111);
//XX is the actual TR signal
usrp_d->set_gpio_attr("RXA", "ATR_XX", 0xFFFF, 0b00010000);
usrp_d->set_gpio_attr("RXA", "ATR_XX", 0xFFFF, 0b00100000);
//0X acts as 'scope sync'
usrp_d->set_gpio_attr("RXA", "ATR_0X", 0xFFFF, 0b01000000);
usrp_d->set_gpio_attr("RXA", "ATR_0X", 0xFFFF, 0b10000000);
// rx config
usrp_d->set_rx_subdev_spec(RXSUBDEV);
usrp_d->set_rx_rate(RXRATE);
std::vector<size_t> rx_chans = RXCHAN;
uhd::tune_request_t rx_tune_request(RXFREQ);
for(auto &channel : rx_chans) {
usrp_d->set_rx_freq(rx_tune_request, channel);
double actual_freq = usrp_d->get_rx_freq(channel);
if (actual_freq != RXFREQ) {
std::cout << "requested rx ctr freq " << RXFREQ << " actual freq " << actual_freq <<std::endl;
}
}
uhd::stream_args_t rx_stream_args("fc32", "sc16");
uhd::stream_cmd_t rx_stream_cmd(uhd::stream_cmd_t::STREAM_MODE_NUM_SAMPS_AND_DONE);
rx_stream_args.channels = rx_chans;
uhd::rx_streamer::sptr rx_stream = usrp_d->get_rx_stream(rx_stream_args);
std::vector<std::complex<float>> buffer(rx_chans.size() * SAMPSPERCHAN);
std::vector<std::complex<float>*> buffer_ptrs;
for(uint32_t i=0; i<rx_chans.size(); i++){
auto ptr = static_cast<std::complex<float>*>(buffer.data() + (i * SAMPSPERCHAN));
buffer_ptrs.push_back(ptr);
}
//tx config
std::vector<size_t> tx_chans = TXCHAN;
usrp_d->set_tx_subdev_spec(TXSUBDEV);
usrp_d->set_tx_rate(TXRATE);
uhd::tune_request_t tx_tune_request(TXFREQ);
for(auto &channel : tx_chans) {
usrp_d->set_tx_freq(tx_tune_request, channel);
double actual_freq = usrp_d->get_tx_freq(channel);
if (actual_freq != RXFREQ) {
std::cout << "requested tx ctr freq " << TXFREQ << " actual freq " << actual_freq <<std::endl;
}
}
uhd::stream_args_t tx_stream_args("fc32", "sc16");
tx_stream_args.channels = tx_chans;
uhd::tx_streamer::sptr tx_stream = usrp_d->get_tx_stream(tx_stream_args);
auto pulse = make_ramped_pulse(TXRATE);
std::vector<std::vector<std::complex<float>>> tx_samples(tx_chans.size(),pulse);
int count = 1;
while(1) {
//std::cout << "Starting round " << count << " with delay " <<DELAY <<std::endl;
auto time_zero = usrp_d->get_time_now() + uhd::time_spec_t(DELAY);
rx_stream_cmd.num_samps = SAMPSPERCHAN;
rx_stream_cmd.stream_now = false;
rx_stream_cmd.time_spec = time_zero;
rx_stream->issue_stream_cmd(rx_stream_cmd);
auto recv = [&](){
double timeout = rx_stream_cmd.time_spec.get_real_secs() -
usrp_d->get_time_now().get_real_secs() +
(SAMPSPERCHAN / RXRATE) + 0.1;
size_t accumulated_samples = 0;
uhd::rx_metadata_t meta;
while(accumulated_samples < SAMPSPERCHAN) {
accumulated_samples += rx_stream->recv(buffer_ptrs, SAMPSPERCHAN, meta, timeout);
timeout = 0.1;
auto error_code = meta.error_code;
switch(error_code) {
case uhd::rx_metadata_t::ERROR_CODE_NONE :
break;
case uhd::rx_metadata_t::ERROR_CODE_TIMEOUT : {
std::cerr << "Timed out!" << std::endl;
exit(-1);
}
case uhd::rx_metadata_t::ERROR_CODE_OVERFLOW : {
std::cerr << "Overflow!" << std::endl;
std::cerr << "OOS: " << meta.out_of_sequence << std::endl;
exit(-1);
}
case uhd::rx_metadata_t::ERROR_CODE_LATE_COMMAND : {
std::cerr << "LATE!" << std::endl;
exit(1);
}
default :
break;
}
//std::cout << "recv " << accumulated_samples << " samples" << std::endl;
}
};
std::thread recv_t(recv);
auto send = [&](double start_time) {
uhd::tx_metadata_t meta;
meta.has_time_spec = true;
auto time_to_send_pulse = uhd::time_spec_t(start_time);
auto pulse_start_time = time_zero + time_to_send_pulse;
meta.time_spec = pulse_start_time;
meta.start_of_burst = true;
uint64_t num_samps_sent = 0;
auto samples_per_buff = tx_samples[0].size();
while (num_samps_sent < samples_per_buff)
{
auto num_samps_to_send = samples_per_buff - num_samps_sent;
num_samps_sent = tx_stream->send(tx_samples, num_samps_to_send, meta);
meta.start_of_burst = false;
meta.has_time_spec= false;
}
meta.end_of_burst = true;
tx_stream->send("", 0, meta);
};
std::vector<double> time_per_pulse = PULSETIMES;
for(int i=0; i<time_per_pulse.size(); i++){
send(time_per_pulse[i]);
}
recv_t.join();
count++;
}
}_______________________________________________
USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
