Hi Rob,
thanks for your reply. Of course, I'd like to clarify as much as
possible. I answer in line.
Cheers
Johannes
On 15.04.20 20:12, Rob Kossler wrote:
Hi Johannes,
I don't really have any direct answers to your questions. But, a couple
of remarks:
* It's not clear to me if you've completely lost access to any N310s
or if you've just lost access through the 1Gb RJ45. If you still
have access via the SFP port, perhaps you can ssh into it and run
"ip addr show" or something like that to determine what the 1Gb port
IP address presently is. Perhaps you can ping that from your network.
I lost access to them completely. They don't show up with
`uhd_find_devices` anymore. They can't be discovered with `nmap -sn`
etc. No ping. Also, both their interfaces are down the RJ45 management
port and the SFP+ port.
* You mentioned that you've had N310 issues that persist after a
reboot. That is curious because it is not my experience. The
issues that I've had have all gone away after rebooting.
That was my hope and experience too.
* The N310 fails to settle some internal init. (e.g. PLL lock) and thus
I can't run any tests e.g. `uhd_usrp_probe` or `benchmark_rate`. Of
course anything else fails too.
* Measured latency jumps wildly. With real-time scheduling and directly
attached USRPs, I usually see e2e latencies (packet entering TX chain,
packet leaving RX chain) below 1ms. After running those USRPs for a
couple of days, this behavior suddenly changed. Now latency follows a
saw line. Jumping to ~.3s and going down gradually to <1ms for one frame
then jumping up again. If I remove my hardware from this application (I
just connect my TX to my RX chain in GRC) latency stays below 1ms. Since
it worked for a while with hardware and then suddenly stopped, I suspect
there's smth wrong with those N310s. Reboot didn't help.
* Not that this will be useful right now, but it is pretty cheap to
buy wifi-controlled smart outlets so that you could remotely
remove/apply power. You can also set a flag on the N310 such that
it will boot automatically following a power failure.
I'll probably have to go that route. And I'm sad I have to.
* My experience with X310s matches yours in that they generally behave
themselves (relative to N310s behavior)
Rob
On Wed, Apr 15, 2020 at 5:36 AM Johannes Demel via USRP-users
<[email protected] <mailto:[email protected]>> wrote:
Hi all,
I want to give you all an update on my experience with my issues so far.
So far I didn't hear back from anyone on the mailing list, thus I went
forward and had someone update my network configuration.
First off, N310s are usable with both management and stream ports
connected. It is just important to assign those ports to different
subnets that are separated via their subnet masks. I assume this is
basic network knowledge which I had chosen to ignore.
My new network setup looks like this:
I have 4 N310s connected. All 4 of them have their management ports
connected to our network with its own subnet etc. Just like all other
machines are connected to this network.
Currently I use 2 machines with Intel X710-DA2 NICs. Each machine is
directly connected to 2 N310s via SFP+ cables. This setup works nicely.
Just put each N310 into its own subet and you're good to go.
From another machine in our network `uhd_find_devices` may look
like this:
Device Address:
serial: XXXXXX
claimed: False
mgmt_addr: x.x.x.149
product: n310
reachable: No
type: n3xx
On the machine where the N310 is connected with its streaming port:
Device Address:
serial: XXXXXX
addr: y.y.y.217
claimed: False
mgmt_addr: x.x.x.149
mgmt_addr: y.y.y.217
product: n310
type: n3xx
Besides `uhd_find_devices` I check each USRP with `uhd_usrp_probe`
which
I consider a good quick check. And further I run `benchmark_rate` with
each USRP to confirm that it is still able to stream samples.
So far so good. BUT:
What I observe is that these N310s tend to disappear from the network
after a week or so. I could use them last Thursday but today half of
them disappeared. That happened before and triggered my initial email.
Even with all N310s connected via their management ports, by now only 2
out of 4 are reachable via network. `uhd_find_devices` can't find them
anymore. Not even their management address.
Also, after a few days all N310s did not work as well as they did after
boot. And it didn't help to reboot them via `reboot now`. My experience
in such situations is that it just helps to have physical access to
them. Unplug them etc. And of course, this is impossible at the moment.
Interestingly, I have 2 X310s connected to another machine and these
X310s are still up and running. I never had issues with these devices
disappearing after a while.
I use all USRPs with UHD 3.15LTS. All SDimages are flashed with this
version.
I use this guide for bmaptool to flash all of them:
https://kb.ettus.com/Writing_the_USRP_File_System_Disk_Image_to_a_SD_Card#Using_bmaptool_to_write_the_disk_image
It's way faster than `dd`.
X310s are in sync with UHD3.15LTS via `uhd_images_downloader &&
uhd_image_loader`.
It would be great to be able to manage those N310s remotely over a
longer period of time. Occasional reboot wouldn't be an issue as
long as
all devices are reliably available without quirks.
Did I stumble over a known issue here? Is this something new? Are my
devices broken? How do I debug this? Is something else going on? Do I
need to provide more info?
Cheers
Johannes
On 27.03.20 09:41, Johannes Demel via USRP-users wrote:
> Hi all,
>
> last week I set up the N310s we have with UHD3.15LTS to run in our
> network. By now, most of them are not accessible remotely
anymore. Since
> I don't have physical access to them anymore, I'd like to figure
out a
> way to reliably manage them before someone fixes the immediate
issue for
> me.
>
> I've seen 2 error patterns.
>
> 1. A N310 may not get a lock on its PLL anymore. Even after a
reboot or
> "force_reinit=1"
>
> 2. N310s disappear from the network. Or they are unresponsive or
I can't
> log into them via ssh anymore to reboot them.
>
> I use Ubuntu 18.04 with UHD3.15LTS without any RFNoC.
>
> Currently, all N310s are only connected via an SFP+ port. In the
past, I
> had issues with an additional connection via the management port.
So I
> currently do not connect them via the management port.
>
> Does it help to connect the USRPs via their management port?
Would I be
> able to manage them more reliably?
>
> How do I make this setup work? Is it sufficient to assign them to
> different subnets? Can I assign them different IP addresses on
the same
> subnet? Do I need to physically separate the two networks?
>
> It would be great to hear from people's experience how to set up
their
> N310s.
>
> Cheers
> Johannes
>
> _______________________________________________
> USRP-users mailing list
> [email protected] <mailto:[email protected]>
> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
_______________________________________________
USRP-users mailing list
[email protected] <mailto:[email protected]>
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
_______________________________________________
USRP-users mailing list
[email protected]
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
