Kevin, I glanced through Captures 0,1,2 and could see the general gist of where it all goes wrong, but it’s not really clear why. The dissector seems out of date, at least it isn’t decoding correctly but I can still follow it.
Capture 0: Packets 2551 through 2606 are the actual sample data packets. Then there is an approx 1.2second pause before UHD, having timed out waiting for ACK’s begins some basic control interaction initiated by the host. Contrast that with Capture 2, where you will see a nicely paced stream of ACK’s (starts at packet 2609) flowing back from the USRP at exact intervals pacing the real time TX operation as the DSP consumes the samples. Capture 1: This goes to the weeds early, streaming never gets close to starting. At around packet 1324 you suddenly see a series of pauses for exactly 0.5 secs each. So a couple of thoughts bordering on guesses: Rate conversion in switches is always a little bit of a minefield when you have a single stream that is faster than the skinny pipe. In this case, operations like the sample burst are streaming from the host faster than 1Gbs into the switch forcing the switch to buffer samples. It’s surprising sometimes how quick these lower cost single chip switches can get buffer starved. Try using 'ethtool' on the Linux host to force the host port to 1Gbps to eliminate rate conversion in the switch as a cause. N210 in general is the most reliable of all USRP’s but it has one quirk, which is that the ethernet port is forced to be 1Gbps and it does no auto negotiation. This often trips up people who connect it to a 100mbit switch or host. I have never actually tried connecting it to 10GBase-T switches…there may be some (switch implementation specific) gotcha’s. Mirroring multiple ports to a single port can affect how the problem manifests because you create new bottle necks in the switch …if the 10G switch is suspect then I suggest interposing a cheap 1G switch that supports mirroring between USRP and 10G switch, that way you’ll have higher confidence exactly what and when went onto the wire. This is also an interesting experiment because it isolates the N210’s ethernet port from the 10G switch…curious to see if it now starts to work OK. I have had this happen when debugging other products. The ICMP port unreachable is likely a red herring, some spurious Linux service rather than UHD originating those packets I suspect. Might be worth looking at the switch internal counters to see if any physical layer error counters are increasing. No smoking gun I’m afraid; your fall back plan might be to stuff quad port 1G cards in your host, they’re cheap if you have the PCIe slots free. -Ian > On Nov 6, 2017, at 3:13 PM, Kevin Krieger via USRP-users > <usrp-users@lists.ettus.com> wrote: > > Hi, > > Any chance to look at these wireshark captures? > > Thanks, > Kevin > > On 10/30/2017 05:30 PM, Kevin Krieger via USRP-users wrote: >> Hi Marcus, >> >> I finally got around to this. >> I've attached 9 captures. >> >> The setup for the captures was: >> XS728T with latest firmware running, factory default >> Computer running tx_bursts has IP of 192.168.10.60 or .67 >> N200 has IP of 192.168.10.107 >> ./tx_bursts --args addr0=192.168.10.107 --freq 10e6 --rate 1e6 >> >> Captures 0,1 and 2 are with both the ports (n200 side, and computer side) >> mirrored to the wireshark capture port, so you essentially see duplicate >> traffic. >> Captures 3,4 and 5 are with only the computer side traffic captured >> Captures 6,7 and 8 are with only the N200 side traffic captured. >> >> Captures 0, 4, and 8 are with the messages "Sent packet: 363 samples" >> appearing, but then the "Waiting for async burst ACK ... fail" message also >> appears. >> Captures 1, 3 and 7 are with the message :Error: Runtimeerror: fifo ctrl >> timed out looking for acks" appearing. >> Captures 2, 5 and 6 are with successful runs of the tx bursts program, so >> instead of 'fail' at the end there is 'success'. >> >> There are txt files with the output of the program for each capture, as well >> as csv files showing the dissected output (only found uhd in the dissectors). >> >> Can you see anything wrong with these runs? Any help is appreciated. >> >> Thanks, >> Kevin >> >> >> On 06/15/2017 02:28 PM, Kevin Krieger via USRP-users wrote: >>> Hi Marcus, >>> >>> Thank you - I was sure that the same subnet was correct but it's good to >>> have confirmation. >>> I have done the dissecting previously, and I recall getting a bunch of "huh >>> what's that?"' messages and then a 'wassup bro' or something but I'll do it >>> again and let you know what I see. (Hilarious descriptions btw!) >>> >>> Kevin >>> >>> >>> On 05/24/2017 08:07 PM, Marcus Müller via USRP-users wrote: >>>> Hi Kevin, Hi Robin, >>>> the same-subnet configuration is the right one, here. There's no routing >>>> ambiguity if you have one card that serves the whole switch. >>>> Kevin, this might be much to ask, but: I'd ask you to both install >>>> wireshark and its development headers (typically, wireshark-dev or >>>> wireshark-devel) from your operating system's package manager. >>>> Then, get the UHD source code, and >>>> >>>> cd /path/to/uhd/tools/dissectors >>>> mkdir build >>>> cd build >>>> cmake -DETTUS_DISSECTOR_NAME=zpu .. >>>> make -j4 && make install >>>> >>>> Ideally, add your user to the group that has access to the raw network >>>> hardware (in case of my OS, that was the "wireshark" group), log out and >>>> back in. >>>> >>>> If that doesn't work, you have to record as root and analyze as your user >>>> (the USRP protocol dissectors got installed to your home directory). >>>> >>>> Then, record the traffic on your 10G interface, and analyze the packets in >>>> question as UHD (some/many will be sample packets, i.e. CVITA). >>>> >>>> Best regards, >>>> >>>> Marcus >>>> On 24.05.2017 20:15, ROBIN TORTORA via USRP-users wrote: >>>>> The only thing that jumps out to me is they are all on the same subnet... >>>>> >>>>> >>>>> That is something I have typically avoided, but if you say it works in >>>>> another configuration, that may not be the issue... >>>>> >>>>>> On May 24, 2017 at 12:12 PM Kevin Krieger via USRP-users >>>>>> <usrp-users@lists.ettus.com> <mailto:usrp-users@lists.ettus.com> wrote: >>>>>> >>>>>> Hi all, >>>>>> >>>>>> I've got a predicament. Here's some background before I describe my >>>>>> problem: >>>>>> We purchased a Netgear xs728T 10GB switch in order to network 16 N200 >>>>>> devices, as well as a processing computer (which has a 10G ethernet >>>>>> card). Our 16 N200 devices are going to be used in a radar configuration >>>>>> where they are all simultaneously either transmitting or receiving. They >>>>>> are all receiving coincident 10MHz and 1PPS signals from 2 octoclocks, >>>>>> which are both fed from a third clock to ensure coincident clocks. They >>>>>> are connected to the switch via cat6 cables ~7feet in length (this item: >>>>>> https://www.newegg.ca/Product/Product.aspx?Item=N82E16812119168&_ga=2.227605844.817843194.1495640587-1295759170.1493911136 >>>>>> >>>>>> <https://www.newegg.ca/Product/Product.aspx?Item=N82E16812119168&_ga=2.227605844.817843194.1495640587-1295759170.1493911136>) >>>>>> >>>>>> The problem: >>>>>> During testing we've found that the example program tx_bursts, when run >>>>>> on one USRP at a time with a modest bandwidth, the "Waiting for async >>>>>> burst ack....failed" message appears about 75% of the time. If I try to >>>>>> use two or more USRPs at a time, then it appears every time. >>>>>> The line used for tx_bursts was typically something like this: >>>>>> ./tx_bursts --args="addr0=192.168.10.104" --freq 10e6 --rate=1e6 >>>>>> --channels="0" >>>>>> for a one usrp test. For a multiple usrp test, I used lines like this: >>>>>> ./tx_bursts >>>>>> --args="addr0=192.168.10.101,addr1=192.168.10.102,addr2=192.168.10.103,addr3=192.168.10.100,addr4=192.168.10.112,addr5=192.168.10.113,addr6=192.168.10.114" >>>>>> --freq 10e6 --rate=1e6 --channels="0,1,2,3,4,5,6" >>>>>> >>>>>> >>>>>> Some evidence & information: >>>>>> We also have a 5 port 1G DLink DGS-1005G. We've tested 4 N200s with >>>>>> tx_bursts on this switch (same cabling) and it works flawlessly. >>>>>> We also have tested a second 10G switch, the 8 port XS708E with 7 N200s >>>>>> with tx_bursts and it works flawlessly. >>>>>> >>>>>> I have wireshark dumps taken via a third machine that is connected to >>>>>> mirrored ports on the xs728T switch. I have attached them in case anyone >>>>>> can tell me if they see something wrong besides the missing acks. >>>>>> The wireshark filter to use is : ip.src == 192.168.10.104 and ip.dst == >>>>>> 192.168.10.104 or ip.src == 192.168.10.104 and ip.dst == 192.168.10.99 >>>>>> >>>>>> The xs728t switch information is: Boot version is 1.0.0.0, SW version is >>>>>> 6.5.1.26 (latest as of testing). >>>>>> I contacted Netgear support, and they sent us a brand new switch, which >>>>>> exhibited the same problem. >>>>>> >>>>>> I'm wondering if there's anyone out there who has had similar issues? Is >>>>>> there anything I can do to get more information or get around this >>>>>> problem? >>>>>> Can anyone see what the root cause might be in the wireshark captures? >>>>>> >>>>>> Any help or pointers are appreciated. Thank you, >>>>>> Kevin >>>>> >>>>> >>>>> >>>>>> _______________________________________________ >>>>>> USRP-users mailing list >>>>>> USRP-users@lists.ettus.com <mailto:USRP-users@lists.ettus.com> >>>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com >>>>>> <http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com> >>>>> >>>>> >>>>> _______________________________________________ >>>>> USRP-users mailing list >>>>> USRP-users@lists.ettus.com <mailto:USRP-users@lists.ettus.com> >>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com >>>>> <http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com> >>>> >>>> >>>> >>>> _______________________________________________ >>>> USRP-users mailing list >>>> USRP-users@lists.ettus.com <mailto:USRP-users@lists.ettus.com> >>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com >>>> <http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com> >>> >>> >>> >>> _______________________________________________ >>> USRP-users mailing list >>> USRP-users@lists.ettus.com <mailto:USRP-users@lists.ettus.com> >>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com >>> <http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com> >> >> >> >> _______________________________________________ >> USRP-users mailing list >> USRP-users@lists.ettus.com <mailto:USRP-users@lists.ettus.com> >> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com >> <http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com> > > _______________________________________________ > USRP-users mailing list > USRP-users@lists.ettus.com > http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
_______________________________________________ USRP-users mailing list USRP-users@lists.ettus.com http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
