During the last 11 days since my last report on 9/25, two of my most unreliable Squeezebox radios have been instrumented with a spectrum analyzer, monitor mode WiFi sniffer adjacent to the radio, access point Ethernet capture, and the 'wlanpoke' and the radios 'messages' logs. This has resulted in gigabytes of captured packets and analyzer videos. During this period, the 'wlanpoke' mitigation software recovered from 1292 wireless connectivity losses with no failures, and the radios kept playing. About a dozen failures have been analyzed, which requires some 802.11 networking education (not enough to date). Possible methods to prevent or reduce the connection failures were also investigated.
Additionally, two cheap IP cameras here have been also experiencing increasingly frequent connectivity dropout. Although they recover on their own, they are no longer reliable for their intended purpose. These failures appeared more recently, well after the Squeezebox radios started failing early in 2020. It seems wireless connectivity issues are more widespread than just the Squeezebox radios. As of now, there has been no breakthrough in either diagnosis or treatment; however, the 'wlanpoke ' mitigation script has been successful. Performance improvement may require updating the proprietary chip EEPROM or CPU '.bin' files. Sorely needed are debug versions of the Atheros driver, related utilities, and the wpa_supplicant to determine who knows what, when. More Information Here is a summary of the progress so far: Environment Four Squeezebox radios connect to a new modest WiFi 802.11G access point on channel 1. The other two are adjacent to and connected to a modern capable AP with multiple other clients on channel 6. Several (13) gadgets including IP cameras connect to yet another modern AP on channel 11. Neighbors access point signals are about 20-30 dBm lower. Spectrum Analyzer A Tektronix 492 spectrum analyzer was set to the channels frequency, with a sweep range of 50 mHz and bandwidth of 1 mHz, resulting in an effective noise floor of about -78 dBm, sufficient to detect expected higher amplitude overloading signals. Capturing an event at a specified point in time is a matter of chance, less than 10%, because a conventional spectrum analyzer receives only one frequency at a time, so cannot capture a complete spectrum at every instant. Video of the analyzer display was captured at 30 fps, of which most frames are duplicates. Given these limitations: o No interfering non-WiFi services operating in or near the Squeezebox radio's WiFi channel have been identified. Put another way, although there may be signals present, they are not so high in amplitude as to make them stand out, and I cannot identify them as of now. o The peak hold spectrum analyzer display shows signals 20-30 db higher than the normal traffic, which might be expected since the analyzer is adjacent to the Squeezebox, and these signals are likely from the radio's WiFi transmitter. Or, maybe not. WiFi Packet Sniffer The "monitor mode" WiFi packet sniffer uses a higher gain antenna and captures more than the Squeezebox radio would be expected to, given its internal antenna. All the relevant packets seem to be captured, including those from neighbors competing APs on the same channel. Some neighbors weaker station packets are apparently not captured. WiFi packets are identified by 6 byte MAC addresses, but their contents are encrypted, which requires the matching with the AP Ethernet packet capture to understand their content. However, most WiFi packets are generated by the AP and station (Squeezebox) MAC layers, with no corresponding Ethernet packets. These include acknowledgement and re transmission frames, access point scan 'probe' requests, etc. As the AP and stations send packets to each other, they must be acknowledged by the receiver, otherwise the sender re-transmits the packet repeatedly for a certain number of times before giving up. o The most striking observation is the large number of packet re transmissions from the Squeezebox even when it reports reasonable SNR (signal to noise ratio) values. This means that the Squeezebox is failing to receive or process the AP's acknowledgement. Also, data frames from the LMS are re transmitted several times before the Squeezebox acknowledges them. This is true for the 'wlanpoke' ping replies from the central router as well. The several millisecond reply time comes from not receiving the first reply packet from the router (via the AP of course). This rules out problems with the Ethernet LAN or router regarding ping failure. o As of now, the mechanism or point at which IP level connectivity is lost is not identified. Still working on this. o From time to time, gratuitous access points from wireless speakers, televisions, IOT gadgets, etc., inside the house pop up on channel 1 with signal levels a bit less than the access point. They respond to Squeezebox AP probe requests, but powering off these devices had no effect on connectivity loss. Ethernet Sniffer The Ethernet sniffer captures packets to and from the WiFi 802.11G access point dedicated to 4 Squeezebox radios on channel 1. o No rogue packets that might cause poor Squeezebox behavior have been detected. o It has not been necessary to match the contents of an encrypted WiFi packet with its unencrypted Ethernet version. o A Windows 10 computer was sending a fair amount of traffic to certain Squeezebox radios. Powering off the computer had no effect. Prophylactic Measures One new prophylactic measure nearly eliminated the worst performer's failures; however, this had little effect on the five other Squeezebox radios. When the worst and best performers switched locations, their performances also switched: the former best became the worst, and the worst became the best. Software Analysis Analysis of the Atheros AR6002 chip and software is severely hampered by lack of a chip specification, plus source code for the driver and accessories. Additionally, my docker build system is not working. Please Help! Secrecy for the driver (and loader) source is apparently unnecessary, as the driver kernel module itself contains the notice: "license=GPL and additional rights." Driver software from previous and subsequent builds has been examined. Evidently, the driver and loader software has limited control over the workings of the chip, which has its own CPU and secret proprietary software that apparently control the MAC and PHY layers, which is likely where the problems lie. Sorely needed are debug versions of the Atheros driver, related utilities, and wpa_supplicant to see who knows what, when. Additionally, the compiled .bin files may be different with different releases, and if so, these might be tried to see if a later version that supports the AR6002 might work better. If all else fails, reverse engineering these files may be required. There are a lot of additional details, pictures, videos, and discussion that could be written and uploaded, if there is any interest. * More Notes A conventional spectrum analyzer samples each frequency as it sweeps through the selected range (e.g., 25, 50, or 100 mHz centered on a WiFi channel). It detects a signal on a frequency only when it happens to be receiving that frequency. A wide sweep frequency requires a correspondingly wide receiver bandwidth (e.g., 1 mHz) to provide multiple sweeps per second, required to catch some glimpse of packet transmissions when the frequencies just happen to coincide. This wide bandwidth increases the noise floor from 96 dBm to about 80 dBm, which fortunately is not an issue when trying to detect overloading signals with much higher amplitude. Rather than a conventional spectrum analyzer, a software defined radio with a low noise amplifier might be a better approach to capturing wide band data. The maximum usable and stable bandwidth of a typical RTL-SDR is about 2.4 MHz, inadequate. The HackRF module offers 20 mHz bandwidth, which is sufficient to cover a WiFi channel, but not adjacent potentially interfering signals. Two can be combined for ~40 mHz bandwidth, but that is getting complicated. Additionally, a very fast computer would be required to keep up with the output. end ------------------------------------------------------------------------ POMdev's Profile: http://forums.slimdevices.com/member.php?userid=70558 View this thread: http://forums.slimdevices.com/showthread.php?t=111663
_______________________________________________ Radio mailing list [email protected] http://lists.slimdevices.com/mailman/listinfo/radio
