Re: [WISPA] Weird problem - 20 seconds latency and other oddness
Marlon K. Schafer (509) 982-2181 wrote: > Except it's not effecting the ap's further out. Even if they were on a > different segment they should still pick up the interference from the > ap's right? Basically, yeah. I think. :) (Good thing I never claimed to know much about RF, innit.) A different picture, so to speak, at http://thedave.us/pics/mvn/wispamap2.gif, which includes some of the relevant bits of the network's physical design. If it were a problem with, say, one of my Trango backhaul links, why does it only affect one of the two SUs on "Trango AP1"? And why does it affect customers on the far end of my licensed 39GHz link? For that matter, note the two overlapping dots. (Green is "good" towers, red is "bad," and the one yellow dot on the left is a tower that we've seen a couple issues with at the same time, but not nearly as bad. Note: not to scale, and please don't make fun of me just because I can't draw straight lines with MS Paint.) The points with overlapping dots show where there are two 2.4GHz APs on the same water tower - the ones facing north are having issues, the ones facing south rarely do. And when the south-facing antennas' customers are having problems, the noise levels and network latency are at their worst (a couple days ago I saw one ping packet that took almost thirty seconds to complete a round-trip, a new record). The backhaul links appear to be solid, because all the while, I can ping the APs themselves, and the managed switches at the key tower locations, with the expected 2-3ms latency and zero packet loss. David Smith MVN.net -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Weird problem - 20 seconds latency and other oddness
Except it's not effecting the ap's further out. Even if they were on a different segment they should still pick up the interference from the ap's right? Marlon (509) 982-2181 Equipment sales (408) 907-6910 (Vonage)Consulting services 42846865 (icq)And I run my own wisp! 64.146.146.12 (net meeting) www.odessaoffice.com/wireless www.odessaoffice.com/marlon/cam - Original Message - From: "Lonnie Nunweiler" <[EMAIL PROTECTED]> To: "WISPA General List" Sent: Tuesday, May 09, 2006 5:55 PM Subject: Re: [WISPA] Weird problem - 20 seconds latency and other oddness I agree it could be noise but a bridge runaway will give you the 10+ second pings and with that much traffic being echoed ALL of your AP and Clients are spewing. It would look like a massive RF flood on the Spectrum Analyzer. Think about what the air wave look like when you have full radio usage. To nearby units and competitors it would be a massive increase in the noise floor. Lonnie On 5/9/06, David E. Smith <[EMAIL PROTECTED]> wrote: Lonnie Nunweiler wrote: > Any confirmation on this? A customer router plugged in with LAN to > the WAN or not getting a DHCP entry or even a DNS entry has caused > many bridges to collapse and appear as if it is noise, simply because > the bridges are all echoing the massive broadcast traffic. There's no DHCP anywhere on the network, and the DHCP UDP ports are filtered out at every POP, so that specifically is a bad example. :) Sorry for not getting back to this, we've had massive weirdness on our dialup gear too (mostly related to moving it). Yes, our network is part bridged/part routed. I'm pretty sure it's a real RF problem, because we pulled out the Bumblebee and my field guy said he saw crazy mad noise all across the 2.4 spectrum a couple days ago, when we were having this weird hiccup. (And another local WISP operator reportedly has had similar issues, though I didn't talk to him personally; that's the boss' department.) We'll probably just have to use some old-school triangulation and such to find out where it is, if it's something we even can find. For instance, today was a cool and cloudy day, and this problem didn't show up all day. Thus, I blame sunspots. :D (Honestly, I'm stumped, but at least we're now reasonably certain it's a real RF issue.) If/when we sort it out, I'll report back. David Smith MVN.net -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- Lonnie Nunweiler Valemount Networks Corporation http://www.star-os.com/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Weird problem - 20 seconds latency and other oddness
Lonnie Nunweiler wrote: > I agree it could be noise but a bridge runaway will give you the 10+ > second pings and with that much traffic being echoed ALL of your AP > and Clients are spewing. It would look like a massive RF flood on the > Spectrum Analyzer. Think about what the air wave look like when you > have full radio usage. To nearby units and competitors it would be a > massive increase in the noise floor. That's certainly possible, but then how do I track down which specific piece of hardware is responsible? As much as you'd enjoy the extra StarOS software sales, I'm not sure it's feasible to wholesale-replace twenty APs. :) If it were, say, a specific radio running amok, I expect that the problem would temporarily disappear for a few seconds while the AP in question is being rebooted. This hasn't (so far) proven to be the case. Example: If the problem were in "AP3", then I'd expect the noise problem in "AP4" (almost ten miles away, running ten channels away) to disappear for at least a few seconds while AP3 reboots. I've tested this with virtually every combination of APs, rebooting all the affected ones (and even a few others) in turn, and watching customers on other APs for a change, and haven't seen one. If it were some kind of network flood, why does it only affect certain tower locations, all of which are at least in vague geographical proximity? (And not other towers twenty miles away?) http://www.thedave.us/pics/mvn/wispamap.jpg is a real quick map of the affected areas. The big green dots are "towers that are doing alright", the big red dots are "towers where weird stuff is happening". (The green dot that's right in the center of the three red dots is a 900MHz tower that I probably shouldn't have put on the map, as the Waverider stuff there has been humming right along all the while.) The next-nearest couple of towers (the green dots near Dix at the far north, and Woodlawn to the east) have occasionally exhibited the same behaviour, but not nearly as often as the three I marked in red. Just about the only explanation that makes much sense to me is, basically, "someone on the north edge of town, or a bit further north, is intermittently spamming RF." Maybe it's something on my network, or someone else's. As always, I'm open to suggestions, ideally ones that come with meaningful ways to test. If/when we find the source, I'll try to follow up with everyone, just to put the whole thing to rest. David Smith MVN.net -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Weird problem - 20 seconds latency and other oddness
I agree it could be noise but a bridge runaway will give you the 10+ second pings and with that much traffic being echoed ALL of your AP and Clients are spewing. It would look like a massive RF flood on the Spectrum Analyzer. Think about what the air wave look like when you have full radio usage. To nearby units and competitors it would be a massive increase in the noise floor. Lonnie On 5/9/06, David E. Smith <[EMAIL PROTECTED]> wrote: Lonnie Nunweiler wrote: > Any confirmation on this? A customer router plugged in with LAN to > the WAN or not getting a DHCP entry or even a DNS entry has caused > many bridges to collapse and appear as if it is noise, simply because > the bridges are all echoing the massive broadcast traffic. There's no DHCP anywhere on the network, and the DHCP UDP ports are filtered out at every POP, so that specifically is a bad example. :) Sorry for not getting back to this, we've had massive weirdness on our dialup gear too (mostly related to moving it). Yes, our network is part bridged/part routed. I'm pretty sure it's a real RF problem, because we pulled out the Bumblebee and my field guy said he saw crazy mad noise all across the 2.4 spectrum a couple days ago, when we were having this weird hiccup. (And another local WISP operator reportedly has had similar issues, though I didn't talk to him personally; that's the boss' department.) We'll probably just have to use some old-school triangulation and such to find out where it is, if it's something we even can find. For instance, today was a cool and cloudy day, and this problem didn't show up all day. Thus, I blame sunspots. :D (Honestly, I'm stumped, but at least we're now reasonably certain it's a real RF issue.) If/when we sort it out, I'll report back. David Smith MVN.net -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- Lonnie Nunweiler Valemount Networks Corporation http://www.star-os.com/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Weird problem - 20 seconds latency and other oddness
Lonnie Nunweiler wrote: > Any confirmation on this? A customer router plugged in with LAN to > the WAN or not getting a DHCP entry or even a DNS entry has caused > many bridges to collapse and appear as if it is noise, simply because > the bridges are all echoing the massive broadcast traffic. There's no DHCP anywhere on the network, and the DHCP UDP ports are filtered out at every POP, so that specifically is a bad example. :) Sorry for not getting back to this, we've had massive weirdness on our dialup gear too (mostly related to moving it). Yes, our network is part bridged/part routed. I'm pretty sure it's a real RF problem, because we pulled out the Bumblebee and my field guy said he saw crazy mad noise all across the 2.4 spectrum a couple days ago, when we were having this weird hiccup. (And another local WISP operator reportedly has had similar issues, though I didn't talk to him personally; that's the boss' department.) We'll probably just have to use some old-school triangulation and such to find out where it is, if it's something we even can find. For instance, today was a cool and cloudy day, and this problem didn't show up all day. Thus, I blame sunspots. :D (Honestly, I'm stumped, but at least we're now reasonably certain it's a real RF issue.) If/when we sort it out, I'll report back. David Smith MVN.net -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Weird problem - 20 seconds latency and other oddness
Any confirmation on this? A customer router plugged in with LAN to the WAN or not getting a DHCP entry or even a DNS entry has caused many bridges to collapse and appear as if it is noise, simply because the bridges are all echoing the massive broadcast traffic. Lonnie On 5/8/06, Lonnie Nunweiler <[EMAIL PROTECTED]> wrote: I suspect your system is bridged. Can you confirm that? Lonnie On 5/8/06, David E. Smith <[EMAIL PROTECTED]> wrote: > Okay, Scriv and I are stumped on this one. > > Over the last couple of weeks, we've started seeing some very odd > oddness on a few of our 2.4GHz POPs. Not all, just some. Here's what > appears to be happening: > > A couple times a day, usually during business hours, something somewhere > generates a massive amount of noise. Connections which report an RF > noise of -90 start showing noise levels of -60 (or worse in some cases), > as reported by our StarOS access point. If it really is RF noise, it's > very broad, as it's simultaneously hitting five or six POPs, some > several miles away, but all at the same time. > > The towers are all running StarOS on Mikrotik RouterBoard hardware, with > a mix of Orinoco and Prism cards, some with amps, some not. Some have > sectored antennas (180 degrees), some have omnis. Between them, the > towers cover just about the entire 2.4 spectrum (obviously, one channel > per access point, but we're using at least channels 1, 4, 6, 8, and 11). > > Those towers are basically identical to several other towers that aren't > affected. > > The other really really weird part is the crazy latency. Pings to the > APs themselves are reliable, and our backhaul links (5.3 and 5.8 GHz) > don't seem to be affected. And pings to our end-customers don't seem to > get lost, they just take their sweet time getting there. While "the > event" is happening, I've seen pings that take in excess of twenty > seconds to complete their round trip. > > 64 bytes from 10.232.175.130: icmp_seq=7 ttl=62 time=27239 ms > > (I think that's my record. In that particular test, there were no > packets lost, they just took a very long time to get there.) > > I've checked or replaced just about everything I can think of in our > network that might cause something like this, and frankly, I'm stumped. > I don't think it's a network problem (traffic bursts or similar) because > of the weird bursts of RF noise. But that'd have to be one helluva burst > of noise to do what it's doing - affecting every channel across ten > miles at once. > > I can go into more detail on any part of the network if you like, though > I think all the likely-relevant details are covered here. > > Help! > > David Smith > MVN.net > -- > WISPA Wireless List: wireless@wispa.org > > Subscribe/Unsubscribe: > http://lists.wispa.org/mailman/listinfo/wireless > > Archives: http://lists.wispa.org/pipermail/wireless/ > -- Lonnie Nunweiler Valemount Networks Corporation http://www.star-os.com/ -- Lonnie Nunweiler Valemount Networks Corporation http://www.star-os.com/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Weird problem - 20 seconds latency and other oddness
He's running Star OS and MT as per the original post. Some of the effected ap's are routers. That's one of the first questions I double-checked too! Marlon (509) 982-2181 Equipment sales (408) 907-6910 (Vonage)Consulting services 42846865 (icq)And I run my own wisp! 64.146.146.12 (net meeting) www.odessaoffice.com/wireless www.odessaoffice.com/marlon/cam - Original Message - From: "Lonnie Nunweiler" <[EMAIL PROTECTED]> To: "WISPA General List" Sent: Monday, May 08, 2006 2:32 PM Subject: Re: [WISPA] Weird problem - 20 seconds latency and other oddness I suspect your system is bridged. Can you confirm that? Lonnie On 5/8/06, David E. Smith <[EMAIL PROTECTED]> wrote: Okay, Scriv and I are stumped on this one. Over the last couple of weeks, we've started seeing some very odd oddness on a few of our 2.4GHz POPs. Not all, just some. Here's what appears to be happening: A couple times a day, usually during business hours, something somewhere generates a massive amount of noise. Connections which report an RF noise of -90 start showing noise levels of -60 (or worse in some cases), as reported by our StarOS access point. If it really is RF noise, it's very broad, as it's simultaneously hitting five or six POPs, some several miles away, but all at the same time. The towers are all running StarOS on Mikrotik RouterBoard hardware, with a mix of Orinoco and Prism cards, some with amps, some not. Some have sectored antennas (180 degrees), some have omnis. Between them, the towers cover just about the entire 2.4 spectrum (obviously, one channel per access point, but we're using at least channels 1, 4, 6, 8, and 11). Those towers are basically identical to several other towers that aren't affected. The other really really weird part is the crazy latency. Pings to the APs themselves are reliable, and our backhaul links (5.3 and 5.8 GHz) don't seem to be affected. And pings to our end-customers don't seem to get lost, they just take their sweet time getting there. While "the event" is happening, I've seen pings that take in excess of twenty seconds to complete their round trip. 64 bytes from 10.232.175.130: icmp_seq=7 ttl=62 time=27239 ms (I think that's my record. In that particular test, there were no packets lost, they just took a very long time to get there.) I've checked or replaced just about everything I can think of in our network that might cause something like this, and frankly, I'm stumped. I don't think it's a network problem (traffic bursts or similar) because of the weird bursts of RF noise. But that'd have to be one helluva burst of noise to do what it's doing - affecting every channel across ten miles at once. I can go into more detail on any part of the network if you like, though I think all the likely-relevant details are covered here. Help! David Smith MVN.net -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- Lonnie Nunweiler Valemount Networks Corporation http://www.star-os.com/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Weird problem - 20 seconds latency and other oddness
Yes. It could also be caused by a bad microwave oven. One that's really bad. But to put out 30dB of always on signal wouldn't be that hard to do. Sure screw up a c to i ratio though :-). This could also be a plastics plant that uses 2.4 to heat pellets. Could be a competitor with a screwed up device/customer that's generating far more than normal traffic etc. After talking to John I believe the cause is probably pretty close to the ground though. He's got omni based systems only 10 miles away and they don't see nearly as much of a change. We'd expect nearly as much noise at those as the close in systems if they source were above the trees. It's a fun one. Probably come down to having a guy in the area with a high gain antenna ready to locate the interference when it happens. Two or three reference points at lines drawn on a map should pin point the interference easily enough. If that's what it really turns out to be ;-) Marlon (509) 982-2181 Equipment sales (408) 907-6910 (Vonage)Consulting services 42846865 (icq)And I run my own wisp! 64.146.146.12 (net meeting) www.odessaoffice.com/wireless www.odessaoffice.com/marlon/cam - Original Message - From: "Michael Watson" <[EMAIL PROTECTED]> To: "WISPA General List" Sent: Monday, May 08, 2006 1:10 PM Subject: Re: [WISPA] Weird problem - 20 seconds latency and other oddness If this was rf noise, Arent hamm operators allowed in 2.4 with higher power limits? Could this account for the 5- 10 mile affected area? -Michael David E. Smith wrote: Okay, Scriv and I are stumped on this one. Over the last couple of weeks, we've started seeing some very odd oddness on a few of our 2.4GHz POPs. Not all, just some. Here's what appears to be happening: A couple times a day, usually during business hours, something somewhere generates a massive amount of noise. Connections which report an RF noise of -90 start showing noise levels of -60 (or worse in some cases), as reported by our StarOS access point. If it really is RF noise, it's very broad, as it's simultaneously hitting five or six POPs, some several miles away, but all at the same time. The towers are all running StarOS on Mikrotik RouterBoard hardware, with a mix of Orinoco and Prism cards, some with amps, some not. Some have sectored antennas (180 degrees), some have omnis. Between them, the towers cover just about the entire 2.4 spectrum (obviously, one channel per access point, but we're using at least channels 1, 4, 6, 8, and 11). Those towers are basically identical to several other towers that aren't affected. The other really really weird part is the crazy latency. Pings to the APs themselves are reliable, and our backhaul links (5.3 and 5.8 GHz) don't seem to be affected. And pings to our end-customers don't seem to get lost, they just take their sweet time getting there. While "the event" is happening, I've seen pings that take in excess of twenty seconds to complete their round trip. 64 bytes from 10.232.175.130: icmp_seq=7 ttl=62 time=27239 ms (I think that's my record. In that particular test, there were no packets lost, they just took a very long time to get there.) I've checked or replaced just about everything I can think of in our network that might cause something like this, and frankly, I'm stumped. I don't think it's a network problem (traffic bursts or similar) because of the weird bursts of RF noise. But that'd have to be one helluva burst of noise to do what it's doing - affecting every channel across ten miles at once. I can go into more detail on any part of the network if you like, though I think all the likely-relevant details are covered here. Help! David Smith MVN.net -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Weird problem - 20 seconds latency and other oddness
I suspect your system is bridged. Can you confirm that? Lonnie On 5/8/06, David E. Smith <[EMAIL PROTECTED]> wrote: Okay, Scriv and I are stumped on this one. Over the last couple of weeks, we've started seeing some very odd oddness on a few of our 2.4GHz POPs. Not all, just some. Here's what appears to be happening: A couple times a day, usually during business hours, something somewhere generates a massive amount of noise. Connections which report an RF noise of -90 start showing noise levels of -60 (or worse in some cases), as reported by our StarOS access point. If it really is RF noise, it's very broad, as it's simultaneously hitting five or six POPs, some several miles away, but all at the same time. The towers are all running StarOS on Mikrotik RouterBoard hardware, with a mix of Orinoco and Prism cards, some with amps, some not. Some have sectored antennas (180 degrees), some have omnis. Between them, the towers cover just about the entire 2.4 spectrum (obviously, one channel per access point, but we're using at least channels 1, 4, 6, 8, and 11). Those towers are basically identical to several other towers that aren't affected. The other really really weird part is the crazy latency. Pings to the APs themselves are reliable, and our backhaul links (5.3 and 5.8 GHz) don't seem to be affected. And pings to our end-customers don't seem to get lost, they just take their sweet time getting there. While "the event" is happening, I've seen pings that take in excess of twenty seconds to complete their round trip. 64 bytes from 10.232.175.130: icmp_seq=7 ttl=62 time=27239 ms (I think that's my record. In that particular test, there were no packets lost, they just took a very long time to get there.) I've checked or replaced just about everything I can think of in our network that might cause something like this, and frankly, I'm stumped. I don't think it's a network problem (traffic bursts or similar) because of the weird bursts of RF noise. But that'd have to be one helluva burst of noise to do what it's doing - affecting every channel across ten miles at once. I can go into more detail on any part of the network if you like, though I think all the likely-relevant details are covered here. Help! David Smith MVN.net -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- Lonnie Nunweiler Valemount Networks Corporation http://www.star-os.com/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Weird problem - 20 seconds latency and other oddness
If this was rf noise, Arent hamm operators allowed in 2.4 with higher power limits? Could this account for the 5- 10 mile affected area? -Michael David E. Smith wrote: Okay, Scriv and I are stumped on this one. Over the last couple of weeks, we've started seeing some very odd oddness on a few of our 2.4GHz POPs. Not all, just some. Here's what appears to be happening: A couple times a day, usually during business hours, something somewhere generates a massive amount of noise. Connections which report an RF noise of -90 start showing noise levels of -60 (or worse in some cases), as reported by our StarOS access point. If it really is RF noise, it's very broad, as it's simultaneously hitting five or six POPs, some several miles away, but all at the same time. The towers are all running StarOS on Mikrotik RouterBoard hardware, with a mix of Orinoco and Prism cards, some with amps, some not. Some have sectored antennas (180 degrees), some have omnis. Between them, the towers cover just about the entire 2.4 spectrum (obviously, one channel per access point, but we're using at least channels 1, 4, 6, 8, and 11). Those towers are basically identical to several other towers that aren't affected. The other really really weird part is the crazy latency. Pings to the APs themselves are reliable, and our backhaul links (5.3 and 5.8 GHz) don't seem to be affected. And pings to our end-customers don't seem to get lost, they just take their sweet time getting there. While "the event" is happening, I've seen pings that take in excess of twenty seconds to complete their round trip. 64 bytes from 10.232.175.130: icmp_seq=7 ttl=62 time=27239 ms (I think that's my record. In that particular test, there were no packets lost, they just took a very long time to get there.) I've checked or replaced just about everything I can think of in our network that might cause something like this, and frankly, I'm stumped. I don't think it's a network problem (traffic bursts or similar) because of the weird bursts of RF noise. But that'd have to be one helluva burst of noise to do what it's doing - affecting every channel across ten miles at once. I can go into more detail on any part of the network if you like, though I think all the likely-relevant details are covered here. Help! David Smith MVN.net -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Weird problem - 20 seconds latency and other oddness
Lots of possible causes (self-interference, interference from other networks, too high an oversubscription/traffic level, etc.) but I'd suggest pulling the AP packet-retransmit percentage statistics and manually creating a bar graph with APs across the bottom and retrans percent on the vertical (Y) axis. Any APs that are running above about 10% or 15% are having RF problems. If all the problem APs are below 10% then it is likely that the latency problem is Ethernet-traffic-level related and not an RF/interference problem. If it turns out to be a non-RF problem, is your network switched (with no MAC-layer filtering or protection from abnormally high traffic levels) or routed? Also, is per-client bandwidth-management in place and working? jack David E. Smith wrote: Okay, Scriv and I are stumped on this one. Over the last couple of weeks, we've started seeing some very odd oddness on a few of our 2.4GHz POPs. Not all, just some. Here's what appears to be happening: A couple times a day, usually during business hours, something somewhere generates a massive amount of noise. Connections which report an RF noise of -90 start showing noise levels of -60 (or worse in some cases), as reported by our StarOS access point. If it really is RF noise, it's very broad, as it's simultaneously hitting five or six POPs, some several miles away, but all at the same time. The towers are all running StarOS on Mikrotik RouterBoard hardware, with a mix of Orinoco and Prism cards, some with amps, some not. Some have sectored antennas (180 degrees), some have omnis. Between them, the towers cover just about the entire 2.4 spectrum (obviously, one channel per access point, but we're using at least channels 1, 4, 6, 8, and 11). Those towers are basically identical to several other towers that aren't affected. The other really really weird part is the crazy latency. Pings to the APs themselves are reliable, and our backhaul links (5.3 and 5.8 GHz) don't seem to be affected. And pings to our end-customers don't seem to get lost, they just take their sweet time getting there. While "the event" is happening, I've seen pings that take in excess of twenty seconds to complete their round trip. 64 bytes from 10.232.175.130: icmp_seq=7 ttl=62 time=27239 ms (I think that's my record. In that particular test, there were no packets lost, they just took a very long time to get there.) I've checked or replaced just about everything I can think of in our network that might cause something like this, and frankly, I'm stumped. I don't think it's a network problem (traffic bursts or similar) because of the weird bursts of RF noise. But that'd have to be one helluva burst of noise to do what it's doing - affecting every channel across ten miles at once. I can go into more detail on any part of the network if you like, though I think all the likely-relevant details are covered here. Help! David Smith MVN.net -- Jack Unger ([EMAIL PROTECTED]) - President, Ask-Wi.Com, Inc. Serving the License-Free Wireless Industry Since 1993 Author of the WISP Handbook - "Deploying License-Free Wireless WANs" True Vendor-Neutral WISP Consulting-Training-Troubleshooting Our next WISP Workshop is June 21-22 in Atlanta, GA. Phone (VoIP Over Broadband Wireless) 818-227-4220 www.ask-wi.com -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Weird problem - 20 seconds latency and other oddness
Are you graphing CPU utilization on your StarOS APs? That might help provide a clue. If someone is getting DOSed or there is a broadcast storm, you might see high CPU utilization before/during the problem. Matt David E. Smith wrote: Okay, Scriv and I are stumped on this one. Over the last couple of weeks, we've started seeing some very odd oddness on a few of our 2.4GHz POPs. Not all, just some. Here's what appears to be happening: A couple times a day, usually during business hours, something somewhere generates a massive amount of noise. Connections which report an RF noise of -90 start showing noise levels of -60 (or worse in some cases), as reported by our StarOS access point. If it really is RF noise, it's very broad, as it's simultaneously hitting five or six POPs, some several miles away, but all at the same time. The towers are all running StarOS on Mikrotik RouterBoard hardware, with a mix of Orinoco and Prism cards, some with amps, some not. Some have sectored antennas (180 degrees), some have omnis. Between them, the towers cover just about the entire 2.4 spectrum (obviously, one channel per access point, but we're using at least channels 1, 4, 6, 8, and 11). Those towers are basically identical to several other towers that aren't affected. The other really really weird part is the crazy latency. Pings to the APs themselves are reliable, and our backhaul links (5.3 and 5.8 GHz) don't seem to be affected. And pings to our end-customers don't seem to get lost, they just take their sweet time getting there. While "the event" is happening, I've seen pings that take in excess of twenty seconds to complete their round trip. 64 bytes from 10.232.175.130: icmp_seq=7 ttl=62 time=27239 ms (I think that's my record. In that particular test, there were no packets lost, they just took a very long time to get there.) I've checked or replaced just about everything I can think of in our network that might cause something like this, and frankly, I'm stumped. I don't think it's a network problem (traffic bursts or similar) because of the weird bursts of RF noise. But that'd have to be one helluva burst of noise to do what it's doing - affecting every channel across ten miles at once. I can go into more detail on any part of the network if you like, though I think all the likely-relevant details are covered here. Help! David Smith MVN.net -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
