Re: [WISPA] The Gremlin, redux
Send a call tag over and we'll ship you my SA. LEAVE it at one of the towers. You'll want a sectored antenna on it. 120* for starters. When the noise hits move to another direction with the antenna. It'll take a while but you'll be able to see what direction it comes from. Then move it to another tower, do the same thing. After 2 or 3 towers you'll be able to see where it's coming from. Just use the max hold function. Hook up a cheap webcam so you can see the screen on the analyzer. Set that up as the active desktop on your puter and check it often. Won't take long to see unusually high peaks. call me and we'll talk over some ideas. 509.988.0260 cell Marlon (509) 982-2181 Equipment sales (408) 907-6910 (Vonage)Consulting services 42846865 (icq)And I run my own wisp! 64.146.146.12 (net meeting) www.odessaoffice.com/wireless www.odessaoffice.com/marlon/cam - Original Message - From: David E. Smith [EMAIL PROTECTED] To: WISPA General List wireless@wispa.org Sent: Friday, October 27, 2006 9:32 AM Subject: Re: [WISPA] The Gremlin, redux Marlon K. Schafer (509) 982-2181 wrote: Scriv needs to hire a good consultant to come check things out! big grin Know any? :D The hard part there would be that it's not, in any way, predictable. We've gone several weeks at a time without this problem appearing, and had days where it showed up several times. Most of the time, I only see it in our network monitor logs, as it came and went within a minute or two. Assuming it's some massive source of interference, I imagine it would be very difficult to triangulate it. (By the time I call even one person to say fire up your spectrum analyzer, it's gone.) First, as I recall, this ONLY effects towers within a 15 mile radius. But it effects ALL towers within that cell. right? Yep. I had a nice map that showed all our towers, and which ones were (and weren't) affected. I may try to dig that up, or recreate it. I've seen customers download or upload massive files (usually ptp stuff) and open up hundreds of connections and whack a tower. If it's the right tower and interferes with the other towers anywhere near it. That's probably not the case, as we've recently taken measures to reduce P2P traffic (a Mikrotik box and some pretty harsh traffic shaping rules). Also, I've seen P2P traffic before, and while it usually slows down that specific AP, I've never seen it affect our backhaul or other nearby towers. We're pretty good in terms on not walking on ourselves. There are no overlaps (AFAIK) between our towers if they're within five miles or so of one another, and facing towards each other. We are using the whole 2.4GHz spectrum, of course, but not all of it in any one place. There are times when radios go bad and start transmitting OUT of band. I've seen wifi stuff flood the whole band. Amps can do that too. We do have a couple of amps here and there, but I don't think any of them are over 200mW. That'd have to be one heckuva runaway amp to cause that much interference. If it were a bad radio, wouldn't it be a more frequent problem instead of one that randomly shows up for a minute or two here and there? Is it possible that your 900 and 5 gig gear ALWAYS has a router at the cpe and the 2.4 doesn't? As I recall the Waverider gear, the ap is a router, that would keep things that would flow on a switch off of the 900 system. Not even close. We've got folks with 900MHz gear and 5.whatever that just plug their radios straight into their PCs. (At least I know that's the case with the 900MHz stuff, as we have a lot of residential customers on it; I can't, off the top of my head, think of any 5.3/5.8 customers without a router, but we don't have that many of them - we're mostly using those bands for inter-tower backhaul.) Are there any towers that are sectorized that point in a direction that makes them unaffected? I wish... Two of them have two 180-degree sector antennas facing in opposite directions, on opposite sides of a water tower (i.e. with twenty feet of steel and water between them), and both sides are affected equally. David Smith MVN.net -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] The Gremlin, redux
We have analyzers already. Thanks for the offer though. Scriv Marlon K. Schafer (509) 982-2181 wrote: Send a call tag over and we'll ship you my SA. LEAVE it at one of the towers. You'll want a sectored antenna on it. 120* for starters. When the noise hits move to another direction with the antenna. It'll take a while but you'll be able to see what direction it comes from. Then move it to another tower, do the same thing. After 2 or 3 towers you'll be able to see where it's coming from. Just use the max hold function. Hook up a cheap webcam so you can see the screen on the analyzer. Set that up as the active desktop on your puter and check it often. Won't take long to see unusually high peaks. call me and we'll talk over some ideas. 509.988.0260 cell Marlon (509) 982-2181 Equipment sales (408) 907-6910 (Vonage)Consulting services 42846865 (icq)And I run my own wisp! 64.146.146.12 (net meeting) www.odessaoffice.com/wireless www.odessaoffice.com/marlon/cam - Original Message - From: David E. Smith [EMAIL PROTECTED] To: WISPA General List wireless@wispa.org Sent: Friday, October 27, 2006 9:32 AM Subject: Re: [WISPA] The Gremlin, redux Marlon K. Schafer (509) 982-2181 wrote: Scriv needs to hire a good consultant to come check things out! big grin Know any? :D The hard part there would be that it's not, in any way, predictable. We've gone several weeks at a time without this problem appearing, and had days where it showed up several times. Most of the time, I only see it in our network monitor logs, as it came and went within a minute or two. Assuming it's some massive source of interference, I imagine it would be very difficult to triangulate it. (By the time I call even one person to say fire up your spectrum analyzer, it's gone.) First, as I recall, this ONLY effects towers within a 15 mile radius. But it effects ALL towers within that cell. right? Yep. I had a nice map that showed all our towers, and which ones were (and weren't) affected. I may try to dig that up, or recreate it. I've seen customers download or upload massive files (usually ptp stuff) and open up hundreds of connections and whack a tower. If it's the right tower and interferes with the other towers anywhere near it. That's probably not the case, as we've recently taken measures to reduce P2P traffic (a Mikrotik box and some pretty harsh traffic shaping rules). Also, I've seen P2P traffic before, and while it usually slows down that specific AP, I've never seen it affect our backhaul or other nearby towers. We're pretty good in terms on not walking on ourselves. There are no overlaps (AFAIK) between our towers if they're within five miles or so of one another, and facing towards each other. We are using the whole 2.4GHz spectrum, of course, but not all of it in any one place. There are times when radios go bad and start transmitting OUT of band. I've seen wifi stuff flood the whole band. Amps can do that too. We do have a couple of amps here and there, but I don't think any of them are over 200mW. That'd have to be one heckuva runaway amp to cause that much interference. If it were a bad radio, wouldn't it be a more frequent problem instead of one that randomly shows up for a minute or two here and there? Is it possible that your 900 and 5 gig gear ALWAYS has a router at the cpe and the 2.4 doesn't? As I recall the Waverider gear, the ap is a router, that would keep things that would flow on a switch off of the 900 system. Not even close. We've got folks with 900MHz gear and 5.whatever that just plug their radios straight into their PCs. (At least I know that's the case with the 900MHz stuff, as we have a lot of residential customers on it; I can't, off the top of my head, think of any 5.3/5.8 customers without a router, but we don't have that many of them - we're mostly using those bands for inter-tower backhaul.) Are there any towers that are sectorized that point in a direction that makes them unaffected? I wish... Two of them have two 180-degree sector antennas facing in opposite directions, on opposite sides of a water tower (i.e. with twenty feet of steel and water between them), and both sides are affected equally. David Smith MVN.net -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] The Gremlin, redux
John Scrivner wrote: Mac, We believe this is truly an outside offender in 2.4 GHz. I have personally seen a carrier that is several times more power than anything I have ever seen. I only saw it for a brief instant though. This interference just does not last long enough to be caught. The high latency is caused by retransmits but I am sure outside interference is what is leading to the need for frames to be sent again. This effects all channels across the 2.4 GHz bands. We have seen the noise floor jump up higher than our radio power levels when this problem happens. What ever is causing this is running higher power than anything we have in the field. We will look at anything, though, to help troubleshoot and I appreciate your ideas. Scriv We've had this too and have never been able to narrow it down. Basiclly, certain areas and without warning or obvious cause, simply become dead for 2.4ghz in that there appears some very powerful inband interference that is not 802.11b/g and has no obvivious source we can find, but the area of effect is fairly localized (using affected customers to tirangulate). We also have a problem within our hometown of repeatedly experiencing burnt out 2.4ghz equipment. Never happens anywhere else in our county wide footprint, just our hometown and across a wide variety of equipment such as smartbridges, atheros, prism 2.5 cards, cb3's, cisco aironet, you name it. Been looking at this problem for years and simply don't understand it. We've had events like this occuring down one particular street for example and an accesspoint not too far away, all in the space of one evening. We've also had events like one subscriber per day (on 2.4ghz) winding up with burnt out prism card, requiring a truck roll and card replacement. Mike- -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
RE: [WISPA] The Gremlin, redux
David, I tend to believe you will find your answer on your network -vs- big bad leak somewhere and the only real suggestion I can offer you would be to do what we do here when we start having weird issues - - and that is to be ready to start unplugging Back Hauls (one at a time) out of their switch..Etc when the trouble starts. Have a ping running to watch for the latency to disappear when you unplug the offending Back Haul. Once you narrow it down to the right leg of the network you can reboot one AP at a time until you find which AP it is coming from by running an extended ping on that leg of the network Don't pull a big dummy like we did here for weeks!! Unplug the servers one at a time too!!! That is where we found one of the servers we host was our culprit. There traffic was of such that it was flooding our switches - - (dirty suckers) :-) GL - keep us informed as to what you find Mac Dearman -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David E. Smith Sent: Friday, October 27, 2006 4:10 PM To: WISPA General List Subject: Re: [WISPA] The Gremlin, redux On Fri, October 27, 2006 3:11 pm, Eric Merkel wrote: 1) Turning off inter-BSS Relay Already done, on most towers. (We do have a couple of towers where one business, with two locations, wants to do VPN-type stuff between 'em.) 2) We block all the typical MS ports(135-139) which broadcast all the time via iptables Done. We block 135-139, 445, and a couple other ports, both TCP and UDP. 3) Packet shape all connections via CBQ on the AP itself to limit how much bandwidth any one customer can consume Mostly done. (For historical reasons, some of our customers are still part of a giant bridged network, and their traffic is shaped in our office not at the AP, but those customers are relatively few and growing fewer by the week.) David Smith MVN.net -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] The Gremlin, redux
Mac Dearman wrote: I tend to believe you will find your answer on your network -vs- big bad leak somewhere and the only real suggestion I can offer you would be to do what we do here when we start having weird issues [ snip: Mac's good advice on how to track down broadcast storms and other network-related problems ] If it were a backhaul problem, though, wouldn't I see high latency on my backhaul links? I haven't, to date. Here's one of the (many) experiments I've done in the past (this one was yesterday, actually): I logged into an affected AP, while the problem was happening, and firewalled off THE WHOLE INTERNET except for one IP address, that being the PC in my office I was using to log into that AP. Then, from that AP, I pinged a CPE associated with it. The problem persisted, high packet loss and 5000-1 millisecond latency. I quickly undid my changes on that AP, then did exactly the same thing on another AP, about twenty miles away, with a different radio card, on a different channel, connected to me through a different backhaul link, and saw exactly the same performance (high packet loss, obscene latency, et cetera). Meanwhile, pings to and through our backhaul links, to the APs themselves, various managed switches at tower locations, and so on, never skipped a beat. (Heck, most of 'em improved a bit, since there wasn't as much pesky customer traffic going through them. :) For that matter, why do I have three towers with a mixture of 2.4GHz and other APs (two with a 900MHz AP, one with a 5.3GHz AP and a 5.8GHz AP) and the non-2.4 customers aren't affected? Keep in mind that, for annoying historical reasons, much of our network is still flat, bridged addresses flying willy-nilly across four counties. If it were a network storm, I'd expect it to hit all our towers, on all channels, and not conveniently skip over the most geographically remote ones. There's also all the nasty logistical problems of my company not having twenty extra hands that we can just have sitting at all our towers for days, or weeks, at a time, waiting for a problem that shows up completely randomly, hoping I can call everyone to start unplugging stuff before the problem magically goes away, but that's another issue altogether ;) Last, remember this really is amazingly random, and usually only shows up for a minute or two at a time, brief enough that I only see it in our logs well after the fact. (Yesterday it visited us for a good twenty minutes, long enough for customers to really notice, and for me to have time to dig into it again.) I certainly don't mean to sound dismissive of any suggestions, it's just that I've tried most of the obvious ones before. David Smith MVN.net -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] The Gremlin, redux
Eric wrote: Looking at the beacon realtime manager and tcpdump, we've never seen an unreasonable # of broadcasts when this is happening. On our network 200 broadcasts per second is pretty typical. When we see it spike to over 1200 per second (6-fold rise), it really drives the latency up. Don't be fooled by total bw metrics ... broadcasts are so short that crippling broadcast storms don't show any spike in total traffic (traffic is actually squeezed out by the broadcasts dominating the radios). I'm thinking tcpdump is like the packet dump that ethereal uses. How do you determine how many broadcasts per second is transiting the AP from it? Just curious. I'm not familiar with beacon realtime manager ... can it tell you how many broadcasts per second are on the air during your events? When you say you've never seen an unreasonable # of broadcasts, how many broadcasts per second do you see during an event? What we do is simply chart broadcasts in out for every radio on our network on a continuous basis. Just about anything will do that, so knowing broadcast levels (and who's causing them) is a piece of cake ... no harder than opening a web page and scrolling to spot the offenders. Rich - Original Message - From: Eric Merkel [EMAIL PROTECTED] To: WISPA General List wireless@wispa.org Sent: Friday, October 27, 2006 3:11 PM Subject: Re: [WISPA] The Gremlin, redux On 10/27/06, Rich Comroe [EMAIL PROTECTED] wrote: We look at the traffic on the tower for abuse and/or virus and don't really find anything. Just to be clear, you've checked your AP broadcast levels during the events and not found found them elevated? We found the most crippling network events were not coming into the network from the outside, but were broadcast storms between 2 or more customers (repeated through the APs). They act similar to the symptoms you cited (a few minutes of extremely elevated latency due to the short term load they place over the rf). Rich We try to mitigate this problem by the following: 1) Turning off inter-BSS Relay 2) We block all the typical MS ports(135-139) which broadcast all the time via iptables 3) Packet shape all connections via CBQ on the AP itself to limit how much bandwidth any one customer can consume Looking at the beacon realtime manager and tcpdump, we've never seen an unreasonable # of broadcasts when this is happening. -Eric -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] The Gremlin, redux
Since your customers are mostly on a bridged network then a single malicious or misbehaving customer radio can take your WHOLE network down. The reason your backhaul is not affected is simply because it has more capacity than the prism AP units. I know you guys are tired of hearing from me about routing versus bridging (You even gave me my own clause) but your problems are precisely what I have been describing for all of these years. The one point you are missing in your overall design is something I have brought up a number of times. A properly functioning bridged system will echo all broadcast traffic throughout the network, thus one system can flood the entire LAN. This leads to the biggest single factor -- your entire network speed is a factor of your slowest AP, since it must rebroadcast all traffic to its users. On a routed design you will still get problems like these slowdowns, but the very important thing to keep in mind is that the slowdown will only take out the one sector. Also (and this should be enough to make you take a hard look at routing) your LAN throughput is the sum total of all of your AP units and thus you might actually start to use your high speed backhaul. Your ping times will go under 2 msec per repeater. Our WAR boards are actually around the 1 msec area. The other thing you have going against you is that you are using a unit based on the Ubicom chipset. Remember the dear old Wet11? That was Ubicom using a prism radio. CB3 is simply a newer version, but still with some issues that seem to be killing networks everywhere. Our last version of the Atheros driver would start to show units could not reconnect and the only way to make the system settle down was to change channels or reboot. What is happening is that the CB3 has trouble when it reconnects to an AP. Something is not quite right with its handshake and it does not get accepted. This causes it to throw a hissy fit and it starts to hammer the AP with reconnection attempts. That massive RF directed at the AP soon causes other units to disconnect and guess what happens if they are a CB3? They try the same techniques the first tried. Suddenly you have this massive reconnect storm and the AP is brought to its knees until you change channels or reboot. Your whole problem is circular and is based on poor choices of both equipment and network design. You chose cheap units that only do a pseudo bridge so you designed your LAN accordingly. You thought you would be safe because you had a good AP, but a LAN is mostly made up of clients, so if the clients are poor, then it follows that your LAN is poor. You have some sectors already routed, but they still have the trouble. I say this is because of the trouble of the clients, which could be at this tower or simply because your tower is visible to them. You are quite likely even using 200 mW prism cards which makes this factor even more probable. So this means you have two issues. 1. Clients that will have to be updated with better firmware or replaced. 2. Your entire system will have to be redesigned for a proper routed design. The way out of this is obvious, but it will be painful and costly. There is going to be a lot of anger directed at me for this message so I am now ducking down to avoid the flames. Regards, Lonnie On 10/28/06, David E. Smith [EMAIL PROTECTED] wrote: Mac Dearman wrote: I tend to believe you will find your answer on your network -vs- big bad leak somewhere and the only real suggestion I can offer you would be to do what we do here when we start having weird issues [ snip: Mac's good advice on how to track down broadcast storms and other network-related problems ] If it were a backhaul problem, though, wouldn't I see high latency on my backhaul links? I haven't, to date. Here's one of the (many) experiments I've done in the past (this one was yesterday, actually): I logged into an affected AP, while the problem was happening, and firewalled off THE WHOLE INTERNET except for one IP address, that being the PC in my office I was using to log into that AP. Then, from that AP, I pinged a CPE associated with it. The problem persisted, high packet loss and 5000-1 millisecond latency. I quickly undid my changes on that AP, then did exactly the same thing on another AP, about twenty miles away, with a different radio card, on a different channel, connected to me through a different backhaul link, and saw exactly the same performance (high packet loss, obscene latency, et cetera). Meanwhile, pings to and through our backhaul links, to the APs themselves, various managed switches at tower locations, and so on, never skipped a beat. (Heck, most of 'em improved a bit, since there wasn't as much pesky customer traffic going through them. :) For that matter, why do I have three towers with a mixture of 2.4GHz and other APs (two with a 900MHz AP, one with a 5.3GHz AP and a 5.8GHz AP) and the non-2.4 customers aren't affected? Keep
Re: [WISPA] The Gremlin, redux
Mac, We believe this is truly an outside offender in 2.4 GHz. I have personally seen a carrier that is several times more power than anything I have ever seen. I only saw it for a brief instant though. This interference just does not last long enough to be caught. The high latency is caused by retransmits but I am sure outside interference is what is leading to the need for frames to be sent again. This effects all channels across the 2.4 GHz bands. We have seen the noise floor jump up higher than our radio power levels when this problem happens. What ever is causing this is running higher power than anything we have in the field. We will look at anything, though, to help troubleshoot and I appreciate your ideas. Scriv Mac Dearman wrote: David, I tend to believe you will find your answer on your network -vs- big bad leak somewhere and the only real suggestion I can offer you would be to do what we do here when we start having weird issues - - and that is to be ready to start unplugging Back Hauls (one at a time) out of their switch..Etc when the trouble starts. Have a ping running to watch for the latency to disappear when you unplug the offending Back Haul. Once you narrow it down to the right leg of the network you can reboot one AP at a time until you find which AP it is coming from by running an extended ping on that leg of the network Don't pull a big dummy like we did here for weeks!! Unplug the servers one at a time too!!! That is where we found one of the servers we host was our culprit. There traffic was of such that it was flooding our switches - - (dirty suckers) :-) GL - keep us informed as to what you find Mac Dearman -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David E. Smith Sent: Friday, October 27, 2006 4:10 PM To: WISPA General List Subject: Re: [WISPA] The Gremlin, redux On Fri, October 27, 2006 3:11 pm, Eric Merkel wrote: 1) Turning off inter-BSS Relay Already done, on most towers. (We do have a couple of towers where one business, with two locations, wants to do VPN-type stuff between 'em.) 2) We block all the typical MS ports(135-139) which broadcast all the time via iptables Done. We block 135-139, 445, and a couple other ports, both TCP and UDP. 3) Packet shape all connections via CBQ on the AP itself to limit how much bandwidth any one customer can consume Mostly done. (For historical reasons, some of our customers are still part of a giant bridged network, and their traffic is shaped in our office not at the AP, but those customers are relatively few and growing fewer by the week.) David Smith MVN.net -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
RE: SPAM ? Re: [WISPA] The Gremlin, redux
VLAN's and Trunking stop all the broadcast traffic. You won't hear me say NOT to static route (yuck) as I have a lot of my network statically routed, but it does NOT out perform my bridged, VLAN'd network segments. I did have a completely flat bridged network and never had my network flooded by a single user with a virus. I am not saying that its not possible, but it never happened to me once in 4 years (and I have 15% of the State of Louisiana covered with wireless) before changing half of my network to a statically routed network. I did have a switch to get flooded in the NOC that actually shut my network down many times before I caught that sucker. I realize that a VLAN'd network can still have trouble, but no more than your statically routed network. There are more ways to skin a cat than one :-) although we know your preferences :-) Mac -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Lonnie Nunweiler Sent: Saturday, October 28, 2006 4:18 PM To: WISPA General List Subject: SPAM ? Re: [WISPA] The Gremlin, redux Importance: Low Since your customers are mostly on a bridged network then a single malicious or misbehaving customer radio can take your WHOLE network down. The reason your backhaul is not affected is simply because it has more capacity than the prism AP units. I know you guys are tired of hearing from me about routing versus bridging (You even gave me my own clause) but your problems are precisely what I have been describing for all of these years. The one point you are missing in your overall design is something I have brought up a number of times. A properly functioning bridged system will echo all broadcast traffic throughout the network, thus one system can flood the entire LAN. This leads to the biggest single factor -- your entire network speed is a factor of your slowest AP, since it must rebroadcast all traffic to its users. On a routed design you will still get problems like these slowdowns, but the very important thing to keep in mind is that the slowdown will only take out the one sector. Also (and this should be enough to make you take a hard look at routing) your LAN throughput is the sum total of all of your AP units and thus you might actually start to use your high speed backhaul. Your ping times will go under 2 msec per repeater. Our WAR boards are actually around the 1 msec area. The other thing you have going against you is that you are using a unit based on the Ubicom chipset. Remember the dear old Wet11? That was Ubicom using a prism radio. CB3 is simply a newer version, but still with some issues that seem to be killing networks everywhere. Our last version of the Atheros driver would start to show units could not reconnect and the only way to make the system settle down was to change channels or reboot. What is happening is that the CB3 has trouble when it reconnects to an AP. Something is not quite right with its handshake and it does not get accepted. This causes it to throw a hissy fit and it starts to hammer the AP with reconnection attempts. That massive RF directed at the AP soon causes other units to disconnect and guess what happens if they are a CB3? They try the same techniques the first tried. Suddenly you have this massive reconnect storm and the AP is brought to its knees until you change channels or reboot. Your whole problem is circular and is based on poor choices of both equipment and network design. You chose cheap units that only do a pseudo bridge so you designed your LAN accordingly. You thought you would be safe because you had a good AP, but a LAN is mostly made up of clients, so if the clients are poor, then it follows that your LAN is poor. You have some sectors already routed, but they still have the trouble. I say this is because of the trouble of the clients, which could be at this tower or simply because your tower is visible to them. You are quite likely even using 200 mW prism cards which makes this factor even more probable. So this means you have two issues. 1. Clients that will have to be updated with better firmware or replaced. 2. Your entire system will have to be redesigned for a proper routed design. The way out of this is obvious, but it will be painful and costly. There is going to be a lot of anger directed at me for this message so I am now ducking down to avoid the flames. Regards, Lonnie On 10/28/06, David E. Smith [EMAIL PROTECTED] wrote: Mac Dearman wrote: I tend to believe you will find your answer on your network -vs- big bad leak somewhere and the only real suggestion I can offer you would be to do what we do here when we start having weird issues [ snip: Mac's good advice on how to track down broadcast storms and other network-related problems ] If it were a backhaul problem, though, wouldn't I see high latency on my backhaul links? I haven't, to date. Here's one of the (many) experiments I've done in the past (this one
RE: [WISPA] The Gremlin, redux
John, Can you set up a CPE to monitor noise floor levels (MRTG..etc) and point them at different towers owned by different companies that may be suspect in your area? If you can identify this - - I have another idea to prove it or release a suspect tower out of suspicion, but that will require a personal phone call :-) Mac Dearman -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of John Scrivner Sent: Saturday, October 28, 2006 4:22 PM To: WISPA General List Subject: Re: [WISPA] The Gremlin, redux Mac, We believe this is truly an outside offender in 2.4 GHz. I have personally seen a carrier that is several times more power than anything I have ever seen. I only saw it for a brief instant though. This interference just does not last long enough to be caught. The high latency is caused by retransmits but I am sure outside interference is what is leading to the need for frames to be sent again. This effects all channels across the 2.4 GHz bands. We have seen the noise floor jump up higher than our radio power levels when this problem happens. What ever is causing this is running higher power than anything we have in the field. We will look at anything, though, to help troubleshoot and I appreciate your ideas. Scriv Mac Dearman wrote: David, I tend to believe you will find your answer on your network -vs- big bad leak somewhere and the only real suggestion I can offer you would be to do what we do here when we start having weird issues - - and that is to be ready to start unplugging Back Hauls (one at a time) out of their switch..Etc when the trouble starts. Have a ping running to watch for the latency to disappear when you unplug the offending Back Haul. Once you narrow it down to the right leg of the network you can reboot one AP at a time until you find which AP it is coming from by running an extended ping on that leg of the network Don't pull a big dummy like we did here for weeks!! Unplug the servers one at a time too!!! That is where we found one of the servers we host was our culprit. There traffic was of such that it was flooding our switches - - (dirty suckers) :-) GL - keep us informed as to what you find Mac Dearman -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David E. Smith Sent: Friday, October 27, 2006 4:10 PM To: WISPA General List Subject: Re: [WISPA] The Gremlin, redux On Fri, October 27, 2006 3:11 pm, Eric Merkel wrote: 1) Turning off inter-BSS Relay Already done, on most towers. (We do have a couple of towers where one business, with two locations, wants to do VPN-type stuff between 'em.) 2) We block all the typical MS ports(135-139) which broadcast all the time via iptables Done. We block 135-139, 445, and a couple other ports, both TCP and UDP. 3) Packet shape all connections via CBQ on the AP itself to limit how much bandwidth any one customer can consume Mostly done. (For historical reasons, some of our customers are still part of a giant bridged network, and their traffic is shaped in our office not at the AP, but those customers are relatively few and growing fewer by the week.) David Smith MVN.net -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] The Gremlin, redux
Hi, You may have covered this before, but I probably missed it in your original post. What equipment are you running for each of the different frequencies (2.4ghz, 900mhz, 5.8ghz, etc.). Is it possible the 2.4ghz AP's are just not able to handle the high packet loads (broadcast storms, viruses, whatever) that the others are? Also, I have never seen high noise on a band actually cause high latency... usually it causes packet loss. My guess is you have an internal problem, not external. Travis Microserv David E. Smith wrote: Mac Dearman wrote: I tend to believe you will find your answer on your network -vs- big bad leak somewhere and the only real suggestion I can offer you would be to do what we do here when we start having weird issues [ snip: Mac's good advice on how to track down broadcast storms and other network-related problems ] If it were a backhaul problem, though, wouldn't I see high latency on my backhaul links? I haven't, to date. Here's one of the (many) experiments I've done in the past (this one was yesterday, actually): I logged into an affected AP, while the problem was happening, and firewalled off THE WHOLE INTERNET except for one IP address, that being the PC in my office I was using to log into that AP. Then, from that AP, I pinged a CPE associated with it. The problem persisted, high packet loss and 5000-1 millisecond latency. I quickly undid my changes on that AP, then did exactly the same thing on another AP, about twenty miles away, with a different radio card, on a different channel, connected to me through a different backhaul link, and saw exactly the same performance (high packet loss, obscene latency, et cetera). Meanwhile, pings to and through our backhaul links, to the APs themselves, various managed switches at tower locations, and so on, never skipped a beat. (Heck, most of 'em improved a bit, since there wasn't as much pesky customer traffic going through them. :) For that matter, why do I have three towers with a mixture of 2.4GHz and other APs (two with a 900MHz AP, one with a 5.3GHz AP and a 5.8GHz AP) and the non-2.4 customers aren't affected? Keep in mind that, for annoying historical reasons, much of our network is still flat, bridged addresses flying willy-nilly across four counties. If it were a network storm, I'd expect it to hit all our towers, on all channels, and not conveniently skip over the most geographically remote ones. There's also all the nasty logistical problems of my company not having twenty extra hands that we can just have sitting at all our towers for days, or weeks, at a time, waiting for a problem that shows up completely randomly, hoping I can call everyone to start unplugging stuff before the problem magically goes away, but that's another issue altogether ;) Last, remember this really is amazingly random, and usually only shows up for a minute or two at a time, brief enough that I only see it in our logs well after the fact. (Yesterday it visited us for a good twenty minutes, long enough for customers to really notice, and for me to have time to dig into it again.) I certainly don't mean to sound dismissive of any suggestions, it's just that I've tried most of the obvious ones before. David Smith MVN.net -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
[WISPA] The Gremlin, redux
This problem was mentioned back in May (see http://lists.wispa.org/pipermail/wireless/2006-May/025354.html ) but it's still here, and I thought it might be worthwhile to bounce it off folks again, to see if anyone has any new insights. Occasionally, completely at random, many of our 2.4GHz APs start showing high packet loss (usually 20-30%) and high latency (upwards of ten seconds - not milliseconds, full seconds) for a few minutes at a time. It always clears itself up eventually, usually after a minute or two, but sometimes it won't go away for ten or fifteen minutes. It's especially annoying when it happens in the middle of the day and all those itchy business customers call in at once. Random facts and tidbits: * I've watched our network traffic, using Mikrotik's torch and StarOS' beacon tools, and I don't see anything that looks like a DDOS, either entering or leaving our network. * It doesn't affect all our towers, just most of them. Specifically, our most remote towers (geographically, not in terms of network topology) are safe. * (This is the Lonnie Nunweiler clause) It affects equally towers that are bridged, routed, and hybrid bridged/routed. * I don't think it's a backhaul problem, because towers on several different kinds of backhaul links are affected simultaneously. (We've got a mix of Alvarion and Trango gear for backhauls, and at least one ancient YDI EX-1.) Also, there's no problem talking between tower locations; pings between different APs take 10ms or so, just like they normally do. * It only affects our 2.4GHz customers - folks on 900MHz gear, 5.3GHz, and 5.8GHz connections don't have any issues while things are going sour. Basically, it looks like someone's got a big giant massive something that spews out insane amounts of 2.4GHz interference, that somehow knocks out all our customers within about a fifteen-mile radius, and they turn it on every so often. Does that conclusion sound reasonable? And if it does, what the heck can I do about it? Frustrated, David Smith MVN.net -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] The Gremlin, redux
Scriv needs to hire a good consultant to come check things out! big grin Anyway, I've seen things like this before. Not this bad, but close enough. There have been a few causes. First, as I recall, this ONLY effects towers within a 15 mile radius. But it effects ALL towers within that cell. right? Assuming I remember that part correctly. I've seen CPE lock it's self into a tx mode. It'll still work, but will drive the system nuts. I've seen customers download or upload massive files (usually ptp stuff) and open up hundreds of connections and whack a tower. If it's the right tower and interferes with the other towers anywhere near it. I've been knocked offline by towers over 30 miles away! OK, that was a ptp wmux system with 8' dishes and 30dB of tx power, but still, it happened. With the new gear I'm using these days I can detect ap's 30+ miles away. If conditions are JUST right, that could add quite a bit of noise locally. There are times when radios go bad and start transmitting OUT of band. I've seen wifi stuff flood the whole band. Amps can do that too. USUALLY it's the ap's that take it in the shorts when something like this happens. After all, they are all up above the trees etc. I've also seen computers take out entire networks when they get infected. If you don't have client to client blocking enabled on your ap's they could be massively overloaded with traffic that you'll not be able to see with a scan tool at the router, it could be traffic that's not even ip traffic! Remember, most of the gear we use with pass netbeui, ipx, appletalk etc. I think the fact that towers that have routers on them between the OTHER ap's and the backhaul should prevent this. Is it possible that your 900 and 5 gig gear ALWAYS has a router at the cpe and the 2.4 doesn't? As I recall the Waverider gear, the ap is a router, that would keep things that would flow on a switch off of the 900 system. Are there any towers that are sectorized that point in a direction that makes them unaffected? Marlon (509) 982-2181 Equipment sales (408) 907-6910 (Vonage)Consulting services 42846865 (icq)And I run my own wisp! 64.146.146.12 (net meeting) www.odessaoffice.com/wireless www.odessaoffice.com/marlon/cam - Original Message - From: David E. Smith [EMAIL PROTECTED] To: wireless@wispa.org Sent: Friday, October 27, 2006 9:39 AM Subject: [WISPA] The Gremlin, redux This problem was mentioned back in May (see http://lists.wispa.org/pipermail/wireless/2006-May/025354.html ) but it's still here, and I thought it might be worthwhile to bounce it off folks again, to see if anyone has any new insights. Occasionally, completely at random, many of our 2.4GHz APs start showing high packet loss (usually 20-30%) and high latency (upwards of ten seconds - not milliseconds, full seconds) for a few minutes at a time. It always clears itself up eventually, usually after a minute or two, but sometimes it won't go away for ten or fifteen minutes. It's especially annoying when it happens in the middle of the day and all those itchy business customers call in at once. Random facts and tidbits: * I've watched our network traffic, using Mikrotik's torch and StarOS' beacon tools, and I don't see anything that looks like a DDOS, either entering or leaving our network. * It doesn't affect all our towers, just most of them. Specifically, our most remote towers (geographically, not in terms of network topology) are safe. * (This is the Lonnie Nunweiler clause) It affects equally towers that are bridged, routed, and hybrid bridged/routed. * I don't think it's a backhaul problem, because towers on several different kinds of backhaul links are affected simultaneously. (We've got a mix of Alvarion and Trango gear for backhauls, and at least one ancient YDI EX-1.) Also, there's no problem talking between tower locations; pings between different APs take 10ms or so, just like they normally do. * It only affects our 2.4GHz customers - folks on 900MHz gear, 5.3GHz, and 5.8GHz connections don't have any issues while things are going sour. Basically, it looks like someone's got a big giant massive something that spews out insane amounts of 2.4GHz interference, that somehow knocks out all our customers within about a fifteen-mile radius, and they turn it on every so often. Does that conclusion sound reasonable? And if it does, what the heck can I do about it? Frustrated, David Smith MVN.net -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] The Gremlin, redux
If it's true that there's a giant something that's spewing noise, you can use a spectrum analyzer and try to identify the noise signature, then triangulate. jack David E. Smith wrote: This problem was mentioned back in May (see http://lists.wispa.org/pipermail/wireless/2006-May/025354.html ) but it's still here, and I thought it might be worthwhile to bounce it off folks again, to see if anyone has any new insights. Occasionally, completely at random, many of our 2.4GHz APs start showing high packet loss (usually 20-30%) and high latency (upwards of ten seconds - not milliseconds, full seconds) for a few minutes at a time. It always clears itself up eventually, usually after a minute or two, but sometimes it won't go away for ten or fifteen minutes. It's especially annoying when it happens in the middle of the day and all those itchy business customers call in at once. Random facts and tidbits: * I've watched our network traffic, using Mikrotik's torch and StarOS' beacon tools, and I don't see anything that looks like a DDOS, either entering or leaving our network. * It doesn't affect all our towers, just most of them. Specifically, our most remote towers (geographically, not in terms of network topology) are safe. * (This is the Lonnie Nunweiler clause) It affects equally towers that are bridged, routed, and hybrid bridged/routed. * I don't think it's a backhaul problem, because towers on several different kinds of backhaul links are affected simultaneously. (We've got a mix of Alvarion and Trango gear for backhauls, and at least one ancient YDI EX-1.) Also, there's no problem talking between tower locations; pings between different APs take 10ms or so, just like they normally do. * It only affects our 2.4GHz customers - folks on 900MHz gear, 5.3GHz, and 5.8GHz connections don't have any issues while things are going sour. Basically, it looks like someone's got a big giant massive something that spews out insane amounts of 2.4GHz interference, that somehow knocks out all our customers within about a fifteen-mile radius, and they turn it on every so often. Does that conclusion sound reasonable? And if it does, what the heck can I do about it? Frustrated, David Smith MVN.net -- Jack Unger ([EMAIL PROTECTED]) - President, Ask-Wi.Com, Inc. Serving the License-Free Wireless Industry Since 1993 Author of the WISP Handbook - Deploying License-Free Wireless WANs True Vendor-Neutral WISP Consulting-Training-Troubleshooting Newsletters Downloadable from http://ask-wi.com/newsletters.html Phone (VoIP Over Broadband Wireless) 818-227-4220 www.ask-wi.com -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] The Gremlin, redux
Jack Unger wrote: If it's true that there's a giant something that's spewing noise, you can use a spectrum analyzer and try to identify the noise signature, then triangulate. If it would just stay broken for a couple hours, I'd love to do that. Sadly, this problem usually just shows up for a minute or two at a time, and never more than about fifteen minutes. The boss and I have tried that before, and the problem is just too intermittent for us to be able to narrow down that way. Of course, our spectrum-fu is not that strong. David Smith MVN.net -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] The Gremlin, redux
On 10/27/06, David E. Smith [EMAIL PROTECTED] wrote: Jack Unger wrote: If it's true that there's a giant something that's spewing noise, you can use a spectrum analyzer and try to identify the noise signature, then triangulate. If it would just stay broken for a couple hours, I'd love to do that. Sadly, this problem usually just shows up for a minute or two at a time, and never more than about fifteen minutes. The boss and I have tried that before, and the problem is just too intermittent for us to be able to narrow down that way. Of course, our spectrum-fu is not that strong. David Smith MVN.net David, We have a similar situation happening mainly on one tower of ours. Basicially it is a StarOS V2 on WRAP boards setup using Prism cards for the AP's. We have 4 90* horizontal sectors. Everyones's signals are great and it runs fine most of the time. Occassionaly we see times where people have 10-20% packet loss. We look at the traffic on the tower for abuse and/or virus and don't really find anything. We've tried different channels and it doesn't seem to help. Other times there is no loss at all. Most of our clients on CB3's but we do have some Orinoco based clients. The Orinoco based clients don't seem to have the problem as much as the CB3's do however. I have not really pinned down what the difference between them would be that would cause the Orinoco's not to show this behaviour even though their signal may be somewhat lower. We've taken a spectrum analyzer up the tower and don't really see any other signals that are really hot out there but it feels like an interefernce problem. Unfortunately, the tower is about an hour drive so catching this while it happens has proved somewhat problematic. In anycase, I feel your pain. I'll let you know if we figure out what this issue is. -Eric -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] The Gremlin, redux
This has been our plan all along. It just will not stay screwed up long enough to get a single heading. The signal level of the interference looks like data but it varies in level so much that finding the heading is not easy. I know spectrum analysis and this one has me stumped. I wish it would interfere and stay that way long enough to track it. I honestly think it is a leaky microwave oven because it runs about long enough to nuke a frozen chicken patty and happens at lunch time quite often! :-) Scriv Jack Unger wrote: If it's true that there's a giant something that's spewing noise, you can use a spectrum analyzer and try to identify the noise signature, then triangulate. jack David E. Smith wrote: This problem was mentioned back in May (see http://lists.wispa.org/pipermail/wireless/2006-May/025354.html ) but it's still here, and I thought it might be worthwhile to bounce it off folks again, to see if anyone has any new insights. Occasionally, completely at random, many of our 2.4GHz APs start showing high packet loss (usually 20-30%) and high latency (upwards of ten seconds - not milliseconds, full seconds) for a few minutes at a time. It always clears itself up eventually, usually after a minute or two, but sometimes it won't go away for ten or fifteen minutes. It's especially annoying when it happens in the middle of the day and all those itchy business customers call in at once. Random facts and tidbits: * I've watched our network traffic, using Mikrotik's torch and StarOS' beacon tools, and I don't see anything that looks like a DDOS, either entering or leaving our network. * It doesn't affect all our towers, just most of them. Specifically, our most remote towers (geographically, not in terms of network topology) are safe. * (This is the Lonnie Nunweiler clause) It affects equally towers that are bridged, routed, and hybrid bridged/routed. * I don't think it's a backhaul problem, because towers on several different kinds of backhaul links are affected simultaneously. (We've got a mix of Alvarion and Trango gear for backhauls, and at least one ancient YDI EX-1.) Also, there's no problem talking between tower locations; pings between different APs take 10ms or so, just like they normally do. * It only affects our 2.4GHz customers - folks on 900MHz gear, 5.3GHz, and 5.8GHz connections don't have any issues while things are going sour. Basically, it looks like someone's got a big giant massive something that spews out insane amounts of 2.4GHz interference, that somehow knocks out all our customers within about a fifteen-mile radius, and they turn it on every so often. Does that conclusion sound reasonable? And if it does, what the heck can I do about it? Frustrated, David Smith MVN.net -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] The Gremlin, redux
We look at the traffic on the tower for abuse and/or virus and don't really find anything. Just to be clear, you've checked your AP broadcast levels during the events and not found found them elevated? We found the most crippling network events were not coming into the network from the outside, but were broadcast storms between 2 or more customers (repeated through the APs). They act similar to the symptoms you cited (a few minutes of extremely elevated latency due to the short term load they place over the rf). Rich - Original Message - From: Eric Merkel [EMAIL PROTECTED] To: WISPA General List wireless@wispa.org Sent: Friday, October 27, 2006 1:07 PM Subject: Re: [WISPA] The Gremlin, redux On 10/27/06, David E. Smith [EMAIL PROTECTED] wrote: Jack Unger wrote: If it's true that there's a giant something that's spewing noise, you can use a spectrum analyzer and try to identify the noise signature, then triangulate. If it would just stay broken for a couple hours, I'd love to do that. Sadly, this problem usually just shows up for a minute or two at a time, and never more than about fifteen minutes. The boss and I have tried that before, and the problem is just too intermittent for us to be able to narrow down that way. Of course, our spectrum-fu is not that strong. David Smith MVN.net David, We have a similar situation happening mainly on one tower of ours. Basicially it is a StarOS V2 on WRAP boards setup using Prism cards for the AP's. We have 4 90* horizontal sectors. Everyones's signals are great and it runs fine most of the time. Occassionaly we see times where people have 10-20% packet loss. We look at the traffic on the tower for abuse and/or virus and don't really find anything. We've tried different channels and it doesn't seem to help. Other times there is no loss at all. Most of our clients on CB3's but we do have some Orinoco based clients. The Orinoco based clients don't seem to have the problem as much as the CB3's do however. I have not really pinned down what the difference between them would be that would cause the Orinoco's not to show this behaviour even though their signal may be somewhat lower. We've taken a spectrum analyzer up the tower and don't really see any other signals that are really hot out there but it feels like an interefernce problem. Unfortunately, the tower is about an hour drive so catching this while it happens has proved somewhat problematic. In anycase, I feel your pain. I'll let you know if we figure out what this issue is. -Eric -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] The Gremlin, redux
Maybe there is a microwave lighting system somewhere nearby. You know what to do with an outdoor light that needs to be shut off.:^) Matt Larsen [EMAIL PROTECTED] John Scrivner wrote: This has been our plan all along. It just will not stay screwed up long enough to get a single heading. The signal level of the interference looks like data but it varies in level so much that finding the heading is not easy. I know spectrum analysis and this one has me stumped. I wish it would interfere and stay that way long enough to track it. I honestly think it is a leaky microwave oven because it runs about long enough to nuke a frozen chicken patty and happens at lunch time quite often! :-) Scriv Jack Unger wrote: If it's true that there's a giant something that's spewing noise, you can use a spectrum analyzer and try to identify the noise signature, then triangulate. jack David E. Smith wrote: This problem was mentioned back in May (see http://lists.wispa.org/pipermail/wireless/2006-May/025354.html ) but it's still here, and I thought it might be worthwhile to bounce it off folks again, to see if anyone has any new insights. Occasionally, completely at random, many of our 2.4GHz APs start showing high packet loss (usually 20-30%) and high latency (upwards of ten seconds - not milliseconds, full seconds) for a few minutes at a time. It always clears itself up eventually, usually after a minute or two, but sometimes it won't go away for ten or fifteen minutes. It's especially annoying when it happens in the middle of the day and all those itchy business customers call in at once. Random facts and tidbits: * I've watched our network traffic, using Mikrotik's torch and StarOS' beacon tools, and I don't see anything that looks like a DDOS, either entering or leaving our network. * It doesn't affect all our towers, just most of them. Specifically, our most remote towers (geographically, not in terms of network topology) are safe. * (This is the Lonnie Nunweiler clause) It affects equally towers that are bridged, routed, and hybrid bridged/routed. * I don't think it's a backhaul problem, because towers on several different kinds of backhaul links are affected simultaneously. (We've got a mix of Alvarion and Trango gear for backhauls, and at least one ancient YDI EX-1.) Also, there's no problem talking between tower locations; pings between different APs take 10ms or so, just like they normally do. * It only affects our 2.4GHz customers - folks on 900MHz gear, 5.3GHz, and 5.8GHz connections don't have any issues while things are going sour. Basically, it looks like someone's got a big giant massive something that spews out insane amounts of 2.4GHz interference, that somehow knocks out all our customers within about a fifteen-mile radius, and they turn it on every so often. Does that conclusion sound reasonable? And if it does, what the heck can I do about it? Frustrated, David Smith MVN.net -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] The Gremlin, redux
On 10/27/06, Rich Comroe [EMAIL PROTECTED] wrote: We look at the traffic on the tower for abuse and/or virus and don't really find anything. Just to be clear, you've checked your AP broadcast levels during the events and not found found them elevated? We found the most crippling network events were not coming into the network from the outside, but were broadcast storms between 2 or more customers (repeated through the APs). They act similar to the symptoms you cited (a few minutes of extremely elevated latency due to the short term load they place over the rf). Rich We try to mitigate this problem by the following: 1) Turning off inter-BSS Relay 2) We block all the typical MS ports(135-139) which broadcast all the time via iptables 3) Packet shape all connections via CBQ on the AP itself to limit how much bandwidth any one customer can consume Looking at the beacon realtime manager and tcpdump, we've never seen an unreasonable # of broadcasts when this is happening. -Eric -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] The Gremlin, redux
On Fri, October 27, 2006 3:11 pm, Eric Merkel wrote: 1) Turning off inter-BSS Relay Already done, on most towers. (We do have a couple of towers where one business, with two locations, wants to do VPN-type stuff between 'em.) 2) We block all the typical MS ports(135-139) which broadcast all the time via iptables Done. We block 135-139, 445, and a couple other ports, both TCP and UDP. 3) Packet shape all connections via CBQ on the AP itself to limit how much bandwidth any one customer can consume Mostly done. (For historical reasons, some of our customers are still part of a giant bridged network, and their traffic is shaped in our office not at the AP, but those customers are relatively few and growing fewer by the week.) David Smith MVN.net -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/