Re: [WISPA] The Gremlin, redux

2006-10-30 Thread Marlon K. Schafer (509) 982-2181
Send a call tag over and we'll ship you my SA.  LEAVE it at one of the 
towers.  You'll want a sectored antenna on it.  120* for starters.  When the 
noise hits move to another direction with the antenna.  It'll take a while 
but you'll be able to see what direction it comes from.  Then move it to 
another tower, do the same thing.


After 2 or 3 towers you'll be able to see where it's coming from.

Just use the max hold function.  Hook up a cheap webcam so you can see the 
screen on the analyzer.  Set that up as the active desktop on your puter and 
check it often.  Won't take long to see unusually high peaks.


call me and we'll talk over some ideas.

509.988.0260 cell

Marlon
(509) 982-2181   Equipment sales
(408) 907-6910 (Vonage)Consulting services
42846865 (icq)And I run my own wisp!
64.146.146.12 (net meeting)
www.odessaoffice.com/wireless
www.odessaoffice.com/marlon/cam



- Original Message - 
From: David E. Smith [EMAIL PROTECTED]

To: WISPA General List wireless@wispa.org
Sent: Friday, October 27, 2006 9:32 AM
Subject: Re: [WISPA] The Gremlin, redux



Marlon K. Schafer (509) 982-2181 wrote:


Scriv needs to hire a good consultant to come check things out!
big grin


Know any? :D

The hard part there would be that it's not, in any way, predictable.
We've gone several weeks at a time without this problem appearing, and
had days where it showed up several times. Most of the time, I only see
it in our network monitor logs, as it came and went within a minute or
two. Assuming it's some massive source of interference, I imagine it
would be very difficult to triangulate it. (By the time I call even one
person to say fire up your spectrum analyzer, it's gone.)


First, as I recall, this ONLY effects towers within a 15 mile radius.
But it effects ALL towers within that cell.  right?


Yep. I had a nice map that showed all our towers, and which ones were
(and weren't) affected. I may try to dig that up, or recreate it.


I've seen customers download or upload massive files (usually ptp stuff)
and open up hundreds of connections and whack a tower.  If it's the
right tower and interferes with the other towers anywhere near it.


That's probably not the case, as we've recently taken measures to reduce
P2P traffic (a Mikrotik box and some pretty harsh traffic shaping
rules). Also, I've seen P2P traffic before, and while it usually slows
down that specific AP, I've never seen it affect our backhaul or other
nearby towers.

We're pretty good in terms on not walking on ourselves. There are no
overlaps (AFAIK) between our towers if they're within five miles or so
of one another, and facing towards each other. We are using the whole
2.4GHz spectrum, of course, but not all of it in any one place.


There are times when radios go bad and start transmitting OUT of band.
I've seen wifi stuff flood the whole band.

Amps can do that too.


We do have a couple of amps here and there, but I don't think any of
them are over 200mW. That'd have to be one heckuva runaway amp to cause
that much interference.

If it were a bad radio, wouldn't it be a more frequent problem instead
of one that randomly shows up for a minute or two here and there?


Is it possible that your 900 and 5 gig gear ALWAYS has a router at the
cpe and the 2.4 doesn't?  As I recall the Waverider gear, the ap is a
router, that would keep things that would flow on a switch off of the
900 system.


Not even close. We've got folks with 900MHz gear and 5.whatever that
just plug their radios straight into their PCs.

(At least I know that's the case with the 900MHz stuff, as we have a lot
of residential customers on it; I can't, off the top of my head, think
of any 5.3/5.8 customers without a router, but we don't have that many
of them - we're mostly using those bands for inter-tower backhaul.)


Are there any towers that are sectorized that point in a direction that
makes them unaffected?


I wish... Two of them have two 180-degree sector antennas facing in
opposite directions, on opposite sides of a water tower (i.e. with
twenty feet of steel and water between them), and both sides are
affected equally.

David Smith
MVN.net
--
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/ 


--
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/


Re: [WISPA] The Gremlin, redux

2006-10-30 Thread John Scrivner

We have analyzers already. Thanks for the offer though.
Scriv

Marlon K. Schafer (509) 982-2181 wrote:

Send a call tag over and we'll ship you my SA.  LEAVE it at one of the 
towers.  You'll want a sectored antenna on it.  120* for starters.  
When the noise hits move to another direction with the antenna.  It'll 
take a while but you'll be able to see what direction it comes from.  
Then move it to another tower, do the same thing.


After 2 or 3 towers you'll be able to see where it's coming from.

Just use the max hold function.  Hook up a cheap webcam so you can see 
the screen on the analyzer.  Set that up as the active desktop on your 
puter and check it often.  Won't take long to see unusually high peaks.


call me and we'll talk over some ideas.

509.988.0260 cell

Marlon
(509) 982-2181   Equipment sales
(408) 907-6910 (Vonage)Consulting services
42846865 (icq)And I run my own wisp!
64.146.146.12 (net meeting)
www.odessaoffice.com/wireless
www.odessaoffice.com/marlon/cam



- Original Message - From: David E. Smith [EMAIL PROTECTED]
To: WISPA General List wireless@wispa.org
Sent: Friday, October 27, 2006 9:32 AM
Subject: Re: [WISPA] The Gremlin, redux



Marlon K. Schafer (509) 982-2181 wrote:


Scriv needs to hire a good consultant to come check things out!
big grin



Know any? :D

The hard part there would be that it's not, in any way, predictable.
We've gone several weeks at a time without this problem appearing, and
had days where it showed up several times. Most of the time, I only see
it in our network monitor logs, as it came and went within a minute or
two. Assuming it's some massive source of interference, I imagine it
would be very difficult to triangulate it. (By the time I call even one
person to say fire up your spectrum analyzer, it's gone.)


First, as I recall, this ONLY effects towers within a 15 mile radius.
But it effects ALL towers within that cell.  right?



Yep. I had a nice map that showed all our towers, and which ones were
(and weren't) affected. I may try to dig that up, or recreate it.

I've seen customers download or upload massive files (usually ptp 
stuff)

and open up hundreds of connections and whack a tower.  If it's the
right tower and interferes with the other towers anywhere near it.



That's probably not the case, as we've recently taken measures to reduce
P2P traffic (a Mikrotik box and some pretty harsh traffic shaping
rules). Also, I've seen P2P traffic before, and while it usually slows
down that specific AP, I've never seen it affect our backhaul or other
nearby towers.

We're pretty good in terms on not walking on ourselves. There are no
overlaps (AFAIK) between our towers if they're within five miles or so
of one another, and facing towards each other. We are using the whole
2.4GHz spectrum, of course, but not all of it in any one place.


There are times when radios go bad and start transmitting OUT of band.
I've seen wifi stuff flood the whole band.

Amps can do that too.



We do have a couple of amps here and there, but I don't think any of
them are over 200mW. That'd have to be one heckuva runaway amp to cause
that much interference.

If it were a bad radio, wouldn't it be a more frequent problem instead
of one that randomly shows up for a minute or two here and there?


Is it possible that your 900 and 5 gig gear ALWAYS has a router at the
cpe and the 2.4 doesn't?  As I recall the Waverider gear, the ap is a
router, that would keep things that would flow on a switch off of the
900 system.



Not even close. We've got folks with 900MHz gear and 5.whatever that
just plug their radios straight into their PCs.

(At least I know that's the case with the 900MHz stuff, as we have a lot
of residential customers on it; I can't, off the top of my head, think
of any 5.3/5.8 customers without a router, but we don't have that many
of them - we're mostly using those bands for inter-tower backhaul.)


Are there any towers that are sectorized that point in a direction that
makes them unaffected?



I wish... Two of them have two 180-degree sector antennas facing in
opposite directions, on opposite sides of a water tower (i.e. with
twenty feet of steel and water between them), and both sides are
affected equally.

David Smith
MVN.net
--
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/ 




--
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/


Re: [WISPA] The Gremlin, redux

2006-10-29 Thread Mike Ireton

John Scrivner wrote:

Mac,
We believe this is truly an outside offender in 2.4 GHz. I have 
personally seen a carrier that is several times more power than anything 
I have ever seen. I only saw it for a brief instant though. This 
interference just does not last long enough to be caught. The high 
latency is caused by retransmits but I am sure outside interference is 
what is leading to the need for frames to be sent again. This effects 
all channels across the 2.4 GHz bands. We have seen the noise floor jump 
up higher than our radio power levels when this problem happens. What 
ever is causing this is running higher power than anything we have in 
the field. We will look at anything, though, to help troubleshoot and I 
appreciate your ideas.

Scriv



We've had this too and have never been able to narrow it down. Basiclly, 
certain areas and without warning or obvious cause, simply become dead 
for 2.4ghz in that there appears some very powerful inband interference 
that is not 802.11b/g and has no obvivious source we can find, but the 
area of effect is fairly localized (using affected customers to 
tirangulate).


We also have a problem within our hometown of repeatedly experiencing 
burnt out 2.4ghz equipment. Never happens anywhere else in our county 
wide footprint, just our hometown and across a wide variety of equipment 
such as smartbridges, atheros, prism 2.5 cards, cb3's, cisco aironet, 
you name it. Been looking at this problem for years and simply don't 
understand it. We've had events like this occuring down one particular 
street for example and an accesspoint not too far away, all in the space 
of one evening. We've also had events like one subscriber per day (on 
2.4ghz) winding up with burnt out prism card, requiring a truck roll and 
card replacement.


Mike-

--
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/


RE: [WISPA] The Gremlin, redux

2006-10-28 Thread Mac Dearman
David,

I tend to believe you will find your answer on your network -vs- big bad
leak somewhere and the only real suggestion I can offer you would be to do
what we do here when we start having weird issues - - and that is to be
ready to start unplugging Back Hauls (one at a time) out of their
switch..Etc when the trouble starts. Have a ping running to watch for the
latency to disappear when you unplug the offending Back Haul. Once you
narrow it down to the right leg of the network you can reboot one AP at a
time until you find which AP it is coming from by running an extended ping
on that leg of the network

 Don't pull a big dummy like we did here for weeks!! Unplug the servers
one at a time too!!! That is where we found one of the servers we host was
our culprit. There traffic was of such that it was flooding our switches - -
(dirty suckers) :-)

GL -  keep us informed as to what you find

Mac Dearman


-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On
Behalf Of David E. Smith
Sent: Friday, October 27, 2006 4:10 PM
To: WISPA General List
Subject: Re: [WISPA] The Gremlin, redux

On Fri, October 27, 2006 3:11 pm, Eric Merkel wrote:

 1) Turning off inter-BSS Relay

Already done, on most towers. (We do have a couple of towers where one
business, with two locations, wants to do VPN-type stuff between 'em.)

 2) We block all the typical MS ports(135-139) which broadcast all the
 time via iptables

Done. We block 135-139, 445, and a couple other ports, both TCP and UDP.

 3) Packet shape all connections via CBQ on the AP itself to limit how
 much bandwidth any one customer can consume

Mostly done. (For historical reasons, some of our customers are still part
of a giant bridged network, and their traffic is shaped in our office not
at the AP, but those customers are relatively few and growing fewer by the
week.)

David Smith
MVN.net
-- 
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/

-- 
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/


Re: [WISPA] The Gremlin, redux

2006-10-28 Thread David E. Smith

Mac Dearman wrote:


I tend to believe you will find your answer on your network -vs- big bad
leak somewhere and the only real suggestion I can offer you would be to do
what we do here when we start having weird issues


[ snip: Mac's good advice on how to track down broadcast storms and 
other network-related problems ]


If it were a backhaul problem, though, wouldn't I see high latency on my 
backhaul links? I haven't, to date.


Here's one of the (many) experiments I've done in the past (this one was 
yesterday, actually): I logged into an affected AP, while the problem 
was happening, and firewalled off THE WHOLE INTERNET except for one IP 
address, that being the PC in my office I was using to log into that AP. 
Then, from that AP, I pinged a CPE associated with it. The problem 
persisted, high packet loss and 5000-1 millisecond latency.


I quickly undid my changes on that AP, then did exactly the same thing 
on another AP, about twenty miles away, with a different radio card, on 
a different channel, connected to me through a different backhaul link, 
and saw exactly the same performance (high packet loss, obscene latency, 
et cetera).


Meanwhile, pings to and through our backhaul links, to the APs 
themselves, various managed switches at tower locations, and so on, 
never skipped a beat. (Heck, most of 'em improved a bit, since there 
wasn't as much pesky customer traffic going through them. :)


For that matter, why do I have three towers with a mixture of 2.4GHz and 
other APs (two with a 900MHz AP, one with a 5.3GHz AP and a 5.8GHz AP) 
and the non-2.4 customers aren't affected? Keep in mind that, for 
annoying historical reasons, much of our network is still flat, 
bridged addresses flying willy-nilly across four counties. If it were a 
network storm, I'd expect it to hit all our towers, on all channels, and 
not conveniently skip over the most geographically remote ones.


There's also all the nasty logistical problems of my company not having 
twenty extra hands that we can just have sitting at all our towers for 
days, or weeks, at a time, waiting for a problem that shows up 
completely randomly, hoping I can call everyone to start unplugging 
stuff before the problem magically goes away, but that's another issue 
altogether ;)


Last, remember this really is amazingly random, and usually only shows 
up for a minute or two at a time, brief enough that I only see it in our 
logs well after the fact. (Yesterday it visited us for a good twenty 
minutes, long enough for customers to really notice, and for me to have 
time to dig into it again.)


I certainly don't mean to sound dismissive of any suggestions, it's just 
that I've tried most of the obvious ones before.


David Smith
MVN.net
--
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/


Re: [WISPA] The Gremlin, redux

2006-10-28 Thread Rich Comroe

Eric wrote:

Looking at the beacon realtime manager and tcpdump, we've never seen
an unreasonable # of broadcasts when this is happening.


On our network 200 broadcasts per second is pretty typical.  When we see it 
spike to over 1200 per second (6-fold rise), it really drives the latency 
up.  Don't be fooled by total bw metrics ... broadcasts are so short that 
crippling broadcast storms don't show any spike in total traffic (traffic is 
actually squeezed out by the broadcasts dominating the radios).  I'm 
thinking tcpdump is like the packet dump that ethereal uses.  How do you 
determine how many broadcasts per second is transiting the AP from it?  Just 
curious.  I'm not familiar with beacon realtime manager ... can it tell 
you how many broadcasts per second are on the air during your events?  When 
you say you've never seen an unreasonable # of broadcasts, how many 
broadcasts per second do you see during an event?  What we do is simply 
chart broadcasts in  out for every radio on our network on a continuous 
basis.  Just about anything will do that, so knowing broadcast levels (and 
who's causing them) is a piece of cake ... no harder than opening a web page 
and scrolling to spot the offenders.


Rich

- Original Message - 
From: Eric Merkel [EMAIL PROTECTED]

To: WISPA General List wireless@wispa.org
Sent: Friday, October 27, 2006 3:11 PM
Subject: Re: [WISPA] The Gremlin, redux



On 10/27/06, Rich Comroe [EMAIL PROTECTED] wrote:

We look at the traffic on the
tower for abuse and/or virus and don't really find anything.

Just to be clear, you've checked your AP broadcast levels during the 
events

and not found found them elevated?  We found the most crippling network
events were not coming into the network from the outside, but were 
broadcast

storms between 2 or more customers (repeated through the APs).  They act
similar to the symptoms you cited (a few minutes of extremely elevated
latency due to the short term load they place over the rf).

Rich



We try to mitigate this problem by the following:

1) Turning off inter-BSS Relay
2) We block all the typical MS ports(135-139) which broadcast all the
time via iptables
3) Packet shape all connections via CBQ on the AP itself to limit how
much bandwidth any one customer can consume

Looking at the beacon realtime manager and tcpdump, we've never seen
an unreasonable # of broadcasts when this is happening.

-Eric
--
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/ 


--
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/


Re: [WISPA] The Gremlin, redux

2006-10-28 Thread Lonnie Nunweiler

Since your customers are mostly on a bridged network then a single
malicious or misbehaving customer radio can take your WHOLE network
down.  The reason your backhaul is not affected is simply because it
has more capacity than the prism AP units.

I know you guys are tired of hearing from me about routing versus
bridging (You even gave me my own clause) but your problems are
precisely what I have been describing for all of these years.

The one point you are missing in your overall design is something I
have brought up a number of times.  A properly functioning bridged
system will echo all broadcast traffic throughout the network, thus
one system can flood the entire LAN.  This leads to the biggest single
factor -- your entire network speed is a factor of your slowest AP,
since it must rebroadcast all traffic to its users.

On a routed design you will still get problems like these slowdowns,
but the very important thing to keep in mind is that the slowdown will
only take out the one sector.  Also (and this should be enough to make
you take a hard look at routing) your LAN throughput is the sum total
of all of your AP units and thus you might actually start to use your
high speed backhaul.  Your ping times will go under 2 msec per
repeater.  Our WAR boards are actually around the 1 msec area.

The other thing you have going against you is that you are using a
unit based on the Ubicom chipset.  Remember the dear old Wet11?  That
was Ubicom using a prism radio.  CB3 is simply a newer version, but
still with some issues that seem to be killing networks everywhere.

Our last version of the Atheros driver would start to show units could
not reconnect and the only way to make the system settle down was to
change channels or reboot.  What is happening is that the CB3 has
trouble when it reconnects to an AP.  Something is not quite right
with its handshake and it does not get accepted.  This causes it to
throw a hissy fit and it starts to hammer the AP with reconnection
attempts.  That massive RF directed at the AP soon causes other units
to disconnect and guess what happens if they are a CB3?  They try the
same techniques the first tried.  Suddenly you have this massive
reconnect storm and the AP is brought to its knees until you change
channels or reboot.

Your whole problem is circular and is based on poor choices of both
equipment and network design.  You chose cheap units that only do a
pseudo bridge so you designed your LAN accordingly.  You thought you
would be safe because you had a good AP, but a LAN is mostly made up
of clients, so if the clients are poor, then it follows that your LAN
is poor.

You have some sectors already routed, but they still have the trouble.
I say this is because of the trouble of the clients, which could be
at this tower or simply because your tower is visible to them.  You
are quite likely even using 200 mW prism cards which makes this factor
even more probable.

So this means you have two issues.  1.  Clients that will have to be
updated with better firmware or replaced.   2.  Your entire system
will have to be redesigned for a proper routed design.

The way out of this is obvious, but it will be painful and costly.
There is going to be a lot of anger directed at me for this message so
I am now ducking down to avoid the flames.

Regards,
Lonnie


On 10/28/06, David E. Smith [EMAIL PROTECTED] wrote:

Mac Dearman wrote:

 I tend to believe you will find your answer on your network -vs- big bad
 leak somewhere and the only real suggestion I can offer you would be to do
 what we do here when we start having weird issues

[ snip: Mac's good advice on how to track down broadcast storms and
other network-related problems ]

If it were a backhaul problem, though, wouldn't I see high latency on my
backhaul links? I haven't, to date.

Here's one of the (many) experiments I've done in the past (this one was
yesterday, actually): I logged into an affected AP, while the problem
was happening, and firewalled off THE WHOLE INTERNET except for one IP
address, that being the PC in my office I was using to log into that AP.
Then, from that AP, I pinged a CPE associated with it. The problem
persisted, high packet loss and 5000-1 millisecond latency.

I quickly undid my changes on that AP, then did exactly the same thing
on another AP, about twenty miles away, with a different radio card, on
a different channel, connected to me through a different backhaul link,
and saw exactly the same performance (high packet loss, obscene latency,
et cetera).

Meanwhile, pings to and through our backhaul links, to the APs
themselves, various managed switches at tower locations, and so on,
never skipped a beat. (Heck, most of 'em improved a bit, since there
wasn't as much pesky customer traffic going through them. :)

For that matter, why do I have three towers with a mixture of 2.4GHz and
other APs (two with a 900MHz AP, one with a 5.3GHz AP and a 5.8GHz AP)
and the non-2.4 customers aren't affected? Keep 

Re: [WISPA] The Gremlin, redux

2006-10-28 Thread John Scrivner

Mac,
We believe this is truly an outside offender in 2.4 GHz. I have 
personally seen a carrier that is several times more power than anything 
I have ever seen. I only saw it for a brief instant though. This 
interference just does not last long enough to be caught. The high 
latency is caused by retransmits but I am sure outside interference is 
what is leading to the need for frames to be sent again. This effects 
all channels across the 2.4 GHz bands. We have seen the noise floor jump 
up higher than our radio power levels when this problem happens. What 
ever is causing this is running higher power than anything we have in 
the field. We will look at anything, though, to help troubleshoot and I 
appreciate your ideas.

Scriv


Mac Dearman wrote:


David,

I tend to believe you will find your answer on your network -vs- big bad
leak somewhere and the only real suggestion I can offer you would be to do
what we do here when we start having weird issues - - and that is to be
ready to start unplugging Back Hauls (one at a time) out of their
switch..Etc when the trouble starts. Have a ping running to watch for the
latency to disappear when you unplug the offending Back Haul. Once you
narrow it down to the right leg of the network you can reboot one AP at a
time until you find which AP it is coming from by running an extended ping
on that leg of the network

Don't pull a big dummy like we did here for weeks!! Unplug the servers
one at a time too!!! That is where we found one of the servers we host was
our culprit. There traffic was of such that it was flooding our switches - -
(dirty suckers) :-)

GL -  keep us informed as to what you find

Mac Dearman


-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On
Behalf Of David E. Smith
Sent: Friday, October 27, 2006 4:10 PM
To: WISPA General List
Subject: Re: [WISPA] The Gremlin, redux

On Fri, October 27, 2006 3:11 pm, Eric Merkel wrote:

 


1) Turning off inter-BSS Relay
   



Already done, on most towers. (We do have a couple of towers where one
business, with two locations, wants to do VPN-type stuff between 'em.)

 


2) We block all the typical MS ports(135-139) which broadcast all the
time via iptables
   



Done. We block 135-139, 445, and a couple other ports, both TCP and UDP.

 


3) Packet shape all connections via CBQ on the AP itself to limit how
much bandwidth any one customer can consume
   



Mostly done. (For historical reasons, some of our customers are still part
of a giant bridged network, and their traffic is shaped in our office not
at the AP, but those customers are relatively few and growing fewer by the
week.)

David Smith
MVN.net
 


--
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/


RE: SPAM ? Re: [WISPA] The Gremlin, redux

2006-10-28 Thread Mac Dearman
VLAN's and Trunking stop all the broadcast traffic. You won't hear me say
NOT to static route (yuck) as I have a lot of my network statically routed,
but it does NOT out perform my bridged, VLAN'd network segments.

 I did have a completely flat bridged network and never had my network
flooded by a single user with a virus. I am not saying that its not
possible, but it never happened to me once in 4 years (and I have 15% of the
State of Louisiana covered with wireless) before changing half of my network
to a statically routed network. I did have a switch to get flooded in the
NOC that actually shut my network down many times before I caught that
sucker.

I realize that a VLAN'd network can still have trouble, but no more than
your statically routed network. There are more ways to skin a cat than one
:-) although we know your preferences :-)


Mac 


-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On
Behalf Of Lonnie Nunweiler
Sent: Saturday, October 28, 2006 4:18 PM
To: WISPA General List
Subject: SPAM ? Re: [WISPA] The Gremlin, redux
Importance: Low

Since your customers are mostly on a bridged network then a single
malicious or misbehaving customer radio can take your WHOLE network
down.  The reason your backhaul is not affected is simply because it
has more capacity than the prism AP units.

I know you guys are tired of hearing from me about routing versus
bridging (You even gave me my own clause) but your problems are
precisely what I have been describing for all of these years.

The one point you are missing in your overall design is something I
have brought up a number of times.  A properly functioning bridged
system will echo all broadcast traffic throughout the network, thus
one system can flood the entire LAN.  This leads to the biggest single
factor -- your entire network speed is a factor of your slowest AP,
since it must rebroadcast all traffic to its users.

On a routed design you will still get problems like these slowdowns,
but the very important thing to keep in mind is that the slowdown will
only take out the one sector.  Also (and this should be enough to make
you take a hard look at routing) your LAN throughput is the sum total
of all of your AP units and thus you might actually start to use your
high speed backhaul.  Your ping times will go under 2 msec per
repeater.  Our WAR boards are actually around the 1 msec area.

The other thing you have going against you is that you are using a
unit based on the Ubicom chipset.  Remember the dear old Wet11?  That
was Ubicom using a prism radio.  CB3 is simply a newer version, but
still with some issues that seem to be killing networks everywhere.

Our last version of the Atheros driver would start to show units could
not reconnect and the only way to make the system settle down was to
change channels or reboot.  What is happening is that the CB3 has
trouble when it reconnects to an AP.  Something is not quite right
with its handshake and it does not get accepted.  This causes it to
throw a hissy fit and it starts to hammer the AP with reconnection
attempts.  That massive RF directed at the AP soon causes other units
to disconnect and guess what happens if they are a CB3?  They try the
same techniques the first tried.  Suddenly you have this massive
reconnect storm and the AP is brought to its knees until you change
channels or reboot.

Your whole problem is circular and is based on poor choices of both
equipment and network design.  You chose cheap units that only do a
pseudo bridge so you designed your LAN accordingly.  You thought you
would be safe because you had a good AP, but a LAN is mostly made up
of clients, so if the clients are poor, then it follows that your LAN
is poor.

You have some sectors already routed, but they still have the trouble.
 I say this is because of the trouble of the clients, which could be
at this tower or simply because your tower is visible to them.  You
are quite likely even using 200 mW prism cards which makes this factor
even more probable.

So this means you have two issues.  1.  Clients that will have to be
updated with better firmware or replaced.   2.  Your entire system
will have to be redesigned for a proper routed design.

The way out of this is obvious, but it will be painful and costly.
There is going to be a lot of anger directed at me for this message so
I am now ducking down to avoid the flames.

Regards,
Lonnie


On 10/28/06, David E. Smith [EMAIL PROTECTED] wrote:
 Mac Dearman wrote:

  I tend to believe you will find your answer on your network -vs- big
bad
  leak somewhere and the only real suggestion I can offer you would be to
do
  what we do here when we start having weird issues

 [ snip: Mac's good advice on how to track down broadcast storms and
 other network-related problems ]

 If it were a backhaul problem, though, wouldn't I see high latency on my
 backhaul links? I haven't, to date.

 Here's one of the (many) experiments I've done in the past (this one

RE: [WISPA] The Gremlin, redux

2006-10-28 Thread Mac Dearman
John,

 Can you set up a CPE to monitor noise floor levels (MRTG..etc) and point
them at different towers owned by different companies that may be suspect in
your area? If you can identify this - - I have another idea to prove it or
release a suspect tower out of suspicion, but that will require a personal
phone call :-)

Mac Dearman

-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On
Behalf Of John Scrivner
Sent: Saturday, October 28, 2006 4:22 PM
To: WISPA General List
Subject: Re: [WISPA] The Gremlin, redux

Mac,
We believe this is truly an outside offender in 2.4 GHz. I have 
personally seen a carrier that is several times more power than anything 
I have ever seen. I only saw it for a brief instant though. This 
interference just does not last long enough to be caught. The high 
latency is caused by retransmits but I am sure outside interference is 
what is leading to the need for frames to be sent again. This effects 
all channels across the 2.4 GHz bands. We have seen the noise floor jump 
up higher than our radio power levels when this problem happens. What 
ever is causing this is running higher power than anything we have in 
the field. We will look at anything, though, to help troubleshoot and I 
appreciate your ideas.
Scriv


Mac Dearman wrote:

David,

I tend to believe you will find your answer on your network -vs- big bad
leak somewhere and the only real suggestion I can offer you would be to do
what we do here when we start having weird issues - - and that is to be
ready to start unplugging Back Hauls (one at a time) out of their
switch..Etc when the trouble starts. Have a ping running to watch for the
latency to disappear when you unplug the offending Back Haul. Once you
narrow it down to the right leg of the network you can reboot one AP at a
time until you find which AP it is coming from by running an extended ping
on that leg of the network

 Don't pull a big dummy like we did here for weeks!! Unplug the servers
one at a time too!!! That is where we found one of the servers we host was
our culprit. There traffic was of such that it was flooding our switches -
-
(dirty suckers) :-)

GL -  keep us informed as to what you find

Mac Dearman


-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On
Behalf Of David E. Smith
Sent: Friday, October 27, 2006 4:10 PM
To: WISPA General List
Subject: Re: [WISPA] The Gremlin, redux

On Fri, October 27, 2006 3:11 pm, Eric Merkel wrote:

  

1) Turning off inter-BSS Relay



Already done, on most towers. (We do have a couple of towers where one
business, with two locations, wants to do VPN-type stuff between 'em.)

  

2) We block all the typical MS ports(135-139) which broadcast all the
time via iptables



Done. We block 135-139, 445, and a couple other ports, both TCP and UDP.

  

3) Packet shape all connections via CBQ on the AP itself to limit how
much bandwidth any one customer can consume



Mostly done. (For historical reasons, some of our customers are still part
of a giant bridged network, and their traffic is shaped in our office not
at the AP, but those customers are relatively few and growing fewer by the
week.)

David Smith
MVN.net
  

-- 
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/

-- 
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/


Re: [WISPA] The Gremlin, redux

2006-10-28 Thread Travis Johnson

Hi,

You may have covered this before, but I probably missed it in your 
original post. What equipment are you running for each of the different 
frequencies (2.4ghz, 900mhz, 5.8ghz, etc.).


Is it possible the 2.4ghz AP's are just not able to handle the high 
packet loads (broadcast storms, viruses, whatever) that the others are?


Also, I have never seen high noise on a band actually cause high 
latency... usually it causes packet loss. My guess is you have an 
internal problem, not external.


Travis
Microserv

David E. Smith wrote:


Mac Dearman wrote:

I tend to believe you will find your answer on your network -vs- big 
bad
leak somewhere and the only real suggestion I can offer you would be 
to do

what we do here when we start having weird issues



[ snip: Mac's good advice on how to track down broadcast storms and 
other network-related problems ]


If it were a backhaul problem, though, wouldn't I see high latency on 
my backhaul links? I haven't, to date.


Here's one of the (many) experiments I've done in the past (this one 
was yesterday, actually): I logged into an affected AP, while the 
problem was happening, and firewalled off THE WHOLE INTERNET except 
for one IP address, that being the PC in my office I was using to log 
into that AP. Then, from that AP, I pinged a CPE associated with it. 
The problem persisted, high packet loss and 5000-1 millisecond 
latency.


I quickly undid my changes on that AP, then did exactly the same thing 
on another AP, about twenty miles away, with a different radio card, 
on a different channel, connected to me through a different backhaul 
link, and saw exactly the same performance (high packet loss, obscene 
latency, et cetera).


Meanwhile, pings to and through our backhaul links, to the APs 
themselves, various managed switches at tower locations, and so on, 
never skipped a beat. (Heck, most of 'em improved a bit, since there 
wasn't as much pesky customer traffic going through them. :)


For that matter, why do I have three towers with a mixture of 2.4GHz 
and other APs (two with a 900MHz AP, one with a 5.3GHz AP and a 
5.8GHz AP) and the non-2.4 customers aren't affected? Keep in mind 
that, for annoying historical reasons, much of our network is still 
flat, bridged addresses flying willy-nilly across four counties. If 
it were a network storm, I'd expect it to hit all our towers, on all 
channels, and not conveniently skip over the most geographically 
remote ones.


There's also all the nasty logistical problems of my company not 
having twenty extra hands that we can just have sitting at all our 
towers for days, or weeks, at a time, waiting for a problem that shows 
up completely randomly, hoping I can call everyone to start unplugging 
stuff before the problem magically goes away, but that's another issue 
altogether ;)


Last, remember this really is amazingly random, and usually only shows 
up for a minute or two at a time, brief enough that I only see it in 
our logs well after the fact. (Yesterday it visited us for a good 
twenty minutes, long enough for customers to really notice, and for me 
to have time to dig into it again.)


I certainly don't mean to sound dismissive of any suggestions, it's 
just that I've tried most of the obvious ones before.


David Smith
MVN.net


--
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/


Re: [WISPA] The Gremlin, redux

2006-10-27 Thread Marlon K. Schafer (509) 982-2181

Scriv needs to hire a good consultant to come check things out!

big grin

Anyway, I've seen things like this before.  Not this bad, but close enough.

There have been a few causes.

First, as I recall, this ONLY effects towers within a 15 mile radius.  But 
it effects ALL towers within that cell.  right?


Assuming I remember that part correctly.  I've seen CPE lock it's self into 
a tx mode.  It'll still work, but will drive the system nuts.


I've seen customers download or upload massive files (usually ptp stuff) and 
open up hundreds of connections and whack a tower.  If it's the right tower 
and interferes with the other towers anywhere near it.


I've been knocked offline by towers over 30 miles away!  OK, that was a ptp 
wmux system with 8' dishes and 30dB of tx power, but still, it happened. 
With the new gear I'm using these days I can detect ap's 30+ miles away.  If 
conditions are JUST right, that could add quite a bit of noise locally.


There are times when radios go bad and start transmitting OUT of band.  I've 
seen wifi stuff flood the whole band.


Amps can do that too.

USUALLY it's the ap's that take it in the shorts when something like this 
happens.  After all, they are all up above the trees etc.


I've also seen computers take out entire networks when they get infected. 
If you don't have client to client blocking enabled on your ap's they could 
be massively overloaded with traffic that you'll not be able to see with a 
scan tool at the router, it could be traffic that's not even ip traffic! 
Remember, most of the gear we use with pass netbeui, ipx, appletalk etc.  I 
think the fact that towers that have routers on them between the OTHER ap's 
and the backhaul should prevent this.


Is it possible that your 900 and 5 gig gear ALWAYS has a router at the cpe 
and the 2.4 doesn't?  As I recall the Waverider gear, the ap is a router, 
that would keep things that would flow on a switch off of the 900 system.


Are there any towers that are sectorized that point in a direction that 
makes them unaffected?


Marlon
(509) 982-2181   Equipment sales
(408) 907-6910 (Vonage)Consulting services
42846865 (icq)And I run my own wisp!
64.146.146.12 (net meeting)
www.odessaoffice.com/wireless
www.odessaoffice.com/marlon/cam



- Original Message - 
From: David E. Smith [EMAIL PROTECTED]

To: wireless@wispa.org
Sent: Friday, October 27, 2006 9:39 AM
Subject: [WISPA] The Gremlin, redux



This problem was mentioned back in May (see
http://lists.wispa.org/pipermail/wireless/2006-May/025354.html ) but
it's still here, and I thought it might be worthwhile to bounce it off
folks again, to see if anyone has any new insights.

Occasionally, completely at random, many of our 2.4GHz APs start showing
high packet loss (usually 20-30%) and high latency (upwards of ten
seconds - not milliseconds, full seconds) for a few minutes at a time.

It always clears itself up eventually, usually after a minute or two,
but sometimes it won't go away for ten or fifteen minutes. It's
especially annoying when it happens in the middle of the day and all
those itchy business customers call in at once.

Random facts and tidbits:

* I've watched our network traffic, using Mikrotik's torch and StarOS'
beacon tools, and I don't see anything that looks like a DDOS, either
entering or leaving our network.
* It doesn't affect all our towers, just most of them. Specifically, our
most remote towers (geographically, not in terms of network topology)
are safe.
* (This is the Lonnie Nunweiler clause) It affects equally towers that
are bridged, routed, and hybrid bridged/routed.
* I don't think it's a backhaul problem, because towers on several
different kinds of backhaul links are affected simultaneously. (We've
got a mix of Alvarion and Trango gear for backhauls, and at least one
ancient YDI EX-1.) Also, there's no problem talking between tower
locations; pings between different APs take 10ms or so, just like they
normally do.
* It only affects our 2.4GHz customers - folks on 900MHz gear, 5.3GHz,
and 5.8GHz connections don't have any issues while things are going sour.

Basically, it looks like someone's got a big giant massive something
that spews out insane amounts of 2.4GHz interference, that somehow
knocks out all our customers within about a fifteen-mile radius, and
they turn it on every so often.

Does that conclusion sound reasonable? And if it does, what the heck can
I do about it?

Frustrated,

David Smith
MVN.net
--
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/



--
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/


Re: [WISPA] The Gremlin, redux

2006-10-27 Thread Jack Unger
If it's true that there's a giant something that's spewing noise, you 
can use a spectrum analyzer and try to identify the noise signature, 
then triangulate.

jack


David E. Smith wrote:


This problem was mentioned back in May (see
http://lists.wispa.org/pipermail/wireless/2006-May/025354.html ) but
it's still here, and I thought it might be worthwhile to bounce it off
folks again, to see if anyone has any new insights.

Occasionally, completely at random, many of our 2.4GHz APs start showing
high packet loss (usually 20-30%) and high latency (upwards of ten
seconds - not milliseconds, full seconds) for a few minutes at a time.

It always clears itself up eventually, usually after a minute or two,
but sometimes it won't go away for ten or fifteen minutes. It's
especially annoying when it happens in the middle of the day and all
those itchy business customers call in at once.

Random facts and tidbits:

* I've watched our network traffic, using Mikrotik's torch and StarOS'
beacon tools, and I don't see anything that looks like a DDOS, either
entering or leaving our network.
* It doesn't affect all our towers, just most of them. Specifically, our
most remote towers (geographically, not in terms of network topology)
are safe.
* (This is the Lonnie Nunweiler clause) It affects equally towers that
are bridged, routed, and hybrid bridged/routed.
* I don't think it's a backhaul problem, because towers on several
different kinds of backhaul links are affected simultaneously. (We've
got a mix of Alvarion and Trango gear for backhauls, and at least one
ancient YDI EX-1.) Also, there's no problem talking between tower
locations; pings between different APs take 10ms or so, just like they
normally do.
* It only affects our 2.4GHz customers - folks on 900MHz gear, 5.3GHz,
and 5.8GHz connections don't have any issues while things are going sour.

Basically, it looks like someone's got a big giant massive something
that spews out insane amounts of 2.4GHz interference, that somehow
knocks out all our customers within about a fifteen-mile radius, and
they turn it on every so often.

Does that conclusion sound reasonable? And if it does, what the heck can
I do about it?

Frustrated,

David Smith
MVN.net


--
Jack Unger ([EMAIL PROTECTED]) - President, Ask-Wi.Com, Inc.
Serving the License-Free Wireless Industry Since 1993
Author of the WISP Handbook - Deploying License-Free Wireless WANs
True Vendor-Neutral WISP Consulting-Training-Troubleshooting
Newsletters Downloadable from http://ask-wi.com/newsletters.html
Phone (VoIP Over Broadband Wireless) 818-227-4220  www.ask-wi.com



--
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/


Re: [WISPA] The Gremlin, redux

2006-10-27 Thread David E. Smith
Jack Unger wrote:
 If it's true that there's a giant something that's spewing noise, you
 can use a spectrum analyzer and try to identify the noise signature,
 then triangulate.

If it would just stay broken for a couple hours, I'd love to do that.

Sadly, this problem usually just shows up for a minute or two at a time,
and never more than about fifteen minutes.

The boss and I have tried that before, and the problem is just too
intermittent for us to be able to narrow down that way. Of course, our
spectrum-fu is not that strong.

David Smith
MVN.net
-- 
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/


Re: [WISPA] The Gremlin, redux

2006-10-27 Thread Eric Merkel

On 10/27/06, David E. Smith [EMAIL PROTECTED] wrote:

Jack Unger wrote:
 If it's true that there's a giant something that's spewing noise, you
 can use a spectrum analyzer and try to identify the noise signature,
 then triangulate.

If it would just stay broken for a couple hours, I'd love to do that.

Sadly, this problem usually just shows up for a minute or two at a time,
and never more than about fifteen minutes.

The boss and I have tried that before, and the problem is just too
intermittent for us to be able to narrow down that way. Of course, our
spectrum-fu is not that strong.

David Smith
MVN.net



David,

We have a similar situation happening mainly on one tower of ours.
Basicially it is a StarOS V2 on WRAP boards setup using Prism cards
for the AP's. We have 4 90* horizontal sectors. Everyones's signals
are great and it runs fine most of the time. Occassionaly we see times
where people have 10-20% packet loss. We look at the traffic on the
tower for abuse and/or virus and don't really find anything. We've
tried different channels and it doesn't seem to help. Other times
there is no loss at all.

Most of our clients on CB3's but we do have some Orinoco based
clients. The Orinoco based clients don't seem to have the problem as
much as the CB3's do however. I have not really pinned down what the
difference between them would be that would cause the Orinoco's not to
show this behaviour even though their signal may be somewhat lower.

We've taken a spectrum analyzer up the tower and don't really see any
other signals that are really hot out there but it feels like an
interefernce problem. Unfortunately, the tower is about an hour drive
so catching this while it happens has proved somewhat problematic.

In anycase, I feel your pain. I'll let you know if we figure out
what this issue is.

-Eric
--
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/


Re: [WISPA] The Gremlin, redux

2006-10-27 Thread John Scrivner
This has been our plan all along. It just will not stay screwed up long 
enough to get a single heading. The signal level of the interference 
looks like data but it varies in level so much that finding the heading 
is not easy. I know spectrum analysis and this one has me stumped. I 
wish it would interfere and stay that way long enough to track it. I 
honestly think it is a leaky microwave oven because it runs about long 
enough to nuke a frozen chicken patty and happens at lunch time quite often!

:-)
Scriv


Jack Unger wrote:

If it's true that there's a giant something that's spewing noise, 
you can use a spectrum analyzer and try to identify the noise 
signature, then triangulate.

jack


David E. Smith wrote:


This problem was mentioned back in May (see
http://lists.wispa.org/pipermail/wireless/2006-May/025354.html ) but
it's still here, and I thought it might be worthwhile to bounce it off
folks again, to see if anyone has any new insights.

Occasionally, completely at random, many of our 2.4GHz APs start showing
high packet loss (usually 20-30%) and high latency (upwards of ten
seconds - not milliseconds, full seconds) for a few minutes at a time.

It always clears itself up eventually, usually after a minute or two,
but sometimes it won't go away for ten or fifteen minutes. It's
especially annoying when it happens in the middle of the day and all
those itchy business customers call in at once.

Random facts and tidbits:

* I've watched our network traffic, using Mikrotik's torch and StarOS'
beacon tools, and I don't see anything that looks like a DDOS, either
entering or leaving our network.
* It doesn't affect all our towers, just most of them. Specifically, our
most remote towers (geographically, not in terms of network topology)
are safe.
* (This is the Lonnie Nunweiler clause) It affects equally towers that
are bridged, routed, and hybrid bridged/routed.
* I don't think it's a backhaul problem, because towers on several
different kinds of backhaul links are affected simultaneously. (We've
got a mix of Alvarion and Trango gear for backhauls, and at least one
ancient YDI EX-1.) Also, there's no problem talking between tower
locations; pings between different APs take 10ms or so, just like they
normally do.
* It only affects our 2.4GHz customers - folks on 900MHz gear, 5.3GHz,
and 5.8GHz connections don't have any issues while things are going 
sour.


Basically, it looks like someone's got a big giant massive something
that spews out insane amounts of 2.4GHz interference, that somehow
knocks out all our customers within about a fifteen-mile radius, and
they turn it on every so often.

Does that conclusion sound reasonable? And if it does, what the heck can
I do about it?

Frustrated,

David Smith
MVN.net




--
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/


Re: [WISPA] The Gremlin, redux

2006-10-27 Thread Rich Comroe

We look at the traffic on the
tower for abuse and/or virus and don't really find anything.


Just to be clear, you've checked your AP broadcast levels during the events 
and not found found them elevated?  We found the most crippling network 
events were not coming into the network from the outside, but were broadcast 
storms between 2 or more customers (repeated through the APs).  They act 
similar to the symptoms you cited (a few minutes of extremely elevated 
latency due to the short term load they place over the rf).


Rich

- Original Message - 
From: Eric Merkel [EMAIL PROTECTED]

To: WISPA General List wireless@wispa.org
Sent: Friday, October 27, 2006 1:07 PM
Subject: Re: [WISPA] The Gremlin, redux



On 10/27/06, David E. Smith [EMAIL PROTECTED] wrote:

Jack Unger wrote:
 If it's true that there's a giant something that's spewing noise, you
 can use a spectrum analyzer and try to identify the noise signature,
 then triangulate.

If it would just stay broken for a couple hours, I'd love to do that.

Sadly, this problem usually just shows up for a minute or two at a time,
and never more than about fifteen minutes.

The boss and I have tried that before, and the problem is just too
intermittent for us to be able to narrow down that way. Of course, our
spectrum-fu is not that strong.

David Smith
MVN.net



David,

We have a similar situation happening mainly on one tower of ours.
Basicially it is a StarOS V2 on WRAP boards setup using Prism cards
for the AP's. We have 4 90* horizontal sectors. Everyones's signals
are great and it runs fine most of the time. Occassionaly we see times
where people have 10-20% packet loss. We look at the traffic on the
tower for abuse and/or virus and don't really find anything. We've
tried different channels and it doesn't seem to help. Other times
there is no loss at all.

Most of our clients on CB3's but we do have some Orinoco based
clients. The Orinoco based clients don't seem to have the problem as
much as the CB3's do however. I have not really pinned down what the
difference between them would be that would cause the Orinoco's not to
show this behaviour even though their signal may be somewhat lower.

We've taken a spectrum analyzer up the tower and don't really see any
other signals that are really hot out there but it feels like an
interefernce problem. Unfortunately, the tower is about an hour drive
so catching this while it happens has proved somewhat problematic.

In anycase, I feel your pain. I'll let you know if we figure out
what this issue is.

-Eric
--
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/ 


--
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/


Re: [WISPA] The Gremlin, redux

2006-10-27 Thread Matt Larsen - Lists

Maybe there is a microwave lighting system somewhere nearby.

You know what to do with an outdoor light that needs to be shut off.:^)

Matt Larsen
[EMAIL PROTECTED]


John Scrivner wrote:
This has been our plan all along. It just will not stay screwed up 
long enough to get a single heading. The signal level of the 
interference looks like data but it varies in level so much that 
finding the heading is not easy. I know spectrum analysis and this one 
has me stumped. I wish it would interfere and stay that way long 
enough to track it. I honestly think it is a leaky microwave oven 
because it runs about long enough to nuke a frozen chicken patty and 
happens at lunch time quite often!

:-)
Scriv


Jack Unger wrote:

If it's true that there's a giant something that's spewing noise, 
you can use a spectrum analyzer and try to identify the noise 
signature, then triangulate.

jack


David E. Smith wrote:


This problem was mentioned back in May (see
http://lists.wispa.org/pipermail/wireless/2006-May/025354.html ) but
it's still here, and I thought it might be worthwhile to bounce it off
folks again, to see if anyone has any new insights.

Occasionally, completely at random, many of our 2.4GHz APs start 
showing

high packet loss (usually 20-30%) and high latency (upwards of ten
seconds - not milliseconds, full seconds) for a few minutes at a time.

It always clears itself up eventually, usually after a minute or two,
but sometimes it won't go away for ten or fifteen minutes. It's
especially annoying when it happens in the middle of the day and all
those itchy business customers call in at once.

Random facts and tidbits:

* I've watched our network traffic, using Mikrotik's torch and StarOS'
beacon tools, and I don't see anything that looks like a DDOS, either
entering or leaving our network.
* It doesn't affect all our towers, just most of them. Specifically, 
our

most remote towers (geographically, not in terms of network topology)
are safe.
* (This is the Lonnie Nunweiler clause) It affects equally towers that
are bridged, routed, and hybrid bridged/routed.
* I don't think it's a backhaul problem, because towers on several
different kinds of backhaul links are affected simultaneously. (We've
got a mix of Alvarion and Trango gear for backhauls, and at least one
ancient YDI EX-1.) Also, there's no problem talking between tower
locations; pings between different APs take 10ms or so, just like they
normally do.
* It only affects our 2.4GHz customers - folks on 900MHz gear, 5.3GHz,
and 5.8GHz connections don't have any issues while things are going 
sour.


Basically, it looks like someone's got a big giant massive something
that spews out insane amounts of 2.4GHz interference, that somehow
knocks out all our customers within about a fifteen-mile radius, and
they turn it on every so often.

Does that conclusion sound reasonable? And if it does, what the heck 
can

I do about it?

Frustrated,

David Smith
MVN.net





--
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/


Re: [WISPA] The Gremlin, redux

2006-10-27 Thread Eric Merkel

On 10/27/06, Rich Comroe [EMAIL PROTECTED] wrote:

We look at the traffic on the
tower for abuse and/or virus and don't really find anything.

Just to be clear, you've checked your AP broadcast levels during the events
and not found found them elevated?  We found the most crippling network
events were not coming into the network from the outside, but were broadcast
storms between 2 or more customers (repeated through the APs).  They act
similar to the symptoms you cited (a few minutes of extremely elevated
latency due to the short term load they place over the rf).

Rich



We try to mitigate this problem by the following:

1) Turning off inter-BSS Relay
2) We block all the typical MS ports(135-139) which broadcast all the
time via iptables
3) Packet shape all connections via CBQ on the AP itself to limit how
much bandwidth any one customer can consume

Looking at the beacon realtime manager and tcpdump, we've never seen
an unreasonable # of broadcasts when this is happening.

-Eric
--
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/


Re: [WISPA] The Gremlin, redux

2006-10-27 Thread David E. Smith
On Fri, October 27, 2006 1:07 pm, Eric Merkel wrote:

 We have a similar situation happening mainly on one tower of ours.
 Basicially it is a StarOS V2 on WRAP boards setup using Prism cards
 for the AP's. We have 4 90* horizontal sectors.

Hm. A few months back, we converted the last of our towers to StarOS V2 on
RouterBoard 500s, mostly with Prism-based cards (but we do have a few
Orinoco and at least one or two Atheros cards in the mix).

Do you also have the insane latency, perchance? (Or even just
higher-than-normal latency?) A few minutes of high packet loss just leads
to a slow (but usable) connection; a few minutes of ten-second latency
renders their connection effectively dead, as everything times out.

Which reminds me, just to try to rule out network traffic another way: If
I log into an AP with a private address (say 10.232.175.1/27) and ping a
wireless client device in that same subnet (10.232.175.20), I *still* see
the same latency and packet loss. The AP knows that those ping packets are
just going out its own wireless interface, so they're not going anywhere
else except through that last-mile hop. And since they're private
addresses, there shouldn't be anything on the public Internet spewing
traffic towards them.

 Most of our clients on CB3's but we do have some Orinoco based
 clients. The Orinoco based clients don't seem to have the problem as
 much as the CB3's do however. I have not really pinned down what the
 difference between them would be that would cause the Orinoco's not to
 show this behaviour even though their signal may be somewhat lower.

Most of my end-users are on CB3s now as well, but we still have a few
Orinoco radios (mostly old RG-1000s with Karlnet firmware), and a few
other random things as well (WRAP boards with Atheros, probably still a
few eight-year-old Maxtechs and Nokia clients, hacked-up Linksys routers
with custom firmware, and Ghu knows what else). As far as I can tell,
they're ALL affected equally.

While I'm still not absolutely convinced that this is an interference
problem, I think I've ruled out everything else.

David Smith
MVN.net
-- 
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/


Re: [WISPA] The Gremlin, redux

2006-10-27 Thread David E. Smith
On Fri, October 27, 2006 3:11 pm, Eric Merkel wrote:

 1) Turning off inter-BSS Relay

Already done, on most towers. (We do have a couple of towers where one
business, with two locations, wants to do VPN-type stuff between 'em.)

 2) We block all the typical MS ports(135-139) which broadcast all the
 time via iptables

Done. We block 135-139, 445, and a couple other ports, both TCP and UDP.

 3) Packet shape all connections via CBQ on the AP itself to limit how
 much bandwidth any one customer can consume

Mostly done. (For historical reasons, some of our customers are still part
of a giant bridged network, and their traffic is shaped in our office not
at the AP, but those customers are relatively few and growing fewer by the
week.)

David Smith
MVN.net
-- 
WISPA Wireless List: wireless@wispa.org

Subscribe/Unsubscribe:
http://lists.wispa.org/mailman/listinfo/wireless

Archives: http://lists.wispa.org/pipermail/wireless/