Eject the di-lithium crystals and get the squirrels out to run the ship :)
On 6/18/2016 8:16 PM, CBB - Jay Fuller wrote:
ok - more details. here is what i know - i'll try not to be too vague.
parts of our network are routed. parts are bridged. parts are NATTED
(well, a lot of it is natted)
The complaints have come from two towers - but the towers are pretty
close to our core.
Tower 1 - it actually has two backhauls right now because we are/were
testing mimosa and haven't finished switching to one backhaul yet.
the tower itself and all associated equipment feeds through a ptp600.
One customer on an epmp 2.4 radio reports the following :
"Every hour or so it will go down for a minute then back up and either
ok or slow" , his definition of slow is it won't run a speedtest...
lol. He says " Thanks. It started right after storm the other night."
So far, that customer is the only report of an issue on that tower.
Now - Tower 2 - downstream from Tower 1 - has the following gear in
the following configuation.
This tower is feeding a large network (sustained 60-80 meg peak times)
. It is on a relatively new Netonix switch
with only the new mimosa backhaul coming through tower 1. There is
nothing else plugged into the switch except
for the mimosa gear connecting to tower 2 and the new backhaul from
our fiber on tower 1.
That is port 2 and port 3 on an rb2011. That connection goes on and
terminates on a mikrotik router that routes
a bunch of stuff behind it. Few problem reports on the mimosa or
anything behind that router.
Port 1 is power - powered from a digital loggers device that is
connected to a solar panel for backup power.
This site is AC fed but can auto-backup to solar if needed.
Port 4 is a 2.4 ubnt radio feeding the tower owner - who is also our
customer service manager. His home has gotten
upwards of 70% sustained packet loss from that radio. (or
something). Problem was temporarily resolved by changing
subnets but the problem returned a few hours later. Anytime I connect
a laptop to his radio (bypassing his internal router,
etc) I get an ip address conflcit on any ip I try. [it is not
actually an IP address conflcit, the IP won't take on my netbook and
when running ipconfig it shows "duplicate" and won't show the ip
assigned). I can get it to assign briefly by powercycling
the radio.
Port 5 - is feeding another tower owner about 15 miles away that feeds
a gated community. That tower owner reports
no connectivity since the storm but I can see all the way to his
equipment (just like i could see to my customer service
manager) - assume he also has 70% packet loss.
Port 6 and Port 7 - one is a ubnt 2.4 radio and one is a 900 mhz
canopy radio. Both seem to work fine. Ubnt 2.4 is
our businest rocket probably passing 24 meg or so. Again, off the
mimosa feeding this tower. This is not routed but
it is on a different subnet (infrastructure is 10.80.3.0/24, subnet
on 2.4 gear and 900 gear is 172.25.61.0/24) The customer on port 4
had a public IP bridged (104.152.x.x/29 or something). So did our
customer service manager
until I tried putting him on a 172.25.61.x address.
Port 8 - feeds another tower that is routed behind a mikrotik. Talked
to a customer on that network via facebook
tonight - he is fine - no drops, no problems, since replacing gear
that was ethernet fried on that tower friday morning.
I want to blame this tower. I've disabled ports for 15-20 minutes at
a time, especially when troubleshooting our customer service manager's
connection. I eventually disabled his private radio and put him on
the customer 2.4 rocket. No luck
there either.
The problem seems closest to this tower. Earlier I did get this
report from someone downstream from the mimosas but there are like 200
people downstream+ and this is the only report i've heard :
"Our internet (Helicon community) Has been up and down for the past
few days. It's taken me nearly 15 minutes to load the facebook page
and get this message to you. Phone 256-747-2436 with any other
questions. It comes and goes in bits and pieces."
I really suspected the customer service manager radio - - that is why
i disabled it. it might be his home radio. i might try
disabling that next....
any other suggestions? This is my current delima...
thank you all!
----- Original Message -----
*From:* CBB - Jay Fuller <mailto:[email protected]>
*To:* [email protected] <mailto:[email protected]>
*Sent:* Saturday, June 18, 2016 7:11 PM
*Subject:* [AFMUG] after the storm...
several segments of our network have reported intermittent
connectivity. in some cases drops "a few times an hour".
I observed at one location coming off a 2.4 loco more dropped
packets than good packets. instead of replacing that one radio,
i thought i might come here for suggestions.
i know a long long time ago we'd heard of ubnt radios losing their
mind and going on an all out broadcast storm. i think i remember
where a firmware release sensed this and disabled the ethernet
port (thus bricking he unit, which is better than it causing
chaos). something is causing chaos, but it is not everywhere. and
of course, it is not getting on other routed segments...
any suggestions other than disabling one port on the switch at
the affected location at a time?
--
