Two things... Look at the logs, on all the devices.. and see if they are doing anything hokey...
and run a pingplotter to ping/trace and catch any or all of these anomalies... Overall if you have happy customers, they may take some time to tell you when something is partially broken.... Ofcourse, we all have the early reporters... Regards. Faisal Imtiaz Snappy Internet & Telecom 7266 SW 48 Street Miami, FL 33155 Tel: 305 663 5518 x 232 Help-desk: (305)663-5518 Option 2 or Email: [email protected] > From: "CBB - Jay Fuller" <[email protected]> > To: [email protected] > Sent: Saturday, June 18, 2016 9:16:23 PM > Subject: Re: [AFMUG] after the storm... > ok - more details. here is what i know - i'll try not to be too vague. > parts of our network are routed. parts are bridged. parts are NATTED (well, a > lot of it is natted) > The complaints have come from two towers - but the towers are pretty close to > our core. > Tower 1 - it actually has two backhauls right now because we are/were testing > mimosa and haven't finished switching to one backhaul yet. the tower itself > and > all associated equipment feeds through a ptp600. One customer on an epmp 2.4 > radio reports the following : > " Every hour or so it will go down for a minute then back up and either ok or > slow" , his definition of slow is it won't run a speedtest... lol. He says " > Thanks. It started right after storm the other night." > So far, that customer is the only report of an issue on that tower. > Now - Tower 2 - downstream from Tower 1 - has the following gear in the > following configuation. > This tower is feeding a large network (sustained 60-80 meg peak times) . It is > on a relatively new Netonix switch > with only the new mimosa backhaul coming through tower 1. There is nothing > else > plugged into the switch except > for the mimosa gear connecting to tower 2 and the new backhaul from our fiber > on > tower 1. > That is port 2 and port 3 on an rb2011. That connection goes on and terminates > on a mikrotik router that routes > a bunch of stuff behind it. Few problem reports on the mimosa or anything > behind > that router. > Port 1 is power - powered from a digital loggers device that is connected to a > solar panel for backup power. > This site is AC fed but can auto-backup to solar if needed. > Port 4 is a 2.4 ubnt radio feeding the tower owner - who is also our customer > service manager. His home has gotten > upwards of 70% sustained packet loss from that radio. (or something). Problem > was temporarily resolved by changing > subnets but the problem returned a few hours later. Anytime I connect a laptop > to his radio (bypassing his internal router, > etc) I get an ip address conflcit on any ip I try. [it is not actually an IP > address conflcit, the IP won't take on my netbook and > when running ipconfig it shows "duplicate" and won't show the ip assigned). I > can get it to assign briefly by powercycling > the radio. > Port 5 - is feeding another tower owner about 15 miles away that feeds a gated > community. That tower owner reports > no connectivity since the storm but I can see all the way to his equipment > (just > like i could see to my customer service > manager) - assume he also has 70% packet loss. > Port 6 and Port 7 - one is a ubnt 2.4 radio and one is a 900 mhz canopy radio. > Both seem to work fine. Ubnt 2.4 is > our businest rocket probably passing 24 meg or so. Again, off the mimosa > feeding > this tower. This is not routed but > it is on a different subnet (infrastructure is 10.80.3.0/24, subnet on 2.4 > gear > and 900 gear is 172.25.61.0/24) The customer on port 4 had a public IP bridged > (104.152.x.x/29 or something). So did our customer service manager > until I tried putting him on a 172.25.61.x address. > Port 8 - feeds another tower that is routed behind a mikrotik. Talked to a > customer on that network via facebook > tonight - he is fine - no drops, no problems, since replacing gear that was > ethernet fried on that tower friday morning. > I want to blame this tower. I've disabled ports for 15-20 minutes at a time, > especially when troubleshooting our customer service manager's connection. I > eventually disabled his private radio and put him on the customer 2.4 rocket. > No luck > there either. > The problem seems closest to this tower. Earlier I did get this report from > someone downstream from the mimosas but there are like 200 people downstream+ > and this is the only report i've heard : > " Our internet (Helicon community) Has been up and down for the past few days. > It's taken me nearly 15 minutes to load the facebook page and get this message > to you. Phone 256-747-2436 with any other questions. It comes and goes in bits > and pieces." > I really suspected the customer service manager radio - - that is why i > disabled > it. it might be his home radio. i might try > disabling that next.... > any other suggestions? This is my current delima... > thank you all! >> ----- Original Message ----- >> From: CBB - Jay Fuller >> To: [email protected] >> Sent: Saturday, June 18, 2016 7:11 PM >> Subject: [AFMUG] after the storm... >> several segments of our network have reported intermittent connectivity. in >> some >> cases drops "a few times an hour". >> I observed at one location coming off a 2.4 loco more dropped packets than >> good >> packets. instead of replacing that one radio, i thought i might come here for >> suggestions. >> i know a long long time ago we'd heard of ubnt radios losing their mind and >> going on an all out broadcast storm. i think i remember where a firmware >> release sensed this and disabled the ethernet port (thus bricking he unit, >> which is better than it causing chaos). something is causing chaos, but it is >> not everywhere. and of course, it is not getting on other routed segments... >> any suggestions other than disabling one port on the switch at the affected >> location at a time?
