I have witnessed this issue first hand for several years. Four for sure, maybe five or six. The very first one I remember is a customer doing Usenet downloads and using what he called an "internet download manager" which I assumed was screwing with TCP ACKs. I believe he was a 4Mbps user at the time and this download manager thing was causing 2 to maybe 2.5x his subscribed rate, as Mike says, on the upstream facing router interface. He shut down or uninstalled the software and it stopped. Yes, this customer is on PTMP fixed wireless. Traffic policing was taking place via MikroTik simple queue at the site router.. I could cut his downstream rate in half and it would follow with double still hitting the backhaul. I could also move his queue all the way to the border router and it was still there at double rate.

BTW, we still have this guy as a customer on fixed wireless. He's been on 25/5Mbps for over a year. And we're about to upgrade him to 50/10Mbps with new gear. 25/5 and 50/10 is a far cry from this claimed "slow" WISP service. This shit ain't cheap to get to bumfsck Illinois so farmer Joe can watch porn and his kids can watch Netflix at the same time. Yup, we have slow NLOS service too, because customers decide they want the rural life buried in a mile of trees while "needing" the city benefits. If you want the gigabits, then move outta the sticks. Running a hundred combined miles of fiber to get to 20 customers that want to pay less than $50/mo is not feasible. /rant off

Another time, maybe three years ago, we had a customer on Canopy 5.7 FSK at 4/1Mbps using the built-in QoS. He was watching Netflix and I saw 8Mbps hitting the AP's ethernet interface. I thought the Canopy scheduler was broken. Until I looked deeper and saw that it was working exactly as designed.. with 50% discard rate on his VC. I want to say this was from LLNW at the time. I could be totally wrong about that, I really don't remember.

Now lets move the Windows 10 updates. A 'buried in the sticks' customer on Canopy 900 FSK. 1.5Mbps/384k. Multiple streams from Microsoft and LLNW at the same time. LLNW alone had maybe 10 streams going and was sending at over 15Mbps on average and at worst about 25Mbps... to a 1.5Mbps subscriber. I could throw in a MikroTik queue upstream which only moved the problem as that 15-25Mbps was still hitting backhaul links. And when I have a 100Mbps link going into the site, 25Mbps is a lot.

We've had numerous customers call in for the last month or two with 'teh innernets is down, my phoen wyfy don't work either'. No, your Windows 10 updates are overloading your service. Shut off your PC to use your internet service. Telling a customer those exact words is ridiculous, but we have to do it.

We had a known issue with a particular licensed microwave vendor's radios that we have in use. It was the ethernet buffer becoming saturated at nowhere near the RF link capacity. They put out a new software release and that was resolved. And that was well before this Windows 10 update overload stuff started.

Normal TCP congestion control behavior works perfectly fine. It's not the network. It's the sender not doing normal TCP stuffs. I don't know why the CDNs and/or Microsoft thinks this is a good idea, but to me, it looks like a DDoS. I'm on some of the same lists as Mike and we know of many others reporting similar issues. A couple to the tune of 50-100Mbps overload destined for 5 or 10Mbps tier subscribers. So thanks to Mike for trying to get a conversation going on this topic. And it's not just us red headed step children WISPs.

On 9/19/2016 10:05 PM, Mike Hammett wrote:

This explains the recent situations (well, not really an explanation, but a bit 
more information from other people). Not so much for the ones going back a year 
or two.

Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Mike Hammett" <na...@ics-il.net>
To: "NANOG" <nanog@nanog.org>
Sent: Monday, September 19, 2016 12:34:48 PM
Subject: CDN Overload?

I participate on a few other mailing lists focused on eyeball networks. For a 
couple years I've been hearing complaints from this CDN or that CDN was 
behaving badly. It's been severely ramping up the past few months. There have 
been some wild allegations, but I would like to develop a bit more standardized 
evidence collection. Initially LimeLight was the only culprit, but recently it 
has been Microsoft as well. I'm not sure if there have been any others.

The principal complaint is that upstream of whatever is doing the rate limiting 
for a given customer there is significantly more capacity being utilized than 
the customer has purchased. This could happen briefly as TCP adjusts to the 
capacity limitation, but in some situations this has persisted for days at a 
time. I'll list out a few situations as best as I can recall them. Some of 
these may even be merges of a couple situations. The point is to show the 
general issue and develop a better process for collecting what exactly is 
happening at the time and how to address it.

One situation had approximately 45 megabit/s of capacity being used up by a 
customer that had a 1.5 megabit/s plan. All other traffic normally held itself 
within the 1.5 megabit/s, but this particular CDN sent excessively more for 
extended periods of time.

An often occurrence has someone with a single digit megabit/s limitation 
consuming 2x - 3x more than their plan on the other side of the rate limiter.

Last month on my own network I saw someone with 2x - 3x being consumed upstream 
and they had *190* connections downloading said data from Microsoft.

The past week or two I've been hearing of people only having a single 
connection downloading at more than their plan rate.

These situations effectively shut out all other Internet traffic to that 
customer or even portion of the network for low capacity NLOS areas. It's a DoS 
caused by downloads. What happened to the days of MS BITS and you didn't even 
notice the download happening? A lot of these guys think that the CDNs are just 
a pile of dicks looking to ruin everyone's day and I'm certain that there are 
at least a couple people at each CDN that aren't that way. ;-)

Lots of rambling, sure. What do I need to have these guys collect as evidence 
of a problem and who should they send it to?

Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

Reply via email to