It really looked to me like they were waiting for 100% packet loss over
1-2 minutes before they'd back off. They pull this shit again and I will
drop 13.0.0.0/8 at the borders. And I don't care what it breaks.
On 7/14/2016 2:25 PM, Craig Schmaderer wrote:
Same here, it didn’t take me very long to assume it was windows 10
updates, so the question is, is this going to be the norm? every
update Tuesday going to f**k the network up? Does anyone have some
good info or idea on what is going to happen?
*From:*Af [mailto:[email protected]] *On Behalf Of *Ken Hohhof
*Sent:* Thursday, July 14, 2016 1:26 PM
*To:* [email protected]
*Subject:* Re: [AFMUG] CDN overload
Wowzers. I guess I feel better knowing it’s not just me, this really
had my head spinning, trying to figure out who was attacking me.
But yeah, they changed something this week. It used to be pretty
common to see 4 parallel sessions, 93 is just crazy, hard to interpret
it any other way than not playing nice and pushing all other traffic
aside.
But we all know ISPs are evil and Silicon Valley can do no wrong, so
it must be our fault somehow.
*From:*Mathew Howard <mailto:[email protected]>
*Sent:*Thursday, July 14, 2016 10:10 AM
*To:*af <mailto:[email protected]>
*Subject:*Re: [AFMUG] CDN overload
Just had another one call in... 93 active http sessions to
13.107.4.50, Microsoft obviously changed something with how they're
doing update... I haven't ever seeing so many complaints generated
from an update before.
On Thu, Jul 14, 2016 at 8:13 AM, Nate Burke <[email protected]
<mailto:[email protected]>> wrote:
I just ran into this yesterday, took down an FSK AP that was
running at 10mb Ethernet. A customer with 2 computers, MS Updates
running in the background. Had about 50 http sessions open to
13.x.x.x addresses.
On 7/14/2016 7:50 AM, Adam Moffett wrote:
Seems like they (MS) should look into promoting a multicast
network for distributing updates.
Or simply limit automatic background updates to 256k (per
destination). If the user clicked the update button, sure get
it to run as fast as possible, but if it's in the background
and they don't even know it's happening then it ought to not
matter how long the download takes.
...of course MS is not likely to care about my opinion on the
matter.
------ Original Message ------
From: "George Skorup" <[email protected]
<mailto:[email protected]>>
To: [email protected] <mailto:[email protected]>
Sent: 7/14/2016 2:33:21 AM
Subject: Re: [AFMUG] CDN overload
I forgot about this. Yes. A little later in the day, I
started to see a lot of 13.n.n.n sources. Microsoft. Yeah,
update Tuesday. Then the same customer would start
receiving from LLNW. Then Akamai. And back to MS again. So
it looks like they're *still* distributing updates across
various CDNs. And believe me, it's not like they were all
hitting this customer at once. One single CDN would try to
send at 5-10X the customer's downlink MIR. Sometimes more.
At one point I saw over 20Mbps for 5-10 minutes. I saw
pretty much the same thing with about 15 other customers
that I looked at. And they were spread across 5-6 towers.
Some directly licensed fed, others farther towards the edge.
DDoS. CDN. Same thing. Or gorilla tactics at the very
least. If the customer calls and says "none of my other
shit works, your internet sucks" what are we supposed to
do? Oh OK, here, we'll turn you up to 12Mbps and see what
that does. Yeah screw that because now the CDN is sending
at 40Mbps! They need to stop fucking with TCP already! And
no, it doesn't matter where I put the policing/shaping.
They still eat up bandwidth on our upstreams. Like you
said before Ken, yeah, it just moves the problem somewhere
else.
On 7/13/2016 11:39 PM, Ken Hohhof wrote:
George, did you identify the application or content
provider, or only the CDN?
I think I started getting hit with the same thing
early yesterday afternoon. At first I thought I was
getting DDOS attacks.
*From:*George Skorup <mailto:[email protected]>
*Sent:*Tuesday, July 12, 2016 6:21 PM
*To:*[email protected] <mailto:[email protected]>
*Subject:*Re: [AFMUG] CDN overload
Yup. LLNW.
On 7/12/2016 5:35 PM, Ken Hohhof wrote:
I assume you torched the traffic and verified it
is all coming from a particular CDN, not a random
bunch of IPs as would be the case with BT. Since
this isn’t your first rodeo.
*From:*George Skorup <mailto:[email protected]>
*Sent:*Tuesday, July 12, 2016 5:31 PM
*To:*[email protected] <mailto:[email protected]>
*Subject:*Re: [AFMUG] CDN overload
Because they dick with TCP.
On 7/12/2016 5:23 PM, Eric Kuhnke wrote:
And why is it the fault of the CDN? It could
be a customer with a 100-peer bittorrent
session downloading 30GB of Ubuntu DVD ISOs.
On Tue, Jul 12, 2016 at 3:13 PM, George Skorup
<[email protected] <mailto:[email protected]>>
wrote:
I have had it with these CDNs sending more
traffic than the last mile can handle. Got
a customer at 1.5Mbps on 900 FSK and
they're sending to her at 15Mbps. Of
course the AP reports RF downlink overloaded.
