Sure.
On 7/19/2016 8:07 PM, Mike Hammett wrote:
Can any of you seeing this problem grab some packet captures the next
time you see it? It's worth actually digging into.
-----
Mike Hammett
Intelligent Computing Solutions <http://www.ics-il.com/>
<https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
Midwest Internet Exchange <http://www.midwest-ix.com/>
<https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
The Brothers WISP <http://www.thebrotherswisp.com/>
<https://www.facebook.com/thebrotherswisp>
<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
------------------------------------------------------------------------
*From: *"Ken Hohhof" <[email protected]>
*To: *[email protected]
*Sent: *Tuesday, July 19, 2016 7:37:42 PM
*Subject: *Re: [AFMUG] CDN overload
I saw it to various customers for 2 days starting mid day last
Tuesday. It has not come back. Even today, which is Tuesday again.
I wonder if it had something to do with the expiration of the free
Windows 10 upgrades on July 29. And then the “Anniversary Update”
rolls out starting Aug. 2.
I had one tower fed via a licensed link but with only a Fast Ethernet
port on the router at one end. Not usually a problem since peak
bandwidth at that tower never approaches 100M. But with 100-150M of
traffic for one customer, the link was being saturated for 5-10 minute
intervals. So clearly they were not following TCP congestion
management. I quickly added another GigE EHWIC module to the Cisco
router so it wouldn’t happen again, but something really nasty was
going on.
*From:* George Skorup <mailto:[email protected]>
*Sent:* Tuesday, July 19, 2016 7:06 PM
*To:* [email protected] <mailto:[email protected]>
*Subject:* Re: [AFMUG] CDN overload
Noop. As I said, Microsuck at one point was sending to a 1.5Mbps
customer at nearly 25Mbps. Confirmed single machine. I believe it was
all the same source address, but like 20 separate streams.
Happened to a guy on Saturday as well. Yet another 1.5Mbps 900MHz
customer. Single PC directly to the radio. I was torching him and saw
about 12Mbps coming from MS's 13.x. Then it would settle for a while
and pick right back up again from LLNW at 6-8Mbps.
That guy opened a ticket and said he was getting less than 100kbps
download speed and no web pages would load. He responded about an hour
later and said everything was normal.
And yet another customer on Wednesday on PMP450 at 12Mbps tier was
being sent over 30Mbps.
On 7/19/2016 2:41 PM, Mike Hammett wrote:
Were all CDNs sending way more than the pipe size or only LimeLight?
Someone at Akamai sent out this message last week regarding a
general increase in usage:
=====
There were two major software updates that spanned Tuesday and
Wednesday which are responsible for the increase you saw.
=====
-----
Mike Hammett
Intelligent Computing Solutions <http://www.ics-il.com/>
<https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
Midwest Internet Exchange <http://www.midwest-ix.com/>
<https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
The Brothers WISP <http://www.thebrotherswisp.com/>
<https://www.facebook.com/thebrotherswisp>
<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
------------------------------------------------------------------------
*From: *"George Skorup" mailto:[email protected]
*To: *[email protected]
*Sent: *Thursday, July 14, 2016 1:33:21 AM
*Subject: *Re: [AFMUG] CDN overload
I forgot about this. Yes. A little later in the day, I started to
see a lot of 13.n.n.n sources. Microsoft. Yeah, update Tuesday.
Then the same customer would start receiving from LLNW. Then
Akamai. And back to MS again. So it looks like they're *still*
distributing updates across various CDNs. And believe me, it's not
like they were all hitting this customer at once. One single CDN
would try to send at 5-10X the customer's downlink MIR. Sometimes
more. At one point I saw over 20Mbps for 5-10 minutes. I saw
pretty much the same thing with about 15 other customers that I
looked at. And they were spread across 5-6 towers. Some directly
licensed fed, others farther towards the edge.
DDoS. CDN. Same thing. Or gorilla tactics at the very least. If
the customer calls and says "none of my other shit works, your
internet sucks" what are we supposed to do? Oh OK, here, we'll
turn you up to 12Mbps and see what that does. Yeah screw that
because now the CDN is sending at 40Mbps! They need to stop
fucking with TCP already! And no, it doesn't matter where I put
the policing/shaping. They still eat up bandwidth on our
upstreams. Like you said before Ken, yeah, it just moves the
problem somewhere else.
On 7/13/2016 11:39 PM, Ken Hohhof wrote:
George, did you identify the application or content provider,
or only the CDN?
I think I started getting hit with the same thing early
yesterday afternoon. At first I thought I was getting DDOS
attacks.
*From:* George Skorup <mailto:[email protected]>
*Sent:* Tuesday, July 12, 2016 6:21 PM
*To:* [email protected] <mailto:[email protected]>
*Subject:* Re: [AFMUG] CDN overload
Yup. LLNW.
On 7/12/2016 5:35 PM, Ken Hohhof wrote:
I assume you torched the traffic and verified it is all
coming from a particular CDN, not a random bunch of IPs as
would be the case with BT. Since this isn’t your first rodeo.
*From:* George Skorup <mailto:[email protected]>
*Sent:* Tuesday, July 12, 2016 5:31 PM
*To:* [email protected] <mailto:[email protected]>
*Subject:* Re: [AFMUG] CDN overload
Because they dick with TCP.
On 7/12/2016 5:23 PM, Eric Kuhnke wrote:
And why is it the fault of the CDN? It could be a
customer with a 100-peer bittorrent session
downloading 30GB of Ubuntu DVD ISOs.
On Tue, Jul 12, 2016 at 3:13 PM, George Skorup
<[email protected] <mailto:[email protected]>> wrote:
I have had it with these CDNs sending more traffic
than the last mile can handle. Got a customer at
1.5Mbps on 900 FSK and they're sending to her at
15Mbps. Of course the AP reports RF downlink
overloaded.
