subject:"\[AFMUG\] CDN Overload"

I have the ear of engineers at Level 3's CDN and Akamai. Working on Microsoft 
now (just sent the e-mail a couple minutes ago). Please continue filling out 
the form if you haven't. More information is better than less. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




- Original Message -

From: "Mike Hammett" <af...@ics-il.net> 
To: af@afmug.com 
Sent: Monday, September 19, 2016 9:43:15 PM 
Subject: [AFMUG] CDN Overload 


Have you seen a CDN overloading a customer? Help me gather information on the 
issue. 

What CDN? 
What have you identified the traffic to be? 
What is the access network? 
Where is the rate limiting done? 
How is the rate limiting done (policing vs. queueing, SFQ, PFIFO, etc,, etc.)? 
What is doing the rate limiting? 
What is the rate-limit set to? 
Upstream of the rate-limiter, what are you seeing for inbound traffic? 
One connection or many? 
How much traffic? 
How does other traffic behave when exceeding the rate limit? 
Where is NAT performed? 
What is doing NAT? 
Shared NAT or isolated to that customer? 
Have you done a packet capture before and after the rate limiter? The NAT 
device? 
Would you be willing to send a filtered packet capture (only the frames that 
relate to this CDN) to the CDN if they want it? 



There have been reports of CDNs sending more traffic than the customer can 
handle and ignores TCP convention to slow down. Trying to investigate this 
thoroughly so we can get the CDN to fix their system. Multiple CDNs have been 
shown to do this. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP

Re: [AFMUG] CDN Overload

2016-09-21 Thread Ken Hohhof

Yes, maybe Xbox updates.

At one time, I thought it was just LLNW that was the problem.

I recently saw this from a Microsoft IP address as well.

 

 

 

From: Af [mailto:af-boun...@afmug.com] On Behalf Of Mike Hammett
Sent: Wednesday, September 21, 2016 7:43 PM
To: af@afmug.com
Subject: Re: [AFMUG] CDN Overload

 

Didn't you have a bunch of problems with LimeLight in the past?



-
Mike Hammett
 <http://www.ics-il.com/> Intelligent Computing Solutions
 <https://www.facebook.com/ICSIL>  
<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>  
<https://www.linkedin.com/company/intelligent-computing-solutions>  
<https://twitter.com/ICSIL> 
 <http://www.midwest-ix.com/> Midwest Internet Exchange
 <https://www.facebook.com/mdwestix>  
<https://www.linkedin.com/company/midwest-internet-exchange>  
<https://twitter.com/mdwestix> 
 <http://www.thebrotherswisp.com/> The Brothers WISP
 <https://www.facebook.com/thebrotherswisp>  
<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> 




  _  

From: "Ken Hohhof" <af...@kwisp.com <mailto:af...@kwisp.com> >
To: af@afmug.com <mailto:af@afmug.com> 
Sent: Tuesday, September 20, 2016 1:54:10 PM
Subject: Re: [AFMUG] CDN Overload

Mike, I know this doesn’t have all the information you are looking for, but 
it’s all I have time to capture right now.  The source IPs seem to be Level3 
CDN, and it’s sending just under 6 Mbps of traffic to a customer rate-limited 
by the tower router to 3 Mbps (Cisco rate limiting which is RED).  The torch 
results are from a Mikrotik router upstream of the tower.  The 10 second torch 
shows around 40 TCP connections.  This seems to be a common pattern, push 
traffic until packet loss is around 50%, with around 50 TCP connections.

 

I tried blocking individual IPs and it was like whack-a-mole, it just added 
more IPs.  Then I blocked 8.0.0.0/8 which did stop the traffic, but I didn’t 
want to leave that in place.  Once I stopped dropping that traffic, it started 
up again.

 

I don’t know what the traffic is, but I suspect Windows 10 update.  It’s a 
little old lady with one desktop computer.  She says it started around 4pm 
yesterday, which seems a little early for Patch Tuesday.  It is making her 
Internet totally unusable, can’t look up directions, can’t check Facebook, 
sporadically gets email.

 

 

From: Mike Hammett <mailto:af...@ics-il.net>  

Sent: Tuesday, September 20, 2016 9:09 AM

To: af@afmug.com <mailto:af@afmug.com>  

Subject: Re: [AFMUG] CDN Overload

 

Can you address the questions I posed in the initial e-mail?



-
Mike Hammett
 <http://www.ics-il.com/> Intelligent Computing Solutions
 <https://www.facebook.com/ICSIL>  
<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>  
<https://www.linkedin.com/company/intelligent-computing-solutions>  
<https://twitter.com/ICSIL> 
 <http://www.midwest-ix.com/> Midwest Internet Exchange
 <https://www.facebook.com/mdwestix>  
<https://www.linkedin.com/company/midwest-internet-exchange>  
<https://twitter.com/mdwestix> 
 <http://www.thebrotherswisp.com/> The Brothers WISP
 <https://www.facebook.com/thebrotherswisp>  
<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> 




  _  

From: "Jim Bouse [Brazos WiFi]" <j...@brazoswifi.com 
<mailto:j...@brazoswifi.com> >
To: af@afmug.com <mailto:af@afmug.com> 
Sent: Tuesday, September 20, 2016 8:58:12 AM
Subject: Re: [AFMUG] CDN Overload

I’ve seen it the most from Limelight.  Don’t know what they are cramming down 
my user’s throats but I suspect it is either Microsoft or Apple.

 

Jim Bouse

Owner

Mobile IT Pro - Brazos WiFi

979-985-5912

j...@brazoswifi.com <mailto:j...@brazoswifi.com>  

 

From: Af [mailto:af-boun...@afmug.com] On Behalf Of Mike Hammett
Sent: Monday, September 19, 2016 10:29 PM
To: af@afmug.com <mailto:af@afmug.com> 
Subject: Re: [AFMUG] CDN Overload

 

Gather evidence, attempt to work cooperatively, then name and shame if 
necessary. But yes, that's close to my intention. If you do your homework 
properly, the greater networking community is very powerful and will back you. 
Those companies are largely ones that will work with you. Forget Amazon, Sony, 
etc. though.

I've heard from people seeing this with Microsoft, Akamai, Limelight and Apple.



-
Mike Hammett
 <http://www.ics-il.com/> Intelligent Computing Solutions
 <https://www.facebook.com/ICSIL>  
<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>  
<https://www.linkedin.com/company/intelligent-computing-solutions>  
<https://twitter.com/ICSIL> 
 <http://www.midwest-ix.com/> Midwest Internet Exchange
 <https://www.facebook.com/mdwestix>  
<https://www.linkedin.com/company/midwest-internet-exchange>  
<https://

Re: [AFMUG] CDN Overload

Didn't you have a bunch of problems with LimeLight in the past? 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




- Original Message -

From: "Ken Hohhof" <af...@kwisp.com> 
To: af@afmug.com 
Sent: Tuesday, September 20, 2016 1:54:10 PM 
Subject: Re: [AFMUG] CDN Overload 




Mike, I know this doesn’t have all the information you are looking for, but 
it’s all I have time to capture right now. The source IPs seem to be Level3 
CDN, and it’s sending just under 6 Mbps of traffic to a customer rate-limited 
by the tower router to 3 Mbps (Cisco rate limiting which is RED). The torch 
results are from a Mikrotik router upstream of the tower. The 10 second torch 
shows around 40 TCP connections. This seems to be a common pattern, push 
traffic until packet loss is around 50%, with around 50 TCP connections. 

I tried blocking individual IPs and it was like whack-a-mole, it just added 
more IPs. Then I blocked 8.0.0.0/8 which did stop the traffic, but I didn’t 
want to leave that in place. Once I stopped dropping that traffic, it started 
up again. 

I don’t know what the traffic is, but I suspect Windows 10 update. It’s a 
little old lady with one desktop computer. She says it started around 4pm 
yesterday, which seems a little early for Patch Tuesday. It is making her 
Internet totally unusable, can’t look up directions, can’t check Facebook, 
sporadically gets email. 





From: Mike Hammett 
Sent: Tuesday, September 20, 2016 9:09 AM 
To: af@afmug.com 
Subject: Re: [AFMUG] CDN Overload 


Can you address the questions I posed in the initial e-mail? 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




- Original Message -

From: "Jim Bouse [Brazos WiFi]" <j...@brazoswifi.com> 
To: af@afmug.com 
Sent: Tuesday, September 20, 2016 8:58:12 AM 
Subject: Re: [AFMUG] CDN Overload 



I’ve seen it the most from Limelight. Don’t know what they are cramming down my 
user’s throats but I suspect it is either Microsoft or Apple. 


Jim Bouse 
Owner 
Mobile IT Pro - Brazos WiFi 
979-985-5912 
j...@brazoswifi.com 



From: Af [mailto:af-boun...@afmug.com] On Behalf Of Mike Hammett 
Sent: Monday, September 19, 2016 10:29 PM 
To: af@afmug.com 
Subject: Re: [AFMUG] CDN Overload 


Gather evidence, attempt to work cooperatively, then name and shame if 
necessary. But yes, that's close to my intention. If you do your homework 
properly, the greater networking community is very powerful and will back you. 
Those companies are largely ones that will work with you. Forget Amazon, Sony, 
etc. though. 

I've heard from people seeing this with Microsoft, Akamai, Limelight and Apple. 



- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




- Original Message -


From: "That One Guy /sarcasm" < thatoneguyst...@gmail.com > 
To: af@afmug.com 
Sent: Monday, September 19, 2016 10:16:26 PM 
Subject: Re: [AFMUG] CDN Overload 
Did you just indicate an intention to get a cdn to alter a corporate policy? I 
have a huge satchel, I mean it could probably hold a couple bowling balls, 
reality only fills it with a couple small pecans. Does it hurt? 



On Sep 19, 2016 9:43 PM, "Mike Hammett" < af...@ics-il.net > wrote: 




Have you seen a CDN overloading a customer? Help me gather information on the 
issue. 

What CDN? 
What have you identified the traffic to be? 
What is the access network? 
Where is the rate limiting done? 
How is the rate limiting done (policing vs. queueing, SFQ, PFIFO, etc,, etc.)? 
What is doing the rate limiting? 
What is the rate-limit set to? 
Upstream of the rate-limiter, what are you seeing for inbound traffic? 
One connection or many? 
How much traffic? 
How does other traffic behave when exceeding the rate limit? 
Where is NAT performed? 
What is doing NAT? 
Shared NAT or isolated to that customer? 
Have you done a packet capture before and after the rate limiter? The NAT 
device? 
Would you be willing to send a filtered packet capture (only the frames that 
relate to this CDN) to the CDN if they want it? 



There have been reports of CDNs sending more traffic than the customer can 
handle and ignores TCP convention to slow down. Trying to investigate this 
thoroughly so we can get the CDN to fix their system. Multiple CDNs have been 
shown to do this. 



- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP

Re: [AFMUG] CDN Overload

https://docs.google.com/spreadsheets/d/1Jdm0dOBf81kSnXEvVfI6ZJbWFNt5AbYUV8CDxGwLSm8/edit?usp=sharing
 

I have made the anonymized answers public. This will obviously have some bias 
to it given that I mostly know fixed wireless operators, but I'm hoping this 
gets some good distribution to catch more platforms. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




- Original Message -

From: "Mike Hammett" <af...@ics-il.net> 
To: af@afmug.com 
Sent: Wednesday, September 21, 2016 9:14:20 AM 
Subject: Re: [AFMUG] CDN Overload 


https://goo.gl/forms/LvgFRsMdNdI8E9HF3 

I have made this into a Google Form to make it easier to track compared to 
randomly formatted responses on multiple mailing lists, Facebook Groups, etc. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




- Original Message -

From: "Mike Hammett" <af...@ics-il.net> 
To: af@afmug.com 
Sent: Monday, September 19, 2016 9:43:11 PM 
Subject: CDN Overload 


Have you seen a CDN overloading a customer? Help me gather information on the 
issue. 

What CDN? 
What have you identified the traffic to be? 
What is the access network? 
Where is the rate limiting done? 
How is the rate limiting done (policing vs. queueing, SFQ, PFIFO, etc,, etc.)? 
What is doing the rate limiting? 
What is the rate-limit set to? 
Upstream of the rate-limiter, what are you seeing for inbound traffic? 
One connection or many? 
How much traffic? 
How does other traffic behave when exceeding the rate limit? 
Where is NAT performed? 
What is doing NAT? 
Shared NAT or isolated to that customer? 
Have you done a packet capture before and after the rate limiter? The NAT 
device? 
Would you be willing to send a filtered packet capture (only the frames that 
relate to this CDN) to the CDN if they want it? 



There have been reports of CDNs sending more traffic than the customer can 
handle and ignores TCP convention to slow down. Trying to investigate this 
thoroughly so we can get the CDN to fix their system. Multiple CDNs have been 
shown to do this. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP

Re: [AFMUG] CDN Overload

https://goo.gl/forms/LvgFRsMdNdI8E9HF3 

I have made this into a Google Form to make it easier to track compared to 
randomly formatted responses on multiple mailing lists, Facebook Groups, etc. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




- Original Message -

From: "Mike Hammett"  
To: af@afmug.com 
Sent: Monday, September 19, 2016 9:43:11 PM 
Subject: CDN Overload 


Have you seen a CDN overloading a customer? Help me gather information on the 
issue. 

What CDN? 
What have you identified the traffic to be? 
What is the access network? 
Where is the rate limiting done? 
How is the rate limiting done (policing vs. queueing, SFQ, PFIFO, etc,, etc.)? 
What is doing the rate limiting? 
What is the rate-limit set to? 
Upstream of the rate-limiter, what are you seeing for inbound traffic? 
One connection or many? 
How much traffic? 
How does other traffic behave when exceeding the rate limit? 
Where is NAT performed? 
What is doing NAT? 
Shared NAT or isolated to that customer? 
Have you done a packet capture before and after the rate limiter? The NAT 
device? 
Would you be willing to send a filtered packet capture (only the frames that 
relate to this CDN) to the CDN if they want it? 



There have been reports of CDNs sending more traffic than the customer can 
handle and ignores TCP convention to slow down. Trying to investigate this 
thoroughly so we can get the CDN to fix their system. Multiple CDNs have been 
shown to do this. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP

Re: [AFMUG] CDN Overload

2016-09-20 Thread castarritt .

What CDN?  Mostly LLNW and Microsoft
What have you identified the traffic to be? Mostly Windows Update
What is the access network? I don't understand the question; PTMP wireless?
Where is the rate limiting done? At the PMP450 Access Point
How is the rate limiting done (policing vs. queueing, SFQ, PFIFO, etc,,
etc.)? However the PMP450 does it; token bucket I think?
What is doing the rate limiting? PMP450
What is the rate-limit set to? Burst up to 15 Mbps, and sustained between
2-10 Mbps depending on plan.
Upstream of the rate-limiter, what are you seeing for inbound traffic? Up
to the burst bandwidth limit, but never higher.
One connection or many? I've seen 30+, but this is rare; it's usually ~5-10.
How much traffic? I don't understand the question.
How does other traffic behave when exceeding the rate limit? Other
applications and services fall apart when most of their packets
get discarded at the PMP450.
Where is NAT performed? NAT to the public IP is performed at our upstream
edge, but there are usually two more layers at the PMP450 CPE, and at the
customer's router.
What is doing NAT? Mikrotik, PMP450, and SOHO router
Shared NAT or isolated to that customer?  Shared public IP for everyone on
that PTMP Access Point.
Have you done a packet capture before and after the rate limiter? The NAT
device? No
Would you be willing to send a filtered packet capture (only the frames
that relate to this CDN) to the CDN if they want it? No

On Mon, Sep 19, 2016 at 9:43 PM, Mike Hammett  wrote:

> Have you seen a CDN overloading a customer? Help me gather information on
> the issue.
>
> What CDN?
> What have you identified the traffic to be?
> What is the access network?
> Where is the rate limiting done?
> How is the rate limiting done (policing vs. queueing, SFQ, PFIFO, etc,,
> etc.)?
> What is doing the rate limiting?
> What is the rate-limit set to?
> Upstream of the rate-limiter, what are you seeing for inbound traffic?
> One connection or many?
> How much traffic?
> How does other traffic behave when exceeding the rate limit?
> Where is NAT performed?
> What is doing NAT?
> Shared NAT or isolated to that customer?
> Have you done a packet capture before and after the rate limiter? The NAT
> device?
> Would you be willing to send a filtered packet capture (only the frames
> that relate to this CDN) to the CDN if they want it?
>
>
>
> There have been reports of CDNs sending more traffic than the customer can
> handle and ignores TCP convention to slow down. Trying to investigate this
> thoroughly so we can get the CDN to fix their system. Multiple CDNs have
> been shown to do this.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions 
> 
> 
> 
> 
> Midwest Internet Exchange 
> 
> 
> 
> The Brothers WISP 
> 
>
>
> 
>

Re: [AFMUG] CDN Overload

2016-09-20 Thread Mike Hammett

Well if you're having issues with Level 3, then you're in luck. I have the ear 
of a Level 3 CDN engineer. They're very anxious to help. Once I get all of the 
information gathered , we'll be able to make some progress. 

I look forward to additional information. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




- Original Message -

From: "Ken Hohhof" <af...@kwisp.com> 
To: af@afmug.com 
Sent: Tuesday, September 20, 2016 1:54:10 PM 
Subject: Re: [AFMUG] CDN Overload 




Mike, I know this doesn’t have all the information you are looking for, but 
it’s all I have time to capture right now. The source IPs seem to be Level3 
CDN, and it’s sending just under 6 Mbps of traffic to a customer rate-limited 
by the tower router to 3 Mbps (Cisco rate limiting which is RED). The torch 
results are from a Mikrotik router upstream of the tower. The 10 second torch 
shows around 40 TCP connections. This seems to be a common pattern, push 
traffic until packet loss is around 50%, with around 50 TCP connections. 

I tried blocking individual IPs and it was like whack-a-mole, it just added 
more IPs. Then I blocked 8.0.0.0/8 which did stop the traffic, but I didn’t 
want to leave that in place. Once I stopped dropping that traffic, it started 
up again. 

I don’t know what the traffic is, but I suspect Windows 10 update. It’s a 
little old lady with one desktop computer. She says it started around 4pm 
yesterday, which seems a little early for Patch Tuesday. It is making her 
Internet totally unusable, can’t look up directions, can’t check Facebook, 
sporadically gets email. 





From: Mike Hammett 
Sent: Tuesday, September 20, 2016 9:09 AM 
To: af@afmug.com 
Subject: Re: [AFMUG] CDN Overload 


Can you address the questions I posed in the initial e-mail? 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




- Original Message -

From: "Jim Bouse [Brazos WiFi]" <j...@brazoswifi.com> 
To: af@afmug.com 
Sent: Tuesday, September 20, 2016 8:58:12 AM 
Subject: Re: [AFMUG] CDN Overload 



I’ve seen it the most from Limelight. Don’t know what they are cramming down my 
user’s throats but I suspect it is either Microsoft or Apple. 


Jim Bouse 
Owner 
Mobile IT Pro - Brazos WiFi 
979-985-5912 
j...@brazoswifi.com 



From: Af [mailto:af-boun...@afmug.com] On Behalf Of Mike Hammett 
Sent: Monday, September 19, 2016 10:29 PM 
To: af@afmug.com 
Subject: Re: [AFMUG] CDN Overload 


Gather evidence, attempt to work cooperatively, then name and shame if 
necessary. But yes, that's close to my intention. If you do your homework 
properly, the greater networking community is very powerful and will back you. 
Those companies are largely ones that will work with you. Forget Amazon, Sony, 
etc. though. 

I've heard from people seeing this with Microsoft, Akamai, Limelight and Apple. 



- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




- Original Message -


From: "That One Guy /sarcasm" < thatoneguyst...@gmail.com > 
To: af@afmug.com 
Sent: Monday, September 19, 2016 10:16:26 PM 
Subject: Re: [AFMUG] CDN Overload 
Did you just indicate an intention to get a cdn to alter a corporate policy? I 
have a huge satchel, I mean it could probably hold a couple bowling balls, 
reality only fills it with a couple small pecans. Does it hurt? 



On Sep 19, 2016 9:43 PM, "Mike Hammett" < af...@ics-il.net > wrote: 




Have you seen a CDN overloading a customer? Help me gather information on the 
issue. 

What CDN? 
What have you identified the traffic to be? 
What is the access network? 
Where is the rate limiting done? 
How is the rate limiting done (policing vs. queueing, SFQ, PFIFO, etc,, etc.)? 
What is doing the rate limiting? 
What is the rate-limit set to? 
Upstream of the rate-limiter, what are you seeing for inbound traffic? 
One connection or many? 
How much traffic? 
How does other traffic behave when exceeding the rate limit? 
Where is NAT performed? 
What is doing NAT? 
Shared NAT or isolated to that customer? 
Have you done a packet capture before and after the rate limiter? The NAT 
device? 
Would you be willing to send a filtered packet capture (only the frames that 
relate to this CDN) to the CDN if they want it? 



There have been reports of CDNs sending more traffic than the customer can 
handle and ignores TCP convention to slow down. Trying to investigate this 
thoroughly so we can get the CDN to fix their system. Multiple CDNs have been 
shown to do this. 



- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP

Re: [AFMUG] CDN Overload

2016-09-20 Thread Mike Hammett

Can you address the questions I posed in the initial e-mail? 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




- Original Message -

From: "Jim Bouse [Brazos WiFi]" <j...@brazoswifi.com> 
To: af@afmug.com 
Sent: Tuesday, September 20, 2016 8:58:12 AM 
Subject: Re: [AFMUG] CDN Overload 



I’ve seen it the most from Limelight. Don’t know what they are cramming down my 
user’s throats but I suspect it is either Microsoft or Apple. 


Jim Bouse 
Owner 
Mobile IT Pro - Brazos WiFi 
979-985-5912 
j...@brazoswifi.com 



From: Af [mailto:af-boun...@afmug.com] On Behalf Of Mike Hammett 
Sent: Monday, September 19, 2016 10:29 PM 
To: af@afmug.com 
Subject: Re: [AFMUG] CDN Overload 


Gather evidence, attempt to work cooperatively, then name and shame if 
necessary. But yes, that's close to my intention. If you do your homework 
properly, the greater networking community is very powerful and will back you. 
Those companies are largely ones that will work with you. Forget Amazon, Sony, 
etc. though. 

I've heard from people seeing this with Microsoft, Akamai, Limelight and Apple. 



- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




- Original Message -


From: "That One Guy /sarcasm" < thatoneguyst...@gmail.com > 
To: af@afmug.com 
Sent: Monday, September 19, 2016 10:16:26 PM 
Subject: Re: [AFMUG] CDN Overload 
Did you just indicate an intention to get a cdn to alter a corporate policy? I 
have a huge satchel, I mean it could probably hold a couple bowling balls, 
reality only fills it with a couple small pecans. Does it hurt? 



On Sep 19, 2016 9:43 PM, "Mike Hammett" < af...@ics-il.net > wrote: 




Have you seen a CDN overloading a customer? Help me gather information on the 
issue. 

What CDN? 
What have you identified the traffic to be? 
What is the access network? 
Where is the rate limiting done? 
How is the rate limiting done (policing vs. queueing, SFQ, PFIFO, etc,, etc.)? 
What is doing the rate limiting? 
What is the rate-limit set to? 
Upstream of the rate-limiter, what are you seeing for inbound traffic? 
One connection or many? 
How much traffic? 
How does other traffic behave when exceeding the rate limit? 
Where is NAT performed? 
What is doing NAT? 
Shared NAT or isolated to that customer? 
Have you done a packet capture before and after the rate limiter? The NAT 
device? 
Would you be willing to send a filtered packet capture (only the frames that 
relate to this CDN) to the CDN if they want it? 



There have been reports of CDNs sending more traffic than the customer can 
handle and ignores TCP convention to slow down. Trying to investigate this 
thoroughly so we can get the CDN to fix their system. Multiple CDNs have been 
shown to do this. 



- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP

Re: [AFMUG] CDN Overload

2016-09-20 Thread Jim Bouse [Brazos WiFi]

I’ve seen it the most from Limelight.  Don’t know what they are cramming down 
my user’s throats but I suspect it is either Microsoft or Apple.

Jim Bouse
Owner
Mobile IT Pro - Brazos WiFi
979-985-5912
j...@brazoswifi.com

From: Af [mailto:af-boun...@afmug.com] On Behalf Of Mike Hammett
Sent: Monday, September 19, 2016 10:29 PM
To: af@afmug.com
Subject: Re: [AFMUG] CDN Overload

Gather evidence, attempt to work cooperatively, then name and shame if 
necessary. But yes, that's close to my intention. If you do your homework 
properly, the greater networking community is very powerful and will back you. 
Those companies are largely ones that will work with you. Forget Amazon, Sony, 
etc. though.

I've heard from people seeing this with Microsoft, Akamai, Limelight and Apple.


-
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]


<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>

From: "That One Guy /sarcasm" 
<thatoneguyst...@gmail.com<mailto:thatoneguyst...@gmail.com>>
To: af@afmug.com<mailto:af@afmug.com>
Sent: Monday, September 19, 2016 10:16:26 PM
Subject: Re: [AFMUG] CDN Overload

Did you just indicate  an intention to get a cdn to alter a corporate policy? I 
have a huge satchel, I mean it could probably hold a couple bowling balls, 
reality only fills it with a couple small pecans. Does it hurt?

On Sep 19, 2016 9:43 PM, "Mike Hammett" 
<af...@ics-il.net<mailto:af...@ics-il.net>> wrote:
Have you seen a CDN overloading a customer? Help me gather information on the 
issue.

What CDN?
What have you identified the traffic to be?
What is the access network?
Where is the rate limiting done?
How is the rate limiting done (policing vs. queueing, SFQ, PFIFO, etc,, etc.)?
What is doing the rate limiting?
What is the rate-limit set to?
Upstream of the rate-limiter, what are you seeing for inbound traffic?
One connection or many?
How much traffic?
How does other traffic behave when exceeding the rate limit?
Where is NAT performed?
What is doing NAT?
Shared NAT or isolated to that customer?
Have you done a packet capture before and after the rate limiter? The NAT 
device?
Would you be willing to send a filtered packet capture (only the frames that 
relate to this CDN) to the CDN if they want it?



There have been reports of CDNs sending more traffic than the customer can 
handle and ignores TCP convention to slow down. Trying to investigate this 
thoroughly so we can get the CDN to fix their system. Multiple CDNs have been 
shown to do this.


-
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]


<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>

Re: [AFMUG] CDN Overload

2016-09-19 Thread Paul Stewart

Oops… can’t read obviously .. might want though to supply example to clarify 
that for some folks ….. ;)


> On Sep 19, 2016, at 11:24 PM, Mike Hammett <af...@ics-il.net> wrote:
> 
> "What is the access network?"
> 
> 
> 
> -
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
>  <https://www.facebook.com/ICSIL> 
> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> 
> <https://www.linkedin.com/company/intelligent-computing-solutions> 
> <https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
>  <https://www.facebook.com/mdwestix> 
> <https://www.linkedin.com/company/midwest-internet-exchange> 
> <https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
>  <https://www.facebook.com/thebrotherswisp>
> 
> 
>  <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> From: "Paul Stewart" <p...@paulstewart.org <mailto:p...@paulstewart.org>>
> To: af@afmug.com <mailto:af@afmug.com>
> Sent: Monday, September 19, 2016 10:11:49 PM
> Subject: Re: [AFMUG] CDN Overload
> 
> Might suggest also what kind of last mile connectivity they have (WISP vs 
> other) to see if something correlates there…. I don’t see or have heard of 
> this on cable, DSL, or FTTH … 
> 
> Paul
> 
> On Sep 19, 2016, at 10:43 PM, Mike Hammett <af...@ics-il.net 
> <mailto:af...@ics-il.net>> wrote:
> 
> Have you seen a CDN overloading a customer? Help me gather information on the 
> issue.
> 
> What CDN?
> What have you identified the traffic to be?
> What is the access network?
> Where is the rate limiting done?
> How is the rate limiting done (policing vs. queueing, SFQ, PFIFO, etc,, etc.)?
> What is doing the rate limiting?
> What is the rate-limit set to?
> Upstream of the rate-limiter, what are you seeing for inbound traffic?
> One connection or many?
> How much traffic?
> How does other traffic behave when exceeding the rate limit?
> Where is NAT performed?
> What is doing NAT?
> Shared NAT or isolated to that customer?
> Have you done a packet capture before and after the rate limiter? The NAT 
> device?
> Would you be willing to send a filtered packet capture (only the frames that 
> relate to this CDN) to the CDN if they want it?
> 
> 
> 
> There have been reports of CDNs sending more traffic than the customer can 
> handle and ignores TCP convention to slow down. Trying to investigate this 
> thoroughly so we can get the CDN to fix their system. Multiple CDNs have been 
> shown to do this.
> 
> 
> 
> -
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
>  <https://www.facebook.com/ICSIL> 
> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> 
> <https://www.linkedin.com/company/intelligent-computing-solutions> 
> <https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
>  <https://www.facebook.com/mdwestix> 
> <https://www.linkedin.com/company/midwest-internet-exchange> 
> <https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
>  <https://www.facebook.com/thebrotherswisp> 
> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>

Re: [AFMUG] CDN Overload

2016-09-19 Thread Mike Hammett

Gather evidence, attempt to work cooperatively, then name and shame if 
necessary. But yes, that's close to my intention. If you do your homework 
properly, the greater networking community is very powerful and will back you. 
Those companies are largely ones that will work with you. Forget Amazon, Sony, 
etc. though. 

I've heard from people seeing this with Microsoft, Akamai, Limelight and Apple. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




- Original Message -

From: "That One Guy /sarcasm" <thatoneguyst...@gmail.com> 
To: af@afmug.com 
Sent: Monday, September 19, 2016 10:16:26 PM 
Subject: Re: [AFMUG] CDN Overload 


Did you just indicate an intention to get a cdn to alter a corporate policy? I 
have a huge satchel, I mean it could probably hold a couple bowling balls, 
reality only fills it with a couple small pecans. Does it hurt? 


On Sep 19, 2016 9:43 PM, "Mike Hammett" < af...@ics-il.net > wrote: 




Have you seen a CDN overloading a customer? Help me gather information on the 
issue. 

What CDN? 
What have you identified the traffic to be? 
What is the access network? 
Where is the rate limiting done? 
How is the rate limiting done (policing vs. queueing, SFQ, PFIFO, etc,, etc.)? 
What is doing the rate limiting? 
What is the rate-limit set to? 
Upstream of the rate-limiter, what are you seeing for inbound traffic? 
One connection or many? 
How much traffic? 
How does other traffic behave when exceeding the rate limit? 
Where is NAT performed? 
What is doing NAT? 
Shared NAT or isolated to that customer? 
Have you done a packet capture before and after the rate limiter? The NAT 
device? 
Would you be willing to send a filtered packet capture (only the frames that 
relate to this CDN) to the CDN if they want it? 



There have been reports of CDNs sending more traffic than the customer can 
handle and ignores TCP convention to slow down. Trying to investigate this 
thoroughly so we can get the CDN to fix their system. Multiple CDNs have been 
shown to do this. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP

Re: [AFMUG] CDN Overload

2016-09-19 Thread Mike Hammett

"What is the access network?" 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




- Original Message -

From: "Paul Stewart" <p...@paulstewart.org> 
To: af@afmug.com 
Sent: Monday, September 19, 2016 10:11:49 PM 
Subject: Re: [AFMUG] CDN Overload 

Might suggest also what kind of last mile connectivity they have (WISP vs 
other) to see if something correlates there…. I don’t see or have heard of this 
on cable, DSL, or FTTH … 


Paul 





On Sep 19, 2016, at 10:43 PM, Mike Hammett < af...@ics-il.net > wrote: 


Have you seen a CDN overloading a customer? Help me gather information on the 
issue. 

What CDN? 
What have you identified the traffic to be? 
What is the access network? 
Where is the rate limiting done? 
How is the rate limiting done (policing vs. queueing, SFQ, PFIFO, etc,, etc.)? 
What is doing the rate limiting? 
What is the rate-limit set to? 
Upstream of the rate-limiter, what are you seeing for inbound traffic? 
One connection or many? 
How much traffic? 
How does other traffic behave when exceeding the rate limit? 
Where is NAT performed? 
What is doing NAT? 
Shared NAT or isolated to that customer? 
Have you done a packet capture before and after the rate limiter? The NAT 
device? 
Would you be willing to send a filtered packet capture (only the frames that 
relate to this CDN) to the CDN if they want it? 



There have been reports of CDNs sending more traffic than the customer can 
handle and ignores TCP convention to slow down. Trying to investigate this 
thoroughly so we can get the CDN to fix their system. Multiple CDNs have been 
shown to do this. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP

Re: [AFMUG] CDN Overload

2016-09-19 Thread That One Guy /sarcasm

Did you just indicate  an intention to get a cdn to alter a corporate
policy? I have a huge satchel, I mean it could probably hold a couple
bowling balls, reality only fills it with a couple small pecans. Does it
hurt?

On Sep 19, 2016 9:43 PM, "Mike Hammett"  wrote:

Have you seen a CDN overloading a customer? Help me gather information on
the issue.

What CDN?
What have you identified the traffic to be?
What is the access network?
Where is the rate limiting done?
How is the rate limiting done (policing vs. queueing, SFQ, PFIFO, etc,,
etc.)?
What is doing the rate limiting?
What is the rate-limit set to?
Upstream of the rate-limiter, what are you seeing for inbound traffic?
One connection or many?
How much traffic?
How does other traffic behave when exceeding the rate limit?
Where is NAT performed?
What is doing NAT?
Shared NAT or isolated to that customer?
Have you done a packet capture before and after the rate limiter? The NAT
device?
Would you be willing to send a filtered packet capture (only the frames
that relate to this CDN) to the CDN if they want it?



There have been reports of CDNs sending more traffic than the customer can
handle and ignores TCP convention to slow down. Trying to investigate this
thoroughly so we can get the CDN to fix their system. Multiple CDNs have
been shown to do this.



-
Mike Hammett
Intelligent Computing Solutions 




Midwest Internet Exchange 



The Brothers WISP

Re: [AFMUG] CDN Overload

2016-09-19 Thread Paul Stewart

Might suggest also what kind of last mile connectivity they have (WISP vs 
other) to see if something correlates there…. I don’t see or have heard of this 
on cable, DSL, or FTTH … 

Paul

> On Sep 19, 2016, at 10:43 PM, Mike Hammett  wrote:
> 
> Have you seen a CDN overloading a customer? Help me gather information on the 
> issue.
> 
> What CDN?
> What have you identified the traffic to be?
> What is the access network?
> Where is the rate limiting done?
> How is the rate limiting done (policing vs. queueing, SFQ, PFIFO, etc,, etc.)?
> What is doing the rate limiting?
> What is the rate-limit set to?
> Upstream of the rate-limiter, what are you seeing for inbound traffic?
> One connection or many?
> How much traffic?
> How does other traffic behave when exceeding the rate limit?
> Where is NAT performed?
> What is doing NAT?
> Shared NAT or isolated to that customer?
> Have you done a packet capture before and after the rate limiter? The NAT 
> device?
> Would you be willing to send a filtered packet capture (only the frames that 
> relate to this CDN) to the CDN if they want it?
> 
> 
> 
> There have been reports of CDNs sending more traffic than the customer can 
> handle and ignores TCP convention to slow down. Trying to investigate this 
> thoroughly so we can get the CDN to fix their system. Multiple CDNs have been 
> shown to do this.
> 
> 
> 
> -
> Mike Hammett
> Intelligent Computing Solutions 
>   
>  
>  
> 
> Midwest Internet Exchange 
>   
>  
> 
> The Brothers WISP 
>   
>

[AFMUG] CDN Overload

2016-09-19 Thread Mike Hammett

Have you seen a CDN overloading a customer? Help me gather information on the 
issue. 

What CDN? 
What have you identified the traffic to be? 
What is the access network? 
Where is the rate limiting done? 
How is the rate limiting done (policing vs. queueing, SFQ, PFIFO, etc,, etc.)? 
What is doing the rate limiting? 
What is the rate-limit set to? 
Upstream of the rate-limiter, what are you seeing for inbound traffic? 
One connection or many? 
How much traffic? 
How does other traffic behave when exceeding the rate limit? 
Where is NAT performed? 
What is doing NAT? 
Shared NAT or isolated to that customer? 
Have you done a packet capture before and after the rate limiter? The NAT 
device? 
Would you be willing to send a filtered packet capture (only the frames that 
relate to this CDN) to the CDN if they want it? 



There have been reports of CDNs sending more traffic than the customer can 
handle and ignores TCP convention to slow down. Trying to investigate this 
thoroughly so we can get the CDN to fix their system. Multiple CDNs have been 
shown to do this. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP

Re: [AFMUG] CDN overload

2016-08-11 Thread Ken Hohhof

At least I haven’t seen a repeat of what happened 4 weeks ago with 100-150 Mbps 
to a 3 Mbps customer.

From: Mike Hammett 
Sent: Thursday, August 11, 2016 8:39 AM
To: af@afmug.com 
Subject: Re: [AFMUG] CDN overload

I saw this in action today from Microsoft. I didn't see 15, but I did see up to 
3 megs for a 1.5 meg customer. I saw up to 190 connections in torch. I took a 
packet capture on the upstream interface and sent it over to one of my 
Microsoft contacts. She may not be the right person, but she's been very 
helpful before so hopefully she can get me to the right person.

-
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

From: "George Skorup" <geo...@cbcast.com>
To: af@afmug.com
Sent: Tuesday, July 12, 2016 5:13:46 PM
Subject: [AFMUG] CDN overload

I have had it with these CDNs sending more traffic than the last mile 
can handle. Got a customer at 1.5Mbps on 900 FSK and they're sending to 
her at 15Mbps. Of course the AP reports RF downlink overloaded.

Re: [AFMUG] CDN overload

2016-08-11 Thread Mike Hammett

I saw this in action today from Microsoft. I didn't see 15, but I did see up to 
3 megs for a 1.5 meg customer. I saw up to 190 connections in torch. I took a 
packet capture on the upstream interface and sent it over to one of my 
Microsoft contacts. She may not be the right person, but she's been very 
helpful before so hopefully she can get me to the right person. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




- Original Message -

From: "George Skorup" <geo...@cbcast.com> 
To: af@afmug.com 
Sent: Tuesday, July 12, 2016 5:13:46 PM 
Subject: [AFMUG] CDN overload 

I have had it with these CDNs sending more traffic than the last mile 
can handle. Got a customer at 1.5Mbps on 900 FSK and they're sending to 
her at 15Mbps. Of course the AP reports RF downlink overloaded.

Re: [AFMUG] CDN overload

2016-07-19 Thread Craig Schmaderer

I had a huge increase in complaints last tuesday and wednesday.  We saw the 
same thing.  I wanted to dig a hole in the sand.

Craig Schmaderer
Cell 402-380-1245
Skywave Wireless, Inc.




On Tue, Jul 19, 2016 at 8:27 PM -0500, "George Skorup" 
<geo...@cbcast.com<mailto:geo...@cbcast.com>> wrote:

Sure.

On 7/19/2016 8:07 PM, Mike Hammett wrote:
Can any of you seeing this problem grab some packet captures the next time you 
see it? It's worth actually digging into.



-
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]


<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>

From: "Ken Hohhof" <af...@kwisp.com><mailto:af...@kwisp.com>
To: af@afmug.com<mailto:af@afmug.com>
Sent: Tuesday, July 19, 2016 7:37:42 PM
Subject: Re: [AFMUG] CDN overload

I saw it to various customers for 2 days starting mid day last Tuesday.  It has 
not come back.  Even today, which is Tuesday again.  I wonder if it had 
something to do with the expiration of the free Windows 10 upgrades on July 29. 
 And then the “Anniversary Update” rolls out starting Aug. 2.

I had one tower fed via a licensed link but with only a Fast Ethernet port on 
the router at one end.  Not usually a problem since peak bandwidth at that 
tower never approaches 100M.  But with 100-150M of traffic for one customer, 
the link was being saturated for 5-10 minute intervals.  So clearly they were 
not following TCP congestion management.  I quickly added another GigE EHWIC 
module to the Cisco router so it wouldn’t happen again, but something really 
nasty was going on.


From: George Skorup<mailto:geo...@cbcast.com>
Sent: Tuesday, July 19, 2016 7:06 PM
To: af@afmug.com<mailto:af@afmug.com>
Subject: Re: [AFMUG] CDN overload

Noop. As I said, Microsuck at one point was sending to a 1.5Mbps customer at 
nearly 25Mbps. Confirmed single machine. I believe it was all the same source 
address, but like 20 separate streams.

Happened to a guy on Saturday as well. Yet another 1.5Mbps 900MHz customer. 
Single PC directly to the radio. I was torching him and saw about 12Mbps coming 
from MS's 13.x. Then it would settle for a while and pick right back up again 
from LLNW at 6-8Mbps.

That guy opened a ticket and said he was getting less than 100kbps download 
speed and no web pages would load. He responded about an hour later and said 
everything was normal.

And yet another customer on Wednesday on PMP450 at 12Mbps tier was being sent 
over 30Mbps.

On 7/19/2016 2:41 PM, Mike Hammett wrote:
Were all CDNs sending way more than the pipe size or only LimeLight?

Someone at Akamai sent out this message last week regarding a general increase 
in usage:


=
There were two major software updates that spanned Tuesday and
Wednesday which are responsible for the increase you saw.
=



-
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]


<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>

From: "George Skorup" mailto:geo...@cbcast.com
To: af@afmu

Re: [AFMUG] CDN overload

2016-07-19 Thread George Skorup

Sure.

On 7/19/2016 8:07 PM, Mike Hammett wrote:
Can any of you seeing this problem grab some packet captures the next
time you see it? It's worth actually digging into.

-
Mike Hammett
Intelligent Computing Solutions <http://www.ics-il.com/>
<https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
Midwest Internet Exchange <http://www.midwest-ix.com/>
<https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
The Brothers WISP <http://www.thebrotherswisp.com/>
<https://www.facebook.com/thebrotherswisp>

<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>

*From: *"Ken Hohhof" <af...@kwisp.com>
*To: *af@afmug.com
*Sent: *Tuesday, July 19, 2016 7:37:42 PM
*Subject: *Re: [AFMUG] CDN overload

I saw it to various customers for 2 days starting mid day last
Tuesday. It has not come back. Even today, which is Tuesday again.
I wonder if it had something to do with the expiration of the free
Windows 10 upgrades on July 29. And then the “Anniversary Update”
rolls out starting Aug. 2.
I had one tower fed via a licensed link but with only a Fast Ethernet
port on the router at one end. Not usually a problem since peak
bandwidth at that tower never approaches 100M. But with 100-150M of
traffic for one customer, the link was being saturated for 5-10 minute
intervals. So clearly they were not following TCP congestion
management. I quickly added another GigE EHWIC module to the Cisco
router so it wouldn’t happen again, but something really nasty was
going on.

*From:* George Skorup <mailto:geo...@cbcast.com>
*Sent:* Tuesday, July 19, 2016 7:06 PM
*To:* af@afmug.com <mailto:af@afmug.com>
*Subject:* Re: [AFMUG] CDN overload
Noop. As I said, Microsuck at one point was sending to a 1.5Mbps
customer at nearly 25Mbps. Confirmed single machine. I believe it was
all the same source address, but like 20 separate streams.

Happened to a guy on Saturday as well. Yet another 1.5Mbps 900MHz
customer. Single PC directly to the radio. I was torching him and saw
about 12Mbps coming from MS's 13.x. Then it would settle for a while
and pick right back up again from LLNW at 6-8Mbps.

That guy opened a ticket and said he was getting less than 100kbps
download speed and no web pages would load. He responded about an hour
later and said everything was normal.

And yet another customer on Wednesday on PMP450 at 12Mbps tier was
being sent over 30Mbps.

On 7/19/2016 2:41 PM, Mike Hammett wrote:

Were all CDNs sending way more than the pipe size or only LimeLight?

Someone at Akamai sent out this message last week regarding a
general increase in usage:

=
There were two major software updates that spanned Tuesday and
Wednesday which are responsible for the increase you saw.
=

-
Mike Hammett
Intelligent Computing Solutions <http://www.ics-il.com/>

<https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
Midwest Internet Exchange <http://www.midwest-ix.com/>

<https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
The Brothers WISP <http://www.thebrotherswisp.com/>
<https://www.facebook.com/thebrotherswisp>

<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
--------
*From: *"George Skorup" mailto:geo...@cbcast.com
*To: *af@afmug.com
*Sent: *Thursday, July 14, 2016 1:33:21 AM
*Subject: *Re: [AFMUG] CDN overload

I forgot about this. Yes. A little later in the day, I started to
see a lot of 13.n.n.n sources. Microsoft. Yeah, update Tuesday.
Then the same customer would start receiving from LLNW. Then
Akamai. And back to MS again. So it looks like they're *still*
distributing updates across various CDNs. And believe me, it's not
like they were all hitting this customer at once. One single CDN
would try to send at 5-10X the customer's downlink MIR. Sometimes
more. At one point I saw over 20Mbps for 5-10 minutes. I saw
pretty much the same thing with about 15 other customers that I
looked at. And they were spread across 5-6 towers. Some directly
licensed fed, others farther towards the edge.

DDoS. CDN. Same thing. Or gorilla tactics at the very least. If
the customer calls and says "none of my other shit works, your
internet sucks&qu

Re: [AFMUG] CDN overload

Can any of you seeing this problem grab some packet captures the next time you 
see it? It's worth actually digging into. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




- Original Message -

From: "Ken Hohhof" <af...@kwisp.com> 
To: af@afmug.com 
Sent: Tuesday, July 19, 2016 7:37:42 PM 
Subject: Re: [AFMUG] CDN overload 




I saw it to various customers for 2 days starting mid day last Tuesday. It has 
not come back. Even today, which is Tuesday again. I wonder if it had something 
to do with the expiration of the free Windows 10 upgrades on July 29. And then 
the “Anniversary Update” rolls out starting Aug. 2. 

I had one tower fed via a licensed link but with only a Fast Ethernet port on 
the router at one end. Not usually a problem since peak bandwidth at that tower 
never approaches 100M. But with 100-150M of traffic for one customer, the link 
was being saturated for 5-10 minute intervals. So clearly they were not 
following TCP congestion management. I quickly added another GigE EHWIC module 
to the Cisco router so it wouldn’t happen again, but something really nasty was 
going on. 





From: George Skorup 
Sent: Tuesday, July 19, 2016 7:06 PM 
To: af@afmug.com 
Subject: Re: [AFMUG] CDN overload 

Noop. As I said, Microsuck at one point was sending to a 1.5Mbps customer at 
nearly 25Mbps. Confirmed single machine. I believe it was all the same source 
address, but like 20 separate streams. 

Happened to a guy on Saturday as well. Yet another 1.5Mbps 900MHz customer. 
Single PC directly to the radio. I was torching him and saw about 12Mbps coming 
from MS's 13.x. Then it would settle for a while and pick right back up again 
from LLNW at 6-8Mbps. 

That guy opened a ticket and said he was getting less than 100kbps download 
speed and no web pages would load. He responded about an hour later and said 
everything was normal. 

And yet another customer on Wednesday on PMP450 at 12Mbps tier was being sent 
over 30Mbps. 


On 7/19/2016 2:41 PM, Mike Hammett wrote: 



Were all CDNs sending way more than the pipe size or only LimeLight? 

Someone at Akamai sent out this message last week regarding a general increase 
in usage: 


= 
There were two major software updates that spanned Tuesday and 
Wednesday which are responsible for the increase you saw. 
= 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




- Original Message -

From: "George Skorup" mailto:geo...@cbcast.com 
To: af@afmug.com 
Sent: Thursday, July 14, 2016 1:33:21 AM 
Subject: Re: [AFMUG] CDN overload 

I forgot about this. Yes. A little later in the day, I started to see a lot of 
13.n.n.n sources. Microsoft. Yeah, update Tuesday. Then the same customer would 
start receiving from LLNW. Then Akamai. And back to MS again. So it looks like 
they're *still* distributing updates across various CDNs. And believe me, it's 
not like they were all hitting this customer at once. One single CDN would try 
to send at 5-10X the customer's downlink MIR. Sometimes more. At one point I 
saw over 20Mbps for 5-10 minutes. I saw pretty much the same thing with about 
15 other customers that I looked at. And they were spread across 5-6 towers. 
Some directly licensed fed, others farther towards the edge. 

DDoS. CDN. Same thing. Or gorilla tactics at the very least. If the customer 
calls and says "none of my other shit works, your internet sucks" what are we 
supposed to do? Oh OK, here, we'll turn you up to 12Mbps and see what that 
does. Yeah screw that because now the CDN is sending at 40Mbps! They need to 
stop fucking with TCP already! And no, it doesn't matter where I put the 
policing/shaping. They still eat up bandwidth on our upstreams. Like you said 
before Ken, yeah, it just moves the problem somewhere else. 


On 7/13/2016 11:39 PM, Ken Hohhof wrote: 





George, did you identify the application or content provider, or only the CDN? 

I think I started getting hit with the same thing early yesterday afternoon. At 
first I thought I was getting DDOS attacks. 





From: George Skorup 
Sent: Tuesday, July 12, 2016 6:21 PM 
To: af@afmug.com 
Subject: Re: [AFMUG] CDN overload 

Yup. LLNW. 


On 7/12/2016 5:35 PM, Ken Hohhof wrote: 





I assume you torched the traffic and verified it is all coming from a 
particular CDN, not a random bunch of IPs as would be the case with BT. Since 
this isn’t your first rodeo. 




From: George Skorup 
Sent: Tuesday, July 12, 2016 5:31 PM 
To: af@afmug.com 
Subject: Re: [AFMUG] CDN overload 

Because they dick with TCP. 


On 7/12/2016 5:23 PM, Eric Kuhnke wrote: 



And why is it the fault of the CDN? It could be a customer with a 100-peer 
bittorrent session downloading 30GB of Ubuntu DVD ISOs. 



On Tue, Jul 12, 2016 at 3:13 PM, George Skorup < geo...@cbcast.com > wrote: 


I have had it with these CDNs sending more t

Re: [AFMUG] CDN overload

2016-07-19 Thread Ken Hohhof

I saw it to various customers for 2 days starting mid day last Tuesday.  It has 
not come back.  Even today, which is Tuesday again.  I wonder if it had 
something to do with the expiration of the free Windows 10 upgrades on July 29. 
 And then the “Anniversary Update” rolls out starting Aug. 2.

I had one tower fed via a licensed link but with only a Fast Ethernet port on 
the router at one end.  Not usually a problem since peak bandwidth at that 
tower never approaches 100M.  But with 100-150M of traffic for one customer, 
the link was being saturated for 5-10 minute intervals.  So clearly they were 
not following TCP congestion management.  I quickly added another GigE EHWIC 
module to the Cisco router so it wouldn’t happen again, but something really 
nasty was going on.


From: George Skorup 
Sent: Tuesday, July 19, 2016 7:06 PM
To: af@afmug.com 
Subject: Re: [AFMUG] CDN overload

Noop. As I said, Microsuck at one point was sending to a 1.5Mbps customer at 
nearly 25Mbps. Confirmed single machine. I believe it was all the same source 
address, but like 20 separate streams.

Happened to a guy on Saturday as well. Yet another 1.5Mbps 900MHz customer. 
Single PC directly to the radio. I was torching him and saw about 12Mbps coming 
from MS's 13.x. Then it would settle for a while and pick right back up again 
from LLNW at 6-8Mbps.

That guy opened a ticket and said he was getting less than 100kbps download 
speed and no web pages would load. He responded about an hour later and said 
everything was normal.

And yet another customer on Wednesday on PMP450 at 12Mbps tier was being sent 
over 30Mbps.


On 7/19/2016 2:41 PM, Mike Hammett wrote:

  Were all CDNs sending way more than the pipe size or only LimeLight?

  Someone at Akamai sent out this message last week regarding a general 
increase in usage:


  =
  There were two major software updates that spanned Tuesday and
  Wednesday which are responsible for the increase you saw.
  =




  -
  Mike Hammett
  Intelligent Computing Solutions

  Midwest Internet Exchange

  The Brothers WISP






--

  From: "George Skorup" mailto:geo...@cbcast.com
  To: af@afmug.com
  Sent: Thursday, July 14, 2016 1:33:21 AM
  Subject: Re: [AFMUG] CDN overload

  I forgot about this. Yes. A little later in the day, I started to see a lot 
of 13.n.n.n sources. Microsoft. Yeah, update Tuesday. Then the same customer 
would start receiving from LLNW. Then Akamai. And back to MS again. So it looks 
like they're *still* distributing updates across various CDNs. And believe me, 
it's not like they were all hitting this customer at once. One single CDN would 
try to send at 5-10X the customer's downlink MIR. Sometimes more. At one point 
I saw over 20Mbps for 5-10 minutes. I saw pretty much the same thing with about 
15 other customers that I looked at. And they were spread across 5-6 towers. 
Some directly licensed fed, others farther towards the edge.

  DDoS. CDN. Same thing. Or gorilla tactics at the very least. If the customer 
calls and says "none of my other shit works, your internet sucks" what are we 
supposed to do? Oh OK, here, we'll turn you up to 12Mbps and see what that 
does. Yeah screw that because now the CDN is sending at 40Mbps! They need to 
stop fucking with TCP already! And no, it doesn't matter where I put the 
policing/shaping. They still eat up bandwidth on our upstreams. Like you said 
before Ken, yeah, it just moves the problem somewhere else.


  On 7/13/2016 11:39 PM, Ken Hohhof wrote:

George, did you identify the application or content provider, or only the 
CDN?

I think I started getting hit with the same thing early yesterday 
afternoon.  At first I thought I was getting DDOS attacks.


From: George Skorup 
Sent: Tuesday, July 12, 2016 6:21 PM
To: af@afmug.com 
    Subject: Re: [AFMUG] CDN overload

Yup. LLNW.


On 7/12/2016 5:35 PM, Ken Hohhof wrote:

  I assume you torched the traffic and verified it is all coming from a 
particular CDN, not a random bunch of IPs as would be the case with BT.  Since 
this isn’t your first rodeo.

  From: George Skorup 
  Sent: Tuesday, July 12, 2016 5:31 PM
  To: af@afmug.com 
  Subject: Re: [AFMUG] CDN overload

  Because they dick with TCP.


  On 7/12/2016 5:23 PM, Eric Kuhnke wrote:

And why is it the fault of the CDN?  It could be a customer with a 
100-peer bittorrent session downloading 30GB of Ubuntu DVD ISOs.


On Tue, Jul 12, 2016 at 3:13 PM, George Skorup <geo...@cbcast.com> 
wrote:

  I have had it with these CDNs sending more traffic than the last mile 
can handle. Got a customer at 1.5Mbps on 900 FSK and they're sending to her at 
15Mbps. Of course the AP reports RF downlink overloaded.

Re: [AFMUG] CDN overload

2016-07-19 Thread George Skorup

Noop. As I said, Microsuck at one point was sending to a 1.5Mbps
customer at nearly 25Mbps. Confirmed single machine. I believe it was
all the same source address, but like 20 separate streams.

Happened to a guy on Saturday as well. Yet another 1.5Mbps 900MHz
customer. Single PC directly to the radio. I was torching him and saw
about 12Mbps coming from MS's 13.x. Then it would settle for a while and
pick right back up again from LLNW at 6-8Mbps.

That guy opened a ticket and said he was getting less than 100kbps
download speed and no web pages would load. He responded about an hour
later and said everything was normal.

And yet another customer on Wednesday on PMP450 at 12Mbps tier was being
sent over 30Mbps.

On 7/19/2016 2:41 PM, Mike Hammett wrote:

Were all CDNs sending way more than the pipe size or only LimeLight?

Someone at Akamai sent out this message last week regarding a general
increase in usage:

=
There were two major software updates that spanned Tuesday and
Wednesday which are responsible for the increase you saw.
=

<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>

*From: *"George Skorup" <geo...@cbcast.com>
*To: *af@afmug.com
*Sent: *Thursday, July 14, 2016 1:33:21 AM
*Subject: *Re: [AFMUG] CDN overload

I forgot about this. Yes. A little later in the day, I started to see
a lot of 13.n.n.n sources. Microsoft. Yeah, update Tuesday. Then the
same customer would start receiving from LLNW. Then Akamai. And back
to MS again. So it looks like they're *still* distributing updates
across various CDNs. And believe me, it's not like they were all
hitting this customer at once. One single CDN would try to send at
5-10X the customer's downlink MIR. Sometimes more. At one point I saw
over 20Mbps for 5-10 minutes. I saw pretty much the same thing with
about 15 other customers that I looked at. And they were spread across
5-6 towers. Some directly licensed fed, others farther towards the edge.

DDoS. CDN. Same thing. Or gorilla tactics at the very least. If the
customer calls and says "none of my other shit works, your internet
sucks" what are we supposed to do? Oh OK, here, we'll turn you up to
12Mbps and see what that does. Yeah screw that because now the CDN is
sending at 40Mbps! They need to stop fucking with TCP already! And no,
it doesn't matter where I put the policing/shaping. They still eat up
bandwidth on our upstreams. Like you said before Ken, yeah, it just
moves the problem somewhere else.

On 7/13/2016 11:39 PM, Ken Hohhof wrote:

George, did you identify the application or content provider, or
only the CDN?
I think I started getting hit with the same thing early yesterday
afternoon. At first I thought I was getting DDOS attacks.
*From:* George Skorup <mailto:geo...@cbcast.com>
*Sent:* Tuesday, July 12, 2016 6:21 PM
*To:* af@afmug.com <mailto:af@afmug.com>
*Subject:* Re: [AFMUG] CDN overload
Yup. LLNW.

On 7/12/2016 5:35 PM, Ken Hohhof wrote:

I assume you torched the traffic and verified it is all coming
from a particular CDN, not a random bunch of IPs as would be
the case with BT. Since this isn’t your first rodeo.
*From:* George Skorup <mailto:geo...@cbcast.com>
*Sent:* Tuesday, July 12, 2016 5:31 PM
*To:* af@afmug.com <mailto:af@afmug.com>
*Subject:* Re: [AFMUG] CDN overload
Because they dick with TCP.

On 7/12/2016 5:23 PM, Eric Kuhnke wrote:

And why is it the fault of the CDN? It could be a
customer with a 100-peer bittorrent session downloading
30GB of Ubuntu DVD ISOs.
On Tue, Jul 12, 2016 at 3:13 PM, George Skorup
<geo...@cbcast.com <mailto:geo...@cbcast.com>> wrote:

I have had it with these CDNs sending more traffic
than the last mile can handle. Got a customer at
1.5Mbps on 900 FSK and they're sending to her at
15Mbps. Of course the AP reports RF downlink overloaded.

Re: [AFMUG] CDN overload

2016-07-19 Thread Josh Luthman

Last Tuesday or Wednesday?  I had 1 customer with issues, but Reddit and
NANOG had some discussion.

Josh Luthman
Office: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373

On Jul 19, 2016 4:37 PM, "Mike Hammett" <af...@ics-il.net> wrote:

> I have seen mention of it outside of WISP land, but it didn't generate
> much conversation.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL>
> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
> <https://www.linkedin.com/company/intelligent-computing-solutions>
> <https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix>
> <https://www.linkedin.com/company/midwest-internet-exchange>
> <https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp>
>
>
> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> ----------
> *From: *"Paul Stewart" <p...@paulstewart.org>
> *To: *af@afmug.com
> *Sent: *Tuesday, July 19, 2016 3:33:39 PM
> *Subject: *Re: [AFMUG] CDN overload
>
> Good points Mike ….
>
> I’ve never seen a CDN do this kind of behaviour before and curious why
> some folks see this occurring …
>
> Paul
>
> On Jul 19, 2016, at 3:53 PM, Mike Hammett <af...@ics-il.net> wrote:
>
> Have you confirmed that these would be a single machine making a single
> request and getting this result?
>
> Could they have started up a bunch of machines at once and it just
> amplified the already "normal" things they do?
>
>
> The next time people are seeing this, could you please do some packet
> captures and look at what is coming down? Is it the same content in
> multiple streams? Is it multiple machines requesting it?
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL>
> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
> <https://www.linkedin.com/company/intelligent-computing-solutions>
> <https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix>
> <https://www.linkedin.com/company/midwest-internet-exchange>
> <https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp>
>
>
> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> --
> *From: *"George Skorup" <geo...@cbcast.com>
> *To: *af@afmug.com
> *Sent: *Tuesday, July 12, 2016 5:13:46 PM
> *Subject: *[AFMUG] CDN overload
>
> I have had it with these CDNs sending more traffic than the last mile
> can handle. Got a customer at 1.5Mbps on 900 FSK and they're sending to
> her at 15Mbps. Of course the AP reports RF downlink overloaded.
>
>
>
>

Re: [AFMUG] CDN overload

I have seen mention of it outside of WISP land, but it didn't generate much 
conversation. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




- Original Message -

From: "Paul Stewart" <p...@paulstewart.org> 
To: af@afmug.com 
Sent: Tuesday, July 19, 2016 3:33:39 PM 
Subject: Re: [AFMUG] CDN overload 

Good points Mike …. 


I’ve never seen a CDN do this kind of behaviour before and curious why some 
folks see this occurring … 


Paul 






On Jul 19, 2016, at 3:53 PM, Mike Hammett < af...@ics-il.net > wrote: 


Have you confirmed that these would be a single machine making a single request 
and getting this result? 

Could they have started up a bunch of machines at once and it just amplified 
the already "normal" things they do? 


The next time people are seeing this, could you please do some packet captures 
and look at what is coming down? Is it the same content in multiple streams? Is 
it multiple machines requesting it? 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




- Original Message -

From: "George Skorup" < geo...@cbcast.com > 
To: af@afmug.com 
Sent: Tuesday, July 12, 2016 5:13:46 PM 
Subject: [AFMUG] CDN overload 

I have had it with these CDNs sending more traffic than the last mile 
can handle. Got a customer at 1.5Mbps on 900 FSK and they're sending to 
her at 15Mbps. Of course the AP reports RF downlink overloaded.

Re: [AFMUG] CDN overload

2016-07-19 Thread Paul Stewart

Good points Mike ….

I’ve never seen a CDN do this kind of behaviour before and curious why some 
folks see this occurring …

Paul

> On Jul 19, 2016, at 3:53 PM, Mike Hammett <af...@ics-il.net> wrote:
> 
> Have you confirmed that these would be a single machine making a single 
> request and getting this result?
> 
> Could they have started up a bunch of machines at once and it just amplified 
> the already "normal" things they do?
> 
> 
> The next time people are seeing this, could you please do some packet 
> captures and look at what is coming down? Is it the same content in multiple 
> streams? Is it multiple machines requesting it?
> 
> 
> 
> -
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
>  <https://www.facebook.com/ICSIL> 
> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> 
> <https://www.linkedin.com/company/intelligent-computing-solutions> 
> <https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
>  <https://www.facebook.com/mdwestix> 
> <https://www.linkedin.com/company/midwest-internet-exchange> 
> <https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
>  <https://www.facebook.com/thebrotherswisp>
> 
> 
>  <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> From: "George Skorup" <geo...@cbcast.com <mailto:geo...@cbcast.com>>
> To: af@afmug.com <mailto:af@afmug.com>
> Sent: Tuesday, July 12, 2016 5:13:46 PM
> Subject: [AFMUG] CDN overload
> 
> I have had it with these CDNs sending more traffic than the last mile 
> can handle. Got a customer at 1.5Mbps on 900 FSK and they're sending to 
> her at 15Mbps. Of course the AP reports RF downlink overloaded.

Re: [AFMUG] CDN overload

Have you confirmed that these would be a single machine making a single request 
and getting this result? 

Could they have started up a bunch of machines at once and it just amplified 
the already "normal " things they do? 


The next time people are seeing this, could you please do some packet captures 
and look at what is coming down? Is it the same content in multiple streams? Is 
it multiple machines requesting it? 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




- Original Message -

From: "George Skorup" <geo...@cbcast.com> 
To: af@afmug.com 
Sent: Tuesday, July 12, 2016 5:13:46 PM 
Subject: [AFMUG] CDN overload 

I have had it with these CDNs sending more traffic than the last mile 
can handle. Got a customer at 1.5Mbps on 900 FSK and they're sending to 
her at 15Mbps. Of course the AP reports RF downlink overloaded.

Re: [AFMUG] CDN overload

Were all CDNs sending way more than the pipe size or only LimeLight? 

Someone at Akamai sent out this message last week regarding a general increase 
in usage: 


= 
There were two major software updates that spanned Tuesday and 
Wednesday which are responsible for the increase you saw. 
= 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




- Original Message -

From: "George Skorup" <geo...@cbcast.com> 
To: af@afmug.com 
Sent: Thursday, July 14, 2016 1:33:21 AM 
Subject: Re: [AFMUG] CDN overload 

I forgot about this. Yes. A little later in the day, I started to see a lot of 
13.n.n.n sources. Microsoft. Yeah, update Tuesday. Then the same customer would 
start receiving from LLNW. Then Akamai. And back to MS again. So it looks like 
they're *still* distributing updates across various CDNs. And believe me, it's 
not like they were all hitting this customer at once. One single CDN would try 
to send at 5-10X the customer's downlink MIR. Sometimes more. At one point I 
saw over 20Mbps for 5-10 minutes. I saw pretty much the same thing with about 
15 other customers that I looked at. And they were spread across 5-6 towers. 
Some directly licensed fed, others farther towards the edge. 

DDoS. CDN. Same thing. Or gorilla tactics at the very least. If the customer 
calls and says "none of my other shit works, your internet sucks" what are we 
supposed to do? Oh OK, here, we'll turn you up to 12Mbps and see what that 
does. Yeah screw that because now the CDN is sending at 40Mbps! They need to 
stop fucking with TCP already! And no, it doesn't matter where I put the 
policing/shaping. They still eat up bandwidth on our upstreams. Like you said 
before Ken, yeah, it just moves the problem somewhere else. 


On 7/13/2016 11:39 PM, Ken Hohhof wrote: 





George, did you identify the application or content provider, or only the CDN? 

I think I started getting hit with the same thing early yesterday afternoon. At 
first I thought I was getting DDOS attacks. 





From: George Skorup 
Sent: Tuesday, July 12, 2016 6:21 PM 
To: af@afmug.com 
Subject: Re: [AFMUG] CDN overload 

Yup. LLNW. 


On 7/12/2016 5:35 PM, Ken Hohhof wrote: 





I assume you torched the traffic and verified it is all coming from a 
particular CDN, not a random bunch of IPs as would be the case with BT. Since 
this isn’t your first rodeo. 




From: George Skorup 
Sent: Tuesday, July 12, 2016 5:31 PM 
To: af@afmug.com 
Subject: Re: [AFMUG] CDN overload 

Because they dick with TCP. 


On 7/12/2016 5:23 PM, Eric Kuhnke wrote: 



And why is it the fault of the CDN? It could be a customer with a 100-peer 
bittorrent session downloading 30GB of Ubuntu DVD ISOs. 



On Tue, Jul 12, 2016 at 3:13 PM, George Skorup < geo...@cbcast.com > wrote: 


I have had it with these CDNs sending more traffic than the last mile can 
handle. Got a customer at 1.5Mbps on 900 FSK and they're sending to her at 
15Mbps. Of course the AP reports RF downlink overloaded.

Re: [AFMUG] CDN overload

Reverting to Windows 7 isn't likely to be good advice. 

Turning off updates also isn't likely to be good advice. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




- Original Message -

From: "David" <m...@davidkunat.com> 
To: af@afmug.com 
Sent: Thursday, July 14, 2016 4:05:01 PM 
Subject: Re: [AFMUG] CDN overload 


We sent this the below instructions out to a section of customers who are on a 
small upstream and were killing on Update Tuesday and the day after. The 
metering option helps if they have PC's connected via wifi. Otherwise disabling 
the windows update service helps too :) A big disclaimer *at your own risk* was 
sent with the instructions We had about 7% of all users complain yesterday, 
and yes, all had windows 10. Most had just been converted over. We also sent 
instructions on how to revert back to windows 7 if they were in the 30 day 
period, and how to install a blocker to block windows 7 if anyone was 
interested. We got a lot of hits on that one and support was busy, but people 
were really happy for the help. 








Set Wi-Fi to Metered 
1. Connected 

Make sure you are connected to the Wi-Fi network that you wish to set as 
metered 2. All Settings 

3. Network & Internet 

4. Advanced options 

You may have to scroll down if you have a lot of networks. Assuming you are 
connected to the network you want to set as metered, hit Advanced options for 
the next step. 
5. Turn on 

Under Metered connection, you can toggle Set as metered connection to On . 

That's it. Now that Wi-Fi connection is set to metered, and the OS will 
throttle data usage for just the barebones like user initiated web browsing or 
email checking. 





Turn off Windows Updates in Windows 10 

This is the best way to make sure that windows does not preform excessive 
updates: 
You can do this using the Windows Update service. 
Go to Control Panel 
Click Administrative Tools 
Double click Services 
In the Services window, scroll down to Windows Update. 
To turn it off, right-click on the process, click on Properties and select 
Disabled. 
Click OK. 
Then right click on the Windows update service and Click "Stop" 
That will take care of Windows Updates not being installed on your machine. 





On Thu, Jul 14, 2016 at 12:29 PM, Mathew Howard < mhoward...@gmail.com > wrote: 



It seems like it's both. sometimes they're all to one IP, and sometimes it's a 
bunch of different IPs... but I guess that could be multiple computers trying 
to update. I don't think it's just 13.x.x.x sources either, it looks to me like 
they're using CDN's too - I saw what looked like the same kind of traffic 
coming from akamai IPs. 





On Thu, Jul 14, 2016 at 2:50 PM, Adam Moffett < dmmoff...@gmail.com > wrote: 




Are the many connections open to the same address or are they spread among 
multiple addresses? 

If they're opening dozens of connections to the same IP, then I think a fairly 
simple rule could cap the number of connections between any given src and dst 
pair. Set it to 10 or 15 and it shouldn't break any sane service. 

If it's multiple src IP's then put a cap on number of connections between 
13.x.x.x src and any of your IP's. 



-- Original Message -- 
From: "Craig Schmaderer" < cr...@skywaveconnect.com > 
To: " af@afmug.com " < af@afmug.com > 
Sent: 7/14/2016 3:25:21 PM 


Subject: Re: [AFMUG] CDN overload 





Same here, it didn’t take me very long to assume it was windows 10 updates, so 
the question is, is this going to be the norm? every update Tuesday going to 
f**k the network up? Does anyone have some good info or idea on what is going 
to happen? 



From: Af [mailto: af-boun...@afmug.com ] On Behalf Of Ken Hohhof 
Sent: Thursday, July 14, 2016 1:26 PM 
To: af@afmug.com 
Subject: Re: [AFMUG] CDN overload 




Wowzers. I guess I feel better knowing it’s not just me, this really had my 
head spinning, trying to figure out who was attacking me. 



But yeah, they changed something this week. It used to be pretty common to see 
4 parallel sessions, 93 is just crazy, hard to interpret it any other way than 
not playing nice and pushing all other traffic aside. 



But we all know ISPs are evil and Silicon Valley can do no wrong, so it must be 
our fault somehow. 








From: Mathew Howard 

Sent: Thursday, July 14, 2016 10:10 AM 

To: af 

Subject: Re: [AFMUG] CDN overload 




Just had another one call in... 93 active http sessions to 13.107.4.50, 
Microsoft obviously changed something with how they're doing update... I 
haven't ever seeing so many complaints generated from an update before. 




On Thu, Jul 14, 2016 at 8:13 AM, Nate Burke < n...@blastcomm.com > wrote: 



I just ran into this yesterday, took down an FSK AP that was running at 10mb 
Ethernet. A customer with 2 computers, MS Updates running in the background. 
Had about 50 http ses

Re: [AFMUG] CDN overload

2016-07-14 Thread That One Guy /sarcasm

A company could probably make a decent amount of revenue maintaining a CDN
IP database as a dynamic service offering

On Thu, Jul 14, 2016 at 3:29 PM, Mathew Howard <mhoward...@gmail.com> wrote:

> It seems like it's both. sometimes they're all to one IP, and sometimes
> it's a bunch of different IPs... but I guess that could be multiple
> computers trying to update. I don't think it's just 13.x.x.x sources
> either, it looks to me like they're using CDN's too - I saw what looked
> like the same kind of traffic coming from akamai IPs.
>
>
>
> On Thu, Jul 14, 2016 at 2:50 PM, Adam Moffett <dmmoff...@gmail.com> wrote:
>
>> Are the many connections open to the same address or are they spread
>> among multiple addresses?
>>
>> If they're opening dozens of connections to the same IP, then I think a
>> fairly simple rule could cap the number of connections between any
>> given src and dst pair.  Set it to 10 or 15 and it shouldn't break any sane
>> service.
>>
>> If it's multiple src IP's then put a cap on number of connections between
>> 13.x.x.x src and any of your IP's.
>>
>> -- Original Message --
>> From: "Craig Schmaderer" <cr...@skywaveconnect.com>
>> To: "af@afmug.com" <af@afmug.com>
>> Sent: 7/14/2016 3:25:21 PM
>> Subject: Re: [AFMUG] CDN overload
>>
>>
>> Same here, it didn’t take me very long to assume it was windows 10
>> updates, so the question is, is this going to be the norm?  every update
>> Tuesday going to f**k the network up?  Does anyone have some good info or
>> idea on what is going to happen?
>>
>>
>>
>> *From:* Af [mailto:af-boun...@afmug.com] *On Behalf Of *Ken Hohhof
>> *Sent:* Thursday, July 14, 2016 1:26 PM
>> *To:* af@afmug.com
>> *Subject:* Re: [AFMUG] CDN overload
>>
>>
>>
>> Wowzers.  I guess I feel better knowing it’s not just me, this really had
>> my head spinning, trying to figure out who was attacking me.
>>
>>
>>
>> But yeah, they changed something this week.  It used to be pretty common
>> to see 4 parallel sessions, 93 is just crazy, hard to interpret it any
>> other way than not playing nice and pushing all other traffic aside.
>>
>>
>>
>> But we all know ISPs are evil and Silicon Valley can do no wrong, so it
>> must be our fault somehow.
>>
>>
>>
>>
>>
>> *From:* Mathew Howard <mhoward...@gmail.com>
>>
>> *Sent:* Thursday, July 14, 2016 10:10 AM
>>
>> *To:* af <af@afmug.com>
>>
>> *Subject:* Re: [AFMUG] CDN overload
>>
>>
>>
>> Just had another one call in... 93 active http sessions to 13.107.4.50,
>> Microsoft obviously changed something with how they're doing update... I
>> haven't ever seeing so many complaints generated from an update before.
>>
>>
>>
>> On Thu, Jul 14, 2016 at 8:13 AM, Nate Burke <n...@blastcomm.com> wrote:
>>
>> I just ran into this yesterday, took down an FSK AP that was running at
>> 10mb Ethernet.  A customer with 2 computers, MS Updates running in the
>> background.  Had about 50 http sessions open to 13.x.x.x addresses.
>>
>>
>>
>> On 7/14/2016 7:50 AM, Adam Moffett wrote:
>>
>> Seems like they (MS) should look into promoting a multicast network for
>> distributing updates.
>>
>>
>>
>> Or simply limit automatic background updates to 256k (per destination).
>> If the user clicked the update button, sure get it to run as fast as
>> possible, but if it's in the background and they don't even know it's
>> happening then it ought to not matter how long the download takes.
>>
>>
>>
>> ...of course MS is not likely to care about my opinion on the matter.
>>
>>
>>
>>
>>
>> -- Original Message --
>>
>> From: "George Skorup" <geo...@cbcast.com>
>>
>> To: af@afmug.com
>>
>> Sent: 7/14/2016 2:33:21 AM
>>
>> Subject: Re: [AFMUG] CDN overload
>>
>>
>>
>> I forgot about this. Yes. A little later in the day, I started to see a
>> lot of 13.n.n.n sources. Microsoft. Yeah, update Tuesday. Then the same
>> customer would start receiving from LLNW. Then Akamai. And back to MS
>> again. So it looks like they're *still* distributing updates across various
>> CDNs. And believe me, it's not like they were all hitting this customer at
>> once. One single CDN would try to send at 5-10X the customer's downlink
>> MIR. Sometimes more. At one point I saw over 20Mbps for 5-10 minu

Re: [AFMUG] CDN overload

2016-07-14 Thread David

We sent this the below instructions out to a section of customers who are
on a small upstream and were killing on Update Tuesday and the day after.
The metering option helps if they have PC's connected via wifi. Otherwise
disabling the windows update service helps too :) A big disclaimer *at your
own risk* was sent with the instructions We had about 7% of all users
complain yesterday, and yes, all had windows 10. Most had just been
converted over. We also sent instructions on how to revert back to windows
7 if they were in the 30 day period, and how to install a blocker to block
windows 7 if anyone was interested. We got a lot of hits on that one and
support was busy, but people were really happy for the help.

Set Wi-Fi to Metered1. Connected

Make sure you are connected to the Wi-Fi network that you wish to set as
metered
2. All Settings

3. Network & Internet

4. Advanced options

You may have to scroll down if you have a lot of networks. Assuming you are
connected to the network you want to set as metered, hit *Advanced options* for
the next step.

5. Turn on

Under *Metered connection,* you can toggle *Set as metered connection* to
*On*.

That's it. Now that Wi-Fi connection is set to metered, and the OS will
throttle data usage for just the barebones like user initiated web browsing
or email checking.

Turn off Windows Updates in Windows 10

This is the best way to make sure that windows does not preform excessive
updates:

You can do this using the Windows Update service.

Go to Control Panel

Click Administrative Tools

Double click Services

In the *Services* window, scroll down to Windows Update.

To turn it off, right-click on the process, click on Properties and select
Disabled.

Click OK.

Then right click on the Windows update service and Click "Stop"

That will take care of Windows Updates not being installed on your machine.

On Thu, Jul 14, 2016 at 12:29 PM, Mathew Howard <mhoward...@gmail.com>
wrote:

> It seems like it's both. sometimes they're all to one IP, and sometimes
> it's a bunch of different IPs... but I guess that could be multiple
> computers trying to update. I don't think it's just 13.x.x.x sources
> either, it looks to me like they're using CDN's too - I saw what looked
> like the same kind of traffic coming from akamai IPs.
>
>
>
> On Thu, Jul 14, 2016 at 2:50 PM, Adam Moffett <dmmoff...@gmail.com> wrote:
>
>> Are the many connections open to the same address or are they spread
>> among multiple addresses?
>>
>> If they're opening dozens of connections to the same IP, then I think a
>> fairly simple rule could cap the number of connections between any
>> given src and dst pair.  Set it to 10 or 15 and it shouldn't break any sane
>> service.
>>
>> If it's multiple src IP's then put a cap on number of connections between
>> 13.x.x.x src and any of your IP's.
>>
>> -- Original Message --
>> From: "Craig Schmaderer" <cr...@skywaveconnect.com>
>> To: "af@afmug.com" <af@afmug.com>
>> Sent: 7/14/2016 3:25:21 PM
>> Subject: Re: [AFMUG] CDN overload
>>
>>
>> Same here, it didn’t take me very long to assume it was windows 10
>> updates, so the question is, is this going to be the norm?  every update
>> Tuesday going to f**k the network up?  Does anyone have some good info or
>> idea on what is going to happen?
>>
>>
>>
>> *From:* Af [mailto:af-boun...@afmug.com] *On Behalf Of *Ken Hohhof
>> *Sent:* Thursday, July 14, 2016 1:26 PM
>> *To:* af@afmug.com
>> *Subject:* Re: [AFMUG] CDN overload
>>
>>
>>
>> Wowzers.  I guess I feel better knowing it’s not just me, this really had
>> my head spinning, trying to figure out who was attacking me.
>>
>>
>>
>> But yeah, they changed something this week.  It used to be pretty common
>> to see 4 parallel sessions, 93 is just crazy, hard to interpret it any
>> other way than not playing nice and pushing all other traffic aside.
>>
>>
>>
>> But we all know ISPs are evil and Silicon Valley can do no wrong, so it
>> must be our fault somehow.
>>
>>
>>
>>
>>
>> *From:* Mathew Howard <mhoward...@gmail.com>
>>
>> *Sent:* Thursday, July 14, 2016 10:10 AM
>>
>> *To:* af <af@afmug.com>
>>
>> *Subject:* Re: [AFMUG] CDN overload
>>
>>
>>
>> Just had another one call in... 93 active http sessions to 13.107.4.50,
>> Microsoft obviously changed something with how they're doing update... I
>> haven't ever seeing so many complaints generated from an update before.
>>
>>
>>
>> On Thu, Jul 14, 2016 at 8:13 AM, Nate Burke <n...@blastcomm.com> wro

Re: [AFMUG] CDN overload

It seems like it's both. sometimes they're all to one IP, and sometimes
it's a bunch of different IPs... but I guess that could be multiple
computers trying to update. I don't think it's just 13.x.x.x sources
either, it looks to me like they're using CDN's too - I saw what looked
like the same kind of traffic coming from akamai IPs.



On Thu, Jul 14, 2016 at 2:50 PM, Adam Moffett <dmmoff...@gmail.com> wrote:

> Are the many connections open to the same address or are they spread among
> multiple addresses?
>
> If they're opening dozens of connections to the same IP, then I think a
> fairly simple rule could cap the number of connections between any
> given src and dst pair.  Set it to 10 or 15 and it shouldn't break any sane
> service.
>
> If it's multiple src IP's then put a cap on number of connections between
> 13.x.x.x src and any of your IP's.
>
> -- Original Message --
> From: "Craig Schmaderer" <cr...@skywaveconnect.com>
> To: "af@afmug.com" <af@afmug.com>
> Sent: 7/14/2016 3:25:21 PM
> Subject: Re: [AFMUG] CDN overload
>
>
> Same here, it didn’t take me very long to assume it was windows 10
> updates, so the question is, is this going to be the norm?  every update
> Tuesday going to f**k the network up?  Does anyone have some good info or
> idea on what is going to happen?
>
>
>
> *From:* Af [mailto:af-boun...@afmug.com] *On Behalf Of *Ken Hohhof
> *Sent:* Thursday, July 14, 2016 1:26 PM
> *To:* af@afmug.com
> *Subject:* Re: [AFMUG] CDN overload
>
>
>
> Wowzers.  I guess I feel better knowing it’s not just me, this really had
> my head spinning, trying to figure out who was attacking me.
>
>
>
> But yeah, they changed something this week.  It used to be pretty common
> to see 4 parallel sessions, 93 is just crazy, hard to interpret it any
> other way than not playing nice and pushing all other traffic aside.
>
>
>
> But we all know ISPs are evil and Silicon Valley can do no wrong, so it
> must be our fault somehow.
>
>
>
>
>
> *From:* Mathew Howard <mhoward...@gmail.com>
>
> *Sent:* Thursday, July 14, 2016 10:10 AM
>
> *To:* af <af@afmug.com>
>
> *Subject:* Re: [AFMUG] CDN overload
>
>
>
> Just had another one call in... 93 active http sessions to 13.107.4.50,
> Microsoft obviously changed something with how they're doing update... I
> haven't ever seeing so many complaints generated from an update before.
>
>
>
> On Thu, Jul 14, 2016 at 8:13 AM, Nate Burke <n...@blastcomm.com> wrote:
>
> I just ran into this yesterday, took down an FSK AP that was running at
> 10mb Ethernet.  A customer with 2 computers, MS Updates running in the
> background.  Had about 50 http sessions open to 13.x.x.x addresses.
>
>
>
> On 7/14/2016 7:50 AM, Adam Moffett wrote:
>
> Seems like they (MS) should look into promoting a multicast network for
> distributing updates.
>
>
>
> Or simply limit automatic background updates to 256k (per destination).
> If the user clicked the update button, sure get it to run as fast as
> possible, but if it's in the background and they don't even know it's
> happening then it ought to not matter how long the download takes.
>
>
>
> ...of course MS is not likely to care about my opinion on the matter.
>
>
>
>
>
> -- Original Message --
>
> From: "George Skorup" <geo...@cbcast.com>
>
> To: af@afmug.com
>
> Sent: 7/14/2016 2:33:21 AM
>
> Subject: Re: [AFMUG] CDN overload
>
>
>
> I forgot about this. Yes. A little later in the day, I started to see a
> lot of 13.n.n.n sources. Microsoft. Yeah, update Tuesday. Then the same
> customer would start receiving from LLNW. Then Akamai. And back to MS
> again. So it looks like they're *still* distributing updates across various
> CDNs. And believe me, it's not like they were all hitting this customer at
> once. One single CDN would try to send at 5-10X the customer's downlink
> MIR. Sometimes more. At one point I saw over 20Mbps for 5-10 minutes. I saw
> pretty much the same thing with about 15 other customers that I looked at.
> And they were spread across 5-6 towers. Some directly licensed fed, others
> farther towards the edge.
>
> DDoS. CDN. Same thing. Or gorilla tactics at the very least. If the
> customer calls and says "none of my other shit works, your internet sucks"
> what are we supposed to do? Oh OK, here, we'll turn you up to 12Mbps and
> see what that does. Yeah screw that because now the CDN is sending at
> 40Mbps! They need to stop fucking with TCP already! And no, it doesn't
> matter where I put the policing/shaping. They still eat up bandwidth on our
> upstreams. Like y

Re: [AFMUG] CDN overload

2016-07-14 Thread Adam Moffett

Are the many connections open to the same address or are they spread 
among multiple addresses?


If they're opening dozens of connections to the same IP, then I think a 
fairly simple rule could cap the number of connections between any given 
src and dst pair.  Set it to 10 or 15 and it shouldn't break any sane 
service.


If it's multiple src IP's then put a cap on number of connections 
between 13.x.x.x src and any of your IP's.


-- Original Message --
From: "Craig Schmaderer" <cr...@skywaveconnect.com>
To: "af@afmug.com" <af@afmug.com>
Sent: 7/14/2016 3:25:21 PM
Subject: Re: [AFMUG] CDN overload

Same here, it didn’t take me very long to assume it was windows 10 
updates, so the question is, is this going to be the norm?  every 
update Tuesday going to f**k the network up?  Does anyone have some 
good info or idea on what is going to happen?




From: Af [mailto:af-boun...@afmug.com] On Behalf Of Ken Hohhof
Sent: Thursday, July 14, 2016 1:26 PM
To:af@afmug.com
Subject: Re: [AFMUG] CDN overload



Wowzers.  I guess I feel better knowing it’s not just me, this really 
had my head spinning, trying to figure out who was attacking me.




But yeah, they changed something this week.  It used to be pretty 
common to see 4 parallel sessions, 93 is just crazy, hard to interpret 
it any other way than not playing nice and pushing all other traffic 
aside.




But we all know ISPs are evil and Silicon Valley can do no wrong, so it 
must be our fault somehow.






From:Mathew Howard

Sent: Thursday, July 14, 2016 10:10 AM

To:af

Subject: Re: [AFMUG] CDN overload



Just had another one call in... 93 active http sessions to 13.107.4.50, 
Microsoft obviously changed something with how they're doing update... 
I haven't ever seeing so many complaints generated from an update 
before.




On Thu, Jul 14, 2016 at 8:13 AM, Nate Burke <n...@blastcomm.com> wrote:

I just ran into this yesterday, took down an FSK AP that was running 
at 10mb Ethernet.  A customer with 2 computers, MS Updates running in 
the background.  Had about 50 http sessions open to 13.x.x.x 
addresses.




On 7/14/2016 7:50 AM, Adam Moffett wrote:

Seems like they (MS) should look into promoting a multicast network 
for distributing updates.




Or simply limit automatic background updates to 256k (per 
destination).  If the user clicked the update button, sure get it to 
run as fast as possible, but if it's in the background and they don't 
even know it's happening then it ought to not matter how long the 
download takes.




...of course MS is not likely to care about my opinion on the matter.





-- Original Message --

From: "George Skorup" <geo...@cbcast.com>

To: af@afmug.com

Sent: 7/14/2016 2:33:21 AM

Subject: Re: [AFMUG] CDN overload



I forgot about this. Yes. A little later in the day, I started to 
see a lot of 13.n.n.n sources. Microsoft. Yeah, update Tuesday. Then 
the same customer would start receiving from LLNW. Then Akamai. And 
back to MS again. So it looks like they're *still* distributing 
updates across various CDNs. And believe me, it's not like they were 
all hitting this customer at once. One single CDN would try to send 
at 5-10X the customer's downlink MIR. Sometimes more. At one point I 
saw over 20Mbps for 5-10 minutes. I saw pretty much the same thing 
with about 15 other customers that I looked at. And they were spread 
across 5-6 towers. Some directly licensed fed, others farther 
towards the edge.


DDoS. CDN. Same thing. Or gorilla tactics at the very least. If the 
customer calls and says "none of my other shit works, your internet 
sucks" what are we supposed to do? Oh OK, here, we'll turn you up to 
12Mbps and see what that does. Yeah screw that because now the CDN 
is sending at 40Mbps! They need to stop fucking with TCP already! 
And no, it doesn't matter where I put the policing/shaping. They 
still eat up bandwidth on our upstreams. Like you said before Ken, 
yeah, it just moves the problem somewhere else.


On 7/13/2016 11:39 PM, Ken Hohhof wrote:

George, did you identify the application or content provider, or 
only the CDN?




I think I started getting hit with the same thing early yesterday 
afternoon.  At first I thought I was getting DDOS attacks.






From:George Skorup

Sent: Tuesday, July 12, 2016 6:21 PM

To:af@afmug.com

Subject: Re: [AFMUG] CDN overload



Yup. LLNW.

On 7/12/2016 5:35 PM, Ken Hohhof wrote:

I assume you torched the traffic and verified it is all coming 
from a particular CDN, not a random bunch of IPs as would be the 
case with BT.  Since this isn’t your first rodeo.




From:George Skorup

Sent: Tuesday, July 12, 2016 5:31 PM

To:af@afmug.com

Subject: Re: [AFMUG] CDN overload



Because they dick with TCP.

On 7/12/2016 5:23 PM, Eric Kuhnke wrote:

And why is it the fault of the CDN?  It could be a customer with 
a 100-peer bittorrent session downloading 30GB of Ubuntu DVD 
ISOs.




On

Re: [AFMUG] CDN overload

2016-07-14 Thread George Skorup

It really looked to me like they were waiting for 100% packet loss over 
1-2 minutes before they'd back off. They pull this shit again and I will 
drop 13.0.0.0/8 at the borders. And I don't care what it breaks.


On 7/14/2016 2:25 PM, Craig Schmaderer wrote:


Same here, it didn’t take me very long to assume it was windows 10 
updates, so the question is, is this going to be the norm?  every 
update Tuesday going to f**k the network up? Does anyone have some 
good info or idea on what is going to happen?


*From:*Af [mailto:af-boun...@afmug.com] *On Behalf Of *Ken Hohhof
*Sent:* Thursday, July 14, 2016 1:26 PM
*To:* af@afmug.com
*Subject:* Re: [AFMUG] CDN overload

Wowzers. I guess I feel better knowing it’s not just me, this really 
had my head spinning, trying to figure out who was attacking me.


But yeah, they changed something this week.  It used to be pretty 
common to see 4 parallel sessions, 93 is just crazy, hard to interpret 
it any other way than not playing nice and pushing all other traffic 
aside.


But we all know ISPs are evil and Silicon Valley can do no wrong, so 
it must be our fault somehow.


*From:*Mathew Howard <mailto:mhoward...@gmail.com>

*Sent:*Thursday, July 14, 2016 10:10 AM

*To:*af <mailto:af@afmug.com>

*Subject:*Re: [AFMUG] CDN overload

Just had another one call in... 93 active http sessions to 
13.107.4.50, Microsoft obviously changed something with how they're 
doing update... I haven't ever seeing so many complaints generated 
from an update before.


On Thu, Jul 14, 2016 at 8:13 AM, Nate Burke <n...@blastcomm.com 
<mailto:n...@blastcomm.com>> wrote:


I just ran into this yesterday, took down an FSK AP that was
running at 10mb Ethernet.  A customer with 2 computers, MS Updates
running in the background.  Had about 50 http sessions open to
13.x.x.x addresses.

On 7/14/2016 7:50 AM, Adam Moffett wrote:

Seems like they (MS) should look into promoting a multicast
network for distributing updates.

Or simply limit automatic background updates to 256k (per
destination).  If the user clicked the update button, sure get
it to run as fast as possible, but if it's in the background
and they don't even know it's happening then it ought to not
matter how long the download takes.

...of course MS is not likely to care about my opinion on the
matter.

-- Original Message --

From: "George Skorup" <geo...@cbcast.com
<mailto:geo...@cbcast.com>>

To: af@afmug.com <mailto:af@afmug.com>

    Sent: 7/14/2016 2:33:21 AM

Subject: Re: [AFMUG] CDN overload

I forgot about this. Yes. A little later in the day, I
started to see a lot of 13.n.n.n sources. Microsoft. Yeah,
update Tuesday. Then the same customer would start
receiving from LLNW. Then Akamai. And back to MS again. So
it looks like they're *still* distributing updates across
various CDNs. And believe me, it's not like they were all
hitting this customer at once. One single CDN would try to
send at 5-10X the customer's downlink MIR. Sometimes more.
At one point I saw over 20Mbps for 5-10 minutes. I saw
pretty much the same thing with about 15 other customers
that I looked at. And they were spread across 5-6 towers.
Some directly licensed fed, others farther towards the edge.

DDoS. CDN. Same thing. Or gorilla tactics at the very
least. If the customer calls and says "none of my other
shit works, your internet sucks" what are we supposed to
do? Oh OK, here, we'll turn you up to 12Mbps and see what
that does. Yeah screw that because now the CDN is sending
at 40Mbps! They need to stop fucking with TCP already! And
no, it doesn't matter where I put the policing/shaping.
They still eat up bandwidth on our upstreams. Like you
said before Ken, yeah, it just moves the problem somewhere
else.

On 7/13/2016 11:39 PM, Ken Hohhof wrote:

George, did you identify the application or content
provider, or only the CDN?

I think I started getting hit with the same thing
early yesterday afternoon.  At first I thought I was
getting DDOS attacks.

*From:*George Skorup <mailto:geo...@cbcast.com>

*Sent:*Tuesday, July 12, 2016 6:21 PM

*To:*af@afmug.com <mailto:af@afmug.com>

*Subject:*Re: [AFMUG] CDN overload

Yup. LLNW.

On 7/12/2016 5:35 PM, Ken Hohhof wrote:

I assume you torched the traffic and verified it
is all coming from a particular CDN, not a ra

Re: [AFMUG] CDN overload

2016-07-14 Thread Craig Schmaderer

Same here, it didn’t take me very long to assume it was windows 10 updates, so 
the question is, is this going to be the norm?  every update Tuesday going to 
f**k the network up?  Does anyone have some good info or idea on what is going 
to happen?

From: Af [mailto:af-boun...@afmug.com] On Behalf Of Ken Hohhof
Sent: Thursday, July 14, 2016 1:26 PM
To: af@afmug.com
Subject: Re: [AFMUG] CDN overload

Wowzers.  I guess I feel better knowing it’s not just me, this really had my 
head spinning, trying to figure out who was attacking me.

But yeah, they changed something this week.  It used to be pretty common to see 
4 parallel sessions, 93 is just crazy, hard to interpret it any other way than 
not playing nice and pushing all other traffic aside.

But we all know ISPs are evil and Silicon Valley can do no wrong, so it must be 
our fault somehow.

From: Mathew Howard<mailto:mhoward...@gmail.com>
Sent: Thursday, July 14, 2016 10:10 AM
To: af<mailto:af@afmug.com>
Subject: Re: [AFMUG] CDN overload

Just had another one call in... 93 active http sessions to 13.107.4.50, 
Microsoft obviously changed something with how they're doing update... I 
haven't ever seeing so many complaints generated from an update before.

On Thu, Jul 14, 2016 at 8:13 AM, Nate Burke 
<n...@blastcomm.com<mailto:n...@blastcomm.com>> wrote:
I just ran into this yesterday, took down an FSK AP that was running at 10mb 
Ethernet.  A customer with 2 computers, MS Updates running in the background.  
Had about 50 http sessions open to 13.x.x.x addresses.

On 7/14/2016 7:50 AM, Adam Moffett wrote:
Seems like they (MS) should look into promoting a multicast network for 
distributing updates.

Or simply limit automatic background updates to 256k (per destination).  If the 
user clicked the update button, sure get it to run as fast as possible, but if 
it's in the background and they don't even know it's happening then it ought to 
not matter how long the download takes.

...of course MS is not likely to care about my opinion on the matter.

-- Original Message --
From: "George Skorup" <geo...@cbcast.com<mailto:geo...@cbcast.com>>
To: af@afmug.com<mailto:af@afmug.com>
Sent: 7/14/2016 2:33:21 AM
Subject: Re: [AFMUG] CDN overload

I forgot about this. Yes. A little later in the day, I started to see a lot of 
13.n.n.n sources. Microsoft. Yeah, update Tuesday. Then the same customer would 
start receiving from LLNW. Then Akamai. And back to MS again. So it looks like 
they're *still* distributing updates across various CDNs. And believe me, it's 
not like they were all hitting this customer at once. One single CDN would try 
to send at 5-10X the customer's downlink MIR. Sometimes more. At one point I 
saw over 20Mbps for 5-10 minutes. I saw pretty much the same thing with about 
15 other customers that I looked at. And they were spread across 5-6 towers. 
Some directly licensed fed, others farther towards the edge.

DDoS. CDN. Same thing. Or gorilla tactics at the very least. If the customer 
calls and says "none of my other shit works, your internet sucks" what are we 
supposed to do? Oh OK, here, we'll turn you up to 12Mbps and see what that 
does. Yeah screw that because now the CDN is sending at 40Mbps! They need to 
stop fucking with TCP already! And no, it doesn't matter where I put the 
policing/shaping. They still eat up bandwidth on our upstreams. Like you said 
before Ken, yeah, it just moves the problem somewhere else.
On 7/13/2016 11:39 PM, Ken Hohhof wrote:
George, did you identify the application or content provider, or only the CDN?

I think I started getting hit with the same thing early yesterday afternoon.  
At first I thought I was getting DDOS attacks.

From: George Skorup<mailto:geo...@cbcast.com>
Sent: Tuesday, July 12, 2016 6:21 PM
To: af@afmug.com<mailto:af@afmug.com>
Subject: Re: [AFMUG] CDN overload

Yup. LLNW.
On 7/12/2016 5:35 PM, Ken Hohhof wrote:
I assume you torched the traffic and verified it is all coming from a 
particular CDN, not a random bunch of IPs as would be the case with BT.  Since 
this isn’t your first rodeo.

From: George Skorup<mailto:geo...@cbcast.com>
Sent: Tuesday, July 12, 2016 5:31 PM
To: af@afmug.com<mailto:af@afmug.com>
Subject: Re: [AFMUG] CDN overload

Because they dick with TCP.
On 7/12/2016 5:23 PM, Eric Kuhnke wrote:
And why is it the fault of the CDN?  It could be a customer with a 100-peer 
bittorrent session downloading 30GB of Ubuntu DVD ISOs.

On Tue, Jul 12, 2016 at 3:13 PM, George Skorup 
<geo...@cbcast.com<mailto:geo...@cbcast.com>> wrote:
I have had it with these CDNs sending more traffic than the last mile can 
handle. Got a customer at 1.5Mbps on 900 FSK and they're sending to her at 
15Mbps. Of course the AP reports RF downlink overloaded.

Re: [AFMUG] CDN overload

2016-07-14 Thread Ken Hohhof

Wowzers.  I guess I feel better knowing it’s not just me, this really had my 
head spinning, trying to figure out who was attacking me.

But yeah, they changed something this week.  It used to be pretty common to see 
4 parallel sessions, 93 is just crazy, hard to interpret it any other way than 
not playing nice and pushing all other traffic aside.

But we all know ISPs are evil and Silicon Valley can do no wrong, so it must be 
our fault somehow.


From: Mathew Howard 
Sent: Thursday, July 14, 2016 10:10 AM
To: af 
Subject: Re: [AFMUG] CDN overload

Just had another one call in... 93 active http sessions to 13.107.4.50, 
Microsoft obviously changed something with how they're doing update... I 
haven't ever seeing so many complaints generated from an update before.


On Thu, Jul 14, 2016 at 8:13 AM, Nate Burke <n...@blastcomm.com> wrote:

  I just ran into this yesterday, took down an FSK AP that was running at 10mb 
Ethernet.  A customer with 2 computers, MS Updates running in the background.  
Had about 50 http sessions open to 13.x.x.x addresses.  


  On 7/14/2016 7:50 AM, Adam Moffett wrote:

Seems like they (MS) should look into promoting a multicast network for 
distributing updates.

Or simply limit automatic background updates to 256k (per destination).  If 
the user clicked the update button, sure get it to run as fast as possible, but 
if it's in the background and they don't even know it's happening then it ought 
to not matter how long the download takes.

...of course MS is not likely to care about my opinion on the matter.


-- Original Message --
From: "George Skorup" <geo...@cbcast.com>
To: af@afmug.com
Sent: 7/14/2016 2:33:21 AM
    Subject: Re: [AFMUG] CDN overload

  I forgot about this. Yes. A little later in the day, I started to see a 
lot of 13.n.n.n sources. Microsoft. Yeah, update Tuesday. Then the same 
customer would start receiving from LLNW. Then Akamai. And back to MS again. So 
it looks like they're *still* distributing updates across various CDNs. And 
believe me, it's not like they were all hitting this customer at once. One 
single CDN would try to send at 5-10X the customer's downlink MIR. Sometimes 
more. At one point I saw over 20Mbps for 5-10 minutes. I saw pretty much the 
same thing with about 15 other customers that I looked at. And they were spread 
across 5-6 towers. Some directly licensed fed, others farther towards the edge.

  DDoS. CDN. Same thing. Or gorilla tactics at the very least. If the 
customer calls and says "none of my other shit works, your internet sucks" what 
are we supposed to do? Oh OK, here, we'll turn you up to 12Mbps and see what 
that does. Yeah screw that because now the CDN is sending at 40Mbps! They need 
to stop fucking with TCP already! And no, it doesn't matter where I put the 
policing/shaping. They still eat up bandwidth on our upstreams. Like you said 
before Ken, yeah, it just moves the problem somewhere else.


  On 7/13/2016 11:39 PM, Ken Hohhof wrote:

George, did you identify the application or content provider, or only 
the CDN?

I think I started getting hit with the same thing early yesterday 
afternoon.  At first I thought I was getting DDOS attacks.


From: George Skorup 
Sent: Tuesday, July 12, 2016 6:21 PM
To: af@afmug.com 
    Subject: Re: [AFMUG] CDN overload

Yup. LLNW.


On 7/12/2016 5:35 PM, Ken Hohhof wrote:

  I assume you torched the traffic and verified it is all coming from a 
particular CDN, not a random bunch of IPs as would be the case with BT.  Since 
this isn’t your first rodeo.

  From: George Skorup 
  Sent: Tuesday, July 12, 2016 5:31 PM
  To: af@afmug.com 
      Subject: Re: [AFMUG] CDN overload

  Because they dick with TCP.


  On 7/12/2016 5:23 PM, Eric Kuhnke wrote:

And why is it the fault of the CDN?  It could be a customer with a 
100-peer bittorrent session downloading 30GB of Ubuntu DVD ISOs.


On Tue, Jul 12, 2016 at 3:13 PM, George Skorup <geo...@cbcast.com> 
wrote:

  I have had it with these CDNs sending more traffic than the last 
mile can handle. Got a customer at 1.5Mbps on 900 FSK and they're sending to 
her at 15Mbps. Of course the AP reports RF downlink overloaded.

Re: [AFMUG] CDN overload

On most of the connections I've checked the traffic is coming from a
13.x.x.x IP (which is Microsoft), but some of them have been various CDN's
IP adresses as well... they all seem to be behaving the same though - lots
and lots of http connections to one IP.

On Thu, Jul 14, 2016 at 11:45 AM, That One Guy /sarcasm <
thatoneguyst...@gmail.com> wrote:

> that wouldnt be CDN traffic, lol unless there is a tech with a windows 10
> laptop connected at the CDN, that would be funny if thats what it boiled
> down to
>
> On Thu, Jul 14, 2016 at 11:33 AM, Bill Prince <part15...@gmail.com> wrote:
>
>> Everyone is aware (I hope) that M$ runs the Windows 10 updates like a
>> bittorrent? Default configuration is to "share" updates with all your
>> neighbors, both on your LAN, and on the internet. This includes the Windows
>> 10 upgrade, which is around 2 or 3 GB.
>>
>> You can turn off that behavior in the settings. Go to Settings->Update &
>> Security->Windows Update->Advanced Options->Choose how updates are
>> delivered.
>>
>> Then UNCHECK "PCs on my local network, and PCs on the Internet" (OR
>> rather) CHECK "PCs on my local network".
>>
>>
>> bp
>> <part15sbs{at}gmail{dot}com>
>>
>>
>> On 7/14/2016 5:50 AM, Adam Moffett wrote:
>>
>> Seems like they (MS) should look into promoting a multicast network for
>> distributing updates.
>>
>> Or simply limit automatic background updates to 256k (per destination).
>> If the user clicked the update button, sure get it to run as fast as
>> possible, but if it's in the background and they don't even know it's
>> happening then it ought to not matter how long the download takes.
>>
>> ...of course MS is not likely to care about my opinion on the matter.
>>
>>
>> -- Original Message --
>> From: "George Skorup" <geo...@cbcast.com>
>> To: af@afmug.com
>> Sent: 7/14/2016 2:33:21 AM
>> Subject: Re: [AFMUG] CDN overload
>>
>>
>> I forgot about this. Yes. A little later in the day, I started to see a
>> lot of 13.n.n.n sources. Microsoft. Yeah, update Tuesday. Then the same
>> customer would start receiving from LLNW. Then Akamai. And back to MS
>> again. So it looks like they're *still* distributing updates across various
>> CDNs. And believe me, it's not like they were all hitting this customer at
>> once. One single CDN would try to send at 5-10X the customer's downlink
>> MIR. Sometimes more. At one point I saw over 20Mbps for 5-10 minutes. I saw
>> pretty much the same thing with about 15 other customers that I looked at.
>> And they were spread across 5-6 towers. Some directly licensed fed, others
>> farther towards the edge.
>>
>> DDoS. CDN. Same thing. Or gorilla tactics at the very least. If the
>> customer calls and says "none of my other shit works, your internet sucks"
>> what are we supposed to do? Oh OK, here, we'll turn you up to 12Mbps and
>> see what that does. Yeah screw that because now the CDN is sending at
>> 40Mbps! They need to stop fucking with TCP already! And no, it doesn't
>> matter where I put the policing/shaping. They still eat up bandwidth on our
>> upstreams. Like you said before Ken, yeah, it just moves the problem
>> somewhere else.
>>
>> On 7/13/2016 11:39 PM, Ken Hohhof wrote:
>>
>> George, did you identify the application or content provider, or only the
>> CDN?
>>
>> I think I started getting hit with the same thing early yesterday
>> afternoon.  At first I thought I was getting DDOS attacks.
>>
>>
>> *From:* George Skorup <geo...@cbcast.com>
>> *Sent:* Tuesday, July 12, 2016 6:21 PM
>> *To:* af@afmug.com
>> *Subject:* Re: [AFMUG] CDN overload
>>
>> Yup. LLNW.
>>
>> On 7/12/2016 5:35 PM, Ken Hohhof wrote:
>>
>> I assume you torched the traffic and verified it is all coming from a
>> particular CDN, not a random bunch of IPs as would be the case with BT.
>> Since this isn’t your first rodeo.
>>
>> *From:* George Skorup <geo...@cbcast.com>
>> *Sent:* Tuesday, July 12, 2016 5:31 PM
>> *To:* af@afmug.com
>> *Subject:* Re: [AFMUG] CDN overload
>>
>> Because they dick with TCP.
>>
>> On 7/12/2016 5:23 PM, Eric Kuhnke wrote:
>>
>> And why is it the fault of the CDN?  It could be a customer with a
>> 100-peer bittorrent session downloading 30GB of Ubuntu DVD ISOs.
>>
>> On Tue, Jul 12, 2016 at 3:13 PM, George Skorup <geo...@cbcast.com> wrote:
>>
>>> I have had it with these CDNs sending more traffic than the last mile
>>> can handle. Got a customer at 1.5Mbps on 900 FSK and they're sending to her
>>> at 15Mbps. Of course the AP reports RF downlink overloaded.
>>>
>>
>>
>>
>>
>>
>>
>>
>
>
> --
> If you only see yourself as part of the team but you don't see your team
> as part of yourself you have already failed as part of the team.
>

Re: [AFMUG] CDN overload

2016-07-14 Thread That One Guy /sarcasm

that wouldnt be CDN traffic, lol unless there is a tech with a windows 10
laptop connected at the CDN, that would be funny if thats what it boiled
down to

On Thu, Jul 14, 2016 at 11:33 AM, Bill Prince <part15...@gmail.com> wrote:

> Everyone is aware (I hope) that M$ runs the Windows 10 updates like a
> bittorrent? Default configuration is to "share" updates with all your
> neighbors, both on your LAN, and on the internet. This includes the Windows
> 10 upgrade, which is around 2 or 3 GB.
>
> You can turn off that behavior in the settings. Go to Settings->Update &
> Security->Windows Update->Advanced Options->Choose how updates are
> delivered.
>
> Then UNCHECK "PCs on my local network, and PCs on the Internet" (OR
> rather) CHECK "PCs on my local network".
>
>
> bp
> <part15sbs{at}gmail{dot}com>
>
>
> On 7/14/2016 5:50 AM, Adam Moffett wrote:
>
> Seems like they (MS) should look into promoting a multicast network for
> distributing updates.
>
> Or simply limit automatic background updates to 256k (per destination).
> If the user clicked the update button, sure get it to run as fast as
> possible, but if it's in the background and they don't even know it's
> happening then it ought to not matter how long the download takes.
>
> ...of course MS is not likely to care about my opinion on the matter.
>
>
> -- Original Message --
> From: "George Skorup" <geo...@cbcast.com>
> To: af@afmug.com
> Sent: 7/14/2016 2:33:21 AM
> Subject: Re: [AFMUG] CDN overload
>
>
> I forgot about this. Yes. A little later in the day, I started to see a
> lot of 13.n.n.n sources. Microsoft. Yeah, update Tuesday. Then the same
> customer would start receiving from LLNW. Then Akamai. And back to MS
> again. So it looks like they're *still* distributing updates across various
> CDNs. And believe me, it's not like they were all hitting this customer at
> once. One single CDN would try to send at 5-10X the customer's downlink
> MIR. Sometimes more. At one point I saw over 20Mbps for 5-10 minutes. I saw
> pretty much the same thing with about 15 other customers that I looked at.
> And they were spread across 5-6 towers. Some directly licensed fed, others
> farther towards the edge.
>
> DDoS. CDN. Same thing. Or gorilla tactics at the very least. If the
> customer calls and says "none of my other shit works, your internet sucks"
> what are we supposed to do? Oh OK, here, we'll turn you up to 12Mbps and
> see what that does. Yeah screw that because now the CDN is sending at
> 40Mbps! They need to stop fucking with TCP already! And no, it doesn't
> matter where I put the policing/shaping. They still eat up bandwidth on our
> upstreams. Like you said before Ken, yeah, it just moves the problem
> somewhere else.
>
> On 7/13/2016 11:39 PM, Ken Hohhof wrote:
>
> George, did you identify the application or content provider, or only the
> CDN?
>
> I think I started getting hit with the same thing early yesterday
> afternoon.  At first I thought I was getting DDOS attacks.
>
>
> *From:* George Skorup <geo...@cbcast.com>
> *Sent:* Tuesday, July 12, 2016 6:21 PM
> *To:* af@afmug.com
> *Subject:* Re: [AFMUG] CDN overload
>
> Yup. LLNW.
>
> On 7/12/2016 5:35 PM, Ken Hohhof wrote:
>
> I assume you torched the traffic and verified it is all coming from a
> particular CDN, not a random bunch of IPs as would be the case with BT.
> Since this isn’t your first rodeo.
>
> *From:* George Skorup <geo...@cbcast.com>
> *Sent:* Tuesday, July 12, 2016 5:31 PM
> *To:* af@afmug.com
> *Subject:* Re: [AFMUG] CDN overload
>
> Because they dick with TCP.
>
> On 7/12/2016 5:23 PM, Eric Kuhnke wrote:
>
> And why is it the fault of the CDN?  It could be a customer with a
> 100-peer bittorrent session downloading 30GB of Ubuntu DVD ISOs.
>
> On Tue, Jul 12, 2016 at 3:13 PM, George Skorup <geo...@cbcast.com> wrote:
>
>> I have had it with these CDNs sending more traffic than the last mile can
>> handle. Got a customer at 1.5Mbps on 900 FSK and they're sending to her at
>> 15Mbps. Of course the AP reports RF downlink overloaded.
>>
>
>
>
>
>
>
>


-- 
If you only see yourself as part of the team but you don't see your team as
part of yourself you have already failed as part of the team.

Re: [AFMUG] CDN overload

2016-07-14 Thread Bill Prince

Everyone is aware (I hope) that M$ runs the Windows 10 updates like a 
bittorrent? Default configuration is to "share" updates with all your 
neighbors, both on your LAN, and on the internet. This includes the 
Windows 10 upgrade, which is around 2 or 3 GB.


You can turn off that behavior in the settings. Go to Settings->Update & 
Security->Windows Update->Advanced Options->Choose how updates are 
delivered.


Then UNCHECK "PCs on my local network, and PCs on the Internet" (OR 
rather) CHECK "PCs on my local network".



bp
<part15sbs{at}gmail{dot}com>

On 7/14/2016 5:50 AM, Adam Moffett wrote:
Seems like they (MS) should look into promoting a multicast network 
for distributing updates.
Or simply limit automatic background updates to 256k (per 
destination).  If the user clicked the update button, sure get it to 
run as fast as possible, but if it's in the background and they don't 
even know it's happening then it ought to not matter how long the 
download takes.

...of course MS is not likely to care about my opinion on the matter.
-- Original Message --
From: "George Skorup" <geo...@cbcast.com <mailto:geo...@cbcast.com>>
To: af@afmug.com <mailto:af@afmug.com>
Sent: 7/14/2016 2:33:21 AM
Subject: Re: [AFMUG] CDN overload
I forgot about this. Yes. A little later in the day, I started to see 
a lot of 13.n.n.n sources. Microsoft. Yeah, update Tuesday. Then the 
same customer would start receiving from LLNW. Then Akamai. And back 
to MS again. So it looks like they're *still* distributing updates 
across various CDNs. And believe me, it's not like they were all 
hitting this customer at once. One single CDN would try to send at 
5-10X the customer's downlink MIR. Sometimes more. At one point I saw 
over 20Mbps for 5-10 minutes. I saw pretty much the same thing with 
about 15 other customers that I looked at. And they were spread 
across 5-6 towers. Some directly licensed fed, others farther towards 
the edge.


DDoS. CDN. Same thing. Or gorilla tactics at the very least. If the 
customer calls and says "none of my other shit works, your internet 
sucks" what are we supposed to do? Oh OK, here, we'll turn you up to 
12Mbps and see what that does. Yeah screw that because now the CDN is 
sending at 40Mbps! They need to stop fucking with TCP already! And 
no, it doesn't matter where I put the policing/shaping. They still 
eat up bandwidth on our upstreams. Like you said before Ken, yeah, it 
just moves the problem somewhere else.


On 7/13/2016 11:39 PM, Ken Hohhof wrote:
George, did you identify the application or content provider, or 
only the CDN?
I think I started getting hit with the same thing early yesterday 
afternoon.  At first I thought I was getting DDOS attacks.

*From:* George Skorup <mailto:geo...@cbcast.com>
*Sent:* Tuesday, July 12, 2016 6:21 PM
*To:* af@afmug.com <mailto:af@afmug.com>
*Subject:* Re: [AFMUG] CDN overload
Yup. LLNW.

On 7/12/2016 5:35 PM, Ken Hohhof wrote:
I assume you torched the traffic and verified it is all coming from 
a particular CDN, not a random bunch of IPs as would be the case 
with BT.  Since this isn’t your first rodeo.

*From:* George Skorup <mailto:geo...@cbcast.com>
*Sent:* Tuesday, July 12, 2016 5:31 PM
*To:* af@afmug.com <mailto:af@afmug.com>
*Subject:* Re: [AFMUG] CDN overload
Because they dick with TCP.

On 7/12/2016 5:23 PM, Eric Kuhnke wrote:
And why is it the fault of the CDN?  It could be a customer with a 
100-peer bittorrent session downloading 30GB of Ubuntu DVD ISOs.
On Tue, Jul 12, 2016 at 3:13 PM, George Skorup <geo...@cbcast.com 
<mailto:geo...@cbcast.com>> wrote:


I have had it with these CDNs sending more traffic than the
last mile can handle. Got a customer at 1.5Mbps on 900 FSK and
they're sending to her at 15Mbps. Of course the AP reports RF
downlink overloaded.

Re: [AFMUG] CDN overload

Just had another one call in... 93 active http sessions to 13.107.4.50,
Microsoft obviously changed something with how they're doing update... I
haven't ever seeing so many complaints generated from an update before.

On Thu, Jul 14, 2016 at 8:13 AM, Nate Burke <n...@blastcomm.com> wrote:

> I just ran into this yesterday, took down an FSK AP that was running at
> 10mb Ethernet.  A customer with 2 computers, MS Updates running in the
> background.  Had about 50 http sessions open to 13.x.x.x addresses.
>
> On 7/14/2016 7:50 AM, Adam Moffett wrote:
>
> Seems like they (MS) should look into promoting a multicast network for
> distributing updates.
>
> Or simply limit automatic background updates to 256k (per destination).
> If the user clicked the update button, sure get it to run as fast as
> possible, but if it's in the background and they don't even know it's
> happening then it ought to not matter how long the download takes.
>
> ...of course MS is not likely to care about my opinion on the matter.
>
>
> -- Original Message --
> From: "George Skorup" <geo...@cbcast.com>
> To: af@afmug.com
> Sent: 7/14/2016 2:33:21 AM
> Subject: Re: [AFMUG] CDN overload
>
>
> I forgot about this. Yes. A little later in the day, I started to see a
> lot of 13.n.n.n sources. Microsoft. Yeah, update Tuesday. Then the same
> customer would start receiving from LLNW. Then Akamai. And back to MS
> again. So it looks like they're *still* distributing updates across various
> CDNs. And believe me, it's not like they were all hitting this customer at
> once. One single CDN would try to send at 5-10X the customer's downlink
> MIR. Sometimes more. At one point I saw over 20Mbps for 5-10 minutes. I saw
> pretty much the same thing with about 15 other customers that I looked at.
> And they were spread across 5-6 towers. Some directly licensed fed, others
> farther towards the edge.
>
> DDoS. CDN. Same thing. Or gorilla tactics at the very least. If the
> customer calls and says "none of my other shit works, your internet sucks"
> what are we supposed to do? Oh OK, here, we'll turn you up to 12Mbps and
> see what that does. Yeah screw that because now the CDN is sending at
> 40Mbps! They need to stop fucking with TCP already! And no, it doesn't
> matter where I put the policing/shaping. They still eat up bandwidth on our
> upstreams. Like you said before Ken, yeah, it just moves the problem
> somewhere else.
>
> On 7/13/2016 11:39 PM, Ken Hohhof wrote:
>
> George, did you identify the application or content provider, or only the
> CDN?
>
> I think I started getting hit with the same thing early yesterday
> afternoon.  At first I thought I was getting DDOS attacks.
>
>
> *From:* George Skorup <geo...@cbcast.com>
> *Sent:* Tuesday, July 12, 2016 6:21 PM
> *To:* af@afmug.com
> *Subject:* Re: [AFMUG] CDN overload
>
> Yup. LLNW.
>
> On 7/12/2016 5:35 PM, Ken Hohhof wrote:
>
> I assume you torched the traffic and verified it is all coming from a
> particular CDN, not a random bunch of IPs as would be the case with BT.
> Since this isn’t your first rodeo.
>
> *From:* George Skorup <geo...@cbcast.com>
> *Sent:* Tuesday, July 12, 2016 5:31 PM
> *To:* af@afmug.com
> *Subject:* Re: [AFMUG] CDN overload
>
> Because they dick with TCP.
>
> On 7/12/2016 5:23 PM, Eric Kuhnke wrote:
>
> And why is it the fault of the CDN?  It could be a customer with a
> 100-peer bittorrent session downloading 30GB of Ubuntu DVD ISOs.
>
> On Tue, Jul 12, 2016 at 3:13 PM, George Skorup <geo...@cbcast.com> wrote:
>
>> I have had it with these CDNs sending more traffic than the last mile can
>> handle. Got a customer at 1.5Mbps on 900 FSK and they're sending to her at
>> 15Mbps. Of course the AP reports RF downlink overloaded.
>>
>
>
>
>
>
>
>

Re: [AFMUG] CDN overload

We had several complaints about this yesterday too

On Thu, Jul 14, 2016 at 8:13 AM, Nate Burke <n...@blastcomm.com> wrote:

> I just ran into this yesterday, took down an FSK AP that was running at
> 10mb Ethernet.  A customer with 2 computers, MS Updates running in the
> background.  Had about 50 http sessions open to 13.x.x.x addresses.
>
> On 7/14/2016 7:50 AM, Adam Moffett wrote:
>
> Seems like they (MS) should look into promoting a multicast network for
> distributing updates.
>
> Or simply limit automatic background updates to 256k (per destination).
> If the user clicked the update button, sure get it to run as fast as
> possible, but if it's in the background and they don't even know it's
> happening then it ought to not matter how long the download takes.
>
> ...of course MS is not likely to care about my opinion on the matter.
>
>
> -- Original Message --
> From: "George Skorup" <geo...@cbcast.com>
> To: af@afmug.com
> Sent: 7/14/2016 2:33:21 AM
> Subject: Re: [AFMUG] CDN overload
>
>
> I forgot about this. Yes. A little later in the day, I started to see a
> lot of 13.n.n.n sources. Microsoft. Yeah, update Tuesday. Then the same
> customer would start receiving from LLNW. Then Akamai. And back to MS
> again. So it looks like they're *still* distributing updates across various
> CDNs. And believe me, it's not like they were all hitting this customer at
> once. One single CDN would try to send at 5-10X the customer's downlink
> MIR. Sometimes more. At one point I saw over 20Mbps for 5-10 minutes. I saw
> pretty much the same thing with about 15 other customers that I looked at.
> And they were spread across 5-6 towers. Some directly licensed fed, others
> farther towards the edge.
>
> DDoS. CDN. Same thing. Or gorilla tactics at the very least. If the
> customer calls and says "none of my other shit works, your internet sucks"
> what are we supposed to do? Oh OK, here, we'll turn you up to 12Mbps and
> see what that does. Yeah screw that because now the CDN is sending at
> 40Mbps! They need to stop fucking with TCP already! And no, it doesn't
> matter where I put the policing/shaping. They still eat up bandwidth on our
> upstreams. Like you said before Ken, yeah, it just moves the problem
> somewhere else.
>
> On 7/13/2016 11:39 PM, Ken Hohhof wrote:
>
> George, did you identify the application or content provider, or only the
> CDN?
>
> I think I started getting hit with the same thing early yesterday
> afternoon.  At first I thought I was getting DDOS attacks.
>
>
> *From:* George Skorup <geo...@cbcast.com>
> *Sent:* Tuesday, July 12, 2016 6:21 PM
> *To:* af@afmug.com
> *Subject:* Re: [AFMUG] CDN overload
>
> Yup. LLNW.
>
> On 7/12/2016 5:35 PM, Ken Hohhof wrote:
>
> I assume you torched the traffic and verified it is all coming from a
> particular CDN, not a random bunch of IPs as would be the case with BT.
> Since this isn’t your first rodeo.
>
> *From:* George Skorup <geo...@cbcast.com>
> *Sent:* Tuesday, July 12, 2016 5:31 PM
> *To:* af@afmug.com
> *Subject:* Re: [AFMUG] CDN overload
>
> Because they dick with TCP.
>
> On 7/12/2016 5:23 PM, Eric Kuhnke wrote:
>
> And why is it the fault of the CDN?  It could be a customer with a
> 100-peer bittorrent session downloading 30GB of Ubuntu DVD ISOs.
>
> On Tue, Jul 12, 2016 at 3:13 PM, George Skorup <geo...@cbcast.com> wrote:
>
>> I have had it with these CDNs sending more traffic than the last mile can
>> handle. Got a customer at 1.5Mbps on 900 FSK and they're sending to her at
>> 15Mbps. Of course the AP reports RF downlink overloaded.
>>
>
>
>
>
>
>
>

Re: [AFMUG] CDN overload

2016-07-14 Thread Nate Burke

I just ran into this yesterday, took down an FSK AP that was running at 
10mb Ethernet.  A customer with 2 computers, MS Updates running in the 
background.  Had about 50 http sessions open to 13.x.x.x addresses.


On 7/14/2016 7:50 AM, Adam Moffett wrote:
Seems like they (MS) should look into promoting a multicast network 
for distributing updates.
Or simply limit automatic background updates to 256k (per 
destination).  If the user clicked the update button, sure get it to 
run as fast as possible, but if it's in the background and they don't 
even know it's happening then it ought to not matter how long the 
download takes.

...of course MS is not likely to care about my opinion on the matter.
-- Original Message --
From: "George Skorup" <geo...@cbcast.com <mailto:geo...@cbcast.com>>
To: af@afmug.com <mailto:af@afmug.com>
Sent: 7/14/2016 2:33:21 AM
Subject: Re: [AFMUG] CDN overload
I forgot about this. Yes. A little later in the day, I started to see 
a lot of 13.n.n.n sources. Microsoft. Yeah, update Tuesday. Then the 
same customer would start receiving from LLNW. Then Akamai. And back 
to MS again. So it looks like they're *still* distributing updates 
across various CDNs. And believe me, it's not like they were all 
hitting this customer at once. One single CDN would try to send at 
5-10X the customer's downlink MIR. Sometimes more. At one point I saw 
over 20Mbps for 5-10 minutes. I saw pretty much the same thing with 
about 15 other customers that I looked at. And they were spread 
across 5-6 towers. Some directly licensed fed, others farther towards 
the edge.


DDoS. CDN. Same thing. Or gorilla tactics at the very least. If the 
customer calls and says "none of my other shit works, your internet 
sucks" what are we supposed to do? Oh OK, here, we'll turn you up to 
12Mbps and see what that does. Yeah screw that because now the CDN is 
sending at 40Mbps! They need to stop fucking with TCP already! And 
no, it doesn't matter where I put the policing/shaping. They still 
eat up bandwidth on our upstreams. Like you said before Ken, yeah, it 
just moves the problem somewhere else.


On 7/13/2016 11:39 PM, Ken Hohhof wrote:
George, did you identify the application or content provider, or 
only the CDN?
I think I started getting hit with the same thing early yesterday 
afternoon.  At first I thought I was getting DDOS attacks.

*From:* George Skorup <mailto:geo...@cbcast.com>
*Sent:* Tuesday, July 12, 2016 6:21 PM
*To:* af@afmug.com <mailto:af@afmug.com>
*Subject:* Re: [AFMUG] CDN overload
Yup. LLNW.

On 7/12/2016 5:35 PM, Ken Hohhof wrote:
I assume you torched the traffic and verified it is all coming from 
a particular CDN, not a random bunch of IPs as would be the case 
with BT.  Since this isn’t your first rodeo.

*From:* George Skorup <mailto:geo...@cbcast.com>
*Sent:* Tuesday, July 12, 2016 5:31 PM
*To:* af@afmug.com <mailto:af@afmug.com>
*Subject:* Re: [AFMUG] CDN overload
Because they dick with TCP.

On 7/12/2016 5:23 PM, Eric Kuhnke wrote:
And why is it the fault of the CDN?  It could be a customer with a 
100-peer bittorrent session downloading 30GB of Ubuntu DVD ISOs.
On Tue, Jul 12, 2016 at 3:13 PM, George Skorup <geo...@cbcast.com 
<mailto:geo...@cbcast.com>> wrote:


I have had it with these CDNs sending more traffic than the
last mile can handle. Got a customer at 1.5Mbps on 900 FSK and
they're sending to her at 15Mbps. Of course the AP reports RF
downlink overloaded.

Re: [AFMUG] CDN overload

2016-07-14 Thread Adam Moffett

Seems like they (MS) should look into promoting a multicast network for 
distributing updates.


Or simply limit automatic background updates to 256k (per destination).  
If the user clicked the update button, sure get it to run as fast as 
possible, but if it's in the background and they don't even know it's 
happening then it ought to not matter how long the download takes.


...of course MS is not likely to care about my opinion on the matter.


-- Original Message --
From: "George Skorup" <geo...@cbcast.com>
To: af@afmug.com
Sent: 7/14/2016 2:33:21 AM
Subject: Re: [AFMUG] CDN overload

I forgot about this. Yes. A little later in the day, I started to see a 
lot of 13.n.n.n sources. Microsoft. Yeah, update Tuesday. Then the same 
customer would start receiving from LLNW. Then Akamai. And back to MS 
again. So it looks like they're *still* distributing updates across 
various CDNs. And believe me, it's not like they were all hitting this 
customer at once. One single CDN would try to send at 5-10X the 
customer's downlink MIR. Sometimes more. At one point I saw over 20Mbps 
for 5-10 minutes. I saw pretty much the same thing with about 15 other 
customers that I looked at. And they were spread across 5-6 towers. 
Some directly licensed fed, others farther towards the edge.


DDoS. CDN. Same thing. Or gorilla tactics at the very least. If the 
customer calls and says "none of my other shit works, your internet 
sucks" what are we supposed to do? Oh OK, here, we'll turn you up to 
12Mbps and see what that does. Yeah screw that because now the CDN is 
sending at 40Mbps! They need to stop fucking with TCP already! And no, 
it doesn't matter where I put the policing/shaping. They still eat up 
bandwidth on our upstreams. Like you said before Ken, yeah, it just 
moves the problem somewhere else.


On 7/13/2016 11:39 PM, Ken Hohhof wrote:
George, did you identify the application or content provider, or only 
the CDN?


I think I started getting hit with the same thing early yesterday 
afternoon.  At first I thought I was getting DDOS attacks.



From:George Skorup
Sent: Tuesday, July 12, 2016 6:21 PM
To:af@afmug.com
Subject: Re: [AFMUG] CDN overload

Yup. LLNW.

On 7/12/2016 5:35 PM, Ken Hohhof wrote:
I assume you torched the traffic and verified it is all coming from a 
particular CDN, not a random bunch of IPs as would be the case with 
BT.  Since this isn’t your first rodeo.


From:George Skorup
Sent: Tuesday, July 12, 2016 5:31 PM
To:af@afmug.com
Subject: Re: [AFMUG] CDN overload

Because they dick with TCP.

On 7/12/2016 5:23 PM, Eric Kuhnke wrote:
And why is it the fault of the CDN?  It could be a customer with a 
100-peer bittorrent session downloading 30GB of Ubuntu DVD ISOs.


On Tue, Jul 12, 2016 at 3:13 PM, George Skorup <geo...@cbcast.com> 
wrote:
I have had it with these CDNs sending more traffic than the last 
mile can handle. Got a customer at 1.5Mbps on 900 FSK and they're 
sending to her at 15Mbps. Of course the AP reports RF downlink 
overloaded.

Re: [AFMUG] CDN overload

2016-07-14 Thread Josh Reynolds

George:
https://www.reddit.com/r/networking/comments/4sqf8k/any_one_else_getting_hammered_by_lime_light/

On Thu, Jul 14, 2016 at 1:33 AM, George Skorup <geo...@cbcast.com> wrote:
> I forgot about this. Yes. A little later in the day, I started to see a lot
> of 13.n.n.n sources. Microsoft. Yeah, update Tuesday. Then the same customer
> would start receiving from LLNW. Then Akamai. And back to MS again. So it
> looks like they're *still* distributing updates across various CDNs. And
> believe me, it's not like they were all hitting this customer at once. One
> single CDN would try to send at 5-10X the customer's downlink MIR. Sometimes
> more. At one point I saw over 20Mbps for 5-10 minutes. I saw pretty much the
> same thing with about 15 other customers that I looked at. And they were
> spread across 5-6 towers. Some directly licensed fed, others farther towards
> the edge.
>
> DDoS. CDN. Same thing. Or gorilla tactics at the very least. If the customer
> calls and says "none of my other shit works, your internet sucks" what are
> we supposed to do? Oh OK, here, we'll turn you up to 12Mbps and see what
> that does. Yeah screw that because now the CDN is sending at 40Mbps! They
> need to stop fucking with TCP already! And no, it doesn't matter where I put
> the policing/shaping. They still eat up bandwidth on our upstreams. Like you
> said before Ken, yeah, it just moves the problem somewhere else.
>
> On 7/13/2016 11:39 PM, Ken Hohhof wrote:
>
> George, did you identify the application or content provider, or only the
> CDN?
>
> I think I started getting hit with the same thing early yesterday afternoon.
> At first I thought I was getting DDOS attacks.
>
>
> From: George Skorup
> Sent: Tuesday, July 12, 2016 6:21 PM
> To: af@afmug.com
> Subject: Re: [AFMUG] CDN overload
>
> Yup. LLNW.
>
> On 7/12/2016 5:35 PM, Ken Hohhof wrote:
>
> I assume you torched the traffic and verified it is all coming from a
> particular CDN, not a random bunch of IPs as would be the case with BT.
> Since this isn’t your first rodeo.
>
> From: George Skorup
> Sent: Tuesday, July 12, 2016 5:31 PM
> To: af@afmug.com
> Subject: Re: [AFMUG] CDN overload
>
> Because they dick with TCP.
>
> On 7/12/2016 5:23 PM, Eric Kuhnke wrote:
>
> And why is it the fault of the CDN?  It could be a customer with a 100-peer
> bittorrent session downloading 30GB of Ubuntu DVD ISOs.
>
> On Tue, Jul 12, 2016 at 3:13 PM, George Skorup <geo...@cbcast.com> wrote:
>>
>> I have had it with these CDNs sending more traffic than the last mile can
>> handle. Got a customer at 1.5Mbps on 900 FSK and they're sending to her at
>> 15Mbps. Of course the AP reports RF downlink overloaded.
>
>
>
>
>
>

Re: [AFMUG] CDN overload

2016-07-14 Thread George Skorup

I forgot about this. Yes. A little later in the day, I started to see a 
lot of 13.n.n.n sources. Microsoft. Yeah, update Tuesday. Then the same 
customer would start receiving from LLNW. Then Akamai. And back to MS 
again. So it looks like they're *still* distributing updates across 
various CDNs. And believe me, it's not like they were all hitting this 
customer at once. One single CDN would try to send at 5-10X the 
customer's downlink MIR. Sometimes more. At one point I saw over 20Mbps 
for 5-10 minutes. I saw pretty much the same thing with about 15 other 
customers that I looked at. And they were spread across 5-6 towers. Some 
directly licensed fed, others farther towards the edge.


DDoS. CDN. Same thing. Or gorilla tactics at the very least. If the 
customer calls and says "none of my other shit works, your internet 
sucks" what are we supposed to do? Oh OK, here, we'll turn you up to 
12Mbps and see what that does. Yeah screw that because now the CDN is 
sending at 40Mbps! They need to stop fucking with TCP already! And no, 
it doesn't matter where I put the policing/shaping. They still eat up 
bandwidth on our upstreams. Like you said before Ken, yeah, it just 
moves the problem somewhere else.


On 7/13/2016 11:39 PM, Ken Hohhof wrote:
George, did you identify the application or content provider, or only 
the CDN?
I think I started getting hit with the same thing early yesterday 
afternoon.  At first I thought I was getting DDOS attacks.

*From:* George Skorup <mailto:geo...@cbcast.com>
*Sent:* Tuesday, July 12, 2016 6:21 PM
*To:* af@afmug.com <mailto:af@afmug.com>
*Subject:* Re: [AFMUG] CDN overload
Yup. LLNW.

On 7/12/2016 5:35 PM, Ken Hohhof wrote:
I assume you torched the traffic and verified it is all coming from a 
particular CDN, not a random bunch of IPs as would be the case with 
BT.  Since this isn’t your first rodeo.

*From:* George Skorup <mailto:geo...@cbcast.com>
*Sent:* Tuesday, July 12, 2016 5:31 PM
*To:* af@afmug.com <mailto:af@afmug.com>
*Subject:* Re: [AFMUG] CDN overload
Because they dick with TCP.

On 7/12/2016 5:23 PM, Eric Kuhnke wrote:
And why is it the fault of the CDN?  It could be a customer with a 
100-peer bittorrent session downloading 30GB of Ubuntu DVD ISOs.
On Tue, Jul 12, 2016 at 3:13 PM, George Skorup <geo...@cbcast.com 
<mailto:geo...@cbcast.com>> wrote:


I have had it with these CDNs sending more traffic than the last
mile can handle. Got a customer at 1.5Mbps on 900 FSK and
they're sending to her at 15Mbps. Of course the AP reports RF
downlink overloaded.

Re: [AFMUG] CDN overload

George, did you identify the application or content provider, or only the CDN?

I think I started getting hit with the same thing early yesterday afternoon.  
At first I thought I was getting DDOS attacks.

From: George Skorup 
Sent: Tuesday, July 12, 2016 6:21 PM
To: af@afmug.com 
Subject: Re: [AFMUG] CDN overload

Yup. LLNW.

On 7/12/2016 5:35 PM, Ken Hohhof wrote:

  I assume you torched the traffic and verified it is all coming from a 
particular CDN, not a random bunch of IPs as would be the case with BT.  Since 
this isn’t your first rodeo.

  From: George Skorup 
  Sent: Tuesday, July 12, 2016 5:31 PM
  To: af@afmug.com 
  Subject: Re: [AFMUG] CDN overload

  Because they dick with TCP.

  On 7/12/2016 5:23 PM, Eric Kuhnke wrote:

And why is it the fault of the CDN?  It could be a customer with a 100-peer 
bittorrent session downloading 30GB of Ubuntu DVD ISOs.

On Tue, Jul 12, 2016 at 3:13 PM, George Skorup <geo...@cbcast.com> wrote:

  I have had it with these CDNs sending more traffic than the last mile can 
handle. Got a customer at 1.5Mbps on 900 FSK and they're sending to her at 
15Mbps. Of course the AP reports RF downlink overloaded.

Re: [AFMUG] CDN overload

I'm sure the tower owner would quote me a price to add a prefab bldg.  Given 
what I pay for a 3 ft dish, I'm not sure I want to know the price for a 
bldg.  I have a DDB cabinet there now.

I have 2 sites on my network with air conditioned buildings, one used to be 
my DS3 feed point years ago, but when you connect to upstreams via 
microwave, you take it where you can get it.

-Original Message- 
From: Josh Reynolds

Sent: Wednesday, July 13, 2016 10:45 AM
To: af@afmug.com
Subject: Re: [AFMUG] CDN overload

Don't have a decently sized tower hut there with HVAC?

On Wed, Jul 13, 2016 at 10:43 AM, Ken Hohhof <af...@kwisp.com> wrote:

Tower.

-Original Message- From: Josh Reynolds
Sent: Wednesday, July 13, 2016 10:15 AM

To: af@afmug.com
Subject: Re: [AFMUG] CDN overload

Is your headend a shack? :)

On Wed, Jul 13, 2016 at 7:06 AM, Ken Hohhof <af...@kwisp.com> wrote:

I wish they made a box that didn't need a data center environment.  Not
looking forward to putting in an outdoor enclosure with HVAC.

-Original Message- From: Josh Reynolds
Sent: Tuesday, July 12, 2016 11:17 PM

To: af@afmug.com
Subject: Re: [AFMUG] CDN overload

Slight correction, you can virtually do whatever you want, you just
can't really block "legal" things, and have to make a good excuse for
"reasonable network management" if you do.

On Tue, Jul 12, 2016 at 11:15 PM, Josh Reynolds <j...@kyneticwifi.com>
wrote:

"limited viability of procera boxes in the open internet era"

You are terribly misinformed.

You can do whatever you want, you just have to put it in some fine
print somewhere. You also can't discriminate and limit a specific
service provider. For instance, you can't shape netflix and not hulu,
but if you wanted to limit each subscriber to "6Mbps Steaming Video",
there's no problem with that.

Procera boxes are INCREDIBLY useful, and not just for traffic shaping.
They also make excellent CGNAT boxes, can help substantially with
DoS/DDoS detection and DoS assistance mechanisms, and give you
excellent DPI into subscriber usage. Knowing what's in use on your
network per subscriber is also substantially helpful when trying to
help a customer with an issue.

support: "You're using all of your bandwidth"

customer: "No I'm not, the kids are in bed and we're not using the
wifi" (they all call it "the wifi" it seems like)

support: "I see 15Mbps of Steam updates going on right now"

customer: "BRB lemmie shut of my son's computer"

*waits for customer speedtest*

customer: "Hey that looks great, thank you VERY MUCH!"

support: "No problem sir/maam. Glad we could help!"

On Tue, Jul 12, 2016 at 10:44 PM, That One Guy /sarcasm
<thatoneguyst...@gmail.com> wrote:

sounded to me like this is a single IP (tcp/udp) connection saturation
scenario, without a serious L7 filter in play its gonna do what its
gonna
do. Powercode for example (historically, not sure about now without
procera)
only applied the cap on new ip connections, established maintained
whatever
it was originally. so if you started an unbroken stream at a 12mb 
burst,

that stream always hung out at 12mb, if your sustained was 3mb, new
streams
were limited to 3mb aggregate, but the 12mb stream prevailed as long as
it
was never considered "new". even when powercode was useful with the
throttling controls, before they threw away their primary benefit in 
the

bandwidth control area to sell their soul to procera with the real time
throttles they took away. with the limited viability of procera boxes 
in

the
open internet era, I can see where this would be a cluster f**k post 
12k

investment.

On Tue, Jul 12, 2016 at 9:44 PM, Josh Reynolds <j...@kyneticwifi.com>
wrote:

The policer at the network edge can be much more aggressive than a
"policer" on an embedded customer network device, and this prevents
that "15Mbps for a 900MHz customer" bandwidth from transitioning
across your backhauls / backbone... per customer.

Procera and other similar solutions can help, yes.

On Tue, Jul 12, 2016 at 8:02 PM, Ken Hohhof <af...@kwisp.com> wrote:
> How does it eliminate the problem, unless you use something like a
> Procera
> to selectively apply policing to the CDN stream, leaving the 
> customer

> some
> bandwidth for other traffic?
>
> From: Josh Reynolds
> Sent: Tuesday, July 12, 2016 7:22 PM
> To: af@afmug.com
> Subject: Re: [AFMUG] CDN overload
>
>
> Shaping/policing at the head end eliminates this problem, and clears
> >  >
> up
> your
> backbone.
>
> On Jul 12, 2016 7:06 PM, "Ken Hohhof" <af...@kwisp.com> wrote:
>>
>> When this happens it basically wipes out that customer’s Internet
>> except
>> for the CDN download, no matter where you do the rate limiting

Re: [AFMUG] CDN overload

2016-07-13 Thread Kurt Fankhauser

If the CDN is not delivering traffic following normal TCP guidelines then I
consider it Malicous traffic and you have rights to block it. Also +1 on
the Procera for diagnosing customer calling saying connection is slow, have
used it many times to determine an XBOX update was hogging entire
connection. I try to give the problem applications less bandwidth but
lately I am leaning towards trying to get more bandwidth to the customer to
eliminate these problems altogether.

On Wed, Jul 13, 2016 at 11:45 AM, Josh Reynolds <j...@kyneticwifi.com>
wrote:

> Don't have a decently sized tower hut there with HVAC?
>
> On Wed, Jul 13, 2016 at 10:43 AM, Ken Hohhof <af...@kwisp.com> wrote:
> > Tower.
> >
> > -Original Message- From: Josh Reynolds
> > Sent: Wednesday, July 13, 2016 10:15 AM
> >
> > To: af@afmug.com
> > Subject: Re: [AFMUG] CDN overload
> >
> > Is your headend a shack? :)
> >
> > On Wed, Jul 13, 2016 at 7:06 AM, Ken Hohhof <af...@kwisp.com> wrote:
> >>
> >> I wish they made a box that didn't need a data center environment.  Not
> >> looking forward to putting in an outdoor enclosure with HVAC.
> >>
> >> -----Original Message- From: Josh Reynolds
> >> Sent: Tuesday, July 12, 2016 11:17 PM
> >>
> >> To: af@afmug.com
> >> Subject: Re: [AFMUG] CDN overload
> >>
> >> Slight correction, you can virtually do whatever you want, you just
> >> can't really block "legal" things, and have to make a good excuse for
> >> "reasonable network management" if you do.
> >>
> >> On Tue, Jul 12, 2016 at 11:15 PM, Josh Reynolds <j...@kyneticwifi.com>
> >> wrote:
> >>>
> >>>
> >>> "limited viability of procera boxes in the open internet era"
> >>>
> >>> You are terribly misinformed.
> >>>
> >>> You can do whatever you want, you just have to put it in some fine
> >>> print somewhere. You also can't discriminate and limit a specific
> >>> service provider. For instance, you can't shape netflix and not hulu,
> >>> but if you wanted to limit each subscriber to "6Mbps Steaming Video",
> >>> there's no problem with that.
> >>>
> >>> Procera boxes are INCREDIBLY useful, and not just for traffic shaping.
> >>> They also make excellent CGNAT boxes, can help substantially with
> >>> DoS/DDoS detection and DoS assistance mechanisms, and give you
> >>> excellent DPI into subscriber usage. Knowing what's in use on your
> >>> network per subscriber is also substantially helpful when trying to
> >>> help a customer with an issue.
> >>>
> >>> support: "You're using all of your bandwidth"
> >>>
> >>> customer: "No I'm not, the kids are in bed and we're not using the
> >>> wifi" (they all call it "the wifi" it seems like)
> >>>
> >>> support: "I see 15Mbps of Steam updates going on right now"
> >>>
> >>> customer: "BRB lemmie shut of my son's computer"
> >>>
> >>> *waits for customer speedtest*
> >>>
> >>> customer: "Hey that looks great, thank you VERY MUCH!"
> >>>
> >>> support: "No problem sir/maam. Glad we could help!"
> >>>
> >>> On Tue, Jul 12, 2016 at 10:44 PM, That One Guy /sarcasm
> >>> <thatoneguyst...@gmail.com> wrote:
> >>>>
> >>>>
> >>>> sounded to me like this is a single IP (tcp/udp) connection saturation
> >>>> scenario, without a serious L7 filter in play its gonna do what its
> >>>> gonna
> >>>> do. Powercode for example (historically, not sure about now without
> >>>> procera)
> >>>> only applied the cap on new ip connections, established maintained
> >>>> whatever
> >>>> it was originally. so if you started an unbroken stream at a 12mb
> burst,
> >>>> that stream always hung out at 12mb, if your sustained was 3mb, new
> >>>> streams
> >>>> were limited to 3mb aggregate, but the 12mb stream prevailed as long
> as
> >>>> it
> >>>> was never considered "new". even when powercode was useful with the
> >>>> throttling controls, before they threw away their primary benefit in
> the
> >>>> bandwidth control area to sell their soul to procera with the real
> time

Re: [AFMUG] CDN overload

2016-07-13 Thread Josh Reynolds

Don't have a decently sized tower hut there with HVAC?

On Wed, Jul 13, 2016 at 10:43 AM, Ken Hohhof <af...@kwisp.com> wrote:
> Tower.
>
> -Original Message- From: Josh Reynolds
> Sent: Wednesday, July 13, 2016 10:15 AM
>
> To: af@afmug.com
> Subject: Re: [AFMUG] CDN overload
>
> Is your headend a shack? :)
>
> On Wed, Jul 13, 2016 at 7:06 AM, Ken Hohhof <af...@kwisp.com> wrote:
>>
>> I wish they made a box that didn't need a data center environment.  Not
>> looking forward to putting in an outdoor enclosure with HVAC.
>>
>> -Original Message- From: Josh Reynolds
>> Sent: Tuesday, July 12, 2016 11:17 PM
>>
>> To: af@afmug.com
>> Subject: Re: [AFMUG] CDN overload
>>
>> Slight correction, you can virtually do whatever you want, you just
>> can't really block "legal" things, and have to make a good excuse for
>> "reasonable network management" if you do.
>>
>> On Tue, Jul 12, 2016 at 11:15 PM, Josh Reynolds <j...@kyneticwifi.com>
>> wrote:
>>>
>>>
>>> "limited viability of procera boxes in the open internet era"
>>>
>>> You are terribly misinformed.
>>>
>>> You can do whatever you want, you just have to put it in some fine
>>> print somewhere. You also can't discriminate and limit a specific
>>> service provider. For instance, you can't shape netflix and not hulu,
>>> but if you wanted to limit each subscriber to "6Mbps Steaming Video",
>>> there's no problem with that.
>>>
>>> Procera boxes are INCREDIBLY useful, and not just for traffic shaping.
>>> They also make excellent CGNAT boxes, can help substantially with
>>> DoS/DDoS detection and DoS assistance mechanisms, and give you
>>> excellent DPI into subscriber usage. Knowing what's in use on your
>>> network per subscriber is also substantially helpful when trying to
>>> help a customer with an issue.
>>>
>>> support: "You're using all of your bandwidth"
>>>
>>> customer: "No I'm not, the kids are in bed and we're not using the
>>> wifi" (they all call it "the wifi" it seems like)
>>>
>>> support: "I see 15Mbps of Steam updates going on right now"
>>>
>>> customer: "BRB lemmie shut of my son's computer"
>>>
>>> *waits for customer speedtest*
>>>
>>> customer: "Hey that looks great, thank you VERY MUCH!"
>>>
>>> support: "No problem sir/maam. Glad we could help!"
>>>
>>> On Tue, Jul 12, 2016 at 10:44 PM, That One Guy /sarcasm
>>> <thatoneguyst...@gmail.com> wrote:
>>>>
>>>>
>>>> sounded to me like this is a single IP (tcp/udp) connection saturation
>>>> scenario, without a serious L7 filter in play its gonna do what its
>>>> gonna
>>>> do. Powercode for example (historically, not sure about now without
>>>> procera)
>>>> only applied the cap on new ip connections, established maintained
>>>> whatever
>>>> it was originally. so if you started an unbroken stream at a 12mb burst,
>>>> that stream always hung out at 12mb, if your sustained was 3mb, new
>>>> streams
>>>> were limited to 3mb aggregate, but the 12mb stream prevailed as long as
>>>> it
>>>> was never considered "new". even when powercode was useful with the
>>>> throttling controls, before they threw away their primary benefit in the
>>>> bandwidth control area to sell their soul to procera with the real time
>>>> throttles they took away. with the limited viability of procera boxes in
>>>> the
>>>> open internet era, I can see where this would be a cluster f**k post 12k
>>>> investment.
>>>>
>>>> On Tue, Jul 12, 2016 at 9:44 PM, Josh Reynolds <j...@kyneticwifi.com>
>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> The policer at the network edge can be much more aggressive than a
>>>>> "policer" on an embedded customer network device, and this prevents
>>>>> that "15Mbps for a 900MHz customer" bandwidth from transitioning
>>>>> across your backhauls / backbone... per customer.
>>>>>
>>>>> Procera and other similar solutions can help, yes.
>>>>>
>>>>> On Tue, Jul 12, 2016 at 8:02 PM, Ken Hohhof <af...@kwisp.com> wr

Re: [AFMUG] CDN overload

Tower.

-Original Message- 
From: Josh Reynolds

Sent: Wednesday, July 13, 2016 10:15 AM
To: af@afmug.com
Subject: Re: [AFMUG] CDN overload

Is your headend a shack? :)

On Wed, Jul 13, 2016 at 7:06 AM, Ken Hohhof <af...@kwisp.com> wrote:

I wish they made a box that didn't need a data center environment.  Not
looking forward to putting in an outdoor enclosure with HVAC.

-Original Message- From: Josh Reynolds
Sent: Tuesday, July 12, 2016 11:17 PM

To: af@afmug.com
Subject: Re: [AFMUG] CDN overload

Slight correction, you can virtually do whatever you want, you just
can't really block "legal" things, and have to make a good excuse for
"reasonable network management" if you do.

On Tue, Jul 12, 2016 at 11:15 PM, Josh Reynolds <j...@kyneticwifi.com>
wrote:

"limited viability of procera boxes in the open internet era"

You are terribly misinformed.

You can do whatever you want, you just have to put it in some fine
print somewhere. You also can't discriminate and limit a specific
service provider. For instance, you can't shape netflix and not hulu,
but if you wanted to limit each subscriber to "6Mbps Steaming Video",
there's no problem with that.

Procera boxes are INCREDIBLY useful, and not just for traffic shaping.
They also make excellent CGNAT boxes, can help substantially with
DoS/DDoS detection and DoS assistance mechanisms, and give you
excellent DPI into subscriber usage. Knowing what's in use on your
network per subscriber is also substantially helpful when trying to
help a customer with an issue.

support: "You're using all of your bandwidth"

customer: "No I'm not, the kids are in bed and we're not using the
wifi" (they all call it "the wifi" it seems like)

support: "I see 15Mbps of Steam updates going on right now"

customer: "BRB lemmie shut of my son's computer"

*waits for customer speedtest*

customer: "Hey that looks great, thank you VERY MUCH!"

support: "No problem sir/maam. Glad we could help!"

On Tue, Jul 12, 2016 at 10:44 PM, That One Guy /sarcasm
<thatoneguyst...@gmail.com> wrote:

sounded to me like this is a single IP (tcp/udp) connection saturation
scenario, without a serious L7 filter in play its gonna do what its 
gonna

do. Powercode for example (historically, not sure about now without
procera)
only applied the cap on new ip connections, established maintained
whatever
it was originally. so if you started an unbroken stream at a 12mb burst,
that stream always hung out at 12mb, if your sustained was 3mb, new
streams
were limited to 3mb aggregate, but the 12mb stream prevailed as long as
it
was never considered "new". even when powercode was useful with the
throttling controls, before they threw away their primary benefit in the
bandwidth control area to sell their soul to procera with the real time
throttles they took away. with the limited viability of procera boxes in
the
open internet era, I can see where this would be a cluster f**k post 12k
investment.

On Tue, Jul 12, 2016 at 9:44 PM, Josh Reynolds <j...@kyneticwifi.com>
wrote:

The policer at the network edge can be much more aggressive than a
"policer" on an embedded customer network device, and this prevents
that "15Mbps for a 900MHz customer" bandwidth from transitioning
across your backhauls / backbone... per customer.

Procera and other similar solutions can help, yes.

On Tue, Jul 12, 2016 at 8:02 PM, Ken Hohhof <af...@kwisp.com> wrote:
> How does it eliminate the problem, unless you use something like a
> Procera
> to selectively apply policing to the CDN stream, leaving the customer
> some
> bandwidth for other traffic?
>
> From: Josh Reynolds
> Sent: Tuesday, July 12, 2016 7:22 PM
> To: af@afmug.com
> Subject: Re: [AFMUG] CDN overload
>
>
> Shaping/policing at the head end eliminates this problem, and clears 
>  >

> up
> your
> backbone.
>
> On Jul 12, 2016 7:06 PM, "Ken Hohhof" <af...@kwisp.com> wrote:
>>
>> When this happens it basically wipes out that customer’s Internet
>> except
>> for the CDN download, no matter where you do the rate limiting.
>> Customer of
>> course assumes their ISP just sucks.  With a lot of education, you 
>>  >>

>> can
>> convince most of them it is actually an aggressive application >>
>> hogging
>> their
>> entire pipe and pushing all the other applications aside.  So I have
>> customers that whenever their VPN to work stops working, they yell
>> upstairs
>> at their kid didn’t I tell you to do your Xbox downloads after I go
>> >> to
>> bed?
>>
>> One view is this isn’t a problem, customer uses bad application, >>
>> feels
>> pain, learns not to do that.  But ever

Re: [AFMUG] CDN overload

2016-07-13 Thread Josh Reynolds

Is your headend a shack? :)

On Wed, Jul 13, 2016 at 7:06 AM, Ken Hohhof <af...@kwisp.com> wrote:
> I wish they made a box that didn't need a data center environment.  Not
> looking forward to putting in an outdoor enclosure with HVAC.
>
> -Original Message- From: Josh Reynolds
> Sent: Tuesday, July 12, 2016 11:17 PM
>
> To: af@afmug.com
> Subject: Re: [AFMUG] CDN overload
>
> Slight correction, you can virtually do whatever you want, you just
> can't really block "legal" things, and have to make a good excuse for
> "reasonable network management" if you do.
>
> On Tue, Jul 12, 2016 at 11:15 PM, Josh Reynolds <j...@kyneticwifi.com>
> wrote:
>>
>> "limited viability of procera boxes in the open internet era"
>>
>> You are terribly misinformed.
>>
>> You can do whatever you want, you just have to put it in some fine
>> print somewhere. You also can't discriminate and limit a specific
>> service provider. For instance, you can't shape netflix and not hulu,
>> but if you wanted to limit each subscriber to "6Mbps Steaming Video",
>> there's no problem with that.
>>
>> Procera boxes are INCREDIBLY useful, and not just for traffic shaping.
>> They also make excellent CGNAT boxes, can help substantially with
>> DoS/DDoS detection and DoS assistance mechanisms, and give you
>> excellent DPI into subscriber usage. Knowing what's in use on your
>> network per subscriber is also substantially helpful when trying to
>> help a customer with an issue.
>>
>> support: "You're using all of your bandwidth"
>>
>> customer: "No I'm not, the kids are in bed and we're not using the
>> wifi" (they all call it "the wifi" it seems like)
>>
>> support: "I see 15Mbps of Steam updates going on right now"
>>
>> customer: "BRB lemmie shut of my son's computer"
>>
>> *waits for customer speedtest*
>>
>> customer: "Hey that looks great, thank you VERY MUCH!"
>>
>> support: "No problem sir/maam. Glad we could help!"
>>
>> On Tue, Jul 12, 2016 at 10:44 PM, That One Guy /sarcasm
>> <thatoneguyst...@gmail.com> wrote:
>>>
>>> sounded to me like this is a single IP (tcp/udp) connection saturation
>>> scenario, without a serious L7 filter in play its gonna do what its gonna
>>> do. Powercode for example (historically, not sure about now without
>>> procera)
>>> only applied the cap on new ip connections, established maintained
>>> whatever
>>> it was originally. so if you started an unbroken stream at a 12mb burst,
>>> that stream always hung out at 12mb, if your sustained was 3mb, new
>>> streams
>>> were limited to 3mb aggregate, but the 12mb stream prevailed as long as
>>> it
>>> was never considered "new". even when powercode was useful with the
>>> throttling controls, before they threw away their primary benefit in the
>>> bandwidth control area to sell their soul to procera with the real time
>>> throttles they took away. with the limited viability of procera boxes in
>>> the
>>> open internet era, I can see where this would be a cluster f**k post 12k
>>> investment.
>>>
>>> On Tue, Jul 12, 2016 at 9:44 PM, Josh Reynolds <j...@kyneticwifi.com>
>>> wrote:
>>>>
>>>>
>>>> The policer at the network edge can be much more aggressive than a
>>>> "policer" on an embedded customer network device, and this prevents
>>>> that "15Mbps for a 900MHz customer" bandwidth from transitioning
>>>> across your backhauls / backbone... per customer.
>>>>
>>>> Procera and other similar solutions can help, yes.
>>>>
>>>> On Tue, Jul 12, 2016 at 8:02 PM, Ken Hohhof <af...@kwisp.com> wrote:
>>>> > How does it eliminate the problem, unless you use something like a
>>>> > Procera
>>>> > to selectively apply policing to the CDN stream, leaving the customer
>>>> > some
>>>> > bandwidth for other traffic?
>>>> >
>>>> > From: Josh Reynolds
>>>> > Sent: Tuesday, July 12, 2016 7:22 PM
>>>> > To: af@afmug.com
>>>> > Subject: Re: [AFMUG] CDN overload
>>>> >
>>>> >
>>>> > Shaping/policing at the head end eliminates this problem, and clears >
>>>> > up
>>>> > your
>>>> > backbone.
>>>>

Re: [AFMUG] CDN overload