Well if you're having issues with Level 3, then you're in luck. I have the ear 
of a Level 3 CDN engineer. They're very anxious to help. Once I get all of the 
information gathered , we'll be able to make some progress. 

I look forward to additional information. 




----- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




----- Original Message -----

From: "Ken Hohhof" <af...@kwisp.com> 
To: af@afmug.com 
Sent: Tuesday, September 20, 2016 1:54:10 PM 
Subject: Re: [AFMUG] CDN Overload 




Mike, I know this doesn’t have all the information you are looking for, but 
it’s all I have time to capture right now. The source IPs seem to be Level3 
CDN, and it’s sending just under 6 Mbps of traffic to a customer rate-limited 
by the tower router to 3 Mbps (Cisco rate limiting which is RED). The torch 
results are from a Mikrotik router upstream of the tower. The 10 second torch 
shows around 40 TCP connections. This seems to be a common pattern, push 
traffic until packet loss is around 50%, with around 50 TCP connections. 

I tried blocking individual IPs and it was like whack-a-mole, it just added 
more IPs. Then I blocked 8.0.0.0/8 which did stop the traffic, but I didn’t 
want to leave that in place. Once I stopped dropping that traffic, it started 
up again. 

I don’t know what the traffic is, but I suspect Windows 10 update. It’s a 
little old lady with one desktop computer. She says it started around 4pm 
yesterday, which seems a little early for Patch Tuesday. It is making her 
Internet totally unusable, can’t look up directions, can’t check Facebook, 
sporadically gets email. 





From: Mike Hammett 
Sent: Tuesday, September 20, 2016 9:09 AM 
To: af@afmug.com 
Subject: Re: [AFMUG] CDN Overload 


Can you address the questions I posed in the initial e-mail? 




----- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




----- Original Message -----

From: "Jim Bouse [Brazos WiFi]" <j...@brazoswifi.com> 
To: af@afmug.com 
Sent: Tuesday, September 20, 2016 8:58:12 AM 
Subject: Re: [AFMUG] CDN Overload 



I’ve seen it the most from Limelight. Don’t know what they are cramming down my 
user’s throats but I suspect it is either Microsoft or Apple. 


Jim Bouse 
Owner 
Mobile IT Pro - Brazos WiFi 
979-985-5912 
j...@brazoswifi.com 



From: Af [mailto:af-boun...@afmug.com] On Behalf Of Mike Hammett 
Sent: Monday, September 19, 2016 10:29 PM 
To: af@afmug.com 
Subject: Re: [AFMUG] CDN Overload 


Gather evidence, attempt to work cooperatively, then name and shame if 
necessary. But yes, that's close to my intention. If you do your homework 
properly, the greater networking community is very powerful and will back you. 
Those companies are largely ones that will work with you. Forget Amazon, Sony, 
etc. though. 

I've heard from people seeing this with Microsoft, Akamai, Limelight and Apple. 



----- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 




----- Original Message -----


From: "That One Guy /sarcasm" < thatoneguyst...@gmail.com > 
To: af@afmug.com 
Sent: Monday, September 19, 2016 10:16:26 PM 
Subject: Re: [AFMUG] CDN Overload 
Did you just indicate an intention to get a cdn to alter a corporate policy? I 
have a huge satchel, I mean it could probably hold a couple bowling balls, 
reality only fills it with a couple small pecans. Does it hurt? 



On Sep 19, 2016 9:43 PM, "Mike Hammett" < af...@ics-il.net > wrote: 




Have you seen a CDN overloading a customer? Help me gather information on the 
issue. 

What CDN? 
What have you identified the traffic to be? 
What is the access network? 
Where is the rate limiting done? 
How is the rate limiting done (policing vs. queueing, SFQ, PFIFO, etc,, etc.)? 
What is doing the rate limiting? 
What is the rate-limit set to? 
Upstream of the rate-limiter, what are you seeing for inbound traffic? 
One connection or many? 
How much traffic? 
How does other traffic behave when exceeding the rate limit? 
Where is NAT performed? 
What is doing NAT? 
Shared NAT or isolated to that customer? 
Have you done a packet capture before and after the rate limiter? The NAT 
device? 
Would you be willing to send a filtered packet capture (only the frames that 
relate to this CDN) to the CDN if they want it? 



There have been reports of CDNs sending more traffic than the customer can 
handle and ignores TCP convention to slow down. Trying to investigate this 
thoroughly so we can get the CDN to fix their system. Multiple CDNs have been 
shown to do this. 



----- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 









Reply via email to