Re: [AFMUG] CDN Overload

the menus though. Still, you can’t leave it that way, how long will Win10 go without contacting the mothership and updating, I think it’s 30 days? From: Af [mailto:af-boun...@afmug.com] On Behalf Of Bill Prince Sent: Sunday, October 2, 2016 10:58 AM To: af@afmug.com Subject: Re: [AFMUG] CDN

Re: [AFMUG] CDN Overload

* Wednesday, September 21, 2016 8:35 PM *To:* af@afmug.com *Subject:* Re: [AFMUG] CDN Overload I have the ear of engineers at Level 3's CDN and Akamai. Working on Microsoft now (just sent the e-mail a couple minutes ago). Please continue filling out the form if you haven't. More informatio

Re: [AFMUG] CDN Overload

October 1, 2016 8:59:57 PM Subject: Re: [AFMUG] CDN Overload So you’re saying their actual engineers don’t talk to customers? That would explain a lot. From: Af [mailto:af-boun...@afmug.com] On Behalf Of Mike Hammett Sent: Saturday, October 1, 2016 8:42 PM To: af@afmug.com Subject:

Re: [AFMUG] CDN Overload

Good point. From: Af [mailto:af-boun...@afmug.com] On Behalf Of Mike Hammett Sent: Saturday, October 1, 2016 9:07 PM To: af@afmug.com Subject: Re: [AFMUG] CDN Overload If you were an actual engineer, would you want to talk to customers? - Mike Hammett <http://www.ics-il.

Re: [AFMUG] CDN Overload

So you’re saying their actual engineers don’t talk to customers? That would explain a lot. From: Af [mailto:af-boun...@afmug.com] On Behalf Of Mike Hammett Sent: Saturday, October 1, 2016 8:42 PM To: af@afmug.com Subject: Re: [AFMUG] CDN Overload That's why I encourage people to fill out

Re: [AFMUG] CDN Overload

" <af...@kwisp.com> To: af@afmug.com Sent: Saturday, October 1, 2016 7:01:47 PM Subject: Re: [AFMUG] CDN Overload I found this wonderful description of the problem and Microsoft support’s totally useless answer: http://answers.microsoft.com/en-us/windows/forum/windows_10-updat

Re: [AFMUG] CDN Overload

On 10/1/16 5:01 PM, Ken Hohhof wrote: I found this wonderful description of the problem and Microsoft support’s totally useless answer: http://answers.microsoft.com/en-us/windows/forum/windows_10-update/windows-update-hogging-bandwidth/f6883871-1de8-4b94-b0b7-fc4cfab158e7?auth=1 That's

Re: [AFMUG] CDN Overload

] On Behalf Of Mike Hammett Sent: Wednesday, September 21, 2016 8:35 PM To: af@afmug.com Subject: Re: [AFMUG] CDN Overload I have the ear of engineers at Level 3's CDN and Akamai. Working on Microsoft now (just sent the e-mail a couple minutes ago). Please continue filling out the form if you haven't

Re: [AFMUG] CDN Overload

I have the ear of engineers at Level 3's CDN and Akamai. Working on Microsoft now (just sent the e-mail a couple minutes ago). Please continue filling out the form if you haven't. More information is better than less. - Mike Hammett Intelligent Computing Solutions Midwest Internet

Re: [AFMUG] CDN Overload

heck Facebook, sporadically gets email. From: Mike Hammett <mailto:af...@ics-il.net> Sent: Tuesday, September 20, 2016 9:09 AM To: af@afmug.com <mailto:af@afmug.com> Subject: Re: [AFMUG] CDN Overload Can you address the questions I posed in the initial e-mail? - M

Re: [AFMUG] CDN Overload

ember 20, 2016 1:54:10 PM Subject: Re: [AFMUG] CDN Overload Mike, I know this doesn’t have all the information you are looking for, but it’s all I have time to capture right now. The source IPs seem to be Level3 CDN, and it’s sending just under 6 Mbps of traffic to a customer rate-limited by

Re: [AFMUG] CDN Overload

more platforms. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Mike Hammett" <af...@ics-il.net> To: af@afmug.com Sent: Wednesday, September 21, 2016 9:14:20 AM Subject: Re: [AFM

Re: [AFMUG] CDN Overload

https://goo.gl/forms/LvgFRsMdNdI8E9HF3 I have made this into a Google Form to make it easier to track compared to randomly formatted responses on multiple mailing lists, Facebook Groups, etc. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers

Re: [AFMUG] CDN Overload

What CDN? Mostly LLNW and Microsoft What have you identified the traffic to be? Mostly Windows Update What is the access network? I don't understand the question; PTMP wireless? Where is the rate limiting done? At the PMP450 Access Point How is the rate limiting done (policing vs. queueing, SFQ,

Re: [AFMUG] CDN Overload

Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Ken Hohhof" <af...@kwisp.com> To: af@afmug.com Sent: Tuesday, September 20, 2016 1:54:10 PM Subject: Re: [AFMUG] CDN Overload Mike, I know this doesn’t have all

Re: [AFMUG] CDN Overload

m Sent: Tuesday, September 20, 2016 8:58:12 AM Subject: Re: [AFMUG] CDN Overload I’ve seen it the most from Limelight. Don’t know what they are cramming down my user’s throats but I suspect it is either Microsoft or Apple. Jim Bouse Owner Mobile IT Pro - Brazos WiFi 979-985-5912 j...@brazoswifi.

Re: [AFMUG] CDN Overload

: Monday, September 19, 2016 10:29 PM To: af@afmug.com Subject: Re: [AFMUG] CDN Overload Gather evidence, attempt to work cooperatively, then name and shame if necessary. But yes, that's close to my intention. If you do your homework properly, the greater networking community is very powerful

Re: [AFMUG] CDN Overload

;https://www.facebook.com/thebrotherswisp> > > > <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> > From: "Paul Stewart" <p...@paulstewart.org <mailto:p...@paulstewart.org>> > To: af@afmug.com <mailto:af@afmug.com> > Sent: Monday, Septe

Re: [AFMUG] CDN Overload

;thatoneguyst...@gmail.com> To: af@afmug.com Sent: Monday, September 19, 2016 10:16:26 PM Subject: Re: [AFMUG] CDN Overload Did you just indicate an intention to get a cdn to alter a corporate policy? I have a huge satchel, I mean it could probably hold a couple bowling balls, reality only fills

Re: [AFMUG] CDN Overload

6 10:11:49 PM Subject: Re: [AFMUG] CDN Overload Might suggest also what kind of last mile connectivity they have (WISP vs other) to see if something correlates there…. I don’t see or have heard of this on cable, DSL, or FTTH … Paul On Sep 19, 2016, at 10:43 PM, Mike Hammett < af...@ics-il.

Re: [AFMUG] CDN Overload

Did you just indicate an intention to get a cdn to alter a corporate policy? I have a huge satchel, I mean it could probably hold a couple bowling balls, reality only fills it with a couple small pecans. Does it hurt? On Sep 19, 2016 9:43 PM, "Mike Hammett" wrote: Have you

Re: [AFMUG] CDN Overload

Might suggest also what kind of last mile connectivity they have (WISP vs other) to see if something correlates there…. I don’t see or have heard of this on cable, DSL, or FTTH … Paul > On Sep 19, 2016, at 10:43 PM, Mike Hammett wrote: > > Have you seen a CDN overloading a

Re: [AFMUG] CDN overload

At least I haven’t seen a repeat of what happened 4 weeks ago with 100-150 Mbps to a 3 Mbps customer. From: Mike Hammett Sent: Thursday, August 11, 2016 8:39 AM To: af@afmug.com Subject: Re: [AFMUG] CDN overload I saw this in action today from Microsoft. I didn't see 15, but I did see up

Re: [AFMUG] CDN overload

I saw this in action today from Microsoft. I didn't see 15, but I did see up to 3 megs for a 1.5 meg customer. I saw up to 190 connections in torch. I took a packet capture on the upstream interface and sent it over to one of my Microsoft contacts. She may not be the right person, but she's

Re: [AFMUG] CDN overload

.com/images/youtubeicon.png] <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> From: "Ken Hohhof" <af...@kwisp.com><mailto:af...@kwisp.com> To: af@afmug.com<mailto:af@afmug.com> Sent: Tuesday, July 19, 2016 7:37:42 PM Subject: Re: [AFMUG] CDN ov

Re: [AFMUG] CDN overload

*From: *"Ken Hohhof" <af...@kwisp.com> *To: *af@afmug.com *Sent: *Tuesday, July 19, 2016 7:37:42 PM *Subject: *Re: [AFMUG] CDN overload I saw it to various customers for 2 days starting mid day last Tuesday. It has not come back. Even today, which is Tues

Re: [AFMUG] CDN overload

...@kwisp.com> To: af@afmug.com Sent: Tuesday, July 19, 2016 7:37:42 PM Subject: Re: [AFMUG] CDN overload I saw it to various customers for 2 days starting mid day last Tuesday. It has not come back. Even today, which is Tuesday again. I wonder if it had something to do with the expira

Re: [AFMUG] CDN overload

they were not following TCP congestion management. I quickly added another GigE EHWIC module to the Cisco router so it wouldn’t happen again, but something really nasty was going on. From: George Skorup Sent: Tuesday, July 19, 2016 7:06 PM To: af@afmug.com Subject: Re: [AFMUG] CDN overload

Re: [AFMUG] CDN overload

- *From: *"George Skorup" <geo...@cbcast.com> *To: *af@afmug.com *Sent: *Thursday, July 14, 2016 1:33:21 AM *Subject: *Re: [AFMUG] CDN overload I forgot about this. Yes. A little later in the day, I started to see a lot of 13.n.n.n sources. Microsoft. Ye

Re: [AFMUG] CDN overload

-exchange> > <https://twitter.com/mdwestix> > The Brothers WISP <http://www.thebrotherswisp.com/> > <https://www.facebook.com/thebrotherswisp> > > > <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> > ------------------ >

Re: [AFMUG] CDN overload

To: af@afmug.com Sent: Tuesday, July 19, 2016 3:33:39 PM Subject: Re: [AFMUG] CDN overload Good points Mike …. I’ve never seen a CDN do this kind of behaviour before and curious why some folks see this occurring … Paul On Jul 19, 2016, at 3:53 PM, Mike Hammett < af...@ics-il.net &

Re: [AFMUG] CDN overload

Good points Mike …. I’ve never seen a CDN do this kind of behaviour before and curious why some folks see this occurring … Paul > On Jul 19, 2016, at 3:53 PM, Mike Hammett wrote: > > Have you confirmed that these would be a single machine making a single > request and

Re: [AFMUG] CDN overload

Have you confirmed that these would be a single machine making a single request and getting this result? Could they have started up a bunch of machines at once and it just amplified the already "normal " things they do? The next time people are seeing this, could you please do some packet

Re: [AFMUG] CDN overload

. = - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "George Skorup" <geo...@cbcast.com> To: af@afmug.com Sent: Thursday, July 14, 2016 1:33:21 AM Subject: Re: [AFMUG] CDN overload

Re: [AFMUG] CDN overload

.com> To: af@afmug.com Sent: Thursday, July 14, 2016 4:05:01 PM Subject: Re: [AFMUG] CDN overload We sent this the below instructions out to a section of customers who are on a small upstream and were killing on Update Tuesday and the day after. The metering option helps if they have PC

Re: [AFMUG] CDN overload

gt;> -- Original Message -- >> From: "Craig Schmaderer" <cr...@skywaveconnect.com> >> To: "af@afmug.com" <af@afmug.com> >> Sent: 7/14/2016 3:25:21 PM >> Subject: Re: [AFMUG] CDN overload >> >> >> Same here, it didn

Re: [AFMUG] CDN overload

riginal Message -- >> From: "Craig Schmaderer" <cr...@skywaveconnect.com> >> To: "af@afmug.com" <af@afmug.com> >> Sent: 7/14/2016 3:25:21 PM >> Subject: Re: [AFMUG] CDN overload >> >> >> Same here, it didn’t take me very

Re: [AFMUG] CDN overload

ur IP's. > > -- Original Message -- > From: "Craig Schmaderer" <cr...@skywaveconnect.com> > To: "af@afmug.com" <af@afmug.com> > Sent: 7/14/2016 3:25:21 PM > Subject: Re: [AFMUG] CDN overload > > > Same here, it didn’t take me very long

Re: [AFMUG] CDN overload

en? From: Af [mailto:af-boun...@afmug.com] On Behalf Of Ken Hohhof Sent: Thursday, July 14, 2016 1:26 PM To:af@afmug.com Subject: Re: [AFMUG] CDN overload Wowzers. I guess I feel better knowing it’s not just me, this really had my head spinning, trying to figure out who was attacking me.

Re: [AFMUG] CDN overload

, July 14, 2016 1:26 PM *To:* af@afmug.com *Subject:* Re: [AFMUG] CDN overload Wowzers. I guess I feel better knowing it’s not just me, this really had my head spinning, trying to figure out who was attacking me. But yeah, they changed something this week. It used to be pretty common to see 4

Re: [AFMUG] CDN overload

] On Behalf Of Ken Hohhof Sent: Thursday, July 14, 2016 1:26 PM To: af@afmug.com Subject: Re: [AFMUG] CDN overload Wowzers. I guess I feel better knowing it’s not just me, this really had my head spinning, trying to figure out who was attacking me. But yeah, they changed something this week. It used

Re: [AFMUG] CDN overload

not playing nice and pushing all other traffic aside. But we all know ISPs are evil and Silicon Valley can do no wrong, so it must be our fault somehow. From: Mathew Howard Sent: Thursday, July 14, 2016 10:10 AM To: af Subject: Re: [AFMUG] CDN overload Just had another one call in... 93

Re: [AFMUG] CDN overload

's >> happening then it ought to not matter how long the download takes. >> >> ...of course MS is not likely to care about my opinion on the matter. >> >> >> -- Original Message -- >> From: "George Skorup" <geo...@cbcast.com> >>

Re: [AFMUG] CDN overload

e update button, sure get it to run as fast as > possible, but if it's in the background and they don't even know it's > happening then it ought to not matter how long the download takes. > > ...of course MS is not likely to care about my opinion on the matter. > > > -- Original Messag

Re: [AFMUG] CDN overload

o:geo...@cbcast.com>> To: af@afmug.com <mailto:af@afmug.com> Sent: 7/14/2016 2:33:21 AM Subject: Re: [AFMUG] CDN overload I forgot about this. Yes. A little later in the day, I started to see a lot of 13.n.n.n sources. Microsoft. Yeah, update Tuesday. Then the same customer would

Re: [AFMUG] CDN overload

t even know it's > happening then it ought to not matter how long the download takes. > > ...of course MS is not likely to care about my opinion on the matter. > > > -- Original Message -- > From: "George Skorup" <geo...@cbcast.com> > To: af@afmug.c

Re: [AFMUG] CDN overload

matter. > > > -- Original Message -- > From: "George Skorup" <geo...@cbcast.com> > To: af@afmug.com > Sent: 7/14/2016 2:33:21 AM > Subject: Re: [AFMUG] CDN overload > > > I forgot about this. Yes. A little later in the day, I started to see a

Re: [AFMUG] CDN overload

getting DDOS attacks. *From:* George Skorup <mailto:geo...@cbcast.com> *Sent:* Tuesday, July 12, 2016 6:21 PM *To:* af@afmug.com <mailto:af@afmug.com> *Subject:* Re: [AFMUG] CDN overload Yup. LLNW. On 7/12/2016 5:35 PM, Ken Hohhof wrote: I assume you torched the traffic and verified it

Re: [AFMUG] CDN overload

OS attacks. From:George Skorup Sent: Tuesday, July 12, 2016 6:21 PM To:af@afmug.com Subject: Re: [AFMUG] CDN overload Yup. LLNW. On 7/12/2016 5:35 PM, Ken Hohhof wrote: I assume you torched the traffic and verified it is all coming from a particular CDN, not a random bunch of IPs as would be

Re: [AFMUG] CDN overload

ndom bunch of IPs as would be the case with BT. > Since this isn’t your first rodeo. > > From: George Skorup > Sent: Tuesday, July 12, 2016 5:31 PM > To: af@afmug.com > Subject: Re: [AFMUG] CDN overload > > Because they dick with TCP. > > On 7/12/2016 5:23 PM, Eric Kuhnke

Re: [AFMUG] CDN overload

rge Skorup <mailto:geo...@cbcast.com> *Sent:* Tuesday, July 12, 2016 6:21 PM *To:* af@afmug.com <mailto:af@afmug.com> *Subject:* Re: [AFMUG] CDN overload Yup. LLNW. On 7/12/2016 5:35 PM, Ken Hohhof wrote: I assume you torched the traffic and verified it is all coming from a particular C

Re: [AFMUG] CDN overload

, July 12, 2016 5:31 PM To: af@afmug.com Subject: Re: [AFMUG] CDN overload Because they dick with TCP. On 7/12/2016 5:23 PM, Eric Kuhnke wrote: And why is it the fault of the CDN? It could be a customer with a 100-peer bittorrent session downloading 30GB of Ubuntu DVD ISOs

Re: [AFMUG] CDN overload

, but when you connect to upstreams via microwave, you take it where you can get it. -Original Message- From: Josh Reynolds Sent: Wednesday, July 13, 2016 10:45 AM To: af@afmug.com Subject: Re: [AFMUG] CDN overload Don't have a decently sized tower hut there with HVAC? On Wed, Jul 13

Re: [AFMUG] CDN overload

tower hut there with HVAC? > > On Wed, Jul 13, 2016 at 10:43 AM, Ken Hohhof <af...@kwisp.com> wrote: > > Tower. > > > > -Original Message- From: Josh Reynolds > > Sent: Wednesday, July 13, 2016 10:15 AM > > > > To: af@afmug.com > > Su

Re: [AFMUG] CDN overload

Don't have a decently sized tower hut there with HVAC? On Wed, Jul 13, 2016 at 10:43 AM, Ken Hohhof <af...@kwisp.com> wrote: > Tower. > > -Original Message- From: Josh Reynolds > Sent: Wednesday, July 13, 2016 10:15 AM > > To: af@afmug.com > Subject: Re: [AFMU

Re: [AFMUG] CDN overload

Tower. -Original Message- From: Josh Reynolds Sent: Wednesday, July 13, 2016 10:15 AM To: af@afmug.com Subject: Re: [AFMUG] CDN overload Is your headend a shack? :) On Wed, Jul 13, 2016 at 7:06 AM, Ken Hohhof <af...@kwisp.com> wrote: I wish they made a box that didn't need

Re: [AFMUG] CDN overload

sh Reynolds > Sent: Tuesday, July 12, 2016 11:17 PM > > To: af@afmug.com > Subject: Re: [AFMUG] CDN overload > > Slight correction, you can virtually do whatever you want, you just > can't really block "legal" things, and have to make a good excuse for > "reasonabl

Re: [AFMUG] CDN overload

I wish they made a box that didn't need a data center environment. Not looking forward to putting in an outdoor enclosure with HVAC. -Original Message- From: Josh Reynolds Sent: Tuesday, July 12, 2016 11:17 PM To: af@afmug.com Subject: Re: [AFMUG] CDN overload Slight correction, you

Re: [AFMUG] CDN overload

>> >> investment. >> >> >> >> On Tue, Jul 12, 2016 at 9:44 PM, Josh Reynolds <j...@kyneticwifi.com> >> wrote: >> >>> >> >>> The policer at the network edge can be much more aggressive than a >> >>> "

Re: [AFMUG] CDN overload

gt;> The policer at the network edge can be much more aggressive than a > >>> "policer" on an embedded customer network device, and this prevents > >>> that "15Mbps for a 900MHz customer" bandwidth from transitioning > >>> across your backhau

Re: [AFMUG] CDN overload

orking, they yell >> >> upstairs >> >> at their kid didn’t I tell you to do your Xbox downloads after I go to >> >> bed? >> >> >> >> One view is this isn’t a problem, customer uses bad application, feels >> >> pain, learns not to

Re: [AFMUG] CDN overload

ing like a > Procera > > to selectively apply policing to the CDN stream, leaving the customer > some > > bandwidth for other traffic? > > > > From: Josh Reynolds > > Sent: Tuesday, July 12, 2016 7:22 PM > > To: af@afmug.com > > Subject: Re: [AFM

Re: [AFMUG] CDN overload

> From: Josh Reynolds > Sent: Tuesday, July 12, 2016 7:22 PM > To: af@afmug.com > Subject: Re: [AFMUG] CDN overload > > > Shaping/policing at the head end eliminates this problem, and clears up your > backbone. > > On Jul 12, 2016 7:06 PM, "Ken Hohhof

Re: [AFMUG] CDN overload

How does it eliminate the problem, unless you use something like a Procera to selectively apply policing to the CDN stream, leaving the customer some bandwidth for other traffic? From: Josh Reynolds Sent: Tuesday, July 12, 2016 7:22 PM To: af@afmug.com Subject: Re: [AFMUG] CDN overload

Re: [AFMUG] CDN overload

problem, which reinforces the idea that it’s >> the ISP’s fault. >> >> >> *From:* Darin Steffl <darin.ste...@mnwifi.com> >> *Sent:* Tuesday, July 12, 2016 5:42 PM >> *To:* af@afmug.com >> *Subject:* Re: [AFMUG] CDN overload >> >> >&g

Re: [AFMUG] CDN overload

t. And people with fat pipes like 50 or 100 Mbps cable Internet > probably don’t experience this problem, which reinforces the idea that it’s > the ISP’s fault. > > > *From:* Darin Steffl <darin.ste...@mnwifi.com> > *Sent:* Tuesday, July 12, 2016 5:42 PM > *To:* af@afmug.

Re: [AFMUG] CDN overload

To: af@afmug.com Subject: Re: [AFMUG] CDN overload Why aren't you rate limiting at the core closer to your upstream? Keep the traffic off your last mile and wireless backhaul network if you can help it. Works much better to throttle at the core instead of CPE. Sent from my smartphone. Please

Re: [AFMUG] CDN overload

That’s really strange .. and have heard that with LLNW before on this list…. but I’ve never actually seen it in action … the traffic should ramp up until it starts to get some loss/degragation and then taper to the point of using as much as possible…. > On Jul 12, 2016, at 3:13 PM, George

Re: [AFMUG] CDN overload

uesday, July 12, 2016 5:31 PM *To:* af@afmug.com <mailto:af@afmug.com> *Subject:* Re: [AFMUG] CDN overload Because they dick with TCP. On 7/12/2016 5:23 PM, Eric Kuhnke wrote: And why is it the fault of the CDN? It could be a customer with a 100-peer bittorrent session downloading 30GB o

Re: [AFMUG] CDN overload

Why aren't you rate limiting at the core closer to your upstream? Keep the traffic off your last mile and wireless backhaul network if you can help it. Works much better to throttle at the core instead of CPE. Sent from my smartphone. Please excuse any typos. On Jul 12, 2016 5:13 PM, "George

Re: [AFMUG] CDN overload

I assume you torched the traffic and verified it is all coming from a particular CDN, not a random bunch of IPs as would be the case with BT. Since this isn’t your first rodeo. From: George Skorup Sent: Tuesday, July 12, 2016 5:31 PM To: af@afmug.com Subject: Re: [AFMUG] CDN overload

Re: [AFMUG] CDN overload

Because they dick with TCP. On 7/12/2016 5:23 PM, Eric Kuhnke wrote: And why is it the fault of the CDN? It could be a customer with a 100-peer bittorrent session downloading 30GB of Ubuntu DVD ISOs. On Tue, Jul 12, 2016 at 3:13 PM, George Skorup

Re: [AFMUG] CDN overload

And why is it the fault of the CDN? It could be a customer with a 100-peer bittorrent session downloading 30GB of Ubuntu DVD ISOs. On Tue, Jul 12, 2016 at 3:13 PM, George Skorup wrote: > I have had it with these CDNs sending more traffic than the last mile can > handle. Got