Re: Anyone seeing peering/connectivity issues in San Jose?

[Brielle]

Further follow up, there appears to be a high amount of congestion between 
CenturyLink and Telia at their peering in California.  

Wonder if there’s another major download event going on?

Sent from my iPad

> On Feb 12, 2020, at 8:40 PM, Brielle  wrote:
> 
> Just a quick followup - someone from DO contacted me offlist to check into 
> possible issues.  I'm sorry for my really crappy wording in the original 
> message - had a long day.
> 
> Basic symptoms seem to be essentially a 'blackhole' somewhere between CL/L3 
> and DO going to the SFO2 data center.
> 
> I lose connectivity for a few minutes at a time - traffic seems to get lost, 
> both IPv4 and IPv6, and then starts responding again.
> 
> It seems to be stable for now, but I'm keeping an eye on it.
> 
> 
>> On 2/12/2020 6:58 PM, Brielle wrote:
>> Anyone else seeing peering / connectivity / peering issues in San Jose this 
>> evening?
>> To one of my servers at Digital Ocean in San Jose from CenturyLink Boise, 
>> ID...
>>   4 2 ms 2 ms 2 ms  boi2-edge-04.inet.qwest.net [63.224.242.57]
>>   514 ms14 ms18 ms  sea-brdr-03.inet.qwest.net [67.14.41.154]
>>   615 ms14 ms14 ms  4.68.75.121
>>   7 *** Request timed out.
>>   8 *** Request timed out.
>>   931 ms28 ms41 ms  138.197.249.181
>>  10 *** Request timed out.
>> Been doing this on and off for the last two hours or so.
> 
> 
> -- 
> Brielle Bruns
> The Summit Open Source Development Group
> http://www.sosdg.org/ http://www.ahbl.org

Re: Anyone seeing peering/connectivity issues in San Jose?

[Brielle]

Just a quick followup - someone from DO contacted me offlist to check 
into possible issues.  I'm sorry for my really crappy wording in the 
original message - had a long day.


Basic symptoms seem to be essentially a 'blackhole' somewhere between 
CL/L3 and DO going to the SFO2 data center.


I lose connectivity for a few minutes at a time - traffic seems to get 
lost, both IPv4 and IPv6, and then starts responding again.


It seems to be stable for now, but I'm keeping an eye on it.


On 2/12/2020 6:58 PM, Brielle wrote:
Anyone else seeing peering / connectivity / peering issues in San Jose 
this evening?


To one of my servers at Digital Ocean in San Jose from CenturyLink 
Boise, ID...


   4 2 ms 2 ms 2 ms  boi2-edge-04.inet.qwest.net 
[63.224.242.57]

   5    14 ms    14 ms    18 ms  sea-brdr-03.inet.qwest.net [67.14.41.154]
   6    15 ms    14 ms    14 ms  4.68.75.121
   7 *    *    * Request timed out.
   8 *    *    * Request timed out.
   9    31 ms    28 ms    41 ms  138.197.249.181
  10 *    *    * Request timed out.


Been doing this on and off for the last two hours or so.




--
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org/ http://www.ahbl.org

Anyone seeing peering/connectivity issues in San Jose?

[Brielle]

Anyone else seeing peering / connectivity / peering issues in San Jose 
this evening?


To one of my servers at Digital Ocean in San Jose from CenturyLink 
Boise, ID...


  4 2 ms 2 ms 2 ms  boi2-edge-04.inet.qwest.net [63.224.242.57]
  514 ms14 ms18 ms  sea-brdr-03.inet.qwest.net [67.14.41.154]
  615 ms14 ms14 ms  4.68.75.121
  7 *** Request timed out.
  8 *** Request timed out.
  931 ms28 ms41 ms  138.197.249.181
 10 *** Request timed out.


Been doing this on and off for the last two hours or so.

--
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org/ http://www.ahbl.org

Re: akamai yesterday - what in the world was that

[Carsten Bormann]

On 2020-02-12, at 20:45, Mike Hammett  wrote:
> 
> Aren't most modern consoles on whether they're "on" or not? IE: It's not a 
> full power up from a dead stop, 0 watts power usage.

https://www.anandtech.com/show/7528/the-xbox-one-mini-review-hardware-analysis/5
says two-digit standby power (which they say is needed for background 
updating).  At least in Germany, nobody sane will leave the thing in that 
expensive mode (a watt-year is $3 here).  Switchable extension power cords are 
being actively marketed here for these power hogs.

Grüße, Carsten

Re: akamai yesterday - what in the world was that

[Seth Mattinen]

On 2/12/20 11:48, Josh Luthman wrote:
In low power state, usually standby, they're connected to the network 
and listen for requests to download a new title (bought online) or 
updates.  I know on the Xbox One side of things this feature is semi-off 
by default as it turns the HDD off to save power, but it's still in 
standby in the sense that it takes only a few seconds to get to a usable 
state.



They can shut down or sleep, it's user choice.

Xbox has a setting for an "instant on" mode. I also had the option to 
check for updates, but when I went to use it yesterday it came up asking 
me to download a system update. And then after it installed that it 
wanted to download a giant update file for Halo. It's supposed to get 
updates on its own if you have both instant on and get updates enabled, 
but it didn't for whatever reason.


On PS4 you choose if you want to turn it off or go into rest mode, but I 
usually choose off because if the power hiccups in the weeks between 
times I get to use it it yells that it wasn't shut down correctly and it 
doesn't self-reboot into rest mode. Even when it was in rest mode, when 
I went to start Overcooked (the only game my wife will co-op play with 
me) it too asked to download an update.


So sure, they can, but it doesn't work reliably and when I have time to 
play *ow I'm going to tell it to download now without caring if it's 
not-my-problem peak time or not. And I'm sure I'm not alone in that 
sentiment. Again, speaking with my end user hat on.

Re: akamai yesterday - what in the world was that

[Tom Deligiannis]

My point was, the user has the option properly configured, but the console
still isn't updating until the console is turned on. I'm not implying that
it doesn't work, I'm simply stating that some users claim to have the
options configured properly but that updates are still not downloading w/o
user interaction.

Tom


On Wed, Feb 12, 2020 at 2:03 PM Josh Luthman 
wrote:

> Because the disks are shut off by default in standby mode.
>
> Josh Luthman
> Office: 937-552-2340
> Direct: 937-552-2343
> 1100 Wayne St
> Suite 1337
> Troy, OH 45373
>
>
> On Wed, Feb 12, 2020 at 2:53 PM Tom Deligiannis 
> wrote:
>
>>   Aren't most modern consoles on whether they're "on" or not? IE: It's
>>> not a full power up from a dead stop, 0 watts power usage.
>>>
>>> I'd think they'd be able to come out of sleep mode on their own,
>>> download the update, then go back to sleep.
>>>
>>
>> Xbox has this feature, but it doesn't work very well. A quick google
>> search shows that many users have their consoles set to receive updates,
>> but that feature doesn't seem to be working properly.
>>
>> Tom
>>
>> On Wed, Feb 12, 2020 at 1:46 PM Mike Hammett  wrote:
>>
>>> Aren't most modern consoles on whether they're "on" or not? IE: It's not
>>> a full power up from a dead stop, 0 watts power usage.
>>>
>>> I'd think they'd be able to come out of sleep mode on their own,
>>> download the update, then go back to sleep.
>>>
>>>
>>>
>>> -
>>> Mike Hammett
>>> Intelligent Computing Solutions 
>>> 
>>> 
>>> 
>>> 
>>> Midwest Internet Exchange 
>>> 
>>> 
>>> 
>>> The Brothers WISP 
>>> 
>>> 
>>> --
>>> *From: *"Seth Mattinen" 
>>> *To: *nanog@nanog.org
>>> *Sent: *Wednesday, February 12, 2020 1:42:21 PM
>>> *Subject: *Re: akamai yesterday - what in the world was that
>>>
>>> On 2/12/20 11:31, Livingood, Jason wrote:
>>> > But I think folks are correct that the issue may be more that a given
>>> gaming device was turned off at night (though no reason that device could
>>> not pre-cache the content from the source). In any case, there should be a
>>> better way to address this. The Internet will see more and more of these
>>> downloads and smoothing the impact out seems prudent for all involved.
>>>
>>>
>>> Putting my end user hat on, I turn off all my consoles when I'm not
>>> using them, often for weeks. When I get home and it looks like I'll have
>>> time to play after dinner I'll turn one of them on and let it
>>> download/install. I don't really care that my off work and dinner times
>>> might not be convenient for my ISP to download giant files. I fully
>>> understand the ISP's perspective, but I'm not going to start leaving my
>>> consoles on 24x7.
>>>
>>> The way to address this used to be this thing called "physical media"
>>> that held games, but nowadays even when I have a game on disc it has to
>>> download at least one massive patch before it will play.
>>>
>>>

Re: akamai yesterday - what in the world was that

[Mike Hammett]

It seems like spinning up the disk if there's an update would be trivial. 
*shrugs* 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Josh Luthman"  
To: "Tom Deligiannis"  
Cc: "Mike Hammett" , "NANOG list"  
Sent: Wednesday, February 12, 2020 2:02:42 PM 
Subject: Re: akamai yesterday - what in the world was that 


Because the disks are shut off by default in standby mode. 





Josh Luthman 
Office: 937-552-2340 
Direct: 937-552-2343 
1100 Wayne St 
Suite 1337 
Troy, OH 45373 



On Wed, Feb 12, 2020 at 2:53 PM Tom Deligiannis < tom.deligian...@gmail.com > 
wrote: 





Aren't most modern consoles on whether they're "on" or not? IE: It's not a full 
power up from a dead stop, 0 watts power usage. 


I'd think they'd be able to come out of sleep mode on their own, download the 
update, then go back to sleep. 




Xbox has this feature, but it doesn't work very well. A quick google search 
shows that many users have their consoles set to receive updates, but that 
feature doesn't seem to be working properly. 


Tom 


On Wed, Feb 12, 2020 at 1:46 PM Mike Hammett < na...@ics-il.net > wrote: 




Aren't most modern consoles on whether they're "on" or not? IE: It's not a full 
power up from a dead stop, 0 watts power usage. 


I'd think they'd be able to come out of sleep mode on their own, download the 
update, then go back to sleep. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 



From: "Seth Mattinen" < se...@rollernet.us > 
To: nanog@nanog.org 
Sent: Wednesday, February 12, 2020 1:42:21 PM 
Subject: Re: akamai yesterday - what in the world was that 

On 2/12/20 11:31, Livingood, Jason wrote: 
> But I think folks are correct that the issue may be more that a given gaming 
> device was turned off at night (though no reason that device could not 
> pre-cache the content from the source). In any case, there should be a better 
> way to address this. The Internet will see more and more of these downloads 
> and smoothing the impact out seems prudent for all involved. 


Putting my end user hat on, I turn off all my consoles when I'm not 
using them, often for weeks. When I get home and it looks like I'll have 
time to play after dinner I'll turn one of them on and let it 
download/install. I don't really care that my off work and dinner times 
might not be convenient for my ISP to download giant files. I fully 
understand the ISP's perspective, but I'm not going to start leaving my 
consoles on 24x7. 

The way to address this used to be this thing called "physical media" 
that held games, but nowadays even when I have a game on disc it has to 
download at least one massive patch before it will play. 

Re: akamai yesterday - what in the world was that

[Chris Adams]

Once upon a time, Mike Hammett  said:
> Aren't most modern consoles on whether they're "on" or not? IE: It's not a 
> full power up from a dead stop, 0 watts power usage. 

The Xbox One kind of does that - it can receive updates (both game and
OS) in that state, but it depends on other settings.  If you plug in an
external hard drive, there's a separate setting that is off by default
(so if a game is on the external drive, it doesn't get updates).
-- 
Chris Adams 

Re: akamai yesterday - what in the world was that

[Josh Luthman]

Because the disks are shut off by default in standby mode.

Josh Luthman
Office: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373


On Wed, Feb 12, 2020 at 2:53 PM Tom Deligiannis 
wrote:

>   Aren't most modern consoles on whether they're "on" or not? IE: It's not
>> a full power up from a dead stop, 0 watts power usage.
>>
>> I'd think they'd be able to come out of sleep mode on their own, download
>> the update, then go back to sleep.
>>
>
> Xbox has this feature, but it doesn't work very well. A quick google
> search shows that many users have their consoles set to receive updates,
> but that feature doesn't seem to be working properly.
>
> Tom
>
> On Wed, Feb 12, 2020 at 1:46 PM Mike Hammett  wrote:
>
>> Aren't most modern consoles on whether they're "on" or not? IE: It's not
>> a full power up from a dead stop, 0 watts power usage.
>>
>> I'd think they'd be able to come out of sleep mode on their own, download
>> the update, then go back to sleep.
>>
>>
>>
>> -
>> Mike Hammett
>> Intelligent Computing Solutions 
>> 
>> 
>> 
>> 
>> Midwest Internet Exchange 
>> 
>> 
>> 
>> The Brothers WISP 
>> 
>> 
>> --
>> *From: *"Seth Mattinen" 
>> *To: *nanog@nanog.org
>> *Sent: *Wednesday, February 12, 2020 1:42:21 PM
>> *Subject: *Re: akamai yesterday - what in the world was that
>>
>> On 2/12/20 11:31, Livingood, Jason wrote:
>> > But I think folks are correct that the issue may be more that a given
>> gaming device was turned off at night (though no reason that device could
>> not pre-cache the content from the source). In any case, there should be a
>> better way to address this. The Internet will see more and more of these
>> downloads and smoothing the impact out seems prudent for all involved.
>>
>>
>> Putting my end user hat on, I turn off all my consoles when I'm not
>> using them, often for weeks. When I get home and it looks like I'll have
>> time to play after dinner I'll turn one of them on and let it
>> download/install. I don't really care that my off work and dinner times
>> might not be convenient for my ISP to download giant files. I fully
>> understand the ISP's perspective, but I'm not going to start leaving my
>> consoles on 24x7.
>>
>> The way to address this used to be this thing called "physical media"
>> that held games, but nowadays even when I have a game on disc it has to
>> download at least one massive patch before it will play.
>>
>>

Re: Peering/Transit eBGP sessions -pet or cattle?

[Baldur Norddahl]

On Tue, Feb 11, 2020 at 12:33 AM Lukas Tribus  wrote:

> > Therefore, if being down for several minutes is not ok, you
> > should invest in dual links to your transits. And connect those
> > to two different routers. If possible with a guarantee the
> > transits use two routers at their end and that divergent fiber
> > paths are used etc.
>
> That is not my experience *at all*. I have always seen my prefixes
> converge in a couple of seconds upstream (vs 2 different Tier1's).
>

This is a bit old but probably still thus:

https://labs.ripe.net/Members/vastur/the-shape-of-a-bgp-update

Quote: "To conclude, we observe that BGP route updates tend to converge
globally in just a few minutes. The propagation of newly announced prefixes
happens almost instantaneously, reaching 50% visibility in just under 10
seconds, revealing a highly responsive global system. Prefix withdrawals
take longer to converge and generate nearly 4 times more BGP traffic, with
the visibility dropping below 10% only after approximately 2 minutes".

Unfortunately they did not test the case of withdrawal from one router
while having the prefix still active at another.


>
> When I saw *minutes* of brownouts in connectivity it was always
> because of ingress prefix convergence (or the lack thereof, due to
> slow FIB programing, then temporary internal routing loops, nasty
> things like that, but never external).
>
>
That is also a significant problem. In the case of a single transit
connection per router, two routers and two providers, there will be a lot
of internal convergence between your two routers in the case of a link
failure. That is also avoided by having both routers having the same
provider connections. That way a router may still have to invalidate many
routes but there will be no loops and the router has loop free alternatives
loaded into memory already (to the other provider). Plus you can use the
simple trick of having a default route as a fall back.

Regards

Baldur

Re: akamai yesterday - what in the world was that

[Tom Deligiannis]

>
>   Aren't most modern consoles on whether they're "on" or not? IE: It's not
> a full power up from a dead stop, 0 watts power usage.
>
> I'd think they'd be able to come out of sleep mode on their own, download
> the update, then go back to sleep.
>

Xbox has this feature, but it doesn't work very well. A quick google search
shows that many users have their consoles set to receive updates, but that
feature doesn't seem to be working properly.

Tom

On Wed, Feb 12, 2020 at 1:46 PM Mike Hammett  wrote:

> Aren't most modern consoles on whether they're "on" or not? IE: It's not a
> full power up from a dead stop, 0 watts power usage.
>
> I'd think they'd be able to come out of sleep mode on their own, download
> the update, then go back to sleep.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions 
> 
> 
> 
> 
> Midwest Internet Exchange 
> 
> 
> 
> The Brothers WISP 
> 
> 
> --
> *From: *"Seth Mattinen" 
> *To: *nanog@nanog.org
> *Sent: *Wednesday, February 12, 2020 1:42:21 PM
> *Subject: *Re: akamai yesterday - what in the world was that
>
> On 2/12/20 11:31, Livingood, Jason wrote:
> > But I think folks are correct that the issue may be more that a given
> gaming device was turned off at night (though no reason that device could
> not pre-cache the content from the source). In any case, there should be a
> better way to address this. The Internet will see more and more of these
> downloads and smoothing the impact out seems prudent for all involved.
>
>
> Putting my end user hat on, I turn off all my consoles when I'm not
> using them, often for weeks. When I get home and it looks like I'll have
> time to play after dinner I'll turn one of them on and let it
> download/install. I don't really care that my off work and dinner times
> might not be convenient for my ISP to download giant files. I fully
> understand the ISP's perspective, but I'm not going to start leaving my
> consoles on 24x7.
>
> The way to address this used to be this thing called "physical media"
> that held games, but nowadays even when I have a game on disc it has to
> download at least one massive patch before it will play.
>
>

Re: akamai yesterday - what in the world was that

[Josh Luthman]

In low power state, usually standby, they're connected to the network and
listen for requests to download a new title (bought online) or updates.  I
know on the Xbox One side of things this feature is semi-off by default as
it turns the HDD off to save power, but it's still in standby in the sense
that it takes only a few seconds to get to a usable state.

Josh Luthman
Office: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373


On Wed, Feb 12, 2020 at 2:46 PM Mike Hammett  wrote:

> Aren't most modern consoles on whether they're "on" or not? IE: It's not a
> full power up from a dead stop, 0 watts power usage.
>
> I'd think they'd be able to come out of sleep mode on their own, download
> the update, then go back to sleep.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions 
> 
> 
> 
> 
> Midwest Internet Exchange 
> 
> 
> 
> The Brothers WISP 
> 
> 
> --
> *From: *"Seth Mattinen" 
> *To: *nanog@nanog.org
> *Sent: *Wednesday, February 12, 2020 1:42:21 PM
> *Subject: *Re: akamai yesterday - what in the world was that
>
> On 2/12/20 11:31, Livingood, Jason wrote:
> > But I think folks are correct that the issue may be more that a given
> gaming device was turned off at night (though no reason that device could
> not pre-cache the content from the source). In any case, there should be a
> better way to address this. The Internet will see more and more of these
> downloads and smoothing the impact out seems prudent for all involved.
>
>
> Putting my end user hat on, I turn off all my consoles when I'm not
> using them, often for weeks. When I get home and it looks like I'll have
> time to play after dinner I'll turn one of them on and let it
> download/install. I don't really care that my off work and dinner times
> might not be convenient for my ISP to download giant files. I fully
> understand the ISP's perspective, but I'm not going to start leaving my
> consoles on 24x7.
>
> The way to address this used to be this thing called "physical media"
> that held games, but nowadays even when I have a game on disc it has to
> download at least one massive patch before it will play.
>
>

Re: akamai yesterday - what in the world was that

[Mike Hammett]

Aren't most modern consoles on whether they're "on" or not? IE: It's not a full 
power up from a dead stop, 0 watts power usage. 


I'd think they'd be able to come out of sleep mode on their own, download the 
update, then go back to sleep. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Seth Mattinen"  
To: nanog@nanog.org 
Sent: Wednesday, February 12, 2020 1:42:21 PM 
Subject: Re: akamai yesterday - what in the world was that 

On 2/12/20 11:31, Livingood, Jason wrote: 
> But I think folks are correct that the issue may be more that a given gaming 
> device was turned off at night (though no reason that device could not 
> pre-cache the content from the source). In any case, there should be a better 
> way to address this. The Internet will see more and more of these downloads 
> and smoothing the impact out seems prudent for all involved. 


Putting my end user hat on, I turn off all my consoles when I'm not 
using them, often for weeks. When I get home and it looks like I'll have 
time to play after dinner I'll turn one of them on and let it 
download/install. I don't really care that my off work and dinner times 
might not be convenient for my ISP to download giant files. I fully 
understand the ISP's perspective, but I'm not going to start leaving my 
consoles on 24x7. 

The way to address this used to be this thing called "physical media" 
that held games, but nowadays even when I have a game on disc it has to 
download at least one massive patch before it will play. 

Re: akamai yesterday - what in the world was that

[Seth Mattinen]

On 2/12/20 11:31, Livingood, Jason wrote:

But I think folks are correct that the issue may be more that a given gaming 
device was turned off at night (though no reason that device could not 
pre-cache the content from the source). In any case, there should be a better 
way to address this. The Internet will see more and more of these downloads and 
smoothing the impact out seems prudent for all involved.



Putting my end user hat on, I turn off all my consoles when I'm not 
using them, often for weeks. When I get home and it looks like I'll have 
time to play after dinner I'll turn one of them on and let it 
download/install. I don't really care that my off work and dinner times 
might not be convenient for my ISP to download giant files. I fully 
understand the ISP's perspective, but I'm not going to start leaving my 
consoles on 24x7.


The way to address this used to be this thing called "physical media" 
that held games, but nowadays even when I have a game on disc it has to 
download at least one massive patch before it will play.

Re: akamai yesterday - what in the world was that

[Livingood, Jason]

(as an aside) Some of this timed & controlled distribution (by the content 
originator) should be possible using IETF Content Delivery Network 
Interconnection (CDNI) standards - see 
https://datatracker.ietf.org/wg/cdni/documents/. The initial RFC 6707 provides 
some background - https://www.rfc-editor.org/rfc/rfc6707.

But I think folks are correct that the issue may be more that a given gaming 
device was turned off at night (though no reason that device could not 
pre-cache the content from the source). In any case, there should be a better 
way to address this. The Internet will see more and more of these downloads and 
smoothing the impact out seems prudent for all involved.

Jason

On 2/12/20, 11:59 AM, "NANOG on behalf of Chris Adams" 
 wrote:

I think security is probably the sticking point for this.  Content
owners don't want anybody having direct access to their files, and as
more content is distributed over HTTPS, content distributors don't wany
anbody having access to their certificates.
--
Chris Adams 

Re: akamai yesterday - what in the world was that

[Scott Weeks]

--
On 2/11/20 6:41 PM, Tom Deligiannis wrote:
> There is a major update that has released today, how's everything 
> looking for everyone?
---


Did anyone else notice a big traffic dip from noon to 8pm local time?  
Strange look on the graphs.

scott

Re: akamai yesterday - what in the world was that

[Scott Weeks]

--
On 2/11/20 6:41 PM, Tom Deligiannis wrote:
> There is a major update that has released today, how's everything 
> looking for everyone?
---


eyeball network here...

It shifted our traffic patterns to earlier peaks.  It started at 9am.  
The first peak was at 3pm and the main peak was at 7pm and the traffic
fell back to normal loads at 1am.  There was increased traffic all 
night, though, as our low traffic was still 4-5Gbps over normal lows 
at the 2-5am period.

Normal peak is ~16Gbps, but this one was ~33Gbps almost all on one
inexpensive link.  The other main link showed and increase of about 
8Gbps with a funny dip between noon and 9pm HST.

scott

Re: akamai yesterday - what in the world was that

[Seth Mattinen]

On 2/12/20 10:02, Jared Mauch wrote:

When you see this please raise it to my attention. I can't promise a resolution 
but will promise clarity in what is going on.



This was in May 2019 so what's done is done at this point, but I will 
forward you the email offlist.

Re: akamai yesterday - what in the world was that

[Jared Mauch]

When you see this please raise it to my attention. I can't promise a resolution 
but will promise clarity in what is going on. 

I know some cities are problematic as we are moving cages or datacenter space 
and have the usual related problems. 

There is always something. 

Sent from my iCar

> On Feb 12, 2020, at 9:36 AM, Seth Mattinen  wrote:
> 
> 
> The wheels of bureaucracy are certainly a problem. The largest peer on our 
> local exchange couldn't even get Akamai to complete a peering turn up because 
> whoever was working on the ticket on the Akamai side got stuck on trying to 
> set up the wrong location. And then months pass, it never got resolved, and 
> then they decided to pull the cache. Akamai had one hand failing to set up 
> new peers and the other hand saying why aren't there more peers, and the two 
> hands never know what the other is doing.

Re: akamai yesterday - what in the world was that

[Seth Mattinen]

The wheels of bureaucracy are certainly a problem. The largest peer on 
our local exchange couldn't even get Akamai to complete a peering turn 
up because whoever was working on the ticket on the Akamai side got 
stuck on trying to set up the wrong location. And then months pass, it 
never got resolved, and then they decided to pull the cache. Akamai had 
one hand failing to set up new peers and the other hand saying why 
aren't there more peers, and the two hands never know what the other is 
doing.

Re: akamai yesterday - what in the world was that

[Denys Fedoryshchenko]

It would be really nice if the major CDNs had virtual machines small
network operators with very expensive regional transport costs could
spin up.  Hit rate would be very low, of course, but the ability to
grab some of these mass-market huge updates and serve them on the
other end of the regional transport at essentially no extra cost would
be great. I'm sure legal arrangements make that difficult, though.

+1

I think primary reason is that many major CDN offload nodes implemented 
such way that they require significant amount of maintenance and 
support. And doesnt matter, small or big ISP - they will have problems, 
and when the company that installed this CDN node is huge, like Facebook 
or Google, to crank all the bureaucratic wheels to change silly power 
supply or HDD - it comes at a huge cost for them. Also add that small 
ISP often dont have 24/7 support shifts, less qualified for complex 
issues, more likely to have poor infrastructure 
(temperature/power/reliability), that means more support expenses.
And they don’t give a damn that because of their "behemothness", they 
increase the digital inequality gap. When a large ISP or ISP cartel 
member enter some regional market, local providers will not be able to 
compete with him, since they cannot afford CDN nodes due traffic volume.


Many of CDN also do questionable "BGP as signalling only" setups with 
proprietary TCP probing/loss, that often doesn't work reliably. Each of 
them is trying to reinvent the wheel, "this time not round, but 
dodecahedral". And when it fails, ISP will waste time of support, until 
it reach someone who understand issue. In most cases, this is a blackbox 
setup, and when problem happens ISP are endlessly trying to explain 
problem to outsourced support, who have very limited access as well, and 
responding like robot according to the his "support workflow", with zero 
feedback to common problems.


Honestly, it's time to develop an open standard for caching content on 
open CDN nodes, which should be easy to use for both content providers 
and ISPs.
For example, at one time existed a special hardcoded "retracker.local" 
server in many torrent clients, which optionally(if resolved on ISP, 
static entry in recursor) was used for the discovery of nearest seeders 
inside network of a local provider.

http://retracker.local/
Maybe it is possible to make a similar scheme, if the content provider 
wants "open" CDN to work, it will set some alternative scheme 
cdn://content.provider.com/path/file or other kind of hint, with content 
validity/authenticity mechanism. After that, the browser will attempt to 
do CDN discovery, for example: "content.provider.com.reservedtld" and 
will push request through it.

I'm sure someone will have a better idea how to do that.

As a result, the installation of such "offloading node" will be just 
installing container/vm and, if the load is increased, increasing the 
number of servers/vm instances.

Re: akamai yesterday - what in the world was that

[Brandon Martin]

On 2/12/20 11:56 AM, Chris Adams wrote:

I think security is probably the sticking point for this.  Content
owners don't want anybody having direct access to their files, and as
more content is distributed over HTTPS, content distributors don't wany
anbody having access to their certificates.


Yeah, these were the "legal" aspects I was referring to above.  Not a 
technical problem, really.  I can't say I'm surprised, and I can think 
of some workarounds, but it's definitely a thing to consider.

--
Brandon Martin

Re: akamai yesterday - what in the world was that

[Chris Adams]

Once upon a time, Brandon Martin  said:
> I guess what I'm looking for is a more "standard product".  Load
> this VM, tell it your preference for upstream use vs. hit rate, let
> it announce some routes into your network, and you take what you
> get.  If you need more, presumably you have the volume to back it
> up.

I think security is probably the sticking point for this.  Content
owners don't want anybody having direct access to their files, and as
more content is distributed over HTTPS, content distributors don't wany
anbody having access to their certificates.
-- 
Chris Adams 

Re: akamai yesterday - what in the world was that

[Brandon Martin]

On 2/12/20 11:22 AM, Seth Mattinen wrote:
My experience is that they want to see lots of traffic growth to stay 
interested. As companies get bigger the minimum bar to play keeps going 
up, and anyone below that bar is stuck relying on transit. Fall below 
the bar or don't show enough growth fast enough and they pull the 
resources away.


Which makes perfect sense for anything that involves physical 
infrastructure, talking to people, etc.  Resources have to be allocated 
reasonably.


I guess what I'm looking for is a more "standard product".  Load this 
VM, tell it your preference for upstream use vs. hit rate, let it 
announce some routes into your network, and you take what you get.  If 
you need more, presumably you have the volume to back it up.


I know somebody had done something like this for Steam at one point 
using a transparent HTTP proxy.  IDK if it still works, and I don't 
think it was actually supported by Valve.


Maybe I'm smoking something, here...
--
Brandon Martin

Re: akamai yesterday - what in the world was that

[Seth Mattinen]

On 2/12/20 8:36 AM, Aaron Gould wrote:

Netflix oca has it figured out, as my fill windows is during off-peak time, 2 
a.m. - 6 am. and I think it's also configurable in the oca portal.



It's not fill, it's that people don't turn on their xbox or whatever 
until after they get home from work and only then does it start 
downloading. Multiply that by 1000 people getting home from work around 
the same time.

RE: akamai yesterday - what in the world was that

[Aaron Gould]

Netflix oca has it figured out, as my fill windows is during off-peak time, 2 
a.m. - 6 am. and I think it's also configurable in the oca portal.

-Aaron

Re: akamai yesterday - what in the world was that

[Seth Mattinen]

On 2/12/20 8:13 AM, Brandon Martin wrote:
It would be really nice if the major CDNs had virtual machines small 
network operators with very expensive regional transport costs could 
spin up.  Hit rate would be very low, of course, but the ability to grab 
some of these mass-market huge updates and serve them on the other end 
of the regional transport at essentially no extra cost would be great. 
I'm sure legal arrangements make that difficult, though.



My experience is that they want to see lots of traffic growth to stay 
interested. As companies get bigger the minimum bar to play keeps going 
up, and anyone below that bar is stuck relying on transit. Fall below 
the bar or don't show enough growth fast enough and they pull the 
resources away.

Re: akamai yesterday - what in the world was that

[Brandon Martin]

On 2/12/20 10:59 AM, Dave Bell wrote:

Night-time for you is daytime for someone else.


This is very true, though I am curious what the international 
demographics are like for COD in particular and many games in general. 
I suspect a lot of them are at least somewhat regional.


I agree that the folks pushing these massive data loads could be 
considerate of the networks they are running on. The steam preload thing 
is a great example. The content becomes available x weeks before launch 
and downloaded to your device at some point in that window. It then 
simply unlocked on launch day.


This works really, really well from what I can see.  I don't even see 
many major OS updates, new game drops, etc. in my stats.  One of my 
networks is too small and hence noisy to really tell, but the other is 
quite predictable.


I'm not sure how much incentive there is for people to implement this 
kind of system though. With the rise of CDN and global caching, people 
care less about the performance of their servers for content 
distribution as it just scales out.


It would be really nice if the major CDNs had virtual machines small 
network operators with very expensive regional transport costs could 
spin up.  Hit rate would be very low, of course, but the ability to grab 
some of these mass-market huge updates and serve them on the other end 
of the regional transport at essentially no extra cost would be great. 
I'm sure legal arrangements make that difficult, though.

--
Brandon Martin

Re: akamai yesterday - what in the world was that

[Dave Bell]

On Wed, 12 Feb 2020 at 14:46, Brandon Martin 
wrote:

>  It would be nice if things could drop overnight to hopefully
> spread things out during the daytime lull some.
>

Night-time for you is daytime for someone else.

I agree that the folks pushing these massive data loads could be
considerate of the networks they are running on. The steam preload thing is
a great example. The content becomes available x weeks before launch and
downloaded to your device at some point in that window. It then simply
unlocked on launch day.

I'm not sure how much incentive there is for people to implement this kind
of system though. With the rise of CDN and global caching, people care less
about the performance of their servers for content distribution as it just
scales out.

Dave

Re: akamai yesterday - what in the world was that

[Jared Mauch]

Our CEO tweeted out a new platform peak yesterday which when you round it from 
the many trailing 9’s is 140T.

https://twitter.com/TomLeightonAKAM/status/1227389107665592320

I want to ensure the bits get through with the least pain as possible.

I’m expecting more of the same in the future, so if you’re not talking to 
someone here and we caused you some pain, I can help.  A number of people 
reached out in the past 24h and I think i’m all caught up on those, but while 
regular IXP peering can help in many places, if we can offload on a high 
capacity PNI that is my preference.  

- Jared

> On Feb 12, 2020, at 10:33 AM, Jason Canady  wrote:
> 
> We saw a higher load overnight, a little bit of a spike last night, but 
> really hard to tell overall with our traffic.  Updates were still going at 
> 8am today.  We run a local/regional WISP.
> 
> On 2/12/20 9:46 AM, Brandon Martin wrote:
>> On 2/11/20 6:41 PM, Tom Deligiannis wrote:
>>> There is a major update that has released today, how's everything looking 
>>> for everyone?
>> 
>> I run a couple distinct very small networks.  Both are transit-only with no 
>> direct peering or local caching and generally sub-gbps.
>> 
>> One set a new 1-min 95% record and did so by nearly 25% over its previous 
>> record.  The other matched its existing record.  The former thankfully has 
>> ample capacity, while the latter thankfully did so before primetime.
>> 
>> These are definitely going to be harder to manage than the primetime hump.  
>> It would be nice if things could drop overnight to hopefully spread things 
>> out during the daytime lull some.  I understand that some people won't pick 
>> it up until they get home and turn on devices (which is often after school 
>> hours for these type of game updates), but spreading things out would be 
>> really nice especially for those of us without local caching.
>> 

Re: akamai yesterday - what in the world was that

[Jason Canady]

We saw a higher load overnight, a little bit of a spike last night, but 
really hard to tell overall with our traffic.  Updates were still going 
at 8am today.  We run a local/regional WISP.


On 2/12/20 9:46 AM, Brandon Martin wrote:

On 2/11/20 6:41 PM, Tom Deligiannis wrote:
There is a major update that has released today, how's everything 
looking for everyone?


I run a couple distinct very small networks.  Both are transit-only 
with no direct peering or local caching and generally sub-gbps.


One set a new 1-min 95% record and did so by nearly 25% over its 
previous record.  The other matched its existing record.  The former 
thankfully has ample capacity, while the latter thankfully did so 
before primetime.


These are definitely going to be harder to manage than the primetime 
hump.  It would be nice if things could drop overnight to hopefully 
spread things out during the daytime lull some.  I understand that 
some people won't pick it up until they get home and turn on devices 
(which is often after school hours for these type of game updates), 
but spreading things out would be really nice especially for those of 
us without local caching.

Re: Flow based architecture in data centers(more specifically Telco Clouds)

[Glen Kent]

Hi,

On Mon, Feb 10, 2020 at 3:22 PM Saku Ytti  wrote:

> On Sun, 9 Feb 2020 at 23:09, Rod Beck 
> wrote:
>
> > I am curious about the distinction about the flow versus non-flow
> architecture for data centers and I am also fascinated by the separate
> issue of WAN architecture for these
>
> Based on the context of the OP's question, he is talking about
> architecture where some components, potentially network devices, are
> flow-aware, instead of doing LPM lookup per packet, they are doing LPM
> lookup per flow.
>

This was exactly my question.


> This comes up every few years in various formats, because with
> flow-lookup you have one expensive LPM lookup per flow and multiple
> cheap LEM lookups. However the LEM table size is unbounded and easily
> abusable leading to a set of very complex problems.
>
> There are of course a lot of variation what OP might mean. Network
> might be for example entirely LEM lookup with extremely small table,
> by using stack of MPLS labels, zero LPM lookups. This architecture
> could be made so that when server needs to send something say video to
> a client, it asks orchestration for permission, telling I need to send
> x GB to DADDR K with rate at least Z and no more than Y, orchestration
> could then tell the server to start sending at time T0 and impose MPLS
> label stack of [l1, l2, l3, l4, l5]
>
> Orchestration would know exactly which links traffic traverses, how
> long will it be utilised and how much free capacity there is. Network
> would be extremely dumb, no IP lookups ever, only thousands of MPLS
> labels in FIB, so entirely on-chip lookups of trivial cost.
>

My question was rather simple.

Many cloud operators use Open Vswitch (OVS) based dataplanes wherein each
packet results in a new flow in the system. The first packet does a lookup
in the slow path which causes the fast path (either OVS-DPDK or a smartNIC
or VPP-like paradigm or something entirely different) to be programmed. All
subsequent packets hit the fast path and reach the VM (which is hosting the
VNF). The advantage of this scheme is that the operator knows the exact
flows existing in their data center and can run some sort of analytics on
that. This obviously becomes harder once you start aggregating the flows or
with mega flows, but you hopefully get the drift.

The other architecture is based on LPM and LEM lookups.

BTW, when i spoke about the Telco Cloud i had meant pure software based
routing. NO hardware. No baremetals and physical network functions. I had
pure VNFs in my mind,

I see that Mellanox (smartNIC) is also programmed using flows. Hence is it
fair to say that most of the current telco cloud architectures are built
around OVS style flow based network devices?

Glen

>
> --
>   ++ytti
>

Re: akamai yesterday - what in the world was that

[Brandon Martin]

On 2/11/20 6:41 PM, Tom Deligiannis wrote:
There is a major update that has released today, how's everything 
looking for everyone?


I run a couple distinct very small networks.  Both are transit-only with 
no direct peering or local caching and generally sub-gbps.


One set a new 1-min 95% record and did so by nearly 25% over its 
previous record.  The other matched its existing record.  The former 
thankfully has ample capacity, while the latter thankfully did so before 
primetime.


These are definitely going to be harder to manage than the primetime 
hump.  It would be nice if things could drop overnight to hopefully 
spread things out during the daytime lull some.  I understand that some 
people won't pick it up until they get home and turn on devices (which 
is often after school hours for these type of game updates), but 
spreading things out would be really nice especially for those of us 
without local caching.


--
Brandon Martin

Telia and Level3 in Toronto

[Drew Weaver]

Has anyone else noticed a bit of a capacity or packet loss issue between Telia 
and Level3 in Toronto?

In my specific case I was able to avoid it because I had a better route but it 
appears that they are trying to push more traffic through that link than it can 
handle.

Thanks,
-Drew

Re: akamai yesterday - what in the world was that

[Aaron Gould]

Good point Bryan... With my single 10 gig pegged out for a few hours sustained, 
I guess it remains to be seen exactly how high that peak would go if I gave it 
more capacity

-Aaron 
- Original Message -
From: Bryan Holloway 
To: Nanog@nanog.org
Sent: Wed, 12 Feb 2020 07:59:20 -0500 (EST)
Subject: Re: akamai yesterday - what in the world was that

Is 10G enough? ;)

We just lit up several 100G Akamai links. Saved the day fo sho ... (this 
time.)


On 2/11/20 8:26 PM, Aaron Gould wrote:
> Huge!  Big as ever.  My aanp links are (were) pegged, seriously.  I will 
> be contacting Akamai about lighting up an additional 10 gig link to my 
> local clusters.  Started at 12 noon central… still going pretty 
> heavily.  Game/update release ?
> 
> -Aaron
> 
> *From:*Tom Deligiannis [mailto:tom.deligian...@gmail.com]
> *Sent:* Tuesday, February 11, 2020 5:41 PM
> *To:* aar...@gvtc.com
> *Cc:* Nanog@nanog.org
> *Subject:* Re: akamai yesterday - what in the world was that
> 
> There is a major update that has released today, how's everything 
> looking for everyone?
> 
> Tom
> 

Re: akamai yesterday - what in the world was that

[Bryan Holloway]

Is 10G enough? ;)

We just lit up several 100G Akamai links. Saved the day fo sho ... (this 
time.)



On 2/11/20 8:26 PM, Aaron Gould wrote:
Huge!  Big as ever.  My aanp links are (were) pegged, seriously.  I will 
be contacting Akamai about lighting up an additional 10 gig link to my 
local clusters.  Started at 12 noon central… still going pretty 
heavily.  Game/update release ?


-Aaron

*From:*Tom Deligiannis [mailto:tom.deligian...@gmail.com]
*Sent:* Tuesday, February 11, 2020 5:41 PM
*To:* aar...@gvtc.com
*Cc:* Nanog@nanog.org
*Subject:* Re: akamai yesterday - what in the world was that

There is a major update that has released today, how's everything 
looking for everyone?


Tom