Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[Thomas King]

Hi Ren et al.,

Thanks for pointing out that some peers do not use the route servers. This 
group can be subdivided in a group of peers not sending any IP prefixes to the 
route servers while maintaining a route server BGP session, and a group of 
peers not even connecting to the route server. The latter do not show up in the 
"BGP session established" section even if they have applied the required IPv4 
changes.

Best regards,
Thomas


On 31.01.19, 02:32, "valdis.kletni...@vt.edu"  wrote:

On Wed, 30 Jan 2019 23:55:40 +, "i3D.net - Martijn Schmidt" said:

> Here: all networks that didn't already change their peering IP are not 
> yet connected to the updated route-server. Some networks are not 
> connected to any route-server. Therefore, those networks did not yet 
> change their peering IP.
>
> I think you can see what's wrong with that statement.. it does not 
> follow. That has nothing to do with peering department resources, but 
> everything to do with the chosen peering strategy.

Under what conditions would somebody be present at the exchange and
not talking to the route server *at all* before the IP change?



smime.p7s
Description: S/MIME cryptographic signature

Re: BGP Experiment

[hank]

On 23/01/2019 19:40, Job Snijders wrote:

I agree with Job.  Continue the experiment and warn us in advance.

-Hank


Dear Ben, all,

I'm not sure this experiment should be canceled. On the public Internet
we MUST assume BGP speakers are compliant with the BGP-4 protocol.
Broken BGP-4 speakers are what they are: broken. They must be fixed, or
the operator must accept the consequences.

"Get a sandbox like every other researcher" is not a fair statement, one
can also posit "Get a compliant BGP-4 implementation like every other
network operator".

When bad guys explicitly seek to target these Asian and Australian
operators you reference (who apparently have not upgraded to the vendor
recommended release), using *valid* BGP updates, will a politely emailed
request help resolve the situation? Of course not!

Stopping the experiment is only treating symptoms, the root cause must
be addressed: broken software.

Kind regards,

Job

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Mark Andrews]

Don’t forget the reverse tree as well.

> On 31 Jan 2019, at 5:40 pm, Hank Nussbacher  wrote:
> 
> On 31/01/2019 07:18, Mark Andrews wrote:
> 
> There is some secret, silent block on my postings to NANOG that the admins 
> have not yet discovered.  In the interim can one of you please proxy to the 
> list that people should not forget about testing their inverse as well - 
> *.in-addr.arpa via https://ednscomp.isc.org/ednscomp
> 
> Regards,
> Hank

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742  INTERNET: ma...@isc.org

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Mark Andrews]

The only ones that could potentially make a “breaking change” on the Feb 1 are 
Google, Cloudflare and Quad9.  They are the public DNS recursive server 
operators that have committed to removing work arounds.  Google has already 
stated publicly that it will be introducing changes gradually and I expect the 
other to also do so.  Name server developers DO NOT have that power.

Google, Cloudflare and Quad9 are needed so the collectively we don’t need to 
deal with “but it works with …”.  That also the reason for the rest of the 
vendors doing it in unison.

What is needed next is for infrastructure zone operators to down load the 
compliance tools and run them on the servers listed in their zones daily and 
then inform the owners of those delegations that their zones are on 
non-compliant servers and give them a dead line to fix them (120 days should be 
enough time).  If the servers aren’t fixed by the dead line the delegation is 
removed until the servers are fixed or replaced with compliant ones.  This will 
catch operators who install out-of-compliance servers and firewalls.  The 
reaction so far by DNS server operators to DNS flag day shows that this is not 
actually unreasonable to require.  The fixed code is out there for both name 
servers and firewalls.

Mark

> On 31 Jan 2019, at 2:49 pm, Christopher Morrow  
> wrote:
> 
> 
> 
> On Wed, Jan 30, 2019 at 6:23 PM Mark Andrews  wrote:
> You do realise that when the day was chosen it was just the date after which 
> new versions of name servers by the original group of Open Source DNS
> 
> you do realize you are proposing to make a breaking change (breaking change 
> to a global system) on a friday.
> delaying until the following monday would not have mattered to you, I'm sure 
> it's going to matter to other folks though.
> 
> thanks,
> -chris
>  
> developers would not have the work arounds incorporated?
> 
> For ISC that will be BIND 9.14.0 and no that will not be available Feb 1 but 
> you can use the development version 9.13 which has had the code for a while 
> now. 
> 
> Individual operators of resolvers will make their own decisions about when to 
> deploy. 
> -- 
> Mark Andrews
> 
> On 31 Jan 2019, at 12:55, Christopher Morrow  wrote:
> 
>> 
>> 
>> On Wed, Jan 30, 2019 at 5:41 PM Jim Popovitch via NANOG  
>> wrote:
>> On Wed, 2019-01-30 at 17:22 -0800, Matthew Petach wrote:
>> > Any chance this could wait until say the Tuesday 
>> > *after* the Superbowl, when we aren't cutting an 
>> > entire religion's worth of potential workers out of 
>> > the workforce available to fix issues in case it 
>> > turns out to be a bigger problem than is expected, 
>> > and when we have less chance of annoying the 
>> > vast army of football-loving fans of every sort? 
>> 
>> IIRC, DNS Flag Day was announce way before last years Super Bowl...
>> what did the people who aren't ready for DNS Flag Day do in the past
>> 364 days that they need a few more days to get ready for?
>> 
>> 
>> Oh, so they had 365 days to plan the time of the event and still picked a 
>> friday for that event?
>> 
>> https://www.opsview.com/resources/system-administrator/blog/three-reasons-why-not-make-major-it-changes-fridays
>> 
>> I see. 

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742  INTERNET: ma...@isc.org

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Jimmy Hess]

On Wed, Jan 30, 2019 at 9:51 PM Christopher Morrow
 wrote:
> you do realize you are proposing to make a breaking change (breaking change to
> a global system) on a friday.  delaying until the following monday would not 
> have

There are reasons to prefer a Friday over other days as well, but the
internet doesn't
schedule around random participant's personal preferences.

Besides,  its a substantial misrepresentation of what the DNS Flag day
is to describe it
as a "breaking change"  made on a certain date  - changes won't finish
in a week, changes
won't finish in two weeks  --- every day of the week may be affected
until the gradual process
of every OS and DNS vendor releasing and every end user upgrading finishes.

Each software vendor and service provider will have their very own
update schedules  regarding
on what exact date the next  version release and every manager of a system with
a DNS Resolver software installation  will have their own  choice on when they
actually install the next update at their site.

Just because all the major maintainers of DNS resolver software agree
all releases after
tomorrow  will remove the workarounds for broken DNS servers/firewalls
that silently discard
queries does not mean every software vendor is shipping their new code
to release on Feb 1,
_and_  every end user is  rushing to upgrade their DNS resolver to
remove the workarounds.

--
-JH

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Christopher Morrow]

On Wed, Jan 30, 2019 at 8:10 PM Keith Medcalf  wrote:

>
> The best time is usually a Wednesday at Noon or 11:00 in the impacted
> timezone.  Of course, if the impact is worldwide then that would probably
> be UT1 :)


that still sounds like: "not friday" right?
>

Calling Jason Lixfeld @ DE-CIX NY

[Marijana Novakovic via NANOG]

Greetings Jason,

Do you see your new IPv4 session UP at DE-CIX NY with LinkedIn? Thanks !

Kind regards, Mara

RE: DNS Flag Day, Friday, Feb 1st, 2019

[Keith Medcalf]

The best time is usually a Wednesday at Noon or 11:00 in the impacted timezone. 
 Of course, if the impact is worldwide then that would probably be UT1 :)

---
The fact that there's a Highway to Hell but only a Stairway to Heaven says a 
lot about anticipated traffic volume.


>-Original Message-
>From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Christopher
>Morrow
>Sent: Wednesday, 30 January, 2019 18:55
>To: Jim Popovitch
>Cc: nanog
>Subject: Re: DNS Flag Day, Friday, Feb 1st, 2019
>
>
>
>On Wed, Jan 30, 2019 at 5:41 PM Jim Popovitch via NANOG
> wrote:
>
>
>   On Wed, 2019-01-30 at 17:22 -0800, Matthew Petach wrote:
>   > Any chance this could wait until say the Tuesday
>   > *after* the Superbowl, when we aren't cutting an
>   > entire religion's worth of potential workers out of
>   > the workforce available to fix issues in case it
>   > turns out to be a bigger problem than is expected,
>   > and when we have less chance of annoying the
>   > vast army of football-loving fans of every sort?
>
>   IIRC, DNS Flag Day was announce way before last years Super
>Bowl...
>   what did the people who aren't ready for DNS Flag Day do in the
>past
>   364 days that they need a few more days to get ready for?
>
>
>
>
>Oh, so they had 365 days to plan the time of the event and still
>picked a friday for that event?
>
>https://www.opsview.com/resources/system-administrator/blog/three-
>reasons-why-not-make-major-it-changes-fridays
>
>I see.

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Christopher Morrow]

On Wed, Jan 30, 2019 at 6:23 PM Mark Andrews  wrote:

> You do realise that when the day was chosen it was just the date after
> which new versions of name servers by the original group of Open Source DNS
>

you do realize you are proposing to make a breaking change (breaking change
to a global system) on a friday.
delaying until the following monday would not have mattered to you, I'm
sure it's going to matter to other folks though.

thanks,
-chris


> developers would not have the work arounds incorporated?
>
> For ISC that will be BIND 9.14.0 and no that will not be available Feb 1
> but you can use the development version 9.13 which has had the code for a
> while now.
>
> Individual operators of resolvers will make their own decisions about when
> to deploy.
> --
> Mark Andrews
>
> On 31 Jan 2019, at 12:55, Christopher Morrow 
> wrote:
>
>
>
> On Wed, Jan 30, 2019 at 5:41 PM Jim Popovitch via NANOG 
> wrote:
>
>> On Wed, 2019-01-30 at 17:22 -0800, Matthew Petach wrote:
>> > Any chance this could wait until say the Tuesday
>> > *after* the Superbowl, when we aren't cutting an
>> > entire religion's worth of potential workers out of
>> > the workforce available to fix issues in case it
>> > turns out to be a bigger problem than is expected,
>> > and when we have less chance of annoying the
>> > vast army of football-loving fans of every sort?
>>
>> IIRC, DNS Flag Day was announce way before last years Super Bowl...
>> what did the people who aren't ready for DNS Flag Day do in the past
>> 364 days that they need a few more days to get ready for?
>>
>>
> Oh, so they had 365 days to plan the time of the event and still picked a
> friday for that event?
>
>
> https://www.opsview.com/resources/system-administrator/blog/three-reasons-why-not-make-major-it-changes-fridays
>
> I see.
>
>

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Mark Andrews]

You do realise that when the day was chosen it was just the date after which 
new versions of name servers by the original group of Open Source DNS 
developers would not have the work arounds incorporated?

For ISC that will be BIND 9.14.0 and no that will not be available Feb 1 but 
you can use the development version 9.13 which has had the code for a while 
now. 

Individual operators of resolvers will make their own decisions about when to 
deploy. 
-- 
Mark Andrews

> On 31 Jan 2019, at 12:55, Christopher Morrow  wrote:
> 
> 
> 
>> On Wed, Jan 30, 2019 at 5:41 PM Jim Popovitch via NANOG  
>> wrote:
>> On Wed, 2019-01-30 at 17:22 -0800, Matthew Petach wrote:
>> > Any chance this could wait until say the Tuesday 
>> > *after* the Superbowl, when we aren't cutting an 
>> > entire religion's worth of potential workers out of 
>> > the workforce available to fix issues in case it 
>> > turns out to be a bigger problem than is expected, 
>> > and when we have less chance of annoying the 
>> > vast army of football-loving fans of every sort? 
>> 
>> IIRC, DNS Flag Day was announce way before last years Super Bowl...
>> what did the people who aren't ready for DNS Flag Day do in the past
>> 364 days that they need a few more days to get ready for?
>> 
> 
> Oh, so they had 365 days to plan the time of the event and still picked a 
> friday for that event?
> 
> https://www.opsview.com/resources/system-administrator/blog/three-reasons-why-not-make-major-it-changes-fridays
> 
> I see. 

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Jim Popovitch via NANOG]

On January 31, 2019 1:55:26 AM UTC, Christopher Morrow 
 wrote:
>On Wed, Jan 30, 2019 at 5:41 PM Jim Popovitch via NANOG
>
>wrote:
>
>> On Wed, 2019-01-30 at 17:22 -0800, Matthew Petach wrote:
>> > Any chance this could wait until say the Tuesday
>> > *after* the Superbowl, when we aren't cutting an
>> > entire religion's worth of potential workers out of
>> > the workforce available to fix issues in case it
>> > turns out to be a bigger problem than is expected,
>> > and when we have less chance of annoying the
>> > vast army of football-loving fans of every sort?
>>
>> IIRC, DNS Flag Day was announce way before last years Super Bowl...
>> what did the people who aren't ready for DNS Flag Day do in the past
>> 364 days that they need a few more days to get ready for?
>>
>>
>Oh, so they had 365 days to plan the time of the event and still picked
>a
>friday for that event?
>
>https://www.opsview.com/resources/system-administrator/blog/three-reasons-why-not-make-major-it-changes-fridays
>
>I see.

Well, you are either ready or you're not ...doing it on a Tuesday morning is 
not going to make any difference at this point.

-Jim P.

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Mark Andrews]

This basically affects sites using really old Windows DNS servers (Microsoft 
decided to make them only respond once with FORMERR so if that message is lost 
they appear to be dead until the timer clears) and those using firewalls that 
block EDNS queries.  If you use such firewalls they are really doing nothing 
useful. 

Most of the other errors reported are benign as far as DNS flag day is 
concerned. 

Also apart from the public DNS resolvers people need to install updated 
software that has the work arounds removed.

Mark
-- 
Mark Andrews

> On 31 Jan 2019, at 12:22, Matthew Petach  wrote:
> 
> 
> 
>> On Wed, Jan 23, 2019 at 4:12 PM Brian Kantor  wrote:
>> Quoting from the web site at https://dnsflagday.net/
> [...] 
>>   The current DNS is unnecessarily slow and suffers from inability  
>>   to deploy new features. To remediate these problems, vendors of
>>   DNS software and also big public DNS providers are going to
>>   remove certain workarounds on February 1st, 2019.
> 
> 
> I would like to note that there is an entire 
> segment of the population that does not 
> interact with technology between sundown 
> on Friday, all the way through Sunday 
> morning.
> 
> Choosing Friday as a day to carry out an 
> operational change of this sort does not 
> seem to have given thought that if things 
> break, there is a possibility they will have 
> to stay broken for at least a full day before 
> the right people can be engaged to work on 
> the issue. 
> 
> In the future, can we try to schedule such events 
> with more consideration on which day the change 
> will take place?
> 
> I will also note that this weekend is the Superbowl 
> in the US; one of the bigger advertising events of the 
> year.  Potentially breaking advertising systems that 
> rely on DNS two days before a major, once-a-year 
> advertising event is *also* somewhat inconsiderate. 
> 
> While I understand that no day will work for everyone, 
> and at some point you just have to pick a day and go 
> for it, I will note that picking the Friday before the 
> Superbowl does seem like a very unfortunate random 
> pick for a day on which to do it. 
> 
> Any chance this could wait until say the Tuesday 
> *after* the Superbowl, when we aren't cutting an 
> entire religion's worth of potential workers out of 
> the workforce available to fix issues in case it 
> turns out to be a bigger problem than is expected, 
> and when we have less chance of annoying the 
> vast army of football-loving fans of every sort? 
> 
> Thanks!
> 
> Matt
> 

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Christopher Morrow]

On Wed, Jan 30, 2019 at 5:41 PM Jim Popovitch via NANOG 
wrote:

> On Wed, 2019-01-30 at 17:22 -0800, Matthew Petach wrote:
> > Any chance this could wait until say the Tuesday
> > *after* the Superbowl, when we aren't cutting an
> > entire religion's worth of potential workers out of
> > the workforce available to fix issues in case it
> > turns out to be a bigger problem than is expected,
> > and when we have less chance of annoying the
> > vast army of football-loving fans of every sort?
>
> IIRC, DNS Flag Day was announce way before last years Super Bowl...
> what did the people who aren't ready for DNS Flag Day do in the past
> 364 days that they need a few more days to get ready for?
>
>
Oh, so they had 365 days to plan the time of the event and still picked a
friday for that event?

https://www.opsview.com/resources/system-administrator/blog/three-reasons-why-not-make-major-it-changes-fridays

I see.

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Jim Popovitch via NANOG]

On Wed, 2019-01-30 at 17:22 -0800, Matthew Petach wrote:
> Any chance this could wait until say the Tuesday 
> *after* the Superbowl, when we aren't cutting an 
> entire religion's worth of potential workers out of 
> the workforce available to fix issues in case it 
> turns out to be a bigger problem than is expected, 
> and when we have less chance of annoying the 
> vast army of football-loving fans of every sort? 

IIRC, DNS Flag Day was announce way before last years Super Bowl...
what did the people who aren't ready for DNS Flag Day do in the past
364 days that they need a few more days to get ready for?

-Jim P.

Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[Mike Hammett]

Some companies just don't join route servers as a policy. It can be annoying if 
you want to talk to them, but I understand there can be various reasons why. It 
gets very annoying when the peering department isn't responsive to manual 
peering requests when they're not on the route server because then they might 
as well not be there at all, as far as you're concerned. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "valdis kletnieks"  
To: "i3D.net - Martijn Schmidt"  
Cc: "North American Network Operators' Group"  
Sent: Wednesday, January 30, 2019 7:32:17 PM 
Subject: Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY 

On Wed, 30 Jan 2019 23:55:40 +, "i3D.net - Martijn Schmidt" said: 

> Here: all networks that didn't already change their peering IP are not 
> yet connected to the updated route-server. Some networks are not 
> connected to any route-server. Therefore, those networks did not yet 
> change their peering IP. 
> 
> I think you can see what's wrong with that statement.. it does not 
> follow. That has nothing to do with peering department resources, but 
> everything to do with the chosen peering strategy. 

Under what conditions would somebody be present at the exchange and 
not talking to the route server *at all* before the IP change? 

Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[valdis . kletnieks]

On Wed, 30 Jan 2019 23:55:40 +, "i3D.net - Martijn Schmidt" said:

> Here: all networks that didn't already change their peering IP are not 
> yet connected to the updated route-server. Some networks are not 
> connected to any route-server. Therefore, those networks did not yet 
> change their peering IP.
>
> I think you can see what's wrong with that statement.. it does not 
> follow. That has nothing to do with peering department resources, but 
> everything to do with the chosen peering strategy.

Under what conditions would somebody be present at the exchange and
not talking to the route server *at all* before the IP change?

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Matthew Petach]

On Wed, Jan 23, 2019 at 4:12 PM Brian Kantor  wrote:

> Quoting from the web site at https://dnsflagday.net/
>
[...]

>   The current DNS is unnecessarily slow and suffers from inability
>   to deploy new features. To remediate these problems, vendors of
>   DNS software and also big public DNS providers are going to
>   remove certain workarounds on February 1st, 2019.
>

I would like to note that there is an entire
segment of the population that does not
interact with technology between sundown
on Friday, all the way through Sunday
morning.

Choosing Friday as a day to carry out an
operational change of this sort does not
seem to have given thought that if things
break, there is a possibility they will have
to stay broken for at least a full day before
the right people can be engaged to work on
the issue.

In the future, can we try to schedule such events
with more consideration on which day the change
will take place?

I will also note that this weekend is the Superbowl
in the US; one of the bigger advertising events of the
year.  Potentially breaking advertising systems that
rely on DNS two days before a major, once-a-year
advertising event is *also* somewhat inconsiderate.

While I understand that no day will work for everyone,
and at some point you just have to pick a day and go
for it, I will note that picking the Friday before the
Superbowl does seem like a very unfortunate random
pick for a day on which to do it.

Any chance this could wait until say the Tuesday
*after* the Superbowl, when we aren't cutting an
entire religion's worth of potential workers out of
the workforce available to fix issues in case it
turns out to be a bigger problem than is expected,
and when we have less chance of annoying the
vast army of football-loving fans of every sort?

Thanks!

Matt

Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[i3D . net - Martijn Schmidt]

On 1/31/19 12:36 AM, Töma Gavrichenkov wrote:
> On Thu, Jan 30, 2019 at 23:10 AM Ren Provo  wrote:
>> You probably should remove sessions with networks
>> explicitly *not* participating in route servers versus
>> displaying them on a global shame list.
> And so it begins — yet another discussion on what does the word
> "responsibility" really mean.
>
> Given that e.g. the peering facility in Amazon, according to an
> adjacent NANOG ML thread, is in deep deep trouble since Nov 2018, just
> shutting down sessions with all of the entries in that shame list is
> likely to cause huge disruption and disappoinment.
>
> --
> Töma

What triggered that part of the discussion is a logical fallacy along 
the lines of: if A is true, then B is true. B is true, therefore A is true.

Here: all networks that didn't already change their peering IP are not 
yet connected to the updated route-server. Some networks are not 
connected to any route-server. Therefore, those networks did not yet 
change their peering IP.

I think you can see what's wrong with that statement.. it does not 
follow. That has nothing to do with peering department resources, but 
everything to do with the chosen peering strategy.

Best regards,
Martijn

Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[Töma Gavrichenkov]

On Thu, Jan 30, 2019 at 23:10 AM Ren Provo  wrote:
> You probably should remove sessions with networks
> explicitly *not* participating in route servers versus
> displaying them on a global shame list.

And so it begins — yet another discussion on what does the word
"responsibility" really mean.

Given that e.g. the peering facility in Amazon, according to an
adjacent NANOG ML thread, is in deep deep trouble since Nov 2018, just
shutting down sessions with all of the entries in that shame list is
likely to cause huge disruption and disappoinment.

--
Töma

Re: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

[Matthew Petach]

On Wed, Jan 30, 2019 at 9:07 AM Christopher Morrow 
wrote:

> And here I always figured it was bespoke knit caps for all the packets in
> cold-weather climes?
> learn something new every day! (also, now I wonder what the people who
> told me they were too busy knitting caps are ACTUALLY doing??)
>

Unfortunately, they're knitting *DATA* caps.

;-P

Sorry, couldn't resist.

Matt

RE: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

[bzs]

Re: Fire

Also fire dept response.

I've ridden with the Boston Fire Dept, extreme cold is a major PITA,
hydrants freeze, you have to work in it going from the heat of the
fire to sub-zero air temps over and over, all while getting soaking
wet, and wind-chill is certainly a factor.

There were always cautionary stories about some firefighter having a
heart attack struggling to get a frozen hydrant open. Whether factual
or not I think it makes a point, no water no firefighting in general.

I could tell you about the fire boats in February...sometimes you need
them.

I just saw a news spot (I believe Chicago) where they had to raise to
multiple alarms on a fairly simple tho working house fire just so they
had enough firefighters to cycle them through a warming truck (I don't
remember any warming truck in Boston tho you could go sit in a truck
cab :-))

Which means thinner coverage for other areas.

-- 
-Barry Shein

Software Tool & Die| b...@theworld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: +1 617-STD-WRLD   | 800-THE-WRLD
The World: Since 1989  | A Public Information Utility | *oo*

Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[Ren Provo]

Hi Thomas,

You probably should remove sessions with networks explicitly *not*
participating in route servers versus displaying them on a global shame
list.

Cheers, -ren

On Wed, Jan 30, 2019 at 5:06 PM Thomas King  wrote:

> Hi all,
>
>
>
> Thanks for your support! This helps us getting all peers on the new IPv4
> space.
>
>
>
> Our looking glass shows which peers already have changed the IP settings
> (see section “BGP session established”) and which peers are still working
> on it (see section “BGP sessions down”):
>
> https://lg.de-cix.net/routeservers/rs1_nyc_ipv4#sessions-up
>
>
>
> Best regards,
> Thomas
>
>
>
>
>
> *From: *NANOG  on behalf of James Stankiewicz <
> stankiew...@njedge.net>
> *Date: *Wednesday, 30. January 2019 at 19:32
> *To: *Jared Mauch 
> *Cc: *North American Network Operators' Group 
> *Subject: *Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY
>
>
>
> Microsoft an Edgecast has not yet made there changes.
>
>
>
> On Wed, Jan 30, 2019 at 12:20 PM Jared Mauch 
> wrote:
>
> Akamai is working on doing our part. Apologies.
>
> Sent from my iCar
>
>
> On Jan 30, 2019, at 12:02 PM, Mehmet Akcin  wrote:
>
> Pinged my contacts in each
>
>
>
> On Wed, Jan 30, 2019 at 05:52 Jason Lixfeld 
> wrote:
>
> Hi,
>
> In late October 2018, DE-CIX announced that they would be renumbering
> their IPv4 address block in New York between 01-28-19 and 01-30-19.
>
> This was followed by numerous reminders in months, weeks and even days
> leading up to the renumbering activity.
>
> The renumbering activity has come and gone, but LinkedIn, Amazon and
> Akamai are still using the old IPs.
>
> If three months has gone by and the numerous reminders that have been sent
> have resulted in these organizations still living on the old IP space, it
> seems to me that there may be some sort of a disconnect between who
> receives the notifications from IXPs and how they are filtered upstream.
>
> I’m hopeful that the eyeballs who read this list are some of those folks
> who should have received the notifications from DE-CIX, or can at least
> filter the info back downstream to whoever can perform the renumbering
> activity.
>
> Thanks.
>
> --
>
> Mehmet
> +1-424-298-1903
>
>
>
>
> --
>
> *Jim Stankiewicz*
>
> *Principal Network Architect*
>
> *NJEdge*
>
> *W:855.832.EDGE(3343)*
>
> *c:201.306.4409*
>
> *[image:
> https://docs.google.com/uc?export=download&id=0B15Cb24EwZVsOUhIa0lWbG9ZT2c&revid=0B15Cb24EwZVsdVB2SlJ3ekFEUllPRDZyMGZ5cUtkbko2bWQ0PQ]*
>
>
>
>
>

Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[Thomas King]

Hi all,

 

Thanks for your support! This helps us getting all peers on the new IPv4 space.

 

Our looking glass shows which peers already have changed the IP settings (see 
section “BGP session established”) and which peers are still working on it (see 
section “BGP sessions down”):

https://lg.de-cix.net/routeservers/rs1_nyc_ipv4#sessions-up

 

Best regards,
Thomas

 

 

From: NANOG  on behalf of James Stankiewicz 

Date: Wednesday, 30. January 2019 at 19:32
To: Jared Mauch 
Cc: North American Network Operators' Group 
Subject: Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

 

Microsoft an Edgecast has not yet made there changes.

 

On Wed, Jan 30, 2019 at 12:20 PM Jared Mauch  wrote:

Akamai is working on doing our part. Apologies. 

Sent from my iCar


On Jan 30, 2019, at 12:02 PM, Mehmet Akcin  wrote:

Pinged my contacts in each 

 

On Wed, Jan 30, 2019 at 05:52 Jason Lixfeld  wrote:

Hi,

In late October 2018, DE-CIX announced that they would be renumbering their 
IPv4 address block in New York between 01-28-19 and 01-30-19.

This was followed by numerous reminders in months, weeks and even days leading 
up to the renumbering activity.

The renumbering activity has come and gone, but LinkedIn, Amazon and Akamai are 
still using the old IPs.

If three months has gone by and the numerous reminders that have been sent have 
resulted in these organizations still living on the old IP space, it seems to 
me that there may be some sort of a disconnect between who receives the 
notifications from IXPs and how they are filtered upstream.

I’m hopeful that the eyeballs who read this list are some of those folks who 
should have received the notifications from DE-CIX, or can at least filter the 
info back downstream to whoever can perform the renumbering activity.

Thanks.

-- 

Mehmet
+1-424-298-1903


 

-- 

Jim Stankiewicz

Principal Network Architect

NJEdge

W:855.832.EDGE(3343)

c:201.306.4409

 

 



smime.p7s
Description: S/MIME cryptographic signature

Re: Amazon Peering

[Tom Beecher]

I'm sure ~ $20k/yr in time cost alone per 10G has nothing to do with
that... :p

Although to be fair, the individual from Amazon who my peering group has
been working with after my first message has been really, really great. As
with many things, the people are great, just not enough resources I'm sure.

On Wed, Jan 30, 2019 at 1:39 PM Denis Fondras  wrote:

> > Yup, super professional of them.
> >
>
> Have you tried to order a port on DirectConnect to check if it was
> hassleless ?
> :p
>

Re: Amazon Peering

[Denis Fondras]

> Yup, super professional of them.
> 

Have you tried to order a port on DirectConnect to check if it was hassleless ?
:p

Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[Marijana Novakovic via NANOG]

Greetings Jason,

Thank you for your kind feedback, we have CM planned for today to do the needed 
change.

Kind regards, Mara



From: NANOG  on behalf of Mike Hammett 

Sent: Wednesday, January 30, 2019 8:41 AM
To: Jason Lixfeld
Cc: North American Network Operators' Group
Subject: Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

A lot of huge companies apparently find it tough to find the $75k to hire one 
more peering person. Not all, though. For many, everything just runs like 
clockwork.



-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com


From: "Jason Lixfeld" 
To: "North American Network Operators' Group" 
Sent: Wednesday, January 30, 2019 7:52:09 AM
Subject: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

Hi,

In late October 2018, DE-CIX announced that they would be renumbering their 
IPv4 address block in New York between 01-28-19 and 01-30-19.

This was followed by numerous reminders in months, weeks and even days leading 
up to the renumbering activity.

The renumbering activity has come and gone, but LinkedIn, Amazon and Akamai are 
still using the old IPs.

If three months has gone by and the numerous reminders that have been sent have 
resulted in these organizations still living on the old IP space, it seems to 
me that there may be some sort of a disconnect between who receives the 
notifications from IXPs and how they are filtered upstream.

I’m hopeful that the eyeballs who read this list are some of those folks who 
should have received the notifications from DE-CIX, or can at least filter the 
info back downstream to whoever can perform the renumbering activity.

Thanks.

Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[James Stankiewicz]

Microsoft an Edgecast has not yet made there changes.

On Wed, Jan 30, 2019 at 12:20 PM Jared Mauch  wrote:

> Akamai is working on doing our part. Apologies.
>
> Sent from my iCar
>
> On Jan 30, 2019, at 12:02 PM, Mehmet Akcin  wrote:
>
> Pinged my contacts in each
>
> On Wed, Jan 30, 2019 at 05:52 Jason Lixfeld 
> wrote:
>
>> Hi,
>>
>> In late October 2018, DE-CIX announced that they would be renumbering
>> their IPv4 address block in New York between 01-28-19 and 01-30-19.
>>
>> This was followed by numerous reminders in months, weeks and even days
>> leading up to the renumbering activity.
>>
>> The renumbering activity has come and gone, but LinkedIn, Amazon and
>> Akamai are still using the old IPs.
>>
>> If three months has gone by and the numerous reminders that have been
>> sent have resulted in these organizations still living on the old IP space,
>> it seems to me that there may be some sort of a disconnect between who
>> receives the notifications from IXPs and how they are filtered upstream.
>>
>> I’m hopeful that the eyeballs who read this list are some of those folks
>> who should have received the notifications from DE-CIX, or can at least
>> filter the info back downstream to whoever can perform the renumbering
>> activity.
>>
>> Thanks.
>>
>> --
> Mehmet
> +1-424-298-1903
>
>

-- 
*Jim Stankiewicz*

*Principal Network Architect*
*NJEdge*
*W:855.832.EDGE(3343)*
*c:201.306.4409*

RE: Amazon Peering

[Naslund, Steve]

AKA Too Big To Care.  Happens a lot.

Steven Naslund
Chicago IL

On Wed, Jan 30, 2019 at 11:36 AM Mike Hammett 
mailto:na...@ics-il.net>> wrote:
Oh, you ordered cross connects for a PNI and they stopped responding 
mid-project? Isn't that nice!


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com


From: "Luca Salvatore via NANOG" mailto:nanog@nanog.org>>
To: "North American Network Operators' Group" 
mailto:nanog@nanog.org>>
Sent: Wednesday, January 30, 2019 9:45:29 AM
Subject: Re: Amazon Peering
Similar experiences here with Amazon.  Initially had semi-regular responses 
from their peering team, they issued LOAs, I ordered the x-connects and then 
radio silence for months.
At the point now where I'm disconnecting x-connects since it's a waste of money.

On Tue, Jan 29, 2019 at 10:49 AM Brooks Swinnerton 
mailto:bswinner...@gmail.com>> wrote:
I also saw sessions come up this weekend, no routes yet though.

On Mon, Jan 28, 2019 at 4:56 PM Tom Beecher 
mailto:beec...@beecher.cc>> wrote:
Mike-

Definitely moving forward now. Someone from Amazon was working with my peering 
group and things started coming up this weekend, so it seems like they're 
catching up pretty good now.

On Thu, Jan 24, 2019 at 2:45 PM Mike Hammett 
mailto:na...@ics-il.net>> wrote:
Let us know your success as well. I'll hold off following up on my requests 
until I see that other people are successful. I don't want to contribute to 
flooding them with requests.


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com


From: "Tom Beecher" mailto:beec...@beecher.cc>>
To: "Jason Lixfeld" mailto:jason%2bna...@lixfeld.ca>>
Cc: "North American Network Operators' Group" 
mailto:nanog@nanog.org>>
Sent: Thursday, January 24, 2019 1:38:51 PM
Subject: Re: Amazon Peering
Thanks Jason. I'll have my peering team take another crack at reaching out and 
see what happens. Appreciate it!

On Thu, Jan 24, 2019 at 2:21 PM Jason Lixfeld 
mailto:jason%2bna...@lixfeld.ca>> wrote:
We circled back with them yesterday on a request we made in late November where 
at the time they said they wouldn’t be turned up until 2019 due to holiday 
network change freeze.

They responded within about 4 hours, thanked us for our patience and 
understanding and said we should expect them to be turned up in about 6 weeks, 
which is apparently their typical timing.

On Jan 24, 2019, at 2:13 PM, Tom Beecher 
mailto:beec...@beecher.cc>> wrote:

I hate to necro-thread , but has anyone seen any movement from Amazon on this? 
I just got a Strongly Worded Message about it, and according to my peering team 
, it's been radio silence for months.


On Sat, Nov 24, 2018 at 12:32 PM JASON BOTHE via NANOG 
mailto:nanog@nanog.org>> wrote:
This is a note I received on Oct18 when checking on a peering request submitted 
on Aug7..

“Apologies for the delays here. We have temporarily frozen IX peering as we 
revise some of our automation processes. I’m hopeful this will be unblocked by 
early November. Thank you for your continued patience.”

On Nov 24, 2018, at 10:59, Darin Steffl 
mailto:darin.ste...@mnwifi.com>> wrote:
It seems wasteful for Amazon to connect to an IX but then ignore peering 
requests for a year.

They have 40G of connectivity but are unresponsive. I'll try emailing all the 
other contacts listed in peeringdb.

Thanks

On Sat, Nov 24, 2018, 10:38 AM Mike Hammett 
mailto:na...@ics-il.net> wrote:
I've e-mailed my contacts there a couple times on people's behalf. No response 
yet.

It seems like a lot of organizations need 1 more person in their peering 
departments.


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com


From: "Darin Steffl" mailto:darin.ste...@mnwifi.com>>
To: "North American Network Operators' Group" 
mailto:nanog@nanog.org>>
Sent: Friday, November 23, 2018 10:21:51 PM
Subject: Amazon Peering
Hey all,

Does anyone have a direct contact to get a peering session established with 
Amazon at an IX? I sent a peering request Dec 2017 and two more times this Sept 
and Nov with no response.

I sent to peer...@amazon.com and received one 
automated response back so I know they received my email but nothing since.



--
Darin Steffl
Minnesota WiFi
www.mnwifi.com
507-634-WiFi
[http://www.snoitulosten.com/wp-content/uploads/2010/01/facebook-small.jpg]
 Like us on Facebook

Re: Amazon Peering

[Luca Salvatore via NANOG]

Yup, super professional of them.

On Wed, Jan 30, 2019 at 11:36 AM Mike Hammett  wrote:

> Oh, you ordered cross connects for a PNI and they stopped responding
> mid-project? Isn't that nice!
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
> --
> *From: *"Luca Salvatore via NANOG" 
> *To: *"North American Network Operators' Group" 
> *Sent: *Wednesday, January 30, 2019 9:45:29 AM
> *Subject: *Re: Amazon Peering
>
> Similar experiences here with Amazon.  Initially had semi-regular
> responses from their peering team, they issued LOAs, I ordered the
> x-connects and then radio silence for months.
> At the point now where I'm disconnecting x-connects since it's a waste of
> money.
>
> On Tue, Jan 29, 2019 at 10:49 AM Brooks Swinnerton 
> wrote:
>
>> I also saw sessions come up this weekend, no routes yet though.
>>
>> On Mon, Jan 28, 2019 at 4:56 PM Tom Beecher  wrote:
>>
>>> Mike-
>>>
>>> Definitely moving forward now. Someone from Amazon was working with my
>>> peering group and things started coming up this weekend, so it seems like
>>> they're catching up pretty good now.
>>>
>>> On Thu, Jan 24, 2019 at 2:45 PM Mike Hammett  wrote:
>>>
 Let us know your success as well. I'll hold off following up on my
 requests until I see that other people are successful. I don't want to
 contribute to flooding them with requests.



 -
 Mike Hammett
 Intelligent Computing Solutions
 http://www.ics-il.com

 Midwest-IX
 http://www.midwest-ix.com

 --
 *From: *"Tom Beecher" 
 *To: *"Jason Lixfeld" 
 *Cc: *"North American Network Operators' Group" 
 *Sent: *Thursday, January 24, 2019 1:38:51 PM
 *Subject: *Re: Amazon Peering

 Thanks Jason. I'll have my peering team take another crack at reaching
 out and see what happens. Appreciate it!

 On Thu, Jan 24, 2019 at 2:21 PM Jason Lixfeld 
 wrote:

> We circled back with them yesterday on a request we made in late
> November where at the time they said they wouldn’t be turned up until 2019
> due to holiday network change freeze.
>
> They responded within about 4 hours, thanked us for our patience and
> understanding and said we should expect them to be turned up in about 6
> weeks, which is apparently their typical timing.
>
> On Jan 24, 2019, at 2:13 PM, Tom Beecher  wrote:
>
> I hate to necro-thread , but has anyone seen any movement from Amazon
> on this? I just got a Strongly Worded Message about it, and according to 
> my
> peering team , it's been radio silence for months.
>
>
> On Sat, Nov 24, 2018 at 12:32 PM JASON BOTHE via NANOG <
> nanog@nanog.org> wrote:
>
>> This is a note I received on Oct18 when checking on a peering request
>> submitted on Aug7..
>>
>> “Apologies for the delays here. We have temporarily frozen IX peering
>> as we revise some of our automation processes. I’m hopeful this will be
>> unblocked by early November. Thank you for your continued patience.”
>>
>> On Nov 24, 2018, at 10:59, Darin Steffl 
>> wrote:
>>
>> It seems wasteful for Amazon to connect to an IX but then ignore
>> peering requests for a year.
>>
>> They have 40G of connectivity but are unresponsive. I'll try emailing
>> all the other contacts listed in peeringdb.
>>
>> Thanks
>>
>> On Sat, Nov 24, 2018, 10:38 AM Mike Hammett >
>>> I've e-mailed my contacts there a couple times on people's behalf.
>>> No response yet.
>>>
>>> It seems like a lot of organizations need 1 more person in their
>>> peering departments.
>>>
>>>
>>>
>>> -
>>> Mike Hammett
>>> Intelligent Computing Solutions
>>> http://www.ics-il.com
>>>
>>> Midwest-IX
>>> http://www.midwest-ix.com
>>>
>>> --
>>> *From: *"Darin Steffl" 
>>> *To: *"North American Network Operators' Group" 
>>> *Sent: *Friday, November 23, 2018 10:21:51 PM
>>> *Subject: *Amazon Peering
>>>
>>> Hey all,
>>>
>>> Does anyone have a direct contact to get a peering session
>>> established with Amazon at an IX? I sent a peering request Dec 2017 and 
>>> two
>>> more times this Sept and Nov with no response.
>>>
>>> I sent to peer...@amazon.com and received one automated response
>>> back so I know they received my email but nothing since.
>>>
>>>
>>>
>>> --
>>> Darin Steffl
>>> Minnesota WiFi
>>> www.mnwifi.com
>>> 507-634-WiFi
>>>  Like us on Facebook
>>> 
>>>
>>>
>

>

Re: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

[Tom Beecher]

Well said. The electrical load shifts, hydraulic systems, airflows
constrained by ice cover, etc, etc, etc. All kinds of things being asked to
do stuff outside or at the edge of specifications.

Hug your local facilities guys when these things happen. (Or bring them
booze.)

On Wed, Jan 30, 2019 at 12:58 PM Naslund, Steve 
wrote:

> >And apparently fire. I wasn’t going to chime in but one of my >providers
> *just* alerted us to an electrical fire in a Minneapolis pop >causing
> loads to failover to ups. Unknown whether weather >conditions contributed
> to the incident.
>
>
>
> Yes, in Chicago we will see an increase in home fires because heating
> systems are being pushed to their limits and people tend to do stupid
> things like run unattended space heaters and try to thaw frozen stuff in
> crazy ways.  In a datacenter you might be pushing electrical loads while
> external electrical components are stressed with temperature.  I have seen
> transformer fires caused by the oil inside not circulating correctly.  You
> end up with hot and cold zones in them.
>
>
>
> Steven Naslund
>
> Chicago IL
>
>

RE: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

[Naslund, Steve]

>And apparently fire. I wasn’t going to chime in but one of my >providers 
>*just* alerted us to an electrical fire in a Minneapolis pop >causing loads to 
>failover to ups. Unknown whether weather >conditions contributed to the 
>incident.

Yes, in Chicago we will see an increase in home fires because heating systems 
are being pushed to their limits and people tend to do stupid things like run 
unattended space heaters and try to thaw frozen stuff in crazy ways.  In a 
datacenter you might be pushing electrical loads while external electrical 
components are stressed with temperature.  I have seen transformer fires caused 
by the oil inside not circulating correctly.  You end up with hot and cold 
zones in them.

Steven Naslund
Chicago IL

RE: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

[Naslund, Steve]

Ironically you don’t really save a lot of energy when it’s this cold because 
the loops are running at high speed and the humidification coils are working 
overtime to keep the RH up in the room.

People think we can bring in all the outside cold we want but the issue then is 
humidity stability.

Steven Naslund
Chicago IL

Re: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

[Mark Tinka]

On 30/Jan/19 19:37, Naslund, Steve wrote:

>
>  
>
> A good HVAC team is critical because we have noted that the building
> management systems often are not flexible enough to automatically deal
> with super extremes and require some human intervention to tell them
> to do things like run heat and cooling simultaneously.  Other actions
> like closing down louvers on evaporators may or may not be automated
> depending on your systems.  If any part of the system does fail in
> these conditions you have to move super quick or you could get serious
> damage fast.  Our biggest monitoring points are flow rate/pressure
> (which could indicate a freeze up beginning or a pump failing), output
> and return temp on the loops.
>

This was one of the things I was concerned about, fluids going smudgy or
just simply freezing up...

Mark.

Re: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

[Paul Zugnoni via NANOG]

And apparently fire. I wasn’t going to chime in but one of my providers
*just* alerted us to an electrical fire in a Minneapolis pop causing loads
to failover to ups. Unknown whether weather conditions contributed to the
incident. PZ

On Wed, Jan 30, 2019 at 09:25 Naslund, Steve  wrote:

> >To the 'infrastructure' question, I think the biggest concerns would >be
> power related. Although we have a DC in Buffalo that is cooled >on
> ambient outside air that has the opposite problem ; it's TOO cold >at the
> moment, so we are cycling most of the hot server exhaust >back into the
> computer rooms to maintain temperatures.
>
>
>
> Exactly what he said.  We actually run cooling and supplemental heating in
> extreme cold.  We need to keep the chiller pulling heat into itself and
> pumps moving on high to keep the outdoor components from freezing up.
> During the summer you might run close to or slightly below freezing on the
> coolant loops but in these conditions you cannot run that low a temp
> because things will freeze up before the coolant returns.  We also have to
> keep the room reasonably warm (50F +).  You also need to watch out for fast
> temp excursions to keep humidity under control.
>
>
>
> The wind speed does make some difference since it is like a fan on your
> evaporator pulling heat out of the cooling loops faster than still air will.
>
>
>
> Steven Naslund
>
> Chicago IL
>
-- 
PZ
Head of Datacenter and Network Infrastructure, Wish
p...@wish.com +1-650-313-3458

RE: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

[Naslund, Steve]

>Exactly what he said.  We actually run cooling and supplemental heating >in 
>extreme cold.  We need to keep the chiller pulling heat into itself and >pumps 
>moving on high to keep the outdoor components from freezing >up.  During the 
>summer you might run close to or slightly below freezing >on the coolant loops 
>but in these conditions you cannot run that low a >temp because things will 
>freeze up before the coolant returns.  We also >have to keep the room 
>reasonably warm (50F +).  You also need to watch >out for fast temp excursions 
>to keep humidity under control.

A good HVAC team is critical because we have noted that the building management 
systems often are not flexible enough to automatically deal with super extremes 
and require some human intervention to tell them to do things like run heat and 
cooling simultaneously.  Other actions like closing down louvers on evaporators 
may or may not be automated depending on your systems.  If any part of the 
system does fail in these conditions you have to move super quick or you could 
get serious damage fast.  Our biggest monitoring points are flow rate/pressure 
(which could indicate a freeze up beginning or a pump failing), output and 
return temp on the loops.

Steven Naslund
Chicago IL

RE: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

[Naslund, Steve]

>To the 'infrastructure' question, I think the biggest concerns would >be power 
>related. Although we have a DC in Buffalo that is cooled >on ambient outside 
>air that has the opposite problem ; it's TOO cold >at the moment, so we are 
>cycling most of the hot server exhaust >back into the computer rooms to 
>maintain temperatures.

Exactly what he said.  We actually run cooling and supplemental heating in 
extreme cold.  We need to keep the chiller pulling heat into itself and pumps 
moving on high to keep the outdoor components from freezing up.  During the 
summer you might run close to or slightly below freezing on the coolant loops 
but in these conditions you cannot run that low a temp because things will 
freeze up before the coolant returns.  We also have to keep the room reasonably 
warm (50F +).  You also need to watch out for fast temp excursions to keep 
humidity under control.

The wind speed does make some difference since it is like a fan on your 
evaporator pulling heat out of the cooling loops faster than still air will.

Steven Naslund
Chicago IL

Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[Jared Mauch]

Akamai is working on doing our part. Apologies. 

Sent from my iCar

> On Jan 30, 2019, at 12:02 PM, Mehmet Akcin  wrote:
> 
> Pinged my contacts in each 
> 
>> On Wed, Jan 30, 2019 at 05:52 Jason Lixfeld  wrote:
>> Hi,
>> 
>> In late October 2018, DE-CIX announced that they would be renumbering their 
>> IPv4 address block in New York between 01-28-19 and 01-30-19.
>> 
>> This was followed by numerous reminders in months, weeks and even days 
>> leading up to the renumbering activity.
>> 
>> The renumbering activity has come and gone, but LinkedIn, Amazon and Akamai 
>> are still using the old IPs.
>> 
>> If three months has gone by and the numerous reminders that have been sent 
>> have resulted in these organizations still living on the old IP space, it 
>> seems to me that there may be some sort of a disconnect between who receives 
>> the notifications from IXPs and how they are filtered upstream.
>> 
>> I’m hopeful that the eyeballs who read this list are some of those folks who 
>> should have received the notifications from DE-CIX, or can at least filter 
>> the info back downstream to whoever can perform the renumbering activity.
>> 
>> Thanks.
>> 
> -- 
> Mehmet
> +1-424-298-1903

RE: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

[Naslund, Steve]

The main issue is infrastructure like power, cable damage, and heating/cooling 
systems.

Power lines tend to go down because anything weak becomes brittle and any 
accident involving a pole tends to cause them to break rather than absorb 
impact.  Also, conduits and manholes that normally might be above freezing 
underground can have standing water freeze breaking splice cases and such.

Large chiller plants need to be run at higher temperatures and speeds to keep 
any outdoor components from freezing up.

Of course, repairs on anything outdoors tend to take a lot longer to resolve.  
Anything that requires digging might be near impossible in these conditions.

So far though in my area (Chicago) my carriers AT&T, Comcast, Cogent, and Level 
3 all seem to be fine so far.  Our current temp is -18.  Wind chill reports at 
-50 to -60 depending where you are.

Steven Naslund
Chicago IL

From: NANOG  On Behalf Of Mark Tinka
Sent: Wednesday, January 30, 2019 10:38 AM
To: North American Network Operators' Group 
Subject: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

For anyone running IP networks in the Midwest, are you having to do anything 
special to keep your networks up?

For the data centres, is this cold front a chance to reduce air conditioning 
costs, or is it actually straining the infrastructure?

I'm curious, from a +27-degree C summer's day here in Johannesburg.

Mark.

Re: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

[Tom Beecher]

To be fair, reporting the the wind chill factor is very meaningful for
health and safety reasons almost everywhere so proper warning is given
about people spending time outside. Minneapolis, and the bigger Canadian
cities have those inside walkways and pedestrian pathways, but they're not
that common elsewhere. I don't think Chicago does for example, and we don't
have that here in Buffalo. Contrary to the rumors, 0F with -40F wind chills
are NOT very common around here.

People need to be warned to take this weather seriously. You might be used
to it, but not everyone in a native that can say that.

To the 'infrastructure' question, I think the biggest concerns would be
power related. Although we have a DC in Buffalo that is cooled on ambient
outside air that has the opposite problem ; it's TOO cold at the moment, so
we are cycling most of the hot server exhaust back into the computer rooms
to maintain temperatures.

On Wed, Jan 30, 2019 at 11:52 AM Mel Beckman  wrote:

> Being a Minnesota native, I can tell you that while it is indeed cold,
> this is nothing new i the Great White North :)  I am amaze a how
> consistently the media overplays the severity of Midwest cold weather as
> some kind of unique phenomenon. They amplify this by reporting the
> wind-chill factor, which is the “what it feels like” equivalent in a cold
> and windy environment. But equipment feels nothing, so windchill is
> irrelevant.
>
> For example, Minneapolis is -20F, but the news media instead reports “-60F
> wind chill”, which, while dramatic, is not meaningful for most purposes. I
> grew up in Minnesota with -30F and lower quite common, and we walked to
> school in those temperatures. You just have to dress well. Minneapolis is
> paved with tunnels and heated skyways to eliminate most outdoor walking
> downtown.
>
> As far as networks go, none of the ISPs I know of do anything different
> than anywhere else in the country. Everyone has backup power. It’s already
> common practice everywhere to exploit cooler winter ambient temperatures to
> reduce HVAC requirements, so that’s not new either. But it gets as hot in
> the Midwest in our summer as it is in SA for you now, so everyone must
> still build out HVAC capacity to cover the hottest days.
>
>  -mel beckman
>
> On Jan 30, 2019, at 8:40 AM, Mark Tinka  wrote:
>
> For anyone running IP networks in the Midwest, are you having to do
> anything special to keep your networks up?
>
> For the data centres, is this cold front a chance to reduce air
> conditioning costs, or is it actually straining the infrastructure?
>
> I'm curious, from a +27-degree C summer's day here in Johannesburg.
>
> Mark.
>
>

Re: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

[Christopher Morrow]

And here I always figured it was bespoke knit caps for all the packets in
cold-weather climes?
learn something new every day! (also, now I wonder what the people who told
me they were too busy knitting caps are ACTUALLY doing??)

On Wed, Jan 30, 2019 at 8:55 AM Bryan Holloway  wrote:

> Approximately 3 hrs ago we lost B-feed at Minneapolis Cologix.
>
> Apparently the local utility requested that they move one side to
> generator due to the weather and high-utilization, and the ATS failed.
>
> But we're up ...
>
>
> On 1/30/19 10:50 AM, Mel Beckman wrote:
> > Being a Minnesota native, I can tell you that while it is indeed cold,
> > this is nothing new i the Great White North :)  I am amaze a how
> > consistently the media overplays the severity of Midwest cold weather as
> > some kind of unique phenomenon. They amplify this by reporting the
> > wind-chill factor, which is the “what it feels like” equivalent in a
> > cold and windy environment. But equipment feels nothing, so windchill is
> > irrelevant.
> >
> > For example, Minneapolis is -20F, but the news media instead reports
> > “-60F wind chill”, which, while dramatic, is not meaningful for most
> > purposes. I grew up in Minnesota with -30F and lower quite common, and
> > we walked to school in those temperatures. You just have to dress well.
> > Minneapolis is paved with tunnels and heated skyways to eliminate most
> > outdoor walking downtown.
> >
> > As far as networks go, none of the ISPs I know of do anything different
> > than anywhere else in the country. Everyone has backup power. It’s
> > already common practice everywhere to exploit cooler winter ambient
> > temperatures to reduce HVAC requirements, so that’s not new either. But
> > it gets as hot in the Midwest in our summer as it is in SA for you now,
> > so everyone must still build out HVAC capacity to cover the hottest days.
> >
> >   -mel beckman
> >
> > On Jan 30, 2019, at 8:40 AM, Mark Tinka  > > wrote:
> >
> >> For anyone running IP networks in the Midwest, are you having to do
> >> anything special to keep your networks up?
> >>
> >> For the data centres, is this cold front a chance to reduce air
> >> conditioning costs, or is it actually straining the infrastructure?
> >>
> >> I'm curious, from a +27-degree C summer's day here in Johannesburg.
> >>
> >> Mark.
>

Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[Mehmet Akcin]

Pinged my contacts in each

On Wed, Jan 30, 2019 at 05:52 Jason Lixfeld  wrote:

> Hi,
>
> In late October 2018, DE-CIX announced that they would be renumbering
> their IPv4 address block in New York between 01-28-19 and 01-30-19.
>
> This was followed by numerous reminders in months, weeks and even days
> leading up to the renumbering activity.
>
> The renumbering activity has come and gone, but LinkedIn, Amazon and
> Akamai are still using the old IPs.
>
> If three months has gone by and the numerous reminders that have been sent
> have resulted in these organizations still living on the old IP space, it
> seems to me that there may be some sort of a disconnect between who
> receives the notifications from IXPs and how they are filtered upstream.
>
> I’m hopeful that the eyeballs who read this list are some of those folks
> who should have received the notifications from DE-CIX, or can at least
> filter the info back downstream to whoever can perform the renumbering
> activity.
>
> Thanks.
>
> --
Mehmet
+1-424-298-1903

Re: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

[Bryan Holloway]

Approximately 3 hrs ago we lost B-feed at Minneapolis Cologix.

Apparently the local utility requested that they move one side to 
generator due to the weather and high-utilization, and the ATS failed.


But we're up ...


On 1/30/19 10:50 AM, Mel Beckman wrote:
Being a Minnesota native, I can tell you that while it is indeed cold, 
this is nothing new i the Great White North :)  I am amaze a how 
consistently the media overplays the severity of Midwest cold weather as 
some kind of unique phenomenon. They amplify this by reporting the 
wind-chill factor, which is the “what it feels like” equivalent in a 
cold and windy environment. But equipment feels nothing, so windchill is 
irrelevant.


For example, Minneapolis is -20F, but the news media instead reports 
“-60F wind chill”, which, while dramatic, is not meaningful for most 
purposes. I grew up in Minnesota with -30F and lower quite common, and 
we walked to school in those temperatures. You just have to dress well. 
Minneapolis is paved with tunnels and heated skyways to eliminate most 
outdoor walking downtown.


As far as networks go, none of the ISPs I know of do anything different 
than anywhere else in the country. Everyone has backup power. It’s 
already common practice everywhere to exploit cooler winter ambient 
temperatures to reduce HVAC requirements, so that’s not new either. But 
it gets as hot in the Midwest in our summer as it is in SA for you now, 
so everyone must still build out HVAC capacity to cover the hottest days.


  -mel beckman

On Jan 30, 2019, at 8:40 AM, Mark Tinka > wrote:


For anyone running IP networks in the Midwest, are you having to do 
anything special to keep your networks up?


For the data centres, is this cold front a chance to reduce air 
conditioning costs, or is it actually straining the infrastructure?


I'm curious, from a +27-degree C summer's day here in Johannesburg.

Mark.

Re: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

[Mel Beckman]

Being a Minnesota native, I can tell you that while it is indeed cold, this is 
nothing new i the Great White North :)  I am amaze a how consistently the media 
overplays the severity of Midwest cold weather as some kind of unique 
phenomenon. They amplify this by reporting the wind-chill factor, which is the 
“what it feels like” equivalent in a cold and windy environment. But equipment 
feels nothing, so windchill is irrelevant.

For example, Minneapolis is -20F, but the news media instead reports “-60F wind 
chill”, which, while dramatic, is not meaningful for most purposes. I grew up 
in Minnesota with -30F and lower quite common, and we walked to school in those 
temperatures. You just have to dress well. Minneapolis is paved with tunnels 
and heated skyways to eliminate most outdoor walking downtown.

As far as networks go, none of the ISPs I know of do anything different than 
anywhere else in the country. Everyone has backup power. It’s already common 
practice everywhere to exploit cooler winter ambient temperatures to reduce 
HVAC requirements, so that’s not new either. But it gets as hot in the Midwest 
in our summer as it is in SA for you now, so everyone must still build out HVAC 
capacity to cover the hottest days.

 -mel beckman

On Jan 30, 2019, at 8:40 AM, Mark Tinka 
mailto:mark.ti...@seacom.mu>> wrote:

For anyone running IP networks in the Midwest, are you having to do anything 
special to keep your networks up?

For the data centres, is this cold front a chance to reduce air conditioning 
costs, or is it actually straining the infrastructure?

I'm curious, from a +27-degree C summer's day here in Johannesburg.

Mark.

Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[Mike Hammett]

A lot of huge companies apparently find it tough to find the $75k to hire one 
more peering person. Not all, though. For many, everything just runs like 
clockwork. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Jason Lixfeld"  
To: "North American Network Operators' Group"  
Sent: Wednesday, January 30, 2019 7:52:09 AM 
Subject: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY 

Hi, 

In late October 2018, DE-CIX announced that they would be renumbering their 
IPv4 address block in New York between 01-28-19 and 01-30-19. 

This was followed by numerous reminders in months, weeks and even days leading 
up to the renumbering activity. 

The renumbering activity has come and gone, but LinkedIn, Amazon and Akamai are 
still using the old IPs. 

If three months has gone by and the numerous reminders that have been sent have 
resulted in these organizations still living on the old IP space, it seems to 
me that there may be some sort of a disconnect between who receives the 
notifications from IXPs and how they are filtered upstream. 

I’m hopeful that the eyeballs who read this list are some of those folks who 
should have received the notifications from DE-CIX, or can at least filter the 
info back downstream to whoever can perform the renumbering activity. 

Thanks. 

Effects of Cold Front on Internet Infrastructure - U.S. Midwest

[Mark Tinka]

For anyone running IP networks in the Midwest, are you having to do
anything special to keep your networks up?

For the data centres, is this cold front a chance to reduce air
conditioning costs, or is it actually straining the infrastructure?

I'm curious, from a +27-degree C summer's day here in Johannesburg.

Mark.

Re: Amazon Peering

[Mike Hammett]

Oh, you ordered cross connects for a PNI and they stopped responding 
mid-project? Isn't that nice! 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Luca Salvatore via NANOG"  
To: "North American Network Operators' Group"  
Sent: Wednesday, January 30, 2019 9:45:29 AM 
Subject: Re: Amazon Peering 


Similar experiences here with Amazon. Initially had semi-regular responses from 
their peering team, they issued LOAs, I ordered the x-connects and then radio 
silence for months. 
At the point now where I'm disconnecting x-connects since it's a waste of 
money. 


On Tue, Jan 29, 2019 at 10:49 AM Brooks Swinnerton < bswinner...@gmail.com > 
wrote: 



I also saw sessions come up this weekend, no routes yet though. 


On Mon, Jan 28, 2019 at 4:56 PM Tom Beecher  wrote: 



Mike- 


Definitely moving forward now. Someone from Amazon was working with my peering 
group and things started coming up this weekend, so it seems like they're 
catching up pretty good now. 


On Thu, Jan 24, 2019 at 2:45 PM Mike Hammett < na...@ics-il.net > wrote: 




Let us know your success as well. I'll hold off following up on my requests 
until I see that other people are successful. I don't want to contribute to 
flooding them with requests. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 



From: "Tom Beecher"  
To: "Jason Lixfeld" < jason+na...@lixfeld.ca > 
Cc: "North American Network Operators' Group" < nanog@nanog.org > 
Sent: Thursday, January 24, 2019 1:38:51 PM 
Subject: Re: Amazon Peering 


Thanks Jason. I'll have my peering team take another crack at reaching out and 
see what happens. Appreciate it! 


On Thu, Jan 24, 2019 at 2:21 PM Jason Lixfeld < jason+na...@lixfeld.ca > wrote: 



We circled back with them yesterday on a request we made in late November where 
at the time they said they wouldn’t be turned up until 2019 due to holiday 
network change freeze. 


They responded within about 4 hours, thanked us for our patience and 
understanding and said we should expect them to be turned up in about 6 weeks, 
which is apparently their typical timing. 





On Jan 24, 2019, at 2:13 PM, Tom Beecher < beec...@beecher.cc > wrote: 


I hate to necro-thread , but has anyone seen any movement from Amazon on this? 
I just got a Strongly Worded Message about it, and according to my peering team 
, it's been radio silence for months. 




On Sat, Nov 24, 2018 at 12:32 PM JASON BOTHE via NANOG < nanog@nanog.org > 
wrote: 





This is a note I received on Oct18 when checking on a peering request submitted 
on Aug7.. 


“Apologies for the delays here. We have temporarily frozen IX peering as we 
revise some of our automation processes. I’m hopeful this will be unblocked by 
early November. Thank you for your continued patience.” 

On Nov 24, 2018, at 10:59, Darin Steffl < darin.ste...@mnwifi.com > wrote: 





It seems wasteful for Amazon to connect to an IX but then ignore peering 
requests for a year. 


They have 40G of connectivity but are unresponsive. I'll try emailing all the 
other contacts listed in peeringdb. 


Thanks 


On Sat, Nov 24, 2018, 10:38 AM Mike Hammett < na...@ics-il.net wrote: 




I've e-mailed my contacts there a couple times on people's behalf. No response 
yet. 

It seems like a lot of organizations need 1 more person in their peering 
departments. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 



From: "Darin Steffl" < darin.ste...@mnwifi.com > 
To: "North American Network Operators' Group" < nanog@nanog.org > 
Sent: Friday, November 23, 2018 10:21:51 PM 
Subject: Amazon Peering 


Hey all, 


Does anyone have a direct contact to get a peering session established with 
Amazon at an IX? I sent a peering request Dec 2017 and two more times this Sept 
and Nov with no response. 


I sent to peer...@amazon.com and received one automated response back so I know 
they received my email but nothing since. 





-- 


Darin Steffl 
Minnesota WiFi 
www.mnwifi.com 
507-634-WiFi 
Like us on Facebook 

Re: Amazon Peering

[Luca Salvatore via NANOG]

Similar experiences here with Amazon.  Initially had semi-regular responses
from their peering team, they issued LOAs, I ordered the x-connects and
then radio silence for months.
At the point now where I'm disconnecting x-connects since it's a waste of
money.

On Tue, Jan 29, 2019 at 10:49 AM Brooks Swinnerton 
wrote:

> I also saw sessions come up this weekend, no routes yet though.
>
> On Mon, Jan 28, 2019 at 4:56 PM Tom Beecher  wrote:
>
>> Mike-
>>
>> Definitely moving forward now. Someone from Amazon was working with my
>> peering group and things started coming up this weekend, so it seems like
>> they're catching up pretty good now.
>>
>> On Thu, Jan 24, 2019 at 2:45 PM Mike Hammett  wrote:
>>
>>> Let us know your success as well. I'll hold off following up on my
>>> requests until I see that other people are successful. I don't want to
>>> contribute to flooding them with requests.
>>>
>>>
>>>
>>> -
>>> Mike Hammett
>>> Intelligent Computing Solutions
>>> http://www.ics-il.com
>>>
>>> Midwest-IX
>>> http://www.midwest-ix.com
>>>
>>> --
>>> *From: *"Tom Beecher" 
>>> *To: *"Jason Lixfeld" 
>>> *Cc: *"North American Network Operators' Group" 
>>> *Sent: *Thursday, January 24, 2019 1:38:51 PM
>>> *Subject: *Re: Amazon Peering
>>>
>>> Thanks Jason. I'll have my peering team take another crack at reaching
>>> out and see what happens. Appreciate it!
>>>
>>> On Thu, Jan 24, 2019 at 2:21 PM Jason Lixfeld 
>>> wrote:
>>>
 We circled back with them yesterday on a request we made in late
 November where at the time they said they wouldn’t be turned up until 2019
 due to holiday network change freeze.

 They responded within about 4 hours, thanked us for our patience and
 understanding and said we should expect them to be turned up in about 6
 weeks, which is apparently their typical timing.

 On Jan 24, 2019, at 2:13 PM, Tom Beecher  wrote:

 I hate to necro-thread , but has anyone seen any movement from Amazon
 on this? I just got a Strongly Worded Message about it, and according to my
 peering team , it's been radio silence for months.


 On Sat, Nov 24, 2018 at 12:32 PM JASON BOTHE via NANOG 
 wrote:

> This is a note I received on Oct18 when checking on a peering request
> submitted on Aug7..
>
> “Apologies for the delays here. We have temporarily frozen IX peering
> as we revise some of our automation processes. I’m hopeful this will be
> unblocked by early November. Thank you for your continued patience.”
>
> On Nov 24, 2018, at 10:59, Darin Steffl 
> wrote:
>
> It seems wasteful for Amazon to connect to an IX but then ignore
> peering requests for a year.
>
> They have 40G of connectivity but are unresponsive. I'll try emailing
> all the other contacts listed in peeringdb.
>
> Thanks
>
> On Sat, Nov 24, 2018, 10:38 AM Mike Hammett 
>> I've e-mailed my contacts there a couple times on people's behalf. No
>> response yet.
>>
>> It seems like a lot of organizations need 1 more person in their
>> peering departments.
>>
>>
>>
>> -
>> Mike Hammett
>> Intelligent Computing Solutions
>> http://www.ics-il.com
>>
>> Midwest-IX
>> http://www.midwest-ix.com
>>
>> --
>> *From: *"Darin Steffl" 
>> *To: *"North American Network Operators' Group" 
>> *Sent: *Friday, November 23, 2018 10:21:51 PM
>> *Subject: *Amazon Peering
>>
>> Hey all,
>>
>> Does anyone have a direct contact to get a peering session
>> established with Amazon at an IX? I sent a peering request Dec 2017 and 
>> two
>> more times this Sept and Nov with no response.
>>
>> I sent to peer...@amazon.com and received one automated response
>> back so I know they received my email but nothing since.
>>
>>
>>
>> --
>> Darin Steffl
>> Minnesota WiFi
>> www.mnwifi.com
>> 507-634-WiFi
>>  Like us on Facebook
>> 
>>
>>

>>>

Re: AT&T Fiber Outage in Holmdel, NJ

[Amauris Rodriguez Martinez]

Hi Mehmet,

Commercial; forgot to mention fiber issues related to AVPN service, 
specifically peering to ASN 13979.

Best Regards,
Amauris

Get Outlook for iOS

From: Mehmet Akcin 
Sent: Tuesday, January 29, 2019 12:43
To: Amauris Rodriguez Martinez
Cc: NANOG
Subject: Re: AT&T Fiber Outage in Holmdel, NJ

commercial or residential?

I have not seen or heard anything about this but paged our AT&T contact asking 
if anything going on..

On Tue, Jan 29, 2019 at 9:11 AM Amauris Rodriguez Martinez 
mailto:amaurisr...@hotmail.com>> wrote:
Did anybody here have fiber issues yesterday night about 9:30 pm?

Best Regards,
Amauris

Get Outlook for iOS

Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[Bryan Holloway]

On 1/30/19 7:52 AM, Jason Lixfeld wrote:

Hi,

In late October 2018, DE-CIX announced that they would be renumbering their 
IPv4 address block in New York between 01-28-19 and 01-30-19.

This was followed by numerous reminders in months, weeks and even days leading 
up to the renumbering activity.

The renumbering activity has come and gone, but LinkedIn, Amazon and Akamai are 
still using the old IPs.

If three months has gone by and the numerous reminders that have been sent have 
resulted in these organizations still living on the old IP space, it seems to 
me that there may be some sort of a disconnect between who receives the 
notifications from IXPs and how they are filtered upstream.

I’m hopeful that the eyeballs who read this list are some of those folks who 
should have received the notifications from DE-CIX, or can at least filter the 
info back downstream to whoever can perform the renumbering activity.

Thanks.



They're not the only ones ... out of all of our peers on that exchange, 
~30% haven't updated as of this writing.


I'm a little reluctant to pull our old address until penetration is a 
little higher.

Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[Jason Lixfeld]

Hi,

In late October 2018, DE-CIX announced that they would be renumbering their 
IPv4 address block in New York between 01-28-19 and 01-30-19.

This was followed by numerous reminders in months, weeks and even days leading 
up to the renumbering activity.

The renumbering activity has come and gone, but LinkedIn, Amazon and Akamai are 
still using the old IPs.

If three months has gone by and the numerous reminders that have been sent have 
resulted in these organizations still living on the old IP space, it seems to 
me that there may be some sort of a disconnect between who receives the 
notifications from IXPs and how they are filtered upstream.

I’m hopeful that the eyeballs who read this list are some of those folks who 
should have received the notifications from DE-CIX, or can at least filter the 
info back downstream to whoever can perform the renumbering activity.

Thanks.