RE: CenturyLink/Level3 feedback

[Jeffrey Hathaway via NANOG]

Greetings,

My personal experience, take it for what it is worth.

Level 3 was extremely slow to turn up a circuit well before the merger and 
seems slower post the merger. I have just never seen level 3 turn up anything 
fast, even in a data center they already were serving other customers in.


Sincerely,
Jeffrey Hathaway
Information Technology • Howard Center Inc.



-Original Message-
From: NANOG  On Behalf Of Jared Mauch
Sent: Friday, July 5, 2019 3:38 PM
To: Stephen Frost 
Cc: nanog 
Subject: Re: CenturyLink/Level3 feedback


CAUTION: This email originated from outside Howard Center. Please use caution 
opening attachments or clicking on web links with this email.




> On Jul 5, 2019, at 3:10 PM, Stephen Frost  wrote:
>
> Greetings,
>
> I have to admit that I was hoping to be able to report to this list
> that CL was able to spin up a new 1G in fairly short order (after all,
> this is what they assured me of when discussing it with them...) but
> it's now been over a month, with them telling me it'll be another
> couple weeks because they need to send a tech out (the wiring and all
> of the equipment has been ready to go, though that also took longer
> than it should have imv...).
>
> And this in an already lit building in northern Virginia, not some
> back of the woods location, small town, or something going across an ocean.

Sometimes you’d be surprised, it may not be straightforward on their end.

Remember, most people here are likely experts at some part or many parts, what 
we do is likely wizardry to others.

I have a saying you’re welcome to steal if you don’t steal it too much:

“We are moving at the speed the organization is capable”.  I suspect that’s the 
case for them in a post-acquisition world trying to sort through all the 
integration work.

- Jared



HowardCenter.org 
[http://howardcenter.org/assets/design/Facebook.jpg] 
  
[http://howardcenter.org/assets/design/Twitter.jpg] 
  
[http://howardcenter.org/assets/design/LinkedIn.jpg] 


CONFIDENTIALITY NOTICE: This e-mail is intended only for the use of the 
individual or entity to which it is addressed and may contain information that 
is patient protected health information, privileged, confidential and exempt 
from disclosure under applicable law. If you have received this communication 
in error, you are hereby notified that any dissemination, distribution or 
copying of this communication is strictly prohibited. Please notify the sender 
by reply e-mail and delete the original message immediately, or notify Howard 
Center, Inc. immediately by forwarded e-mail to our Privacy Officer, 
da...@howardcenter.org. Thank you.

Re: Puerto Rico Internet Exchange

[Bill Woodcock]

.Org, .pr, and a couple of root letters should be on our Puerto Rico node 
already, along with several hundred other TLDs. 

-Bill


> On Jul 6, 2019, at 17:00, Rubens Kuhl  wrote:
> 
> 
> It would be interesting if ICANN, Verisign and Afilias were able to join the 
> IX as well making the root and .com/.net/.org/.pr zones available even if the 
> island is cut off from the globe. There is so much fixation in bits per 
> second while IX'es are resiliency tools, more than bandwidth saving tools. 
> 
> 
> Rubens
> 
> 
>> On Sat, Jul 6, 2019 at 6:19 PM Mehmet Akcin  wrote:
>> Hey there, just a very brief update
>> 
>> We are in the process of RE-launching Internet Exchange in San Juan, Puerto 
>> Rico in a few weeks. We've got multiple networks in San Juan agreed to join 
>> the IX in a common neutral point.  If you are able to help with the project 
>> or interested in learning more about it, please contact me offlist. 
>> (especially if you are in Puerto rico)
>> 
>> Once everything is operational and the website is set up, I hope to contact 
>> back and update once we've got mrtg, etc is operational.
>> 
>> thank you

Re: Puerto Rico Internet Exchange

[cyrus ramirez via NANOG]

I'm interested as well.
Cyrus Ramirez

Sent from Yahoo Mail on Android 
 
  On Sat, Jul 6, 2019 at 6:01 PM, Rubens Kuhl wrote:   
It would be interesting if ICANN, Verisign and Afilias were able to join the IX 
as well making the root and .com/.net/.org/.pr zones available even if the 
island is cut off from the globe. There is so much fixation in bits per second 
while IX'es are resiliency tools, more than bandwidth saving tools. 

Rubens

On Sat, Jul 6, 2019 at 6:19 PM Mehmet Akcin  wrote:

Hey there, just a very brief update
We are in the process of RE-launching Internet Exchange in San Juan, Puerto 
Rico in a few weeks. We've got multiple networks in San Juan agreed to join the 
IX in a common neutral point.  If you are able to help with the project or 
interested in learning more about it, please contact me offlist. (especially if 
you are in Puerto rico)
Once everything is operational and the website is set up, I hope to contact 
back and update once we've got mrtg, etc is operational.
thank you
  

Re: Puerto Rico Internet Exchange

[Rubens Kuhl]

It would be interesting if ICANN, Verisign and Afilias were able to join
the IX as well making the root and .com/.net/.org/.pr zones available even
if the island is cut off from the globe. There is so much fixation in bits
per second while IX'es are resiliency tools, more than bandwidth saving
tools.


Rubens


On Sat, Jul 6, 2019 at 6:19 PM Mehmet Akcin  wrote:

> Hey there, just a very brief update
>
> We are in the process of RE-launching Internet Exchange in San Juan,
> Puerto Rico in a few weeks. We've got multiple networks in San Juan agreed
> to join the IX in a common neutral point.  If you are able to help with the
> project or interested in learning more about it, please contact me offlist.
> (especially if you are in Puerto rico)
>
> Once everything is operational and the website is set up, I hope to
> contact back and update once we've got mrtg, etc is operational.
>
> thank you
>

Re: CloudFlare issues?

[Matt Corallo]

Oops, I mean with a script which removes such routes if there is an
encompassing route which a different upstream takes, as obviously the
more-specific would otherwise still win.

Matt

On 7/6/19 5:44 PM, Matt Corallo wrote:
> On my test net I take ROA_INVALIDs and convert them to unreachables with
> a low preference (ie so that any upstreams taking only the shorter path
> will be selected, but so that such packets will never be routed).
> 
> Obviously this isn't a well-supported operation, but I'm curious what
> people think of such an approach? If you really want to treat
> ROA_INVALID as "this is probably a hijack", you don't really want to be
> sending the hijacker traffic.
> 
> Of course if upstreams are rejecting ROA_INVALID you can still have the
> same problem one network away, but its an interesting result for
> testing, especially since it rejects a bunch of crap in China where CT
> has reassigned prefixes with covering ROAs to customers who re-announce
> on their own ASN (which appears to be common).
> 
> Matt
> 
> On 7/6/19 4:05 PM, Brett Frankenberger wrote:
>> On Thu, Jul 04, 2019 at 11:46:05AM +0200, Mark Tinka wrote:
>>> I finally thought about this after I got off my beer high :-).
>>>
>>> Some of our customers complained about losing access to Cloudflare's
>>> resources during the Verizon debacle. Since we are doing ROV and
>>> dropping Invalids, this should not have happened, given most of
>>> Cloudflare's IPv4 and IPv6 routes are ROA'd.
>>
>> These were more-specifics, though.  So if you drop all the
>> more-specifics as failing ROV, then you end up following the valid
>> shorter prefix to the destination.  Quite possibly that points at the
>> upstream which sent you the more-specific which you rejected, at which
>> point your packets end up same going to the same place they would have
>> gone if you had accepted the invalid more-specific.
>>
>> Two potential issues here:  First, if you don't have an upstream who
>> is also rejecting the invalid routes, then anywhere you send the
>> packets, they're going to follow the more-specific.  Second, even if
>> you do have an upstream that is rejecting the invalid routes, ROV won't
>> cause you to prefer the less-specific from an upstream that is
>> rejecting the invalid routes over a less-specific from an upstream that
>> is accepting the invalid routes.
>>
>> For example:
>>if upstream A sends you:
>>   10.0.0.0./16 valid
>>and upstream B sends you
>>   10.0.0.0/16 valid
>>   10.0.0.0/17 invalid
>>   10.0.128.0/17 invalid
>> you want send to send the packet to A.  But ROV won't cause that, and if
>> upstream B is selected by your BGP decision criteria (path length,
>> etc.), you're packets will ultimately follow the more-specific.
>>
>> (Of course, the problem is can occur more than one network away.  Even
>> if you do send to upstream A, there's no guarantee that A's
>> less-specifics aren't pointed at another network that does have the
>> more-specifics.  But at least you give them a fighting chance by
>> sending them to A.)
>>
>>  -- Brett
>>

Re: CloudFlare issues?

[Matt Corallo]

On my test net I take ROA_INVALIDs and convert them to unreachables with
a low preference (ie so that any upstreams taking only the shorter path
will be selected, but so that such packets will never be routed).

Obviously this isn't a well-supported operation, but I'm curious what
people think of such an approach? If you really want to treat
ROA_INVALID as "this is probably a hijack", you don't really want to be
sending the hijacker traffic.

Of course if upstreams are rejecting ROA_INVALID you can still have the
same problem one network away, but its an interesting result for
testing, especially since it rejects a bunch of crap in China where CT
has reassigned prefixes with covering ROAs to customers who re-announce
on their own ASN (which appears to be common).

Matt

On 7/6/19 4:05 PM, Brett Frankenberger wrote:
> On Thu, Jul 04, 2019 at 11:46:05AM +0200, Mark Tinka wrote:
>> I finally thought about this after I got off my beer high :-).
>>
>> Some of our customers complained about losing access to Cloudflare's
>> resources during the Verizon debacle. Since we are doing ROV and
>> dropping Invalids, this should not have happened, given most of
>> Cloudflare's IPv4 and IPv6 routes are ROA'd.
> 
> These were more-specifics, though.  So if you drop all the
> more-specifics as failing ROV, then you end up following the valid
> shorter prefix to the destination.  Quite possibly that points at the
> upstream which sent you the more-specific which you rejected, at which
> point your packets end up same going to the same place they would have
> gone if you had accepted the invalid more-specific.
> 
> Two potential issues here:  First, if you don't have an upstream who
> is also rejecting the invalid routes, then anywhere you send the
> packets, they're going to follow the more-specific.  Second, even if
> you do have an upstream that is rejecting the invalid routes, ROV won't
> cause you to prefer the less-specific from an upstream that is
> rejecting the invalid routes over a less-specific from an upstream that
> is accepting the invalid routes.
> 
> For example:
>if upstream A sends you:
>   10.0.0.0./16 valid
>and upstream B sends you
>   10.0.0.0/16 valid
>   10.0.0.0/17 invalid
>   10.0.128.0/17 invalid
> you want send to send the packet to A.  But ROV won't cause that, and if
> upstream B is selected by your BGP decision criteria (path length,
> etc.), you're packets will ultimately follow the more-specific.
> 
> (Of course, the problem is can occur more than one network away.  Even
> if you do send to upstream A, there's no guarantee that A's
> less-specifics aren't pointed at another network that does have the
> more-specifics.  But at least you give them a fighting chance by
> sending them to A.)
> 
>  -- Brett
> 

Puerto Rico Internet Exchange

[Mehmet Akcin]

Hey there, just a very brief update

We are in the process of RE-launching Internet Exchange in San Juan, Puerto
Rico in a few weeks. We've got multiple networks in San Juan agreed to join
the IX in a common neutral point.  If you are able to help with the project
or interested in learning more about it, please contact me offlist.
(especially if you are in Puerto rico)

Once everything is operational and the website is set up, I hope to contact
back and update once we've got mrtg, etc is operational.

thank you

Re: CloudFlare issues?

[Brett Frankenberger]

On Thu, Jul 04, 2019 at 11:46:05AM +0200, Mark Tinka wrote:
> I finally thought about this after I got off my beer high :-).
> 
> Some of our customers complained about losing access to Cloudflare's
> resources during the Verizon debacle. Since we are doing ROV and
> dropping Invalids, this should not have happened, given most of
> Cloudflare's IPv4 and IPv6 routes are ROA'd.

These were more-specifics, though.  So if you drop all the
more-specifics as failing ROV, then you end up following the valid
shorter prefix to the destination.  Quite possibly that points at the
upstream which sent you the more-specific which you rejected, at which
point your packets end up same going to the same place they would have
gone if you had accepted the invalid more-specific.

Two potential issues here:  First, if you don't have an upstream who
is also rejecting the invalid routes, then anywhere you send the
packets, they're going to follow the more-specific.  Second, even if
you do have an upstream that is rejecting the invalid routes, ROV won't
cause you to prefer the less-specific from an upstream that is
rejecting the invalid routes over a less-specific from an upstream that
is accepting the invalid routes.

For example:
   if upstream A sends you:
  10.0.0.0./16 valid
   and upstream B sends you
  10.0.0.0/16 valid
  10.0.0.0/17 invalid
  10.0.128.0/17 invalid
you want send to send the packet to A.  But ROV won't cause that, and if
upstream B is selected by your BGP decision criteria (path length,
etc.), you're packets will ultimately follow the more-specific.

(Of course, the problem is can occur more than one network away.  Even
if you do send to upstream A, there's no guarantee that A's
less-specifics aren't pointed at another network that does have the
more-specifics.  But at least you give them a fighting chance by
sending them to A.)

 -- Brett