Re: FCC workshop: Security vulnerabilities within our communications networks

[Christopher Morrow]

looks like our best and brightest have the problem resolved, phew!
we're all safe now.

On Tue, Jun 25, 2019 at 5:26 PM Sean Donelan  wrote:
>
> On Fri, 21 Jun 2019, Sean Donelan wrote:
> > Federal Communications Commissioner Geoffrey Starks is holding a workshop
> > next week, June 27, 2019, to hear from interested parties on how to address
> > the national security threats posed by insecure equipment within our
> > communications networks.
>
> The agenda for the FCC network security workshop has been published.
>
> https://www.fcc.gov/document/commissioner-starks-releases-agenda-network-security-workshop
>

Re: Public Subnet re-assignments

[Mel Beckman]

Michel is right. This is a common configuration error: failing to have the mask 
agree on all interfaces. This is indeed what you would see.

 -mel

On Jun 25, 2019, at 4:07 PM, Michel Py 
mailto:michel...@tsisemi.com>> wrote:

>  Scott wrote :
> No nothing like that. I'm just removing the .0/30 and 4/30 subnets and adding 
> .0/29.
> To  your previous question, yes .0 and .3 are unused. Once I change the 
> subnet .3
> becomes a usable IP and it's getting hammered with traffic, causing packet 
> loss.

You change the subnet mask on both sides, right ?

Looks to me like expected behavior. On the sending router, with a /30 mask the 
.3 address is not usable, so the sending router does not send traffic.
When you change to the /29 mask, .3 becomes usable, the sending router ARPs it, 
and starts sending traffic.

In a way, that is possibly good news, as it allows you do find out that you may 
have a DOS or a DDOS attack going on your .3 address.

Michel.



On 6/25/19 3:30 PM, Mel Beckman wrote:
> Also, what do you mean by “join to /30 public subnets to a /29”? You can’t 
> overlap subnets, if that’s what you’re thinking.
>
>  -mel
>
>> On Jun 25, 2019, at 3:27 PM, Mel Beckman 
>> mailto:m...@beckman.org>> wrote:
>>
>> You’re using just the two middle IPs in the four that make up the /30 set, 
>> right? IOW, the subnet x.x.x.0/30 should have .0 and .3 unused (they’re 
>> broadcast), and you use .1 and .2.
>>
>> -mel
>>
>>> On Jun 25, 2019, at 9:41 AM, Scott 
>>> mailto:sc...@viviotech.net>> wrote:
>>>
>>> First, sorry if this is a bit of a noob question.
>>>
>>> I'm trying to find a way of preventing a slew of traffic to an IP, or
>>> IP's, when I join two /30 public subnets to a /29. It appears that while
>>> the ranges are /30 someone is trying to brute-force the network and/or
>>> broadcast addresses for the ranges. When I change them to be a /29, now
>>> the router sees the traffic and starts dropping packets. Are there any
>>> suggestions for mitigating this behavior or is it just the nature of the
>>> beast?
>>>
>>> --
>>> 101010
>>>
>>>
--
101010

TSI Disclaimer:  This message and any files or text attached to it are intended 
only for the recipients named above and contain information that may be 
confidential or privileged. If you are not the intended recipient, you must not 
forward, copy, use or otherwise disclose this communication or the information 
contained herein. In the event you have received this message in error, please 
notify the sender immediately by replying to this message, and then delete all 
copies of it from your system. Thank you!...

Re: 80.67.75.0/24 (Akamai) announced by Kazakhtelecom

[Jared Mauch]

Yes. I saw this earlier today. It’s complex to unwind so many years of this 
config :-)

Sent from my iCar

> On Jun 25, 2019, at 5:42 PM, Eric Dugas  wrote:
> 
> Got alerts for 80.67.75.0/24 (Akamai) normally announced by Tier1 providers 
> routed by a long AS path from our of our peers:
> 
> 80.67.75.0/24 AS path: 9002 9198 43727 6762 2914 23454 23454 I, 
> validation-state: unknown
> 80.67.64.0/19 AS path: 1299 3257 34164
> 
> I just got home and it seems Akamai already reacted:
> 
> 80.67.75.0/24 AS path: 174 2914 23454 23454
>  AS path: 1299 2914 23454 23454
> 
> Less damage (as far as I can see) but what a bad week for BGP so far...

RE: Public Subnet re-assignments

[Michel Py]

>  Scott wrote :
> No nothing like that. I'm just removing the .0/30 and 4/30 subnets and adding 
> .0/29.
> To  your previous question, yes .0 and .3 are unused. Once I change the 
> subnet .3
> becomes a usable IP and it's getting hammered with traffic, causing packet 
> loss.

You change the subnet mask on both sides, right ?

Looks to me like expected behavior. On the sending router, with a /30 mask the 
.3 address is not usable, so the sending router does not send traffic.
When you change to the /29 mask, .3 becomes usable, the sending router ARPs it, 
and starts sending traffic.

In a way, that is possibly good news, as it allows you do find out that you may 
have a DOS or a DDOS attack going on your .3 address.

Michel.



On 6/25/19 3:30 PM, Mel Beckman wrote:
> Also, what do you mean by “join to /30 public subnets to a /29”? You can’t 
> overlap subnets, if that’s what you’re thinking.
>
>  -mel
>
>> On Jun 25, 2019, at 3:27 PM, Mel Beckman  wrote:
>>
>> You’re using just the two middle IPs in the four that make up the /30 set, 
>> right? IOW, the subnet x.x.x.0/30 should have .0 and .3 unused (they’re 
>> broadcast), and you use .1 and .2.
>>
>> -mel
>>
>>> On Jun 25, 2019, at 9:41 AM, Scott  wrote:
>>>
>>> First, sorry if this is a bit of a noob question.
>>>
>>> I'm trying to find a way of preventing a slew of traffic to an IP, or
>>> IP's, when I join two /30 public subnets to a /29. It appears that while
>>> the ranges are /30 someone is trying to brute-force the network and/or
>>> broadcast addresses for the ranges. When I change them to be a /29, now
>>> the router sees the traffic and starts dropping packets. Are there any
>>> suggestions for mitigating this behavior or is it just the nature of the
>>> beast?
>>>
>>> --
>>> 101010
>>>
>>>
--
101010

TSI Disclaimer:  This message and any files or text attached to it are intended 
only for the recipients named above and contain information that may be 
confidential or privileged. If you are not the intended recipient, you must not 
forward, copy, use or otherwise disclose this communication or the information 
contained herein. In the event you have received this message in error, please 
notify the sender immediately by replying to this message, and then delete all 
copies of it from your system. Thank you!...

Re: Public Subnet re-assignments

[Mel Beckman]

If the sources are from many different IPs, it could be a DDoS attack that you 
simply didn’t notice before. You can black-hole individual IPs using a /32 
null0 route. That will at least stop your border router from trying to ARP the 
destination, reducing broadcast traffic on the subnet. In fact, it’s a good 
idea to configure /32 null0 routes for IPs you don’t use. Those IPs can’t then 
be scanned. 

 -mel

> On Jun 25, 2019, at 3:50 PM, Scott  wrote:
> 
> No nothing like that. I'm just removing the .0/30 and 4/30 subnets and
> adding .0/29.
> 
> To  your previous question, yes .0 and .3 are unused. Once I change the
> subnet .3 becomes a usable IP and it's getting hammered with traffic,
> causing packet loss.
> 
> On 6/25/19 3:30 PM, Mel Beckman wrote:
>> Also, what do you mean by “join to /30 public subnets to a /29”? You can’t 
>> overlap subnets, if that’s what you’re thinking.
>> 
>> -mel
>> 
>>> On Jun 25, 2019, at 3:27 PM, Mel Beckman  wrote:
>>> 
>>> You’re using just the two middle IPs in the four that make up the /30 set, 
>>> right? IOW, the subnet x.x.x.0/30 should have .0 and .3 unused (they’re 
>>> broadcast), and you use .1 and .2.
>>> 
>>> -mel
>>> 
 On Jun 25, 2019, at 9:41 AM, Scott  wrote:
 
 First, sorry if this is a bit of a noob question.
 
 I'm trying to find a way of preventing a slew of traffic to an IP, or
 IP's, when I join two /30 public subnets to a /29. It appears that while
 the ranges are /30 someone is trying to brute-force the network and/or
 broadcast addresses for the ranges. When I change them to be a /29, now
 the router sees the traffic and starts dropping packets. Are there any
 suggestions for mitigating this behavior or is it just the nature of the
 beast?
 
 -- 
 101010
 
 
> -- 
> 101010
> 

Re: Public Subnet re-assignments

[Scott Weeks]

--- sc...@viviotech.net wrote:
From: Scott 

To  your previous question, yes .0 and .3 are 
unused. Once I change the subnet .3 becomes a 
usable IP and it's getting hammered with 
traffic, causing packet loss.
--


Is it legitimate traffic or DDoS stuff?

scott

Re: Public Subnet re-assignments

[Scott]

No nothing like that. I'm just removing the .0/30 and 4/30 subnets and
adding .0/29.

To  your previous question, yes .0 and .3 are unused. Once I change the
subnet .3 becomes a usable IP and it's getting hammered with traffic,
causing packet loss.

On 6/25/19 3:30 PM, Mel Beckman wrote:
> Also, what do you mean by “join to /30 public subnets to a /29”? You can’t 
> overlap subnets, if that’s what you’re thinking.
>
>  -mel
>
>> On Jun 25, 2019, at 3:27 PM, Mel Beckman  wrote:
>>
>> You’re using just the two middle IPs in the four that make up the /30 set, 
>> right? IOW, the subnet x.x.x.0/30 should have .0 and .3 unused (they’re 
>> broadcast), and you use .1 and .2.
>>
>> -mel
>>
>>> On Jun 25, 2019, at 9:41 AM, Scott  wrote:
>>>
>>> First, sorry if this is a bit of a noob question.
>>>
>>> I'm trying to find a way of preventing a slew of traffic to an IP, or
>>> IP's, when I join two /30 public subnets to a /29. It appears that while
>>> the ranges are /30 someone is trying to brute-force the network and/or
>>> broadcast addresses for the ranges. When I change them to be a /29, now
>>> the router sees the traffic and starts dropping packets. Are there any
>>> suggestions for mitigating this behavior or is it just the nature of the
>>> beast?
>>>
>>> -- 
>>> 101010
>>>
>>>
-- 
101010

Re: Public Subnet re-assignments

[Mel Beckman]

Also, what do you mean by “join to /30 public subnets to a /29”? You can’t 
overlap subnets, if that’s what you’re thinking.

 -mel

> On Jun 25, 2019, at 3:27 PM, Mel Beckman  wrote:
> 
> You’re using just the two middle IPs in the four that make up the /30 set, 
> right? IOW, the subnet x.x.x.0/30 should have .0 and .3 unused (they’re 
> broadcast), and you use .1 and .2.
> 
> -mel
> 
>> On Jun 25, 2019, at 9:41 AM, Scott  wrote:
>> 
>> First, sorry if this is a bit of a noob question.
>> 
>> I'm trying to find a way of preventing a slew of traffic to an IP, or
>> IP's, when I join two /30 public subnets to a /29. It appears that while
>> the ranges are /30 someone is trying to brute-force the network and/or
>> broadcast addresses for the ranges. When I change them to be a /29, now
>> the router sees the traffic and starts dropping packets. Are there any
>> suggestions for mitigating this behavior or is it just the nature of the
>> beast?
>> 
>> -- 
>> 101010
>> 
>> 
> 

Re: Public Subnet re-assignments

[Mel Beckman]

You’re using just the two middle IPs in the four that make up the /30 set, 
right? IOW, the subnet x.x.x.0/30 should have .0 and .3 unused (they’re 
broadcast), and you use .1 and .2.

 -mel

> On Jun 25, 2019, at 9:41 AM, Scott  wrote:
> 
> First, sorry if this is a bit of a noob question.
> 
> I'm trying to find a way of preventing a slew of traffic to an IP, or
> IP's, when I join two /30 public subnets to a /29. It appears that while
> the ranges are /30 someone is trying to brute-force the network and/or
> broadcast addresses for the ranges. When I change them to be a /29, now
> the router sees the traffic and starts dropping packets. Are there any
> suggestions for mitigating this behavior or is it just the nature of the
> beast?
> 
> -- 
> 101010
> 
> 

80.67.75.0/24 (Akamai) announced by Kazakhtelecom

[Eric Dugas]

Got alerts for 80.67.75.0/24 (Akamai) normally announced by Tier1 providers 
routed by a long AS path from our of our peers:

80.67.75.0/24 AS path: 9002 9198 43727 6762 2914 23454 23454 I, 
validation-state: unknown
80.67.64.0/19 AS path: 1299 3257 34164

I just got home and it seems Akamai already reacted:
80.67.75.0/24 AS path: 174 2914 23454 23454
AS path: 1299 2914 23454 23454

Less damage (as far as I can see) but what a bad week for BGP so far...

Re: Are network operators morons? [was: CloudFlare issues?]

[Randy Bush]

>> perhaps the good side of this saga is that it may be an inflection
>> point
> I doubt it.
> The greyer my hair gets, the crankier I get.

i suspect i am a bit ahead of you there

but i used to think that the public would never become aware of privacy
issues.  snowen bumped that ball and tim cook spiked it.  and it is
getting more and more air time.

randy

Re: FCC workshop: Security vulnerabilities within our communications networks

[Sean Donelan]

On Fri, 21 Jun 2019, Sean Donelan wrote:
Federal Communications Commissioner Geoffrey Starks is holding a workshop 
next week, June 27, 2019, to hear from interested parties on how to address 
the national security threats posed by insecure equipment within our 
communications networks.


The agenda for the FCC network security workshop has been published.

https://www.fcc.gov/document/commissioner-starks-releases-agenda-network-security-workshop

Re: Are network operators morons? [was: CloudFlare issues?]

[Sean Donelan]

On Tue, 25 Jun 2019, Randy Bush wrote:

perhaps the good side of this saga is that it may be an inflection point


I doubt it.

The greyer my hair gets, the crankier I get.

Re: Are network operators morons? [was: CloudFlare issues?]

[Randy Bush]

perhaps the good side of this saga is that it may be an inflection point

randy

Re: CloudFlare issues?

[Randy Bush]

>> Respectfully, I believe Cloudflare’s public comments today have been
>> a real disservice. This blog post, and your CEO on Twitter today,
>> took every opportunity to say “DAMN THOSE MORONS AT 701!”. They’re
>> not.
> 
> I presume that seeing a CF blog post isn’t regular for you. :-).

never seen such a thing :)

amidst all this conjecturbation and blame casting, have any of the
parties *directly* involved, i.e. 701 and their customer, issued any
sort of post mortem from which we might learn?

randy

Public Subnet re-assignments

[Scott]

First, sorry if this is a bit of a noob question.

I'm trying to find a way of preventing a slew of traffic to an IP, or
IP's, when I join two /30 public subnets to a /29. It appears that while
the ranges are /30 someone is trying to brute-force the network and/or
broadcast addresses for the ranges. When I change them to be a /29, now
the router sees the traffic and starts dropping packets. Are there any
suggestions for mitigating this behavior or is it just the nature of the
beast?

-- 
101010

Re: BGP filtering study resources (Was: CloudFlare issues?)

[Alex Band]

For further community-driven RPKI information there is:

https://rpki.readthedocs.io/ 

Along with an FAQ:

https://rpki.readthedocs.io/en/latest/about/faq.html

Cheers,

-Alex

> On 25 Jun 2019, at 17:55, BATTLES, TIM  wrote:
> 
> https://www.nccoe.nist.gov/projects/building-blocks/secure-inter-domain-routing
>  
> Timothy A Battles
> Chief Security Office
> 314-280-4578
> tb2...@att.com
> 12976 Hollenberg Dr
> Bridgeton, MO 63044
>  
> The information contained in this e-mail, including any attachment(s), is 
> intended solely for use by the named addressee(s).  If you are not the 
> intended recipient, or a person designated as responsible for delivering such 
> messages to the intended recipient, you are not authorized to disclose, copy, 
> distribute or retain this message, in whole or in part, without written 
> authorization from the sender.  This e-mail may contain proprietary, 
> confidential or privileged information. If you have received this message in 
> error, please notify the sender immediately. 
>  
>  
> From: NANOG  On Behalf Of Tom Beecher
> Sent: Tuesday, June 25, 2019 9:42 AM
> To: Job Snijders 
> Cc: NANOG 
> Subject: Re: BGP filtering study resources (Was: CloudFlare issues?)
>  
> Job also enjoys having his ID checked. Can we get a best practices link added 
> to the list for that?
>  
> On Tue, Jun 25, 2019 at 10:27 AM Job Snijders  wrote:
> Dear Stephen,
> 
> On Tue, Jun 25, 2019 at 07:04:12AM -0700, Stephen Satchell wrote:
> > On 6/25/19 2:25 AM, Katie Holly wrote:
> > > Disclaimer: As much as I dislike Cloudflare (I used to complain
> > > about them a lot on Twitter), this is something I am absolutely
> > > agreeing with them. Verizon failed to do the most basic of network
> > > security, and it will happen again, and again, and again...
> > 
> > I used to be a quality control engineer in my career, so I have a
> > question to ask from the perspective of a QC guy:  what is the Best
> > Practice for minimizing, if not totally preventing, this sort of
> > problem?  Is there a "cookbook" answer to this?
> > 
> > (I only run edge networks now, and don't have BGP to worry about.  If
> > my current $dayjob goes away -- they all do -- I might have to get
> > back into the BGP game, so this is not an idle query.)
> > 
> > Somehow "just be careful and clueful" isn't the right answer.
> 
> Here are some resources which maybe can serve as a starting point for
> anyone interested in the problem space:
> 
> presentation: Architecting robust routing policies
> pdf: 
> https://ripe77.ripe.net/presentations/59-RIPE77_Snijders_Routing_Policy_Architecture.pdf
> video: 
> https://ripe77.ripe.net/archive/video/Job_Snijders-B._BGP_Policy_Update-20181017-140440.mp4
> 
> presentation: Practical Everyday BGP filtering "Peerlocking"
> pdf: http://instituut.net/~job/NANOG67_NTT_peerlocking_JobSnijders.pdf
> video: https://www.youtube.com/watch?v=CSLpWBrHy10
> 
> RFC 8212 ("EBGP default deny") and why we should ask our vendors like
> Cisco IOS, IOS XE, NX-OS, Juniper, Arista, Brocade, etc... to be
> compliant with this RFC:
> slides 2-14: 
> http://largebgpcommunities.net/presentations/ITNOG3-Job_Snijders_Recent_BGP_Innovations.pdf
> skip to the rfc8212 part: https://youtu.be/V6Wsq66-f40?t=854
> compliance tracker: http://github.com/bgp/RFC8212
> 
> The NLNOG Day in Fall 2018 has a wealth of RPKI related presentations
> and testimonies: https://nlnog.net/nlnog-day-2018/
> 
> Finally, there is the NLNOG BGP Filter Guide: http://bgpfilterguide.nlnog.net/
> If you spot errors or have suggestions, please submit them via github
> https://github.com/nlnog/bgpfilterguide
> 
> Please let me or the group know should you require further information,
> I love talking about this topic ;-)
> 
> Kind regards,
> 
> Job

RE: BGP filtering study resources (Was: CloudFlare issues?)

[BATTLES, TIM]

https://www.nccoe.nist.gov/projects/building-blocks/secure-inter-domain-routing

Timothy A Battles
Chief Security Office
314-280-4578
tb2...@att.com
12976 Hollenberg Dr
Bridgeton, MO 63044

The information contained in this e-mail, including any attachment(s), is 
intended solely for use by the named addressee(s).  If you are not the intended 
recipient, or a person designated as responsible for delivering such messages 
to the intended recipient, you are not authorized to disclose, copy, distribute 
or retain this message, in whole or in part, without written authorization from 
the sender.  This e-mail may contain proprietary, confidential or privileged 
information. If you have received this message in error, please notify the 
sender immediately.


From: NANOG  On Behalf Of Tom Beecher
Sent: Tuesday, June 25, 2019 9:42 AM
To: Job Snijders 
Cc: NANOG 
Subject: Re: BGP filtering study resources (Was: CloudFlare issues?)

Job also enjoys having his ID checked. Can we get a best practices link added 
to the list for that?

On Tue, Jun 25, 2019 at 10:27 AM Job Snijders 
mailto:j...@ntt.net>> wrote:
Dear Stephen,

On Tue, Jun 25, 2019 at 07:04:12AM -0700, Stephen Satchell wrote:
> On 6/25/19 2:25 AM, Katie Holly wrote:
> > Disclaimer: As much as I dislike Cloudflare (I used to complain
> > about them a lot on Twitter), this is something I am absolutely
> > agreeing with them. Verizon failed to do the most basic of network
> > security, and it will happen again, and again, and again...
>
> I used to be a quality control engineer in my career, so I have a
> question to ask from the perspective of a QC guy:  what is the Best
> Practice for minimizing, if not totally preventing, this sort of
> problem?  Is there a "cookbook" answer to this?
>
> (I only run edge networks now, and don't have BGP to worry about.  If
> my current $dayjob goes away -- they all do -- I might have to get
> back into the BGP game, so this is not an idle query.)
>
> Somehow "just be careful and clueful" isn't the right answer.

Here are some resources which maybe can serve as a starting point for
anyone interested in the problem space:

presentation: Architecting robust routing policies
pdf: 
https://ripe77.ripe.net/presentations/59-RIPE77_Snijders_Routing_Policy_Architecture.pdf
video: 
https://ripe77.ripe.net/archive/video/Job_Snijders-B._BGP_Policy_Update-20181017-140440.mp4

presentation: Practical Everyday BGP filtering "Peerlocking"
pdf: 
http://instituut.net/~job/NANOG67_NTT_peerlocking_JobSnijders.pdf
video: 
https://www.youtube.com/watch?v=CSLpWBrHy10

RFC 8212 ("EBGP default deny") and why we should ask our vendors like
Cisco IOS, IOS XE, NX-OS, Juniper, Arista, Brocade, etc... to be
compliant with this RFC:
slides 2-14: 
http://largebgpcommunities.net/presentations/ITNOG3-Job_Snijders_Recent_BGP_Innovations.pdf
skip to the rfc8212 part: 
https://youtu.be/V6Wsq66-f40?t=854
compliance tracker: 
http://github.com/bgp/RFC8212

The NLNOG Day in Fall 2018 has a wealth of RPKI related presentations
and testimonies: 

How Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline Today

[Martin J. Levy]

Cloudflare blog on the outage is out.
https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-knocked-large-parts-of-the-internet-offline-today/

Martin

On Mon, Jun 24, 2019 at 3:57 AM Dmitry Sherman  wrote:

> Hello are there any issues with CloudFlare services now?
>
> Dmitry Sherman
> dmi...@interhost.net
> Interhost Networks Ltd
> Web: http://www.interhost.co.il
> fb: https://www.facebook.com/InterhostIL
> Office: (+972)-(0)74-7029881 Fax: (+972)-(0)53-7976157
>
>

Re: AWS 40/100G wholesale Express-Route ?

[Paul Zugnoni via NANOG]

Hi,
Some quick terminology to be clear, AWS uses the term "Direct Connect"
whereas MS Azure uses "Express Route". Right now, max link bandwidths
each:
  - AWS Direct Connect: 10G.
  - Azure Express Route: 100G (though I'm unsure this is available in
every location)
  - GCP Dedicated Interconnect: 10G (100G in beta)

If your equipment is not local to a location where your desired cloud
provider has the handoff, then you must have some carrier service in
between. There are several partners of these cloud providers. You can
buy a circuit from them to reach a port on the cloud provider side.
The partner handles the lower layer connectivity into the cloud
provider gear. Some partners may offer above 10G but you should review
in detail how they manage the bandwidth to be sure you get the
performance you need.

PZ

On Mon, Jun 24, 2019 at 10:00 AM Jérôme Nicolle  wrote:
>
> Hello everyone,
>
> I was wondering, is there any way to get more than a 10G port for a PNI
> with AWS customers ?
>
> Right now I'm looking at 4 ridiculously expensive X-Cos (on two
> locations, so that makes 8) to establish a redundant 40Gbps backhaul,
> where I have 40/100G ports available.
>
> How could we deal with that ? Is there an "off-market" offering for
> higher speed interconnects ?
>
> Best regards,
>
> --
> Jérôme Nicolle
> +33 6 19 31 27 14
>


-- 
PZ
Head of Datacenter and Network Infrastructure, Wish
p...@wish.com +1-650-313-3458

Re: BGP filtering study resources (Was: CloudFlare issues?)

[Tom Beecher]

Job also enjoys having his ID checked. Can we get a best practices link
added to the list for that?

On Tue, Jun 25, 2019 at 10:27 AM Job Snijders  wrote:

> Dear Stephen,
>
> On Tue, Jun 25, 2019 at 07:04:12AM -0700, Stephen Satchell wrote:
> > On 6/25/19 2:25 AM, Katie Holly wrote:
> > > Disclaimer: As much as I dislike Cloudflare (I used to complain
> > > about them a lot on Twitter), this is something I am absolutely
> > > agreeing with them. Verizon failed to do the most basic of network
> > > security, and it will happen again, and again, and again...
> >
> > I used to be a quality control engineer in my career, so I have a
> > question to ask from the perspective of a QC guy:  what is the Best
> > Practice for minimizing, if not totally preventing, this sort of
> > problem?  Is there a "cookbook" answer to this?
> >
> > (I only run edge networks now, and don't have BGP to worry about.  If
> > my current $dayjob goes away -- they all do -- I might have to get
> > back into the BGP game, so this is not an idle query.)
> >
> > Somehow "just be careful and clueful" isn't the right answer.
>
> Here are some resources which maybe can serve as a starting point for
> anyone interested in the problem space:
>
> presentation: Architecting robust routing policies
> pdf:
> https://ripe77.ripe.net/presentations/59-RIPE77_Snijders_Routing_Policy_Architecture.pdf
> video:
> https://ripe77.ripe.net/archive/video/Job_Snijders-B._BGP_Policy_Update-20181017-140440.mp4
>
> presentation: Practical Everyday BGP filtering "Peerlocking"
> pdf: http://instituut.net/~job/NANOG67_NTT_peerlocking_JobSnijders.pdf
> video: https://www.youtube.com/watch?v=CSLpWBrHy10
>
> RFC 8212 ("EBGP default deny") and why we should ask our vendors like
> Cisco IOS, IOS XE, NX-OS, Juniper, Arista, Brocade, etc... to be
> compliant with this RFC:
> slides 2-14:
> http://largebgpcommunities.net/presentations/ITNOG3-Job_Snijders_Recent_BGP_Innovations.pdf
> skip to the rfc8212 part: https://youtu.be/V6Wsq66-f40?t=854
> compliance tracker: http://github.com/bgp/RFC8212
>
> The NLNOG Day in Fall 2018 has a wealth of RPKI related presentations
> and testimonies: https://nlnog.net/nlnog-day-2018/
>
> Finally, there is the NLNOG BGP Filter Guide:
> http://bgpfilterguide.nlnog.net/
> If you spot errors or have suggestions, please submit them via github
> https://github.com/nlnog/bgpfilterguide
>
> Please let me or the group know should you require further information,
> I love talking about this topic ;-)
>
> Kind regards,
>
> Job
>

Re: Are network operators morons? [was: CloudFlare issues?]

[Christopher Morrow]

(thanks, btw, again)

On Tue, Jun 25, 2019 at 8:33 AM Patrick W. Gilmore  wrote:
> It is not like 701 is causing problems every week, or even ever year. If you 
> think this one incident proves they are ‘morons’, you are only showing you 
> are neither experienced nor mature enough to make that judgement.
>

I would be shocked if 701 is no longer filtering customers by default.
I know they weren't filtering 'peers'.

it seems like the particular case yesterday was a missed customer
prefix-list :( which is sad, but happens.
the japan incident seems to be the other type, I'd guess.

-chris

BGP filtering study resources (Was: CloudFlare issues?)

[Job Snijders]

Dear Stephen,

On Tue, Jun 25, 2019 at 07:04:12AM -0700, Stephen Satchell wrote:
> On 6/25/19 2:25 AM, Katie Holly wrote:
> > Disclaimer: As much as I dislike Cloudflare (I used to complain
> > about them a lot on Twitter), this is something I am absolutely
> > agreeing with them. Verizon failed to do the most basic of network
> > security, and it will happen again, and again, and again...
> 
> I used to be a quality control engineer in my career, so I have a
> question to ask from the perspective of a QC guy:  what is the Best
> Practice for minimizing, if not totally preventing, this sort of
> problem?  Is there a "cookbook" answer to this?
> 
> (I only run edge networks now, and don't have BGP to worry about.  If
> my current $dayjob goes away -- they all do -- I might have to get
> back into the BGP game, so this is not an idle query.)
> 
> Somehow "just be careful and clueful" isn't the right answer.

Here are some resources which maybe can serve as a starting point for
anyone interested in the problem space:

presentation: Architecting robust routing policies
pdf: 
https://ripe77.ripe.net/presentations/59-RIPE77_Snijders_Routing_Policy_Architecture.pdf
video: 
https://ripe77.ripe.net/archive/video/Job_Snijders-B._BGP_Policy_Update-20181017-140440.mp4

presentation: Practical Everyday BGP filtering "Peerlocking"
pdf: http://instituut.net/~job/NANOG67_NTT_peerlocking_JobSnijders.pdf
video: https://www.youtube.com/watch?v=CSLpWBrHy10

RFC 8212 ("EBGP default deny") and why we should ask our vendors like
Cisco IOS, IOS XE, NX-OS, Juniper, Arista, Brocade, etc... to be
compliant with this RFC:
slides 2-14: 
http://largebgpcommunities.net/presentations/ITNOG3-Job_Snijders_Recent_BGP_Innovations.pdf
skip to the rfc8212 part: https://youtu.be/V6Wsq66-f40?t=854
compliance tracker: http://github.com/bgp/RFC8212

The NLNOG Day in Fall 2018 has a wealth of RPKI related presentations
and testimonies: https://nlnog.net/nlnog-day-2018/

Finally, there is the NLNOG BGP Filter Guide: http://bgpfilterguide.nlnog.net/
If you spot errors or have suggestions, please submit them via github
https://github.com/nlnog/bgpfilterguide

Please let me or the group know should you require further information,
I love talking about this topic ;-)

Kind regards,

Job

Re: CloudFlare issues?

[Ca By]

On Tue, Jun 25, 2019 at 7:06 AM Stephen Satchell  wrote:

> On 6/25/19 2:25 AM, Katie Holly wrote:
> > Disclaimer: As much as I dislike Cloudflare (I used to complain about
> > them a lot on Twitter), this is something I am absolutely agreeing with
> > them. Verizon failed to do the most basic of network security, and it
> > will happen again, and again, and again...
>
> I used to be a quality control engineer in my career, so I have a
> question to ask from the perspective of a QC guy:  what is the Best
> Practice for minimizing, if not totally preventing, this sort of
> problem?  Is there a "cookbook" answer to this?
>
> (I only run edge networks now, and don't have BGP to worry about.  If my
> current $dayjob goes away -- they all do -- I might have to get back
> into the BGP game, so this is not an idle query.)
>
> Somehow "just be careful and clueful" isn't the right answer.


1. Know what to expect — create policy to enforce routes and paths that you
expect, knowing sometimes this may be very broad

2. Enforce what you expect — drop routes and session that do not conform

3.  Use all the internal tools in series as layers of defense —
as-path-list with regex, ip prefix lists, max-routes — they work in series
and all must match. Shoving everything into a route-map is not best,
because what happens when that policy breaks?  Good to have layers.

4. Use irr, rpki, and alarming as external ecosystem tools.

5. Dont run noction or ios, unsafe defaults.

6. When on the phone with your peer, verbally check to make sure they
double check their policy.  Dont assume.





>

Re: Are network operators morons? [was: CloudFlare issues?]

[Mark Tinka]

On 25/Jun/19 14:59, Adam Kennedy via NANOG wrote:

>
>
> I believe there probably is a happy medium we can all meet, sort of
> our own ISP DMZ, where we can help one another in the simple mistakes
> or cut each other some slack in those difficult times. I like to think
> NANOG is that place.

Isn't that the point of NOG's, and why we rack so many air miles each
year trying to meet each other and break bread (or something) while
checking the Competition Hats at the door?

Mark.

Re: CloudFlare issues?

[Aftab Siddiqui]

Hi Stephen,


> I used to be a quality control engineer in my career, so I have a
> question to ask from the perspective of a QC guy:  what is the Best
> Practice for minimizing, if not totally preventing, this sort of
> problem?  Is there a "cookbook" answer to this?
>

As suggested by Job in the thread above,

- deploy RPKI based BGP Origin validation (with invalid == reject)
- apply maximum prefix limits on all EBGP sessions
- ask your router vendor to comply with RFC 8212 ('default deny')
- turn off your 'BGP optimizers' --> You actually don't need that at
all. I survived without any optimizer.

Aslo, read RFC7454 and join MANRS :)

Regards,
Aftab Siddiqui

Re: CloudFlare issues?

[Stephen Satchell]

On 6/25/19 2:25 AM, Katie Holly wrote:
> Disclaimer: As much as I dislike Cloudflare (I used to complain about
> them a lot on Twitter), this is something I am absolutely agreeing with
> them. Verizon failed to do the most basic of network security, and it
> will happen again, and again, and again...

I used to be a quality control engineer in my career, so I have a
question to ask from the perspective of a QC guy:  what is the Best
Practice for minimizing, if not totally preventing, this sort of
problem?  Is there a "cookbook" answer to this?

(I only run edge networks now, and don't have BGP to worry about.  If my
current $dayjob goes away -- they all do -- I might have to get back
into the BGP game, so this is not an idle query.)

Somehow "just be careful and clueful" isn't the right answer.

Re: Are network operators morons? [was: CloudFlare issues?]

[Adam Kennedy via NANOG]

Now with that out of the way...  The mentality of everyone working together
for a Better Internet (tm) is sort of a mantra of WISPA and WISPs in
general. It is a mantra that has puzzled me and perplexed my own feelings
as a network engineer. Do I want a better overall experience for my users
and customers? Absolutely. Do I strive to make our network the best...
pause... in the world? Definitely. Should I do the same to help a
neighboring ISP, a competitor? This is where I scratch my head. You would
absolutely think that we would all want a better overall Internet. One that
we can depend on in times of need. One that we can be proud of. But we are
driven, unfortunately, by our C-level execs to shun the competition and do
whatever we can to get a leg up on everyone else. While this is good for
the bottom line it is not exactly a healthy mentality to pit everyone
against each other. It causes animosity between providers and we end up
blaming each other for something simple and then claim they are stupid. A
mistake that may be easy to make, a mistake that we have probably made
ourselves a few times, perhaps a mistake we can learn to shrug off.

I believe there probably is a happy medium we can all meet, sort of our own
ISP DMZ, where we can help one another in the simple mistakes or cut each
other some slack in those difficult times. I like to think NANOG is that
place.

--

Adam Kennedy, Network & Systems Engineer

adamkenn...@watchcomm.net

*Watch Communications*

(866) 586-1518






On Tue, Jun 25, 2019 at 8:50 AM Matthew Walster  wrote:

>
>
> On Tue, 25 Jun 2019, 14:31 Patrick W. Gilmore,  wrote:
>
>> I must be old. All I can think is Kids These Days, and maybe Get Off My
>> BGP, er Lawn.
>>
>
> Maybe they ought to [puts on shades] mind their MANRS.
>
> M (scuttling away)
>
>>

Re: Are network operators morons? [was: CloudFlare issues?]

[Matthew Walster]

On Tue, 25 Jun 2019, 14:31 Patrick W. Gilmore,  wrote:

> I must be old. All I can think is Kids These Days, and maybe Get Off My
> BGP, er Lawn.
>

Maybe they ought to [puts on shades] mind their MANRS.

M (scuttling away)

>

Are network operators morons? [was: CloudFlare issues?]

[Patrick W. Gilmore]

[Removing the attribution, because many people have made statements like this 
over the last day - or year. Just selecting this one as a succinct and recent 
example to illustrate the point.]

>> This blog post, and your CEO on Twitter today, took every opportunity to say 
>> “DAMN THOSE MORONS AT 701!”.
> Damn those morons at 701, period.

I must be old. All I can think is Kids These Days, and maybe Get Off My BGP, er 
Lawn.

Any company running a large, high complex infrastructure is going to make 
mistakes. Period.

It is not like 701 is causing problems every week, or even ever year. If you 
think this one incident proves they are ‘morons’, you are only showing you are 
neither experienced nor mature enough to make that judgement.

To be clear, they may well be morons. I no longer know many people architecting 
and operating 701’s backbone, so I cannot tell you first-hand how smart they 
are. Maybe they are stupid, but exceptionally lucky. However, the facts at hand 
do not support your blanket assertion, and making it does not speak well of you.

OTOH, I do have first-hand experience with previous CF blog posts, and to say 
they spin things in their favor is being generous. But then, it’s a blog post, 
i.e. Marketing. What else would you expect?


I know it is anathema to the ethos of the network engineers & architects to 
work together instead of hurling insults, but it would probably result in a 
better Internet. And isn’t that what we all (supposedly) want?

-- 
TTFN,
patrick

Re: CloudFlare issues?

[Tom Beecher]

Verizon Business / Enterprise is the access network, aka 701/2/3.

Verizon Media Group is the CDNs/Media side. Digital Media Services (
Edgecast ) , Yahoo, AOL. 15133 / 10310 / 1668. ( The entity formerly named
Oath, created when Yahoo was acquired. )

On Tue, Jun 25, 2019 at 06:54 Hank Nussbacher  wrote:

> On 25/06/2019 08:17, Christopher Morrow wrote:
> > On Tue, Jun 25, 2019 at 12:49 AM Hank Nussbacher 
> wrote:
> >> On 25/06/2019 03:03, Tom Beecher wrote:
> >>> Disclaimer : I am a Verizon employee via the Yahoo acquisition. I do
> >>> not work on 701.  My comments are my own opinions only.
> >>>
> >>> Respectfully, I believe Cloudflare’s public comments today have been a
> >>> real disservice. This blog post, and your CEO on Twitter today, took
> >>> every opportunity to say “DAMN THOSE MORONS AT 701!”. They’re not.
> >>>
> >>>
> >> Perhaps suggest to VZ management to use their blog:
> >> https://www.verizondigitalmedia.com/blog/
> > #coughwrongvz
> >
> > I think anyway - you probably mean:
> > https://enterprise.verizon.com/
> This post is unrelated to Verizon Enterprise?
>
> https://www.verizondigitalmedia.com/blog/2019/06/exponential-global-growth-at-75-tbps/
>
> -Hank
> >
> > GoodLuck! I think it's 3 clicks to: "www22.verizon.com" which gets
> > even moar fun!
> > The NOC used to answer if you called: +1-800-900-0241
> > which is in their whois records...
> >
> >> to contrandict what CF blogged about?
> >>
> >> -Hank
> >>
>
>

Re: CloudFlare issues?

[Hank Nussbacher]

On 25/06/2019 08:17, Christopher Morrow wrote:

On Tue, Jun 25, 2019 at 12:49 AM Hank Nussbacher  wrote:

On 25/06/2019 03:03, Tom Beecher wrote:

Disclaimer : I am a Verizon employee via the Yahoo acquisition. I do
not work on 701.  My comments are my own opinions only.

Respectfully, I believe Cloudflare’s public comments today have been a
real disservice. This blog post, and your CEO on Twitter today, took
every opportunity to say “DAMN THOSE MORONS AT 701!”. They’re not.



Perhaps suggest to VZ management to use their blog:
https://www.verizondigitalmedia.com/blog/

#coughwrongvz

I think anyway - you probably mean:
https://enterprise.verizon.com/

This post is unrelated to Verizon Enterprise?
https://www.verizondigitalmedia.com/blog/2019/06/exponential-global-growth-at-75-tbps/

-Hank


GoodLuck! I think it's 3 clicks to: "www22.verizon.com" which gets
even moar fun!
The NOC used to answer if you called: +1-800-900-0241
which is in their whois records...


to contrandict what CF blogged about?

-Hank

Re: CloudFlare issues?

[Rich Kulawiec]

On Mon, Jun 24, 2019 at 09:39:13PM -0400, Ross Tajvar wrote:
> A technical one - see below from CF's blog post:
> "It is unfortunate that while we tried both e-mail and phone calls to reach
> out to Verizon, at the time of writing this article (over 8 hours after the
> incident), we have not heard back from them, nor are we aware of them
> taking action to resolve the issue."

Which is why an operation the size of Verizon should be able to manage
the trivial task of monitoring its RFC 2142 role addresses 24x7 with a
response time measured in minutes.   And not just Verizon: every large
operation should be doing the same.  There is no excuse for failure
to implement this rudimentary operational practice.

[ And let me add that a very good way to deal with mail sent to those
addresses is to use procmail to pre-sort based on who it's from.  Every
time a message is received from a new source, a new procmail rule should
be added to classify it appropriately.  Over time, this makes it very
easy to identify traffic from clueful people vs. traffic from idiots,
and thus to readily discern what needs to be triaged first. ]

---rsk

Re: CloudFlare issues?

[Katie Holly]

Disclaimer : I am a Verizon employee via the Yahoo acquisition. I do not work 
on 701.  My comments are my own opinions only.

Disclaimer: As much as I dislike Cloudflare (I used to complain about them a 
lot on Twitter), this is something I am absolutely agreeing with them. Verizon 
failed to do the most basic of network security, and it will happen again, and 
again, and again...


This blog post, and your CEO on Twitter today, took every opportunity to say 
“DAMN THOSE MORONS AT 701!”.

Damn those morons at 701, period.


But do we know they didn’t?

They didn't, otherwise yesterday's LSE could have been prevented.


Do we know it was there and just setup wrong?

If it virtually exists but does not work, it does not exist.


Did another change at another time break what was there?

What's not there can not be changed. Keep in mind, another well-known route 
leak happened back in 2017 when Google leaked routes towards Verizon and 
Verizon silently accepted and propagated all of them without filtering. 
Probably nothing has changed since then.


Shouldn’t we be working on facts?

They have stated the facts.


to take a harder stance on the BGP optimizer that generated he bogus routes

The BGP optimizer was only the trigger for this event, the actual 
mis-configuration happened between 396531 and 701. IDGAF if 396531 or one of 
their peers uses a BGP optimizer, 701 should have filtered those out, but they 
decided to not do that instead.


You’re right to use this as a lever to push for proper filtering , RPKI, best 
practices.

Yes, and 701 should follow those "best practices".

Point being, I do network stuff since around 10 years and started doing BGP and 
internet routing related stuff only around three years ago and even _I_ can 
follow best practices. And if I have the knowledge about those things and can 
follow best practices, Verizon SURELY has enough resources to do so as well!

On 6/25/19 2:03 AM, Tom Beecher wrote:

Disclaimer : I am a Verizon employee via the Yahoo acquisition. I do not work 
on 701.  My comments are my own opinions only.

Respectfully, I believe Cloudflare’s public comments today have been a real 
disservice. This blog post, and your CEO on Twitter today, took every 
opportunity to say “DAMN THOSE MORONS AT 701!”. They’re not.

You are 100% right that 701 should have had some sort of protection mechanism 
in place to prevent this. But do we know they didn’t? Do we know it was there 
and just setup wrong? Did another change at another time break what was there? 
I used 701 many  jobs ago and they absolutely had filtering in place; it saved 
my bacon when I screwed up once and started readvertising a full table from a 
2nd provider. They smacked my session down an I got a nice call about it.

You guys have repeatedly accused them of being dumb without even speaking to 
anyone yet from the sounds of it. Shouldn’t we be working on facts?

Should they have been easier to reach once an issue was detected? Probably. 
They’re certainly not the first vendor to have a slow response time though. 
Seems like when an APAC carrier takes 18 hours to get back to us, we write it 
off as the cost of doing business.

It also would have been nice, in my opinion, to take a harder stance on the BGP 
optimizer that generated he bogus routes, and the steel company that failed BGP 
101 and just gladly reannounced one upstream to another. 701 is culpable for 
their mistakes, but there doesn’t seem like there is much appetite to shame the 
other contributors.

You’re right to use this as a lever to push for proper filtering , RPKI, best 
practices. I’m 100% behind that. We can all be a hell of a lot better at what 
we do. This stuff happens more than it should, but less than it could.

But this industry is one big ass glass house. What’s that thing about stones 
again?

On Mon, Jun 24, 2019 at 18:06 Justin Paine via NANOG mailto:nanog@nanog.org>> wrote:

FYI for the group -- we just published this: 
https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-knocked-large-parts-of-the-internet-offline-today/


_
*Justin Paine*
Director of Trust & Safety
PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D
101 Townsend St., San Francisco, CA 94107 





On Mon, Jun 24, 2019 at 2:25 PM Mark Tinka mailto:mark.ti...@seacom.mu>> wrote:



On 24/Jun/19 18:09, Pavel Lunin wrote:

 >
 > Hehe, I haven't seen this text before. Can't agree more.
 >
 > Get your tie back on Job, nobody listened again.
 >
 > More seriously, I see no difference between prefix hijacking and the
 > so called bgp optimisation based on completely fake announces on
 > behalf of other people.
 >
 > If ever your upstream or any other party who your company pays money
 > to does this dirty thing, now it's just