Verizon’s ESInet

[Ryan Finnesey via NANOG]

Would someone from Verizon or someone familiar with Verizon’s ESInet please 
contact me off list .

Ryan

LACNOG2021 - Call for Presentations (Deadline extension)

[Hernan Moguilevsky]

Hi NANOG,
 

The LACNOG Program Committee has extended the deadline for the
submission of proposals for LACNOG2021, up to August 6, 2021. All the
papers meeting the requirements specified in the call below, will be
considered

 

LACNOG 2021 - Call for Presentations

 

 

LACNOG, the Latin American and Caribbean Network Operators Group, will
hold its LACNOG 2021 conference from 11 to 15 October 2021 together with
the LACNIC 36 event.

 

The LACNOG 2021 Program Committee invites the Internet community to
submit their presentation proposals for the event.

 

In line with the spirit of LACNOG, proposed topics must be geared
towards Internet development in the region. The following is a
non-exhaustive list of some of the topics of interest for the LACNOG
2021 meeting:

 

●    Network operation and professional experiences, success stories

●    Internet of Things

●    MANRS

●    Community Networks

●    IPv6 integration and deployment

●    Experiences involving botnets, malware, spam, viruses, denial
of service attacks and exploit techniques

●    IP network architecture, sizing, configuration and administration

●    Routing and switching protocols, including unicast, multicast,
anycast, SDN, etc.

●    End-user applications (e.g., e-mail, HTTP, DNS, NFVs)

●    Value-added services such as VPNs, distributed systems, cloud
computing, etc.

●    Peering, Internet traffic exchange, IXPs

●    Network data security and management, attack mitigation

●    Network monitoring, performance, measurements and telemetry

●    Network automation, evolution and convergence

●    Infrastructure and physical transport, including optical and
wireless networks

●    Legislation, regulations and Internet governance issues

●    Research and education

 

Possible presentation formats include: 

 

●    Lightning talk: brief, 10-minute presentation (including a
space for Q)

●    Presentation: 20-minute presentation (including a space for Q)

●    Posters: includes a single-page PDF (A2or smaller) with the
basic information of the presentation plus a 2- to 5-minute video with
the presentation

 

Deadlines for the 2021 call for proposals will be as follows:

 

●    Reception of proposals: 25June to 6 August, 2021

●    Proposals will be accepted until: 6 August 2021 at 23:59 UTC-3
(Uruguay time)

●    Evaluation by the Program Committee: 7August to 16 August 2021

●    Announcement of results: 17 August 2021

●    Deadline for submitting the final presentation: 17 August - 17
September 2021

●    Final presentations will be accepted until: 17 September 2021
at 23:59 UTC-3 (Uruguay time)

●    Event date: 11 to 15 October 2021

 

Applicants must submit a summary and a draft of the slides of their
proposed presentation along with a brief biography, for which they must
use the form available athttps://eventos.nog.lat/e/lacnog2021


 

If your work is selected, you authorize LACNOG and LACNIC to publish
your name, photograph, biography and final work in the event program.

  

Speakers presenting their work at the LACNOG 2021 conference will
receive a certificate acknowledging their participation.

 

Guidelines for Submitting a Presentation for LACNOG have been prepared
that contain a description of the criteria that will be considered when
evaluating each proposal, presentation format details and other
information. These Guidelines are available
at https://www.lacnog.org/guiapresentaciones/
.

 

Communications with the Program Committee will be handled
through p...@lacnog.org .

 

We thank you in advance for your attention and look forward to receiving
your proposals for LACNOG 2021.

 

The Program Committee

 

 

Re: Global Akamai Outage

[Lukas Tribus]

Hello!

On Mon, 26 Jul 2021 at 17:50, heasley  wrote:
>
> Mon, Jul 26, 2021 at 02:20:39PM +0200, Lukas Tribus:
> > rpki-client 7.1 emits a new per VRP attribute: expires, which makes it
> > possible for RTR servers to stop considering outdated VRP's:
> > https://github.com/rpki-client/rpki-client-openbsd/commit/9e48b3b6ad416f40ac3b5b265351ae0bb13ca925
>
> Since rpki-client removes "outdated" (expired) VRPs, how does an RTR
> server "stop considering" something that does not exist from its PoV?

rpki-client can only remove outdated VRP's, if it a) actually runs and
b) if it successfully completes a validation cycle. It also needs to
do this BEFORE the RTR server distributes data.

If rpki-client for whatever reason doesn't complete a validation cycle
[doesn't start, crashes, cannot write to the file] it will not be able
to update the file, which stayrtr reads and distributes.

If your VM went down with both rpki-client and stayrtr, and it stays
down for 2 days (maybe a nasty storage or virtualization problem or
maybe this just a PSU failure in a SPOF server), when the VM comes
backup, stayrtr will read and distribute 2 days old data - after all -
rpki-client is a periodic cronjob while stayrtr will start
immediately, so there will be plenty of time to distribute obsolete
VRP's. Just because you have another validator and RTR server in
another region that was always available, doesn't mean that the
erroneous and obsolete data served by this server will be ignored.

There are more reasons and failure scenarios why this 2 piece setup
(periodic RPKI validation, separate RTR daemon) can become a "split
brain". As you implement more complicated setups (a single global RPKI
validation result is distributed to regional RTR servers - the
cloudflare approach), things get even more complicated. Generally I
prefer the all in one approach for these reasons (FORT validator).

At least if it crashes, it takes down the RTR server with it:

https://github.com/NICMx/FORT-validator/issues/40#issuecomment-695054163


But I have to emphasize that all those are just examples. Unknown bugs
or corner cases can lead to similar behavior in "all in one" daemons
like Fort and Routinator. That's why specific improvements absolutely
do not mean we don't have to monitor the RTR servers.


lukas

DNS contact for raytheon

[Wes Hardaker]

If someone from Raytheon could contact me about their DNS zones I'd
appreciate it.

-- 
Wes Hardaker 
USC/ISI

Re: Global Akamai Outage

[Mark Tinka]

On 7/26/21 17:50, heasley wrote:


Since rpki-client removes "outdated" (expired) VRPs, how does an RTR
server "stop considering" something that does not exist from its PoV?

Did you mean that it can warn about impending expiration?


StayRTR reads the VRP data generated by rpki-client.

Mark.

Re: Global Akamai Outage

[heasley]

Mon, Jul 26, 2021 at 02:20:39PM +0200, Lukas Tribus:
> rpki-client 7.1 emits a new per VRP attribute: expires, which makes it
> possible for RTR servers to stop considering outdated VRP's:
> https://github.com/rpki-client/rpki-client-openbsd/commit/9e48b3b6ad416f40ac3b5b265351ae0bb13ca925

Since rpki-client removes "outdated" (expired) VRPs, how does an RTR
server "stop considering" something that does not exist from its PoV?

Did you mean that it can warn about impending expiration?

Re: 1G/10G BaseT switch recommendation

[Brandon Martin]

On 7/22/21 2:46 PM, Drew Weaver wrote:

Hello everyone,

I’m looking for recommendations from the community on 48x10G RJ45/4-6 
SFP28 (uplink ports) switches that people actually like working with.


Features are VPC or non-vendor specific equivalent, L2/L3 BGP/OSPFv3, 
ACLs, functional CoPP and some sort of API to manage them. [the CLI 
would work, my lib can handle most Networking OS CLIs anyway]


My problem point is coming from the RJ45 requirement, most vendors have 
one switch that they sell that is RJ45 at 10G or at the most one in each 
line (enterprise/datacenter) and they seem to be almost an afterthought. 
[probably because SFP28 is better in every way if you are already using 
fiber at the endpoint] sadly, we are not.


I just want to make sure I am not excluding any vendors from my research.

I appreciate any suggestions or recommendations. Can even keep it 
off-list if you want.


Arista 7160-48TC6 gets pretty close to your requirements, I think.  It 
has 48x10GBASE-T ports + 6xQSFP28 100GBASE-R.  The QSFP28 support 
breakout with SR4 or PLR4 optics AFAIK , so you can use them for 25GbE 
if you prefer.


You'll need the FLX-LITE license to officially get access to the layer 3 
features IIRC.

--
Brandon Martin

Re: Spamhaus ASN-DROP list

[Steve Linford]

Hi,

Contact the SBL team via the Lookup form at https://check.spamhaus.org/

The form says 'IP or Domain' but it will also look up ASNs so just put your ASN 
in. That will allow you to create a ticket with the right team and the issue 
should then get dealt with fairly quickly.

Regards,

  Steve Linford
  The Spamhaus Project
  https://www.spamhaus.org
  

> On 23 Jul 2021, at 09:08, Siyuan Miao  wrote:
> 
> Hi All,
> 
> One of our ASNs has been listed in the Spamhaus ASN-DROP list before it was 
> assigned to us. 
> 
> We emailed them last year but didn't get a response. Could anyone from 
> Spamhaus contact us off the list?
> 
> Best Regards,
> Siyuan
> 

Re: Global Akamai Outage

[Mark Tinka]

On 7/26/21 14:20, Lukas Tribus wrote:


Some specific failure scenarios are currently being addressed, but
this doesn't make monitoring optional:

rpki-client 7.1 emits a new per VRP attribute: expires, which makes it
possible for RTR servers to stop considering outdated VRP's:
https://github.com/rpki-client/rpki-client-openbsd/commit/9e48b3b6ad416f40ac3b5b265351ae0bb13ca925

stayrtr (a gortr fork), will consider this attribute in the future:
https://github.com/bgp/stayrtr/issues/3


I was just about to cite these two as improving this particular issue in 
upcoming releases.


I am running RPKI-Client + StayRTR, alongside Fort, and yes, while 
monitoring should be standard, improvements in the validation and RTR 
objectives will also go a long way in mitigating these issues.


What's quickly happening in this space is that not all validators and 
RTR servers are going to made equal. There are a number of options 
currently available (both deprecated and current), but I expect that we 
may settle on just a handful, as experience increases. And in what 
remains, I anticipate that they will be bolstered to consider these very 
problems.


Mark.

Re: Global Akamai Outage

[Lukas Tribus]

Hello,


On Mon, 26 Jul 2021 at 11:40, Mark Tinka  wrote:
> I can count, on my hands, the number of RPKI-related outages that we
> have experienced, and all of them have turned out to be a
> misunderstanding of how ROA's work, either by customers or some other
> network on the Internet. The good news is that all of those cases were
> resolved within a few hours of notifying the affected party.

That's good, but the understanding of operational issues in the RPKI
systems in the wild is underwhelming, we are bound to make the same
mistakes of DNS all over again.

Yes, a complete failure of an RTR server theoretically does not have
big negative effects in networks. But failure of RPKI validation with
a separate RTR server can lead to outdated VRP's on the routers, just
as RTR server bugs will, which is why monitoring not only for
availability but also whether the data is actually not outdated is
*very* necessary.


Here some examples (both of operators POV as well as actual failure scenarios):


https://mailman.nanog.org/pipermail/nanog/2020-August/208982.html

> we are at fault for not deploying the validation service in a redundant
> setup and for failing at monitoring the service. But we did so because
> we thought it not to be too important, because a failed validation
> service should simply lead to no validation, not a crashed router.

In this case a RTR client bug crashed the router. But the point is
that it is not clear that setting up RPKI validators and RTR servers
is a serious endeavor and monitoring it is not optional.



https://github.com/cloudflare/gortr/issues/82

> we noticed that one the ROAs was wrong. When I pulled output.json
> from octorpki (/output.json), it had the correct value. However when
> I ran rtrdump, it had different ASN value for the prefix. Restarting
> gortr process did fix it. Sending SIGHUP did not.



https://github.com/RIPE-NCC/rpki-validator-3/issues/264

> yesterday we saw a unexpected ROA propagation delay.
>
> After updating a ROA in the RIPE lirportal, NTT, Telia and Cogent
> saw the update within an hour, but a specific rpki validator
> 3.1-2020.08.06.14.39 in a third party network did not converge
> for more than 4 hours.


I wrote a naive nagios script to check for stalled serials on a RTR server:
https://github.com/lukastribus/rtrcheck

and talked about it in this his blog post (shameless plug):
https://labs.ripe.net/author/lukas_tribus/rpki-rov-about-stale-rtr-servers-and-how-to-monitor-them/

This is on the validation/network side. On the CA side, similar issues apply.

I believe we still lack a few high level outages caused by
insufficient reliability in the RPKI stacks for people to start taking
it seriously.


Some specific failure scenarios are currently being addressed, but
this doesn't make monitoring optional:

rpki-client 7.1 emits a new per VRP attribute: expires, which makes it
possible for RTR servers to stop considering outdated VRP's:
https://github.com/rpki-client/rpki-client-openbsd/commit/9e48b3b6ad416f40ac3b5b265351ae0bb13ca925

stayrtr (a gortr fork), will consider this attribute in the future:
https://github.com/bgp/stayrtr/issues/3



cheers,
lukas

Re: Global Akamai Outage

[Mark Tinka]

On 7/26/21 07:25, Saku Ytti wrote:


Doesn't matter. And I'm not trying to say RPKI is a bad thing. I like
that we have good AS:origin mapping that is verifiable and machine
readable, that part of the solution will be needed for many
applications which intend to improve the Internet by some metric.
And of course adding any complexity will have some rearing problems,
particularly if the problem it attempts to address is infrequently
occurring, so it would be naive not to expect an increased rate of
outages while maturing it.


Yes, while RPKI fixes problems that genuinely occur infrequently, it's 
intended to work very well for when those problems do occur, especially 
the intentional hijacks, because when they do occur, it disrupts quite a 
large part of the Internet, even if for a few minutes or couple of 
hours. So from that standpoint, RPKI does add value.


Where I do agree with you is that we should restrain ourselves from 
applying RPKI to use-cases that are non-core to its reasons for 
existence, e.g., AS0.


I can count, on my hands, the number of RPKI-related outages that we 
have experienced, and all of them have turned out to be a 
misunderstanding of how ROA's work, either by customers or some other 
network on the Internet. The good news is that all of those cases were 
resolved within a few hours of notifying the affected party.


Mark.