Re: A survey on BGP MRAI timer values in practice

[Saku Ytti]

On Wed, 9 Jun 2021 at 01:18, Adam Thompson  wrote:

If your work results in actionable recommendations such as "don't use BGP
> out-delay timers to mitigate XYZ in circumstance LMNO, do ABC instead",
> that's fantastic.  Please keep us advised, and do post aggregated survey
> results here once you close the survey.
>

What is actionable? What is the goal? The question as OP presented contains
some assumptions

a) better convergence is needed
b) MRAI is important part of the solution space

Neither are provable. We already know how to make DFZ convergence really
fast (or at least orders of magnitude faster than it is), that information
exists, but that isn't deployed because customers are not asking for it, so
providers are not aware that there is room for improvements.

Things don't optimise to be as good as they can be, things optimise to be
as bad as the market allows them to be. And the market accepts the DFZ
convergence.

If you do decide to optimise for DFZ convergence, without commercial
pressure, you will risk lower availability, because you'll be using
configuration less tested by other customers and everyone knows how
terrible quality every NOS is. Everyone finds novel bugs, in the same damn
protocols we've ran +20 years. It's like running Windows and Linux and
regularly finding out listing files in a directory breaks your service,
year after year after year.

For those who are interested in better convergence
   - change your interface down reporting to 0 (there may be delay before
interface down is reported to system, so that optical protection works
without causing outage)
   - use 'add-path' or at least 'best-external' in iBGP, so that you always
have backup eBGP route immediately available once best is invalidated
(normally you have lot of delay to find next best, once you lose your best
eBGP)
   - tie your route validity to IGP, so you can invalidate your BGP the
moment IGP disappears
   - ensure IGP converges fast (another topic)
   - set MRAI to 0
   - use PIC edge
   - ensure your BGP NLRI can be as large as MTU allows
   - ensure your convergence isn't bottle necked by slow peer in group
   - ensure you are not dropping received TCP packets on punt path
   - ensure your fast external fallover works (eBGP down, on int down) this
is quite easy to break
   - then ensure everyone else in the DFZ does the same thing



But from a business POV, don't do any of this, you will have more bugs and
lower availability and your customers will be less happy.



> I *am *specifically interested in the answer to "Have you ever had to
> adjust BGP out-delay with any of your peers, and why?"  It would be great
> if we could derive that answer from the survey results, but anecdotal
> replies here would also be helpful.  All you larger(-than-me) network
> operators out there: when would I need to use out-delay?  Why?  What does
> it accomplish?
>
> Good luck in reformulating your survey to get better engagement,
> -Adam
>
> *Adam Thompson*
> Consultant, Infrastructure Services
> [image: 1593169877849]
> 100 - 135 Innovation Drive
> Winnipeg, MB, R3T 6A8
> (204) 977-6824 or 1-800-430-6404 (MB only)
> athomp...@merlin.mb.ca
> www.merlin.mb.ca
>
> --
> *From:* NANOG  on behalf
> of Saku Ytti 
> *Sent:* June 8, 2021 01:06
> *To:* shahr...@cs.umass.edu 
> *Cc:* nanog list ; Arun Venkataramani 
> *Subject:* Re: A survey on BGP MRAI timer values in practice
>
> On Mon, 7 Jun 2021 at 19:32,  wrote:
>
> > We often read that the Internet (i.e. BGP) has a long convergence delay.
> > But why is it so slow? And can we (researchers) do anything about it?
>
> Create business incentives to improve it. This is a non-technical
> problem, we've long had technical tools to make it fast, there just
> isn't incentive to make it fast. Customers are not asking operators
> for better convergence speeds.
>
> > Please help us out to find out by answering our short anonymous survey
> > (<10 minutes).
>
> Can you tell me what have you done so far? What are the default MRAI
> values for each AFI/SAFI for IOS, IOS-XR, Junos, SROS, VRP and EOS?
> Then people responding don't have to check what their NOS does, they
> can refer to your table and tell the default value, since this is what
> >99% will be using.
>
> Now your survey has built-in selection bias, people who answer it are
> people who know what it is and who are concerned about it and have
> changed it, this is not a representative group and you will start your
> work with very bad data.
>
>
>
> --
>   ++ytti
>


-- 
  ++ytti

Re: BCP38 on public-facing Ubuntu servers

[Stephen Satchell]

On 6/8/21 2:38 PM, Fran via NANOG wrote:

Hey,

to my knowledge there is no IPv6 equivalent for 
net.ipv4.conf.all.rp_filter.


Therefore I use netfilter to do the RP filtering for both address families.

ip(6)tables -t raw -I PREROUTING -m rpfilter --invert -j DROP

Using the raw tables less resources are used, but you could also choose 
other tables.

Details abour rpfilter can be found here [1].

This can also be achieved using nftables [2].


I've been in discussions on how to filter packets with bad source 
addresses on several mailing lists, including this one.  For the last 
few weeks, I've been search for all the information I can find for how 
Linux implements rp_filter...which appears to have some holes.


Looking at /proc/sys/net/ipv6, there is no knob for rp_filter, so if 
your system is IPv6 enabled you have to use the built-in firewall.


For IPv4, I found kernel documentation, but it doesn't tell the whole 
story.  For that, I had to comb the kernel sources to find out all the 
details of rp_filter.  I've prepared a RFC letter of what I think I 
found, to be sent to the kernel developers.  Here is the text of what 
I'll be sending, with any constructive criticism I get from here:


Letter begins:

After looking at the source that appears to implement rp_filter
linux/net/ipv4/fib_frontend.c
I believe that I now understand the tests rp_filter performs to
validate the source address when net.ipv4.conf.*.rp_filter is
set to one or two for a given interface.

Does the new paragraph I have written accurately reflect what
happens?  If so, then I find out how to submit a patch to add the
clarification to the kernel document.

Description of rp_filter from
https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt

rp_filter - INTEGER
0 - No source validation.
1 - Strict mode as defined in RFC3704 Strict Reverse Path
Each incoming packet is tested against the FIB and if the
interface is not the best reverse path the packet check will
fail. By default failed packets are discarded.
2 - Loose mode as defined in RFC3704 Loose Reverse Path
Each incoming packet's source address is also tested against
the FIB and if the source address is not reachable via any
interface the packet check will fail.

[*new text here]

Current recommended practice in RFC3704 is to enable strict mode
to prevent IP spoofing from DDos attacks. If using asymmetric
routing or other complicated routing, then loose mode is
recommended.

The max value from conf/{all,interface}/rp_filter is used
when doing source validation on the {interface}.

Default value is 0. Note that some distributions enable it
in startup scripts.


Recommended addition where marked with "[*new text here]":
rp_filter will examine the source address of an incoming IP
packet by performing an FIB lookup.  In loose mode (value 2),
the packet is rejected if the source address is neither
UNICAST nor LOCAL nor IPSEC.  For strict mode (value 1) the
interface indicated by the FIB entry must also match the
interface on which the packet arrived.

Re: BCP38 on public-facing Ubuntu servers

[Fran via NANOG]

Hey,



to my knowledge there is no IPv6 equivalent for net.ipv4.conf.all.rp_filter.

Therefore I use netfilter to do the RP filtering for both address families.



ip(6)tables -t raw -I PREROUTING -m rpfilter --invert -j DROP



Using the raw tables less resources are used, but you could also choose other 
tables.
Details abour rpfilter can be found here [1].


This can also be achieved using nftables [2].


Best

Fran

[1] https://ipset.netfilter.org/iptables-extensions.man.html#lbBX
[2] 
https://wiki.nftables.org/wiki-nftables/index.php/Matching_routing_information



On 04.06.21 20:43, Jay Vosburgh wrote:

Grant Taylor via NANOG  wrote:


On 6/3/21 8:44 AM, William Herrin wrote:

rp_filter is great until your network is slightly less than a perfect
hierarchy. Then your Linux "router" starts mysteriously dropping packets
and, as with allow_local, Linux doesn't have any way to generate logs
about it so you end up with these mysteriously unexplained packet
discards matching no conceivable rule in iptables... This failure has
too often been the bane of my existence when using Linux for advanced
networking.


I don't remember the particulars, but I thought that was the domain of
log_martians (net.ipv4.conf.*.log_martians).

Without log_martians or explicitly looking for such, no, you won't get any
indication of such drops.


Yes, enabling the log_martians sysctl will generate a kernel log
message for each rp_filter failure (subject to rate limiting).  There
are also stat counters in /proc/net/stat/rt_cache (one line per CPU) for
in_martian_dst and in_martian_src which increment regardless of the
log_martians setting.

The rp_filter sysctl defaults to strict mode (== 1) on Ubuntu,
but can be set to loose mode (== 2); the difference is, essentially, in
strict mode the reverse path must be the same interface as the ingress
interface, whereas in loose mode the reverse path can be any interface
(as long as the source address is reachable).

https://www.kernel.org/doc/Documentation/networking/ip-sysctl.rst

-J

---
-Jay Vosburgh, jay.vosbu...@canonical.com

Re: A survey on BGP MRAI timer values in practice

[Adam Thompson]

+1 to Saku's concerns - I simply ignored the survey because I wasn't sure what 
MRAI was, and I wasn't sure what my values would be.  But I have time to be 
interested right now, so a-spelunking I go...

The term "MRAI" does not appear anywhere in Arista's or Extreme's 
documentation.  Nor does this timer interval appear in any BGP-related "show" 
output, on any of my platforms, that I can see.

I've found out that "out-delay" (not "delay out") is synonymous with MRAI and 
seems widely used.

I found "out-delay" in an Arista technote, and now I know how to override it on 
Aristas, and the default is zero (0).  Unfortunately, "show" commands on EOS 
will only show you the current out-delay if it is non-zero, which makes 
reporting it a bit difficult.

Extreme's MLXe platform doesn't appear to support an out-delay/MRAI knob at 
all, at least as far as I can tell.  I know there are several other current and 
former MLXe operators here, maybe one of them will know?

Based on my limited history with NANOG-L, I guess your initial email might have 
been seen by perhaps 20 people who immediately knew what you were talking 
about, and 2000 who didn't.  (I don't actually know subscriber numbers, that's 
totally a WAG.  And maybe more people have touched out-delay knobs than I 
think.  Dunno.)

 This illustrates the gap between academia 
and industry - academics can research a narrow topic, and come at issues from a 
theoretic standpoint using unfamiliar terminology.  As a practitioner, I get 
told to add carrier X as a peer by end-of-day Friday, and "just make it work".  
I wasn't even able to understand your initial question, because I haven't spent 
a semester understanding the intricacies of BGP propagation - I just know it 
usually works, knobs exist that I shouldn't fiddle with, and that's good enough 
for my job. 

If your work results in actionable recommendations such as "don't use BGP 
out-delay timers to mitigate XYZ in circumstance LMNO, do ABC instead", that's 
fantastic.  Please keep us advised, and do post aggregated survey results here 
once you close the survey.

I am specifically interested in the answer to "Have you ever had to adjust BGP 
out-delay with any of your peers, and why?"  It would be great if we could 
derive that answer from the survey results, but anecdotal replies here would 
also be helpful.  All you larger(-than-me) network operators out there: when 
would I need to use out-delay?  Why?  What does it accomplish?

Good luck in reformulating your survey to get better engagement,
-Adam

Adam Thompson
Consultant, Infrastructure Services
[1593169877849]
100 - 135 Innovation Drive
Winnipeg, MB, R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
athomp...@merlin.mb.ca
www.merlin.mb.ca


From: NANOG  on behalf of Saku 
Ytti 
Sent: June 8, 2021 01:06
To: shahr...@cs.umass.edu 
Cc: nanog list ; Arun Venkataramani 
Subject: Re: A survey on BGP MRAI timer values in practice

On Mon, 7 Jun 2021 at 19:32,  wrote:

> We often read that the Internet (i.e. BGP) has a long convergence delay.
> But why is it so slow? And can we (researchers) do anything about it?

Create business incentives to improve it. This is a non-technical
problem, we've long had technical tools to make it fast, there just
isn't incentive to make it fast. Customers are not asking operators
for better convergence speeds.

> Please help us out to find out by answering our short anonymous survey
> (<10 minutes).

Can you tell me what have you done so far? What are the default MRAI
values for each AFI/SAFI for IOS, IOS-XR, Junos, SROS, VRP and EOS?
Then people responding don't have to check what their NOS does, they
can refer to your table and tell the default value, since this is what
>99% will be using.

Now your survey has built-in selection bias, people who answer it are
people who know what it is and who are concerned about it and have
changed it, this is not a representative group and you will start your
work with very bad data.



--
  ++ytti

Spoofer Report for NANOG for May 2021

[CAIDA Spoofer Project]

In response to feedback from operational security communities,
CAIDA's source address validation measurement project
(https://spoofer.caida.org) is automatically generating monthly
reports of ASes originating prefixes in BGP for systems from which
we received packets with a spoofed source address.
We are publishing these reports to network and security operations
lists in order to ensure this information reaches operational
contacts in these ASes.

This report summarises tests conducted within usa, can.

Inferred improvements during May 2021:
ASNName   Fixed-By
5078   ONENET-AS-12021-05-19
55016  IMPER-AS-1 2021-05-25

Further information for the inferred remediation is available at:
https://spoofer.caida.org/remedy.php

Source Address Validation issues inferred during May 2021:
ASNName   First-Spoofed Last-Spoofed
3356   LEVEL32016-03-06   2021-05-05
209CENTURYLINK-US-LEGACY-QWEST   2016-08-16   2021-05-25
20412  CLARITY-TELECOM   2016-09-30   2021-05-31
6181   FUSE-NET  2016-10-10   2021-05-29
11427  TWC-11427-TEXAS   2016-10-21   2021-05-28
174COGENT-1742016-10-21   2021-05-11
30036  MEDIACOM-ENTERPRISE-BUSINESS  2016-11-16   2021-05-25
22898  ATLINK2016-12-16   2021-05-28
63296  AWBROADBAND   2017-09-01   2021-05-27
546PARSONS-PGS-1 2017-11-20   2021-05-20
393564 SPOKANE   2018-06-05   2021-05-18
33452  RW2018-09-19   2021-05-26
20448  VPNTRANET-LLC 2018-09-20   2021-05-28
5078   ONENET-AS-1   2020-04-06   2021-05-24
6391   URBAN-15  2020-05-29   2021-05-20
208188 PUGET-SOUND-NETWORKS  2020-12-04   2021-05-30
62684  XLBROADBAND   2021-05-19   2021-05-19
13876  FIBER-64  2021-05-20   2021-05-27
26397  OPTIPUTER 2021-05-24   2021-05-24
3754   NYSERNET3 2021-05-24   2021-05-24
3756   NYSERNET3 2021-05-24   2021-05-26
20150  ANYNODE   2021-05-29   2021-05-29

Further information for these tests where we received spoofed
packets is available at:
https://spoofer.caida.org/recent_tests.php?country_include=usa,can_block=1

Please send any feedback or suggestions to spoofer-i...@caida.org

Re: Fastly CDN down globally?

[Robert Webb]

Fix has already been applied. Might take some time for things to get back
to normal since this was a global outage.

On Tue, Jun 8, 2021, 8:24 AM Tony McCrory  wrote:

> Affected for example:-
>
> theguardian.co.uk
> nytimes.com
> fastly.com
>
> All return 503 Service Unavailable from Varnish Cache
>
> Tony
>

Re: Fastly CDN Down?

[Karl Auer]

On Tue, 2021-06-08 at 18:11 +0800, Jason Cooper via NANOG wrote:
> I just got an email from my CI alert that says registry.terraform.io
> is experiencing HTTP 503.
> And not just terraform, repo.maven.apache.org/maven2 also seems down.
> Even with their own website: fastly.com
> Is there anyone at Fastly?

According to fastly, the problem has been resolved. See 
--https://status.fastly.com/incidents/vpk0ssybt3bj

Regards, K.

-- 
~~~
Karl Auer (ka...@biplane.com.au)
http://www.biplane.com.au/kauer

Fastly CDN down globally?

[Tony McCrory]

Affected for example:-

theguardian.co.uk
nytimes.com
fastly.com

All return 503 Service Unavailable from Varnish Cache

Tony

Fastly CDN Down?

[Jason Cooper via NANOG]

I just got an email from my CI alert that says registry.terraform.io is
experiencing HTTP 503.
And not just terraform, repo.maven.apache.org/maven2 also seems down.
Even with their own website: fastly.com

Is there anyone at Fastly?

--
Jason

Re: DANE of SMTP Survey

[Mark Tinka]

On 6/3/21 23:41, babydr DBA James W. Laferriere wrote:



The Signing of the 'Zone' ,  Can the 'Zone' be signed by a 
self-signed key ?  Or MUST I (and others) rely on a external 
certificate authority ?


Mind you I notice in rfc6487 (note(s)) about self-signed 
certificates .
So Maybe I am being a bit over worried about having to spend more 
money just to keep my 2 ip-ranges routing in light of the RPKI 
initative(s) .


Which Mr. Andrews response below answers quite succinctly ,


Indeed! Thanks, Mark.

Yeah, it's never been obvious or apparent to me that self-signed keys 
for DNSSEC would not be honoured.


My personal zone, as well as my company's one, are both self-signed. 
They've both been working reasonably well, so far.


Mark.

Re: A survey on BGP MRAI timer values in practice

[Saku Ytti]

On Mon, 7 Jun 2021 at 19:32,  wrote:

> We often read that the Internet (i.e. BGP) has a long convergence delay.
> But why is it so slow? And can we (researchers) do anything about it?

Create business incentives to improve it. This is a non-technical
problem, we've long had technical tools to make it fast, there just
isn't incentive to make it fast. Customers are not asking operators
for better convergence speeds.

> Please help us out to find out by answering our short anonymous survey
> (<10 minutes).

Can you tell me what have you done so far? What are the default MRAI
values for each AFI/SAFI for IOS, IOS-XR, Junos, SROS, VRP and EOS?
Then people responding don't have to check what their NOS does, they
can refer to your table and tell the default value, since this is what
>99% will be using.

Now your survey has built-in selection bias, people who answer it are
people who know what it is and who are concerned about it and have
changed it, this is not a representative group and you will start your
work with very bad data.



-- 
  ++ytti