Re: plea for comcast/sprint handoff debug help

[Randy Bush]

> If there is a covering less specific ROA issued by a parent, this will
> then result in RPKI invalid routes.

i.e. the upstream kills the customer.  not a wise business model.

> The fall-back may help in cases where there is an accidental outage of
> the RRDP server (for as long as the rsync servers can deal with the
> load)

folk try different software, try different configurations, realize that
having their CA gooey exposed because they wanted to serve rrdp and
block, ...

randy, finding the fort rp to be pretty solid!

Re: Incorrect GeoIP filtering of 185.83.72.0/22

[TJ Trout]

http://thebrotherswisp.com/index.php/geo-and-vpn/

If you find anything not on our list let me know

On Fri, Oct 30, 2020, 6:38 PM Adam Pavlidis  wrote:

> Hello,
>
> We are reaching out to NANOG since the following issue is mostly observed
> in US-based service providers.
>
> We are advertising the prefix *185.83.72.0/22 *,
> that seems to be blocked by various popular US-based services, thus our
> customers originating from this prefix have trouble reaching such services.
> Indicatively:
>
> Atlassian (AMAZON), Adobe (AKAMAI) .
>
> To the best of our understanding, the blocking is enforced due to US / EU
> sanctions against Iran. The prefix was purchased by us - Lamda Hellix SA (
> *AS56910* based in *GREECE, EUROPE* ) - approximately 8 months ago.
>
> We followed the necessary process as instructed by RIPE for changing the
> ownership of the prefix.
>
> Therefore, we kindly ask those of you that use/maintain/operate GeoIP
> databases to update your records.
>
> Most importantly, we would be grateful if you could share with us (
> n...@lamdahellix.com) which Geo databases are mainly used in the US for
> this purpose.
>
> Kind regards,
>
> Adam Pavlidis
>

Re: Incorrect GeoIP filtering of 185.83.72.0/22

[Brian Ellwood]

Adam,

ip2location.com has that IP block listed as "(DCH) Data Center/Web 
Hosting/Transit” which we’ve seen cause issues for residential users in the 
past, most notably on Cogent IP space.

We worked with supp...@ip2location.com to have the address block re-analyzed 
and updated to "(ISP) Fixed Line ISP” which resolved many of the issues users 
had accessing various streaming CDN services.

HTH

> On Oct 30, 2020, at 08:27, Adam Pavlidis  wrote:
> 
> Hello,
> 
> We are reaching out to NANOG since the following issue is mostly observed in 
> US-based service providers.
> 
> We are advertising the prefix 185.83.72.0/22, that seems to be blocked by 
> various popular US-based services, thus our customers originating from this 
> prefix have trouble reaching such services. Indicatively:
> 
> Atlassian (AMAZON), Adobe (AKAMAI) .
> 
> To the best of our understanding, the blocking is enforced due to US / EU 
> sanctions against Iran. The prefix was purchased by us - Lamda Hellix SA ( 
> AS56910 based in GREECE, EUROPE ) - approximately 8 months ago.
> 
> We followed the necessary process as instructed by RIPE for changing the 
> ownership of the prefix.
> 
> Therefore, we kindly ask those of you that use/maintain/operate GeoIP 
> databases to update your records.
> 
> Most importantly, we would be grateful if you could share with us 
> (n...@lamdahellix.com) which Geo databases are mainly used in the US for this 
> purpose.
> 
> Kind regards,
> 
> Adam Pavlidis
> 

Re: Newbie Questions: How-to remove spurious IRR records (and keep them out for good)?

[Rubens Kuhl]

YMMV, but my take:
1 - You should worry a little, but not much. Filters allowing unwanted
announcements might be created using these erroneous IRR records, but
they won't do any damage by themselves. An actual wrong BGP
announcement is required for any damage to happen, and even without
those IRR records, a wrong announcement will cause some havoc since
not everyone builds filters based on IRR and not everyone runs RPKI
validation.
2 - Most IRR databases will take reports from the RIR-registered
contact of the block seriously. Some databases will react faster than
others; for instance, in TC any such objects will be removed upon
knowledge and if the maintainer recreates those objects, the
maintainer may be permanently excluded from the database.
3 - Unfortunately there is not much you can do since this is caused by
relaxed submission filtering at IRR databases, The RIR-connected IRR
databases are usually very good in preventing such, but the
independent ones usually are not. IRRd versions prior to v4 (thanks
NTT for v4) are also more prone to accept non-compliant records and
can only eliminate them after inclusion.


Rubens



On Fri, Oct 30, 2020 at 10:11 PM Pirawat WATANAPONGSE  wrote:
>
>
> Dear Guru(s),
>
>
> I am seeking advice concerning someone else announcing IRR records on 
> resources belonging to me.
> [I was referred to this mailing list from the DNS-OARC community.]
>
> Context:
> I have already registered all my IP address blocks with ROA/RPKI
> [evidence: 
> https://stat.ripe.net/widget/as-routing-consistency#w.resource=AS9411]
> However, HE reports a lot of spurious IRR records on my resources
> [example: https://bgp.he.net/net/158.108.0.0/16#_irr]
>
> Question #1:
> Should I worry about those spurious records [Yes | No | Depends]?
> My fear is that other sites might accept those records without checking and 
> thus be misled somewhere else, since ROV is not yet the behavior-in-majority.
> My other reasoning is that, if we are not going to keep accurate records, why 
> bother keeping them at all anyway.
>
> Question #2:
> What can I do about it [in case the answer to Question #1 is Yes]?
> Should I notify those Database Admins? Will they consider me a nuisance?
> And most importantly: Will they erase those records for me, or will they just 
> ignore me?
>
> Question #3:
> Did I not do something that can prevent those spurious records from happening 
> in the first place?
> And, anything I can do now to prevent it from ever happening again?
>
>
> Thanks in advance for your advice(s),
>
> Pirawat.
>

Re: dark fiber connection between 111 E 8th and Coresite NYC1 or NYC2

[Mehmet Akcin]

You can visit https://live.infrapedia.com (no login required anymore..) and
see the providers

On Fri, Oct 30, 2020 at 18:15 Eric Germann  wrote:

> Looking for a recommendation of a provider who can give us a dark fiber
> cross connect or an L2 connection between the two in the subject for an AWS
> Direct Connect out of Coresite
>
> Thanks
>
> Eric
>
> --
Mehmet
+1-424-298-1903

Re: dark fiber connection between 111 E 8th and Coresite NYC1 or NYC2

[Robert DeVita]

You can probably order an extended XC from Digital realty between the two sites.

Robert DeVita
Founder & CEO
Mejeticks
c. 469-441-8864
e. radev...@mejeticks.com

From: NANOG  on behalf of Eric 
Germann 
Sent: Tuesday, October 27, 2020 3:30:00 PM
To: nanog@nanog.org 
Subject: dark fiber connection between 111 E 8th and Coresite NYC1 or NYC2

Looking for a recommendation of a provider who can give us a dark fiber cross 
connect or an L2 connection between the two in the subject for an AWS Direct 
Connect out of Coresite

Thanks

Eric

RPKI over RSYNC vs RRDP (Was: plea for comcast/sprint handoff debug help)

[Job Snijders]

On Fri, Oct 30, 2020 at 12:47:44PM +0100, Alex Band wrote:
> > On 30 Oct 2020, at 01:10, Randy Bush  wrote:
> > i'll see your blog post and raise you a peer reviewed academic paper
> > and two rfcs :)
> 
> For the readers wondering what is going on here: there is a reason
> there is only a vague mention to two RFCs instead of the specific
> paragraph where it says that Relying Party software must fall back to
> rsync immediately if RRDP is temporarily unavailable. That is because
> this section doesn’t exist.

*skeptical face* Alex, you got it backwards: the section that does not
exist, is to *not* fall back to rsync. But on the other hand, there are
ample RFC sections which outline rsync is the mandatory-to-implement
protocol. Starts at RFC 6481 Section 3: "The publication repository
MUST be available using rsync".

Even the RRDP RFC itself (RFC 8182) describes that RSYNC and RRDP
*co-exist*. I think this co-existence was factored into both the design
of RPKIoverRSYNC and subsequently RPKIoverRRDP. An rsync publication
point does not become invalid because of the demise of an
once-upon-a-time valid RRDP publication point.

Only a few weeks ago a large NIR (IDNIC) disabled their RRDP service
because somehow the RSYNC and RRDP repositories were out-of-sync with
each other. The RRDP service remained disabled for a number of days
until they repaired their RPKI Certificate Authority service.

I suppose that during this time, Routinator was unable to receive any
updates related to the IDNIC CA (pinned to RRDP -> because of a prior
successful fetch prior to the partial IDNIC RPKI outage). This in turn
deprived the IDNIC subordinate Resource Holders the ability to update
their Route Origin Authorization attestations (from Routinator's
perspective).

Given that RRDP is an *optional* protocol in the RPKI stack, it doesn't
make sense to me to strictly pin fetching operations to RRDP: Over time
(months, years), a CA could enable / disable / enable / disable RRDP
service, while listing the RRDP URI as a valid SIA, amongst other valid
SIAs.

An analogy to DNS: A website operator may add  records to indicate
IPv6 reachability, but over time may also remove the  record if
there (temporarily) is some kind of issue with the IPv6 service. The
Internet operations community of course encourages everyone to add 
records, and IPv6 Happy Eyeballs were a concept to for a long time even
*favor* IPv6 over IPv4 to help improve IPv6 adoption, but a dual-stack
browser will always try to make benefit of the redundancy that exists
through the two address families.

RSYNC and RRDP should be viewed in a similar context as v4 vs v6, but
unlike with IPv4 and IPv6, I am convinced that RSYNC can be deprecated
in the span of 3 or 4 years, the draft-sidrops-bruijnzeels-deprecate-rsync
document is helping towards that goal! 

> Be that as it may, operators can rest assured that if consensus goes
> against our logic, we will change our design.

Please change the implementation a little bit (0.8.1). I think it is too
soon for the internet wide 'rsync to RRDP' migration project to be
declared complete and successfull, and this actually hampers the
transition to RRDP.

Pinning to RRDP *forever* violates the principle-of-least-astonishment
in a world where draft-sidrops-bruijnzeels-deprecate-rsync-00 was
published only as recent as November 2019. That draft now is a working
group document, and it will probably take another 1 or 2 years before it
is published as RFC.

Section 5 of 'draft-deprecate-rsync' says RRDP *SHOULD* be used when it
is available. Thus it logically follows, when it is not available, the
lowest common denominator is to be used: rsync. After all, the Issuing
CA put an RSYNC URI in the 'Subject Information Access' (SIA). Who knows
better than the CA?

The ability to publish routing intentions, and for others to honor the
intentions of the CA is what RPKI is all about. When the CA says
delegated RPKI data is available at both an RSYNC URI and an RRDP URI,
both are valid network entrypoints to the publication point. The
resource holder's X.509 signature even is on those 'reference to there'
directions (URIs)! :-)

If I can make a small suggestion: make 0.8.1 fall back to rsync after
waiting an hour or so, (meanwhile polling to see if the the RRDP service
restores). This way the network operator takes advantage of both
transport protocols, whichever is available, with a clear preference to
try RRDP first, then eventually rsync.

RPKI was designed in such a way that it can be transported even over
printed paper, usb stick, bluetooth, vinyl, rsync, and also https (as
rrdp). Because RPKI data is signed using the X.509 framework, the
transportation method really is irrelevant. IP holders can publish RPKI
data via horse + cart, and still make productive use of it!

Routinator's behavior is not RFC compliant, and has tangible effects in
the default-free zone.

Regards,

Job

Re: Apple Catalina Appears to Introduce Massive Jitter - SOLVED!

[Aaron Atac via NANOG]

Hi Mark,

I'm running a MacBook Pro (Retina, 15-inch, Mid 2015) with Mojave 10.14.6 
(latest).
I've always had location services off (including all system services within).

I haven't seen any jitter issues on my end.

Along with, no matter if I have bluetooth turned on with my wireless mouse and 
keyboard connected or not, I see no consistent change in jitter.

Even further, enabling location services (including all system services within) 
and bluetooth enabled, I'm not seeing any change in jitter.

I know you said earlier this issue seems to have started somewhere _around_ 
Mojave so just my two cents.

-Aaron

Oct 30, 2020, 12:08 by mark.ti...@seacom.com:

> Hi all.
>  
>  So I may have fixed this for my end, and hopefully others may be  able 
> to use the same fix.
>  
>  After a tip from Karl Auerbach and this link:
>  
>      > https://developer.apple.com/forums/thread/97805
>  
>  ... I was able to fix the problem by disabling Bluetooth. 
>  
>  However, disabling Bluetooth was not enough. I also had to disable  all 
> Location Services.
>  
>  After that, I re-enabled Location Services and only allowed for  two 
> features:
>  
>      - NetSpot
>      - Find My Mac
>  
>  With just those two location services, as well as Bluetooth  disabled, I 
> have no more high jitter.
>  
>  App performance like Zoom and Youtube uploads are now crisp, with  0.0% 
> packet loss.
>  
>  So looks like that Bluetooth is a huge problem. Confirmed by  opening 
> the "Console" app, and adding "scan" in the filter bar,  top right.
>  
>  A peak latency of 13.5ms after 300 packets:
>  
>  Marks-MacBook-Pro.local(172.16.0.239)
>   
>   
> 2020-10-30T21:06:05+0200
>  Keys:  Help   Display mode   Restart statistics   Order of  fields   quit
>   
>   
>    Packets   
> Pings
>   Host
>   
>      Loss%   Snt   Last   Avg 
>  Best  Wrst StDev
>   1.172.16.0.254  
>   
>      0.0%   300    3.1   4.8  
>  2.2  13.5   1.9
>  
>  Mark.
>  
>  

Anyone have a contact at Microsoft RE: Microsoft Defender SmartScreen removal?

[Matt Rauch via NANOG]

Hello,

We have a hosting customer who had a brand new domain flagged from day one.
Reported the site safe several weeks ago, and still no change. Microsoft
Support chat said it could take a couple weeks to review.

Matt Rauch

Re: 100G over 100 km of dark fiber

[Lady Benjamin PD Cannon]

Very impressive!  Can you share your fiber type and link-loss?
—L.B.

Lady Benjamin PD Cannon, ASCE
6x7 Networks & 6x7 Telecom, LLC 
CEO 
b...@6by7.net 
"The only fully end-to-end encrypted global telecommunications company in the 
world.”
FCC License KJ6FJJ



> On Oct 30, 2020, at 9:03 AM, Tarko Tikan  wrote:
> 
> hey,
> 
>> If it’s just a single 100G channel needed you could try 100GBASE-ZR4. 
>> Specified for 80km, 30db power budget they could actually reach more the 
>> 80km.
>> Dispersion should also be „no" problem in the 1310nm length. I have to say 
>> that I never tried this on 100km distance without coherent solutions.
> 
> Just to add to my original suggestion, we just did 100G-ZR4 over 30dB link 
> with pre-FEC BER 3.174E-11.
> 
> As OP is asking for a solution for 25dB I don't see any reason why ZR4 would 
> not work and why you would need coherent, amplifiers or any other additional 
> solution except when you are limited by QSFP28 SFF power.
> 
> -- 
> tarko

Incorrect GeoIP filtering of 185.83.72.0/22

[Adam Pavlidis]

Hello,

We are reaching out to NANOG since the following issue is mostly observed
in US-based service providers.

We are advertising the prefix *185.83.72.0/22 *,
that seems to be blocked by various popular US-based services, thus our
customers originating from this prefix have trouble reaching such services.
Indicatively:

Atlassian (AMAZON), Adobe (AKAMAI) .

To the best of our understanding, the blocking is enforced due to US / EU
sanctions against Iran. The prefix was purchased by us - Lamda Hellix SA (
*AS56910* based in *GREECE, EUROPE* ) - approximately 8 months ago.

We followed the necessary process as instructed by RIPE for changing the
ownership of the prefix.

Therefore, we kindly ask those of you that use/maintain/operate GeoIP
databases to update your records.

Most importantly, we would be grateful if you could share with us (
n...@lamdahellix.com) which Geo databases are mainly used in the US for this
purpose.

Kind regards,

Adam Pavlidis

Re: plea for comcast/sprint handoff debug help

[Tim Bruijnzeels]

Hi Job, all,

> On 30 Oct 2020, at 11:06, Job Snijders  wrote:
> 
> On Thu, Oct 29, 2020 at 09:14:16PM +0100, Alex Band wrote:
>> In fact, we argue that it's actually a bad idea to do so:
>> 
>> https://blog.nlnetlabs.nl/why-routinator-doesnt-fall-back-to-rsync/
>> 
>> We're interested to hear views on this from both an operational and
>> security perspective.
> 
> I don't see a compelling reason to not use rsync when RRDP is
> unavailable.
> 
> Quoting from the blog post:
> 
>"While this isn’t threatening the integrity of the RPKI – all data
>is cryptographically signed making it really difficult to forge data
>– it is possible to withhold information or replay old data."
> 
> RRDP does not solve the issue of withholding data or replaying old data.
> The RRDP protocol /also/ is unauthenticated, just like rsync. The RRDP
> protocol basically is rsync wrapped in XML over HTTPS.
> 
> Withholding of information is detected through verification of RPKI
> manifests (something Routinator didn't verify up until last week!),
> and replaying of old data is addressed by checking validity dates and
> CRLs (something Routinator also didn't do until last week!).
> 
> Of course I see advantages to this industry mainly using RRDP, but those
> are not security advantages. The big migration towards RRDP can happen
> somewhere in the next few years.


Routinator does TLS verification when it encounters an RRDP repository. If the 
repository cannot be reached, or its HTTPS certificate is somehow invalid, it 
will use rsync instead. It's only after it found a *valid* HTTPS connection, 
that it refuses to fall back.

There is a security angle here.

Malicious-in-the-middle attacks can lead an RP to a bogus HTTPS server and 
force the software to downgrade to rsync, which has no channel security. The 
software can then be given old data (new ROAs can be withheld), or the attacker 
can simply withhold a single object. With the stricter publication point 
completeness validation introduced by RFC6486-bis this will lead to the 
rejecting of all ROAs published there.

The result is the exact same problem that Randy et al.'s research pointed at. 
If there is a covering less specific ROA issued by a parent, this will then 
result in RPKI invalid routes.

The fall-back may help in cases where there is an accidental outage of the RRDP 
server (for as long as the rsync servers can deal with the load), but it 
increases the attack surface for repositories that keep their RRDP server 
available.

Regards,
Tim



> 
> The arguments brought forward in the blog post don't make sense to me.
> The '150,000' number in the blog post seems a number pulled from thin
> air.
> 
> Regards,
> 
> Job

RE: [SPAM] Re: plea for comcast/sprint handoff debug help

[p.fazio]

please remove me from list


 Original Message 
Subject: [SPAM] Re: plea for comcast/sprint handoff debug help
From: Alex Band 
Date: Thu, October 29, 2020 2:14 pm
To: Randy Bush 
Cc: North American Network Operators' Group 


> On 28 Oct 2020, at 16:58, Randy Bush  wrote:
> 
>> tl;dr:
>> 
>> comcast: does your 50.242.151.5 westin router receive the announcement
>> of 147.28.0.0/20 from sprint's westin router 144.232.9.61?
> 
> tl;dr: diagnosed by comcast.  see our short paper to be presented at imc
>   tomorrow https://archive.psg.com/200927.imc-rp.pdf
> 
> lesson: route origin relying party software may cause as much damage as
> 	it ameliorates
> 
> randy

To clarify this for the readers here: there is an ongoing research experiment where connectivity to the RRDP and rsync endpoints of several RPKI publication servers is being purposely enabled and disabled for prolonged periods of time. This is perfectly fine of course.

While the resulting paper presented at IMC is certainly interesting, having relying party software fall back to rsync when RRDP is unavailable is not a requirement specified in any RFC, as the paper seems to suggest. In fact, we argue that it's actually a bad idea to do so:

https://blog.nlnetlabs.nl/why-routinator-doesnt-fall-back-to-rsync/

We're interested to hear views on this from both an operational and security perspective.

-Alex

Re: Apple Catalina Appears to Introduce Massive Jitter

[David Curado]

I was curious, so poked at this... my results from a macbook pro 2019
running Catalina 10.15.3

sudo /usr/local/sbin/mtr -r 10.200.200.200

Start: 2020-10-29T14:09:08-0400
HOST: bos-mp36c   Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 10.200.200.200 0.0%10   11.9  63.7   9.0 340.2 104.1

sudo sysctl net.link.generic.system.hwcksum_tx=0
net.link.generic.system.hwcksum_tx: 1 -> 0
sudo sysctl net.link.generic.system.hwcksum_rx=0
net.link.generic.system.hwcksum_rx: 1 -> 0
sudo ifconfig en0 -rxcsum

sudo /usr/local/sbin/mtr -r 10.200.200.200

Start: 2020-10-29T14:09:43-0400
HOST: bos-mp36c   Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 10.200.200.200 0.0%10   19.8  13.6   9.8  20.9   4.5

So, seems to make things better. =-)



On Thu, Oct 29, 2020 at 11:50 AM Mark Tinka  wrote:

>
>
> On 10/29/20 16:08, J. Hellenthal wrote:
>
> > I believe I have seen the same thing with a Mid 2015 11,4 running
> catalina. Not diagnosing further because I could not find a reason for it
> fast enough and not sure if it really had an impact at the moment…. but
> could you try the following
> >
> >
> > sudo sysctl net.link.generic.system.hwcksum_tx=0
> > sudo sysctl net.link.generic.system.hwcksum_rx=0
> > sudo ifconfig en0 -rxcsum
>
> Thanks, I'll have a sniff.
>
> > If you have some specific tests to run I would be willing to run them
> here on Big Sur with the same laptop but I have nothing now that runs
> Catalina
>
> One of my mates found the same issue on Big Sur (beta) on a 2013 MacBook
> Pro.
>
> Just a simple mtr test to your local home router's IP address should,
> over wi-fi, should show you the jitter.
>
> Mark.
>

RE: [SPAM] Re: Apple Catalina Appears to Introduce Massive Jitter

[p.fazio]

PLEASE REMOVE ME FROM THE LISTTHANK YOU


 Original Message 
Subject: [SPAM] Re: Apple Catalina Appears to Introduce Massive Jitter
From: colin johnston 
Date: Thu, October 29, 2020 11:12 am
To: Mark Tinka 
Cc: NANOG 

Hey Mark,Good shout with debug, same issue seen on MacBook Air with Catalina 10.15.6 beta, pings upto 150ms seeniMac with Sierra zero jitter and usually sub 1m pingsNow need to find out why, I never noticed as wife using the MacBook Air :(I cant yet update to big sur since need lots of sad space, need to cutdown on university docs me thinksColOn 29 Oct 2020, at 12:07, Mark Tinka  wrote:   Hi all.  I've been on High Sierra for several years now due to a limitation with an app that couldn't deal with Apple's latest rounds of system permissions since Mojave. Eventually, I gave up on waiting for them to fix it and upgraded my older Butterfly keyboard laptop to Catalina 4 weeks ago.  At the same time, I picked up the new Magic keyboard laptop 2 weeks ago which came with Catalina.  Over the past week, I've been troubleshooting a massive jitter issue on Catalina, just between itself and my home router. For control, I have a Windows PC (tower-top) using a wireless adapter to connect to my home network. That has no jitter at all.  I have noticed as much as 300ms+ jitter on Catalina.  I then asked a few friends around the world to run tests for me on their own Catalina installations to their local router over wi-fi, and the results are the same. Jitter so high that what should be a 1ms - 5ms latency can (for a short period) jump to 200ms+, 300ms+, 400ms+.  On the off-chance that it is an issue with the new wireless chips on the later MacBook models, one of my friends tested the same on a 2013 MacBook Pro running a beta version of Big Sur. Same story!  Another friend in South East Asia, testing on a 2018 13-inch MacBook Pro running Catalina, also had the same issue.  A Google search suggests that this is some known issue since Mojave, to do with Location Services, and some other apps, in a non-deterministic way:      https://apple.stackexchange.com/questions/263638/macbook-pro-experiencing-ping-spikes-to-local-router  For me, even after disabling all or some Location Services features, the problem remains.  Is anyone else seeing this on their Catalina Mac's while on wi-fi? If so, does anyone know what's going on here?  Ideally, this wouldn't matter if it was just a cosmetic issue - but I do actually see physical impact to performance of network access to/from the laptop, which has all the hallmarks of high jitter and/or packet loss.   An app like Zoom, which can display network performance data for a session in real-time, does indicate nominal packet loss for audio and video on this device, while other devices on the same WLAN are happy.  Thoughts?  Mark.   

Re: DE-CIX - Wednesday 11:00am - 1:30pm Eastern

[Ed dAgostino]

All,


For clarification, DE-CIX New York operates over a dozen switches as part of 
our local switch fabric.   ONE of our switches malfunctioned for about a two 
hour period  prior to rebooting and that caused problems for customer networks 
connected to that switch during that period.   All other switches were not 
impacted and the DE-CIX New York exchange did not go down.


Ed d'Agostino


From: NANOG  on behalf of Matt 
Hoppes 
Sent: Thursday, October 29, 2020 11:29:03 AM
To: North American Network Operators' Group; outages-discuss...@outages.org
Subject: DE-CIX - Wednesday 11:00am - 1:30pm Eastern

Does anyone here know any more about what crashed in the DE-CIX NYC
exchange yesterday between approximately 11am and 1:30pm?

Between those times I'm told the Nokia switching fabric locked up and
ultimately rebooted.

Re: Apple Catalina Appears to Introduce Massive Jitter

[Cory Sell via NANOG]

Might be worth disabling each AP to see if there's one out there having an 
issue playing nice with the MacBook. Also try different combinations of two APs 
working together. It's possible the MacBook is flip flopping because the power 
levels are fighting each other.

Does the Mac have this issue at your local coffee shop or another establishment 
with Wi-Fi? You can try to rule out the AirPort card in the Mac itself.

Sent from ProtonMail mobile

 Original Message 
On Oct 29, 2020, 7:55 AM, Mark Tinka wrote:

> On 10/29/20 14:40, Jared Mauch wrote:
>
>> I know there was a recent fix Apple did for devices talking to UBNT APs
>> for their handsets, perhaps there's a similar fix needed on your side?
>>
>> I have all UBNT at home for wireless and periodically have some random
>> issues which I can't explain, but for the most part have things tuned to 
>> ensure
>> there's little to no interference.
>
> I am running TP-Link AP's. Two of them are Google OnHub, which was built
> by TP-Link, and the 3rd one is the TP-Link Archer C6.
>
> So they all support 802.11ac, which is where my device spends most of
> its time (5GHz).
>
> No interference from my neighbors (separated by thick walls), and I am
> running separate channels for both frequencies per device.
>
> Also, no other wireless device is suffering like this.
>
>> Do you see the same when hardwired? I keep many of my devices hardwired
>> to avoid odd jitter issues.
>
> No issues on the wire at all. Quite perfect.
>
> Like you, I hard-wire all fixed devices (TV's, a/v receivers, satellite
> decoders, gaming consoles, energy meters, e.t.c.). The only devices on
> wireless are mobile devices, tablets, laptops and the Windows PC which
> is hooked up via wi-fi as well.
>
>> I also saw some older versions of the Pulse Secure
>> VPN add the behavior you describe, including the more uptime the slower it 
>> would
>> get.
>
> I use Viscosity as an SSL/VPN client. The issue is the same whether it
> is enabled or offline.
>
> Mark.

dark fiber connection between 111 E 8th and Coresite NYC1 or NYC2

[Eric Germann]

Looking for a recommendation of a provider who can give us a dark fiber cross 
connect or an L2 connection between the two in the subject for an AWS Direct 
Connect out of Coresite

Thanks

Eric

Newbie Questions: How-to remove spurious IRR records (and keep them out for good)?

[Pirawat WATANAPONGSE]

Dear Guru(s),


I am seeking advice concerning someone else announcing IRR records on
resources belonging to me.
[I was referred to this mailing list from the DNS-OARC community.]

Context:
I have already registered all my IP address blocks with ROA/RPKI
[evidence:
https://stat.ripe.net/widget/as-routing-consistency#w.resource=AS9411]
However, HE reports a lot of spurious IRR records on my resources
[example: https://bgp.he.net/net/158.108.0.0/16#_irr]

Question #1:
Should I worry about those spurious records [Yes | No | Depends]?
My fear is that other sites might accept those records without checking and
thus be misled somewhere else, since ROV is not yet the
behavior-in-majority.
My other reasoning is that, if we are not going to keep accurate records,
why bother keeping them at all anyway.

Question #2:
What can I do about it [in case the answer to Question #1 is Yes]?
Should I notify those Database Admins? Will they consider me a nuisance?
And most importantly: Will they erase those records for me, or will they
just ignore me?

Question #3:
Did I not do something that can prevent those spurious records from
happening in the first place?
And, anything I can do now to prevent it from ever happening again?


Thanks in advance for your advice(s),

Pirawat.

Re: Disney+ geolocation error for 213.134.224.0/19

[Johan Hedberg]

I had a similar issue here in Sweden. The contact point listed at 
http://thebrotherswisp.com/index.php/geo-and-vpn/ 
 
(netad...@disneystreaming.com ) helped me 
with this pretty quickly.

— 
Johan Hedberg


> On 25 Oct 2020, at 11:48, Sander Steffann  wrote:
> 
> Hi,
> 
> Anybody around from Disney+?  my main customer (Solcon) is an ISP in the 
> Netherlands. One of our ranges is 213.134.224.0/19 and it seems to be 
> classified as non-Netherlands. The official support channel doesn't get any 
> further than "you must be using a VPN" even though we are the ISP and it's 
> our own address space...
> 
> Any assistance would be much appreciated!
> 
> Cheers,
> Sander
> 

outlook inbound email issues?

[nanog]

I have a client who is receiving 50% of their mail on outlook servers 
for the past few hours.


MX records point to:
x-com.mail.protection.outlook.com has address 104.47.38.36
x-com.mail.protection.outlook.com has address 104.47.36.36

Anyone aware of outlook issues or have someone they can poke to check 
into this?  (Both answer smtp requests, guessing a queue is backed up 
somewhere.)


thanks
bill

--
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Configuring of MACsec for three EX4300 Switches

[switch999--- via NANOG]

Hi, 
 
following only the required configuration of 
https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/macsec-configuring-mx-series.html
 
for 
# Configuring MACsec Using Static Connectivity Association Key (CAK) Mode 
 
works fine for two switches, but with a third EX4300 in the middle not. 
 
Thus, could anyone please help what is required to ensure connectivity through 
three EX4300? 
 
Even the configuration (A; with several tries) on the outer sides switches such 
as 
e.g. given for (one port) per switch 
jack@cs2# set security macsec connectivity-association ca1 mka eapol-address 
provider-bridge  
jack@cs2# set security macsec connectivity-association ca1 mka eapol-address 
lldp-multicast  
jack@cs2# set protocols layer2-control mac-rewrite interface ge-0/0/13 protocol 
ieee8021 
worked not for the three EX4300. 
 
Tunneling through a EX4200, in the middle (via vlan, snippet see below) worked 
fine, even without the  
configuration (A) at the outer sides switches, only with the most important 
commands 
given in 
https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/macsec-configuring-mx-series.html.
 
 
Any idea why tunneling through the middle EX4300 failed? (Used version: 
17.3R3-S9.3!) 
 
Regards, 
Jack 
 
 
# PS: What is the equivalent code for EX4300 from the EX4200 code 
    vlan-id 55;  
    dot1q-tunneling { 
    layer2-protocol-tunneling { 
    all; 
    }   
 

RE: Linux router network cards

[Toke Høiland-Jørgensen via NANOG]

micah anderson  writes:

> Thanks for the reply.
>
> Philip Loenneker  writes:
>> Take a look at the Mellanox ConnectX 5 series of cards. They handle
>> DPDK, PVRDMA (basically SR-IOV that allows live migration between
>> hosts), and can even process packets within the NIC for some
>
> From what I can tell, SR-IOV/PVRDMA aren't really useful for me in
> building a router that wont be doing any virtualization.
>
> If the card can do DPDK, can it do XDP?

The Connect-X 5 has excellent XDP support - it's what we used for the
XDP paper:

https://github.com/xdp-project/xdp-paper#hardware

-Toke

AS33132 / Crown Castle pulling a Lumen

[Adam Korab]

If anybody that can help with CCF continuing to announce a prefix four hours 
after the originating session was shutdown, please contact me unicast. 

A ticket has been opened with their NOC but progress is not forthcoming. 

Thanks,

Adam

Re: APOLOGIES: QB server hiccups

[Jim Popovitch via NANOG]

On Thu, 2020-10-22 at 18:04 +, Paul Nash wrote:
> Autocorrect changed a misspelled recipient to “nanog”.
> 

Not quite.  What happened was sometime in the past Brian sent an email
to NANOG from a domain publishing a DMARC record.  Mailman on nanog@
wraps such an email and (like it or not) sets the From: as "Brian 
via NANOG " and then various MUAs decide to save that,
and the next time someone starts typing "Bry..." into their To: field it
pulls up "Brian ... via ,,," (which nobody every pays close enough
attention to).  The blame can be spread around wide, but it is what it
is and everyone should be aware of it.

-Jim P. ("Who the hell would code something like that?!?!")

Re: A study on community-triggered updates in BGP

[Thomas Krenc]

Hi Jeff,

We have tested FRR (v6.0.2) indeed and found that duplicates are not
suppressed by default.

We will publish more detailed results and configurations on the website
soon.

Thomas

On 10/21/20 4:35 PM, Jeff Tantsura wrote:
>
> Hi Thomas,
>
> We had a similar discussion on FRR slack, there are some duplicates
> indeed.
> Are you planing to test FRR at some point in time?
>
> Cheers,
> Jeff
> On Oct 21, 2020, 3:58 PM -0700, Jakob Heitz (jheitz) via NANOG
> , wrote:
>> Thomas,
>>
>> I confirmed your case and took a look at the code.
>> The outbound duplicate suppression function tries to detect
>> duplicates without actually storing or recreating the
>> previously sent update, so it misses some cases.
>>
>> Your use case is a good one. We will check to see if we can
>> detect it without compromising significantly on resource usage.
>> Thank you for raising the issue.
>>
>> Regards,
>> Jakob.
>>
>> -Original Message-
>> Date: Tue, 20 Oct 2020 04:48:37 -0700
>> From: Thomas Krenc 
>>
>> Hi Jakob.
>>
>> The simple configuration below allows communities to be forwarded
>> (send-community-ebgp), but are cleaned at egress (using route-policy and
>> community-set).
>>
>> In the experiment, the router receives announcements with altering
>> community attributes only, from the internal peer. After the filter is
>> applied, the router sends duplicates to the external peer.
>>
>> Also, In a slightly different setup, the router sends duplicates due to
>> changes in the next-hop only.
>>
>> best regards
>> Thomas
>>
>> ---
>>
>> RP/0/0/CPU0:ios(config)#show running-config
>> Tue Oct 20 02:56:24.230 UTC
>> Building configuration...
>> !! IOS XR Configuration 6.0.1
>> !! Last configuration change at Tue Oct 20 02:56:02 2020 by cisco
>> !
>> interface MgmtEth0/0/CPU0/0
>> ?shutdown
>> !
>> interface GigabitEthernet0/0/0/0
>> ?ipv4 address 10.12.0.2 255.255.255.252
>> !
>> interface GigabitEthernet0/0/0/1
>> ?ipv4 address 10.20.0.1 255.255.255.252
>> !
>> community-set all
>> ? *:*
>> end-set
>> !
>> route-policy nofilter
>> ? pass
>> end-policy
>> !
>> route-policy egressfilter
>> ? delete community in all
>> ? pass
>> end-policy
>> !
>> router bgp 65002
>> ?bgp router-id 10.12.0.2
>> ?address-family ipv4 unicast
>> !
>> ?neighbor 10.12.0.1
>> ? remote-as 65001
>> ? address-family ipv4 unicast
>> ?? send-community-ebgp
>> ?? route-policy egressfilter out
>> !
>> ?neighbor 10.20.0.2
>> ? remote-as 65002
>> ? address-family ipv4 unicast
>> !
>> end
>>
>> On 10/17/20 3:59 PM, Jakob Heitz (jheitz) via NANOG wrote:
>>> IOS-XR has duplicate update suppression logic for EBGP sessions,
>>> not for IBGP sessions.
>>>
>>> If you are using EBGP and seeing a fault in the duplicate update
>>> suppression logic in IOS-XR, please let me know configs and details
>>> of the experiment.
>>>
>>> Regards,
>>> Jakob.
>>>
>>> -Original Message-
>>> Date: Thu, 15 Oct 2020 18:35:58 -0700
>>> From: Thomas Krenc 
>>>
>>> Dear NANOG,
>>>
>>> As a team of researchers from NPS and TU Berlin, we are investigating
>>> the impact of BGP community attributes on the update behavior
>>> between ASes.
>>>
>>> We find that when a route is associated with multiple distinct community
>>> attributes it does not only lead to multiple announcement at the tagging
>>> AS, but also at neighboring ASes, if communities are not filtered
>>> properly. This behavior is wide-spread.
>>>
>>> In order to better understand our observations, we have performed a
>>> series of laboratory experiments using Cisco IOS, Junos OS, as well as
>>> the BIRD daemon.
>>>
>>> We find that - by default - all tested routers generate announcements
>>> with changing community attributes, even when other attributes do not
>>> change. In addition, when communities are filtered at egress, Cisco und
>>> BIRD send duplicate announcements (Juniper does not).
>>>
>>> Since our findings are limited to observations in public data as well as
>>> few router implementations, we would like to share our research and
>>> kindly ask you to have a look at:
>>>
>>> ??? https://www.cmand.org/communityexploration/
>>>
>>> There, we provide some resources documenting our research, as well as
>>> open questions. We greatly appreciate any feedback and insights you can
>>> offer. Also, please don't hesitate to contact us directly:
>>>
>>> ??? communityexploration AT cmand DOT org
>>>
>>> best regards
>>>
>>> Thomas Krenc
>>> Postdoctoral Researcher
>>> Naval Postgraduate School

Re: Apple Catalina Appears to Introduce Massive Jitter - SOLVED!

[J. Hellenthal via NANOG]

Heya ! Doug!

Yeah I wouldn’t put this on BT either. On the other hand it seems that whether 
the scheduler is newreno or cubic that this situation persists pasts my 
previous suggestions. Seems tho that when you put strain on an upload that the 
jitter gets considerably worse... 90m out of a 100m link it starts to get to 
1000+ms

-- 
 J. Hellenthal

The fact that there's a highway to Hell but only a stairway to Heaven says a 
lot about anticipated traffic volume.

> On Oct 30, 2020, at 16:59, Doug Barton  wrote:
> I would hesitate to blame BT. I have a macbook pro from ~1 year ago, on 
> Catalina, and I use BT extensively ... mouse, keyboard, and headset. I do 
> have location services trimmed down to just find my mac.
> 
> I ran: ping -c 1000 -i 0.1 
> 
> 1000 packets transmitted, 998 packets received, 0.2% packet loss
> round-trip min/avg/max/stddev = 1.255/2.378/9.095/0.634 ms
> 
> One thing that may contribute to blaming BT however is if you are using wifi 
> on 2.4G only, and/or preferring it, as BT operates in the same frequency 
> range neighborhood. My macbook is connected using 5G.
> 
> Happy to compare other settings if there is interest.
> 
> Doug
> 
> 
>> On 10/30/20 12:08 PM, Mark Tinka wrote:
>> Hi all.
>> So I may have fixed this for my end, and hopefully others may be able to use 
>> the same fix.
>> After a tip from Karl Auerbach and this link:
>> https://developer.apple.com/forums/thread/97805
>> ... I was able to fix the problem by disabling Bluetooth.
>> However, disabling Bluetooth was not enough. I also had to disable all 
>> Location Services.
>> After that, I re-enabled Location Services and only allowed for two features:
>> - NetSpot
>> - Find My Mac
>> With just those two location services, as well as Bluetooth disabled, I have 
>> no more high jitter.
>> App performance like Zoom and Youtube uploads are now crisp, with 0.0% 
>> packet loss.
>> So looks like that Bluetooth is a huge problem. Confirmed by opening the 
>> "Console" app, and adding "scan" in the filter bar, top right.
>> A peak latency of 13.5ms after 300 packets:
>> Marks-MacBook-Pro.local (172.16.0.239) 2020-10-30T21:06:05+0200
>> Keys:  Help   Display mode   Restart statistics   Order of fields   quit
>> Packets   Pings
>>  Host Loss%   Snt   Last   Avg  Best  Wrst StDev
>>  1. 172.16.0.254 0.0%   3003.1   4.8   2.2  13.5   1.9
>> Mark.


smime.p7s
Description: S/MIME cryptographic signature

Re: Apple Catalina Appears to Introduce Massive Jitter - SOLVED!

[Doug Barton]

I would hesitate to blame BT. I have a macbook pro from ~1 year ago, on 
Catalina, and I use BT extensively ... mouse, keyboard, and headset. I 
do have location services trimmed down to just find my mac.


I ran: ping -c 1000 -i 0.1 

1000 packets transmitted, 998 packets received, 0.2% packet loss
round-trip min/avg/max/stddev = 1.255/2.378/9.095/0.634 ms

One thing that may contribute to blaming BT however is if you are using 
wifi on 2.4G only, and/or preferring it, as BT operates in the same 
frequency range neighborhood. My macbook is connected using 5G.


Happy to compare other settings if there is interest.

Doug


On 10/30/20 12:08 PM, Mark Tinka wrote:

Hi all.

So I may have fixed this for my end, and hopefully others may be able to 
use the same fix.


After a tip from Karl Auerbach and this link:

https://developer.apple.com/forums/thread/97805

... I was able to fix the problem by disabling Bluetooth.

However, disabling Bluetooth was not enough. I also had to disable all 
Location Services.


After that, I re-enabled Location Services and only allowed for two 
features:


     - NetSpot
     - Find My Mac

With just those two location services, as well as Bluetooth disabled, I 
have no more high jitter.


App performance like Zoom and Youtube uploads are now crisp, with 0.0% 
packet loss.


So looks like that Bluetooth is a huge problem. Confirmed by opening the 
"Console" app, and adding "scan" in the filter bar, top right.


A peak latency of 13.5ms after 300 packets:

Marks-MacBook-Pro.local (172.16.0.239) 2020-10-30T21:06:05+0200
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
Packets   Pings
  Host Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. 172.16.0.254 0.0%   300    3.1   4.8   2.2  13.5   1.9

Mark.

Re: 100G over 100 km of dark fiber

[Sander Steffann]

Hi,

On 30-10-2020 15:33, Dale W. Carder wrote:


You may also find that 100G PAM4 could work.  There are some vendors that
sell the optic, and an outboard EDFA + DCF pizza box.


We are about to deploy these on a couple of dark fibers:
https://www.solid-optics.com/product/edfamux-multiplexer-amplifier-dispersion-compensation-dwdm-mux-edfa/

They have amplified and dispersion compensated 8x100G to be used with 
PAM4 optics, and a pass-through port to connect existing 1G/10G MUXes to 
(which can have their own amplification if necessary).


They can provide models with different sets of channels of you need that 
(nice when cascading them with existing 1/10G MUXes). IIRC next year 
they can also build in a power meter so you can do remote monitoring.


If you're interested I can let you know how much we like them in a few 
months ;)


Cheers,
Sander

Re: urpf - evil?

[Martijn Schmidt via NANOG]

Hi Baldur,

You are at risk of facilitating spoofed and/or reflection DDoS attacks if you 
don't implement BCP38.. that's why uRPF exists. :)

Best regards,
Martijn

From: NANOG  on behalf of 
Baldur Norddahl 
Sent: 30 October 2020 20:29
To: nanog@nanog.org 
Subject: urpf - evil?

Hello

While working on my ACLs I noticed that I was successful in blocking some 
apparently spoofed IPv6 traffic. The destination was Facebook and the source 
was IPv6 range belonging to a mobile operator that sells 4G Wifi router based 
solutions.

So thinking about how and why a few customers end up sending packets to our 
network with the wrong source, I came up with a theory (not validated): What if 
the customer connects his 4G Wifi router to one of the LAN ports of our CPE (or 
visa versa)? His computer would then pick up an IPv6 range from both ISPs along 
with two default routes. But only one default route would be used, and in this 
case that was apparently the default route going to our network. But still his 
computer might use the IPv6 address from the other ISP as source and therefore 
he ends up "spoofing" by sending that to us. We deliver the packets to Facebook 
and I assume Facebook will route the replies just fine through the other ISP.

Now the thing is that my impression is that it actually works so long I do not 
actively block it with uRPF or ACLs on our edge. I have learned that spoofing 
is evil and I should be blocking this - but why am I sabotaging something that 
apparently is doing just fine at some customers?

Regards,

Baldur

urpf - evil?

[Baldur Norddahl]

Hello

While working on my ACLs I noticed that I was successful in blocking some
apparently spoofed IPv6 traffic. The destination was Facebook and the
source was IPv6 range belonging to a mobile operator that sells 4G Wifi
router based solutions.

So thinking about how and why a few customers end up sending packets to our
network with the wrong source, I came up with a theory (not validated):
What if the customer connects his 4G Wifi router to one of the LAN ports of
our CPE (or visa versa)? His computer would then pick up an IPv6 range from
both ISPs along with two default routes. But only one default route would
be used, and in this case that was apparently the default route going to
our network. But still his computer might use the IPv6 address from the
other ISP as source and therefore he ends up "spoofing" by sending that to
us. We deliver the packets to Facebook and I assume Facebook will route the
replies just fine through the other ISP.

Now the thing is that my impression is that it actually works so long I do
not actively block it with uRPF or ACLs on our edge. I have learned that
spoofing is evil and I should be blocking this - but why am I sabotaging
something that apparently is doing just fine at some customers?

Regards,

Baldur

Re: Apple Catalina Appears to Introduce Massive Jitter - SOLVED!

[Mark Tinka]

Hi all.

So I may have fixed this for my end, and hopefully others may be able to 
use the same fix.


After a tip from Karl Auerbach and this link:

    https://developer.apple.com/forums/thread/97805

... I was able to fix the problem by disabling Bluetooth.

However, disabling Bluetooth was not enough. I also had to disable all 
Location Services.


After that, I re-enabled Location Services and only allowed for two 
features:


    - NetSpot
    - Find My Mac

With just those two location services, as well as Bluetooth disabled, I 
have no more high jitter.


App performance like Zoom and Youtube uploads are now crisp, with 0.0% 
packet loss.


So looks like that Bluetooth is a huge problem. Confirmed by opening the 
"Console" app, and adding "scan" in the filter bar, top right.


A peak latency of 13.5ms after 300 packets:

Marks-MacBook-Pro.local (172.16.0.239) 2020-10-30T21:06:05+0200
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
Packets   Pings
 Host Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 172.16.0.254 0.0%   300    3.1   4.8   2.2  13.5   1.9

Mark.

Weekly Routing Table Report

[Routing Analysis Role Account]

This is an automated weekly mailing describing the state of the Internet
Routing Table as seen from APNIC's router in Japan.

The posting is sent to APOPS, NANOG, AfNOG, SANOG, PacNOG, SAFNOG
TZNOG, MENOG, BJNOG, SDNOG, CMNOG, LACNOG and the RIPE Routing WG.

Daily listings are sent to bgp-st...@lists.apnic.net

For historical data, please see http://thyme.rand.apnic.net.

If you have any comments please contact Philip Smith .

Routing Table Report   04:00 +10GMT Sat 31 Oct, 2020

Report Website: http://thyme.rand.apnic.net
Detailed Analysis:  http://thyme.rand.apnic.net/current/

Analysis Summary


BGP routing table entries examined:  828067
Prefixes after maximum aggregation (per Origin AS):  316529
Deaggregation factor:  2.62
Unique aggregates announced (without unneeded subnets):  396695
Total ASes present in the Internet Routing Table: 69827
Prefixes per ASN: 11.86
Origin-only ASes present in the Internet Routing Table:   60016
Origin ASes announcing only one prefix:   24837
Transit ASes present in the Internet Routing Table:9811
Transit-only ASes present in the Internet Routing Table:298
Average AS path length visible in the Internet Routing Table:   4.3
Max AS path length visible:  30
Max AS path prepend of ASN ( 45582)  27
Prefixes from unregistered ASNs in the Routing Table:   994
Number of instances of unregistered ASNs:   995
Number of 32-bit ASNs allocated by the RIRs:  33990
Number of 32-bit ASNs visible in the Routing Table:   28151
Prefixes from 32-bit ASNs in the Routing Table:  128583
Number of bogon 32-bit ASNs visible in the Routing Table:14
Special use prefixes present in the Routing Table:1
Prefixes being announced from unallocated address space:515
Number of addresses announced to Internet:   2864617216
Equivalent to 170 /8s, 190 /16s and 151 /24s
Percentage of available address space announced:   77.4
Percentage of allocated address space announced:   77.4
Percentage of available address space allocated:  100.0
Percentage of address space in use by end-sites:   99.5
Total number of prefixes smaller than registry allocations:  279502

APNIC Region Analysis Summary
-

Prefixes being announced by APNIC Region ASes:   216070
Total APNIC prefixes after maximum aggregation:   63548
APNIC Deaggregation factor:3.40
Prefixes being announced from the APNIC address blocks:  211683
Unique aggregates announced from the APNIC address blocks:86759
APNIC Region origin ASes present in the Internet Routing Table:   10986
APNIC Prefixes per ASN:   19.27
APNIC Region origin ASes announcing only one prefix:   3098
APNIC Region transit ASes present in the Internet Routing Table:   1600
Average APNIC Region AS path length visible:4.4
Max APNIC Region AS path length visible: 30
Number of APNIC region 32-bit ASNs visible in the Routing Table:   6108
Number of APNIC addresses announced to Internet:  779017600
Equivalent to 46 /8s, 110 /16s and 221 /24s
APNIC AS Blocks4608-4864, 7467-7722, 9216-10239, 17408-18431
(pre-ERX allocations)  23552-24575, 37888-38911, 45056-46079, 55296-56319,
   58368-59391, 63488-64098, 64297-64395, 131072-143673
APNIC Address Blocks 1/8,  14/8,  27/8,  36/8,  39/8,  42/8,  43/8,
49/8,  58/8,  59/8,  60/8,  61/8, 101/8, 103/8,
   106/8, 110/8, 111/8, 112/8, 113/8, 114/8, 115/8,
   116/8, 117/8, 118/8, 119/8, 120/8, 121/8, 122/8,
   123/8, 124/8, 125/8, 126/8, 133/8, 150/8, 153/8,
   163/8, 171/8, 175/8, 180/8, 182/8, 183/8, 202/8,
   203/8, 210/8, 211/8, 218/8, 219/8, 220/8, 221/8,
   222/8, 223/8,

ARIN Region Analysis Summary


Prefixes being announced by ARIN Region ASes:241495
Total ARIN prefixes after maximum aggregation:   111528
ARIN Deaggregation factor: 2.17
Prefixes being announced from the ARIN address blocks:   239329
Unique aggregates announced from the ARIN address blocks:114696
ARIN Region origin ASes present in the Internet Routing Table:18672
ARIN Prefixes per ASN:12.82
ARIN 

bgp dampening and anycast networks (particularly cloudflare)

[David Hubbard]

Hi all, was curious if anyone has found it necessary to alter their route 
dampening rules related to anycast networks, and Cloudflare especially?  I’ve 
got a customer whose target web server has been going intermittently 
inaccessible from a very geographically distant Cloudflare location (AU), while 
no reports of issues from anywhere closer to the US.  I’m seeing a bunch of 
their /24’s dampened on my side in several locations, and they appear to be 
networks that favor or are specific to AU, so I’m thinking that’s the issue.  
I’m going to whitelist their ASN, but perhaps I need to work on the policy to 
be more tolerant of flaps compared to years past with the increase in anycast 
use?

Thanks,

David

Re: 100G over 100 km of dark fiber

[Jared Brown]

The 100 km leg completes a ring.
 

Jared
 

Sent: Friday, October 30, 2020
From: "Ben Cannon" 
To: "Jared Brown" 
Cc: nanog@nanog.org
Subject: Re: 100G over 100 km of dark fiber

You could break this into 10x 10g coherent lanes, but you’re going to end up 
back close to coherent 100g prices.
 
You’re at the threshold distance where you’re past all the short range tech and 
are seriously pushing it - whereas the 100g coherent tech is just taking off.  
 
How important is this link?
 

Ms. Benjamin PD Cannon, ASCE
6x7 Networks & 6x7 Telecom, LLC 
CEO 
b...@6by7.net[mailto:b...@6by7.net]
"The only fully end-to-end encrypted global telecommunications company in the 
world.”
FCC License KJ6FJJ

 On Oct 30, 2020, at 7:19 AM, Jared Brown  wrote:
 
Hello NANOG!

I need to push 100G over 100 km of dark fiber. Since there are no 100G 
pluggable optics with this reach (~25 dB), I have been offered coherent 
transport systems to solve my problem. This is all good and well, except total 
system costs start from high five figures.

So, my question is, do I have any other options?

I can't help noticing that you can break out a 100G QSFP into four 25G QSFPs. 
25G DWDM systems are relatively inexpensive (low five figures), but can you 
make 25G DWDM go 100 km?

I only need the one 100G, so I don't really need a highly scalable DWDM system. 
I can't put anything midspan, or if I could it would cost more than just going 
with a coherent system.


Jared

Re: 100G over 100 km of dark fiber

[Tarko Tikan]

hey,


If it’s just a single 100G channel needed you could try 100GBASE-ZR4. Specified 
for 80km, 30db power budget they could actually reach more the 80km.
Dispersion should also be „no" problem in the 1310nm length. I have to say that 
I never tried this on 100km distance without coherent solutions.


Just to add to my original suggestion, we just did 100G-ZR4 over 30dB 
link with pre-FEC BER 3.174E-11.


As OP is asking for a solution for 25dB I don't see any reason why ZR4 
would not work and why you would need coherent, amplifiers or any other 
additional solution except when you are limited by QSFP28 SFF power.


--
tarko

Re: 100G over 100 km of dark fiber

[Ben Cannon]

You could break this into 10x 10g coherent lanes, but you’re going to end up 
back close to coherent 100g prices.

You’re at the threshold distance where you’re past all the short range tech and 
are seriously pushing it - whereas the 100g coherent tech is just taking off.  

How important is this link?

Ms. Benjamin PD Cannon, ASCE
6x7 Networks & 6x7 Telecom, LLC 
CEO 
b...@6by7.net
"The only fully end-to-end encrypted global telecommunications company in the 
world.”

FCC License KJ6FJJ



> On Oct 30, 2020, at 7:19 AM, Jared Brown  wrote:
> 
> Hello NANOG!
> 
> I need to push 100G over 100 km of dark fiber. Since there are no 100G 
> pluggable optics with this reach (~25 dB), I have been offered coherent 
> transport systems to solve my problem. This is all good and well, except 
> total system costs start from high five figures.
> 
> So, my question is, do I have any other options?
> 
> I can't help noticing that you can break out a 100G QSFP into four 25G QSFPs. 
> 25G DWDM systems are relatively inexpensive (low five figures), but can you 
> make 25G DWDM go 100 km?
> 
> I only need the one 100G, so I don't really need a highly scalable DWDM 
> system. I can't put anything midspan, or if I could it would cost more than 
> just going with a coherent system.
> 
> 
> Jared

Re: 100G over 100 km of dark fiber

[Nick Hilliard]

Dale W. Carder wrote on 30/10/2020 14:33:

You may also find that 100G PAM4 could work.


not at 100km. This would be outside the dispersion tolerance limits for 
pam4.


Nick

RE: 100G over 100 km of dark fiber

[Brian Turnbow via NANOG]

Hi jared 

as others have pointed out there are lots of options


inphi offers these
https://www.inphi.com/products/colorz/


or use a box like packetlight, here is a Arista solution brief 
https://www.arista.com/assets/data/pdf/Whitepapers/Arista_Packetlight_100G_Extension_Solution.pdf
and if you serch for openline systems there are a few that do smaller systems 
2/4/8 ports that are available
FS even offers this
https://www.fs.com/de-en/specials/100g-fmx-transport-platform-103.html
more expensive than optics  but an alternative and you can stay in low 5 
figures.

cisco has the ncs55a2 with these
https://www.cisco.com/c/en/us/products/collateral/routers/network-convergence-system-5500-series/datasheet-c78-743732.html
that costs like a gazzilion dollars but your company may have great discounts...


HTH

Brian


> -Original Message-
> From: NANOG  On Behalf Of
> Jared Brown
> Sent: Friday, October 30, 2020 3:19 PM
> To: nanog@nanog.org
> Subject: 100G over 100 km of dark fiber
> 
> Hello NANOG!
> 
> I need to push 100G over 100 km of dark fiber. Since there are no 100G
> pluggable optics with this reach (~25 dB), I have been offered coherent
> transport systems to solve my problem. This is all good and well, except total
> system costs start from high five figures.
> 
> So, my question is, do I have any other options?
> 
> I can't help noticing that you can break out a 100G QSFP into four 25G QSFPs.
> 25G DWDM systems are relatively inexpensive (low five figures), but can you
> make 25G DWDM go 100 km?
> 
> I only need the one 100G, so I don't really need a highly scalable DWDM
> system. I can't put anything midspan, or if I could it would cost more than 
> just
> going with a coherent system.
> 
> 
> Jared

Re: plea for comcast/sprint handoff debug help

[Tom Beecher]

Alex:

When I follow the RFC rabbit hole :

RFC6481 :  A Profile for Resource Certificate Repository Structure

The publication repository MUST be available using rsync
>  [RFC5781 ] [RSYNC 
> ].  Support of additional 
> retrieval mechanisms
>  is the choice of the repository operator.  The supported
>  retrieval mechanisms MUST be consistent with the accessMethod
>  element value(s) specified in the SIA of the associated CA or
>  EE certificate.
>
>
Then :

RFC8182 :  The RPKI Repository Delta Protocol (RRDP)

 This document allows the use of RRDP as an additional repository
>distribution mechanism for RPKI.  In time, RRDP may replace rsync
>[RSYNC ] as the only 
> mandatory-to-implement repository distribution
>mechanism.  However, this transition is outside of the scope of this
>document.
>
>
Is this not the case then that currently rsync is still mandatory, even if
RRDP is in place?  Or is there a more current RFC that has defined the
transition that I did not locate?

On Fri, Oct 30, 2020 at 7:49 AM Alex Band  wrote:

>
> > On 30 Oct 2020, at 01:10, Randy Bush  wrote:
> >
> > i'll see your blog post and raise you a peer reviewed academic paper and
> > two rfcs :)
>
> For the readers wondering what is going on here: there is a reason there
> is only a vague mention to two RFCs instead of the specific paragraph where
> it says that Relying Party software must fall back to rsync immediately if
> RRDP is temporarily unavailable. That is because this section doesn’t
> exist. The point is that there is no bug and in fact, Routinator has a
> carefully thought out strategy to deal with transient outages. Moreover, we
> argue that our strategy is the better choice, both operationally and from a
> security standpoint.
>
> The paper shows that Routinator is the most used RPKI relying party
> software, and we know many of you here rely on it for route origin
> validation in a production environment. We take this responsibility and
> therefore this matter very seriously, and would not want you to think we
> have been careless in our software design. Quite the opposite.
>
> We have made several attempts within the IETF to have a discussion on
> technical merit, where aspects such as overwhelming an rsync server with
> traffic, or using aggressive fallback to rsync as an entry point to a
> downgrade attack have been brought forward. Our hope was that our arguments
> would be considered on technical merit, but that did not happen yet. Be
> that as it may, operators can rest assured that if consensus goes against
> our logic, we will change our design.
>
> > perhaps go over to your unbound siblings and discuss this analog.
>
> The mention of Unbound DNS resolver in this context is interesting,
> because we have in fact discussed our strategy with the developers on this
> team as there is a lot to be learned from other standards and operational
> experiences.
>
> We feel very strongly about this matter because the claim that using our
> software negatively affects Internet routing robustness strikes at the core
> of NLnet Labs’ existence: our reputation and our mission to work for the
> good of the Internet. They are the core values that make it possible for a
> not-for-profit foundation like ours to make free, liberally licensed open
> source software.
>
> We’re proud of what we’ve been able to achieve and look forward to a
> continued open discussion with the community.
>
> Respectfully,
>
> Alex
>

Re: 100G over 100 km of dark fiber

[Brandon Martin]

On 10/30/20 10:27 AM, Vincentz Petzholtz wrote:

If it’s just a single 100G channel needed you could try 100GBASE-ZR4. Specified 
for 80km, 30db power budget they could actually reach more the 80km.
Dispersion should also be „no" problem in the 1310nm length. I have to say that 
I never tried this on 100km distance without coherent solutions.


If you end up marginal with the ZR4 at the receive end, you may be able 
to use a silicon optical preamp to make up the margin.


If you are at the noise floor already (ZR4 receivers are pretty good), 
you may just need to dump more power into the line.  I recently came 
across the PDFA (like an EDFA but the fiber is doped with Praseodymium 
instead of Erbium) which operate on the 1310 band and can push upwards 
of 20-23dBm of power out.  They are not cheap (low 5 figures), but they 
are cheaper than coherent and will still let you run on 1310 so you 
don't have chromatic dispersion issues.  With good (ER4 or ZR4) 
receivers good down to -20dBm or so, you've got 30-33dB of link budget 
which should just about make your 100km if it's decent fiber.  They're 
also usable as a pre-amp (different configuration optimized for gain and 
noise figure instead of power, similar to EDFA preamps) and have 
performance potentially better than silicon amps in that situation.


The 1550 PAM4 pizza box EDFA+Mux+DCM that others have mentioned may also 
work and end up even cheaper, but I normally see those for 40-60km or 
"up to" 80km with them really pushing the link budget at that point.


Honestly, I'd be tempted to just suck it up and do a coherent solution, 
though I admit it would probably be at least 2x the cost.  You can 
probably get a 200G carrier, though.

--
Brandon Martin

Re: 100G over 100 km of dark fiber

[Dale W. Carder]

You may also find that 100G PAM4 could work.  There are some vendors that
sell the optic, and an outboard EDFA + DCF pizza box.

Dale

Thus spake Tarko Tikan (ta...@lanparty.ee) on Fri, Oct 30, 2020 at 04:25:58PM 
+0200:
> hey,
> 
> > I need to push 100G over 100 km of dark fiber. Since there are no 100G 
> > pluggable optics with this reach (~25 dB), I have been offered coherent 
> > transport systems to solve my problem. This is all good and well, except 
> > total system costs start from high five figures.
> 
> 100G-ZR4 QSFP28 is on the market and works. Just watch out for power
> limitations, your typical DC switch might not support it but proper stuff
> has no problems providing 6.5W of power.
> 
> -- 
> tarko

Re: 100G over 100 km of dark fiber

[Vincentz Petzholtz]

Hi Jared,

If it’s just a single 100G channel needed you could try 100GBASE-ZR4. Specified 
for 80km, 30db power budget they could actually reach more the 80km.
Dispersion should also be „no" problem in the 1310nm length. I have to say that 
I never tried this on 100km distance without coherent solutions.

Best regards,
Vincentz
PS: https://www.flexoptix.net/en/transceiver/q-161hg-80.html?co10728=100306

> Am 30.10.2020 um 15:19 schrieb Jared Brown :
> 
> Hello NANOG!
> 
> I need to push 100G over 100 km of dark fiber. Since there are no 100G 
> pluggable optics with this reach (~25 dB), I have been offered coherent 
> transport systems to solve my problem. This is all good and well, except 
> total system costs start from high five figures.
> 
> So, my question is, do I have any other options?
> 
> I can't help noticing that you can break out a 100G QSFP into four 25G QSFPs. 
> 25G DWDM systems are relatively inexpensive (low five figures), but can you 
> make 25G DWDM go 100 km?
> 
> I only need the one 100G, so I don't really need a highly scalable DWDM 
> system. I can't put anything midspan, or if I could it would cost more than 
> just going with a coherent system.
> 
> 
> Jared



signature.asc
Description: Message signed with OpenPGP

Re: 100G over 100 km of dark fiber

[Tarko Tikan]

hey,


I need to push 100G over 100 km of dark fiber. Since there are no 100G 
pluggable optics with this reach (~25 dB), I have been offered coherent 
transport systems to solve my problem. This is all good and well, except total 
system costs start from high five figures.


100G-ZR4 QSFP28 is on the market and works. Just watch out for power 
limitations, your typical DC switch might not support it but proper 
stuff has no problems providing 6.5W of power.


--
tarko

100G over 100 km of dark fiber

[Jared Brown]

Hello NANOG!

I need to push 100G over 100 km of dark fiber. Since there are no 100G 
pluggable optics with this reach (~25 dB), I have been offered coherent 
transport systems to solve my problem. This is all good and well, except total 
system costs start from high five figures.

So, my question is, do I have any other options?

I can't help noticing that you can break out a 100G QSFP into four 25G QSFPs. 
25G DWDM systems are relatively inexpensive (low five figures), but can you 
make 25G DWDM go 100 km?

I only need the one 100G, so I don't really need a highly scalable DWDM system. 
I can't put anything midspan, or if I could it would cost more than just going 
with a coherent system.


Jared

Re: plea for comcast/sprint handoff debug help

[Alex Band]

> On 30 Oct 2020, at 01:10, Randy Bush  wrote:
> 
> i'll see your blog post and raise you a peer reviewed academic paper and
> two rfcs :)

For the readers wondering what is going on here: there is a reason there is 
only a vague mention to two RFCs instead of the specific paragraph where it 
says that Relying Party software must fall back to rsync immediately if RRDP is 
temporarily unavailable. That is because this section doesn’t exist. The point 
is that there is no bug and in fact, Routinator has a carefully thought out 
strategy to deal with transient outages. Moreover, we argue that our strategy 
is the better choice, both operationally and from a security standpoint.

The paper shows that Routinator is the most used RPKI relying party software, 
and we know many of you here rely on it for route origin validation in a 
production environment. We take this responsibility and therefore this matter 
very seriously, and would not want you to think we have been careless in our 
software design. Quite the opposite.

We have made several attempts within the IETF to have a discussion on technical 
merit, where aspects such as overwhelming an rsync server with traffic, or 
using aggressive fallback to rsync as an entry point to a downgrade attack have 
been brought forward. Our hope was that our arguments would be considered on 
technical merit, but that did not happen yet. Be that as it may, operators can 
rest assured that if consensus goes against our logic, we will change our 
design.

> perhaps go over to your unbound siblings and discuss this analog.

The mention of Unbound DNS resolver in this context is interesting, because we 
have in fact discussed our strategy with the developers on this team as there 
is a lot to be learned from other standards and operational experiences. 

We feel very strongly about this matter because the claim that using our 
software negatively affects Internet routing robustness strikes at the core of 
NLnet Labs’ existence: our reputation and our mission to work for the good of 
the Internet. They are the core values that make it possible for a 
not-for-profit foundation like ours to make free, liberally licensed open 
source software. 

We’re proud of what we’ve been able to achieve and look forward to a continued 
open discussion with the community.

Respectfully,

Alex

Re: plea for comcast/sprint handoff debug help

[Job Snijders]

On Thu, Oct 29, 2020 at 09:14:16PM +0100, Alex Band wrote:
> In fact, we argue that it's actually a bad idea to do so:
> 
> https://blog.nlnetlabs.nl/why-routinator-doesnt-fall-back-to-rsync/
>
> We're interested to hear views on this from both an operational and
> security perspective.

I don't see a compelling reason to not use rsync when RRDP is
unavailable.

Quoting from the blog post:

"While this isn’t threatening the integrity of the RPKI – all data
is cryptographically signed making it really difficult to forge data
– it is possible to withhold information or replay old data."

RRDP does not solve the issue of withholding data or replaying old data.
The RRDP protocol /also/ is unauthenticated, just like rsync. The RRDP
protocol basically is rsync wrapped in XML over HTTPS.

Withholding of information is detected through verification of RPKI
manifests (something Routinator didn't verify up until last week!),
and replaying of old data is addressed by checking validity dates and
CRLs (something Routinator also didn't do until last week!).

Of course I see advantages to this industry mainly using RRDP, but those
are not security advantages. The big migration towards RRDP can happen
somewhere in the next few years.

The arguments brought forward in the blog post don't make sense to me.
The '150,000' number in the blog post seems a number pulled from thin
air.

Regards,

Job

Re: Apple Catalina Appears to Introduce Massive Jitter

[Mark Tinka]

On 10/29/20 20:14, David Curado wrote:

I was curious, so poked at this... my results from a macbook pro 2019 
running Catalina 10.15.3


sudo /usr/local/sbin/mtr -r 10.200.200.200

Start: 2020-10-29T14:09:08-0400
HOST: bos-mp36c                   Loss%   Snt Last   Avg  Best  Wrst StDev
  1.|-- 10.200.200.200             0.0%    10 11.9  63.7   9.0 340.2 104.1

sudo sysctl net.link.generic.system.hwcksum_tx=0
net.link.generic.system.hwcksum_tx: 1 -> 0
sudo sysctl net.link.generic.system.hwcksum_rx=0
net.link.generic.system.hwcksum_rx: 1 -> 0
sudo ifconfig en0 -rxcsum

sudo /usr/local/sbin/mtr -r 10.200.200.200

Start: 2020-10-29T14:09:43-0400
HOST: bos-mp36c                   Loss%   Snt Last   Avg  Best  Wrst StDev
  1.|-- 10.200.200.200             0.0%    10 19.8  13.6   9.8  20.9   4.5

So, seems to make things better. =-)


I tried it again this morning, on the off-chance that the stars didn't 
align yesterday. Same deal:


sh-3.2# sysctl net.link.generic.system.hwcksum_tx=0
net.link.generic.system.hwcksum_tx: 1 -> 0
sh-3.2# sysctl net.link.generic.system.hwcksum_rx=0
net.link.generic.system.hwcksum_rx: 1 -> 0
sh-3.2# ifconfig en0 -rxcsum

Marks-MacBook-Pro.local (172.16.0.239) 2020-10-30T08:39:44+0200
Keys:  Help   Display mode   Restart statistics   Order of fields quit
Packets   Pings
 Host Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 172.16.0.254 0.0%    28    1.9  28.7   1.9 218.9  56.1

Mark.