Re: Mailing list SPF Failure

[Hank Nussbacher]

On 17/05/2024 5:45, Karl Auer wrote:

On Thu, 2024-05-16 at 19:27 -0700, Michael Thomas wrote:

On 5/16/24 7:22 PM, Scott Q. wrote:

Mike, you do realize Google/Gmail rejects e-mails with
invalid/missing SPF right ?

I was receiving the mail while NANOG had no SPF record, so no? Any
receiver would be really stupid take a single signal as
disqualifying.

For small-scale senders, it's either or both. For large-scale senders
(5000+ per day) it's both.

At least according to this:

https://support.google.com/a/answer/81126


I think some may have missed these announcements:

https://labs.ripe.net/author/fergalc/enhancing-email-delivery-at-the-ripe-ncc/

https://blog.google/products/gmail/gmail-security-authentication-spam-protection/


Regards,

Hank

Re: puck not responding

[Hank Nussbacher]

On 29/02/2024 17:21, Jared Mauch wrote:

On behalf of cisco-nsp and outages - we salute you.

-Hank




On Feb 28, 2024, at 1:30 AM, Daniel Marks via NANOG  wrote:

We’re getting rocked by storms here in Michigan, could be related.


[ brief version of what happened from what I can tell reconstructing things]

I was alerted ~4am US/E yesterday about the issue.  This machine has been 
generously hosted by my previous employer for quite some time, funnily enough 
it was 7 years ago almost to the day since I started my current employment.

The IPMI was not responsive and the machine was located in 350 Cermak, on a 
floor that was not impacted with the heat/cold event.

I have been meaning to move things off and on, but never quite had the 
motivation to tackle the task.  Yesterday forced my hand.

Once I confirmed that we could get the machine out of the colocation facility 
(thank you again NTT) I drove from Michigan to Chicago, got lunch and picked up 
the machine and headed back to the colocation that I have in Michigan at the 
123Net/DetroitIX site.

Once I had a console on it, I determined that this old machine had a few things 
that had been gradually updated and upgraded over time, not all the filesystem 
options were set correctly and after some tune2fs options were set and fstab 
updated to ensure everything is migrated fully from ext2 -> ext4 the system was 
able to be booted without issues.

Afterwards I’ve determined that there is still a hardware related problem, so I 
am now going to move it to new hardware later today schedule permitting as I 
want to go onsite and make sure that the I/O is performant.

Feb 28 22:09:05 kernel: Memory: 32816872K/33544380K available (20480K kernel 
code, 3276K rwdata, 14748K rodata, 4588K init, 4892K bss, 727248K reserved, 0K 
cma-reserved)
Feb 29 00:20:07 kernel: Memory: 16326408K/16767164K available (20480K kernel 
code, 3276K rwdata, 14748K rodata, 4588K init, 4892K bss, 440496K reserved, 0K 
cma-reserved)

Not quite a great thing when nobody is onsite and the machine requires being 
power cycled and the amount of memory changes.

If you are seeing any other issues, do let me know, I did move the IPv4 space 
but have renumbered for v6, so if you use my free secondary dns service, and 
your own vanity name, you will need to update your  records.

If you are seeing any reachability issues let me know, there should be ROA and 
other objects in place for things.

Sorry everyone got this email, feel bad it’s like when warren asked the list 
some personal details :-)

- Jared


(Even more details: changing disk images from qcow -> qcow2 and other things like ext2 
-> ext3/4 over all the years as the machine has gone from Linux -> FreeBSD -> 
Linux again and other things is always a fun way to keep bringing your legacy around with 
you, it’s good overall)

puck not responding

[Hank Nussbacher]

Any clue as to when bgp.he.net will be back?

[Hank Nussbacher]

Thanks,

Hank

Re: Issue with Geolocation in Lasvegas

[Hank Nussbacher]

On 04/01/2024 9:13, Raja Sekhar Gullapalli via NANOG wrote:

https://www.iucc.ac.il/en/blog/2021-05-google-geo-location/

:-)

Regards,
Hank


Team,

We are having issues in our lasvegas office & it shows geolocation in 
all browsers as Israel instead of US region when we access 
news.google.com in our PC.


Our public ip is 129.46.96.20.

Can you help to whom we can contact to resolve the issue.

Regards,

Raja

Re: Any comprehensive listing of where Google's IPs originate from?

[Hank Nussbacher]

On 04/12/2023 16:09, Drew Weaver wrote:

Although not an answer to your specific question, when I need to reduce 
latency to a Google cloud region I use:

https://gcping.com/

Regards,
Hank


Hello,

We are trying to reduce latency to a region in Google Cloud which we 
are in the same city of. Latency is currently about 22ms rt for the 
traffic to go 9 miles.


I am having the hardest time finding any comprehensive list of what 
exchanges, transit, etc their IP addresses are being announced over.


Specifically trying to get closer to addresses in these prefixes:

34.162.192.0/18

34.162.64.0/18

Any info is greatly appreciated.

Thanks,

-Drew

BGP hijack?

[Hank Nussbacher]

We just had every single prefix in AS378 start being announced by AS2027.

Every announcement by AS2027 is failing RPKI yet being propagated a bit.
Is this yet another misbehaving device or an actual attack?


Thanks,

Hank

Re: Low to Mid Range DWDM Platforms

[Hank Nussbacher]

On 06/10/2023 16:07, Mike Hammett wrote:

I  have found that for low end DWDM solutions, 
https://www.packetlight.com/ has always been the cheapest available.


Regards,
Hank


I've been using various forms of passive WDM for years. I have a couple 
different projects on my plate that require me to look at the next level of 
platform.

In some projects, I'll be looking for needing to have someone long distances of 
glass without any electronics. Some spans could be over 60 miles.

In some projects, I'll need to transport multiple 100-gig waves.

What is the landscape like between basic passive and something like a 30 
terabit Ciena? I know of multiple vendors in that space, but I like to learn 
more about what features I need and what features I don't need from somewhere 
other than the vendor's mouth. Obviously, the most reliability at the least 
cost as well.

Thanks.



-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

Re: Providing geofeed info to Google

[Hank Nussbacher]

Old topic: if one doesn't have access to https://isp.google.com how does 
one update their geo-location data so Google sees it?



Thanks,

Hank


On Tue, Aug 30, 2022 at 12:34:41PM -0700, Hugo Slabbert wrote:

Google folks:

I see historical reference to needing to use the Google Peering Portal (
http://peering.google.com) if you need to provide Google with geofeed info
for GeoIP info on network blocks, ref
https://mailman.nanog.org/pipermail/nanog/2015-May/075229.html.

Is that still the case?  Are there any avenues to provide Google with
geofeed info if you're *not* currently peering with 15169? Or to get access
to just the geofeed update portion of the Peering Portal?

Re: v6 route mess frm AS266970

[Hank Nussbacher]

On 29/08/2023 18:41, Randy Bush wrote:

is a massive route leak not even menntioned when it is only ipv6?

the guess i heard was it looked like a classic config reorigination
disaster.

randy


Has the route leak been resolved?    BGPstream still shows it as active:

https://bgpstream.crosswork.cisco.com/


RPKI only worked where it is implemented.

I saw one path via Lumen (AS3356) and was disappointed to see it based 
on their blog from 2.5 years ago:


https://blog.lumen.com/lumen-enhances-routing-security-with-resource-public-key-infrastructure-rpki/ 



"Once implemented, Lumen will use RPKI route validation on all BGP 
sessions for both customers and peers. Lumen’s RPKI validation servers 
download the ROAs, examine them, then send the tables to routers that 
can determine the validity of an IP prefix."



MANRS confirms that AS3356 does not do much RPKI (see attachment).


Regards,

Hank

Re: Geolocastion and FF and Whatsapp

[Hank Nussbacher]

On 18/08/2023 16:36, J. Hellenthal wrote:

Private (incognito) in FF gives English page.

Interesting.

Thanks,
Hank


Move your FF profile out of the way and reopen FF... diff results ?



On Aug 18, 2023, at 00:45, Hank Nussbacher  wrote:

When I go to https://www.whatsapp.com/ I see the page in Latvian - 
but only on FF.


When using Chrome I see the page in English.

So who is doing bad geolocation?  FF or Whatsapp?


Thanks,

Hank




--

J. Hellenthal

The fact that there's a highway to Hell but only a stairway to Heaven 
says a lot about anticipated traffic volume.

Geolocastion and FF and Whatsapp

[Hank Nussbacher]

When I go to https://www.whatsapp.com/ I see the page in Latvian - but 
only on FF.


When using Chrome I see the page in English.

So who is doing bad geolocation?  FF or Whatsapp?


Thanks,

Hank

Re: Do ISP's collect and analyze traffic of users?

[Hank Nussbacher]

On 19/05/2023 15:27, Justin Streiner wrote:

It amazes me how people can focus on Netflow metadata and ignore things 
like Microsoft telemetry data from every Windows box, or ignore the 
massive amount of html cookies that are traded by companies or how 
almost every corporate firewall or anti-spam box "reports" back to the 
mother ship and sends tons of information via secret channels like 
hashed DNS lookups just to be avoided.


Regards,
Hank

There are already so many different ways that organizations can find 
out all sorts of information about individual users, as others have 
noted (social media interactions, mobile location/GPS data, call/text 
history, interactions with specific sites, etc), that there probably 
isn't much incentive for many providers to harvest data beyond what is 
needed for troubleshooting and capacity planning.  Plus, gathering 
more data - potentially down to the level packet payload - is not an 
easy problem to solve (read: expensive) and doesn't scale well at all. 
100G links are very common today, and 400G is becoming so.  I doubt 
that many infrastructure providers would be able to justify the major 
investments in extra infrastructure to support this, for a revenue 
stream that likely wouldn't match that investment, which would make 
such an investment a loss-leader.


Content providers - particularly social media platforms - have a 
somewhat different business model, but those providers already have 
many different ways to harvest and sell large troves of user data.


Thank you
jms

Re: Best Linux (or BSD) hosted BGP?

[Hank Nussbacher]

On 02/05/2023 17:56, Warren Kumari wrote:

For those that like FRR:
https://thehackernews.com/2023/05/researchers-uncover-new-bgp-flaws-in.html

Regards,
Hank


+lots.

I've used a number of Linux routing thingies (BIRD, Quagga, 
VyOS/Ubiquiti, OpenBGPd, ExBGP), and FRR is (for me at least) by far 
the friendliest. It's trivial to spin this up on a cloud VM and start 
announcing a prefix.


For doing something like Anycast though (where you are mostly just 
announcing a route on demand), ExaBGP is great.


W


On Mon, May 01, 2023 at 2:03 PM, Jean Franco  wrote:

https://frrouting.org/ 

RPKI in Lacnic?

[Hank Nussbacher]

Just got this:


Possible TA malfunction or incomplete VRP file: 100.00% of the ROAs 
disappeared from lacnic


DETAILS:
--
Event type:   rpki-diff
When event started:   2023-04-15 17:17:30 UTC
Last event:   2023-04-15 17:17:30 UTC


Only us or everyone?


Thanks,

Hank

Re: Starlink routing

[Hank Nussbacher]

On 23/01/2023 0:42, Michael Thomas wrote:
I read in the Economist that the gen of starlink satellites will have 
the ability to route messages between each satellite. Would 
conventional routing protocols be up to such a challenge? Or would it 
have to be custom made for that problem? And since a lot of companies 
and countries are getting on that action, it seems like fertile ground 
for (bad) wheel reinvention?


Mike


For further reading try:

https://www.internetsociety.org/wp-content/uploads/2022/11/Perspectives-on-LEO-Satellites.pdf


-Hank

Re: Getting Fiber to My Town by Jared Mauch

[Hank Nussbacher via NANOG]

On 10/09/2020 15:16, Jared Mauch wrote:

Go Jared:
https://www.dailystar.co.uk/tech/news/man-who-built-broadband-avoid-27717560

-Hank





On Sep 10, 2020, at 8:06 AM, Jared Brown  wrote:

I believe this belongs here:

Getting Fiber to My Town by Jared Mauch
https://www.youtube.com/watch?v=ASXJgvy3mEg (YouTube video of NLnog 
presentation)
https://nlnog.net/static/live/nlnog_live_sep_2020_jared.pdf (slides for 
presentation)
https://news.ycombinator.com/item?id=24424910#24430901 (discussion on Hacker 
News with Jared participating)
https://washftth.com/ (project homepage)

I find this an interesting description of how to apply skills that we normally 
only use at work to solve connectivity issues at home. Quite timely too, as 
home connectivity is needed more than ever.


As my kids are in the other room on 4x zooms at once, the prior connection 
could not have survived the load with them and me working as well.

I’m working with the PC on giving another version of this talk (I want to 
provide more financial details) for an upcoming meeting.

Stay tuned to the NANOG Agenda :-)

- Jared

Re: Test email

[Hank Nussbacher]

On 20/06/2022 11:30, Peter Potvin wrote:

I did not send this to the list.  I assume the admins are testing out 
what has been blocking my emails for the past month and somehow this 
email slipped thru.  Just ignore and delete.


-Hank

Why did moderation let this through the filters? I don't believe that 
testing email functionality is the intended use case of the NANOG 
mailing list.


Also worth noting that the website for the domain this came from says 
the owner of the site specializes in "anti-spam", which based on this 
email alone doesn't look to be the case. Anyone else agree?


Regards,
Peter


On Mon., Jun. 20, 2022, 4:15 a.m. , > wrote:



Hello,

Checking Email Functionality.

Hosting Support
Thank you,


The information contained in this message may be privileged, 
confidential and protected from disclosure. This message is intended 
only for the designated recipient(s). It is subject to access, review 
and disclosure by the sender's Email System Administrator. If you have 
received this message in error, please advise by return e-mail so that 
our address records can be corrected and please delete immediately 
without reading, copying or forwarding to others. Any unauthorized 
review, use, disclosure or distribution is prohibited.

Copyright © 2022 Accuris Technologies Ltd. All Rights Reserved.

L'information contenue dans ce message pourrait être de nature 
privilégiée, confidentielle et protégée contre toute divulgation. Ce 
message est destiné à l'usage exclusif du(des) destinataire(s) visé(s). 
Le gestionnaire de système du courrier électronique de l'expéditeur 
pourrait avoir accès à ce message, l'examiner et le divulguer. Si ce 
message vous est transmis par erreur, veuillez nous en aviser par 
courrier électronique à notre adresse, afin que l'on puisse corriger nos 
registres, puis veuillez le supprimer immédiatement, sans le lire, le 
copier ou le transmettre à des tiers. Tout examen, toute utilisation, 
divulgation ou distribution non autorisé de cette information est interdit.

Droit d'auteur © 2022 Accuris Technologies Ltd. Tous droits réservés.

Re: Newbie x Cisco IOS-XR x ROV: BCP to not harassing peer(s)

[Hank Nussbacher]

On 14/05/2022 00:16, Jakob Heitz (jheitz) via NANOG wrote:



'RPKI-dropped-only' causes the dropped routes to be stored. This will prevent
the unnecessary route-refreshes described above. It does not prevent all
route-refreshes, but uses significantly less memory than 'RPKI-tested-only'

Regards,
Jakob.


In the end, the reason for all this RPKI-thingy is to prevent route 
spoofing by malicious actors.  It sure would be nice if someone from the 
top 20: https://asrank.caida.org/ would be able to have an auto-updated 
site that showed all RPKI dropped from their end.


This would complement https://bgpstream.crosswork.cisco.com/ for those 
of us who want to know who is trying to hijack our routes at the core.


Regards,
Hank

Re: Sabotage: several severed cables at the origin of a major internet outage in France

[Hank Nussbacher]

On 27/04/2022 17:29, Nick Hilliard wrote:

https://twitter.com/apb_laudrain/status/1519252859598032898

-Hank


+ pics:

https://twitter.com/acontios_net/status/1519296590015606787
https://twitter.com/acontios_net/status/1519280710762348545
https://twitter.com/acontios_net/status/1519276453350805504

Nick

Paul Ferguson wrote on 27/04/2022 15:17:

On 4/27/22 7:08 AM, Sean Donelan wrote:



Multiple physical cable cuts in multiple diverse locations in France.

Several networks that connect the internet infrastructures of major 
French cities were cut overnight, in a short interval. A state source 
evokes with "the Obs" a "coordinated malicious act", which confirms 
SFR and Free affected. An investigation has been opened.


https://www.nouvelobs.com/faits-divers/20220427.OBS57722/plusieurs-cables-sectionnes-a-l-origine-d-une-importante-panne-internet-en-france.html 





English language news article, fwiw:

https://www.telegraph.co.uk/world-news/2022/04/27/internet-multiple-cities-across-france-suspected-sabotage/ 



Cheers,

- ferg

Re: Any sign of supply chain returning to normal?

[Hank Nussbacher]

On 23/04/2022 01:28, na...@jima.us wrote:

Ordered a pair of ASR9906s in Jan 2022 with delivery Aug 2022.

-Hank


Anecdotally, I had a pair of Nexus 93180s that I ordered in May 2021 show up in 
February 2022, so 9 months. The estimated ship date got punted several times 
(probably due to being preempted by folks employing the approach Laura outlined 
;-) ).

I haven't ordered anything since then, but I understand that 4-8 months isn't 
unexpected, still.

- Jima

From: NANOG  On Behalf Of Drew Weaver
Sent: Friday, April 22, 2022 07:24
To: 'nanog@nanog.org' 
Subject: Any sign of supply chain returning to normal?

I'm not sure if this is the right place for this discussion but I can't think 
of anywhere better to ask.

Has anyone seen any progress whatsoever on supply chain issues with networking 
hardware?

I've noticed that primary market lead times have been increasing and at the 
same time secondary market pricing has also been going higher at the same time, 
still.

What have you seen?

SATCOM terminals under attack in Europe

[Hank Nussbacher]

https://www.reversemode.com/2022/03/satcom-terminals-under-attack-in-europe.html


-Hank

Re: Russia to disconnect from global Internet

[Hank Nussbacher]

Bill Woodcock wrote:


> This applies exclusively to Russian federal government networks, not 
ISPs or telecom operators.



https://twitter.com/krisnova/status/1500590779047170048?s=12

says otherwise.


-Hank

Re: identity.cisco.com certificate has expired

[Hank Nussbacher]

On 05/03/2022 21:08, Matthew Huff wrote:

Arghh…

Just an FYI, id.cisco.com is fubar’ed. Hopefully cisco has already fixed 
it and the proxies/caches/cdns just need to timeout, but just in case 
anyone knows a contact at Cisco’s ops group…


You mean in addition to all the other Cisco boxes that are fubered already?
https://www.itnews.com.au/news/old-quovadis-certificate-issue-starting-to-break-cisco-products-576794
https://tools.cisco.com/security/center/resources/Q-CA-Root-Change

-Hank



*Matthew Huff*| Director of Technical Operations | OTA Management LLC

/Office: 914-460-4039/

/mh...@ox.com  | //www.ox.com /

*...*

Conflicts and fiber cuts

[Hank Nussbacher]

As the discussion rages on NANOG, RIPE, CENTR and many other 
uber-technical forums, I would like to see whether we can focus on what 
we know best - networking.  Perhaps a weekly report of fiber cuts 
throughout Europe (starting from Feb 15) and the RFO that the carrier 
provided.  Of especial interest would be undersea/underocean cuts or 
strange outages that the carrier cannot explain.  Perhaps we can then 
map where some nation/state is sabotaging fiber or tapping into such fiber.



Anyone willing to run with this?


-Hank

Re: enom giving Google a bad name

[Hank Nussbacher]

On 16/01/2022 19:57, Rubens Kuhl wrote:

On Sun, Jan 16, 2022 at 2:38 PM Hank Nussbacher  wrote:


Many of you might be following the enom weekend fiasco:

https://twitter.com/enomsupport/status/1482621466151571456

https://twitter.com/enomsupport/status/1482707275529678849

https://enomstatus.com/

Thousands of domains have been knocked out.


Because they used URL redirection services, right ? Domains under
management are unlikely to be affected unless the registrant needs a
domain update.


From what I see enom's nameservers are down.

-Hank

enom giving Google a bad name

[Hank Nussbacher]

Many of you might be following the enom weekend fiasco:

https://twitter.com/enomsupport/status/1482621466151571456

https://twitter.com/enomsupport/status/1482707275529678849

https://enomstatus.com/

Thousands of domains have been knocked out.


But I just found out that Google is an enom reseller:

https://help.enom.com/hc/en-us/articles/115005222367-My-Domain-was-registered-through-G-Suite-Google-Apps

"Google handles all of the billing and renewals of your domain name, 
while Enom offers technical support to manage your domain."



So who takes responsibility when a fiasco happens like this: Google or Enom?


-Hank

Re: Contact request AS 6453

[Hank Nussbacher]

On 15/01/2022 10:00, jim deleskie wrote:

Did you try:
https://www.peeringdb.com/net/437

peering-pol...@as6453.net
peering-...@as6453.net
ip...@tatacommunications.com

Regards,
Hank

Have you found anyone.  Not there any more but can probably still find 
someone for you.


-jim

On Thu, Jan 13, 2022, 10:11 AM Drew Weaver > wrote:


Does anyone have a contact for AS 6453 or are there any AS 6453
folks on list?

__ __

Seeing some routing trouble from their customers to the US.

__ __

Thanks,

-Drew

__ __

Re: Cloudflare Abuse Contact

[Hank Nussbacher]

On 07/01/2022 21:35, Töma Gavrichenkov wrote:

I would try n...@cloudflare.com based on:
https://www.peeringdb.com/net/4224

Regards,
Hank


Peace,

On Fri, Jan 7, 2022 at 8:42 PM Mike Hale  wrote:

The abuse email sends an auto-responder that tells you to use the web form.
The web form is centered around their web hosting business; I figured
I'd try general, but you can't submit it without punching in a URL
that is hosted by Cloudflare (and they validate it ... you can't do
https://bogus.site).

What I'm seeing is a ton of abusive DNS traffic that's causing some
issues, and there's no abuse form that works for this scenario.


Most probably, that means that the company doesn't have any counter
abuse process whatsoever for requests like yours, so no matter where
you push that, there won't be any action.

Having said that, the aforementioned form accepts "https[: slash
slash]cloudflare.com" as a valid URL so chances are requests to that
URL are treated in the general sense.

In the meantime, are you sure you'll be able to support your case with
data?  DNS is *mostly* a connection-less protocol, so how do you know
these queries are coming from Cloudflare and not from a spoofed
source?

Lastly, have you tried to block the problematic Cloudflare IP range to
see what would happen?  E.g. does 1.1.1.1 still resolve your domains
then, etc.?

--
Töma

Re: Anyone seeing ping corruption?

[Hank Nussbacher]

On 20/12/2021 04:41, J Doe wrote:



Hi,

Out of curiosity - does anyone know why Google is truncating ICMP 
responses ?


As Google has stated in many forums and I quote:
"Google Public DNS is a Domain Name System service, not an ICMP network 
testing service."


-Hank

Re: Log4j mitigation

[Hank Nussbacher]

On 13/12/2021 15:28, bofh139 wrote:

Now we have the long journey from Java, Ldap, DNS, Birds and Coffee Cups

behind us. Does anyone else have any advice on prevention?


Scan your systems:
https://github.com/logpresso/CVE-2021-44228-Scanner
https://github.com/fullhunt/log4j-scan

-Hank

Re: Latency/Packet Loss on ASR1006

[Hank Nussbacher]

On 07/12/2021 17:32, Blake Hudson wrote:

Suggestion: move this thread to cisco-nsp where you might find more 
assistance.


Regards,
Hank



On 11/26/2021 1:09 PM, Colin Legendre wrote:

Hi,

We have ...

ASR1006  that has following cards...
1 x ESP40
1 x SIP40
4 x SPA-1x10GE-L-V2
1 x 6TGE
1 x RP2

We've been having latency and packet loss during peak periods...

We notice all is good until we reach 50% utilization on output of...

'show platform hardware qfp active datapath utilization summary'

Literally ... 47% good... 48% good... 49% latency to next hop goes 
from 1ms to 15-20ms... 50% we see 1-2% packet-loss and 30-40ms 
latency... 53% we see 60-70ms latency and 8-10% packet loss.


Is this expected... the ESP40 can only really push 20G and then starts 
to have performance issues?





I haven't experienced that across about a dozen ASR 1ks. Though I just 
checked and we are not pushing any of our ESP's over 50% currently (the 
closest we have is an ESP 40 doing 18Gbps). However, I'm pretty sure 
we've pushed older ESPs (5, 10's, and 20's) to ~75% or so in the past.


Given the components you have, I would have expected your router to 
handle 40Gbps input and 40Gbps output. That could either be 40Gbps into 
the 6 port card [and 40Gbps out of the four 1 port cards] or it could be 
40Gbps input that is spread across the 6 port and 1 port cards [that is 
then output across both cards as well].


Despite other comments, I think your components are well matched. The 
only non-obvious thing here is that the 6 port card only has a ~40Gbps 
connection to the backplane so you cannot use all 6 ports at full 
bandwidth. I think this router is well suited to handle 20-30Gbps of 
customer demand doing standard destination based routing (if you're 
doing traffic shaping, NAT, tunnelling, or something else more involved 
than extended ACLs you may need something beefier at those traffic levels).

Re: .bv ccTLD

[Hank Nussbacher]

On 04/12/2021 00:45, Jay R. Ashworth wrote:

My favorite youtuber has just pointed out that Bougainville will separate
formally from Papua New Guinea in 2027, which, surprisingly, is only 5 or 6
years from now.

So I looked up .bv, and of course... it's assigned to Bouvet Island, an
uninhabited island whose political superior says anything that might go in
that TLD will go in .no instead. [Wikipedia]

So, what's the actual status of .bv?  Assigned, or reserved?  And if it
is reserved at the 3166 secretariat level, can they reassign it?

NORID might try to make a case that BV is the common corporate abbreviation
in their political subdivision... but they're not selling those domains now,
so that doesn't seem compelling.

Anyone here got a buddy on the secretariat?  :-)

Cheers,
-- jra


All handling of ccTLDs are handled via the ccNSO of ICANN:
https://ccnso.icann.org/en

For example ccTLD retirement is a multi-year and perhaps multi-decade 
process:

https://ccnso.icann.org/sites/default/files/field-attached/ccpdp3-retirement-vote-report-05aug21-en.pdf
https://community.icann.org/pages/viewpage.action?pageId=64081623=/64081623/166266006/Final%20Report%20ccPDP3%20Retirement%20-%20June%202021.pdf

Regards,
Hank

Re: Google location question

[Hank Nussbacher]

On 23/11/2021 22:31, Chuck Church wrote:

Old issue.  Everyone encounters this at some point:
https://www.iucc.ac.il/en/blog/2021-05-google-geo-location/

You can try reporting it to Google:
https://support.google.com/websearch/workflow/9308722?hl=en
and wait a month or so to see if the issue gets resolved.

Regards,
Hank


NANOG,

     Sorry if this is the wrong forum, but I figured we 
could all use a distraction from IPV4 expansion for a bit.  We’re facing 
a problem with corporate use of browsers and google location services.  
Our users across North and South America seem to be getting location 
info for Rio de Janeiro, BR as of the last few days.  All of our users 
when browsing hit general internet via a NAT pool for IPs out of 
Atlanta, GA.  That hasn’t changed in over a year.  The IP space itself 
and the upstream routers PTR haven’t changed for longer than that.


     For a normal laptop without a GPS chip and using 
default settings, it would seem that something else must be informing 
google of location.  We’ve had a few theories.  I discovered our Cisco 
wireless in RIO office is broadcasting SSID.  The same SSID name we use 
in all offices.  But I believe that google uses BSSID which is MAC 
based.  And would be different between sites, as each have their own 
wireless controller, and certainly different APs.  At least that’s what 
I believe to be true, that google didn’t associate our standard SSID 
name with a physical location.


     I’m curious if anyone has ever dealt with this before.  
I don’t want to blame our workstation folks for something that might be 
a geolocation issue, either based on wi-fi being detected, or a messed 
up geolocation for our IP space.  Is there any place in Google-land 
where you can see exactly why it thinks you’re in a given location?  I 
did see some articles on API usage to view values, but nothing that 
shows you the process when it takes all variables into effect.


Thanks,

Chuck

BGP hujack by AS25478?

[Hank Nussbacher]

Based on my own observation as well as via bgpstream there was a massive 
BGP hijack attempt last night by IHOME-AS iHome LLC, RU (AS 25478).


Lasted about 10 minutes.  Noction?  Finger faddle? Malicious?


Thanks,

Hank

Re: Internet history

[Hank Nussbacher]

On 21/10/2021 21:52, Patrick W. Gilmore wrote:

On Oct 21, 2021, at 2:37 PM, Michael Thomas  wrote:


[changed to a more appropriate subject]

On 10/20/21 3:52 PM, Grant Taylor via NANOG wrote:

On 10/20/21 3:26 PM, Michael Thomas wrote:

Just as an interesting aside if you're interested in the history of networking, 
When Wizards Stayed Up Late is quite elucidating.


+10 to Where Wizards Stay Up Late.

I recently re-acquired (multiple copies of) it.  (Multiple because I wanted the 
same edition that I couldn't locate after multiple moves.)



One of the things about the book was that it finally confirmed for me what I had heard but thought 
might be apocryphal which was that one of my co-workers at Cisco (Charlie Klein) was the first one 
to receive a packet on ARPAnet. I guess it sent an "l" and then immediately crashed. They 
fixed the problem and the next time they got "login:". It also casts shade on another 
early well known person which gives me some amount of schadenfreude.


It was “LO”, and Mr. Kline sent the packets, but you got it essentially right.

Source: https://uclaconnectionlab.org/internet-museum/

The last picture confirms Mr. Kline sent the LO and crashed the WHOLE INTERENT (FSVO “Internet”) just a couple seconds after it started. 


Reminds me of the time the entire Swift network crashed when the capital 
of Ecuador (Quito) was added to the network. :-)


-Hank

Geolocation accuracy

[Hank Nussbacher]

Can anyone recommend a geo-location service with high city accuracy?
Maxmind, for most countries (broadband, which does move) is below 50% 
accuracy (they claim 68% accuracy for USA cities):

https://www.maxmind.com/en/geoip2-city-accuracy-comparison?country==city=excluding

Thanks,

Hank

Re: DNS pulling BGP routes?

[Hank Nussbacher]

On 06/10/2021 22:38, Jon Lewis wrote:

But I just don't understand why this is a good idea at all. Network 
topology is not DNS's bailiwick so using it as a trigger to withdraw 
routes seems


Everything I've seen posted about this (whether from Facebook directly, 
or others) is so vague as to what happened, that I think everyone's just 
making assumptions based on their own experiences or best guesses as to 
what really happened.


Better question is why do we not see any FB netadmins on NANOG?  I'm not 
talking about October 2021 but rather over the past 3-5 years how many 
FB techies have posted here like we see people from Google, Cloudflare, 
Akamai, etc.?


-Hank

Re: Facebook post-mortems...

[Hank Nussbacher]

On 05/10/2021 21:11, Randy Monroe via NANOG wrote:
Updated: 
https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/ 


Lets try to breakdown this "engineering" blog posting:

- "During one of these routine maintenance jobs, a command was issued 
with the intention to assess the availability of global backbone 
capacity, which unintentionally took down all the connections in our 
backbone network"


Can anyone guess as to what command FB issued that would cause them to 
withdraw all those prefixes?


- "it was not possible to access our data centers through our normal 
means because their networks were down, and second, the total loss of 
DNS broke many of the internal tools we’d normally use to investigate 
and resolve outages like this.  Our primary and out-of-band network 
access was down..."


Does this mean that FB acknowledges that the loss of DNS broke their OOB 
access?


-Hank

Re: Facebook post-mortems...

[Hank Nussbacher]

On 05/10/2021 13:17, Hauke Lampe wrote:

On 05.10.21 07:22, Hank Nussbacher wrote:


Thanks for the posting.  How come they couldn't access their routers via
their OOB access?


My speculative guess would be that OOB access to a few outbound-facing
routers per DC does not help much if a configuration error withdraws the
infrastructure prefixes down to the rack level while dedicated OOB to
each RSW would be prohibitive.

https://research.fb.com/wp-content/uploads/2021/03/Running-BGP-in-Data-Centers-at-Scale_final.pdf



Thanks for sharing that article.  But OOB access involves exactly that - 
Out Of Band - meaning one doesn't depend on any infrastructure prefixes 
or DFZ announced prefixes.  OOB access is usually via a local ADSL or 
wireless modem connected to the BFR.  The article does not discuss OOB 
at all.


Regards,
Hank

Re: Facebook post-mortems...

[Hank Nussbacher]

On 05/10/2021 05:53, Patrick W. Gilmore wrote:

Update about the October 4th outage

https://engineering.fb.com/2021/10/04/networking-traffic/outage/



Thanks for the posting.  How come they couldn't access their routers via 
their OOB access?


-Hank

Re: massive facebook outage presently

[Hank Nussbacher]

On 04/10/2021 22:05, Jason Kuehl wrote:

BGP related:
https://twitter.com/SGgrc/status/1445116435731296256
as also related by FB CTO:
https://twitter.com/atoonk/status/1445121351707070468

-Hank

https://twitter.com/disclosetv/status/1445100931947892736?s=20 



On Mon, Oct 4, 2021 at 3:01 PM Tony Wicks > wrote:


Didn't write that part of the automation script and that coder left...

 > I got a mail that Facebook was leaving NLIX. Maybe someone
botched the
 > script so they took down all BGP sessions instead of just NLIX
and now
 > they can't access the equipment to put it back... :-)




--
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.ku...@gmail.com 

FYI: NANOG and ICANN

[Hank Nussbacher]

https://www.icann.org/en/announcements/details/icann-signs-a-memorandum-of-understanding-with-nanog-27-9-2021-en

Regards,
Hank

Re: An update on the AfriNIC situation

[Hank Nussbacher]

On 27/08/2021 18:36, Bill Woodcock wrote:

As many of you are aware, AfriNIC is under legal attack by Heng Lu / “Cloud 
Innovation.”

John Curran just posted an excellent summary of the current state of affairs 
here:


https://teamarin.net/2021/08/27/afrinic-and-the-stability-of-the-internet-number-registry-system/

If, like me, you feel like chipping in a little bit of money to help AfriNIC 
make payroll despite Heng having gotten their bank accounts frozen, some of the 
African ISP associations have put together a fund, which you can donate to here:

https://www.tespok.co.ke/?page_id=14001

It’s an unfortunate situation, but the African Internet community has really 
pulled together to defend themselves, and they’ve got a lot less resources than 
most of us do.

-Bill



Why not just use the RIR Joint Stability Fund which was created for 
exactly these situations before asking for donations?


Details here:
https://www.nro.net/accountability/rir-accountability/joint-rir-stability-fund/

Regards,
Hank

Caveat: The views expressed above are solely my own and do not express 
the views or opinions of my employer

Re: netflow in the core used for surveillance

[Hank Nussbacher]

On 26/08/2021 00:13, Randy Bush wrote:

https://www.vice.com/en/article/jg84yy/data-brokers-netflow-data-team-cymru

used to get dissidents, activists, and journos killed

at, comcast, ... zayo, please tell us you do not do this.

randy



I'm confused.  Quoting from the article:
"In a recent research report on an Israeli spyware vendor called 
Candiru, Citizen Lab thanked Team Cymru.


Thanks to Team Cymru for providing access to their Pure Signal Recon 
product. Their tool’s ability to show Internet traffic telemetry from 
the past three months provided the breakthrough we needed to identify 
the initial victim from Candiru’s infrastructure," the report reads. 
Citizen Lab did not respond to multiple requests for comment."


So Team Cymru helped expose themselves as to getting dissidents, 
activists and journalists killed?


-Hank
Caveat: The views expressed above are solely my own and do not express 
the views or opinions of my employer

Re: Setting sensible max-prefix limits

[Hank Nussbacher]

On 18/08/2021 13:03, Chriztoffer Hansen wrote:

On Wed, 18 Aug 2021 at 11:33, Lars Prehn  wrote:

I guess for long standing peers one could just eyeball it, e.g., current
prefix count + some safety margin. How does that work for new peers?


If you have automation in place. Another approach is to count the
received prefix. Store the counted value in a database. Based on the
avg prefix count over X (time period). Add e.g. 10 - 25 % headroom
over the avg prefix count and use the calculated value as the
max-prefix limit?


That works but all too often people forget to update it.  Set a 
quarterly reminder in your calendar to check max-prefix setting.


-Hank

Re: "Tactical" /24 announcements

[Hank Nussbacher]

On 12/08/2021 17:59, William Herrin wrote:


If you prune the routes from the Routing Information Base instead, for
any widely accepted size (i.e. /24 or shorter netmask) you break the
Internet. 


How does this break the Internet?  I would think it would just result in 
sub-optimal routing (provided there is a covering larger prefix) but 
everything should continue to work.  Clue me in, please.


-Hank

Re: "Tactical" /24 announcements

[Hank Nussbacher]

On 09/08/2021 18:47, Billy Croan wrote:

How does the community feel about using /24 originations in BGP as a
tactical advantage against potential bgp hijackers?

All of our allocations are larger and those prefixes we announce for
clients as well usually are.  But we had a request recently to
originate everything as distinct /24 prefixes, to reduce the effect of
a potential bgp hijack.  It seemed a little bit like a tragedy of the
commons situation.

Is this seen as route table pollution, or a necessary evil in today's world?
How many routers out there today would be affected if everyone did this?
Are there any big networks that drop or penalize announcements like this?



In addition to what everyone else said, announcing /24s will not help 
you one bit since ASNs announce /25s, /26s, /27s, etc. Attached is a 
7800+ line text file sorted by ASN with prefixes being announced that 
are more specific than /24 (only /25+/26+/27 listed).


This is based on http://www.ris.ripe.net/dumps/riswhoisdump.IPv4.gz from 
about a month ago.


That dump lists all the IPv4 prefixes seen in the collective of latest 
RIS table dumps, together with origin AS and number of peers that passed 
the routes to RIS.


So good luck with announcing /24s.

Regards,
Hank


more-specifics-sorted-by-asn.7z
Description: Binary data

Re: Where to get IPv4 block these day

[Hank Nussbacher]

Been a while since I had to deal with NetOps stuff. Was
wondering, where do you go these days to get IPv4 blocks? It
seems like getting assignments is hard due to exhaustion. I have
found some "Auction" sites but it all feels very scammy. Any
info would be appreciated.


-James


https://www.ripe.net/manage-ips-and-asns/resource-transfers-and-mergers/brokers
https://www.ripe.net/manage-ips-and-asns/resource-transfers-and-mergers/listing
https://www.ripe.net/manage-ips-and-asns/resource-transfers-and-mergers/transfer-statistics/within-ripe-ncc-service-region/ipv4-transfer-statistics

-Hank

Re: Global Akamai Outage

[Hank Nussbacher]

On 25/07/2021 09:18, Saku Ytti wrote:

Hey,

Not a critique against Akamai specifically, it applies just the same
to me. Everything seems so complex and fragile.


Complex systems are apt to break and only a very limited set of tier-3 
engineers will understand what needs to be done to fix it.


KISS

-Hank

Re: Global Akamai Outage

[Hank Nussbacher]

On 23/07/2021 09:24, Hank Nussbacher wrote:

From Akamai.  How companies and vendors should report outages:

[07:35 UTC on July 24, 2021] Update:

Root Cause:

This configuration directive was sent as part of preparation for 
independent load balancing control of a forthcoming product. Updates to 
the configuration directive for this load balancing component have 
routinely been made on approximately a weekly basis. (Further changes to 
this configuration channel have been blocked until additional safety 
measures have been implemented, as noted in Corrective and Preventive 
Actions.)


The load balancing configuration directive included a formatting error. 
As a safety measure, the load balancing component disregarded the 
improper configuration and fell back to a minimal configuration. In this 
minimal state, based on a VIP-only configuration, it did not support 
load balancing for Enhanced TLS slots greater than 6145.


The missing load balancing data meant that the Akamai authoritative DNS 
system for the akamaiedge.net zone would not receive any directive for 
how to respond to DNS queries for many Enhanced TLS slots. The 
authoritative DNS system will respond with a SERVFAIL when there is no 
directive, as during localized failures resolvers will retry an 
alternate authority.


The zoning process used for deploying configuration changes to the 
network includes an alert check for potential issues caused by the 
configuration changes. The zoning process did result in alerts during 
the deployment. However, due to how the particular safety check was 
configured, the alerts for this load balancing component did not prevent 
the configuration from continuing to propagate, and did not result in 
escalation to engineering SMEs. The input safety check on the load 
balancing component also did not automatically roll back the change upon 
detecting the error.


Contributing Factors:

    The internal alerting which was specific to the load balancing 
component did not result in blocking the configuration from propagating 
to the network, and did not result in an escalation to the SMEs for the 
component.
    The alert and associated procedure indicating widespread SERVFAILs 
potentially due to issues with mapping systems did not lead to an 
appropriately urgent and timely response.
    The internal alerting which fired and was escalated to SMEs was for 
a separate component which uses the load balancing data. This internal 
alerting initially fired for the Edge DNS system rather than the mapping 
system, which delayed troubleshooting potential issues with the mapping 
system and the load balancing component which had the configuration 
change. Subsequent internal alerts more clearly indicated an issue with 
the mapping system.
    The impact to the Enhanced TLS service affected Akamai staff access 
to internal tools and websites, which delayed escalation of alerts, 
troubleshooting, and especially initiation of the incident process.


Short Term

Completed:

    Akamai completed rolling back the configuration change at 16:44 UTC 
on July 22, 2021.

    Blocked any further changes to the involved configuration channel.
    Other related channels are being reviewed and may be subject to a 
similar block as reviews take place. Channels will be unblocked after 
additional safety measures are assessed and implemented where needed.


In Progress:

    Validate and strengthen the safety checks for the configuration 
deployment zoning process
    Increase the sensitivity and priority of alerting for high rates of 
SERVFAILs.


Long Term

In Progress:

    Reviewing and improving input safety checks for mapping components.
    Auditing critical systems to identify gaps in monitoring and 
alerting, then closing unacceptable gaps.





On 22/07/2021 19:34, Mark Tinka wrote:

https://edgedns.status.akamai.com/

Mark.



[18:30 UTC on July 22, 2021] Update:

Akamai experienced a disruption with our DNS service on July 22, 2021. 
The disruption began at 15:45 UTC and lasted for approximately one 
hour. Affected customer sites were significantly impacted for 
connections that were not established before the incident began.


Our teams identified that a change made in a mapping component was 
causing the issue, and in order to mitigate it we rolled the change 
back at approximately 16:44 UTC. We can confirm this was not a 
cyberattack against Akamai's platform. Immediately following the 
rollback, the platform stabilized and DNS services resumed normal 
operations. At this time the incident is resolved, and we are 
monitoring to ensure that traffic remains stable.

Re: Global Akamai Outage

[Hank Nussbacher]

On 22/07/2021 19:34, Mark Tinka wrote:

https://edgedns.status.akamai.com/

Mark.



[18:30 UTC on July 22, 2021] Update:

Akamai experienced a disruption with our DNS service on July 22, 2021. 
The disruption began at 15:45 UTC and lasted for approximately one hour. 
Affected customer sites were significantly impacted for connections that 
were not established before the incident began.


Our teams identified that a change made in a mapping component was 
causing the issue, and in order to mitigate it we rolled the change back 
at approximately 16:44 UTC. We can confirm this was not a cyberattack 
against Akamai's platform. Immediately following the rollback, the 
platform stabilized and DNS services resumed normal operations. At this 
time the incident is resolved, and we are monitoring to ensure that 
traffic remains stable.

Re: Google Geo Location Issues

[Hank Nussbacher]

On 30/06/2021 17:43, Jason Kuehl wrote:

Once you have access to isp.google.com your problems are far from over. 
 You would assume that they would use whois info to know which prefix 
belongs to your ASN.  That would be wrong.  If you have, for example, a 
multi-homed customer and you provided them with a smaller prefix (/27) 
from within your larger prefix (/19), and since they are multihomed some 
other ASN announces that smaller prefix (/27), Google will see that and 
invalidate any geo-location info you provide for your /19.  Google 
follows the BGP routing table as authoritative for ownership of prefixes.


Regards,
Hank

Caveat: The views expressed above are solely my own and do not express 
the views or opinions of my employer



This is want I'm working on setting up, here is hoping they approve my 
account.


On Wed, Jun 30, 2021 at 10:30 AM Benjamin Hatton > wrote:


If they are your subnets, and you have your own AS (and possibly
enough traffic to Google, I'm not sure what exactly their criteria
is but we were able to make an account with ~2Gbps peak and no
peering relationships) you can create an account with the Google ISP
Portal (isp.google.com ) which has a self
service tool to upload a csv that will update your prefixes in the
Google IP Geolocation database.  Took ~2 weeks after upload for it
to be reflected in live data.

*Ben Hatton*

*Network Engineer | Haefele Connect*
d:(607)589-8000 | b...@haefeleconnect.com 

**


On Wed, Jun 30, 2021 at 10:21 AM Nate Burke mailto:n...@blastcomm.com>> wrote:

Same here, YoutubeTV Geolocation problem, has one of my subnets
in Tulsa instead of Chicago.  Ticket open for 8 months.  Every
reply was that it was 'with engineering' and no ETA.  I just got
a notice from them last week that they're closing the ticket and
sent a survey to fill out to rate the support received.  I told
them that the issue was still not resolved and never heard back
from them.

On 6/30/2021 8:55 AM, Josh Luthman wrote:

Been months since I was told they'd get it fixed. To be fair
they did say they weren't sure on how long it would take.  I
feel like I've been forgotten about.

Josh Luthman
24/7 Help Desk: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373


On Wed, Jun 30, 2021 at 9:18 AM Mike Hammett mailto:na...@ics-il.net>> wrote:

I've discovered that if you *CAN* get a Google ISP
account, you can manage it all there.

If you can't, well, you're up shit creek without a paddle.



-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com 

Midwest-IX
http://www.midwest-ix.com 



*From: *"Jason Kuehl" mailto:jason.w.ku...@gmail.com>>
*To: *"NANOG" mailto:nanog@nanog.org>>
*Sent: *Tuesday, June 29, 2021 6:25:06 PM
*Subject: *Google Geo Location Issues

I'm looking for a contact, email, number, smoke signals
for someone at Google I can talk to on geolocation issue.
For some reason Google has labeled our IP ranges
as Belarus when we're located in the states. If anyone can
point me at any contact I would be really happy..

.

-- 
Sincerely,


Jason W Kuehl
Cell 920-419-8983
jason.w.ku...@gmail.com 





--
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.ku...@gmail.com 

Re: Google Geo Location Issues

[Hank Nussbacher]

On 30/06/2021 02:25, Jason Kuehl wrote:

Good luck.  I had a case where Israeli users were located in Iceland. 
Took me well over a month to get it resolved.  You can read about it in 
my blog entry:

https://www.iucc.ac.il/en/blog/2021-05-google-geo-location/

Just heard of two more incidents this week where other Israeli users 
from different ISPs are seen as being in Belgium and the other is seen 
as being in Greece.


I have told those 2 new incidents to try:
https://support.google.com/websearch/workflow/9308722?hl=en
and wait a month to see if the issue gets resolved (as stated on that 
Google page).


Clearly some process since about March 2021 is broken inside Google 
geo-location land.


Regards,
Hank

Caveat: The views expressed above are solely my own and do not express 
the views or opinions of my employer



I'm looking for a contact, email, number, smoke signals for someone at 
Google I can talk to on geolocation issue. For some reason Google has 
labeled our IP ranges as Belarus when we're located in the states. If 
anyone can point me at any contact I would be really happy..


.

--
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.ku...@gmail.com 

Re: shadowserver.org

[Hank Nussbacher]

What is the difference between shodan.io and shadowserver.org ? Jean

Just those 2?  Greynoise maps them all.  See an old preso from 2018:
https://www.slideshare.net/andrewwantsyou/identifying-and-correlating-internetwide-scan-traffic-to-newsworthy-security-events
See slide 7 for a 4 year old list which has only grown :-)

-Hank

Re: shadowserver.org

[Hank Nussbacher]

On 28/06/2021 06:19, Scott Aldrich wrote:

Anyone have an idea how to get HE/ShadowServer,org servers to stop
attempting to penetrate the comcast drop at my house?

Their website claims altruism.. but my logs dont support that claim.

Scott



Scott,

Did you look at:
https://www.shadowserver.org/what-we-do/network-reporting/dns-open-resolvers-report/
https://scan.shadowserver.org/dns/

If you still think they are penetrating you see their section of 
blocklisting:


To be removed from this set of scanning you will need to send an email 
to dnsscan [at] shadowserver [dot] org with the specific CIDR's that you 
would like to have removed. You will have to be the verifiable owner of 
these CIDR's and be able to prove that fact. Any address space that is 
blocklisted will be publicly available here: 
https://scan.shadowserver.org/dns/exclude.html


Regards,
Hank

Re: ROA coverage info

[Hank Nussbacher]

On 24/08/2020 17:49, Rayhaan Jaufeerally (NANOG) wrote:
There's also this site run by NIST: https://rpki-monitor.antd.nist.gov/ 
 which contains further breakdowns


Anyone know why https://rpki-monitor.antd.nist.gov/ is down?

Thanks,
Hank

Re: Google uploading your plain text passwords

[Hank Nussbacher]

On 12/06/2021 08:31, Damian Menscher via NANOG wrote:



The Chrome password manager is convenient, and the sync can be 
incredibly handy (I can sign into stuff on different computers or even 
my phone without needing to copy over the passwords), but you might 
consider leaving your highest-value passwords out of that system, or 
really any system.  Personally, my financial passwords are not known by 
Chrome, myself, or even my password manager.  (Yes, you heard that right 
-- no single entity knows the passwords.  How?  By using a simple 
secret-splitting scheme -- I memorize part of the password, and my 
password manager stores the rest.)


Or:
https://doubleoctopus.com/

-Hank



Damian

Re: Google IP Geolocation

[Hank Nussbacher]

On 22/04/2021 11:36, Hank Nussbacher wrote:

Jared wrote earlier:

I've had a similar issue in the past trying to get ready to peer with them. I wanted portal access to look at things. I may yet post a geofeed file just because. 

(I was also rejected a portal account, didn't escalate to friends at google because I know my traffic is under a gig for now). 

- Jared 


Don't bother.  We have Google ISP Portal access.  We updated our 
geo-location feed file as they requested:

http://noc.ilan.net.il/GGC/iucc-geo-feed-for-google.csv
15 of 16 prefixes were processed on April 30.  Till today all our users 
in Israel are still located in Iceland and have now started to learn 
Icelandic in order to complete Google pages.


Regards,
Hank



The issues that others had earlier this month just hit us this morning.

Users in Israel (132.74.0.0/15) trying to access Google.com or 
Youtube.com appear as coming from Iceland (see screenshot).


Change happened overnight.  Someone internally in Google's geo-location 
group typo'ed Israel as Iceland.


We have GGC access and Google ISP Portal access but why should we have 
to change the geolocation which worked well since forever just because 
someone internally messed up?



Regards,

Hank

Re: Google IP Geolocation

[Hank Nussbacher]

On 22/04/2021 11:36, Hank Nussbacher wrote:

The issues that others had earlier this month just hit us this morning.

Users in Israel (132.74.0.0/15) trying to access Google.com or 
Youtube.com appear as coming from Iceland (see screenshot).


Change happened overnight.  Someone internally in Google's geo-location 
group typo'ed Israel as Iceland.


We have GGC access and Google ISP Portal access but why should we have 
to change the geolocation which worked well since forever just because 
someone internally messed up?



Regards,

Hank




Now I am hearing that other universities in Israel have seen this issue 
since as early as April 4.


-Hank

Re: Google IP Geolocation

[Hank Nussbacher]

The issues that others had earlier this month just hit us this morning.

Users in Israel (132.74.0.0/15) trying to access Google.com or 
Youtube.com appear as coming from Iceland (see screenshot).


Change happened overnight.  Someone internally in Google's geo-location 
group typo'ed Israel as Iceland.


We have GGC access and Google ISP Portal access but why should we have 
to change the geolocation which worked well since forever just because 
someone internally messed up?



Regards,

Hank

Re: BGP and The zero window edge

[Hank Nussbacher]

On 22/04/2021 02:24, Job Snijders via NANOG wrote:

On Wed, Apr 21, 2021 at 09:22:57PM +, Jakob Heitz (jheitz) wrote:

I'd like to get some data on what actually happened in the real cases
and analyze it.

[snip]

TCP zero window is possible, but many other things could
cause it too.


Indeed. There could be a number of reasons that caused it.

Switchings away from TCP win=0 towards "Zombie Routes":

*RIGHT NOW* (at the moment of writing), there are a number of zombie
route visible in the IPv6 Default-Free Zone:

One example is 
http://lg.ring.nlnog.net/prefix_detail/lg01/ipv6?q=2a0b:6b86:d15::/48

 2a0b:6b86:d15::/48 via:
 BGP.as_path: 204092 57199 35280 6939 42615 42615 212232
 BGP.as_path: 208627 207910 57199 35280 6939 42615 42615 212232
 BGP.as_path: 208627 207910 57199 35280 6939 42615 42615 212232
 (first announced April 15th, last withdrawn April 15th, 2021)
 
Another one is http://lg.ring.nlnog.net/prefix_detail/lg01/ipv6?q=2a0b:6b86:d24::/48


 2a0b:6b86:d24::/48 via:
 BGP.as_path: 201701 9002 6939 42615 212232
 BGP.as_path: 34927 9002 6939 42615 212232
 BGP.as_path: 207960 34927 9002 6939 42615 212232
 BGP.as_path: 44103 50673 9002 6939 42615 212232
 BGP.as_path: 208627 207910 34927 9002 6939 42615 212232
 BGP.as_path: 3280 34927 9002 6939 42615 212232
 BGP.as_path: 206628 34927 9002 6939 42615 212232
 BGP.as_path: 208627 207910 34927 9002 6939 42615 212232
 (first announced March 24th, last withdrawn March 24th, 2021)

Just now, I literally rebooted the BGP speaker behind lg.ring.nlnog.net
to make ensure that those routes are not stuck in the BGP looking glass
itself.

2a0b:6b86:d24::/48 was first announced on March 24th, 2021, and
withdrawn at the end of March 24th, 2021 by the originator, and now
almost a month later, this prefix still is visible in the default-free
zone despite WITHDRAW messages having been sent and the AS 212232
operator confirming they are not announcing that IP prefix anywhere.

I checked the AS 6939 Looking glass, but the d24::/48 route is not
visible in the http://lg.he.net/ web interface. This leads me to believe
the the route got stuck somewhere along way in either of 201701, 204092,
206628, 207910, 207960, 208627, 3280, 34927, 35280, 44103, 50673, 57199,
and/or 9002.

This implies indeed might be multiple reasons a BGP route gets stuck
('stuck' as in - a WITHDRAW was not generated, or ignored). Perhaps on
any one of these edges there is a very high Out Queue for one reason or
another:

 34927 9002
 206628 34927
 44103 50673
 207960 34927
 3280 34927
 9002 6939
 201701 9002
 208627 207910

I'm not sure all the these sightings of stuck routes can be pinpointed
to one specific BGP vendor (or one bug).


I would guess that all the stuck route sightings manifest from one 
undiscovered TCP library bug that some BGP vendors are all commonly using.


-Hank




Kind regards,

Job

Re: Perhaps it's time to think about enhancements to the NANOG list...?

[Hank Nussbacher]

On 20/03/2021 21:34, Stan Barber wrote:

+1

-Hank


+1 from the peanut gallery

On Sat, Mar 20, 2021 at 2:30 PM Allen Kitchen 
mailto:allenmckinleykitc...@gmail.com>> 
wrote:




On Sat, Mar 20, 2021 at 2:07 PM Randy Bush mailto:ra...@psg.com>> wrote:

i do not find the volume or diversity on the nanog list problematic.
in fact, i suspect its diversity and openness are major factors in
it being the de facto global anything-ops list.  perhaps we do not
need to fix that.

randy


+1 (or as much more as I can be credited for)

..Allen

RPKI invalid logs?

[Hank Nussbacher]

Is there a place where one can examine RPKI invalid logs for a specific 
date & time or even better logs showing those that dropped RPKI invalid 
announcements?



Thanks,

Hank

Re: bgp.he.net?

[Hank Nussbacher]

  
  
On 18/02/2021 15:08, Hank Nussbacher
  wrote:


  
  
  Is it down?
  
  
  -Hank
  

Back up.


-Hank

  

bgp.he.net?

[Hank Nussbacher]

  
  
Is it down?


-Hank

  

Re: Problems with newish IP block assignment issues from ARIN

[Hank Nussbacher]

  
  
On 08/02/2021 22:14, Justin Wilson
  (Lists) wrote:


  
It acts like the IP block was blacklisted at some point and got on some bad lists but I don’t want ti limit myself to that theory.  I have opened up a ticket with ARIN asking for any guidance.  Has anyone ran into this with new space assigned? Any tools, sites, etc. I can use to do further troubleshooting.  The IP block does not appear to have any blacklisted IPs according to MX toolbox, and some others.



Try:
http://multirbl.valli.org/lookup/


-Hank




  

The block in question is 134.195.44.0/22.  It has been RPKI certified and has IRR entries.

Thanks in advance


Justin Wilson
j...@mtin.net

—
https://j2sw.com - All things jsw (AS209109)
https://blog.j2sw.com - Podcast and Blog





  

Re: RTBH and Flowspec Measurements - Stop guessing when the attack will over

[Hank Nussbacher]

  
  
On 02/02/2021 19:08, Douglas Fischer
  wrote:


  
  
Well... That is a point
  of view!
  And I must respect that.
  
  Against this position, there are several companies, including
  some tier 1, that sells this as an $extra$.
  
  About the "Please break me at my earliest inconvenience."
  part:
  I believe that the same type of prefix filtering that
  applies to Downstream-BGP-Routes applies to RTBH and Flowspec.
  So, exactly as in common BGP Route-Filtering:
  - If the network operator does it correctly, it should work
  correctly.
  - If the network operator deals with that without the needed
  skills, expertise, attention+devotion, wrong things will come
  up.

  

You forgot to mention software bugs:
https://kb.juniper.net/InfoCenter/index?page=content=JSA11101=SIRT_1=LIST


Note what Juniper states:
Workaround:
  There are no viable workarounds for this issue



-Hank




  

  
  But, this still does not helps to find a solution do an
  organization A that sends some flowspec our RTBH to
  organization B(presuming organization B will accept that), 
  and organization B do some reports of what is match with that
  flowspec or RTBH.
  
  That, in my opinion, is the only way to stop guessing how long
  will an attack will last, and start to define the end of a
  flowspec/RTBH action based on real information related to
  that.
  I want to close the feedback loop.



  
  
  
Em ter., 2 de fev. de 2021 às
  13:07, Tom Beecher  escreveu:


  Personally, I would absolutely, positively,
never ever under any circumstances provide access to a 3rd
party company to push a FlowSpec rule or trigger RTBH on my
networks. No way.  You would be handing over a nuclear
trigger and saying "Please break me at my earliest
inconvenience." 
  
  
On Tue, Feb 2, 2021 at
  5:56 AM Douglas Fischer 
  wrote:


  
OK, but do you
  know any company the sells de Flowspec as a service,
  in the way that the Attack Identifications are not
  made by their equipment, just receiving de
  BGP-FlowSpec and applying that rules on that
  equipments... And even then give back to the customer
  some way to access those statistics?
  
  I just know one or two that do that, and(sadly) they
  do it on fancy web reports or PDFs.
  Without any chance of using that as structured data do
  feedback the anomaly detection tools to determine if
  already it is the time to remove that Flowsperc rule.
  
  What I'm looking for is something like:
  A) XML/JSON/CSV files streamed to my equipment from
  the Flowspec Upstream Equipments saying "Heepend that,
  that, and that." Almost in real time.
  B) NetFlow/IPFIX/SFlow streamed to my equipment from
  the Upstream Equipment, restricted to the
  DST-Address that matches to the IP blocks that were
  involved to the Flowspec or RTBH that I Annouced to
  then.
  C) Any other idea that does the job of gives me the
  visibility of what is happening with FlowSpec-rules,
  or RTBH on theyr network.
  



  
  
  
Em seg., 1 de fev. de
  2021 às 22:07, Dobbins, Roland 
  escreveu:


  



  On Feb 2, 2021, at 00:34,
Douglas Fischer 
wrote:
  


  

  
Or even know if already there is a solution
to that and I'm trying to invent the wheel.
  


  

  



   

Re: Centurylink having a bad morning?

[Hank Nussbacher]

  
  
On 30/08/2020 20:08, Baldur Norddahl
  wrote:


https://blog.cloudflare.com/analysis-of-todays-centurylink-level-3-outage/


Sounds like Flowspec possibly blocking
  tcp/179 might be the cause.


But that is Cloudflare speculation.


Regards,
  Hank
Caveat: The views expressed above are
  solely my own and do not express the views or opinions of my
  employer 




  
  
An outage is what it is. I am not worried about outages. We
  have multiple transits to deal with that.


It is the keep announcing prefixes after withdrawal from
  peers and customers that is the huge problem here. That is
  killing all the effort and money I put into having redundancy.
  It is sabotage of my network after I cut the ties. I do not
  want to be a customer at an outlet who has a system that will
  do that. Luckily we do not currently have a contract and now
  they will have to convince me it is safe for me to make a
  contract with them. If that is impossible I guess I won't be
  getting a contract with them.


But I disagree in that it would be impossible. They need to
  make a good report telling exactly what went wrong and how
  they changed the design, so something like this can not happen
  again. The basic design of BGP is such that this should not
  happen easily if at all. They did something unwise. Did they
  make a route reflector based on a database or something?


Regards,


Baldur


  On Sun, Aug 30, 2020 at 5:13
PM Mike Bolitho 
wrote:
  
  
Exactly. And asking that they somehow prove
  this won't happen again is impossible.
  
  - Mike Bolitho



  On Sun, Aug 30, 2020,
8:10 AM Drew Weaver 
wrote:
  
  

  
I’m not defending them but I am
  sure it isn’t intentional.
 

  From: NANOG
thenap@nanog.org>
On Behalf Of Baldur Norddahl
Sent: Sunday, August 30, 2020 9:28 AM
To: nanog@nanog.org
Subject: Re: Centurylink having a bad
morning?

 

  How is that acceptable
behaviour? I shall remember never to make a
contract with these guys until they can prove
that they won't advertise my prefixes after I
pull them. Under any circumstances. 

 

  
søn. 30. aug. 2020 15.14
  skrev Joseph Jenkins :
  
  

  Finally got through on
their support line and spoke to level1. The
only thing the tech could say was it was an
issue with BGP route reflectors and it
started about 3am(pacific). They were still
trying to isolate the issue. I've tried
failing over my circuits and no go, the
traffic just dies as L3 won't stop
advertising my routes.

 

  
On Sun, Aug 30, 2020 at
  5:21 AM Drew Weaver via NANOG 
  wrote:
  
  

  
Hello,
 
Woke up this
  morning to a bunch of reports of
  issues with connectivity had to shut
  down some Level3/CTL connections to
  get it to return to normal.
 
As of right now
  their support portal won’t load:
  https://www.centurylink.com/business/login/

Re: Centurylink having a bad morning? [EXTERNAL]

[Hank Nussbacher]

  
  
On 30/08/2020 18:22, Joseph Jenkins
  wrote:


  
  Well at least it looks like the issue is starting
to resolve  and stuff is coming back up.
  
  
On Sun, Aug 30, 2020 at 8:21
  AM Matt Hoppes 
  wrote:

Is
  this what happens when your entire network is database driven?

  



See:
https://status.ctl.io/
and specifically:
https://status.ctl.io/history/f19a0555-abbd-4038-91cb-b55a7645c1f5


Regards,
  Hank
Caveat: The views expressed above are solely my own and do not
  express the views or opinions of my employer 

  

Bottlenecks and link upgrades

[Hank Nussbacher]

  
  
At what point do commercial ISPs upgrade links in their backbone
  as well as peering and transit links that are congested?  At 80%
  capacity?  90%?  95%?  


Thanks,
  Hank


Caveat: The views expressed above are solely my own and do not
  express the views or opinions of my employer 

  

ISPs are hit hardest by COVID-19 disruption

[Hank Nussbacher]

  
  
https://betanews.com/2020/08/04/isps-covid-19-disruption/


Really?


-Hank


Caveat: The views expressed above are solely my own and do not
  express the views or opinions of my employer 

  

Re: BGP route hijack by AS10990

[Hank Nussbacher]

  
  
On 01/08/2020 00:50, Mark Tinka wrote:


  

On 31/Jul/20 23:38, Sabri Berisha wrote:


  
Kudos to Telia for admitting their mistakes, and fixing their processes.

  
  
Considering Telia's scope and "experience", that is one thing. But for
the general good of the Internet, the number of intended or
unintentional route hijacks in recent years, and all the noise that
rises on this and other lists each time we have such incidents (this
won't be the last), Telia should not have waited to be called out in
order to get this fixed.

Do we know if they are fixing this on just this customer of theirs, or
all their customers? I know this has been their filtering policy with us
(SEACOM) since 2014, as I pointed out earlier today. There has not been
a shortage of similar incidents between now and then, where the
community has consistently called for more deliberate and effective
route filtering across inter-AS arrangements.




AS  level filtering is easy.  IP prefix level filtering is hard. 
  Especially when you are in the top 200:
https://asrank.caida.org/


That being said, and due to these BGP "polluters" constantly
  doing the same thing, wouldn't an easy fix be to use the
  max-prefix/prefix-limit option:
https://www.cisco.com/c/en/us/support/docs/ip/border-gateway-protocol-bgp/25160-bgp-maximum-prefix.html
https://www.juniper.net/documentation/en_US/junos/topics/reference/configuration-statement/prefix-limit-edit-protocols-bgp.html


For every BGP peer,  the ISP determines what the current
  max-prefix currently is.  Then add in 2% and set the max-prefix. 

An errant BGP polluter would then only have limited damage to the
  Internet routing table.
Not the greatest solution, but easy to implement via a one line
  change on every BGP peer.


Smaller ISPs can easily do it on their 10 BGP peers so as to
  limit damage as to what they will hear from their neighbors.



-Hank


Caveat: The views expressed above are solely my own and do not
  express the views or opinions of my employer 



  

Re: BGP route hijack by AS10990

[Hank Nussbacher]

  
  
On 30/07/2020 20:32, Sadiq Saif wrote:


  On Thu, 30 Jul 2020, at 13:09, Patrick Schultz wrote:

  
so, bgp optimizers... again?

-- 
Patrick

  
  
More like shame on Telia for not filtering properly.




But wait - MANRS indicates that Telia does everything right:
https://www.manrs.org/isps/participants/?gv_search=telia=any
How can that be?


-Hank
Caveat: The views expressed above are solely my own and do not
  express the views or opinions of my employer 

  

Re: BGP route hijack by AS10990

[Hank Nussbacher]

  
  
On 30/07/2020 05:46, Clinton Work
  wrote:


See:
https://bgpstream.com/event/245264
https://bgpstream.com/event/245265


-Hank


Caveat: The views expressed above are
  solely my own and do not express the views or opinions of my
  employer 




  We saw a bunch of our IP blocks hijacked by AS10990 from 19:15 MDT until 20:23 MDT.   Anybody else have problems with that. 

ASpath:  1299 7219 10990

50.92.0.0/17	AS10990
198.166.0.0/17	 AS10990
198.166.128.0/17	AS10990
162.157.128.0/17	AS10990
162.157.0.0/17	AS10990
50.92.128.0/17	AS10990



--
Clinton Work
Airdrie, AB




  

Re: Survey on the use of IP blacklists for threat mitigation

[Hank Nussbacher]

  
  
On 16/06/2020 22:08, J. Hellenthal via
  NANOG wrote:


This issue was raised in Reddit and
  Github:
https://www.reddit.com/r/sysadmin/comments/h149em/calls_to_replace_blacklist_whitelist_black_hat/
https://www.techspot.com/news/85631-github-replace-terms-whitelist-blacklist-masterslave-racially-insensitive.html
and is not limited to the term
  "blacklist".


-Hank


Note: the views expressed above are my
  own and do not necessarily reflect the views of my employer



  
  Guess we all better start rewriting all of the documentation out
  there because some PC marketing snowflake wants to get extra
  brownie points and attention for classifying a color in RGB into a
  racial divide for which it never originated.
  
  
  blacklists are not always deny/block/disallow and conformed
of things that allow you to take actions whatever your choosing
upon their contents and your policies.
  
  
  What’s next ? redlisting ? Don’t offend the Russians ... blue
? Don’t want to offend the police ...
  
  
  Leave this crap off the list, it’s not helping anyone.
  
  
  SMH
  


  -- 
   J.
  Hellenthal
  

  The
fact that there's a highway to Hell but only a stairway to
Heaven says a lot about anticipated traffic volume.

  On Jun 16, 2020, at 13:27, Ryan Landry
 wrote:

  


  
In kind, I'd like to encourage the use of
  terms like permit/accept list or deny/block list.
  
  
  Respectfully,
  -Ryan



  On Tue, Jun 16, 2020 at
11:33 AM Rachee Singh 
wrote:
  
  
Hi NANOG community,
  
  We are a group of researchers studying the use of IP
  blacklists as a mechanism to mitigate security threats
  -- particularly over the IPv6 Internet. We would like
  to understand if and how you use IP blacklists to
  secure your networks. Please consider taking our short
  survey: https://forms.gle/ZEsxyiBivJAfLF7e6
  
  The survey will be anonymous unless you choose to
  identify yourself.
  
  Thanks,
  Rachee
  UMass Amherst

  

  

  


  

IBM Cloud global outage caused by "incorrect" BGP routing

[Hank Nussbacher]

  
  
https://www.bleepingcomputer.com/news/technology/ibm-cloud-global-outage-caused-by-incorrect-bgp-routing/


-Hank


Note: the views expressed above are my own and do not necessarily
  reflect the views of my employer

  

Spike in traffic to Google caches?

[Hank Nussbacher]

Did anyone notice a huge jump in traffic today between 11:30-11:40 (GMT) 
directed at Google and Akamai caches coming from Amazon and Google?

Gaming updates?

Thanks,
Hank
Caveat: The views expressed above are solely my own and do not express 
the views or opinions of my employer

Re: Backhoe season?

[Hank Nussbacher]

On 26/03/2020 20:02, Aaron Gould wrote:

Numerous gov'ts and municipalities, which had planned constructions jobs 
but postponed them to the summer due to heavy traffic volume, have 
started to implement all those construction jobs, which includes backhoes.


-Hank


I heard, and am seeing that construction type jobs don't seem to be affected 
much with the virus shutdown.  I mean I see guys building homes and working on 
roads all around me...  furthermore, we've heard of a couple fiber cuts that 
have brought portions of our network down a couple times in the last week or so.

-Aaron

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of William Herrin
Sent: Thursday, March 26, 2020 12:57 PM
To: nanog@nanog.org
Subject: Backhoe season?

Howdy,

With so much work shut down, I'm curious how backhoe season is shaping
up this year? How do the circuit and fiber outage numbers look?

Regards,
Bill Herrin

Re: Gmail email blocking is off the rails (again)

[Hank Nussbacher]

On 04/12/2019 05:04, Matthew Pounsett wrote:

Cute way to promote Google Groups over Mailman.  Gotta give 'em credit 
for being creative :-)


-Hank



For some reason Gmail has started blocking mailman administrative 
emails to someone who's an admin on a list I host.  Their SMTP 552 
error message points to 
, which implies the 
"problem" is the URLs in the email, but is otherwise completely unhelpful.


If anyone here has any pull with Gmail postmasters, could you please 
suggest to them that they whitelist messages that are as consistent 
and well-known as mailman's admin and moderator messages?

Re: "Using Cloud Resources to Dramatically Improve Internet Routing"

[Hank Nussbacher]

On 07/10/2019 17:42, Stephane Bortzmeyer wrote:

On Fri, Oct 04, 2019 at 03:52:26PM -0400,
  Phil Pishioneri  wrote
  a message of 9 lines which said:


Using Cloud Resources to Dramatically Improve Internet Routing
UMass Amherst researchers to use cloud-based ‘logically centralized
control’

Executive summary: it's SDN for BGP. Centralizing Internet routing,
what could go wrong? (As the authors say, "One reason is there is no
single entity that has a big picture of what is going on, no
manager". I wonder who will be Internet's manager.)


Centralized Internet routing - sounds like DoH for BGP.

What could possibly go wrong?!

-Hank

Re: Art and Tech is madness

[Hank Nussbacher]

On 05/09/2019 08:09, Kasper Adel wrote:

No.  This is art & tech from 12 years ago:
https://www.youtube.com/watch?v=_y36fG2Oba0

-Hank

In SPRING a time when segment and routing had no mismatch, a time when 
isis and ospf ate a forbidden encap, all they had to do was forward 
bgp like its hot, but crazy flapping doesnt leave any real LDP without 
some real FSM check, My dynamic unnumbered neighbor.



Suddenly, Out of order, an AS is overridden, we see frames dropping, 
we sniff a bit and it turns out, sfps are burning, we are in a place 
right now where ping and pong are jittery, their latency is tested, 
they cant strengthen their icmp bond with a warm bfd message, how can 
they keep everyone in ACK, safe from teardown and dampening, with this 
kind of ixp relationship??! but oh admin, we know forwarding works in 
its own mysterious ways. We are left with two non rfc compliant 
scavengers, bastard 802.1ah fools in a leaky yet shaped, buffer 
display of some runts and nimbles, and a giant too.


They start their life of a packet, leaving one interface to a 
neighbor, from an adjacency to a peer, an endless loop, its a prefix 
hijack, but as they move from one stack to another, finding their way 
through a tunnel of memory failures and RMAs, one hell of an LSP ride, 
through firewall horrors and MTU mismatches, leaving behind, a sea of 
syslog messages and snmp alarms. Anyway, Their ttl expired and one 
funny access list abruptly denies them life, sending them to Null0, 
where they can be peacefully discarded.



Thats what tech does to yeh

Looking for Cloudfront clue

[Hank Nussbacher]

Can someone with routing/BGP/peering clue in AWS's Cloudfront, please 
contact me offlist?


Thanks,

Hank

Re: Mx204 alternative

[Hank Nussbacher]

On 02/09/2019 11:16, Mark Tinka wrote:


On 8/Aug/19 05:33, Brandon Martin wrote:

  


MX204 is a very nice pizza box router for service providers.  I'm not
aware of anything quite like it in terms of having a mature control
plane.  I like the JunOS config language better than Cisco-style that
most other folks use.

The MX204 is pretty hard to beat. It fits well as a peering/transit
router, as well as a Metro-E router where you need a 100Gbps ring to
carry 10Gbps customers, as well as downstream cheaper routers that will
do sub-10Gbps quite nicely.

That said, at least for the Metro, I still believe a lighter version of
the MX204, with dense 1Gbps capability, is still needed. Been asking
since 2007.

Mark.


What about handling LAG on 1Gb/sec links?  That is a major showstopper 
if indeed it is missing:


https://www.juniper.net/documentation/en_US/junos/topics/reference/configuration-statement/speed-gigether-options.html
•    On MX10003 and MX204 routers, rate selectability at PIC 
level and port level does not support 1-Gbps speed.
•    On MX10003 and MX204 routers, the interface name prefix 
must be xe.
•    On MX10003 and MX204 routers, even after configuring 1-Gbps 
speed, the protocol continues to advertise the bandwidth as 10-Gigabit 
Ethernet.
•    On MX10003 and MX204 routers, Link Aggregation Group (LAG) 
is supported on 10-Gbps speed only. It is not supported on 1-Gbps speed.


-Hank

Re: Corporate Identity Theft: Azuki, LLC -- AS13389, 216.179.128.0/17

[Hank Nussbacher]

On 15/08/2019 06:16, Ronald F. Guilmette wrote:

- If the resource owner is no where to be found, why should we as a
community care?

I'm so glad you asked.


Regardless, in -either- the case where no heir can be found -or- in the
case where the rightful heir is either just too dumb or just too lazy
to take the minimal steps necessary to reclaim the property (and/or before
this has ocurred) the community should care because the kind of people who
either steal or squat on IPv4 blocks are, almost without exception, not the
kind of people who anybody sane wants to be accepting packets from, let
alone peering with.  There is, in my opinion and experience, a high
degree of correlation between skulduggery with respect to -obtaining-
(illicitly) IPv4 address blocks and using those addresses in a manner
which is not at all conducive to the general welfare of the Internet or
its users.


So if the rightful is apathetic, then won't these new "malicious blocks" 
just end up in numerous blacklists and all the illegal activity being 
performed from those usurped blocks will just be blocked in the end?  
Since the RIRs won't do much(as much as we have tried) why not just 
leave it be (as much as it may hurt to do that) and let the bad blocks 
just become part of the blacklist sludgepool?



Report it on some webpage and call it "Internet
Resources stolen", document every incident as you do via email, send a
copy to the appropriate RIR and upstream ISP allowing the hijack in
question to show that you did the appropriate effort and we can then
move on.

I can and will stop posting here, and go off an blog about this stuff
instead, if the consensus is that I'm utterly off-topic or utterly
uninteresting and useless.  But a few folks have told me they find
this stuff interesting, and it has operational significance, I think.
So for now, at least, I'd like to continue to share here.


Suggestion: post here a link to your new blog for every incident you 
find.  State here something like "/22 stolen from  registered in 
country aaa by yyy located in country bbb".  Those that are interested 
will click on the link and I suggest you allow comments on every blog 
post so that people can respond and comment.


Regards,

Hank

Re: RPKI adoption

[Hank Nussbacher]

On 14/08/2019 06:24, John Curran wrote:


When you did that Whois look up at the ARIN website, you did agree to 
terms of use for the Whois service which contains indemnification 
provisions and are legally enforceable. 



If you instead used a command line interface (e.g. "whois -h 
whois.arin.net  …”), then you received output 
from ARIN’s Whois server along with notice of the applicable terms of 
service…  I would observe that continued use at that point has been 
held to indicate agreement on your part [ref: Register.com 
, Inc. v. Verio, Inc., 356 F.3d 393 (2d Cir. 2004)]


Thanks,
/John

John Curran
President and CEO
American Registry for Internet Numbers

Just like to add kudos to John for being open and responsive on this 
list and other lists to numerous issues and questions in regards to 
ARIN.  Not many CEOs are willing or able to respond as you do.


Thanks for your time and effort,

-Hank

Re: Corporate Identity Theft: Azuki, LLC -- AS13389, 216.179.128.0/17

[Hank Nussbacher]

On 13/08/2019 22:17, Ronald F. Guilmette wrote:

Just as an observer to your long resource theft postings:
- Do you attempt to contact directly the organization or person who have 
had their resource taken over?

- Do they care or are they apathetic?
- If the resource owner is no where to be found, why should we as a 
community care?  Report it on some webpage and call it "Internet 
Resources stolen", document every incident as you do via email, send a 
copy to the appropriate RIR and upstream ISP allowing the hijack in 
question to show that you did the appropriate effort and we can then 
move on.


Regards,
Hank

Re: Bgpmon alternatives?

[Hank Nussbacher]

On 18/07/2019 08:44, Töma Gavrichenkov wrote:

On Thu, Jul 18, 2019 at 3:16 AM TJ Trout  wrote:

Anyone know of a hosted alternative to bgpmon? I'm testing
Qrator but I can't determine if it will notify in real-time of a
prefix hijack?

Qrator guy there.
Real-time notifications are there but are only available on a
commercial basis, because basically real time is expensive to compute.
The rest is free.

--
Töma

What about once a day notification of BGP hijack?  Is that also 
expensive to compute?  I have an account and cannot find any 
documentation of realtime notifications nor its cost.  All I found was 
this - https://qrator.net/en/pricing .   Can you send links to the BGP 
hijack notification service and its cost?


Thanks,
-Hank

Re: Performance metrics used in commercial BGP route optimizers

[Hank Nussbacher]

On 16/07/2019 20:41, Job Snijders wrote:
On Tue, Jul 16, 2019 at 3:33 PM Mike Hammett > wrote:


More like do whatever you want in your own house as long as you
don't infringe upon others.


That's where the rub is; when using "BGP optimisers" to influence 
public Internet routing, you cannot guarantee you won't infringe upon 
others.


The argument against route optimizers (assuming appropriate
ingress\egress filters) is a religious one and should be treated
as such.

There is a difference between BGP optimizers and route optimizers.  When 
was the last time you heard a complain about Akamai screwing up the 
global routing table over the past 12 years:


https://www.akamai.com/us/en/about/news/press/2007-press/akamai-introduces-advanced-communications-protocol-for-accelerating-dynamic-applications.jsp

https://developer.akamai.com/legacy/learn/Optimization/SureRoute.html

-Hank

Re: CloudFlare issues?

[Hank Nussbacher]

On 25/06/2019 08:17, Christopher Morrow wrote:

On Tue, Jun 25, 2019 at 12:49 AM Hank Nussbacher  wrote:

On 25/06/2019 03:03, Tom Beecher wrote:

Disclaimer : I am a Verizon employee via the Yahoo acquisition. I do
not work on 701.  My comments are my own opinions only.

Respectfully, I believe Cloudflare’s public comments today have been a
real disservice. This blog post, and your CEO on Twitter today, took
every opportunity to say “DAMN THOSE MORONS AT 701!”. They’re not.



Perhaps suggest to VZ management to use their blog:
https://www.verizondigitalmedia.com/blog/

#coughwrongvz

I think anyway - you probably mean:
https://enterprise.verizon.com/

This post is unrelated to Verizon Enterprise?
https://www.verizondigitalmedia.com/blog/2019/06/exponential-global-growth-at-75-tbps/

-Hank


GoodLuck! I think it's 3 clicks to: "www22.verizon.com" which gets
even moar fun!
The NOC used to answer if you called: +1-800-900-0241
which is in their whois records...


to contrandict what CF blogged about?

-Hank

Re: CloudFlare issues?

[Hank Nussbacher]

On 25/06/2019 03:03, Tom Beecher wrote:
Disclaimer : I am a Verizon employee via the Yahoo acquisition. I do 
not work on 701.  My comments are my own opinions only.


Respectfully, I believe Cloudflare’s public comments today have been a 
real disservice. This blog post, and your CEO on Twitter today, took 
every opportunity to say “DAMN THOSE MORONS AT 701!”. They’re not.




Perhaps suggest to VZ management to use their blog:
https://www.verizondigitalmedia.com/blog/
to contradict what CF blogged about?

-Hank

Re: Russian Anal Probing + Malware

[Hank Nussbacher]

On 24/06/2019 00:23, Randy Bush wrote:

e.g. i am aware of researchers scanning to see patching spread and
trying to make a conext paper dreadline this week or infocom next month.

hard to tell the sheep from the goats and the wolf from the sheep.  i
get the appended.  sheep or wholf?  i sure do not claim to be smart
enough to know.  but i sure am glad others are .

Greynoise can be your friend:
https://greynoise.io/about
https://viz.greynoise.io/table

-Hank



randy

---

Re: Bgpmon alternatives?

[Hank Nussbacher]

On 16/06/2019 12:28, Töma Gavrichenkov wrote:
On Sun, Jun 16, 2019, 4:57 AM TJ Trout > wrote:


Any simple and easy bgpmon alternatives you guys could recommend?



https://radar.qrator.net/

(this is not an advertisement!)

--
Töma

I have been a subscribed member to your service for a number of years 
and do not see where I can receive an email pushed to my my inbox of a 
suspected BGP hijack.  Can that be added?


Regards,

Hank

Cisco Crosswork Network Insights - or how to destroy a useful service

[Hank Nussbacher]

I have started to use Cisco Crosswork Network Insights which is the 
replacement for BGPmon and I am shocked at how Cisco has managed to 
destroy a useful tool.I have had a paid 50 prefix account since the day 
BGPmon became available and helped two clients implement a 500 prefix 
license over the past 4 years.None will be buying Cisco Crosswork 
Network Insights, based on my recommendation.


I really don’t know where to begin since there is so much to dislike in 
this new GUI.I will try to give you just a small taste but I suggest you 
request a 90 day trial license and try it out for yourself.


This was not designed by someone who deals with BGP hijacks or who 
manages a network.It was probably given to some GUI developer with a 
minimal understanding of what the users needed.How do I know this?Take 
for example the main configuration menu: 
https://crosswork.cisco.com/#/configuration with the first tab of 
“prefixes”.On that page there is *no* mention of which ASN the prefix is 
associated with.That of course was fundamental in the BGPmon menu: 
https://portal.bgpmon.net/myprefixes.php


Or take for example its “express configuration”, where you insert an ASN 
and it automatically finds all prefixes and creates a policy.But does it 
know the name of the ASN?Nope.Something again that was basic in BGPmon 
via: https://portal.bgpmon.net/myasn.php is non-existent in CNI.


Or how about the alarms one gets to an email?Want to see how that looks?

From: Crosswork Admin [mailto:ad...@crosswork.cisco.com]
Sent: 15 May 2019 11:39
To: Hank Nussbacher 
Subject: CCNI Notification

Active alarm count 1 starting at 2019-05-15 08:34:42.960762315 + 
UTC. Please click on the link for each alarm below:

https://crosswork.cisco.com/#/alarm/ba7c5084-f05d-4c12-a17f-be9e815d6647

Compare that with what we used to get:


Possible Prefix Hijack (Code: 10)


Your prefix:99.201.0.0/16:
Prefix Description:Kuku net
Update time:2018-08-12 17:50 (UTC)
Detected by #peers:140
Detected prefix:99.201.131.0/24
Announced by:AS46 (BGP hijacking Ltd)
Upstream AS:AS11 (Clueless ISP allowing customer hijacking Ltd)
ASpath:55 44 33 11 46
Alert 
details:https://portal.bgpmon.net/alerts.php?details_id=830521190

Mark as false alert:https://portal.bgpmon.net/fp.php?aid=830521190

That is just a small sampling.Maybe two years down the road, Cisco will 
speak to customers first before destroying a useful service.


Anyone else trying this out and feels the same or feels differently?

Disappointed,
Hank

Re: Widespread Firefox issues

[Hank Nussbacher]

On 05/05/2019 00:04, Lee wrote:

On 5/4/19, Mark Foster  wrote:

Official update from Mozilla:

https://blog.mozilla.org/addons/2019/05/04/update-regarding-add-ons-in-firefox/

where they say
   Please note: The fix does not apply to Firefox ESR
which is what I'm running, so
   about:config
change
   xpinstall.signatures.required
to false, restart and all my extensions now show
   xxx could not be verified for use in Firefox.  Proceed with caution.
but at least they're all enabled again :)


Am running FF 66.0.3 and did the above but still Avira Browser Safety 
and Cisco Webex still show up as disabled.


What am I missing?

Thanks,

Hank




Lee

Re: Open Petition for ARIN-prop-266: BGP Hijacking is an ARIN Policy Violation

[Hank Nussbacher]

On 27/04/2019 06:44, William Herrin wrote:
On Fri, Apr 26, 2019 at 7:48 PM Owen DeLong > wrote:
> Do you honestly believe that hijackings are being committed by ARIN 
members or even ARIN resource holders that have signed RSAs with ARIN?


Wasn't Softlayer (an ARIN resource holder) called out on this list 
about 14 hours ago for hijacking a couple /24s? And honest mistake no 
doubt but come on man, the hijackings happen.


I don't think the proposal is talking about valid mistakes.  The 
proposal is talking about active, repetitive, BGP hijacking.  If you 
disagree with the proposal, can you state what your proposed solution is 
for BGP hijacks?  What should we as a community do to prevent them from 
happening before some government/int'l agency mandates what they 
consider would be their solution?    Or do we just continue to drumbeat 
MANRS, post major BGP hijacks on NANOG and carry-on as we have for the 
past decade?



-Hank

Re: A Deep Dive on the Recent Widespread DNS Hijacking

[Hank Nussbacher]

On 25/02/2019 11:37, Ask Bjørn Hansen wrote:



On Feb 24, 2019, at 22:03, Hank Nussbacher  wrote:

Did you have a CAA record defined and if not, why not?

If the attacker got a CA to issue the cert because they changed the DNS server 
to be their own, a CAA record wouldn’t have helped (or at least been even 
easier to thwart than DNSSEC).


Yes if an attacker pwned the DNS then game over no matter what. I go 
under the assumption that the attacker was not able to take over the DNS 
system but rather other things along the way, in which case CAA should 
be of some assistance.


-Hank




Ask

Re: A Deep Dive on the Recent Widespread DNS Hijacking

[Hank Nussbacher]

On 25/02/2019 07:20, Bill Woodcock wrote:

On Feb 24, 2019, at 7:41 PM, Montgomery, Douglas (Fed)  wrote:
In the 3rd attack noted below, do we know if the CA that issued the DV CERTS 
does DNSSEC validation on its DNS challenge queries?

We know that neither Comodo nor Let's Encrypt were DNSSEC validating before 
issuing certs.  The Let’s Encrypt guys at least seemed interested in learning 
from their mistake.  Can’t say as much of Comodo.

 -Bill


Bill,

Did you have a CAA record defined and if not, why not?

-Hank

Re: Real-time BGP hijacking detection: ARTEMIS-1.0.0 just released

[Hank Nussbacher]

On 21/12/2018 17:10, Jared Mauch wrote:

So expect now BGP hijackers to announce /25s from here on in.  They 
generally adopt BCPs faster than providers.


-Hank


Folks have studied announcing a /25 etc.. and it can help because many 
providers will accept them.. it won’t get everyone, but longer than /24 
prefixes do help.

- Jared


On Dec 21, 2018, at 10:07 AM, Kody Vicknair  wrote:

I'm curious, If the highjacked prefix is a /24 (subset of your much larger /22) 
and you can only tie the highjacked prefix, at that point how effective is the 
mitigation outside of a default bgp route selection process?






-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Vasileios Kotronis
Sent: Thursday, December 20, 2018 11:23 AM
To: nanog@nanog.org
Subject: Real-time BGP hijacking detection: ARTEMIS-1.0.0 just released

Dear operators,

FORTH's INSPIRE group and CAIDA are delighted to announce the public release of 
the ARTEMIS BGP prefix hijacking detection tool, available as open-source 
software at https://github.com/FORTH-ICS-INSPIRE/artemis

ARTEMIS is designed to be operated by an AS in order to monitor BGP for potential 
hijacking attempts against its own prefixes. The system detects such attacks within 
seconds, enabling immediate mitigation. The current release has been tested at a 
major greek ISP, a dual-homed edge academic network, and a major US R 
backbone network.

We would be happy if you'd give it a try and provide feedback. Feel free to 
make pull requests on GitHub and help us make this a true community project.

ARTEMIS is funded by European Research Council (ERC) grant agreement no.
338402 (NetVolution Project), the RIPE NCC Community Projects 2017, the Comcast 
Innovation Fund, US NSF grants OAC-1848641 and CNS-1423659 and US DHS S 
contract HHSP233201600012C.

Best regards,
Vasileios

--
===
Vasileios Kotronis
Postdoctoral Researcher, member of the INSPIRE Group INSPIRE = INternet 
Security, Privacy, and Intelligence REsearch Telecommunications and Networks 
Lab (TNL) Foundation for Research and Technology - Hellas (FORTH) Leoforos 
Plastira 100, Heraklion 70013, Greece e-mail : vkotro...@ics.forth.gr
url: http://inspire.edu.gr
===

Re: Should ISP block child pornography?

[Hank Nussbacher]

On 07/12/2018 20:48, Max Tulyev wrote:
Yes, you may nullroute some IP with some site, but as the collateral 
damage you will block part of Cloudflare or Amazon, for example. So 
you have to buy and install additional equipment and software to do it 
a bit less painful. That's not so cheap, that should be planned, 
brought, installed, checked and personal should be learned. After 
that, your system will be capable to block some website for ~90% of 
your customers will not proactively avoid blocking. And for *NONE* who 
will, as CP addicts, terrorists, blackmarkets, gambling, porn and 
others do.
It is even more complex.  As you said filtering by IP address causing 
collateral damage to multi-host sites.
But there are sites that use primarily IPv6 addresses so you need to 
filter  not only IPv4 but IPv6 as well.
Also, sites change their IP address after they find out they are 
blocked, so you need a cron job which checks the IP addresses every 
10-15 minutes and updates the filters (if you are willing to accept 
collateral damage).


But when requested to block a FQDN, and filtering by IPv4 or IPv6 is not 
an option, again there are issues.


You filter/block in your central DNS server, but what about the user at 
home who is using 8.8.8.8 or 9.9.9.9?  Or the corporate link to some 
Fortune 500 company with their own DNS servers that bypass the ISP 
servers.  So now you are in a situation where you have to divert/capture 
*all *udp/53 and tcp/53 and pass it to some scrubbing server which will 
only block the requests to the forbidden FQDNs.   Oh but wait, what 
about DoH?


Governments that require ISPs to block "certain" sites have no clue what 
is required technologically to adhere to their demands.


-Hank