TLS issues with Lumen (CenturyLink)

[Andrew Gray]

Can someone from Lumen/CenturyLink contact me off-list?

We are seeing issues with some traffic coming from customers inside 
AS209 (and only AS209) that appear to be hitting some sort of 
in-the-middle inspection that is causing TLS issues (showing that the 
certificates are invalid).


Thanks!

[NANOG-announce] N88 Sponsorships, Peering Forum, Final Call for Presentations + CaribNOG 25 Empowers Underserved Students + Inspires Community

[Nanog News]

*Become a Sponsor of NANOG 88!*
*Invest in the Strength of the Community We've Built *

*Sponsorship Benefits: *

• Showcase your newest technologies + solutions
• Increase your brand’s visibility + reach
• Amplify your organization’s message
• Connect with industry influencers + decision makers
• Empower people + inspire change

*MORE INFO*


*Creating Global Connectivity by Connecting Globally*
*CaribNOG 25 Empowers Underserved Students + Inspires Community*

Network Engineer II at Akamai Technologies, Aaron Atac, is a part of the
millennial community at NANOG and serves on the Outreach and Programming
Committee. This was his first time attending CaribNOG, and "events like
these" originally motivated him to get involved.

Atac has a shared mission in helping these communities inspired by his dad,
who grew up in Turkey with limited access to education.

*READ NOW
*

*Sign up for the NANOG 88 Peering Forum *
*Meet + Greet Peers in this 90 Min Session *

The Peering Coordination Forum applications will remain open until 20
applications have been received or until the deadline date of 05, June.

The forum provides time for attendees to meet and network with others in
the peering community present at NANOG.


*MORE INFO* 

*Fundamentals of Designing and Deploying Computer Networks*
*ISOC Presents Moderated Course with Instructor *

This course will discuss the fundamentals of networking, Ethernet, and WIFI
technologies. It will additionally teach the planning, design, and
deployment of simple LANs and cover how to connect a LAN to the Internet.

*MORE INFO
*

*Deadline for N88 Presentations is Approaching!*
*Presentation Submission Deadline is Monday, 24 April*

The NANOG PC is looking to schedule over 1,600 minutes of content between
General Session and Breakout Rooms for NANOG 88 - so don’t wait!

*MORE INFO * 
___
NANOG-announce mailing list
NANOG-announce@nanog.org
https://mailman.nanog.org/mailman/listinfo/nanog-announce

N88 Sponsorships, Peering Forum, Final Call for Presentations + CaribNOG 25 Empowers Underserved Students + Inspires Community

[Nanog News]

*Become a Sponsor of NANOG 88!*
*Invest in the Strength of the Community We've Built *

*Sponsorship Benefits: *

• Showcase your newest technologies + solutions
• Increase your brand’s visibility + reach
• Amplify your organization’s message
• Connect with industry influencers + decision makers
• Empower people + inspire change

*MORE INFO*


*Creating Global Connectivity by Connecting Globally*
*CaribNOG 25 Empowers Underserved Students + Inspires Community*

Network Engineer II at Akamai Technologies, Aaron Atac, is a part of the
millennial community at NANOG and serves on the Outreach and Programming
Committee. This was his first time attending CaribNOG, and "events like
these" originally motivated him to get involved.

Atac has a shared mission in helping these communities inspired by his dad,
who grew up in Turkey with limited access to education.

*READ NOW
*

*Sign up for the NANOG 88 Peering Forum *
*Meet + Greet Peers in this 90 Min Session *

The Peering Coordination Forum applications will remain open until 20
applications have been received or until the deadline date of 05, June.

The forum provides time for attendees to meet and network with others in
the peering community present at NANOG.


*MORE INFO* 

*Fundamentals of Designing and Deploying Computer Networks*
*ISOC Presents Moderated Course with Instructor *

This course will discuss the fundamentals of networking, Ethernet, and WIFI
technologies. It will additionally teach the planning, design, and
deployment of simple LANs and cover how to connect a LAN to the Internet.

*MORE INFO
*

*Deadline for N88 Presentations is Approaching!*
*Presentation Submission Deadline is Monday, 24 April*

The NANOG PC is looking to schedule over 1,600 minutes of content between
General Session and Breakout Rooms for NANOG 88 - so don’t wait!

*MORE INFO * 

Re: ABQNOG -- May 4, 2023

[Chris Grundemann]

Thanks for sharing, John. I'm excited for this event, long overdue!

See everyone there - find me wherever the green chile is being served. =)

~Chris



On Mon, Apr 3, 2023 at 8:58 PM John Osmon  wrote:

> For folks that might be in the southwest US (and any that want to
> visit!), we're going to hold an operators group meeting on May 4,
> 2023 in Albuquerque, New Mexico.
>
> Come to the land of green chile chessburgers, and meet some of the
> local operators.  This inaugural meeting is free.  We hope to
> see you in May!
>
> http://abqnog.org
>
>
>
>

-- 
@ChrisGrundemann
http://chrisgrundemann.com

Re: DNS resolution for hhs.gov

[Bjørn Mork]

Interestingly enough, the company behind this mess decided to sign it:

 bjorn@canardo:~$ dig dhhs.gov @158.74.30.99 +nsid|grep NSID
 ; NSID: 4c 65 69 64 6f 73 20 62 75 69 6c 64 20 57 2e 56 45 52 4e 41 20 32 30 
32 33 ("Leidos build W.VERNA 2023")


Guessing this was done by "security professionals" from
https://www.leidos.com/




Bjørn

Mark Andrews  writes:

> The nameservers are not answering all in scope questions being sent to the 
> servers.  Something is blocking or not generating NXDOMAIN responses.  This 
> impacts on QNAME minimisation queries that usually elicit a NXDOMAIN 
> response.  This happens irrespective of DNSSEC records being requested so I 
> doubt that it is a fragmentation issue.
>
> Both _.dhhs.gov  and foobar.dhhs.gov 
>  time out but dhhs.gov  itself 
> doesn’t.
>
> % dig _.dhhs.gov @158.74.30.103 +dnssec
> ;; communications error to 158.74.30.103#53: timed out
> ;; communications error to 158.74.30.103#53: timed out
> ;; communications error to 158.74.30.103#53: timed out
>
> ; <<>> DiG 9.19.11-dev <<>> _.dhhs.gov @158.74.30.103 +dnssec
> ;; global options: +cmd
> ;; no servers could be reached
>
> % dig dhhs.gov @158.74.30.103 +dnssec
>
> ; <<>> DiG 9.19.11-dev <<>> dhhs.gov @158.74.30.103 +dnssec
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18125
> ;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
> ;; WARNING: recursion requested but not available
>
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags: do; udp: 4096
> ; COOKIE: d939ecfdb6cd2d902678cca26435eb2dd6fcebd65fe5c58f (good)
> ;; QUESTION SECTION:
> ;dhhs.gov. IN A
>
> ;; ANSWER SECTION:
> dhhs.gov. 9000 IN A 52.7.111.176
> dhhs.gov. 9000 IN RRSIG A 8 2 9000 20230416000149 20230410230149 11710 
> dhhs.gov. YCEsecATdJEHs3OtxQs/kE2A/37/mzgUpGLzQwrPP9xqaGmBq2mDteKx 
> QyUnh0JuURBq0Qy1htxsOD9kX4dxSxUNCEO7/KHw0AOoIbnh2+GL8kc3 
> jKB2jkcN+whA9+CqThto020nLSCXcgdm7qOfyNBUFICoYNtVrd7/lLCJ kho=
> dhhs.gov. 9000 IN RRSIG A 8 2 9000 20230416000149 20230410230149 21469 
> dhhs.gov. OkEdR/ofhV+JogwAkZtLmHyxn3pK2E4zaGUV786kKbtQrI6SzetCk+sC 
> Db3W0LrYRZy1BEqqxZeRnLXVEjyyyKfnYMRPtoP3sCTLPuuDeu8oDmhw 
> eniXLbJ10od6YWywgQDl2bYrTLEt6R8+TGG7up446TGgRk9wOV/uU2Jb d+U=
>
> ;; Query time: 308 msec
> ;; SERVER: 158.74.30.103#53(158.74.30.103) (UDP)
> ;; WHEN: Wed Apr 12 09:20:13 AEST 2023
> ;; MSG SIZE  rcvd: 417
>
> % dig foobar.dhhs.gov @158.74.30.103 +dnssec
> ;; communications error to 158.74.30.103#53: timed out
> ;; communications error to 158.74.30.103#53: timed out
> ;; communications error to 158.74.30.103#53: timed out
>
> ; <<>> DiG 9.19.11-dev <<>> foobar.dhhs.gov @158.74.30.103 +dnssec
> ;; global options: +cmd
> ;; no servers could be reached
>
> % dig foobar.dhhs.gov @158.74.30.103 
> ;; communications error to 158.74.30.103#53: timed out
> ;; communications error to 158.74.30.103#53: timed out
> ;; communications error to 158.74.30.103#53: timed out
>
> ; <<>> DiG 9.19.11-dev <<>> foobar.dhhs.gov @158.74.30.103
> ;; global options: +cmd
> ;; no servers could be reached
>
> % 
>
>> On 12 Apr 2023, at 01:12, Samuel Jackson  wrote:
>> 
>> I wanted to run this by everyone to make sure I am not the one losing my 
>> mind over this.
>> 
>> A dig +trace cob.cms.hhs.gov fails for me as it looks like the NS for 
>> hhs.gov does not seem to resolve the hostname.
>> 
>> However dig +trace cms.hhs.gov resolves and so does dig +trace 
>> eclkc.ohs.acf.hhs.gov
>> 
>> However if I simply ask my local resolver to resolve cob.cms.hhs.gov, it 
>> works. Any thoughts on why this is the case?
>> 
>> Thanks,
>> 

Your DNS Servers are not working correctly.

[Mark Andrews]

I work for a DNS vendor and saw reports about DNS resolution errors when 
looking up names under dhhs.gov.
It looks like your servers are not returning non-existence answers over UDP 
which breaks servers that are trying to do DNS QNAME minimisation (See RFC 
7816).

Below are three queries that the servers should be capable of answering if they 
are following the DNS protocol correctly.  dhhs.gov is answered but 
foobar.dhhs.gov doesn’t return anything and I would expect a NXDOMAIN (Name 
Error) response.  Additionally 355.dhhs.gov should be returning a 
NODATA/NOERROR response at a minimum as it part of your DNS servers names.

If I ask the same questions over TCP instead of UDP I get answers.

This really smells like a misconfigured firewall.

Mark

% dig dhhs.gov @158.74.30.99

; <<>> DiG 9.19.11-dev <<>> dhhs.gov @158.74.30.99
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59012
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 7b8cd5530b5fa45190ac7ac264364fe858d1f83093c6da62 (good)
;; QUESTION SECTION:
;dhhs.gov. IN A

;; ANSWER SECTION:
dhhs.gov. 9000 IN A 52.7.111.176

;; Query time: 243 msec
;; SERVER: 158.74.30.99#53(158.74.30.99) (UDP)
;; WHEN: Wed Apr 12 16:30:00 AEST 2023
;; MSG SIZE  rcvd: 81

% dig foobar.dhhs.gov @158.74.30.99
;; communications error to 158.74.30.99#53: timed out
;; communications error to 158.74.30.99#53: timed out
;; communications error to 158.74.30.99#53: timed out

; <<>> DiG 9.19.11-dev <<>> foobar.dhhs.gov @158.74.30.99
;; global options: +cmd
;; no servers could be reached

[ant-7641:~/git/bind9] marka% dig 355.dhhs.gov @158.74.30.99
;; communications error to 158.74.30.99#53: timed out
;; communications error to 158.74.30.99#53: timed out
;; communications error to 158.74.30.99#53: timed out

; <<>> DiG 9.19.11-dev <<>> 355.dhhs.gov @158.74.30.99
;; global options: +cmd
;; no servers could be reached

% 

% dig dhhs.gov @158.74.30.99 +tcp

; <<>> DiG 9.19.11-dev <<>> dhhs.gov @158.74.30.99 +tcp
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18254
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 710a14c38e16a91fd4060d86643652ecca2dce18d21e3144 (good)
;; QUESTION SECTION:
;dhhs.gov. IN A

;; ANSWER SECTION:
dhhs.gov. 9000 IN A 52.7.111.176

;; Query time: 246 msec
;; SERVER: 158.74.30.99#53(158.74.30.99) (TCP)
;; WHEN: Wed Apr 12 16:42:52 AEST 2023
;; MSG SIZE  rcvd: 81

% dig 355.dhhs.gov @158.74.30.99 +tcp

; <<>> DiG 9.19.11-dev <<>> 355.dhhs.gov @158.74.30.99 +tcp
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56223
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: e10fe6bd8dccc0ed038bbff1643652fb582c8d51b5d3a25c (good)
;; QUESTION SECTION:
;355.dhhs.gov. IN A

;; AUTHORITY SECTION:
dhhs.gov. 3600 IN SOA rh120ns1.368.dhhs.gov. hostmaster.psc.hhs.gov. 2023021759 
1200 300 2419200 3600

;; Query time: 246 msec
;; SERVER: 158.74.30.99#53(158.74.30.99) (TCP)
;; WHEN: Wed Apr 12 16:43:07 AEST 2023
;; MSG SIZE  rcvd: 137


% 
 -- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742  INTERNET: ma...@isc.org