Re: [EXTERNAL] Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[Validin Axon]

Thank you, Jim. Who is the vendor responsible?

Kenneth

On Tue, Apr 23, 2024 at 4:24 PM Rampley, Jim F 
wrote:

>
>
> Hi Kenneth,
>
>
>
> We have been working internally and with our third-party domain reputation
> source to get your domain removed from their malware list.
>
>
>
> Jim
>
>
>
> *From: *NANOG  on behalf
> of Validin Axon 
> *Date: *Tuesday, April 23, 2024 at 2:15 PM
> *To: *Tom Beecher 
> *Cc: *NANOG 
> *Subject: *[EXTERNAL] Re: Help with removing DNS shinkhole FP from
> Charter/Spectrum
>
>
>
> CAUTION: The e-mail below is from an external source. Please exercise
> caution before opening attachments, clicking links, or following guidance.
>
>
>
> Tom,
>
>
>
> Thank you for this! It is very interesting that the behavior is
> intermittent. A friend of mine who tested it this weekend saw correct
> answers on IPv6 and incorrect answers on IPv4.
>
>
>
> Kenneth
>
>
>
> On Tue, Apr 23, 2024 at 2:56 PM Tom Beecher  wrote:
>
> Validin, made an interesting observation on this. I am also a Spectrum
> residential customer,  none of their equipment, run my own DNS server
> (pihole).
>
>
>
> My DHCP Assigned DNS servers are
>
> 2001:1998:f00:1::1
> 2001:1998:f00:2::1
>
> bash-3.2$ dig -x 2001:1998:f00:1::1 +short
> dns-cac-lb-01.rr.com.
> bash-3.2$ dig -x 2001:1998:f00:2::1 +short
> dns-cac-lb-02.rr.com.
> bash-3.2$
>
>
> bash-3.2$ dig dns-cac-lb-01.rr.com +short
> 209.18.47.61
> bash-3.2$ dig dns-cac-lb-02.rr.com +short
> 209.18.47.62
> bash-3.2$
>
> bash-3.2$ dig @209.18.47.61 validin.com +short
> 157.245.112.183
> 137.184.54.107
> bash-3.2$ dig @209.18.47.62 validin.com +short
> 157.245.112.183
> 137.184.54.107
> bash-3.2$
>
> bash-3.2$ dig @2001:1998:f00:1::1 validin.com +short
> 127.0.0.54
> bash-3.2$
>
> bash-3.2$ dig @2001:1998:f00:2::1 validin.com +short
> 127.0.0.54
> bash-3.2$
>
> Same servers on V4 were returning correct info, but on V6 were not.
>
> However, a few minutes later :
>
> bash-3.2$ dig @2001:1998:f00:1::1 validin.com +short
> 157.245.112.183
> 137.184.54.107
> bash-3.2$ dig @2001:1998:f00:2::1 validin.com +short
> 157.245.112.183
> 137.184.54.107
> bash-3.2$
>
> Deltas :
>
> bash-3.2$ dig @2001:1998:f00:1::1  validin.com
>
> ; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com
> ; (1 server found)
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42329
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
>
> ;; QUESTION SECTION:
> ;validin.com.   IN  A
>
> ;; ANSWER SECTION:
> validin.com.60  IN  A   127.0.0.54
>
> ;; Query time: 37 msec
> ;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1)
> ;; WHEN: Tue Apr 23 13:50:03 EDT 2024
> ;; MSG SIZE  rcvd: 45
>
> bash-3.2$
>
> bash-3.2$ dig @2001:1998:f00:1::1 validin.com
>
> ; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com
> ; (1 server found)
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9667
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
>
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 512
> ;; QUESTION SECTION:
> ;validin.com.   IN  A
>
> ;; ANSWER SECTION:
> validin.com.600 IN  A   157.245.112.183
> validin.com.600 IN  A   137.184.54.107
>
> ;; Query time: 157 msec
> ;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1)
> ;; WHEN: Tue Apr 23 14:19:20 EDT 2024
> ;; MSG SIZE  rcvd: 72
>
> bash-3.2$
>
>
>
> Seems like quite possibly they are intermittently caching bunk data from
> something.
>
>
>
>
>
> On Tue, Apr 23, 2024 at 1:39 PM Validin Axon  wrote:
>
> Hi Jason,
>
>
>
> > I suspect what’s happened is an incorrect assumption that DNS is even
> the issue here. Because you mentioned Spectrum Shield, I suspect it is not.
>
>
>
> I appreciate the response and links. However, I've been told repeatedly by
> Spectrum that they're not blocking with Spectrum Shield. Despite these
> assurances, I've filled out a removal request through their published
> removal process several times, and the response I received stated that
> we're not being blocked. This check agrees with that:
>
> https://www.spectrum.net/support/forms/verify_url_security
>
>
>
> "Security Shield Is Not Blocking This Site
>
> The URL provided is not being blocked by Spectrum Security Shield
&g

Re: [EXTERNAL] Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[Rampley, Jim F]

Hi Kenneth,

We have been working internally and with our third-party domain reputation 
source to get your domain removed from their malware list.

Jim

From: NANOG  on behalf of 
Validin Axon 
Date: Tuesday, April 23, 2024 at 2:15 PM
To: Tom Beecher 
Cc: NANOG 
Subject: [EXTERNAL] Re: Help with removing DNS shinkhole FP from 
Charter/Spectrum

CAUTION: The e-mail below is from an external source. Please exercise caution 
before opening attachments, clicking links, or following guidance.

Tom,

Thank you for this! It is very interesting that the behavior is intermittent. A 
friend of mine who tested it this weekend saw correct answers on IPv6 and 
incorrect answers on IPv4.

Kenneth

On Tue, Apr 23, 2024 at 2:56 PM Tom Beecher 
mailto:beec...@beecher.cc>> wrote:
Validin, made an interesting observation on this. I am also a Spectrum 
residential customer,  none of their equipment, run my own DNS server (pihole).

My DHCP Assigned DNS servers are

2001:1998:f00:1::1
2001:1998:f00:2::1

bash-3.2$ dig -x 2001:1998:f00:1::1 +short
dns-cac-lb-01.rr.com<http://dns-cac-lb-01.rr.com>.
bash-3.2$ dig -x 2001:1998:f00:2::1 +short
dns-cac-lb-02.rr.com<http://dns-cac-lb-02.rr.com>.
bash-3.2$


bash-3.2$ dig dns-cac-lb-01.rr.com<http://dns-cac-lb-01.rr.com> +short
209.18.47.61
bash-3.2$ dig dns-cac-lb-02.rr.com<http://dns-cac-lb-02.rr.com> +short
209.18.47.62
bash-3.2$

bash-3.2$ dig @209.18.47.61<http://209.18.47.61> 
validin.com<http://validin.com> +short
157.245.112.183
137.184.54.107
bash-3.2$ dig @209.18.47.62<http://209.18.47.62> 
validin.com<http://validin.com> +short
157.245.112.183
137.184.54.107
bash-3.2$

bash-3.2$ dig @2001:1998:f00:1::1 validin.com<http://validin.com> +short
127.0.0.54
bash-3.2$

bash-3.2$ dig @2001:1998:f00:2::1 validin.com<http://validin.com> +short
127.0.0.54
bash-3.2$

Same servers on V4 were returning correct info, but on V6 were not.

However, a few minutes later :

bash-3.2$ dig @2001:1998:f00:1::1 validin.com<http://validin.com> +short
157.245.112.183
137.184.54.107
bash-3.2$ dig @2001:1998:f00:2::1 validin.com<http://validin.com> +short
157.245.112.183
137.184.54.107
bash-3.2$

Deltas :

bash-3.2$ dig @2001:1998:f00:1::1  validin.com<http://validin.com>

; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com<http://validin.com>
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42329
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;validin.com<http://validin.com>.   IN  A

;; ANSWER SECTION:
validin.com<http://validin.com>.60  IN  A   127.0.0.54

;; Query time: 37 msec
;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1)
;; WHEN: Tue Apr 23 13:50:03 EDT 2024
;; MSG SIZE  rcvd: 45

bash-3.2$

bash-3.2$ dig @2001:1998:f00:1::1 validin.com<http://validin.com>

; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com<http://validin.com>
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9667
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;validin.com<http://validin.com>.   IN  A

;; ANSWER SECTION:
validin.com<http://validin.com>.600 IN  A   
157.245.112.183
validin.com<http://validin.com>.600 IN  A   
137.184.54.107

;; Query time: 157 msec
;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1)
;; WHEN: Tue Apr 23 14:19:20 EDT 2024
;; MSG SIZE  rcvd: 72

bash-3.2$

Seems like quite possibly they are intermittently caching bunk data from 
something.


On Tue, Apr 23, 2024 at 1:39 PM Validin Axon 
mailto:a...@validin.com>> wrote:
Hi Jason,

> I suspect what’s happened is an incorrect assumption that DNS is even the 
> issue here. Because you mentioned Spectrum Shield, I suspect it is not.

I appreciate the response and links. However, I've been told repeatedly by 
Spectrum that they're not blocking with Spectrum Shield. Despite these 
assurances, I've filled out a removal request through their published removal 
process several times, and the response I received stated that we're not being 
blocked. This check agrees with that:
https://www.spectrum.net/support/forms/verify_url_security

"Security Shield Is Not Blocking This Site
The URL provided is not being blocked by Spectrum Security Shield
The URL you entered should be accessible."
Further, checking Spectrum DNS servers on the Spectrum network show that my 
company's main domain and all subdomains resolve to 127.0.0.54. So, if 
CujoAI/Spectrum Shield are not using DNS query responses to control access, 
then it's not CujoAI/Spectrum Shi

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[Validin Axon]

Tom,

Thank you for this! It is very interesting that the behavior is
intermittent. A friend of mine who tested it this weekend saw correct
answers on IPv6 and incorrect answers on IPv4.

Kenneth

On Tue, Apr 23, 2024 at 2:56 PM Tom Beecher  wrote:

> Validin, made an interesting observation on this. I am also a Spectrum
> residential customer,  none of their equipment, run my own DNS server
> (pihole).
>
> My DHCP Assigned DNS servers are
>
> 2001:1998:f00:1::1
> 2001:1998:f00:2::1
>
> bash-3.2$ dig -x 2001:1998:f00:1::1 +short
> dns-cac-lb-01.rr.com.
> bash-3.2$ dig -x 2001:1998:f00:2::1 +short
> dns-cac-lb-02.rr.com.
> bash-3.2$
>
>
> bash-3.2$ dig dns-cac-lb-01.rr.com +short
> 209.18.47.61
> bash-3.2$ dig dns-cac-lb-02.rr.com +short
> 209.18.47.62
> bash-3.2$
>
> bash-3.2$ dig @209.18.47.61 validin.com +short
> 157.245.112.183
> 137.184.54.107
> bash-3.2$ dig @209.18.47.62 validin.com +short
> 157.245.112.183
> 137.184.54.107
> bash-3.2$
>
> bash-3.2$ dig @2001:1998:f00:1::1 validin.com +short
> 127.0.0.54
> bash-3.2$
>
> bash-3.2$ dig @2001:1998:f00:2::1 validin.com +short
> 127.0.0.54
> bash-3.2$
>
> Same servers on V4 were returning correct info, but on V6 were not.
>
> However, a few minutes later :
>
> bash-3.2$ dig @2001:1998:f00:1::1 validin.com +short
> 157.245.112.183
> 137.184.54.107
> bash-3.2$ dig @2001:1998:f00:2::1 validin.com +short
> 157.245.112.183
> 137.184.54.107
> bash-3.2$
>
> Deltas :
>
> bash-3.2$ dig @2001:1998:f00:1::1  validin.com
>
> ; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com
> ; (1 server found)
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42329
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
>
> ;; QUESTION SECTION:
> ;validin.com.   IN  A
>
> ;; ANSWER SECTION:
> validin.com.60  IN  A   127.0.0.54
>
> ;; Query time: 37 msec
> ;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1)
> ;; WHEN: Tue Apr 23 13:50:03 EDT 2024
> ;; MSG SIZE  rcvd: 45
>
> bash-3.2$
>
> bash-3.2$ dig @2001:1998:f00:1::1 validin.com
>
> ; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com
> ; (1 server found)
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9667
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
>
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 512
> ;; QUESTION SECTION:
> ;validin.com.   IN  A
>
> ;; ANSWER SECTION:
> validin.com.600 IN  A   157.245.112.183
> validin.com.600 IN  A   137.184.54.107
>
> ;; Query time: 157 msec
> ;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1)
> ;; WHEN: Tue Apr 23 14:19:20 EDT 2024
> ;; MSG SIZE  rcvd: 72
>
> bash-3.2$
>
> Seems like quite possibly they are intermittently caching bunk data from
> something.
>
>
> On Tue, Apr 23, 2024 at 1:39 PM Validin Axon  wrote:
>
>> Hi Jason,
>>
>> > I suspect what’s happened is an incorrect assumption that DNS is even
>> the issue here. Because you mentioned Spectrum Shield, I suspect it is not.
>>
>> I appreciate the response and links. However, I've been told repeatedly
>> by Spectrum that they're not blocking with Spectrum Shield. Despite these
>> assurances, I've filled out a removal request through their published
>> removal process several times, and the response I received stated that
>> we're not being blocked. This check agrees with that:
>> https://www.spectrum.net/support/forms/verify_url_security
>>
>> "Security Shield Is Not Blocking This Site
>> The URL provided is not being blocked by Spectrum Security Shield
>> The URL you entered should be accessible."
>>
>> Further, checking Spectrum DNS servers on the Spectrum network show that
>> my company's main domain and all subdomains resolve to 127.0.0.54. So, if
>> CujoAI/Spectrum Shield are not using DNS query responses to control access,
>> then it's not CujoAI/Spectrum Shield that is responsible for the incorrect
>> DNS response. Using a different recursive resolve, I can resolve our
>> domains just fine. I can also resolve other domains that point to the same
>> IPs as the sinkholed domain just fine. However, many people use the
>> Spectrum default DNS servers and cannot access my website because of this.
>>
>> > You should contact Charter/Spectrum to have them investigate what their
>> system might be blocking this content.
>>
>> I have tried, for months, including spending many hours on chat and phone
>> support, to reach someone within Spectrum support who is capable of both
>> understanding and directing me to someone who can fix the problem, but it
>> hasn't happened yet. I've asked to talk to someone on the DNS team and was
>> given a flat "No." I've posted here hoping that someone in the
>> ISP-connected world knows SOMEONE at Spectrum, Akamai, or whichever company
>> is actually responsible for the Spectrum DNS servers who can provide a
>> remediation path.
>>
>> 

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[Tom Beecher]

Validin, made an interesting observation on this. I am also a Spectrum
residential customer,  none of their equipment, run my own DNS server
(pihole).

My DHCP Assigned DNS servers are

2001:1998:f00:1::1
2001:1998:f00:2::1

bash-3.2$ dig -x 2001:1998:f00:1::1 +short
dns-cac-lb-01.rr.com.
bash-3.2$ dig -x 2001:1998:f00:2::1 +short
dns-cac-lb-02.rr.com.
bash-3.2$


bash-3.2$ dig dns-cac-lb-01.rr.com +short
209.18.47.61
bash-3.2$ dig dns-cac-lb-02.rr.com +short
209.18.47.62
bash-3.2$

bash-3.2$ dig @209.18.47.61 validin.com +short
157.245.112.183
137.184.54.107
bash-3.2$ dig @209.18.47.62 validin.com +short
157.245.112.183
137.184.54.107
bash-3.2$

bash-3.2$ dig @2001:1998:f00:1::1 validin.com +short
127.0.0.54
bash-3.2$

bash-3.2$ dig @2001:1998:f00:2::1 validin.com +short
127.0.0.54
bash-3.2$

Same servers on V4 were returning correct info, but on V6 were not.

However, a few minutes later :

bash-3.2$ dig @2001:1998:f00:1::1 validin.com +short
157.245.112.183
137.184.54.107
bash-3.2$ dig @2001:1998:f00:2::1 validin.com +short
157.245.112.183
137.184.54.107
bash-3.2$

Deltas :

bash-3.2$ dig @2001:1998:f00:1::1  validin.com

; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42329
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;validin.com.   IN  A

;; ANSWER SECTION:
validin.com.60  IN  A   127.0.0.54

;; Query time: 37 msec
;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1)
;; WHEN: Tue Apr 23 13:50:03 EDT 2024
;; MSG SIZE  rcvd: 45

bash-3.2$

bash-3.2$ dig @2001:1998:f00:1::1 validin.com

; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9667
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;validin.com.   IN  A

;; ANSWER SECTION:
validin.com.600 IN  A   157.245.112.183
validin.com.600 IN  A   137.184.54.107

;; Query time: 157 msec
;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1)
;; WHEN: Tue Apr 23 14:19:20 EDT 2024
;; MSG SIZE  rcvd: 72

bash-3.2$

Seems like quite possibly they are intermittently caching bunk data from
something.


On Tue, Apr 23, 2024 at 1:39 PM Validin Axon  wrote:

> Hi Jason,
>
> > I suspect what’s happened is an incorrect assumption that DNS is even
> the issue here. Because you mentioned Spectrum Shield, I suspect it is not.
>
> I appreciate the response and links. However, I've been told repeatedly by
> Spectrum that they're not blocking with Spectrum Shield. Despite these
> assurances, I've filled out a removal request through their published
> removal process several times, and the response I received stated that
> we're not being blocked. This check agrees with that:
> https://www.spectrum.net/support/forms/verify_url_security
>
> "Security Shield Is Not Blocking This Site
> The URL provided is not being blocked by Spectrum Security Shield
> The URL you entered should be accessible."
>
> Further, checking Spectrum DNS servers on the Spectrum network show that
> my company's main domain and all subdomains resolve to 127.0.0.54. So, if
> CujoAI/Spectrum Shield are not using DNS query responses to control access,
> then it's not CujoAI/Spectrum Shield that is responsible for the incorrect
> DNS response. Using a different recursive resolve, I can resolve our
> domains just fine. I can also resolve other domains that point to the same
> IPs as the sinkholed domain just fine. However, many people use the
> Spectrum default DNS servers and cannot access my website because of this.
>
> > You should contact Charter/Spectrum to have them investigate what their
> system might be blocking this content.
>
> I have tried, for months, including spending many hours on chat and phone
> support, to reach someone within Spectrum support who is capable of both
> understanding and directing me to someone who can fix the problem, but it
> hasn't happened yet. I've asked to talk to someone on the DNS team and was
> given a flat "No." I've posted here hoping that someone in the
> ISP-connected world knows SOMEONE at Spectrum, Akamai, or whichever company
> is actually responsible for the Spectrum DNS servers who can provide a
> remediation path.
>
> Regards,
>
> Kenneth
>
> On Tue, Apr 23, 2024 at 12:59 PM 'Livingood, Jason' via axon <
> a...@validin.com> wrote:
>
>> > However, there's no correction process for Spectrum's DNS sinkhole
>>
>> > But back to the topic: someone mentioned to me that Spectrum may not be
>> the direct providers for the DNS services they provide to their customers.
>> If anyone knows anything about how I might discover and reach out to the
>> people responsible, please let me know.
>>

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[Validin Axon]

Hi Jason,

> I suspect what’s happened is an incorrect assumption that DNS is even the
issue here. Because you mentioned Spectrum Shield, I suspect it is not.

I appreciate the response and links. However, I've been told repeatedly by
Spectrum that they're not blocking with Spectrum Shield. Despite these
assurances, I've filled out a removal request through their published
removal process several times, and the response I received stated that
we're not being blocked. This check agrees with that:
https://www.spectrum.net/support/forms/verify_url_security

"Security Shield Is Not Blocking This Site
The URL provided is not being blocked by Spectrum Security Shield
The URL you entered should be accessible."

Further, checking Spectrum DNS servers on the Spectrum network show that my
company's main domain and all subdomains resolve to 127.0.0.54. So, if
CujoAI/Spectrum Shield are not using DNS query responses to control access,
then it's not CujoAI/Spectrum Shield that is responsible for the incorrect
DNS response. Using a different recursive resolve, I can resolve our
domains just fine. I can also resolve other domains that point to the same
IPs as the sinkholed domain just fine. However, many people use the
Spectrum default DNS servers and cannot access my website because of this.

> You should contact Charter/Spectrum to have them investigate what their
system might be blocking this content.

I have tried, for months, including spending many hours on chat and phone
support, to reach someone within Spectrum support who is capable of both
understanding and directing me to someone who can fix the problem, but it
hasn't happened yet. I've asked to talk to someone on the DNS team and was
given a flat "No." I've posted here hoping that someone in the
ISP-connected world knows SOMEONE at Spectrum, Akamai, or whichever company
is actually responsible for the Spectrum DNS servers who can provide a
remediation path.

Regards,

Kenneth

On Tue, Apr 23, 2024 at 12:59 PM 'Livingood, Jason' via axon <
a...@validin.com> wrote:

> > However, there's no correction process for Spectrum's DNS sinkhole
>
> > But back to the topic: someone mentioned to me that Spectrum may not be
> the direct providers for the DNS services they provide to their customers.
> If anyone knows anything about how I might discover and reach out to the
> people responsible, please let me know.
>
>
>
> I suspect what’s happened is an incorrect assumption that DNS is even the
> issue here. Because you mentioned Spectrum Shield, I suspect it is not.
>
> Spectrum Shield (
> https://www.spectrum.com/resources/internet-wifi/benefits-of-spectrum-security-shield)
> is a customer-managed security protection service built into their gateways
> (I assume you can turn it off). The malware and content detection engine
> behind that is very likely run by CujoAI (https://cujo.com/) and it does
> not use DNS query/response exchanges as the control mechanism (in part to
> counter-act DNS-changing malware or malware using its own DoH channel for
> example).
>
> You should contact Charter/Spectrum to have them investigate what their
> system might be blocking this content.
>
> Comcast (where I work) runs a similar system (
> https://www.xfinity.com/support/articles/using-xfinity-xfi-advanced-security)
> and maintains a site to report these sorts of issues (
> https://www.xfinity.com/support/articles/report-blocked-website).
>
> Jason
>
>
>
>
>
>
>
>
>

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[Livingood, Jason via NANOG]

> However, there's no correction process for Spectrum's DNS sinkhole
> But back to the topic: someone mentioned to me that Spectrum may not be the 
> direct providers for the DNS services they provide to their customers. If 
> anyone knows anything about how I might discover and reach out to the people 
> responsible, please let me know.

I suspect what’s happened is an incorrect assumption that DNS is even the issue 
here. Because you mentioned Spectrum Shield, I suspect it is not.

Spectrum Shield 
(https://www.spectrum.com/resources/internet-wifi/benefits-of-spectrum-security-shield)
 is a customer-managed security protection service built into their gateways (I 
assume you can turn it off). The malware and content detection engine behind 
that is very likely run by CujoAI (https://cujo.com/) and it does not use DNS 
query/response exchanges as the control mechanism (in part to counter-act 
DNS-changing malware or malware using its own DoH channel for example).

You should contact Charter/Spectrum to have them investigate what their system 
might be blocking this content.

Comcast (where I work) runs a similar system 
(https://www.xfinity.com/support/articles/using-xfinity-xfi-advanced-security) 
and maintains a site to report these sorts of issues 
(https://www.xfinity.com/support/articles/report-blocked-website).

Jason

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[John R. Levine]

I'm not sure where you saw that message, but I got this message via email
after I submitted an unblock request with Spectrum Shield:


We have reviewed your request to unblock validin.com. This site was not

found to be blocked by Spectrum Shield and should be accessible from your
browser.


Sigh.


I've cleaned up everything I could from that botched blocklist aggregation.
However, there's no correction process for Spectrum's DNS sinkhole, and I'm
not even sure that's how our domain got mixed up there. The support staff
I've spoken with have denied the existence of DNS sinkholing at Spectrum,
and demonstrated they lack the basic technical sophistication needed to
understand the concept.


Yeah, that's the problem.  And given stuff like this link below, I 
wouldn't expect their legal department to be any better.  Clearly there is 
someone somewhere who is competent because their network mostly works, but 
damned if I know how to find them.


https://www.theverge.com/2022/7/29/23282522/charter-spectrum-customer-murder-forged-terms-of-service

R's,
John

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[John R. Levine]

Bill is absolutely correct. The spammers lost their case because they 
were demonstrably spammers.


No, really they did not.  I read the decisions.  Have you?  Hint: under 
CAN SPAM a great deal of spam is completely legal so it didn't matter.


We’ve had accidental black hole cases with *US* providers that removed 
the block once they received a C If they don’t have iron clad proof 
in hand. (More than just a few complaints and no traffic analysis), it’s 
just the least risky response.


I will believe that there are people that cave in response to threats like 
this, but again, there is no case law to support it.


R's,
John

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[William Herrin]

On Mon, Apr 22, 2024 at 5:54 PM Validin Axon  wrote:
> Hi Bill,
>
> I'm not sure where you saw that message, but I got this
> message via email after I submitted an unblock request with Spectrum Shield:

Howdy,

That was Christopher, not me. But you should check the talos link I
sent you privately. Also https://ipcheck.proofpoint.com/. Whatever
they're detecting, it didn't happen last year.

Regards,
Bill Herrin


-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[Validin Axon]

Hi Bill,

I'm not sure where you saw that message, but I got this message via email
after I submitted an unblock request with Spectrum Shield:

> We have reviewed your request to unblock validin.com. This site was not
found to be blocked by Spectrum Shield and should be accessible from your
browser.
>
> Thank you,
>
> Spectrum

My company's domain got caught up in some lazy copy/pasting from this blog
post last year that cited my company as a source for the data. Someone
copy/pasted the whole page, which included my company's domain name, and
that made it to a few AV OTX pulses and VT collections:
https://gi7w0rm.medium.com/uncovering-ddgroup-a-long-time-threat-actor-d3b3020625a4

I've cleaned up everything I could from that botched blocklist aggregation.
However, there's no correction process for Spectrum's DNS sinkhole, and I'm
not even sure that's how our domain got mixed up there. The support staff
I've spoken with have denied the existence of DNS sinkholing at Spectrum,
and demonstrated they lack the basic technical sophistication needed to
understand the concept. They've each ultimately told me that each affected
customer would need to reach out to the Spectrum customer service, which
would then help that customer change their DNS settings to another DNS
provider. Of course, the last thing I'd want to do with a potential
customer is ask them to go through that painful process. I also have no
idea how many potential users or customers can't reach me and simply give
up without letting me know.

Lastly, I AM a Spectrum customer. My home internet service is Spectrum. If
it weren't for that, I'd be truly SOL because support would just ignore me.
But, they they claim the issue is resolved from their perspective because I
can simply change my DNS settings.

But back to the topic: someone mentioned to me that Spectrum may not be the
direct providers for the DNS services they provide to their customers. If
anyone knows anything about how I might discover and reach out to the
people responsible, please let me know. :-)

Regards,

Kenneth

On Mon, Apr 22, 2024 at 8:07 PM Christopher Morrow 
wrote:

> “We checked the website you are trying to access for malicious and
> spear-phishing content and found it likely to be unsafe.”
>
> perhaps charter thinks there's a reason to not permit folks to access
> a possibly dangerous site?
> (it's also possible it just got cough up amongst some other stuff in
> the hosting provider's space, nothing jumps out in passive-dns
> lokoups.)
>
> On Mon, Apr 22, 2024 at 7:39 PM William Herrin  wrote:
> >
> > On Mon, Apr 22, 2024 at 4:00 PM John Levine  wrote:
> > > It appears that William Herrin  said:
> > > >If you can't reach a technical POC, use the legal one. Your lawyer can
> >
> > > The only response to a letter like that is "we run our network to
> > > serve our customers and manage it the way we think is best" and you
> > > know what, they're right.
> >
> > Hi John,
> >
> > Respectfully, you're mistaken. Look up "tortious interference."
> >
> > Operators have considerable legal leeway to block traffic for cause,
> > or even by mistake if corrected upon notification, but a lawyer who
> > blows off a cease-and-desist letter without investigating it with the
> > tech staff has committed malpractice. The lawyer doesn't want to
> > commit malpractice. You write the lawyer via certified mail, he's
> > going to talk to the tech staff and you're going to get a response. At
> > that point, you have an open communication pathway to get things
> > fixed. Which was the problem to be solved.
> >
> >
> > > Having said that, I suspect the least bad alternative if you can't
> > > find an out of band contact is to get some of the Spectrum customers
> > > who can't reach you to complain. They're customers, you aren't.
> >
> > My results going through the support front-door at large companies for
> > oddball problems have been less than stellar. Has your experience
> > truly been different?
> >
> > Regards,
> > Bill Herrin
> >
> >
> > --
> > William Herrin
> > b...@herrin.us
> > https://bill.herrin.us/
>

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[Mel Beckman]

Bill is absolutely correct. The spammers lost their case because they were 
demonstrably spammers. We’ve had accidental black hole cases with *US* 
providers that removed the block once they received a C If they don’t have 
iron clad proof in hand. (More than just a few complaints and no traffic 
analysis), it’s just the least risky response.

That doesn’t work well with overseas providers, though, because they’re 
essentially immune to U.S. litigation unless the plaintiff has deep pockets.

 -mel

On Apr 22, 2024, at 5:21 PM, William Herrin  wrote:

On Mon, Apr 22, 2024 at 5:07 PM John R. Levine  wrote:
a complaint would have to show that the
blocking was malicious rather than merited or accidental.  In this case it
seems probably accidental, but for all I know there might have been bad
traffic to merit a block.

Hi John,

I'll try not to belabor it, but accidental that isn't corrected upon
formal legal notification becomes negligent and negligent has more or
less the same legal status as malicious.

The spammers lost because the networks published a terms of use
document that the spammers unambiguously violated. Even though it
interfered with the spammer's business, the block was merited so the
preponderance of the evidence fell in favor of the service provider.

Regards,
Bill Herrin


--
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[William Herrin]

On Mon, Apr 22, 2024 at 5:07 PM John R. Levine  wrote:
> a complaint would have to show that the
> blocking was malicious rather than merited or accidental.  In this case it
> seems probably accidental, but for all I know there might have been bad
> traffic to merit a block.

Hi John,

I'll try not to belabor it, but accidental that isn't corrected upon
formal legal notification becomes negligent and negligent has more or
less the same legal status as malicious.

The spammers lost because the networks published a terms of use
document that the spammers unambiguously violated. Even though it
interfered with the spammer's business, the block was merited so the
preponderance of the evidence fell in favor of the service provider.

Regards,
Bill Herrin


-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[John R. Levine]

On Mon, 22 Apr 2024, William Herrin wrote:

Respectfully, you're mistaken. Look up "tortious interference."


I'm familiar with it.

But I am also familar with many cases were spammers have sued network 
operators claiming that they're falsely defamed, so the operator has to 
deliver their mail.  They have without exception lost.  If you can find 
actual cases where a court forced an operator to deliver a third party's 
traffic I would like to hear about it.*


43 USC 230(c)(A) provides extremely broad protection for "good faith" 
blocking, which means that a complaint would have to show that the 
blocking was malicious rather than merited or accidental.  In this case it 
seems probably accidental, but for all I know there might have been bad 
traffic to merit a block.


Here's one of the cases where a spammer lost:

https://jl.ly/Email/holomaxx.html
https://jl.ly/Email/holo4.html

And here's one where the judge rejected tortious interference:

https://jl.ly/Email/spamarrest.html


My results going through the support front-door at large companies for
oddball problems have been less than stellar. Has your experience
truly been different?


No, it's terrible, and Spectrum is particularly bad.  I am now in month 
three of trying to get them to route a /24 to my host that belongs to one 
of my users, and their responses can be summarized as very complex 
exegeses of "duh?"


But bogus lawyer letters will just make things worse.

R's,
John

* - let's stay away for now from the Texas and Florida social network 
common carrier laws which are a whole other can of s*

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[Christopher Morrow]

“We checked the website you are trying to access for malicious and
spear-phishing content and found it likely to be unsafe.”

perhaps charter thinks there's a reason to not permit folks to access
a possibly dangerous site?
(it's also possible it just got cough up amongst some other stuff in
the hosting provider's space, nothing jumps out in passive-dns
lokoups.)

On Mon, Apr 22, 2024 at 7:39 PM William Herrin  wrote:
>
> On Mon, Apr 22, 2024 at 4:00 PM John Levine  wrote:
> > It appears that William Herrin  said:
> > >If you can't reach a technical POC, use the legal one. Your lawyer can
>
> > The only response to a letter like that is "we run our network to
> > serve our customers and manage it the way we think is best" and you
> > know what, they're right.
>
> Hi John,
>
> Respectfully, you're mistaken. Look up "tortious interference."
>
> Operators have considerable legal leeway to block traffic for cause,
> or even by mistake if corrected upon notification, but a lawyer who
> blows off a cease-and-desist letter without investigating it with the
> tech staff has committed malpractice. The lawyer doesn't want to
> commit malpractice. You write the lawyer via certified mail, he's
> going to talk to the tech staff and you're going to get a response. At
> that point, you have an open communication pathway to get things
> fixed. Which was the problem to be solved.
>
>
> > Having said that, I suspect the least bad alternative if you can't
> > find an out of band contact is to get some of the Spectrum customers
> > who can't reach you to complain. They're customers, you aren't.
>
> My results going through the support front-door at large companies for
> oddball problems have been less than stellar. Has your experience
> truly been different?
>
> Regards,
> Bill Herrin
>
>
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[William Herrin]

On Mon, Apr 22, 2024 at 4:00 PM John Levine  wrote:
> It appears that William Herrin  said:
> >If you can't reach a technical POC, use the legal one. Your lawyer can

> The only response to a letter like that is "we run our network to
> serve our customers and manage it the way we think is best" and you
> know what, they're right.

Hi John,

Respectfully, you're mistaken. Look up "tortious interference."

Operators have considerable legal leeway to block traffic for cause,
or even by mistake if corrected upon notification, but a lawyer who
blows off a cease-and-desist letter without investigating it with the
tech staff has committed malpractice. The lawyer doesn't want to
commit malpractice. You write the lawyer via certified mail, he's
going to talk to the tech staff and you're going to get a response. At
that point, you have an open communication pathway to get things
fixed. Which was the problem to be solved.


> Having said that, I suspect the least bad alternative if you can't
> find an out of band contact is to get some of the Spectrum customers
> who can't reach you to complain. They're customers, you aren't.

My results going through the support front-door at large companies for
oddball problems have been less than stellar. Has your experience
truly been different?

Regards,
Bill Herrin


-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[John Levine]

It appears that William Herrin  said:
>On Sun, Apr 21, 2024 at 6:21 PM Validin Axon  wrote:
>> Looking for some help/advice. Spectrum is sinkholing my company's domain, 
>> validin[.]com, to 127.0.0.54.
>
>Howdy,
>
>If you can't reach a technical POC, use the legal one. Your lawyer can
>find the appropriate recipient and write a cease-and-desist letter for
>you. After that, it's -their- lawyer's problem to track down the
>correct technical people.

No, that is terrible advice.  In the immortal acronym of Laura Atkins, TWSD.

The only response to a letter like that is "we run our network to
serve our customers and manage it the way we think is best" and you
know what, they're right. It is absolutely legal to block traffic you
think is malicious, even if you are wrong, and there is case law.

Having said that, I suspect the least bad alternative if you can't
find an out of band contact is to get some of the Spectrum customers
who can't reach you to complain. They're customers, you aren't.

R's,
John

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[Validin Axon]

Hi Mel,

I appreciate the suggestion. During my earlier research, I'd noticed that
as well. However, the DNS block includes all validin.com subdomains, covering
those on completely different ASNs. It also does NOT affect other domains
that resolve to the exact same IP addresses (e.g., validin.net). So, I'm
inclined to think it's not that simple, unfortunately.

I'd considered switching domains, but that doesn't guarantee the problem
wouldn't just reappear again, and it'd impact the search engine ranking
we've built up. We rely 100% on inbound, so that'd be a big set back.

Warm regards,

Kenneth

On Mon, Apr 22, 2024 at 10:29 AM Mel Beckman  wrote:

>
> UCEPROTECTL3 137.184.54.107 was listed
>
> I notice from MXToolbox.com that your domain’s IP address is on the
> UCEPROTECTL3 blacklist.
>
> This is a notoriously evil blacklist that charges people for removal. This
> may be why Spectrum is blackholing your domain. Most respectable ISPs won’t
> use it. But Spectrum…
>
> There is no delisting procedure without making a “donation” to the UCEPROTECT3
> black sparrow account. They’re famous for blacklisting large swaths of IP
> addresses that catch up innocent parties that have never spammed a flea.
>
> -mel
>
>
> On Apr 22, 2024, at 7:24 AM, Mel Beckman  wrote:
>
>  I notice you’re on the UCEPROTECT3 blacklist:
>
> 
> Network Tools: DNS,IP,Email
> 
> mxtoolbox.com
> 
> 
>
> UCEPROTECTL3 137.184.54.107 was listed
>  This is a notoriously evil blacklist that charges people for removal.
> This may be why Spectrum is blackholing your domain. Most respectable ISPs
> won’t use it. But Spectrum…
>
> There is no delisting procedure without making a “donation” to the UCEPROTECT3
> black sparrow account. They’re famous for blacklisting large swaths of IP
> addresses that catch up innocent parties that have never spammed a flea.
>
> -mel
>
> On Apr 22, 2024, at 4:51 AM, Validin Axon  wrote:
>
> 
> Looking for some help/advice. Spectrum is sinkholing my company's domain,
> validin[.]com, to 127.0.0.54. The sinkhole responses come from their
> recursive DNS servers, 209.18.47.61 and 209.18.47.62, which are defaults
> for and in use by many of their customers and are only reachable from
> within the Spectrum network. I've had 4 people over the last week (think:
> customers, prospects, etc) who use Charter/Spectrum tell me that they have
> difficulty accessing my website as a result of this sinkhole behavior. This
> behavior is causing reputational harm to my company.
>
> I've personally confirmed this behavior from the Spectrum network (I am
> also a customer) using dig to test their DNS servers:
> ```
> $ dig +short @209.18.47.61 validin.com
> 127.0.0.54
> $ dig +short @209.18.47.62 validin.com
> 127.0.0.54
> ```
>  Using Cloudflare/Google/etc works correctly:
> ```
> $ dig +short @1.1.1.1 validin.com
> 137.184.54.107
> 157.245.112.183
> $ dig +short @8.8.8.8 validin.com
> 157.245.112.183
> 137.184.54.107
> ```
>
> I suspect my domain was blocklisted last year when a threat researcher
> included my domain name in a blog post about a threat they were
> investigating and cited my company as the source for their data. Someone
> scraped that post, and my company's domain was accidentally added to
> two Alient Vault OTX pulses and at least one collection on Virus Total. I
> removed the domain via false positive reporting from everything I could.
> However, it appears that being added to Spectrum's DNS sinkhole list is
> effectively permanent and there's no clear path for false positive
> remediation.
>
> I've tried the official Spectrum support lines for months to no avail, and
> recently tried reaching out on Twitter, but have had no success there
> either. I'm clearly not able to find the right people through these routes,
> as none of the people I reach understand the difference between a DNS
> sinkhole and an IP block list and don't appear to be aware that DNS
> blocklisting is a separate behavior from their opt-in content filtering via
> Security Shield.
>
> So, if someone could please help me find the team or individual
> responsible for Spectrum's DNS sinkhole behavior, I would be exceptionally
> grateful. :-)
>
> As I mentioned, this is causing reputation harm, so switching my own DNS
> servers is not sufficient. People who need to reach me, can't. So, I would
> appreciate any other help or advice you have,
>
> Kenneth
>
>

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[Mel Beckman]

I notice from MXToolbox.com that your domain’s IP address is on the 
UCEPROTECTL3 blacklist.

This is a notoriously evil blacklist that charges people for removal. This may 
be why Spectrum is blackholing your domain. Most respectable ISPs won’t use it. 
But Spectrum…

There is no delisting procedure without making a “donation” to the UCEPROTECT3 
black sparrow account. They’re famous for blacklisting large swaths of IP 
addresses that catch up innocent parties that have never spammed a flea.

-mel

On Apr 22, 2024, at 4:51 AM, Validin Axon  wrote:


Looking for some help/advice. Spectrum is sinkholing my company's domain, 
validin[.]com, to 127.0.0.54. The sinkhole responses come from their recursive 
DNS servers, 209.18.47.61 and 209.18.47.62, which are defaults for and in use 
by many of their customers and are only reachable from within the Spectrum 
network. I've had 4 people over the last week (think: customers, prospects, 
etc) who use Charter/Spectrum tell me that they have difficulty accessing my 
website as a result of this sinkhole behavior. This behavior is causing 
reputational harm to my company.

I've personally confirmed this behavior from the Spectrum network (I am also a 
customer) using dig to test their DNS servers:
```
$ dig +short @209.18.47.61 validin.com
127.0.0.54
$ dig +short @209.18.47.62 validin.com
127.0.0.54
```
 Using Cloudflare/Google/etc works correctly:
```
$ dig +short @1.1.1.1 validin.com
137.184.54.107
157.245.112.183
$ dig +short @8.8.8.8 validin.com
157.245.112.183
137.184.54.107
```

I suspect my domain was blocklisted last year when a threat researcher included 
my domain name in a blog post about a threat they were investigating and cited 
my company as the source for their data. Someone scraped that post, and my 
company's domain was accidentally added to two Alient Vault OTX pulses and at 
least one collection on Virus Total. I removed the domain via false positive 
reporting from everything I could. However, it appears that being added to 
Spectrum's DNS sinkhole list is effectively permanent and there's no clear path 
for false positive remediation.

I've tried the official Spectrum support lines for months to no avail, and 
recently tried reaching out on Twitter, but have had no success there either. 
I'm clearly not able to find the right people through these routes, as none of 
the people I reach understand the difference between a DNS sinkhole and an IP 
block list and don't appear to be aware that DNS blocklisting is a separate 
behavior from their opt-in content filtering via Security Shield.

So, if someone could please help me find the team or individual responsible for 
Spectrum's DNS sinkhole behavior, I would be exceptionally grateful. :-)

As I mentioned, this is causing reputation harm, so switching my own DNS 
servers is not sufficient. People who need to reach me, can't. So, I would 
appreciate any other help or advice you have,

Kenneth

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[William Herrin]

On Sun, Apr 21, 2024 at 6:21 PM Validin Axon  wrote:
> Looking for some help/advice. Spectrum is sinkholing my company's domain, 
> validin[.]com, to 127.0.0.54.

Howdy,

If you can't reach a technical POC, use the legal one. Your lawyer can
find the appropriate recipient and write a cease-and-desist letter for
you. After that, it's -their- lawyer's problem to track down the
correct technical people.

Incidentally, for folks who choose to interdict DNS: whatever your
reasons, pointing the DNS to a loopback IP is bad practice. Really bad
practice. Minimum good practice points it to a web site you control
which provides enough information to get delisted. And provides you
with a test point where you can collect information about what you've
caused to be interdicted.

Regards,
Bill Herrin


-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/