Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[John R. Levine]

I'm not sure where you saw that message, but I got this message via email
after I submitted an unblock request with Spectrum Shield:


We have reviewed your request to unblock validin.com. This site was not

found to be blocked by Spectrum Shield and should be accessible from your
browser.


Sigh.


I've cleaned up everything I could from that botched blocklist aggregation.
However, there's no correction process for Spectrum's DNS sinkhole, and I'm
not even sure that's how our domain got mixed up there. The support staff
I've spoken with have denied the existence of DNS sinkholing at Spectrum,
and demonstrated they lack the basic technical sophistication needed to
understand the concept.


Yeah, that's the problem.  And given stuff like this link below, I 
wouldn't expect their legal department to be any better.  Clearly there is 
someone somewhere who is competent because their network mostly works, but 
damned if I know how to find them.


https://www.theverge.com/2022/7/29/23282522/charter-spectrum-customer-murder-forged-terms-of-service

R's,
John

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[John R. Levine]

Bill is absolutely correct. The spammers lost their case because they 
were demonstrably spammers.


No, really they did not.  I read the decisions.  Have you?  Hint: under 
CAN SPAM a great deal of spam is completely legal so it didn't matter.


We’ve had accidental black hole cases with *US* providers that removed 
the block once they received a C If they don’t have iron clad proof 
in hand. (More than just a few complaints and no traffic analysis), it’s 
just the least risky response.


I will believe that there are people that cave in response to threats like 
this, but again, there is no case law to support it.


R's,
John

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[William Herrin]

On Mon, Apr 22, 2024 at 5:54 PM Validin Axon  wrote:
> Hi Bill,
>
> I'm not sure where you saw that message, but I got this
> message via email after I submitted an unblock request with Spectrum Shield:

Howdy,

That was Christopher, not me. But you should check the talos link I
sent you privately. Also https://ipcheck.proofpoint.com/. Whatever
they're detecting, it didn't happen last year.

Regards,
Bill Herrin


-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[Validin Axon]

Hi Bill,

I'm not sure where you saw that message, but I got this message via email
after I submitted an unblock request with Spectrum Shield:

> We have reviewed your request to unblock validin.com. This site was not
found to be blocked by Spectrum Shield and should be accessible from your
browser.
>
> Thank you,
>
> Spectrum

My company's domain got caught up in some lazy copy/pasting from this blog
post last year that cited my company as a source for the data. Someone
copy/pasted the whole page, which included my company's domain name, and
that made it to a few AV OTX pulses and VT collections:
https://gi7w0rm.medium.com/uncovering-ddgroup-a-long-time-threat-actor-d3b3020625a4

I've cleaned up everything I could from that botched blocklist aggregation.
However, there's no correction process for Spectrum's DNS sinkhole, and I'm
not even sure that's how our domain got mixed up there. The support staff
I've spoken with have denied the existence of DNS sinkholing at Spectrum,
and demonstrated they lack the basic technical sophistication needed to
understand the concept. They've each ultimately told me that each affected
customer would need to reach out to the Spectrum customer service, which
would then help that customer change their DNS settings to another DNS
provider. Of course, the last thing I'd want to do with a potential
customer is ask them to go through that painful process. I also have no
idea how many potential users or customers can't reach me and simply give
up without letting me know.

Lastly, I AM a Spectrum customer. My home internet service is Spectrum. If
it weren't for that, I'd be truly SOL because support would just ignore me.
But, they they claim the issue is resolved from their perspective because I
can simply change my DNS settings.

But back to the topic: someone mentioned to me that Spectrum may not be the
direct providers for the DNS services they provide to their customers. If
anyone knows anything about how I might discover and reach out to the
people responsible, please let me know. :-)

Regards,

Kenneth

On Mon, Apr 22, 2024 at 8:07 PM Christopher Morrow 
wrote:

> “We checked the website you are trying to access for malicious and
> spear-phishing content and found it likely to be unsafe.”
>
> perhaps charter thinks there's a reason to not permit folks to access
> a possibly dangerous site?
> (it's also possible it just got cough up amongst some other stuff in
> the hosting provider's space, nothing jumps out in passive-dns
> lokoups.)
>
> On Mon, Apr 22, 2024 at 7:39 PM William Herrin  wrote:
> >
> > On Mon, Apr 22, 2024 at 4:00 PM John Levine  wrote:
> > > It appears that William Herrin  said:
> > > >If you can't reach a technical POC, use the legal one. Your lawyer can
> >
> > > The only response to a letter like that is "we run our network to
> > > serve our customers and manage it the way we think is best" and you
> > > know what, they're right.
> >
> > Hi John,
> >
> > Respectfully, you're mistaken. Look up "tortious interference."
> >
> > Operators have considerable legal leeway to block traffic for cause,
> > or even by mistake if corrected upon notification, but a lawyer who
> > blows off a cease-and-desist letter without investigating it with the
> > tech staff has committed malpractice. The lawyer doesn't want to
> > commit malpractice. You write the lawyer via certified mail, he's
> > going to talk to the tech staff and you're going to get a response. At
> > that point, you have an open communication pathway to get things
> > fixed. Which was the problem to be solved.
> >
> >
> > > Having said that, I suspect the least bad alternative if you can't
> > > find an out of band contact is to get some of the Spectrum customers
> > > who can't reach you to complain. They're customers, you aren't.
> >
> > My results going through the support front-door at large companies for
> > oddball problems have been less than stellar. Has your experience
> > truly been different?
> >
> > Regards,
> > Bill Herrin
> >
> >
> > --
> > William Herrin
> > b...@herrin.us
> > https://bill.herrin.us/
>

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[Mel Beckman]

Bill is absolutely correct. The spammers lost their case because they were 
demonstrably spammers. We’ve had accidental black hole cases with *US* 
providers that removed the block once they received a C If they don’t have 
iron clad proof in hand. (More than just a few complaints and no traffic 
analysis), it’s just the least risky response.

That doesn’t work well with overseas providers, though, because they’re 
essentially immune to U.S. litigation unless the plaintiff has deep pockets.

 -mel

On Apr 22, 2024, at 5:21 PM, William Herrin  wrote:

On Mon, Apr 22, 2024 at 5:07 PM John R. Levine  wrote:
a complaint would have to show that the
blocking was malicious rather than merited or accidental.  In this case it
seems probably accidental, but for all I know there might have been bad
traffic to merit a block.

Hi John,

I'll try not to belabor it, but accidental that isn't corrected upon
formal legal notification becomes negligent and negligent has more or
less the same legal status as malicious.

The spammers lost because the networks published a terms of use
document that the spammers unambiguously violated. Even though it
interfered with the spammer's business, the block was merited so the
preponderance of the evidence fell in favor of the service provider.

Regards,
Bill Herrin


--
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[William Herrin]

On Mon, Apr 22, 2024 at 5:07 PM John R. Levine  wrote:
> a complaint would have to show that the
> blocking was malicious rather than merited or accidental.  In this case it
> seems probably accidental, but for all I know there might have been bad
> traffic to merit a block.

Hi John,

I'll try not to belabor it, but accidental that isn't corrected upon
formal legal notification becomes negligent and negligent has more or
less the same legal status as malicious.

The spammers lost because the networks published a terms of use
document that the spammers unambiguously violated. Even though it
interfered with the spammer's business, the block was merited so the
preponderance of the evidence fell in favor of the service provider.

Regards,
Bill Herrin


-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[John R. Levine]

On Mon, 22 Apr 2024, William Herrin wrote:

Respectfully, you're mistaken. Look up "tortious interference."


I'm familiar with it.

But I am also familar with many cases were spammers have sued network 
operators claiming that they're falsely defamed, so the operator has to 
deliver their mail.  They have without exception lost.  If you can find 
actual cases where a court forced an operator to deliver a third party's 
traffic I would like to hear about it.*


43 USC 230(c)(A) provides extremely broad protection for "good faith" 
blocking, which means that a complaint would have to show that the 
blocking was malicious rather than merited or accidental.  In this case it 
seems probably accidental, but for all I know there might have been bad 
traffic to merit a block.


Here's one of the cases where a spammer lost:

https://jl.ly/Email/holomaxx.html
https://jl.ly/Email/holo4.html

And here's one where the judge rejected tortious interference:

https://jl.ly/Email/spamarrest.html


My results going through the support front-door at large companies for
oddball problems have been less than stellar. Has your experience
truly been different?


No, it's terrible, and Spectrum is particularly bad.  I am now in month 
three of trying to get them to route a /24 to my host that belongs to one 
of my users, and their responses can be summarized as very complex 
exegeses of "duh?"


But bogus lawyer letters will just make things worse.

R's,
John

* - let's stay away for now from the Texas and Florida social network 
common carrier laws which are a whole other can of s*

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[Christopher Morrow]

“We checked the website you are trying to access for malicious and
spear-phishing content and found it likely to be unsafe.”

perhaps charter thinks there's a reason to not permit folks to access
a possibly dangerous site?
(it's also possible it just got cough up amongst some other stuff in
the hosting provider's space, nothing jumps out in passive-dns
lokoups.)

On Mon, Apr 22, 2024 at 7:39 PM William Herrin  wrote:
>
> On Mon, Apr 22, 2024 at 4:00 PM John Levine  wrote:
> > It appears that William Herrin  said:
> > >If you can't reach a technical POC, use the legal one. Your lawyer can
>
> > The only response to a letter like that is "we run our network to
> > serve our customers and manage it the way we think is best" and you
> > know what, they're right.
>
> Hi John,
>
> Respectfully, you're mistaken. Look up "tortious interference."
>
> Operators have considerable legal leeway to block traffic for cause,
> or even by mistake if corrected upon notification, but a lawyer who
> blows off a cease-and-desist letter without investigating it with the
> tech staff has committed malpractice. The lawyer doesn't want to
> commit malpractice. You write the lawyer via certified mail, he's
> going to talk to the tech staff and you're going to get a response. At
> that point, you have an open communication pathway to get things
> fixed. Which was the problem to be solved.
>
>
> > Having said that, I suspect the least bad alternative if you can't
> > find an out of band contact is to get some of the Spectrum customers
> > who can't reach you to complain. They're customers, you aren't.
>
> My results going through the support front-door at large companies for
> oddball problems have been less than stellar. Has your experience
> truly been different?
>
> Regards,
> Bill Herrin
>
>
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[William Herrin]

On Mon, Apr 22, 2024 at 4:00 PM John Levine  wrote:
> It appears that William Herrin  said:
> >If you can't reach a technical POC, use the legal one. Your lawyer can

> The only response to a letter like that is "we run our network to
> serve our customers and manage it the way we think is best" and you
> know what, they're right.

Hi John,

Respectfully, you're mistaken. Look up "tortious interference."

Operators have considerable legal leeway to block traffic for cause,
or even by mistake if corrected upon notification, but a lawyer who
blows off a cease-and-desist letter without investigating it with the
tech staff has committed malpractice. The lawyer doesn't want to
commit malpractice. You write the lawyer via certified mail, he's
going to talk to the tech staff and you're going to get a response. At
that point, you have an open communication pathway to get things
fixed. Which was the problem to be solved.


> Having said that, I suspect the least bad alternative if you can't
> find an out of band contact is to get some of the Spectrum customers
> who can't reach you to complain. They're customers, you aren't.

My results going through the support front-door at large companies for
oddball problems have been less than stellar. Has your experience
truly been different?

Regards,
Bill Herrin


-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[John Levine]

It appears that William Herrin  said:
>On Sun, Apr 21, 2024 at 6:21 PM Validin Axon  wrote:
>> Looking for some help/advice. Spectrum is sinkholing my company's domain, 
>> validin[.]com, to 127.0.0.54.
>
>Howdy,
>
>If you can't reach a technical POC, use the legal one. Your lawyer can
>find the appropriate recipient and write a cease-and-desist letter for
>you. After that, it's -their- lawyer's problem to track down the
>correct technical people.

No, that is terrible advice.  In the immortal acronym of Laura Atkins, TWSD.

The only response to a letter like that is "we run our network to
serve our customers and manage it the way we think is best" and you
know what, they're right. It is absolutely legal to block traffic you
think is malicious, even if you are wrong, and there is case law.

Having said that, I suspect the least bad alternative if you can't
find an out of band contact is to get some of the Spectrum customers
who can't reach you to complain. They're customers, you aren't.

R's,
John

Re: Question about mutual transit and complex BGP peering

[Matthew Petach]

On Mon, Apr 22, 2024 at 7:35 AM Sriram, Kotikalapudi (Fed) via NANOG <
nanog@nanog.org> wrote:

> Requesting responses to the following questions. Would be helpful in some
> IETF work in progress.
>
> Q1: Consider an AS peering relationship that is complex (or hybrid)
> meaning, for example, provider-to-customer (P2C) for one set of prefixes
> and lateral peers (i.e., transit-free peer-to-peer (P2P)) for another set
> of prefixes.  Are these diverse relationships usually segregated, i.e., P2C
> on one BGP session and P2P on another?  How often they might co-exist
> within one single BGP session?
>
>
Every time I've been in relationships like this, the fundamental answer is
always "follow the money".

If there's dollars flowing relative to the "provider-to-customer"
relationship, but no dollars flowing along the "peer-to-peer" relationship,
you need a solid way to determine which bits are taking the zero-dollar
pathway, and which bits are taking the non-zero-dollar pathway.

Whatever means are available to positively distinguish the traffic on an
unambiguous basis that both networks agree on is what determines the setup.

In many cases, separate physical ports with separate BGP sessions (and
sometimes even separate VRFs) is the only way that both parties fully trust
all the right bits
are being accounted for in each case.

In other relationships, flow data is considered adequate to determine how
much traffic is zero dollar, and how much traffic is non-zero dollar.  In
that case, it can be a single BGP session, single port.



> Q2: Consider an AS peering relationship that is mutual transit (i.e., P2C
> relationship in each direction for all prefixes).  Is this supported within
> one single BGP session?  How often the ASes might setup two separate BGP
> sessions between them -- one for P2C in one direction (AS A to AS B) and
> the other for P2C in the opposite direction (AS B to AS A)?
>

This is just a variant of a normal peer-to-peer relationship, most likely
with a traffic ratio involved.
In most of these situations, as long as the traffic is within the defined
ratio, accounting for the
bits isn't worth it; sending a bill from A to B for $X, and a different
bill from B to A for $+$Y where $Y is
generally much smaller than $X is more headache than it's worth.
And once the ratio goes outside of the prescribed range, you're not really
mutual transit anymore, you're provider to customer,
and the only wrinkle is which one considers themselves the provider, and
which considers themselves the customer.
Witness Level 3 versus Comcast versus Netflix from years ago:
https://arstechnica.com/tech-policy/2010/12/comcastlevel3/
https://publicknowledge.org/netflix-cdn-v-the-cable-guys-or-comcast-v-level-3-part-deux-peering-payback/

Again--when everything is within ratio, and pipes aren't full, no need for
separate ports or separate BGP
sessions.

Once things start to fill up, though, then things get ugly.  That's when
different sessions come into play,
with some traffic being shunted to congested sessions, while the two sides
battle it out.

It still comes down to the same fundamental rule, though--follow the
money.   ^_^;

Thanks!

Matt




> Thank you.
>
> Sriram
> Kotikalapudi Sriram, US NIST
>

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[Validin Axon]

Hi Mel,

I appreciate the suggestion. During my earlier research, I'd noticed that
as well. However, the DNS block includes all validin.com subdomains, covering
those on completely different ASNs. It also does NOT affect other domains
that resolve to the exact same IP addresses (e.g., validin.net). So, I'm
inclined to think it's not that simple, unfortunately.

I'd considered switching domains, but that doesn't guarantee the problem
wouldn't just reappear again, and it'd impact the search engine ranking
we've built up. We rely 100% on inbound, so that'd be a big set back.

Warm regards,

Kenneth

On Mon, Apr 22, 2024 at 10:29 AM Mel Beckman  wrote:

>
> UCEPROTECTL3 137.184.54.107 was listed
>
> I notice from MXToolbox.com that your domain’s IP address is on the
> UCEPROTECTL3 blacklist.
>
> This is a notoriously evil blacklist that charges people for removal. This
> may be why Spectrum is blackholing your domain. Most respectable ISPs won’t
> use it. But Spectrum…
>
> There is no delisting procedure without making a “donation” to the UCEPROTECT3
> black sparrow account. They’re famous for blacklisting large swaths of IP
> addresses that catch up innocent parties that have never spammed a flea.
>
> -mel
>
>
> On Apr 22, 2024, at 7:24 AM, Mel Beckman  wrote:
>
>  I notice you’re on the UCEPROTECT3 blacklist:
>
> 
> Network Tools: DNS,IP,Email
> 
> mxtoolbox.com
> 
> 
>
> UCEPROTECTL3 137.184.54.107 was listed
>  This is a notoriously evil blacklist that charges people for removal.
> This may be why Spectrum is blackholing your domain. Most respectable ISPs
> won’t use it. But Spectrum…
>
> There is no delisting procedure without making a “donation” to the UCEPROTECT3
> black sparrow account. They’re famous for blacklisting large swaths of IP
> addresses that catch up innocent parties that have never spammed a flea.
>
> -mel
>
> On Apr 22, 2024, at 4:51 AM, Validin Axon  wrote:
>
> 
> Looking for some help/advice. Spectrum is sinkholing my company's domain,
> validin[.]com, to 127.0.0.54. The sinkhole responses come from their
> recursive DNS servers, 209.18.47.61 and 209.18.47.62, which are defaults
> for and in use by many of their customers and are only reachable from
> within the Spectrum network. I've had 4 people over the last week (think:
> customers, prospects, etc) who use Charter/Spectrum tell me that they have
> difficulty accessing my website as a result of this sinkhole behavior. This
> behavior is causing reputational harm to my company.
>
> I've personally confirmed this behavior from the Spectrum network (I am
> also a customer) using dig to test their DNS servers:
> ```
> $ dig +short @209.18.47.61 validin.com
> 127.0.0.54
> $ dig +short @209.18.47.62 validin.com
> 127.0.0.54
> ```
>  Using Cloudflare/Google/etc works correctly:
> ```
> $ dig +short @1.1.1.1 validin.com
> 137.184.54.107
> 157.245.112.183
> $ dig +short @8.8.8.8 validin.com
> 157.245.112.183
> 137.184.54.107
> ```
>
> I suspect my domain was blocklisted last year when a threat researcher
> included my domain name in a blog post about a threat they were
> investigating and cited my company as the source for their data. Someone
> scraped that post, and my company's domain was accidentally added to
> two Alient Vault OTX pulses and at least one collection on Virus Total. I
> removed the domain via false positive reporting from everything I could.
> However, it appears that being added to Spectrum's DNS sinkhole list is
> effectively permanent and there's no clear path for false positive
> remediation.
>
> I've tried the official Spectrum support lines for months to no avail, and
> recently tried reaching out on Twitter, but have had no success there
> either. I'm clearly not able to find the right people through these routes,
> as none of the people I reach understand the difference between a DNS
> sinkhole and an IP block list and don't appear to be aware that DNS
> blocklisting is a separate behavior from their opt-in content filtering via
> Security Shield.
>
> So, if someone could please help me find the team or individual
> responsible for Spectrum's DNS sinkhole behavior, I would be exceptionally
> grateful. :-)
>
> As I mentioned, this is causing reputation harm, so switching my own DNS
> servers is not sufficient. People who need to reach me, can't. So, I would
> appreciate any other help or advice you have,
>
> Kenneth
>
>

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[Mel Beckman]

I notice from MXToolbox.com that your domain’s IP address is on the 
UCEPROTECTL3 blacklist.

This is a notoriously evil blacklist that charges people for removal. This may 
be why Spectrum is blackholing your domain. Most respectable ISPs won’t use it. 
But Spectrum…

There is no delisting procedure without making a “donation” to the UCEPROTECT3 
black sparrow account. They’re famous for blacklisting large swaths of IP 
addresses that catch up innocent parties that have never spammed a flea.

-mel

On Apr 22, 2024, at 4:51 AM, Validin Axon  wrote:


Looking for some help/advice. Spectrum is sinkholing my company's domain, 
validin[.]com, to 127.0.0.54. The sinkhole responses come from their recursive 
DNS servers, 209.18.47.61 and 209.18.47.62, which are defaults for and in use 
by many of their customers and are only reachable from within the Spectrum 
network. I've had 4 people over the last week (think: customers, prospects, 
etc) who use Charter/Spectrum tell me that they have difficulty accessing my 
website as a result of this sinkhole behavior. This behavior is causing 
reputational harm to my company.

I've personally confirmed this behavior from the Spectrum network (I am also a 
customer) using dig to test their DNS servers:
```
$ dig +short @209.18.47.61 validin.com
127.0.0.54
$ dig +short @209.18.47.62 validin.com
127.0.0.54
```
 Using Cloudflare/Google/etc works correctly:
```
$ dig +short @1.1.1.1 validin.com
137.184.54.107
157.245.112.183
$ dig +short @8.8.8.8 validin.com
157.245.112.183
137.184.54.107
```

I suspect my domain was blocklisted last year when a threat researcher included 
my domain name in a blog post about a threat they were investigating and cited 
my company as the source for their data. Someone scraped that post, and my 
company's domain was accidentally added to two Alient Vault OTX pulses and at 
least one collection on Virus Total. I removed the domain via false positive 
reporting from everything I could. However, it appears that being added to 
Spectrum's DNS sinkhole list is effectively permanent and there's no clear path 
for false positive remediation.

I've tried the official Spectrum support lines for months to no avail, and 
recently tried reaching out on Twitter, but have had no success there either. 
I'm clearly not able to find the right people through these routes, as none of 
the people I reach understand the difference between a DNS sinkhole and an IP 
block list and don't appear to be aware that DNS blocklisting is a separate 
behavior from their opt-in content filtering via Security Shield.

So, if someone could please help me find the team or individual responsible for 
Spectrum's DNS sinkhole behavior, I would be exceptionally grateful. :-)

As I mentioned, this is causing reputation harm, so switching my own DNS 
servers is not sufficient. People who need to reach me, can't. So, I would 
appreciate any other help or advice you have,

Kenneth

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[William Herrin]

On Sun, Apr 21, 2024 at 6:21 PM Validin Axon  wrote:
> Looking for some help/advice. Spectrum is sinkholing my company's domain, 
> validin[.]com, to 127.0.0.54.

Howdy,

If you can't reach a technical POC, use the legal one. Your lawyer can
find the appropriate recipient and write a cease-and-desist letter for
you. After that, it's -their- lawyer's problem to track down the
correct technical people.

Incidentally, for folks who choose to interdict DNS: whatever your
reasons, pointing the DNS to a loopback IP is bad practice. Really bad
practice. Minimum good practice points it to a web site you control
which provides enough information to get delisted. And provides you
with a test point where you can collect information about what you've
caused to be interdicted.

Regards,
Bill Herrin


-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Question about mutual transit and complex BGP peering

[Sriram, Kotikalapudi (Fed) via NANOG]

Requesting responses to the following questions. Would be helpful in some IETF 
work in progress.
 
Q1: Consider an AS peering relationship that is complex (or hybrid) meaning, 
for example, provider-to-customer (P2C) for one set of prefixes and lateral 
peers (i.e., transit-free peer-to-peer (P2P)) for another set of prefixes.  Are 
these diverse relationships usually segregated, i.e., P2C on one BGP session 
and P2P on another?  How often they might co-exist within one single BGP 
session?

Q2: Consider an AS peering relationship that is mutual transit (i.e., P2C 
relationship in each direction for all prefixes).  Is this supported within one 
single BGP session?  How often the ASes might setup two separate BGP sessions 
between them -- one for P2C in one direction (AS A to AS B) and the other for 
P2C in the opposite direction (AS B to AS A)?

Thank you.

Sriram
Kotikalapudi Sriram, US NIST

Re: constant FEC errors juniper mpc10e 400g

[Mark Tinka]

On 4/22/24 09:47, Vasilenko Eduard via NANOG wrote:


Assume that some carrier has 10k FBB subscribers in a particular municipality 
(without any hope of considerably increasing this number).
2Mbps is the current average per household in the busy hour, pretty uniform 
worldwide.
You could multiply it by 8/7 if you like to add wireless -> not much would 
change.
2*2*10GE (2*10GE on the ring in every direction) is 2 times than needed to 
carry 10k subscribers.
The optical ring may be less than 20 municipalities - it is very common.
Hence, the upgrade of old extremely cheap 10GE DWDM systems (for 40 lambdas) 
makes sense for some carriers.
It depends on the population density and the carrier market share.
10GE for the WAN side would not be dead in the next 5 years because 2Mbps per 
household would not grow very fast in the future - this logistic curve is close 
to a plateau.
PS: It is probably not the case for Africa where new subscribers are connected 
to the Internet at a fast rate.


As a function of how much Internet there is in Africa, there really 
aren't that many optical transport service providers. Some 
countries/cities/towns have more than they need, others have just one. 
But in general, you would say there is massive room for improvement if 
you surveyed the entire continent.


Typically, it will be the incumbents, alongside 2 or 3 competitives. In 
fact, in some African countries, only the incumbent may be large enough 
to run an optical backbone, with all the competitives leasing capacity 
from them.


It is not uncommon to find the closest competitor to an incumbent for 
terrestrial services being the mobile network operator, purely because 
they have some excess capacity left over from having to build the 
backbone for their core business, mobile. And, they are flush with cash, 
so a loss-making terrestrial backhaul business can be covered by the 
month's sales in SIM cards.


Truly independent transport providers are few and far between because 
access to dark fibre is not easy (either its lack of availability, the 
incumbent refusing to sell it, or its high price). For the few 
independent transport providers that do spring up, they will focus on a 
limited set of hot routes, and because competition on those routes may 
be wanting, prices and capacity would not be terribly attractive.


So the bulk of Africa's Internet really relies on a handful of key 
African wholesale IP Transit providers taking great effort into 
extending their network as deep into cities as they can, and using their 
size to negotiate the best prices for terrestrial backhaul from the few 
optical network operators that the market has. Those providers then sell 
to the local and regional ISP's, who don't have to worry about running a 
backbone.


All this means is that for those operators that run an optical backbone, 
especially nationally, 10G carriers are very, very legacy. If they still 
have them, it'd be a spin-off off the main core to support some old SDH 
customers that are too comfortable to move to Ethernet.


Metro backhaul and last mile FNO's (fibre network operators) who have 
invested in extending access into homes and businesses are a different 
story, with most countries having a reasonable handful of options 
customers can choose from. Like national backhaul, there is plenty of 
room for improvement - in some markets more than others - but unlike 
national backhaul, not as constrained for choice or price.


Mark.

Help with removing DNS shinkhole FP from Charter/Spectrum

[Validin Axon]

Looking for some help/advice. Spectrum is sinkholing my company's domain,
validin[.]com, to 127.0.0.54. The sinkhole responses come from their
recursive DNS servers, 209.18.47.61 and 209.18.47.62, which are defaults
for and in use by many of their customers and are only reachable from
within the Spectrum network. I've had 4 people over the last week (think:
customers, prospects, etc) who use Charter/Spectrum tell me that they have
difficulty accessing my website as a result of this sinkhole behavior. This
behavior is causing reputational harm to my company.

I've personally confirmed this behavior from the Spectrum network (I am
also a customer) using dig to test their DNS servers:
```
$ dig +short @209.18.47.61 validin.com
127.0.0.54
$ dig +short @209.18.47.62 validin.com
127.0.0.54
```
 Using Cloudflare/Google/etc works correctly:
```
$ dig +short @1.1.1.1 validin.com
137.184.54.107
157.245.112.183
$ dig +short @8.8.8.8 validin.com
157.245.112.183
137.184.54.107
```

I suspect my domain was blocklisted last year when a threat researcher
included my domain name in a blog post about a threat they were
investigating and cited my company as the source for their data. Someone
scraped that post, and my company's domain was accidentally added to
two Alient Vault OTX pulses and at least one collection on Virus Total. I
removed the domain via false positive reporting from everything I could.
However, it appears that being added to Spectrum's DNS sinkhole list is
effectively permanent and there's no clear path for false positive
remediation.

I've tried the official Spectrum support lines for months to no avail, and
recently tried reaching out on Twitter, but have had no success there
either. I'm clearly not able to find the right people through these routes,
as none of the people I reach understand the difference between a DNS
sinkhole and an IP block list and don't appear to be aware that DNS
blocklisting is a separate behavior from their opt-in content filtering via
Security Shield.

So, if someone could please help me find the team or individual responsible
for Spectrum's DNS sinkhole behavior, I would be exceptionally grateful. :-)

As I mentioned, this is causing reputation harm, so switching my own DNS
servers is not sufficient. People who need to reach me, can't. So, I would
appreciate any other help or advice you have,

Kenneth

RE: constant FEC errors juniper mpc10e 400g

[Vasilenko Eduard via NANOG]

Assume that some carrier has 10k FBB subscribers in a particular municipality 
(without any hope of considerably increasing this number).
2Mbps is the current average per household in the busy hour, pretty uniform 
worldwide.
You could multiply it by 8/7 if you like to add wireless -> not much would 
change.
2*2*10GE (2*10GE on the ring in every direction) is 2 times than needed to 
carry 10k subscribers.
The optical ring may be less than 20 municipalities - it is very common.
Hence, the upgrade of old extremely cheap 10GE DWDM systems (for 40 lambdas) 
makes sense for some carriers.
It depends on the population density and the carrier market share.
10GE for the WAN side would not be dead in the next 5 years because 2Mbps per 
household would not grow very fast in the future - this logistic curve is close 
to a plateau.
PS: It is probably not the case for Africa where new subscribers are connected 
to the Internet at a fast rate.
Ed/
-Original Message-
From: NANOG  On Behalf Of 
Tarko Tikan
Sent: Saturday, April 20, 2024 19:19
To: nanog@nanog.org
Subject: Re: constant FEC errors juniper mpc10e 400g

hey,

> That said, I don't expect any subsea cables getting built in the next 
> 3 years and later will have 10G as a product on the SLTE itself... it 
> wouldn't be worth the spectrum.

10G wavelengths for new builds died about 10 years ago when coherent 100G 
became available, submarine or not. Putting 10G into same system is not really 
feasible at all.

--
tarko