Re: (IETF I-D): Implications of IPv6 Addressing on Security Operations (Fwd: New Version Notification for draft-gont-opsec-ipv6-addressing-00.txt)

[Fernando Gont]

Hi, Daniel,

On 7/2/23 21:20, Daniel Marks via NANOG wrote:
Anecdotal but I've seen hacked AWS accounts with Cloudformation scripts 
to create and destroy lots of tiny instances to rotate through IPv4 
addresses.


As with everything, the question is always "what's the level of effort 
that is required".


If an attacker is given the option to:

1) Hack an AWS account, and then script the creation of through-away VMs 
just to be able to change the IP address each time, or,


2) Stay on the same machine, and be able to (even legitimately) use 
2**64 addresses without even the need to hack any terraform scripts



They will probably go for #2. And aside of their choices, #1 requires 
more skills than #2.




Being able to rotate through IP addresses is not a new thing, 
I'm sure we all have networks in mind when we think of garbage/malicious 
traffic just over IPv4 alone.


The difference is in the scale at which this is possible with IPv6, and 
how high (or low) the bar is to do it.



There are some strange implementations of IPv6 that end up having a lot 
of dissociated users grouped together in a /64 (i.e. Linode, AT 
Wireless, etc)


Therein probably lies some good advice .. i.e., that to the extent that 
is possible, folks refrain from sharing the same /64 across 
unrelated/disassociated users.


Thanks,
--
Fernando Gont
SI6 Networks
e-mail: fg...@si6networks.com
PGP Fingerprint: F242 FF0E A804 AF81 EB10 2F07 7CA1 321D 663B B494

Re: (IETF I-D): Implications of IPv6 Addressing on Security Operations (Fwd: New Version Notification for draft-gont-opsec-ipv6-addressing-00.txt)

[Sabri Berisha]

- On Feb 7, 2023, at 5:04 PM, Fernando Gont fg...@si6networks.com wrote:

> On 7/2/23 21:43, Sabri Berisha wrote:
>> - On Feb 7, 2023, at 4:20 PM, nanog nanog@nanog.org wrote:

Hi,

>>> Anecdotal but I've seen hacked AWS accounts with Cloudformation scripts
>>> to create and destroy lots of tiny instances to rotate through IPv4
>>> addresses.
>> 
>> If only AWS would care about hacked AWS accounts.
> 
> Do they lose or earn money when accounts are hacked?

I guess that depends if the credit card on file is expired...

Thanks,

Sabri

Re: (IETF I-D): Implications of IPv6 Addressing on Security Operations (Fwd: New Version Notification for draft-gont-opsec-ipv6-addressing-00.txt)

[Fernando Gont]

On 7/2/23 21:43, Sabri Berisha wrote:

- On Feb 7, 2023, at 4:20 PM, nanog nanog@nanog.org wrote:

Hi,


Anecdotal but I've seen hacked AWS accounts with Cloudformation scripts
to create and destroy lots of tiny instances to rotate through IPv4
addresses.


If only AWS would care about hacked AWS accounts.


Do they lose or earn money when accounts are hacked?


--
Fernando Gont
SI6 Networks
e-mail: fg...@si6networks.com
PGP Fingerprint: F242 FF0E A804 AF81 EB10 2F07 7CA1 321D 663B B494

Re: (IETF I-D): Implications of IPv6 Addressing on Security Operations (Fwd: New Version Notification for draft-gont-opsec-ipv6-addressing-00.txt)

[Sabri Berisha]

- On Feb 7, 2023, at 4:20 PM, nanog nanog@nanog.org wrote:

Hi,

> Anecdotal but I've seen hacked AWS accounts with Cloudformation scripts
> to create and destroy lots of tiny instances to rotate through IPv4
> addresses. 

If only AWS would care about hacked AWS accounts.

Thanks,

Sabri

Re: (IETF I-D): Implications of IPv6 Addressing on Security Operations (Fwd: New Version Notification for draft-gont-opsec-ipv6-addressing-00.txt)

[Daniel Marks via NANOG]

Anecdotal but I've seen hacked AWS accounts with Cloudformation scripts 
to create and destroy lots of tiny instances to rotate through IPv4 
addresses. Being able to rotate through IP addresses is not a new thing, 
I'm sure we all have networks in mind when we think of garbage/malicious 
traffic just over IPv4 alone.


Internally at my company (corp network that went all in on IPv6), we 
have a script that looks through logs and will ban an entire /64 for a 
period of time if it has more than a few banned IP addresses in the same 
subnet (I think ~10 /128s is a 30 minute ban, but we're still tuning it).


There are some strange implementations of IPv6 that end up having a lot 
of dissociated users grouped together in a /64 (i.e. Linode, AT 
Wireless, etc) which makes me hesitant to implement automatic /64 bans 
for 1 or 2 spammy IP addresses. At one point we accidentally banned a 
large portion of US users on AT when we banned 2600:387:f:10::/64


On 2/6/2023 10:05 PM, William Herrin wrote:

On Mon, Feb 6, 2023 at 6:43 PM Fernando Gont  wrote:

On 6/2/23 20:39, Owen DeLong wrote:

After all, they’re only collecting addresses to ban at the rate they’re 
actually being used to send packets.

Yeah, but the whole point of banning is that the banned address is
actually used by an attacker subsequently,

You both have valuable points here. Listen to each other.

On the one hand, sophisticated attackers already scatter attacks
between source addresses to evade protection software. Attackers who
don't have control over their computer's IP address do not. This is
not new and IPv6 does not really change that picture.

On the other hand, there are so many addresses in a /64 that an
attacker can literally use a fresh one for each and every probe he
sends. Without a process for advancing the /128 ban to a /64 ban (and
releasing it once activity stops), reactive firewalls are likely to
become less and less effective.

Regards,
Bill Herrin

Re: Lima, OH Spectrum/Charter Severe Node/Hop Latency Issues

[Ross Tajvar]

For those who haven't seen it (i.e. Austin), here is "the guide" on how to
troubleshoot correctly with traceroute:
https://archive.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N47_Sun.pdf

ICMP is deprioritized by any normal router. Non-cascading loss does not
indicate a problem of any kind. The NOC doesn't care because nothing is
wrong, and the OSP team definitely doesn't care because ICMP is several
layers above OSP and is therefore not their problem.

On Tue, Feb 7, 2023 at 5:11 PM Kevin Shymkiw  wrote:

> ICMP response time from a router/device is not a great way to judge if
> there is an issue or not.  The devices generally have control plane
> policing and responding to ICMP is not prioritized at all.
>
> I would suggest your engineer setup something on their end of the
> connection that you can ping, and start there.
>
> Leverage something like smoke ping to something on their LAN, or even the
> public IP on their RG/Modem.
>
> Kevin
>
> On Tue, Feb 7, 2023 at 15:01 Austin Ayers via NANOG 
> wrote:
>
>> Hello all,
>>
>> One of my NetOps engineers resides in Lima, Ohio and they are receiving
>> terrible bufferbloat, packet loss, and random disconnects.
>>
>> I have been pinging 24.33.160.213 (Lima, OH Spectrum/Chart Node) and it's
>> rejecting a ton of packets. This has been going on for weeks.
>>
>> Node having problems: lag-1.limaohid01h.netops.charter.com
>>
>> NOC seems like they don't care, same with OSP in the field.
>>
>> There is no reason why this hop (#13) should have up to 613ms ping times.
>>
>> Thank you,
>> Austin
>>
>>
>>

Re: (IETF I-D): Implications of IPv6 Addressing on Security Operations (Fwd: New Version Notification for draft-gont-opsec-ipv6-addressing-00.txt)

[Fernando Gont]

Hi, Bill,

On 7/2/23 01:26, William Herrin wrote:

On Mon, Feb 6, 2023 at 7:40 PM Fernando Gont  wrote:

On 7/2/23 00:05, William Herrin wrote:

On the one hand, sophisticated attackers already scatter attacks
between source addresses to evade protection software.


Whereas in the IPv6 case , you normally have at least a /64 without
restriction. You might have a /56 or /48 thanks to your ISP, or simply a
/48 thanks to some free tunnelbroker provider...


That's not what's actually happening. 


Well, this *is* happening. -- trust me :-)



What's happening is a mix of
your computer gets one address unless you bother to enable DHCP/PD, or
your CPE gets an IPv6 block and your computer does SLAAC and/or DHCP
to assign itself a single IPv6 address. A lot of the probing is coming
from hijacked computers, so they have the address they have.

Sophisticated attackers can do more with the address blocks they get
from their own service providers. But sophisticated attackers could
spin up VMs with stolen credit cards, hijack BGP and do all manner of
things with IPv4 and IPv6 too.


You can use a /48 pretty legitimately without stealing any credit cards 
or spinning extra VMs...


Thanks,
--
Fernando Gont
SI6 Networks
e-mail: fg...@si6networks.com
PGP Fingerprint: F242 FF0E A804 AF81 EB10 2F07 7CA1 321D 663B B494

Re: Lima, OH Spectrum/Charter Severe Node/Hop Latency Issues

[Kevin Shymkiw]

ICMP response time from a router/device is not a great way to judge if
there is an issue or not.  The devices generally have control plane
policing and responding to ICMP is not prioritized at all.

I would suggest your engineer setup something on their end of the
connection that you can ping, and start there.

Leverage something like smoke ping to something on their LAN, or even the
public IP on their RG/Modem.

Kevin

On Tue, Feb 7, 2023 at 15:01 Austin Ayers via NANOG  wrote:

> Hello all,
>
> One of my NetOps engineers resides in Lima, Ohio and they are receiving
> terrible bufferbloat, packet loss, and random disconnects.
>
> I have been pinging 24.33.160.213 (Lima, OH Spectrum/Chart Node) and it's
> rejecting a ton of packets. This has been going on for weeks.
>
> Node having problems: lag-1.limaohid01h.netops.charter.com
>
> NOC seems like they don't care, same with OSP in the field.
>
> There is no reason why this hop (#13) should have up to 613ms ping times.
>
> Thank you,
> Austin
>
>
>

RE: Lima, OH Spectrum/Charter Severe Node/Hop Latency Issues

[Ryan Hamel]

Austin,

 

If you run MTRs or traceroutes through the node, is there any other
additional packet loss seen in the path, and at the destination? What does
the reverse MTR or traceroute look like? The attached image was stripped out
by the mailing list system.

 

Bufferbloat is controlled at the firewall level, which is different from
packet loss and disconnects.

 

Ryan

 

From: NANOG  On Behalf Of Austin
Ayers via NANOG
Sent: Tuesday, February 7, 2023 1:49 PM
To: nanog@nanog.org
Subject: Lima, OH Spectrum/Charter Severe Node/Hop Latency Issues

 

Hello all, 

 

One of my NetOps engineers resides in Lima, Ohio and they are receiving
terrible bufferbloat, packet loss, and random disconnects.

 

I have been pinging 24.33.160.213 (Lima, OH Spectrum/Chart Node) and it's
rejecting a ton of packets. This has been going on for weeks.

 

Node having problems: lag-1.limaohid01h.netops.charter.com

 

NOC seems like they don't care, same with OSP in the field.

 

There is no reason why this hop (#13) should have up to 613ms ping times. 

 

Thank you,

Austin

 

Lima, OH Spectrum/Charter Severe Node/Hop Latency Issues

[Austin Ayers via NANOG]

Hello all,

One of my NetOps engineers resides in Lima, Ohio and they are receiving 
terrible bufferbloat, packet loss, and random disconnects.

I have been pinging 24.33.160.213 (Lima, OH Spectrum/Chart Node) and it's 
rejecting a ton of packets. This has been going on for weeks.

Node having problems: lag-1.limaohid01h.netops.charter.com

NOC seems like they don't care, same with OSP in the field.

There is no reason why this hop (#13) should have up to 613ms ping times.

Thank you,
Austin

[cid:d929237e-d932-4060-9dbb-f4b99b12afa4]

Re: About emails impersonating Path Network

[Michael Thomas]

On 2/7/23 11:33 AM, Jay Hennigan wrote:

On 2/7/23 11:18, Michael Thomas wrote:

FWIW, lookalike domains can and do happen with http too. Nothing 
unique about that to email.


Then the bad guys throw in the occasional Cyrillic, etc. character 
that looks like a Roman one and things get even more fun.


At least with spear-phishing attacks you can bound the problem detection 
investigation since you know what your own domain's legit names are. 
Beyond that, I have no idea if any of the mailbox providers are doing 
anything about lookalike attacks. Email at least has the advantage that 
it is in hands of a user's provider who could care. CA's I'm sure 
couldn't care less.


Mike

Re: About emails impersonating Path Network

[Jay Hennigan]

On 2/7/23 11:18, Michael Thomas wrote:

FWIW, lookalike domains can and do happen with http too. Nothing unique 
about that to email.


Then the bad guys throw in the occasional Cyrillic, etc. character that 
looks like a Roman one and things get even more fun.


--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: About emails impersonating Path Network

[Michael Thomas]

On 2/7/23 6:09 AM, Rich Kulawiec wrote:

On Mon, Feb 06, 2023 at 12:41:43PM -0800, Michael Thomas wrote:

This seems like a perfect object lesson on why you should use DKIM and SPF
and make sure the sending domain can set up a p=reject policy for DMARC.

But it's not.  DKIM and SPF are mostly useless against competently executed
email forgery -- and can even be exploited to make it worse.  Thanks to
a combination of increasingly bad user interfaces, user ignorance,
TLD proliferation, and the ill-advised conflation of identification with
authenticity, it's not currently possible to stop email forgery in any
meaningful sense of the word "stop".


There has been a real failing on the UI side, but that not the fault of 
the authentication protocols. Thunderbird which is what I use is 
completely useless and silent on authentication. For others who 
implement some UI indications, some of them aren't very obvious and 
often tentative. There is a Usenix paper which researched email auth and 
part of it was on user visible feedback. The long and short is that it 
was useful, but not a silver bullet. A large part of the problem from my 
read of it is that it wasn't ubiquitous and standardized ala the Lock 
icon on browsers. It's easy to make fun of that, but it is clearly 
better than nothing. MUA vendors are always at liberty to bring their 
requirements for extensions for protocols or even new protocols if it 
would help the user experience by guiding how MUA's make the UI 
decisions on the advice of senders. It's pretty clear that they find 
this niche at best. That is a pity because some uniformity could make 
this a positive feedback loop and up the utility greatly.


FWIW, lookalike domains can and do happen with http too. Nothing unique 
about that to email.


Mike

Re: About emails impersonating Path Network

[Martin Hannigan]

On Tue, Feb 7, 2023 at 11:59 AM J. Hellenthal via NANOG 
wrote:

> Your only option is to monitor the generic tld's atp and register them
> yourself. Clone attacks are real, impersonation has been around since
> centuries and yes, its an attack vector but only impacting your customers.
> There is a vector you can pursue, its action by lawsuit. Would you rather
> pay the registration of the domain or would you rather pay the retainer
> costs of lawyers ...
>
> This is what the free web permits. Your only choice at this point is the
> retainer fee and intergovernmental practices.
>
>
> PeeringDB may be able to implement different practices but I have a pretty
> good feeling they are at their wits end to void practices that impact its
> "yellow pages" service.
>

[ PDB product committee hat on ]

I'll make sure this gets visibility with the PeeringDB  product/ops
committee and see if we can contribute here. If we can help make it harder
I'm sure we would.

Warm regards,

-M<

Re: About emails impersonating Path Network

[Rafael Possamai]

I've found this article before and implemented it for domains that we own, but 
do not use for e-mail purposes. 
https://www.gov.uk/guidance/protect-domains-that-dont-send-email

Might be worth checking it out.

Cheers,
Rafael

- Original message -
From: Konrad Zemek 
To: nanog@nanog.org
Subject: About emails impersonating Path Network
Date: Monday, February 06, 2023 12:25

Hi Nanog,

It looks like someone with an axe to grind against our company has decided to 
email every AS contact they could find on PeeringDB, impersonating us and 
sometimes spoofing our domains.

We're aware of the emails and are sorry for the inconvenience. We've since 
added SPF records to the domains we own but don't use (the perps have since 
name-squatted some new ones). We're also actively working with law enforcement 
on the matter.

Thanks
Konrad Zemek
CTO Path Network
AS396998

[NANOG-announce] Watch NANOG Hackathon, N87 Socials, + More

[Nanog News]

*WATCH Hackathon Welcome Session*
*Meet Project Vendors, Get Project Ideas, + More *

*Hybrid NANOG 87 Hackathon kicked-off last Friday, 3 Feb. and will continue
until Sunday, 12 Feb. *

Watch video to meet project vendors, get ideas for potential Hackathon
projects, and check out resources to ensure a successful Hackathon.

*WATCH NOW* 

*Connect with Old + New Friends at NANOG 87 Socials!*
*Don't Miss out on NANOG 87 Social Events Next Week*

*Our community is what makes NANOG special. *Connect with the phenomenal
women and men that make up the heart of our mission at our NANOG Social
events, taking place during NANOG 87.  Check out full details below!

* MORE INFO *


*Can't Make it to Atlanta? Still Time to Register Virtually*
NANOG 87 will Kick Off Monday!

There is still time to register for NANOG 87.

Can't make it to Atlanta? Join us virtually! Stream presentations,
participate in real-time chat forums, and enjoy the 360 camera views of
NANOG 87 General Session + more!

*REGISTER NOW* 


*What's Hot on The Mailing List? *
*Participate or Find a Tech Solution in the Community Forum Mailing List *

Remember, a NANOG account must be created to access the Community Forum.

*Current Trending Conversations:*

• Spectrum (legacy TWC) Infrastructure - Contact Off List
 Replies 26

• Smaller than a /24 for BGP?
 Replies 32

• Typical last mile battery runtime (protecting against power cuts)
 Replies 31

• 202203071610.AYC Re: Making Use of 240/4 NetBlockReplies
338

• Increasing problems with geolocation/IPv4 access
 Replies 13



*WHAT'S HOT* 
___
NANOG-announce mailing list
NANOG-announce@nanog.org
https://mailman.nanog.org/mailman/listinfo/nanog-announce

Watch NANOG Hackathon, N87 Socials, + More

[Nanog News]

*WATCH Hackathon Welcome Session*
*Meet Project Vendors, Get Project Ideas, + More *

*Hybrid NANOG 87 Hackathon kicked-off last Friday, 3 Feb. and will continue
until Sunday, 12 Feb. *

Watch video to meet project vendors, get ideas for potential Hackathon
projects, and check out resources to ensure a successful Hackathon.

*WATCH NOW* 

*Connect with Old + New Friends at NANOG 87 Socials!*
*Don't Miss out on NANOG 87 Social Events Next Week*

*Our community is what makes NANOG special. *Connect with the phenomenal
women and men that make up the heart of our mission at our NANOG Social
events, taking place during NANOG 87.  Check out full details below!

* MORE INFO *


*Can't Make it to Atlanta? Still Time to Register Virtually*
NANOG 87 will Kick Off Monday!

There is still time to register for NANOG 87.

Can't make it to Atlanta? Join us virtually! Stream presentations,
participate in real-time chat forums, and enjoy the 360 camera views of
NANOG 87 General Session + more!

*REGISTER NOW* 


*What's Hot on The Mailing List? *
*Participate or Find a Tech Solution in the Community Forum Mailing List *

Remember, a NANOG account must be created to access the Community Forum.

*Current Trending Conversations:*

• Spectrum (legacy TWC) Infrastructure - Contact Off List
 Replies 26

• Smaller than a /24 for BGP?
 Replies 32

• Typical last mile battery runtime (protecting against power cuts)
 Replies 31

• 202203071610.AYC Re: Making Use of 240/4 NetBlockReplies
338

• Increasing problems with geolocation/IPv4 access
 Replies 13



*WHAT'S HOT* 

Yondoo provided router, has "password" as admin pw, won't let us change it

[TACACS Macaque via NANOG]

Hi,

Long time lurker, first time poster. Sorry in advance if this is the wrong 
forum for something like this.

My mom's ISP (Yondoo) seems to be providing DOCSIS 3.1 CPE (Customer Premises 
Equipment) with a built-in router, without providing the ability to change the 
admin password from "password" on it.


​

Their customer service rep said that this is not only WAI, but also wanted to 
charge her $50 to have a tech come out and change it. Which is obviously less 
than ideal.

That aside, this seems like a pretty egregious security standard which, from my 
understanding, can have fairly dire security implications... e.g., DNS server 
settings can be pointed at whatever someone wants here.

My mom is elderly and had already fallen victim to a call center scammer a 
couple years ago. They briefly took control over her laptop before she called 
for backup. So I'm just a little concerned that we have no control over 
changing this router's admin password — from “password” — in a pinch, without 
waiting for a truck roll && shelling out $50.

I've sent her a DOCSIS 3.1 modem that doesn't have a router built-in, in hopes 
that they'll let us bring our own. She does have Google Wifi, but we can't even 
put their router into bridge mode. So she would be double NATed and have no 
control over changing the admin password on the first router.

Anyone have any experience with Yondoo? I've tried reaching out to them on 
multiple fronts, but have yet to hear back from them on this. A tech is 
scheduled to come out tomorrow, so the plan is to beg (bribe?) them to let us 
use our own modem and then take it from there.

Thanks,
Todd

Re: Caribnog email list

[Stephen Lee]

Thanks Biil, David. This has been sorted.

Best,
Stephen

On Sat, 4 Feb 2023 at 13:30, Bill Woodcock  wrote:
>
> Forwarded to the maintainers.
>
> -Bill
>
>
>
> > On Feb 4, 2023, at 6:44 PM, David Bass  wrote:
> >
> > Anyone on here run it?  The URL to sign up on the website doesn’t seem to 
> > work at the moment.
>
>

Re: About emails impersonating Path Network

[Rich Kulawiec]

On Mon, Feb 06, 2023 at 12:41:43PM -0800, Michael Thomas wrote:
> This seems like a perfect object lesson on why you should use DKIM and SPF
> and make sure the sending domain can set up a p=reject policy for DMARC.

But it's not.  DKIM and SPF are mostly useless against competently executed
email forgery -- and can even be exploited to make it worse.  Thanks to
a combination of increasingly bad user interfaces, user ignorance,
TLD proliferation, and the ill-advised conflation of identification with
authenticity, it's not currently possible to stop email forgery in any
meaningful sense of the word "stop".

Let me illustrate part -- *part*, only part because a full explanation
would take many pages -- of this problem by example.  There are many
variations on this theme, so after reading it you're tempted to say
"But":  don't.  Because for every one of those "buts" there's a counter,
and again, expounding those would take many pages.  Focus on the concept
here, not the precise details of the particular example I've chosen.

Let's suppose that example.net is the target of an impersonation campaign.
And let's suppose that I'm the attacker.  Thanks to ICANN and unchecked
registrar greed, I can now register example.lol or example.guide or
example.life or any of a thousand variations.  Thanks to unscrupulous
hosting operations who don't care who their customers are or what
they're doing and can't be bothered to perform even perfunctory due
diligence, I'll have no problem getting a web/mail host for example.lol.

I can then clone example.net's web site, modifying the URLs as appropriate.
I can also set up MX records, DKIM, SPF, etc. all of which are syntactically
and semantically correct for example.lol.  It shouldn't be too difficult
to figure out which email addresses are valid at example.net: I could
scrape their web site, or go through the NANOG or IETF or other mailing
lists, or scrape social media, or just write to them from a freemail account
somewhere and see what turns up.  I can replicate those at example.lol.
 
So what I have now is a web site and mail system that copies example.net
in every detail EXCEPT for the TLD.  And that matters little, (a) because
all-singing all-dancing email interfaces are increasingly getting dumber
and hiding the actual email addresses of correspondents and (b) even if
they expose it, e.g.:

From: Some Person 

nearly every user out there will not pick up on the TLD.  (If you think
"lol" is a bit too obvious, then go through the list.  There are plenty
of others that aren't.  Not that it matters much, because I could always
just namesquat on example-pro in the .net TLD or some other variation
on that theme, just like was done to Path Networks in this instance.
And nearly every user out there won't catch that either.)

And as the chef's kiss on this, an increasing number of email user
interfaces will check the DKIM record and dutifully mark messages
like this as "authentic" -- which, from the viewpoint of DKIM, they
absolutely are.  Users will see the green address bar or checkmark or
whatever signifies this, and their brains will turn off, and they'll
proceed on the assumption that such messages are authentic.

TL;DR: email anti-forgery technologies are useless wallpaper slapped
over an underlying set of serious problems that nobody has any interest
in fixing because they're highly profitable problems.  They've completely
failed to justify their cost/complexity and are readily exploited as part
of attacks.

---rsk