RE: Are any of you starting to get AI robocalls?

[Sean Pedersen]

Yep. Add it to the list of IRS scams, fake arrest warrants, credit repair, free 
vacations, etc. The rate of calls has increased dramatically in the past year, 
especially with the "neighborhood scam" where they spoof their CLID to a local 
area code and prefix +  through  and blast you with calls, trying to 
trick you into thinking it's someone local and thus important or legitimate.

I have a second phone I use for work and on-call, so that goes on DND from 6PM 
to 6AM with a VIP list of people/numbers that can ring through. No problems 
there, and somehow that number isn't (yet) on anyone's list, so I don't get 
many calls.

On my personal cell, I started to use an app called Hiya that has been pretty 
successful. It's available for both iPhone and Android. It powers a lot of the 
carrier-specific apps like AT Call Protect, but unlike them, it doesn't suck. 
It's a giant database of reports that rate calling numbers and classify them as 
fraud, scam, neighborhood spoofing, etc. and you can flag them or route them 
right to voicemail. The only time it doesn’t work is when it hasn't updated its 
list in a little while and a few sneak through. They just realized a premium 
version that added some features. I haven't explored it yet.

Went from about 20 calls a week to almost nothing.

Carriers seem to be either uncapable or unwilling to address the issue other 
than the occasional lip-service reply about "taking customer's $variable 
seriously." 

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of William Herrin
Sent: Tuesday, April 3, 2018 3:32 PM
To: nanog@nanog.org
Subject: Are any of you starting to get AI robocalls?

Howdy.

Have any of you started to get AI robocalls? I've had a couple of
calls recently where I get the connect silence of a predictive dialer
followed by a woman speaking with call center background noise. She
gives her name and asks how I'm doing. The first time it happened it
seemed off for reasons I can't quite articulate, so I asked: "Are you
a robot or a person?" She responded "yes" and then launched in to a
sales pitch. The next time I asked, "where can I direct your call?"
She responded "that's good" and launched in to her pitch.

Regards,
Bill Herrin


-- 
William Herrin  her...@dirtside.com  b...@herrin.us
Dirtside Systems . Web: 

RE: Proof of ownership; when someone demands you remove a prefix

[Sean Pedersen]

I appreciate everyone's input and will incorporate it into our internal 
policies going forward. 

I also want to assure everyone who has taken the time to read or respond that 
we're going about this methodically; our customer is involved and is responding 
promptly and their customer is has opened a case with the RIR. We're in the 
process of following up with the RIR. Our goal is not to cause an 'operational 
headache' for anyone, but exactly the opposite.

Thanks again for all of your feedback and responses.

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Naslund, Steve
Sent: Tuesday, March 13, 2018 11:59 AM
To: nanog@nanog.org
Subject: Re: Proof of ownership; when someone demands you remove a prefix

The fact that it is a newer customer would make me talk to the RIR direct and 
verify that a dispute is really in progress.  I would also look at some looking 
glasses and see if the prefix is being announced elsewhere, if so that might 
indicate that your customer is indeed stepping on a legit owner.  I would also 
make it clear to the new customer that they are on thin ice here to light a 
fire under their process.  Let them know that it is up to them to convince you 
that they are the legit owner.  No one wants to lose a customer but they are 
threatening your business and putting you in legal jeopardy if they are not 
legit.

Steven Naslund
Chicago IL

>-Original Message-
>From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Sean Pedersen
>Sent: Tuesday, March 13, 2018 12:39 PM
>To: nanog@nanog.org
>Subject: RE: Proof of ownership; when someone demands you remove a prefix
>
>This is more or less the situation we're in. We contacted the customer and 
>they informed us the matter is in dispute with the RIR and that their 
>>customer (the assignee) is in the process of resolving the issue. We have to 
>allow them time to accomplish this. I've asked for additional information >to 
>help us understand the nature of the dispute. In that time we received another 
>request to stop announcing the prefix(s) in addition to a new set of 
>>prefixes, and a threat to contact our upstream providers as well as ARIN - 
>which is not the RIR the disputed resources are allocated to.
>
>This is a new(er) customer, so there is some merit to dropping the prefix and 
>letting them sort it out based on the current RIR contact(s). However, >there 
>is obvious concern over customer service and dropping such a large block of 
>IPs. 
>
>I'm definitely leaning toward "let the customer (or customer's customer) and 
>the RIR sort it out" if the POC validates the request weighed responsibly 
>>against customer age. However, from a customer service perspective, I think 
>we owe it to our customers to make sure a request is legitimate before we 
>>knock them offline. With a limited toolset to validate that information, I 
>can't help but feel conflicted.
>
>I appreciate all the feedback this thread has generated so far!

RE: Proof of ownership; when someone demands you remove a prefix

[Sean Pedersen]

This is more or less the situation we're in. We contacted the customer and they 
informed us the matter is in dispute with the RIR and that their customer (the 
assignee) is in the process of resolving the issue. We have to allow them time 
to accomplish this. I've asked for additional information to help us understand 
the nature of the dispute. In that time we received another request to stop 
announcing the prefix(s) in addition to a new set of prefixes, and a threat to 
contact our upstream providers as well as ARIN - which is not the RIR the 
disputed resources are allocated to.

This is a new(er) customer, so there is some merit to dropping the prefix and 
letting them sort it out based on the current RIR contact(s). However, there is 
obvious concern over customer service and dropping such a large block of IPs. 

I'm definitely leaning toward "let the customer (or customer's customer) and 
the RIR sort it out" if the POC validates the request weighed responsibly 
against customer age. However, from a customer service perspective, I think we 
owe it to our customers to make sure a request is legitimate before we knock 
them offline. With a limited toolset to validate that information, I can't help 
but feel conflicted.

I appreciate all the feedback this thread has generated so far!

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Naslund, Steve
Sent: Tuesday, March 13, 2018 8:27 AM
To: nanog@nanog.org
Subject: RE: Proof of ownership; when someone demands you remove a prefix

Yes, absolutely go with the RIR.  Only thing I might adjust it whether I let 
the customer launch a dispute with the RIR before or after I make the change 
and to me that would depend on the preponderance of the evidence either way.  I 
might give the long term customer the reasonable doubt.  A new customer with a 
new advertisement not so much.  Talk to your legal people of course but I would 
think if the RIR could verify a dispute in progress, you are covered until the 
dispute is resolved.  Seems legally reasonable to me and shows due diligence on 
your part without you getting in the middle.

Steven Naslund
Chicago IL

>Hi Sean,
>
>There is a definitive technical means. It's called contact the POC published 
>in WHOIS by the RIR and ask. It isn't flawless and you don't have to like >it, 
>but there it is.
>
>If you contacted the POC and the POC replied stop, you stop. If the POC was 
>hijacked at the RIR, that's between your customer and the RIR.
>The RIR has a standard process and an expert team for dealing with these 
>situations. It's their job.
>
>Regards,
>Bill Herrin

RE: Proof of ownership; when someone demands you remove a prefix

[Sean Pedersen]

In this case we defaulted to trusting our customer and their LOA over a 
stranger on the Internet and asked our customer to review the request. 
Unfortunately, that doesn't necessarily mean a stranger on the Internet isn't 
the actual assignee. A means to definitively prove "ownership" from a technical 
angle would be great.

In the example provided in my original e-mail, it appears that an IP broker or 
related scammer gained access to the assignee's RIR account and made some 
object updates (e-mail, country, etc.) that they could use to "prove" they had 
authority to make the request. I assume their offer of proof would have been to 
send us an email from the dubious @yahoo.com account they had listed as the 
admin contact. 

I agree with a private response that I received that at some point lawyers 
probably need to take over if a technical solution to verification is not 
reached. 

I'm not terribly current on resource certification, but would RPKI play a role 
here? It looks like its application is limited to authenticating the 
announcement of resources to prevent route hijacking. If you've authorized a 
3rd party to announce your routes, could you assign a certificate to that 3rd 
party for a specific resource and then revoke it if they are no longer 
authorized? Would it matter if someone gains access to your RIR/LIR account and 
revokes the certificate? This would assume protocol compatibility, that 
everyone is using it, etc. 

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Jason Hellenthal
Sent: Monday, March 12, 2018 6:40 PM
To: George William Herbert 
Cc: nanog@nanog.org
Subject: Re: Proof of ownership; when someone demands you remove a prefix

How about signed ownership ? (https://keybase.io) if you are able to update the 
record … and it is able to be signed then shouldn’t that be proof enough of 
ownership of the ASN ?

If you can update a forward DNS record then you can have the reverse record 
updated in the same sort of fashion and signed by a third party to provide 
first party of authoritative ownership… Assuming you have an assigned ASN and 
the admin has taken the time to let alone understand the concept and properly 
prove the identity in the first place… (EV cert ?)


Just a light opinion from … https://jhackenthal.keybase.pub

Trust is a big issue these days and validation even worse given SSL trust.

-- 

The fact that there's a highway to Hell but only a stairway to Heaven says a 
lot about anticipated traffic volume.





> On Mar 12, 2018, at 21:20, George William Herbert  
> wrote:
> 
> Ownership?...
> 
> (Duck)
> 
> -george 
> 
> Sent from my iPhone
> 
>> On Mar 12, 2018, at 4:11 PM, Randy Bush  wrote:
>> 
>> it's a real shame there is no authorative cryptographically verifyable
>> attestation of address ownership.

RE: Proof of ownership; when someone demands you remove a prefix

[Sean Pedersen]

Without revealing too much identifying information, the prefix is allocated to 
a 3rd party that is a customer of our customer. We have a signed LOA on hand 
that matches the RIR database object details (names, prefix, etc.), and the 
request to stop announcing came from another 3rd party that does not appear to 
be related to either our customer or their customer.

Both the individual making the demand as well as the 3rd party that "owns" the 
prefix are in industries that suggest things are not entirely above-board. The 
email came from a IP broker domain whose TLD is an eastern European country.

At this point I'm going to have to rely on our customer's POC, whom I've 
already contacted, to verify whether or not this is true and err in their 
favor. 

I was just curious what others have experienced. Since so much of the Internet 
is "best effort" in terms of validation, I wasn't sure if there was much else 
that could be done.

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of n...@imap.cc
Sent: Monday, March 12, 2018 12:08 PM
To: nanog@nanog.org
Subject: Re: Proof of ownership; when someone demands you remove a prefix

I've seen this type of situation come up more than a few times with the shadier 
IP brokers that lease and don't care who they lease to, for example Logicweb, 
Cloudinnovation ( see 
bgp.he.net/search?search[search]=cloudinnovation+OR+%22cloud+innovation%22 ), 
Digital Energy-host1plus. The ranges get abused to hell and back for garbage 
traffic selling, rate limit bypassing, scraping, proxies, banned from 
youtube/google/etc for view and like farms, and then thrown away, and the 
leaser tries to get them unannounced quickly for further resale.



On Mon, Mar 12, 2018, at 11:57 AM, Matt Harris wrote:
> On Mon, Mar 12, 2018 at 1:46 PM, Sean Pedersen <spedersen.li...@gmail.com>
> wrote:
> 
> > We recently received a demand to stop announcing a "fraudulent" prefix. Is
> > there an industry best practice when handling these kind of requests? Do
> > you
> > have personal or company-specific preferences or requirements? To the best
> > of my knowledge, we've rarely, if ever, received such a request. This is
> > relatively new territory.
> >
> 
> This could definitely be an attempt at a DoS attack, and wouldn't be the
> first time I've heard of something like this being done as such.
> 
> I thought about requesting they make changes to their RIR database objects
> > to confirm ownership, but all that does is verify that person has access to
> > the account tied to the ORG/resource, not ownership. Current entries in the
> > database list the same ORG and contact that signed the LOA. When do you get
> > to the point where things look "good enough" to believe someone?
> >
> 
> They may also be leasing one chunk of space from an organization without
> actually having access to the RIR db too - in that case, they could ask the
> org they are leasing from to put in a SWIP with the RIR, but if they don't
> choose to, then that's not a hard requirement.
> 
> On the same token, having access to the org account at the RIR pretty much
> makes you as legitimate as you're going to be as far as any of us can
> really tell.  If there's an issue where the RIR account has been
> compromised, then that issue lies between the RIR and their customer, and
> isn't really your business because you have no way to know whatsoever.
> 
> 
> > Has anyone gone so far as to make the requestor provide something like a
> > notarized copy stating ownership? Have you ever gotten legal departments
> > involved? The RIR?
> >
> 
> A notarized copy stating *ownership* seems overboard.  Lots of
> organizations lease IPv4 space, and lots more now since depletion in many
> regions, and their use of it is entirely legitimate in accordance with
> their contractual rights established in the lease agreement with the
> owner.  I'd probably think about looking at the contact info in the RIR
> whois and ask them, if I had a situation like this myself.  Ultimately, the
> RIR's contact which would be in their whois db should be authoritative more
> so than anyone else.  I doubt the RIR would be able to say much if you
> contacted them beyond that everything that isn't in whois isn't something
> they'd share publicly.
> 
> Take care,
> Matt

Proof of ownership; when someone demands you remove a prefix

[Sean Pedersen]

We recently received a demand to stop announcing a "fraudulent" prefix. Is
there an industry best practice when handling these kind of requests? Do you
have personal or company-specific preferences or requirements? To the best
of my knowledge, we've rarely, if ever, received such a request. This is
relatively new territory.

 

In this case we have a signed LOA on file for that prefix and I've reached
out to our customer to verify the validity of the sender's request. The
sender claims to have proof that they are authorized to speak on behalf of
the owner. I will wait until I hear from our customer before I consider a
response to the sender. I don't get a real sense of legitimacy from the
sender making the request. No one else announces the prefix. Nothing about
the request appears to be legitimate, especially considering the sender.

 

I thought about requesting they make changes to their RIR database objects
to confirm ownership, but all that does is verify that person has access to
the account tied to the ORG/resource, not ownership. Current entries in the
database list the same ORG and contact that signed the LOA. When do you get
to the point where things look "good enough" to believe someone?

 

Has anyone gone so far as to make the requestor provide something like a
notarized copy stating ownership? Have you ever gotten legal departments
involved? The RIR?

 

RE: Gonna be a long day for anybody with CPE that does WPA2..

[Sean Pedersen]

Cisco's PSIRT:
https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-
sa-20171016-wpa

Some fixes appear to be available, or will be soon.

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of
valdis.kletni...@vt.edu
Sent: Monday, October 16, 2017 12:38 AM
To: nanog@nanog.org
Subject: Gonna be a long day for anybody with CPE that does WPA2..

Looks like WPA2 may have just become the new WEP.

And it looks like we're all going to be reflashing a lot of devices.

"The proof-of-concept exploit is called KRACK, short for Key Reinstallation
Attacks. The research has been a closely guarded secret for weeks ahead of a
coordinated disclosure that's scheduled for 8 a.m. Monday, east coast time.
An
advisory the US CERT recently distributed to about 100 organizations
described
the research this way:

"US-CERT has become aware of several key management vulnerabilities in the
4-way handshake of the Wi-Fi Protected Access II (WPA2) security protocol.
The
impact of exploiting these vulnerabilities includes decryption, packet
replay,
TCP connection hijacking, HTTP content injection, and others. Note that as
protocol-level issues, most or all correct implementations of the standard
will
be affected. The CERT/CC and the reporting researcher KU Leuven, will be
publicly disclosing these vulnerabilities on 16 October 2017."

https://arstechnica.com/information-technology/2017/10/severe-flaw-in-wpa2-p
rotocol-leaves-wi-fi-traffic-open-to-eavesdropping/

RE: Internet access for security consultants - pen tests, attack traffic, bulk e-mail, etc.

[Sean Pedersen]

Quick note to thank everyone for their input. So far everything that has been 
suggested publicly and privately has been right in line with what were already 
putting on paper so this provided some great feedback and confirmation that 
we’re going in the right direction.

Internet access for security consultants - pen tests, attack traffic, bulk e-mail, etc.

[Sean Pedersen]

We were recently approached by a company that does security consulting. Some
of the functions they perform include discovery scans, penetration testing,
bulk e-mail generation (phishing, malware, etc.), hosting fake botnets -
basically, they'd be generating a lot of bad network traffic. Targeted at
specific clients/customers, but still bad. As an ISP, this is new territory
for us and there are some concerns about potential impact, abuse reports,
reputation, authorization to perform such tests, etc. 

 

Does anyone have experience in this area that would be willing to offer
advice?

 

 

RE: VXLAN for WAN Pseudowires?

[Sean Pedersen]

Been there, got the t-shirt ...

VXLAN was not designed to be a direct replacement for L2VPN. It was built to
scale L2 broadcast domains. With Cisco, L2 control protocols like STP are
not supported. Can't speak for other vendors. So what someone would
typically expect from EoMPLS and L2 protocol tunneling, they're not going to
get with a VXLAN substitute. 

If all you're looking to do is create a virtual Ethernet cable between two
L3 interfaces with absolutely no L2 protocol tunneling involved, it works
like a champ. We use them for that purpose, building virtual cross-connects
between routers, as well as in a few virtual environments to overlay L2
networks across a BGP CLOS. 

I would not recommend VXLAN to replace L2VPNs unless you plan to wait for /
hope for L2 protocol tunneling support. You would not get a direct
replacement for your current VPLS circuits. Segment routing is in the same
boat - no L2VPN support until next year, and then it's supposed to be
limited to -EX and -FX Nexus chassis.

I can't speak on the subject of using it across a WAN; we dump our VXLAN
tunnels to ASR9Ks at the edge where traditional MPLS takes over. Generally,
it's not recommended to try and extend a LAN-based protocol across a WAN.
While it's L3 traffic once the VXLAN overlay takes over, I would look into
the control plane requirements, overhead, etc. before even considering it.

NCS is what Cisco currently recommends for a scalable MPLS fabric. There's
also the ASR9000V, which acts as a satellite / remote line card of larger
ASR9K routers, but you'd still want some kind of aggregation in there to
scale or your ASR9K port cost is going to start to hurt.

Hit me up off-list if you want to know a little more in detail.

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Simon Lockhart
Sent: Thursday, July 20, 2017 2:12 AM
To: nanog@nanog.org
Subject: VXLAN for WAN Pseudowires?

All,

I'm currently going through a network design for an upgrade for one of the
networks I run. Much of the wide-area traffic on the network is used purely
to transport Ethernet tail circuits back from an edge PoP to a core PoP. 
Currently we're using Extreme X460 and X670 switches to achieve this,
carrying the tail circuits within VPLS.

Two things are making me look at a change of solution for this - firstly
Extreme have stated that they're not interested in the service provider
market any more (and reflected this in significant reductions in discounts),
and secondly we need to look at higher bandwidth port options (40G + 100G,
particularly for backhaul circuits).

As we're primarily a Cisco house, I've been looking at suitable
replacements, and the Nexus 9k range looks good - 92160YC and 9236C in
particular. However, this would mean a shift from VPLS to VXLAN. We're also
looking at Cisco-like products, such as the Arista range.

We've been doing some testing in the lab, and so far, things look good -
it's easy to configure, and appears to do the job of getting packets from A
to B.

We do have two concerns, though:

1) Cisco are strongly advising against using the Nexus switches in a WAN
   scenario - as they're designed for "datacentre" use. They've so far said 
   they can't find anyone who can help validate designs using Nexus, and 
   instead are pushing us towards the NCS-5000 series switches. Same
chipset,
   but 2-3 times the price! NCS does, however, support VPLS, so would be an
   easier drop-in to our existing network.

2) Traffic engineering - we don't have a lot of requirement for this, but do
   have a small number of customers who buy A and B circuits, and require
them
   to be routed across different paths on our network. This is easy with
MPLS
   using explicit LSPs, but we've not yet worked out how to achieve the same
   thing in VXLAN.

So, my question to the community is - have any of you used VXLAN as a
wide-area layer 2 transport technology? Any pros or cons? Gotchas? Scare
stories?
Recommendations? Am I trying to shoot myself in the foot?

Many thanks,

Simon

RE: ISP billing - data collection, correlation, and billing

[Sean Pedersen]

Flexible NetFlow is the best option for us. I haven’t seen a lot of products 
outside of open source that work with telemetry at the moment, and nothing that 
has billing functions. We’re far too lean to be able to rely on open source / 
devops at this point. That’s been a hard lesson to teach the company. 

 

It’s definitely on our internal roadmap, though.

 

From: Michael Krygeris [mailto:m...@krygerism.com] 
Sent: Sunday, July 16, 2017 9:35 AM
To: Lee Howard <l...@asgard.org>; Luke Guillory <lguill...@reservetele.com>; 
Sean Pedersen <spedersen.li...@gmail.com>; nanog@nanog.org
Subject: Re: ISP billing - data collection, correlation, and billing

 

 

On Sat, Jul 15, 2017 at 4:48 AM Lee Howard <l...@asgard.org 
<mailto:l...@asgard.org> > wrote:

 

Depending on the capability of the exporter, you don't need to export full flow 
information. With Cisco's Flexible Netflow you can define the aggregation in 
the flow cache you are monitoring. You are not required to use a 7-tuple.

An aggregation could be something basic like this:

Source interface

Destination interface 

Octets

Packets

 

This would give you SNMP equivalent for byte accounting on interfaces without 
requiring full flow accounting IF you're not forced to do sampling and IF you 
have flexible netflow.

 

Another much more recent method around SNMP (sorry SNMP, I'm over you) is 
streaming telemetry, which is part of Netconf/YANG/OpenConfig.

This is more of a push method for these yang data models(the relevent one here 
being snmp interfaces table).

It already exists on some Juniper and Cisco platforms.

Mike Krygeris

 



On 7/14/17, 2:47 PM, "NANOG on behalf of Luke Guillory"
<nanog-boun...@nanog.org <mailto:nanog-boun...@nanog.org>  on behalf of 
lguill...@reservetele.com <mailto:lguill...@reservetele.com> > wrote:

>On the HFC / CMTS side of things we have IPDR which I believe has some
>open source collectors out there. I'm not sure that IPDR is used much
>outside of the HFC world though.

IPDR was my first thought as an alternative to SNMP. Is its accuracy
comparable? It’s been included into TR-069, so it’s theoretically
available to telcos, too. And usage-based billing is part of it’s purpose.


Not sure I’d want to use NetFlow/IPFIX, since by nature it tracks flows,
not bits, and I don’t want to record flows. But I’d be interested in
hearing others’ experience.


Lee

ISP billing - data collection, correlation, and billing

[Sean Pedersen]

I went back a few years in the archives and found a few odd references, but
not much discussion. I'm curious what some other approaches are to
usage-based billing, both the practice of generating/correlating data and
the billing itself.

 

We bill based on use/95th percentile and our system is rudimentary on its
best day. We use SNMP and interface descriptions to generate data and
correlate it with customers. This works for the most part, but leaves a lot
to be desired. 

 

Ours is one method, and I've seen others who use NetFlow data in a similar
fashion, with the assumption that the NetFlow data is correlated either via
IP address or source interface. 

 

Most of the systems I've seen are full-fledged CRM, billing, and OSS, which
is a little overkill for us at the moment. I think we would have issues
trying to integrate such a multi-headed beast into our organization at this
time.

 

What methods do you use to collect and correlate data?

What systems, if any, do you use?

 

I'm a little in the dark as the billing/OSS side of things is outside of my
normal scope and would appreciate any recommendations.

 

Thanks!

RE: Looking for Cisco ASR9000v feedback

[Sean Pedersen]

Yeah - look for bundles if possible. I know it cut about 3/4 of the cost off of 
an NCS5K that we were looking at in a ASR9K satellite config.

Also, if you're doing satellite on the 9000V, I believe support for that 
feature is going away in a future version of IOS-XR. Double-check w/ your 
account team.

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Tom Hill
Sent: Tuesday, June 6, 2017 7:53 AM
To: nanog@nanog.org
Subject: Re: Looking for Cisco ASR9000v feedback

On 06/06/17 15:34, Erik Sundberg wrote:
> Looking for the pro's, con's, and the gotcha's of moving our 1G ports to the 
> 9000V.

The nV licenses for one. Talk about printing money.

-- 
Tom