RE: What are these Google IPs hammering on my DNS server?

[Michael Hare via NANOG]

Damian-

Not Google or ISCs fault, our customers have made some decisions that have 
exasperated the issues.  By and away the biggest problem facing my customers is 
that they have chosen a stateful border firewall that collapses due to session 
exhaustion and they put everything, including aDNS, behind said firewall.  “If 
it hurts, don’t do it” comes to mind, but out of my hands.

At quick glance following the ISC link I didn’t see the compute infrastructure 
[core count] needed to get 1Mpps.  There is an obvious difference between 99% 
load of ~500rps and 1M, so we can maybe advise to not undersize ADNS if that's 
an issue.

I'm an ISP engineer and am generally not the directly affected party, so I 
don't get to pick these implementation details for my customers.  I appreciate 
the background and suggestions from you and others on this thread like Mark.  
That's an interesting comment about DNSSEC that I hadn't considered.

-Michael

From: Damian Menscher 
Sent: Monday, December 4, 2023 12:21 PM
To: Michael Hare 
Cc: John R. Levine ; nanog@nanog.org
Subject: Re: What are these Google IPs hammering on my DNS server?

Google Public DNS (8.8.8.8) attempts to identify and filter abuse, and while we 
think we're fairly effective for large attacks (eg, those above 1Mpps), it gets 
more challenging (due to risk of false positives) to adequately filter small 
attacks.  I should note that we generally see the attack traffic coming from 
botnets, or forwarding resolvers that blend the attack traffic with legitimate 
traffic.

Based on ISC BIND load-tests [0], a single DNS server can handle O(1Mpps).  
Also, no domain should be served by a single DNS server, so O(1Mpps) seems like 
a safe lower-bound for small administrative domains (larger ones will have more 
redundancy/capacity).  Based on these estimates, we haven't treated mitigation 
of small attacks as a high priority.  If O(25Kpps) attacks are causing real 
problems for the community, I'd appreciate that feedback and some hints as to 
why your experience differs from the ISC BIND load-tests.  With a better 
understanding of the pain-points, we may be able to improve our filtering a 
bit, though I suspect we're nearing the limits of what is attainable.

Since it was mentioned up-thread, I'd caution against dropping queries from 
likely-legitimate recursives, as that will lead to a retry storm that you won't 
like (based on a few reports of authoritatives who suffered outages, the retry 
storm increased demand by 30x and they initially misdiagnosed the root cause as 
a DDoS).  The technically correct (if not entirely practical) mitigation for a 
DNS cache-busting attack laundered through open recursives is to deploy DNSSEC 
and issue NSEC/NSEC3 responses to allow the recursives to cache the 
non-existence of the randomized labels.

[0] https://www.isc.org/blogs/bind-performance-september-2023/

Damian
--
Damian Menscher :: Security Reliability Engineer :: Google :: AS15169

On Sun, Dec 3, 2023 at 1:22 PM Michael Hare via NANOG 
mailto:nanog@nanog.org>> wrote:
John-

This is little consolation, but at AS3128, I see the same thing to our 
downstream at times, claiming to come from both 13335 and 15169 often 
simultaneously at the tune of 25Kpps , "assuming it's not spoofed", which is 
pragmatically impossible to prove for me given our indirect relationships with 
these companies.  When I see these events, I typically also see a wide variety 
of country codes participating simultaneously.  Again, assuming it's not 
spoofed.  To me it just looks like effective harassment with 13335/15169 
helping out.  I pine for the internet of the 1990s.

Recent events in GMT for us were the following, curious if you see the same
~ Nov 26 05:40
~ Nov 30 00:40
~ Nov 30 05:55

Application agnostic, on the low $ end for "fixes", if it's either do something 
or face an outage, I've found some utility in short term automated DSCP 
coloring on ingress paired with light touch policing as close to the end host 
as possible, which at least keeps things mostly working during times of 
conformance.  Cheap/fast and working ... most of the time.  Definitely not 
great or complete at all, and a role I'd rather not play as an educational 
ISP/enterprise.

So what are most folks doing to survive crap like this?  Nothing/waiting it 
out?  Oursourcing DNS?  Scrubbing appliance?  Poormans stuff like I mention 
above?

-Michael

> -Original Message-
> From: NANOG 
> mailto:wisc@nanog.org>> On
> Behalf Of John R. Levine
> Sent: Sunday, December 3, 2023 1:18 PM
> To: Peter Potvin 
> mailto:peter.pot...@accuristechnologies.ca>>
> Cc: nanog@nanog.org<mailto:nanog@nanog.org>
> Subject: Re: What are these Google IPs hammering on my DNS server?
>
> > Did a bit of digging on Google's developer site and came across this:
> > https://developers.google.com/speed/public-
> dns/faq#locations_of_ip_address_

RE: What are these Google IPs hammering on my DNS server?

[Michael Hare via NANOG]

John-

This is little consolation, but at AS3128, I see the same thing to our 
downstream at times, claiming to come from both 13335 and 15169 often 
simultaneously at the tune of 25Kpps , "assuming it's not spoofed", which is 
pragmatically impossible to prove for me given our indirect relationships with 
these companies.  When I see these events, I typically also see a wide variety 
of country codes participating simultaneously.  Again, assuming it's not 
spoofed.  To me it just looks like effective harassment with 13335/15169 
helping out.  I pine for the internet of the 1990s.

Recent events in GMT for us were the following, curious if you see the same
~ Nov 26 05:40
~ Nov 30 00:40
~ Nov 30 05:55

Application agnostic, on the low $ end for "fixes", if it's either do something 
or face an outage, I've found some utility in short term automated DSCP 
coloring on ingress paired with light touch policing as close to the end host 
as possible, which at least keeps things mostly working during times of 
conformance.  Cheap/fast and working ... most of the time.  Definitely not 
great or complete at all, and a role I'd rather not play as an educational 
ISP/enterprise.

So what are most folks doing to survive crap like this?  Nothing/waiting it 
out?  Oursourcing DNS?  Scrubbing appliance?  Poormans stuff like I mention 
above?

-Michael 

> -Original Message-
> From: NANOG  On
> Behalf Of John R. Levine
> Sent: Sunday, December 3, 2023 1:18 PM
> To: Peter Potvin 
> Cc: nanog@nanog.org
> Subject: Re: What are these Google IPs hammering on my DNS server?
> 
> > Did a bit of digging on Google's developer site and came across this:
> > https://developers.google.com/speed/public-
> dns/faq#locations_of_ip_address_ranges_google_public_dns_uses_to_send_
> queries
> >
> > Looks like the IPs you mentioned belong to Google's public DNS resolver
> > based on that list on their site. They could also be spoofed though from a
> > DNS AMP attack, so keep that in mind.
> 
> Per my recent message, the replies are tiny so if it's an amplification
> attack, it's a very incompetent one.  The queries are case randomized so I
> guess it's really Google.  Sigh.
> 
> If anyone is wondering, I have a passive aggressive countermeasure against
> some overqueriers that returns ten NS referral names, and then 25 random
> IP addresses for each of those names, but I don't do that to Google.
> 
> R's,
> John
> 
> > --
> > *Accuris Technologies Ltd.*
> >
> >
> > On Sun, Dec 3, 2023 at 1:51 PM John Levine  wrote:
> >
> >> At contacts.abuse.net, I have a little stunt DNS server that provides
> >> domain contact info, e.g.:
> >>
> >> $ host -t txt comcast.net.contacts.abuse.net
> >> comcast.net.contacts.abuse.net descriptive text "ab...@comcast.net"
> >>
> >> $ host -t hinfo comcast.net.contacts.abuse.net
> >> comcast.net.contacts.abuse.net host information "lookup" "comcast.net"
> >>
> >> Every once in a while someone decides to look up every domain in the
> >> world and DoS'es it until I update my packet filters. This week it's
> >> been this set of IPs that belong to Google. I don't think they're
> >> 8.8.8.8. Any idea what they are? Random Google Cloud customers? A
> >> secret DNS mapping project?
> >>
> >>  172.253.1.133
> >>  172.253.206.36
> >>  172.253.1.130
> >>  172.253.206.37
> >>  172.253.13.196
> >>  172.253.255.36
> >>  172.253.13.197
> >>  172.253.1.131
> >>  172.253.255.35
> >>  172.253.255.37
> >>  172.253.1.132
> >>  172.253.13.193
> >>  172.253.1.129
> >>  172.253.255.33
> >>  172.253.206.35
> >>  172.253.255.34
> >>  172.253.206.33
> >>  172.253.206.34
> >>  172.253.13.194
> >>  172.253.13.195
> >>  172.71.125.63
> >>  172.71.117.60
> >>  172.71.133.51
> >>
> >> R's,
> >> John
> >>
> >
> 
> Regards,
> John Levine, jo...@taugh.com, Primary Perpetrator of "The Internet for
> Dummies",
> Please consider the environment before reading this e-mail. https://jl.ly

RE: Long hops on international paths

[Michael Hare via NANOG]

Paul-

You said: "... would decide to configure MPLS paths between Chicago and distant 
international locations ..."

AS3128 runs MPLS and it's probable someone might correct me here, but for a IGP 
backbone area I think it's common for there to be a full mesh of LSPs via 
either LDP, RSVP, SR etc.  AS3128 is a small regional and we operate in that 
way across 60+ nodes.  I don't know if it's common for someone with a global 
footprint like 1299 to have a contiguous global MPLS backbone, but the point of 
my reply was to say it's not impossible to think 1299 has a global MPLS mesh 
between major POPs.

-Michael

> -Original Message-
> From: NANOG  On
> Behalf Of PAUL R BARFORD
> Sent: Tuesday, January 18, 2022 8:16 AM
> To: Saku Ytti 
> Cc: Esteban Carisimo ;
> nanog@nanog.org; Fabian E. Bustamante 
> Subject: Re: Long hops on international paths
> 
> Hello Saku,
> 
> Thank you for the summary.  We're clear about the fact that what we're
> seeing are MLPS paths - that was not in question.  What we are not clear
> about and the reason for the post is why the provider - zayo.telia in this 
> case
> - would decide to configure MPLS paths between Chicago and distant
> international locations.  We assumed we would see hops in traceroute
> between Chicago and coastal locations and then hops that transited
> submarine infrastructure followed by hops to large population centers.
> 
> Regards, PB
> 
> 
> From: Saku Ytti 
> Sent: Tuesday, January 18, 2022 12:50 AM
> To: PAUL R BARFORD 
> Cc: Lukas Tribus ; Esteban Carisimo
> ; nanog@nanog.org
> ; Fabian E. Bustamante
> 
> Subject: Re: Long hops on international paths
> 
> 1) all (meaning all hitting the zayo.telia) your traceroutes originate
> from University in Chicago
> 2) the zayo.telia device is physically close to the university
> 3) we should expect physically close-by backbone device to be present
> in disproportionate amount of traceroutes
> 4) almost certainly zayo.telia is imposing the MPLS label of TTL 255,
> _NOT_ copying IP TTL, therefore until MPLS label is popped, TTL is not
> expiring. I.e. you are seeing ingressPE and egress PE ot Telia, you
> are not seeing any P routers.
> 
> This is not esoteric knowledge, but a fairly basic Internet concept. I
> am worried you are missing too much context to produce actionable
> output from your work. It might be interesting to see your curriculum,
> why this confusion arose, why it seems logical that the reason must be
> that almost all waves are terminated there, because it would not seem
> logical for people practising in the field who have even cursory
> understanding, this implies problems in the curriculum.
> 
> On Tue, 18 Jan 2022 at 07:21, PAUL R BARFORD  wrote:
> >
> > Please find the examples for the case of Telia below.
> >
> > FROM jfk-us (jfk-us.team-probing.c008820.20201002.warts.gz)
> >
> >
> >
> > traceroute from 216.66.30.102 (Ark probe hosted in New York City, NY, US.
> No AS info found) to 223.114.235.32 (MAXMIXD: Turpan, CN)
> >
> > 1  216.66.30.101  0.365 ms
> >
> > 2  62.115.49.173  3.182 ms
> >
> > 3  *
> >
> > 4  62.115.137.59  17.453 ms [x] (chi-b23-link.ip.twelve99.net., CAIDA-
> GEOLOC -> Chicago, IL, US)
> >
> > 5  62.115.117.48  59.921 ms [x] (sea-b2-link.ip.twelve99.net., RIPE-IPMAP ->
> Seattle, WA, US)
> >
> > 6  62.115.171.221  69.993 ms
> >
> > 7  223.120.6.53  69.378 ms
> >
> > 8  223.120.12.34  226.225 ms
> >
> > 9  221.183.55.110  237.475 ms
> >
> > 10  221.183.25.201  238.697 ms
> >
> > 11  221.176.16.213  242.296 ms
> >
> > 12  221.183.36.62  352.695 ms
> >
> > 13  221.183.39.2  300.166 ms
> >
> > 14  117.191.8.118  316.270 ms
> >
> > 15  *
> >
> > 16  *
> >
> > 17  *
> >
> > 18  *
> >
> > 19  *
> >
> >
> >
> >
> >
> > FROM ord-us (ord-us.team-probing.c008820.20201002.warts.gz)
> >
> >
> >
> > traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at
> Depaul University-AS20120) to 109.25.215.237 (237.215.25.109.rev.sfr.net.,
> MAXMIXD: La Crau, FR)
> >
> > 1  140.192.218.129  0.795 ms
> >
> > 2  140.192.9.124  0.603 ms
> >
> > 3  64.124.44.158  1.099 ms
> >
> > 4  64.125.31.172  3.047 ms
> >
> > 5  *
> >
> > 6  64.125.15.65  1.895 ms  [x] (zayo.telia.ter1.ord7.us.zip.zayo.com.,
> CAIDA-GEOLOC -> Chicago, IL, US)
> >
> > 7  62.115.118.59  99.242 ms[x] (prs-b3-link.ip.twelve99.net., CAIDA-
> GEOLOC -> Paris, FR)
> >
> > 8  62.115.154.23  105.214 ms
> >
> > 9  77.136.10.6  119.021 ms
> >
> > 10  77.136.10.6  118.830 ms
> >
> > 11  80.118.89.202  118.690 ms
> >
> > 12  80.118.89.234  118.986 ms
> >
> > 13  109.24.108.66  119.159 ms
> >
> > 14  109.25.215.237  126.085 ms
> >
> >
> >
> >
> >
> > traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at
> Depaul University-AS20120) to 84.249.89.93 (dsl-tkubng12-54f959-
> 93.dhcp.inet.fi., MAXMIXD: Turku, FI)
> >
> > 1  140.192.218.129  0.243 ms
> >
> > 2  140.192.9.124  0.326 ms
> >
> > 3  64.124.44.158  0.600 ms
> >
> > 4  *
> >
> > 5  *
> >
> > 6  

RE: BGP Route Monitoring

[Michael Hare via NANOG]

Re: Adam's advice about IOS/XR SNMP access to VRF, while this experience may be 
a bit dated [IOS XR 5.x], in production we have used "snmp-server community-map 
$x context $y".  I will say we weren't pleased, we noticed that context 
switches didn't work well.  For example if our poller tried to simultaneously 
poll the global community and the vrf community at the same time, the results 
were non deterministic.  Maybe this has been improved in later versions, or if 
you always have a single poller that will always be sequential, you may never 
see this.

Although more work BMP is probably the better/safer approach.

-Michael

From: NANOG  On Behalf Of Adam 
Thompson
Sent: Thursday, January 6, 2022 12:41 PM
To: Sandoiu Mihai ; nanog@nanog.org
Subject: RE: BGP Route Monitoring

Most monitoring products allow you to monitor custom SNMP OIDs, and your entire 
BGP RIB is - usually - exposed via SNMP.
Most monitoring products also treat "missing" OIDs specially, and can alert on 
that fact.
At least, that's how I would start doing it.
We use Observium here, and it can do what you want, albeit with a little bit of 
futzing around in the Custom OID and Alerts sections.

Cisco does weird things with getting SNMP data from VRFs, though, so... YMMV.  
I know there used to be a Cisco-proprietary way to select which VRF you were 
polling common OIDs from, but don't remember the details.
-Adam

Adam Thompson
Consultant, Infrastructure Services
[MERLIN]
100 - 135 Innovation Drive
Winnipeg, MB, R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
athomp...@merlin.mb.ca
www.merlin.mb.ca

From: NANOG 
mailto:nanog-bounces+athompson=merlin.mb...@nanog.org>>
 On Behalf Of Sandoiu Mihai
Sent: Thursday, January 6, 2022 4:35 AM
To: nanog@nanog.org
Subject: BGP Route Monitoring

Hi

I am looking for a route monitoring product that does the following:
-checks if a specific bgp route from a specific neighbor is present the BGP 
table (in some vrf, not necessarily internet routed vrf) of an ASR9K running 
IOS XR
-sends a syslog message or an alarm if the route goes missing

The use case is the following: we are receiving same routes over 2 or more bgp 
peerings, due to best route we cannot really see at the moment if one of the 
routes ceased to be received over a certain peering.

Alternative approach: a product that measures the number of bgp received 
prefixes from a certain peer.

Do you know of such product that is readily available and does not require ssh 
sessions to the routers and parsing the outputs?
I am trying to find a solution that does not require much scripting or 
customization.

Many thanks.

Regards
Mihai

RE: Partial vs Full tables

[Michael Hare via NANOG]

Mark (and others),

I used to run loose uRPF on peering/transit links for AS3128 because I used to 
think that tightening the screws was always the "right thing to do".   

I instrumented at 60s granularity with vendor J uRPF drop counters on these 
links.  Drops during steady state [bgp converged] were few [Kbps].  Drops 
during planned maintenance were at much rates for a few minutes.

What was happening: I advertise a handful of routes to transit/peers from 
multiple ASBR.  Typically my ASBR sees 800K FIB and a few million RIB routes  
We all know this takes a good amount of time to churn..   

For planned maintenance of ASBR A [cold boot upgrades], if recovery didn't 
include converging my inbound routes before doing eBGP advertising, I'd be 
tossing packets due to loose uRPF.  

Remember during this time 'ASBR B'  in my AS is happy egressing traffic as soon 
as 'ASBR A' advertises my dozen or so prefixes via eBGP, I start to see return 
traffic much sooner than before 'ASBR A' has converged.  No more specific 
return route yet other than maybe default for a few minutes if unlucky..  The 
result is bit bucket networkwide despite ASBR B functioning just fine.

Maybe everyone already convergences inbound before advertising eBGP and I made 
a rookie mistake, but what about unplanned events?

For me the summary is that I was causing more collateral damage than good 
[verified by time series data], so I turned off loose URPF.  YMMV.

-Michael

> -Original Message-
> From: NANOG  On Behalf Of Mark Tinka
> Sent: Thursday, June 11, 2020 12:14 PM
> To: nanog@nanog.org
> Subject: Re: Partial vs Full tables
> 
> 
> 
> On 10/Jun/20 19:31, William Herrin wrote:
> 
> >
> > Sorry, it'd be pre-coffee if I drank coffee and I was overly harsh
> > here. Let me back up:
> >
> > The most basic spoofing protection is: don't accept remote packets
> > pretending to be from my IP address.
> >
> > Strict mode URPF extends this to networks: don't accept packets on
> > interfaces where I know for sure the source host isn't in that
> > direction. It works fine in network segments whose structure requires
> > routes to be perfectly symmetrical: on every interface, the packet for
> > every source can only have been from one particular next hop, the same
> > one that advertises acceptance of packets with that destination. The
> > use of BGP breaks the symmetry requirement so close to always that you
> > may as well think of it as always. Even with a single transit or a
> > partial table. Don't use strict mode URPF on BGP speakers.
> >
> > Loose mode URPF is... broken. It was a valiant attempt to extend
> > reverse path filtering into networks with asymmetry but I've yet to
> > discover a use where there wasn't some faulty corner case. If you
> > think you want to use loose mode RPF, trust me: you've already passed
> > the point where any RPF was going to be helpful to you. Time to set it
> > aside and solve the problem a different way.
> 
> We don't run Loose Mode on peering routers because they don't carry a
> full table. If anyone sent the wrong packets that way, they wouldn't be
> able to leave the box anyway.
> 
> We do run Loose Mode on transit routers, no issues thus far.
> 
> We do run Strict Mode on customer-facing links that are stub-homed to us
> (DIA). We also run Loose Mode on customer-facing links that buy transit
> (BGP).
> 
> But mostly, BCP-38 deployed at the edge (peering, transit and customer
> routers) also goes a long way in protecting the network.
> 
> Mark.

RE: Partial vs Full tables

[Michael Hare via NANOG]

Saku-

> In internal network, instead of having a default route in iBGP or IGP,
> you should have the same loopback address in every full DFZ router and
> advertise that loopback in IGP. Then non fullDFZ routers should static
> route default to that loopback, always reaching IGP closest full DFZ
> router.

Just because DFZ role device can advertise loopback unconditionally in IGP 
doesn't mean the DFZ actually has a valid eBGP or iBGP session to another DFZ.  
It may be contrived but could this not be a possible way to blackhole nearby 
PEs..?   

We currently take a full RIB and I am currently doing full FIB.  I'm currently 
choosing to create a default aggregate for downstream default-only connectors 
based on something like

 from {
protocol bgp;
as-path-group transit-providers;
route-filter 0.0.0.0/0 prefix-length-range /8-/10;
route-type external;
}

Of course there is something functionally equivalent for v6.  I have time 
series data on the count of routes contributing to the aggregate which helps a 
bit with ease of mind of default being pulled when it shouldn't be.  Like all 
tricks of this type I recognize this is susceptible to default being 
synthesized when it shouldn't be.

I'm considering an approach similar to Tore's blog where at some point I keep 
the full RIB but selectively populate the FIB.  Tore, care to comment on why 
you decided to filter the RIB as well?

-Michael

RE: Long AS Path

[Michael Hare]

Couldn't one make the same argument with respect to filtering private ASNs from 
the global table?  Unlike filtering of RFC1918 and the like a private ASN in 
the path isn't likely to leak RFC1918 like traffic, yet I believe several major 
ISPs have done just that.  This topic was discussed ~1 year ago on NANOG.

I do filter private ASNs but have not yet filtered long AS paths.  Before I did 
it I had to contact a major CDN because I would have dropped their route, in 
the end costing me money (choosing transit vs peering).

In the end, it is indeed risk vs reward, with risk being undefined behavior.  
It's plausible to speculate that not every path length bug has been squashed 
(or might not be re-introduced).

-Michael

> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Hunter
> Fuller
> Sent: Monday, June 26, 2017 9:40 AM
> To: James Bensley ; nanog@nanog.org
> Subject: Re: Long AS Path
> 
> This could just be ignorance, but based on this thread, I'm not sure what
> risk we would be managing, as DFZ router operators, by filtering those
> paths. They seem silly, but harmless (similar to, for instance, painting a
> nyan cat on a graph by announcing prefixes at certain times).
> 
> On Sun, Jun 25, 2017 at 6:32 AM James Bensley 
> wrote:
> 
> > On 24 June 2017 at 13:10, Mel Beckman  wrote:
> > > James,
> > >
> > > By "experienced by someone else" I mean someone who is not one of
> your
> > customers.
> > >
> > > The better strategy, I think, is to not filter long paths unless you
> > have a reason to see their creating a problem. Otherwise you're just
> > operating on superstition, no?
> > >
> > > -mel via cell
> >
> > Hi Mel,
> >
> > I mean this as a rhetorical question as we could talk until the end of
> > time about this; what is the difference between operating on
> > superstition and trying to be pro-active? Both for me fall under the
> > category of "risk management".
> >
> > Cheers,
> > James.
> >
> --
> 
> --
> Hunter Fuller
> Network Engineer
> VBH Annex B-5
> +1 256 824 5331
> 
> Office of Information Technology
> The University of Alabama in Huntsville
> Systems and Infrastructure

RE: Bogon ASN Filter Policy

[Michael Hare]

I'm not against the theory of what is being proposed, but I was surprised to 
see little discussion of this announcement on list.

Upon examination on my view of the DFZ from AS3128 I see over 400 upstream 
routes falling into this category, mostly in the 64512 - 65534 range.  Based on 
our flow bandwidth stats we chose to reach out to several origin ASN, two 
fairly well known, as a courtesy.

For the *TT's who are planning on implementing shortly, have you went through a 
similar diagnostic effort and what might you share or report on such endeavors?

-Michael

> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Arnold Nipper
> Sent: Wednesday, June 08, 2016 12:37 AM
> To: Jay Borkenhagen ; nanog@nanog.org
> Subject: Re: Bogon ASN Filter Policy
> 
> On 03.06.2016 15:08, Jay Borkenhagen wrote:
> 
> > AT/as7018 is also now in the process of updating its as-path bogon
> > filters to match those cited below.  We have long employed such
> > filters, and our changes at this time are primarily to extend them to
> > prohibit as23456 and the reserved blocks > as65535.
> >
> > So to Job and Adam and anyone else who deploys such filters: Thanks!
> > I would like to extend to you this laurel, and hearty handshake...
> >
> 
> Well done, NTT, GTT, AT You may want to notice that most of the IXP
> around the world which operate route servers since long do strict
> filtering. Both on ASN as well as on prefixes. So it's really nice to
> see, that the big ISP take care as well now.
> 
> As I have learnt yesterday at ENOG11 a way more challenging issue is to
> cope with route leaks.
> 
> 
> Cheers and cu in chi
> Arnold
> 
> >
> > On 02-June-2016, Adam Davenport writes:
> >  > I personally applaud this effort as initiatives like this that help
> >  > prevent the global propagation of Bogons and other "bad things" only
> >  > serves to help us all.  With that said, notice went out to potentially
> >  > affected GTT / AS3257 customers this week that by the end of June we too
> >  > will be filtering prefixes that contain any of the Bogon ASNs listed
> >  > below in the in the as-path.  I highly encourage other networks to
> >  > follow suit, as again it only helps us all.
> >  >
> >  > Thanks Job for kicking this one off, and I look forward to others to
> >  > doing the same!
> >  >
> >  > Adam Davenport / adam.davenp...@gtt.net
> >  >
> >  >
> >  >
> >  > On 6/2/16 3:41 PM, Job Snijders wrote:
> >  > > Dear fellow network operators,
> >  > >
> >  > > In July 2016, NTT Communications' Global IP Network AS2914 will deploy
> a
> >  > > new routing policy to block Bogon ASNs from its view of the 
> > default-free
> >  > > zone. This notification is provided as a courtesy to the network
> >  > > community at large.
> >  > >
> >  > > After the Bogon ASN filter policy has been deployed, AS 2914 will not
> >  > > accept route announcements from any eBGP neighbor which contains a
> Bogon
> >  > > ASN anywhere in the AS_PATH or its atomic aggregate attribute.
> >  > >
> >  > > The reasoning behind this policy is twofold:
> >  > >
> >  > >  - Private or Reserved ASNs have no place in the public DFZ. 
> > Barring
> >  > >these from the DFZ helps improve accountability and dampen
> >  > >accidental exposure of internal routing artifacts.
> >  > >
> >  > >  - All AS2914 devices support 4-byte ASNs. Any occurrence of 
> > "23456"
> >  > >in the DFZ is a either a misconfiguration or software issue.
> >  > >
> >  > > We are undertaking this effort to improve the quality of routing data 
> > as
> >  > > part of the global ecosystem. This should improve the security posture
> >  > > and provide additional certainty [1] to those undertaking network
> >  > > troubleshooting.
> >  > >
> >  > > Bogon ASNs are currently defined as following:
> >  > >
> >  > >  0   # Reserved RFC7607
> >  > >  23456   # AS_TRANS RFC6793
> >  > >  64496-64511 # Reserved for use in docs and code 
> > RFC5398
> >  > >  64512-65534 # Reserved for Private Use RFC6996
> >  > >  65535   # Reserved RFC7300
> >  > >  65536-65551 # Reserved for use in docs and code 
> > RFC5398
> >  > >  65552-131071# Reserved
> >  > >  42-4294967294   # Reserved for Private Use RFC6996
> >  > >  4294967295  # Reserved RFC7300
> >  > >
> >  > > A current overview of what are considered Bogon ASNs is maintained at
> >  > > NTT's Routing Policies page [2]. The IANA Autonomous System Number
> >  > > Registry [3] is closely tracked and the NTT Bogon ASN definitions are
> >  > > updated accordingly.
> >  > >
> >  > > We encourage network operators to consider deploying similar policies.
> >  > > Configuration examples for various platforms can be found here [4].
> >  > >
> >  > > NTT staff is monitoring current occurrences of Bogon ASNs in the 
> > routing
> >  

RE: LLDP via SNMP

[Michael Hare]

My experience with Juniper has been mixed, experience with 12.X and 13.X made 
me wonder if I had a poor understanding of how to do a proper decode or if 
Juniper's implementation itself had issues as I would often get incomplete 
results.

We've now grab over XML and have been pleased with that choice to the point I 
no longer see it as a "resort" but the most efficient way for us to move 
forward.  I used to open JTAC cases on each SNMP problem I'd come across [ CoS, 
mac-accounting ] and it generally wasn't a good use of our time.

-Michael

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Spencer Ryan
Sent: Thursday, May 26, 2016 9:51 AM
To: a...@kentik.com
Cc: North American Network Operators' Group 
Subject: Re: LLDP via SNMP

We use Observium for most of our SNMP monitoring, and it correctly pulls
LLDP and CDP data from all of our Cisco and Arista gear.


*Spencer Ryan* | Senior Systems Administrator | sr...@arbor.net
*Arbor Networks*
+1.734.794.5033 (d) | +1.734.846.2053 (m)
www.arbornetworks.com

On Thu, May 26, 2016 at 10:47 AM, Avi Freedman 
wrote:

>
> Have had the question come up a few times, so I wanted to poll the
> community to see...
>
> For those who are monitoring LLDP, how have you found the SNMP MIB support
> support for it on Juniper, Cisco, Brocade, Arista, and others?
>
> Wondering if you've needed to resort to CLI scraping or APIs to get the
> data?
>
> Thanks,
>
> Avi Freedman
> CEO, Kentik
>
>

RE: strategies to mitigate DNS amplification attacks in ISP network

[Michael Hare]

Martin-

I represent a statewide educational network running Juniper gear that is a 
quasi-enterprise.  I think efforts depend on size and type of network.  We are 
testing an approach that involves;

1) whitelisting known local resolvers, well behaved cloud DNS resolvers. 
2) on ingress, policing non-conforming traffic that matches UDP src port 53 dst 
port unreserved bytes > 1400
3) on ingress, queuing fragments to high packet loss priority [you better 
understand how fragments are used in your network before doing this]
4) If you have Juniper gear look into prefix-action
5) RTBH if required

This obviously doesn't work on the core of the internet.

Other good tips: 
* strengthen [anycast] your local DNS resolvers and consider a scheme that 
allows you to change their outgoing address on the fly.
* push [some] of your external authoritative DNS to the cloud.

-Michael

> > -Original Message-
> > From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Martin T
> > Sent: Tuesday, December 01, 2015 11:00 AM
> > To: nanog@nanog.org
> > Subject: strategies to mitigate DNS amplification attacks in ISP network
> >
> > Hi,
> >
> > as around 40% of ASNs allow at least partial IPv4 address spoofing in
> > their network(http://spoofer.csail.mit.edu/summary.php) and there are
> > around 30 million open-resolvers(http://openresolverproject.org/) in
> > the Internet, then DNS amplification traffic is daily occasion for
> > ISPs. This in probably mainly because RPF checks and DNS
> > RRL(https://kb.isc.org/article/AA-01000/0/A-Quick-Introduction-to-Response-
> > Rate-Limiting.html)
> > are not ubiquitously implemented, recursive requests without any ACLs
> > in DNS servers are often allowed, it requires little effort from
> > attackers point of view and is effective attack method. Unfortunately,
> > there seems to be very limited number of countermeasures for ISPs. Few
> > which I can think of:
> >
> > 1) higher capacity backbone links - I'm not sure if this can be
> > considered a mitigation method, but at least it can help to affect
> > smaller amount of customers if traffic volumes are not very high
> >
> >
> > 2) rate-limit incoming DNS traffic flows on peering and uplink ports -
> > here I mean something similar to iptables "recent"
> > module(http://www.netfilter.org/documentation/HOWTO/netfilter-
> extensions-
> > HOWTO-3.html#ss3.16)
> > which allows certain number of certain type of packets in a configured
> > time-slot per IP. However, such functionality is probably not common
> > on edge or backbone routers.
> >
> >
> > Tracking the packet state does definitely not work because state table
> > should be synchronized between all the routers in the network and
> > again, this requires Internet-routers to have stateful firewall
> > functionality. In addition, one also needs to allow new DNS
> > connections from Internet to its network.
> > If one simply polices incoming DNS traffic on uplink and peering
> > ports(for example if baseline DNS traffic is 5Mbps, then policer is
> > set to 50Mbps), then legitimate customers DNS traffic is also affected
> > in case of actual attack occurs and policer starts to drop DNS
> > traffic, i.e. policer has no way to distinguish between the legitimate
> > and non-legitimate incoming DNS traffic.
> >
> >
> > Am I wrong in some points? What are the common practices to mitigate
> > DNS amplification attacks in ISP network?
> >
> >
> > thanks,
> > Martin

Re: US DOJ victim letter

[Michael Hare]

AS2381 has also received them, we are no further along in this than you are.

On 1/19/2012 2:59 PM, Jay Hennigan wrote:

We have received three emails from the US Department of Justice Victim
Notification System to our ARIN POC address advising us that we may be
the victim of a crime.  Headers look legit.

We have been frustrated in trying to follow the rabbit hole to get any
useful information.  we've jumped through hoops to get passwords that
don't work and attempted to navigate a voice-mail system that resembles
the twisty maze of passages all different from an old text adventure
game.

This *seems* to be legit, and I would think that the end result is
likely to be a list of IP addresses associated with infected hosts.

Has anyone else received the email?  Is it legit?  If so has anyone
successfully navigated the maze, and if so how?  Is it worth it?

(And why don't they just send the list of infected IPs to the ARIN
contact in the first place?)

--
Jay Hennigan - CCIE #7880 - Network Engineering - j...@impulse.net
Impulse Internet Service  -  http://www.impulse.net/
Your local telephone and internet company - 805 884-6323 - WB6RDV

Re: ouch..

[Michael Hare]

You seem to have accidentally put an 'R' between your E and X ;)

On 9/14/2011 4:05 PM, Scott Weeks wrote:



--- brandon.galbra...@gmail.com wrote:
From: Brandon Galbraithbrandon.galbra...@gmail.com

Juniper: Who needs to waste time with pathetic marketing videos when you're
gear just works.
---


Unless it's the ERX series.  Blech, it puts a bad taste in my mouth just 
writing the acronym ERX.  Thank $deity that I don't have to work on those 
anymore...

scott

Re: IPv6 end user addressing

[Michael Hare]

On 8/10/2011 8:46 PM, William Herrin wrote:

On Wed, Aug 10, 2011 at 9:32 PM, Owen DeLongo...@delong.com  wrote:

Someday, I expect the pantry to have a barcode reader on it connected back
a computer setup for the kitchen someday.  Most of us already use barcode
readers when we shop so its not a big step to home use.


Nah... That's short-term thinking. The future holds advanced pantries with
RFID sensors that know what is in the pantry and when they were manufactured,
what their expiration date is, etc.


And since your can of creamed corn is globally addressable, the rest
of the world knows what's in your pantry too. ;)


I can't believe no one has made a joke yet about a kernel.

Sorry for the bad joke,
-Michael



Regards,
Bill Herrin

Re: ISP support for use of 4-byte ASNs in peering

[Michael Hare]

While attempting to focus on ISPs there is still [unbelievably] a vendor 
support issue.  You may consider this a procurement failure, but the 
fact remains that some products [Cisco me3400e] have yet to implement 
support.


-Michael

On 8/9/2011 9:24 AM, Nick Hilliard wrote:

On 09/08/2011 14:47, John Curran wrote:

   At ARIN, we are still having parties returning 4-byte ASN's (seeking 2-byte 
instead),
   indicating that the 4-byte ones are not sufficiently accepted in peering to 
be usable.
   This is obviously a less than desirable situation, and it appears that it is 
not trending
   towards resolution at this time.


At INEX, we see 60% of IXP connections which can handle ASN32 natively.
However, INEX is a small IXP and I haven't seen similar figures from other
IXPs which could validate this 60/40 split.

Having said that, in the IXP world most new service providers connect into
route servers, so there is often no perceived requirement for direct
ASN32-ASN16 interconnection - the intersection of new service providers
and ASN32 holders is quite large.  And if you really want a bilateral
peering relationship, there's no reason not to use AS23456.


Thoughts?


- interior BGP community management is great fun with an ASN32, oh yes.

- i don't have much sympathy for people who whine about not being able to
support ASN32 peerings.  There is no good reason for this these days.

- from personal experience, I understand why ASN32 is less popular.
However, it's certainly usable.

Nick

Re: ISP support for use of 4-byte ASNs in peering

[Michael Hare]

Granted I've never worked outside academia but the me3400 is otherwise a 
cost effective gig-e demarc for very simple bgp multihoming.


They have made a strategic decision to not implement a simple software 
update support RFC4893 [it has been 4 years] and to set an artificial 
price point on entry.  Fine, that is their choice.  There are other 
vendors offering 4 byte ASN in simliar products at or near this price 
point and we'll probably have to move to them.


-Michael

On 8/9/2011 10:38 AM, Nick Hilliard wrote:

On 09/08/2011 15:45, Michael Hare wrote:

While attempting to focus on ISPs there is still [unbelievably] a vendor
support issue.  You may consider this a procurement failure, but the fact
remains that some products [Cisco me3400e] have yet to implement support.


the me3400 is a metro core ethernet switch with L3 extensions.  It's not
intended as a border router.  If you use the wrong tool for the job...

Nick