Re: Open source Netflow analysis for monitoring AS-to-AS traffic

[Brian Knight via NANOG]

Thanks to all who took the time to comment and make suggestions. 

To summarize the private messages, one respondent suggested Argus as a
collector. Another mentioned that they are still using AS-Stats. 

I'm drawn to Akvorado. I like the self-contained nature of the
application. NF collector, database, and modern web GUI are all bundled
in one docker container. The full-featured demo [5] is fantastic. That
the app can enrich the Netflow data with BMP is an added bonus. 

The best part is, the GUI has the report viz I need, and it is actually
the default visualization in the demo. It also has the graph types that
I didn't know I needed, like the Sankey graph. 

FlowViewer looks interesting as well. I suspect getting the reports
right may take some time, given the amount of GUI filtering options. 

pmacct and Argus seem to be capable tools that have been around for a
long time, but I haven't seen a concise stack building guide to get
Netflow data into a good GUI using these. Looks like there are some
older Docker images available for both. I could write my own SQL or roll
my own stack, but I'd much rather spend my time on other things. 

I appreciate the conversation around sFlow. I actually wasn't aware that
XR supported it. AS path probably doesn't add a whole lot of value given
that I'm focused on flows across our IP transit circuits. I'm able to
determine my next AS hop simply by looking at the flow's associated
tuple of (flow exporter, interface). I can use other tools like
RouteViews or RIPE's RIS to determine the destination AS's upstreams if
needed. The rest of the path is probably not too helpful for determining
peering opportunities. 

I think I'm going to get Akvorado running in my environment. If that
doesn't pan out, I'll likely go back to AS-Stats. 

Can those running Akvorado comment on their system specs? The only spec
I've seen is a mention in this blog post [6]: "Akvorado is performant
enough to handle 100 000 flows per second with 64 GB of RAM and 24 vCPU.
With 2 TB of disk, you should expect to keep data for a few years." 

Thanks again all, 

-Brian 

On 2024-03-26 19:04, Brian Knight via NANOG wrote:

> What's presently the most commonly used open source toolset for monitoring 
> AS-to-AS traffic?
> 
> I want to see with which ASes I am exchanging the most traffic across my 
> transits and IX links. I want to look for opportunities to peer so I can 
> better sell expansion of peering to upper management. 
> 
> Our routers are mostly $VENDOR_C_XR so Netflow support is key.
> 
> In the past, I've used AS-Stats [1] for this purpose. However, it is 
> particularly CPU and disk IO intensive. Also, it has not been actively 
> maintained since 2017.
> 
> InfluxDB wants to sell me [2] on Telegraf + InfluxDB + Chronograf + 
> Kapacitor, but I can't find any clear guide on what hardware I would need for 
> that, never mind how to set up the software. It does appear to have an open 
> source option, however. 
> 
> pmacct seems to be good at gathering Netflow, but doesn't seem to analyze 
> data. I don't see any concise howto guides for setting this up for my 
> purpose, however. 
> 
> I'm aware Kentik does this very well, but I have no budget at the moment, my 
> testing window is longer than the 30 day trial, and we are not prepared to 
> share our Netflow data with a third party. 
> 
> Elastiflow [3] appears to have been open source [4] at one time in the past, 
> but no longer. Since it too appears to be hosted, I have the same objections 
> as I do with Kentik above. 
> 
> On-list and off-list replies are welcome. 
> 
> Thanks, 
> 
> -Brian

 

Links:
--
[1] https://github.com/manuelkasper/AS-Stats
[2] https://www.influxdata.com/what-are-netflow-and-sflow/
[3] https://www.elastiflow.com/
[4] https://github.com/robcowart/elastiflow?tab=readme-ov-file
[5] https://demo.akvorado.net/
[6] https://vincent.bernat.ch/en/blog/2022-akvorado-flow-collector

Open source Netflow analysis for monitoring AS-to-AS traffic

[Brian Knight via NANOG]

What's presently the most commonly used open source toolset for
monitoring AS-to-AS traffic?

I want to see with which ASes I am exchanging the most traffic across my
transits and IX links. I want to look for opportunities to peer so I can
better sell expansion of peering to upper management. 

Our routers are mostly $VENDOR_C_XR so Netflow support is key.

In the past, I've used AS-Stats [1] for this purpose. However, it is
particularly CPU and disk IO intensive. Also, it has not been actively
maintained since 2017.

InfluxDB wants to sell me [2] on Telegraf + InfluxDB + Chronograf +
Kapacitor, but I can't find any clear guide on what hardware I would
need for that, never mind how to set up the software. It does appear to
have an open source option, however. 

pmacct seems to be good at gathering Netflow, but doesn't seem to
analyze data. I don't see any concise howto guides for setting this up
for my purpose, however. 

I'm aware Kentik does this very well, but I have no budget at the
moment, my testing window is longer than the 30 day trial, and we are
not prepared to share our Netflow data with a third party. 

Elastiflow [3] appears to have been open source [4] at one time in the
past, but no longer. Since it too appears to be hosted, I have the same
objections as I do with Kentik above. 

On-list and off-list replies are welcome. 

Thanks, 

-Brian 
  

Links:
--
[1] https://github.com/manuelkasper/AS-Stats
[2] https://www.influxdata.com/what-are-netflow-and-sflow/
[3] https://www.elastiflow.com/
[4] https://github.com/robcowart/elastiflow?tab=readme-ov-file

Re: The Reg does 240/4

[Brian Knight via NANOG]

Depends what size block is being traded. Prices for /16 and larger have been flat since 2021.One thing is for sure: the cost for any size block has not dropped back to 2013 levels.Consider also that providers are starting to pass the charges onto their customers, like $DAYJOB-1 (an NSP) and now AWS this year. Those who may not be trading address blocks are starting to feel the bite.-BrianOn Feb 15, 2024, at 5:31 PM, Tom Beecher  wrote:$/IPv4 address peaked in 2021, and has been declining since. On Thu, Feb 15, 2024 at 16:05 Brian Knight via NANOG <nanog@nanog.org> wrote:On 2024-02-15 13:10, Lyndon Nerenberg (VE7TFX/VE6BBM) wrote:
> I've said it before, and I'll say it again:
> 
>   The only thing stopping global IPv6 deployment is
>   Netflix continuing to offer services over IPv4.
> 
> If Netflix dropped IPv4, you would see IPv6 available *everywhere*
> within a month.

As others have noted, and to paraphrase a long-ago quote from this 
mailing list, I'm sure all of Netflix's competitors hope Netflix does 
that.

I remain hopeful that the climbing price of unique, available IPv4 
addresses eventually forces migration to v6. From my armchair, only 
through economics will this situation will be resolved.

> --lyndon

-Brian

Re: The Reg does 240/4

[Brian Knight via NANOG]

On 2024-02-15 13:10, Lyndon Nerenberg (VE7TFX/VE6BBM) wrote:

I've said it before, and I'll say it again:

  The only thing stopping global IPv6 deployment is
  Netflix continuing to offer services over IPv4.

If Netflix dropped IPv4, you would see IPv6 available *everywhere*
within a month.


As others have noted, and to paraphrase a long-ago quote from this 
mailing list, I'm sure all of Netflix's competitors hope Netflix does 
that.


I remain hopeful that the climbing price of unique, available IPv4 
addresses eventually forces migration to v6. From my armchair, only 
through economics will this situation will be resolved.



--lyndon


-Brian

Re: One Can't Have It Both Ways Re: Streamline the CG-NAT Re: EzIP Re: IPv4 address block

[Brian Knight via NANOG]

On 2024-01-13 04:03, Brett O'Hara wrote:

They have no interest in trying new things or making new technology 
work without a solid financial reason and there is none for them 
implementing ipv6.


When I left $DAYJOB-1 almost 2 years ago, they had just finished 
increasing fees on IPv4 blocks (larger than /29) that had already been 
assigned to customers.


This wasn't on new assignments only. It was applied to all Internet 
customers with /28 and larger assignments that were already assigned and 
working at the time of the increase.


I know $DAYJOB-1 weren't alone in the NSP industry.

Also, one very large cloud provider I use for personal projects is 
charging additional fees for IPv4 starting this year. My cost for (3) 
IPv4 addresses went from zero to $10.80/mo/ip, jacking up my bill about 
20%. These were IPs assigned to my services 6-7 years ago.


There is a financial reason looming. I grant you that, at the moment, it 
may still be low enough to be considered "cost of doing business" for 
nearly all enterprises. But it's exerting force like a glacier does; 
slowly, irregularly, yet inexorably; so it can be difficult to see.


-Brian

Re: Your Input Needed: Can ROA Replace LOA? – Short Survey (7 mins)

[Brian Knight via NANOG]

On 2023-11-15 21:47, Christopher Hawker wrote:


Hello everyone,

Aftab Siddiqui is currently exploring the possibility of using Route 
Object Authorisations (ROAs) as a potential replacement to LOAs. 
Separate to this (and unknowing of Aftab's research), I had started a 
discussion on the RPKI Community guild on Discord 
(https://discord.gg/9jYcqpbdRE) discussing the usage of ROAs instead of 
LOAs.


An LOA, or "Letter of Authority" / "Letter of Authorization," is a 
formal document granting permission for third parties to take specific 
actions regarding network resources or services. In the service 
provider industry, its primary use is for advertising address resources 
(IPv4/v6 and ASN). When an organization intends to announce its IP 
prefixes through its own or a transit provider's ASN to the global 
internet, it typically needs to provide an LOA to their transit 
provider, confirming their custodianship or ownership of the resources.


I've found WHOIS is a good enough resource for this purpose. The SPs 
that are delegating prefixes are good about using SWIP to show 
assignment.


North American SPs are motivated to keep SWIP assignments up to date 
because of ARIN's requirement to demonstrate usage of IP resources for 
IP block transfers.


I think I've needed to request an LOA from a customer for this purpose 
just once in the past 10 years because the SWIP wasn't done. IIRC the 
assigning provider did a SWIP instead.



RPKI ROA, stands for "Resource Public Key Infrastructure Route Origin 
Authorization," is part of a security framework designed to validate 
the authenticity of internet routing information. It involves a 
digitally signed object that specifies which Autonomous Systems (ASes) 
are permitted to announce specific IP address prefixes.


Could you please take a moment to fill out our brief survey? Your 
feedback will play a crucial role in our understanding of this topic.


Survey Link: https://www.surveymonkey.com/r/JCHLWBB

Thanks,
Christopher Hawker


-Brian

Re: Zayo woes

[Brian Knight via NANOG]

On 2023-09-19 09:41, Matthew Petach wrote:


On Tue, Sep 19, 2023 at 7:19AM Mike Hammett  wrote:


[...]

I've never understood companies that acquire and don't completely 
integrate as quickly as they can.


Ah, spoken with the voice of someone who's never been in the position 
of:

a) acquiring a company not-much-smaller-than-you that
b) runs on completely different hardware and software and
c) your executives have promised there will be cost savings after the 
merger due to "synergies" between the two companies.

^_^;

Let's say you're an all J shop; your scripts, your tooling, everything 
expects to be talking to J devices.


Your executives buy a company that has almost the same size 
network--but it's all C devices running classic IOS.


You can go to your executives and tell them "hey, to integrate quickly 
with our network and tooling, we need to swap out all their C gear for 
J gear; it's gonna cost an extra $50M"
The executives respond by pointing at c) above, and denying the request 
for money to convert the acquired network to J.


You can go to your network and say "hey, we need to revamp our tooling 
and systems to understand how to speak to C and J devices equally, in 
spite of wildly different syntaxes for route-maps and the like-it's 
going to take 4 more developer headcount to rewrite all the systems."
The executives respond by pointing at c) above, and deny the request 
for developer headcount to rewrite your software systems.


Never mind C vs J, the difference in supported features alone is enough 
to cause heartburn. Example: the acquired company supports and offers 
E-LAN service; the acquiree doesn't. From a systems perspective, the 
acquiree has no way to track those without dev effort.


And I guarantee IT's #1 focus is not generating route maps or interface 
config. They're focused on processes and reports for the money people. 
If the engineering org has no developers, you're either running parallel 
or you're in for some long nights.



The general result of acquisitions of similar-sized companies is that 
the infrastructure runs in parallel, slowly getting converted over and 
unified as gear needs to be replaced, or sites are phased out--because 
any other course of action costs more money than the executives had 
promised the shareholders, the board, or the VCs, depending on what 
stage your company is at.


Swift integrations cost money, and most acquisitions promise cost 
savings instead of warning of increased costs due to integration.


That's why most companies don't integrate quickly.  :(

Matt


-Brian

Re: Scheduled outage -- Nationwide no driver license updates this weekend

[Brian Knight via NANOG]

It seems to say more about fluctuating funding and IT management.I seem to recall an issue with the FAA’s NOTAM / TFR database a few weeks back, one that grounded all flights one fine morning. Wasn’t network-related, but the articles I read about the application’s architecture and fault-tolerance make it seem… lightweight. :) And I would guess that outage was far more impactful than this one is likely to be.-BrianOn Mar 1, 2023, at 9:45 AM, Jason Leschnik  wrote:Says a lot about the architecture of the application and redundancy. I'd love to know what the failover looks like in a worst-case scenario  On Sun, 26 Feb 2023 at 10:51, Aaron de Bruyn via NANOG  wrote:If we have downtime, we lose revenue, customers, sleep, etc...If the government does it, what are you going to do?  Get your license somewhere else?-AOn Sat Feb 25, 2023, 11:39 PM GMT, Christopher Morrow wrote:On Sat, Feb 25, 2023 at 6:12 PM Sean Donelan  wrote:Verizon network maintenance will impact access to the “National DriverRegister,” a system that motor vehicle offices around the country need tocheck before handing out a license.Wait, what year is it?how is a network maintenance on what seems like a fairly critical system goingto cause a total outage of said system?I think we time traveled back to 1990 here...All 50 states and D.C. participate in the National Driver Register, adatabase maintained by the National Highway Traffic Safety Administration.The register contains information about drivers who have had their drivingprivileges revoked, suspended or denied due to serious traffic violations,such as driving under the influence of alcohol or drugs, reckless drivingor excessive speeding.The scheduled maintenance should be finished by Monday, in case you neededto update your driver's license or planned to do some reckless drivingthis weekend.

Re: DMARC ViolationAS21299 - 46.42.196.0/24 ASN prepending 255 times

[Brian Knight via NANOG]

Ask your upstream providers for a BGP community tag that lowers localpref below 
100 within their network. Set that community tag on any backup routes along 
with your (moderate) path prepending.

The backup upstream will then install that route only if there is no other way 
to get to your AS.

The path prepending remains so that other providers can more quickly install 
the primary route when it comes back.

Those two actions together have always made any backup route behave correctly 
for me.

-Brian

> On Mar 25, 2022, at 4:57 PM, Adam Thompson  wrote:
> 
> 
> Tom, how exactly does someone “ride the 0/0” train in the DFZ?
>  
> I’m connected to both commercial internet and NREN, and unfortunately-long 
> paths are not uncommon in this scenario, in order to do traffic steering.  If 
> there’s another solution that affects global inbound traffic distributions, 
> I’d love to hear about it (and so would a lot of my peers in edu).
>  
> If there were a usable way to “dump” the excessively-long path only as long 
> as a better path was already known by at least one edge router, that might be 
> workable, but you’d have to keep track of it somewhere to reinstall it if the 
> primary route went away… at which point you may as well have not dropped it 
> in the first place.
>  
> -Adam
>  
> Adam Thompson
> Consultant, Infrastructure Services
> 
> 100 - 135 Innovation Drive
> Winnipeg, MB, R3T 6A8
> (204) 977-6824 or 1-800-430-6404 (MB only)
> athomp...@merlin.mb.ca
> www.merlin.mb.ca
>  
> From: NANOG  On Behalf Of Tom 
> Beecher
> Sent: Friday, March 25, 2022 4:13 PM
> To: Paschal Masha 
> Cc: nanog 
> Subject: Re: DMARC ViolationAS21299 - 46.42.196.0/24 ASN prepending 255 times
>  
> The best practice with regards to as_path length is to have an edge filter 
> that dumps any prefix with a length longer than say 10. Depending on the 
> situation, might even be able to go smaller. 
>  
> At a certain point, keeping that route around does nothing for you, just 
> shoot it and ride the 0/0 train. 
>  
> On Fri, Mar 25, 2022 at 4:09 AM Paschal Masha  
> wrote:
> :) probably the longest prepend in the world.
> 
> A thought though, is it breaking any standard or best practice procedures?
> 
> Regards 
> Paschal Masha | Engineering 
> Skype ID: paschal.masha
> 
> - Original Message -
> From: "Erik Sundberg" 
> To: "nanog" 
> Sent: Friday, March 25, 2022 6:43:38 AM
> Subject: DMARC ViolationAS21299 - 46.42.196.0/24 ASN prepending 255 times
> 
> If anyone from AS21299 is lurking on Nanog. Please reduce your AS prepends 
> for 46.42.196.0/24 from 255 prepends to a more reasonable number of prepends 
> let's say 20. Thanks! 
> 
> This is a Kazakhstan register IP Block and ASN 
> 
> 
> 
> Network Next Hop Metric LocPrf Weight Path 
> 
> *> 46.42.196.0/24 x.x.x.x 0 100 0 2914 174 3216 3216 35168 21299 21299 21299 
> 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 
> 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 
> 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 
> 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 
> 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 
> 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 
> 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 
> 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 
> 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 
> 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 
> 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 
> 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 
> 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 
> 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 
> 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 
> 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 
> 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 
> 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 21299 
> 21299 21299 21299 21299 21 299 21299 21299 21299 21299 21299 21299 21299 
> 21299 21299 21299 21299 21299 21299 i 
> 
> 
> 
> 
> 
> 
> 
> 
> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or 
> previous e-mail messages attached to it may contain confidential information 
> that is legally privileged. If you are not the intended recipient, or a 
> person responsible for delivering it to the intended recipient, you are 
> hereby notified that any disclosure, copying, distribution or use of any of 
> the information contained in or attached to this transmission is STRICTLY 
> PROHIBITED. If you have received this transmission in error please 

Re: Authoritative Resources for Public DNS Pinging

[Brian Knight via NANOG]

On 2022-02-10 11:42, John Todd wrote:

"The Prudent Mariner never relies solely on any single aid to 
navigation"


It's best to ping multiple targets, and take action only if all targets 
do not return replies.


For route tracking a la $VENDOR_C's IP SLA, if possible, we'll ping 
next-hop IP, one target on our network, and one off our network.  
Withdraw the default route only if all three targets fail to return 
replies.



John Todd - jt...@quad9.net
General Manager - Quad9 Recursive Resolver


./btk

Re: Can it really be this quiet?

[Brian Knight via NANOG]

Also, lots of people out sick with the ‘rona. Fortunately, Omicron seems much 
less harmful than other variants.

Hope all are staying safe and well.

-Brian

> On Jan 3, 2022, at 2:06 PM, Josh Luthman  wrote:
> 
> 
> Likely a parallel between vacation, ie people not touching things, and things 
> staying working.
> 
> Josh Luthman
> 24/7 Help Desk: 937-552-2340
> Direct: 937-552-2343
> 1100 Wayne St
> Suite 1337
> Troy, OH 45373
> 
> 
>> On Mon, Jan 3, 2022 at 2:56 PM Job Snijders via NANOG  
>> wrote:
>> Hi Allen,
>> 
>> Yes, it can be this quiet. It’s good news, it means the thing is mostly 
>> working :-)
>> 
>> I wish everyone a happy and calm 2022!
>> 
>> Kind regards,
>> 
>> Job
>> 
>>> On Mon, 3 Jan 2022 at 20:47, Allen McKinley Kitchen (gmail) 
>>>  wrote:
>>> Or has NANOG also succumbed to a signed-integer date problem?
>>> 
>>> Waiting to see what I get back..
>>> 
>>> ..Allen

Re: IPv6 woes - RFC

[Brian Knight via NANOG]

On 2021-09-04 23:33, Mark Tinka wrote:

On 9/5/21 04:49, John Levine wrote:

I have asked my ISP about IPv6 and their answer is that that they're 
not opposed to
it but since I am the only person who has asked for it, it's quite low 
on the list

of things to do.


Supporting the routing and forwarding of IP addresses is just about
the most basic thing any ISP should do.

If that is low on their to-do list, what else could they possibly be 
doing?



$DAYJOB (at a business SP) is much busier installing more VPNs in the 
form of SDWAN than anything IPv6 related.  There is a hell of a lot more 
customer demand for tools that route packets with finer control than 
just dest-based routing, not to mention the security add-ons.


The fact that folks can achieve multi-homed Internet connectivity 
without the financial burden of a /24 helps SDWAN's case.  Turns out Mom 
and Pop, and even John Q. down the street, wants fast failover in case 
of circuit trouble just like the big boys.


That said, I do hear prospects and customers asking for IPv6 more often 
now than a year ago.  But it's nowhere near the popularity of SDWAN: 
It's gone from maybe 2-4 requests per year in 2019 to 2-4 requests per 
quarter today.


Sorry, the market *still* doesn't care much about IPv6.  Believe me, I 
hope people will continue getting interested, but I'm done holding my 
breath.


Maybe we should collectively focus instead on raising the default MTU 
for the Internet to 2000 bytes to accommodate all these tunnels. ;)




Mark.



-Brian

Re: DPDK and energy efficiency

[Brian Knight via NANOG]

On 2021-03-05 15:40, Eric Kuhnke wrote:

For comparison purposes, I'm curious about the difference in wattage 
results between:


a) Your R640 at 420W running DPDK

b) The same R640 hardware temporarily booted from a Ubuntu server live 
USB, in which some common CPU stress and memory disk/IO benchmarks are 
being run to intentionally load the system to 100% to characterize its 
absolute maximum AC load wattage.


We've got a few more hosts waiting to be deployed that are configured 
almost identically.  I'll see what we can do.


I'm guessing those tests would pull slightly more power than the vEdge 
hosts, just because there's not much disk IO that happens on a 
networking VM.  These hosts have four SSDs for local storage.


What's the delta between the 420W and absolute maximum load the server 
is capable of pulling on the 208VAC side?


https://manpages.ubuntu.com/manpages/artful/man1/stress-ng.1.html


Server PS maximum input wattage is 900W.  Present draw of 2.0A @ 208V is 
~420W, so 420/900 = 46.67%


One possible factor is whether ESXI is configured to pass the pci-e 
devices directly through to the guest VM, or if there is any 
abstraction in between. For non-ESXI stuff, in the world of Xen or KVM 
there's many different ways that a guest domU can access a dom0's 
network devices, some of which can have impact on overall steady-state 
wattage consumed by the system.


The 420W server has its interfaces routed through the ESXI kernel.  
We're moving quickly to SR-IOV on new servers.


If the greatest possible efficiency is desired for a number of 1U 
things, one thing to look at would be something similar to the open 
compute platform single centralized AC to DC power units, and servers 
that don't each have their own discrete 110-240VAC single or dual power 
supplies. In terms of cubic meters of air moved per hour vs wattage, 
the fans found in 1U servers are really quite inefficient. As a 
randomly chosen example of 12VDC 40mm (1U server height) fan:


https://www.shoppui.com/documents/9HV0412P3K001.pdf

If you have a single 12.0VDC fan that's a maximum load of 1.52A, that's 
a possible load of up to 18.24W for just *one* 40mm height fan. And 
your typical high speed dual socket 1U server may have up to eight or 
ten of those, in the typical front to back wind tunnel configuration. 
Normally fans won't be running at full speed, so each one won't be a 
18W load, but more like 10-12W per fan is totally normal. Plus two at 
least two more fans in both hot swap power supplies. Under heavy load I 
would not be surprised at all to say that 80W to 90W of your R640's 
total 420W load is ventilation.


Which is of course dependent on the environmentals.  Fan speeds on our 
two servers are 25% for the 260W vs. 29% for 420W, so not much 
difference.  Inlet temp on both is ~17C.


I checked out another R640 heavily loaded with vEdge VMs, and it's 
pulling similar power, 415W, but the fan speed is at 45%, because inlet 
temp is 22C.


The TDP for the Xeon 6152 is 140W, which seems middle-of-the-road.  From 
the quick survey I did of Dell's configurator, the R640 can take CPUs up 
to 205W.  So we have headroom in terms of cooling.


In a situation where you're running out of power before you run out of 
rack space, look at some 1.5U and 2U high chassist that use 60mm height 
fans, which are much more efficient in ratio of air moved per time 
period vs watts.


Or ask the colo to turn the A/C lower ;)  (that moves the power problem 
elsewhere, I know)


Thanks,

-Brian

Re: DPDK and energy efficiency

[Brian Knight via NANOG]

On 2021-03-05 12:22, Etienne-Victor Depasquale wrote:


Sure, here goes:

https://www.surveymonkey.com/results/SM-BJ9FCT6K9/


Thanks for sharing these results.  We run DPDK workloads (Cisco nee 
Viptela vEdge Cloud) on ESXI.  Fwiw, a quick survey of a few of our Dell 
R640s running mostly vEdge workloads shows the PS output wattage is 
about 60% higher than a non-vEdge workload: 420W vs 260W.  PS input 
amperage is 2.0A@208V vs 1.4A, a 42% difference.  Processor type is Xeon 
6152.  Stats obtained from the iDRAC lights-out management module.


vEdge does not do any limiting of polling by default, and afaik the 
software has no support for any kind of limiting.  It will poll the 
network driver on every core assigned to the VM for max performance, 
except for one core which is assigned to the control plane.


I'm usually more concerned about the lack of available CPU cores.  The 
CPU usage forces us not to oversubscribe the VM hosts, which means we 
must provision vEdges less densely and buy more gear sooner.  Plus, the 
increased power demand means we can fit about 12 vEdge servers per 
cabinet instead of 17.  (Power service is 30A 208V, maximum of 80% 
usage.)


OTOH, I face far fewer questions about vEdge Cloud performance problems 
than I do on other virtual platforms.




Cheers,

Etienne



Thanks again,

-Brian

Re: Famous operational issues

[Brian Knight via NANOG]

On 2021-02-17 13:28, John Kristoff wrote:

On Wed, 17 Feb 2021 14:07:54 -0500
John Curran  wrote:


I have no idea what outages were most memorable for others, but the
Stanford transfer switch explosion in October 1996 resulted in a much
of the Internet in the Bay Area simply not being reachable for
several days.


Thanks John.

This reminds me of two I've not seen anyone mention yet.  Both
coincidentally in the Chicago area that I learned before my entry
into netops full time.  One was a flood:

  

The other, at the dawn of an earlier era:

  



I wouldn't necessarily put those two in the top 3, but by some standard
for many they were certainly very significant and noteworthy.

John


Thanks for sharing these links John.  I was personally affected by the 
Hinsdale CO fire when I was a kid.  At the time, my family lived on the 
southern border of Hinsdale in the adjacent town of Burr Ridge.  It was 
weird like a power outage: you're reminded of the loss of service every 
time you perform the simple act of requesting service, picking up the 
phone or toggling a light switch.  But it lasted a lot longer than any 
loss of power: It was six or seven weeks that, to this day, felt a lot 
longer.


Anytime we needed to talk to someone long-distance, we had to drive to a 
cousin's house to make the call.  To talk to anyone local, you'd have to 
physically go and show up unannounced.  At 11 years old, I was the 
bicycle messenger between our house and my great-grandmother, who lived 
about two blocks away.  My mother and father kept the cars gassed up and 
extra fuel on hand in case there was an emergency.


Dad ran a home improvement business out of the house, so new business 
ground to a halt.  Mom worked for a publishing company, so their release 
dates were impacted.  The local grocery store's scanners wouldn't work, 
so they had to punch the orders into the register by hand, using the 
paper sticker prices on the items.


I clearly remember from the local papers that they had to special-order 
the replacement 5ESS at enormous cost.  I saw the big brick building 
after the fire with the burn marks around the front door.  In late May 
and early June, the Greyhound buses with the workers were parked around 
the block, power plants outside with huge cables snaking in right 
through the wide open front door.


When we heard that dial tone at last, everyone was happier than an 
iPhone with full bars. Lol


We're spoiled for choice in telecom networks these days.  Also, 
facilities management have learned plenty of lessons since then.  Like, 
install and maintain an FM-200 fire suppression system.  But 
nevertheless, sometimes when I step into a colo, I think of that outage 
and the impact it had.


-Brian

Re: Ingress filtering on transits, peers, and IX ports

[Brian Knight via NANOG]

As a final update to this thread, we started blocking spoofed and 
invalid traffic as of early Thursday morning Nov 19th.  So far, knock on 
wood, no reports of issues from our customer base.


In addition, I've been able to verify with the security research team's 
test tool that we are no longer responding to the spoofed DNS requests.


The ACL was implemented as follows:

Ingress

* Deny to and from bogon networks, where bogon is either source or dest
* Deny invalid TCP and UDP ports (currently only port 0) [log]
* Permit to and from transit / peer / IX connected subnets
* For IPv6, also permit link-local IPs (fe80::/10)
* Deny to and from multicast ranges 224.0.0.0/4 and ff00::/8
* Permit ICMP / traceroute over UDP to infrastructure
* Deny all other traffic to infrastructure [log]
* Permit from customer PI / PA space
* Deny from originated aggregate space [log]
* Permit all traffic to customer PI / PA space
* Permit all traffic to aggregate space
* Deny any any [log]

Egress

* Deny to and from bogon networks
* Deny invalid ports [log]
* Permit to and from transit / peer / IX connected subnets
* For IPv6, also permit link-local IPs
* Deny to and from multicast range
* Permit all traffic from any source to customer PI / PA space
* Permit all traffic from customer PI / PA space
* Permit all traffic from aggregate space
* Deny any any [log]

Below I've included the specific $VENDOR_C config I implemented for the 
filtering, sans specifics on our IP blocks.  I hope folks find this 
useful as a guide to their own efforts, and constructive criticism is 
always welcome.


Future work includes:

* Tightening the rules permitting access to/from the transit / peer / IX 
connected subnets, while keeping the ACL general enough for use on all 
Internet-facing interfaces
* Automation of updates to aggregate and customer IP blocks (looking at 
using the irrpt project for this)


Once more, to those who provided valuable input, thank you very much 
indeed!


-Brian


!-

! Static ACLs for Service Provider BCP 84 Compliance
! IOS XR config

! IPv4

object-group network ipv4 IPV4-BOGON
  description Invalid IPV4 networks
  0.0.0.0/8
  10.0.0.0/8
  100.64.0.0/10
  127.0.0.0/8
  169.254.0.0/16
  172.16.0.0/12
  192.0.0.0/24
  192.0.2.0/24
  192.168.0.0/16
  198.18.0.0/15
  198.51.100.0/24
  203.0.113.0/24
  240.0.0.0/4
exit

object-group network ipv4 IPV4-TRAN-WAN
  description Transit WAN PtP subnets
  [Point to point /30's go here]
exit

object-group network ipv4 IPV4-IX
  description IX subnets
  [IX /24 and /23 subnets here]
exit

object-group network ipv4 IPV4-PEER-WAN
  description Direct peer WAN PtP subnets
  [Direct peer WAN IPs go here]
exit

object-group network ipv4 IPV4-BGP-AGG
  description ARIN IPV4 Aggregate Blocks
  [Aggregated IP blocks go here]
exit

object-group network ipv4 IPV4-INFRA
  description Infrastructure subnets to be protected
  [List of loopback blocks and backbone / core PtP /30's here]
exit

object-group network ipv4 IPV4-BACKDOOR-HOSTS
  description Hosts observed to be sending valid traffic via Internet
  [One-off hosts, active TCP or UDP traffic was observed during data 
collection]

exit

object-group network ipv4 IPV4-CUST
  [full list of all customer IP blocks]
  [Includes customer PI blocks, disaggregated PA from other providers,]
  [and PA assigned from your aggregate space]
exit

object-group port TCPUDP-BLOCKED
  eq 0
  [additional ports to be generally blocked, list here]
exit

ipv4 access-list IPV4-INET-IN
  10 remark BCP 84 for transits, IX, and peering
  101 remark *** Block bogon networks as src or dest ***
  110 deny ipv4 net-group IPV4-BOGON any
  111 deny ipv4 any net-group IPV4-BOGON
  201 remark *** Blocked protocols ***
  210 deny udp any port-group TCPUDP-BLOCKED any log
  211 deny udp any any port-group TCPUDP-BLOCKED log
  212 deny tcp any port-group TCPUDP-BLOCKED any log
  213 deny tcp any any port-group TCPUDP-BLOCKED log
  301 remark *** Transit, IX, peer connected networks ***
  310 permit ipv4 net-group IPV4-PEER-WAN any
  311 permit ipv4 any net-group IPV4-PEER-WAN
  312 permit ipv4 net-group IPV4-TRAN-WAN any
  313 permit ipv4 any net-group IPV4-TRAN-WAN
  314 permit ipv4 net-group IPV4-IX any
  315 permit ipv4 any net-group IPV4-IX
  401 remark *** Block multicast ***
  410 deny ipv4 224.0.0.0/4 any
  411 deny ipv4 any 224.0.0.0/4
  501 remark *** Protect infrastructure subnets ***
  510 deny icmp any net-group IPV4-INFRA fragments log
  511 permit icmp any net-group IPV4-INFRA
  512 permit udp any range 1024 65535 net-group IPV4-INFRA range 33435 
33535
  513 permit udp any range 33435 33535 net-group IPV4-INFRA range 1024 
65535

  515 deny ipv4 any net-group IPV4-INFRA
  601 remark *** Customer Inet BGP Announced Prefixes ***
  620 permit ipv4 net-group IPV4-CUST any
  640 permit ipv4 net-group IPV4-BACKDOOR-HOSTS any
  701 remark *** Block originated networks ***
  710 deny ipv4 net-group IPV4-BGP-AGG any log
  801 remark *** Permit 

Re: Ingress filtering on transits, peers, and IX ports

[Brian Knight via NANOG]

Randy, thank you for the reminder to look also at what services (L4 
ports) should be generally blocked.


As I was implementing a similar rule for logging purposes, I discovered 
an oddity with $VENDOR_C_XR ACLs.  I created the following:


object-group port TCPUDP-BLOCKED
  eq 0
  eq sunrpc
  eq 445
  range 137 139
exit

ipv4 access-list IPV4-INET-IN
  10 remark BCP 84 for transits, IX, and peering
  101 remark *** Block bogon networks as src or dest ***
  110 deny ipv4 net-group IPV4-BOGON any
  111 deny ipv4 any net-group IPV4-BOGON
  201 remark *** Blocked protocols PERMIT FOR NOW ***
  210 permit udp any port-group TCPUDP-BLOCKED any log
  211 permit udp any any port-group TCPUDP-BLOCKED log
  212 permit tcp any port-group TCPUDP-BLOCKED any log
  213 permit tcp any any port-group TCPUDP-BLOCKED log
[snip]

ipv4 access-list IPV4-INET-OUT
  10 remark BCP 84 for transits, IX, and peering
  101 remark *** Block bogon networks as src or dest ***
  110 deny ipv4 net-group IPV4-BOGON any
  111 deny ipv4 any net-group IPV4-BOGON
  201 remark *** Blocked protocols PERMIT FOR NOW ***
  210 permit udp any port-group TCPUDP-BLOCKED any log
  211 permit udp any any port-group TCPUDP-BLOCKED log
  212 permit tcp any port-group TCPUDP-BLOCKED any log
  213 permit tcp any any port-group TCPUDP-BLOCKED log
[snip]

After I did this, logs on our syslog server started growing like crazy.  
It was full of entries like:


2020-10-21T01:47:17-05:00,info,RP/0/RSP1/CPU0:Oct 21 01:47:17.972 CDT: 
ipv4_acl_mgr[305]: %ACL-IPV4_ACL-6-IPACCESSLOGP : access-list 
IPV4-INET-OUT (210) permit udp on.net.ip.adr(0) -> off.net.ip.adr(0), 5 
packets
2020-10-21T02:01:08-05:00,info,RP/0/RSP0/CPU0:Oct 21 02:01:08.490 CDT: 
ipv4_acl_mgr[263]: %ACL-IPV4_ACL-6-IPACCESSLOGP : access-list 
IPV4-INET-IN (210) permit udp off.net.ip.adr(0) -> on.net.ip.adr(0), 58 
packets


After wondering why in the world my customers were sending so much data 
on port 0, I found a few different sources saying that port 0 is 
commonly used in place of valid information when dealing with fragments. 
 Turns out that $VENDOR_C_XR does this too.


It wasn't clear why fragments would be matching that rule until I found 
the right vendor doc.  The router will pass IP fragments with a "permit" 
ACL line as long as that fragment's layer 3 info matches the layer 3 
information in the ACL.  The router logs the packet similar the above: 
L4 protocol with source and dest port = 0.  From the doc:


-

For an access-list entry containing Layer 3 and Layer 4 information:
• The entry is applied to non-fragmented packets and initial fragments.
• If the entry matches and is a permit statement, the packet or
fragment is permitted.
• If the entry matches and is a deny statement, the packet or fragment
is denied.

The entry is also applied to non-initial fragments in the following
manner. Because non-initial fragments contain only Layer 3 information,
only the Layer 3 portion of an access-list entry can be applied. If the
Layer 3 portion of the access-list entry matches, and
• If the entry is a permit statement, the non-initial fragment is
permitted.
• If the entry is a deny statement, the next access-list entry is
processed.
The deny statements are handled differently for non-initial
fragments versus non-fragmented or initial fragments.

-

Since my rule's L3 info was permit any source to any destination, any IP 
fragment would match the rule, be passed, and be logged.  The solution 
was to add rules explicitly permitting fragments above the layer 4 
rules:


ipv4 access-list IPV4-INET-IN
  10 remark BCP 84 for transits, IX, and peering
  101 remark *** Block bogon networks as src or dest ***
  110 deny ipv4 net-group IPV4-BOGON any
  111 deny ipv4 any net-group IPV4-BOGON
  201 remark *** Blocked protocols PERMIT FOR NOW ***
  203 permit ipv4 net-group IPV4-CUST any fragments
  204 permit ipv4 net-group IPV4-BACKDOOR-HOSTS any fragments
  205 permit ipv4 any net-group IPV4-BGP-AGG fragments
  206 permit ipv4 any net-group IPV4-CUST fragments
  210 permit udp any port-group TCPUDP-BLOCKED any log
  211 permit udp any any port-group TCPUDP-BLOCKED log
  212 permit tcp any port-group TCPUDP-BLOCKED any log
  213 permit tcp any any port-group TCPUDP-BLOCKED log

Logs are a lot calmer now in terms of new lines per minute, and far more 
relevant.  When we switch those rules to deny statements, we can 
eliminate the rules specifically permitting fragments.


Looks like $VENDOR_J makes things so much simpler for this task.

Thanks,


-Brian


On 2020-10-20 00:18, Randy Bush wrote:

term blocked-ports {
from {
protocol [ tcp udp ];
first-fragment;
destination-port
	[ 0 sunrpc 135 netbios-ns netbios-dgm netbios-ssn 111 445 syslog 
11211];

}
then {
sample;
discard;
}
}

and i block all external access to weak devices such as switches, pdus,
ipmi, ...

randy

Re: Ingress filtering on transits, peers, and IX ports

[Brian Knight via NANOG]

Thanks to the folks who responded to my messages on and off-list.  A 
couple of folks have asked me to summarize the responses that I 
received.


* Static ACL is currently the best way to protect a multi-homed network. 
 Loose RPF may be used if bogon filtering is more important, but it does 
not provide anti-spoofing security.


* Protect your infrastructure subnets with the ingress ACL [BCP 84 sec 
3.2].  Loopbacks and point-to-point circuits can benefit from this.  In 
the draft ACL, for example, I permit ICMP and traceroute over UDP, and 
block all else.


* Do an egress ACL also, to prevent clutter from reaching the rest of 
the 'Net.  Permit only your aggregate and customer prefixes going 
outbound.


* As I worked through putting the ACLs together, I found that if one 
implements an egress ACL, then customer prefixes must be enumerated 
anyway.  Once those are in an object group, it's easy to add an entry to 
the ingress ACL permitting traffic destined to customer PI space and 
aggregate space.  Seems better than just permitting all traffic in.


Our ACLs, both v4 and v6, now look like the following:

Ingress

* Deny to and from bogon networks, where bogon is either source or dest
* Permit to and from WAN PtP subnets
* For IPv6, also permit link-local IPs (fe80::/10)
* Deny to and from multicast ranges 224.0.0.0/4 and ff00::/8
* Permit ICMP / traceroute over UDP to infrastructure
* Deny all other traffic to infrastructure
* Permit from customer PI / PA space
* Deny from originated aggregate space
* Permit all traffic to customer PI / PA space
* Permit all traffic to aggregate space
* Deny any any

Egress

* Deny to and from bogon networks
* Permit to and from WAN PtP subnets
* For IPv6, also permit link-local IPs
* Deny to and from multicast range
* Permit all traffic from customer PI / PA space
* Permit all traffic from aggregate space
* Deny any any

We have started implementing the ACLs by blocking the bogon traffic 
only.  The other deny rules are set up as permit rules for now with 
logging turned on.  I'll review matching traffic before I switch the 
rules to deny.


Future work also includes automating the updates to the object groups 
via IRR.


BTW, Team Cymru didn't have any guidance around IPv6 bogons, so I put 
together the below object group based on the IANA IPv6 allocation list: 
https://www.iana.org/assignments/ipv6-unicast-address-assignments/ipv6-unicast-address-assignments.xhtml. 
 Obviously this is only for space not yet allocated to RIRs.


object-group network ipv6 IPV6-BOGON
  description Invalid IPV6 networks
  ::/3
  4000::/3
  6000::/3
  8000::/3
  a000::/3
  c000::/3
  e000::/4
  f000::/5
  f800::/6
  fc00::/7
  fe00::/9
  fec0::/10
exit

Thanks,

-Brian



On 2020-10-14 17:43, Brian Knight wrote:

So I have put together what I think is a reasonable and complete ACL.
From my time in the enterprise world, I know that a good ingress ACL
filters out traffic sourcing from:

* Bogon blocks, like 0.0.0.0/8, 127.0.0.0/8, RFC1918 space, etc
(well-documented in
https://team-cymru.com/community-services/bogon-reference/bogon-reference-http/)
* RIR-assigned blocks I am announcing to the rest of the world

However, I recognized a SP-specific case where we could receive
legitimate traffic sourcing from our own IP blocks: customers running
multi-homed BGP where we have assigned PA space to them.  So I added
"permit" statements for traffic sourcing from these blocks.

Also, we have direct peering links that are numbered within our
assigned prefixes.  So we can use the same ACL with these peer
interfaces and continue to have BGP work, I added "permit" statements
for these point-to-point subnets.

So the order of the statements is:

* Permit where source is direct peer PtP networks
* Permit where source is BGP customer PA prefix
* Deny where source is bogon
* Deny where source is our advertised prefixes
* Permit all other traffic

I considered BGP customer PI prefixes to be out of scope for ingress
filtering, since the customer is likely to be multi-homing.  Should we
consider filtering them?

The Team Cymru Secure IOS Template
[https://www.cymru.com/Documents/secure-ios-template.html] also
references an ICMP fragment drop entry on the ingress ACL.  I think
that's good for an enterprise network, but as an SP, I'm very hesitant
to include this.  Is this included in anyone else's transit / peer /
IX ACL?

Is there anything else that I'm not thinking of?

Thanks,

-Brian


On 2020-10-14 09:25, Brian Knight via NANOG wrote:

Hi Marcos,

Thanks for your reply.  But I am looking for guidance on traffic
filtering, not BGP prefix filtering.

I have looked at BCP 84, and it's a good overview of the methods
available to an ISP.  My questions are more operational in nature and
are not covered by the document.  Of the choices presented in BCP 84,
what do folks really use?  If it's an ACL, what challenges have there
been with updates?  etc.

-Brian


On 2020-10-13 18:52, Marc

Re: Ingress filtering on transits, peers, and IX ports

[Brian Knight via NANOG]

So I have put together what I think is a reasonable and complete ACL.  
From my time in the enterprise world, I know that a good ingress ACL 
filters out traffic sourcing from:


* Bogon blocks, like 0.0.0.0/8, 127.0.0.0/8, RFC1918 space, etc 
(well-documented in 
https://team-cymru.com/community-services/bogon-reference/bogon-reference-http/)

* RIR-assigned blocks I am announcing to the rest of the world

However, I recognized a SP-specific case where we could receive 
legitimate traffic sourcing from our own IP blocks: customers running 
multi-homed BGP where we have assigned PA space to them.  So I added 
"permit" statements for traffic sourcing from these blocks.


Also, we have direct peering links that are numbered within our assigned 
prefixes.  So we can use the same ACL with these peer interfaces and 
continue to have BGP work, I added "permit" statements for these 
point-to-point subnets.


So the order of the statements is:

* Permit where source is direct peer PtP networks
* Permit where source is BGP customer PA prefix
* Deny where source is bogon
* Deny where source is our advertised prefixes
* Permit all other traffic

I considered BGP customer PI prefixes to be out of scope for ingress 
filtering, since the customer is likely to be multi-homing.  Should we 
consider filtering them?


The Team Cymru Secure IOS Template 
[https://www.cymru.com/Documents/secure-ios-template.html] also 
references an ICMP fragment drop entry on the ingress ACL.  I think 
that's good for an enterprise network, but as an SP, I'm very hesitant 
to include this.  Is this included in anyone else's transit / peer / IX 
ACL?


Is there anything else that I'm not thinking of?

Thanks,

-Brian


On 2020-10-14 09:25, Brian Knight via NANOG wrote:

Hi Marcos,

Thanks for your reply.  But I am looking for guidance on traffic
filtering, not BGP prefix filtering.

I have looked at BCP 84, and it's a good overview of the methods
available to an ISP.  My questions are more operational in nature and
are not covered by the document.  Of the choices presented in BCP 84,
what do folks really use?  If it's an ACL, what challenges have there
been with updates?  etc.

-Brian


On 2020-10-13 18:52, Marcos Manoni wrote:

Hi, Brian

Check RFC3704/BCP84 Ingress Filtering for Multihomed Networks (Updated
by: RFC8704 Enhanced Feasible-Path uRPF).

Ingress Access Lists require typically manual maintenance, but are
the most bulletproof when done properly; typically, ingress access
lists are best fit between the edge and the ISP when the
configuration is not too dynamic if strict RPF is not an option,
between ISPs if the number of used prefixes is low, or as an
additional layer of protection


Ingress filters Best Practices
https://www.ripe.net/support/training/material/bgp-operations-and-security-training-course/BGP-Slides-Single.pdf
*Don’t accept BOGON ASNs
*Don’t accept BOGON prefixes
*Don’t accept your own prefix
*Don’t accept default (unless you requested it)
*Don’t accept prefixes that are too specific
*Don’t accept if AS Path is too long
*Create filters based on Internet Routing Registries

And also BGP Best Current Practices by Philip Smith
http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf

Regards.

El mar., 13 oct. 2020 a las 19:52, Brian Knight via NANOG
() escribió:


Hi Mel,

My understanding of uRPF is:

* Strict mode will permit a packet only if there is a route for the 
source IP in the RIB, and that route points to the interface where 
the packet was received


* Loose mode will permit a packet if there is a route for the source 
IP in the RIB.  It does not matter where the route is pointed.


Strict mode won't work for us, because with our multi-homed transits 
and IX peers, we will almost certainly drop a legitimate packet 
because the best route is through another transit.


Loose mode won't work for us, because all of our own prefixes are in 
our RIB, and thus the uRPF check on a transit would never block 
anything.


Or am I missing something?

Thanks,

-Brian

On 2020-10-13 17:22, Mel Beckman wrote:

You can also use Unicast Reverse Path Forwarding. RPF is more 
efficient than ACLs, and has the added advantage of not requiring 
maintenance. In a nutshell, if your router has a route to a prefix in 
its local RIB, then incoming packets from a border interface having a 
matching source IP will be dropped.


RPF has knobs and dials to make it work for various ISP environments. 
Implement it carefully (as is be standing next to the router involved 
:)


Here's a Cisco brief on the topic:


https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwarding





I think all router vendors support this feature. Here's a similar 
article by Juniper:


https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/interfaces-configuring-unicast-rpf.html


-mel beckman

On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG  
wrote:


We recent

Re: Ingress filtering on transits, peers, and IX ports

[Brian Knight via NANOG]

Hi Eric, 

I shot a message over the folk who did the testing for more info about
their test.  If I'm able to find anything useful in our logs from their
detail, I'll post it to the list. 

The message referenced DNS cache poisoning and DDOS amplification, so it
seemed fairly general and more focused on whether ASes accepted spoofed
traffic.  They also referenced the new NXNSAttack, which I did not know
about previously. 

Thanks, 

-Brian 

On 2020-10-13 20:49, Eric Kuhnke wrote:

> Aside from the BCPs currently being discussed for ingress filtering, I would 
> be very interested in seeing what this traffic looked like from the 
> perspective of your DNS servers' logs. 
> 
> I assume you're talking about customer facing recursive/caching resolvers, 
> and not authoritative-only nameservers.  
> 
> Considering that one can run an instance of an anycasted recursive 
> nameserver, under heavy load for a very large number of clients, on a $600 1U 
> server these days... I wonder what exactly the threat model is. 
> 
> Reverse amplification of DNS traffic returning to the spoofed IPs for DoS 
> purposes, such as to cause the nameserver to DoS a single /32 endpoint IP 
> being targeted, as in common online gaming disputes?  
> 
> What volume of pps or Mbps would appear as spurious traffic as a result of 
> this attack? 
> 
> On Tue, Oct 13, 2020 at 3:14 PM Brian Knight via NANOG  
> wrote: 
> 
>> We recently received an email notice from a group of security 
>> researchers who are looking at the feasibility of attacks using spoofed 
>> traffic.  Their methodology, in broad strokes, was to send traffic to 
>> our DNS servers with a source IP that looked like it came from our 
>> network.  Their attacks were successful, and they included a summary of 
>> what they found.  So this message has started an internal conversation 
>> on what traffic we should be filtering and how.
>> 
>> This security test was not about BCP 38 for ingress traffic from our 
>> customers, nor was it about BGP ingress filtering.  This tested our 
>> ingress filtering from the rest of the Internet.
>> 
>> It seems to me like we should be filtering traffic with spoofed IPs on 
>> our transit, IX, and peering links.  I have done this filtering in the 
>> enterprise world extensively, and it's very helpful to keep out bad 
>> traffic.  BCP 84 also discusses ingress filtering for SP's briefly and 
>> seems to advocate for it.
>> 
>> We have about 15 IP blocks allocated to us.  With a network as small as 
>> ours, I chose to go with a static ACL approach to start the 
>> conversation.  I crafted a static ACL, blocking all ingress traffic 
>> sourced from any of our assigned IP ranges.  I made sure to include:
>> 
>> * Permit entries for P-t-P WAN subnets on peering links
>> * Permit entries for IP assignments to customers running multi-homed BGP
>> * The "permit ipv4 any any" at the end :)
>> 
>> The questions I wanted to ask the SP community are:
>> 
>> * What traffic filtering do you do on your transits, on IX ports, and 
>> your direct peering links?
>> * How is it accomplished?  Through static ACL or some flavor of uRPF?
>> * If you use static ACLs, what is the administrative overhead like?  
>> What makes it easy or difficult to update?
>> * How did you test your filters when they were implemented?
>> 
>> Thanks a lot,
>> 
>> -Brian

Re: Ingress filtering on transits, peers, and IX ports

[Brian Knight via NANOG]

Hi Marcos,

Thanks for your reply.  But I am looking for guidance on traffic 
filtering, not BGP prefix filtering.


I have looked at BCP 84, and it's a good overview of the methods 
available to an ISP.  My questions are more operational in nature and 
are not covered by the document.  Of the choices presented in BCP 84, 
what do folks really use?  If it's an ACL, what challenges have there 
been with updates?  etc.


-Brian


On 2020-10-13 18:52, Marcos Manoni wrote:

Hi, Brian

Check RFC3704/BCP84 Ingress Filtering for Multihomed Networks (Updated
by: RFC8704 Enhanced Feasible-Path uRPF).

Ingress Access Lists require typically manual maintenance, but are
the most bulletproof when done properly; typically, ingress access
lists are best fit between the edge and the ISP when the
configuration is not too dynamic if strict RPF is not an option,
between ISPs if the number of used prefixes is low, or as an
additional layer of protection


Ingress filters Best Practices
https://www.ripe.net/support/training/material/bgp-operations-and-security-training-course/BGP-Slides-Single.pdf
*Don’t accept BOGON ASNs
*Don’t accept BOGON prefixes
*Don’t accept your own prefix
*Don’t accept default (unless you requested it)
*Don’t accept prefixes that are too specific
*Don’t accept if AS Path is too long
*Create filters based on Internet Routing Registries

And also BGP Best Current Practices by Philip Smith
http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf

Regards.

El mar., 13 oct. 2020 a las 19:52, Brian Knight via NANOG
() escribió:


Hi Mel,

My understanding of uRPF is:

* Strict mode will permit a packet only if there is a route for the 
source IP in the RIB, and that route points to the interface where the 
packet was received


* Loose mode will permit a packet if there is a route for the source 
IP in the RIB.  It does not matter where the route is pointed.


Strict mode won't work for us, because with our multi-homed transits 
and IX peers, we will almost certainly drop a legitimate packet 
because the best route is through another transit.


Loose mode won't work for us, because all of our own prefixes are in 
our RIB, and thus the uRPF check on a transit would never block 
anything.


Or am I missing something?

Thanks,

-Brian

On 2020-10-13 17:22, Mel Beckman wrote:

You can also use Unicast Reverse Path Forwarding. RPF is more 
efficient than ACLs, and has the added advantage of not requiring 
maintenance. In a nutshell, if your router has a route to a prefix in 
its local RIB, then incoming packets from a border interface having a 
matching source IP will be dropped.


RPF has knobs and dials to make it work for various ISP environments. 
Implement it carefully (as is be standing next to the router involved 
:)


Here's a Cisco brief on the topic:


https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwarding





I think all router vendors support this feature. Here's a similar 
article by Juniper:


https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/interfaces-configuring-unicast-rpf.html


-mel beckman

On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG  
wrote:


We recently received an email notice from a group of security 
researchers who are looking at the feasibility of attacks using 
spoofed traffic.  Their methodology, in broad strokes, was to send 
traffic to our DNS servers with a source IP that looked like it came 
from our network.  Their attacks were successful, and they included a 
summary of what they found.  So this message has started an internal 
conversation on what traffic we should be filtering and how.


This security test was not about BCP 38 for ingress traffic from our 
customers, nor was it about BGP ingress filtering.  This tested our 
ingress filtering from the rest of the Internet.


It seems to me like we should be filtering traffic with spoofed IPs on 
our transit, IX, and peering links.  I have done this filtering in the 
enterprise world extensively, and it's very helpful to keep out bad 
traffic.  BCP 84 also discusses ingress filtering for SP's briefly and 
seems to advocate for it.


We have about 15 IP blocks allocated to us.  With a network as small 
as ours, I chose to go with a static ACL approach to start the 
conversation.  I crafted a static ACL, blocking all ingress traffic 
sourced from any of our assigned IP ranges.  I made sure to include:


* Permit entries for P-t-P WAN subnets on peering links

* Permit entries for IP assignments to customers running multi-homed 
BGP


* The "permit ipv4 any any" at the end :)

The questions I wanted to ask the SP community are:

* What traffic filtering do you do on your transits, on IX ports, and 
your direct peering links?


* How is it accomplished?  Through static ACL or some flavor of uRPF?

* If you use static ACLs, what is the administrative overhead like?  
What makes it easy or difficult to update?


* How did you

Re: Ingress filtering on transits, peers, and IX ports

[Brian Knight via NANOG]

Hi Mel, 

My understanding of uRPF is: 

* Strict mode will permit a packet only if there is a route for the
source IP in the RIB, and that route points to the interface where the
packet was received 

* Loose mode will permit a packet if there is a route for the source IP
in the RIB.  It does not matter where the route is pointed. 

Strict mode won't work for us, because with our multi-homed transits and
IX peers, we will almost certainly drop a legitimate packet because the
best route is through another transit. 

Loose mode won't work for us, because all of our own prefixes are in our
RIB, and thus the uRPF check on a transit would never block anything. 

Or am I missing something? 

Thanks, 

-Brian 

On 2020-10-13 17:22, Mel Beckman wrote:

> You can also use Unicast Reverse Path Forwarding. RPF is more efficient than 
> ACLs, and has the added advantage of not requiring maintenance. In a 
> nutshell, if your router has a route to a prefix in its local RIB, then 
> incoming packets from a border interface having a matching source IP will be 
> dropped. 
> 
> RPF has knobs and dials to make it work for various ISP environments. 
> Implement it carefully (as is be standing next to the router involved :) 
> 
> Here's a Cisco brief on the topic: 
> 
> https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwarding
>  
> 
> I think all router vendors support this feature. Here's a similar article by 
> Juniper: 
> 
> https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/interfaces-configuring-unicast-rpf.html
>  
> 
> -mel beckman
> 
>> On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG  wrote:
> 
>> We recently received an email notice from a group of security researchers 
>> who are looking at the feasibility of attacks using spoofed traffic.  Their 
>> methodology, in broad strokes, was to send traffic to our DNS servers with a 
>> source IP that looked like it came from our network.  Their attacks were 
>> successful, and they included a summary of what they found.  So this message 
>> has started an internal conversation on what traffic we should be filtering 
>> and how.
> 
>> This security test was not about BCP 38 for ingress traffic from our 
>> customers, nor was it about BGP ingress filtering.  This tested our ingress 
>> filtering from the rest of the Internet.
> 
>> It seems to me like we should be filtering traffic with spoofed IPs on our 
>> transit, IX, and peering links.  I have done this filtering in the 
>> enterprise world extensively, and it's very helpful to keep out bad traffic. 
>>  BCP 84 also discusses ingress filtering for SP's briefly and seems to 
>> advocate for it.
> 
>> We have about 15 IP blocks allocated to us.  With a network as small as 
>> ours, I chose to go with a static ACL approach to start the conversation.  I 
>> crafted a static ACL, blocking all ingress traffic sourced from any of our 
>> assigned IP ranges.  I made sure to include:
> 
>> * Permit entries for P-t-P WAN subnets on peering links
> 
>> * Permit entries for IP assignments to customers running multi-homed BGP
> 
>> * The "permit ipv4 any any" at the end :)
> 
>> The questions I wanted to ask the SP community are:
> 
>> * What traffic filtering do you do on your transits, on IX ports, and your 
>> direct peering links?
> 
>> * How is it accomplished?  Through static ACL or some flavor of uRPF?
> 
>> * If you use static ACLs, what is the administrative overhead like?  What 
>> makes it easy or difficult to update?
> 
>> * How did you test your filters when they were implemented?
> 
>> Thanks a lot,
> 
>> -Brian

Ingress filtering on transits, peers, and IX ports

[Brian Knight via NANOG]

We recently received an email notice from a group of security 
researchers who are looking at the feasibility of attacks using spoofed 
traffic.  Their methodology, in broad strokes, was to send traffic to 
our DNS servers with a source IP that looked like it came from our 
network.  Their attacks were successful, and they included a summary of 
what they found.  So this message has started an internal conversation 
on what traffic we should be filtering and how.


This security test was not about BCP 38 for ingress traffic from our 
customers, nor was it about BGP ingress filtering.  This tested our 
ingress filtering from the rest of the Internet.


It seems to me like we should be filtering traffic with spoofed IPs on 
our transit, IX, and peering links.  I have done this filtering in the 
enterprise world extensively, and it's very helpful to keep out bad 
traffic.  BCP 84 also discusses ingress filtering for SP's briefly and 
seems to advocate for it.


We have about 15 IP blocks allocated to us.  With a network as small as 
ours, I chose to go with a static ACL approach to start the 
conversation.  I crafted a static ACL, blocking all ingress traffic 
sourced from any of our assigned IP ranges.  I made sure to include:


* Permit entries for P-t-P WAN subnets on peering links
* Permit entries for IP assignments to customers running multi-homed BGP
* The "permit ipv4 any any" at the end :)

The questions I wanted to ask the SP community are:

* What traffic filtering do you do on your transits, on IX ports, and 
your direct peering links?

* How is it accomplished?  Through static ACL or some flavor of uRPF?
* If you use static ACLs, what is the administrative overhead like?  
What makes it easy or difficult to update?

* How did you test your filters when they were implemented?

Thanks a lot,

-Brian