Re: How many IPs can I block before taking a performance hit?
So here is a related question - I want to implement something like what some of you folks seem to have in place with dynamically updated blacklists and reloading pf on the fly. With a redundant pair of firewalls should I be doing this on the MASTER only? I'm just wondering about reloading pf on the BACKUP and because state tables are synced is there going to be any kind of negative effect on MASTER in terms of performance or even the fact that MASTER will be doing the reload at the same time. It is relatively easy to detect whether or not I am MASTER and then only do that if so. Should I be doing that or will it matter?
Re: How many IPs can I block before taking a performance hit?
‐‐ Original Message ‐‐‐ On Wednesday, August 12, 2020 7:11 AM, Alan McKay wrote: > Hey folks, > > This is one that is difficult to test in a test environment. > > I've got OpenBSD 6.5 on a relatively new pair of servers each with 8G RAM. > > With some scripting I'm looking at feeding block IPs to the firewalls > to block bad-guys in near real time, but in theory if we got attacked > by a bot net or something like that, it could result in a few thousand > IPs being blocked. Possibly even 10s of thousands. > > Are there any real-world data out there on how big of a block list we > can handle without impacting performance? > > We're doing the standard /etc/blacklist to load a table and then have > a block on the table right at the top of the ruleset. > > thanks, > -Alan > On our network, we maintain a running block of scanners but IP addresses are removed from the list after several hours of no scanning. One thing that has been useful for us is to create three sets of IP addresses from our allocation of IP addresses. One set ( currently with 9 IP addresses) allows incoming access from anywhere in the world. Another set (currently 18 IP addresses) allows incoming access from the US only. The third set (the remainder of our /24) allows no incoming access. Of course, each host may have its own rules to limit access to the services actually needed. Note that this applies to normal traffic. Regardless of where it originates, things like chargen are blocked for both incoming and outgoing traffic. Every afternoon, we download the current IPv4 and IPv6 address blocks for the US from http://www.ipdeny.com/ipblocks/data/aggregated/us-aggregated.zone for IPv4 and http://www.ipdeny.com/ipv6/ipaddresses/blocks/us.zone for IPv6. Thus, we use these lists to permit access to our "US only" hosts. The IP source of attempts to scan our IP addresses in the third set above are automatically added to the block of scanners to be blocked. These blocks are then applied to all incoming traffic. Thus, if someone tries to scan IP addresses of hosts that provide no services on the Internet, they are also blocked from connecting to any of our hosts for several hours. So if 192.0.2.20, for example, is seen as trying to scan our network, they will be blocked from accessing any of our network for a little while. During that time, connections to a service at 192.0.2.20 from our network are still permitted since it isn't entirely impossible that the interpretation of it being a network scan is an error. Walt
Re: How many IPs can I block before taking a performance hit?
Hi, I have a script that downloads "badhosts" from a site that continuously updates through a distrubed network. I currently limit my blocklist to 450,000 ip addresses. real mem = 4261072896 (4063MB) avail mem = 4119322624 (3928MB) bios0: PC Engines apu2 -pa-r-- blocklist Addresses: 45 Cleared: Tue May 26 18:45:08 2020 References: [ Anchors: 0 Rules: 1 ] Evaluations: [ NoMatch: 3794791 Match: 1172204 ] In/Block: [ Packets: 1172204 Bytes: 61337613 ] In/Match: [ Packets: 0 Bytes: 0 ] In/Pass: [ Packets: 0 Bytes: 0 ] In/XPass: [ Packets: 0 Bytes: 0 ] Out/Block: [ Packets: 0 Bytes: 0 ] Out/Match: [ Packets: 0 Bytes: 0 ] Out/Pass: [ Packets: 0 Bytes: 0 ] Out/XPass: [ Packets: 0 Bytes: 0 ] Cheers, Steve W. On 12/08/2020 6:11 a.m., Alan McKay wrote: Hey folks, This is one that is difficult to test in a test environment. I've got OpenBSD 6.5 on a relatively new pair of servers each with 8G RAM. With some scripting I'm looking at feeding block IPs to the firewalls to block bad-guys in near real time, but in theory if we got attacked by a bot net or something like that, it could result in a few thousand IPs being blocked. Possibly even 10s of thousands. Are there any real-world data out there on how big of a block list we can handle without impacting performance? We're doing the standard /etc/blacklist to load a table and then have a block on the table right at the top of the ruleset. thanks, -Alan
Re: How many IPs can I block before taking a performance hit?
Hi, as the tables are stored in RAM anyway during thee processing it’s moreless matter of how fast are your DIMMs / CPU. I’m usually work with several tables with cca 30 K records - no impact on the performance so far. S pozdravem / Kind regards Martin Sukaný UNIX Engineer, Developer, DevOps specialist xmpp: mar...@sukany.cz phone: +420 776 275 713 email: mar...@sukany.cz l: https://www.linkedin.com/in/martins6 > 12. 8. 2020 v 14:22, Stuart Harland : > > This is one of those “How long is a piece of string” examples. > > You don’t give a lot in the way of specifications so as to come up with a > reasonble guess. But the guesses are meaningless anyway, as the packet > filtering subsystems are pretty efficient and very rapid. > > In reality with sufficient CPU clock speed and memory for the state tables, > you should be able to simultaneously block thousands and thousands, if not > more. > > Not particularly scientific, but there we are. > > Stuart > >> On 12 Aug 2020, at 13:11, Alan McKay wrote: >> >> Hey folks, >> >> This is one that is difficult to test in a test environment. >> >> I've got OpenBSD 6.5 on a relatively new pair of servers each with 8G RAM. >> >> With some scripting I'm looking at feeding block IPs to the firewalls >> to block bad-guys in near real time, but in theory if we got attacked >> by a bot net or something like that, it could result in a few thousand >> IPs being blocked. Possibly even 10s of thousands. >> >> Are there any real-world data out there on how big of a block list we >> can handle without impacting performance? >> >> We're doing the standard /etc/blacklist to load a table and then have >> a block on the table right at the top of the ruleset. >> >> thanks, >> -Alan >> >> -- >> "You should sit in nature for 20 minutes a day. >> Unless you are busy, then you should sit for an hour" >>- Zen Proverb >> >
Re: How many IPs can I block before taking a performance hit?
On 2020-08-12, Tomasz Rola wrote: > Is there a way to have listing of offending IPs and perhaps grouping > them into /nn subnets - other than writing oneself the script? aggregate6, in packages. It will be slow on a large list, of course. > Something as easy as awk might suffice, I guess - and then instead of > five rules, just one rule for a subnet. If IPs are close enough to > form a subnet (now, what is "close enough", there might be interesting > problem). Of course, this way, some IPs will be excluded even if > they did nothing wrong (yet). it doesn't do this "fuzzy matching" though, it purely converts a fully filled subnet to the relevant prefix. e.g. $ printf '1.0.0.0\n1.0.0.1\n1.0.0.2\n' | aggregate6 1.0.0.0/31 1.0.0.2/32
Re: How many IPs can I block before taking a performance hit?
On 2020-08-12 05:11, Alan McKay wrote: Hey folks, This is one that is difficult to test in a test environment. I've got OpenBSD 6.5 on a relatively new pair of servers each with 8G RAM. With some scripting I'm looking at feeding block IPs to the firewalls to block bad-guys in near real time, but in theory if we got attacked by a bot net or something like that, it could result in a few thousand IPs being blocked. Possibly even 10s of thousands. Are there any real-world data out there on how big of a block list we can handle without impacting performance? We're doing the standard /etc/blacklist to load a table and then have a block on the table right at the top of the ruleset. thanks, -Alan At Otto said, if you're using tables, then you should be fine. I'm doing geoip blocking and all sorts of filtering using a pf table that contains over 200 undecillion addresses (that obviously includes CIDR block expansion): # Entries (+-) 9482 addresses added. 10859 addresses deleted. # Entries (expanded CIDR blocks) IPv4 addresses in table: 966545967 IPv6 addresses in table: 298179424470603435988810818668701155328 fw$ wc -l < /etc/pf-badhost.txt 146541
Re: How many IPs can I block before taking a performance hit?
On Wed, Aug 12, 2020 at 03:00:03PM +0200, Martin Sukany wrote: > Hi, > > as the tables are stored in RAM anyway during thee processing it’s > moreless matter of how fast are your DIMMs / CPU. I’m usually work > with several tables with cca 30 K records - no impact on the > performance so far. So, for as long as the table(s) do not spill out of cpu's cache, it is going to be a not so huge problem. If you run memtest, the difference between various caches is big, but cache vs ram is huge. Is there a way to have listing of offending IPs and perhaps grouping them into /nn subnets - other than writing oneself the script? Something as easy as awk might suffice, I guess - and then instead of five rules, just one rule for a subnet. If IPs are close enough to form a subnet (now, what is "close enough", there might be interesting problem). Of course, this way, some IPs will be excluded even if they did nothing wrong (yet). Another nice thing to have might be a utility which looks for rules and disables those which did not fired up during last x seconds (by looking up through firewall logs, perhaps). I have no idea if there is such utility and am not sure how to look it up. -- Regards, Tomasz Rola -- ** A C programmer asked whether computer had Buddha's nature. ** ** As the answer, master did "rm -rif" on the programmer's home** ** directory. And then the C programmer became enlightened... ** ** ** ** Tomasz Rola mailto:tomasz_r...@bigfoot.com **
Re: How many IPs can I block before taking a performance hit?
Wow over 160 MILLION (yes I screamed that) IPs! How much RAM is in your system? On Wed, Aug 12, 2020 at 10:26 AM infoomatic wrote: > > We have ~30,000 entries in our table blocking networks and > single ip addresses, all in all at the moment exactly 169,471,974 hosts > being blocked. No idea what your criteria is for "performance impact", > but we have no issues. > > > On 12.08.20 14:11, Alan McKay wrote: > > Hey folks, > > > > This is one that is difficult to test in a test environment. > > > > I've got OpenBSD 6.5 on a relatively new pair of servers each with 8G RAM. > > > > With some scripting I'm looking at feeding block IPs to the firewalls > > to block bad-guys in near real time, but in theory if we got attacked > > by a bot net or something like that, it could result in a few thousand > > IPs being blocked. Possibly even 10s of thousands. > > > > Are there any real-world data out there on how big of a block list we > > can handle without impacting performance? > > > > We're doing the standard /etc/blacklist to load a table and then have > > a block on the table right at the top of the ruleset. > > > > thanks, > > -Alan > > > -- "You should sit in nature for 20 minutes a day. Unless you are busy, then you should sit for an hour" - Zen Proverb
Re: How many IPs can I block before taking a performance hit?
We have ~30,000 entries in our table blocking networks and single ip addresses, all in all at the moment exactly 169,471,974 hosts being blocked. No idea what your criteria is for "performance impact", but we have no issues. On 12.08.20 14:11, Alan McKay wrote: > Hey folks, > > This is one that is difficult to test in a test environment. > > I've got OpenBSD 6.5 on a relatively new pair of servers each with 8G RAM. > > With some scripting I'm looking at feeding block IPs to the firewalls > to block bad-guys in near real time, but in theory if we got attacked > by a bot net or something like that, it could result in a few thousand > IPs being blocked. Possibly even 10s of thousands. > > Are there any real-world data out there on how big of a block list we > can handle without impacting performance? > > We're doing the standard /etc/blacklist to load a table and then have > a block on the table right at the top of the ruleset. > > thanks, > -Alan >
Re: How many IPs can I block before taking a performance hit?
On Wed, Aug 12, 2020 at 08:11:14AM -0400, Alan McKay wrote: > Hey folks, > > This is one that is difficult to test in a test environment. > > I've got OpenBSD 6.5 on a relatively new pair of servers each with 8G RAM. > > With some scripting I'm looking at feeding block IPs to the firewalls > to block bad-guys in near real time, but in theory if we got attacked > by a bot net or something like that, it could result in a few thousand > IPs being blocked. Possibly even 10s of thousands. > > Are there any real-world data out there on how big of a block list we > can handle without impacting performance? > > We're doing the standard /etc/blacklist to load a table and then have > a block on the table right at the top of the ruleset. > > thanks, > -Alan > > -- > "You should sit in nature for 20 minutes a day. > Unless you are busy, then you should sit for an hour" > - Zen Proverb > Typical answer: "it depends". Having in the order of 10k of rules might not be a smart idea. But if you are using tables you should do fine for many, many IPs. -Otto
Re: How many IPs can I block before taking a performance hit?
This is one of those “How long is a piece of string” examples. You don’t give a lot in the way of specifications so as to come up with a reasonble guess. But the guesses are meaningless anyway, as the packet filtering subsystems are pretty efficient and very rapid. In reality with sufficient CPU clock speed and memory for the state tables, you should be able to simultaneously block thousands and thousands, if not more. Not particularly scientific, but there we are. Stuart > On 12 Aug 2020, at 13:11, Alan McKay wrote: > > Hey folks, > > This is one that is difficult to test in a test environment. > > I've got OpenBSD 6.5 on a relatively new pair of servers each with 8G RAM. > > With some scripting I'm looking at feeding block IPs to the firewalls > to block bad-guys in near real time, but in theory if we got attacked > by a bot net or something like that, it could result in a few thousand > IPs being blocked. Possibly even 10s of thousands. > > Are there any real-world data out there on how big of a block list we > can handle without impacting performance? > > We're doing the standard /etc/blacklist to load a table and then have > a block on the table right at the top of the ruleset. > > thanks, > -Alan > > -- > "You should sit in nature for 20 minutes a day. > Unless you are busy, then you should sit for an hour" > - Zen Proverb >
How many IPs can I block before taking a performance hit?
Hey folks, This is one that is difficult to test in a test environment. I've got OpenBSD 6.5 on a relatively new pair of servers each with 8G RAM. With some scripting I'm looking at feeding block IPs to the firewalls to block bad-guys in near real time, but in theory if we got attacked by a bot net or something like that, it could result in a few thousand IPs being blocked. Possibly even 10s of thousands. Are there any real-world data out there on how big of a block list we can handle without impacting performance? We're doing the standard /etc/blacklist to load a table and then have a block on the table right at the top of the ruleset. thanks, -Alan -- "You should sit in nature for 20 minutes a day. Unless you are busy, then you should sit for an hour" - Zen Proverb
