In response to Charles: For #2, after you empty actionunban, write a script to iterate thru your ipset and for each IP issue: fail2ban-client set <JAIL> unbanip <IP> now fail2ban forgets about them and since actionunban is empty they're still in the ipset. No duplicate data.
For #3, some admins don't save on shutdown nor restore on boot their ipset. I don't. I just let the system build it from scratch. For Tom and all: I too use Shorewall. I actually create my ipsets with a timeout: ipset -exist create fail2ban-Ip hash:ip timeout 86400 # ------------------------------------------------------------------------------ my action: [INCLUDES] before = iptables-blocktype.conf [Definition] # start, stop, et. al. handled by Shorewall actioncheck = actionstart = actionstop = actionban = ipset -exist add fail2ban-<name> <ip> timeout <my_timeout> #actionunban = ipset -exist del fail2ban-<name> <ip> actionunban = [Init] name = Ip my_timeout = 3600 # ------------------------------------------------------------------------------ my jail: [ProFTP] #enabled = false enabled = true filter = proftpd action = my_ipset_ip[my_timeout=86400] logpath = /var/log/proftpd/system.log maxretry = 3 findtime = 7200 bantime = 60 Note: an ipset timeout value of zero IS a permanent ban (man ipset). Change <my_timeout> as needed in each jail. Bill On 2/12/2016 9:16 AM, Tom Hendrikx wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Hi, > > Maybe an interesting side note: fail2ban is built to quickly ban *and* > unban problematic ip addresses. The whole nature of fail2ban is (IMHO) > in the fact that it automatically unbans ip addresses after a while. > > However, you state that you have a list of 17000 ip adresses that are > permanently banned. There is no reason to have fail2ban maintain this > list. I fixed this by having an action in f2b that sent the addresses to > the shorewall blacklist (which IS devised for perm bans). An empty unban > action and irrelevant unban time complete the setup. > > > You can replace shorewall with any other solution you like, of course. > But keeping the perm bans in f2b is IMHO simply using the wrong tool > for the job. > > Regards, > Tom > > On 12-02-16 14:49, Charles Bradshaw wrote: >> Bill, >> >> Sorry again, I actually miss read your first reply. I read >> actionban instead of actionunban. >> >> I am indeed saving and restoring the ipset. At least, that's what I >> used to do until I found fail2ban taking hours to shutdown. Last >> time I hit the boot button after about an hour with the result that >> the ipset was left intact. ipset has built in and well documented >> method for backup and restore. >> >> While I understand your proposed method and see how it would work, >> I make the following observations: >> >> 1 - Your method has a certain pragmatic elegance, but is devious >> and will certainly confuse the uninitiated! >> >> 2 - I can see how your method will work if implemented from square >> one, but what about the 17000 odds IP which have been previously >> band with a ban time of forever? I've been running the particular >> jail with bantime = -1 for well over a year now. >> >> 3 - Why store anything at all in an external database. Ipsets are >> just that, a highly efficient linked to iptables database. The >> botnet problem is increasing rapidly. Today I'm seeing 8/hour >> originally it was 2 or 3. In the meantime > 17000 IP have been >> permanently banned. That says there are botnets out there with >> orders more than 10000 infected machines! We know not when this >> will, in effect, escalate to Denial of Service! Several hours to >> shutdown is a kind of DNS! >> >> Back on a pragmatic front, storing and manipulating vast amounts >> of duplicate data is simply not good practice. If you look out >> there you will find much discussion on the subject of how to unban >> the inadvertently banned. I might be wrong, but I suspect because >> sqlite permanent banning was implemented without due consideration >> of the consequences on existing installations. >> >> I think what I really need to understand now is; how does fail2ban >> 'think' an IP is banned or not. Where is the database? When is it >> written/read? In what version of fail2ban did sqlite get >> implemented. At present my /var/lib/fail2ban/fail2ban.sqlite3 has >> 7.9MB of entries. >> >> I ask again how do I turn sqlite activity off? Just point me at >> the documentation. >> >> Charles Bradshaw >> >> On Thu, 2016-02-11 at 22:31 -0500, Bill Shirley wrote: >>> When you said: This leaves the ipset intact. I made the the >>> assumption, maybe incorrectly, that you were saving your ipset >>> with some utility on shutdown and restoring after a re-boot. >>> >>> If that IS the case then change your jail to: bantime = 60 >>> >>> and make actionunban empty in your .local action: #actionunban = >>> ipset -exist del fail2ban-<name> <ip> actionunban = >>> >>> fail2ban will ban the IP address and in one minute it will unban >>> it. However, with actionunban being empty, the IP address will >>> not be removed from the ipset. So now fail2ban thinks very few, >>> if any, addresses are banned. With very few addresses to >>> 'remove', shutdown should be quick. >>> >>> Bill >>> >>> >>> On 2/11/2016 7:03 PM, Charles Bradshaw wrote: >>> >>>> Thanks Bill, >>>> >>>> Sorry I'm being a bit dim. Do you mean to temporarily modify >>>> the actionban in /etc/fail2ban/action.d/myaction.conf before >>>> the shutdown? How does that affect the shutdown? I can see how >>>> it affects the restart but eh.. no action actionban no bans at >>>> all after restart! >>>> >>>> Surely deleting the actionstop clause altogether, thus >>>> preventing deletion of the ipset and a modified actionstart to >>>> do nothing if the ipset already exists. Then neither start nor >>>> stop take time. >>>> >>>> I see the new sqlite behavior, but then where is the reference >>>> to dbfile forcing all the bans into >>>> /var/lib/fail2ban/fail2ban.sqlite3 it is not in my >>>> fail2ban.conf! If its use is default behaviour how do I >>>> disable it? >>>> >>>> On Thu, 2016-02-11 at 12:19 -0500, Bill Shirley wrote: >>>>> Try using an empty actionunban in your action and set the >>>>> bantime = 60 in your jail. This way fail2ban thinks it's >>>>> unbanning after a minute. fail2ban shutdown should be >>>>> quick. >>>>> >>>>> Bill >>>>> >>>>> On 2/11/2016 5:15 AM, Charles Bradshaw wrote: >>>>>> Hello list, >>>>>> >>>>>> I am running fail2ban.noarch 0.9.3-1.el6.1 as installed >>>>>> from the CentOS repository. >>>>>> >>>>>> I have one ipset jail which over time has accumulated more >>>>>> than 17000 permanent bans. This is causing a severe problem >>>>>> during restarts. (obviously!) >>>>>> >>>>>> First it would take many hours to shut down fail2ban >>>>>> gracefully the solution is to force a power down. This >>>>>> leaves the ipset intact. >>>>>> >>>>>> Next when the fail2ban server restarts it takes a similar >>>>>> many hours for the server to redundantly restore the bans >>>>>> from the database to the already intact ipset. >>>>>> >>>>>> This a ridiculous process! The whole purpose of ipsets is >>>>>> to efficiently hold vast numbers of blocked IPs. >>>>>> >>>>>> The most importantly problem here is fail2ban is preventing >>>>>> fast clean shutdowns. Understand 17000 bans is nothing! an >>>>>> ipset can efficiently hold > 65K, under which circumstances >>>>>> the shutdown and restart delays would extend to weeks!! The >>>>>> startup delay is not a severe problem except that 17000 >>>>>> emails and all the disk activity is a total pain in the >>>>>> ass. >>>>>> >>>>>> So the question is: how to turn off fail2ban gracefully >>>>>> without these ridiculous delays. >>>>>> >>>>>> Also note when fail2ban shuts down the ipset entries in >>>>>> iptables do not get deleted, but that's another story. >>>>>> >>>>>> Thanks in advance, Charles Bradshaw >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> ------------------------------------------------------------------ > - ------------ >>>>>> > Site24x7 APM Insight: Get Deep Visibility into Application Performance >>>>>> APM + Mobile APM + RUM: Monitor 3 App instances at just >>>>>> $35/Month Monitor end-to-end web transactions and take >>>>>> corrective actions now Troubleshoot faster and improve >>>>>> end-user experience. Signup Now! >>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 >>>>>> >>>>>> > _______________________________________________ >>>>>> Fail2ban-users mailing list >>>>>> [email protected] >>>>>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users >>>>> > - ------------------------------------------------------------------------ > - ------ >>>>> Site24x7 APM Insight: Get Deep Visibility into Application >>>>> Performance APM + Mobile APM + RUM: Monitor 3 App instances >>>>> at just $35/Month Monitor end-to-end web transactions and >>>>> take corrective actions now Troubleshoot faster and improve >>>>> end-user experience. Signup Now! >>>>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 >>>>> >>>>> > _______________________________________________ >>>>> Fail2ban-users mailing list >>>>> [email protected] >>>>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users >>>> >>>> -------------------------------------------------------------------- > - ---------- >>>> > Site24x7 APM Insight: Get Deep Visibility into Application Performance >>>> APM + Mobile APM + RUM: Monitor 3 App instances at just >>>> $35/Month Monitor end-to-end web transactions and take >>>> corrective actions now Troubleshoot faster and improve end-user >>>> experience. Signup Now! >>>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 >>>> >>>> > _______________________________________________ >>>> Fail2ban-users mailing list >>>> [email protected] >>>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users >>> --------------------------------------------------------------------- > - --------- >>> > Site24x7 APM Insight: Get Deep Visibility into Application Performance >>> APM + Mobile APM + RUM: Monitor 3 App instances at just >>> $35/Month Monitor end-to-end web transactions and take corrective >>> actions now Troubleshoot faster and improve end-user experience. >>> Signup Now! >>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 >>> _______________________________________________ Fail2ban-users >>> mailing list [email protected] >>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users >> >> >> ---------------------------------------------------------------------- > - -------- >> > Site24x7 APM Insight: Get Deep Visibility into Application Performance >> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month >> Monitor end-to-end web transactions and take corrective actions >> now Troubleshoot faster and improve end-user experience. Signup >> Now! >> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 >> _______________________________________________ Fail2ban-users >> mailing list [email protected] >> https://lists.sourceforge.net/lists/listinfo/fail2ban-users >> > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2 > > iQIcBAEBCAAGBQJWvelNAAoJEJPfMZ19VO/1qX4P/1XjghWFSDPUY1S40/lxsXtb > MTyk6UWcD0AqqSMVzcLfV+eK+WGNJqwgTGvyQ7xECurrHCDFQrXeTCkfYVyRw7Hn > RH6IswOdjZFeP8gP3kDErdu932ccXiBdu9pBkSQQlO+M2+ISWnPr2hKJla80jDKM > n0xK+vRjrUjlUormlRIFdDmtY0ITdGrN2j3yzAUDrgCxw/nHlsD+aA2ohbEs5jtp > 1EGyOZ5Bz3VXlKlQ2uEIx3kYK4brGUplX0tG+KkValgb8KSmjaxItcez4kyut9oM > GjvwHVNz4Pc/5lsvBXovj8deDwXPHi8jzqxUiBrlmmXFGHB7Fk1Xo4gG5UJOJvtt > m2MS2f16/dqNSBQBDmkj5p9XUFsT2/2TtyC3CNTNgdUHWI7kppUoW7DICvQPJlKM > q1k6oLHeNKfqkM/NvJsOb0A+nHlVjRUDmvZQeAygwcvdPpsYUdnzvFJckx1GltVI > 7yasTR68afWEFvnz6Cc1Up/R0zQV2LS7nxT/F1s5yhcy/+jGzPsPSBJm/xqjZiF+ > DZ3Ii7omUaH9SgjqhuVIK0UgW571nLJoNMENXu6rLgu/kQt+gS5grKzXTvVhUUPS > iqZHKxC3rBk4lg3JeprjV2bN3AA5mbS93MCBqfpzc/YiIpSH/E3i5xZI8Hb1M2VK > WMwHBjXeNjmuZjOGAe5C > =GnPZ > -----END PGP SIGNATURE----- > > ------------------------------------------------------------------------------ > Site24x7 APM Insight: Get Deep Visibility into Application Performance > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month > Monitor end-to-end web transactions and take corrective actions now > Troubleshoot faster and improve end-user experience. Signup Now! > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > _______________________________________________ > Fail2ban-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/fail2ban-users ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 _______________________________________________ Fail2ban-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/fail2ban-users
