I stand corrected. I replied off list because we are drifting off topic expressing opinions.
On Fri, 2016-02-12 at 17:42 +0100, Tom Hendrikx wrote: > Hi, > > Please don't reply off-list. See comments below. > > On 12-02-16 17:17, Charles Bradshaw wrote: > > Hi Tom > > > > Humm.. yes we agree that storing vast numbers of bans in fail2ban > > is the wrong approach. Duplicate data always is. > > > > I'm unfamiliar shorewall, I'm almost sure ipset is iptables > > equivalent of shorewall blacklists. > > You are wrong. Shorewall is a management wrapper around > iptables/ipset, just like f2b is. It is just better suited for > managing static rules. > > > The whole point is ipsets are fast, efficient and separate and > > therefore ease the maintenance problem. > > Yes. But f2b is not fast. Which is why you should not look for a > solution to your problem within f2b. > > > Looking out there, it's pretty obvious the uninitiated are having > > problems with fail2bans complexity, especially with permanent bans > > and how to unban the inadvertent ones. > > Your bans are static, not dynamic, so *don't* try to manage it with a > tool that is dedicated towards managing dynamic bans. KISS principle > applies here. > > > > > I'm not even sure that permanent banning is a good idea. I started > > out trying to reduce the frequency of fake attempts at my forum, > > but after more than 12 months I'm still seeing an increase in the > > number of bans per hour! > > If you've read up on botnets, you known this si game you'll lose. The > bots won't disappear, and it wont help you to ban them forever. But I > wasn't really looking into this part of your problem, just trying to > solve your "how do I manage a lot of perm bans efficiently" problem. > > > > > If you know how to cleanly disable the sqlite functionality I would > > be grateful for the heads-up. > > > > Charles Bradshaw > > > > On Fri, 2016-02-12 at 15:16 +0100, Tom Hendrikx wrote: > >> Hi, > >> > >> Maybe an interesting side note: fail2ban is built to quickly ban > >> *and* unban problematic ip addresses. The whole nature of > >> fail2ban is (IMHO) in the fact that it automatically unbans ip > >> addresses after a while. > >> > >> However, you state that you have a list of 17000 ip adresses that > >> are permanently banned. There is no reason to have fail2ban > >> maintain this list. I fixed this by having an action in f2b that > >> sent the addresses to the shorewall blacklist (which IS devised > >> for perm bans). An empty unban action and irrelevant unban time > >> complete the setup. > >> > >> > >> You can replace shorewall with any other solution you like, of > >> course. But keeping the perm bans in f2b is IMHO simply using the > >> wrong tool for the job. > >> > >> Regards, Tom > >> > >> On 12-02-16 14:49, Charles Bradshaw wrote: > >>> Bill, > >>> > >>> Sorry again, I actually miss read your first reply. I read > >>> actionban instead of actionunban. > >>> > >>> I am indeed saving and restoring the ipset. At least, that's > >>> what I used to do until I found fail2ban taking hours to > >>> shutdown. Last time I hit the boot button after about an hour > >>> with the result that the ipset was left intact. ipset has built > >>> in and well documented method for backup and restore. > >>> > >>> While I understand your proposed method and see how it would > >>> work, I make the following observations: > >>> > >>> 1 - Your method has a certain pragmatic elegance, but is > >>> devious and will certainly confuse the uninitiated! > >>> > >>> 2 - I can see how your method will work if implemented from > >>> square one, but what about the 17000 odds IP which have been > >>> previously band with a ban time of forever? I've been running > >>> the particular jail with bantime = -1 for well over a year > >>> now. > >>> > >>> 3 - Why store anything at all in an external database. Ipsets > >>> are just that, a highly efficient linked to iptables database. > >>> The botnet problem is increasing rapidly. Today I'm seeing > >>> 8/hour originally it was 2 or 3. In the meantime > 17000 IP > >>> have been permanently banned. That says there are botnets out > >>> there with orders more than 10000 infected machines! We know > >>> not when this will, in effect, escalate to Denial of Service! > >>> Several hours to shutdown is a kind of DNS! > >>> > >>> Back on a pragmatic front, storing and manipulating vast > >>> amounts of duplicate data is simply not good practice. If you > >>> look out there you will find much discussion on the subject of > >>> how to unban the inadvertently banned. I might be wrong, but I > >>> suspect because sqlite permanent banning was implemented > >>> without due consideration of the consequences on existing > >>> installations. > >>> > >>> I think what I really need to understand now is; how does > >>> fail2ban 'think' an IP is banned or not. Where is the database? > >>> When is it written/read? In what version of fail2ban did sqlite > >>> get implemented. At present my > >>> /var/lib/fail2ban/fail2ban.sqlite3 has 7.9MB of entries. > >>> > >>> I ask again how do I turn sqlite activity off? Just point me > >>> at the documentation. > >>> > >>> Charles Bradshaw > >>> > >>> On Thu, 2016-02-11 at 22:31 -0500, Bill Shirley wrote: > >>>> When you said: This leaves the ipset intact. I made the the > >>>> assumption, maybe incorrectly, that you were saving your > >>>> ipset with some utility on shutdown and restoring after a > >>>> re-boot. > >>>> > >>>> If that IS the case then change your jail to: bantime = 60 > >>>> > >>>> and make actionunban empty in your .local action: > >>>> #actionunban = ipset -exist del fail2ban-<name> <ip> > >>>> actionunban = > >>>> > >>>> fail2ban will ban the IP address and in one minute it will > >>>> unban it. However, with actionunban being empty, the IP > >>>> address will not be removed from the ipset. So now fail2ban > >>>> thinks very few, if any, addresses are banned. With very few > >>>> addresses to 'remove', shutdown should be quick. > >>>> > >>>> Bill > >>>> > >>>> > >>>> On 2/11/2016 7:03 PM, Charles Bradshaw wrote: > >>>> > >>>>> Thanks Bill, > >>>>> > >>>>> Sorry I'm being a bit dim. Do you mean to temporarily > >>>>> modify the actionban in > >>>>> /etc/fail2ban/action.d/myaction.conf before the shutdown? > >>>>> How does that affect the shutdown? I can see how it affects > >>>>> the restart but eh.. no action actionban no bans at all > >>>>> after restart! > >>>>> > >>>>> Surely deleting the actionstop clause altogether, thus > >>>>> preventing deletion of the ipset and a modified actionstart > >>>>> to do nothing if the ipset already exists. Then neither > >>>>> start nor stop take time. > >>>>> > >>>>> I see the new sqlite behavior, but then where is the > >>>>> reference to dbfile forcing all the bans into > >>>>> /var/lib/fail2ban/fail2ban.sqlite3 it is not in my > >>>>> fail2ban.conf! If its use is default behaviour how do I > >>>>> disable it? > >>>>> > >>>>> On Thu, 2016-02-11 at 12:19 -0500, Bill Shirley wrote: > >>>>>> Try using an empty actionunban in your action and set > >>>>>> the bantime = 60 in your jail. This way fail2ban thinks > >>>>>> it's unbanning after a minute. fail2ban shutdown should > >>>>>> be quick. > >>>>>> > >>>>>> Bill > >>>>>> > >>>>>> On 2/11/2016 5:15 AM, Charles Bradshaw wrote: > >>>>>>> Hello list, > >>>>>>> > >>>>>>> I am running fail2ban.noarch 0.9.3-1.el6.1 as > >>>>>>> installed from the CentOS repository. > >>>>>>> > >>>>>>> I have one ipset jail which over time has accumulated > >>>>>>> more than 17000 permanent bans. This is causing a > >>>>>>> severe problem during restarts. (obviously!) > >>>>>>> > >>>>>>> First it would take many hours to shut down fail2ban > >>>>>>> gracefully the solution is to force a power down. This > >>>>>>> leaves the ipset intact. > >>>>>>> > >>>>>>> Next when the fail2ban server restarts it takes a > >>>>>>> similar many hours for the server to redundantly > >>>>>>> restore the bans from the database to the already > >>>>>>> intact ipset. > >>>>>>> > >>>>>>> This a ridiculous process! The whole purpose of ipsets > >>>>>>> is to efficiently hold vast numbers of blocked IPs. > >>>>>>> > >>>>>>> The most importantly problem here is fail2ban is > >>>>>>> preventing fast clean shutdowns. Understand 17000 bans > >>>>>>> is nothing! an ipset can efficiently hold > 65K, under > >>>>>>> which circumstances the shutdown and restart delays > >>>>>>> would extend to weeks!! The startup delay is not a > >>>>>>> severe problem except that 17000 emails and all the > >>>>>>> disk activity is a total pain in the ass. > >>>>>>> > >>>>>>> So the question is: how to turn off fail2ban > >>>>>>> gracefully without these ridiculous delays. > >>>>>>> > >>>>>>> Also note when fail2ban shuts down the ipset entries > >>>>>>> in iptables do not get deleted, but that's another > >>>>>>> story. > >>>>>>> > >>>>>>> Thanks in advance, Charles Bradshaw > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> ---------------------------------------------------------------- > -- > >> > >>>>>>> > ------------ > >>>>>>> > >>>>>>> > >> Site24x7 APM Insight: Get Deep Visibility into Application > >> Performance > >>>>>>> APM + Mobile APM + RUM: Monitor 3 App instances at > >>>>>>> just $35/Month Monitor end-to-end web transactions and > >>>>>>> take corrective actions now Troubleshoot faster and > >>>>>>> improve end-user experience. Signup Now! > >>>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > >>>>>>> > >>>>>>> > >> > >>>>>>> > _______________________________________________ > >>>>>>> Fail2ban-users mailing list > >>>>>>> [email protected] > >>>>>>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users > >>>>>> > >>>>>> > >>>>>>> > >> > >>>>>>> > ------------------------------------------------------------------------ > >> ------ > >>>>>> Site24x7 APM Insight: Get Deep Visibility into > >>>>>> Application Performance APM + Mobile APM + RUM: Monitor 3 > >>>>>> App instances at just $35/Month Monitor end-to-end web > >>>>>> transactions and take corrective actions now Troubleshoot > >>>>>> faster and improve end-user experience. Signup Now! > >>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > >>>>>> > >>>>>> > >> > >>>>>> > _______________________________________________ > >>>>>> Fail2ban-users mailing list > >>>>>> [email protected] > >>>>>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users > >>>>> > >>>>> > >>>>> > >>>>>> > -------------------------------------------------------------------- > >> ---------- > >>>>> > >>>>> > >> Site24x7 APM Insight: Get Deep Visibility into Application > >> Performance > >>>>> APM + Mobile APM + RUM: Monitor 3 App instances at just > >>>>> $35/Month Monitor end-to-end web transactions and take > >>>>> corrective actions now Troubleshoot faster and improve > >>>>> end-user experience. Signup Now! > >>>>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > >>>>> > >>>>> > >> > >>>>> > _______________________________________________ > >>>>> Fail2ban-users mailing list > >>>>> [email protected] > >>>>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users > >>>> > >>>> > >>>>> > --------------------------------------------------------------------- > >> --------- > >>>> > >>>> > >> Site24x7 APM Insight: Get Deep Visibility into Application > >> Performance > >>>> APM + Mobile APM + RUM: Monitor 3 App instances at just > >>>> $35/Month Monitor end-to-end web transactions and take > >>>> corrective actions now Troubleshoot faster and improve > >>>> end-user experience. Signup Now! > >>>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > >>>> _______________________________________________ > >>>> Fail2ban-users mailing list > >>>> [email protected] > >>>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users > >>> > >>> > >>> > >>> -------------------------------------------------------------------- > -- > >> > >>> > -------- > >>> > >>> > >> Site24x7 APM Insight: Get Deep Visibility into Application > >> Performance > >>> APM + Mobile APM + RUM: Monitor 3 App instances at just > >>> $35/Month Monitor end-to-end web transactions and take > >>> corrective actions now Troubleshoot faster and improve end-user > >>> experience. Signup Now! > >>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > >>> _______________________________________________ > >>> Fail2ban-users mailing list > >>> [email protected] > >>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users > >>> > >> > >> --------------------------------------------------------------------- > --------- > >> > >> > Site24x7 APM Insight: Get Deep Visibility into Application Performance > >> APM + Mobile APM + RUM: Monitor 3 App instances at just > >> $35/Month Monitor end-to-end web transactions and take corrective > >> actions now Troubleshoot faster and improve end-user experience. > >> Signup Now! > >> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > >> _______________________________________________ Fail2ban-users > >> mailing list [email protected] > >> https://lists.sourceforge.net/lists/listinfo/fail2ban-users > > > > > > ------------------------------------------------------------------------------ > Site24x7 APM Insight: Get Deep Visibility into Application Performance > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month > Monitor end-to-end web transactions and take corrective actions now > Troubleshoot faster and improve end-user experience. Signup Now! > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > _______________________________________________ > Fail2ban-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/fail2ban-users ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 _______________________________________________ Fail2ban-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/fail2ban-users
