Bug#554162: fail2ban: sometimes(frequently) fails to load, iptable rules with multiple jails
I created a patch for this bug. It is not very extensive. In fact it changes just the bare minimum needed to prevent the problem. I don't add a queue then have all the commands run by a single thread as Yaroslav suggested, I just added a lock to executeCmd() in the server/action.py file to prevent calling iptables or other shell commands concurrently. The patch can be found at: https://sourceforge.net/tracker/?func=detailaid=3413485group_id=121032atid=689046 (my apologies if that link wraps to the next line) Michael Saavedra -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#554162: fail2ban: sometimes(frequently) fails to load, iptable rules with multiple jails
it is awesome to have a brain and know where the hands come from, isn't it? ;-) THANK YOU Michael -- that looks like you solved our biggest problem and solved it right. Shame on me that I had started envisioning some complicated queue process -- locking imho is sufficient and due to simplicity of the implementation -- preferable. I have committed your patch upstream (sorry -- I haven't yet migrated it once and for all from SVN to GIT, so authorship is only in the comment) and uploaded fix to Debian. thanks again ;-) On Fri, 23 Sep 2011, Michael Saavedra wrote: I created a patch for this bug. It is not very extensive. In fact it changes just the bare minimum needed to prevent the problem. I don't add a queue then have all the commands run by a single thread as Yaroslav suggested, I just added a lock to executeCmd() in the server/action.py file to prevent calling iptables or other shell commands concurrently. The patch can be found at: https://sourceforge.net/tracker/?func=detailaid=3413485group_id=121032atid=689046 (my apologies if that link wraps to the next line) Michael Saavedra -- =--= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#554162: fail2ban: sometimes(frequently) fails to load iptable rules with multiple jails
Hi,
sorry for late answer.
I have removed action.py patch and put
sleep ${RANDOM:0:1}.${RANDOM: -1:1}
onto first line of actionstart and actionstop
So I get more sleep values without big delay - it should be from 0.0 to 3.9, I
think
It seems to work nicely now
With regards
Libor
Dne čtvrtek 05 Listopad 2009 16:17:13 Yaroslav Halchenko napsal(a):
oki doki... it is indeed the fact here -- whenever multiple
processes/threads/whatever contact iptables in parallel
$ for f in {1..5}; do cn=fail2ban-proftpd$f; echo $cn; /bin/sh -c
iptables -N $cn; iptables -A $cn -j RETURN; iptables -I INPUT -p tcp -m
multiport --dports ftp,ftp-data,ftps,ftps-data -j $cn ; done
fail2ban-proftpd1
fail2ban-proftpd2
fail2ban-proftpd3
fail2ban-proftpd4
fail2ban-proftpd5
$ for f in {1..5}; do cn=fail2ban-proftpd$f; echo $cn; iptables -D INPUT
-p tcp -m multiport --dports ftp,ftp-data,ftps,ftps-data -j $cn; iptables
-F $cn; iptables -X $cn; done fail2ban-proftpd1
fail2ban-proftpd2
fail2ban-proftpd3
fail2ban-proftpd4
fail2ban-proftpd5
So -- everything was cool, lets try now start them in parallel:
$ for f in {1..5}; do cn=fail2ban-proftpd$f; echo $cn; /bin/sh -c
iptables -N $cn; iptables -A $cn -j RETURN; iptables -I INPUT -p tcp -m
multiport --dports ftp,ftp-data,ftps,ftps-data -j $cn ; done
fail2ban-proftpd1
[2] 7312
fail2ban-proftpd2
[3] 7314
fail2ban-proftpd3
[4] 7315
fail2ban-proftpd4
[5] 7322
fail2ban-proftpd5
iptables: Invalid argument. Run `dmesg' for more information.
iptables: Resource temporarily unavailable.
$
$ for f in {1..5}; do cn=fail2ban-proftpd$f; echo $cn; iptables -D INPUT
-p tcp -m multiport --dports ftp,ftp-data,ftps,ftps-data -j $cn; iptables
-F $cn; iptables -X $cn; done fail2ban-proftpd1
fail2ban-proftpd2
fail2ban-proftpd3
iptables: No chain/target/match by that name.
fail2ban-proftpd4
fail2ban-proftpd5
So, here we got it -- iptables might not digest nicely a bulk of requests
done in parallel. Even if I set affinity for children tasks to run on the
same core with taskset (so there could not be real parallelism) -- same
shit.
You get more fun if you go beyond 5 ;) (all those missing .so etc messages)
BTW -- there were no further information in dmesg
So, as a temporary solution I would just advise you to add arbitrary sleep
time at the beginning of each chain initiation/shutdown(just in case),
smth like:
actionstart = sleep ${RANDOM:0:1}
iptables -N fail2ban-name
iptables -A fail2ban-name -j RETURN
iptables -I INPUT -p protocol -m multiport --dports port
-j fail2ban-name
actionstop = sleep ${RANDOM:0:1}
iptables -D INPUT -p protocol -m multiport --dports port
-j fail2ban-name iptables -F fail2ban-name
iptables -X fail2ban-name
I guess average delay of 5 sec would be tolerable, right? ;)
if you have better sleep then use 0.${RANDOM:0:1} ;-)
On Thu, 05 Nov 2009, Libor Klepáč wrote:
Hi,
You are welcome.
Let's see if someone else can confirm this behaviour. Maybe they will
have other ideas.
signature.asc
Description: This is a digitally signed message part.
Bug#554162: fail2ban: sometimes(frequently) fails to load iptable rules with multiple jails
Hi,
I have done some testing without the patch
I have tested
- putting sleep between commands (I have tested this yesterday too) - it
didn't help
- putting all commands in one line, separated with ; - it didn't help
- putting all commands in one line, separated with - it didn't help
- putting all commands in one line, separated with ; and I put () around
whole line - it failed differently, it seems, it separates commands somewhere
on it's own
-
sh: -c: line 1: syntax error: unexpected end of file
2009-11-04 09:34:33,313 fail2ban.actions.action: ERROR (iptables -N fail2ban-
postfix returned 200
-
now the question -- what is your /bin/sh? ;)
could you try different kernel? I've tried on 2.6.26-2-amd64
/bin/sh is bash 3.2-5
Sorry , I cannot use another kernel now, this is our main production server.
Well, I really don't know, where the problem might be, so I'll stick to this
patch for now, it works for me.
We have some filtering of smtp port based on results from amavis and 554 from
postfix, we have around 100-200 hosts banned in this jail (for 30 minutes) all
day, I think it filtres lots of spam - I can send it as some wishlist bug - but
i think it's based on some filter I have downloaded somewhere, so I don't
remember licence
Libor
Dne úterý 03 Listopad 2009 22:09:10 Yaroslav Halchenko napsal(a):
Patch is working, but i can't use action_mw
(output goes like this
---
2009-11-03 21:04:02,138 fail2ban.actions.action: ERROR printf %b
Subject: [Fail2Ban] cyrus: started
From: Fail2Ban fail2ban
well -- that is what the patch you've applied accomplished
unfortunately :-/ since the mailing command is a multiline printf
command. imho instead of that patch I would simple tune up the action
for banning -- just add sleep 1 (or sleep 0.1 if system has that recent
sleep ;)) after each command and see how that helps
or may be alternatively just placing all commands on 1 line with ;
between them -- so they should start as a one command (you said that
system tolerated that fine)
All commands in one line work ok from command line.
interesting -- thanks for checking
Sequence is also ok. Fail2ban launches actions in threads?
to say the truth -- I don't know exactly... from what I know, it just
calls system() from stdlib, which should call /bin/sh -c '.'
I still think that the issue is deeper underground (iptables return
before actually completing modification of the table etc) although
myself could not replicate it on none of boxes (biefie enough: 8 cores,
64GB RAM etc) with smth like
for f in {1..100}; do /bin/sh -c iptables -N fail2ban-proftpd; iptables -A
fail2ban-proftpd -j RETURN; iptables -I INPUT -p tcp -m multiport --dports
ftp,ftp-data,ftps,ftps-data -j fail2ban-proftpd; iptables -D INPUT -p tcp
-m multiport --dports ftp,ftp-data,ftps,ftps-data -j fail2ban-proftpd;
iptables -F fail2ban-proftpd; iptables -X fail2ban-proftpd; done
now the question -- what is your /bin/sh? ;)
could you try different kernel? I've tried on 2.6.26-2-amd64
signature.asc
Description: This is a digitally signed message part.
Bug#554162: fail2ban: sometimes(frequently) fails to load iptable rules with multiple jails
Thank you Libor for all the testing! sorry that none has helped how many jails do you have? fail2ban is threading to handle each jail in a separate thread (iirc) -- I guess problem might lie there and some people reported similar issue in the context of high number of jails On Wed, 04 Nov 2009, Libor Klepáč wrote: Hi, I have done some testing without the patch I have tested - putting sleep between commands (I have tested this yesterday too) - it didn't help - putting all commands in one line, separated with ; - it didn't help - putting all commands in one line, separated with - it didn't help - putting all commands in one line, separated with ; and I put () around whole line - it failed differently, it seems, it separates commands somewhere on it's own - sh: -c: line 1: syntax error: unexpected end of file 2009-11-04 09:34:33,313 fail2ban.actions.action: ERROR (iptables -N fail2ban- postfix returned 200 - -- .-. =-- /v\ = Keep in touch// \\ (yoh@|www.)onerussian.com Yaroslav Halchenko /( )\ ICQ#: 60653192 Linux User^^-^^[17] -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#554162: fail2ban: sometimes(frequently) fails to load iptable rules with multiple jails
Hi, there seems to be some problem with sequence of commands run from python. Maybe our server is too powerfull? (poweredge 2950 with two 4 core processors, old server was some old pentium4 with ht, there were no such problems) If i run those commands separately from command line ,there is no problem. But if whole (multiline) actionstart is run, there seems to be problem with order of commands. Problem is, that iptables -I INPUT -p tcp -m multiport --dports ftp,ftp-data,ftps,ftps-data -j fail2ban-proftpd is run in time, when there is no such chain, so kernel tries to autoload module for it (but there is no such module) (you can try it by running this command) You get this iptables v1.4.4: Couldn't load target `fail2ban- proftpd':/lib/xtables/libipt_fail2ban-proftpd.so: cannot open shared object file: No such file or directory Patch from sf explodes actionstart by newline and launches each line separately (maybe with some locking? I can't read python) With regards Libor Dne úterý 03 Listopad 2009 16:00:30 Yaroslav Halchenko napsal(a): brr... are you sure you just have some problem with iptables? what if whenever there is no fail2ban running you run those commands while being root iptables -N fail2ban-proftpd iptables -A fail2ban-proftpd -j RETURN iptables -I INPUT -p tcp -m multiport --dports ftp,ftp-data,ftps,ftps-data -j fail2ban-proftpd will you get at some point that iptables v1.4.4: Couldn't load target `fail2ban-proftpd':/lib/xtables/libipt_fail2ban-proftpd.so: cannot open shared object file: No such file or directory or would be there any other error reported? On Tue, 03 Nov 2009, Libor Klepac wrote: Package: fail2ban Version: 0.8.4-1 Severity: important Hi, I have problems with using several jails. It was there before, but didn't hit so hard as today, when i was unable to restart fail2ban cleanly , i was trying it for maybe 50 times (changing configs, installing python2.4 ...etc). I seems to fail to execute iptables in correct orderd leading to something like this in its output signature.asc Description: This is a digitally signed message part.
Bug#554162: fail2ban: sometimes(frequently) fails to load iptable rules with multiple jails
Patch is working, but i can't use action_mw
(output goes like this
---
2009-11-03 21:04:02,138 fail2ban.actions.action: ERROR printf %b Subject:
[Fail2Ban] cyrus: started
From: Fail2Ban fail2ban
well -- that is what the patch you've applied accomplished
unfortunately :-/ since the mailing command is a multiline printf
command. imho instead of that patch I would simple tune up the action
for banning -- just add sleep 1 (or sleep 0.1 if system has that recent
sleep ;)) after each command and see how that helps
or may be alternatively just placing all commands on 1 line with ;
between them -- so they should start as a one command (you said that
system tolerated that fine)
All commands in one line work ok from command line.
interesting -- thanks for checking
Sequence is also ok. Fail2ban launches actions in threads?
to say the truth -- I don't know exactly... from what I know, it just
calls system() from stdlib, which should call /bin/sh -c '.'
I still think that the issue is deeper underground (iptables return
before actually completing modification of the table etc) although
myself could not replicate it on none of boxes (biefie enough: 8 cores,
64GB RAM etc) with smth like
for f in {1..100}; do /bin/sh -c iptables -N fail2ban-proftpd; iptables -A
fail2ban-proftpd -j RETURN; iptables -I INPUT -p tcp -m multiport --dports
ftp,ftp-data,ftps,ftps-data -j fail2ban-proftpd; iptables -D INPUT -p tcp -m
multiport --dports ftp,ftp-data,ftps,ftps-data -j fail2ban-proftpd; iptables -F
fail2ban-proftpd; iptables -X fail2ban-proftpd; done
now the question -- what is your /bin/sh? ;)
could you try different kernel? I've tried on 2.6.26-2-amd64
--
.-.
=-- /v\ =
Keep in touch// \\ (yoh@|www.)onerussian.com
Yaroslav Halchenko /( )\ ICQ#: 60653192
Linux User^^-^^[17]
--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#554162: fail2ban: sometimes(frequently) fails to load iptable rules with multiple jails
Package: fail2ban Version: 0.8.4-1 Severity: important Hi, I have problems with using several jails. It was there before, but didn't hit so hard as today, when i was unable to restart fail2ban cleanly , i was trying it for maybe 50 times (changing configs, installing python2.4 ...etc). I seems to fail to execute iptables in correct orderd leading to something like this in its output - iptables: Resource temporarily unavailable. iptables: No chain/target/match by that name. iptables v1.4.4: Couldn't load target `fail2ban-proftpd':/lib/xtables/libipt_fail2ban-proftpd.so: cannot open shared object file: No such file or directory Try `iptables -h' or 'iptables --help' for more information. 2009-11-03 14:29:14,591 fail2ban.actions.action: ERROR iptables -N fail2ban-proftpd iptables -A fail2ban-proftpd -j RETURN iptables -I INPUT -p tcp -m multiport --dports ftp,ftp-data,ftps,ftps-data -j fail2ban-proftpd returned 200 - Seems like last rule is executed when chain is not ready This problem is reported upstream, i think http://sourceforge.net/tracker/?func=detailaid=2870788group_id=121032atid=689044 With patch http://sourceforge.net/tracker/?func=detailaid=2857096group_id=121032atid=689046 Patch applies fine and seems to help, but I have to use action = %(action_)s instead of action_m action_m = %(banaction)s[name=%(__name__)s, port=%(port)s, protocol=%(protocol)s] %(mta)s[name=%(__name__)s, dest=%(destemail)s, protocol=%(protocol)s] action = %(action_m)s or provided action = %(action_mw)s With regards Libor -- System Information: Debian Release: squeeze/sid APT prefers testing APT policy: (500, 'testing') Architecture: amd64 (x86_64) Kernel: Linux 2.6.29-1-amd64 (SMP w/8 CPU cores) Locale: LANG=en_US.UTF8, LC_CTYPE=en_US.UTF8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/bash Versions of packages fail2ban depends on: ii lsb-base 3.2-20 Linux Standard Base 3.2 init scrip ii python2.5.4-2An interactive high-level object-o ii python-central0.6.11 register and build utility for Pyt Versions of packages fail2ban recommends: ii iptables 1.4.4-2administration tools for packet fi ii whois 4.7.32 an intelligent whois client Versions of packages fail2ban suggests: ii bsd-mailx [mailx] 8.1.2-0.20081101cvs-2 A simple mail user agent pn python-gamin none(no description available) -- no debconf information -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#554162: fail2ban: sometimes(frequently) fails to load iptable rules with multiple jails
brr... are you sure you just have some problem with iptables? what if whenever there is no fail2ban running you run those commands while being root iptables -N fail2ban-proftpd iptables -A fail2ban-proftpd -j RETURN iptables -I INPUT -p tcp -m multiport --dports ftp,ftp-data,ftps,ftps-data -j fail2ban-proftpd will you get at some point that iptables v1.4.4: Couldn't load target `fail2ban-proftpd':/lib/xtables/libipt_fail2ban-proftpd.so: cannot open shared object file: No such file or directory or would be there any other error reported? On Tue, 03 Nov 2009, Libor Klepac wrote: Package: fail2ban Version: 0.8.4-1 Severity: important Hi, I have problems with using several jails. It was there before, but didn't hit so hard as today, when i was unable to restart fail2ban cleanly , i was trying it for maybe 50 times (changing configs, installing python2.4 ...etc). I seems to fail to execute iptables in correct orderd leading to something like this in its output -- .-. =-- /v\ = Keep in touch// \\ (yoh@|www.)onerussian.com Yaroslav Halchenko /( )\ ICQ#: 60653192 Linux User^^-^^[17] -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#554162: fail2ban: sometimes(frequently) fails to load iptable rules with multiple jails
Hi, i usually write little confusing ;) Patch is working, but i can't use action_mw (output goes like this --- 2009-11-03 21:04:02,138 fail2ban.actions.action: ERROR printf %b Subject: [Fail2Ban] cyrus: started From: Fail2Ban fail2ban To: fail2...@xxx\n Hi,\n The jail cyrus has been started successfully.\n Regards,\n Fail2Ban | /usr/sbin/sendmail -f fail2ban fail2...@xxx FAILED /bin/sh: -c: line 0: unexpected EOF while looking for matching `' /bin/sh: -c: line 1: syntax error: unexpected end of file 2009-11-03 21:04:02,142 fail2ban.actions.action: ERROR printf %b Hi,\n returned 200 /bin/sh: The: command not found 2009-11-03 21:04:02,145 fail2ban.actions.action: ERROR The jail http has been started successfully.\n returned 7f00 /bin/sh: Regards,n: command not found 2009-11-03 21:04:02,148 fail2ban.actions.action: ERROR Regards,\n returned 7f00 /bin/sh: -c: line 0: unexpected EOF while looking for matching `' /bin/sh: -c: line 1: syntax error: unexpected end of file 2009-11-03 21:04:02,151 fail2ban.actions.action: ERROR Fail2Ban|mail -s [Fail2Ban] http: started fail2...@xxx returned 200 2009-11-03 21:04:03,152 fail2ban.actions.action: ERROR printf %b Hi,\n The jail http has been started successfully.\n Regards,\n Fail2Ban|mail -s [Fail2Ban] http: started fail2...@xxx FAILED --- All commands in one line work ok from command line. I tried little python script -- import os realCmd = echo 1\necho 2\necho 3\necho 11\necho 12\necho 13\necho 21\necho 22\necho 23\necho 31\necho 32\necho 33 retcode = os.system(realCmd) -- Sequence is also ok. Fail2ban launches actions in threads? Libor Dne úterý 03 Listopad 2009 20:14:07 Yaroslav Halchenko napsal(a): d'oh me -- I've read your original incorrectly (that you applied patch and it didn't help ;-)) . Just out of research -- could you please run in shell all 3 commands in a single line like iptables -N fail2ban-proftpd; iptables -A fail2ban-proftpd -j RETURN; iptables -I INPUT -p tcp -m multiport --dports ftp,ftp-data,ftps,ftps-data -j fail2ban-proftpd I would really not expect some out of order execution from Python's os.system and it might be that real problem is buried deeper. On Tue, 03 Nov 2009, Libor Klepáč wrote: Hi, there seems to be some problem with sequence of commands run from python. Maybe our server is too powerfull? (poweredge 2950 with two 4 core processors, old server was some old pentium4 with ht, there were no such problems) If i run those commands separately from command line ,there is no problem. But if whole (multiline) actionstart is run, there seems to be problem with order of commands. Problem is, that iptables -I INPUT -p tcp -m multiport --dports ftp,ftp-data,ftps,ftps-data -j fail2ban-proftpd is run in time, when there is no such chain, so kernel tries to autoload module for it (but there is no such module) (you can try it by running this command) signature.asc Description: This is a digitally signed message part.
Bug#554162: fail2ban: sometimes(frequently) fails to load iptable rules with multiple jails
d'oh me -- I've read your original incorrectly (that you applied patch and it didn't help ;-)) . Just out of research -- could you please run in shell all 3 commands in a single line like iptables -N fail2ban-proftpd; iptables -A fail2ban-proftpd -j RETURN; iptables -I INPUT -p tcp -m multiport --dports ftp,ftp-data,ftps,ftps-data -j fail2ban-proftpd I would really not expect some out of order execution from Python's os.system and it might be that real problem is buried deeper. On Tue, 03 Nov 2009, Libor Klepáč wrote: Hi, there seems to be some problem with sequence of commands run from python. Maybe our server is too powerfull? (poweredge 2950 with two 4 core processors, old server was some old pentium4 with ht, there were no such problems) If i run those commands separately from command line ,there is no problem. But if whole (multiline) actionstart is run, there seems to be problem with order of commands. Problem is, that iptables -I INPUT -p tcp -m multiport --dports ftp,ftp-data,ftps,ftps-data -j fail2ban-proftpd is run in time, when there is no such chain, so kernel tries to autoload module for it (but there is no such module) (you can try it by running this command) -- .-. =-- /v\ = Keep in touch// \\ (yoh@|www.)onerussian.com Yaroslav Halchenko /( )\ ICQ#: 60653192 Linux User^^-^^[17] -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
