NGINX BOTCHECK

Debian 9.2 

$ dpkg -l fail2ban
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                     Version           Architecture      Description
+++-========================-=================-=================-=====================================================
ii  fail2ban                 0.9.6-2           all               ban hosts that 
cause multiple authentication errors


-----------------------------------------------------
$ cat /etc/fail2ban/filter.d/nginx-botsearch.conf |grep -v ^#

[INCLUDES]

before = botsearch-common.conf

[Definition]

failregex = ^<HOST> \- \S+ \[\] \"(GET|POST|HEAD) \/<block> \S+\" 404 .+$
            ^ \[error\] \d+#\d+: \*\d+ (\S+ )?\"\S+\" (failed|is not found) 
\(2\: No such file or directory\), client\: <HOST>\, server\: \S*\, request: 
\"(GET|POST|HEAD) \/<block> \S+\"\, .*?$

ignoreregex = 

——————————————————————————

Sample log lines:
# grep bot www.example.co.uk_access.log
66.249.75.148 - - [13/Mar/2018:12:33:58 +0000] 
TLSv1.2/ECDHE-RSA-AES256-GCM-SHA384 "GET /robots.txt HTTP/1.1" 404 162 "-" 
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.75.148 - - [13/Mar/2018:12:33:58 +0000] 
TLSv1.2/ECDHE-RSA-AES256-GCM-SHA384 "GET /ads.txt HTTP/1.1" 404 162 "-" 
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
157.55.39.236 - - [13/Mar/2018:15:04:19 +0000] 
TLSv1.2/ECDHE-RSA-AES256-GCM-SHA384 "GET / HTTP/1.1" 302 5 "-" "Mozilla/5.0 
(compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"














> On 13 Mar 2018, at 11:07, Tom Hendrikx <t...@whyscream.net> wrote:
> 
> Hi,
> 
> Please keep replies on-list, don't e-mail me privately.
> 
> Can you post:
> - OS version you're running
> - fail2ban version you're running
> - contents of the /etc/fail2ban/filter.d/dovecot.conf file, so we can
> extend the current regex
> 
> For nginx, please create a new thread and supply the same information,
> along with some sample log lines.
> 
> Kind regards,
> 
>       Tom
> 
> 
> On 12-03-18 21:03, Sophie Loewenthal wrote:
>> Hi,  Thanks for the fail2ban-regex checker. I checked nginx and this also 
>> seemed not to work.  Again I have the ciphers listed when they connect.
>> 
>> 
>> 
>> **** NGINX *****
>> # fail2ban-regex mx10.example.co.uk_access.log '^<HOST> \- \S+ \[\] 
>> \"(GET|POST|HEAD) \/<block> \S+\" 404 .+$'
>> Running tests
>> =============
>> Use   failregex line : ^<HOST> \- \S+ \[\] \"(GET|POST|HEAD) \/<block> \S...
>> Use         log file : mx10.example.co.uk_access.log
>> Use         encoding : UTF-8
>> 
>> Results
>> =======
>> Failregex: 0 total
>> Ignoreregex: 0 total
>> Date template hits:
>> |- [# of hits] date format
>> |  [10] Day(?P<_sep>[-/])MON(?P=_sep)Year[ 
>> :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
>> `-
>> 
>> Lines: 10 lines, 0 ignored, 0 matched, 10 missed
>> [processed in 0.00 sec]
>> 
>> |- Missed line(s):
>> |  207.46.13.127 - - [12/Mar/2018:11:52:42 +0000] 
>> TLSv1.2/ECDHE-RSA-AES256-GCM-SHA384 "GET / HTTP/1.1" 302 5 "-" "Mozilla/5.0 
>> (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
>> |  184.105.247.194 - - [12/Mar/2018:14:25:42 +0000] 
>> TLSv1.2/ECDHE-RSA-AES128-GCM-SHA256 "GET / HTTP/1.1" 302 5 "-" "-"
>> |  183.129.160.229 - - [12/Mar/2018:15:21:21 +0000] 
>> TLSv1.2/ECDHE-RSA-AES256-GCM-SHA384 "GET 
>> /farm/libs/modules/tween/tween.min.js HTTP/1.1" 404 162 "-" "Mozilla/5.0 
>> (Macintosh; Intel Mac OS X 10.11; rv:47.0) Gecko/20100101 Firefox/47.0"
>> |  207.46.13.104 - - [12/Mar/2018:15:48:45 +0000] 
>> TLSv1.2/ECDHE-RSA-AES256-GCM-SHA384 "GET / HTTP/1.1" 302 5 "-" "Mozilla/5.0 
>> (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
>> |  207.46.13.127 - - [12/Mar/2018:16:15:41 +0000] 
>> TLSv1.2/ECDHE-RSA-AES256-GCM-SHA384 "GET / HTTP/1.1" 302 5 "-" "Mozilla/5.0 
>> (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
>> |  66.249.75.148 - - [12/Mar/2018:16:37:47 +0000] 
>> TLSv1.2/ECDHE-RSA-AES256-GCM-SHA384 "GET /robots.txt HTTP/1.1" 404 162 "-" 
>> "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
>> |  66.249.75.144 - - [12/Mar/2018:16:37:47 +0000] 
>> TLSv1.2/ECDHE-RSA-AES256-GCM-SHA384 "GET /ads.txt HTTP/1.1" 404 162 "-" 
>> "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
>> |  207.46.13.45 - - [12/Mar/2018:19:01:28 +0000] 
>> TLSv1.2/DHE-RSA-AES256-GCM-SHA384 "GET /robots.txt HTTP/1.1" 404 162 "-" 
>> "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
>> |  207.46.13.45 - - [12/Mar/2018:19:01:29 +0000] 
>> TLSv1.2/DHE-RSA-AES256-GCM-SHA384 "GET /robots.txt HTTP/1.1" 404 162 "-" 
>> "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
>> |  40.77.167.54 - - [12/Mar/2018:19:01:34 +0000] 
>> TLSv1.2/ECDHE-RSA-AES256-GCM-SHA384 "GET / HTTP/1.1" 302 5 "-" "Mozilla/5.0 
>> (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
>> `-
>> 
>> 
>> 
>> 
>> 
>> ***** DOVECOT ******
>> # fail2ban-regex /var/log/mail.log '^%(__prefix_line)s(?:pop3|imap)-login: 
>> (?:Info: )?(?:Aborted login|Disconnected)(?::(?: [^ \(]+)+)? \((?:auth 
>> failed, \d+ attempts( in \d+ secs)?|tried to use (disabled|disallowed) \S+ 
>> auth)\):( user=<[^>]+>,)?( method=\S+,)? rip=<HOST>(?:, lip=\S+)?(?:, TLS(?: 
>> handshaking(?:: SSL_accept\(\) failed: error:[\dA-F]+:SSL 
>> routines:[TLS\d]+_GET_CLIENT_HELLO:unknown protocol)?)?(: Disconnected)?)?(, 
>> session=<\S+>)?\s*$'
>> 
>> Running tests
>> =============
>> Use   failregex line : ^%(__prefix_line)s(?:pop3|imap)-login: (?:Info: )?...
>> Use         log file : /var/log/mail.log
>> Use         encoding : UTF-8
>> 
>> Results
>> =======
>> Failregex: 0 total
>> Ignoreregex: 0 total
>> Date template hits:
>> |- [# of hits] date format
>> |  [3014] (?:DAY )?MON Day 24hour:Minute:Second(?:\.Microseconds)?(?: Year)?
>> `-
>> 
>> Lines: 3014 lines, 0 ignored, 0 matched, 3014 missed
>> [processed in 0.38 sec]
>> Missed line(s): too many to print.  Use --print-all-missed to print all 3014 
>> lines
>> 
>> 
>> 
>> best,
>> Sophie 
>> 
>> 
>> 
>> 
>> 
>>> On 12 Mar 2018, at 10:47, Tom Hendrikx <t...@whyscream.net> wrote:
>>> 
>>> Hi,
>>> 
>>> 
>>> you can test this using the fail2ban-regex tool. When I use one of your
>>> example lines, it doesn't match on my setup (ubuntu 16.04, fail2ban
>>> 0.9.3). The similar logline from own setup doesn match:
>>> 
>>> Feb 19 03:02:33 alison dovecot: imap-login: Disconnected (auth failed, 1
>>> attempts in 7 secs): user=<john...@example.net>, method=PLAIN,
>>> rip=127.0.0.1, lip=127.0.0.1, TLS, session=<e1LxFYdlwKbes9bZ>
>>> 
>>> The latest config file for dovecot in github is completely different
>>> from the one I'm using, but also lacks support for this AFAICS.
>>> 
>>> I guess we could come up with a regex that would support your log lines too.
>>> 
>>> Kind regards,
>>>     Tom
>>> 
>>> On 12-03-18 10:02, Sophie Loewenthal wrote:
>>>> Hi, 
>>>> 
>>>> Sorry for the delay. Flu.
>>>> 
>>>> Will fail2ban act on these example lines below with the extra cipher 
>>>> details?
>>>> 
>>>> I know the lines below would not trigger actions because there are not 
>>>> enough failures in the log. Normally dovecot does not have the TLS/cipher 
>>>> part logged. Will the regexes still matched correctly?
>>>> 
>>>> 
>>>> Mar 11 08:52:04 mx10 dovecot: imap-login: Disconnected (auth failed, 1 
>>>> attempts in 2 secs): hid...@example.co.uk>, method=PLAIN, 
>>>> rip=125.69.11.254, lip=10.1.1.100, TLS: Disconnected, TLSv1 with cipher 
>>>> DHE-RSA-AES256-SHA (256/256 bits)
>>>> Mar 11 10:18:24 mx10 dovecot: imap-login: Disconnected (auth failed, 1 
>>>> attempts in 2 secs): ju...@example.co.uk>, method=PLAIN, rip=37.59.8.29, 
>>>> lip=10.1.1.100, TLS: Disconnected, TLSv1 with cipher ECDHE-RSA-AES256-SHA 
>>>> (256/256 bits)
>>>> Mar 11 11:48:04 mx10 dovecot: imap-login: Disconnected (auth failed, 1 
>>>> attempts in 2 secs): neoc...@example.co.uk>, method=PLAIN, 
>>>> rip=178.216.98.75, lip=10.1.1.100, TLS: Disconnected, TLSv1 with cipher 
>>>> ECDHE-RSA-AES256-SHA (256/256 bits)
>>>> Mar 11 13:37:39 mx10 dovecot: imap-login: Aborted login (auth failed, 4 
>>>> attempts in 26 secs): junk4>, method=PLAIN, rip=71.213.169.18, 
>>>> lip=10.1.1.100, TLS, TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)
>>>> Mar 11 13:37:40 mx10 dovecot: imap-login: Aborted login (auth failed, 4 
>>>> attempts in 26 secs): junk4>, method=PLAIN, rip=187.67.197.100, 
>>>> lip=10.1.1.100, TLS, TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)
>>>> Mar 11 22:35:24 mx10 dovecot: imap-login: Disconnected (auth failed, 1 
>>>> attempts in 2 secs): hid...@example.co.uk>, method=PLAIN, 
>>>> rip=182.100.218.83, lip=10.1.1.100, TLS, TLSv1 with cipher 
>>>> DHE-RSA-AES256-SHA (256/256 bits)
>>>> 
>>>> 
>>>> The jails are enabled in the config. I’ve not see a match for 3 months 
>>>> since I installed the server.
>>>> [dovecot]
>>>> port    = imap,imaps,sieve
>>>> logpath = %(dovecot_log)s
>>>> backend = %(dovecot_backend)s
>>>> 
>>>> [sieve]
>>>> port   = smtp,465,submission
>>>> logpath = %(dovecot_log)s
>>>> backend = %(dovecot_backend)s
>>>> 
>>>> 
>>>> 
>>>> 
>>>>> On 6 Mar 2018, at 10:50, Tom Hendrikx <t...@whyscream.net> wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>> On 06-03-18 08:59, Sophie Loewenthal wrote:
>>>>>> Morning, 
>>>>>> 
>>>>>> My logging from and postfix dovecot is in this format:
>>>>>> 
>>>>>> Mar  6 07:49:45 mx dovecot: imap-login: Login: sop...@example.com>, 
>>>>>> method=PLAIN, rip=94.19.2.3, lip=1.31.1.3, mpid=10655, TLS, TLSv1.2 with 
>>>>>> cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
>>>>>> 
>>>>>> Mar  6 07:55:36 mx postfix/smtpd[10793]: Anonymous TLS connection 
>>>>>> established from unknown[94.19.2.3]: TLSv1.2 with cipher 
>>>>>> ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
>>>>>> 
>>>>>> How can I adapt the filter to pick this up? I don’t think the regex in  
>>>>>> filter.d/postfix.conf|dovecot.conf will pick these changed lines up 
>>>>>> because they have the ciphers included, will they?
>>>>> 
>>>>> Lines that are not understood/matched by fail2ban are ignored.
>>>>> 
>>>>> I don't think these lines signify anything that fail2ban should act on,
>>>>> but please explain what you would like fail2ban to do, based on those
>>>>> log lines?
>>>>> 
>>>>>> 
>>>>>> Best wishes,
>>>>>> 
>>>>>> Sophie 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> ------------------------------------------------------------------------------
>>>>>> Check out the vibrant tech community on one of the world's most
>>>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>>>> _______________________________________________
>>>>>> Fail2ban-users mailing list
>>>>>> Fail2ban-users@lists.sourceforge.net
>>>>>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users
>>>>>> 
>>>>> 
>>>>> ------------------------------------------------------------------------------
>>>>> Check out the vibrant tech community on one of the world's most
>>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>>> _______________________________________________
>>>>> Fail2ban-users mailing list
>>>>> Fail2ban-users@lists.sourceforge.net
>>>>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users
>>>> 
>>>> 
>>>> ------------------------------------------------------------------------------
>>>> Check out the vibrant tech community on one of the world's most
>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>> _______________________________________________
>>>> Fail2ban-users mailing list
>>>> Fail2ban-users@lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users
>>>> 
>>> 
>>> ------------------------------------------------------------------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>> _______________________________________________
>>> Fail2ban-users mailing list
>>> Fail2ban-users@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users
>> 
> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Fail2ban-users mailing list
> Fail2ban-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/fail2ban-users


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Fail2ban-users mailing list
Fail2ban-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fail2ban-users

Reply via email to