The error in that article is that he sets up the false need to "filter the
noise". Fail2ban does this by default. IMO, the article's approach adds
unnecessary server load.
\\\\Greg
________________________________
From: Tony Collins <t...@evilplan.org.uk>
Sent: Tuesday, October 17, 2017 2:03:07 AM
To: fail2ban-users@lists.sourceforge.net
Subject: Re: [Fail2ban-users] ubuntu 16.04 / Lines: 682 lines, 0 ignored, 25
matched, 657 missed
Hi again
I understand why you've come to that conclusion, but in all honesty I don't
think f2b is meant to be used that way.
F2b processes only what you tell it to. It assumes everything is harmless
unless you tell it it's harmful. If you've got a harmless log line that also
contains stuff that might get caught in a failregex filter, that's when you add
an ignoreregex. So, if something isn't covered by a "failregex" or an
"ignoreregex" in your conf file, it will be "missed", which is how it should be.
I'd be interested in others' opinions, but I think that web page is giving bad
advice on adding ignoreregexes.
Basically, we know what the threats are - if it's ssh, basically we've got that
sewn up with our existing failregex. Anything that isn't successfully
authorised is caught by the failregex - the sshd program is pretty much
complete at this point, so every possible failure type is already there in the
code, and f2b's devs have added pretty much all of them to the conf file.
Everything else can be safely "missed". That's how I do it. I don't even go
through my ssh log files, because the f2b devs add the necessary failregexes
(and updates to the program contain any necessary changes). Your pastebin file
looks exactly like mine, but I'd still do it that way - I assume we can "miss"
everything unless there's a reason not to.
Let me give a broader usage example: web servers. Every day, thousands of lines
get added, and each one can be slightly different. A new user agent, a new
search engine; an RSS feed that is asking for specific comments, with a link
that changes every half an hour. Some query strings (the
?user=tony&page=991&colour=blue stuff) are totally safe, some might be attacks.
If I added all the lines that I know are safe, my ignoreregex will be massive
and complex and it will get more complex every day, because I need to get more
and more specific as new safe/unsafe things come up. To me, that's an
impractical use of f2b, so all I do is look for threats, bots, scrapers.
(Actually, one way I do it is, I've got a filter/jail that traps anything that
hits my server more than a certain number of times in 10 seconds - it informs
me so I can decide whether to ban it; it doesn't ban it itself, the jail simply
emails me; it's only partially effective because many bots now scrape much more
slowly in order to avoid detection, so I've got other things for them). I only
add "ignores" if something would otherwise be caught in a fail filter but I
don't want it to
It's my belief that we cannot use f2b the way that web page is advising us to.
We should work on the assumption that everything can be "missed" unless we
explicitly want to do something with it.
I use "ignore" for things that might trip my fail filter but I want to make an
exception for that regex (for example, a filter to catch downloading bots, but
I might want to allow one bot to download unhindered). That's the most
efficient way of using f2b.
With all that in mind, this is the most likely answer to your question: the
lines that are being "missed" are being missed cos you haven't set up
ignoreregexes to examine and then ignore them. You've got failregexes that seem
to be working just fine, but not ignoreregexes. Remember, if it's not
explicitly included in a failregex or an ignoreregex, it will be "missed".
Could you show us your sshd.conf file please? There shouldn't be anything
'identifiable' in there, so you're safe to send it here. We can take a look and
see if there *should* be more "ignored" lines according to your conf. If you've
got things in there already, then it might be a matter of editing them so they
"ignore" more. But genuinely I think the results you showed were correct:
almost everything should be "missed" cos we're not interested in it.
Sorry for the length - I hope it hasn't completely muddied the waters :-)
-tony
Tony Collins
On 16 October 2017 at 20:42, A
<publicf...@bak.rr.com<mailto:publicf...@bak.rr.com>> wrote:
This got sucked into moderator heaven due to the length; I'm sure I went over
100 lines. Since it has yet to be approved, I'm resending - modified to pass
the length test.
According to the best reference I've found so far,
http://www.the-art-of-web.com/system/fail2ban-filters/ "Missed" means there is
no regex that 'match/fails' nor is there a regex matching the expression that
is 'ignored'. Missing.
Based on that link's wording - and I agree with it - nothing should be missing,
everything should either match/fail or be ignored. By ignoring non-essential
and/or repetitive messages, it makes --print-all-missed actually useful by only
printing lines that have not been previously examined and either matched or
ignored.
How long does it take you to go through 6000+ lines and assess whether there is
a threat on any given day? If you're not reviewing it, then how do you know if
there's a threat? What if it's 100,000 lines? How do you know if you didn't
notice a line you should have noticed if it's buried within thousands of lines
that you could, and should have safely ignored with a "simple" regex? I don't
understand why you would not want to use your computer to filter out 5999 of
them to find the single line that's actually important for you to review
carefully - and then either ignore or fail.
As to the specifics of the missing messages, I hope you don't mind, but having
thought about it a bit more, I prefer not to post the info in a public forum
for security reasons and I hope the moderator rejects my previous message.
Instead please go to https://pastebin.com/ibJjn49a the link is good for a
week, which limits my exposure somewhat.
Thanks again for the assist!
- A
On 10/15/2017 10:33 PM, Tony Collins wrote:
Hi. OK, can I ask: why did you expect every line to fail ('match')? Have you
been through the log file and noticed that every log entry is an attack? I am
asking because a normal log file would contain a mixture of missed, matched and
ignored lines.
In my current log file, if I run the same command, I get this:
Lines: 9622 lines, 0 ignored, 3585 matched, 6037 missed
You can see there's a "healthy" mix of matched and missed, which is exactly
what a normal log file should show.
If you have checked and you believe that fail2ban should've 'failed' the other
657 lines, we need to see the "missed" lines.
To do that, please run the fail2ban-regex command again but add this to the
then: --print-all-missed
Then copy 100 of them into your reply so we can examine them.
Tony Collins
On 16 October 2017 at 04:16, A
<publicf...@bak.rr.com<mailto:publicf...@bak.rr.com>> wrote:
Thank you for the assist. The issue is that 657 lines were missed.
Lines: 682 lines, 0 ignored, 25 matched, 657 missed [processed in 0.45 sec]
On 10/15/2017 01:41 PM, Tony Collins wrote:
Is it just me? I can't tell what the issue is!
On Sun, 15 Oct 2017 at 21:02, A
<publicf...@bak.rr.com<mailto:publicf...@bak.rr.com>> wrote:
I can't be the first to encounter this... does anyone have a fix for the below
please?
Thank you in advance!
- Andrew
# fail2ban-regex /var/log/auth.log.1 /etc/fail2ban/filter.d/sshd.
conf
Running tests
=============
Use failregex filter file : sshd, basedir: /etc/fail2ban
Use maxlines : 10
Use log file : /var/log/auth.log.1
Use encoding : UTF-8
Results
=======
Failregex: 25 total
|- #) [# of hits] regular expression
| 3) [10] ^\s*(<[^.]+\.[^.]+>)?\s*(?:\S+ )?(?:kernel: \[ *\d+\.\d+\]
)?(?:@vserver_\S+
)?(?:(?:\[\d+\])?:\s+[\[\(]?sshd(?:\(\S+\))?[\]\)]?:?|[\[\(]?sshd(?:\(\S+\))?[\]\)]?:?(?:\[\d+\])?:?)?\s(?:\[ID
\d+ \S+\])?\s*Failed \S+ for .*? from <HOST>(?: port \d*)?(?: ssh\d*)?(:
(ruser .*|(\S+ ID \S+ \(serial \d+\) CA )?\S+ (?:[\da-f]{2}:){15}[\da-f]{2}(,
client user ".*", client host ".*")?))?\s*$
| 5) [5] ^\s*(<[^.]+\.[^.]+>)?\s*(?:\S+ )?(?:kernel: \[ *\d+\.\d+\]
)?(?:@vserver_\S+
)?(?:(?:\[\d+\])?:\s+[\[\(]?sshd(?:\(\S+\))?[\]\)]?:?|[\[\(]?sshd(?:\(\S+\))?[\]\)]?:?(?:\[\d+\])?:?)?\s(?:\[ID
\d+ \S+\])?\s*[iI](?:llegal|nvalid) user .* from <HOST>\s*$
| 16) [10] ^\s*(<[^.]+\.[^.]+>)?\s*(?:\S+ )?(?:kernel: \[ *\d+\.\d+\]
)?(?:@vserver_\S+
)?(?:(?:\[\d+\])?:\s+[\[\(]?sshd(?:\(\S+\))?[\]\)]?:?|[\[\(]?sshd(?:\(\S+\))?[\]\)]?:?(?:\[\d+\])?:?)?\s(?:\[ID
\d+ \S+\])?\s*pam_unix\(sshd:auth\):\s+authentication
failure;\s*logname=\S*\s*uid=\d*\s*euid=\d*\s*tty=\S*\s*ruser=\S*\s*rhost=<HOST>\s.*$
`-
Ignoreregex: 0 total
Date template hits:
|- [# of hits] date format
| [682] (?:DAY )?MON Day 24hour:Minute:Second(?:\.Microseconds)?(?: Year)?
`-
Lines: 682 lines, 0 ignored, 25 matched, 657 missed [processed in 0.45 sec]
Missed line(s): too many to print. Use --print-all-missed to print all 657
lines
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org!
http://sdm.link/slashdot_______________________________________________
Fail2ban-users mailing list
Fail2ban-users@lists.sourceforge.net<mailto:Fail2ban-users@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/fail2ban-users
--
-- Tony Collins
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Fail2ban-users mailing list
Fail2ban-users@lists.sourceforge.net<mailto:Fail2ban-users@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/fail2ban-users
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Fail2ban-users mailing list
Fail2ban-users@lists.sourceforge.net<mailto:Fail2ban-users@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/fail2ban-users
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Fail2ban-users mailing list
Fail2ban-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fail2ban-users
