Hi again
I understand why you've come to that conclusion, but in all honesty I don't
think f2b is meant to be used that way.
F2b processes only what you tell it to. It assumes everything is harmless
unless you tell it it's harmful. If you've got a harmless log line that
also contains stuff that might get caught in a failregex filter, that's
when you add an ignoreregex. So, if something isn't covered by a
"failregex" or an "ignoreregex" in your conf file, it will be "missed",
which is how it should be.
I'd be interested in others' opinions, but I think that web page is giving
bad advice on adding ignoreregexes.
Basically, we know what the threats are - if it's ssh, basically we've got
that sewn up with our existing failregex. Anything that isn't successfully
authorised is caught by the failregex - the sshd program is pretty much
complete at this point, so every possible failure type is already there in
the code, and f2b's devs have added pretty much all of them to the conf
file. Everything else can be safely "missed". That's how I do it. I don't
even go through my ssh log files, because the f2b devs add the necessary
failregexes (and updates to the program contain any necessary changes).
Your pastebin file looks exactly like mine, but I'd still do it that way -
I assume we can "miss" everything unless there's a reason not to.
Let me give a broader usage example: web servers. Every day, thousands of
lines get added, and each one can be slightly different. A new user agent,
a new search engine; an RSS feed that is asking for specific comments, with
a link that changes every half an hour. Some query strings (the
?user=tony&page=991&colour=blue stuff) are totally safe, some might be
attacks.
If I added all the lines that I know are safe, my ignoreregex will be
massive and complex and it will get more complex every day, because I need
to get more and more specific as new safe/unsafe things come up. To me,
that's an impractical use of f2b, so all I do is look for threats, bots,
scrapers. (Actually, one way I do it is, I've got a filter/jail that traps
anything that hits my server more than a certain number of times in 10
seconds - it informs me so I can decide whether to ban it; it doesn't ban
it itself, the jail simply emails me; it's only partially effective because
many bots now scrape much more slowly in order to avoid detection, so I've
got other things for them). I only add "ignores" if something would
otherwise be caught in a fail filter but I don't want it to
It's my belief that we cannot use f2b the way that web page is advising us
to. We should work on the assumption that everything can be "missed" unless
we explicitly want to do something with it.
I use "ignore" for things that might trip my fail filter but I want to make
an exception for that regex (for example, a filter to catch downloading
bots, but I might want to allow one bot to download unhindered). That's the
most efficient way of using f2b.
With all that in mind, *this is the most likely answer to your question*:
the lines that are being "missed" are being missed cos you haven't set up
ignoreregexes to examine and then ignore them. You've got failregexes that
seem to be working just fine, but not ignoreregexes. Remember, if it's not
explicitly included in a failregex or an ignoreregex, it will be "missed".
Could you show us your sshd.conf file please? There shouldn't be anything
'identifiable' in there, so you're safe to send it here. We can take a look
and see if there *should* be more "ignored" lines according to your conf.
If you've got things in there already, then it might be a matter of editing
them so they "ignore" more. But genuinely I think the results you showed
were correct: almost everything should be "missed" cos we're not interested
in it.
Sorry for the length - I hope it hasn't completely muddied the waters :-)
-tony
Tony Collins
On 16 October 2017 at 20:42, A <publicf...@bak.rr.com> wrote:
> This got sucked into moderator heaven due to the length; I'm sure I went
> over 100 lines. Since it has yet to be approved, I'm resending - modified
> to pass the length test.
>
> According to the best reference I've found so far,
> http://www.the-art-of-web.com/system/fail2ban-filters/ "Missed" means
> there is no regex that 'match/fails' nor is there a regex matching the
> expression that is 'ignored'. Missing.
>
> Based on that link's wording - and I agree with it - nothing should be
> missing, everything should either match/fail or be ignored. By ignoring
> non-essential and/or repetitive messages, it makes --print-all-missed
> actually useful by only printing lines that have not been previously
> examined and either matched or ignored.
>
> How long does it take you to go through 6000+ lines and assess whether
> there is a threat on any given day? If you're not reviewing it, then how
> do you know if there's a threat? What if it's 100,000 lines? How do you
> know if you didn't notice a line you should have noticed if it's buried
> within thousands of lines that you could, and should have safely ignored
> with a "simple" regex? I don't understand why you would not want to use
> your computer to filter out 5999 of them to find the single line that's
> actually important for you to review carefully - and then either ignore or
> fail.
>
> As to the specifics of the missing messages, I hope you don't mind, but
> having thought about it a bit more, I prefer not to post the info in a
> public forum for security reasons and I hope the moderator rejects my
> previous message. Instead please go to https://pastebin.com/ibJjn49a the
> link is good for a week, which limits my exposure somewhat.
>
> Thanks again for the assist!
>
> - A
>
>
>
> On 10/15/2017 10:33 PM, Tony Collins wrote:
>
> Hi. OK, can I ask: why did you expect every line to fail ('match')? Have
> you been through the log file and noticed that every log entry is an
> attack? I am asking because a normal log file would contain a mixture of
> missed, matched and ignored lines.
>
> In my current log file, if I run the same command, I get this:
>
> *Lines: 9622 lines, 0 ignored, 3585 matched, 6037 missed*
>
> You can see there's a "healthy" mix of matched and missed, which is
> exactly what a normal log file should show.
>
> If you have checked and you believe that fail2ban should've 'failed' the
> other 657 lines, we need to see the "missed" lines.
>
> To do that, please run the fail2ban-regex command again but add this to
> the then: --print-all-missed
>
> Then copy 100 of them into your reply so we can examine them.
>
>
>
> Tony Collins
>
> On 16 October 2017 at 04:16, A <publicf...@bak.rr.com> wrote:
>
>> Thank you for the assist. The issue is that 657 lines were missed.
>> Lines: 682 lines, 0 ignored, 25 matched, 657 missed [processed in 0.45
>> sec]
>>
>>
>> On 10/15/2017 01:41 PM, Tony Collins wrote:
>>
>> Is it just me? I can't tell what the issue is!
>>
>>
>> On Sun, 15 Oct 2017 at 21:02, A <publicf...@bak.rr.com> wrote:
>>
>>> I can't be the first to encounter this... does anyone have a fix for
>>> the below please?
>>>
>>> Thank you in advance!
>>>
>>> - Andrew
>>>
>>> # fail2ban-regex /var/log/auth.log.1 /etc/fail2ban/filter.d/sshd.
>>> conf
>>>
>>> Running tests
>>> =============
>>>
>>> Use failregex filter file : sshd, basedir: /etc/fail2ban
>>> Use maxlines : 10
>>> Use log file : /var/log/auth.log.1
>>> Use encoding : UTF-8
>>>
>>>
>>> Results
>>> =======
>>>
>>> Failregex: 25 total
>>> |- #) [# of hits] regular expression
>>> | 3) [10] ^\s*(<[^.]+\.[^.]+>)?\s*(?:\S+ )?(?:kernel: \[ *\d+\.\d+\]
>>> )?(?:@vserver_\S+ )?(?:(?:\[\d+\])?:\s+[\[\(]?ss
>>> hd(?:\(\S+\))?[\]\)]?:?|[\[\(]?sshd(?:\(\S+\))?[\]\)]?:?(?:\[\d+\])?:?)?\s(?:\[ID
>>> \d+ \S+\])?\s*Failed \S+ for .*? from <HOST>(?: port \d*)?(?: ssh\d*)?(:
>>> (ruser .*|(\S+ ID \S+ \(serial \d+\) CA )?\S+
>>> (?:[\da-f]{2}:){15}[\da-f]{2}(,
>>> client user ".*", client host ".*")?))?\s*$
>>> | 5) [5] ^\s*(<[^.]+\.[^.]+>)?\s*(?:\S+ )?(?:kernel: \[ *\d+\.\d+\]
>>> )?(?:@vserver_\S+ )?(?:(?:\[\d+\])?:\s+[\[\(]?ss
>>> hd(?:\(\S+\))?[\]\)]?:?|[\[\(]?sshd(?:\(\S+\))?[\]\)]?:?(?:\[\d+\])?:?)?\s(?:\[ID
>>> \d+ \S+\])?\s*[iI](?:llegal|nvalid) user .* from <HOST>\s*$
>>> | 16) [10] ^\s*(<[^.]+\.[^.]+>)?\s*(?:\S+ )?(?:kernel: \[ *\d+\.\d+\]
>>> )?(?:@vserver_\S+ )?(?:(?:\[\d+\])?:\s+[\[\(]?ss
>>> hd(?:\(\S+\))?[\]\)]?:?|[\[\(]?sshd(?:\(\S+\))?[\]\)]?:?(?:\[\d+\])?:?)?\s(?:\[ID
>>> \d+ \S+\])?\s*pam_unix\(sshd:auth\):\s+authentication
>>> failure;\s*logname=\S*\s*uid=\d*\s*euid=\d*\s*tty=\S*\s*ruse
>>> r=\S*\s*rhost=<HOST>\s.*$
>>> `-
>>>
>>> Ignoreregex: 0 total
>>>
>>> Date template hits:
>>> |- [# of hits] date format
>>> | [682] (?:DAY )?MON Day 24hour:Minute:Second(?:\.Microseconds)?(?:
>>> Year)?
>>> `-
>>>
>>> Lines: 682 lines, 0 ignored, 25 matched, 657 missed [processed in 0.45
>>> sec]
>>> Missed line(s): too many to print. Use --print-all-missed to print all
>>> 657 lines
>>> ------------------------------------------------------------
>>> ------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot______
>>> _________________________________________
>>> Fail2ban-users mailing list
>>> Fail2ban-users@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users
>>>
>> --
>> -- Tony Collins
>>
>>
>>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
>
> _______________________________________________
> Fail2ban-users mailing
> listFail2ban-users@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/fail2ban-users
>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Fail2ban-users mailing list
> Fail2ban-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/fail2ban-users
>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Fail2ban-users mailing list
Fail2ban-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fail2ban-users
