Sorry to resurrect an old thread, but I finally got my local decoders to work with Nginx logs when they are being sent to syslog rather than /var/log/nginx.
Here's what worked in order to trigger the standard web rules from web_rules.xml . Note that other agents that have 'regular' Nginx logs in /var/log/nginx also still work (they are parsed by the core OSSEC web-accesslog decoder). <decoder name="nginx-syslog"> <program_name>^nginx</program_name> <type>web-log</type> </decoder> <decoder name="nginx-syslog_2"> <parent>nginx-syslog</parent> <type>web-log</type> <regex offset="after_parent">(\d*.\d*.\d*.\d*)</regex> <order>srcip</order> </decoder> <decoder name="nginx-syslog_2"> <parent>nginx-syslog</parent> <type>web-log</type> <regex offset="after_parent">(\d*/\w*/\d*)</regex> <order>date</order> </decoder> <decoder name="nginx-syslog_2"> <parent>nginx-syslog</parent> <type>web-log</type> <regex offset="after_parent">:(\d*:\d*:\d*)</regex> <order>time</order> </decoder> <decoder name="nginx-syslog_2"> <parent>nginx-syslog</parent> <type>web-log</type> <!-- +0000] "GET /foobar HTTP/1.1" 404 --> <regex offset="after_parent">\s+\S+\s+"(\S+)\s+(\S+)\s+\S+"\s+(\d+)</regex> <order>action, url, id</order> </decoder> I am sure it could be simplified, but for now I'm happy as it actually trips the rules as desired. On Wednesday, June 30, 2021 at 7:59:59 AM UTC+10 Miguel Jacq wrote: > Hi Yana, > > Thank you for the reply. > > It's not really a 'fresh' installation. I did install it using the system > PCRE2, with `PCRE2_SYSTEM=yes ./install.sh` > > I don't think the issue is with PCRE as such but the fact that the nginx > logs are arriving in the syslog, and therefore the decoder regex somehow > needs tweaking since the format of the log is not this traditional Nginx > log: > > 11.22.33.44 - - [29/Jun/2021:21:55:30 +0000] "GET / HTTP/1.1" 200 467 "-" > "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) > Chrome/98 Safari/537.4" > > > ... but instead because it's coming into rsyslog, it looks like this: > > Jun 29 21:55:30 example.com nginx: 11.22.33.44 - - [29/Jun/2021:21:55:30 > +0000] "GET / HTTP/1.1" 200 467 "-" "Mozilla/5.0 (Windows NT 6.2; WOW64) > AppleWebKit/537.4 (KHTML, like Gecko) Chrome/98 Safari/537.4" > > The web-accesslog decoder looks like this, I wonder if it needs overriding > somehow to account for the extra fields and maybe that's why the PCRE > matches aren't able to 'detect' the status code in the usual way? > > <decoder name="web-accesslog"> > <type>web-log</type> > <prematch_pcre2>^\S+ \S+ \S+ \[\S+ \S\d+\] "\w+ \S+ HTTP\S+" > </prematch_pcre2> > <pcre2>^(\S+) \S+ (\S+) \[\S+ \S\d+\] </pcre2> > <pcre2>"(\w+) (\S+) HTTP\S+" (\d+) </pcre2> > <order>srcip, srcuser, action, url, id</order> > </decoder> > > I'll see if hopefully anyone else has any tips. > > Thanks again > > On Tuesday, June 29, 2021 at 8:26:06 PM UTC+10 [email protected] wrote: > >> Hi, >> >> My apologies for the late response. Is your installation a fresh >> installation? It seems that from version 3.4, you must have the pcre2-10.32 >> sources installed in *src/external. *You can obtain them by running: >> >> *cd ossec-hids-* * >> *wget https://ftp.pcre.org/pub/pcre/pcre2-10.32.tar.gz >> <https://ftp.pcre.org/pub/pcre/pcre2-10.32.tar.gz> * >> *tar xzf pcre2-10.32.tar.gz -C src/external* >> >> After this, if you decide to use the pcre2-10.32 sources, you must set >> the PCRE"_SYSTEM variable to *no:* >> >> *cd ossec-hids-* * >> *PCRE2_SYSTEM=no ./install.sh* >> >> If you decide to use the system's PCRE2, set the PCRE2_SYSTEM variable to >> *yes:* >> >> *cd ossec-hids-** >> *PCRE2_SYSTEM=yes ./install.sh* >> >> You can check more information about this here >> <http://www.ossec.net/docs/docs/manual/installation/installation-requirements.html> >> . >> >> Hope I was helpful. Let me know if you need anything else. >> >> Regards, >> Yana. >> >> On Tuesday, June 22, 2021 at 12:00:49 AM UTC+2 [email protected] wrote: >> >>> Thanks Yana, >>> >>> With the original 'id_pcre2' in rules 31120 and 31122, and my custom >>> decoder per the original post, I get this: >>> >>> ossec-testrule: Type one log per line. >>> >>> Jun 21 12:35:37 example.com nginx: 22.33.44.55 - - >>> [21/Jun/2021:12:35:37 +0000] "GET /something?bad HTTP/1.1" 500 10372 " >>> https://something.com"; "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 >>> (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.5083.400 >>> QQBrowser/10.0.972.400" >>> >>> >>> **Phase 1: Completed pre-decoding. >>> full event: 'Jun 21 12:35:37 example.com nginx: 22.33.44.55 - - >>> [21/Jun/2021:12:35:37 +0000] "GET /something?bad HTTP/1.1" 500 10372 " >>> https://something.com"; "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 >>> (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.5083.400 >>> QQBrowser/10.0.972.400"' >>> hostname: 'example.com' >>> program_name: 'nginx' >>> log: '22.33.44.55 - - [21/Jun/2021:12:35:37 +0000] "GET >>> /something?bad HTTP/1.1" 500 10372 "https://something.com"; "Mozilla/5.0 >>> (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.26 >>> Safari/537.36 Core/1.63.5083.400 QQBrowser/10.0.972.400"' >>> >>> **Phase 2: Completed decoding. >>> decoder: 'web-accesslog' >>> >>> **Phase 3: Completed filtering (rules). >>> Rule id: '31100' >>> Level: '0' >>> Description: 'Access log messages grouped.' >>> >>> >>> >>> If I change the <id_prce2> to <match> and remove the ^ in the 50/500 >>> match string, for rules 31120 and 31122, I get this: >>> >>> ossec-testrule: Type one log per line. >>> >>> Jun 21 12:35:37 example.com nginx: 22.33.44.55 - - >>> [21/Jun/2021:12:35:37 +0000] "GET /something?bad HTTP/1.1" 500 10372 " >>> https://something.com"; "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 >>> (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.5083.400 >>> QQBrowser/10.0.972.400" >>> >>> >>> **Phase 1: Completed pre-decoding. >>> full event: 'Jun 21 12:35:37 example.com nginx: 22.33.44.55 - - >>> [21/Jun/2021:12:35:37 +0000] "GET /something?bad HTTP/1.1" 500 10372 " >>> https://something.com"; "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 >>> (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.5083.400 >>> QQBrowser/10.0.972.400"' >>> hostname: 'example.com' >>> program_name: 'nginx' >>> log: '22.33.44.55 - - [21/Jun/2021:12:35:37 +0000] "GET >>> /something?bad HTTP/1.1" 500 10372 "https://something.com"; "Mozilla/5.0 >>> (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.26 >>> Safari/537.36 Core/1.63.5083.400 QQBrowser/10.0.972.400"' >>> >>> **Phase 2: Completed decoding. >>> decoder: 'web-accesslog' >>> >>> **Phase 3: Completed filtering (rules). >>> Rule id: '31122' >>> Level: '5' >>> Description: 'Web server 500 error code (Internal Error).' >>> **Alert to be generated. >>> >>> >>> >>> Thanks >>> >>> On Tuesday, June 22, 2021 at 12:55:41 AM UTC+10 [email protected] >>> wrote: >>> >>>> Hi Miguel, >>>> >>>> Could you please paste the output coming from *ossec-logtest* after >>>> pasting these logs? >>>> >>>> Waiting for your reply, >>>> Yana. >>>> >>>> On Monday, June 21, 2021 at 12:29:56 PM UTC+2 [email protected] wrote: >>>> >>>>> Hi, >>>>> >>>>> I am running a system whereby Nginx traffic logs are being sent from a >>>>> Docker container to a remote syslog server, where they arrive in that >>>>> remote syslog server's /var/log/syslog. This remote server is also the >>>>> one >>>>> running OSSEC. >>>>> >>>>> As a result, the Nginx logs look like this in the syslog - note the ' >>>>> example.com' is effectively the 'program_name' which is the >>>>> identifier of the container itself. >>>>> >>>>> Jun 20 15:52:09 example.com nginx: 11.22.33.44 - - >>>>> [20/Jun/2021:15:52:09 +0000] "GET /something/ HTTP/1.1" 500 7910 " >>>>> https://example.com/"; "Mozilla/5.0 (Windows NT 5.1; rv:52.0) >>>>> Gecko/20100101 Firefox/52.0" >>>>> >>>>> My problem is that the OSSEC rules are not recognising the Nginx logs, >>>>> because they are in the syslog. >>>>> >>>>> To 'half' solve that, I added this custom decoder which I borrowed >>>>> from https://github.com/wazuh/wazuh/issues/352: >>>>> >>>>> <decoder name="web-accesslog"> >>>>> <type>web-log</type> >>>>> <program_name>nginx|apache</program_name> >>>>> </decoder> >>>>> >>>>> Now, this is good because the above example log message will now >>>>> appear as rule 31101 'Access log messages grouped'. Progress! >>>>> >>>>> However, note that the log message was a 500 internal server error. It >>>>> seems that despite landing in 31101 thanks to the custom decoder, the >>>>> other >>>>> 'child' rules in web_rules.xml are not applying, e.g 31122: >>>>> >>>>> <rule id="31122" level="5"> >>>>> <if_sid>31120</if_sid> >>>>> <id_pcre2>^500</id_pcre2> >>>>> <options>alert_by_email</options> >>>>> <description>Web server 500 error code (Internal >>>>> Error).</description> >>>>> <group>system_error,</group> >>>>> </rule> >>>>> >>>>> It doesn't seem to hit this error, it just stays as 31101 according to >>>>> ossec-logtest. >>>>> >>>>> I am assuming it's the id_pcre2 not picking up the '500' because of >>>>> the extra fields when it's from syslog? As a guess? >>>>> >>>>> If I change both rule 31120 and rule 31122 to use <match>50</match> >>>>> and <match>500</match> respectively, then it works, and rule 31122 fires >>>>> for the above. But not if it uses id_pcre2 *or* if it uses ^ at the start >>>>> of the match - both make it skip. >>>>> >>>>> I'm not so great at regexes - so I would really appreciate any help to >>>>> get the standard web rules detecting the above Nginx log message when >>>>> it's >>>>> coming as a 'syslog' message. >>>>> >>>>> I am running OSSEC 3.6.0 on Ubuntu 18.04. >>>>> >>>>> Thanks! >>>>> >>>> -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/ossec-list/9f41f196-191b-41ef-80cc-26dba086795bn%40googlegroups.com.
