Ok, I can reproduce the segfault with a bare minimum. Here is what I did: 1) I spun up a brand new server (clone of my current ossec server), installed a fresh 2.8.1 ossec server. Modified local_rules.xml to look like this:
<!-- @(#) $Id: ./etc/rules/local_rules.xml, 2011/09/08 dcid Exp $ - Example of local rules for OSSEC. - - Copyright (C) 2009 Trend Micro Inc. - All rights reserved. - - This program is a free software; you can redistribute it - and/or modify it under the terms of the GNU General Public - License (version 2) as published by the FSF - Free Software - Foundation. - - License details: http://www.ossec.net/en/licensing.html --> <!-- Modify it at your will. --> <group name="local,syslog,"> <!-- move from level 7 to level 5 6/25/2013 --> <rule id="533" level="5" overwrite="yes"> <if_sid>530</if_sid> <match>ossec: output: 'netstat -tan</match> <check_diff /> <description>Listened ports status (netstat) changed (new port opened or closed).</description> </rule> </group> <!-- SYSLOG,LOCAL --> <!-- EOF --> I did NOT add any agents, but let that run for about an hour. 2) I then stopped the ossec server, ran manage_agents, added one agent and restarted the ossec server. NOTE, I did not transfer the agent key, so the agent was not trying to communicate yet. I let that run for about an hour. 3) I transferred the key to the agent - this is an existing agent, I just added the new key and change the <server-ip> to point to my new test server and restarted the agent and server. I begin to see communication between server and agent. I let it run it's initial scans - about 10 minutes. 4) Then I opened a new listen port to try and trigger the netstat -tan rule using nc -l 7777 on the agent. 5) ossec-analysisd segfaulted after about 7 minues. Let me know if you want any of the output or if you want me to run any other tests. On Friday, December 12, 2014 11:14:47 AM UTC-6, [email protected] wrote: > > I get the segfault with ONLY the posted rule in my local_rules.xml. In > other words, with a completely vanilla install of 2.8.1, by just adding > that rule to local_rules.xml and restarting. Unfortunately, I can't > remember if I needed to add agents before I got the segfault. I'm happy to > spin up a new instance and test it out though. > > On Friday, December 12, 2014 10:52:25 AM UTC-6, dan (ddpbsd) wrote: >> >> On Fri, Dec 12, 2014 at 11:33 AM, <[email protected]> wrote: >> > I'm so sorry. This IS 2.8.1 code. My post says v1.7 and 1.8, but I >> meant v >> > 2.7 and 2.8. >> > >> >> I'm not able to reproduce this, could you provide your entire >> local_rules.xml? >> >> > On Friday, December 12, 2014 9:59:34 AM UTC-6, dan (ddpbsd) wrote: >> >> >> >> On Fri, Dec 12, 2014 at 10:43 AM, <[email protected]> wrote: >> >> > I recently migrated our ossec server to new hardware, and at the >> same >> >> > time >> >> > upgraded from v 1.7 to 1.8. Unfortunately ossec-analysisd kept >> >> > segfaulting >> >> > in the new environment. It's taken me a couple of weeks to narrow it >> >> > down, >> >> > and it seems to be consistently caused by the following rule from my >> >> > local_rules.xml (downgrade of a base rule from level 7 to level 5): >> >> > >> >> > <rule id="533" level="5" overwrite="yes"> >> >> > <if_sid>530</if_sid> >> >> > <match>ossec: output: 'netstat -tan</match> >> >> > <check_diff /> >> >> > <description>Listened ports status (netstat) changed (new port >> >> > opened or >> >> > closed).</description> >> >> > </rule> >> >> > >> >> > This rule has been running under v1.7 in our old environment for >> over a >> >> > year. For now, I've just commented out this rule and all is good, >> but I >> >> > thought it might be helpful to the community if we figure out what's >> >> > going >> >> > on - is it a bug, or some boneheaded misconfiguration on my part. >> I'm >> >> > happy >> >> > to provide more detail if needed. >> >> > >> >> > Current environment is Amazon AWS VPC. All servers are running >> Amazon >> >> > Linux >> >> > 64-bit AMI's (based on RHEL/CentOS). >> >> > [root@smithers ossec]# uname -a >> >> > Linux smithers 3.14.23-22.44.amzn1.x86_64 #1 SMP Tue Nov 11 23:07:48 >> UTC >> >> > 2014 x86_64 x86_64 x86_64 GNU/Linux >> >> > >> >> > Initially I tried to migrate everything that was relevant >> (client.keys, >> >> > ossec.conf, local_rules.xml, rids files, etc). But with constant >> >> > crashes, I >> >> > backed everything out and used the following basic process to narrow >> >> > down >> >> > what was happening: >> >> > >> >> > 1) I installed v1.8 server, and ran the default install with no >> agents >> >> > for >> >> > about 24 hours. All good. >> >> > 2) Added one agent (fresh install, new key, no custom rules), ran >> for 24 >> >> > hours. All good. >> >> > 3) Added remaining 7 agents (fresh install, new key, no custom >> rules), >> >> > ran >> >> > for 24 hours. All good. >> >> > 4) Added my local_rules.xml, analysisd segfaulted within a few >> minutes. >> >> > 5) Removed local_rules.xml, ran for 24 hours with no problems. >> >> > 6) Added local_rules.xml back, analysisd segfaulted after a few >> minutes. >> >> > 7) Removed local_rules.xml. Started adding one rule at a time, and >> >> > running >> >> > for 6 - 12 hours with each new rule. >> >> > 8) The above rule seems to consistently be the problem. HOWEVER, >> since >> >> > it >> >> > can take anywhere from a few minutes to a few hours (up to 5) for >> the >> >> > segfault to occur, it is possible the above rule is not the problem. >> >> > >> >> > I did try running analysisd with gdb to get some more information, >> but >> >> > I'm >> >> > not confident that I got all the ossec processes started correctly >> when >> >> > I >> >> > did it. I can post all the output from that run if someone wants it. >> >> > >> >> > Here is an example of the output in /var/log/messages: >> >> > >> >> > Dec 9 12:02:03 smithers kernel: [513311.628392] >> ossec-analysisd[12455]: >> >> > segfault at 0 ip (null) sp 00007fff152c88f8 error 14 in >> >> > ossec-analysisd[400000+65000] >> >> > >> >> > Following the segfault, these are the ossec processes running >> (although >> >> > sometimes syscheckd and/or monitord is not running): >> >> > >> >> > [root@smithers ossec]# ps -ef | grep ossec >> >> > ossecm 12447 1 0 09:58 ? 00:00:00 >> >> > /var/ossec/bin/ossec-maild >> >> > root 12451 1 0 09:58 ? 00:00:00 >> >> > /var/ossec/bin/ossec-execd >> >> > root 12470 1 0 09:58 ? 00:00:03 >> >> > /var/ossec/bin/ossec-syscheckd >> >> > ossec 12473 1 0 09:58 ? 00:00:00 >> >> > /var/ossec/bin/ossec-monitord >> >> > root 13327 13302 0 13:12 pts/0 00:00:00 grep ossec >> >> > >> >> > Here is the ossec.log from start to finish (this time it took a >> couple >> >> > of >> >> > hours for the segfault to occur): >> >> > >> >> > [root@smithers ossec]# tail -f logs/ossec.log >> >> > 2014/12/09 09:58:28 ossec-testrule: INFO: Reading local decoder >> file. >> >> [snip] >> >> > 2014/12/09 12:28:39 ossec-monitord: socketerr (not available). >> >> > 2014/12/09 12:28:39 ossec-monitord(1224): ERROR: Error sending >> message >> >> > to >> >> > queue. >> >> > >> >> > Let me know if anything else would be helpful. >> >> > >> >> >> >> I'm not seeing this issue on post 2.8.1 code, any chance you can >> >> upgrade to something remotely recent? >> >> >> >> > -- >> >> > >> >> > --- >> >> > You received this message because you are subscribed to the Google >> >> > Groups >> >> > "ossec-list" group. >> >> > To unsubscribe from this group and stop receiving emails from it, >> send >> >> > an >> >> > email to [email protected]. >> >> > For more options, visit https://groups.google.com/d/optout. >> > >> > -- >> > >> > --- >> > You received this message because you are subscribed to the Google >> Groups >> > "ossec-list" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an >> > email to [email protected]. >> > For more options, visit https://groups.google.com/d/optout. >> > -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
