I get the segfault with ONLY the posted rule in my local_rules.xml. In other words, with a completely vanilla install of 2.8.1, by just adding that rule to local_rules.xml and restarting. Unfortunately, I can't remember if I needed to add agents before I got the segfault. I'm happy to spin up a new instance and test it out though.
On Friday, December 12, 2014 10:52:25 AM UTC-6, dan (ddpbsd) wrote: > > On Fri, Dec 12, 2014 at 11:33 AM, <[email protected] <javascript:>> > wrote: > > I'm so sorry. This IS 2.8.1 code. My post says v1.7 and 1.8, but I meant > v > > 2.7 and 2.8. > > > > I'm not able to reproduce this, could you provide your entire > local_rules.xml? > > > On Friday, December 12, 2014 9:59:34 AM UTC-6, dan (ddpbsd) wrote: > >> > >> On Fri, Dec 12, 2014 at 10:43 AM, <[email protected]> wrote: > >> > I recently migrated our ossec server to new hardware, and at the same > >> > time > >> > upgraded from v 1.7 to 1.8. Unfortunately ossec-analysisd kept > >> > segfaulting > >> > in the new environment. It's taken me a couple of weeks to narrow it > >> > down, > >> > and it seems to be consistently caused by the following rule from my > >> > local_rules.xml (downgrade of a base rule from level 7 to level 5): > >> > > >> > <rule id="533" level="5" overwrite="yes"> > >> > <if_sid>530</if_sid> > >> > <match>ossec: output: 'netstat -tan</match> > >> > <check_diff /> > >> > <description>Listened ports status (netstat) changed (new port > >> > opened or > >> > closed).</description> > >> > </rule> > >> > > >> > This rule has been running under v1.7 in our old environment for over > a > >> > year. For now, I've just commented out this rule and all is good, but > I > >> > thought it might be helpful to the community if we figure out what's > >> > going > >> > on - is it a bug, or some boneheaded misconfiguration on my part. I'm > >> > happy > >> > to provide more detail if needed. > >> > > >> > Current environment is Amazon AWS VPC. All servers are running Amazon > >> > Linux > >> > 64-bit AMI's (based on RHEL/CentOS). > >> > [root@smithers ossec]# uname -a > >> > Linux smithers 3.14.23-22.44.amzn1.x86_64 #1 SMP Tue Nov 11 23:07:48 > UTC > >> > 2014 x86_64 x86_64 x86_64 GNU/Linux > >> > > >> > Initially I tried to migrate everything that was relevant > (client.keys, > >> > ossec.conf, local_rules.xml, rids files, etc). But with constant > >> > crashes, I > >> > backed everything out and used the following basic process to narrow > >> > down > >> > what was happening: > >> > > >> > 1) I installed v1.8 server, and ran the default install with no > agents > >> > for > >> > about 24 hours. All good. > >> > 2) Added one agent (fresh install, new key, no custom rules), ran for > 24 > >> > hours. All good. > >> > 3) Added remaining 7 agents (fresh install, new key, no custom > rules), > >> > ran > >> > for 24 hours. All good. > >> > 4) Added my local_rules.xml, analysisd segfaulted within a few > minutes. > >> > 5) Removed local_rules.xml, ran for 24 hours with no problems. > >> > 6) Added local_rules.xml back, analysisd segfaulted after a few > minutes. > >> > 7) Removed local_rules.xml. Started adding one rule at a time, and > >> > running > >> > for 6 - 12 hours with each new rule. > >> > 8) The above rule seems to consistently be the problem. HOWEVER, > since > >> > it > >> > can take anywhere from a few minutes to a few hours (up to 5) for the > >> > segfault to occur, it is possible the above rule is not the problem. > >> > > >> > I did try running analysisd with gdb to get some more information, > but > >> > I'm > >> > not confident that I got all the ossec processes started correctly > when > >> > I > >> > did it. I can post all the output from that run if someone wants it. > >> > > >> > Here is an example of the output in /var/log/messages: > >> > > >> > Dec 9 12:02:03 smithers kernel: [513311.628392] > ossec-analysisd[12455]: > >> > segfault at 0 ip (null) sp 00007fff152c88f8 error 14 in > >> > ossec-analysisd[400000+65000] > >> > > >> > Following the segfault, these are the ossec processes running > (although > >> > sometimes syscheckd and/or monitord is not running): > >> > > >> > [root@smithers ossec]# ps -ef | grep ossec > >> > ossecm 12447 1 0 09:58 ? 00:00:00 > >> > /var/ossec/bin/ossec-maild > >> > root 12451 1 0 09:58 ? 00:00:00 > >> > /var/ossec/bin/ossec-execd > >> > root 12470 1 0 09:58 ? 00:00:03 > >> > /var/ossec/bin/ossec-syscheckd > >> > ossec 12473 1 0 09:58 ? 00:00:00 > >> > /var/ossec/bin/ossec-monitord > >> > root 13327 13302 0 13:12 pts/0 00:00:00 grep ossec > >> > > >> > Here is the ossec.log from start to finish (this time it took a > couple > >> > of > >> > hours for the segfault to occur): > >> > > >> > [root@smithers ossec]# tail -f logs/ossec.log > >> > 2014/12/09 09:58:28 ossec-testrule: INFO: Reading local decoder file. > >> [snip] > >> > 2014/12/09 12:28:39 ossec-monitord: socketerr (not available). > >> > 2014/12/09 12:28:39 ossec-monitord(1224): ERROR: Error sending > message > >> > to > >> > queue. > >> > > >> > Let me know if anything else would be helpful. > >> > > >> > >> I'm not seeing this issue on post 2.8.1 code, any chance you can > >> upgrade to something remotely recent? > >> > >> > -- > >> > > >> > --- > >> > You received this message because you are subscribed to the Google > >> > Groups > >> > "ossec-list" group. > >> > To unsubscribe from this group and stop receiving emails from it, > send > >> > an > >> > email to [email protected]. > >> > For more options, visit https://groups.google.com/d/optout. > > > > -- > > > > --- > > You received this message because you are subscribed to the Google > Groups > > "ossec-list" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to [email protected] <javascript:>. > > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
