On Fri, Dec 12, 2014 at 5:37 PM, <[email protected]> wrote: > Ok, I can reproduce the segfault with a bare minimum. Here is what I did: > > 1) I spun up a brand new server (clone of my current ossec server), > installed a fresh 2.8.1 ossec server. Modified local_rules.xml to look like > this: > > <!-- @(#) $Id: ./etc/rules/local_rules.xml, 2011/09/08 dcid Exp $ > > - Example of local rules for OSSEC. > - > - Copyright (C) 2009 Trend Micro Inc. > - All rights reserved. > - > - This program is a free software; you can redistribute it > - and/or modify it under the terms of the GNU General Public > - License (version 2) as published by the FSF - Free Software > - Foundation. > - > - License details: http://www.ossec.net/en/licensing.html > --> > <!-- Modify it at your will. --> > > <group name="local,syslog,"> > <!-- move from level 7 to level 5 6/25/2013 --> > <rule id="533" level="5" overwrite="yes"> > <if_sid>530</if_sid> > <match>ossec: output: 'netstat -tan</match> > <check_diff /> > <description>Listened ports status (netstat) changed (new port opened or > closed).</description> > </rule> > </group> <!-- SYSLOG,LOCAL --> > <!-- EOF --> > > I did NOT add any agents, but let that run for about an hour. > > 2) I then stopped the ossec server, ran manage_agents, added one agent and > restarted the ossec server. NOTE, I did not transfer the agent key, so the > agent was not trying to communicate yet. I let that run for about an hour. > > 3) I transferred the key to the agent - this is an existing agent, I just > added the new key and change the <server-ip> to point to my new test server > and restarted the agent and server. I begin to see communication between > server and agent. I let it run it's initial scans - about 10 minutes. > > 4) Then I opened a new listen port to try and trigger the netstat -tan rule > using nc -l 7777 on the agent. > > 5) ossec-analysisd segfaulted after about 7 minues. > > Let me know if you want any of the output or if you want me to run any other > tests. >
Wow, thanks. I was able to reproduce it with this info. > > > On Friday, December 12, 2014 11:14:47 AM UTC-6, [email protected] wrote: >> >> I get the segfault with ONLY the posted rule in my local_rules.xml. In >> other words, with a completely vanilla install of 2.8.1, by just adding that >> rule to local_rules.xml and restarting. Unfortunately, I can't remember if >> I needed to add agents before I got the segfault. I'm happy to spin up a new >> instance and test it out though. >> >> On Friday, December 12, 2014 10:52:25 AM UTC-6, dan (ddpbsd) wrote: >>> >>> On Fri, Dec 12, 2014 at 11:33 AM, <[email protected]> wrote: >>> > I'm so sorry. This IS 2.8.1 code. My post says v1.7 and 1.8, but I >>> > meant v >>> > 2.7 and 2.8. >>> > >>> >>> I'm not able to reproduce this, could you provide your entire >>> local_rules.xml? >>> >>> > On Friday, December 12, 2014 9:59:34 AM UTC-6, dan (ddpbsd) wrote: >>> >> >>> >> On Fri, Dec 12, 2014 at 10:43 AM, <[email protected]> wrote: >>> >> > I recently migrated our ossec server to new hardware, and at the >>> >> > same >>> >> > time >>> >> > upgraded from v 1.7 to 1.8. Unfortunately ossec-analysisd kept >>> >> > segfaulting >>> >> > in the new environment. It's taken me a couple of weeks to narrow it >>> >> > down, >>> >> > and it seems to be consistently caused by the following rule from my >>> >> > local_rules.xml (downgrade of a base rule from level 7 to level 5): >>> >> > >>> >> > <rule id="533" level="5" overwrite="yes"> >>> >> > <if_sid>530</if_sid> >>> >> > <match>ossec: output: 'netstat -tan</match> >>> >> > <check_diff /> >>> >> > <description>Listened ports status (netstat) changed (new port >>> >> > opened or >>> >> > closed).</description> >>> >> > </rule> >>> >> > >>> >> > This rule has been running under v1.7 in our old environment for >>> >> > over a >>> >> > year. For now, I've just commented out this rule and all is good, >>> >> > but I >>> >> > thought it might be helpful to the community if we figure out what's >>> >> > going >>> >> > on - is it a bug, or some boneheaded misconfiguration on my part. >>> >> > I'm >>> >> > happy >>> >> > to provide more detail if needed. >>> >> > >>> >> > Current environment is Amazon AWS VPC. All servers are running >>> >> > Amazon >>> >> > Linux >>> >> > 64-bit AMI's (based on RHEL/CentOS). >>> >> > [root@smithers ossec]# uname -a >>> >> > Linux smithers 3.14.23-22.44.amzn1.x86_64 #1 SMP Tue Nov 11 23:07:48 >>> >> > UTC >>> >> > 2014 x86_64 x86_64 x86_64 GNU/Linux >>> >> > >>> >> > Initially I tried to migrate everything that was relevant >>> >> > (client.keys, >>> >> > ossec.conf, local_rules.xml, rids files, etc). But with constant >>> >> > crashes, I >>> >> > backed everything out and used the following basic process to narrow >>> >> > down >>> >> > what was happening: >>> >> > >>> >> > 1) I installed v1.8 server, and ran the default install with no >>> >> > agents >>> >> > for >>> >> > about 24 hours. All good. >>> >> > 2) Added one agent (fresh install, new key, no custom rules), ran >>> >> > for 24 >>> >> > hours. All good. >>> >> > 3) Added remaining 7 agents (fresh install, new key, no custom >>> >> > rules), >>> >> > ran >>> >> > for 24 hours. All good. >>> >> > 4) Added my local_rules.xml, analysisd segfaulted within a few >>> >> > minutes. >>> >> > 5) Removed local_rules.xml, ran for 24 hours with no problems. >>> >> > 6) Added local_rules.xml back, analysisd segfaulted after a few >>> >> > minutes. >>> >> > 7) Removed local_rules.xml. Started adding one rule at a time, and >>> >> > running >>> >> > for 6 - 12 hours with each new rule. >>> >> > 8) The above rule seems to consistently be the problem. HOWEVER, >>> >> > since >>> >> > it >>> >> > can take anywhere from a few minutes to a few hours (up to 5) for >>> >> > the >>> >> > segfault to occur, it is possible the above rule is not the problem. >>> >> > >>> >> > I did try running analysisd with gdb to get some more information, >>> >> > but >>> >> > I'm >>> >> > not confident that I got all the ossec processes started correctly >>> >> > when >>> >> > I >>> >> > did it. I can post all the output from that run if someone wants it. >>> >> > >>> >> > Here is an example of the output in /var/log/messages: >>> >> > >>> >> > Dec 9 12:02:03 smithers kernel: [513311.628392] >>> >> > ossec-analysisd[12455]: >>> >> > segfault at 0 ip (null) sp 00007fff152c88f8 error 14 in >>> >> > ossec-analysisd[400000+65000] >>> >> > >>> >> > Following the segfault, these are the ossec processes running >>> >> > (although >>> >> > sometimes syscheckd and/or monitord is not running): >>> >> > >>> >> > [root@smithers ossec]# ps -ef | grep ossec >>> >> > ossecm 12447 1 0 09:58 ? 00:00:00 >>> >> > /var/ossec/bin/ossec-maild >>> >> > root 12451 1 0 09:58 ? 00:00:00 >>> >> > /var/ossec/bin/ossec-execd >>> >> > root 12470 1 0 09:58 ? 00:00:03 >>> >> > /var/ossec/bin/ossec-syscheckd >>> >> > ossec 12473 1 0 09:58 ? 00:00:00 >>> >> > /var/ossec/bin/ossec-monitord >>> >> > root 13327 13302 0 13:12 pts/0 00:00:00 grep ossec >>> >> > >>> >> > Here is the ossec.log from start to finish (this time it took a >>> >> > couple >>> >> > of >>> >> > hours for the segfault to occur): >>> >> > >>> >> > [root@smithers ossec]# tail -f logs/ossec.log >>> >> > 2014/12/09 09:58:28 ossec-testrule: INFO: Reading local decoder >>> >> > file. >>> >> [snip] >>> >> > 2014/12/09 12:28:39 ossec-monitord: socketerr (not available). >>> >> > 2014/12/09 12:28:39 ossec-monitord(1224): ERROR: Error sending >>> >> > message >>> >> > to >>> >> > queue. >>> >> > >>> >> > Let me know if anything else would be helpful. >>> >> > >>> >> >>> >> I'm not seeing this issue on post 2.8.1 code, any chance you can >>> >> upgrade to something remotely recent? >>> >> >>> >> > -- >>> >> > >>> >> > --- >>> >> > You received this message because you are subscribed to the Google >>> >> > Groups >>> >> > "ossec-list" group. >>> >> > To unsubscribe from this group and stop receiving emails from it, >>> >> > send >>> >> > an >>> >> > email to [email protected]. >>> >> > For more options, visit https://groups.google.com/d/optout. >>> > >>> > -- >>> > >>> > --- >>> > You received this message because you are subscribed to the Google >>> > Groups >>> > "ossec-list" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> > an >>> > email to [email protected]. >>> > For more options, visit https://groups.google.com/d/optout. > > -- > > --- > You received this message because you are subscribed to the Google Groups > "ossec-list" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
