On Mon, Dec 15, 2014 at 8:47 AM, dan (ddp) <[email protected]> wrote: > On Fri, Dec 12, 2014 at 5:37 PM, <[email protected]> wrote: >> Ok, I can reproduce the segfault with a bare minimum. Here is what I did: >> >> 1) I spun up a brand new server (clone of my current ossec server), >> installed a fresh 2.8.1 ossec server. Modified local_rules.xml to look like >> this: >> >> <!-- @(#) $Id: ./etc/rules/local_rules.xml, 2011/09/08 dcid Exp $ >> >> - Example of local rules for OSSEC. >> - >> - Copyright (C) 2009 Trend Micro Inc. >> - All rights reserved. >> - >> - This program is a free software; you can redistribute it >> - and/or modify it under the terms of the GNU General Public >> - License (version 2) as published by the FSF - Free Software >> - Foundation. >> - >> - License details: http://www.ossec.net/en/licensing.html >> --> >> <!-- Modify it at your will. --> >> >> <group name="local,syslog,"> >> <!-- move from level 7 to level 5 6/25/2013 --> >> <rule id="533" level="5" overwrite="yes"> >> <if_sid>530</if_sid> >> <match>ossec: output: 'netstat -tan</match> >> <check_diff /> >> <description>Listened ports status (netstat) changed (new port opened or >> closed).</description> >> </rule> >> </group> <!-- SYSLOG,LOCAL --> >> <!-- EOF --> >> >> I did NOT add any agents, but let that run for about an hour. >> >> 2) I then stopped the ossec server, ran manage_agents, added one agent and >> restarted the ossec server. NOTE, I did not transfer the agent key, so the >> agent was not trying to communicate yet. I let that run for about an hour. >> >> 3) I transferred the key to the agent - this is an existing agent, I just >> added the new key and change the <server-ip> to point to my new test server >> and restarted the agent and server. I begin to see communication between >> server and agent. I let it run it's initial scans - about 10 minutes. >> >> 4) Then I opened a new listen port to try and trigger the netstat -tan rule >> using nc -l 7777 on the agent. >> >> 5) ossec-analysisd segfaulted after about 7 minues. >> >> Let me know if you want any of the output or if you want me to run any other >> tests. >> > > Wow, thanks. I was able to reproduce it with this info. >
Sometimes. It apparently doesn't like crashing when running under gdb. >> >> >> On Friday, December 12, 2014 11:14:47 AM UTC-6, [email protected] wrote: >>> >>> I get the segfault with ONLY the posted rule in my local_rules.xml. In >>> other words, with a completely vanilla install of 2.8.1, by just adding that >>> rule to local_rules.xml and restarting. Unfortunately, I can't remember if >>> I needed to add agents before I got the segfault. I'm happy to spin up a new >>> instance and test it out though. >>> >>> On Friday, December 12, 2014 10:52:25 AM UTC-6, dan (ddpbsd) wrote: >>>> >>>> On Fri, Dec 12, 2014 at 11:33 AM, <[email protected]> wrote: >>>> > I'm so sorry. This IS 2.8.1 code. My post says v1.7 and 1.8, but I >>>> > meant v >>>> > 2.7 and 2.8. >>>> > >>>> >>>> I'm not able to reproduce this, could you provide your entire >>>> local_rules.xml? >>>> >>>> > On Friday, December 12, 2014 9:59:34 AM UTC-6, dan (ddpbsd) wrote: >>>> >> >>>> >> On Fri, Dec 12, 2014 at 10:43 AM, <[email protected]> wrote: >>>> >> > I recently migrated our ossec server to new hardware, and at the >>>> >> > same >>>> >> > time >>>> >> > upgraded from v 1.7 to 1.8. Unfortunately ossec-analysisd kept >>>> >> > segfaulting >>>> >> > in the new environment. It's taken me a couple of weeks to narrow it >>>> >> > down, >>>> >> > and it seems to be consistently caused by the following rule from my >>>> >> > local_rules.xml (downgrade of a base rule from level 7 to level 5): >>>> >> > >>>> >> > <rule id="533" level="5" overwrite="yes"> >>>> >> > <if_sid>530</if_sid> >>>> >> > <match>ossec: output: 'netstat -tan</match> >>>> >> > <check_diff /> >>>> >> > <description>Listened ports status (netstat) changed (new port >>>> >> > opened or >>>> >> > closed).</description> >>>> >> > </rule> >>>> >> > >>>> >> > This rule has been running under v1.7 in our old environment for >>>> >> > over a >>>> >> > year. For now, I've just commented out this rule and all is good, >>>> >> > but I >>>> >> > thought it might be helpful to the community if we figure out what's >>>> >> > going >>>> >> > on - is it a bug, or some boneheaded misconfiguration on my part. >>>> >> > I'm >>>> >> > happy >>>> >> > to provide more detail if needed. >>>> >> > >>>> >> > Current environment is Amazon AWS VPC. All servers are running >>>> >> > Amazon >>>> >> > Linux >>>> >> > 64-bit AMI's (based on RHEL/CentOS). >>>> >> > [root@smithers ossec]# uname -a >>>> >> > Linux smithers 3.14.23-22.44.amzn1.x86_64 #1 SMP Tue Nov 11 23:07:48 >>>> >> > UTC >>>> >> > 2014 x86_64 x86_64 x86_64 GNU/Linux >>>> >> > >>>> >> > Initially I tried to migrate everything that was relevant >>>> >> > (client.keys, >>>> >> > ossec.conf, local_rules.xml, rids files, etc). But with constant >>>> >> > crashes, I >>>> >> > backed everything out and used the following basic process to narrow >>>> >> > down >>>> >> > what was happening: >>>> >> > >>>> >> > 1) I installed v1.8 server, and ran the default install with no >>>> >> > agents >>>> >> > for >>>> >> > about 24 hours. All good. >>>> >> > 2) Added one agent (fresh install, new key, no custom rules), ran >>>> >> > for 24 >>>> >> > hours. All good. >>>> >> > 3) Added remaining 7 agents (fresh install, new key, no custom >>>> >> > rules), >>>> >> > ran >>>> >> > for 24 hours. All good. >>>> >> > 4) Added my local_rules.xml, analysisd segfaulted within a few >>>> >> > minutes. >>>> >> > 5) Removed local_rules.xml, ran for 24 hours with no problems. >>>> >> > 6) Added local_rules.xml back, analysisd segfaulted after a few >>>> >> > minutes. >>>> >> > 7) Removed local_rules.xml. Started adding one rule at a time, and >>>> >> > running >>>> >> > for 6 - 12 hours with each new rule. >>>> >> > 8) The above rule seems to consistently be the problem. HOWEVER, >>>> >> > since >>>> >> > it >>>> >> > can take anywhere from a few minutes to a few hours (up to 5) for >>>> >> > the >>>> >> > segfault to occur, it is possible the above rule is not the problem. >>>> >> > >>>> >> > I did try running analysisd with gdb to get some more information, >>>> >> > but >>>> >> > I'm >>>> >> > not confident that I got all the ossec processes started correctly >>>> >> > when >>>> >> > I >>>> >> > did it. I can post all the output from that run if someone wants it. >>>> >> > >>>> >> > Here is an example of the output in /var/log/messages: >>>> >> > >>>> >> > Dec 9 12:02:03 smithers kernel: [513311.628392] >>>> >> > ossec-analysisd[12455]: >>>> >> > segfault at 0 ip (null) sp 00007fff152c88f8 error 14 in >>>> >> > ossec-analysisd[400000+65000] >>>> >> > >>>> >> > Following the segfault, these are the ossec processes running >>>> >> > (although >>>> >> > sometimes syscheckd and/or monitord is not running): >>>> >> > >>>> >> > [root@smithers ossec]# ps -ef | grep ossec >>>> >> > ossecm 12447 1 0 09:58 ? 00:00:00 >>>> >> > /var/ossec/bin/ossec-maild >>>> >> > root 12451 1 0 09:58 ? 00:00:00 >>>> >> > /var/ossec/bin/ossec-execd >>>> >> > root 12470 1 0 09:58 ? 00:00:03 >>>> >> > /var/ossec/bin/ossec-syscheckd >>>> >> > ossec 12473 1 0 09:58 ? 00:00:00 >>>> >> > /var/ossec/bin/ossec-monitord >>>> >> > root 13327 13302 0 13:12 pts/0 00:00:00 grep ossec >>>> >> > >>>> >> > Here is the ossec.log from start to finish (this time it took a >>>> >> > couple >>>> >> > of >>>> >> > hours for the segfault to occur): >>>> >> > >>>> >> > [root@smithers ossec]# tail -f logs/ossec.log >>>> >> > 2014/12/09 09:58:28 ossec-testrule: INFO: Reading local decoder >>>> >> > file. >>>> >> [snip] >>>> >> > 2014/12/09 12:28:39 ossec-monitord: socketerr (not available). >>>> >> > 2014/12/09 12:28:39 ossec-monitord(1224): ERROR: Error sending >>>> >> > message >>>> >> > to >>>> >> > queue. >>>> >> > >>>> >> > Let me know if anything else would be helpful. >>>> >> > >>>> >> >>>> >> I'm not seeing this issue on post 2.8.1 code, any chance you can >>>> >> upgrade to something remotely recent? >>>> >> >>>> >> > -- >>>> >> > >>>> >> > --- >>>> >> > You received this message because you are subscribed to the Google >>>> >> > Groups >>>> >> > "ossec-list" group. >>>> >> > To unsubscribe from this group and stop receiving emails from it, >>>> >> > send >>>> >> > an >>>> >> > email to [email protected]. >>>> >> > For more options, visit https://groups.google.com/d/optout. >>>> > >>>> > -- >>>> > >>>> > --- >>>> > You received this message because you are subscribed to the Google >>>> > Groups >>>> > "ossec-list" group. >>>> > To unsubscribe from this group and stop receiving emails from it, send >>>> > an >>>> > email to [email protected]. >>>> > For more options, visit https://groups.google.com/d/optout. >> >> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "ossec-list" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
