I'm going to post this here as well as the github issue. I might have a fix for this in my diff_overwrite branch (https://github.com/ddpbsd/ossec-hids/tree/diff_overwrite). Please test and report findings. It seems to be working for me, but changes to the way rules are handled in analysisd can have a much bigger impact than intended, so the more testing we can get (of all analysisd things) the better off we'll be.
On Tue, Dec 16, 2014 at 9:16 AM, <[email protected]> wrote: > Ok, Thanks for the advice. > > On Tuesday, December 16, 2014 7:54:01 AM UTC-6, dan (ddpbsd) wrote: >> >> I've been playing with it for a while and can't figure it out. You >> should probably post an issue to github >> (https://github.com/ossec/ossec-hids). The devs may pay attention to >> it there. >> >> On Mon, Dec 15, 2014 at 2:12 PM, <[email protected]> wrote: >> > I decided to give gdb a try again. I can get it to segfault pretty >> > consistently. Here is my output. >> > >> > [root@ossectst ossec]# ps -ef | grep ossec >> > root 5018 4903 0 12:39 pts/2 00:00:00 tail -f logs/ossec.log >> > root 5675 2407 0 12:50 pts/1 00:00:00 grep ossec >> > [root@ossectst ossec]# service ossec start >> > Starting OSSEC: [ OK ] >> > [root@ossectst ossec]# ps -ef | grep ossec >> > root 5018 4903 0 12:39 pts/2 00:00:00 tail -f logs/ossec.log >> > ossecm 5711 1 0 12:50 ? 00:00:00 >> > /var/ossec/bin/ossec-maild >> > root 5715 1 0 12:50 ? 00:00:00 >> > /var/ossec/bin/ossec-execd >> > ossec 5719 1 0 12:50 ? 00:00:00 >> > /var/ossec/bin/ossec-analysisd >> > root 5723 1 0 12:50 ? 00:00:00 >> > /var/ossec/bin/ossec-logcollector >> > ossecr 5728 1 0 12:50 ? 00:00:00 >> > /var/ossec/bin/ossec-remoted >> > root 5734 1 0 12:50 ? 00:00:00 >> > /var/ossec/bin/ossec-syscheckd >> > ossec 5737 1 0 12:50 ? 00:00:00 >> > /var/ossec/bin/ossec-monitord >> > root 5747 2407 0 12:50 pts/1 00:00:00 grep ossec >> > [root@ossectst ossec]# gdb /var/ossec/bin/ossec-analysisd 5719 >> > GNU gdb (GDB) Amazon Linux (7.6.1-51.24.amzn1) >> > Copyright (C) 2013 Free Software Foundation, Inc. >> > License GPLv3+: GNU GPL version 3 or later >> > <http://gnu.org/licenses/gpl.html> >> > This is free software: you are free to change and redistribute it. >> > There is NO WARRANTY, to the extent permitted by law. Type "show >> > copying" >> > and "show warranty" for details. >> > This GDB was configured as "x86_64-amazon-linux-gnu". >> > For bug reporting instructions, please see: >> > <http://www.gnu.org/software/gdb/bugs/>... >> > Reading symbols from /var/ossec/bin/ossec-analysisd...done. >> > Attaching to program: /var/ossec/bin/ossec-analysisd, process 5719 >> > Reading symbols from /lib64/libm.so.6...Reading symbols from >> > /usr/lib/debug/lib64/libm-2.17.so.debug...done. >> > done. >> > Loaded symbols for /lib64/libm.so.6 >> > Reading symbols from /lib64/libc.so.6...Reading symbols from >> > /usr/lib/debug/lib64/libc-2.17.so.debug...done. >> > done. >> > Loaded symbols for /lib64/libc.so.6 >> > Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from >> > /usr/lib/debug/lib64/ld-2.17.so.debug...done. >> > done. >> > Loaded symbols for /lib64/ld-linux-x86-64.so.2 >> > Reading symbols from /lib64/libnss_files.so.2...Reading symbols from >> > /usr/lib/debug/lib64/libnss_files-2.17.so.debug...done. >> > done. >> > Loaded symbols for /lib64/libnss_files.so.2 >> > 0x00007f9f2b003b53 in __recvfrom_nocancel () at >> > ../sysdeps/unix/syscall-template.S:81 >> > 81 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) >> > (gdb) bt >> > #0 0x00007f9f2b003b53 in __recvfrom_nocancel () at >> > ../sysdeps/unix/syscall-template.S:81 >> > #1 0x0000000000430569 in OS_RecvUnix (socket=4, sizet=6144, >> > ret=0x7fffe0f85870 "1:/var/log/maillog") at os_net.c:539 >> > #2 0x0000000000403612 in OS_ReadMSG (m_queue=4) at analysisd.c:754 >> > #3 0x0000000000403262 in main (argc=1, argv=0x7fffe0f87268) at >> > analysisd.c:555 >> > (gdb) cont >> > Continuing. >> > >> > Program received signal SIGSEGV, Segmentation fault. >> > 0x0000000000000000 in ?? () >> > (gdb) where >> > #0 0x0000000000000000 in ?? () >> > #1 0x0000000000404a0e in OS_CheckIfRuleMatch (lf=0x1b0a490, >> > curr_node=0x1addcd0) at analysisd.c:1631 >> > #2 0x0000000000404a53 in OS_CheckIfRuleMatch (lf=0x1b0a490, >> > curr_node=0x1adcac0) at analysisd.c:1654 >> > #3 0x0000000000404a53 in OS_CheckIfRuleMatch (lf=0x1b0a490, >> > curr_node=0x1ad53f0) at analysisd.c:1654 >> > #4 0x0000000000404a53 in OS_CheckIfRuleMatch (lf=0x1b0a490, >> > curr_node=0x199c4e0) at analysisd.c:1654 >> > #5 0x0000000000403a5e in OS_ReadMSG (m_queue=4) at analysisd.c:984 >> > #6 0x0000000000403262 in main (argc=1, argv=0x7fffe0f87268) at >> > analysisd.c:555 >> > (gdb) list >> > 76 #else >> > 77 >> > 78 /* This is a "normal" system call stub: if there is an error, >> > 79 it returns -1 and sets errno. */ >> > 80 >> > 81 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) >> > 82 ret >> > 83 T_PSEUDO_END (SYSCALL_SYMBOL) >> > 84 >> > 85 #endif >> > (gdb) bt full >> > #0 0x0000000000000000 in ?? () >> > No symbol table info available. >> > #1 0x0000000000404a0e in OS_CheckIfRuleMatch (lf=0x1b0a490, >> > curr_node=0x1addcd0) at analysisd.c:1631 >> > currently_rule = 0x1add7c0 >> > #2 0x0000000000404a53 in OS_CheckIfRuleMatch (lf=0x1b0a490, >> > curr_node=0x1adcac0) at analysisd.c:1654 >> > child_node = 0x1addcd0 >> > child_rule = 0x0 >> > currently_rule = 0x1adc770 >> > #3 0x0000000000404a53 in OS_CheckIfRuleMatch (lf=0x1b0a490, >> > curr_node=0x1ad53f0) at analysisd.c:1654 >> > child_node = 0x1adcac0 >> > child_rule = 0x0 >> > currently_rule = 0x1ad5ff0 >> > #4 0x0000000000404a53 in OS_CheckIfRuleMatch (lf=0x1b0a490, >> > curr_node=0x199c4e0) at analysisd.c:1654 >> > child_node = 0x1ad53f0 >> > child_rule = 0x0 >> > currently_rule = 0x199c210 >> > #5 0x0000000000403a5e in OS_ReadMSG (m_queue=4) at analysisd.c:984 >> > rulenode_pt = 0x199c4e0 >> > i = 765 >> > msg = "1:(itchy) 10.0.1.0->netstat -tan |grep LISTEN |grep -v >> > 127.0.0.1 | sort\000ossec: output: 'netstat -tan |grep LISTEN |grep -v >> > 127.0.0.1 | sort':\ntcp 0 0 0.0.0.0:22", ' ' <repeats 18 >> > times>, >> > "0.0.0.0:*", ' ' <repeats 19 times>... >> > lf = 0x1b0a490 >> > ---Type <return> to continue, or q <return> to quit--- >> > stats_rule = 0x1b08b00 >> > #6 0x0000000000403262 in main (argc=1, argv=0x7fffe0f87268) at >> > analysisd.c:555 >> > c = -2 >> > m_queue = 4 >> > test_config = 0 >> > run_foreground = 0 >> > debug_level = 0 >> > dir = 0x444bc0 "/var/ossec" >> > user = 0x444bcb "ossec" >> > group = 0x444bcb "ossec" >> > uid = 503 >> > gid = 502 >> > cfg = 0x444bd1 "/var/ossec/etc/ossec.conf" >> > (gdb) cont >> > Continuing. >> > >> > Program terminated with signal SIGSEGV, Segmentation fault. >> > The program no longer exists. >> > (gdb) quit >> > [root@ossectst ossec]# service ossec stop >> > Stopping OSSEC: [ OK ] >> > [root@ossectst ossec]# tail /var/log/messages >> > ... snip ... >> > Dec 15 10:45:51 ossectst kernel: [ 2082.001287] ossec-analysisd[3772]: >> > segfault at 0 ip (null) sp 00007fffa4fa8218 error 14 in >> > ossec-analysisd[400000+65000] >> > >> > >> > >> > On Monday, December 15, 2014 8:56:33 AM UTC-6, dan (ddpbsd) wrote: >> >> >> >> On Mon, Dec 15, 2014 at 8:47 AM, dan (ddp) <[email protected]> wrote: >> >> > On Fri, Dec 12, 2014 at 5:37 PM, <[email protected]> wrote: >> >> >> Ok, I can reproduce the segfault with a bare minimum. Here is what >> >> >> I >> >> >> did: >> >> >> >> >> >> 1) I spun up a brand new server (clone of my current ossec server), >> >> >> installed a fresh 2.8.1 ossec server. Modified local_rules.xml to >> >> >> look >> >> >> like >> >> >> this: >> >> >> >> >> >> <!-- @(#) $Id: ./etc/rules/local_rules.xml, 2011/09/08 dcid Exp $ >> >> >> >> >> >> - Example of local rules for OSSEC. >> >> >> - >> >> >> - Copyright (C) 2009 Trend Micro Inc. >> >> >> - All rights reserved. >> >> >> - >> >> >> - This program is a free software; you can redistribute it >> >> >> - and/or modify it under the terms of the GNU General Public >> >> >> - License (version 2) as published by the FSF - Free Software >> >> >> - Foundation. >> >> >> - >> >> >> - License details: http://www.ossec.net/en/licensing.html >> >> >> --> >> >> >> <!-- Modify it at your will. --> >> >> >> >> >> >> <group name="local,syslog,"> >> >> >> <!-- move from level 7 to level 5 6/25/2013 --> >> >> >> <rule id="533" level="5" overwrite="yes"> >> >> >> <if_sid>530</if_sid> >> >> >> <match>ossec: output: 'netstat -tan</match> >> >> >> <check_diff /> >> >> >> <description>Listened ports status (netstat) changed (new port >> >> >> opened or >> >> >> closed).</description> >> >> >> </rule> >> >> >> </group> <!-- SYSLOG,LOCAL --> >> >> >> <!-- EOF --> >> >> >> >> >> >> I did NOT add any agents, but let that run for about an hour. >> >> >> >> >> >> 2) I then stopped the ossec server, ran manage_agents, added one >> >> >> agent >> >> >> and >> >> >> restarted the ossec server. NOTE, I did not transfer the agent key, >> >> >> so >> >> >> the >> >> >> agent was not trying to communicate yet. I let that run for about an >> >> >> hour. >> >> >> >> >> >> 3) I transferred the key to the agent - this is an existing agent, I >> >> >> just >> >> >> added the new key and change the <server-ip> to point to my new test >> >> >> server >> >> >> and restarted the agent and server. I begin to see communication >> >> >> between >> >> >> server and agent. I let it run it's initial scans - about 10 >> >> >> minutes. >> >> >> >> >> >> 4) Then I opened a new listen port to try and trigger the netstat >> >> >> -tan >> >> >> rule >> >> >> using nc -l 7777 on the agent. >> >> >> >> >> >> 5) ossec-analysisd segfaulted after about 7 minues. >> >> >> >> >> >> Let me know if you want any of the output or if you want me to run >> >> >> any >> >> >> other >> >> >> tests. >> >> >> >> >> > >> >> > Wow, thanks. I was able to reproduce it with this info. >> >> > >> >> >> >> Sometimes. It apparently doesn't like crashing when running under gdb. >> >> >> >> >> >> >> >> >> >> >> On Friday, December 12, 2014 11:14:47 AM UTC-6, [email protected] >> >> >> wrote: >> >> >>> >> >> >>> I get the segfault with ONLY the posted rule in my local_rules.xml. >> >> >>> In >> >> >>> other words, with a completely vanilla install of 2.8.1, by just >> >> >>> adding that >> >> >>> rule to local_rules.xml and restarting. Unfortunately, I can't >> >> >>> remember if >> >> >>> I needed to add agents before I got the segfault. I'm happy to spin >> >> >>> up >> >> >>> a new >> >> >>> instance and test it out though. >> >> >>> >> >> >>> On Friday, December 12, 2014 10:52:25 AM UTC-6, dan (ddpbsd) wrote: >> >> >>>> >> >> >>>> On Fri, Dec 12, 2014 at 11:33 AM, <[email protected]> wrote: >> >> >>>> > I'm so sorry. This IS 2.8.1 code. My post says v1.7 and 1.8, but >> >> >>>> > I >> >> >>>> > meant v >> >> >>>> > 2.7 and 2.8. >> >> >>>> > >> >> >>>> >> >> >>>> I'm not able to reproduce this, could you provide your entire >> >> >>>> local_rules.xml? >> >> >>>> >> >> >>>> > On Friday, December 12, 2014 9:59:34 AM UTC-6, dan (ddpbsd) >> >> >>>> > wrote: >> >> >>>> >> >> >> >>>> >> On Fri, Dec 12, 2014 at 10:43 AM, <[email protected]> wrote: >> >> >>>> >> > I recently migrated our ossec server to new hardware, and at >> >> >>>> >> > the >> >> >>>> >> > same >> >> >>>> >> > time >> >> >>>> >> > upgraded from v 1.7 to 1.8. Unfortunately ossec-analysisd >> >> >>>> >> > kept >> >> >>>> >> > segfaulting >> >> >>>> >> > in the new environment. It's taken me a couple of weeks to >> >> >>>> >> > narrow it >> >> >>>> >> > down, >> >> >>>> >> > and it seems to be consistently caused by the following rule >> >> >>>> >> > from my >> >> >>>> >> > local_rules.xml (downgrade of a base rule from level 7 to >> >> >>>> >> > level >> >> >>>> >> > 5): >> >> >>>> >> > >> >> >>>> >> > <rule id="533" level="5" overwrite="yes"> >> >> >>>> >> > <if_sid>530</if_sid> >> >> >>>> >> > <match>ossec: output: 'netstat -tan</match> >> >> >>>> >> > <check_diff /> >> >> >>>> >> > <description>Listened ports status (netstat) changed (new >> >> >>>> >> > port >> >> >>>> >> > opened or >> >> >>>> >> > closed).</description> >> >> >>>> >> > </rule> >> >> >>>> >> > >> >> >>>> >> > This rule has been running under v1.7 in our old environment >> >> >>>> >> > for >> >> >>>> >> > over a >> >> >>>> >> > year. For now, I've just commented out this rule and all is >> >> >>>> >> > good, >> >> >>>> >> > but I >> >> >>>> >> > thought it might be helpful to the community if we figure out >> >> >>>> >> > what's >> >> >>>> >> > going >> >> >>>> >> > on - is it a bug, or some boneheaded misconfiguration on my >> >> >>>> >> > part. >> >> >>>> >> > I'm >> >> >>>> >> > happy >> >> >>>> >> > to provide more detail if needed. >> >> >>>> >> > >> >> >>>> >> > Current environment is Amazon AWS VPC. All servers are >> >> >>>> >> > running >> >> >>>> >> > Amazon >> >> >>>> >> > Linux >> >> >>>> >> > 64-bit AMI's (based on RHEL/CentOS). >> >> >>>> >> > [root@smithers ossec]# uname -a >> >> >>>> >> > Linux smithers 3.14.23-22.44.amzn1.x86_64 #1 SMP Tue Nov 11 >> >> >>>> >> > 23:07:48 >> >> >>>> >> > UTC >> >> >>>> >> > 2014 x86_64 x86_64 x86_64 GNU/Linux >> >> >>>> >> > >> >> >>>> >> > Initially I tried to migrate everything that was relevant >> >> >>>> >> > (client.keys, >> >> >>>> >> > ossec.conf, local_rules.xml, rids files, etc). But with >> >> >>>> >> > constant >> >> >>>> >> > crashes, I >> >> >>>> >> > backed everything out and used the following basic process to >> >> >>>> >> > narrow >> >> >>>> >> > down >> >> >>>> >> > what was happening: >> >> >>>> >> > >> >> >>>> >> > 1) I installed v1.8 server, and ran the default install with >> >> >>>> >> > no >> >> >>>> >> > agents >> >> >>>> >> > for >> >> >>>> >> > about 24 hours. All good. >> >> >>>> >> > 2) Added one agent (fresh install, new key, no custom rules), >> >> >>>> >> > ran >> >> >>>> >> > for 24 >> >> >>>> >> > hours. All good. >> >> >>>> >> > 3) Added remaining 7 agents (fresh install, new key, no >> >> >>>> >> > custom >> >> >>>> >> > rules), >> >> >>>> >> > ran >> >> >>>> >> > for 24 hours. All good. >> >> >>>> >> > 4) Added my local_rules.xml, analysisd segfaulted within a >> >> >>>> >> > few >> >> >>>> >> > minutes. >> >> >>>> >> > 5) Removed local_rules.xml, ran for 24 hours with no >> >> >>>> >> > problems. >> >> >>>> >> > 6) Added local_rules.xml back, analysisd segfaulted after a >> >> >>>> >> > few >> >> >>>> >> > minutes. >> >> >>>> >> > 7) Removed local_rules.xml. Started adding one rule at a >> >> >>>> >> > time, >> >> >>>> >> > and >> >> >>>> >> > running >> >> >>>> >> > for 6 - 12 hours with each new rule. >> >> >>>> >> > 8) The above rule seems to consistently be the problem. >> >> >>>> >> > HOWEVER, >> >> >>>> >> > since >> >> >>>> >> > it >> >> >>>> >> > can take anywhere from a few minutes to a few hours (up to 5) >> >> >>>> >> > for >> >> >>>> >> > the >> >> >>>> >> > segfault to occur, it is possible the above rule is not the >> >> >>>> >> > problem. >> >> >>>> >> > >> >> >>>> >> > I did try running analysisd with gdb to get some more >> >> >>>> >> > information, >> >> >>>> >> > but >> >> >>>> >> > I'm >> >> >>>> >> > not confident that I got all the ossec processes started >> >> >>>> >> > correctly >> >> >>>> >> > when >> >> >>>> >> > I >> >> >>>> >> > did it. I can post all the output from that run if someone >> >> >>>> >> > wants >> >> >>>> >> > it. >> >> >>>> >> > >> >> >>>> >> > Here is an example of the output in /var/log/messages: >> >> >>>> >> > >> >> >>>> >> > Dec 9 12:02:03 smithers kernel: [513311.628392] >> >> >>>> >> > ossec-analysisd[12455]: >> >> >>>> >> > segfault at 0 ip (null) sp 00007fff152c88f8 error >> >> >>>> >> > 14 >> >> >>>> >> > in >> >> >>>> >> > ossec-analysisd[400000+65000] >> >> >>>> >> > >> >> >>>> >> > Following the segfault, these are the ossec processes running >> >> >>>> >> > (although >> >> >>>> >> > sometimes syscheckd and/or monitord is not running): >> >> >>>> >> > >> >> >>>> >> > [root@smithers ossec]# ps -ef | grep ossec >> >> >>>> >> > ossecm 12447 1 0 09:58 ? 00:00:00 >> >> >>>> >> > /var/ossec/bin/ossec-maild >> >> >>>> >> > root 12451 1 0 09:58 ? 00:00:00 >> >> >>>> >> > /var/ossec/bin/ossec-execd >> >> >>>> >> > root 12470 1 0 09:58 ? 00:00:03 >> >> >>>> >> > /var/ossec/bin/ossec-syscheckd >> >> >>>> >> > ossec 12473 1 0 09:58 ? 00:00:00 >> >> >>>> >> > /var/ossec/bin/ossec-monitord >> >> >>>> >> > root 13327 13302 0 13:12 pts/0 00:00:00 grep ossec >> >> >>>> >> > >> >> >>>> >> > Here is the ossec.log from start to finish (this time it took >> >> >>>> >> > a >> >> >>>> >> > couple >> >> >>>> >> > of >> >> >>>> >> > hours for the segfault to occur): >> >> >>>> >> > >> >> >>>> >> > [root@smithers ossec]# tail -f logs/ossec.log >> >> >>>> >> > 2014/12/09 09:58:28 ossec-testrule: INFO: Reading local >> >> >>>> >> > decoder >> >> >>>> >> > file. >> >> >>>> >> [snip] >> >> >>>> >> > 2014/12/09 12:28:39 ossec-monitord: socketerr (not >> >> >>>> >> > available). >> >> >>>> >> > 2014/12/09 12:28:39 ossec-monitord(1224): ERROR: Error >> >> >>>> >> > sending >> >> >>>> >> > message >> >> >>>> >> > to >> >> >>>> >> > queue. >> >> >>>> >> > >> >> >>>> >> > Let me know if anything else would be helpful. >> >> >>>> >> > >> >> >>>> >> >> >> >>>> >> I'm not seeing this issue on post 2.8.1 code, any chance you >> >> >>>> >> can >> >> >>>> >> upgrade to something remotely recent? >> >> >>>> >> >> >> >>>> >> > -- >> >> >>>> >> > >> >> >>>> >> > --- >> >> >>>> >> > You received this message because you are subscribed to the >> >> >>>> >> > Google >> >> >>>> >> > Groups >> >> >>>> >> > "ossec-list" group. >> >> >>>> >> > To unsubscribe from this group and stop receiving emails from >> >> >>>> >> > it, >> >> >>>> >> > send >> >> >>>> >> > an >> >> >>>> >> > email to [email protected]. >> >> >>>> >> > For more options, visit https://groups.google.com/d/optout. >> >> >>>> > >> >> >>>> > -- >> >> >>>> > >> >> >>>> > --- >> >> >>>> > You received this message because you are subscribed to the >> >> >>>> > Google >> >> >>>> > Groups >> >> >>>> > "ossec-list" group. >> >> >>>> > To unsubscribe from this group and stop receiving emails from >> >> >>>> > it, >> >> >>>> > send >> >> >>>> > an >> >> >>>> > email to [email protected]. >> >> >>>> > For more options, visit https://groups.google.com/d/optout. >> >> >> >> >> >> -- >> >> >> >> >> >> --- >> >> >> You received this message because you are subscribed to the Google >> >> >> Groups >> >> >> "ossec-list" group. >> >> >> To unsubscribe from this group and stop receiving emails from it, >> >> >> send >> >> >> an >> >> >> email to [email protected]. >> >> >> For more options, visit https://groups.google.com/d/optout. >> > >> > -- >> > >> > --- >> > You received this message because you are subscribed to the Google >> > Groups >> > "ossec-list" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> > an >> > email to [email protected]. >> > For more options, visit https://groups.google.com/d/optout. > > -- > > --- > You received this message because you are subscribed to the Google Groups > "ossec-list" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
