Ok, Thanks for the advice. On Tuesday, December 16, 2014 7:54:01 AM UTC-6, dan (ddpbsd) wrote: > > I've been playing with it for a while and can't figure it out. You > should probably post an issue to github > (https://github.com/ossec/ossec-hids). The devs may pay attention to > it there. > > On Mon, Dec 15, 2014 at 2:12 PM, <[email protected] <javascript:>> wrote: > > I decided to give gdb a try again. I can get it to segfault pretty > > consistently. Here is my output. > > > > [root@ossectst ossec]# ps -ef | grep ossec > > root 5018 4903 0 12:39 pts/2 00:00:00 tail -f logs/ossec.log > > root 5675 2407 0 12:50 pts/1 00:00:00 grep ossec > > [root@ossectst ossec]# service ossec start > > Starting OSSEC: [ OK ] > > [root@ossectst ossec]# ps -ef | grep ossec > > root 5018 4903 0 12:39 pts/2 00:00:00 tail -f logs/ossec.log > > ossecm 5711 1 0 12:50 ? 00:00:00 > /var/ossec/bin/ossec-maild > > root 5715 1 0 12:50 ? 00:00:00 > /var/ossec/bin/ossec-execd > > ossec 5719 1 0 12:50 ? 00:00:00 > > /var/ossec/bin/ossec-analysisd > > root 5723 1 0 12:50 ? 00:00:00 > > /var/ossec/bin/ossec-logcollector > > ossecr 5728 1 0 12:50 ? 00:00:00 > /var/ossec/bin/ossec-remoted > > root 5734 1 0 12:50 ? 00:00:00 > > /var/ossec/bin/ossec-syscheckd > > ossec 5737 1 0 12:50 ? 00:00:00 > > /var/ossec/bin/ossec-monitord > > root 5747 2407 0 12:50 pts/1 00:00:00 grep ossec > > [root@ossectst ossec]# gdb /var/ossec/bin/ossec-analysisd 5719 > > GNU gdb (GDB) Amazon Linux (7.6.1-51.24.amzn1) > > Copyright (C) 2013 Free Software Foundation, Inc. > > License GPLv3+: GNU GPL version 3 or later > > <http://gnu.org/licenses/gpl.html> > > This is free software: you are free to change and redistribute it. > > There is NO WARRANTY, to the extent permitted by law. Type "show > copying" > > and "show warranty" for details. > > This GDB was configured as "x86_64-amazon-linux-gnu". > > For bug reporting instructions, please see: > > <http://www.gnu.org/software/gdb/bugs/>... > > Reading symbols from /var/ossec/bin/ossec-analysisd...done. > > Attaching to program: /var/ossec/bin/ossec-analysisd, process 5719 > > Reading symbols from /lib64/libm.so.6...Reading symbols from > > /usr/lib/debug/lib64/libm-2.17.so.debug...done. > > done. > > Loaded symbols for /lib64/libm.so.6 > > Reading symbols from /lib64/libc.so.6...Reading symbols from > > /usr/lib/debug/lib64/libc-2.17.so.debug...done. > > done. > > Loaded symbols for /lib64/libc.so.6 > > Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from > > /usr/lib/debug/lib64/ld-2.17.so.debug...done. > > done. > > Loaded symbols for /lib64/ld-linux-x86-64.so.2 > > Reading symbols from /lib64/libnss_files.so.2...Reading symbols from > > /usr/lib/debug/lib64/libnss_files-2.17.so.debug...done. > > done. > > Loaded symbols for /lib64/libnss_files.so.2 > > 0x00007f9f2b003b53 in __recvfrom_nocancel () at > > ../sysdeps/unix/syscall-template.S:81 > > 81 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) > > (gdb) bt > > #0 0x00007f9f2b003b53 in __recvfrom_nocancel () at > > ../sysdeps/unix/syscall-template.S:81 > > #1 0x0000000000430569 in OS_RecvUnix (socket=4, sizet=6144, > > ret=0x7fffe0f85870 "1:/var/log/maillog") at os_net.c:539 > > #2 0x0000000000403612 in OS_ReadMSG (m_queue=4) at analysisd.c:754 > > #3 0x0000000000403262 in main (argc=1, argv=0x7fffe0f87268) at > > analysisd.c:555 > > (gdb) cont > > Continuing. > > > > Program received signal SIGSEGV, Segmentation fault. > > 0x0000000000000000 in ?? () > > (gdb) where > > #0 0x0000000000000000 in ?? () > > #1 0x0000000000404a0e in OS_CheckIfRuleMatch (lf=0x1b0a490, > > curr_node=0x1addcd0) at analysisd.c:1631 > > #2 0x0000000000404a53 in OS_CheckIfRuleMatch (lf=0x1b0a490, > > curr_node=0x1adcac0) at analysisd.c:1654 > > #3 0x0000000000404a53 in OS_CheckIfRuleMatch (lf=0x1b0a490, > > curr_node=0x1ad53f0) at analysisd.c:1654 > > #4 0x0000000000404a53 in OS_CheckIfRuleMatch (lf=0x1b0a490, > > curr_node=0x199c4e0) at analysisd.c:1654 > > #5 0x0000000000403a5e in OS_ReadMSG (m_queue=4) at analysisd.c:984 > > #6 0x0000000000403262 in main (argc=1, argv=0x7fffe0f87268) at > > analysisd.c:555 > > (gdb) list > > 76 #else > > 77 > > 78 /* This is a "normal" system call stub: if there is an error, > > 79 it returns -1 and sets errno. */ > > 80 > > 81 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) > > 82 ret > > 83 T_PSEUDO_END (SYSCALL_SYMBOL) > > 84 > > 85 #endif > > (gdb) bt full > > #0 0x0000000000000000 in ?? () > > No symbol table info available. > > #1 0x0000000000404a0e in OS_CheckIfRuleMatch (lf=0x1b0a490, > > curr_node=0x1addcd0) at analysisd.c:1631 > > currently_rule = 0x1add7c0 > > #2 0x0000000000404a53 in OS_CheckIfRuleMatch (lf=0x1b0a490, > > curr_node=0x1adcac0) at analysisd.c:1654 > > child_node = 0x1addcd0 > > child_rule = 0x0 > > currently_rule = 0x1adc770 > > #3 0x0000000000404a53 in OS_CheckIfRuleMatch (lf=0x1b0a490, > > curr_node=0x1ad53f0) at analysisd.c:1654 > > child_node = 0x1adcac0 > > child_rule = 0x0 > > currently_rule = 0x1ad5ff0 > > #4 0x0000000000404a53 in OS_CheckIfRuleMatch (lf=0x1b0a490, > > curr_node=0x199c4e0) at analysisd.c:1654 > > child_node = 0x1ad53f0 > > child_rule = 0x0 > > currently_rule = 0x199c210 > > #5 0x0000000000403a5e in OS_ReadMSG (m_queue=4) at analysisd.c:984 > > rulenode_pt = 0x199c4e0 > > i = 765 > > msg = "1:(itchy) 10.0.1.0->netstat -tan |grep LISTEN |grep -v > > 127.0.0.1 | sort\000ossec: output: 'netstat -tan |grep LISTEN |grep -v > > 127.0.0.1 | sort':\ntcp 0 0 0.0.0.0:22", ' ' <repeats 18 > times>, > > "0.0.0.0:*", ' ' <repeats 19 times>... > > lf = 0x1b0a490 > > ---Type <return> to continue, or q <return> to quit--- > > stats_rule = 0x1b08b00 > > #6 0x0000000000403262 in main (argc=1, argv=0x7fffe0f87268) at > > analysisd.c:555 > > c = -2 > > m_queue = 4 > > test_config = 0 > > run_foreground = 0 > > debug_level = 0 > > dir = 0x444bc0 "/var/ossec" > > user = 0x444bcb "ossec" > > group = 0x444bcb "ossec" > > uid = 503 > > gid = 502 > > cfg = 0x444bd1 "/var/ossec/etc/ossec.conf" > > (gdb) cont > > Continuing. > > > > Program terminated with signal SIGSEGV, Segmentation fault. > > The program no longer exists. > > (gdb) quit > > [root@ossectst ossec]# service ossec stop > > Stopping OSSEC: [ OK ] > > [root@ossectst ossec]# tail /var/log/messages > > ... snip ... > > Dec 15 10:45:51 ossectst kernel: [ 2082.001287] ossec-analysisd[3772]: > > segfault at 0 ip (null) sp 00007fffa4fa8218 error 14 in > > ossec-analysisd[400000+65000] > > > > > > > > On Monday, December 15, 2014 8:56:33 AM UTC-6, dan (ddpbsd) wrote: > >> > >> On Mon, Dec 15, 2014 at 8:47 AM, dan (ddp) <[email protected]> wrote: > >> > On Fri, Dec 12, 2014 at 5:37 PM, <[email protected]> wrote: > >> >> Ok, I can reproduce the segfault with a bare minimum. Here is what > I > >> >> did: > >> >> > >> >> 1) I spun up a brand new server (clone of my current ossec server), > >> >> installed a fresh 2.8.1 ossec server. Modified local_rules.xml to > look > >> >> like > >> >> this: > >> >> > >> >> <!-- @(#) $Id: ./etc/rules/local_rules.xml, 2011/09/08 dcid Exp $ > >> >> > >> >> - Example of local rules for OSSEC. > >> >> - > >> >> - Copyright (C) 2009 Trend Micro Inc. > >> >> - All rights reserved. > >> >> - > >> >> - This program is a free software; you can redistribute it > >> >> - and/or modify it under the terms of the GNU General Public > >> >> - License (version 2) as published by the FSF - Free Software > >> >> - Foundation. > >> >> - > >> >> - License details: http://www.ossec.net/en/licensing.html > >> >> --> > >> >> <!-- Modify it at your will. --> > >> >> > >> >> <group name="local,syslog,"> > >> >> <!-- move from level 7 to level 5 6/25/2013 --> > >> >> <rule id="533" level="5" overwrite="yes"> > >> >> <if_sid>530</if_sid> > >> >> <match>ossec: output: 'netstat -tan</match> > >> >> <check_diff /> > >> >> <description>Listened ports status (netstat) changed (new port > >> >> opened or > >> >> closed).</description> > >> >> </rule> > >> >> </group> <!-- SYSLOG,LOCAL --> > >> >> <!-- EOF --> > >> >> > >> >> I did NOT add any agents, but let that run for about an hour. > >> >> > >> >> 2) I then stopped the ossec server, ran manage_agents, added one > agent > >> >> and > >> >> restarted the ossec server. NOTE, I did not transfer the agent key, > so > >> >> the > >> >> agent was not trying to communicate yet. I let that run for about an > >> >> hour. > >> >> > >> >> 3) I transferred the key to the agent - this is an existing agent, I > >> >> just > >> >> added the new key and change the <server-ip> to point to my new test > >> >> server > >> >> and restarted the agent and server. I begin to see communication > >> >> between > >> >> server and agent. I let it run it's initial scans - about 10 > minutes. > >> >> > >> >> 4) Then I opened a new listen port to try and trigger the netstat > -tan > >> >> rule > >> >> using nc -l 7777 on the agent. > >> >> > >> >> 5) ossec-analysisd segfaulted after about 7 minues. > >> >> > >> >> Let me know if you want any of the output or if you want me to run > any > >> >> other > >> >> tests. > >> >> > >> > > >> > Wow, thanks. I was able to reproduce it with this info. > >> > > >> > >> Sometimes. It apparently doesn't like crashing when running under gdb. > >> > >> >> > >> >> > >> >> On Friday, December 12, 2014 11:14:47 AM UTC-6, [email protected] > wrote: > >> >>> > >> >>> I get the segfault with ONLY the posted rule in my local_rules.xml. > >> >>> In > >> >>> other words, with a completely vanilla install of 2.8.1, by just > >> >>> adding that > >> >>> rule to local_rules.xml and restarting. Unfortunately, I can't > >> >>> remember if > >> >>> I needed to add agents before I got the segfault. I'm happy to spin > up > >> >>> a new > >> >>> instance and test it out though. > >> >>> > >> >>> On Friday, December 12, 2014 10:52:25 AM UTC-6, dan (ddpbsd) wrote: > >> >>>> > >> >>>> On Fri, Dec 12, 2014 at 11:33 AM, <[email protected]> wrote: > >> >>>> > I'm so sorry. This IS 2.8.1 code. My post says v1.7 and 1.8, but > I > >> >>>> > meant v > >> >>>> > 2.7 and 2.8. > >> >>>> > > >> >>>> > >> >>>> I'm not able to reproduce this, could you provide your entire > >> >>>> local_rules.xml? > >> >>>> > >> >>>> > On Friday, December 12, 2014 9:59:34 AM UTC-6, dan (ddpbsd) > wrote: > >> >>>> >> > >> >>>> >> On Fri, Dec 12, 2014 at 10:43 AM, <[email protected]> wrote: > >> >>>> >> > I recently migrated our ossec server to new hardware, and at > the > >> >>>> >> > same > >> >>>> >> > time > >> >>>> >> > upgraded from v 1.7 to 1.8. Unfortunately ossec-analysisd > kept > >> >>>> >> > segfaulting > >> >>>> >> > in the new environment. It's taken me a couple of weeks to > >> >>>> >> > narrow it > >> >>>> >> > down, > >> >>>> >> > and it seems to be consistently caused by the following rule > >> >>>> >> > from my > >> >>>> >> > local_rules.xml (downgrade of a base rule from level 7 to > level > >> >>>> >> > 5): > >> >>>> >> > > >> >>>> >> > <rule id="533" level="5" overwrite="yes"> > >> >>>> >> > <if_sid>530</if_sid> > >> >>>> >> > <match>ossec: output: 'netstat -tan</match> > >> >>>> >> > <check_diff /> > >> >>>> >> > <description>Listened ports status (netstat) changed (new > >> >>>> >> > port > >> >>>> >> > opened or > >> >>>> >> > closed).</description> > >> >>>> >> > </rule> > >> >>>> >> > > >> >>>> >> > This rule has been running under v1.7 in our old environment > for > >> >>>> >> > over a > >> >>>> >> > year. For now, I've just commented out this rule and all is > >> >>>> >> > good, > >> >>>> >> > but I > >> >>>> >> > thought it might be helpful to the community if we figure out > >> >>>> >> > what's > >> >>>> >> > going > >> >>>> >> > on - is it a bug, or some boneheaded misconfiguration on my > >> >>>> >> > part. > >> >>>> >> > I'm > >> >>>> >> > happy > >> >>>> >> > to provide more detail if needed. > >> >>>> >> > > >> >>>> >> > Current environment is Amazon AWS VPC. All servers are > running > >> >>>> >> > Amazon > >> >>>> >> > Linux > >> >>>> >> > 64-bit AMI's (based on RHEL/CentOS). > >> >>>> >> > [root@smithers ossec]# uname -a > >> >>>> >> > Linux smithers 3.14.23-22.44.amzn1.x86_64 #1 SMP Tue Nov 11 > >> >>>> >> > 23:07:48 > >> >>>> >> > UTC > >> >>>> >> > 2014 x86_64 x86_64 x86_64 GNU/Linux > >> >>>> >> > > >> >>>> >> > Initially I tried to migrate everything that was relevant > >> >>>> >> > (client.keys, > >> >>>> >> > ossec.conf, local_rules.xml, rids files, etc). But with > constant > >> >>>> >> > crashes, I > >> >>>> >> > backed everything out and used the following basic process to > >> >>>> >> > narrow > >> >>>> >> > down > >> >>>> >> > what was happening: > >> >>>> >> > > >> >>>> >> > 1) I installed v1.8 server, and ran the default install with > no > >> >>>> >> > agents > >> >>>> >> > for > >> >>>> >> > about 24 hours. All good. > >> >>>> >> > 2) Added one agent (fresh install, new key, no custom rules), > >> >>>> >> > ran > >> >>>> >> > for 24 > >> >>>> >> > hours. All good. > >> >>>> >> > 3) Added remaining 7 agents (fresh install, new key, no > custom > >> >>>> >> > rules), > >> >>>> >> > ran > >> >>>> >> > for 24 hours. All good. > >> >>>> >> > 4) Added my local_rules.xml, analysisd segfaulted within a > few > >> >>>> >> > minutes. > >> >>>> >> > 5) Removed local_rules.xml, ran for 24 hours with no > problems. > >> >>>> >> > 6) Added local_rules.xml back, analysisd segfaulted after a > few > >> >>>> >> > minutes. > >> >>>> >> > 7) Removed local_rules.xml. Started adding one rule at a > time, > >> >>>> >> > and > >> >>>> >> > running > >> >>>> >> > for 6 - 12 hours with each new rule. > >> >>>> >> > 8) The above rule seems to consistently be the problem. > HOWEVER, > >> >>>> >> > since > >> >>>> >> > it > >> >>>> >> > can take anywhere from a few minutes to a few hours (up to 5) > >> >>>> >> > for > >> >>>> >> > the > >> >>>> >> > segfault to occur, it is possible the above rule is not the > >> >>>> >> > problem. > >> >>>> >> > > >> >>>> >> > I did try running analysisd with gdb to get some more > >> >>>> >> > information, > >> >>>> >> > but > >> >>>> >> > I'm > >> >>>> >> > not confident that I got all the ossec processes started > >> >>>> >> > correctly > >> >>>> >> > when > >> >>>> >> > I > >> >>>> >> > did it. I can post all the output from that run if someone > wants > >> >>>> >> > it. > >> >>>> >> > > >> >>>> >> > Here is an example of the output in /var/log/messages: > >> >>>> >> > > >> >>>> >> > Dec 9 12:02:03 smithers kernel: [513311.628392] > >> >>>> >> > ossec-analysisd[12455]: > >> >>>> >> > segfault at 0 ip (null) sp 00007fff152c88f8 error > 14 > >> >>>> >> > in > >> >>>> >> > ossec-analysisd[400000+65000] > >> >>>> >> > > >> >>>> >> > Following the segfault, these are the ossec processes running > >> >>>> >> > (although > >> >>>> >> > sometimes syscheckd and/or monitord is not running): > >> >>>> >> > > >> >>>> >> > [root@smithers ossec]# ps -ef | grep ossec > >> >>>> >> > ossecm 12447 1 0 09:58 ? 00:00:00 > >> >>>> >> > /var/ossec/bin/ossec-maild > >> >>>> >> > root 12451 1 0 09:58 ? 00:00:00 > >> >>>> >> > /var/ossec/bin/ossec-execd > >> >>>> >> > root 12470 1 0 09:58 ? 00:00:03 > >> >>>> >> > /var/ossec/bin/ossec-syscheckd > >> >>>> >> > ossec 12473 1 0 09:58 ? 00:00:00 > >> >>>> >> > /var/ossec/bin/ossec-monitord > >> >>>> >> > root 13327 13302 0 13:12 pts/0 00:00:00 grep ossec > >> >>>> >> > > >> >>>> >> > Here is the ossec.log from start to finish (this time it took > a > >> >>>> >> > couple > >> >>>> >> > of > >> >>>> >> > hours for the segfault to occur): > >> >>>> >> > > >> >>>> >> > [root@smithers ossec]# tail -f logs/ossec.log > >> >>>> >> > 2014/12/09 09:58:28 ossec-testrule: INFO: Reading local > decoder > >> >>>> >> > file. > >> >>>> >> [snip] > >> >>>> >> > 2014/12/09 12:28:39 ossec-monitord: socketerr (not > available). > >> >>>> >> > 2014/12/09 12:28:39 ossec-monitord(1224): ERROR: Error > sending > >> >>>> >> > message > >> >>>> >> > to > >> >>>> >> > queue. > >> >>>> >> > > >> >>>> >> > Let me know if anything else would be helpful. > >> >>>> >> > > >> >>>> >> > >> >>>> >> I'm not seeing this issue on post 2.8.1 code, any chance you > can > >> >>>> >> upgrade to something remotely recent? > >> >>>> >> > >> >>>> >> > -- > >> >>>> >> > > >> >>>> >> > --- > >> >>>> >> > You received this message because you are subscribed to the > >> >>>> >> > Google > >> >>>> >> > Groups > >> >>>> >> > "ossec-list" group. > >> >>>> >> > To unsubscribe from this group and stop receiving emails from > >> >>>> >> > it, > >> >>>> >> > send > >> >>>> >> > an > >> >>>> >> > email to [email protected]. > >> >>>> >> > For more options, visit https://groups.google.com/d/optout. > >> >>>> > > >> >>>> > -- > >> >>>> > > >> >>>> > --- > >> >>>> > You received this message because you are subscribed to the > Google > >> >>>> > Groups > >> >>>> > "ossec-list" group. > >> >>>> > To unsubscribe from this group and stop receiving emails from > it, > >> >>>> > send > >> >>>> > an > >> >>>> > email to [email protected]. > >> >>>> > For more options, visit https://groups.google.com/d/optout. > >> >> > >> >> -- > >> >> > >> >> --- > >> >> You received this message because you are subscribed to the Google > >> >> Groups > >> >> "ossec-list" group. > >> >> To unsubscribe from this group and stop receiving emails from it, > send > >> >> an > >> >> email to [email protected]. > >> >> For more options, visit https://groups.google.com/d/optout. > > > > -- > > > > --- > > You received this message because you are subscribed to the Google > Groups > > "ossec-list" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to [email protected] <javascript:>. > > For more options, visit https://groups.google.com/d/optout. >
-- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
