I recently migrated our ossec server to new hardware, and at the same time
upgraded from v 1.7 to 1.8. Unfortunately ossec-analysisd kept segfaulting
in the new environment. It's taken me a couple of weeks to narrow it down,
and it seems to be consistently caused by the following rule from my
local_rules.xml (downgrade of a base rule from level 7 to level 5):
<rule id="533" level="5" overwrite="yes">
<if_sid>530</if_sid>
<match>ossec: output: 'netstat -tan</match>
<check_diff />
<description>Listened ports status (netstat) changed (new port opened
or closed).</description>
</rule>
This rule has been running under v1.7 in our old environment for over a
year. For now, I've just commented out this rule and all is good, but I
thought it might be helpful to the community if we figure out what's going
on - is it a bug, or some boneheaded misconfiguration on my part. I'm happy
to provide more detail if needed.
Current environment is Amazon AWS VPC. All servers are running Amazon Linux
64-bit AMI's (based on RHEL/CentOS).
[root@smithers ossec]# uname -a
Linux smithers 3.14.23-22.44.amzn1.x86_64 #1 SMP Tue Nov 11 23:07:48 UTC
2014 x86_64 x86_64 x86_64 GNU/Linux
Initially I tried to migrate everything that was relevant (client.keys,
ossec.conf, local_rules.xml, rids files, etc). But with constant crashes, I
backed everything out and used the following basic process to narrow down
what was happening:
1) I installed v1.8 server, and ran the default install with no agents for
about 24 hours. All good.
2) Added one agent (fresh install, new key, no custom rules), ran for 24
hours. All good.
3) Added remaining 7 agents (fresh install, new key, no custom rules), ran
for 24 hours. All good.
4) Added my local_rules.xml, analysisd segfaulted within a few minutes.
5) Removed local_rules.xml, ran for 24 hours with no problems.
6) Added local_rules.xml back, analysisd segfaulted after a few minutes.
7) Removed local_rules.xml. Started adding one rule at a time, and running
for 6 - 12 hours with each new rule.
8) The above rule seems to consistently be the problem. HOWEVER, since it
can take anywhere from a few minutes to a few hours (up to 5) for the
segfault to occur, it is possible the above rule is not the problem.
I did try running analysisd with gdb to get some more information, but I'm
not confident that I got all the ossec processes started correctly when I
did it. I can post all the output from that run if someone wants it.
Here is an example of the output in /var/log/messages:
Dec 9 12:02:03 smithers kernel: [513311.628392] ossec-analysisd[12455]:
segfault at 0 ip (null) sp 00007fff152c88f8 error 14 in
ossec-analysisd[400000+65000]
Following the segfault, these are the ossec processes running (although
sometimes syscheckd and/or monitord is not running):
[root@smithers ossec]# ps -ef | grep ossec
ossecm 12447 1 0 09:58 ? 00:00:00 /var/ossec/bin/ossec-maild
root 12451 1 0 09:58 ? 00:00:00 /var/ossec/bin/ossec-execd
root 12470 1 0 09:58 ? 00:00:03
/var/ossec/bin/ossec-syscheckd
ossec 12473 1 0 09:58 ? 00:00:00
/var/ossec/bin/ossec-monitord
root 13327 13302 0 13:12 pts/0 00:00:00 grep ossec
Here is the ossec.log from start to finish (this time it took a couple of
hours for the segfault to occur):
[root@smithers ossec]# tail -f logs/ossec.log
2014/12/09 09:58:28 ossec-testrule: INFO: Reading local decoder file.
2014/12/09 09:58:28 ossec-testrule: INFO: Started (pid: 12426).
2014/12/09 09:58:28 ossec-maild: INFO: Started (pid: 12447).
2014/12/09 09:58:28 ossec-execd: INFO: Started (pid: 12451).
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading local decoder file.
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'rules_config.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'pam_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'sshd_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'telnetd_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'syslog_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'arpwatch_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'symantec-av_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'symantec-ws_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'pix_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'named_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'smbd_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'vsftpd_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'pure-ftpd_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'proftpd_rules.xml'
2014/12/09 09:58:28 ossec-remoted: INFO: Started (pid: 12463).
2014/12/09 09:58:28 ossec-remoted: INFO: Started (pid: 12464).
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'ms_ftpd_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'ftpd_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'hordeimp_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'roundcube_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'wordpress_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'cimserver_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'vpopmail_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'vmpop3d_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'courier_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'web_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'web_appsec_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'apache_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'nginx_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'php_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'mysql_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'postgresql_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'ids_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'squid_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'firewall_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'cisco-ios_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'netscreenfw_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'sonicwall_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'postfix_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'sendmail_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'imapd_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'mailscanner_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'dovecot_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'ms-exchange_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'racoon_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'vpn_concentrator_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'spamd_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'msauth_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'mcafee_av_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'trend-osce_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'ms-se_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'zeus_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'solaris_bsm_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'vmware_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'ms_dhcp_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'asterisk_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'ossec_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'attack_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'openbsd_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'clam_av_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'dropbear_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Reading rules file:
'local_rules.xml'
2014/12/09 09:58:28 ossec-analysisd: INFO: Total rules enabled: '1315'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file: '/etc/mtab'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file: '/etc/mnttab'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file: '/etc/hosts.deny'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file:
'/etc/mail/statistics'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file: '/etc/random-seed'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file: '/etc/adjtime'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file: '/etc/httpd/logs'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file: '/etc/utmpx'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file: '/etc/wtmpx'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file: '/etc/cups/certs'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file: '/etc/dumpdates'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file:
'/etc/svc/volatile'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file:
'C:\WINDOWS/System32/LogFiles'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file: 'C:\WINDOWS/Debug'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file:
'C:\WINDOWS/WindowsUpdate.log'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file:
'C:\WINDOWS/iis6.log'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file:
'C:\WINDOWS/system32/wbem/Logs'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file:
'C:\WINDOWS/system32/wbem/Repository'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file:
'C:\WINDOWS/Prefetch'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file:
'C:\WINDOWS/PCHEALTH/HELPCTR/DataColl'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file:
'C:\WINDOWS/SoftwareDistribution'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file: 'C:\WINDOWS/Temp'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file:
'C:\WINDOWS/system32/config'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file:
'C:\WINDOWS/system32/spool'
2014/12/09 09:58:28 ossec-analysisd: INFO: Ignoring file:
'C:\WINDOWS/system32/CatRoot'
2014/12/09 09:58:28 ossec-analysisd: INFO: White listing IP: '127.0.0.1'
2014/12/09 09:58:28 ossec-analysisd: INFO: White listing IP: '10.0.0.2'
2014/12/09 09:58:28 ossec-analysisd: INFO: 2 IPs in the white list for
active response.
2014/12/09 09:58:28 ossec-analysisd: INFO: White listing Hostname:
'localhost.localdomain'
2014/12/09 09:58:28 ossec-analysisd: INFO: 1 Hostname(s) in the white list
for active response.
2014/12/09 09:58:28 ossec-analysisd: INFO: Started (pid: 12455).
2014/12/09 09:58:29 ossec-remoted(4111): INFO: Maximum number of agents
allowed: '256'.
2014/12/09 09:58:29 ossec-remoted(1410): INFO: Reading authentication keys
file.
2014/12/09 09:58:29 ossec-remoted: INFO: Assigning counter for agent
krusty: '128:7748'.
2014/12/09 09:58:29 ossec-remoted: INFO: Assigning counter for agent
wiggum: '62:7688'.
2014/12/09 09:58:29 ossec-remoted: INFO: Assigning counter for agent itchy:
'3:9196'.
2014/12/09 09:58:29 ossec-remoted: INFO: Assigning counter for agent
sideshowbob: '2:8216'.
2014/12/09 09:58:29 ossec-remoted: INFO: Assigning counter for agent homer:
'292:7438'.
2014/12/09 09:58:29 ossec-remoted: INFO: Assigning counter for agent marge:
'151:1683'.
2014/12/09 09:58:29 ossec-remoted: INFO: Assigning counter for agent bart:
'0:4926'.
2014/12/09 09:58:29 ossec-remoted: INFO: Assigning counter for agent
scratchy: '100:2784'.
2014/12/09 09:58:29 ossec-remoted: INFO: Assigning sender counter: 0:4697
2014/12/09 09:58:29 ossec-monitord: INFO: Started (pid: 12473).
2014/12/09 09:58:31 ossec-analysisd: INFO: Connected to '/queue/alerts/ar'
(active-response queue)
2014/12/09 09:58:31 ossec-analysisd: INFO: Connected to
'/queue/alerts/execq' (exec queue)
2014/12/09 09:58:33 ossec-syscheckd: INFO: Started (pid: 12470).
2014/12/09 09:58:33 ossec-rootcheck: INFO: Started (pid: 12470).
2014/12/09 09:58:33 ossec-syscheckd: INFO: Monitoring directory: '/etc'.
2014/12/09 09:58:33 ossec-syscheckd: INFO: Monitoring directory: '/usr/bin'.
2014/12/09 09:58:33 ossec-syscheckd: INFO: Monitoring directory:
'/usr/sbin'.
2014/12/09 09:58:33 ossec-syscheckd: INFO: Monitoring directory: '/bin'.
2014/12/09 09:58:33 ossec-syscheckd: INFO: Monitoring directory: '/sbin'.
2014/12/09 09:58:34 ossec-logcollector(1950): INFO: Analyzing file:
'/var/log/messages'.
2014/12/09 09:58:34 ossec-logcollector(1950): INFO: Analyzing file:
'/var/log/secure'.
2014/12/09 09:58:34 ossec-logcollector(1950): INFO: Analyzing file:
'/var/log/maillog'.
2014/12/09 09:58:34 ossec-logcollector(1950): INFO: Analyzing file:
'/var/log/httpd/error_log'.
2014/12/09 09:58:34 ossec-logcollector(1950): INFO: Analyzing file:
'/var/log/httpd/access_log'.
2014/12/09 09:58:34 ossec-logcollector: INFO: Monitoring output of
command(360): df -h
2014/12/09 09:58:34 ossec-logcollector: INFO: Monitoring full output of
command(360): netstat -tan |grep LISTEN |grep -v 127.0.0.1 | sort
2014/12/09 09:58:34 ossec-logcollector: INFO: Monitoring full output of
command(360): last -n 5
2014/12/09 09:58:34 ossec-logcollector: INFO: Started (pid: 12459).
2014/12/09 09:59:35 ossec-syscheckd: INFO: Starting syscheck scan
(forwarding database).
2014/12/09 09:59:35 ossec-syscheckd: INFO: Starting syscheck database
(pre-scan).
2014/12/09 10:04:31 ossec-syscheckd: INFO: Finished creating syscheck
database (pre-scan completed).
2014/12/09 10:04:43 ossec-syscheckd: INFO: Ending syscheck scan (forwarding
database).
2014/12/09 10:05:03 ossec-rootcheck: INFO: Starting rootcheck scan.
2014/12/09 10:08:03 ossec-rootcheck: INFO: Ending rootcheck scan.
2014/12/09 12:02:04 ossec-logcollector: socketerr (not available).
2014/12/09 12:02:04 ossec-logcollector(1224): ERROR: Error sending message
to queue.
2014/12/09 12:02:07 ossec-logcollector(1210): ERROR: Queue
'/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2014/12/09 12:02:07 ossec-logcollector(1211): ERROR: Unable to access
queue: '/var/ossec/queue/ossec/queue'. Giving up..
2014/12/09 12:02:10 ossec-remoted: socketerr (not available).
2014/12/09 12:02:10 ossec-remoted(1210): ERROR: Queue '/queue/ossec/queue'
not accessible: 'Connection refused'.
2014/12/09 12:02:13 ossec-remoted(1210): ERROR: Queue '/queue/ossec/queue'
not accessible: 'Connection refused'.
2014/12/09 12:02:13 ossec-remoted(1211): ERROR: Unable to access queue:
'/queue/ossec/queue'. Giving up..
2014/12/09 12:24:39 ossec-monitord: socketerr (not available).
2014/12/09 12:24:39 ossec-monitord(1224): ERROR: Error sending message to
queue.
2014/12/09 12:24:39 ossec-monitord: socketerr (not available).
2014/12/09 12:24:39 ossec-monitord(1224): ERROR: Error sending message to
queue.
2014/12/09 12:24:39 ossec-monitord: socketerr (not available).
2014/12/09 12:24:39 ossec-monitord(1224): ERROR: Error sending message to
queue.
2014/12/09 12:28:39 ossec-monitord: socketerr (not available).
2014/12/09 12:28:39 ossec-monitord(1224): ERROR: Error sending message to
queue.
2014/12/09 12:28:39 ossec-monitord: socketerr (not available).
2014/12/09 12:28:39 ossec-monitord(1224): ERROR: Error sending message to
queue.
Let me know if anything else would be helpful.
--
---
You received this message because you are subscribed to the Google Groups
"ossec-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
