Hi Erik, I have no clue of what is going on (well, I know that analysisd is dying), but we can try to find it out.
Can you do the following (or all of them)? -Start analysisd with the debug flags (-d -d) -Run strace on analysisd before it dies (or something similar on solaris -kdump?) If we can't find out what is going on with it, it would be nice to re-compile ossec with debug enabled to see what is going on... Thanks for the report, -- Daniel B. Cid dcid ( at ) ossec.net On 6/10/07, Erik Delfgaauw <[EMAIL PROTECTED]> wrote: > Hi folks, > > OSSEC Server is crashing after some time, it happens time after time, in > this cycle which started on 2007/06/09 at 23:34:56 it happens on 2007/06/10 > at 02:55:35, here's some information: > > ==========[UNAME > -A]==================================================================================================== > > SunOS sola 5.9 Generic_118558-39 sun4u sparc SUNW,Sun-Blade-100 Solaris > > ==========[OSSEC.LOG]==================================================================================================== > > 2007/06/09 23:34:56 ossec-maild: E-Mail notification disabled. Clean Exit. > 2007/06/09 23:34:56 ossec-execd: Started (pid: 1689). > 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'rules_config.xml' > 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'pam_rules.xml' > 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'sshd_rules.xml' > 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'telnetd_rules.xml' > 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'syslog_rules.xml' > 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: > 'arpwatch_rules.xml' > 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: > 'symantec-av_rules.xml' > 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'pix_rules.xml' > 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'named_rules.xml' > 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'smbd_rules.xml' > 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'vsftpd_rules.xml' > 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: > 'pure-ftpd_rules.xml' > 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'proftpd_rules.xml' > 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'ms_ftpd_rules.xml' > 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: > 'hordeimp_rules.xml' > 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: > 'vpopmail_rules.xml' > 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'web_rules.xml' > 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'apache_rules.xml' > 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'ids_rules.xml' > 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'squid_rules.xml' > 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: > 'firewall_rules.xml' > 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: > 'netscreenfw_rules.xml' > 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'postfix_rules.xml' > 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: > 'sendmail_rules.xml' > 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'imapd_rules.xml' > 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: > 'mailscanner_rules.xml' > 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: > 'ms-exchange_rules.xml' > 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'racoon_rules.xml' > 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: > 'vpn_concentrator_rules.xml' > 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'spamd_rules.xml' > 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'msauth_rules.xml' > 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'attack_rules.xml' > 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'zeus_rules.xml' > 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'ossec_rules.xml' > 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'local_rules.xml' > 2007/06/09 23:34:57 ossec-analysisd: Total rules enabled: '559' > 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/mtab' > 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/mnttab' > 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/hosts.deny' > 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/mail/statistics' > 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/random-seed' > 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/adjtime' > 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/httpd/logs' > 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/utmpx' > 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/wtmpx' > 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/cups/certs' > 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: > 'C:\WINDOWS/System32/LogFiles' > 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: 'C:\WINDOWS/Debug' > 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: > 'C:\WINDOWS/WindowsUpdate.log' > 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: 'C:\WINDOWS/iis6.log' > 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: > 'C:\WINDOWS/system32/wbem/Logs' > 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: > 'C:\WINDOWS/system32/wbem/Repository' > 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: 'C:\WINDOWS/Prefetch' > 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: > 'C:\WINDOWS/PCHEALTH/HELPCTR/DataColl' > 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: > 'C:\WINDOWS/SoftwareDistribution' > 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: 'C:\WINDOWS/Temp' > 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: > 'C:\WINDOWS/system32/config' > 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: > 'C:\WINDOWS/system32/spool' > 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: > 'C:\WINDOWS/system32/CatRoot' > 2007/06/09 23:34:57 ossec-analysisd: White listing IP: '127.0.0.1' > 2007/06/09 23:34:57 ossec-analysisd: White listing IP: '10.6.1.250' > 2007/06/09 23:34:57 ossec-analysisd: 2 IPs in the white list for active > response. > 2007/06/09 23:34:57 ossec-analysisd: White listing Hostname: > 'localhost.localdomain' > 2007/06/09 23:34:57 ossec-analysisd: 1 Hostname(s) in the white list for > active response. > 2007/06/09 23:34:57 ossec-analysisd: Started (pid: 1694). > 2007/06/09 23:34:57 ossec-logcollector: DEBUG: Waiting main daemons to > settle. > 2007/06/09 23:34:57 ossec-remoted: Started (pid: 1702). > 2007/06/09 23:34:57 ossec-remoted: Started (pid: 1703). > 2007/06/09 23:34:57 ossec-remoted: Assigning counter for agent cent: > '1:8010'. > 2007/06/09 23:34:57 ossec-remoted: Assigning counter for agent wall: > '1:4126'. > 2007/06/09 23:34:57 ossec-remoted: Assigning sender counter: 0:199 > 2007/06/09 23:34:57 ossec-monitord: Started (pid: 1711). > 2007/06/09 23:34:59 ossec-syscheckd: Started (pid: 1706). > 2007/06/09 23:34:59 ossec-rootcheck: Started (pid: 1706). > 2007/06/09 23:35:00 ossec-analysisd: Connected to '/queue/alerts/ar' > (active-response queue) > 2007/06/09 23:35:00 ossec-analysisd: Connected to '/queue/alerts/execq' > (exec queue) > 2007/06/09 23:35:03 ossec-logcollector: (unix_domain) Maximum send buffer > set to: '16384'. > 2007/06/09 23:35:03 ossec-logcollector: DEBUG: Entering LogCollectorStart(). > 2007/06/09 23:35:03 ossec-logcollector(1950): Analyzing file: > '/var/log/authlog'. > 2007/06/09 23:35:03 ossec-logcollector(1950): Analyzing file: > '/var/log/syslog'. > 2007/06/09 23:35:03 ossec-logcollector: Started (pid: 1698). > 2007/06/10 01:03:46 ossec-logcollector: DEBUG: Reading syslog message: 'Jun > 10 01:03:44 sola EEPROM_SECURITY: [ID 702911 auth.info] > security-#badlogins=0' > 2007/06/10 01:11:25 ossec-logcollector: DEBUG: Reading syslog message: 'Jun > 10 01:11:23 sola syslogd: configuration restart' > 2007/06/10 01:12:15 ossec-logcollector: DEBUG: Reading syslog message: 'Jun > 10 01:12:14 sola syslogd: going down on signal 15' > 2007/06/10 02:55:35 ossec-logcollector: DEBUG: Reading syslog message: 'Jun > 10 02:55:33 sola genunix: [ID 457380 kern.notice] NOTICE: core_log: > ossec-analysisd[1694] core dump failed, errno=2: > /var/core/core_sola_ossec-analysisd_201_201_1181436933_1694' > 2007/06/10 02:55:35 ossec-logcollector: socketerr (not available). > 2007/06/10 02:55:35 ossec-logcollector(1224): Error sending message to > queue. > 2007/06/10 02:55:38 ossec-logcollector(1210): Queue > '/opt/ossec/queue/ossec/queue' not accessible: 'Destination address > required'. > 2007/06/10 02:55:38 ossec-logcollector(1211): Unable to access queue: > '/opt/ossec/queue/ossec/queue'. Giving up.. > 2007/06/10 02:55:44 ossec-remoted: socketerr (not available). > 2007/06/10 03:19:17 ossec-monitord: socketerr (not available). > 2007/06/10 03:19:17 ossec-monitord(1224): Error sending message to queue. > 2007/06/10 03:19:17 ossec-monitord: socketerr (not available). > 2007/06/10 03:19:17 ossec-monitord(1224): Error sending message to queue. > 2007/06/10 06:55:53 ossec-syscheckd: socketerr (not available). > 2007/06/10 06:55:53 ossec-syscheckd(1224): Error sending message to queue. > 2007/06/10 06:55:56 ossec-syscheckd(1210): Queue > '/opt/ossec/queue/ossec/queue' not accessible: 'Destination address > required'. > 2007/06/10 06:55:56 ossec-syscheckd(1211): Unable to access queue: > '/opt/ossec/queue/ossec/queue'. Giving up.. > > ==========[OSSEC-INIT.CONF]==================================================================================================== > > DIRECTORY="/opt/ossec" > VERSION="v1.2" > DATE="Sun May 20 00:43:09 MEST 2007" > TYPE="server" > > ==========[OSSEC.CONF]==================================================================================================== > > <ossec_config> > <global> > <email_notification>no</email_notification> > </global> > > <rules> > <include>rules_config.xml</include> > <include>pam_rules.xml</include> > <include>sshd_rules.xml</include> > <include>telnetd_rules.xml</include> > <include>syslog_rules.xml</include> > <include>arpwatch_rules.xml</include> > <include>symantec-av_rules.xml</include> > <include>pix_rules.xml</include> > <include>named_rules.xml</include> > <include>smbd_rules.xml</include> > <include>vsftpd_rules.xml</include> > <include>pure-ftpd_rules.xml</include> > <include>proftpd_rules.xml</include> > <include>ms_ftpd_rules.xml</include> > <include>hordeimp_rules.xml</include> > <include>vpopmail_rules.xml</include> > <include>web_rules.xml</include> > <include>apache_rules.xml</include> > <include>ids_rules.xml</include> > <include>squid_rules.xml</include> > <include>firewall_rules.xml</include> > <include>netscreenfw_rules.xml</include> > <include>postfix_rules.xml</include> > <include>sendmail_rules.xml</include> > <include>imapd_rules.xml</include> > <include>mailscanner_rules.xml</include> > <include>ms-exchange_rules.xml</include> > <include>racoon_rules.xml</include> > <include>vpn_concentrator_rules.xml</include> > <include>spamd_rules.xml</include> > <include>msauth_rules.xml</include> > <!-- <include>policy_rules.xml</include> --> > <include>attack_rules.xml</include> > <include>zeus_rules.xml</include> > <include>ossec_rules.xml</include> > <include>local_rules.xml</include> > </rules> > > <syscheck> > <!-- Frequency that syscheck is executed - default to every 6 hours --> > <frequency>21600</frequency> > > <!-- Directories to check (perform all possible verifications) --> > <directories > check_all="yes">/etc,/usr/bin,/usr/sbin</directories> > <directories check_all="yes">/bin,/sbin</directories> > > <!-- Files/directories to ignore --> > <ignore>/etc/mtab</ignore> > <ignore>/etc/mnttab</ignore> > <ignore>/etc/hosts.deny</ignore> > <ignore>/etc/mail/statistics</ignore> > <ignore>/etc/random-seed</ignore> > <ignore>/etc/adjtime</ignore> > <ignore>/etc/httpd/logs</ignore> > <ignore>/etc/utmpx</ignore> > <ignore>/etc/wtmpx</ignore> > <ignore>/etc/cups/certs</ignore> > > <!-- Windows files to ignore --> > <ignore>C:\WINDOWS/System32/LogFiles</ignore> > <ignore>C:\WINDOWS/Debug</ignore> > <ignore>C:\WINDOWS/WindowsUpdate.log</ignore> > <ignore>C:\WINDOWS/iis6.log</ignore> > <ignore>C:\WINDOWS/system32/wbem/Logs</ignore> > <ignore>C:\WINDOWS/system32/wbem/Repository</ignore> > <ignore>C:\WINDOWS/Prefetch</ignore> > <ignore>C:\WINDOWS/PCHEALTH/HELPCTR/DataColl</ignore> > <ignore>C:\WINDOWS/SoftwareDistribution</ignore> > <ignore>C:\WINDOWS/Temp</ignore> > <ignore>C:\WINDOWS/system32/config</ignore> > <ignore>C:\WINDOWS/system32/spool</ignore> > <ignore>C:\WINDOWS/system32/CatRoot</ignore> > </syscheck> > > <rootcheck> > > <rootkit_files>/opt/ossec/etc/shared/rootkit_files.txt</rootkit_files> > > <rootkit_trojans>/opt/ossec/etc/shared/rootkit_trojans.txt</rootkit_trojans> > </rootcheck> > > <global> > <white_list>127.0.0.1</white_list> > <white_list>^localhost.localdomain$</white_list> > <white_list> 10.6.1.250</white_list> > </global> > > <remote> > <connection>secure</connection> > </remote> > > <alerts> > <log_alert_level>1</log_alert_level> > </alerts> > > <command> > <name>host-deny</name> > <executable>host-deny.sh</executable> > <expect>srcip</expect> > <timeout_allowed>yes</timeout_allowed> > </command> > > <command> > <name>firewall-drop</name> > <executable>firewall-drop.sh</executable> > <expect>srcip</expect> > <timeout_allowed>yes</timeout_allowed> > </command> > > <command> > <name>disable-account</name> > <executable>disable-account.sh</executable> > <expect>user</expect> > <timeout_allowed>yes</timeout_allowed> > </command> > > <command> > <name>route-null</name> > <executable>route-null.sh</executable> > <expect>srcip</expect> > <timeout_allowed>yes</timeout_allowed> > </command> > > > <!-- Active Response Config --> > <active-response> > <!-- This response is going to execute the host-deny > - command for every event that fires a rule with > - level (severity) >= 6. > - The IP is going to be blocked for 600 seconds. > --> > <command>host-deny</command> > <location>local</location> > <level>6</level> > <timeout>600</timeout> > </active-response> > > <active-response> > <!-- Firewall Drop response. Block the IP for > - 600 seconds on the firewall (iptables, > - ipfilter, etc). > --> > <command>firewall-drop</command> > <location>local</location> > <level>6</level> > <timeout>600</timeout> > </active-response> > > <!-- Files to monitor (localfiles) --> > > <localfile> > <log_format>syslog</log_format> > <location>/var/log/authlog</location> > </localfile> > > <localfile> > <log_format>syslog</log_format> > <location>/var/log/syslog</location> > </localfile> > </ossec_config> > > > Hope you can get a clue from all this. > > Many thanks! > > Erik >
