On Mon, 2009-03-30 at 06:58 -0400, John A. Sullivan III wrote: > On Tue, 2009-03-24 at 11:49 -0400, John A. Sullivan III wrote: > > Here it is. There is another problem. My apologies for wondering why > > the list was so slow to respond. I am not receiving any email from the > > list including Nerijus' response below. I only received your direct > > responses, Daniel. Does one need a gmail account to use googlegroups? > > > > In any event, here is the bzip2 file. Thanks - John > > > > On Tue, 2009-03-24 at 11:44 -0300, Daniel Cid wrote: > > > Yes, try zipping it and sending to the list (or directly to my email > > > if you think it may contain confidential > > > information). It will certainly help us debug this issue. > > > > > > Thanks, > > > > > > -- > > > Daniel B. Cid > > > dcid ( at ) ossec.net > > > > > > On Fri, Mar 20, 2009 at 3:13 AM, Nerijus Krukauskas > > > <[email protected]> wrote: > > > > > > > > On 19/03/2009, John A. Sullivan III <[email protected]> > > > > wrote: > > > >> > > > >> Thanks, Daniel. I have the trace but it is a 40 MB file. How shall I > > > >> send it to you? - John > > > > > > > > I believe that if you try to zip it, it's gonna be something around 4 > > > > MB... :) > > > > > > > > -- > > > > http://nk99.org/ > > > > > Hello, all. I do have some more information on this serious bug. It > has now bitten us on two out of two vservers. > > We first thought it might have to do with our use of wildcards in the > localfile definitions, e.g., > <localfile> > <log_format>syslog</log_format> > <location>/vservers/[a-zA-Z0-9]*/var/log/maillog</location> > </localfile> > So we pulled them all out. We still had the same problem. However, it > did seem to be coincidental with not being able to find specified files. > We had mistyped some file names and paths and saw this in the error logs > before the service spun out of control: > > 2009/03/30 04:57:14 ossec-syscheckd: INFO: Starting syscheck scan (db). > 2009/03/30 04:58:41 ossec-logcollector(1103): ERROR: Unable to open file > '/vservers/w01/var/log/httpd/ssipki.error_log'. > 2009/03/30 04:58:41 ossec-logcollector(1103): ERROR: Unable to open file > '/vservers/w01/var/log/httpd/ssipki.access_log'. > 2009/03/30 04:58:41 ossec-logcollector(1103): ERROR: Unable to open file > '/var/log/dirsrv/admin-serv/error'. > 2009/03/30 04:58:41 ossec-logcollector(1103): ERROR: Unable to open file > '/var/log/dirsrv/admin-serv/access'. > 2009/03/30 04:58:41 ossec-logcollector(1103): ERROR: Unable to open file > '/var/log/dirsrv/slapd-ldap01/errors'. > 2009/03/30 05:00:51 ossec-logcollector(1103): ERROR: Unable to open file > '/vservers/w01/var/log/httpd/ssipki.error_log'. > 2009/03/30 05:00:51 ossec-logcollector(1103): ERROR: Unable to open file > '/vservers/w01/var/log/httpd/ssipki.access_log'. > 2009/03/30 05:00:51 ossec-logcollector(1103): ERROR: Unable to open file > '/var/log/dirsrv/admin-serv/error'. > 2009/03/30 05:00:51 ossec-logcollector(1103): ERROR: Unable to open file > '/var/log/dirsrv/admin-serv/access'. > 2009/03/30 05:00:51 ossec-logcollector(1103): ERROR: Unable to open file > '/var/log/dirsrv/slapd-ldap01/errors'. > 2009/03/30 05:03:01 ossec-logcollector(1103): ERROR: Unable to open file > '/vservers/w01/var/log/httpd/ssipki.error_log'. > 2009/03/30 05:03:01 ossec-logcollector(1103): ERROR: Unable to open file > '/vservers/w01/var/log/httpd/ssipki.access_log'. > 2009/03/30 05:03:01 ossec-logcollector(1103): ERROR: Unable to open file > '/var/log/dirsrv/admin-serv/error'. > 2009/03/30 05:03:01 ossec-logcollector(1103): ERROR: Unable to open file > '/var/log/dirsrv/admin-serv/access'. > 2009/03/30 05:03:01 ossec-logcollector(1103): ERROR: Unable to open file > '/var/log/dirsrv/slapd-ldap01/errors'. > 2009/03/30 05:05:11 ossec-logcollector(1103): ERROR: Unable to open file > '/vservers/w01/var/log/httpd/ssipki.error_log'. > 2009/03/30 05:05:11 ossec-logcollector(1103): ERROR: Unable to open file > '/vservers/w01/var/log/httpd/ssipki.access_log'. > 2009/03/30 05:05:11 ossec-logcollector(1103): ERROR: Unable to open file > '/var/log/dirsrv/admin-serv/error'. > 2009/03/30 05:05:11 ossec-logcollector(1103): ERROR: Unable to open file > '/var/log/dirsrv/admin-serv/access'. > 2009/03/30 05:05:11 ossec-logcollector(1103): ERROR: Unable to open file > '/var/log/dirsrv/slapd-ldap01/errors'. > 2009/03/30 05:07:21 ossec-logcollector(1103): ERROR: Unable to open file > '/vservers/w01/var/log/httpd/ssipki.error_log'. > 2009/03/30 05:07:21 ossec-logcollector(1103): ERROR: Unable to open file > '/vservers/w01/var/log/httpd/ssipki.access_log'. > 2009/03/30 05:07:21 ossec-logcollector(1103): ERROR: Unable to open file > '/var/log/dirsrv/admin-serv/error'. > 2009/03/30 05:07:21 ossec-logcollector(1103): ERROR: Unable to open file > '/var/log/dirsrv/admin-serv/access'. > 2009/03/30 05:07:21 ossec-logcollector(1103): ERROR: Unable to open file > '/var/log/dirsrv/slapd-ldap01/errors'. > 2009/03/30 05:09:32 ossec-logcollector(1904): INFO: File not available, > ignoring it: '/vservers/w01/var/log/httpd/ssipki.error_log'. > 2009/03/30 05:09:32 ossec-logcollector(1904): INFO: File not available, > ignoring it: '/vservers/w01/var/log/httpd/ssipki.access_log'. > 2009/03/30 05:09:32 ossec-logcollector(1904): INFO: File not available, > ignoring it: '/var/log/dirsrv/admin-serv/error'. > 2009/03/30 05:09:32 ossec-logcollector(1904): INFO: File not available, > ignoring it: '/var/log/dirsrv/admin-serv/access'. > 2009/03/30 05:09:32 ossec-logcollector(1904): INFO: File not available, > ignoring it: '/var/log/dirsrv/slapd-ldap01/errors'. > 2009/03/30 05:16:10 ossec-syscheckd: INFO: Ending syscheck scan (db). > > On our second vserver, we did try wildcards in the directories > definitions. That gave us the following before spinning out of control: > 2009/03/30 05:49:22 ossec-syscheckd: Error opening directory: > '/user/local/sbin': No such file or directory > 2009/03/30 05:49:22 ossec-syscheckd: Error opening directory: > '/vservers/*/etc': No such file or directory > 2009/03/30 05:49:22 ossec-syscheckd: Error opening directory: > '/vservers/*/usr/bin': No such file or directory > 2009/03/30 05:49:22 ossec-syscheckd: Error opening directory: > '/vservers/*/usr/sbin': No such file or directory > 2009/03/30 05:49:22 ossec-syscheckd: Error opening directory: > '/vservers/*/bin': No such file or directory > 2009/03/30 05:49:22 ossec-syscheckd: Error opening directory: > '/vservers/*/sbin': No such file or directory > 2009/03/30 05:49:22 ossec-syscheckd: Error opening directory: > '/vservers/*/usr/local/bin': No such file or directory > 2009/03/30 05:49:22 ossec-syscheckd: Error opening directory: > '/vservers/*/user/local/sbin': No such file or directory > 2009/03/30 05:49:22 ossec-syscheckd: Error opening directory: > '/vservers/*/usr/local/etc': No such file or directory > 2009/03/30 05:51:22 ossec-syscheckd: INFO: Starting syscheck scan (db). > > Having corrected the paths in the first vserver and taken out the wild > cards, it seems to be behaving itself. However, not being able to use > wild cards or regex's in the directories and localfiles definitions is > certainly inconvenient when we anticipate hundreds of virtual machines > on some of these systems. > > That still leaves us with the base problem. It appears that if ossec > syscheckd encounters enough missing files, it does spin out of control > and requires a power cycle of the system to recover. Thanks - John > > PS - I'm still not receiving any emails from the mail list. > Oops! I spoke to soon. The first vserver just went out of control but again, it is about missing files. We had defined some directories we knew didn't have any files just in case they were populated in the future. We would hope we could do that to prevent human error. Here is what the logs showed before CPU usage spiked to 100%:
2009/03/30 06:22:20 ossec-syscheckd: Error opening directory: '/user/local/sbin': No such file or directory 2009/03/30 06:23:07 ossec-syscheckd: Error opening directory: '/vservers/ns02/user/local/sbin': No such file or directory 2009/03/30 06:23:57 ossec-syscheckd: Error opening directory: '/vservers/w01/user/local/sbin': No such file or directory 2009/03/30 06:25:18 ossec-syscheckd: Error opening directory: '/vservers/pg01/user/local/sbin': No such file or directory 2009/03/30 06:26:43 ossec-syscheckd: Error opening directory: '/vservers/ld01/user/local/sbin': No such file or directory 2009/03/30 06:28:43 ossec-syscheckd: INFO: Starting syscheck scan (db). -- John A. Sullivan III Open Source Development Corporation +1 207-985-7880 [email protected] http://www.spiritualoutreach.com Making Christianity intelligible to secular society
