On Tue, 2009-03-24 at 11:49 -0400, John A. Sullivan III wrote:
> Here it is. There is another problem. My apologies for wondering why
> the list was so slow to respond. I am not receiving any email from the
> list including Nerijus' response below. I only received your direct
> responses, Daniel. Does one need a gmail account to use googlegroups?
>
> In any event, here is the bzip2 file. Thanks - John
>
> On Tue, 2009-03-24 at 11:44 -0300, Daniel Cid wrote:
> > Yes, try zipping it and sending to the list (or directly to my email
> > if you think it may contain confidential
> > information). It will certainly help us debug this issue.
> >
> > Thanks,
> >
> > --
> > Daniel B. Cid
> > dcid ( at ) ossec.net
> >
> > On Fri, Mar 20, 2009 at 3:13 AM, Nerijus Krukauskas
> > <[email protected]> wrote:
> > >
> > > On 19/03/2009, John A. Sullivan III <[email protected]> wrote:
> > >>
> > >> Thanks, Daniel. I have the trace but it is a 40 MB file. How shall I
> > >> send it to you? - John
> > >
> > > I believe that if you try to zip it, it's gonna be something around 4
> > > MB... :)
> > >
> > > --
> > > http://nk99.org/
> > >
Hello, all. I do have some more information on this serious bug. It
has now bitten us on two out of two vservers.
We first thought it might have to do with our use of wildcards in the
localfile definitions, e.g.,
<localfile>
<log_format>syslog</log_format>
<location>/vservers/[a-zA-Z0-9]*/var/log/maillog</location>
</localfile>
So we pulled them all out. We still had the same problem. However, it
did seem to be coincidental with not being able to find specified files.
We had mistyped some file names and paths and saw this in the error logs
before the service spun out of control:
2009/03/30 04:57:14 ossec-syscheckd: INFO: Starting syscheck scan (db).
2009/03/30 04:58:41 ossec-logcollector(1103): ERROR: Unable to open file
'/vservers/w01/var/log/httpd/ssipki.error_log'.
2009/03/30 04:58:41 ossec-logcollector(1103): ERROR: Unable to open file
'/vservers/w01/var/log/httpd/ssipki.access_log'.
2009/03/30 04:58:41 ossec-logcollector(1103): ERROR: Unable to open file
'/var/log/dirsrv/admin-serv/error'.
2009/03/30 04:58:41 ossec-logcollector(1103): ERROR: Unable to open file
'/var/log/dirsrv/admin-serv/access'.
2009/03/30 04:58:41 ossec-logcollector(1103): ERROR: Unable to open file
'/var/log/dirsrv/slapd-ldap01/errors'.
2009/03/30 05:00:51 ossec-logcollector(1103): ERROR: Unable to open file
'/vservers/w01/var/log/httpd/ssipki.error_log'.
2009/03/30 05:00:51 ossec-logcollector(1103): ERROR: Unable to open file
'/vservers/w01/var/log/httpd/ssipki.access_log'.
2009/03/30 05:00:51 ossec-logcollector(1103): ERROR: Unable to open file
'/var/log/dirsrv/admin-serv/error'.
2009/03/30 05:00:51 ossec-logcollector(1103): ERROR: Unable to open file
'/var/log/dirsrv/admin-serv/access'.
2009/03/30 05:00:51 ossec-logcollector(1103): ERROR: Unable to open file
'/var/log/dirsrv/slapd-ldap01/errors'.
2009/03/30 05:03:01 ossec-logcollector(1103): ERROR: Unable to open file
'/vservers/w01/var/log/httpd/ssipki.error_log'.
2009/03/30 05:03:01 ossec-logcollector(1103): ERROR: Unable to open file
'/vservers/w01/var/log/httpd/ssipki.access_log'.
2009/03/30 05:03:01 ossec-logcollector(1103): ERROR: Unable to open file
'/var/log/dirsrv/admin-serv/error'.
2009/03/30 05:03:01 ossec-logcollector(1103): ERROR: Unable to open file
'/var/log/dirsrv/admin-serv/access'.
2009/03/30 05:03:01 ossec-logcollector(1103): ERROR: Unable to open file
'/var/log/dirsrv/slapd-ldap01/errors'.
2009/03/30 05:05:11 ossec-logcollector(1103): ERROR: Unable to open file
'/vservers/w01/var/log/httpd/ssipki.error_log'.
2009/03/30 05:05:11 ossec-logcollector(1103): ERROR: Unable to open file
'/vservers/w01/var/log/httpd/ssipki.access_log'.
2009/03/30 05:05:11 ossec-logcollector(1103): ERROR: Unable to open file
'/var/log/dirsrv/admin-serv/error'.
2009/03/30 05:05:11 ossec-logcollector(1103): ERROR: Unable to open file
'/var/log/dirsrv/admin-serv/access'.
2009/03/30 05:05:11 ossec-logcollector(1103): ERROR: Unable to open file
'/var/log/dirsrv/slapd-ldap01/errors'.
2009/03/30 05:07:21 ossec-logcollector(1103): ERROR: Unable to open file
'/vservers/w01/var/log/httpd/ssipki.error_log'.
2009/03/30 05:07:21 ossec-logcollector(1103): ERROR: Unable to open file
'/vservers/w01/var/log/httpd/ssipki.access_log'.
2009/03/30 05:07:21 ossec-logcollector(1103): ERROR: Unable to open file
'/var/log/dirsrv/admin-serv/error'.
2009/03/30 05:07:21 ossec-logcollector(1103): ERROR: Unable to open file
'/var/log/dirsrv/admin-serv/access'.
2009/03/30 05:07:21 ossec-logcollector(1103): ERROR: Unable to open file
'/var/log/dirsrv/slapd-ldap01/errors'.
2009/03/30 05:09:32 ossec-logcollector(1904): INFO: File not available,
ignoring it: '/vservers/w01/var/log/httpd/ssipki.error_log'.
2009/03/30 05:09:32 ossec-logcollector(1904): INFO: File not available,
ignoring it: '/vservers/w01/var/log/httpd/ssipki.access_log'.
2009/03/30 05:09:32 ossec-logcollector(1904): INFO: File not available,
ignoring it: '/var/log/dirsrv/admin-serv/error'.
2009/03/30 05:09:32 ossec-logcollector(1904): INFO: File not available,
ignoring it: '/var/log/dirsrv/admin-serv/access'.
2009/03/30 05:09:32 ossec-logcollector(1904): INFO: File not available,
ignoring it: '/var/log/dirsrv/slapd-ldap01/errors'.
2009/03/30 05:16:10 ossec-syscheckd: INFO: Ending syscheck scan (db).
On our second vserver, we did try wildcards in the directories
definitions. That gave us the following before spinning out of control:
2009/03/30 05:49:22 ossec-syscheckd: Error opening directory:
'/user/local/sbin': No such file or directory
2009/03/30 05:49:22 ossec-syscheckd: Error opening directory:
'/vservers/*/etc': No such file or directory
2009/03/30 05:49:22 ossec-syscheckd: Error opening directory:
'/vservers/*/usr/bin': No such file or directory
2009/03/30 05:49:22 ossec-syscheckd: Error opening directory:
'/vservers/*/usr/sbin': No such file or directory
2009/03/30 05:49:22 ossec-syscheckd: Error opening directory:
'/vservers/*/bin': No such file or directory
2009/03/30 05:49:22 ossec-syscheckd: Error opening directory:
'/vservers/*/sbin': No such file or directory
2009/03/30 05:49:22 ossec-syscheckd: Error opening directory:
'/vservers/*/usr/local/bin': No such file or directory
2009/03/30 05:49:22 ossec-syscheckd: Error opening directory:
'/vservers/*/user/local/sbin': No such file or directory
2009/03/30 05:49:22 ossec-syscheckd: Error opening directory:
'/vservers/*/usr/local/etc': No such file or directory
2009/03/30 05:51:22 ossec-syscheckd: INFO: Starting syscheck scan (db).
Having corrected the paths in the first vserver and taken out the wild
cards, it seems to be behaving itself. However, not being able to use
wild cards or regex's in the directories and localfiles definitions is
certainly inconvenient when we anticipate hundreds of virtual machines
on some of these systems.
That still leaves us with the base problem. It appears that if ossec
syscheckd encounters enough missing files, it does spin out of control
and requires a power cycle of the system to recover. Thanks - John
PS - I'm still not receiving any emails from the mail list.
--
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
[email protected]
http://www.spiritualoutreach.com
Making Christianity intelligible to secular society
