Hello again. On Fri, Apr 20, 2012 at 7:40 PM, Christopher Moraes <[email protected]>wrote:
> > On restart, syscheck starts running a scan and analysisd will be receiving > the information right away and comparing it with the data in the syscheck > DB. (syscheck only reads the file info and analysisd compares the present > state with the information in the DB) > > Since your trace revealed a lot of reads on the queue/syscheck/syscheck > file, I'm still guessing that it is the size of this file which is causing > analysisd to use a lot of CPU. > > Is it possible for you to test this by clearing up the syscheck DB and > restarting ossec. If analysisd does not hog the CPU then you know that the > size of the syscheck file is the issue. > I think you are absolutely correct about the syscheck run on ossec restart. Just check ossec's logfile now and noticed this: 2012/04/20 12:38:44 ossec-syscheckd: INFO: Starting syscheck scan (forwarding database). 2012/04/20 12:38:44 ossec-syscheckd: INFO: Starting syscheck database (pre-scan). 2012/04/20 12:38:44 ossec-syscheckd: INFO: Initializing real time file monitoring (not started). 2012/04/21 17:54:14 ossec-syscheckd: INFO: Finished creating syscheck database (pre-scan completed). 2012/04/21 17:54:27 ossec-syscheckd: INFO: Ending syscheck scan (forwarding database). 2012/04/21 17:54:47 ossec-syscheckd: INFO: Starting real time file monitoring. Which means the "initial" scan took about 29 hours (yay!), which caused analysisd to use the CPU like nuts. At his point, the fact that OSSEC waits for 29 hours before activating real-time monitoring is really starting to be an issue. > - why the /var/ossec/queue/syscheck/syscheck file never gets smaller >> [snip] >> > Short answer - to keep history about the file changes. Once ossec > detects a file change, it comments out the existing entry in the syscheck > DB and adds a new entry at the end of the file. If you have auto_ignore > set to "no", this will cause the DB to keep growing over time. > auto_ignore is set to "yes". # grep auto_ignore /var/ossec/etc/* /var/ossec/etc/ossec.conf: <!-- change - auto_ignore yes to ignore logfiles that change often --> /var/ossec/etc/ossec.conf: <auto_ignore>yes</auto_ignore> However on the server and all agents the syscheck config specifies that /etc is to be monitored with "report_changes" set to yes. Any other monitored folder does not have report_changes activated. > > - why does the analysisd re-read the whole file - including >> events/files/folders from months ago or even one year ago - i noticed that >> on the strace. > > > Which file are you referring to? > The same file i was asking in the previous line :) aka /var/ossec/queue/syscheck/syscheck. Stupid question: what happens if i clear the DB (except for the file getting way smaller)? Since I care only about changes that happen from this moment on to a file (and i keep the logs of any other previous changes), I don't think the file change data logged in the file would be really useful. Did i get this right? > > > - why is it random-seeking and reading only in 4K blocks (can the read >> buffer be tuned?) >> >> Analysisd gets a filename and its sum data and then does a full scan of > the syscheck DB file to find that filename. After every file is goes back > to the beginning of the file and does a full scan again. > Oh man. On Apr 23, dan (ddp) [email protected] wrote: I thought auto_ignore was a server side setting. After 3 change > notifications you shouldn't get any more for that file. How quickly it > changes doesn't matter. > auto_ignore (as far as i know) is documented as a server side setting. And on this server is set to "yes". However, i still get notifications. And i still need to find some spare time and dig more on this issue (the "spam" email we're getting is a bit annoying). The format isn't exactly optimal. The files don't get smaller until > you make them smaller (clearing the DB). > As i asked before: Stupid question: what happens if i clear the DB (except for the file getting way smaller)? The process isn't exactly optimal. It's something that's being looked > into, but I don't think much has been done yet. > > - why is it random-seeking and reading only in 4K blocks (can the read > > buffer be tuned?) > > > > No idea. > My guess is that it 4K is the size of some buffer, however i have not had any time to look at the source code to check. >From what you know, did anything change in this matter between OSSEC 2.5.1 and 2.6.0? What i mean is: upgrading from 2.5.1 to 2.6.0 would fix anything related to this issue? (besides me cleaning the DB). Thank you for your time. V.
