If anybody else happens to experience this issue, Daniel and I were able to determine that it was due to a rather large fts-queue file (4.4M in my case). Removing the fts-queue file and letting OSSEC create a new file allowed ossec-analysisd and ossec-logtest to start instantly without excessive CPU usage. Daniel is going to work on improving the code that reads the fts-queue file.
Regards, Doug Burks http://securityonion.blogspot.com/ On Tue, Mar 9, 2010 at 2:41 PM, Doug Burks <[email protected]> wrote: > Hi Daniel, > > Thanks for your response. We're running OSSEC 2.3 on CentOS 5.4. > Nothing unusual in ossec.conf or local_rules.xml (I sent you a direct > email with a copy of my local rules). We have 33 agents total (24 > Windows, 9 Linux). All agents are running 2.3 as well. ossec-logtest > is exhibiting the same behavior; would it be affected by agents? Is > there any additional logging that I can enable to determine what is > taking so much time and CPU? > > Thanks, > Doug Burks > > On Mar 9, 7:41 am, Daniel Cid <[email protected]> wrote: >> Hi Doug, >> >> I have no clue to what might be going on... syscheckd taking long >> doesn't matter, >> because it "sleeps" in the middle to save some CPU. All normal.. >> >> For analysisd and log-test to take that long, there must be something in your >> rules or environment that's causing all that delay. I never had this >> problem before... >> What version are you using? Which OS? How many agents pointing to that box? >> >> Thanks, >> >> -- >> Daniel B. Cid >> dcid ( at ) ossec.net >> >> >> >> On Fri, Mar 5, 2010 at 10:53 AM, Doug Burks <[email protected]> wrote: >> > Yes, I saw that the log file showed a 3-minute gap between syscheckd >> > starting and finishing pre-scan. However, ossec-syscheckd is not the >> > process that is taking up 100% CPU. ossec-analysisd takes 100% CPU >> > for 3 minutes. ossec-logtest does the same thing, and I wouldn't >> > expect it to do anything with syscheckd. >> >> > I've looked at 2 other OSSEC installs and neither of them exhibit this >> > behavior. When starting OSSEC, they do show the standard 3-minute >> > syscheckd gap in the log file, but there is NO process taking 100% CPU >> > for any amount of time. Also, starting ossec-logtest on these other >> > OSSEC installs is instantaneous with no excessive CPU usage. >> >> > What would cause ossec-analysisd and ossec-logtest to hit 100% CPU >> > usage for 3 minutes? Any ideas, Daniel Cid? >> >> > Thanks, >> > Doug Burks >> >> > On Mar 4, 4:02 pm, Joshua Gimer <[email protected]> wrote: >> >> On Thu, Mar 4, 2010 at 12:11 PM, Doug Burks <[email protected]> wrote: >> >> > As I mentioned in my previous message, ossec-logtest takes about 3 >> >> > minutes before it will accept input. During this time, it is stuck at >> >> > 100% CPU usage. ossec-analysisd does the same thing when starting >> >> > OSSEC. After the 3 minutes is up, ossec-analysisd settles down to >> >> > about 30% CPU usage. >> >> >> > .... >> >> > 2010/03/04 13:59:55 ossec-syscheckd: INFO: Starting syscheck database >> >> > (pre-scan). >> >> > 2010/03/04 14:02:41 ossec-syscheckd: INFO: Finished creating syscheck >> >> > database (pre-scan completed). >> >> >> > Is this normal? >> >> >> > Thanks, >> >> > Doug Burks >> >> >> The majority of the time is being spent starting the syscheck database. >> >> Google seems to have a few results of OSSEC start logs that show around a >> >> 3 >> >> minute start as well. >> >> >> -- >> >> Thx >> >> Joshua Gimer > -- Doug Burks, GCIA, GSEC, CISSP http://securityonion.blogspot.com
