Hi Jeremy, On Mon, Jan 10, 2011 at 2:54 PM, Jeremy Lee <[email protected]> wrote: > I classified it as "Syslog" (does the classification make much of a > difference though?). # of lines per entry varies greatly since it's > non-standard. I really wish the app was designed to log in standard format, > but at this point that would be a minor change request that would be shifted > probably all the way to the back of the queue for development items :( >
The different log formats are read differently. Apache logs look different than syslog messages. I was hoping the multi-line log type might help, but that seems to require a set number of lines per message. > In either case, that would make sense - I can foresee logging of 1000 events > per second easily during peak times. And if over half the logs consist of > 25+ newlines, that would exponentially consume resources. I did a quick line > count of the log file from that day and there were over 40 million lines in > a 24 hour period! I'd have to do more analysis to figure out the breakdown > per hour, but I'm pretty sure my theory is holding that this particular log > file is a bit much for OSSEC to handle at least when volume/web traffic is > high for us. > <kludge> If the log file ONLY receives logs from this application, you could write a script to read the messages and convert them into something more useful. </kludge> > On Mon, Jan 10, 2011 at 11:49 AM, dan (ddp) <[email protected]> wrote: >> >> Hi Jeremy, >> >> On Mon, Jan 10, 2011 at 11:35 AM, Jeremy Lee <[email protected]> wrote: >> > Thanks for the response Daniel. Unfortunately, I didn't get to measure # >> > of >> > events per second for the particular log. One thing I was also wondering >> > though: would the size of each log potentially create issues? The log is >> > in >> > non-standard format and so one log entry may end up containing 25+ >> > newlines. >> > I would have to guess that OSSEC may read each newline as a new entry? >> > I'm >> > wondering if that may have been throwing it off...? >> > >> >> What log format do you have these classified as? Is there a a set >> number of lines in each entry (I'm guessing not, but worth checking >> ;))? >> It's possible that each "line\n" is considered a new log message, so >> even though you've technically got 1000 events, OSSEC sees 25000. >> >> > On Mon, Jan 10, 2011 at 7:51 AM, Daniel Cid <[email protected]> >> > wrote: >> >> >> >> Do you know how much events/logs it created per second? On the agent >> >> side it should >> >> be able to easily handle many thousands of logs per second without any >> >> tunning... >> >> >> >> thanks, >> >> >> >> >> >> On Sat, Jan 8, 2011 at 1:37 PM, jplee3 <[email protected]> wrote: >> >> > Hi all, >> >> > >> >> > We experienced 90%+ utliziation from the ossec-agentd on a couple >> >> > servers this morning and I was able to trace it to a log file that >> >> > grew extremely large in a relatively short time window (1gb of growth >> >> > within a 2-3 hour time window). After removing the log for OSSEC to >> >> > monitor, agentd quieted down. Prior to this the systems were almost >> >> > to >> >> > the point of being hosed. This is on Linux btw. I read an article >> >> > mentioning excessive # of events at least in Windows will be the >> >> > catalyst of issues like this. >> >> > >> >> > In any case, is there a breakdown on what OSSEC's limitation/capacity >> >> > is in respect to this? Of course, I'm sure it's partly dependent on >> >> > hardware capacity. One of the servers, in the same load-balanced vip, >> >> > did not experience these symptoms. The difference is that this >> >> > particular server is running on leveled-up hardware (DL380 G4 versus >> >> > the ones with issues: DL360 G4's) and and a newer OS. I believe the >> >> > DL360 G4's are both running 2.6.12-1.1372_FC3smp (Fedora Core >> >> > 3?!?!?!) >> >> > while the DL380 G4 is on RH (not sure which version but if I had to >> >> > guess a more recent version of RHEL). >> >> > >> >> > Besides the 'obvious' rhetoric: "upgrade your hardware and software!" >> >> > - is there any other possible avenues to investigate with why the >> >> > ossec-agentd was unable to handle the volume of log growth? Obviously >> >> > it's the rate at which events are generated in the logs as we >> >> > understand that OSSEC does not read in the entire file at once, etc >> >> > >> >> > Any ideas or suggestions? >> >> > >> >> > >> >> > TIA! >> >> > >> >> > >> > >> > > >
