I classified it as "Syslog" (does the classification make much of a difference though?). # of lines per entry varies greatly since it's non-standard. I really wish the app was designed to log in standard format, but at this point that would be a minor change request that would be shifted probably all the way to the back of the queue for development items :(
In either case, that would make sense - I can foresee logging of 1000 events per second easily during peak times. And if over half the logs consist of 25+ newlines, that would exponentially consume resources. I did a quick line count of the log file from that day and there were over 40 million lines in a 24 hour period! I'd have to do more analysis to figure out the breakdown per hour, but I'm pretty sure my theory is holding that this particular log file is a bit much for OSSEC to handle at least when volume/web traffic is high for us. On Mon, Jan 10, 2011 at 11:49 AM, dan (ddp) <[email protected]> wrote: > Hi Jeremy, > > On Mon, Jan 10, 2011 at 11:35 AM, Jeremy Lee <[email protected]> wrote: > > Thanks for the response Daniel. Unfortunately, I didn't get to measure # > of > > events per second for the particular log. One thing I was also wondering > > though: would the size of each log potentially create issues? The log is > in > > non-standard format and so one log entry may end up containing 25+ > newlines. > > I would have to guess that OSSEC may read each newline as a new entry? > I'm > > wondering if that may have been throwing it off...? > > > > What log format do you have these classified as? Is there a a set > number of lines in each entry (I'm guessing not, but worth checking > ;))? > It's possible that each "line\n" is considered a new log message, so > even though you've technically got 1000 events, OSSEC sees 25000. > > > On Mon, Jan 10, 2011 at 7:51 AM, Daniel Cid <[email protected]> > wrote: > >> > >> Do you know how much events/logs it created per second? On the agent > >> side it should > >> be able to easily handle many thousands of logs per second without any > >> tunning... > >> > >> thanks, > >> > >> > >> On Sat, Jan 8, 2011 at 1:37 PM, jplee3 <[email protected]> wrote: > >> > Hi all, > >> > > >> > We experienced 90%+ utliziation from the ossec-agentd on a couple > >> > servers this morning and I was able to trace it to a log file that > >> > grew extremely large in a relatively short time window (1gb of growth > >> > within a 2-3 hour time window). After removing the log for OSSEC to > >> > monitor, agentd quieted down. Prior to this the systems were almost to > >> > the point of being hosed. This is on Linux btw. I read an article > >> > mentioning excessive # of events at least in Windows will be the > >> > catalyst of issues like this. > >> > > >> > In any case, is there a breakdown on what OSSEC's limitation/capacity > >> > is in respect to this? Of course, I'm sure it's partly dependent on > >> > hardware capacity. One of the servers, in the same load-balanced vip, > >> > did not experience these symptoms. The difference is that this > >> > particular server is running on leveled-up hardware (DL380 G4 versus > >> > the ones with issues: DL360 G4's) and and a newer OS. I believe the > >> > DL360 G4's are both running 2.6.12-1.1372_FC3smp (Fedora Core 3?!?!?!) > >> > while the DL380 G4 is on RH (not sure which version but if I had to > >> > guess a more recent version of RHEL). > >> > > >> > Besides the 'obvious' rhetoric: "upgrade your hardware and software!" > >> > - is there any other possible avenues to investigate with why the > >> > ossec-agentd was unable to handle the volume of log growth? Obviously > >> > it's the rate at which events are generated in the logs as we > >> > understand that OSSEC does not read in the entire file at once, etc > >> > > >> > Any ideas or suggestions? > >> > > >> > > >> > TIA! > >> > > >> > > > > > >
