Thanks for the ideas, Dan! In fact, there is a pre-existing script that runs every minute which basically greps for specific keywords, etc. If there are greater than 50 it spits information out into an email.
The initial reason I was utilizing OSSEC was to eliminate this script. However, it seems like I'll need to modify it a little and have OSSEC work in conjunction with it to be most effective while not compromising performance. On Mon, Jan 10, 2011 at 12:00 PM, dan (ddp) <[email protected]> wrote: > Hi Jeremy, > > On Mon, Jan 10, 2011 at 2:54 PM, Jeremy Lee <[email protected]> wrote: > > I classified it as "Syslog" (does the classification make much of a > > difference though?). # of lines per entry varies greatly since it's > > non-standard. I really wish the app was designed to log in standard > format, > > but at this point that would be a minor change request that would be > shifted > > probably all the way to the back of the queue for development items :( > > > > The different log formats are read differently. Apache logs look > different than syslog messages. I was hoping the multi-line log type > might help, but that seems to require a set number of lines per > message. > > > In either case, that would make sense - I can foresee logging of 1000 > events > > per second easily during peak times. And if over half the logs consist of > > 25+ newlines, that would exponentially consume resources. I did a quick > line > > count of the log file from that day and there were over 40 million lines > in > > a 24 hour period! I'd have to do more analysis to figure out the > breakdown > > per hour, but I'm pretty sure my theory is holding that this particular > log > > file is a bit much for OSSEC to handle at least when volume/web traffic > is > > high for us. > > > > <kludge> If the log file ONLY receives logs from this application, you > could write a script to read the messages and convert them into > something more useful. </kludge> > > > On Mon, Jan 10, 2011 at 11:49 AM, dan (ddp) <[email protected]> wrote: > >> > >> Hi Jeremy, > >> > >> On Mon, Jan 10, 2011 at 11:35 AM, Jeremy Lee <[email protected]> wrote: > >> > Thanks for the response Daniel. Unfortunately, I didn't get to measure > # > >> > of > >> > events per second for the particular log. One thing I was also > wondering > >> > though: would the size of each log potentially create issues? The log > is > >> > in > >> > non-standard format and so one log entry may end up containing 25+ > >> > newlines. > >> > I would have to guess that OSSEC may read each newline as a new entry? > >> > I'm > >> > wondering if that may have been throwing it off...? > >> > > >> > >> What log format do you have these classified as? Is there a a set > >> number of lines in each entry (I'm guessing not, but worth checking > >> ;))? > >> It's possible that each "line\n" is considered a new log message, so > >> even though you've technically got 1000 events, OSSEC sees 25000. > >> > >> > On Mon, Jan 10, 2011 at 7:51 AM, Daniel Cid <[email protected]> > >> > wrote: > >> >> > >> >> Do you know how much events/logs it created per second? On the agent > >> >> side it should > >> >> be able to easily handle many thousands of logs per second without > any > >> >> tunning... > >> >> > >> >> thanks, > >> >> > >> >> > >> >> On Sat, Jan 8, 2011 at 1:37 PM, jplee3 <[email protected]> wrote: > >> >> > Hi all, > >> >> > > >> >> > We experienced 90%+ utliziation from the ossec-agentd on a couple > >> >> > servers this morning and I was able to trace it to a log file that > >> >> > grew extremely large in a relatively short time window (1gb of > growth > >> >> > within a 2-3 hour time window). After removing the log for OSSEC to > >> >> > monitor, agentd quieted down. Prior to this the systems were almost > >> >> > to > >> >> > the point of being hosed. This is on Linux btw. I read an article > >> >> > mentioning excessive # of events at least in Windows will be the > >> >> > catalyst of issues like this. > >> >> > > >> >> > In any case, is there a breakdown on what OSSEC's > limitation/capacity > >> >> > is in respect to this? Of course, I'm sure it's partly dependent on > >> >> > hardware capacity. One of the servers, in the same load-balanced > vip, > >> >> > did not experience these symptoms. The difference is that this > >> >> > particular server is running on leveled-up hardware (DL380 G4 > versus > >> >> > the ones with issues: DL360 G4's) and and a newer OS. I believe the > >> >> > DL360 G4's are both running 2.6.12-1.1372_FC3smp (Fedora Core > >> >> > 3?!?!?!) > >> >> > while the DL380 G4 is on RH (not sure which version but if I had to > >> >> > guess a more recent version of RHEL). > >> >> > > >> >> > Besides the 'obvious' rhetoric: "upgrade your hardware and > software!" > >> >> > - is there any other possible avenues to investigate with why the > >> >> > ossec-agentd was unable to handle the volume of log growth? > Obviously > >> >> > it's the rate at which events are generated in the logs as we > >> >> > understand that OSSEC does not read in the entire file at once, etc > >> >> > > >> >> > Any ideas or suggestions? > >> >> > > >> >> > > >> >> > TIA! > >> >> > > >> >> > > >> > > >> > > > > > >
