On 16 February 2011 21:52, Elazar Leibovich <[email protected]> wrote:
> I never used it, but I saw ads about splunk for log management. > > http://www.splunk.com/ > Thanks. I'm aware of Splunk, our Customer Engineering people use it. However: 1. It's bloody expensive (they license by size of uncompressed data fed). 2. It doesn't seem to scale (it uses a single instance of an RDBMS database, can be very slow even before we start feeding the interesting log files into it. We currently limit its use to feeding of very specific CSV-formatted log files) 3. I'm not aware of automatic log analysis capability in it - it's nice to graph, draw statistics and maybe generate reports (when it finally comes up with a result), but not for alert generation. Cheers, --Amos > 2011/2/16 Amos Shapira <[email protected]> > >> Hello, >> >> >> As part of PCI-DSS compliance I'm working on (ref: >> http://en.wikipedia.org/wiki/Payment_Card_Industry_Data_Security_Standard), >> we need to implement automatic log file analysis and alerting. (It's also a >> Good Thing(TM) to have such a thing in place in general). >> >> LogWatch is not enough since it can't handle the amount of logs generated >> by our system (we generate ~6Gb of compressed HTTP daemon access log files >> every 24 hours alone, not to mention many other log files and more to come >> as we progress with PCI compliance) and still requires someone to manually >> go through its reports. >> >> Instead, I see many ads for commercial systems which can analyse log files >> in near real time and generate custom alerts about suspicious activity >> outside a learned activity pattern. These systems cost a fortune. >> >> On the other hand - I saw mentions of open-source system which dump log >> files onto a NoSQL database and achieve the same functionality with free >> tools. >> >> Alas - I lost the references for the later. >> >> Closest thing I found is Flume (https://github.com/cloudera/flume). >> Someone tells me that it also does the actual analysis but I don't see this >> mentioned on its web site. >> >> Does anyone else here have an idea about such systems? >> >> Thanks, >> >> --Amos >> Does anyone >> >> _______________________________________________ >> Linux-il mailing list >> [email protected] >> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il >> >> >
_______________________________________________ Linux-il mailing list [email protected] http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
