How many log lines do you think you would be collecting? We ended up using syslog-ng to receive all the data, and syslog-ng hands it off to splunk along with writing it to files. For the most part splunk is great, but sometimes its just easier to go through the logs from the command line. Splunk can get bogged down at times and can also get somewhat pricey. We have about 5 million log lines a day going into our logging setup (damn mail servers), and the most usable way to use splunk was to split the load up between multiple splunk servers so the query load is somewhat distributed.
Mike Devlin Manager of Operations boston..com On Fri, May 15, 2009 at 10:13 AM, Sean Lutner <[email protected]> wrote: > I'd second the recommendation for splunk. It's a fantastic product, is easy > to setup and would provide you with a way to aggregate and then easily > search over all your data. Aggregation is the easy part, the searching, > correlation, etc is not easy. I've implemented splunk at three different > places and am in the middle of a deployment currently. > > > On May 15, 2009, at 9:49 AM, seph wrote: > > Mike Sprague <[email protected]> writes: >> >> I work for a web hosting company with about a thousand linux servers. >>> We're discussing options on how to process the logs mainly from our mail >>> and web servers to make troubleshooting easier. We're not really >>> looking for long term storage; just a better way to be able to search >>> the logs to diagnose either specific customer issues, broad system >>> attacks, issues across a pool of servers or issues with a specific >>> server. >>> >> >> splunk would be the obvious commercial product for this space. >> >> http://www.opensyslog.com is a online hosted log consolidation tool. In >> beta. >> >> I've seen a variety of open source things in the log processing world. I >> don't think they're generally very flashy, and I'm not sure what the >> current favorites are. googling around syslog processing finds some >> suggestions. >> >> seph >> >> _______________________________________________ >> bblisa mailing list >> [email protected] >> http://www.bblisa.org/mailman/listinfo/bblisa >> >> > > _______________________________________________ > bblisa mailing list > [email protected] > http://www.bblisa.org/mailman/listinfo/bblisa >
_______________________________________________ bblisa mailing list [email protected] http://www.bblisa.org/mailman/listinfo/bblisa
