I've been using logstash + elasticsearch for a while now with gridinit.com rolling my own front end to search the data. The tire gem is useful for this. Quite frankly I love logstash + elasticsearch. The latter scales/distributes fairly well (in same geo area)
I also implemented a similar setup but with kibana for the front end for a client who wanted to index logs from 6 app servers and a mammoth PG server, all up which is indexing about 1.8M events per hour. That was done on a single server. So yeah, forget splunk, look at logstash + elasticsearch and kibana On Thursday, October 25, 2012 2:10:56 PM UTC+11, Michael Pearson wrote: > > I haven't worked with splunk since it first came out in 2006 or so. I know > mainly that it's quite expensive. > > Have you checked out logstash? http://logstash.net/ > > We need to pull in some log file aggregation & management at BikeExchange, > so I'll be interested to see where this thread goes. > > On Thu, Oct 25, 2012 at 2:00 PM, Craig Read <[email protected]<javascript:> > > wrote: > >> I'm currently parsing some large log files and populating a rails db with >> 'key' pieces of information from those logs via ActiveResource. >> They're not 'web logs', and each line can have totally different data >> (including how the data is structured), so most of the tools I see around >> aren't applicable to my particular use case. Also, only about 0.05% of the >> data is actually relevant, so I'm looking for a 'better way'. >> >> I did see mention of 'treating logs as data' (and Splunk in particular) >> on the latest TW Tech Radar. >> Is anybody using Splunk <http://www.splunk.com/> with (or without) the >> ruby-splunk <https://github.com/beezly/ruby-splunk> gem? >> If so, do you recommend it, or is there a better approach to doing this? >> >> Cheers, >> >> -- >> Craig Read >> >> @Catharz >> https://github.com/Catharz >> http://stackoverflow.com/users/158893/catharz >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Ruby or Rails Oceania" group. >> To post to this group, send email to [email protected]<javascript:> >> . >> To unsubscribe from this group, send email to >> [email protected] <javascript:>. >> For more options, visit this group at >> http://groups.google.com/group/rails-oceania?hl=en. >> > > > > -- > Michael Pearson > > > -- You received this message because you are subscribed to the Google Groups "Ruby or Rails Oceania" group. To view this discussion on the web visit https://groups.google.com/d/msg/rails-oceania/-/76h6YkeaaLIJ. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/rails-oceania?hl=en.
