If you don't have those sort of $$ and are happy to trust your logs to a cloud service, I've had a great experience with Papertrail.
It aggregates your logs, makes them easily searchable via a web interface, and even lets you set up notification hooks to alert you if logs that match a particular regex occur. You can archive your logs to S3. Oh, and you can just sit and watch the tail of your aggregated logs via a web interface if that sort of thing interests you (I swear it helps you grok an app, lol). It is free for less than 100MB of logs/month. I've never got above $7 for 1Gb logs/month, but it does get more expensive quickly if you exceed a few Gb of logs per month. Before I tried Papertrail, I tried Loggly and found it to be nowhere near as useful. If anyone is interested, I have Chef recipes to set up Papertrail that I'd be happy to share with folks. Not that it is hard, quite the opposite actually. Cheers, Chris @cmaitchison On 26/10/2012, at 8:58 AM, Julian Doherty <[email protected]> wrote: Splunk is awesome. We're using it at Envato, and I used it a bit at Lonely Planet. Super expensive (five figure+ $$$ for most installs), but does pretty much anything you want it to do in terms of aggregating + searching logs. Is invaluable for debugging and diagnosing production issues in realtime that would have been impossible before. You need to do a bit of setup and learning to get the most value out of it. Main thing is setting it up so it understands the log format(s) of your app(s) and can parse info out of it. It does a lot of things out of the box, and there are plugins for Rails logging etc. Cheers Julian @madlep On 25/10/2012, at 2:00 PM, Craig Read wrote: > I'm currently parsing some large log files and populating a rails db with > 'key' pieces of information from those logs via ActiveResource. > They're not 'web logs', and each line can have totally different data > (including how the data is structured), so most of the tools I see around > aren't applicable to my particular use case. Also, only about 0.05% of the > data is actually relevant, so I'm looking for a 'better way'. > > I did see mention of 'treating logs as data' (and Splunk in particular) on > the latest TW Tech Radar. > Is anybody using Splunk with (or without) the ruby-splunk gem? > If so, do you recommend it, or is there a better approach to doing this? > > Cheers, > > -- > Craig Read > > @Catharz > https://github.com/Catharz > http://stackoverflow.com/users/158893/catharz > > > -- > You received this message because you are subscribed to the Google Groups > "Ruby or Rails Oceania" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/rails-oceania?hl=en. -- You received this message because you are subscribed to the Google Groups "Ruby or Rails Oceania" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/rails-oceania?hl=en. -- You received this message because you are subscribed to the Google Groups "Ruby or Rails Oceania" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/rails-oceania?hl=en.
