Oren, I am glad that you bring up this issue. As you know, I have been working on the problem of logging and analytics for Heroku apps for a while now, and have recently finished building a modular architecture to support the capture, analysis, and acting on logging data --beyond the passive reporting of low-level events, to encompass user-defined, application-level events and actions.
In short, our solution provides 'logging and analytics' as a service. Through a simple gem and our hosted service, you can: * automatically record web request information and have it sent (over a secure channel) to our servers, * programmatically log any kind of semi-structured data (arbitrary schema-less key/value pairs), anywhere in your application, and * have full access to your data through a RESTful API and a powerful query language On top of this basic architecture we have built powerful tools to analyze and report on the recorded data --addressing the issues raised in this thread by Jamie and Arun, and supporting many other use cases as well. For example, we have: * a tool to 'tail -f' whatever your applications are recording while filtering out unwanted records (e.g. list only 404s) * a tool that lets you group your data by different criteria and compute totals over the groups (e.g., count of searches performed, by search term) * an application to create a private RSS feed with the log entries that match arbitrary criteria (e.g, new accounts as they are created, along with relevant information about them) * dashboards and charts to get a quick picture of what's going on in your applications (e.g., number of requests by unit of time, or number of accounts by gender, etc.) Thanks to the RESTful API and the powerful query language, the architecture is highly extensible and makes it easy to write scripts that actively monitor the data and react to it. We have also written, for example, * a tiny Rack middleware that records HTTP_X_HEROKU_QUEUE_DEPTH, and a separate monitoring application that runs regular queries against the database. If in a given interval it finds that the log values exceed a user-defined value, the script makes calls to the Heroku API to start new dynos --providing automatic up-scaling for your apps. * a monitoring tool that sends you email whenever the number of failed logging attempts per unit of time exceeds a certain threshold. We are ready to begin a private beta, and are looking for volunteers. If you're interested, please send me an email to [email protected]. I'll be delighted to give out free accounts and see what people think of it and, more importantly, what solutions and ideas you build with our solution. Thanks, - Agustin -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
