On 13-01-03 11:23 AM, Hermann Lauer wrote: > On Thu, Jan 03, 2013 at 09:12:41AM -0500, Francois Mikus wrote: >> Hermann, there is already an existing Shinken module to send data to >> Graphite (carbon), have you looked at it? > Yes, when we started to test graphite as storage for data instead of > rrds. Before we used rrd, hence the name shinken2rrd. > >>> perfdata from shinken into carbon-cache (besides other sources using the >>> same minimalistic >>> udp protocol). >>> >>> If there is interrest in adding the two broker modules (as asked some time >>> ago) I will > The modules are used here since over a year ago (so before the specialialized > graphite one where available) and are using templates to allow generating > free format (text) udp packets. So they are not specialised for graphite > in any way. > > One of the problems we look here at the moment is how to get two kind > of consolidation functions in graphite/whisper - rrd's can build up for > example "max" and "average" rra's from one primary datapoint. Of course > input data could be duplicated with the templates in our broker modules, > but a solution for duplication to whisper databases inside the carbon-cache > would be better as then our packet formats from other sources besides shinken > needs not to be changed. Well... That is one key difference between rrd and whisper(graphite) databases.
Whisper is one metric per file and one consolidation function. So you have to choose, min, max, avg, median, etc. The obvious solution for a statistician is to use a better consolidation function so that you do not need multiple ones. This may be possible with whisper databases but I am not convinced. With the Ceres database format which allocates dynamically database space, it is possible to use more advanced statistical consolidation functions based on changing slopes. I will post the link to the algorithm description. This is something I aim to get a statistics student to work on in the future as a masters project or as a summer student project. In the mean time, you can compute the min/max/average/etc dynamically at graph generation based on the unconsolidated tiime-series data. But once consolidated, you obviously have to choose one method. Having N consolidation methods for a single data point is little sad. As it doubles the amount of disk space used, even considering the is only a single timestamp (in rrd) and multiple timestamps in Graphite (.wsp). The right place for doing determining multiple consolidation methods in the current architecture is upstream from carbon-cache to keep it consistent with how Graphite works. Changing it in carbon-cache would take a lot more testing and risk breaking something, IMO. So for now, do it upstream in carbon aggregator or in Shinken module(shinken2rrd, or Shinken Graphite module). Which is not a big problem. The Graphite module for Shinken can send data using pickled batches for efficient communications and can deal with disconnections, it can easily be extended to do a little magic. Sending more data to Graphite is not the issue, but it will mean more data points, 2x or even 3x the number of data points to update and more disk size. This becomes a problem when you start to exceed 300K basic data points. On my end that is how I will deal with it, bidding my time until a better consolidation algorithm is available. I hope this helps put things in perspective, Francois > Thanks for any idea on that, Francois. > Greetings > Hermann > ------------------------------------------------------------------------------ Master HTML5, CSS3, ASP.NET, MVC, AJAX, Knockout.js, Web API and much more. Get web development skills now with LearnDevNow - 350+ hours of step-by-step video tutorials by Microsoft MVPs and experts. SALE $99.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122812 _______________________________________________ Shinken-devel mailing list Shinken-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/shinken-devel
