Hello,
Hmm.. I had not read through the code for the holt-winter implementation
with Graphite, I had assumed that the implementation had configuration
options. HW algorithm has a short and long term period, smoothing and
upper/lower confidence bands which are typically all configurable items
making HW very flexible. Though HW is NOT meant to be used on server
network interfaces of servers or even uplinks, it is meant to be used on
things that have clearly defined patterns or that aggregates enough data
flows that it creates a identifieable pattern, for example: An internet
gateway, an AS internet router.
Way back with Cricket, I would configure HW with a two week period, so
that it would compare this week versus last week, plus you define a
short term seasonality (period) that also defines if the algorithm will
adapt more easily to fast short term changes. In a corporate network
with three WAN carriers, 150 sites, a couple internet access points, and
one major datacenter, there were only 4 links that warranted the use of
HW with active alerts. Every other HW active time-series was
informational only.
For use with time-series that have more variability it is highly
suggested to have the HW anomaly be confirmed by a second metric before
generating a Warning or generating an Error and notification. That is
where the Shinken internal rules become useful (as we discussed a ways
back :-)).
Once again, let me reiterate that I think what you are doing is great
(re-using MongoDB), implementing anomaly detection and leveraging
internal rules. Just make sure that you keep the minimum amount of data
in MongoDB for the algorithm because the data is also typically exported
to PNP or Graphite for graphing.
Keep up the great work. I look forward in testing this feature.
As to the second point, using Graphite... What I am trying to say is
that, we (I include myself here) should really get a good shinken
graphite_check, that can use Graphites native functions to compute
alerts from short and mid-term trends in graphite. The wealth of API
functions can make it very powerful and flexible. There is an article on
the News page about this.. :-) Of course if we can cover directly in
Shinken some of the most common use cases that even better.
Cheers,
Francois
On 13-01-10 9:51 AM, nap wrote:
On Thu, Jan 3, 2013 at 3:12 PM, Francois Mikus <fmi...@acktomic.com
<mailto:fmi...@acktomic.com>> wrote:
Happy new year from this side of the ocean!
Hi :)
[...]
Graphite has built-in holt-winters support, trendlines, functions
for comparing a time period versus a previous time period, etc.
Yes, the main code is at
graphite/webapp/graphite/render/functions.py:def
holtWintersAnalysis(series, bootstrap=None). But like I said to
Hermann, HW is a nightmare to manage, even if we add to Graphite
parameters to it, instead of the hard coded parameters, you will have
to hack several times the parameters to get something near mine. The
key is the dimension where you smoothing (in fact it's not a
smoothing, it's an update of an average based on history). When you
got "quick" changes during the weeks (or 2 months are Graphite is
using it this cycle), you will have an O(N) error accumulation (each
average compute will add it's error from the previous value). For
values like disks it's not a problem, the variation is slow, so HW
will work without too much problems. But for things as network
consumption (and I think you know what I means here) what is the rule
for 8h55 and 9h05? None. All users are connection (in fact nearer to
9h20 because of coffee pause...).
That's why I used my algorithm, to bypass this O(N) errors problem.
One other thing, is the computation time. In Graphite, you can have
HW, but it will look at 2 months value for just one value GET. Here we
are doing something nearer to reporting than to classic transactional.
Aggregate/pre-compute (tell the name you want here) is the key. O(1)
versus O(N) computation -> I choose the first one :)
We should concentrate on integrating better calling of the
graphing functions of graphite and a (new) storage mechanism that
is only used for Shinken.
I don't understand this part. Do you means integrate this in
carbon-agregator? (to bypass the 0(n) thing). I personally don't have
time for this, but I can explain the algorithm to Graphite folks if
they want it ;)
What you are doing is great, but in the longer term it may be
better to focus on a backend that has more support and ecosystem
and that can be used natively within Shinken. I think the ideas
you have can be applied against Graphite to provide really really
good stuff!!
Yes, should be better of course. But I don't have time to learn
another tool code for hacking such thing. I need it now for a project,
and here it work well. That's ok for me :D
Jean
Hermann, there is already an existing Shinken module to send data
to Graphite (carbon), have you looked at it?
Cheers,
xkilian
------------------------------------------------------------------------------
Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current
with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
MVPs and experts. ON SALE this month only -- learn more at:
http://p.sf.net/sfu/learnmore_122712
_______________________________________________
Shinken-devel mailing list
Shinken-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/shinken-devel
------------------------------------------------------------------------------
Master HTML5, CSS3, ASP.NET, MVC, AJAX, Knockout.js, Web API and
much more. Get web development skills now with LearnDevNow -
350+ hours of step-by-step video tutorials by Microsoft MVPs and experts.
SALE $99.99 this month only -- learn more at:
http://p.sf.net/sfu/learnmore_122812
_______________________________________________
Shinken-devel mailing list
Shinken-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/shinken-devel
