On 14 March 2011 20:42, Phil Steitz <[email protected]> wrote: > On 3/14/11 12:34 PM, Luc Maisonobe wrote: >> Le 14/03/2011 15:33, Benson Margulies a écrit : >>> Please excuse the following ignorant question. >>> >>> I want to maintain summary statistics of a rate. At each 'event', I >>> know the number of characters and the time it took to process them, >>> and I want to maintain summary statistics for the rate of >>> chars/second. I imagine that I'm missing something basic, but I don't >>> see how to do this. >> You should define some windows width, either in terms of a time span >> (all events in the last n seconds) or in terms of number of events (last >> n events). >> >> In [math], we do not provide (yet) anything for maintaining such a data >> structure, you'll have to maintain the events in this slot by yourself, >> with something similar to a FIFO. > > I am not sure I understand what the problem is exactly, but if what > you need is simply "rolling" statistics, where a dataset of 0,...,n > values are maintained with the newest values replacing the oldest, > we do in fact support that in > o.a.c.math.descriptive.DescriptiveStatistics.
In JMeter we needed to display long running percentiles without using excess memory, and someone came up with the idea of using buckets for ranges of values. So instead of keeping details on each sample elapsed time, we increment the count for the appropriate bucket. If the range of values is too large to use a single bucket for each value, each bucket can represent a range of values. These ranges can potentially be non-uniform though that does complicate the calculations. JMeter actually uses a TreeMap for the values and counts - the values need to be sorted in order to calculate percentiles. Depending on the data-set, it might be possible to used fixed arrays instead of the TreeMap. > Phil >> When you have your data available, each time a new event is added or >> removed from the ones that belong to the window, you can fetch compute >> the statistics you want on this data (min, max, mean, median, standard >> deviation ...) and wait for next addition/removel to recompute it again. >> >> Another thing we discussed some months ago (but did not implement yet) >> is a way to compute an approximation of percentiles in a flow of data >> without storing them. There is an interesting algorithm for it that was >> developed for the needs of telecommunication companies, I think it may >> be of interest to you. This would provide results like : currently 95% >> of the characters are processed in n milliseconds. would you be >> interested in us implementing this feature ? >> >> Luc --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
