On Fri, Jan 30, 2009 at 1:58 AM, Antony Blakey <[email protected]> wrote: > > On 30/01/2009, at 4:27 PM, Paul Davis wrote: > >> On Fri, Jan 30, 2009 at 12:32 AM, Antony Blakey <[email protected]> >> wrote: >>> >>> On 30/01/2009, at 9:56 AM, Paul Davis wrote: >>> >>>> The way that stats are calculated currently with the dependent >>>> variable being time could cause some issues in implementing more >>>> statistics. With my extremely limited knowledge of stats I think >>>> moving that to be dependent on the number of requests might be better. >>>> This is something that hopefully someone out there knows more about. >>>> (This is in terms of "avg for last 5 minutes" vs "avg for last 100 >>>> requests", (the later of the two making stddev type stats >>>> calculateable on the fly in constant memory.) >>> >>> The problem with using # of requests is that depending on your data, each >>> request may take a long time. I have this problem at the moment: 1008 >>> documents in a 3.5G media database. During a compact, the status in >>> _active_tasks updates every 1000 documents, so you can imagine how useful >>> that is :/ I thought it had hung (and neither the beam.smp CPU time nor >>> the >>> IO requests were a good indicator). I spent some time chasing this down >>> as a >>> bug before realising the problems was in the status granularity! >>> >> >> Actually I don't think that affects my question at all. It may change >> how we report things though. As in, it may be important to be able to >> report things that are not single increment/decrement conditions but >> instead allow for adding arbitrary floating point numbers to the >> number of recorded data points. > > I think I have the wrong end of the stick here - my problem was with the > granularity of updates, not with the basis of calculation. >
Heh. Well, we can only measure what we know. And in the interest of simplicity I think the granularity is gonna have to stick to pretty much per request. Also you're flying with 300 MiB docs? perhaps its time to chop or store in FTP? > Antony Blakey > ------------- > CTO, Linkuistics Pty Ltd > Ph: 0438 840 787 > > It is no measure of health to be well adjusted to a profoundly sick society. > -- Jiddu Krishnamurti > > >
