Hi James, I'd like to make a longer comment here to make it easier to discuss.
> [...] > > Note the proposal to alter how Timer metrics are exposed in an incompatible > way (I argue this is OK because you can't really make use of these metrics > now). I am not sure I follow your argument around `Timer`. It is similar to a gauge caching the last value and an associated statistics calculated from a time series. I have never used Prometheus, but a brief look at the Prometheus docs seems to suggest that a `Timer` could be mapped onto a Prometheus summary type with minimal modifications (namely, by adding a `sum` value that you propose as sole replacement). I believe that exposing statistics is useful, and moving all `Timer` metrics to counters (cumulative value and number of samples) would leads to information loss. Since most of your criticism of `Timer` is about it its associated statistics, maybe we can make fixes to libprocess' `TimeSeries` and the derived `Statistics` to make them more usable. Right now `Statistics` seems to be more apt for dealing with timing measurements where one probably worries more about the long tail of the distribution (it only exposes the median and higher percentiles). It seems that if one would e.g., make the exposed percentiles configurable, it should be possible to expose a useful characterization of the underlying distribution (think: box plot). It might be that one would need to revisit how `TimeSeries` sparsifies older data to make sure the quantiles we expose are meaningful. > First, note that the “allocator/mesos/allocation_run_ms/count” sample is not > useful at all. It has the semantics of a saturating counter that saturates at > the size of the bounded time series. To address this, there is another metric > “allocator/mesos/allocation_runs”, which tracks the actual count of > allocation runs (3161331.00 in this case). If you plot this counter over time > (ie. as a rate), it will be zero for all time once it reaches saturation. In > the case of allocation runs, this is almost all the time, since 1000 > allocations will be performed within a few hours. While `count` is not a useful measure of the behavior of the measured datum, it is critical to assess whether the derived statistic is meaningful (sample size). Like you write, it becomes less interesting once enough data was collected. > Finally, while the derived statistics metrics can be informative, they are > actually less expressive than a raw histogram would be. A raw histogram of > timed values would allow an observer to distinguish cases where there are > clear performance bands (e.g. when allocation times cluster at either 15ms or > 200ms), but the percentile statistics obscure this information. I would argue that is more a problem of `Statistics` only reporting percentiles from the far out, large value tail. Would e.g., the reported percentiles be placed more evenly it should be possible to recognize bands. After all percentiles are just samples from the cumulative distribution from which one can derived the underlying distribution (with some resolution) by taking a derivative. Note that a poorly binned and ranged histogram can obscure features as well. Reporting percentiles/quantiles has the advantage of adapting to the data automatically. Cheers, Benjamin
