Re: [Architecture] Using HdrHistogram as a reservoir in Carbon Metrics

[Isuru Perera]

I completed the HdrHIstogram integration with Carbon Metrics. Please check
the commits in my personal repository for now [1]. I'll send a PR later.

Thanks!

[1] https://github.com/chrishantha/carbon-metrics/commits/master

On Mon, Oct 17, 2016 at 11:30 AM, Malith Jayasinghe 
wrote:

> Two important configuration parameters  of HDR (High Dynamic Range)
> histogram are 1) integer value range and 2) the value precision.
>
> For example, let's assume that  the range is 0 - 3,600,000,000 and the
> value precision = 3 ( (means that the accuracy of any value within the
> range will be better than 1/1000 of the value)
>
> Under these settings HDR histogram can  track and analyze the counts of
> observed response times ranging between 1 microsecond and 1 hour in
> magnitude with the following resolutions [1]:
>
> 1 microsecond resolution up to 1 millisecond,
> 1 millisecond (or better) resolution up to one second
> 1 second (or better) resolution up to 1,000 seconds.
> 3.6 seconds (or better) resolution up to 1 hour
>
> The memory footprint HDR depends on the dynamic range and precision chosen
> (As Isuru already pointed for a given dynamic range and precision the
> memory foot print is fixed). There is a way to compute the memory foot
> print using these values [1].
>
> [1] https://github.com/HdrHistogram/HdrHistogram
>
> On Fri, Oct 14, 2016 at 9:33 AM, Isuru Perera  wrote:
>
>> Hi,
>>
>> In Carbon Metrics [1], the TImers and Histograms use the default
>> reservoir provided by the Dropwizard Metrics [2] library.
>>
>> Currently there is no support to change this reservoir implementation in
>> Carbon Metrics. Therefore we are working on supporting a set of reservoirs
>> in Carbon Metrics.
>>
>> The HdrHistogram [3] is a popular histogram implementation and it has lot
>> of useful features for us. For more info, see [4]. One major advantage for
>> us is that HDR Histogram maintains a fixed cost in both space and time.
>>
>> When using HdrHistogram as reservoir, we can also a solve a problem with
>> the default Exponentially Decaying Reservoir when publishing metrics events
>> to WSO2 Data Analytics Server. The problem is explained in [5]. With HDR
>> Histogram, we can reset the snapshot when reporting to WSO2 DAS.
>>
>> We are going to integrate HDR Histogram into the next Carbon Metrics
>> release, which is based on Carbon 5.x
>>
>> We don't have plans to do any release of Carbon Metrics based on Carbon
>> 4.x. If anyone needs the HDR Histogram as a reservoir, we'll be able to do
>> changes in 1.x.x [6] branch and do a release.
>>
>> Thanks!
>>
>> Best Regards,
>>
>> [1] https://github.com/wso2/carbon-metrics
>> [2] http://metrics.dropwizard.io
>> [3] https://github.com/HdrHistogram/HdrHistogram
>> [4] http://hdrhistogram.github.io/HdrHistogram/
>> [5] http://taint.org/2014/01/16/145944a.html
>> [6] https://github.com/wso2/carbon-metrics/branches
>>
>> --
>> Isuru Perera
>> Associate Technical Lead | WSO2, Inc. | http://wso2.com/
>> Lean . Enterprise . Middleware
>>
>> about.me/chrishantha
>> Contact: +IsuruPereraWSO2 
>>
>
>
>
> --
> Malith Jayasinghe
>
> WSO2, Inc. (http://wso2.com)
> Email   : mali...@wso2.com
> Mobile : 0770704040
> Lean . Enterprise . Middleware
>



-- 
Isuru Perera
Associate Technical Lead | WSO2, Inc. | http://wso2.com/
Lean . Enterprise . Middleware

about.me/chrishantha
Contact: +IsuruPereraWSO2 
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Using HdrHistogram as a reservoir in Carbon Metrics

[Malith Jayasinghe]

Two important configuration parameters  of HDR (High Dynamic Range)
histogram are 1) integer value range and 2) the value precision.

For example, let's assume that  the range is 0 - 3,600,000,000 and the
value precision = 3 ( (means that the accuracy of any value within the
range will be better than 1/1000 of the value)

Under these settings HDR histogram can  track and analyze the counts of
observed response times ranging between 1 microsecond and 1 hour in
magnitude with the following resolutions [1]:

1 microsecond resolution up to 1 millisecond,
1 millisecond (or better) resolution up to one second
1 second (or better) resolution up to 1,000 seconds.
3.6 seconds (or better) resolution up to 1 hour

The memory footprint HDR depends on the dynamic range and precision chosen
(As Isuru already pointed for a given dynamic range and precision the
memory foot print is fixed). There is a way to compute the memory foot
print using these values [1].

[1] https://github.com/HdrHistogram/HdrHistogram

On Fri, Oct 14, 2016 at 9:33 AM, Isuru Perera  wrote:

> Hi,
>
> In Carbon Metrics [1], the TImers and Histograms use the default reservoir
> provided by the Dropwizard Metrics [2] library.
>
> Currently there is no support to change this reservoir implementation in
> Carbon Metrics. Therefore we are working on supporting a set of reservoirs
> in Carbon Metrics.
>
> The HdrHistogram [3] is a popular histogram implementation and it has lot
> of useful features for us. For more info, see [4]. One major advantage for
> us is that HDR Histogram maintains a fixed cost in both space and time.
>
> When using HdrHistogram as reservoir, we can also a solve a problem with
> the default Exponentially Decaying Reservoir when publishing metrics events
> to WSO2 Data Analytics Server. The problem is explained in [5]. With HDR
> Histogram, we can reset the snapshot when reporting to WSO2 DAS.
>
> We are going to integrate HDR Histogram into the next Carbon Metrics
> release, which is based on Carbon 5.x
>
> We don't have plans to do any release of Carbon Metrics based on Carbon
> 4.x. If anyone needs the HDR Histogram as a reservoir, we'll be able to do
> changes in 1.x.x [6] branch and do a release.
>
> Thanks!
>
> Best Regards,
>
> [1] https://github.com/wso2/carbon-metrics
> [2] http://metrics.dropwizard.io
> [3] https://github.com/HdrHistogram/HdrHistogram
> [4] http://hdrhistogram.github.io/HdrHistogram/
> [5] http://taint.org/2014/01/16/145944a.html
> [6] https://github.com/wso2/carbon-metrics/branches
>
> --
> Isuru Perera
> Associate Technical Lead | WSO2, Inc. | http://wso2.com/
> Lean . Enterprise . Middleware
>
> about.me/chrishantha
> Contact: +IsuruPereraWSO2 
>



-- 
Malith Jayasinghe

WSO2, Inc. (http://wso2.com)
Email   : mali...@wso2.com
Mobile : 0770704040
Lean . Enterprise . Middleware
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture