Hi,

*@Fabian:* Thank you very much! Is the summit open, can I join?

*@Bryan:*
I think I would make the decision if it would make sense for the Prometheus 
Server to be able to process the Exemplars on _count first. If so, then I 
would look into the different clients and their APIs. 

For Histograms, I don't think this should result in a client API change, I 
think this should only affect the behavior of the implementation. Also, I 
think source, binary, and behavioral compatibility can be kept. I think it 
does not matter where the Exemplar is coming from (directly from the user 
or from the sampler), the interesting part happens after the implementation 
got the Exemplar. In the case of Histograms, this would mean not just 
updating the reference of the Exemplar of the current bucket but also 
updating an extra reference to the latest Exemplar (that actually belongs 
to _count). So the histogram would hold references to N+1 exemplars where N 
is the number of buckets and the +1 Exemplar is the latest recorded one.

For Summaries, this needs a client API change (an addition) since right now 
Summaries do not have Exemplars support. I think this should be very 
similar to the Exemplars support of Counters.

I would like to call out two things:
- I'm not an expert of any of the Prometheus clients (I only used the Java 
and the JavaScript clients).
- I think I would not even be affected by the changes of the client APIs 
since Micrometer is not using these, it directly creates a 
Collector.MetricFamilySamples.Sample instead (that can accept Exemplars as 
of today). So if the Prometheus server could process Exemplars on _count, I 
think my use-case should be covered. But I would definitely add the support 
to the client APIs too so that users who use the clients can enjoy these 
features.
 
Thanks,
Jonatan

On Friday, November 4, 2022 at 1:09:22 AM UTC-7 bjbo...@gmail.com wrote:

> >>> Also, what do you think about supporting _count in Histograms (since 
> Histogram extends Summary with buckets)?
> >> How exactly would you change this while remaining backwards-compatible 
> for existing users?
> > As far as I can see, adding Exemplars to _count should be a 
> backward-compatible change (it is an addition).
>
> I would like to see what you can see.  Please spell it out for me.
>
> The current API on Histogram is ObserveWithExemplar(), and it adds the 
> exemplar to a _bucket metric.
> Would you change the behaviour of that API?  Would you add a new API?  
>
> Bryan
>
> On Friday, 4 November 2022 at 05:46:39 UTC jonatan...@gmail.com wrote:
>
>> Hi,
>>
>> Sorry for the late reply let me go on-by-one, please bear with me. :)
>>
>> *>I recommend taking that question to an OpenMetrics list.* 
>> Thanks, I will open a thread there too based on the discussion here.
>>
>> *>Prometheus could independently decide to go beyond what OpenMetrics 
>> says*
>> Since Micrometer uses the Prometheus Java Client this would solve the 
>> issue for us but if it makes sense it would be great to have it 
>> standardized later(?) (in OM).
>>
>> *>Maybe your point is that exemplars are tied to the _bucket metrics and 
>> not the _count metric?*
>> Yes, that's exactly what I'm saying with the caveat that I think this 
>> would be useful for Summaries too not only Histograms.
>>
>> *>How exactly would you change this while remaining backwards-compatible 
>> for existing users?*
>> As far as I can see, adding Exemplars to _count should be a 
>> backward-compatible change (it is an addition).
>> Would users be broken because of this?
>>
>> *>Perhaps the upcoming "native histograms" or "sparse histograms" feature 
>> will suit what you need?*
>> I'm not sure but I'm not only talking about histograms, I'm also talking 
>> about Summary.
>>
>> *>In an OM Histogram, the +Inf bucket fulfills exactly the same function*
>>
>>
>> *as the _count (spec says: "The +Inf bucket counts all requests.") Soif 
>> you would like an examplar on the _count of a Histogram, you can aswell use 
>> an exemplar on the +Inf bucket.*
>>
>> I think I disagree with the second sentence. Let's say you have an 
>> application where processing the first request is significantly slower than 
>> the rest (lazy init, populating caches, GC, establishing connections, 
>> etc.). In this environment (I think this is true for lots of apps nowadays) 
>> it can easily happen that the +Inf bucket will be populated with an 
>> Exemplar for the first request and it will never get updated because the 
>> app will never be as slow as it was for the first request. Also, nothing 
>> guarantees that the +Inf bucker will have an exemplar, maybe the processing 
>> was faster than that. As far as I understand, exemplars are not like 
>> cumulative "le" counters so incrementing a bucket does not mean updating an 
>> exemplar (quite the opposite, maybe all of the buckets will be incremented 
>> but only one will get a new Exemplar). This is also true for apps that can 
>> get significantly faster over time (i.e.: JIT).  I think a solution here 
>> would be give_me_the_last_updated_bucket(my_histogram) or adding 
>> Exemplar to _count.
>>
>> *>Side note: In Java this would be particularly useful because the 
>> popular Spring Boot framework exposes a Summary 
>> http_server_requests_seconds by default that look like this (no quantiles, 
>> just _count and _sum)*
>> This was actually one of the drivers of this request, users are asking 
>> for this from us. I also find it extremely useful: without this I need to 
>> create an additional counter which does not seem right since I already have 
>> one.
>>
>> Thanks,
>> Jonatan
>>
>> On Tuesday, October 18, 2022 at 6:29:27 AM UTC-7 fab...@fstab.de wrote:
>>
>>> Side note: In Java this would be particularly useful because the popular 
>>> Spring Boot framework exposes a Summary http_server_requests_seconds by 
>>> default that look like this (no quantiles, just _count and _sum):
>>>
>>> # HELP http_server_requests_seconds  
>>> # TYPE http_server_requests_seconds summary
>>> http_server_requests_seconds_count{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/",}
>>>  1.0
>>> http_server_requests_seconds_sum{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/",}
>>>  1.014687278
>>>
>>> I think this is pretty useful, you can get request rates and error rates 
>>> out of it. If Prometheus / OpenMetrics had support for Exemplars on the 
>>> _count, users could find example traces per HTTP status and URI.
>>>
>>> Fabian
>>>
>>> On Tue, Oct 18, 2022 at 3:05 PM Bjoern Rabenstein <bjo...@rabenste.in> 
>>> wrote:
>>>
>>>> On 06.10.22 14:45, 'Fabian Stäber' via Prometheus Developers wrote:
>>>> > 
>>>> > Great question from the CNCF Slack: What's the reason why we don't 
>>>> allow 
>>>> > Exemplars for _count in Summary metrics?
>>>> > 
>>>> > What do you think? Any reason why Exemplars don't work in _count in 
>>>> > Summaries? Would that be something we could consider supporting?
>>>>
>>>> The _count of a Summary _and_ the _count of a Histogram (both
>>>> conventional as well as the new native ones) is essentially a counter
>>>> within the larger "structured" metric of a Summary/Histogram.
>>>>
>>>> From that perspective, it should have the option of attaching an
>>>> examplar, as a regular Counter has, too.
>>>>
>>>> My speculation why it doesn't in OpenMetrics:
>>>>
>>>> In an OM Histogram, the +Inf bucket fulfills exactly the same function
>>>> as the _count (spec says: "The +Inf bucket counts all requests.") So
>>>> if you would like an examplar on the _count of a Histogram, you can as
>>>> well use an exemplar on the +Inf bucket.
>>>>
>>>> That obviously doesn't help in the case of a Summary, but I guess the
>>>> rationale is that Histograms are generally to be preferred over
>>>> Summaries, and therefore didn't get the thourough treatment when it
>>>> came to exemplars.
>>>>
>>>>
>>>> However, even if you really dislike the precalculated quantiles in
>>>> Summaries, there is still the case of a Summary without quantiles. I
>>>> think adding exemplars to such a Summary is as much needed as adding
>>>> exemplars to any regular Counter.
>>>>
>>>> -- 
>>>> Björn Rabenstein
>>>> [PGP-ID] 0x851C3DA17D748D03
>>>> [email] bjo...@rabenste.in
>>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/aeb85db0-20e8-4365-a3d5-6bf956ce1cf9n%40googlegroups.com.

Reply via email to