[jira] [Commented] (MESOS-6918) Prometheus exporter endpoints for metrics
[
https://issues.apache.org/jira/browse/MESOS-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484034#comment-16484034
]
Kevin Klues commented on MESOS-6918:
Which endpoints is this targeted for specifically?
Useful metrics are kind of spread out all over the place at the moment.
{noformat}
masters:
/metrics/snapshot
/state -- per agent resource usage stats are in here
agents:
/metrics/snapshot
/monitor/statistics
{noformat}
> Prometheus exporter endpoints for metrics
> -
>
> Key: MESOS-6918
> URL: https://issues.apache.org/jira/browse/MESOS-6918
> Project: Mesos
> Issue Type: Bug
> Components: statistics
>Reporter: James Peach
>Assignee: James Peach
>Priority: Major
>
> There are a couple of [Prometheus|https://prometheus.io] metrics exporters
> for Mesos, of varying quality. Since the Mesos stats system actually knows
> about statistics data types and semantics, and Mesos has reasonable HTTP
> support we could add Prometheus metrics endpoints to directly expose
> statistics in [Prometheus wire
> format|https://prometheus.io/docs/instrumenting/exposition_formats/],
> removing the need for operators to run separate exporter processes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-6918) Prometheus exporter endpoints for metrics
[ https://issues.apache.org/jira/browse/MESOS-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389066#comment-16389066 ] James Peach commented on MESOS-6918: > [~jamespeach], do you think it's feasible to target some of this work for 1.6? Yes I think it's doable. > Prometheus exporter endpoints for metrics > - > > Key: MESOS-6918 > URL: https://issues.apache.org/jira/browse/MESOS-6918 > Project: Mesos > Issue Type: Bug > Components: statistics >Reporter: James Peach >Assignee: James Peach >Priority: Major > > There are a couple of [Prometheus|https://prometheus.io] metrics exporters > for Mesos, of varying quality. Since the Mesos stats system actually knows > about statistics data types and semantics, and Mesos has reasonable HTTP > support we could add Prometheus metrics endpoints to directly expose > statistics in [Prometheus wire > format|https://prometheus.io/docs/instrumenting/exposition_formats/], > removing the need for operators to run separate exporter processes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MESOS-6918) Prometheus exporter endpoints for metrics
[ https://issues.apache.org/jira/browse/MESOS-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388599#comment-16388599 ] Zhitao Li commented on MESOS-6918: -- [~jamespeach], do you think it's feasible to target some of this work for 1.6? We are interested in reusing some functionalities here. > Prometheus exporter endpoints for metrics > - > > Key: MESOS-6918 > URL: https://issues.apache.org/jira/browse/MESOS-6918 > Project: Mesos > Issue Type: Bug > Components: statistics >Reporter: James Peach >Assignee: James Peach >Priority: Major > > There are a couple of [Prometheus|https://prometheus.io] metrics exporters > for Mesos, of varying quality. Since the Mesos stats system actually knows > about statistics data types and semantics, and Mesos has reasonable HTTP > support we could add Prometheus metrics endpoints to directly expose > statistics in [Prometheus wire > format|https://prometheus.io/docs/instrumenting/exposition_formats/], > removing the need for operators to run separate exporter processes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MESOS-6918) Prometheus exporter endpoints for metrics
[
https://issues.apache.org/jira/browse/MESOS-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195412#comment-16195412
]
James Peach commented on MESOS-6918:
Summary from our discussion:
- retain the existing {{Timer}} value that holds the duration of the last sample
- capture total duration (monotonic sum) for {{Timer}}s in their time series
- capture total sample count for {{Timer}}s in their time series
- replace the {{Semantics}} enum with a {{monotonic}} marker (enum or bool or
something)
> Prometheus exporter endpoints for metrics
> -
>
> Key: MESOS-6918
> URL: https://issues.apache.org/jira/browse/MESOS-6918
> Project: Mesos
> Issue Type: Bug
> Components: statistics
>Reporter: James Peach
>Assignee: James Peach
>
> There are a couple of [Prometheus|https://prometheus.io] metrics exporters
> for Mesos, of varying quality. Since the Mesos stats system actually knows
> about statistics data types and semantics, and Mesos has reasonable HTTP
> support we could add Prometheus metrics endpoints to directly expose
> statistics in [Prometheus wire
> format|https://prometheus.io/docs/instrumenting/exposition_formats/],
> removing the need for operators to run separate exporter processes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MESOS-6918) Prometheus exporter endpoints for metrics
[
https://issues.apache.org/jira/browse/MESOS-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194713#comment-16194713
]
James Peach commented on MESOS-6918:
{quote}
...all metric types (potentially future ones as well) ... But are we sure about
this is the right/only criterion? (The examples cited in the design doc don't
consistently define this and none defines it as "semantics") Could there be
other dimensions / features to classify metrics?
{quote}
Obviously there could be a large number of ways to classify how metrics behave.
However 2 is sufficient for all the metrics we have today. I am not trying to
boil the ocean here and invent the high temple of metrics systems; I'm just
trying to improve the one we already have. I think you are willfully reading
too much into this.
{quote}
We already have metric types that are called Counter and Gauge
{quote}
As I explained in person (and maybe in the design doc?), the class type is not
the preferred implementation because there are more class types than semantic
types. It's perfectly reasonable to use {{Counter}} to publish a metric that
has {{GAUGE}} semantics, and obviously we already use {{GAUGE}} in both
{{Counter}} and {{Timer}} class. It's pretty clear that we should separate the
implementation classes from the logical model.
{quote}
keep the MetricsProcess logic generic.
{quote}
{{MetricsProcess}} is not generic. It emits a specific format. I'm fine with
shuffling code around, but there's so little it isn't worth a separate
abstraction IMHO.
{quote}
I think we can keep the meaning the existing field Timer.value() (the last
sampled value).
{quote}
As I explained in the doc, this value is not helpful at all. I don't think
there's any point in keeping it.
{quote}
provide Prometheus its required info.
{quote}
Making {{Timer}} a {{COUNTER}} is not specific to Prometheus, it is necessary
to make {{Timer}} values useful at all.
> Prometheus exporter endpoints for metrics
> -
>
> Key: MESOS-6918
> URL: https://issues.apache.org/jira/browse/MESOS-6918
> Project: Mesos
> Issue Type: Bug
> Components: statistics
>Reporter: James Peach
>Assignee: James Peach
>
> There are a couple of [Prometheus|https://prometheus.io] metrics exporters
> for Mesos, of varying quality. Since the Mesos stats system actually knows
> about statistics data types and semantics, and Mesos has reasonable HTTP
> support we could add Prometheus metrics endpoints to directly expose
> statistics in [Prometheus wire
> format|https://prometheus.io/docs/instrumenting/exposition_formats/],
> removing the need for operators to run separate exporter processes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MESOS-6918) Prometheus exporter endpoints for metrics
[
https://issues.apache.org/jira/browse/MESOS-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194161#comment-16194161
]
Yan Xu commented on MESOS-6918:
---
[~bmahler] let's chat about the reviews? [~jpe...@apache.org] and I have
already discussed this offline and I have added comments to the design doc and
the reviews. Here's the summary:
- I am not convinced about the newly introduced {{enum Semantics \{COUNTER,
GAUGE\}}}. We already have metric *types* that are called {{Counter}} and
{{Gauge}} and I think people could be confused about Counter the semantics and
Counter the type, for example.
-- I understand that the semantics is supposed to help express:
bq. {{Timer}}'s value should be cumulative / monotonically increasing
(because it's more useful that way, as explained in the design doc) but this
enum seems to try to suggest that all metric types (potentially future ones as
well) can and should be classified into one of the two buckets. But are we sure
about this is the right/only criterion? (The examples cited in the design doc
don't consistently define this and none defines it as "semantics") Could there
be other dimensions / features to classify metrics? To me
{{s/Semantics/Monotonicity/}} would have been clearer but I am not sure about
the usefulness of that either.
-- The use of this enum right now is just to pass the metric type info down to
the Prometheus formatter. We can just define {{enum Type \{COUNTER, GAUGE,
TIMER\}}} and pass it down.
- I hope we confine the Prometheus logic in a
`metrics/formatters/prometheus.hpp|cpp` file and keep the {{MetricsProcess}}
logic generic.
- I think we can keep the meaning the existing field {{Timer.value()}} (the
last sampled value). We can add a new field {{sum}} in the {{TimeSeries}}
alongside the new {{total}} (can we name it something like {{totalCount}}?) to
provide Prometheus its required info.
> Prometheus exporter endpoints for metrics
> -
>
> Key: MESOS-6918
> URL: https://issues.apache.org/jira/browse/MESOS-6918
> Project: Mesos
> Issue Type: Bug
> Components: statistics
>Reporter: James Peach
>Assignee: James Peach
>
> There are a couple of [Prometheus|https://prometheus.io] metrics exporters
> for Mesos, of varying quality. Since the Mesos stats system actually knows
> about statistics data types and semantics, and Mesos has reasonable HTTP
> support we could add Prometheus metrics endpoints to directly expose
> statistics in [Prometheus wire
> format|https://prometheus.io/docs/instrumenting/exposition_formats/],
> removing the need for operators to run separate exporter processes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MESOS-6918) Prometheus exporter endpoints for metrics
[
https://issues.apache.org/jira/browse/MESOS-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158964#comment-16158964
]
James Peach commented on MESOS-6918:
{quote}
I understand that complex Prometheus metric types such as summary require some
more data than what is currently provided so we need to add them somewhere. But
they should be added to Mesos' existing "core" metrics classes only if they are
generic (backwards compatible) improvements that make sense regardless of the
Prometheus support. I believe this is indeed your goal but we need to
articulate how current timer/statistics modeling is lacking/wrong.
{quote}
The {{Statistics}} class currently generates enough information to support
Prometheus summary metrics. I added some discussion about {{Timer}} metrics to
the doc.
{quote}
There are some patches that remove history from simple metrics because it
doesn't make sense. Should the history then be put in another base type that
Counter and Gauge don't derive from?
{quote}
I don't see any need for that right now.
{quote}
After the above is done, is Prometheus merely a specific format? If that's the
case, can we encapsulate the formatting logic into a formatter class/method
instead of the main metric endpoint actor?
{quote}
Yes we might be able to do something along those lines.
> Prometheus exporter endpoints for metrics
> -
>
> Key: MESOS-6918
> URL: https://issues.apache.org/jira/browse/MESOS-6918
> Project: Mesos
> Issue Type: Bug
> Components: statistics
>Reporter: James Peach
>Assignee: James Peach
>
> There are a couple of [Prometheus|https://prometheus.io] metrics exporters
> for Mesos, of varying quality. Since the Mesos stats system actually knows
> about statistics data types and semantics, and Mesos has reasonable HTTP
> support we could add Prometheus metrics endpoints to directly expose
> statistics in [Prometheus wire
> format|https://prometheus.io/docs/instrumenting/exposition_formats/],
> removing the need for operators to run separate exporter processes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MESOS-6918) Prometheus exporter endpoints for metrics
[ https://issues.apache.org/jira/browse/MESOS-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154486#comment-16154486 ] James Peach commented on MESOS-6918: Design doc: https://docs.google.com/document/d/1j1TkckxGrKixvAUoz_TJRl-YayFMCNIUYv8Cq19Kal8 > Prometheus exporter endpoints for metrics > - > > Key: MESOS-6918 > URL: https://issues.apache.org/jira/browse/MESOS-6918 > Project: Mesos > Issue Type: Bug > Components: statistics >Reporter: James Peach >Assignee: James Peach > > There are a couple of [Prometheus|https://prometheus.io] metrics exporters > for Mesos, of varying quality. Since the Mesos stats system actually knows > about statistics data types and semantics, and Mesos has reasonable HTTP > support we could add Prometheus metrics endpoints to directly expose > statistics in [Prometheus wire > format|https://prometheus.io/docs/instrumenting/exposition_formats/], > removing the need for operators to run separate exporter processes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (MESOS-6918) Prometheus exporter endpoints for metrics
[
https://issues.apache.org/jira/browse/MESOS-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150848#comment-16150848
]
Yan Xu commented on MESOS-6918:
---
I think [~bmahler]'s questions (and mine below) suggest we need a (mini) design
doc about the overarching methodology here. The summary of each review and the
existing comments in this JIRA are not providing enough of a high level design
so the justification for each patch is not clear enough.
A few more questions:
- I understand that complex Prometheus metric types such as {{summary}} require
some more data than what is currently provided so we need to add them
somewhere. But they should be added to Mesos' existing "core" metrics classes
only if they are generic (backwards compatible) improvements that make sense
regardless of the Prometheus support. I believe this is indeed your goal but we
need to articulate how current timer/statistics modeling is lacking/wrong.
-- There are some patches that remove history from simple metrics because it
doesn't make sense. Should the history then be put in another base type that
{{Counter}} and {{Gauge}} don't derive from?
- After the above is done, is Prometheus merely a specific format? If that's
the case, can we encapsulate the formatting logic into a formatter class/method
instead of the main metric endpoint actor?
> Prometheus exporter endpoints for metrics
> -
>
> Key: MESOS-6918
> URL: https://issues.apache.org/jira/browse/MESOS-6918
> Project: Mesos
> Issue Type: Bug
> Components: statistics
>Reporter: James Peach
>Assignee: James Peach
>
> There are a couple of [Prometheus|https://prometheus.io] metrics exporters
> for Mesos, of varying quality. Since the Mesos stats system actually knows
> about statistics data types and semantics, and Mesos has reasonable HTTP
> support we could add Prometheus metrics endpoints to directly expose
> statistics in [Prometheus wire
> format|https://prometheus.io/docs/instrumenting/exposition_formats/],
> removing the need for operators to run separate exporter processes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MESOS-6918) Prometheus exporter endpoints for metrics
[
https://issues.apache.org/jira/browse/MESOS-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136119#comment-16136119
]
Benjamin Mahler commented on MESOS-6918:
[~jpe...@apache.org] [~xujyan] ideally you could shepherd but with my feedback
involved, I do want to see what the changes to the existing metrics code /
functionality will be. I'll include some thoughts here:
# I'm a little confused about the relationship between the enum type and a
"multi-metric" type like {{Timer}}, where {{Timer}} includes some sub-metrics
that are counters and some sub-metrics that are gauges, the current patches
seem to express a single type for {{Timer}} which feels off.
# There seems to be a breaking change to the timer metric that expresses the
most recent measurement, probably that one needs to be deprecated in place of a
{{/total_time}} or we should rationalize why it's ok to change the meaning
(maybe it's ok, not sure).
Those are the main things that I haven't spent enough time to come up with
suggestions for, haven't looked at the endpoint change yet.
> Prometheus exporter endpoints for metrics
> -
>
> Key: MESOS-6918
> URL: https://issues.apache.org/jira/browse/MESOS-6918
> Project: Mesos
> Issue Type: Bug
> Components: statistics
>Reporter: James Peach
>Assignee: James Peach
>
> There are a couple of [Prometheus|https://prometheus.io] metrics exporters
> for Mesos, of varying quality. Since the Mesos stats system actually knows
> about statistics data types and semantics, and Mesos has reasonable HTTP
> support we could add Prometheus metrics endpoints to directly expose
> statistics in [Prometheus wire
> format|https://prometheus.io/docs/instrumenting/exposition_formats/],
> removing the need for operators to run separate exporter processes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MESOS-6918) Prometheus exporter endpoints for metrics
[ https://issues.apache.org/jira/browse/MESOS-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135996#comment-16135996 ] Yan Xu commented on MESOS-6918: --- [~klueska] [~bmahler] if you guys don't have cycles I can shepherd this but of course let's sync on the overall approach (if deemed necessary as I go over the code or if you are so inclined). > Prometheus exporter endpoints for metrics > - > > Key: MESOS-6918 > URL: https://issues.apache.org/jira/browse/MESOS-6918 > Project: Mesos > Issue Type: Bug > Components: statistics >Reporter: James Peach >Assignee: James Peach > > There are a couple of [Prometheus|https://prometheus.io] metrics exporters > for Mesos, of varying quality. Since the Mesos stats system actually knows > about statistics data types and semantics, and Mesos has reasonable HTTP > support we could add Prometheus metrics endpoints to directly expose > statistics in [Prometheus wire > format|https://prometheus.io/docs/instrumenting/exposition_formats/], > removing the need for operators to run separate exporter processes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (MESOS-6918) Prometheus exporter endpoints for metrics
[ https://issues.apache.org/jira/browse/MESOS-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102201#comment-16102201 ] James Peach commented on MESOS-6918: [~klueska], [~bmahler] Are either of you able to officially shepherd this? > Prometheus exporter endpoints for metrics > - > > Key: MESOS-6918 > URL: https://issues.apache.org/jira/browse/MESOS-6918 > Project: Mesos > Issue Type: Bug > Components: statistics >Reporter: James Peach >Assignee: James Peach > > There are a couple of [Prometheus|https://prometheus.io] metrics exporters > for Mesos, of varying quality. Since the Mesos stats system actually knows > about statistics data types and semantics, and Mesos has reasonable HTTP > support we could add Prometheus metrics endpoints to directly expose > statistics in [Prometheus wire > format|https://prometheus.io/docs/instrumenting/exposition_formats/], > removing the need for operators to run separate exporter processes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (MESOS-6918) Prometheus exporter endpoints for metrics
[
https://issues.apache.org/jira/browse/MESOS-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16086917#comment-16086917
]
James Peach commented on MESOS-6918:
Implementation notes:
* I'm proposing this only for the {{/metrics/snapshot}} endpoint
* Prometheus (at least v2) doesn't send an {{Accept}} header, so we can't mux
on that to produce Prometheus output. We need to require a separate endpoint or
query parameter.
* [~klueska] suggests using a query parameter to explicitly request Prometheus
output
* The standard endpoint for exporters is {{/metrics}} (this is the prometheus
configuration default)
* The mapping from {{Accept}} to exposition format is done in
[expfmt.Negotiate|https://github.com/prometheus/common/blob/9a94032291f2192936512bab367bc45e77990d6a/expfmt/encode.go#L41].
The text format is the default.
* As per [#2788|https://github.com/prometheus/prometheus/issues/2788],
Prometheus v2 doesn't support the protobuf exposition; {{text/html+gzip}} is
recommended.
* Prometheus does send a {{Accept-Encoding: gzip}} header.
* Prometheus will send the {{X-Prometheus-Scrape-Timeout-Seconds}} header with
the request timeout it is planning to use.
> Prometheus exporter endpoints for metrics
> -
>
> Key: MESOS-6918
> URL: https://issues.apache.org/jira/browse/MESOS-6918
> Project: Mesos
> Issue Type: Bug
> Components: statistics
>Reporter: James Peach
>Assignee: James Peach
>
> There are a couple of [Prometheus|https://prometheus.io] metrics exporters
> for Mesos, of varying quality. Since the Mesos stats system actually knows
> about statistics data types and semantics, and Mesos has reasonable HTTP
> support we could add Prometheus metrics endpoints to directly expose
> statistics in [Prometheus wire
> format|https://prometheus.io/docs/instrumenting/exposition_formats/],
> removing the need for operators to run separate exporter processes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
