On 8/2/19 11:11 π.μ., Willy Tarreau wrote:
> Hi Christopher,
>
>
> On Thu, Feb 07, 2019 at 10:09:52PM +0100, Christopher Faulet wrote:
>> Hi,
>>
>> This patch adds a new component in contrib. It is a Prometheus exporter for
>> HAProxy.
> (...)
>
> Thanks for doing this. After reading the whole patch, I measure how
> uninteresting an experience this must have been! After all, using C
> to produce yet-another-format is akin to using python to write
> yet-another-http-proxy :-)
>
> I totally agree with your approach of placing it under contrib/. After
> all we've already done the same with other extensions explicitly
> targetting products (mod_security, mod_defender, systemd, ...).
> We support standards, not products. And products which respect standards
> are naturally supported via the standards, so there is indeed no reason
> for opening a new can of worms by inviting $PRODUCT_OF_THE_MONTH into
> src/ especially when these products change faster than our maintenance
> cycle.
>
> In my opinion the right place for a stats exporter is : outside. However
> I'm well aware that our export formats are not necessarily friendly to
> such exporters. For example, the fact that prometheus uses this funny
> ordering forces a gateway to keep all metrics in memory before being
> able to dump them. It's not cool either. We could have a long-term
> approach consisting in trying to implement multiple tree walk methods
> combined with a few formats so that implementing external exporters in
> various languages becomes trivial. In this case such tools could provide
> high quality agents to collect our metrics by default without having to
> work around some limitations or constraints.
>
> This is probably a design discussion to have for the long term here on
> the list : what are the main stats export mechanisms desired in field.
> I can imagine that most agents will want to poll haproxy and dump the
> whole stats once in a while, some will rely on it to send a full dump
> once in a while (this may already become an issue during reloads), some
> might possibly want to subscribe to change notification of certain
> metrics, or receive a diff from the previous dump once in a while. And
> for all these variations I guess we may have to implement 2 or 3 dump
> styles :
> - Front -> Back -> Server -> metric
> - { Front, Back, Server } -> metric
> - metric: { front, back, server }
>
> I don't know if I'm completely off or not, but I do think that those who
> have experience with such tools should definitely join the discussion to
> share their observations and deployment difficulties met in field.
> There are mainly two ways to get metrics out of software: 1. Push, where foobar software uploads stats to a remote/local entry point. Graphite is one of the most used system for this. 2. Pull, the metrics pipeline/infra scrapes foobar software to fetch metrics. The above is the easy part of getting metrics, the most challenging is data types(counters, gauges, summaries) and format. Graphite has a very simply format and data type, you send strings over TCP connections: metricname value timestamp(epoch) where metricname looks like a FS tree: loadbalancers.edge.lb-01.haproxy.frontend.www_haproxy_org.lbtol 124 1549661217 Prometheus uses the pull method and it is a bit more complicated. You also have "proxy" systems like telegraf/fluentd that can work on any method and build a bridge between foobar software and any metric pipeline. All those "proxy" systems allow you to write any transformer you want, so the options are countless. I have to agree with you that supporting all possible combinations is quite hard. Some software support both, some only one of them. More and more you see new software have "instrumentation" out of the box using the pull method and default to Prometheus model. I personally find the cvs data we get out of STATS socket easy to use. I can easily write a software to support both models. I have written one to support the Push method and use Graphite as a metric pipeline and it would be trivial to write an exporter for Prometheus and for another X system in two years. I prefer the foorbar software to give me raw data and let me decide how I will use them. I don't want to have any kind of aggregation at the source of the metrics or any other funny things, which can use issues when I do aggregation at the store or during visualization. My 2cents, Pavlos
signature.asc
Description: OpenPGP digital signature
