Edit: I think James extension system trivially allows to do that.
I'll have a quick poc at exporting metrics to Graphite this weekend (if
Lucas sleeps long enough) and feed it as an example to the James community.
Regards,
Benoit
On 10/06/2022 16:41, Benoit TELLIER wrote:
Hello Jean,
For the record, we are moving forward, we will keep the ESReporter for
now not to be blocked by orthogonal discussions.
On 10/06/2022 16:01, Jean Helou wrote:
Hey benoit,
First I am happy to see this component raise such interest, and gets so
much attention!
For the record, I want to establish that I don't use nor actually
like ES
for such use cases so the ES part doesn't really matter to me.
I do care a lot about metrics collection and I generally dislike
being tied
to a single system.
However it's very easy to spend loads of time on technical issue that
delivers little value to end users.
My Prometheus deployments works well and I don't intend to invest more
of the topic. I don't think I will beat you on that ;-)
Today you (linagora?) are happy with prometheus and the platform I
target
supports it so I will be using it too.
What worries me is the total lack of choice, the difficulty to add said
choice in the current james ( and I did try) despite boasting a metrics
collection system that is supposed to have pluggable implementations.
I do consider that metrics collection brings a lot of value to end
users,
and that being able to integrate in an existing metrics collection
infrastructure is an important point for james adoption beyond your
deployments :D
1. I (we?) would be happy to work on real world customer contracts on
the topic. Also contributions are welcomed on the topic.
2. Today we could add an extension mechanism exposing metrics and
enabling exports.
Concretely, takes a dropwizard MetricRegistry in, outputs metrics out.
The downside is that we enforce the dropwizard choice onto out users,
thus making it harder to change this component when required.
So yes, when we are done with the pulsar/blob repository thing I will
probably spend some time to either submit a couple modules for
established
and stable protocols that won't require much maintenance beyond the
occasional version bump ( graphite and collectd reporters most likely
since
they are 1st party modules of dropwizard metrics ) or to implement
opentelemetry in a way that doesn't require such internal bindings,
brings
additional benefits in terms of features such as tracing and is more
actively maintained than dropwizard metrics.
Also, ELK stack is not used for metrics.
actually it is, APMs and metrics ingestion have been part of the
elastic
solution for quite a while not that I have actually used them
However, there are many monitoring systems based on push instead of
pull.
The debate has been going for a long time and will probably be
going on
forever ( google lists 54 million results for 'push vs pull monitoring
systems' ) it would be nice for james to support at least one of each
model
even if pull is currently the favored approach.
Prometheus is popular these days because it is deployed internally in
kubernetes which is at the top of the hype curve but james should
probably
support at least one push based protocol
You know we `could` support many things it do not mean that we should.
We are a small community with limited means, the maintenance effort
most
of the time falls on the same people.
Our efforts should be directed to things that are useful, not a
deprecated metric module with alternatives that targets another Elastic
Search version.
Note that I don't push to keep the es module, I push for having at least
one push based alternative to prometheus out of the box if only to
demonstrate/document how plugging a dropwizard metrics reporter in james
can be achieved if you don't use prometheus
or at least document how to plug
one in for people who build their own assembly.
+1 thanks for opening with consensus. (was about to do the same ;-) )
I agree we could have an extension mechanism for metric exporters.
Ideally underlying metric library should come up with an SPI so we, as
application writers, don't need to care.\
I'm not going so far as to want an SPI for dropwizard metrics
reporters, as
I said a couple integrations for the reporters provided by metrics
itself
to document how to integrate such a reporter is just fine.
As I wrote above I will eventually contribute such an integration if
it has
not been done before I get to it.
Though, if there's a potential move to open telemetry, then adding our
own custom extension mechanism is likely not the right thing.
I would first attempt to implement the graphite/collectd reporter
binding,
then look into opentelemetry since the former is supposed to be less
work
than the latter.
A lot less actually
https://metrics.dropwizard.io/4.2.0/manual/collectd.html //
https://metrics.dropwizard.io/3.1.0/manual/graphite/ :-)
It should be trivial. The hardest would be to keep the classpath of
non collectd users clean, which could be achieved via an extension
mechanism located into
https://github.com/apache/james-project/tree/master/third-party. You
would drop the JAR into extension-jars folder, register your metric
reporter FQDN into extensions.properties and go. Even user supplied
Jars would be supported.
CF
https://github.com/apache/james-project/blob/master/server/apps/distributed-app/docs/modules/ROOT/pages/operate/metrics.adoc
Upon recent documentation on the topic we likely forgot to update
legacy
documentation. That's a fail that will be fixed timely.
I have been unable to navigate to that document on the current
website :(
I am not sure how an end user is supposed to discover this ...
cheers
jean
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
