Hello Jean,
First I am happy to see this component raise such interest, and gets so
much attention!
Hopefully this means somebody would give it the love it deserves from
years, enable it to run on recent ElasticSearch versions, hardens its
behaviours and underlying libraries?
In depth answers inlined.
On 09/06/2022 17:09, Jean Helou wrote:
Hello all,
We IMO need to add that direct reporting (for logs or metrics) from the
application to ElasticSearch is an anti-pattern, and pretty uncommon.
Doesn't sound quite right :
Pushing logs from the app the ES is exactly what the whole ELK thing is all
about. Given the amount of literature that promotes it and the popularity
of logstash-logback-encoder [1], I would like to argue that this is not all
that uncommon :)
Logstash serves as a buffer, distinct from the APP, dedicated to this
purpose.
To my knowledge getting the L (logstash) out of the stack is not very
popular, and that's my point here.
Also, ELK stack is not used for metrics.
I'm less familiar with the metrics side of things and the current elastic
documentation [2] seems to mix both a prometheus like ingester and a jvm
agent that pushes apm metrics to ES which seems to offer a migration path
for people still using ES for metrics monitoring.
Yes.
However, there are many monitoring systems based on push instead of pull.
The debate has been going for a long time and will probably be going on
forever ( google lists 54 million results for 'push vs pull monitoring
systems' ) it would be nice for james to support at least one of each model
even if pull is currently the favored approach.
Prometheus is popular these days because it is deployed internally in
kubernetes which is at the top of the hype curve but james should probably
support at least one push based protocol
You know we `could` support many things it do not mean that we should.
We are a small community with limited means, the maintenance effort most
of the time falls on the same people.
Our efforts should be directed to things that are useful, not a
deprecated metric module with alternatives that targets another Elastic
Search version.
or at least document how to plug
one in for people who build their own assembly.
+1 thanks for opening with consensus. (was about to do the same ;-) )
I agree we could have an extension mechanism for metric exporters.
Ideally underlying metric library should come up with an SPI so we, as
application writers, don't need to care.\
Though, if there's a potential move to open telemetry, then adding our
own custom extension mechanism is likely not the right thing.
I think the es metrics reporter has not been really maintained and/or used
since es v6.
Note that the ES module is still the *first* metrics monitoring mechanism
listed in the james documentation [3]
That is a fail. This page needs to be updated.
Let's make the documentation better then ;-)
and the *only* one listed for
distributed james[4].
That is a fail. This page needs to be updated.
Let's make the documentation better then ;-)
That same documentation has no mention of the
deprecation nor of prometheus, there is lots of information about JMX and
folder monitoring (quite frankly I don't know how much of it is actually
still relevant ( especially with the removal of FileDir IIRC )).
CF
https://github.com/apache/james-project/blob/master/server/apps/distributed-app/docs/modules/ROOT/pages/operate/metrics.adoc
Upon recent documentation on the topic we likely forgot to update legacy
documentation. That's a fail that will be fixed timely.
Also note that
- there is no other options for push based monitoring model in the current
code base
- implementing such an alternative has proven not to be trivial ( we spent
a couple evenings with matthieu trying to write an influxdb reporter when
assembling the first pulsar based app, we *failed* and gave up by plugging
the ES reporter in the pulsar app)
If james drops the only existing push based dropwizard reporter
integration, I would very very much like to see documentation on how to
write and integrate another (maybe the metrics-graphite [6] could be used
as a sample since there is a readily available graphite-statsd docker image
[7] ?) Matthieu and I can't currently revisit this subject, we are both
limited by our available free time and we are currently fully occupied
subduing the mailrepository subs system so it accepts using the blob store
based implementation in the pulsar assembly ( fun quiz can you tell looking
only at the documentation and code comments the difference between :
MailRepositoryStore, MailRepositoryUrlStore and MailRepository all of which
are in mailrepository-api )
Game accepted.
- Mail repository is storage for email, with their processing context
(long term storage, differs from mail queue which is a flow).
- Mail repository are identified by their URL
- Mail repository can be created through the use of mail repository
store by supplying an URL
- MailRepositoryUrlStore is an implementation detail of
MailRepositoryStore, and brings persistance to mail repositories (that
are created through webadmin, configuration changes etc..)
Ideally MailRepositoryUrlStore should not have had been in the API.
Now to have a MailRepositoryStore not based on Cassandra, the memory
implementation is good enough if manual creation of mail repository is
forbidden (akka through webadmin) and if configuration is homogeneous in
the James cluster.
One of the things we discussed with Matthieu as a possible contrib after
the pulsar mailqueue based assembly was implementing metrics-api using
opentelemetry[8] and to drop the need for custom reporter integration in
favor of configuration based reporters[9] ( don't hesitate to beat us to
it ! :p )
Sounds fair.
However it's very easy to spend loads of time on technical issue that
delivers little value to end users.
My Prometheus deployments works well and I don't intend to invest more
of the topic. I don't think I will beat you on that ;-)
Cheers,
Benoit
Cheers
jean
PS while writing this message I realized that 3.7.0 is not listed on
https://james.apache.org/server/release-notes.html despite having a blog
announcement
https://james.apache.org/james/update/2022/03/21/james-3.7.0.html
[1] https://github.com/logfellow/logstash-logback-encoder
[2]
https://www.elastic.co/guide/en/observability/master/monitor-java-app.html
[4] https://james.apache.org/server/monitor.html
[5]
https://james.apache.org/server/manage-guice-distributed-james.html#Basic_Monitoring
[6] https://metrics.dropwizard.io/4.2.0/manual/graphite.html
[7] https://hub.docker.com/r/graphiteapp/graphite-statsd/
[8] https://opentelemetry.io/
[9]
https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md#exporters
On Thu, Jun 9, 2022 at 9:21 AM Rene Cordier <rcord...@apache.org> wrote:
I would like to add that it seems this module has been deprecated when
James migrated from es6 to es7, as stated in the documentation of the
configuration here for example:
https://github.com/apache/james-project/blob/master/src/site/xdoc/server/config-elasticsearch.xml#L137
For likely all the reasons given below.
So it makes sense to remove it now that we are migrating to es8.
Regards,
Rene.
On 09/06/2022 14:13, Benoit TELLIER wrote:
+1
We IMO need to add that direct reporting (for logs or metrics) from the
application to ElasticSearch is an anti-pattern, and pretty uncommon.
It's usage led, at least at Linagora, to metrics/logs loss. The code
quality of these extensions was low and proved several time problematic.
Alternatives:
- Anything compatible with prometheus (pretty much all of metrics
ecosystem)
- Things like fluentbit to scrap logs from the container/a file and
move it to your analytic stack
Regards,
Benoit
On 09/06/2022 11:02, Rene Cordier wrote:
Hello guys,
As working on the upgrade to ElasticSearch 8.2, I've been wondering if
it was still worth it to keep the elasticsearch metrics reporter module.
Since we expose a metrics endpoint that tools like prometheus can just
call to scrap metrics out of James, I think the es metrics reporter
has not been really maintained and/or used since es v6.
As such, we decided to remove it. It allows to reduce the number of
modules loaded by James and also reduce the cost of maintenance.
But maybe other people are still using it? If a part of the community
still want to use it with the switch to ES8, they can put it back and
migrate it, but we would appreciate that they can assume the
maintenance of it as well.
Also removed the logback-elasticsearch-appender. It's not correlated
to our code and easy to add the JAR back on the classpath if needed.
The PR in question: https://github.com/apache/james-project/pull/1018
Don't hesitate to answer this thread if you have any concerns.
Cheers,
Rene.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
