Hello Justin, I have just added Prometheus plugin on test servers. It works fine, but there are some broker metrics which are not exposed. For example, we have pair of primary and backup servers, and we have configured alert for attribute "ReplicaSync" which indicates if both servers are synchronized. Is it possible to expose broker attributes as metrics using plugin?
Recently we used Telegraf Jolokia input which can gather any attribute from any mbean. Later we used JMX exporter which exports almost all metrics, but there were some broker metrics which are not exported (for example, broker version). -- Best regards, Aleksandr Milovidov -----Original Message----- From: Justin Bertram <jbert...@apache.org> Sent: Tuesday, September 19, 2023 7:22 PM To: users@activemq.apache.org Subject: Re: Artemis first message age metric It's been awhile now, but when I played with the Prometheus JMX exporter I recall my experience was the same. It's use required integration with the platform MBean server which had to be used in lieu of the MBean server managed by the broker which meant that the role-based access control configured in management.xml wouldn't work for other JMX/Jolokia clients (e.g. the web console). This is a non-starter for lots of users. It also means you're potentially running 2 embedded web servers - the one for the broker and the one for the Prometheus JMX exporter. Some folks may not mind this, but it adds a new attack vector for security which a lot of users don't like. You can craft a reliable stuck-consumer detection mechanism using the metrics available via the metrics manager, but it can't rely on a single metric (as noted previously). Also, keep in mind that the broker itself has slow/stuck consumer detection [1]. Justin [1] https://activemq.apache.org/components/artemis/documentation/latest/slow-consumers.html On Tue, Sep 19, 2023 at 10:42 AM MILOVIDOV Aleksandr <aleksandr.milovi...@raiffeisen.ru.invalid> wrote: > Thanks Justin, > > I think we have to optimize monitoring, and I'll check how to use > artemis-prometheus-metrics-plugin. > > Our previous metrics solution was: run Telegraf on each host, use > Telegraf Jolokia input, and send metrics to InfluxDB. > JMX exporter seems to work better, but there was only one problem. > We had to add option -Dcom.sun.management.jmxremote=true to make JMX > Exporter gather JVM metrics. It does not gather these metrics when > this option is turned off. > But It seems that this option disables ACL on JMX, and any > authenticated user has access to any operation. So I have restricted > JMX access to localhost only. > > We used firstMessageAge metric because in IBM MQ we have frequently > faced problems when it looks like client is working but messages are > stuck in queue. > The most useful metric in this situation is delivering count > (uncommitted messages in IBM), it shows messages which are consumed but not > acknowledged. > And I did not measure performance impact of using these metrics. > > > -- > Best regards, > Aleksandr Milovidov > > -----Original Message----- > From: Justin Bertram <jbert...@apache.org> > Sent: Monday, September 18, 2023 9:46 PM > To: users@activemq.apache.org > Subject: Re: Artemis first message age metric > > I believe it would be possible to modify the getFirstMessageAge > operation so that it returned 0 if there were no messages. > > However, I would recommend a different approach. The > getFirstMessageAge operation is actually fairly "heavy" and not generally > recommended. > Furthermore, the age of the first message isn't meaningful in and of > itself in this scenario because if the consumerCount is 0 then by > definition no messages can be stuck. A robust stuck-message detection > mechanism must, at the very least, verify that consumerCount > 0. > Also, instead of using the age of the first message I recommend > inspecting messagesAcknowledged over time. For example, if the > consumerCount > 0 and messagesAcknowledged remains unchanged for 60 > seconds then messages (or more likely *consumers*) may be stuck. If > you're using Prometheus then I believe you can use a range vector selector > [1] for this kind of operation. > > Also, consider using the pluggable metrics functionality [2] rather > than the Prometheus JMX exporter. There is already a Prometheus plugin > [3] available and that would save you the headache of configuring the > Prometheus JMX Exporter and should provide better performance as well. > If there's a specific reason you can't use the pluggable metrics > functionality I'd love to hear your feedback. We're always trying to > improve things for real-world use-cases. > > > Justin > > [1] > > https://prometheus.io/docs/prometheus/latest/querying/basics/#range-ve > ctor-selectors > [2] > > https://activemq.apache.org/components/artemis/documentation/latest/me > trics.html [3] > https://github.com/rh-messaging/artemis-prometheus-metrics-plugin > > On Mon, Sep 18, 2023 at 3:41 AM MILOVIDOV Aleksandr > <aleksandr.milovi...@raiffeisen.ru.invalid> wrote: > > > Hi All, > > > > We are using metrics to check if messages are stuck in ActiveMQ > > Artemis queues. One of these metrics is first message age attribute. > > For example, IBM MQ has similar metric. It returns oldest message > > age if queue monitoring is turned on and returns 0 when queue is empty. > > > > When we tried to implement similar approach for Artemis, we faced a > > problem. This attribute is empty if there are no messages in the queue. > > I cannot figure out how to make an alert fire when this attribute is > > greater than threshold, and return to normal when there are no > > messages in the queue. > > > > We are using Prometheus with JMX exporter. I tried to write complex > > PromQL queries like: > > > > absent(artemis_first_message_age{broker="broker_name ") * 0 or > > artemis_first_message_age{broker="broker_name"} > 0 > > > > But it does not work because absent() function returns only one > > value for all series of the metric (and returns nothing if any value > > is > present). > > And it is not possible to write all queue names (including dynamic > > queues or queues which will be defined in the future) in the alert > definition. > > > > Is it possible to make broker return zero first message age when > > there are no messages in the queue? > > > > -- > > Best regards, > > Aleksandr Milovidov > > > > > > ----------------------------------- > > > > This message and any attachment are confidential and may be > > privileged or otherwise protected from disclosure. If you are not > > the intended recipient any use, distribution, copying or disclosure > > is strictly prohibited. If you have received this message in error, > > please notify the sender immediately either by telephone or by > > e-mail and delete this message and any attachment from your system. > > Correspondence via e-mail is for information purposes only. AO > > Raiffeisenbank neither makes nor accepts legally binding statements > > by e-mail unless otherwise > agreed. > > > > ----------------------------------- > > > ----------------------------------- > > This message and any attachment are confidential and may be privileged > or otherwise protected from disclosure. If you are not the intended > recipient any use, distribution, copying or disclosure is strictly > prohibited. If you have received this message in error, please notify > the sender immediately either by telephone or by e-mail and delete > this message and any attachment from your system. Correspondence via > e-mail is for information purposes only. AO Raiffeisenbank neither > makes nor accepts legally binding statements by e-mail unless otherwise > agreed. > > ----------------------------------- > ----------------------------------- This message and any attachment are confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient any use, distribution, copying or disclosure is strictly prohibited. If you have received this message in error, please notify the sender immediately either by telephone or by e-mail and delete this message and any attachment from your system. Correspondence via e-mail is for information purposes only. AO Raiffeisenbank neither makes nor accepts legally binding statements by e-mail unless otherwise agreed. -----------------------------------
