Thanks Justin, I think we have to optimize monitoring, and I'll check how to use artemis-prometheus-metrics-plugin.
Our previous metrics solution was: run Telegraf on each host, use Telegraf Jolokia input, and send metrics to InfluxDB. JMX exporter seems to work better, but there was only one problem. We had to add option -Dcom.sun.management.jmxremote=true to make JMX Exporter gather JVM metrics. It does not gather these metrics when this option is turned off. But It seems that this option disables ACL on JMX, and any authenticated user has access to any operation. So I have restricted JMX access to localhost only. We used firstMessageAge metric because in IBM MQ we have frequently faced problems when it looks like client is working but messages are stuck in queue. The most useful metric in this situation is delivering count (uncommitted messages in IBM), it shows messages which are consumed but not acknowledged. And I did not measure performance impact of using these metrics. -- Best regards, Aleksandr Milovidov -----Original Message----- From: Justin Bertram <jbert...@apache.org> Sent: Monday, September 18, 2023 9:46 PM To: users@activemq.apache.org Subject: Re: Artemis first message age metric I believe it would be possible to modify the getFirstMessageAge operation so that it returned 0 if there were no messages. However, I would recommend a different approach. The getFirstMessageAge operation is actually fairly "heavy" and not generally recommended. Furthermore, the age of the first message isn't meaningful in and of itself in this scenario because if the consumerCount is 0 then by definition no messages can be stuck. A robust stuck-message detection mechanism must, at the very least, verify that consumerCount > 0. Also, instead of using the age of the first message I recommend inspecting messagesAcknowledged over time. For example, if the consumerCount > 0 and messagesAcknowledged remains unchanged for 60 seconds then messages (or more likely *consumers*) may be stuck. If you're using Prometheus then I believe you can use a range vector selector [1] for this kind of operation. Also, consider using the pluggable metrics functionality [2] rather than the Prometheus JMX exporter. There is already a Prometheus plugin [3] available and that would save you the headache of configuring the Prometheus JMX Exporter and should provide better performance as well. If there's a specific reason you can't use the pluggable metrics functionality I'd love to hear your feedback. We're always trying to improve things for real-world use-cases. Justin [1] https://prometheus.io/docs/prometheus/latest/querying/basics/#range-vector-selectors [2] https://activemq.apache.org/components/artemis/documentation/latest/metrics.html [3] https://github.com/rh-messaging/artemis-prometheus-metrics-plugin On Mon, Sep 18, 2023 at 3:41 AM MILOVIDOV Aleksandr <aleksandr.milovi...@raiffeisen.ru.invalid> wrote: > Hi All, > > We are using metrics to check if messages are stuck in ActiveMQ > Artemis queues. One of these metrics is first message age attribute. > For example, IBM MQ has similar metric. It returns oldest message age > if queue monitoring is turned on and returns 0 when queue is empty. > > When we tried to implement similar approach for Artemis, we faced a > problem. This attribute is empty if there are no messages in the queue. > I cannot figure out how to make an alert fire when this attribute is > greater than threshold, and return to normal when there are no > messages in the queue. > > We are using Prometheus with JMX exporter. I tried to write complex > PromQL queries like: > > absent(artemis_first_message_age{broker="broker_name ") * 0 or > artemis_first_message_age{broker="broker_name"} > 0 > > But it does not work because absent() function returns only one value > for all series of the metric (and returns nothing if any value is present). > And it is not possible to write all queue names (including dynamic > queues or queues which will be defined in the future) in the alert definition. > > Is it possible to make broker return zero first message age when there > are no messages in the queue? > > -- > Best regards, > Aleksandr Milovidov > > > ----------------------------------- > > This message and any attachment are confidential and may be privileged > or otherwise protected from disclosure. If you are not the intended > recipient any use, distribution, copying or disclosure is strictly > prohibited. If you have received this message in error, please notify > the sender immediately either by telephone or by e-mail and delete > this message and any attachment from your system. Correspondence via > e-mail is for information purposes only. AO Raiffeisenbank neither > makes nor accepts legally binding statements by e-mail unless otherwise > agreed. > > ----------------------------------- > ----------------------------------- This message and any attachment are confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient any use, distribution, copying or disclosure is strictly prohibited. If you have received this message in error, please notify the sender immediately either by telephone or by e-mail and delete this message and any attachment from your system. Correspondence via e-mail is for information purposes only. AO Raiffeisenbank neither makes nor accepts legally binding statements by e-mail unless otherwise agreed. -----------------------------------
