Good point on the ’most recent dequeued’ message being a good marker. I think ideally we wouldn’t have to browse the message or have a sliding window of updated metrics for all messages, but achieving both of those is probably unavoidable.
I think first “all time” and “next to dispatch” are useful enough to paint the picture of flow health. Next draft of metrics: (’next’ Aka ‘head’ or ’next’ message to be dequeued— requires a browse) nextMessageEnqueuedTimestamp (from the message’s brokerInTime) nextMessageID (‘last’ aka the most recent message to be ack’d — processed in-flight) lastEnqueuedMessageTimestamp (from brokerInTime on the message) lastDequeuedTimestamp (clock timestamp of when the message was dequeued) lastDequeuedMessageTimestamp (from brokerInTime on the message) lastDequeuedMessageID (‘First’ aka first all time in the life of the destination since boot — processed in-flight) firstEnqueuedMessageTimestamp (from brokerInTime) firstDequeuedTimestamp (wall clock time when the message was dequeued) firstDequeuedMessageTimestamp (from brokerInTime) Calculations: Queue processing latency can then be calculated as = lastDequeuedMessageTimestamp - lastDequeuedTimestamp Oldest message age on the queue (when queueSize > 0): lastEnqueuedMessageTimestamp Thoughts? -Matt Pavlovich > On Jan 25, 2022, at 1:11 PM, Endre Stølsvik <en...@stolsvik.com> wrote: > > Hi! > > Thanks for the positive feedback! > > From the issue AMQ-8463, I do not quite understand the terminology. > > The original name I came up with for this property was > "headMessageBrokerInTime", to indicate that it was the "message at head of > queue", and the *amqMessage.getBrokerInTime()* of that message. > However, when looking at corresponding functionality in QueueControl(Impl) > in Artemis, I found that the existing functionality there was called > "firstMessageTimestamp", and thought that it would make sense to align the > names when it is a new feature anyway. > For comparison, in RabbitMQ, the corresponding feature's name is a mix of > those two, "headMessageTimestamp" - > https://www.rabbitmq.com/rabbitmqctl.8.html#head_message_timestamp, but you > evidently want to install a small plugin to get it on all messages: > https://github.com/rabbitmq/rabbitmq-message-timestamp, as there is > something with small messages not hitting the message store and thus not > actually getting the timestamp set in default config: > https://github.com/deadtrickster/prometheus_rabbitmq_exporter/issues/15#issuecomment-376684478 > > (From that last issue comment, this sentence resonates: *"The timestamp of > the head message is the best metric available to evaluate the entire > system's ability to keep up with demand."* > .. but I would also add the queue length, which already is present in the > Destination info from StatisticsBroker(Plugin) as 'size'. Thus, with these > two properties available, it is possible to make a remote monitoring system > for all destinations on ActiveMQ, which is ideal for what I want to > accomplish!) > > So, back to the AMQ-8463, I am not sure what these two lines means: > > *> "firstMessageTimestamp - the timestamp of the first message to traverse > the destination" > "lastMessageTimestamp - the timestamp of the most recent > message to traverse the destination (aka 'message age' for messages still > on the queue)"* > > If you by the first line mean "the timestamp of the first message to *ever > traverse the destination in the lifetime of this ActiveMQ instance*", I > believe one ends up with a different concept here than those other two > brokers, and it does not accomplish any of my goals of knowing the > liveliness of the queues. > > The second concept I would rather give an even more specific name, like > "mostRecentDequeuedMessageTimestamp" or something like that, if I > understood you correctly. > > HOWEVER: The big point of the property that I need and have implemented, is > that you basically get two monitoring features in one go: If a message at > the head of the queue is *old*, then it *either* means that the consumer(s) > is stuck or slow, OR it means that there is a substantial queue, where the > messages get old *while waiting on the queue*. By also getting the queue > length, you can distinguish between those two cases. > > But crucially: If there *is no message at the head of the queue*, then from > a monitoring perspective wrt. asserting that there is progress/liveliness > on the queue (wrt. consumers), you are actually in the clear! (If you want > to monitor that there is actually messages flowing at all, one may consider > the already existing properties enqueueCount, dequeueCount, and/or > dispatchCount.) > > > *> "One side benefit of this approach is that you’d have the timestamp > values to work with even when the queue is empty."* > > This means that - assuming that I understood the concept of your two > suggested timestamps and that comment above - neither of those two values > will actually accomplish what I need: I *do not want* a timestamp if there > are no messages on the queue, as the queue is then empty, and evidently > consumers are keeping up! Getting a timestamp of the last message to > traverse would undermine the functionality, as I could then not use this > value alone. Just consider a situation where a message traversed and was > consumed properly, but then there was no new message for 2 hours. > > Kind regards, > Endre Stølsvik > > > On Tue, Jan 25, 2022 at 4:42 PM Matt Pavlovich <mattr...@gmail.com> wrote: > >> Endre- >> >> Thank you for the contribution! I think this is a good metric to track on >> the _destination_ itself.. along w/ the lastMessageTimestamp. Other >> products have it and it provides good observability feedback as to the >> traffic progressing over the destinations. >> >> One side benefit of this approach is that you’d have the timestamp values >> to work with even when the queue is empty. >> >> I’ve created JIRA AMQ-8463 <https://issues.apache.org/jira/browse/AMQ-8463> >> to track this and will target it for 5.17.0 and 5.16.5. If you have other >> use cases or scenarios, please add your notes to the JIRA. >> >> Thanks! >> Matt Pavlovich >> >>> On Jan 24, 2022, at 6:13 PM, Endre Stølsvik <en...@stolsvik.com> wrote: >>> >>> Hi! >>> >>> I have a small contribution I was hoping that you would consider. >>> >>> The feature is in the StatisticsBroker(Plugin), where I have added an >>> optional request property, which if present will on the reply message >>> include the "BrokerInTime" for the first message in the destination, as a >>> field "firstMessageTimestamp". >>> >>> How do you want me to do such a suggestion? Should I first open an issue >> / >>> feature request, and then do a PR for it, or just submit a PR directly? >>> >>> Kind regards, >>> Endre Stølsvik >> >>
