On Tue, Jan 25, 2022 at 11:55 PM Matt Pavlovich <mattr...@gmail.com> wrote:
> > > On Jan 25, 2022, at 4:36 PM, Endre Stølsvik <en...@stolsvik.com> wrote: > > > I still do not see the point of the "first message on destination since > > boot of ActiveMQ": What is the use of this? Why would I in an operational > > context care? But of course, if it has zero cost, then why not! ;-) > > First-ever message tells you how long you’ve been processing messages in > the history of the boot of the broker. Head and Tail only provide you the > active window. > I understand that, I just fail to see the value. The broker should just always be up. When it is booted due to maintenance or upgrade, that is .. unfortunate. But why such a rather random point in time has any value wrt. to statistics, I do not see. But that doesn't really matter! > > > I also still don't quite understand the value of knowing any timestamps > of > > messages that *have been* on the queue, i.e. "last to be dispatched" or > ".. > > dequeued", or "last to be enqueued". If there are no messages on the > queue, > > then there is no timestamp for head or last. That indeed also means that > > there is no "timespan" on the queue, and the *current* latency is 0. If > the > > head == last, i.e. 1 message, then the span is also 0, which also makes > > sense - but that one message might be old. > > This tells you the last time a message was consumed. > This is useful when the queue is empty. Again, this is about total life of > the queue, not just the active window. > Yes, okay - I can see that when looking over your queues, this would have some value. With the dequeCount and last message processed, you will have an idea of the activity of the queue. (Of course, having this in a graph over time instead would be of much more value, ref. metrics export.) > > > Evaluating the time it took to process the last message that was > processed > > often bears little information about the throughput of messages: One > > message might be heavy to process, while the next is light and fast. In > > addition, we have GC pauses and similar that can make single message > > processings times differ wildly. > > Currently, ActiveMQ has min, max and average processing times. The ability > to view how the last message performed provides you the snapshot info on > the current consumer(s). Are you inching towards being a slow consumer > situation that could trigger a slow consumer policy? ie.. abort the > connection? > > This also aligns w/ IBM MQ which has LAST_GET_DATE/TIME and would ease > their transition. > Easing transition is valuable in itself. > > > If you keep the lastEnqueuedMessageTimestamp on the destination, then it > > will be very fast to find the value for lastMessageTimestamp: If there > are > >> 0 messages on queue, then return that directly, if there are 0 messages > on > > queue, then return "null". > > This metric is different than current ‘head’ and ’tail’ metrics. This is > about entire life of the queue. If there haven’t been messages on a queue > for hours, you’d want to know that to go track down why producers aren’t > sending. > Yes, I understand that - this was meant as an idea for actually finding the answer to last/tailMessageTimestamp. IIUC, the "algorithm" I described yields the same result as spooling back to the last message of the queue, and returning the BrokerInTime of that, if any. Only without the spooling. > > > There is a bit of a point to not include *too* much information in those > > statistics reply messages - the set of datapoints should have an > > operational value. I have several thousand queues, and query for ">", so > I > > will get several thousand reply messages. I'd rather query more often, > than > > get dozens of datapoints that is of little value. > > The idea is to re-use the metrics from the destination for the plugin. It > should do very little actual ‘work’. > I was simply thinking about the girth of the actual StatisticsPlugin's reply messages. > > > > I could change the PR to reflect the wanted name change for > > firstMessageTimestamp (to headMessageTimestamp or whatever you would > want), > > but it will be a bit heavier for me to dive deeper down into the ActiveMQ > > internals to include those other datapoints. > > I’ve got some other Destination enhancements planned for 5.16.x.. I’m > happy to incorporate your concepts in a comprehensive ’statistics update’ > PR. > Yeah okay. The current PR is already tested and running on my setup, and has tests - so you could consider merging it so that this feature is in, and then continue from there. > > > At any rate, I will be happy no matter what things are named, if I only > get > > the "BrokerInTime" timestamp of the current first message on the > > destination, and that this is "null" (not set/0) if there are no message > on > > the destination! > > Let me update the requirements draft w/ a new set to include ‘head’ and > ’tail’ values. > I would argue that "last" is better than "tail", as tail in a linked list context means "all elements except the head". Then again, in the Deque JavaDoc, they refer to "head" and "tail". > > > PS: If you do not merge my PR, then I hope that you could include the > other > > small change in the PR, i.e. the improvement of how to request > > "null-termination" of the replies, as I found the existing solution to be > > quite hard to decipher how to enable! Setting a query message property to > > enable features of the query was much simpler than making a long and > > somewhat convoluted query destination. > > Let’s revisit this once we’ve settled on design for destination metrics. > Perhaps you submit a new PR once those changes are in. > Again, the PR is already there - also improving the current feature of null-termination (making it way simpler to use), a microscopic performance improvement, and a tiny bit of cleanup. > > > PPS: Note that in the current implementation in the PR, the gathering of > > the information is async, and it is thus possible that size=1, while > > firstMessageTimestamp is "null" - and also the opposite, that size=0, > while > > firstMessageTimestamp is set. It will both be hard and probably pretty > > costly to make this consistent (due to synchronization/blocking), and in > an > > operational monitoring setting, it doesn't matter at all: Those values > are > > evaluated independently, and any transient skew is of no importance. > > Not sure about null.. maybe we’ll need to use 0 or -1 to indicate some > “non-value” concept. > Yeah, that's why i wrote "null", in quotes. The current PR doesn't include the value in the MapMessage if there is no message. If you want a value, I'd personally prefer 0 to -1. > > > PPPS: Another thing to consider at a later point, is to include good > > metrics, as in Prometheus/Grafana-style. The current solution for > ActiveMQ > > seems to be to hook on a JMX exporter, but adding this natively to > ActiveMQ > > would probably give value. Spring's Micrometer is pretty nice, selling > > itself as "slf4j for metrics" / "Vendor-neutral application metrics > > facade": If you instrument using Micrometer, you can get metrics > gathering > > by Prometheus, but also 18 other such solutions. https://micrometer.io/. > By > > exposing all the information we've discussed as meters, you could really > > get a good view of throughput, queue sizes, latencies etc. > > Metrics collection systems are like popular music acts, there is a new > ‘best’ every few months ;-) > Micrometer actually seems pretty well thought out, and the "slf4j for metrics" angle makes tons of sense. There seems to be a bit of uptake. > I think as long as we provide standard ways to access the data, > incorporating them into any system is straight-forward. > As mentioned, the current solution to getting metrics from ActiveMQ is to also fire up a "sidecar" JMX Exporter. IMHO, this is hardly a fantastic solution; I believe it would be a massive asset of ActiveMQ to be able to be metrics-instrumentable with the simple enabling of a switch in the config, like the StatisticsPlugin. Or maybe just default enabled with a default Prometheus scrape endpoint (as that basically is a defacto standard) - as is the web console and jolokia? But of course, this is a complete tangent to the discussion at hand! Kind regards, Endre Stølsvik.
