[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590180#comment-16590180 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user michaelandrepearce commented on the issue: https://github.com/apache/activemq-artemis/pull/2262 @cshannon we have an issue where we have reported message count that never get dispatched, it was thought it was metric issue only, e.g. difference between whats in the messageferences and the counter (e.g. the messages aren't really there), but it looks like a more serious issue with the messages not being delivered for one reason or another and the messages really are in messageferences. They fun of a not very easy to replicate issue. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > Fix For: 2.5.0 > > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1659#comment-1659 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on the issue: https://github.com/apache/activemq-artemis/pull/2262 @michaelandrepearce - Do you have an actual use case or example where it's not reliable? The new metric gets used for size stuff as well so I don't think it should be changed for consistency and should be fixed if there is an issue However I also see you closed this out so is there not actually an issue here? > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > Fix For: 2.5.0 > > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589861#comment-16589861 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user michaelandrepearce closed the pull request at: https://github.com/apache/activemq-artemis/pull/2262 > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > Fix For: 2.5.0 > > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589860#comment-16589860 ] ASF GitHub Bot commented on ARTEMIS-1663: - GitHub user michaelandrepearce opened a pull request: https://github.com/apache/activemq-artemis/pull/2262 ARTEMIS-1663 fix disparity on messagecount Reverting a change done in 1663. It seems the new messagecount != the original messageReferences.size all the time, it is more reliable to use the original messageReferences size as this is the actual collection's size. You can merge this pull request into a Git repository by running: $ git pull https://github.com/michaelandrepearce/activemq-artemis AREMIS1663-FIX Alternatively you can review and apply these changes as the patch at: https://github.com/apache/activemq-artemis/pull/2262.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2262 commit 2f9e30cde211950bf00cd9a00c05e09ae4dc6fea Author: Michael André Pearce Date: 2018-08-23T07:50:34Z ARTEMIS-1663 fix disparity on messagecount Reverting a change done in 1663. It seems the new messagecount != the original messageReferences.size all the time, it is more reliable to use the original messageReferences size as this is the actual collection's size. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > Fix For: 2.5.0 > > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364199#comment-16364199 ] Christopher L. Shannon commented on ARTEMIS-1663: - [~clebertsuconic] - good catch, I had increased that encodeSize because I initially was storing a size value there in an early version of this patch but then took it out and apparently forgot to revert back the encodeSize to the original > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > Fix For: 2.5.0 > > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364189#comment-16364189 ] ASF subversion and git services commented on ARTEMIS-1663: -- Commit 9b104930c25beeba646d92ab38c6038a11f1e227 in activemq-artemis's branch refs/heads/master from Clebert Suconic [ https://git-wip-us.apache.org/repos/asf?p=activemq-artemis.git;h=9b10493 ] ARTEMIS-1663 Fixing Encoding on PageCountPendingImpl The PageCountPendingImpl was increasing the encode size without using its full allocation. This was causing issues on replication as the encode is also used to determine the size of the packets. however the packets were not receive the full allocated data causing missing packets on the replication and test failures. This is fixing the issue > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > Fix For: 2.5.0 > > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357231#comment-16357231 ] ASF subversion and git services commented on ARTEMIS-1663: -- Commit ea70af15a3fbfc1b6ac86589e1e2e04a79ca3e23 in activemq-artemis's branch refs/heads/master from [~cshannon] [ https://git-wip-us.apache.org/repos/asf?p=activemq-artemis.git;h=ea70af1 ] ARTEMIS-1663 - Add new message count and size metrics Adding new metrics for tracking message counts and sizes on a Queue. This includes tracking metrics for pending, delivering and scheduled messages. The paging store also tracks message size now. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357233#comment-16357233 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user asfgit closed the pull request at: https://github.com/apache/activemq-artemis/pull/1853 > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357232#comment-16357232 ] ASF subversion and git services commented on ARTEMIS-1663: -- Commit 58c4660426eceabecfaecaad42716562d5e2fd92 in activemq-artemis's branch refs/heads/master from Clebert Suconic [ https://git-wip-us.apache.org/repos/asf?p=activemq-artemis.git;h=58c4660 ] ARTEMIS-1663 Tweak on Paging test Making sure it JournalCompatibilityTest/paging is actually paging > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355830#comment-16355830 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on the issue: https://github.com/apache/activemq-artemis/pull/1853 Had to update the PR again as something got messed up when rebasing > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355808#comment-16355808 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166700570 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/paging/cursor/impl/PageSubscriptionImpl.java --- @@ -439,7 +453,7 @@ public void confirmPosition(final Transaction tx, final PagePosition position) t public void ackTx(final Transaction tx, final PagedReference reference) throws Exception { confirmPosition(tx, reference.getPosition()); - counter.increment(tx, -1); + counter.increment(tx, -1, -getPersistentSize(reference)); --- End diff -- @michaelandrepearce - So because of this line here we can't really default the persistentSize to -1, it needs to be 0. This call here will store a new PageCountRecordInc() record to the journal with a negative count and negative size on message ack to decrement the counters. Because of this a negative can actually be a valid value and really 0 should mean not set (we should never have a message size of 0) > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355801#comment-16355801 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on the issue: https://github.com/apache/activemq-artemis/pull/1853 @clebertsuconic - I did some cleanup work and I added a couple of basic tests to the compatibility test. It doesn't do much but show that the current snapshot can load a 2.4 journal successfully (and it can load the paging store as well). We can expand the tests with more if you want. Also, I removed the persistentSize from the PageCountPendingImpl class as the size (and the count apparently) aren't actually used and not needed. On recovery we can recover the size from the paged reference. Do you think this is ready to merge? > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355535#comment-16355535 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user clebertsuconic commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166636796 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/persistence/impl/journal/codec/PageCountPendingImpl.java --- @@ -62,21 +67,41 @@ public long getPageID() { return pageID; } + /** +* @return the persistentSize +*/ + @Override + public long getPersistentSize() { + return persistentSize; + } + + /** +* @param persistentSize the persistentSize to set +*/ + public void setPersistentSize(long persistentSize) { + this.persistentSize = persistentSize; + } + @Override public int getEncodeSize() { - return DataConstants.SIZE_LONG * 2; + return DataConstants.SIZE_LONG * 3; } @Override public void encode(ActiveMQBuffer buffer) { buffer.writeLong(queueID); buffer.writeLong(pageID); + buffer.writeLong(persistentSize); --- End diff -- JournalCompatibilityTest under compatibility-tests. I don't think the test is using paging. we should just extend the test to use paging. I can help you with that if you have questions. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355515#comment-16355515 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166632761 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/persistence/impl/journal/codec/PageCountPendingImpl.java --- @@ -62,21 +67,41 @@ public long getPageID() { return pageID; } + /** +* @return the persistentSize +*/ + @Override + public long getPersistentSize() { + return persistentSize; + } + + /** +* @param persistentSize the persistentSize to set +*/ + public void setPersistentSize(long persistentSize) { + this.persistentSize = persistentSize; + } + @Override public int getEncodeSize() { - return DataConstants.SIZE_LONG * 2; + return DataConstants.SIZE_LONG * 3; } @Override public void encode(ActiveMQBuffer buffer) { buffer.writeLong(queueID); buffer.writeLong(pageID); + buffer.writeLong(persistentSize); --- End diff -- I can add a compatibility test. The value will just be a default (@michaelandrepearce wants me to use -1 instead of 0 and i guess we just would ignore it if not set for the count). Where's the journal test located in the test suite? > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355503#comment-16355503 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user clebertsuconic commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166630869 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/persistence/impl/journal/codec/PageCountPendingImpl.java --- @@ -62,21 +67,41 @@ public long getPageID() { return pageID; } + /** +* @return the persistentSize +*/ + @Override + public long getPersistentSize() { + return persistentSize; + } + + /** +* @param persistentSize the persistentSize to set +*/ + public void setPersistentSize(long persistentSize) { + this.persistentSize = persistentSize; + } + @Override public int getEncodeSize() { - return DataConstants.SIZE_LONG * 2; + return DataConstants.SIZE_LONG * 3; } @Override public void encode(ActiveMQBuffer buffer) { buffer.writeLong(queueID); buffer.writeLong(pageID); + buffer.writeLong(persistentSize); --- End diff -- OIC.. you already implemented it.. I would like to have a compatibility test validating that.. there's a journal test on the new compatibility testsuite already... but not one with paging.. and validating the counters... If you have a way to do it.. it would be great.. otherwise let me know and I will do it. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355500#comment-16355500 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166630379 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/persistence/impl/journal/codec/PageCountPendingImpl.java --- @@ -62,21 +67,41 @@ public long getPageID() { return pageID; } + /** +* @return the persistentSize +*/ + @Override + public long getPersistentSize() { + return persistentSize; + } + + /** +* @param persistentSize the persistentSize to set +*/ + public void setPersistentSize(long persistentSize) { + this.persistentSize = persistentSize; + } + @Override public int getEncodeSize() { - return DataConstants.SIZE_LONG * 2; + return DataConstants.SIZE_LONG * 3; } @Override public void encode(ActiveMQBuffer buffer) { buffer.writeLong(queueID); buffer.writeLong(pageID); + buffer.writeLong(persistentSize); --- End diff -- @clebertsuconic - here is an example of where the size is now persisted > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355496#comment-16355496 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on the issue: https://github.com/apache/activemq-artemis/pull/1853 @clebertsuconic - The metric in the page counters is persisted along with the page counters now. When the page counters are written to the journal that metric is encoded to disk and reloaded on restart. Older versions of the journal are compatible, they will just not have that size value loaded as it won't have been saved previously. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355475#comment-16355475 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user clebertsuconic commented on the issue: https://github.com/apache/activemq-artemis/pull/1853 @cshannon although looking at the PR now you added persistent size into page counters... but as I read it, it is not persistent... We should go all the way in.. i.e... persist the new size metric along with the page counters... or all the way back.. i.e... only give user's paging metrics through page size... Implementing the counter on the journal is no big deal.. but I would like to keep it compatible... older versions of the journal should still be compatible. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355408#comment-16355408 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on the issue: https://github.com/apache/activemq-artemis/pull/1853 Oh ok I get it now. Yeah you are right that the PagingStore interface has a method to get the number of pages and the page size which can be used to get an idea of consumption. This value is exposed through AddressControl so users can see it. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355388#comment-16355388 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user clebertsuconic commented on the issue: https://github.com/apache/activemq-artemis/pull/1853 I was just saying paging metrics should be exposed. Even if the files are to be GCed but if something happened and they are not the user should have the metrics exposed about pages. I think we do have them but i don’t remember now. I’m away from a computer and I can’t check now. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355384#comment-16355384 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on the issue: https://github.com/apache/activemq-artemis/pull/1853 @clebertsuconic - I don't think we need to re-calculate anything if the values are missing for paging. The point of this metric isn't to give the user information on storage consumption as that data already exists. They can find out how many page files are being used and disk usage already, etc. The point of these metrics is to know exactly how much data on a queue is left to consume. So there might be a 5 megabyte page file but only 2 messages left to consume and the rest of the file is just waiting to be GC'd. This metric only includes the 2 messages to be consumed and not the rest of the file. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355380#comment-16355380 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user clebertsuconic commented on the issue: https://github.com/apache/activemq-artemis/pull/1853 I think paging should be measured just in number of pages (files) on the address. That would give users an idea about storage consumption. It wouldn’t indeed be practical to load it to calculate sizes. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355376#comment-16355376 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166601621 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/persistence/impl/journal/codec/PageCountRecordInc.java --- @@ -26,17 +26,20 @@ private int value; + private long persistentSize; --- End diff -- @michaelandrepearce - That is fine I will make those values -1, but I don't think we would try and re-calculate a correct value. When loading the journal you'd have to re-load the messages to calculate from the paging store which we don't want to do. I think the proper behavior is to just ignore it and not use the value in the calculations. Then over time as new records come in the values would be filled in and correct. If I change the value to -1 on the record, are you thinking we should just have the getter do the negative check and return 0 if it's -1 or were you thinking to just return -1 from the getter and then check for negative everywhere else the value is used? > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355123#comment-16355123 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user michaelandrepearce commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166545869 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/persistence/impl/journal/codec/PageCountRecordInc.java --- @@ -26,17 +26,20 @@ private int value; + private long persistentSize; --- End diff -- Seems my comment is removed due to update. Should be -1. Reason for this is that if it isn't set e.g. have read up an old journal as such decode wont set it, and you know you need to calculate it to get a correct value. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354536#comment-16354536 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on the issue: https://github.com/apache/activemq-artemis/pull/1853 I updated the PR with getPersistentSize() everywhere along with comments to try and make it more clear what the value represented. I can tweak it some more if people still think it's not clear. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354492#comment-16354492 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on the issue: https://github.com/apache/activemq-artemis/pull/1853 Ok I can just use getPersistentSize() then for everything, I will update the PR shortly. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354489#comment-16354489 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user clebertsuconic commented on the issue: https://github.com/apache/activemq-artemis/pull/1853 I would include the sum you mentioned for large message. The large message file size plus the large message encode size. That translates fine as getPersjstentsjze > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354477#comment-16354477 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on the issue: https://github.com/apache/activemq-artemis/pull/1853 @clebertsuconic - hmm so I'm starting to like you original idea of just using getEncodeSize() more as the the message might be non-durable and in that case the encode size is still used but it isn't on disk so that would be a little misleading. However there is still the issue of what to do aboutt large messages? Maybe we need a name like fullEncodeSize() or something so that it is implies it is the full message including the body? > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354413#comment-16354413 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on the issue: https://github.com/apache/activemq-artemis/pull/1853 getPersistentSize() is fine with me as it specifies that the size represents the entire persisted size of the message > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354353#comment-16354353 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user clebertsuconic commented on the issue: https://github.com/apache/activemq-artemis/pull/1853 @cshannon you may not like getEncodeSize().. but doing message.getMessageSize() doesn't make it any clearer..we need to find a better name? maybe getPersistentSize()? message.get *Message* size to me has the same meaning of message.getSize(). > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354349#comment-16354349 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on the issue: https://github.com/apache/activemq-artemis/pull/1853 @clebertsuconic - Fair enough, but I think I should call it getMessageSize() instead (i already did this in a couple places) The reason why I would use that name is because the value is slightly different than getEncodeSize(). For the normal message case, getMessageSize() and geEncodeSize() are equal. But for a large message getMessageSize() will be getEncodeSize() + the body size. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354341#comment-16354341 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user clebertsuconic commented on the issue: https://github.com/apache/activemq-artemis/pull/1853 i would rename getSize() as getEncodeSize() on the interfaces then. getSize could be dubious and users may get the wrong expectation of the meaning. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354319#comment-16354319 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166401530 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/persistence/impl/journal/codec/PageCountPendingImpl.java --- @@ -43,6 +44,8 @@ public PageCountPendingImpl(long queueID, long pageID, int inc) { long pageID; + long size; --- End diff -- @michaelandrepearce - did you have in mind having the getter in the class handle the -1 (ie have the getSize() method check for -1 and return 0 if negative) or having the getSize() method return the -1 as is and then have the code using the value explicitly do the check for negative? > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354235#comment-16354235 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166388367 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/persistence/impl/journal/codec/PageCountPendingImpl.java --- @@ -43,6 +44,8 @@ public PageCountPendingImpl(long queueID, long pageID, int inc) { long pageID; + long size; --- End diff -- The issue is backwards compatibility. These values are now being stored to the journal and old records won't have a size. So by defaulting to a 0 value for when an old record is it makes everything simple. Yes I could make it -1 but then I have to add checks for -1 versus just using 0 as a default and then the logic is clean as we don't need to handle negative edge cases. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354223#comment-16354223 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user michaelandrepearce commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166387776 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/persistence/impl/journal/codec/PageCountPendingImpl.java --- @@ -43,6 +44,8 @@ public PageCountPendingImpl(long queueID, long pageID, int inc) { long pageID; + long size; --- End diff -- Eg it’s typical for -1 to be used to represent null vs 0 as no size , slight semantic difference but important none the less > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354222#comment-16354222 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user michaelandrepearce commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166387479 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/persistence/impl/journal/codec/PageCountPendingImpl.java --- @@ -43,6 +44,8 @@ public PageCountPendingImpl(long queueID, long pageID, int inc) { long pageID; + long size; --- End diff -- There’s a difference between no size (0) and not being set (-1) the logic calculating should be aware of this as -1 means it would need to be calculated > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354219#comment-16354219 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166386744 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/persistence/impl/journal/codec/PageCountPendingImpl.java --- @@ -43,6 +44,8 @@ public PageCountPendingImpl(long queueID, long pageID, int inc) { long pageID; + long size; --- End diff -- Why would we use -1 and not 0? These values default to 0 because we don't want the counter to change if the size is not set. If we default to -1 then the counters will go negative where as if they are 0 when not set then nothing changes with the counters which is what we want. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354213#comment-16354213 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user michaelandrepearce commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166385486 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/persistence/impl/journal/codec/PageCountRecord.java --- @@ -26,18 +26,21 @@ private long value; + private long size; --- End diff -- Default to -1 > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354214#comment-16354214 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user michaelandrepearce commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166385388 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/persistence/impl/journal/codec/PageCountPendingImpl.java --- @@ -43,6 +44,8 @@ public PageCountPendingImpl(long queueID, long pageID, int inc) { long pageID; + long size; --- End diff -- Should default be -1 > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354212#comment-16354212 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user michaelandrepearce commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166385592 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/persistence/impl/journal/codec/PageCountRecordInc.java --- @@ -26,17 +26,20 @@ private int value; + private long size; --- End diff -- Default to -1 > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354198#comment-16354198 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on the issue: https://github.com/apache/activemq-artemis/pull/1853 @clebertsuconic - No, I'm not trying to measure the size on the JVM. I want to measure the size of the messages in general (including how much space is on disk) as a metric to know how much data is left to consume on the Queue. The most accurate way is to use the encode size as this is the amount of space taken up on disk. For large messages this also includes the body size. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354131#comment-16354131 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user clebertsuconic commented on the issue: https://github.com/apache/activemq-artemis/pull/1853 You are getting the getEncodeSize() as the size... what is what you want to measure? how much memory it uses on the JVM? We have some calculations we do on paging.. perhaps that's what you want to expose? perhaps we should also review those calculations. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354104#comment-16354104 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on the issue: https://github.com/apache/activemq-artemis/pull/1853 @michaelandrepearce - ok made those 2 changes so it is ready for another look > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354043#comment-16354043 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166346352 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/paging/cursor/PagedReferenceImpl.java --- @@ -53,6 +53,8 @@ private Object protocolData; + private Long messageSize; --- End diff -- Yep, I'll set this to -1 to mean not set. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354042#comment-16354042 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166346269 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/paging/cursor/impl/PageSubscriptionImpl.java --- @@ -1309,4 +1336,68 @@ public void remove() { public void close() { } } + + + private static class PagePositionSize { --- End diff -- I will fix this and extend PagePosition as if the position is incremented that is fine as the size will just go back to 0 unless set. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353964#comment-16353964 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user michaelandrepearce commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166328137 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/paging/cursor/PagedReferenceImpl.java --- @@ -53,6 +53,8 @@ private Object protocolData; + private Long messageSize; --- End diff -- Could this be primitive? and either -1 or 0 to denote empty or not set in logic, just objects = GC, and we're putting effort in to remove them on the hot path as much as possible to keep GC pressure as low as possible. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353957#comment-16353957 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user michaelandrepearce commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166325405 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/paging/cursor/impl/PageSubscriptionCounterImpl.java --- @@ -374,8 +403,43 @@ private ItemOper(PageSubscriptionCounterImpl counter, long id, int add) { @Override public void afterCommit(Transaction tx) { for (ItemOper oper : operations) { -oper.counter.incrementProcessed(oper.id, oper.amount); +oper.counter.incrementProcessed(oper.id, oper.amount, oper.size); } } } + + private static class PendingCounter { + final long id; + final AtomicInteger count; --- End diff -- Yes this i what i meant , ill have a look at the updated PR in a bit/tomorrow. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353955#comment-16353955 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user michaelandrepearce commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166325203 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/paging/cursor/impl/PageSubscriptionImpl.java --- @@ -1309,4 +1336,68 @@ public void remove() { public void close() { } } + + + private static class PagePositionSize { --- End diff -- But this new object still holds PagePosition so its not been removed. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353835#comment-16353835 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on the issue: https://github.com/apache/activemq-artemis/pull/1853 @michaelandrepearce - I updated my PR, take a look and let me know what you think. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353816#comment-16353816 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166283882 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/paging/cursor/impl/PageSubscriptionCounterImpl.java --- @@ -374,8 +403,43 @@ private ItemOper(PageSubscriptionCounterImpl counter, long id, int add) { @Override public void afterCommit(Transaction tx) { for (ItemOper oper : operations) { -oper.counter.incrementProcessed(oper.id, oper.amount); +oper.counter.incrementProcessed(oper.id, oper.amount, oper.size); } } } + + private static class PendingCounter { + final long id; + final AtomicInteger count; --- End diff -- I'm about to push an updated PR using the field updater classes. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353768#comment-16353768 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166272476 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/paging/cursor/impl/PageSubscriptionImpl.java --- @@ -1309,4 +1336,68 @@ public void remove() { public void close() { } } + + + private static class PagePositionSize { --- End diff -- The reason I didn't extend is because PagePosition has methods like nextMessage() which increment the message number, etc. If that is called then the stored size doesn't really make sense anymore as it won't match which is why I just created a new object. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353767#comment-16353767 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166272177 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/persistence/impl/journal/codec/PageCountRecordIncV2.java --- @@ -0,0 +1,72 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.artemis.core.persistence.impl.journal.codec; + +import org.apache.activemq.artemis.api.core.ActiveMQBuffer; +import org.apache.activemq.artemis.utils.DataConstants; + +public class PageCountRecordIncV2 extends PageCountRecordInc { --- End diff -- Also a good point, I can get rid of the extra objects and just do a check on the buffer for readableBytes() is greater than 0. Thanks. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353766#comment-16353766 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166271726 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/paging/cursor/PageSubscription.java --- @@ -161,4 +165,9 @@ * @throws Exception */ void onDeletePage(Page deletedPage) throws Exception; + + AtomicLong getDeliveredCount(); --- End diff -- Good point, I will make that change. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353764#comment-16353764 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166271561 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/paging/cursor/impl/PageSubscriptionCounterImpl.java --- @@ -374,8 +403,43 @@ private ItemOper(PageSubscriptionCounterImpl counter, long id, int add) { @Override public void afterCommit(Transaction tx) { for (ItemOper oper : operations) { -oper.counter.incrementProcessed(oper.id, oper.amount); +oper.counter.incrementProcessed(oper.id, oper.amount, oper.size); } } } + + private static class PendingCounter { + final long id; + final AtomicInteger count; --- End diff -- What do you mean by atomic updater, do you mean using something like AtomicIntegerFieldUpdater ? > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353312#comment-16353312 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user michaelandrepearce commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166179369 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/paging/cursor/PageSubscription.java --- @@ -161,4 +165,9 @@ * @throws Exception */ void onDeletePage(Page deletedPage) throws Exception; + + AtomicLong getDeliveredCount(); --- End diff -- not sure would want to expose an atomic outside the object as it allows other code to update it unexpectedly, should probably be primitive and the object should return get on the atomic. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353309#comment-16353309 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user michaelandrepearce commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166179104 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/server/impl/QueuePendingMessageMetrics.java --- @@ -0,0 +1,108 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.artemis.core.server.impl; + +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicLong; + +import com.google.common.base.Preconditions; + +import org.apache.activemq.artemis.core.server.ActiveMQServerLogger; +import org.apache.activemq.artemis.core.server.MessageReference; +import org.apache.activemq.artemis.core.server.Queue; + +public class QueuePendingMessageMetrics { + + private final AtomicInteger messageCount = new AtomicInteger(0); --- End diff -- Could all these atomics be primitives with static atomic updates, just to reduce object size. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353304#comment-16353304 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user michaelandrepearce commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166178773 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/persistence/impl/journal/codec/PageCountRecordV2.java --- @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.artemis.core.persistence.impl.journal.codec; + +import org.apache.activemq.artemis.api.core.ActiveMQBuffer; +import org.apache.activemq.artemis.utils.DataConstants; + +public class PageCountRecordV2 extends PageCountRecord { --- End diff -- Ditto as per other comment if v2 is needed. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353303#comment-16353303 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user michaelandrepearce commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166178732 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/persistence/impl/journal/codec/PageCountRecordIncV2.java --- @@ -0,0 +1,72 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.artemis.core.persistence.impl.journal.codec; + +import org.apache.activemq.artemis.api.core.ActiveMQBuffer; +import org.apache.activemq.artemis.utils.DataConstants; + +public class PageCountRecordIncV2 extends PageCountRecordInc { --- End diff -- Is a new V2 needed? could have added size to PageCountRecord default to , and checked for readable bytes before read the long, to be back compatible. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353301#comment-16353301 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user michaelandrepearce commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166178431 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/paging/cursor/impl/PageSubscriptionImpl.java --- @@ -1309,4 +1336,68 @@ public void remove() { public void close() { } } + + + private static class PagePositionSize { --- End diff -- Why not just extend PagePosition (saves on objects) > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353300#comment-16353300 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user michaelandrepearce commented on a diff in the pull request: https://github.com/apache/activemq-artemis/pull/1853#discussion_r166178175 --- Diff: artemis-server/src/main/java/org/apache/activemq/artemis/core/paging/cursor/impl/PageSubscriptionCounterImpl.java --- @@ -374,8 +403,43 @@ private ItemOper(PageSubscriptionCounterImpl counter, long id, int add) { @Override public void afterCommit(Transaction tx) { for (ItemOper oper : operations) { -oper.counter.incrementProcessed(oper.id, oper.amount); +oper.counter.incrementProcessed(oper.id, oper.amount, oper.size); } } } + + private static class PendingCounter { + final long id; + final AtomicInteger count; --- End diff -- could this be an int, with a static atomic updater (ditto with the atomic below also) > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352728#comment-16352728 ] ASF GitHub Bot commented on ARTEMIS-1663: - Github user cshannon commented on the issue: https://github.com/apache/activemq-artemis/pull/1853 This PR is quite large so I would appreciate if multiple people take a look to make sure it looks ok. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1663) Add enhanced message count and size metrics for Queues
[ https://issues.apache.org/jira/browse/ARTEMIS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352726#comment-16352726 ] ASF GitHub Bot commented on ARTEMIS-1663: - GitHub user cshannon opened a pull request: https://github.com/apache/activemq-artemis/pull/1853 ARTEMIS-1663 - Add new message count and size metrics Adding new metrics for tracking message counts and sizes on a Queue. This includes tracking metrics for pending, delivering and scheduled messages. The paging store also tracks message size now. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cshannon/activemq-artemis ARTEMIS-1663 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/activemq-artemis/pull/1853.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1853 commit ce0b980b4e90a5c95ed88ac697d87cac0605789c Author: Christopher L. Shannon (cshannon) Date: 2018-02-05T18:24:31Z ARTEMIS-1663 - Add new message count and size metrics Adding new metrics for tracking message counts and sizes on a Queue. This includes tracking metrics for pending, delivering and scheduled messages. The paging store also tracks message size now. > Add enhanced message count and size metrics for Queues > -- > > Key: ARTEMIS-1663 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1663 > Project: ActiveMQ Artemis > Issue Type: New Feature > Components: Broker >Affects Versions: 2.4.0 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon >Priority: Major > > The purpose of this Jira is to enhance the message count metrics and to add > size metrics for Queues. By size metrics I mean the encoded size of all the > messages (for large messages this will include body as well). Right now we > track memory usage but it is also very useful to know exactly how much data > is pending on a Queue to be consumed. > The scope of this will be support for tracking the count/size of pending > messages, count/size of delivering messages, and count/size of scheduled > messages. For each of these categories durable size metrics will be tracked > as well. The paging store will also support tracking of size metrics as well. > These new metrics will be exposed through JMX to allow users to view the > statistics. Also by tracking these new values it will allow for enhanced > features in the future (ie new flow control options) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)