[jira] [Commented] (ARTEMIS-2213) Clock drift causing server halt
[ https://issues.apache.org/jira/browse/ARTEMIS-2213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16730899#comment-16730899 ] ASF GitHub Bot commented on ARTEMIS-2213: - Github user wy96f commented on the issue: https://github.com/apache/activemq-artemis/pull/2481 @franz1981 thanks for your advice. We've enabled gc log and run the nano program. Any news will be reported:) > Clock drift causing server halt > --- > > Key: ARTEMIS-2213 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2213 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: Broker >Affects Versions: 2.6.3 >Reporter: yangwei >Priority: Critical > > In our production cluster some brokers crashed. There is nothing unusual in > the dump stack. After digging into code, we found component was incorrectly > expired. When clock drifted back, left time was less than enter time. If the > component was not entered in default 12ms, it would be expired and server > was halted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-2214) Cache durable&priority in PagedReference to avoid blocks in consuming paged messages
[ https://issues.apache.org/jira/browse/ARTEMIS-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16730814#comment-16730814 ] ASF GitHub Bot commented on ARTEMIS-2214: - Github user michaelandrepearce commented on the issue: https://github.com/apache/activemq-artemis/pull/2482 Just looking over this a bit more. As the page ref needs to be a small as possible. The whole point of paging is to remove it from memory to be able to scale. With regards to priority i dont see this being accessed or affecting a hot path its only on access to message references. In regards to a consumer closing and having unhandled messages. This isnt on hot path imo, and shouldnt be tuning for that. > Cache durable&priority in PagedReference to avoid blocks in consuming paged > messages > > > Key: ARTEMIS-2214 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2214 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: Broker >Affects Versions: 2.6.3 >Reporter: Qihong Xu >Priority: Major > Attachments: stacks.txt > > > We recently performed a test on artemis broker and found a severe performance > issue. > When paged messages are being consumed, decrementMetrics in > QueuePendingMessageMetrics will try to ‘getMessage’ to check whether they are > durable or not. In this way queue will be locked for a long time because page > may be GCed and need to be reload entirely. Other operations rely on queue > will be blocked at this time, which cause a significant TPS drop. Detailed > stacks are attached below. > This also happens when consumer is closed and messages are pushed back to the > queue, artemis will check priority on return if these messages are paged. > To solve the issue, durable and priority need to be cached in PagedReference > just like messageID, transactionID and so on. I have applied a patch to fix > the issue. Any review is appreciated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARTEMIS-2218) AMQP 1.0 issue with Shared Subscription
[ https://issues.apache.org/jira/browse/ARTEMIS-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vuk Vasic updated ARTEMIS-2218: --- Affects Version/s: 2.2.0 2.4.0 2.5.0 2.6.1 > AMQP 1.0 issue with Shared Subscription > --- > > Key: ARTEMIS-2218 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2218 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: AMQP >Affects Versions: 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.6.1, 2.6.2, 2.6.3 > Environment: For testing i used Vormeo ActiveMQ Docker image. > Image is: *vromero/activemq-artemis:2.6.3-alpine* > >Reporter: Vuk Vasic >Priority: Blocker > Attachments: sample amqp 1.PNG, sample amqp 2.PNG, sample amqp 3.PNG > > > For some reason after running *ActiveMQ Artemis (2.6.3)* only first topic > created can be used for shared subscription. > Steps to reproduce: > # Create 2 consumers (with same Container ID and Subscription Name, > Capabilities are: shared, topic and global) and 1 producer. The Artemis has 2 > producers in admin panel and uses round-robin for deliver of messages. (See > sample amqp 1.PNG). Topic is named *sample4* > # Remove consumers and producers for *sample4* > # Create 2 consumers (with same Container ID and Subscription Name, > Capabilities are: shared, topic and global) and 1 producer. The Artemis has > only 1 producers in admin panel and it delivers message to only one listener. > Topic is named *sample5*. (See sample amqp 2.PNG) > # Repeat step 1 and now the topic *sample4* is also delivering to only one > consumer. (See sample amqp 3.PNG) > > It is interesting that if you use same topic multiple times, everything > works, But as soon as you create new topic no shared subscription over AMQP > 1.0 are not working. > > *Update* > This works fine on version 1.5.6. I also tested 2.3.0 and it does not work > too. > > *New update* > Version 2.0 and 2.1 also works fine. This seems to be an issue from version > 2.2 to 2.6.3. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARTEMIS-2218) AMQP 1.0 issue with Shared Subscription
[ https://issues.apache.org/jira/browse/ARTEMIS-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vuk Vasic updated ARTEMIS-2218: --- Description: For some reason after running *ActiveMQ Artemis (2.6.3)* only first topic created can be used for shared subscription. Steps to reproduce: # Create 2 consumers (with same Container ID and Subscription Name, Capabilities are: shared, topic and global) and 1 producer. The Artemis has 2 producers in admin panel and uses round-robin for deliver of messages. (See sample amqp 1.PNG). Topic is named *sample4* # Remove consumers and producers for *sample4* # Create 2 consumers (with same Container ID and Subscription Name, Capabilities are: shared, topic and global) and 1 producer. The Artemis has only 1 producers in admin panel and it delivers message to only one listener. Topic is named *sample5*. (See sample amqp 2.PNG) # Repeat step 1 and now the topic *sample4* is also delivering to only one consumer. (See sample amqp 3.PNG) It is interesting that if you use same topic multiple times, everything works, But as soon as you create new topic no shared subscription over AMQP 1.0 are not working. *Update* This works fine on version 1.5.6. I also tested 2.3.0 and it does not work too. *New update* Version 2.0 and 2.1 also works fine. This seems to be an issue from version 2.2 to 2.6.3. was: For some reason after running *ActiveMQ Artemis (2.6.3)* only first topic created can be used for shared subscription. Steps to reproduce: # Create 2 consumers (with same Container ID and Subscription Name, Capabilities are: shared, topic and global) and 1 producer. The Artemis has 2 producers in admin panel and uses round-robin for deliver of messages. (See sample amqp 1.PNG). Topic is named *sample4* # Remove consumers and producers for *sample4* # Create 2 consumers (with same Container ID and Subscription Name, Capabilities are: shared, topic and global) and 1 producer. The Artemis has only 1 producers in admin panel and it delivers message to only one listener. Topic is named *sample5*. (See sample amqp 2.PNG) # Repeat step 1 and now the topic *sample4* is also delivering to only one consumer. (See sample amqp 3.PNG) It is interesting that if you use same topic multiple times, everything works, But as soon as you create new topic no shared subscription over AMQP 1.0 are not working. *Update* This works fine on version 1.5.6. I also tested 2.3.0 and it does not work too. > AMQP 1.0 issue with Shared Subscription > --- > > Key: ARTEMIS-2218 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2218 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: AMQP >Affects Versions: 2.3.0, 2.6.0, 2.6.2, 2.6.3 > Environment: For testing i used Vormeo ActiveMQ Docker image. > Image is: *vromero/activemq-artemis:2.6.3-alpine* > >Reporter: Vuk Vasic >Priority: Blocker > Attachments: sample amqp 1.PNG, sample amqp 2.PNG, sample amqp 3.PNG > > > For some reason after running *ActiveMQ Artemis (2.6.3)* only first topic > created can be used for shared subscription. > Steps to reproduce: > # Create 2 consumers (with same Container ID and Subscription Name, > Capabilities are: shared, topic and global) and 1 producer. The Artemis has 2 > producers in admin panel and uses round-robin for deliver of messages. (See > sample amqp 1.PNG). Topic is named *sample4* > # Remove consumers and producers for *sample4* > # Create 2 consumers (with same Container ID and Subscription Name, > Capabilities are: shared, topic and global) and 1 producer. The Artemis has > only 1 producers in admin panel and it delivers message to only one listener. > Topic is named *sample5*. (See sample amqp 2.PNG) > # Repeat step 1 and now the topic *sample4* is also delivering to only one > consumer. (See sample amqp 3.PNG) > > It is interesting that if you use same topic multiple times, everything > works, But as soon as you create new topic no shared subscription over AMQP > 1.0 are not working. > > *Update* > This works fine on version 1.5.6. I also tested 2.3.0 and it does not work > too. > > *New update* > Version 2.0 and 2.1 also works fine. This seems to be an issue from version > 2.2 to 2.6.3. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARTEMIS-2218) AMQP 1.0 issue with Shared Subscription
[ https://issues.apache.org/jira/browse/ARTEMIS-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vuk Vasic updated ARTEMIS-2218: --- Description: For some reason after running *ActiveMQ Artemis (2.6.3)* only first topic created can be used for shared subscription. Steps to reproduce: # Create 2 consumers (with same Container ID and Subscription Name, Capabilities are: shared, topic and global) and 1 producer. The Artemis has 2 producers in admin panel and uses round-robin for deliver of messages. (See sample amqp 1.PNG). Topic is named *sample4* # Remove consumers and producers for *sample4* # Create 2 consumers (with same Container ID and Subscription Name, Capabilities are: shared, topic and global) and 1 producer. The Artemis has only 1 producers in admin panel and it delivers message to only one listener. Topic is named *sample5*. (See sample amqp 2.PNG) # Repeat step 1 and now the topic *sample4* is also delivering to only one consumer. (See sample amqp 3.PNG) It is interesting that if you use same topic multiple times, everything works, But as soon as you create new topic no shared subscription over AMQP 1.0 are not working. *Update* This works fine on version 1.5.6. I also tested 2.3.0 and it does not work too. was: For some reason after running *ActiveMQ Artemis (2.6.3)* only first topic created can be used for shared subscription. Steps to reproduce: # Create 2 consumers (with same Container ID and Subscription Name, Capabilities are: shared, topic and global) and 1 producer. The Artemis has 2 producers in admin panel and uses round-robin for deliver of messages. (See sample amqp 1.PNG). Topic is named *sample4* # Remove consumers and producers for *sample4* # Create 2 consumers (with same Container ID and Subscription Name, Capabilities are: shared, topic and global) and 1 producer. The Artemis has only 1 producers in admin panel and it delivers message to only one listener. Topic is named *sample5*. (See sample amqp 2.PNG) # Repeat step 1 and now the topic *sample4* is also delivering to only one consumer. (See sample amqp 3.PNG) It is interesting that if you use same topic multiple times, everything works, But as soon as you create new topic no shared subscription over AMQP 1.0 are not working. > AMQP 1.0 issue with Shared Subscription > --- > > Key: ARTEMIS-2218 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2218 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: AMQP >Affects Versions: 2.3.0, 2.6.0, 2.6.2, 2.6.3 > Environment: For testing i used Vormeo ActiveMQ Docker image. > Image is: *vromero/activemq-artemis:2.6.3-alpine* > >Reporter: Vuk Vasic >Priority: Blocker > Attachments: sample amqp 1.PNG, sample amqp 2.PNG, sample amqp 3.PNG > > > For some reason after running *ActiveMQ Artemis (2.6.3)* only first topic > created can be used for shared subscription. > Steps to reproduce: > # Create 2 consumers (with same Container ID and Subscription Name, > Capabilities are: shared, topic and global) and 1 producer. The Artemis has 2 > producers in admin panel and uses round-robin for deliver of messages. (See > sample amqp 1.PNG). Topic is named *sample4* > # Remove consumers and producers for *sample4* > # Create 2 consumers (with same Container ID and Subscription Name, > Capabilities are: shared, topic and global) and 1 producer. The Artemis has > only 1 producers in admin panel and it delivers message to only one listener. > Topic is named *sample5*. (See sample amqp 2.PNG) > # Repeat step 1 and now the topic *sample4* is also delivering to only one > consumer. (See sample amqp 3.PNG) > > It is interesting that if you use same topic multiple times, everything > works, But as soon as you create new topic no shared subscription over AMQP > 1.0 are not working. > > *Update* > This works fine on version 1.5.6. I also tested 2.3.0 and it does not work > too. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARTEMIS-2218) AMQP 1.0 issue with Shared Subscription
[ https://issues.apache.org/jira/browse/ARTEMIS-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vuk Vasic updated ARTEMIS-2218: --- Affects Version/s: (was: 2.6.1) 2.3.0 > AMQP 1.0 issue with Shared Subscription > --- > > Key: ARTEMIS-2218 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2218 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: AMQP >Affects Versions: 2.3.0, 2.6.0, 2.6.2, 2.6.3 > Environment: For testing i used Vormeo ActiveMQ Docker image. > Image is: *vromero/activemq-artemis:2.6.3-alpine* > >Reporter: Vuk Vasic >Priority: Blocker > Attachments: sample amqp 1.PNG, sample amqp 2.PNG, sample amqp 3.PNG > > > For some reason after running *ActiveMQ Artemis (2.6.3)* only first topic > created can be used for shared subscription. > Steps to reproduce: > # Create 2 consumers (with same Container ID and Subscription Name, > Capabilities are: shared, topic and global) and 1 producer. The Artemis has 2 > producers in admin panel and uses round-robin for deliver of messages. (See > sample amqp 1.PNG). Topic is named *sample4* > # Remove consumers and producers for *sample4* > # Create 2 consumers (with same Container ID and Subscription Name, > Capabilities are: shared, topic and global) and 1 producer. The Artemis has > only 1 producers in admin panel and it delivers message to only one listener. > Topic is named *sample5*. (See sample amqp 2.PNG) > # Repeat step 1 and now the topic *sample4* is also delivering to only one > consumer. (See sample amqp 3.PNG) > > It is interesting that if you use same topic multiple times, everything > works, But as soon as you create new topic no shared subscription over AMQP > 1.0 are not working. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARTEMIS-2218) AMQP 1.0 issue with Shared Subscription
Vuk Vasic created ARTEMIS-2218: -- Summary: AMQP 1.0 issue with Shared Subscription Key: ARTEMIS-2218 URL: https://issues.apache.org/jira/browse/ARTEMIS-2218 Project: ActiveMQ Artemis Issue Type: Bug Components: AMQP Affects Versions: 2.6.3, 2.6.2, 2.6.1, 2.6.0 Environment: For testing i used Vormeo ActiveMQ Docker image. Image is: *vromero/activemq-artemis:2.6.3-alpine* Reporter: Vuk Vasic Attachments: sample amqp 1.PNG, sample amqp 2.PNG, sample amqp 3.PNG For some reason after running *ActiveMQ Artemis (2.6.3)* only first topic created can be used for shared subscription. Steps to reproduce: # Create 2 consumers (with same Container ID and Subscription Name, Capabilities are: shared, topic and global) and 1 producer. The Artemis has 2 producers in admin panel and uses round-robin for deliver of messages. (See sample amqp 1.PNG). Topic is named *sample4* # Remove consumers and producers for *sample4* # Create 2 consumers (with same Container ID and Subscription Name, Capabilities are: shared, topic and global) and 1 producer. The Artemis has only 1 producers in admin panel and it delivers message to only one listener. Topic is named *sample5*. (See sample amqp 2.PNG) # Repeat step 1 and now the topic *sample4* is also delivering to only one consumer. (See sample amqp 3.PNG) It is interesting that if you use same topic multiple times, everything works, But as soon as you create new topic no shared subscription over AMQP 1.0 are not working. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-2216) Use a specific executor for pageSyncTimer
[ https://issues.apache.org/jira/browse/ARTEMIS-2216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16730776#comment-16730776 ] ASF GitHub Bot commented on ARTEMIS-2216: - Github user michaelandrepearce commented on the issue: https://github.com/apache/activemq-artemis/pull/2484 Comments on this are: We have shared executor and thread pools as a way to pool these its part artemis design so that these can be controlled, eg the number of threads are configurable. And theres uses cases for why, we dont just add executors. If we add a new pool it should be configurable like others. Whilst your use case this significantly improves things. As you note this change isnt entirely positive to all users as it does impact use cases where people may care more on the consumer side (i am one one of those). If anything if the broker is saturated its more important consumers for us can dequeue and if anything producers backpressure. As if consumers cant keep up and producer peak continues you can very soon end up situation where paging depth will just grow and grow. It would be good to get more stats therefore on how as producer rates ramup how consumer rates ramp down. And crticial cross over points. Along with number of remote consumers and producers. All of this could be mute if there is a way to eliminate the negative effect this change has. > Use a specific executor for pageSyncTimer > - > > Key: ARTEMIS-2216 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2216 > Project: ActiveMQ Artemis > Issue Type: Improvement >Affects Versions: 2.6.3 >Reporter: Qihong Xu >Priority: Major > > Improving throughput on paging mode is one of our concerns since our cluster > uses paging a lot. > We found that pageSyncTimer in PagingStoreImpl shared the same executor with > pageCursorProvider from thread pool. In heavy load scenario like hundreds of > consumers receiving messages simultaneously, it became difficult for > pageSyncTimer to get the executor due to race condition. Therefore page sync > was delayed and producers suffered low throughput. > > To achieve higher performance we assign a specific executor to pageSyncTimer > to avoid racing. And we run a small-scale test on a single modified broker. > > Broker: 4C/8G/500G SSD > Producer: 200 threads, non-transactional send > Consumer 200 threads, transactional receive > Message text size: 100-200 bytes randomly > AddressFullPolicy: PAGE > > Test result： > | |Only Send TPS|Only Receive TPS|Send&Receive TPS| > |Original ver|38k|33k|3k/30k| > |Modified ver|38k|34k|30k/12.5k| > > The chart above shows that on modified broker send TPS improves from “poor” > to “extremely fast”, while receive TPS drops from “extremely fast” to > “not-bad” under heavy load. Considering consumer systems usually have a long > processing chain after receiving messages, we don’t need too fast receive > TPS. Instead, we want to guarantee send TPS to cope with traffic peak and > lower producer’s delay time. Moreover, send and receive TPS in total raises > from 33k to about 43k. From all above this trade-off seems beneficial and > acceptable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-2214) Cache durable&priority in PagedReference to avoid blocks in consuming paged messages
[ https://issues.apache.org/jira/browse/ARTEMIS-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16730623#comment-16730623 ] ASF GitHub Bot commented on ARTEMIS-2214: - Github user qihongxu commented on the issue: https://github.com/apache/activemq-artemis/pull/2482 > Nice catch!!! > If you care about checking perf of paging please try to profile the broker with https://github.com/jvm-profiling-tools/async-profiler using lock/cpu events too: it will allows to get profile data without affecting performance, helping to find issues like this one :) Thanks for your advice! We used to record performance from producer/consumer side. Later on we will try to dig into broker side by using tools as recommended :) > Cache durable&priority in PagedReference to avoid blocks in consuming paged > messages > > > Key: ARTEMIS-2214 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2214 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: Broker >Affects Versions: 2.6.3 >Reporter: Qihong Xu >Priority: Major > Attachments: stacks.txt > > > We recently performed a test on artemis broker and found a severe performance > issue. > When paged messages are being consumed, decrementMetrics in > QueuePendingMessageMetrics will try to ‘getMessage’ to check whether they are > durable or not. In this way queue will be locked for a long time because page > may be GCed and need to be reload entirely. Other operations rely on queue > will be blocked at this time, which cause a significant TPS drop. Detailed > stacks are attached below. > This also happens when consumer is closed and messages are pushed back to the > queue, artemis will check priority on return if these messages are paged. > To solve the issue, durable and priority need to be cached in PagedReference > just like messageID, transactionID and so on. I have applied a patch to fix > the issue. Any review is appreciated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-2217) ‘MQTTSessionState’ in the ‘SESSIONS ConcurrentHashMap’ never be removed
[ https://issues.apache.org/jira/browse/ARTEMIS-2217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16730615#comment-16730615 ] ASF GitHub Bot commented on ARTEMIS-2217: - GitHub user onlyMIT opened a pull request: https://github.com/apache/activemq-artemis/pull/2485 ARTEMIS-2217 ‘MQTTSessionState’ in the ‘SESSIONS ConcurrentHashMap’ n… …ever be removed ‘MQTTSessionState’ in the ‘SESSIONS ConcurrentHashMap’ should be removed when the conusmer （cleanSession is true） connection is closed You can merge this pull request into a Git repository by running: $ git pull https://github.com/onlyMIT/activemq-artemis ARTEMIS-2217 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/activemq-artemis/pull/2485.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2485 commit c4c951c3a2cf74f211d9b7c17cad48f27b725ff5 Author: onlyMIT Date: 2018-12-29T08:53:04Z ARTEMIS-2217 ‘MQTTSessionState’ in the ‘SESSIONS ConcurrentHashMap’ never be removed ‘MQTTSessionState’ in the ‘SESSIONS ConcurrentHashMap’ should be removed when the conusmer （cleanSession is true） connection is closed > ‘MQTTSessionState’ in the ‘SESSIONS ConcurrentHashMap’ never be removed > --- > > Key: ARTEMIS-2217 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2217 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: MQTT >Affects Versions: 2.7.0 >Reporter: Shiping Liang >Priority: Major > > ‘MQTTSessionState’ in the ‘SESSIONS ConcurrentHashMap’ should be removed when > the conusmer （cleanSession is true） connection is closed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-2215) largemessage have been consumed but not deleted from the disk during backup and live sync
[
https://issues.apache.org/jira/browse/ARTEMIS-2215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16730609#comment-16730609
]
ASF GitHub Bot commented on ARTEMIS-2215:
-
Github user franz1981 commented on a diff in the pull request:
https://github.com/apache/activemq-artemis/pull/2483#discussion_r244470771
--- Diff:
artemis-server/src/main/java/org/apache/activemq/artemis/core/persistence/impl/journal/JournalStorageManager.java
---
@@ -461,8 +462,7 @@ void deleteLargeMessageFile(final LargeServerMessage
largeServerMessage) throws
try {
if (isReplicated() && replicator.isSynchronizing()) {
synchronized (largeMessagesToDelete) {
--- End diff --
If it's now using CHM there is no need to sync on it
> largemessage have been consumed but not deleted from the disk during backup
> and live sync
> -
>
> Key: ARTEMIS-2215
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2215
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Affects Versions: 2.6.3
>Reporter: yuebao
>Priority: Major
>
> During the backup and live synchronization, the client consumes the
> largemessage, then the live crash(the performCachedLargeMessageDeletes method
> is not executed), after the live startup, the largemessages that have been
> consumed are not deleted from the disk.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARTEMIS-2215) largemessage have been consumed but not deleted from the disk during backup and live sync
[
https://issues.apache.org/jira/browse/ARTEMIS-2215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16730610#comment-16730610
]
ASF GitHub Bot commented on ARTEMIS-2215:
-
Github user franz1981 commented on a diff in the pull request:
https://github.com/apache/activemq-artemis/pull/2483#discussion_r244470859
--- Diff:
artemis-server/src/main/java/org/apache/activemq/artemis/core/persistence/impl/journal/AbstractJournalStorageManager.java
---
@@ -193,7 +193,7 @@ public static JournalContent getType(byte type) {
protected final Map
mapPersistedAddressSettings = new ConcurrentHashMap<>();
- protected final Set largeMessagesToDelete = new HashSet<>();
+ protected final Map largeMessagesToDelete =
new ConcurrentHashMap<>();
--- End diff --
It is possible to use a primitive version of the map ie using primitive
longs instead of boxed types
> largemessage have been consumed but not deleted from the disk during backup
> and live sync
> -
>
> Key: ARTEMIS-2215
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2215
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Affects Versions: 2.6.3
>Reporter: yuebao
>Priority: Major
>
> During the backup and live synchronization, the client consumes the
> largemessage, then the live crash(the performCachedLargeMessageDeletes method
> is not executed), after the live startup, the largemessages that have been
> consumed are not deleted from the disk.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (ARTEMIS-2217) ‘MQTTSessionState’ in the ‘SESSIONS ConcurrentHashMap’ never be removed
Shiping Liang created ARTEMIS-2217: -- Summary: ‘MQTTSessionState’ in the ‘SESSIONS ConcurrentHashMap’ never be removed Key: ARTEMIS-2217 URL: https://issues.apache.org/jira/browse/ARTEMIS-2217 Project: ActiveMQ Artemis Issue Type: Bug Components: MQTT Affects Versions: 2.7.0 Reporter: Shiping Liang ‘MQTTSessionState’ in the ‘SESSIONS ConcurrentHashMap’ should be removed when the conusmer （cleanSession is true） connection is closed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARTEMIS-2216) Use a specific executor for pageSyncTimer
[ https://issues.apache.org/jira/browse/ARTEMIS-2216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qihong Xu updated ARTEMIS-2216: --- Description: Improving throughput on paging mode is one of our concerns since our cluster uses paging a lot. We found that pageSyncTimer in PagingStoreImpl shared the same executor with pageCursorProvider from thread pool. In heavy load scenario like hundreds of consumers receiving messages simultaneously, it became difficult for pageSyncTimer to get the executor due to race condition. Therefore page sync was delayed and producers suffered low throughput. To achieve higher performance we assign a specific executor to pageSyncTimer to avoid racing. And we run a small-scale test on a single modified broker. Broker: 4C/8G/500G SSD Producer: 200 threads, non-transactional send Consumer 200 threads, transactional receive Message text size: 100-200 bytes randomly AddressFullPolicy: PAGE Test result： | |Only Send TPS|Only Receive TPS|Send&Receive TPS| |Original ver|38k|33k|3k/30k| |Modified ver|38k|34k|30k/12.5k| The chart above shows that on modified broker send TPS improves from “poor” to “extremely fast”, while receive TPS drops from “extremely fast” to “not-bad” under heavy load. Considering consumer systems usually have a long processing chain after receiving messages, we don’t need too fast receive TPS. Instead, we want to guarantee send TPS to cope with traffic peak and lower producer’s delay time. Moreover, send and receive TPS in total raises from 33k to about 43k. From all above this trade-off seems beneficial and acceptable. was: Improve paging throughput by using a specific executor for pageSyncTimer Improving throughput on paging mode is one of our concerns since our cluster uses paging a lot. We found that pageSyncTimer in PagingStoreImpl shared the same executor with pageCursorProvider from thread pool. In heavy load scenario like hundreds of consumers receiving messages simultaneously, it became difficult for pageSyncTimer to get the executor due to race condition. Therefore page sync was delayed and producers suffered low throughput. To achieve higher performance we assign a specific executor to pageSyncTimer to avoid racing. And we run a small-scale test on a single modified broker. Broker: 4C/8G/500G SSD Producer: 200 threads, non-transactional send Consumer 200 threads, transactional receive Message text size: 100-200 bytes randomly AddressFullPolicy: PAGE Test result： | |Only Send TPS|Only Receive TPS|Send&Receive TPS| |Original ver|38k|33k|3k/30k| |Modified ver|38k|34k|30k/12.5k| The chart above shows that on modified broker send TPS improves from “poor” to “extremely fast”, while receive TPS drops from “extremely fast” to “not-bad” under heavy load. Considering consumer systems usually have a long processing chain after receiving messages, we don’t need too fast receive TPS. Instead, we want to guarantee send TPS to cope with traffic peak and lower producer’s delay time. Moreover, send and receive TPS in total raises from 33k to about 43k. From all above this trade-off seems beneficial and acceptable. > Use a specific executor for pageSyncTimer > - > > Key: ARTEMIS-2216 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2216 > Project: ActiveMQ Artemis > Issue Type: Improvement >Affects Versions: 2.6.3 >Reporter: Qihong Xu >Priority: Major > > Improving throughput on paging mode is one of our concerns since our cluster > uses paging a lot. > We found that pageSyncTimer in PagingStoreImpl shared the same executor with > pageCursorProvider from thread pool. In heavy load scenario like hundreds of > consumers receiving messages simultaneously, it became difficult for > pageSyncTimer to get the executor due to race condition. Therefore page sync > was delayed and producers suffered low throughput. > > To achieve higher performance we assign a specific executor to pageSyncTimer > to avoid racing. And we run a small-scale test on a single modified broker. > > Broker: 4C/8G/500G SSD > Producer: 200 threads, non-transactional send > Consumer 200 threads, transactional receive > Message text size: 100-200 bytes randomly > AddressFullPolicy: PAGE > > Test result： > | |Only Send TPS|Only Receive TPS|Send&Receive TPS| > |Original ver|38k|33k|3k/30k| > |Modified ver|38k|34k|30k/12.5k| > > The chart above shows that on modified broker send TPS improves from “poor” > to “extremely fast”, while receive TPS drops from “extremely fast” to > “not-bad” under heavy load. Considering consumer systems usually have a long > processing chain after receiving messages, we don’t need too fast receive > TPS. Instead, we want to guarantee send TPS to cope with traffic peak and > lower producer’s delay time. Mor
[jira] [Commented] (ARTEMIS-2215) largemessage have been consumed but not deleted from the disk during backup and live sync
[ https://issues.apache.org/jira/browse/ARTEMIS-2215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16730596#comment-16730596 ] ASF GitHub Bot commented on ARTEMIS-2215: - GitHub user CNNJYB opened a pull request: https://github.com/apache/activemq-artemis/pull/2483 ARTEMIS-2215 largemessage have been consumed but not deleted During the backup and live synchronization, the client consumes the largemessage, then the live crash(the performCachedLargeMessageDeletes method is not executed), after the live startup, the largemessages that have been consumed are not deleted from the disk. You can merge this pull request into a Git repository by running: $ git pull https://github.com/CNNJYB/activemq-artemis dev-largemessagenotdelete Alternatively you can review and apply these changes as the patch at: https://github.com/apache/activemq-artemis/pull/2483.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2483 commit bf28c751af152956d1a12aa2237502c0a14fc4e8 Author: yb <17061955@...> Date: 2018-12-29T08:09:48Z ARTEMIS-2215 largemessage have been consumed but not deleted from the disk during backup and live sync > largemessage have been consumed but not deleted from the disk during backup > and live sync > - > > Key: ARTEMIS-2215 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2215 > Project: ActiveMQ Artemis > Issue Type: Bug >Affects Versions: 2.6.3 >Reporter: yuebao >Priority: Major > > During the backup and live synchronization, the client consumes the > largemessage, then the live crash(the performCachedLargeMessageDeletes method > is not executed), after the live startup, the largemessages that have been > consumed are not deleted from the disk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARTEMIS-2216) Use a specific executor for pageSyncTimer
Qihong Xu created ARTEMIS-2216: -- Summary: Use a specific executor for pageSyncTimer Key: ARTEMIS-2216 URL: https://issues.apache.org/jira/browse/ARTEMIS-2216 Project: ActiveMQ Artemis Issue Type: Improvement Affects Versions: 2.6.3 Reporter: Qihong Xu Improve paging throughput by using a specific executor for pageSyncTimer Improving throughput on paging mode is one of our concerns since our cluster uses paging a lot. We found that pageSyncTimer in PagingStoreImpl shared the same executor with pageCursorProvider from thread pool. In heavy load scenario like hundreds of consumers receiving messages simultaneously, it became difficult for pageSyncTimer to get the executor due to race condition. Therefore page sync was delayed and producers suffered low throughput. To achieve higher performance we assign a specific executor to pageSyncTimer to avoid racing. And we run a small-scale test on a single modified broker. Broker: 4C/8G/500G SSD Producer: 200 threads, non-transactional send Consumer 200 threads, transactional receive Message text size: 100-200 bytes randomly AddressFullPolicy: PAGE Test result： | |Only Send TPS|Only Receive TPS|Send&Receive TPS| |Original ver|38k|33k|3k/30k| |Modified ver|38k|34k|30k/12.5k| The chart above shows that on modified broker send TPS improves from “poor” to “extremely fast”, while receive TPS drops from “extremely fast” to “not-bad” under heavy load. Considering consumer systems usually have a long processing chain after receiving messages, we don’t need too fast receive TPS. Instead, we want to guarantee send TPS to cope with traffic peak and lower producer’s delay time. Moreover, send and receive TPS in total raises from 33k to about 43k. From all above this trade-off seems beneficial and acceptable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARTEMIS-2215) largemessage have been consumed but not deleted from the disk during backup and live sync
yuebao created ARTEMIS-2215: --- Summary: largemessage have been consumed but not deleted from the disk during backup and live sync Key: ARTEMIS-2215 URL: https://issues.apache.org/jira/browse/ARTEMIS-2215 Project: ActiveMQ Artemis Issue Type: Bug Affects Versions: 2.6.3 Reporter: yuebao During the backup and live synchronization, the client consumes the largemessage, then the live crash(the performCachedLargeMessageDeletes method is not executed), after the live startup, the largemessages that have been consumed are not deleted from the disk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
