[jira] [Updated] (ARTEMIS-2586) Inifinite Block in AMQ212054 after transient DB-error
[ https://issues.apache.org/jira/browse/ARTEMIS-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rico Neubauer updated ARTEMIS-2586: --- Component/s: (was: AMQP) > Inifinite Block in AMQ212054 after transient DB-error > - > > Key: ARTEMIS-2586 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2586 > Project: ActiveMQ Artemis > Issue Type: Bug >Affects Versions: 2.10.1 > Environment: This is Ubuntu 18.04 and Oracle DB, but don't think it's > that relevant for the issue. >Reporter: Rico Neubauer >Priority: Major > Attachments: 2019-11-28_threaddump_01.txt, > 2019-12-04_threaddump_01.txt, Message-Counts.png, artemis.xml, > initial-error.txt, log-extract.txt, writerIndex-Credits.PNG > > > Hi, > Would like to describe a quite severe situation which was expirienced in a > long-running test with 2 out of 3 instances/machines. > We are running Karaf with Artemis 2.10.1. > After some time (see screenshot), first one, then after a while a 2nd > instance came to a complete stop. > Looking into the logs and thread-dumps revealed the following (same for bith > instances): > # There was a temporary problem connecting to the DB ({{connection reset by > peer}}and {{Closed Connection }}) > # This resulted (due to handling on our side) in an > {{IllegalStateException}}/{{Error during two phase commit}} being thrown back > to Artemis. > # After this, there is no messaging possible anymore at all and the > following log repeats: > {noformat} > AMQ212054: Destination address=DLQ is blocked. If the system is configured to > block make sure you consume messages on this configuration.{noformat} > (system is not configured to block, see attached config) > which comes from threads like these, trying to obtain credits for sending: > > {noformat} > "Thread-93 (ActiveMQ-client-global-threads)" Id=2001 in TIMED_WAITING on > lock=java.util.concurrent.Semaphore$NonfairSync@1f9a57e0 > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1332) > at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:582) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerCreditsImpl.actualAcquire(ClientProducerCreditsImpl.java:73) > at > org.apache.activemq.artemis.core.client.impl.AbstractProducerCreditsImpl.acquireCredits(AbstractProducerCreditsImpl.java:77) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:301) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:275) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:128) > at > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:485) > at > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:195) > at > com.seeburger.engine.jms.MessageReceiverBase.sendToDLQ(MessageReceiverBase.java:571) > at > com.seeburger.engine.jms.MessageReceiverBase.handleException(MessageReceiverBase.java:493) > at > com.seeburger.engine.jms.MessageReceiverBase.onMessage(MessageReceiverBase.java:387) > at > org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:110) > at > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1031) > at > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50) > at > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1154) > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) > at > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) > at > org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$431/1769898766.run(Unknown > Source) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > Locked synchronizers: count = 1 > - java.util.concurrent.ThreadPoolExecutor$Worker@bc49fcf > {noformat} > which will never succeed, since the credits seem to no
[jira] [Updated] (ARTEMIS-2586) Inifinite Block in AMQ212054 after transient DB-error
[ https://issues.apache.org/jira/browse/ARTEMIS-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rico Neubauer updated ARTEMIS-2586: --- Attachment: artemis.xml > Inifinite Block in AMQ212054 after transient DB-error > - > > Key: ARTEMIS-2586 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2586 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: AMQP >Affects Versions: 2.10.1 > Environment: This is Ubuntu 18.04 and Oracle DB, but don't think it's > that relevant for the issue. >Reporter: Rico Neubauer >Priority: Major > Attachments: 2019-11-28_threaddump_01.txt, > 2019-12-04_threaddump_01.txt, Message-Counts.png, artemis.xml, > initial-error.txt, log-extract.txt, writerIndex-Credits.PNG > > > Hi, > Would like to describe a quite severe situation which was expirienced in a > long-running test with 2 out of 3 instances/machines. > We are running Karaf with Artemis 2.10.1. > After some time (see screenshot), first one, then after a while a 2nd > instance came to a complete stop. > Looking into the logs and thread-dumps revealed the following (same for bith > instances): > # There was a temporary problem connecting to the DB ({{connection reset by > peer}}and {{Closed Connection }}) > # This resulted (due to handling on our side) in an > {{IllegalStateException}}/{{Error during two phase commit}} being thrown back > to Artemis. > # After this, there is no messaging possible anymore at all and the > following log repeats: > {noformat} > AMQ212054: Destination address=DLQ is blocked. If the system is configured to > block make sure you consume messages on this configuration.{noformat} > (system is not configured to block, see attached config) > which comes from threads like these, trying to obtain credits for sending: > > {noformat} > "Thread-93 (ActiveMQ-client-global-threads)" Id=2001 in TIMED_WAITING on > lock=java.util.concurrent.Semaphore$NonfairSync@1f9a57e0 > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1332) > at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:582) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerCreditsImpl.actualAcquire(ClientProducerCreditsImpl.java:73) > at > org.apache.activemq.artemis.core.client.impl.AbstractProducerCreditsImpl.acquireCredits(AbstractProducerCreditsImpl.java:77) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:301) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:275) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:128) > at > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:485) > at > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:195) > at > com.seeburger.engine.jms.MessageReceiverBase.sendToDLQ(MessageReceiverBase.java:571) > at > com.seeburger.engine.jms.MessageReceiverBase.handleException(MessageReceiverBase.java:493) > at > com.seeburger.engine.jms.MessageReceiverBase.onMessage(MessageReceiverBase.java:387) > at > org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:110) > at > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1031) > at > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50) > at > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1154) > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) > at > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) > at > org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$431/1769898766.run(Unknown > Source) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > Locked synchronizers: count = 1 > - java.util.concurrent.ThreadPoolExecutor$Worker@bc49fcf > {noformat} > which will never succeed, since the
[jira] [Updated] (ARTEMIS-2586) Inifinite Block in AMQ212054 after transient DB-error
[ https://issues.apache.org/jira/browse/ARTEMIS-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rico Neubauer updated ARTEMIS-2586: --- Description: Hi, Would like to describe a quite severe situation which was expirienced in a long-running test with 2 out of 3 instances/machines. We are running Karaf with Artemis 2.10.1. After some time (see screenshot), first one, then after a while a 2nd instance came to a complete stop. Looking into the logs and thread-dumps revealed the following (same for bith instances): # There was a temporary problem connecting to the DB ({{connection reset by peer}}and {{Closed Connection }}) # This resulted (due to handling on our side) in an {{IllegalStateException}}/{{Error during two phase commit}} being thrown back to Artemis. # After this, there is no messaging possible anymore at all and the following log repeats: {noformat} AMQ212054: Destination address=DLQ is blocked. If the system is configured to block make sure you consume messages on this configuration.{noformat} (system is not configured to block, see attached config) which comes from threads like these, trying to obtain credits for sending: {noformat} "Thread-93 (ActiveMQ-client-global-threads)" Id=2001 in TIMED_WAITING on lock=java.util.concurrent.Semaphore$NonfairSync@1f9a57e0 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1332) at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:582) at org.apache.activemq.artemis.core.client.impl.ClientProducerCreditsImpl.actualAcquire(ClientProducerCreditsImpl.java:73) at org.apache.activemq.artemis.core.client.impl.AbstractProducerCreditsImpl.acquireCredits(AbstractProducerCreditsImpl.java:77) at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:301) at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:275) at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:128) at org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:485) at org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:195) at com.seeburger.engine.jms.MessageReceiverBase.sendToDLQ(MessageReceiverBase.java:571) at com.seeburger.engine.jms.MessageReceiverBase.handleException(MessageReceiverBase.java:493) at com.seeburger.engine.jms.MessageReceiverBase.onMessage(MessageReceiverBase.java:387) at org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:110) at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1031) at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50) at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1154) at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) at org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$431/1769898766.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) Locked synchronizers: count = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@bc49fcf {noformat} which will never succeed, since the credits seem to no suffice (see heap-dump screenshot) >From my point of view, the thrown IllegalStateException should not lead to the >system going in this non-recoverable state, what do you think, is there >something that can be enhanced? [Fastthread-Link|https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjAvMDEvMy8tLTIwMTktMTItMDRfdGhyZWFkZHVtcF8wMS50eHQtLTEzLTM4LTE1OzstLTIwMTktMTEtMjhfdGhyZWFkZHVtcF8wMS50eHQtLTEzLTM4LTE1] In case it helps: The 2 instances are still in this state (since September) and I can fetch additional information or debug them on request. was: Hi, Would like to describe a quite severe situation which was expirienced in a long-running test with 2 out of 3 instances/machines. We are running Karaf with Artemis 2.10.1. After some time (see screenshot), first one, then after a while a 2nd instance came to a complete
[jira] [Updated] (ARTEMIS-2586) Inifinite Block in AMQ212054 after transient DB-error
[ https://issues.apache.org/jira/browse/ARTEMIS-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rico Neubauer updated ARTEMIS-2586: --- Description: Hi, Would like to describe a quite severe situation which was expirienced in a long-running test with 2 out of 3 instances/machines. We are running Karaf with Artemis 2.10.1. After some time (see screenshot), first one, then after a while a 2nd instance came to a complete stop. Looking into the logs and thread-dumps revealed the following (same for bith instances): # There was a temporary problem connecting to the DB ({{connection reset by peer}}and {{Closed Connection }}) # This resulted (due to handling on our side) in an {{IllegalStateException}}/{{Error during two phase commit}} being thrown back to Artemis. # After this, there is no messaging possible anymore at all and the following log repeats: {noformat} AMQ212054: Destination address=DLQ is blocked. If the system is configured to block make sure you consume messages on this configuration.{noformat} (system is not condifured to block, see attached config) which comes from threads like these, trying to obtain credits for sending: {noformat} "Thread-93 (ActiveMQ-client-global-threads)" Id=2001 in TIMED_WAITING on lock=java.util.concurrent.Semaphore$NonfairSync@1f9a57e0 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1332) at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:582) at org.apache.activemq.artemis.core.client.impl.ClientProducerCreditsImpl.actualAcquire(ClientProducerCreditsImpl.java:73) at org.apache.activemq.artemis.core.client.impl.AbstractProducerCreditsImpl.acquireCredits(AbstractProducerCreditsImpl.java:77) at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:301) at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:275) at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:128) at org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:485) at org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:195) at com.seeburger.engine.jms.MessageReceiverBase.sendToDLQ(MessageReceiverBase.java:571) at com.seeburger.engine.jms.MessageReceiverBase.handleException(MessageReceiverBase.java:493) at com.seeburger.engine.jms.MessageReceiverBase.onMessage(MessageReceiverBase.java:387) at org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:110) at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1031) at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50) at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1154) at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) at org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$431/1769898766.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) Locked synchronizers: count = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@bc49fcf {noformat} which will never succeed, since the credits seem to no suffice (see heap-dump screenshot) >From my point of view, the thrown IllegalStateException should not lead to the >system going in this non-recoverable state, what do you think, is there >something that can be enhanced? [Fastthread-Link|https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjAvMDEvMy8tLTIwMTktMTItMDRfdGhyZWFkZHVtcF8wMS50eHQtLTEzLTM4LTE1OzstLTIwMTktMTEtMjhfdGhyZWFkZHVtcF8wMS50eHQtLTEzLTM4LTE1] In case it helps: The 2 instances are still in this state (since September) and I can fetch additional information or debug them on request. was: Hi, Would like to describe a quite severe situation which was expirienced in a long-running test with 2 out of 3 instances/machines. We are running Karaf with Artemis 2.10.1. After some time (see screenshot), first one, then after a while a 2nd instance came to a complete
[jira] [Updated] (ARTEMIS-2586) Inifinite Block in AMQ212054 after transient DB-error
[ https://issues.apache.org/jira/browse/ARTEMIS-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rico Neubauer updated ARTEMIS-2586: --- Attachment: writerIndex-Credits.PNG > Inifinite Block in AMQ212054 after transient DB-error > - > > Key: ARTEMIS-2586 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2586 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: AMQP >Affects Versions: 2.10.1 > Environment: This is Ubuntu 18.04 and Oracle DB, but don't think it's > that relevant for the issue. >Reporter: Rico Neubauer >Priority: Major > Attachments: 2019-11-28_threaddump_01.txt, > 2019-12-04_threaddump_01.txt, Message-Counts.png, initial-error.txt, > log-extract.txt, writerIndex-Credits.PNG > > > Hi, > Would like to describe a quite severe situation which was expirienced in a > long-running test with 2 out of 3 instances/machines. > We are running Karaf with Artemis 2.10.1. > After some time (see screenshot), first one, then after a while a 2nd > instance came to a complete stop. > Looking into the logs and thread-dumps revealed the following (same for bith > instances): > # There was a temporary problem connecting to the DB (\{{connection reset by > peer}}and \{{Closed Connection }}) > # This resulted (due to handling on our side) in an > \{{IllegalStateException}}/\{{Error during two phase commit}} being thrown > back to Artemis. > # After this, there is no messaging possible anymore at all and the > following log repeats: > {noformat} > AMQ212054: Destination address=DLQ is blocked. If the system is configured to > block make sure you consume messages on this configuration.{noformat} > which comes from threads like these, trying to obtain credits for sending: > > {noformat} > "Thread-93 (ActiveMQ-client-global-threads)" Id=2001 in TIMED_WAITING on > lock=java.util.concurrent.Semaphore$NonfairSync@1f9a57e0 > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1332) > at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:582) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerCreditsImpl.actualAcquire(ClientProducerCreditsImpl.java:73) > at > org.apache.activemq.artemis.core.client.impl.AbstractProducerCreditsImpl.acquireCredits(AbstractProducerCreditsImpl.java:77) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:301) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:275) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:128) > at > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:485) > at > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:195) > at > com.seeburger.engine.jms.MessageReceiverBase.sendToDLQ(MessageReceiverBase.java:571) > at > com.seeburger.engine.jms.MessageReceiverBase.handleException(MessageReceiverBase.java:493) > at > com.seeburger.engine.jms.MessageReceiverBase.onMessage(MessageReceiverBase.java:387) > at > org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:110) > at > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1031) > at > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50) > at > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1154) > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) > at > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) > at > org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$431/1769898766.run(Unknown > Source) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > Locked synchronizers: count = 1 > - java.util.concurrent.ThreadPoolExecutor$Worker@bc49fcf > {noformat} > which will never succeed, since the credits seem to no suffice (see heap-dump > screenshot)
[jira] [Updated] (ARTEMIS-2586) Inifinite Block in AMQ212054 after transient DB-error
[ https://issues.apache.org/jira/browse/ARTEMIS-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rico Neubauer updated ARTEMIS-2586: --- Attachment: Message-Counts.png > Inifinite Block in AMQ212054 after transient DB-error > - > > Key: ARTEMIS-2586 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2586 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: AMQP >Affects Versions: 2.10.1 > Environment: This is Ubuntu 18.04 and Oracle DB, but don't think it's > that relevant for the issue. >Reporter: Rico Neubauer >Priority: Major > Attachments: 2019-11-28_threaddump_01.txt, > 2019-12-04_threaddump_01.txt, Message-Counts.png, initial-error.txt, > log-extract.txt, writerIndex-Credits.PNG > > > Hi, > Would like to describe a quite severe situation which was expirienced in a > long-running test with 2 out of 3 instances/machines. > We are running Karaf with Artemis 2.10.1. > After some time (see screenshot), first one, then after a while a 2nd > instance came to a complete stop. > Looking into the logs and thread-dumps revealed the following (same for bith > instances): > # There was a temporary problem connecting to the DB (\{{connection reset by > peer}}and \{{Closed Connection }}) > # This resulted (due to handling on our side) in an > \{{IllegalStateException}}/\{{Error during two phase commit}} being thrown > back to Artemis. > # After this, there is no messaging possible anymore at all and the > following log repeats: > {noformat} > AMQ212054: Destination address=DLQ is blocked. If the system is configured to > block make sure you consume messages on this configuration.{noformat} > which comes from threads like these, trying to obtain credits for sending: > > {noformat} > "Thread-93 (ActiveMQ-client-global-threads)" Id=2001 in TIMED_WAITING on > lock=java.util.concurrent.Semaphore$NonfairSync@1f9a57e0 > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1332) > at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:582) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerCreditsImpl.actualAcquire(ClientProducerCreditsImpl.java:73) > at > org.apache.activemq.artemis.core.client.impl.AbstractProducerCreditsImpl.acquireCredits(AbstractProducerCreditsImpl.java:77) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:301) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:275) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:128) > at > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:485) > at > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:195) > at > com.seeburger.engine.jms.MessageReceiverBase.sendToDLQ(MessageReceiverBase.java:571) > at > com.seeburger.engine.jms.MessageReceiverBase.handleException(MessageReceiverBase.java:493) > at > com.seeburger.engine.jms.MessageReceiverBase.onMessage(MessageReceiverBase.java:387) > at > org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:110) > at > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1031) > at > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50) > at > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1154) > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) > at > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) > at > org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$431/1769898766.run(Unknown > Source) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > Locked synchronizers: count = 1 > - java.util.concurrent.ThreadPoolExecutor$Worker@bc49fcf > {noformat} > which will never succeed, since the credits seem to no suffice (see heap-dump > screenshot) >
[jira] [Updated] (ARTEMIS-2586) Inifinite Block in AMQ212054 after transient DB-error
[ https://issues.apache.org/jira/browse/ARTEMIS-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rico Neubauer updated ARTEMIS-2586: --- Attachment: 2019-12-04_threaddump_01.txt > Inifinite Block in AMQ212054 after transient DB-error > - > > Key: ARTEMIS-2586 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2586 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: AMQP >Affects Versions: 2.10.1 > Environment: This is Ubuntu 18.04 and Oracle DB, but don't think it's > that relevant for the issue. >Reporter: Rico Neubauer >Priority: Major > Attachments: 2019-11-28_threaddump_01.txt, > 2019-12-04_threaddump_01.txt, Message-Counts.png, initial-error.txt, > log-extract.txt, writerIndex-Credits.PNG > > > Hi, > Would like to describe a quite severe situation which was expirienced in a > long-running test with 2 out of 3 instances/machines. > We are running Karaf with Artemis 2.10.1. > After some time (see screenshot), first one, then after a while a 2nd > instance came to a complete stop. > Looking into the logs and thread-dumps revealed the following (same for bith > instances): > # There was a temporary problem connecting to the DB (\{{connection reset by > peer}}and \{{Closed Connection }}) > # This resulted (due to handling on our side) in an > \{{IllegalStateException}}/\{{Error during two phase commit}} being thrown > back to Artemis. > # After this, there is no messaging possible anymore at all and the > following log repeats: > {noformat} > AMQ212054: Destination address=DLQ is blocked. If the system is configured to > block make sure you consume messages on this configuration.{noformat} > which comes from threads like these, trying to obtain credits for sending: > > {noformat} > "Thread-93 (ActiveMQ-client-global-threads)" Id=2001 in TIMED_WAITING on > lock=java.util.concurrent.Semaphore$NonfairSync@1f9a57e0 > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1332) > at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:582) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerCreditsImpl.actualAcquire(ClientProducerCreditsImpl.java:73) > at > org.apache.activemq.artemis.core.client.impl.AbstractProducerCreditsImpl.acquireCredits(AbstractProducerCreditsImpl.java:77) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:301) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:275) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:128) > at > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:485) > at > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:195) > at > com.seeburger.engine.jms.MessageReceiverBase.sendToDLQ(MessageReceiverBase.java:571) > at > com.seeburger.engine.jms.MessageReceiverBase.handleException(MessageReceiverBase.java:493) > at > com.seeburger.engine.jms.MessageReceiverBase.onMessage(MessageReceiverBase.java:387) > at > org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:110) > at > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1031) > at > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50) > at > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1154) > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) > at > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) > at > org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$431/1769898766.run(Unknown > Source) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > Locked synchronizers: count = 1 > - java.util.concurrent.ThreadPoolExecutor$Worker@bc49fcf > {noformat} > which will never succeed, since the credits seem to no suffice (see heap-dump >
[jira] [Updated] (ARTEMIS-2586) Inifinite Block in AMQ212054 after transient DB-error
[ https://issues.apache.org/jira/browse/ARTEMIS-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rico Neubauer updated ARTEMIS-2586: --- Attachment: 2019-11-28_threaddump_01.txt > Inifinite Block in AMQ212054 after transient DB-error > - > > Key: ARTEMIS-2586 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2586 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: AMQP >Affects Versions: 2.10.1 > Environment: This is Ubuntu 18.04 and Oracle DB, but don't think it's > that relevant for the issue. >Reporter: Rico Neubauer >Priority: Major > Attachments: 2019-11-28_threaddump_01.txt, > 2019-12-04_threaddump_01.txt, Message-Counts.png, initial-error.txt, > log-extract.txt, writerIndex-Credits.PNG > > > Hi, > Would like to describe a quite severe situation which was expirienced in a > long-running test with 2 out of 3 instances/machines. > We are running Karaf with Artemis 2.10.1. > After some time (see screenshot), first one, then after a while a 2nd > instance came to a complete stop. > Looking into the logs and thread-dumps revealed the following (same for bith > instances): > # There was a temporary problem connecting to the DB (\{{connection reset by > peer}}and \{{Closed Connection }}) > # This resulted (due to handling on our side) in an > \{{IllegalStateException}}/\{{Error during two phase commit}} being thrown > back to Artemis. > # After this, there is no messaging possible anymore at all and the > following log repeats: > {noformat} > AMQ212054: Destination address=DLQ is blocked. If the system is configured to > block make sure you consume messages on this configuration.{noformat} > which comes from threads like these, trying to obtain credits for sending: > > {noformat} > "Thread-93 (ActiveMQ-client-global-threads)" Id=2001 in TIMED_WAITING on > lock=java.util.concurrent.Semaphore$NonfairSync@1f9a57e0 > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1332) > at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:582) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerCreditsImpl.actualAcquire(ClientProducerCreditsImpl.java:73) > at > org.apache.activemq.artemis.core.client.impl.AbstractProducerCreditsImpl.acquireCredits(AbstractProducerCreditsImpl.java:77) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:301) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:275) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:128) > at > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:485) > at > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:195) > at > com.seeburger.engine.jms.MessageReceiverBase.sendToDLQ(MessageReceiverBase.java:571) > at > com.seeburger.engine.jms.MessageReceiverBase.handleException(MessageReceiverBase.java:493) > at > com.seeburger.engine.jms.MessageReceiverBase.onMessage(MessageReceiverBase.java:387) > at > org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:110) > at > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1031) > at > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50) > at > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1154) > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) > at > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) > at > org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$431/1769898766.run(Unknown > Source) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > Locked synchronizers: count = 1 > - java.util.concurrent.ThreadPoolExecutor$Worker@bc49fcf > {noformat} > which will never succeed, since the credits seem to no suffice (see heap-dump >
[jira] [Updated] (ARTEMIS-2586) Inifinite Block in AMQ212054 after transient DB-error
[ https://issues.apache.org/jira/browse/ARTEMIS-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rico Neubauer updated ARTEMIS-2586: --- Attachment: log-extract.txt > Inifinite Block in AMQ212054 after transient DB-error > - > > Key: ARTEMIS-2586 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2586 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: AMQP >Affects Versions: 2.10.1 > Environment: This is Ubuntu 18.04 and Oracle DB, but don't think it's > that relevant for the issue. >Reporter: Rico Neubauer >Priority: Major > Attachments: 2019-11-28_threaddump_01.txt, > 2019-12-04_threaddump_01.txt, Message-Counts.png, initial-error.txt, > log-extract.txt, writerIndex-Credits.PNG > > > Hi, > Would like to describe a quite severe situation which was expirienced in a > long-running test with 2 out of 3 instances/machines. > We are running Karaf with Artemis 2.10.1. > After some time (see screenshot), first one, then after a while a 2nd > instance came to a complete stop. > Looking into the logs and thread-dumps revealed the following (same for bith > instances): > # There was a temporary problem connecting to the DB (\{{connection reset by > peer}}and \{{Closed Connection }}) > # This resulted (due to handling on our side) in an > \{{IllegalStateException}}/\{{Error during two phase commit}} being thrown > back to Artemis. > # After this, there is no messaging possible anymore at all and the > following log repeats: > {noformat} > AMQ212054: Destination address=DLQ is blocked. If the system is configured to > block make sure you consume messages on this configuration.{noformat} > which comes from threads like these, trying to obtain credits for sending: > > {noformat} > "Thread-93 (ActiveMQ-client-global-threads)" Id=2001 in TIMED_WAITING on > lock=java.util.concurrent.Semaphore$NonfairSync@1f9a57e0 > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1332) > at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:582) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerCreditsImpl.actualAcquire(ClientProducerCreditsImpl.java:73) > at > org.apache.activemq.artemis.core.client.impl.AbstractProducerCreditsImpl.acquireCredits(AbstractProducerCreditsImpl.java:77) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:301) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:275) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:128) > at > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:485) > at > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:195) > at > com.seeburger.engine.jms.MessageReceiverBase.sendToDLQ(MessageReceiverBase.java:571) > at > com.seeburger.engine.jms.MessageReceiverBase.handleException(MessageReceiverBase.java:493) > at > com.seeburger.engine.jms.MessageReceiverBase.onMessage(MessageReceiverBase.java:387) > at > org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:110) > at > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1031) > at > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50) > at > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1154) > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) > at > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) > at > org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$431/1769898766.run(Unknown > Source) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > Locked synchronizers: count = 1 > - java.util.concurrent.ThreadPoolExecutor$Worker@bc49fcf > {noformat} > which will never succeed, since the credits seem to no suffice (see heap-dump > screenshot) > From
[jira] [Updated] (ARTEMIS-2586) Inifinite Block in AMQ212054 after transient DB-error
[ https://issues.apache.org/jira/browse/ARTEMIS-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rico Neubauer updated ARTEMIS-2586: --- Attachment: initial-error.txt > Inifinite Block in AMQ212054 after transient DB-error > - > > Key: ARTEMIS-2586 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2586 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: AMQP >Affects Versions: 2.10.1 > Environment: This is Ubuntu 18.04 and Oracle DB, but don't think it's > that relevant for the issue. >Reporter: Rico Neubauer >Priority: Major > Attachments: initial-error.txt > > > Hi, > Would like to describe a quite severe situation which was expirienced in a > long-running test with 2 out of 3 instances/machines. > We are running Karaf with Artemis 2.10.1. > After some time (see screenshot), first one, then after a while a 2nd > instance came to a complete stop. > Looking into the logs and thread-dumps revealed the following (same for bith > instances): > # There was a temporary problem connecting to the DB (\{{connection reset by > peer}}and \{{Closed Connection }}) > # This resulted (due to handling on our side) in an > \{{IllegalStateException}}/\{{Error during two phase commit}} being thrown > back to Artemis. > # After this, there is no messaging possible anymore at all and the > following log repeats: > {noformat} > AMQ212054: Destination address=DLQ is blocked. If the system is configured to > block make sure you consume messages on this configuration.{noformat} > which comes from threads like these, trying to obtain credits for sending: > > {noformat} > "Thread-93 (ActiveMQ-client-global-threads)" Id=2001 in TIMED_WAITING on > lock=java.util.concurrent.Semaphore$NonfairSync@1f9a57e0 > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1332) > at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:582) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerCreditsImpl.actualAcquire(ClientProducerCreditsImpl.java:73) > at > org.apache.activemq.artemis.core.client.impl.AbstractProducerCreditsImpl.acquireCredits(AbstractProducerCreditsImpl.java:77) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:301) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:275) > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:128) > at > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:485) > at > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:195) > at > com.seeburger.engine.jms.MessageReceiverBase.sendToDLQ(MessageReceiverBase.java:571) > at > com.seeburger.engine.jms.MessageReceiverBase.handleException(MessageReceiverBase.java:493) > at > com.seeburger.engine.jms.MessageReceiverBase.onMessage(MessageReceiverBase.java:387) > at > org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:110) > at > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1031) > at > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50) > at > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1154) > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) > at > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) > at > org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$431/1769898766.run(Unknown > Source) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > Locked synchronizers: count = 1 > - java.util.concurrent.ThreadPoolExecutor$Worker@bc49fcf > {noformat} > which will never succeed, since the credits seem to no suffice (see heap-dump > screenshot) > From my point of view, the thrown IllegalStateException should not lead to > the system going in this non-recoverable state, what