[jira] [Commented] (ARTEMIS-2586) Inifinite Block in AMQ212054 after transient DB-error
[
https://issues.apache.org/jira/browse/ARTEMIS-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010517#comment-17010517
]
Rico Neubauer commented on ARTEMIS-2586:
[~jbertram] Thanks for your reply and your hints.
No, we are not acknowledging the message before the send.
I will set up a test which provokes the situation and check whether
acknowledging resolves the situation (whereas this would actually not be the
desired flow).
About the AMQP - yes, it was because of the "AMQ" prefix - sorry for the
confusion I caused here. Just removed it as component.
> Inifinite Block in AMQ212054 after transient DB-error
> -
>
> Key: ARTEMIS-2586
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2586
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Affects Versions: 2.10.1
> Environment: This is Ubuntu 18.04 and Oracle DB, but don't think it's
> that relevant for the issue.
>Reporter: Rico Neubauer
>Priority: Major
> Attachments: 2019-11-28_threaddump_01.txt,
> 2019-12-04_threaddump_01.txt, Message-Counts.png, artemis.xml,
> initial-error.txt, log-extract.txt, writerIndex-Credits.PNG
>
>
> Hi,
> Would like to describe a quite severe situation which was expirienced in a
> long-running test with 2 out of 3 instances/machines.
> We are running Karaf with Artemis 2.10.1.
> After some time (see screenshot), first one, then after a while a 2nd
> instance came to a complete stop.
> Looking into the logs and thread-dumps revealed the following (same for bith
> instances):
> # There was a temporary problem connecting to the DB ({{connection reset by
> peer}}and {{Closed Connection }})
> # This resulted (due to handling on our side) in an
> {{IllegalStateException}}/{{Error during two phase commit}} being thrown back
> to Artemis.
> # After this, there is no messaging possible anymore at all and the
> following log repeats:
> {noformat}
> AMQ212054: Destination address=DLQ is blocked. If the system is configured to
> block make sure you consume messages on this configuration.{noformat}
> (system is not configured to block, see attached config)
> which comes from threads like these, trying to obtain credits for sending:
>
> {noformat}
> "Thread-93 (ActiveMQ-client-global-threads)" Id=2001 in TIMED_WAITING on
> lock=java.util.concurrent.Semaphore$NonfairSync@1f9a57e0
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1332)
> at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:582)
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerCreditsImpl.actualAcquire(ClientProducerCreditsImpl.java:73)
> at
> org.apache.activemq.artemis.core.client.impl.AbstractProducerCreditsImpl.acquireCredits(AbstractProducerCreditsImpl.java:77)
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:301)
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:275)
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:128)
> at
> org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:485)
> at
> org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:195)
> at
> com.seeburger.engine.jms.MessageReceiverBase.sendToDLQ(MessageReceiverBase.java:571)
> at
> com.seeburger.engine.jms.MessageReceiverBase.handleException(MessageReceiverBase.java:493)
> at
> com.seeburger.engine.jms.MessageReceiverBase.onMessage(MessageReceiverBase.java:387)
> at
> org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:110)
> at
> org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1031)
> at
> org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50)
> at
> org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1154)
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$431/1769898766.run(Unknown
> Source)
> at
>
[jira] [Commented] (ARTEMIS-2586) Inifinite Block in AMQ212054 after transient DB-error
[
https://issues.apache.org/jira/browse/ARTEMIS-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17008035#comment-17008035
]
Justin Bertram commented on ARTEMIS-2586:
-
Are you acknowledging the message before you attempt to send it to the DLQ? If
not, perhaps this is triggering this situation. For what it's worth the broker
can handle sending messages to a DLQ so there's a possibility you can simplify
your application here and eliminate this problem as well. You might also try
eliminating producer flow control on the client by specifying
{{producerWindowSize=-1}} on the client's URL.
Aside from that I think I'd need a reproducible test-case to investigate
further.
bq. Just used "AMQP" as component since, the logged error has it in its name.
You are right: we are using the core protocol.
>From what I can tell there's no mention of AMQP in any of the attachments.
>Perhaps you're referring to message code "AMQ212054" which has the "AMQ"
>prefix (which is a 3-letter code referencing ActiveMQ)?
> Inifinite Block in AMQ212054 after transient DB-error
> -
>
> Key: ARTEMIS-2586
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2586
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: AMQP
>Affects Versions: 2.10.1
> Environment: This is Ubuntu 18.04 and Oracle DB, but don't think it's
> that relevant for the issue.
>Reporter: Rico Neubauer
>Priority: Major
> Attachments: 2019-11-28_threaddump_01.txt,
> 2019-12-04_threaddump_01.txt, Message-Counts.png, artemis.xml,
> initial-error.txt, log-extract.txt, writerIndex-Credits.PNG
>
>
> Hi,
> Would like to describe a quite severe situation which was expirienced in a
> long-running test with 2 out of 3 instances/machines.
> We are running Karaf with Artemis 2.10.1.
> After some time (see screenshot), first one, then after a while a 2nd
> instance came to a complete stop.
> Looking into the logs and thread-dumps revealed the following (same for bith
> instances):
> # There was a temporary problem connecting to the DB ({{connection reset by
> peer}}and {{Closed Connection }})
> # This resulted (due to handling on our side) in an
> {{IllegalStateException}}/{{Error during two phase commit}} being thrown back
> to Artemis.
> # After this, there is no messaging possible anymore at all and the
> following log repeats:
> {noformat}
> AMQ212054: Destination address=DLQ is blocked. If the system is configured to
> block make sure you consume messages on this configuration.{noformat}
> (system is not configured to block, see attached config)
> which comes from threads like these, trying to obtain credits for sending:
>
> {noformat}
> "Thread-93 (ActiveMQ-client-global-threads)" Id=2001 in TIMED_WAITING on
> lock=java.util.concurrent.Semaphore$NonfairSync@1f9a57e0
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1332)
> at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:582)
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerCreditsImpl.actualAcquire(ClientProducerCreditsImpl.java:73)
> at
> org.apache.activemq.artemis.core.client.impl.AbstractProducerCreditsImpl.acquireCredits(AbstractProducerCreditsImpl.java:77)
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:301)
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:275)
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:128)
> at
> org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:485)
> at
> org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:195)
> at
> com.seeburger.engine.jms.MessageReceiverBase.sendToDLQ(MessageReceiverBase.java:571)
> at
> com.seeburger.engine.jms.MessageReceiverBase.handleException(MessageReceiverBase.java:493)
> at
> com.seeburger.engine.jms.MessageReceiverBase.onMessage(MessageReceiverBase.java:387)
> at
> org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:110)
> at
> org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1031)
> at
> org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50)
> at
>
[jira] [Commented] (ARTEMIS-2586) Inifinite Block in AMQ212054 after transient DB-error
[
https://issues.apache.org/jira/browse/ARTEMIS-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007974#comment-17007974
]
Rico Neubauer commented on ARTEMIS-2586:
Config attached now - sorry, missed it.
Just used "AMQP" as component since, the logged error has it in its name. You
are right: we are using the core protocol.
The exceptions thrown originate in the JDBC-driver's networ layer, while the
MessageReceiverBase#onMessage tries to commit. This exception is handled by
MessageReceiverBase and message is tried to be sent to DLQ, while looking at
it, maybe this handling leads to the credit exhaustion, since the handling is
still part of our #onMessage code, i.e. there is not yet a return/exception
back to JMS-server done, when the blocking send is in progress:
e.g.
{noformat}
"Thread-83 (ActiveMQ-client-global-threads)" Id=1727 in TIMED_WAITING on
lock=java.util.concurrent.Semaphore$NonfairSync@85eb557
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1332)
at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:582)
at
org.apache.activemq.artemis.core.client.impl.ClientProducerCreditsImpl.actualAcquire(ClientProducerCreditsImpl.java:73)
at
org.apache.activemq.artemis.core.client.impl.AbstractProducerCreditsImpl.acquireCredits(AbstractProducerCreditsImpl.java:77)
at
org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:301)
at
org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:275)
at
org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:128)
at
org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:485)
at
org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:195)
at
com.company.engine.jms.MessageReceiverBase.sendToDLQ(MessageReceiverBase.java:571)
at
com.company.engine.jms.MessageReceiverBase.handleException(MessageReceiverBase.java:493)
at
com.company.engine.jms.MessageReceiverBase.onMessage(MessageReceiverBase.java:387){noformat}
> Inifinite Block in AMQ212054 after transient DB-error
> -
>
> Key: ARTEMIS-2586
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2586
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: AMQP
>Affects Versions: 2.10.1
> Environment: This is Ubuntu 18.04 and Oracle DB, but don't think it's
> that relevant for the issue.
>Reporter: Rico Neubauer
>Priority: Major
> Attachments: 2019-11-28_threaddump_01.txt,
> 2019-12-04_threaddump_01.txt, Message-Counts.png, artemis.xml,
> initial-error.txt, log-extract.txt, writerIndex-Credits.PNG
>
>
> Hi,
> Would like to describe a quite severe situation which was expirienced in a
> long-running test with 2 out of 3 instances/machines.
> We are running Karaf with Artemis 2.10.1.
> After some time (see screenshot), first one, then after a while a 2nd
> instance came to a complete stop.
> Looking into the logs and thread-dumps revealed the following (same for bith
> instances):
> # There was a temporary problem connecting to the DB ({{connection reset by
> peer}}and {{Closed Connection }})
> # This resulted (due to handling on our side) in an
> {{IllegalStateException}}/{{Error during two phase commit}} being thrown back
> to Artemis.
> # After this, there is no messaging possible anymore at all and the
> following log repeats:
> {noformat}
> AMQ212054: Destination address=DLQ is blocked. If the system is configured to
> block make sure you consume messages on this configuration.{noformat}
> (system is not configured to block, see attached config)
> which comes from threads like these, trying to obtain credits for sending:
>
> {noformat}
> "Thread-93 (ActiveMQ-client-global-threads)" Id=2001 in TIMED_WAITING on
> lock=java.util.concurrent.Semaphore$NonfairSync@1f9a57e0
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1332)
> at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:582)
> at
>
[jira] [Commented] (ARTEMIS-2586) Inifinite Block in AMQ212054 after transient DB-error
[
https://issues.apache.org/jira/browse/ARTEMIS-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007825#comment-17007825
]
Justin Bertram commented on ARTEMIS-2586:
-
Are the exceptions logged in [^initial-error.txt] being thrown from
{{com.company.engine.jms.MessageReceiverBase.onMessage()}}?
> Inifinite Block in AMQ212054 after transient DB-error
> -
>
> Key: ARTEMIS-2586
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2586
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: AMQP
>Affects Versions: 2.10.1
> Environment: This is Ubuntu 18.04 and Oracle DB, but don't think it's
> that relevant for the issue.
>Reporter: Rico Neubauer
>Priority: Major
> Attachments: 2019-11-28_threaddump_01.txt,
> 2019-12-04_threaddump_01.txt, Message-Counts.png, initial-error.txt,
> log-extract.txt, writerIndex-Credits.PNG
>
>
> Hi,
> Would like to describe a quite severe situation which was expirienced in a
> long-running test with 2 out of 3 instances/machines.
> We are running Karaf with Artemis 2.10.1.
> After some time (see screenshot), first one, then after a while a 2nd
> instance came to a complete stop.
> Looking into the logs and thread-dumps revealed the following (same for bith
> instances):
> # There was a temporary problem connecting to the DB ({{connection reset by
> peer}}and {{Closed Connection }})
> # This resulted (due to handling on our side) in an
> {{IllegalStateException}}/{{Error during two phase commit}} being thrown back
> to Artemis.
> # After this, there is no messaging possible anymore at all and the
> following log repeats:
> {noformat}
> AMQ212054: Destination address=DLQ is blocked. If the system is configured to
> block make sure you consume messages on this configuration.{noformat}
> (system is not configured to block, see attached config)
> which comes from threads like these, trying to obtain credits for sending:
>
> {noformat}
> "Thread-93 (ActiveMQ-client-global-threads)" Id=2001 in TIMED_WAITING on
> lock=java.util.concurrent.Semaphore$NonfairSync@1f9a57e0
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1332)
> at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:582)
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerCreditsImpl.actualAcquire(ClientProducerCreditsImpl.java:73)
> at
> org.apache.activemq.artemis.core.client.impl.AbstractProducerCreditsImpl.acquireCredits(AbstractProducerCreditsImpl.java:77)
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:301)
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:275)
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:128)
> at
> org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:485)
> at
> org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:195)
> at
> com.seeburger.engine.jms.MessageReceiverBase.sendToDLQ(MessageReceiverBase.java:571)
> at
> com.seeburger.engine.jms.MessageReceiverBase.handleException(MessageReceiverBase.java:493)
> at
> com.seeburger.engine.jms.MessageReceiverBase.onMessage(MessageReceiverBase.java:387)
> at
> org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:110)
> at
> org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1031)
> at
> org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50)
> at
> org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1154)
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$431/1769898766.run(Unknown
> Source)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> Locked
[jira] [Commented] (ARTEMIS-2586) Inifinite Block in AMQ212054 after transient DB-error
[
https://issues.apache.org/jira/browse/ARTEMIS-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007595#comment-17007595
]
Justin Bertram commented on ARTEMIS-2586:
-
Also, you've used the {{AMQP}} component on this JIRA but from the stack-traces
it appears you're using the core protocol. Please clarify what protocol you're
using.
> Inifinite Block in AMQ212054 after transient DB-error
> -
>
> Key: ARTEMIS-2586
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2586
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: AMQP
>Affects Versions: 2.10.1
> Environment: This is Ubuntu 18.04 and Oracle DB, but don't think it's
> that relevant for the issue.
>Reporter: Rico Neubauer
>Priority: Major
> Attachments: 2019-11-28_threaddump_01.txt,
> 2019-12-04_threaddump_01.txt, Message-Counts.png, initial-error.txt,
> log-extract.txt, writerIndex-Credits.PNG
>
>
> Hi,
> Would like to describe a quite severe situation which was expirienced in a
> long-running test with 2 out of 3 instances/machines.
> We are running Karaf with Artemis 2.10.1.
> After some time (see screenshot), first one, then after a while a 2nd
> instance came to a complete stop.
> Looking into the logs and thread-dumps revealed the following (same for bith
> instances):
> # There was a temporary problem connecting to the DB ({{connection reset by
> peer}}and {{Closed Connection }})
> # This resulted (due to handling on our side) in an
> {{IllegalStateException}}/{{Error during two phase commit}} being thrown back
> to Artemis.
> # After this, there is no messaging possible anymore at all and the
> following log repeats:
> {noformat}
> AMQ212054: Destination address=DLQ is blocked. If the system is configured to
> block make sure you consume messages on this configuration.{noformat}
> (system is not configured to block, see attached config)
> which comes from threads like these, trying to obtain credits for sending:
>
> {noformat}
> "Thread-93 (ActiveMQ-client-global-threads)" Id=2001 in TIMED_WAITING on
> lock=java.util.concurrent.Semaphore$NonfairSync@1f9a57e0
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1332)
> at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:582)
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerCreditsImpl.actualAcquire(ClientProducerCreditsImpl.java:73)
> at
> org.apache.activemq.artemis.core.client.impl.AbstractProducerCreditsImpl.acquireCredits(AbstractProducerCreditsImpl.java:77)
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:301)
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:275)
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:128)
> at
> org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:485)
> at
> org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:195)
> at
> com.seeburger.engine.jms.MessageReceiverBase.sendToDLQ(MessageReceiverBase.java:571)
> at
> com.seeburger.engine.jms.MessageReceiverBase.handleException(MessageReceiverBase.java:493)
> at
> com.seeburger.engine.jms.MessageReceiverBase.onMessage(MessageReceiverBase.java:387)
> at
> org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:110)
> at
> org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1031)
> at
> org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50)
> at
> org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1154)
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$431/1769898766.run(Unknown
> Source)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
>
[jira] [Commented] (ARTEMIS-2586) Inifinite Block in AMQ212054 after transient DB-error
[
https://issues.apache.org/jira/browse/ARTEMIS-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007593#comment-17007593
]
Justin Bertram commented on ARTEMIS-2586:
-
{quote}(system is not configured to block, see attached config)
{quote}
I don't see any config attached.
> Inifinite Block in AMQ212054 after transient DB-error
> -
>
> Key: ARTEMIS-2586
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2586
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: AMQP
>Affects Versions: 2.10.1
> Environment: This is Ubuntu 18.04 and Oracle DB, but don't think it's
> that relevant for the issue.
>Reporter: Rico Neubauer
>Priority: Major
> Attachments: 2019-11-28_threaddump_01.txt,
> 2019-12-04_threaddump_01.txt, Message-Counts.png, initial-error.txt,
> log-extract.txt, writerIndex-Credits.PNG
>
>
> Hi,
> Would like to describe a quite severe situation which was expirienced in a
> long-running test with 2 out of 3 instances/machines.
> We are running Karaf with Artemis 2.10.1.
> After some time (see screenshot), first one, then after a while a 2nd
> instance came to a complete stop.
> Looking into the logs and thread-dumps revealed the following (same for bith
> instances):
> # There was a temporary problem connecting to the DB ({{connection reset by
> peer}}and {{Closed Connection }})
> # This resulted (due to handling on our side) in an
> {{IllegalStateException}}/{{Error during two phase commit}} being thrown back
> to Artemis.
> # After this, there is no messaging possible anymore at all and the
> following log repeats:
> {noformat}
> AMQ212054: Destination address=DLQ is blocked. If the system is configured to
> block make sure you consume messages on this configuration.{noformat}
> (system is not configured to block, see attached config)
> which comes from threads like these, trying to obtain credits for sending:
>
> {noformat}
> "Thread-93 (ActiveMQ-client-global-threads)" Id=2001 in TIMED_WAITING on
> lock=java.util.concurrent.Semaphore$NonfairSync@1f9a57e0
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1332)
> at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:582)
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerCreditsImpl.actualAcquire(ClientProducerCreditsImpl.java:73)
> at
> org.apache.activemq.artemis.core.client.impl.AbstractProducerCreditsImpl.acquireCredits(AbstractProducerCreditsImpl.java:77)
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:301)
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:275)
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:128)
> at
> org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:485)
> at
> org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:195)
> at
> com.seeburger.engine.jms.MessageReceiverBase.sendToDLQ(MessageReceiverBase.java:571)
> at
> com.seeburger.engine.jms.MessageReceiverBase.handleException(MessageReceiverBase.java:493)
> at
> com.seeburger.engine.jms.MessageReceiverBase.onMessage(MessageReceiverBase.java:387)
> at
> org.apache.activemq.artemis.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:110)
> at
> org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1031)
> at
> org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50)
> at
> org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1154)
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$431/1769898766.run(Unknown
> Source)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> Locked synchronizers: count = 1
>
