[
https://issues.apache.org/jira/browse/AMQCPP-753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Katherine Pully updated AMQCPP-753:
-----------------------------------
Description:
When an ActiveMQConnection fails before a message acknowledgement can be
delivered, only decaf exceptions are caught. However, the check for a failed
connection ultimately results in a CMS exception, which means that the consumer
read lock does not get released. This will result in a deadlock when the
session tries to acquire the consumer write lock (for example, when cleaning up
the connection).
I have attached a stack trace from such a deadlock, which occurs when the
connection is cleaned up. The relevant portion (edited for brevity and clarity,
though the attached is the original), is:
{code:java}
0 decaf::util::concurrent::ExecutorKernel::Worker::run()
ThreadPoolExecutor.cpp:184
1 decaf::util::concurrent::ExecutorKernel::runWorker
(decaf::util::concurrent::ExecutorKernel::Worker*) ThreadPoolExecutor.cpp:738
2 activemq::core::OnExceptionRunnable::run() ActiveMQConnection.cpp:439
3 activemq::core::ActiveMQConnection::cleanup() ActiveMQConnection.cpp:839
4 activemq::core::kernels::ActiveMQSessionKernel::dispose()
ActiveMQSessionKernel.cpp:371
5 decaf::util::concurrent::locks::AbstractQueuedSynchronizer::acquire(int)
AbstractQueuedSynchronizer.cpp:1565
6 decaf::util::concurrent::locks::SynchronizerState::acquireQueued((anonymous
namespace)::Node*, int) AbstractQueuedSynchronizer.cpp:711
7 decaf::util::concurrent::locks::LockSupport::park() LockSupport.cpp:54
8 decaf::internal::util::concurrent::Threading::park(decaf::lang::Thread*)
Threading.cpp:1345
9
decaf::internal::util::concurrent::PlatformThread::interruptibleWaitOnCondition(_opaque_pthread_cond_t*,
_opaque_pthread_mutex_t*,
decaf::internal::util::concurrent::CompletionCondition&) PlatformThread.cpp:210
10 _pthread_cond_wait
11 __psynch_cvwait{code}
The issue can be produced by using a client-acknowledge strategy and adding a
substantial (10+ seconds) call to sleep before acknowledging the message, and
then breaking the connection.
The exception is originally thrown by
[ActiveMQConnection::checkClosedOrFailed|#L1329].] These exceptions become
ActiveMQ exceptions [here|#L1257],] and then to CMS Exceptions [here|#L1426].]
The only exceptions caught in
[ActiveMQSessionKernel::acknowledge|https://github.com/apache/activemq-cpp/blob/master/activemq-cpp/src/main/activemq/core/kernels/ActiveMQSessionKernel.cpp#L508]
are [decaf
exceptions|https://github.com/apache/activemq-cpp/blob/master/activemq-cpp/src/main/activemq/core/kernels/ActiveMQSessionKernel.cpp#L518];
when a CMS exception is caught, the [consumer read
lock|https://github.com/apache/activemq-cpp/blob/master/activemq-cpp/src/main/activemq/core/kernels/ActiveMQSessionKernel.cpp#L510]
is not released.
This issue can be fixed by catching the CMS exceptions.
was:
When an ActiveMQConnection fails before a message acknowledgement can be
delivered, only decaf exceptions are caught. However, the check for a failed
connection ultimately results in a CMS exception. As a result, this exception
is not handled and the consumer read lock does not get released. This will
result in a deadlock when the session tries to acquire the consumer write lock
(for example, when cleaning up the connection).
I have attached a stack trace from such a deadlock, which occurs when the
connection is cleaned up. The relevant portion (edited for brevity and clarity,
though the attached is the original), is:
{code:java}
0 decaf::util::concurrent::ExecutorKernel::Worker::run()
ThreadPoolExecutor.cpp:184
1 decaf::util::concurrent::ExecutorKernel::runWorker
(decaf::util::concurrent::ExecutorKernel::Worker*) ThreadPoolExecutor.cpp:738
2 activemq::core::OnExceptionRunnable::run() ActiveMQConnection.cpp:439
3 activemq::core::ActiveMQConnection::cleanup() ActiveMQConnection.cpp:839
4 activemq::core::kernels::ActiveMQSessionKernel::dispose()
ActiveMQSessionKernel.cpp:371
5 decaf::util::concurrent::locks::AbstractQueuedSynchronizer::acquire(int)
AbstractQueuedSynchronizer.cpp:1565
6 decaf::util::concurrent::locks::SynchronizerState::acquireQueued((anonymous
namespace)::Node*, int) AbstractQueuedSynchronizer.cpp:711
7 decaf::util::concurrent::locks::LockSupport::park() LockSupport.cpp:54
8 decaf::internal::util::concurrent::Threading::park(decaf::lang::Thread*)
Threading.cpp:1345
9
decaf::internal::util::concurrent::PlatformThread::interruptibleWaitOnCondition(_opaque_pthread_cond_t*,
_opaque_pthread_mutex_t*,
decaf::internal::util::concurrent::CompletionCondition&) PlatformThread.cpp:210
10 _pthread_cond_wait
11 __psynch_cvwait{code}
The issue can be produced by using a client-acknowledge strategy and adding a
substantial (10+ seconds) call to sleep before acknowledging the message, and
then breaking the connection.
The exception is originally thrown by
[ActiveMQConnection::checkClosedOrFailed|#L1329].] These exceptions become
ActiveMQ exceptions [here|#L1257],] and then to CMS Exceptions [here|#L1426].]
The only exceptions caught in
[ActiveMQSessionKernel::acknowledge|https://github.com/apache/activemq-cpp/blob/master/activemq-cpp/src/main/activemq/core/kernels/ActiveMQSessionKernel.cpp#L508]
are [decaf
exceptions|https://github.com/apache/activemq-cpp/blob/master/activemq-cpp/src/main/activemq/core/kernels/ActiveMQSessionKernel.cpp#L518];
when a CMS exception is caught, the [consumer read
lock|https://github.com/apache/activemq-cpp/blob/master/activemq-cpp/src/main/activemq/core/kernels/ActiveMQSessionKernel.cpp#L510]
is not released.
This issue can be fixed by catching the CMS exceptions.
> Deadlock when ActiveMQConection fails before message acknowledgement can be
> delivered
> -------------------------------------------------------------------------------------
>
> Key: AMQCPP-753
> URL: https://issues.apache.org/jira/browse/AMQCPP-753
> Project: ActiveMQ C++ Client
> Issue Type: Bug
> Components: Decaf
> Affects Versions: 3.9.5
> Environment: Unix/Linux
> Reporter: Katherine Pully
> Assignee: Timothy A. Bish
> Priority: Major
> Attachments: recreated_hang.txt
>
>
> When an ActiveMQConnection fails before a message acknowledgement can be
> delivered, only decaf exceptions are caught. However, the check for a failed
> connection ultimately results in a CMS exception, which means that the
> consumer read lock does not get released. This will result in a deadlock when
> the session tries to acquire the consumer write lock (for example, when
> cleaning up the connection).
> I have attached a stack trace from such a deadlock, which occurs when the
> connection is cleaned up. The relevant portion (edited for brevity and
> clarity, though the attached is the original), is:
> {code:java}
> 0 decaf::util::concurrent::ExecutorKernel::Worker::run()
> ThreadPoolExecutor.cpp:184
> 1 decaf::util::concurrent::ExecutorKernel::runWorker
> (decaf::util::concurrent::ExecutorKernel::Worker*) ThreadPoolExecutor.cpp:738
> 2 activemq::core::OnExceptionRunnable::run() ActiveMQConnection.cpp:439
> 3 activemq::core::ActiveMQConnection::cleanup() ActiveMQConnection.cpp:839
> 4 activemq::core::kernels::ActiveMQSessionKernel::dispose()
> ActiveMQSessionKernel.cpp:371
> 5 decaf::util::concurrent::locks::AbstractQueuedSynchronizer::acquire(int)
> AbstractQueuedSynchronizer.cpp:1565
> 6
> decaf::util::concurrent::locks::SynchronizerState::acquireQueued((anonymous
> namespace)::Node*, int) AbstractQueuedSynchronizer.cpp:711
> 7 decaf::util::concurrent::locks::LockSupport::park() LockSupport.cpp:54
> 8 decaf::internal::util::concurrent::Threading::park(decaf::lang::Thread*)
> Threading.cpp:1345
> 9
> decaf::internal::util::concurrent::PlatformThread::interruptibleWaitOnCondition(_opaque_pthread_cond_t*,
> _opaque_pthread_mutex_t*,
> decaf::internal::util::concurrent::CompletionCondition&)
> PlatformThread.cpp:210
> 10 _pthread_cond_wait
> 11 __psynch_cvwait{code}
> The issue can be produced by using a client-acknowledge strategy and adding a
> substantial (10+ seconds) call to sleep before acknowledging the message, and
> then breaking the connection.
> The exception is originally thrown by
> [ActiveMQConnection::checkClosedOrFailed|#L1329].] These exceptions become
> ActiveMQ exceptions [here|#L1257],] and then to CMS Exceptions
> [here|#L1426].] The only exceptions caught in
> [ActiveMQSessionKernel::acknowledge|https://github.com/apache/activemq-cpp/blob/master/activemq-cpp/src/main/activemq/core/kernels/ActiveMQSessionKernel.cpp#L508]
> are [decaf
> exceptions|https://github.com/apache/activemq-cpp/blob/master/activemq-cpp/src/main/activemq/core/kernels/ActiveMQSessionKernel.cpp#L518];
> when a CMS exception is caught, the [consumer read
> lock|https://github.com/apache/activemq-cpp/blob/master/activemq-cpp/src/main/activemq/core/kernels/ActiveMQSessionKernel.cpp#L510]
> is not released.
> This issue can be fixed by catching the CMS exceptions.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
