[
https://issues.apache.org/jira/browse/AMQCPP-553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14359528#comment-14359528
]
Ilya commented on AMQCPP-553:
-----------------------------
Seems like the issue is not with BitSet itself, but with the way it's used by
ActiveMQMessageAudit::isDuplicate().
Here is the full stacktrace responsible for allocations:
{noformat}
0x5BC4DAB: decaf::util::BitSet::ensureCapacity(int) (BitSet.cpp:773)
0x5BC50B5: decaf::util::BitSet::set(int) (BitSet.cpp:642)
0x5898171:
activemq::core::ActiveMQMessageAudit::isDuplicate(decaf::lang::Pointer<activemq::commands::MessageId,
decaf::util::concurrent::atomic::AtomicRefCounter>) const
(ActiveMQMessageAudit.cpp:193)
0x58D34BE:
activemq::core::ConnectionAudit::isDuplicate(activemq::core::Dispatcher*,
decaf::lang::Pointer<activemq::commands::Message,
decaf::util::concurrent::atomic::AtomicRefCounter>) (ConnectionAudit.cpp:115)
0x5823C44:
activemq::core::ActiveMQConnection::isDuplicate(activemq::core::Dispatcher*,
decaf::lang::Pointer<activemq::commands::Message,
decaf::util::concurrent::atomic::AtomicRefCounter>)
(ActiveMQConnection.cpp:1917)
0x591173D:
activemq::core::kernels::ActiveMQConsumerKernel::dispatch(decaf::lang::Pointer<activemq::commands::MessageDispatch,
decaf::util::concurrent::atomic::AtomicRefCounter> const&)
(ActiveMQConsumerKernel.cpp:1527)
0x58C3452:
activemq::core::ActiveMQSessionExecutor::dispatch(decaf::lang::Pointer<activemq::commands::MessageDispatch,
decaf::util::concurrent::atomic::AtomicRefCounter> const&)
(ActiveMQSessionExecutor.cpp:156)
0x58C3DE3: activemq::core::ActiveMQSessionExecutor::iterate()
(ActiveMQSessionExecutor.cpp:181)
0x59FCF93: activemq::threads::DedicatedTaskRunner::run()
(DedicatedTaskRunner.cpp:141)
0x5B5FBFD: (anonymous namespace)::runCallback(void*) (Threading.cpp:266)
0x5B5F972: (anonymous namespace)::threadEntryMethod(void*) (Threading.cpp:254)
0x61FD9CF: start_thread (in /lib64/libpthread-2.12.so)
0x74BDB0B: clone (in /lib64/libc-2.12.so)
{noformat}
At first I thought that the issue must be that producer sequence id in the
following code keeps growing, requiring larger underlying array, as more
messages are processed:
{code}
184 long long index = msgId->getProducerSequenceId();
185 if (index >= 0) {
186 int scaledIndex = (int) index;
187 if (index > Integer::MAX_VALUE) {
188 scaledIndex = (int)(index - Integer::MAX_VALUE);
189 }
190
191 answer = bits->get(scaledIndex);
192 if (!answer) {
193 bits->set(scaledIndex, true);
194 }
195 }
{code}
But then, for one ActiveMQDestination instance, it should stop at
Integer::MAX_VALUE/8, which is 256Mb, and since the test has 12 instances at a
time, the limit must be 3Gb.
So I let the test run for a while, but the memory usage growth did not stop at
this figure, reaching ~4Gb and crashing with a segfault in 6 minutes.
Having looked at the code a little bit more, I now suspect that the reason
might be that the following map (declared in
./activemq-cpp/src/main/activemq/core/ConnectionAudit.cpp)
{code}
53 StlMap<Pointer<ActiveMQDestination>, Pointer<ActiveMQMessageAudit>,
ActiveMQDestination::COMPARATOR> destinations;
{code}
is not purged of pointers to deleted ActiveMQDestination objects.
I can repackage the attached stress test as AMQCPP unit test if needed.
> Deadlock and memory leak under stress
> -------------------------------------
>
> Key: AMQCPP-553
> URL: https://issues.apache.org/jira/browse/AMQCPP-553
> Project: ActiveMQ C++ Client
> Issue Type: Bug
> Components: CMS Impl
> Affects Versions: 3.8.3
> Environment: x86_64
> Linux 3.13.0-36
> gcc 4.4.7
> Reporter: Ilya
> Assignee: Timothy Bish
> Attachments: activemq-cpp-deadlock-fix.patch, amqcpp-deadlock.tar.xz,
> amqcpp-deadlock.tar.xz, massif.out.22958.xz
>
>
> Please have a look at the attached stress test. It consists of 12 reader
> threads, that create and destroy cms::MessageConsumers in a loop, and 4
> writer threads, that send cms::TextMessages in a loop. Reader threads
> deadlock in less than 1 minute on my machine. To run it, simply issue
> {quote}
> make test
> {quote}
> I traced the issue down to inconsistent mutex acquisition order by the
> following two threads:
> {quote}
> thread 7 (Thread 0x7fa691fce700 (LWP 28088))
> (gdb) bt
> #0 0x00007fa68eb405bc in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007fa68fea1c06 in (anonymous namespace)::doMonitorEnter
> (monitor=0x7fa668007ff0, thread=0x7fa64c0071d0) at
> decaf/internal/util/concurrent/Threading.cpp:664
> #2 0x00007fa68ff15dcb in decaf::util::concurrent::Lock::lock
> (this=0x7fa691fcd6b0) at decaf/util/concurrent/Lock.cpp:54
> #3 0x00007fa68ff15ee5 in decaf::util::concurrent::Lock::Lock
> (this=<value optimized out>, object=<value optimized out>,
> intiallyLocked=<value optimized out>)
> at decaf/util/concurrent/Lock.cpp:32
> #4 0x00007fa68fc4da38 in
> activemq::core::kernels::ActiveMQConsumerKernel::dispatch
> (this=0x7fa680010a20, dispatch=...) at
> activemq/core/kernels/ActiveMQConsumerKernel.cpp:1527
> #5 0x00007fa68fc06584 in
> activemq::core::ActiveMQSessionExecutor::dispatch (this=0x7fa65c005300,
> dispatch=...) at activemq/core/ActiveMQSessionExecutor.cpp:156
> #6 0x00007fa68fc06f15 in
> activemq::core::ActiveMQSessionExecutor::iterate (this=0x7fa65c005300) at
> activemq/core/ActiveMQSessionExecutor.cpp:181
> #7 0x00007fa68fd3fdf5 in activemq::threads::DedicatedTaskRunner::run
> (this=0x7fa64c004ab0) at activemq/threads/DedicatedTaskRunner.cpp:141
> Waiting for internal->listenerMutex, that is held by thread 6
> Acquisition order:
> ActiveMQConsumerKernel::internal->unconsumedMessages
> ActiveMQConsumerKernel::internal->listenerMutex
> (gdb) frame 1
> (gdb) p/x monitor->owner->handle
> $9 = 0x7fa691fbd700
> thread 6 (Thread 0x7fa691fbd700 (LWP 28091))
> (gdb) bt
> #0 0x00007fa68eb405bc in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x00007fa68fea1c06 in (anonymous namespace)::doMonitorEnter
> (monitor=0x7fa6680071e0, thread=0x7fa6246a13d0) at
> decaf/internal/util/concurrent/Threading.cpp:664
> #2 0x00007fa68ff15dcb in decaf::util::concurrent::Lock::lock
> (this=0x7fa691fbc6f0) at decaf/util/concurrent/Lock.cpp:54
> #3 0x00007fa68ff15ee5 in decaf::util::concurrent::Lock::Lock
> (this=<value optimized out>, object=<value optimized out>,
> intiallyLocked=<value optimized out>)
> at decaf/util/concurrent/Lock.cpp:32
> #4 0x00007fa68fc3c21a in
> activemq::core::SimplePriorityMessageDispatchChannel::dequeueNoWait
> (this=0x7fa68000eee0) at
> activemq/core/SimplePriorityMessageDispatchChannel.cpp:95
> #5 0x00007fa68fc40f3c in
> activemq::core::kernels::ActiveMQConsumerKernel::iterate
> (this=0x7fa680010a20) at activemq/core/kernels/ActiveMQConsumerKernel.cpp:1701
> #6 0x00007fa68fc8ab23 in
> activemq::core::kernels::ActiveMQSessionKernel::iterateConsumers
> (this=0x7fa65c003960) at activemq/core/kernels/ActiveMQSessionKernel.cpp:1370
> #7 0x00007fa68fc06eb9 in
> activemq::core::ActiveMQSessionExecutor::iterate (this=0x7fa65c005300) at
> activemq/core/ActiveMQSessionExecutor.cpp:173
> #8 0x00007fa68fd3fdf5 in activemq::threads::DedicatedTaskRunner::run
> (this=0x7fa624603e00) at activemq/threads/DedicatedTaskRunner.cpp:141
> Waiting for mutex, that is held by thread 7
> Acquisition order:
> ActiveMQSessionKernel::config->consumerLock
> ActiveMQConsumerKernel::internal->listenerMutex
> ActiveMQConsumerKernel::internal->unconsumedMessages
> (gdb) frame 1
> (gdb) p/x monitor->owner->handle
> $10 = 0x7fa691fce700
> {quote}
> With the attached patch, the stress test no longer deadlocks. However, it
> starts consuming memory in BitSet::ensureCapacity() via
> ActiveMQConnection::isDuplicate(). See attached massif output for details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
