[
https://issues.apache.org/activemq/browse/AMQCPP-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=58050#action_58050
]
Eddie Fast commented on AMQCPP-277:
-----------------------------------
Just an update on this issue. We've upgraded to 3.1.1 and still have this
deadlock issue.
I've been able to reproduce on both the example main.cpp, as well as the new
unit test that was added, testCreateManyConsumersAndSetListeners().
In the unit test, I was getting deadlocks about 90% of the time in the 10 or so
tests I ran, and it seemed to deadlock after around 70-150 consumers had been
created.
This was in the ReleaseDLL config. The only change I made was to increase the
number of consumers to 500 just because the deadlocks were happening around 100
or so. Also, I'm using the .dll versions of APR, so I made that change to the
project.
I guess this could be a problem of how I'm compiling APR and/or Activemq.
That's always sitting in the back of my mind. I'm assuming I did it the
"standard" way. But like I said, we don't have this problem when we revert
back to 2.2.2, and that was compiled the same way.
I'm going to attach callstacks for all threads from a deadlock in the unit
test. Let me know what else I can provide to help narrow this down.
In this case, it deadlocked with ix=164. The callstacks look the same
regardless of how many it has created.
List of threads:
0 > 4456 Main Thread Main Thread
decaf::internal::util::concurrent::ConditionImpl::wait Normal 0
0 1804 Worker Thread `anonymous namespace'::threadWorker
decaf::internal::util::concurrent::ConditionImpl::wait Normal 0
0 5704 Worker Thread `anonymous namespace'::threadWorker
decaf::internal::util::concurrent::ConditionImpl::wait Normal 0
0 7408 Worker Thread `anonymous namespace'::threadWorker
decaf::internal::util::concurrent::ConditionImpl::wait Normal 0
0 3312 Worker Thread `anonymous namespace'::threadWorker
decaf::internal::util::concurrent::ConditionImpl::wait Normal 0
---------------
4456 Main Thread
ntdll.dll!771df871()
[Frames below may be incorrect and/or missing, no symbols loaded for
ntdll.dll]
ntdll.dll!771df871()
KernelBase.dll!763c0816()
kernel32.dll!74fb1138()
activemq-cpp.dll!decaf::internal::util::concurrent::ConditionImpl::wait(decaf::util::concurrent::ConditionHandle
* condition=0x00bc23a0, __int64 mills=4294967295, __int64 nanos=0) Line 110 +
0x10 bytes C++
activemq-cpp.dll!decaf::util::concurrent::Mutex::wait(__int64
millisecs=4294967295, int nanos=0) Line 124 + 0x20 bytes C++
activemq-cpp.dll!decaf::lang::Thread::join(__int64
millisecs=4294967295, unsigned int nanos=0) Line 464 + 0x36 bytes C++
activemq-cpp.dll!decaf::lang::Thread::join() Line 421 C++
activemq-cpp.dll!activemq::threads::DedicatedTaskRunner::shutdown()
Line 83 C++
activemq-cpp.dll!activemq::core::ActiveMQSessionExecutor::stop() Line
110 C++
activemq-cpp.dll!activemq::core::ActiveMQSession::stop() Line 807
C++
activemq-cpp.dll!activemq::core::ActiveMQConsumer::setMessageListener(cms::MessageListener
* listener=0x0018fadc) Line 523 C++
>
> vs2005-activemq-unittests.exe!activemq::core::ActiveMQSessionTest::testCreateManyConsumersAndSetListeners()
> Line 127 C++
vs2005-activemq-unittests.exe!CppUnit::TestCaller<activemq::core::ActiveMQSessionTest>::runTest()
Line 173 C++
vs2005-activemq-unittests.exe!CppUnit::TestCaseMethodFunctor::operator()()
Line 33 C++
vs2005-activemq-unittests.exe!CppUnit::DefaultProtector::protect(const
CppUnit::Functor & functor={...}, const CppUnit::ProtectorContext &
context={...}) Line 15 + 0x11 bytes C++
vs2005-activemq-unittests.exe!CppUnit::ProtectorChain::ProtectFunctor::operator()()
Line 20 + 0x13 bytes C++
vs2005-activemq-unittests.exe!CppUnit::ProtectorChain::protect(const
CppUnit::Functor & functor={...}, const CppUnit::ProtectorContext &
context={...}) Line 79 C++
vs2005-activemq-unittests.exe!CppUnit::TestResult::protect(const
CppUnit::Functor & functor={...}, CppUnit::Test * test=0x00b1ba08, const
std::basic_string<char,std::char_traits<char>,std::allocator<char> > &
shortDescription="") Line 178 + 0x18 bytes C++
vs2005-activemq-unittests.exe!CppUnit::TestCase::run(CppUnit::TestResult *
result=0x00b3a770) Line 92 + 0x4a bytes C++
vs2005-activemq-unittests.exe!CppUnit::TestComposite::doRunChildTests(CppUnit::TestResult
* controller=0x00b3a770) Line 64 + 0x14 bytes C++
vs2005-activemq-unittests.exe!CppUnit::TestComposite::run(CppUnit::TestResult *
result=0x00b3a770) Line 24 C++
vs2005-activemq-unittests.exe!CppUnit::TestComposite::doRunChildTests(CppUnit::TestResult
* controller=0x00b3a770) Line 64 + 0x14 bytes C++
vs2005-activemq-unittests.exe!CppUnit::TestComposite::run(CppUnit::TestResult *
result=0x00b3a770) Line 24 C++
vs2005-activemq-unittests.exe!CppUnit::TestResult::runTest(CppUnit::Test *
test=0x00b39390) Line 146 C++
vs2005-activemq-unittests.exe!CppUnit::TestRunner::run(CppUnit::TestResult &
controller={...}, const
std::basic_string<char,std::char_traits<char>,std::allocator<char> > &
testPath="") Line 97 C++
vs2005-activemq-unittests.exe!CppUnit::TextTestRunner::run(std::basic_string<char,std::char_traits<char>,std::allocator<char>
> testName="", bool doWait=false, bool doPrintResult=true, bool
doPrintProgress=true) Line 66 C++
vs2005-activemq-unittests.exe!main(int argc=1, char * *
argv=0x00c38050) Line 111 + 0x24 bytes C++
vs2005-activemq-unittests.exe!__tmainCRTStartup() Line 586 + 0x17
bytes C
kernel32.dll!74fb3677()
ntdll.dll!771f9d72()
ntdll.dll!771f9d45()
vs2005-activemq-unittests.exe!___sse2_available_init() + 0x3cff2 bytes
C++
fffb5c8d()
---------------
1804 Worker Thread
ntdll.dll!771df871()
[Frames below may be incorrect and/or missing, no symbols loaded for
ntdll.dll]
ntdll.dll!771df871()
KernelBase.dll!763c0816()
kernel32.dll!74fb1138()
>
> activemq-cpp.dll!decaf::internal::util::concurrent::ConditionImpl::wait(decaf::util::concurrent::ConditionHandle
> * condition=0x00b408c8, __int64 mills=4294967295, __int64 nanos=0) Line 110
> + 0x10 bytes C++
activemq-cpp.dll!decaf::internal::util::concurrent::ConditionImpl::wait(decaf::util::concurrent::ConditionHandle
* condition=0x00b408c8) Line 71 + 0x11 bytes C++
activemq-cpp.dll!decaf::util::concurrent::Mutex::wait() Line 95 + 0xf
bytes C++
activemq-cpp.dll!decaf::util::StlQueue<std::pair<decaf::lang::Runnable
*,decaf::util::concurrent::TaskListener *> >::wait() Line 269 C++
activemq-cpp.dll!decaf::util::concurrent::ThreadPool::deQueueTask()
Line 151 C++
activemq-cpp.dll!decaf::util::concurrent::PooledThread::run() Line 68
+ 0x1d bytes C++
activemq-cpp.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang::ThreadProperties
* properties=0x00b40c28) Line 133 + 0x11 bytes C++
activemq-cpp.dll!`anonymous namespace'::threadWorker(void *
arg=0x00b40c28) Line 204 + 0x9 bytes C++
msvcr90.dll!727d3433()
msvcr90.dll!727d34c7()
kernel32.dll!74fb3677()
ntdll.dll!771f9d72()
ntdll.dll!771f9d45()
---------------
5704 Worker Thread
ntdll.dll!771df871()
[Frames below may be incorrect and/or missing, no symbols loaded for
ntdll.dll]
ntdll.dll!771df871()
KernelBase.dll!763c0816()
kernel32.dll!74fb1138()
>
> activemq-cpp.dll!decaf::internal::util::concurrent::ConditionImpl::wait(decaf::util::concurrent::ConditionHandle
> * condition=0x00b408c8, __int64 mills=4294967295, __int64 nanos=0) Line 110
> + 0x10 bytes C++
activemq-cpp.dll!decaf::internal::util::concurrent::ConditionImpl::wait(decaf::util::concurrent::ConditionHandle
* condition=0x00b408c8) Line 71 + 0x11 bytes C++
activemq-cpp.dll!decaf::util::concurrent::Mutex::wait() Line 95 + 0xf
bytes C++
activemq-cpp.dll!decaf::util::StlQueue<std::pair<decaf::lang::Runnable
*,decaf::util::concurrent::TaskListener *> >::wait() Line 269 C++
activemq-cpp.dll!decaf::util::concurrent::ThreadPool::deQueueTask()
Line 151 C++
activemq-cpp.dll!decaf::util::concurrent::PooledThread::run() Line 68
+ 0x1d bytes C++
activemq-cpp.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang::ThreadProperties
* properties=0x00b40fa8) Line 133 + 0x11 bytes C++
activemq-cpp.dll!`anonymous namespace'::threadWorker(void *
arg=0x00b40fa8) Line 204 + 0x9 bytes C++
msvcr90.dll!727d3433()
msvcr90.dll!727d34c7()
kernel32.dll!74fb3677()
ntdll.dll!771f9d72()
ntdll.dll!771f9d45()
---------------
7408 Worker Thread
ntdll.dll!771df871()
[Frames below may be incorrect and/or missing, no symbols loaded for
ntdll.dll]
ntdll.dll!771df871()
KernelBase.dll!763c0816()
msctf.dll!76cf1651()
kernel32.dll!74fb1138()
>
> activemq-cpp.dll!decaf::internal::util::concurrent::ConditionImpl::wait(decaf::util::concurrent::ConditionHandle
> * condition=0x00b408c8, __int64 mills=4294967295, __int64 nanos=0) Line 110
> + 0x10 bytes C++
activemq-cpp.dll!decaf::internal::util::concurrent::ConditionImpl::wait(decaf::util::concurrent::ConditionHandle
* condition=0x00b408c8) Line 71 + 0x11 bytes C++
activemq-cpp.dll!decaf::util::concurrent::Mutex::wait() Line 95 + 0xf
bytes C++
activemq-cpp.dll!decaf::util::StlQueue<std::pair<decaf::lang::Runnable
*,decaf::util::concurrent::TaskListener *> >::wait() Line 269 C++
activemq-cpp.dll!decaf::util::concurrent::ThreadPool::deQueueTask()
Line 151 C++
activemq-cpp.dll!decaf::util::concurrent::PooledThread::run() Line 68
+ 0x1d bytes C++
activemq-cpp.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang::ThreadProperties
* properties=0x00b41408) Line 133 + 0x11 bytes C++
activemq-cpp.dll!`anonymous namespace'::threadWorker(void *
arg=0x00b41408) Line 204 + 0x9 bytes C++
msvcr90.dll!727d3433()
msvcr90.dll!727d34c7()
kernel32.dll!74fb3677()
ntdll.dll!771f9d72()
ntdll.dll!771f9d45()
---------------
3312 Worker Thread
ntdll.dll!771df871()
[Frames below may be incorrect and/or missing, no symbols loaded for
ntdll.dll]
ntdll.dll!771df871()
KernelBase.dll!763c0816()
kernel32.dll!74fb1138()
>
> activemq-cpp.dll!decaf::internal::util::concurrent::ConditionImpl::wait(decaf::util::concurrent::ConditionHandle
> * condition=0x00b485b0, __int64 mills=4294967295, __int64 nanos=0) Line 110
> + 0x10 bytes C++
activemq-cpp.dll!decaf::internal::util::concurrent::ConditionImpl::wait(decaf::util::concurrent::ConditionHandle
* condition=0x00b485b0) Line 71 + 0x11 bytes C++
activemq-cpp.dll!decaf::util::concurrent::Mutex::wait() Line 95 + 0xf
bytes C++
activemq-cpp.dll!decaf::util::StlQueue<decaf::lang::Pointer<activemq::commands::Command,decaf::lang::AtomicRefCounter>
>::wait() Line 269 C++
activemq-cpp.dll!activemq::transport::mock::InternalCommandListener::run()
Line 75 C++
activemq-cpp.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang::ThreadProperties
* properties=0x00b43050) Line 133 + 0x11 bytes C++
activemq-cpp.dll!`anonymous namespace'::threadWorker(void *
arg=0x00b43050) Line 204 + 0x9 bytes C++
msvcr90.dll!727d3433()
msvcr90.dll!727d34c7()
kernel32.dll!74fb3677()
ntdll.dll!771f9d72()
ntdll.dll!771f9d45()
> Freeze when creating multiple Consumers
> ---------------------------------------
>
> Key: AMQCPP-277
> URL: https://issues.apache.org/activemq/browse/AMQCPP-277
> Project: ActiveMQ C++ Client
> Issue Type: Bug
> Affects Versions: 3.1
> Environment: Windows 7 x64, ActiveMQ Server 5.3, Java 6.0.17, Visual
> Studio 2008 SP1, Windows 7 SDK, APR 1.3.9
> Reporter: Eddie Fast
> Assignee: Timothy Bish
> Fix For: 3.2.0
>
>
> We create multiple MessageConsumers with different selectors. This can be
> upwards of 50 or so consumers per client.
> We recently upgraded to 3.1 from 2.2.2. We also upgraded our ActiveMQ
> Server to 5.3 from 5.2.
> After upgrading, we've been seeing random, but frequent freezing in our
> clients. I've narrowed it down to where it creates the MessageConsumer and
> calls setMessageListener();
> It seems to be stuck, waiting for a thread to join.
> I've modified the main.cpp to exhibit this behavior, but I've only been able
> to reproduce it randomly. It seems to happen more frequently if you stick a
> breakpoint on this line in Thread.cpp, Thread::join():
> Thread::join( INFINITE, 0 );
> Here's the callstack where it freezes, which is consistent in our clients as
> well as the main.cpp example:
> ntdll.dll!76fff871()
> [Frames below may be incorrect and/or missing, no symbols loaded for
> ntdll.dll]
> ntdll.dll!76fff871()
> KernelBase.dll!75e20816()
> kernel32.dll!755b1138()
>
> activemq-cppd.dll!decaf::internal::util::concurrent::ConditionImpl::wait(decaf::util::concurrent::ConditionHandle
> * condition=0x0239e0e0, __int64 mills=4294967295, __int64 nanos=0) Line 110
> + 0x10 bytes C++
> activemq-cppd.dll!decaf::util::concurrent::Mutex::wait(__int64
> millisecs=4294967295, int nanos=0) Line 124 + 0x20 bytes C++
> activemq-cppd.dll!decaf::lang::Thread::join(__int64
> millisecs=4294967295, unsigned int nanos=0) Line 464 + 0x36 bytes C++
> activemq-cppd.dll!decaf::lang::Thread::join() Line 421 C++
> > activemq-cppd.dll!activemq::threads::DedicatedTaskRunner::shutdown()
> > Line 83 C++
> activemq-cppd.dll!activemq::core::ActiveMQSessionExecutor::stop() Line
> 110 C++
> activemq-cppd.dll!activemq::core::ActiveMQSession::stop() Line 807
> C++
>
> activemq-cppd.dll!activemq::core::ActiveMQConsumer::setMessageListener(cms::MessageListener
> * listener=0x0018fe0c) Line 523 C++
> vs2005-activemq-example.exe!HelloWorldConsumer::run() Line 247 + 0x5b
> bytes C++
>
> activemq-cppd.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang::ThreadProperties
> * properties=0x01df7f68) Line 133 + 0x11 bytes C++
> activemq-cppd.dll!`anonymous namespace'::threadWorker(void *
> arg=0x01df7f68) Line 204 + 0x9 bytes C++
> msvcr90d.dll!_callthreadstartex() Line 348 + 0xf bytes C
> msvcr90d.dll!_threadstartex(void * ptd=0x01dfafd8) Line 331 C
> To modify main.cpp, it's pretty simple. Instead of creating 1
> MessageConsumer in HelloWorldConsumer::run(), I've created 50:
> {quote}
> // put this line with the class variables
> std::vector<MessageConsumer*> consumers;
> for ( int i = 0; i < 50; i++ )
> {{
> // Create a MessageConsumer from the Session to the Topic or Queue
> consumers.push_back( session->createConsumer( destination ) );
> consumers.back()->setMessageListener( this );
> }}
> {quote}
> I'm using the standard .conf for the server. I've reverted to 2.2.2 and
> have no issues.
> Any ideas?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
