Memory leak/segv's with frequent reconnections (identified)
-----------------------------------------------------------
Key: AMQCPP-207
URL: https://issues.apache.org/activemq/browse/AMQCPP-207
Project: ActiveMQ C++ Client
Issue Type: Bug
Components: CMS Impl, Decaf, Openwire
Affects Versions: 2.2.1
Environment: Windows XP, Vistual Studio 2005 (C++)
Reporter: Michael J. Kitchin
Assignee: Timothy Bish
If a CMS consumer client reconnects frequently a segv will arise with
reliability (it's guaranteed for us within a hundred reconnections or so).
Additionally, in researching this we seem to have stumbled across a related
memory leak.
When we shut down our consumer in the following order, as suggested by the
sample code:
- delete destination
- delete consumer
- close session
- close connection (*)
- delete session
- delete connection
...a segv will occur at the step marked with a (*), due to effects observable
in these thread stack traces:
Thread #1:
> activemq-cppd.dll!activemq::transport::IOTransport::close() Line 158
> C++
activemq-cppd.dll!activemq::transport::TransportFilter::close() Line
222 + 0x1b bytes C++
activemq-cppd.dll!activemq::transport::filters::TcpTransport::close()
Line 147 + 0x8 bytes C++
activemq-cppd.dll!activemq::transport::filters::ResponseCorrelator::close()
Line 290 + 0x19 bytes C++
activemq-cppd.dll!activemq::connector::openwire::OpenWireFormatNegotiator::close()
Line 257 + 0x1b bytes C++
activemq-cppd.dll!activemq::connector::openwire::OpenWireConnector::close()
Line 178 + 0x19 bytes C++
activemq-cppd.dll!activemq::core::ActiveMQConnectionData::close() Line
89 + 0x19 bytes C++
activemq-cppd.dll!activemq::core::ActiveMQConnectionData::~ActiveMQConnectionData()
Line 77 C++
activemq-cppd.dll!activemq::core::ActiveMQConnectionData::`vector
deleting destructor'() + 0x54 bytes C++
activemq-cppd.dll!activemq::core::ActiveMQConnection::close() Line 160
+ 0x37 bytes C++
Thread #2:
activemq-cppd.dll!decaf::net::SocketInputStream::read(unsigned char *
buffer=0x08b4dcb8, unsigned int offset=0, unsigned int bufferSize=1) Line 170
+ 0x18 bytes C++
> activemq-cppd.dll!decaf::io::BufferedInputStream::bufferData() Line
> 260 + 0x30 bytes C++
activemq-cppd.dll!decaf::io::BufferedInputStream::read(unsigned char *
targetBuffer=0x0a80f7d8, unsigned int offset=0, unsigned int
targetBufferSize=4) Line 181 + 0x8 bytes C++
activemq-cppd.dll!decaf::io::DataInputStream::readAllData(unsigned char
* buffer=0x0a80f7d8, unsigned int length=4) Line 360 + 0x25 bytes C++
activemq-cppd.dll!decaf::io::DataInputStream::readInt() Line 167
C++
activemq-cppd.dll!activemq::connector::openwire::OpenWireFormat::unmarshal(decaf::io::DataInputStream
* dis=0x08bd66d8) Line 227 + 0xf bytes C++
activemq-cppd.dll!activemq::connector::openwire::OpenWireCommandReader::readCommand()
Line 71 + 0x1c bytes C++
activemq-cppd.dll!activemq::transport::IOTransport::run() Line 188 +
0x15 bytes C++
activemq-cppd.dll!decaf::lang::Thread::runCallback(apr_thread_t *
self=0x08bb3ea0, void * param=0x08bd6828) Line 113 + 0x15 bytes C++
libapr-1.dll!dummy_worker(void * opaque=0x08bb3ea0) Line 80 C
...where, in thread #2, the SocketInputStream interior to the
BufferedInputStream (the inputStream member) has been NULL'ed between two
method calls shown below, ending at bufferedinputstream.cpp:260:
// Get the number of bytes currently available on the input stream
// that could be read without blocking.
std::size_t available = inputStream->available();
// Calculate the number of bytes that we can read. Always >= 1 byte!
std::size_t bytesToRead = max( (std::size_t)1, min( available,
getUnusedBytes() ) );
// Read the bytes from the input stream.
int bytesRead = inputStream->read( getTail(), 0, bytesToRead );
....as a result of the stream close() call in thread #1. The close() call is in
place to break any waiting IO so the executor thread may terminate in an
orderly fashion, but the stream and/or iotransport calls don't anticipate a
case where in-stream processing is underway at the time its internal state is
reset.
This suggests either:
(1) more careful synchronization of the compound stream's state
(2) realigning the compound stream's member data
...I would vote the former, and could suggest some options.
Additionally, when reviewing the code in in thread #1 we found the following,
beginning at iotransport.cpp:150:
// We have to close the input stream before
// we stop the thread. this will force us to
// wake up the thread if it's stuck in a read
// (which is likely). Otherwise, the join that
// follows will block forever.
if( inputStream != NULL ){
inputStream->close();
inputStream = NULL;
}
// Wait for the thread to die.
if( thread != NULL ){
thread->join();
delete thread;
thread = NULL;
}
// Close the output stream.
if( outputStream != NULL ){
outputStream->close();
outputStream = NULL;
}
...our familiarity with decaf is limited, but it appears inputStream and
outputStream are conventional pointers, so should therefore be deleted prior to
being set to NULL to avoid leaking memory. To minimize the other potential segv
edge cases, we would suggest doing the close() calls before completing the
thread join(), then deleting/NULL'ing the pointers afterwards.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
