[
https://issues.apache.org/jira/browse/QPID-5072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13740805#comment-13740805
]
Gordon Sim commented on QPID-5072:
----------------------------------
I checked in a fix similar in nature to the attached patch, which at least
ensures the session is removed from the attached set, freeing up the name.
The leak appear to be coming from the store:
==581==
==581== HEAP SUMMARY:
==581== in use at exit: 2,011,611 bytes in 18,291 blocks
==581== total heap usage: 14,423,045 allocs, 14,404,754 frees, 1,481,971,039
bytes allocated
==581==
==581== Thread 1:
==581== 2,002,358 (188,968 direct, 1,813,390 indirect) bytes in 1,817 blocks
are definitely lost in loss record 126 of 126
==581== at 0x4A08301: operator new(unsigned long) (vg_replace_malloc.c:298)
==581== by 0x65684F7:
mrg::msgstore::MessageStoreImpl::store(qpid::broker::PersistableQueue const*,
mrg::msgstore::TxnCtxt*, boost::intrusive_ptr<qpid::broker::PersistableMessage>
const&, bool) (in /home/gordon/projects/qpid-git/build/main/src/legacystore.so)
==581== by 0x6568035:
mrg::msgstore::MessageStoreImpl::enqueue(qpid::broker::TransactionContext*,
boost::intrusive_ptr<qpid::broker::PersistableMessage> const&,
qpid::broker::PersistableQueue const&) (in
/home/gordon/projects/qpid-git/build/main/src/legacystore.so)
==581== by 0x5036153:
qpid::broker::MessageStoreModule::enqueue(qpid::broker::TransactionContext*,
boost::intrusive_ptr<qpid::broker::PersistableMessage> const&,
qpid::broker::PersistableQueue const&) (in
/home/gordon/projects/qpid-git/build/main/src/libqpidbroker.so.2.0.0)
==581== by 0x4FB2203:
qpid::broker::Queue::enqueue(qpid::broker::TransactionContext*,
qpid::broker::Message&) (in
/home/gordon/projects/qpid-git/build/main/src/libqpidbroker.so.2.0.0)
==581== by 0x4FAD63E: qpid::broker::Queue::deliverTo(qpid::broker::Message,
qpid::broker::TxBuffer*) (in
/home/gordon/projects/qpid-git/build/main/src/libqpidbroker.so.2.0.0)
==581== by 0x4FAD4A0: qpid::broker::Queue::deliver(qpid::broker::Message,
qpid::broker::TxBuffer*) (in
/home/gordon/projects/qpid-git/build/main/src/libqpidbroker.so.2.0.0)
==581== by 0x4FED4B3:
qpid::broker::DeliverableMessage::deliverTo(boost::shared_ptr<qpid::broker::Queue>
const&) (in
/home/gordon/projects/qpid-git/build/main/src/libqpidbroker.so.2.0.0)
==581== by 0x4F8AD44:
qpid::broker::Exchange::doRoute(qpid::broker::Deliverable&,
boost::shared_ptr<std::vector<boost::shared_ptr<qpid::broker::Exchange::Binding>,
std::allocator<boost::shared_ptr<qpid::broker::Exchange::Binding> > > const>)
(in /home/gordon/projects/qpid-git/build/main/src/libqpidbroker.so.2.0.0)
==581== by 0x4FF16C1:
qpid::broker::DirectExchange::route(qpid::broker::Deliverable&) (in
/home/gordon/projects/qpid-git/build/main/src/libqpidbroker.so.2.0.0)
==581== by 0x5067755:
qpid::broker::SemanticState::route(qpid::broker::Message&,
qpid::broker::Deliverable&) (in
/home/gordon/projects/qpid-git/build/main/src/libqpidbroker.so.2.0.0)
==581== by 0x50843B6:
qpid::broker::SessionState::handleContent(qpid::framing::AMQFrame&,
qpid::framing::SequenceNumber const&) (in
/home/gordon/projects/qpid-git/build/main/src/libqpidbroker.so.2.0.0)
==581==
==581== LEAK SUMMARY:
==581== definitely lost: 188,968 bytes in 1,817 blocks
==581== indirectly lost: 1,813,390 bytes in 16,354 blocks
==581== possibly lost: 0 bytes in 0 blocks
==581== still reachable: 9,253 bytes in 120 blocks
==581== suppressed: 0 bytes in 0 blocks
==581== Reachable blocks (those to which a pointer was found) are not shown.
==581== To see them, rerun with: --leak-check=full --show-reachable=yes
==581==
==581== For counts of detected and suppressed errors, rerun with: -v
==581== Use --track-origins=yes to see where uninitialised values come from
==581== ERROR SUMMARY: 361 errors from 4 contexts (suppressed: 2 from 2)
In the code:
try {
if (queue) {
boost::intrusive_ptr<DataTokenImpl> dtokp(new DataTokenImpl);
dtokp->addRef();
I suspect the line above may be the issue. This will prevent the DataTokenImpl
being deleted even when the dtokp pointer is removed. I assume this is being
done because the journal needs to explicitly decrement that when done. Probably
that step is not happening when the enqueue fails due to insufficient capacity.
dtokp->setSourceMessage(message);
dtokp->set_external_rid(true);
dtokp->set_rid(message->getPersistenceId()); // set the messageID
into the Journal header (record-id)
JournalImpl* jc =
static_cast<JournalImpl*>(queue->getExternalQueueStore());
if (txn->getXid().empty()) {
jc->enqueue_data_record(&buff[0], size, size, dtokp.get(),
!message->isPersistent());
(I faked up a similar exception in the broker itself, and running valgrind
without the store loaded but with a siimilar error pattern, there is no leak
reported by valgrind).
> [C++ broker] SessionManager does not forget sessions when broker drops
> connection after journal error, leading to memory leak
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: QPID-5072
> URL: https://issues.apache.org/jira/browse/QPID-5072
> Project: Qpid
> Issue Type: Bug
> Components: C++ Broker
> Affects Versions: 0.22
> Reporter: Pavel Moravec
> Attachments: bz991413.patch, memoryLeak-in-sessionManager.sh,
> test_00903524_threaded.java
>
>
> Description of problem:
> Broker rejects session re-subscription (on a new AMQP+TCP connection) after
> the previous was dropped due to a journal error. This bug in broker:
> - is in fact a memory leak, as session manager map is not cleared when
> requested
> - it affects Java client during failover, when original journal problem has
> been already resolved, but the client can't even attach the same session.
> Version-Release number of selected component (if applicable):
> any (teste 0.18 and also 0.22 broker)
> How reproducible:
> 100%
> Steps to Reproduce:
> 1) run attached Java reproducer on a fresh broker:
> java test_00903524_threaded 1
> 2) check qpidd logs for errors
> Actual results:
> 2013-08-02 11:39:04 [Protocol] error Unexpected exception: Enqueue capacity
> threshold exceeded on queue "testQueue".
> (/builddir/build/BUILD/qpid-0.22/cpp/src/qpid/legacystore/JournalImpl.cpp:594)
> 2013-08-02 11:39:04 [Protocol] error Connection
> qpid.10.34.1.141:5672-10.34.1.141:42635 closed by error: Enqueue capacity
> threshold exceeded on queue "testQueue".
> (/builddir/build/BUILD/qpid-0.22/cpp/src/qpid/legacystore/JournalImpl.cpp:594)(501)
> 2013-08-02 11:39:14 [Protocol] error Channel exception: session-busy: Session
> already attached: [email protected]
> (/builddir/build/BUILD/qpid-0.22/cpp/src/qpid/broker/SessionManager.cpp:55)
> 2013-08-02 11:39:14 [Protocol] error Channel exception: not-attached: Channel
> 1 is not attached
> (/builddir/build/BUILD/qpid-0.22/cpp/src/qpid/amqp_0_10/SessionHandler.cpp:39)
> ..
> (few thousands of the latest error)
> Expected results:
> just the journal error be seen (and multiple times), not the "Session already
> attached" one or "Channel 1 is not attached" one.
> Additional info:
> The broker management _deletes_ the session, as the broker logs:
> 2013-08-02 11:39:04 [Management] trace Management object marked deleted:
> org.apache.qpid.broker:session:36332225-5c5b-4077-b8ec-820555253a89
> org.apache.qpid.broker:session:36332225-5c5b-4077-b8ec-820555253a89
> (deleted)
> 2013-08-02 11:39:10 [Management] trace Deleting V1 properties
> 0-68-1--14(org.apache.qpid.broker:session:36332225-5c5b-4077-b8ec-820555253a89)
> len=164
> 2013-08-02 11:39:10 [Management] trace Deleting V1 statistics
> 0-68-1--14(org.apache.qpid.broker:session:36332225-5c5b-4077-b8ec-820555253a89)
> len=127
> 2013-08-02 11:39:10 [Management] trace Deleting V2
> map={_create_ts:1375436343054221056, _delete_ts:1375436344249095890,
> _object_id:{_agent_epoch:68,
> _object_name:org.apache.qpid.broker:session:36332225-5c5b-4077-b8ec-820555253a89},
> _schema_id:{_class_name:session, _hash:1aaa08d0-c118-ff78-0956-47b9ac9c6849,
> _package_name:org.apache.qpid.broker, _type:_data},
> _update_ts:1375436343054221056, _values:{TxnCommits:0, TxnCount:0,
> TxnRejects:0, TxnStarts:0, attached:True, channelId:0, clientCredit:0,
> connectionRef:{_agent_epoch:68,
> _object_name:org.apache.qpid.broker:connection:qpid.10.34.1.141:5672-10.34.1.141:42635},
> detachedLifespan:0, name:36332225-5c5b-4077-b8ec-820555253a89,
> unackedMessages:0,
> vhostRef:{_object_name:org.apache.qpid.broker:vhost:org.apache.qpid.broker:broker:amqp-broker,/}}}
> But session manager inside does not forget the sessions/channels..
> Reproduction for memory leak:
> run attached script
> Weird is, even fixing the bug for Java client, some steady memory increase is
> still present..
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
