[
https://issues.apache.org/jira/browse/QPID-7987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16858431#comment-16858431
]
Alex Rudyy edited comment on QPID-7987 at 6/7/19 9:29 AM:
----------------------------------------------------------
Hi Utkarsh,
Thanks for volunteering for the task.
Unfortunately, I did not put much details for the problem into JIRA description
and cannot recollect what exactly was happening . I tried to reproduce the
problem as suggested in my earlier comment, but, that did not replicate the
issue so far.
The Qpid Broker utilizes direct memory for keeping message content/headers. The
direct memory is managed via {{QpidByteBuffer}} objects which are the wrappers
around java {{ByteBuffer}}. They have a reference counter mechanism. Every time
when buffer is duplicated, sliced, or any other operation invoked resulting in
creation of new {{QpidByteBuffer}} instance from the original one, the
reference counter is incremented. When duplicate/slice/view is disposed the
reference counter is decremented. When counter reaches 0, the pooled byte
buffer is returned into pool, otherwise, the instance of {{QpidByteBuffer}} is
garbage collected eventually.
Method {{QpidByteBuffer#dispose()}} has to be invoked every time when
{{QpidByteBuffer}} instance is not needed any more. If {{dispose}} is not
invoked, that can result in leakage of {{QpidByteBuffer}} s .
This JIRA was raised to track a race condition when {{transaction}} is idle or
inactive for some time (The application does not commit/rollback the
transaction for a while). It seems when AMQP 0-10 session is closed by
transaction timeout functionality and a new message is arriving at the same
time, there is a possibility for a race when incoming message
{{QpidByteBuffer}} s are not disposed. As result, the {{QpidByteBuffer}} leak
can occur.
Taking that the issue occurrence frequency was quite rear and 0-10 is a legacy
protocol, the issue was not addressed promptly. Since than, some changes had
been made in {{transaction timeout}} functionality which is currently closing
impacted connection rather than session (for consistency reasons). It looks
like that those changes might prevented the issue from occurrence in my
reproduction attempts.
I will try to reproduce the problem again later. If I will not manage to do it,
I will close the JIRA with "Cannot reproduce".
Potentially, you can try to replicate the issue by writing a test which can
send the messages when transaction timeout functionality is closing underlying
connection. You can follow existing integration tests
[TransactionTimeoutTest|https://github.com/apache/qpid-broker-j/blob/master/systests/qpid-systests-jms_1.1/src/test/java/org/apache/qpid/systests/jms_1_1/extensions/transactiontimeout/TransactionTimeoutTest.java].
It is going to be tricky to catch the moment...
Please, note that issue only affects AMQP 0-10 code
was (Author: alex.rufous):
Hi Utkarsh,
Thanks for volunteering for the task.
Unfortunately, I did not put much details for the problem into JIRA description
and cannot recollect what exactly was happening . I tried to reproduce the
problem as suggested in my earlier comment, but, that did not replicate the
issue so far.
The Qpid Broker utilizes direct memory for keeping message content/headers. The
direct memory is managed via {{QpidByteBuffer}} objects which are the wrappers
around java {{ByteBuffer}}. They have a reference counter mechanism. Every time
when buffer is duplicated, sliced, or any other operation invoked resulting in
creation of new {{QpidByteBuffer}} instance from the original one, the
reference counter is incremented. When duplicate/slice/view is disposed the
reference counter is decremented. When counter reaches 0, the pooled byte
buffer is returned into pool, otherwise, the instance of {{QpidByteBuffer}} is
garbage collected eventually.
Method {{QpidByteBuffer#dispose()}} has to be invoked every time when
{{QpidByteBuffer}} instance is not needed any more. If {{dispose}} is not
invoked, that can result in leakage of {{QpidByteBuffer}} s .
This JIRA was raised to track a race condition when {{transaction}} is idle or
inactive for some time (The application does not commit/rollback the
transaction for a while). It seems when AMQP 0-10 session is closed by
transaction timeout functionality and a new message is arriving at the same
time, there is a possibility for a race when incoming message
{{QpidByteBuffer}} s are not disposed. As result, the {{QpidByteBuffer}} leak
can occur.
Taking that the issue occurrence frequency was quite rear and 0-10 is a legacy
protocol, the issue was not addressed promptly. Since than, some changes had
been made in {{transaction timeout}} functionality which is currently closing
impacted connection rather than session (for consistency reasons). It looks
like that those changes might prevented the issue from occurrence in my
reproduction attempts.
I will try to reproduce the problem again later. If I will not manage to do it,
I will close the JIRA with "Cannot reproduce".
Potentially, you can try to replicate the issue by writing a test which can
send the messages when transaction timeout functionality is closing underlying
connection. You can follow existing integration tests
[TransactionTimeoutTest|https://github.com/apache/qpid-broker-j/blob/master/systests/qpid-systests-jms_1.1/src/test/java/org/apache/qpid/systests/jms_1_1/extensions/transactiontimeout/TransactionTimeoutTest.java].
It is going to be tricky to catch the moment...
Please, note that issue only affect AMQP 0-10 code
> [Qpid Broker-J][AMQP 0-10] QpidByteBuffer can leak when local transaction
> open/idle time exceeds the maximum open/idle timeout causing session close
> but a new message arrives at the same time when session is in process of close
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: QPID-7987
> URL: https://issues.apache.org/jira/browse/QPID-7987
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
> Affects Versions: qpid-java-6.1.4, qpid-java-broker-7.0.0, qpid-java-6.1.5
> Reporter: Alex Rudyy
> Priority: Major
> Attachments: 0002-instrumentation.patch
>
>
> {{QpidByteBuffer}} can leak when broker operates with configured open and
> idle transaction close timeouts (
> {{virtualhost.storeTransactionOpenTimeoutClose}} or/and
> {{virtualhost.storeTransactionIdleTimeoutClose}}) and the transaction timeout
> occurs causing the Broker to close the underlying session but at the same
> time a new message arrives. The new message {{QpidByteBuffer}} might not be
> disposed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
