[
https://issues.apache.org/jira/browse/QPID-8276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16767683#comment-16767683
]
Alex Rudyy commented on QPID-8276:
----------------------------------
The fixes were committed against wrong JIRA:
Commit 94de25eb9fb8be6e6deba38a72afcf7b14ce1d0b in qpid-broker-j's branch
refs/heads/master from Alex Rudyy
[ https://gitbox.apache.org/repos/asf?p=qpid-broker-j.git;h=94de25e ]
QPID-8273: [Broker-J] Set notify work on channel reaching end-of-read-stream
Commit ae6fecffad5edb4abde74f1e4d2e2060702523f2 in qpid-broker-j's branch
refs/heads/7.1.x from Alex Rudyy
[ https://gitbox.apache.org/repos/asf?p=qpid-broker-j.git;h=ae6fecf ]
QPID-8273: [Broker-J] Set notify work on channel reaching end-of-read-stream
(cherry picked from commit 94de25eb9fb8be6e6deba38a72afcf7b14ce1d0b)
Commit 2e72e832d9b8274cc09f7f1cfba313a40f8cc0da in qpid-broker-j's branch
refs/heads/7.0.x from Alex Rudyy
[ https://gitbox.apache.org/repos/asf?p=qpid-broker-j.git;h=2e72e83 ]
QPID-8273: [Broker-J] Set notify work on channel reaching end-of-read-stream
(cherry picked from commit 94de25eb9fb8be6e6deba38a72afcf7b14ce1d0b)
> [Broker-J] Broker can leak closed NonBlockingConnection objects and
> eventually run out of heap memory
> -----------------------------------------------------------------------------------------------------
>
> Key: QPID-8276
> URL: https://issues.apache.org/jira/browse/QPID-8276
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
> Affects Versions: qpid-java-broker-7.0.3, qpid-java-broker-7.0.2,
> qpid-java-broker-7.0.0, qpid-java-broker-7.0.1, qpid-java-6.1.7,
> qpid-java-broker-7.1.0, qpid-java-broker-7.0.4, qpid-java-broker-7.0.5,
> qpid-java-broker-7.0.6
> Reporter: Alex Rudyy
> Assignee: Alex Rudyy
> Priority: Critical
> Fix For: qpid-java-broker-7.0.7, qpid-java-broker-7.1.1
>
>
> The Qpid Broker-J can leak closed NonBlockingConnection objects.
> The heap dump analysis of impacted broker instance revealed that leaked
> {{NonBlockingConnection}} objects are accumulated in
> {{SelectorThread.SelectionTask#_unscheduledConnections}} belonging to AMQP
> port IO pool. They have no ticker set and no state changed flag set
> ({{NonBlockingConnection#isStateChanged() == false)}}. As result, the
> NonBlockingConnection objects are not removed from
> {{SelectorThread#_unscheduledConnections}} on invocation of
> {{SelectorThread.SelectionTask#processUnscheduledConnections()}} called from
> {{SelectorThread.SelectionTask#performSelect()}}.
> The {{NonBlockingConnection}} and underlying model object are in closed state.
> It seems that leaked {{NonBlockingConnection}} was closed as part of
> invocation {{NonBlockingConnection#doWork()}}. The connection was
> unregistered on {{VirtualHost}} IO pool and re-registered with port IO pool
> as part of invocation {{NetworkConnectionScheduler#processConnection}} At
> first, it was stored in collection
> {{SelectorThread.SelectionTask#_unregisteredConnections}}. Later on, it was
> moved from {{SelectorThread.SelectionTask#_unregisteredConnections}} to
> {{SelectorThread.SelectionTask#_unscheduledConnections}} as part of
> invocation {{SelectorThread.SelectionTask#reregisterUnregisteredConnections}}
> and stack there afterwards.
> The TLS transport was used in leaked connection, but, I think that connection
> with plain transport can be leaked as well.
> I suspect that connections were leaked in result of following scenario:
> * Invocation of {{SocketChannel#read(java.nio.ByteBuffer[])}} returned
> {{-1}} in {{NonBlockingConnection#readFromNetwork}}.
> * The flag {{NonBlockingConnection#_closed}} was set to {{true}}. The method
> {{ProtocolEngine#notifyWork()}} was not invoked to set {{state changed}} flag
> to {{true}}
> * The execution of {{NonBlockingConnection#doWork()}} ended up it connection
> shutdown (due to {{NonBlockingConnection#_closed}} being set) and following
> re-scheduling the connection on port IO scheduler. The latter resulted in
> connection being put into
> {{SelectorThread.SelectionTask#_unscheduledConnections}} as described above.
> It seems that opening and closing frequent connections with connection life
> span {{>10s}} (required for tickers to be removed) can ended-up in
> connections being leaked as described in scenario above. It looks like
> connections which are closed orderly or closed in result of {{IOException}}
> being thrown from socket read/write operation are not effected by the defect.
> The impacted broker instance can eventually crash with out of memory error.
> Broker memory monitoring and periodic broker restarts can mitigate the issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
