[
https://issues.apache.org/jira/browse/PROTON-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400483#comment-16400483
]
Miha Plesko commented on PROTON-1791:
-------------------------------------
Hi again guys,
a) [~aconway] the patch you pointed me to regarding file descriptor leakage
unfortunately doesn't change anything. It seems like it's not the right place
to invoke `io.close` there although I don't know why. The stalled sockets
remain.
b) To workaround heartbeating problem we have to invoke _process_ function o
ConnectionDriver instance periodically (every 5 seconds):
{code:java}
connection_driver.process
{code}
Only then the connection is not closed with "local-idle-timeout expired" error
every 2 or 3 minutes. It didn't help if we were sending messages to the queue,
nor opening and closing dummy session on connection. In fact, we weren't even
able to send more than one message to the queue because _on_sendable_ callback
was only invoked once after _connection.open_receiver_. I'm not sure why this
happens (is it a bug? is it the ActiveMQ configuration?), but I didn't have
time to dive into.
To address both problems (i.e. socket leakage and heartbeat sending) we had to
implement it like this
[https://github.com/ManageIQ/manageiq-providers-nuage/pull/72/files] . If you
can, I'd kindly ask you guys for a review on that small PR. I realize you can't
really test it without access to ActiveMQ, but since you're much more familiar
with the qpid_proton internals than myself, I'd really value your thoughts.
Best Regards,
Miha
> TCP sockets remain open in CLOSE_WAIT state
> -------------------------------------------
>
> Key: PROTON-1791
> URL: https://issues.apache.org/jira/browse/PROTON-1791
> Project: Qpid Proton
> Issue Type: Bug
> Components: ruby-binding
> Affects Versions: proton-c-0.21.0
> Environment: Confirmed on Ubuntu 16.04 and RHEL 7.4
> Confirmed on qpid_proton 0.19.0 and 0.21.0
> Reporter: Miha Plesko
> Assignee: Alan Conway
> Priority: Major
> Labels: bug
> Fix For: proton-c-0.22.0
>
> Attachments: idle-disconnect.png
>
>
> Hi guys,
> thanks for developing the awesome qpid_proton ruby gem, we're using it on
> daily basis!
> However, recently we noticed following error in our server log:
> Too many open files - socket(2) for "172.16.117.189" port 5672
> After some research it turns out that qpid_proton process is having
> increasingly
> more and more following file descriptors open:
> $ lsof -ap 108533
> ruby 108533 miha 116u IPv4 562438 0t0 TCP
> 172.16.117.189:53626->147.75.102.132:amqp (CLOSE_WAIT)
> ruby 108533 miha 197u IPv4 561644 0t0 TCP
> 172.16.117.189:53630->147.75.102.132:amqp (CLOSE_WAIT)
> ruby 108533 miha 311u IPv4 560657 0t0 TCP
> 172.16.117.189:53634->147.75.102.132:amqp (CLOSE_WAIT)
> ruby 108533 miha 549u IPv4 565342 0t0 TCP
> 172.16.117.189:53642->147.75.102.132:amqp (CLOSE_WAIT)
> ruby 108533 miha 576u IPv4 565122 0t0 TCP
> 172.16.117.189:53650->147.75.102.132:amqp (CLOSE_WAIT)
> ruby 108533 miha 603u IPv4 565738 0t0 TCP
> 172.16.117.189:53654->147.75.102.132:amqp (CLOSE_WAIT)
> ruby 108533 miha 630u IPv4 563021 0t0 TCP
> 172.16.117.189:53658->147.75.102.132:amqp (CLOSE_WAIT)
> ruby 108533 miha 657u IPv4 568361 0t0 TCP
> 172.16.117.189:53662->147.75.102.132:amqp (CLOSE_WAIT)
> ruby 108533 miha 666u IPv4 563027 0t0 TCP
> 172.16.117.189:53666->147.75.102.132:amqp (CLOSE_WAIT)
> ruby 108533 miha 675u IPv4 567538 0t0 TCP
> 172.16.117.189:53670->147.75.102.132:amqp (CLOSE_WAIT)
> ruby 108533 miha 684u IPv4 567998 0t0 TCP
> 172.16.117.189:53678->147.75.102.132:amqp (CLOSE_WAIT)
> ruby 108533 miha 690u IPv4 574709 0t0 TCP
> 172.16.117.189:53686->147.75.102.132:amqp (CLOSE_WAIT)
> ruby 108533 miha 693u IPv4 578725 0t0 TCP
> 172.16.117.189:53694->147.75.102.132:amqp (CLOSE_WAIT)
> ruby 108533 miha 696u IPv4 576840 0t0 TCP
> 172.16.117.189:53698->147.75.102.132:amqp (CLOSE_WAIT)
> ruby 108533 miha 699u IPv4 577819 0t0 TCP
> 172.16.117.189:53702->147.75.102.132:amqp (CLOSE_WAIT)
> ruby 108533 miha 702u IPv4 582192 0t0 TCP
> 172.16.117.189:53710->147.75.102.132:amqp (CLOSE_WAIT)
> ruby 108533 miha 705u IPv4 582861 0t0 TCP
> 172.16.117.189:53714->147.75.102.132:amqp (CLOSE_WAIT)
> ruby 108533 miha 708u IPv4 577363 0t0 TCP
> 172.16.117.189:53718->147.75.102.132:amqp (CLOSE_WAIT)
> ruby 108533 miha 711u IPv4 578175 0t0 TCP
> 172.16.117.189:53722->147.75.102.132:amqp (CLOSE_WAIT)
> ruby 108533 miha 714u IPv4 587172 0t0 TCP
> 172.16.117.189:53730->147.75.102.132:amqp (CLOSE_WAIT)
> ruby 108533 miha 717u IPv4 584387 0t0 TCP
> 172.16.117.189:53734->147.75.102.132:amqp (CLOSE_WAIT)
> ...
> I think the CLOSE_WAIT status of file descriptor indicates that the TCP
> connection has already been closed, but the file descriptor wasn't closed.
> After 9 hours or so there are enough of such file descriptors for OS to
> complain about it.
> We did all we could to close connections gracefully:
> connection.container.stop
> connection.close
> connection = nil
> but nothing seems to help. A simple but expensive workaround is to manually
> invoke Ruby's garbage collection,
> but ideally `connection.close` would close the file descriptor.
> May I kindly ask you to look at this?
> Thank you and Best Regards,
> Miha
> PS: The error occurs both on Ubuntu 16.04 and RHEL 7.4
> PS2: The error occurs both on qpid_proton 0.19.0 and 0.21.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
