[
https://issues.apache.org/jira/browse/QPID-6297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14273060#comment-14273060
]
Pavel Moravec commented on QPID-6297:
-------------------------------------
Trivial reproducer:
0) decrease TCP retries:
echo 2 > /proc/sys/net/ipv4/tcp_retries2
1) Run this script that runs receiver.fetch(timeout=10) in a loop:
#!/usr/bin/env python
from qpid.messaging import *
import datetime
conn = Connection("localhost:5672", reconnect=1)
timeout=10
try:
conn.open()
sess = conn.session()
recv = sess.receiver("testQueue;{create:always}")
while (1):
print "%s: before fetch, timeout=%s" %(datetime.datetime.now(), timeout)
msg = Message()
try:
msg = recv.fetch(timeout=timeout)
except ReceiverError, e:
print e
print "%s: after fetch, msg=%s" %(datetime.datetime.now(), msg)
sess.close()
except ReceiverError, e:
print e
except KeyboardInterrupt:
pass
conn.close()
2) simulate network outage:
iptables -A OUTPUT -p tcp --dport 5672 -j REJECT; date
3) Once the script logs "No handlers could be found for logger
"qpid.messaging"", flush iptables
4) Wait few seconds for the backtrace
> Python client (qpid.messaging) raises KeyError insead of reconnecting
> ---------------------------------------------------------------------
>
> Key: QPID-6297
> URL: https://issues.apache.org/jira/browse/QPID-6297
> Project: Qpid
> Issue Type: Bug
> Components: Python Client
> Affects Versions: 0.22
> Environment: EL6
> Reporter: Jeff Ortel
> Attachments: goferBug.cap
>
>
> Description of problem:
> Having some temporary network outage causing gofer loses TCP connection to
> AMQP broker, it does not try to reconnect.
> How reproducible:
> 100%
> Steps to Reproduce:
> 1. Just to speedup reproducer, lower kernel tunable net.ipv4.tcp_retries2 to
> e.g. 4:
> echo 4 > /proc/sys/net/ipv4/tcp_retries2
> 2. Have consumer connected (with auto-reconnect enabled and heartbeats not
> enabled) and receiver open on a queue address and check its TCP connections
> to AMQP broker:
> netstat -anp | grep 5671
> (there should be 2 TCP connections)
> 3. Emulate network outage via iptables:
> iptables -A OUTPUT -p tcp --dport 5671 -j REJECT
> 4. Monitor /var/log/messages; once it logs WARNING "recoverable error", flush
> iptables (iptables -F).
> 5. Wait few seconds.
> 6. Check gofer TCP connections:
> netstat -anp | grep 5671
> Actual results:
> 6. shows just 1 TCP connection
> /var/log/messages repeatedly logs:
> Dec 1 16:39:02 pmoravec-rhel6-3 goferd:
> [ERROR][pulp.agent.a726580c-5f1e-4a79-9f11-de0adc52c1e9]
> gofer.transport.qpid.consumer:117 - 046d2084-b0f1-4de4-a039-89499d9e680d
> Dec 1 16:39:02 pmoravec-rhel6-3 goferd:
> [ERROR][pulp.agent.a726580c-5f1e-4a79-9f11-de0adc52c1e9]
> gofer.transport.qpid.consumer:117 - Traceback (most recent call last): File
> "/usr/lib/python2.6/site-packages/gofer/transport/qpid/consumer.py", line
> 113, in get return self.__receiver.fetch(timeout=timeout) File "<string>",
> line 6, in fetch File
> "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 1030, in
> fetch self._ecwait(lambda: self.linked) File
> "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in
> _ecwait result = self._ewait(lambda: self.closed or predicate(), timeout)
> File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line
> 993, in _ewait result = self.session._ewait(lambda: self.error or
> predicate(), timeout) File
> "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 580, in
> _ewait result = self.connection._ewait(lambda: self.error or predicate(),
> timeout) File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py",
> line 219, in _ewait self.check_error() File
> "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 212, in
> check_error raise e InternalError: Traceback (most recent call last): File
> "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 660, in
> write op.dispatch(self) File "/usr/lib/python2.6/site-packages/qpid/ops.py",
> line 84, in dispatch getattr(target, handler)(self, *args) File
> "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 877, in
> do_session_detached sst = self._sessions.pop(dtc.channel) KeyError: 'pop():
> dictionary is empty'
> Expected results:
> 2nd TCP connection re-established, no errors in /var/log/messages
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
