---------- Forwarded message ---------- From: Roman Sokolkov <rsokol...@gmail.com> Date: 2011/12/28 Subject: Re: [Openstack] reconnecting to RabbitMQ To: Ask Solem <a...@rabbitmq.com>
Hi! I tried to make Rabbit HA for Openstack. My first e-mail was little innacurate. I use two nodes with active/active rabbits with mirrored queues. Additionally I use Virtual IP. And all it managed by Pacemaker. Openstack uses this remote rabbitmq via VIP. I hardly reboot node with VIP and VIP moves to other node. In this case, connections on Openstack host staying forever in ESTABLISHED state(in py-amqlib 0.6.1 no tcp keepalive options). Openstack think that all good and doesn't try to reconnect. But I if correctly stop RabbitMQ, connections closed correctly and Openstack will try to reconnect. I solve this issue by adding tcp_keepalive options in py-amqlib(transport.py). It's very interesting what implemented in KAMQP. And how heartbeats work? 2011/12/28 Ask Solem <a...@rabbitmq.com> > Not sure how I can help until the problem is identified. > Kombu does not do automatic reconnecting by default, but provides > the primitives necessary to implement it. > > amqplib does not support heartbeats so there may be cases > where the client/server does not properly detect that the socket > has been closed, this usually only a problem with some particular > intermediate > network components like routers and firewalls. Support for heartbeats > is currently in the works, and I already have a rudimentary implementation > at http://github.com/ask/kamqp/ if someone wants to try it out (though > that > probably requires some hacking). > > > On 28 Dec 2011, at 10:51, Alexis Richardson wrote: > > > Hi all, > > > > I am bcc'ing the author of Kombu in case he can help. > > > > alexis > > > > > > On Sun, Dec 25, 2011 at 2:11 AM, Chris Behrens > > <chris.behr...@rackspace.com> wrote: > >> Hi Roman, > >> > >> Are you using diablo or trunk (both of which use kombu by default and > should support reconnecting)? > >> > >> If so, can you 'egrep' for the following strings in logs and provide > the output? You can reply to me directly so we're not spamming this list. > >> > >> "Unable to connect to AMQP server" > >> "Connected to AMQP server" > >> "Re-established AMQP queues" > >> "AMQP server on.*unreachable" > >> 'Failed to consume" > >> "Failed to publish" > >> > >> Thanks, > >> > >> - Chris > >> > >> > >> On Dec 24, 2011, at 1:56 PM, Roman Sokolkov wrote: > >> > >>> Hello! > >>> > >>> If for some reason rabbitmq has been down some time or connection was > aborted, nova couldn't reconnect automatically and I need to restart all > nova services manually. > >>> > >>> Additionally after restarting rabbit, If I make some call to API. For > example, "nova floating-ip-list" only > >>> API reconnects automatically. Other nova services staying disconnected. > >>> > >>> Are there mechanisms to monitor establising of connections? And it > difficult to add them? > >>> > >>> -- > >>> Best regards, Roman Sokolkov > >>> _______________________________________________ > >>> Mailing list: https://launchpad.net/~openstack > >>> Post to : openstack@lists.launchpad.net > >>> Unsubscribe : https://launchpad.net/~openstack > >>> More help : https://help.launchpad.net/ListHelp > >> > >> > >> _______________________________________________ > >> Mailing list: https://launchpad.net/~openstack > >> Post to : openstack@lists.launchpad.net > >> Unsubscribe : https://launchpad.net/~openstack > >> More help : https://help.launchpad.net/ListHelp > > -- Regards, Roman Sokolkov -- Regards, Roman Sokolkov
_______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
