[Bug 1449056] Re: rabbitmq not ready after restart
Reviewed: https://review.openstack.org/186641 Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=6bc905c3488a93fa87776bcd0af7e362a90b082f Submitter: Jenkins Branch:master commit 6bc905c3488a93fa87776bcd0af7e362a90b082f Author: Tony Breeds t...@bakeyournoodle.com Date: Fri May 15 12:51:43 2015 +1000 Change the restart_rpc_backend loop to accomodate async rabbitmq Some distros have converted to systemd for starting RabbitMQ. This has resulted in: --- [Call Trace] ./stack.sh:904:restart_rpc_backend /home/stack/projects/openstack/openstack-dev/devstack/lib/rpc_backend:201:die [ERROR] /home/stack/projects/openstack/openstack-dev/devstack/lib/rpc_backend:201 Failed to set rabbitmq password Error on exit World dumping... see /opt/stack/logs/worlddump-2015-05-29-031618.txt for details --- Because 'restart_service rabbitmq-server' returns before the server is ready to accept connections. Alter the retry loop to only restart the rabbitmq-server every second time through the loop. Allowing time for the slow rabbit to start. Closes-Bug: 1449056 Change-Id: Ibb291c1ecfd109f9ed10b5f194933364985cc1ce ** Changed in: devstack Status: In Progress = Fix Released -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to rabbitmq-server in Ubuntu. https://bugs.launchpad.net/bugs/1449056 Title: rabbitmq not ready after restart To manage notifications about this bug go to: https://bugs.launchpad.net/devstack/+bug/1449056/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1449056] Re: rabbitmq not ready after restart
I tested http://launchpadlibrarian.net/208123057/rabbitmq- server_3.5.1-2_all.deb and it works fine for me. It would be great to see this backported to vivid. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to rabbitmq-server in Ubuntu. https://bugs.launchpad.net/bugs/1449056 Title: rabbitmq not ready after restart To manage notifications about this bug go to: https://bugs.launchpad.net/devstack/+bug/1449056/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1449056] Re: rabbitmq not ready after restart
@james-page: Thanks! Forgive me for not understanding the process but is it possible to get the fix in vivid as well? I assume there will need to be some QA process around that. I'm happy to verify any proposed builds. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to rabbitmq-server in Ubuntu. https://bugs.launchpad.net/bugs/1449056 Title: rabbitmq not ready after restart To manage notifications about this bug go to: https://bugs.launchpad.net/devstack/+bug/1449056/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1449056] Re: rabbitmq not ready after restart
This bug was fixed in the package rabbitmq-server - 3.5.1-2 --- rabbitmq-server (3.5.1-2) unstable; urgency=medium [ Tony Breeds ] * systemd: Ensure that rabbitmq has started before marking service as running (LP: #1449056). [ James Page ] * systemd: Drop use of /etc/default/rabbitmq-server. -- James Page james.p...@ubuntu.com Tue, 02 Jun 2015 11:40:59 +0100 ** Changed in: rabbitmq-server (Ubuntu) Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to rabbitmq-server in Ubuntu. https://bugs.launchpad.net/bugs/1449056 Title: rabbitmq not ready after restart To manage notifications about this bug go to: https://bugs.launchpad.net/devstack/+bug/1449056/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1449056] Re: rabbitmq not ready after restart
I've uploaded this to Debian unstable; it will autosync as soon as launchpad notices. ** Changed in: rabbitmq-server (Ubuntu) Status: New = Fix Committed ** Changed in: rabbitmq-server (Ubuntu) Importance: Undecided = Medium ** Changed in: rabbitmq-server (Ubuntu) Milestone: None = ubuntu-15.06 -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to rabbitmq-server in Ubuntu. https://bugs.launchpad.net/bugs/1449056 Title: rabbitmq not ready after restart To manage notifications about this bug go to: https://bugs.launchpad.net/devstack/+bug/1449056/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1449056] Re: rabbitmq not ready after restart
The attachment rabbitmq-sync-systemd.patch seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the patch flag from the attachment, remove the patch tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team. [This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.] ** Tags added: patch -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to rabbitmq-server in Ubuntu. https://bugs.launchpad.net/bugs/1449056 Title: rabbitmq not ready after restart To manage notifications about this bug go to: https://bugs.launchpad.net/devstack/+bug/1449056/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1449056] Re: rabbitmq not ready after restart
Trying to restate the problem a little morew clearly as after we've looked into this the 'Bug Description' is really more of a symptom. The current systemd sverice definition for rabbitmq-server completes before the rabbit process is actually available. This means that scripted environments that start rabbitmq-server and then try to talk to it fall subject to a race (see Bug Description). The best solution is to package erlang-sd_notify[1] and patch rabbit 3.5 with something like[2] and then change the systemd service type from 'simple' to 'notify'. That fix is probably better for debian unstable / wily. For testing/vivid I propose Adding a new script that will wait for the daemon to be available and calling that from from the service file. [1] https://github.com/lemenkov/erlang-sd_notify [2] http://pkgs.fedoraproject.org/cgit/rabbitmq-server.git/tree/rabbitmq-server-0001-Add-systemd-notify-support.patch?h=f21 -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to rabbitmq-server in Ubuntu. https://bugs.launchpad.net/bugs/1449056 Title: rabbitmq not ready after restart To manage notifications about this bug go to: https://bugs.launchpad.net/devstack/+bug/1449056/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1449056] Re: rabbitmq not ready after restart
** Changed in: devstack Assignee: Dr. Jens Rosenboom (j-rosenboom-j) = Tony Breeds (o-tony) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to rabbitmq-server in Ubuntu. https://bugs.launchpad.net/bugs/1449056 Title: rabbitmq not ready after restart To manage notifications about this bug go to: https://bugs.launchpad.net/devstack/+bug/1449056/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1449056] Re: rabbitmq not ready after restart
Update package to 1) include rabbitmq-server-wait: A helper script that uses the same environment file as rabbitmq-server and waits until the server is up. 2) Call the new file from the systemd service. This ensures that the server is available when systemctl start rabbitmq- server exits. ** Patch added: rabbitmq-sync-systemd.patch https://bugs.launchpad.net/devstack/+bug/1449056/+attachment/4407815/+files/rabbitmq-sync-systemd.patch -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to rabbitmq-server in Ubuntu. https://bugs.launchpad.net/bugs/1449056 Title: rabbitmq not ready after restart To manage notifications about this bug go to: https://bugs.launchpad.net/devstack/+bug/1449056/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1449056] Re: rabbitmq not ready after restart
Looks like Fedora hit similar (but not identical) issues. https://bugzilla.redhat.com/show_bug.cgi?id=1103524 The answer there was to patch systemd_notify support into rabbit. ** Bug watch added: Red Hat Bugzilla #1103524 https://bugzilla.redhat.com/show_bug.cgi?id=1103524 ** Also affects: rabbitmq-server (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to rabbitmq-server in Ubuntu. https://bugs.launchpad.net/bugs/1449056 Title: rabbitmq not ready after restart To manage notifications about this bug go to: https://bugs.launchpad.net/devstack/+bug/1449056/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1449056] Re: rabbitmq not ready after restart
** Changed in: devstack Assignee: Tony Breeds (o-tony) = Dr. Jens Rosenboom (j-rosenboom-j) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to rabbitmq-server in Ubuntu. https://bugs.launchpad.net/bugs/1449056 Title: rabbitmq not ready after restart To manage notifications about this bug go to: https://bugs.launchpad.net/devstack/+bug/1449056/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
