Re: [openstack-dev] devstack: Unable to restart rabbitmq-server
Hi, Le lundi 17 mars 2014, 09:19:03 John Eckersberg a écrit : There are a couple of known bugs that can prevent rabbitmq-server from starting on F20. First one (same bug, two BZs) is related to SELinux and port probing: https://bugzilla.redhat.com/show_bug.cgi?id=1032595#c8 https://bugzilla.redhat.com/show_bug.cgi?id=998682 I just got this issue and loose 1 hour because of it. It's not easy to identify the bug (hint: rotation of the audit log). I'm ashamed. I had exactly the same issue 6 months ago, but I forgot it and how to fix it :-( Until the issue is fixed in RabbitMQ or in the SELinux policy of RabbitMQ, I suggest this change in DevStack to warn users of the bug: https://review.openstack.org/#/c/84482/ Note: I'm running DevStack on Fedora 20. Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] devstack: Unable to restart rabbitmq-server
Victor Stinner victor.stin...@enovance.com writes: Hi, Le lundi 17 mars 2014, 09:19:03 John Eckersberg a écrit : There are a couple of known bugs that can prevent rabbitmq-server from starting on F20. First one (same bug, two BZs) is related to SELinux and port probing: https://bugzilla.redhat.com/show_bug.cgi?id=1032595#c8 https://bugzilla.redhat.com/show_bug.cgi?id=998682 I just got this issue and loose 1 hour because of it. It's not easy to identify the bug (hint: rotation of the audit log). I'm ashamed. I had exactly the same issue 6 months ago, but I forgot it and how to fix it :-( Until the issue is fixed in RabbitMQ or in the SELinux policy of RabbitMQ, I suggest this change in DevStack to warn users of the bug: https://review.openstack.org/#/c/84482/ Note: I'm running DevStack on Fedora 20. Victor An update has been submitted for Fedora 20 which corrects this issue: https://admin.fedoraproject.org/updates/rabbitmq-server-3.1.5-4.fc20 Please give it a try and if it works for you, be sure to leave karma on the update so it gets pushed to stable faster. -John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] devstack: Unable to restart rabbitmq-server
Le mardi 1 avril 2014, 12:17:39 John Eckersberg a écrit : An update has been submitted for Fedora 20 which corrects this issue: https://admin.fedoraproject.org/updates/rabbitmq-server-3.1.5-4.fc20 Please give it a try and if it works for you, be sure to leave karma on the update so it gets pushed to stable faster. Oh great! I tested it and it works with SELinux configured in Enforcing. I tried to submit a +1 using bodhi but it requests a password, and I don't know which password it is. Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] devstack: Unable to restart rabbitmq-server
My apologies! I mistakenly thought that SELinux was permissive, it wasn't! Making it permissive, rabbitmq-server gets started everytime without any ssues now. thanx, deepak openstack-dev@lists.openstack.org To: openstack-dev@lists.openstack.org Hi List, It been few hours and I tried everything from ensuring /etc/hosts, /etc/hostname etc (per google results) and rabbitmq-server still doesn't start. I am using latest devstack as of today on F20 Below is the error I see [stack@devstack-vm ~]$ sudo systemctl status rabbitmq-server.service rabbitmq-server.service - RabbitMQ broker Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; disabled) Active: failed (Result: timeout) since Mon 2014-03-17 07:20:47 UTC; 14s ago Process: 30065 ExecStopPost=/usr/bin/rm /var/run/rabbitmq/pid (code=exited, status=0/SUCCESS) Process: 30027 ExecStop=/usr/lib/rabbitmq/bin/rabbitmqctl stop (code=exited, status=0/SUCCESS) Process: 29879 ExecStart=/usr/lib/rabbitmq/bin/rabbitmq-server (code=killed, signal=TERM) Main PID: 29879 (code=killed, signal=TERM) CGroup: /system.slice/rabbitmq-server.service Mar 17 07:19:11 devstack-vm.localdomain rabbitmqctl[29880]: pid is 29879 ... Mar 17 07:19:12 devstack-vm.localdomain rabbitmq-server[29879]: RabbitMQ 3.1.5. Copyright (C) 2007-2013 GoPivotal, Inc. Mar 17 07:19:12 devstack-vm.localdomain rabbitmq-server[29879]: ## ## Licensed under the MPL. See http://www.rabbitmq.com/ Mar 17 07:19:12 devstack-vm.localdomain rabbitmq-server[29879]: ## ## Mar 17 07:19:12 devstack-vm.localdomain rabbitmq-server[29879]: ## Logs: /var/log/rabbitmq/rab...@devstack-vm.log Mar 17 07:19:12 devstack-vm.localdomain rabbitmq-server[29879]: ## ##/var/log/rabbitmq/rab...@devstack-vm-sasl.log Mar 17 07:19:12 devstack-vm.localdomain rabbitmq-server[29879]: ## Mar 17 07:20:41 devstack-vm.localdomain systemd[1]: rabbitmq-server.service operation timed out. Stopping. Mar 17 07:20:41 devstack-vm.localdomain rabbitmqctl[30027]: Stopping and halting node 'rabbit@devstack-vm' ... Mar 17 07:20:46 devstack-vm.localdomain rabbitmq-server[29879]: Starting broker... completed with 0 plugins. Mar 17 07:20:47 devstack-vm.localdomain rabbitmqctl[29880]: ...done. Mar 17 07:20:47 devstack-vm.localdomain rabbitmqctl[30027]: ...done. Mar 17 07:20:47 devstack-vm.localdomain systemd[1]: Failed to start RabbitMQ broker. Mar 17 07:20:47 devstack-vm.localdomain systemd[1]: Unit rabbitmq-server.service entered failed state. [stack@devstack-vm ~]$ sudo systemctl start rabbitmq-server.service Any additional things I can try to === debug info ... [stack@devstack-vm ~]$ hostname -s devstack-vm [stack@devstack-vm ~]$ sudo cat /etc/hostname devstack-vm.localdomain [stack@devstack-vm ~]$ hostname devstack-vm.localdomain [stack@devstack-vm ~]$ sudo cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 devstack-vm ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 [stack@devstack-vm ~]$ sudo cat /etc/hostname devstack-vm.localdomain [stack@devstack-vm ~]$ hostname -s devstack-vm [stack@devstack-vm ~]$ ___ OpenStack-dev mailing listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] devstack: Unable to restart rabbitmq-server
On Mon, Mar 17, 2014 at 02:27:29PM +0530, Deepak C Shetty wrote: Hi List, It been few hours and I tried everything from ensuring /etc/hosts, /etc/hostname etc (per google results) and rabbitmq-server still doesn't start. I am using latest devstack as of today on F20 Below is the error I see [stack@devstack-vm ~]$ sudo systemctl status rabbitmq-server.service rabbitmq-server.service - RabbitMQ broker Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; disabled) Active: failed (Result: timeout) since Mon 2014-03-17 07:20:47 UTC; 14s ago Process: 30065 ExecStopPost=/usr/bin/rm /var/run/rabbitmq/pid (code=exited, status=0/SUCCESS) Process: 30027 ExecStop=/usr/lib/rabbitmq/bin/rabbitmqctl stop (code=exited, status=0/SUCCESS) Process: 29879 ExecStart=/usr/lib/rabbitmq/bin/rabbitmq-server (code=killed, signal=TERM) Main PID: 29879 (code=killed, signal=TERM) CGroup: /system.slice/rabbitmq-server.service I don't know much about RabbitMQ itself, but a few things that may give you some debugging clues from systemd journal (look up the man page for what the switches mean): Show all logs of priority error: $ journalctl -p err Some variations: $ journalctl /usr/sbin/rabbitmq-server $ journalctl -u rabbitmq-server -l -p err $ journalctl -u rabbitmq-server -l --since=yesterday -p err -- /kashyap ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] devstack: Unable to restart rabbitmq-server
Deepak C Shetty deepa...@redhat.com writes: Hi List, It been few hours and I tried everything from ensuring /etc/hosts, /etc/hostname etc (per google results) and rabbitmq-server still doesn't start. I am using latest devstack as of today on F20 There are a couple of known bugs that can prevent rabbitmq-server from starting on F20. First one (same bug, two BZs) is related to SELinux and port probing: https://bugzilla.redhat.com/show_bug.cgi?id=1032595#c8 https://bugzilla.redhat.com/show_bug.cgi?id=998682 Second one is a race condition in Erlang. If you are repeatedly unable to start rabbitmq-server, it's probably not this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1059913 I have a patched rabbitmq-server package which includes the fixes for these two issues, if you'd like to try it and see if it helps your issue. And if it helps, please comment on the bug(s) to encourage the maintainer to pull them into the package :) http://jeckersb.fedorapeople.org/rabbitmq-server-3.1.5-3.fc20.noarch.rpm Hope that helps, John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] devstack: Unable to restart rabbitmq-server
I've also had devstack somehow end up starting qpid, then try to start rabbit on F20. In this case it seems sufficient to stop qpid then re-run devstack. I haven't had time to track down the issue yet. Best Regards, Solly Ross - Original Message - From: John Eckersberg jecke...@redhat.com To: Deepak C Shetty deepa...@redhat.com, openstack-dev@lists.openstack.org Sent: Monday, March 17, 2014 9:19:03 AM Subject: Re: [openstack-dev] devstack: Unable to restart rabbitmq-server Deepak C Shetty deepa...@redhat.com writes: Hi List, It been few hours and I tried everything from ensuring /etc/hosts, /etc/hostname etc (per google results) and rabbitmq-server still doesn't start. I am using latest devstack as of today on F20 There are a couple of known bugs that can prevent rabbitmq-server from starting on F20. First one (same bug, two BZs) is related to SELinux and port probing: https://bugzilla.redhat.com/show_bug.cgi?id=1032595#c8 https://bugzilla.redhat.com/show_bug.cgi?id=998682 Second one is a race condition in Erlang. If you are repeatedly unable to start rabbitmq-server, it's probably not this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1059913 I have a patched rabbitmq-server package which includes the fixes for these two issues, if you'd like to try it and see if it helps your issue. And if it helps, please comment on the bug(s) to encourage the maintainer to pull them into the package :) http://jeckersb.fedorapeople.org/rabbitmq-server-3.1.5-3.fc20.noarch.rpm Hope that helps, John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev