Re: [openstack-dev] devstack: Unable to restart rabbitmq-server

2014-04-01 Thread Victor Stinner
Hi,

Le lundi 17 mars 2014, 09:19:03 John Eckersberg a écrit :
 There are a couple of known bugs that can prevent rabbitmq-server from
 starting on F20.
 
 First one (same bug, two BZs) is related to SELinux and port probing:
 https://bugzilla.redhat.com/show_bug.cgi?id=1032595#c8
 https://bugzilla.redhat.com/show_bug.cgi?id=998682

I just got this issue and loose 1 hour because of it. It's not easy to 
identify the bug (hint: rotation of the audit log).

I'm ashamed. I had exactly the same issue 6 months ago, but I forgot it and 
how to fix it :-(

Until the issue is fixed in RabbitMQ or in the SELinux policy of RabbitMQ, I 
suggest this change in DevStack to warn users of the bug:

https://review.openstack.org/#/c/84482/

Note: I'm running DevStack on Fedora 20.

Victor

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] devstack: Unable to restart rabbitmq-server

2014-04-01 Thread John Eckersberg
Victor Stinner victor.stin...@enovance.com writes:
 Hi,

 Le lundi 17 mars 2014, 09:19:03 John Eckersberg a écrit :
 There are a couple of known bugs that can prevent rabbitmq-server from
 starting on F20.
 
 First one (same bug, two BZs) is related to SELinux and port probing:
 https://bugzilla.redhat.com/show_bug.cgi?id=1032595#c8
 https://bugzilla.redhat.com/show_bug.cgi?id=998682

 I just got this issue and loose 1 hour because of it. It's not easy to 
 identify the bug (hint: rotation of the audit log).

 I'm ashamed. I had exactly the same issue 6 months ago, but I forgot it and 
 how to fix it :-(

 Until the issue is fixed in RabbitMQ or in the SELinux policy of RabbitMQ, I 
 suggest this change in DevStack to warn users of the bug:

 https://review.openstack.org/#/c/84482/

 Note: I'm running DevStack on Fedora 20.

 Victor

An update has been submitted for Fedora 20 which corrects this issue:

https://admin.fedoraproject.org/updates/rabbitmq-server-3.1.5-4.fc20

Please give it a try and if it works for you, be sure to leave karma on
the update so it gets pushed to stable faster.

-John

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] devstack: Unable to restart rabbitmq-server

2014-04-01 Thread Victor Stinner
Le mardi 1 avril 2014, 12:17:39 John Eckersberg a écrit :
 An update has been submitted for Fedora 20 which corrects this issue:
 
 https://admin.fedoraproject.org/updates/rabbitmq-server-3.1.5-4.fc20
 
 Please give it a try and if it works for you, be sure to leave karma on
 the update so it gets pushed to stable faster.

Oh great! I tested it and it works with SELinux configured in Enforcing.

I tried to submit a +1 using bodhi but it requests a password, and I don't 
know which password it is.

Victor

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] devstack: Unable to restart rabbitmq-server

2014-03-18 Thread Deepak Shetty
My apologies! I mistakenly thought that SELinux was permissive, it wasn't!
Making it permissive, rabbitmq-server gets started everytime without any
ssues now.

thanx,
deepak








 openstack-dev@lists.openstack.org
  To: openstack-dev@lists.openstack.org

 Hi List,
 It been few hours and I tried everything from ensuring /etc/hosts,
 /etc/hostname etc (per google results) and rabbitmq-server still doesn't
 start. I am using latest devstack as of today on F20

 Below is the error I see

 [stack@devstack-vm ~]$ sudo systemctl status rabbitmq-server.service
 rabbitmq-server.service - RabbitMQ broker
Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service;
 disabled)
Active: failed (Result: timeout) since Mon 2014-03-17 07:20:47 UTC;
 14s ago
   Process: 30065 ExecStopPost=/usr/bin/rm /var/run/rabbitmq/pid
 (code=exited, status=0/SUCCESS)
   Process: 30027 ExecStop=/usr/lib/rabbitmq/bin/rabbitmqctl stop
 (code=exited, status=0/SUCCESS)
   Process: 29879 ExecStart=/usr/lib/rabbitmq/bin/rabbitmq-server
 (code=killed, signal=TERM)
  Main PID: 29879 (code=killed, signal=TERM)
CGroup: /system.slice/rabbitmq-server.service

 Mar 17 07:19:11 devstack-vm.localdomain rabbitmqctl[29880]: pid is 29879 ...
 Mar 17 07:19:12 devstack-vm.localdomain rabbitmq-server[29879]: RabbitMQ
 3.1.5. Copyright (C) 2007-2013 GoPivotal, Inc.
 Mar 17 07:19:12 devstack-vm.localdomain rabbitmq-server[29879]: ##
 ##  Licensed under the MPL.  See http://www.rabbitmq.com/
 Mar 17 07:19:12 devstack-vm.localdomain rabbitmq-server[29879]: ##  ##
 Mar 17 07:19:12 devstack-vm.localdomain rabbitmq-server[29879]:
 ##  Logs: /var/log/rabbitmq/rab...@devstack-vm.log
 Mar 17 07:19:12 devstack-vm.localdomain rabbitmq-server[29879]: ##
 ##/var/log/rabbitmq/rab...@devstack-vm-sasl.log
 Mar 17 07:19:12 devstack-vm.localdomain rabbitmq-server[29879]: ##
 Mar 17 07:20:41 devstack-vm.localdomain systemd[1]:
 rabbitmq-server.service operation timed out. Stopping.
 Mar 17 07:20:41 devstack-vm.localdomain rabbitmqctl[30027]: Stopping and
 halting node 'rabbit@devstack-vm' ...
 Mar 17 07:20:46 devstack-vm.localdomain rabbitmq-server[29879]: Starting
 broker... completed with 0 plugins.
 Mar 17 07:20:47 devstack-vm.localdomain rabbitmqctl[29880]: ...done.
 Mar 17 07:20:47 devstack-vm.localdomain rabbitmqctl[30027]: ...done.
 Mar 17 07:20:47 devstack-vm.localdomain systemd[1]: Failed to start
 RabbitMQ broker.
 Mar 17 07:20:47 devstack-vm.localdomain systemd[1]: Unit
 rabbitmq-server.service entered failed state.
 [stack@devstack-vm ~]$ sudo systemctl start rabbitmq-server.service


 Any additional things I can try to
 ===
 debug info ...

 [stack@devstack-vm ~]$ hostname -s
 devstack-vm

 [stack@devstack-vm ~]$ sudo cat /etc/hostname
 devstack-vm.localdomain

 [stack@devstack-vm ~]$  hostname
 devstack-vm.localdomain

 [stack@devstack-vm ~]$ sudo cat /etc/hosts
 127.0.0.1   localhost localhost.localdomain localhost4
 localhost4.localdomain4  devstack-vm
 ::1 localhost localhost.localdomain localhost6
 localhost6.localdomain6


 [stack@devstack-vm ~]$ sudo cat /etc/hostname
 devstack-vm.localdomain

 [stack@devstack-vm ~]$ hostname -s
 devstack-vm
 [stack@devstack-vm ~]$


 ___
 OpenStack-dev mailing 
 listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] devstack: Unable to restart rabbitmq-server

2014-03-17 Thread Kashyap Chamarthy
On Mon, Mar 17, 2014 at 02:27:29PM +0530, Deepak C Shetty wrote:
 Hi List,
 It been few hours and I tried everything from ensuring
 /etc/hosts, /etc/hostname etc (per google results) and
 rabbitmq-server still doesn't start. I am using latest devstack as
 of today on F20
 
 Below is the error I see
 
 [stack@devstack-vm ~]$ sudo systemctl status rabbitmq-server.service
 rabbitmq-server.service - RabbitMQ broker
Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service;
 disabled)
Active: failed (Result: timeout) since Mon 2014-03-17 07:20:47
 UTC; 14s ago
   Process: 30065 ExecStopPost=/usr/bin/rm /var/run/rabbitmq/pid
 (code=exited, status=0/SUCCESS)
   Process: 30027 ExecStop=/usr/lib/rabbitmq/bin/rabbitmqctl stop
 (code=exited, status=0/SUCCESS)
   Process: 29879 ExecStart=/usr/lib/rabbitmq/bin/rabbitmq-server
 (code=killed, signal=TERM)
  Main PID: 29879 (code=killed, signal=TERM)
CGroup: /system.slice/rabbitmq-server.service

I don't know much about RabbitMQ itself, but a few things that may give
you some debugging clues from systemd journal (look up the man page for
what the switches mean):

Show all logs of priority error:

$ journalctl -p err

Some variations:

$ journalctl /usr/sbin/rabbitmq-server
$ journalctl -u rabbitmq-server -l -p err
$ journalctl -u rabbitmq-server -l --since=yesterday -p err


-- 
/kashyap

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] devstack: Unable to restart rabbitmq-server

2014-03-17 Thread John Eckersberg
Deepak C Shetty deepa...@redhat.com writes:
 Hi List,
  It been few hours and I tried everything from ensuring /etc/hosts, 
 /etc/hostname etc (per google results) and rabbitmq-server still doesn't 
 start. I am using latest devstack as of today on F20

There are a couple of known bugs that can prevent rabbitmq-server from
starting on F20.

First one (same bug, two BZs) is related to SELinux and port probing:
https://bugzilla.redhat.com/show_bug.cgi?id=1032595#c8
https://bugzilla.redhat.com/show_bug.cgi?id=998682

Second one is a race condition in Erlang.  If you are repeatedly unable
to start rabbitmq-server, it's probably not this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1059913

I have a patched rabbitmq-server package which includes the fixes for
these two issues, if you'd like to try it and see if it helps your
issue.  And if it helps, please comment on the bug(s) to encourage the
maintainer to pull them into the package :)

http://jeckersb.fedorapeople.org/rabbitmq-server-3.1.5-3.fc20.noarch.rpm

Hope that helps,
John

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] devstack: Unable to restart rabbitmq-server

2014-03-17 Thread Solly Ross
I've also had devstack somehow end up starting qpid, then try to start rabbit 
on F20.  In this case
it seems sufficient to stop qpid then re-run devstack.  I haven't had time to 
track down the issue yet.

Best Regards,
Solly Ross

- Original Message -
From: John Eckersberg jecke...@redhat.com
To: Deepak C Shetty deepa...@redhat.com, openstack-dev@lists.openstack.org
Sent: Monday, March 17, 2014 9:19:03 AM
Subject: Re: [openstack-dev] devstack: Unable to restart rabbitmq-server

Deepak C Shetty deepa...@redhat.com writes:
 Hi List,
  It been few hours and I tried everything from ensuring /etc/hosts, 
 /etc/hostname etc (per google results) and rabbitmq-server still doesn't 
 start. I am using latest devstack as of today on F20

There are a couple of known bugs that can prevent rabbitmq-server from
starting on F20.

First one (same bug, two BZs) is related to SELinux and port probing:
https://bugzilla.redhat.com/show_bug.cgi?id=1032595#c8
https://bugzilla.redhat.com/show_bug.cgi?id=998682

Second one is a race condition in Erlang.  If you are repeatedly unable
to start rabbitmq-server, it's probably not this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1059913

I have a patched rabbitmq-server package which includes the fixes for
these two issues, if you'd like to try it and see if it helps your
issue.  And if it helps, please comment on the bug(s) to encourage the
maintainer to pull them into the package :)

http://jeckersb.fedorapeople.org/rabbitmq-server-3.1.5-3.fc20.noarch.rpm

Hope that helps,
John

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev