It seems that I as chasing down the wrong rabbit hole… Our controller node runs on a VM and has had sever hiccups the last few days (pausing for up to 10 seconds from time to time). We have moved the VM to another host, the pauses disappeared, and I can delete as many VMs at a time as I like…
cheers jc -- SWITCH Jens-Christian Fischer, Peta Solutions Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland phone +41 44 268 15 15, direct +41 44 268 15 71 [email protected] http://www.switch.ch http://www.switch.ch/stories On 11.07.2014, at 15:45, Brebner, Gavin <[email protected]> wrote: > > we change /etc/security/limits.conf (actually add an entry to > /etc/security/limits.conf.d) for the rabbit user. > > We just do that for the non-compute nodes. > > Gavin > > > From: Jens-Christian Fischer [mailto:[email protected]] > Sent: Friday, July 11, 2014 3:41 PM > To: Brebner, Gavin > Cc: [email protected] > Subject: Re: [Openstack] Unable to delete instances > > There is just one queue that is building up: notifications.info - there are > no consumers for it. > > I have upped the ulimit for the rabbitmq server (by changing > /etc/default/rabbitmq-server) but that hasn’t made any difference. > > Are you also upping the limits on the compute nodes? > > cheers > jc > > -- > SWITCH > Jens-Christian Fischer, Peta Solutions > Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland > phone +41 44 268 15 15, direct +41 44 268 15 71 > [email protected] > http://www.switch.ch > > http://www.switch.ch/stories > > On 11.07.2014, at 12:50, Brebner, Gavin <[email protected]> wrote: > > > Delete issues in my experience are typically linked to messaging getting > stuck (or dead hosts). I would have a look at the > RMQ queue lengths and see if any are building up. > > Have you adjusted your open file descriptor limits ? The default Linux limit > (1024) is NOT enough in our experience for any reasonable > sized system. We’ve seen that cause messaging outages. > > Gavin > > From: Jens-Christian Fischer [mailto:[email protected]] > Sent: Friday, July 11, 2014 12:32 PM > To: Thomas Bernard > Cc: [email protected] > Subject: Re: [Openstack] Unable to delete instances > > Did you ever come to the bottom of this? > > We are seeing the same thing with a brand new Icehouse / Ceph installation > (using the stock distributions) > > Restarting nova-compute on the compute nodes usually makes the instances go > away (or resetting the state, and then restarting nova-compute) > > We see this problem intermittent but frequent enough to be really annoying. > > The log files have so far not shown anything conclusive: I got an “Unknown > auth strategy” error, but according to > https://bugs.launchpad.net/python-neutronclient/+bug/1297309 this is a red > herring. > > Anyone else running into this problem or having solved it? > > cheers > Jens-Christian > -- > SWITCH > Jens-Christian Fischer, Peta Solutions > Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland > phone +41 44 268 15 15, direct +41 44 268 15 71 > [email protected] > http://www.switch.ch > > http://www.switch.ch/stories > > On 22.05.2014, at 16:19, Thomas Bernard <[email protected]> wrote: > > > > Hi, > > I just finished my upgrade to icehouse, everything works just fine but... i > can't delete my instances! > > When i delete instance (from cli or from horizon) : > > * The VM goes to "deleting"state > * Vm are destroy from KVM > * But nova list & horizon continue to display the "deleting" state > > Some time after, the Vm state goes to "ERROR" and in nova-compute.log i can > see : "Unauthorized: Unknown auth strategy\n" (from neutronclient) > > Any idea? > > Thx! > > > > > > _______________________________________________ > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : [email protected] > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : [email protected] Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
