Hi Sam, We upgraded to Icehouse over the weekend and had some issues with Rabbit.
The number of RabbitMQ connections went from ~70 to ~380 post-upgrade and we had reports of users being unable to use the metadata service and that instances took longer to boot as they waited for metadata. I noticed that there were several more nova-api-metadata services running on our compute nodes than before (nova-network multi-host environment). I found the config option "metadata_workers" which is set to the number of cpus by default. I changed it to "2" and restarted everything across the board. Our total number of Rabbit connections are now sitting at ~150. I'm not sure if this helps at all. I wasn't running Icehouse long enough to see if any of the reply_xxxxxx queues filled up with messages or not. The times that I checked showed that this was not happening (and is still not happening). Joe On Mon, Aug 25, 2014 at 5:17 PM, Sam Morrison <[email protected]> wrote: > Hi, > > Since upgrading to Icehouse we have seen increased issues with messaging > relating to RabbitMQ. > > 1. We often get reply_xxxxxx queues starting to fill up with unacked > messages. To fix this we need to restart the offending service. Usually > nova-api or nova-compute. > > 2. If you kill a node so as to force an *ungraceful* disconnect of rabbit > the connection “object?” still sticks around in rabbit. Starting the > service again means there are now 2 consumers. The new one and the phantom > old one. This then leads to messages piling up in the unacked queue. This > feels like a rabbit bug to me but just thought I’d mention it here too. > > > We have have a setup that includes icehouse computes and havana computes > in the same cloud and we only see this on the icehouse computes. This is > using Trusty and RabbitMQ 3.3.4 > > > Has anyone seen anything like this too? > > Thanks, > Sam > > > _______________________________________________ > OpenStack-operators mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : [email protected] Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
