[Bug 1472712] Re: Using SSL with rabbitmq prevents communication between nova-compute and conductor after latest nova updates
Finally got to the bottom of this. The issue lies in python-amqp rather than python-oslo.messaging. The current trusty version of python-amqp (1.3.3) has a bug that is fixed in 1.4.4 (see http://amqp.readthedocs.org/en/latest/changelog.html#version-1-4-4). I tried backporting the Juno/Utopic version (1.4.5) for Trusty and everything works just fine now. I will shortly propose an SRU to get python-amqp fixed in Trusty. ** Package changed: python-oslo.messaging (Ubuntu) = python-amqp (Ubuntu) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to python-oslo.messaging in Ubuntu. https://bugs.launchpad.net/bugs/1472712 Title: Using SSL with rabbitmq prevents communication between nova-compute and conductor after latest nova updates To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1472712/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1472712] Re: Using SSL with rabbitmq prevents communication between nova-compute and conductor after latest nova updates
OK upon further investigation i have found some trace of a root cause. Oslo.messaging always uses a timeout of 1 second when polling queues and connections. This appears to be too small when using ssl and frequently results in SSLError/timeout which cause all threads to fail and reconnect and fail again repeatedly thus resulting in the number of connections rising fast and rpc not working, hence why compute and conductor are not able to communicate. I've played around with alternative timeout values and I get much better results even with a value of 2s instead of 1s. I'll propose an initial workaround patch shortly so we can get out of this bind for now but I think we'll ultimately need a more intelligent solution than what oslo.messaging support in this version. ** Changed in: python-oslo.messaging (Ubuntu) Status: Confirmed = In Progress ** Changed in: python-oslo.messaging (Ubuntu) Assignee: (unassigned) = Edward Hope-Morley (hopem) ** Changed in: python-oslo.messaging (Ubuntu) Importance: Undecided = High -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to python-oslo.messaging in Ubuntu. https://bugs.launchpad.net/bugs/1472712 Title: Using SSL with rabbitmq prevents communication between nova-compute and conductor after latest nova updates To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1472712/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1472712] Re: Using SSL with rabbitmq prevents communication between nova-compute and conductor after latest nova updates
** Tags added: sts -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to python-oslo.messaging in Ubuntu. https://bugs.launchpad.net/bugs/1472712 Title: Using SSL with rabbitmq prevents communication between nova-compute and conductor after latest nova updates To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1472712/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1472712] Re: Using SSL with rabbitmq prevents communication between nova-compute and conductor after latest nova updates
A bit more info from my end. I've been trying out different scenarios and it seems that this is constrained to Trusty Icehouse using python- oslo.messaging version 1.3.0-0ubuntu1.2 configured to connect to rabbitmq-server using ssl e.g. my nova.conf has: rabbit_userid = nova rabbit_virtual_host = openstack rabbit_password = gr6Mx2FJhC8NH3P4dBRGH8tYT39s6LLcMfJChKM6dtb3rpN5wfkRWVBcMLdhqp58 rabbit_host = 10.5.6.86 rabbit_use_ssl = True rabbit_port = 5671 kombu_ssl_ca_certs = /etc/nova/rabbit-client-ca.pem I've played around with reverting back to 1.3.0-0ubuntu1 (which does not appear to exhibit the issue) and re-adding patches one-by-one and have found that simply adding the patch for bug 1400268 causes the issue to occur. So, question is what is it about that patch that causes these issues? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to python-oslo.messaging in Ubuntu. https://bugs.launchpad.net/bugs/1472712 Title: Using SSL with rabbitmq prevents communication between nova-compute and conductor after latest nova updates To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1472712/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1472712] Re: Using SSL with rabbitmq prevents communication between nova-compute and conductor after latest nova updates
** Also affects: python-oslo.messaging (Ubuntu) Importance: Undecided Status: New ** Changed in: oslo.messaging Status: Confirmed = Invalid ** Changed in: nova Status: New = Invalid ** Changed in: python-oslo.messaging (Ubuntu) Status: New = Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to python-oslo.messaging in Ubuntu. https://bugs.launchpad.net/bugs/1472712 Title: Using SSL with rabbitmq prevents communication between nova-compute and conductor after latest nova updates To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1472712/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
