[Bug 985489] Re: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt
** Changed in: nova Status: Incomplete = Invalid -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/985489 Title: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/985489/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 985489] Re: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt
Moving to incomplete, I feel like we're waiting on the patch from @wangpan. And it's not clear this bug is moving forward without it. ** Tags removed: canonistack ** Changed in: nova Status: Confirmed = Incomplete ** Changed in: nova Importance: High = Medium ** Tags added: compute messaging -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/985489 Title: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/985489/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 985489] Re: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt
@wangpan I'm revisiting this bug, do you have a link to the change that you proposed? Even if you didn't think it was adequate I'd like to see your approach. -Mike -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/985489 Title: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/985489/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 985489] Re: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt
We just saw this with havana on precise. ii libvirt0 1.1.1-0ubuntu8.5~cloud0 library for interfacing with different virtualization systems ii nova-compute-kvm 1:2013.2.2-0ubuntu1~cloud0 OpenStack Compute - compute node (KVM) As you can see we're using the Ubuntu Cloud Archive for these havana packages. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/985489 Title: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/985489/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 985489] Re: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt
In addition, this seemed to hit three compute nodes (the full set in a test deployment) in relatively quick succession after a few days of normal operation. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/985489 Title: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/985489/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 985489] Re: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt
Hi all, after we updating our qemu to qemu-kvm 1.1.2+dfsg-6 under debian wheezy 7.0(linux kernel is 3.2.39), this question has disappeared for more than three months, someone else can try this way if you encountered this issue. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/985489 Title: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/985489/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 985489] Re: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt
@wangpan (hzwangpan) two things. 1 I've forgot the URL of multi-thread/process lock of libvirt (I faced this problem in essex, one year more time.). My solution is compiling libvirt from source code which is 1.XXX version. 2 Seems different with, I'll try my best to submit a patch. I hope it can solve you problem. This patch maybe a little long. I solve it in essex, I have to move it to grizzly. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/985489 Title: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/985489/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 985489] Re: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt
Hi Ji you, two things: 1. Can you tell me more info about the multi-process/multi-threads lock bugs of libvirt? such as bug report URL or something like this. 2. In our env, the 'virsh' command is running OK and there is only one libvirtd process when this bug occurs, this may be a difference with yours. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/985489 Title: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/985489/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 985489] Re: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt
Description of this problem I have met this problem before. This bug is mainly caused by libvirt openstack-compute driver for libvirt. When you meet this problem. You may type command. $ virsh list --all If this command get stucked, you may face this problem. Another interesting appearance is that, if you type $ ps aux | grep libvirtd You may find two libvirtd service is running. Such as, .. /usr/sbin/libvirtd -d ../usr/sbin/libvirtd -d You kill one process, then libvirt will be ok. But sometimes later, the bug will come out again. Reasons of this problem Why this bug comes out? 1 This bug mainly caused by libvirt. The older verion of libvirt has multi-process/multi-threads lock bugs. Sometimes, when multi-processes access libvirt/libvirtd services by libvirt-api, this error will happen. Especially, if you access libvirt from differe process or threads, then this bug will come out. All accesses of libvirt will get stucked. 2 The driver of libvirt in OpenStack, access libvirt/libvirtd service too frequently. Especially, when you have a lot of vms running on libvirt. Solutions for this bug: 1 You have to update you libvirt/libvirtd services. My advice is your libvirt version should 0.9.13. 2 We may have some methods to get lower frequent of these accesses. I think I can fix this Bug. ^_^. I have fix it in my env of openstack. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/985489 Title: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/985489/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 985489] Re: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt
** Changed in: nova Status: Confirmed = In Progress ** Changed in: nova Assignee: (unassigned) = wangpan (hzwangpan) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/985489 Title: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/985489/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 985489] Re: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt
** Changed in: nova Assignee: wangpan (hzwangpan) = (unassigned) ** Changed in: nova Status: In Progress = Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/985489 Title: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/985489/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 985489] Re: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt
** Changed in: nova Status: Confirmed = In Progress ** Changed in: nova Assignee: Michael H Wilson (geekinutah) = wangpan (hzwangpan) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/985489 Title: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/985489/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 985489] Re: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt
** Changed in: nova Assignee: wangpan (hzwangpan) = (unassigned) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/985489 Title: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/985489/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 985489] Re: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt
Sorry for Michael H Wilson, I commited a patch relative to this bug, so OpenStack Hudson automatically assign it to me, but the patch can't reslove this bug eventually, and I can't reassign it to you now, you may need to get it again. ** Changed in: nova Status: In Progress = Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/985489 Title: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/985489/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 985489] Re: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt
** Changed in: nova Assignee: (unassigned) = Michael H Wilson (geekinutah) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/985489 Title: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/985489/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 985489] Re: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt
I find this issue in my stable folsom, but unortunately it occurs randomly, and my libvirt version is 0.9.12-5(debian wheezy). log in the nova-compute.log: 2013-02-26 02:48:45 DEBUG nova.manager [-] Running periodic task ComputeManager.update_available_resource from (pid=35879) periodic_tasks /usr/local/lib/python2.7/dist-packages/nova/manager.py:200 2013-02-26 02:48:45 DEBUG nova.utils [-] Got semaphore compute_resources for method update_available_resource... from (pid=35879) inner /usr/local/lib/python2.7/dist-packages/nova/utils.py:803 ==and then the periodic task stopped until 2013-02-26 09:37:48 ERROR nova.manager [-] Error during ComputeManager.update_available_resource: Domain not found: no domain with matching id 1519 .. 2013-02-26 09:37:48 DEBUG nova.manager [-] Running periodic task ComputeManager._report_driver_status from (pid=35879) periodic_tasks /usr/local/lib/python2.7/dist-packages/nova/manager.py:200 2013-02-26 09:37:48 INFO nova.compute.manager [-] Updating host status 2013-02-26 09:37:48 DEBUG nova.virt.libvirt.driver [-] Updating host stats from (pid=35879) update_status /usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py:3602 ==and then the periodic task stopped again until I restart nova-compute service At the second time when this issue occurs, I try to restart libvirtd and don't restart nova-compute, the nova-compute continues to running after I restart libvirtd! so I believe this issue is resulted from libvirtd. We may need to add a timeout when nova-compute call libvirt apis. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/985489 Title: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/985489/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 985489] Re: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt
** Summary changed: - nova-compute stops processing compute.$HOSTNAME occasionally + nova-compute stops processing compute.$HOSTNAME occasionally on libvirt -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/985489 Title: nova-compute stops processing compute.$HOSTNAME occasionally on libvirt To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/985489/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
