Re: [openstack-dev] [Nova] nova-compute deadlock

2014-08-04 Thread Qin Zhao
Dear stackers, FYI. Eventually I report this problem to libguestfs. A workaround has been included into libguestfs code to fix it. Thanks for your supporting! https://bugzilla.redhat.com/show_bug.cgi?id=1123007 On Sat, Jun 7, 2014 at 3:27 AM, Qin Zhao chaoc...@gmail.com wrote: Yuriy, And I

Re: [openstack-dev] [Nova] nova-compute deadlock

2014-06-07 Thread Richard W.M. Jones
On Sat, May 31, 2014 at 01:25:04AM +0800, Qin Zhao wrote: Hi all, When I run Icehouse code, I encountered a strange problem. The nova-compute service becomes stuck, when I boot instances. I report this bug in https://bugs.launchpad.net/nova/+bug/1313477. After thinking several days, I

Re: [openstack-dev] [Nova] nova-compute deadlock

2014-06-06 Thread Qin Zhao
Yuriy, And I think if we use proxy object of multiprocessing, the green thread will not switch during we call libguestfs. Is that correct? On Fri, Jun 6, 2014 at 2:44 AM, Qin Zhao chaoc...@gmail.com wrote: Hi Yuriy, I read multiprocessing source code just now. Now I feel it may not solve

Re: [openstack-dev] [Nova] nova-compute deadlock

2014-06-05 Thread laserjetyang
Will this patch of Python fix your problem? *http://bugs.python.org/issue7213 http://bugs.python.org/issue7213* On Wed, Jun 4, 2014 at 10:41 PM, Qin Zhao chaoc...@gmail.com wrote: Hi Zhu Zhu, Thank you for reading my diagram! I need to clarify that this problem does not occur during

Re: [openstack-dev] [Nova] nova-compute deadlock

2014-06-05 Thread Yuriy Taraday
This behavior of os.pipe() has changed in Python 3.x so it won't be an issue on newer Python (if only it was accessible for us). From the looks of it you can mitigate the problem by running libguestfs requests in a separate process (multiprocessing.managers comes to mind). This way the only

Re: [openstack-dev] [Nova] nova-compute deadlock

2014-06-05 Thread Qin Zhao
Hi, Thanks for reading my bug! I think this patch can not fix this problem now, because pipe2() requires Python 3.3. On Thu, Jun 5, 2014 at 6:17 PM, laserjetyang laserjety...@gmail.com wrote: Will this patch of Python fix your problem? *http://bugs.python.org/issue7213

Re: [openstack-dev] [Nova] nova-compute deadlock

2014-06-05 Thread Qin Zhao
Hi Yuriy, Thanks for reading my bug! You are right. Python 3.3 or 3.4 should not have this issue, since they have can secure the file descriptor. Before OpenStack move to Python 3, we may still need a solution. Calling libguestfs in a separate process seems to be a way. This way, Nova code can

Re: [openstack-dev] [Nova] nova-compute deadlock

2014-06-05 Thread Yuriy Taraday
Please take a look at https://docs.python.org/2.7/library/multiprocessing.html#managers - everything is already implemented there. All you need is to start one manager that would serve all your requests to libguestfs. The implementation in stdlib will provide you with all exceptions and return

Re: [openstack-dev] [Nova] nova-compute deadlock

2014-06-05 Thread Qin Zhao
Hi Yuriy, I read multiprocessing source code just now. Now I feel it may not solve this problem very easily. For example, let us assume that we will use the proxy object in Manager's process to call libguestfs. In manager.py, I see it needs to create a pipe, before fork the child process. The

Re: [openstack-dev] [Nova] nova-compute deadlock

2014-06-03 Thread Zhu Zhu
Hi Qin Zhao, Thanks for raising this issue and analysis. According to the issue description and happen scenario(https://docs.google.com/drawings/d/1pItX9urLd6fmjws3BVovXQvRg_qMdTHS-0JhYfSkkVc/pub?w=960h=720), if that's the case, concurrent mutiple KVM spawn instances(with both config drive