Re: [vdsm] thread pool implementation
On 03/25/2014 03:15 PM, Francesco Romani wrote: - Original Message - From: Saggi Mizrahi smizr...@redhat.com To: Francesco Romani from...@redhat.com Cc: vdsm-devel vdsm-devel@lists.fedorahosted.org, Yaniv Bronheim ybron...@redhat.com Sent: Tuesday, March 25, 2014 1:45:22 PM Subject: Re: thread pool implementation The thing that worries me the most is stuck threads. I hate them! Could we move to multiple libvirt connections scheme? Where if a call takes too long we just close the connection. I know that the call is still running in libvirt but then it's their problem and not my problem. That way the thread pool doesn't need to handle this use case making it much simpler. Because apart from the problem of libvirt calls getting stuck we just need a run of the mill threadpool solution. It is an interesting point. I'll investigate the multiple libvirt connection Idea. we support multiple libvirt connections with libvirtconnection.py. however, vdsm uses only one instance during its run. might be that this was the initial thought of the libvirtconnection get function. but still, if the request gets stuck, how multiply connections help us? we still need threads to avoid freezing other flows. afaiu you plan to cover each libvirt call in a thread from the pool? we can still use one connection for that. if each thread will establish new session to libvirt, it might cause very long delay (depends on the load) Bests, -- Yaniv Bronhaim. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Share some topic on meeting
On 03/11/2014 04:57 AM, Doron Fediuck wrote: - Original Message - From: Chuan Liao (Jason Liao, HPservers-Core-OE-PSC) chuan.l...@hp.com To: aba...@redhat.com Cc: dfedi...@redhat.com, Chegu Vinod chegu_vi...@hp.com, Shang-Chun Liang (David Liang, HPservers-Core-OE-PSC) shangchun.li...@hp.com Sent: Monday, March 10, 2014 2:48:22 PM Subject: Share some topic on meeting Hi I am Jason Liao from HP. One of NUMA and guest NUMA feature developer. In order to let the feature work with others, we need to add per CPU usage info sampling data into VDSM. The workflow is mostly like the total CPU usage info sampling method, and have more data in getstats function. I am not sure everybody agree with these change. So I want to discuss this on the meeting. Best Regards, Jason Liao Hey Jason, Thanks for sharing and initiating the work. im forwarding the mail to vdsm-devel which is the list for that question we have vdsm talk biweekly on monday's and it still happens and beat same time as always, although this week it was canceled. hope to hear you there in two weeks. adding more fields to getVdsStats requires also engine side of course in db and mapping vdsm response. but first, can you elaborate more about those fields you want to add? how often those parameter get changed? Regards, Yaniv Bronhaim Added some of the VDSM maintainers, as Ayal no longer handles it. -- Yaniv Bronhaim. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] [PATCH] monkey_patch: Fail if it is too late to monkey-patch
On 03/08/2014 03:22 PM, Nir Soffer wrote: - Original Message - From: Nir Soffer nsof...@redhat.com To: vdsm-de...@fedorahosted.org Sent: Saturday, March 8, 2014 3:15:34 PM Subject: [vdsm] [PATCH] monkey_patch: Fail if it is too late to monkey-patch Please review this patch for pthreading: https://github.com/nirs/pthreading/commit/406abe5c0df1c09e509687c99f637799387fbad2 The patch above was replaced with this one: https://github.com/nirs/pthreading/commit/490e837e0afed37823803d9e180a5d3eb17cb82c Nir ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel +1 thanks! -- Yaniv Bronhaim. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
[vdsm] Fwd: Fwd: suggested patch for python-pthreading
back to the topic I would like to continue pushing the fix in. Nir, iirc you suggested a patch to validate that pthreading.monkey_patch() is being called before using the native thread and pthreading module, right? any comments about that part? Original Message Subject: Fwd: [vdsm] suggested patch for python-pthreading Date: Tue, 18 Feb 2014 09:32:59 -0500 (EST) From: Nir Soffer nsof...@redhat.com To: Yaniv Bronheim ybron...@redhat.com - Forwarded Message - From: Yaniv Bronheim ybron...@redhat.com To: Nir Soffer nsof...@redhat.com Cc: Dan Kenigsberg dan...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Wednesday, February 5, 2014 6:41:28 PM Subject: Re: [vdsm] suggested patch for python-pthreading - Original Message - From: Nir Soffer nsof...@redhat.com To: Dan Kenigsberg dan...@redhat.com Cc: Yaniv Bronheim ybron...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Tuesday, February 4, 2014 4:39:55 PM Subject: Re: [vdsm] suggested patch for python-pthreading - Original Message - From: Dan Kenigsberg dan...@redhat.com To: Nir Soffer nsof...@redhat.com Cc: Yaniv Bronheim ybron...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Tuesday, February 4, 2014 3:51:02 PM Subject: Re: [vdsm] suggested patch for python-pthreading On Tue, Feb 04, 2014 at 05:14:51AM -0500, Nir Soffer wrote: - Original Message - From: Yaniv Bronheim ybron...@redhat.com To: Nir Soffer nsof...@redhat.com Cc: VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Tuesday, February 4, 2014 11:53:52 AM Subject: Re: [vdsm] suggested patch for python-pthreading - Original Message - From: Nir Soffer nsof...@redhat.com To: Yaniv Bronheim ybron...@redhat.com Cc: VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Tuesday, February 4, 2014 11:39:56 AM Subject: Re: [vdsm] suggested patch for python-pthreading - Original Message - From: Yaniv Bronheim ybron...@redhat.com To: VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Tuesday, February 4, 2014 11:04:37 AM Subject: [vdsm] suggested patch for python-pthreading according to coredumps we found in the scope of the bug [1] we opened [2] that suggested to override python's implementation of thread.allocate_lock in each coredump we saw few threads stuck with the bt: #16 0x7fcb69288c93 in PyEval_CallObjectWithKeywords (func=0x2527820, arg=0x7fcb6972f050, kw=value optimized out) at Python/ceval.c:3663 #17 0x7fcb692ba7ba in t_bootstrap (boot_raw=0x250a820) at Modules/threadmodule.c:428 #18 0x7fcb68fa3851 in start_thread (arg=0x7fcb1bfff700) at pthread_create.c:301 #19 0x7fcb6866694d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 in pystack the threads were stuck in /usr/lib64/python2.6/threading.py (513): __bootstrap_inner in bootstrap_inner we use thread.allocate_lock which python-pthreading does not override. we suggest the following commit: From 9d89e9be1a379b3d93b23dd54a381b9ca0973ebc Mon Sep 17 00:00:00 2001 From: Yaniv Bronhaim ybron...@redhat.com Date: Mon, 3 Feb 2014 19:24:30 +0200 Subject: [PATCH] Mocking thread.allocate_lock with Lock imp Signed-off-by: Yaniv Bronhaim ybron...@redhat.com --- pthreading.py | 4 1 file changed, 4 insertions(+) diff --git a/pthreading.py b/pthreading.py index 916ca7f..96df42c 100644 --- a/pthreading.py +++ b/pthreading.py @@ -132,6 +132,10 @@ def monkey_patch(): Thus, Queue and SocketServer can easily enjoy them. +import thread + +thread.allocate_lock = Lock + import threading threading.Condition = Condition -- 1.8.3.1 [1] https://bugzilla.redhat.com/show_bug.cgi?id=1022036 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1060749 Replacing allocate_lock in thread is correct. However, since threading copies thread.allocate_lock, and you don't control import order, you have to monkeypatch threading.allocate_lock as well. The full should be: import thread thread.allocate_lock = Lock import threading threading.allocate_lock = Lock threading.Lock = Lock Nir thanks Nir for the reply and review I guess you meant threading._allocate_lock, but we monkey patch it in an order so we do control the import order.. not sure its necessary Correct, threading._allocate_lock. And we also have to
Re: [vdsm] A question about the SPM operation permission in VDSM
On 01/31/2013 03:55 PM, Federico Simoncelli wrote:
- Original Message -
From: Shu Ming shum...@linux.vnet.ibm.com
To: VDSM Project Development vdsm-devel@lists.fedorahosted.org
Sent: Tuesday, October 30, 2012 7:22:18 AM
Subject: [vdsm] A question about the SPM operation permission in VDSM
Hi,
In the VDSM code about some SPM operations like HSM.deleteImage(), It
is found that VDSM doesn't check if the operation will be launched
on a SPM host or not. It only checks if the storage pool is already
acquired by one SPM host, but not necessary the same host as the SPM
operation is delivered to. The code is like this:
HSM.deleteImage()
{
...
HSM._spmSchedule()
{
self.validateSPM(spUUID) --- Only check if the storage pool was
acquired by one host, but not necessary this host
}
...
}
So it really depends on the node management application AKA
ovirt-engine to dispatch the SPM operations to the right VDSM host.
Hi Shu,
validateSPM is:
def validateSPM(self, spUUID):
pool = self.getPool(spUUID)
if pool.spmRole != sp.SPM_ACQUIRED:
raise se.SpmStatusError(spUUID)
despite its ambiguous name SPM_ACQUIRED refers only to the spmRole
of the current host. That said, vdsm actually checks before running
deleteImage if the host is actually the SPM or not. Eventually you can
verify it running deleteImage on an HSM host, it should fail with:
# vdsClient 0 deleteImage ...
Not SPM: ('spUUID',)
VDSM doesn't block API functions at all, it allows you to perform any
api functions in any host whether if its SPM or not. What seems wrong,
due to that SPM operations are allowed only if VDSM internal state is
set to SPM, so why we publish code that we don't use.. seems like it was
changed during the time (the dispatcher should hide from the API
functions that only the SPM can perform). Now we have the validation as
part of sp.py (pool operations), that uses securable.py for that. little
bit complicated... but this is how it currently works..
--
Yaniv Bronhaim.
RedHat, Israel
09-7692289
054-7744187
___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] starting up vdsm and svdsm
On 12/30/2012 12:11 PM, Alon Bar-Lev wrote: - Original Message - From: ybronhei ybron...@redhat.com To: VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Sunday, December 30, 2012 11:53:17 AM Subject: [vdsm] starting up vdsm and svdsm Currently, vdsm deamon script starts (respawn) vdsm process as vdsm user, and during that vdsm starts super vdsm process as root. * when vdsm dies, svdsm process supposed to notice and exit by itself. * if svdsm dies, in the next proxy call by vdsm, vdsm supposed to start new svdsm instance after verifying that old instance is dead and call it. This is super vdsm and vdsm startup design in general. -- In current implementation we have some edge cases that we need to handle: * vsdm can try to communicate with old instance of svdsm. * vdsm can kill wrong process that got last svdsm pid before starting the new instance of svdsm. * vdsm can try to communicate with svdsm before svdsm starts to serve requests. And i guess there are many more possible senerios that can cause bugs.. As it seems, it doesn't make sense that vdsm manages root process, and can kill it.. What if svdsm will be the manager of vdsm: * deamon script starts up svdsm instance instead of vdsm * svdsm forks vdsm and changes its privilege (uid to vdsm) * vdsm receives as parameter svdsm pid for sudo requests * when vdsm dies, svdsm will start new instance of vdsm automatically, and note the crash reason to syslog. * svdsm starts with respawn, so when svdsm dies, vdsm dies also as its son process, and another instance of svdsm will start automatically and start new instance of vdsm. Sounds like much correcter and easier to maintain design. I am not sure I understand why these two services are related. As far as I understand svdsm is a total stateless slave without any logic, to provide privilege escalation to vdsm. Why does it need to be restarted if vdsm changes state? I expect it to exist as a separate service (init.d/systemd) and vdsmd to depend on that service. What am I missing? There is no relation between the states of the two. The only relation is that if one is up the other one must also be up and serve requests. for that case, systemd dependencies can solve the issue.. Instead of using systemd dependency and split the vdsmd service, I suggested to change the current relation and startup progress. Separate to two services that depended on each-other is another option to consider. Also, svdsm can init whatever vdsm needs and is limited to do as a vdsm user (check log permissions, clean old temporary files and so on if needed..) Royce Lv has already started to work on such design here http://gerrit.ovirt.org/#/c/4145/ I want to update it and push it forward. I would like to hear more opinions and point of views on this change, Thanks. -- Yaniv Bronhaim. RedHat, Israel 09-7692289 054-7744187 ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel -- Yaniv Bronhaim. RedHat, Israel 09-7692289 054-7744187 ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] starting up vdsm and svdsm
On 12/30/2012 01:45 PM, Alon Bar-Lev wrote: - Original Message - From: ybronhei ybron...@redhat.com To: Alon Bar-Lev alo...@redhat.com Cc: VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Sunday, December 30, 2012 1:40:03 PM Subject: Re: [vdsm] starting up vdsm and svdsm On 12/30/2012 12:11 PM, Alon Bar-Lev wrote: - Original Message - From: ybronhei ybron...@redhat.com To: VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Sunday, December 30, 2012 11:53:17 AM Subject: [vdsm] starting up vdsm and svdsm Currently, vdsm deamon script starts (respawn) vdsm process as vdsm user, and during that vdsm starts super vdsm process as root. * when vdsm dies, svdsm process supposed to notice and exit by itself. * if svdsm dies, in the next proxy call by vdsm, vdsm supposed to start new svdsm instance after verifying that old instance is dead and call it. This is super vdsm and vdsm startup design in general. -- In current implementation we have some edge cases that we need to handle: * vsdm can try to communicate with old instance of svdsm. * vdsm can kill wrong process that got last svdsm pid before starting the new instance of svdsm. * vdsm can try to communicate with svdsm before svdsm starts to serve requests. And i guess there are many more possible senerios that can cause bugs.. As it seems, it doesn't make sense that vdsm manages root process, and can kill it.. What if svdsm will be the manager of vdsm: * deamon script starts up svdsm instance instead of vdsm * svdsm forks vdsm and changes its privilege (uid to vdsm) * vdsm receives as parameter svdsm pid for sudo requests * when vdsm dies, svdsm will start new instance of vdsm automatically, and note the crash reason to syslog. * svdsm starts with respawn, so when svdsm dies, vdsm dies also as its son process, and another instance of svdsm will start automatically and start new instance of vdsm. Sounds like much correcter and easier to maintain design. I am not sure I understand why these two services are related. As far as I understand svdsm is a total stateless slave without any logic, to provide privilege escalation to vdsm. Why does it need to be restarted if vdsm changes state? I expect it to exist as a separate service (init.d/systemd) and vdsmd to depend on that service. What am I missing? There is no relation between the states of the two. The only relation is that if one is up the other one must also be up and serve requests. for that case, systemd dependencies can solve the issue.. This is a dependency. Why the other when must also be up? From what I understand, there is no problem in having svdsm up anyway, hence no mutual dependency. If we do have mutual dependency we should work to eliminate that anyway. right, svdsm can be up anyway.. but must be up and serve to let vdsm to be operational. and we can solve that by normal systemd dependency, as vdsm will require svdsm service and run it in type=forking (this way we guarantee that the parent exists and runs before starting). I don't see any other mutual dependency between the two.. Instead of using systemd dependency and split the vdsmd service, I suggested to change the current relation and startup progress. Any benefit in that over using proper system services? Maybe not.. just need to consider differences between systemd and initctl (upstart) dependencies mechanism if we decide to rely on system services. And they work quite different.. If we don't want to have 2 implementations for that as we have now with some if-else statements in vdsmd.init script, it might help to keep it in one service that splits to 2 processes.. Separate to two services that depended on each-other is another option to consider. Also, svdsm can init whatever vdsm needs and is limited to do as a vdsm user (check log permissions, clean old temporary files and so on if needed..) Royce Lv has already started to work on such design here http://gerrit.ovirt.org/#/c/4145/ I want to update it and push it forward. I would like to hear more opinions and point of views on this change, Thanks. -- Yaniv Bronhaim. RedHat, Israel 09-7692289 054-7744187 ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel -- Yaniv Bronhaim. RedHat, Israel 09-7692289 054-7744187 -- Yaniv Bronhaim. RedHat, Israel 09-7692289 054-7744187 ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] [Engine-devel] Host bios information
, in the json-rpc base model, calls are not only cheaper, you also have batch calls. This means you can send multiple requests as one message and have VDSM send you the responses as one message once all tasks completed. This makes splitting aggregated methods to smaller methods painless and with minimal overhead. - Original Message - From: Shu Ming shum...@linux.vnet.ibm.com To: ybronhei ybron...@redhat.com Cc: VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Thursday, December 13, 2012 11:04:09 AM Subject: Re: [vdsm] Host bios information After a quick review of the wiki page, it was stated that dmidecode gave too much informations. Only five fields will be displayed in the hardware tab, Manufactory, Version, Family, UUID and serial number. For Family, it is mean the CPU core's family. And it confuses me a bit with the CPU name and CPU type fields in general tab. I think we should chose the best one to characterizethe CPU type. ybronhei: Today in the Api we display general information about the host that vdsm export by getCapabilities Api. We decided to add bios information as part of the information that is displayed in UI under host's general sub-tab. To summaries the feature - We'll modify General tab to Software Information and add another tab for Hardware Information which will include all the bios data that we'll decide to gather from the host and display. Following this feature page: http://www.ovirt.org/Features/Design/HostBiosInfo for more details. All the parameters that can be displayed are mentioned in the wiki. I would greatly appreciate your comments and questions. Thanks. -- --- 舒明 Shu Ming Open Virtualization Engineerning; CSTL, IBM Corp. Tel: 86-10-82451626 Tieline: 9051626 E-mail: shum...@cn.ibm.com or shum...@linux.vnet.ibm.com Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel ___ Engine-devel mailing list engine-de...@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel -- Yaniv Bronhaim. RedHat, Israel 09-7692289 054-7744187 ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Host bios information
On 12/05/2012 04:32 PM, Adam Litke wrote: On Wed, Dec 05, 2012 at 11:05:21AM +0200, ybronhei wrote: Today in the Api we display general information about the host that vdsm export by getCapabilities Api. We decided to add bios information as part of the information that is displayed in UI under host's general sub-tab. To summaries the feature - We'll modify General tab to Software Information and add another tab for Hardware Information which will include all the bios data that we'll decide to gather from the host and display. Following this feature page: http://www.ovirt.org/Features/Design/HostBiosInfo for more details. All the parameters that can be displayed are mentioned in the wiki. I would greatly appreciate your comments and questions. Seems good to me but I would like to throw out one suggestion. getVdsCapabilities is already a huge command that does a lot of time consuming things. As part of the vdsm API refactoring, we are going to start favoring small and concise APIs over bag APIs. Perhaps we should just add a new verb: Host.getVdsBiosInfo() that returns only this information. It leads to modification also in how the engine collects the parameters with the new api request and I'm not sure if we should get into this.. Now we have specific known way of how engine requests for capabilities, when and how it effects the status of the host that is shown via the UI. To simplify this feature I prefer to use the current way of gathering and providing host's information. If we'll decide to split the host's capabilities api, it needs to get rfcs mail of its own because it changes engine's internal flows and it makes this feature to something much more influential. -- Yaniv Bronhaim. RedHat, Israel 09-7692289 054-7744187 ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] [VDSM][RFC] hsm service standalone
On 12/03/2012 07:22 PM, Saggi Mizrahi wrote: HSM is not a package it's an application. Currently it and the rest of VDSM share the same process but they use RPC to communicate. This is done so that one day we can actually have them run as different processes. HSM is not something you import, it's a daemon you communicate with. - Original Message - From: Dan Kenigsberg dan...@redhat.com To: Saggi Mizrahi smizr...@redhat.com Cc: Sheldon shao...@linux.vnet.ibm.com, a...@linux.vnet.ibm.com, vdsm-devel@lists.fedorahosted.org, Zheng Sheng ZS Zhou zhshz...@cn.ibm.com Sent: Monday, December 3, 2012 12:01:28 PM Subject: Re: [vdsm] [VDSM][RFC] hsm service standalone On Mon, Dec 03, 2012 at 11:35:44AM -0500, Saggi Mizrahi wrote: There are a bunch of precondition to having HSM pulled out. On simple things is that someone needs to go through storage/misc.py and utils.py and move all the code out to logical packages. There also needs to be a bit of a rearrangement of the code files so they can import the reusable code properly. I am also still very much against putting core VDSM in to site-packages. Would you elaborate on your position? Do you mind the wrong implications this may give about API stability? Contrary, It may help to split internal api and external (means rpc between vdsm to engine and rpc between vdsm and hsm), and it might be easier to understand the process flow and control the request from engine. For example, for migration you will receive vdsm request for migration, and that will initiate all the internal processes that we do during migration by sending requests to hsm service. The engine doesn't supposed to care about the internal rpc and those abilities does not supposed to be exposed via vdsm api if we don't allow those specific hsm operation from engine, but via HSM-python service that won't be available for the engine. Might be a good division. But lots of work indeed.. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel -- Yaniv Bronhaim. RedHat, Israel 09-7692289 054-7744187 ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
