Re: [vdsm] VDSM Repository Reorganization
On Tue, Feb 12, 2013 at 01:58:01AM +0100, Ewoud Kohl van Wijngaarden wrote: On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote: It is some time now that we are discussing an eventual repository reorganization for vdsm. In fact I'm sure that we all experienced at least once the discomfort of having several modules scattered around the tree. The main goal of the reorganization would be to place the modules in their proper location so that they can be used (imported) without any special change (or hack) even when the code is executed inside the development repository (e.g. tests). Recently there has been an initial proposal about moving some of these modules: http://gerrit.ovirt.org/#/c/11858/ That spawned an interesting discussion that must involve the entire community; in fact before starting any work we should try to converge on a decision for the final repository structure in order to minimize the discomfort for the contributors that will be forced to rebase their pending gerrit patches. Even if the full reorganization won't happen in a short time I think we should plan the entire structure now and then eventually move only few relevant modules to their final location. To start the discussion I'm attaching here a sketch of the vdsm repository structure that I envision: . |-- client | |-- [...] | `-- vdsClient.py |-- common | |-- [...] | |-- betterPopen | | `-- [...] | `-- vdsm | |-- [...] | `-- config.py |-- contrib | |-- [...] | |-- nfs-check.py | `-- sos |-- daemon | |-- [...] | |-- supervdsm.py | `-- vdsmd `-- tool |-- [...] `-- vdsm-tool Wondering if common should be named lib since you'll most likely want to install it to the python lib dir anyway. Minor bikeshedding. Why the distinction between client and tool? Do you expect other scripts to be added and give them all their own directory? I'd expect that most of their code would mostly live in vdsm and the actual executables are thin wrappers. This would match up with the scripts dir distutils has. vdsm-tool is to be shipped with the server only. It is a tool that helps the admin (and the sysv/systemd service) configure the host for vdsm. It's code is part of common (or lib) only to make it easier to use it with the default sys.path. Where do the hooks go? They, too, are part of the daemon, but I think they can stay as a top-level directory as they are now. This would allow any component (daemon, client, tool, etc...) to run in place with the only addition of PYTHONPATH=$(top_srcdir)/common. The tests would need also the additional component path but that is a given since it's what they should be testing. +1 on fewer hacks to load the environment. Once the components are in the proper place we can eventually start using distutils inside the Makefile.am files in order to simplify the installation process (and also as verification that our repository structure is complying with the python standards). +1 IMHO use of auto* should be minimized when developing python. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] VDSM Repository Reorganization
On Tue, Feb 19, 2013 at 03:53:46PM -0500, Saggi Mizrahi wrote: I'm not sure what's the purpose of having different versions of the client/server on the same machine. The software repository is one and it should provide both (as they're built from the same source). This is the standard way of delivering client/server applications in all the distributions. We can change that but we must have a good reason. There isn't really a reason. But, as I said, you don't want them to depend on each other or have the schema in it's own rpm. This means that you have to distribute them separately. I also want to allow to update the client on a host without updating the server. This is because you may want to have a script that works across the cluster but not update all the hosts. Now, even though you will use only old methods, the schema itself might become unparsable by old implementations. This should never happen. Right now each symbol in the schema is represented by a single OrderedDict and the parsing code just loads the schema file into the a list of these dicts. Once loaded, the vdsmapi module categorizes symbols according to the top-level keys. Unrecognized symbol types are simply skipped. -- Adam Litke a...@us.ibm.com IBM Linux Technology Center ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] VDSM Repository Reorganization
In order to make progress on the file reorg, I want to summarize the discussion and propose that a consensus has been reached regarding placement of the schema file. The current code has a routine find_schema() that can locate the schema file in the development source tree or in an installed location. Therefore, it only needs to appear in the source tree in a single location and we will not need any symlinks for this purpose. Recently, the API handling code (schema and parsing module) have been split into their own rpm. This should solve the installation problem since any entity that needs access to the schema and parser should simply install the vdsm-api rpm. -- Adam Litke a...@us.ibm.com IBM Linux Technology Center ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] VDSM Repository Reorganization
Hi Vinzenz, On 02/18/2013 05:43 PM, Vinzenz Feenstra wrote: It would be nice to come to an agreement any time soon. I would like to apply all the change as soon as possible. I would not like to see this go into the depth of a dead hole. The absence of feedback typically means one of 6 things: * No-one read it * No-one understood it * No-one cared * No-one is paying attention to you any more * It's so ridiculous it's unworthy of comment * Everyone who read it agrees I suggest you assume the last one, and proceed until someone starts shouting at you ;-) Thanks, Dave. -- Dave Neary - Community Action and Impact Open Source and Standards, Red Hat - http://community.redhat.com Ph: +33 9 50 71 55 62 / Cell: +33 6 77 01 92 13 ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] VDSM Repository Reorganization
- Original Message - From: Saggi Mizrahi smizr...@redhat.com To: Adam Litke a...@us.ibm.com Cc: VDSM Project Development vdsm-devel@lists.fedorahosted.org, Dan Kenigsberg dan...@redhat.com, Vinzenz Feenstra vfeen...@redhat.com, Ayal Baron aba...@redhat.com, Federico Simoncelli fsimo...@redhat.com Sent: Monday, February 18, 2013 8:50:30 PM Subject: Re: VDSM Repository Reorganization - Original Message - From: Adam Litke a...@us.ibm.com To: Federico Simoncelli fsimo...@redhat.com Cc: VDSM Project Development vdsm-devel@lists.fedorahosted.org, Dan Kenigsberg dan...@redhat.com, Saggi Mizrahi smizr...@redhat.com, Vinzenz Feenstra vfeen...@redhat.com, Ayal Baron aba...@redhat.com Sent: Tuesday, February 12, 2013 3:08:09 PM Subject: Re: VDSM Repository Reorganization On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote: It is some time now that we are discussing an eventual repository reorganization for vdsm. In fact I'm sure that we all experienced at least once the discomfort of having several modules scattered around the tree. The main goal of the reorganization would be to place the modules in their proper location so that they can be used (imported) without any special change (or hack) even when the code is executed inside the development repository (e.g. tests). Recently there has been an initial proposal about moving some of these modules: http://gerrit.ovirt.org/#/c/11858/ That spawned an interesting discussion that must involve the entire community; in fact before starting any work we should try to converge on a decision for the final repository structure in order to minimize the discomfort for the contributors that will be forced to rebase their pending gerrit patches. Even if the full reorganization won't happen in a short time I think we should plan the entire structure now and then eventually move only few relevant modules to their final location. To start the discussion I'm attaching here a sketch of the vdsm repository structure that I envision: . |-- client | |-- [...] | `-- vdsClient.py |-- common | |-- [...] | |-- betterPopen | | `-- [...] | `-- vdsm | |-- [...] | `-- config.py |-- contrib | |-- [...] | |-- nfs-check.py | `-- sos |-- daemon | |-- [...] | |-- supervdsm.py | `-- vdsmd `-- tool |-- [...] `-- vdsm-tool The schema file vdsmapi-schema.json (as well as the python module to parse it) are needed by the server and clients. Initially I'd think it should be installed in 'common', but a client does not need things like betterPopen. Any recommendation on where the schema/API definition should live? Well they both should have the file but when installed both should have their own version of the file depending on the version installed of the client or the server. This is so that vdsm-cli doesn't depend on vdsm or vice-versa. You can't have them share the file since if one is installed with a version of the schema where the schema syntax changed the client\server will fail to parse the schema. I'm not sure what's the purpose of having different versions of the client/server on the same machine. The software repository is one and it should provide both (as they're built from the same source). This is the standard way of delivering client/server applications in all the distributions. We can change that but we must have a good reason. As for development, I think the least bad solution is to put it in contrib with symlinks that have relative paths. |--daemon | |-- [...] | `-- vdsmapi-schema.json - ../contrib/vdsmapi-schema.json |--client | |-- [...] | `-- vdsmapi-schema.json - ../contrib/vdsmapi-schema.json |--contrib | |-- [...] | `-- vdsmapi-schema.json : . Git knows how to handle symlinks and symlinks are relative to the location of the symlink. +1. -- Federico ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] VDSM Repository Reorganization
- Original Message - From: Federico Simoncelli fsimo...@redhat.com To: Saggi Mizrahi smizr...@redhat.com Cc: VDSM Project Development vdsm-devel@lists.fedorahosted.org, Dan Kenigsberg dan...@redhat.com, Vinzenz Feenstra vfeen...@redhat.com, Ayal Baron aba...@redhat.com, Adam Litke a...@us.ibm.com Sent: Tuesday, February 19, 2013 11:27:59 AM Subject: Re: VDSM Repository Reorganization - Original Message - From: Saggi Mizrahi smizr...@redhat.com To: Adam Litke a...@us.ibm.com Cc: VDSM Project Development vdsm-devel@lists.fedorahosted.org, Dan Kenigsberg dan...@redhat.com, Vinzenz Feenstra vfeen...@redhat.com, Ayal Baron aba...@redhat.com, Federico Simoncelli fsimo...@redhat.com Sent: Monday, February 18, 2013 8:50:30 PM Subject: Re: VDSM Repository Reorganization - Original Message - From: Adam Litke a...@us.ibm.com To: Federico Simoncelli fsimo...@redhat.com Cc: VDSM Project Development vdsm-devel@lists.fedorahosted.org, Dan Kenigsberg dan...@redhat.com, Saggi Mizrahi smizr...@redhat.com, Vinzenz Feenstra vfeen...@redhat.com, Ayal Baron aba...@redhat.com Sent: Tuesday, February 12, 2013 3:08:09 PM Subject: Re: VDSM Repository Reorganization On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote: It is some time now that we are discussing an eventual repository reorganization for vdsm. In fact I'm sure that we all experienced at least once the discomfort of having several modules scattered around the tree. The main goal of the reorganization would be to place the modules in their proper location so that they can be used (imported) without any special change (or hack) even when the code is executed inside the development repository (e.g. tests). Recently there has been an initial proposal about moving some of these modules: http://gerrit.ovirt.org/#/c/11858/ That spawned an interesting discussion that must involve the entire community; in fact before starting any work we should try to converge on a decision for the final repository structure in order to minimize the discomfort for the contributors that will be forced to rebase their pending gerrit patches. Even if the full reorganization won't happen in a short time I think we should plan the entire structure now and then eventually move only few relevant modules to their final location. To start the discussion I'm attaching here a sketch of the vdsm repository structure that I envision: . |-- client | |-- [...] | `-- vdsClient.py |-- common | |-- [...] | |-- betterPopen | | `-- [...] | `-- vdsm | |-- [...] | `-- config.py |-- contrib | |-- [...] | |-- nfs-check.py | `-- sos |-- daemon | |-- [...] | |-- supervdsm.py | `-- vdsmd `-- tool |-- [...] `-- vdsm-tool The schema file vdsmapi-schema.json (as well as the python module to parse it) are needed by the server and clients. Initially I'd think it should be installed in 'common', but a client does not need things like betterPopen. Any recommendation on where the schema/API definition should live? Well they both should have the file but when installed both should have their own version of the file depending on the version installed of the client or the server. This is so that vdsm-cli doesn't depend on vdsm or vice-versa. You can't have them share the file since if one is installed with a version of the schema where the schema syntax changed the client\server will fail to parse the schema. I'm not sure what's the purpose of having different versions of the client/server on the same machine. The software repository is one and it should provide both (as they're built from the same source). This is the standard way of delivering client/server applications in all the distributions. We can change that but we must have a good reason. There isn't really a reason. But, as I said, you don't want them to depend on each other or have the schema in it's own rpm. This means that you have to distribute them separately. I also want to allow to update the client on a host without updating the server. This is because you may want to have a script that works across the cluster but not update all the hosts. Now, even though you will use only old methods, the schema itself might become unparsable by old implementations. As for development, I think the least bad solution is to put it in contrib with symlinks that have relative paths. |--daemon | |-- [...] | `-- vdsmapi-schema.json - ../contrib/vdsmapi-schema.json |--client | |-- [...] | `-- vdsmapi-schema.json - ../contrib/vdsmapi-schema.json |--contrib | |-- [...] | `--
Re: [vdsm] VDSM Repository Reorganization
Hi everyone, It would be nice to come to an agreement any time soon. I would like to apply all the change as soon as possible. I would not like to see this go into the depth of a dead hole. Thanks. On 02/13/2013 03:54 PM, Vinzenz Feenstra wrote: On 02/13/2013 03:08 PM, Yaniv Bronheim wrote: First lets illustrate the current repository structure: - tests - vdsm - storage - gluster - betterPopen ... and more unused directories - vdsm_api - vdsm_cli - vdsm_hooks - vdsm_log - vdsm_reg - vdsm_tools - vds_bootstrap - contrib As I understand your proposal, you added 'common' and 'deamon' and left the rest the way it is. So vdsm can be renamed to vdsm_core and can include sub folders for deamon and common\lib python files. It'll be something like: - vdsm_core - deamon - vdsmd .. - more deamon scripts - lib - independent tools - can also be part of vdsm_tools, as it doesn't part of the vdsm_core - utils.py - betterPopen - dmidecode ... - more unrelated utils that vdsm uses - core - vdsm - API.py - clientIF.py - config.py defines.py constants.py ... should be part of vdsm_core - libvirtconnection.py ... core files of vdsm - storage ... as it now without files that can be and should be moved to lib folder (e.g. threadPool, task, taskManager and inc..) the rest we'll stay the same. about the installation location it something else.. In my opinion this arrangement is nice to have, but not a must - we'll still need to set PYTHONPATH when we run from different locations (what you called making an hack), so as I see it, it just for comfort.. no? Maybe I missed something.. (And tests can be split to subfolders for each part - core\tests lib\tests and inc..) - Original Message - From: Federico Simoncelli fsimo...@redhat.com To: Adam Litke a...@us.ibm.com Cc: VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Wednesday, February 13, 2013 1:06:55 AM Subject: Re: [vdsm] VDSM Repository Reorganization - Original Message - From: Adam Litke a...@us.ibm.com To: Federico Simoncelli fsimo...@redhat.com Cc: VDSM Project Development vdsm-devel@lists.fedorahosted.org, Dan Kenigsberg dan...@redhat.com, Saggi Mizrahi smizr...@redhat.com, Vinzenz Feenstra vfeen...@redhat.com, Ayal Baron aba...@redhat.com Sent: Tuesday, February 12, 2013 9:08:09 PM Subject: Re: VDSM Repository Reorganization On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote: It is some time now that we are discussing an eventual repository reorganization for vdsm. In fact I'm sure that we all experienced at least once the discomfort of having several modules scattered around the tree. The main goal of the reorganization would be to place the modules in their proper location so that they can be used (imported) without any special change (or hack) even when the code is executed inside the development repository (e.g. tests). Recently there has been an initial proposal about moving some of these modules: http://gerrit.ovirt.org/#/c/11858/ That spawned an interesting discussion that must involve the entire community; in fact before starting any work we should try to converge on a decision for the final repository structure in order to minimize the discomfort for the contributors that will be forced to rebase their pending gerrit patches. Even if the full reorganization won't happen in a short time I think we should plan the entire structure now and then eventually move only few relevant modules to their final location. To start the discussion I'm attaching here a sketch of the vdsm repository structure that I envision: . |-- client | |-- [...] | `-- vdsClient.py |-- common | |-- [...] | |-- betterPopen | | `-- [...] | `-- vdsm | |-- [...] | `-- config.py |-- contrib | |-- [...] | |-- nfs-check.py | `-- sos |-- daemon | |-- [...] | |-- supervdsm.py | `-- vdsmd `-- tool |-- [...] `-- vdsm-tool The schema file vdsmapi-schema.json (as well as the python module to parse it) are needed by the server and clients. Initially I'd think it should be installed in 'common', but a client does not need things like betterPopen. Any recommendation on where the schema/API definition should live? The problem is that betterPopen is misplaced as it's used only by the daemon. Anyway let's not mix the three different aspects: +1 - repository structure Well that's what is supposed to be the topic. - installation location - packaging Those are supposed to be off topic considerint $SUBJECT That said in my opinion it can remain in common (which can be renamed to lib as Ewoud suggested) and be installed in the site-lib *but* be packaged only with the daemon (while we're waiting for it to become fully independent and be moved out of the repo). Where such things could be put in a folder 'extra' for now. (As an example) I suppose that the code
Re: [vdsm] VDSM Repository Reorganization
- Original Message - From: Adam Litke a...@us.ibm.com To: Federico Simoncelli fsimo...@redhat.com Cc: VDSM Project Development vdsm-devel@lists.fedorahosted.org, Dan Kenigsberg dan...@redhat.com, Saggi Mizrahi smizr...@redhat.com, Vinzenz Feenstra vfeen...@redhat.com, Ayal Baron aba...@redhat.com Sent: Tuesday, February 12, 2013 3:08:09 PM Subject: Re: VDSM Repository Reorganization On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote: It is some time now that we are discussing an eventual repository reorganization for vdsm. In fact I'm sure that we all experienced at least once the discomfort of having several modules scattered around the tree. The main goal of the reorganization would be to place the modules in their proper location so that they can be used (imported) without any special change (or hack) even when the code is executed inside the development repository (e.g. tests). Recently there has been an initial proposal about moving some of these modules: http://gerrit.ovirt.org/#/c/11858/ That spawned an interesting discussion that must involve the entire community; in fact before starting any work we should try to converge on a decision for the final repository structure in order to minimize the discomfort for the contributors that will be forced to rebase their pending gerrit patches. Even if the full reorganization won't happen in a short time I think we should plan the entire structure now and then eventually move only few relevant modules to their final location. To start the discussion I'm attaching here a sketch of the vdsm repository structure that I envision: . |-- client | |-- [...] | `-- vdsClient.py |-- common | |-- [...] | |-- betterPopen | | `-- [...] | `-- vdsm | |-- [...] | `-- config.py |-- contrib | |-- [...] | |-- nfs-check.py | `-- sos |-- daemon | |-- [...] | |-- supervdsm.py | `-- vdsmd `-- tool |-- [...] `-- vdsm-tool The schema file vdsmapi-schema.json (as well as the python module to parse it) are needed by the server and clients. Initially I'd think it should be installed in 'common', but a client does not need things like betterPopen. Any recommendation on where the schema/API definition should live? Well they both should have the file but when installed both should have their own version of the file depending on the version installed of the client or the server. This is so that vdsm-cli doesn't depend on vdsm or vice-versa. You can't have them share the file since if one is installed with a version of the schema where the schema syntax changed the client\server will fail to parse the schema. As for development, I think the least bad solution is to put it in contrib with symlinks that have relative paths. |--daemon | |-- [...] | `-- vdsmapi-schema.json - ../contrib/vdsmapi-schema.json |--client | |-- [...] | `-- vdsmapi-schema.json - ../contrib/vdsmapi-schema.json |--contrib | |-- [...] | `-- vdsmapi-schema.json : . Git knows how to handle symlinks and symlinks are relative to the location of the symlink. We could also just select the daemon or the client folder and put the real file there and a symlink in the other but I feel it's like choosing which one of your children is the main user of a schema file. -- Adam Litke a...@us.ibm.com IBM Linux Technology Center ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] VDSM Repository Reorganization
First lets illustrate the current repository structure: - tests - vdsm - storage - gluster - betterPopen ... and more unused directories - vdsm_api - vdsm_cli - vdsm_hooks - vdsm_log - vdsm_reg - vdsm_tools - vds_bootstrap - contrib As I understand your proposal, you added 'common' and 'deamon' and left the rest the way it is. So vdsm can be renamed to vdsm_core and can include sub folders for deamon and common\lib python files. It'll be something like: - vdsm_core - deamon - vdsmd .. - more deamon scripts - lib - independent tools - can also be part of vdsm_tools, as it doesn't part of the vdsm_core - utils.py - betterPopen - dmidecode ... - more unrelated utils that vdsm uses - core - vdsm - API.py - clientIF.py - config.py defines.py constants.py ... should be part of vdsm_core - libvirtconnection.py ... core files of vdsm - storage ... as it now without files that can be and should be moved to lib folder (e.g. threadPool, task, taskManager and inc..) the rest we'll stay the same. about the installation location it something else.. In my opinion this arrangement is nice to have, but not a must - we'll still need to set PYTHONPATH when we run from different locations (what you called making an hack), so as I see it, it just for comfort.. no? Maybe I missed something.. (And tests can be split to subfolders for each part - core\tests lib\tests and inc..) - Original Message - From: Federico Simoncelli fsimo...@redhat.com To: Adam Litke a...@us.ibm.com Cc: VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Wednesday, February 13, 2013 1:06:55 AM Subject: Re: [vdsm] VDSM Repository Reorganization - Original Message - From: Adam Litke a...@us.ibm.com To: Federico Simoncelli fsimo...@redhat.com Cc: VDSM Project Development vdsm-devel@lists.fedorahosted.org, Dan Kenigsberg dan...@redhat.com, Saggi Mizrahi smizr...@redhat.com, Vinzenz Feenstra vfeen...@redhat.com, Ayal Baron aba...@redhat.com Sent: Tuesday, February 12, 2013 9:08:09 PM Subject: Re: VDSM Repository Reorganization On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote: It is some time now that we are discussing an eventual repository reorganization for vdsm. In fact I'm sure that we all experienced at least once the discomfort of having several modules scattered around the tree. The main goal of the reorganization would be to place the modules in their proper location so that they can be used (imported) without any special change (or hack) even when the code is executed inside the development repository (e.g. tests). Recently there has been an initial proposal about moving some of these modules: http://gerrit.ovirt.org/#/c/11858/ That spawned an interesting discussion that must involve the entire community; in fact before starting any work we should try to converge on a decision for the final repository structure in order to minimize the discomfort for the contributors that will be forced to rebase their pending gerrit patches. Even if the full reorganization won't happen in a short time I think we should plan the entire structure now and then eventually move only few relevant modules to their final location. To start the discussion I'm attaching here a sketch of the vdsm repository structure that I envision: . |-- client | |-- [...] | `-- vdsClient.py |-- common | |-- [...] | |-- betterPopen | | `-- [...] | `-- vdsm | |-- [...] | `-- config.py |-- contrib | |-- [...] | |-- nfs-check.py | `-- sos |-- daemon | |-- [...] | |-- supervdsm.py | `-- vdsmd `-- tool |-- [...] `-- vdsm-tool The schema file vdsmapi-schema.json (as well as the python module to parse it) are needed by the server and clients. Initially I'd think it should be installed in 'common', but a client does not need things like betterPopen. Any recommendation on where the schema/API definition should live? The problem is that betterPopen is misplaced as it's used only by the daemon. Anyway let's not mix the three different aspects: - repository structure - installation location - packaging That said in my opinion it can remain in common (which can be renamed to lib as Ewoud suggested) and be installed in the site-lib *but* be packaged only with the daemon (while we're waiting for it to become fully independent and be moved out of the repo). I suppose that the code dealing with the schema will be shared by both the client and the daemon and it will be placed in common (or lib) and the schema itself can go there too. To be honest I started envisioning client/common/daemon/tool as pure-python directories (as they are once installed) so maybe the schema could be placed somewhere else with other miscellaneous files (./schema? ./config?), but I wouldn't stress over
Re: [vdsm] VDSM Repository Reorganization
On 02/13/2013 03:08 PM, Yaniv Bronheim wrote: First lets illustrate the current repository structure: - tests - vdsm - storage - gluster - betterPopen ... and more unused directories - vdsm_api - vdsm_cli - vdsm_hooks - vdsm_log - vdsm_reg - vdsm_tools - vds_bootstrap - contrib As I understand your proposal, you added 'common' and 'deamon' and left the rest the way it is. So vdsm can be renamed to vdsm_core and can include sub folders for deamon and common\lib python files. It'll be something like: - vdsm_core - deamon - vdsmd .. - more deamon scripts - lib - independent tools - can also be part of vdsm_tools, as it doesn't part of the vdsm_core - utils.py - betterPopen - dmidecode ... - more unrelated utils that vdsm uses - core - vdsm - API.py - clientIF.py - config.py defines.py constants.py ... should be part of vdsm_core - libvirtconnection.py ... core files of vdsm - storage ... as it now without files that can be and should be moved to lib folder (e.g. threadPool, task, taskManager and inc..) the rest we'll stay the same. about the installation location it something else.. In my opinion this arrangement is nice to have, but not a must - we'll still need to set PYTHONPATH when we run from different locations (what you called making an hack), so as I see it, it just for comfort.. no? Maybe I missed something.. (And tests can be split to subfolders for each part - core\tests lib\tests and inc..) - Original Message - From: Federico Simoncelli fsimo...@redhat.com To: Adam Litke a...@us.ibm.com Cc: VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Wednesday, February 13, 2013 1:06:55 AM Subject: Re: [vdsm] VDSM Repository Reorganization - Original Message - From: Adam Litke a...@us.ibm.com To: Federico Simoncelli fsimo...@redhat.com Cc: VDSM Project Development vdsm-devel@lists.fedorahosted.org, Dan Kenigsberg dan...@redhat.com, Saggi Mizrahi smizr...@redhat.com, Vinzenz Feenstra vfeen...@redhat.com, Ayal Baron aba...@redhat.com Sent: Tuesday, February 12, 2013 9:08:09 PM Subject: Re: VDSM Repository Reorganization On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote: It is some time now that we are discussing an eventual repository reorganization for vdsm. In fact I'm sure that we all experienced at least once the discomfort of having several modules scattered around the tree. The main goal of the reorganization would be to place the modules in their proper location so that they can be used (imported) without any special change (or hack) even when the code is executed inside the development repository (e.g. tests). Recently there has been an initial proposal about moving some of these modules: http://gerrit.ovirt.org/#/c/11858/ That spawned an interesting discussion that must involve the entire community; in fact before starting any work we should try to converge on a decision for the final repository structure in order to minimize the discomfort for the contributors that will be forced to rebase their pending gerrit patches. Even if the full reorganization won't happen in a short time I think we should plan the entire structure now and then eventually move only few relevant modules to their final location. To start the discussion I'm attaching here a sketch of the vdsm repository structure that I envision: . |-- client | |-- [...] | `-- vdsClient.py |-- common | |-- [...] | |-- betterPopen | | `-- [...] | `-- vdsm | |-- [...] | `-- config.py |-- contrib | |-- [...] | |-- nfs-check.py | `-- sos |-- daemon | |-- [...] | |-- supervdsm.py | `-- vdsmd `-- tool |-- [...] `-- vdsm-tool The schema file vdsmapi-schema.json (as well as the python module to parse it) are needed by the server and clients. Initially I'd think it should be installed in 'common', but a client does not need things like betterPopen. Any recommendation on where the schema/API definition should live? The problem is that betterPopen is misplaced as it's used only by the daemon. Anyway let's not mix the three different aspects: +1 - repository structure Well that's what is supposed to be the topic. - installation location - packaging Those are supposed to be off topic considerint $SUBJECT That said in my opinion it can remain in common (which can be renamed to lib as Ewoud suggested) and be installed in the site-lib *but* be packaged only with the daemon (while we're waiting for it to become fully independent and be moved out of the repo). Where such things could be put in a folder 'extra' for now. (As an example) I suppose that the code dealing with the schema will be shared by both the client and the daemon and it will be placed in common (or lib) and the schema itself can go there too. To be honest I started envisioning client/common/daemon/tool as pure-python directories (as they are once
[vdsm] VDSM Repository Reorganization
It is some time now that we are discussing an eventual repository reorganization for vdsm. In fact I'm sure that we all experienced at least once the discomfort of having several modules scattered around the tree. The main goal of the reorganization would be to place the modules in their proper location so that they can be used (imported) without any special change (or hack) even when the code is executed inside the development repository (e.g. tests). Recently there has been an initial proposal about moving some of these modules: http://gerrit.ovirt.org/#/c/11858/ That spawned an interesting discussion that must involve the entire community; in fact before starting any work we should try to converge on a decision for the final repository structure in order to minimize the discomfort for the contributors that will be forced to rebase their pending gerrit patches. Even if the full reorganization won't happen in a short time I think we should plan the entire structure now and then eventually move only few relevant modules to their final location. To start the discussion I'm attaching here a sketch of the vdsm repository structure that I envision: . |-- client | |-- [...] | `-- vdsClient.py |-- common | |-- [...] | |-- betterPopen | | `-- [...] | `-- vdsm | |-- [...] | `-- config.py |-- contrib | |-- [...] | |-- nfs-check.py | `-- sos |-- daemon | |-- [...] | |-- supervdsm.py | `-- vdsmd `-- tool |-- [...] `-- vdsm-tool This would allow any component (daemon, client, tool, etc...) to run in place with the only addition of PYTHONPATH=$(top_srcdir)/common. The tests would need also the additional component path but that is a given since it's what they should be testing. Once the components are in the proper place we can eventually start using distutils inside the Makefile.am files in order to simplify the installation process (and also as verification that our repository structure is complying with the python standards). Please get involved, share your feedback and proposals. Thanks, -- Federico ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] VDSM Repository Reorganization
On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote: It is some time now that we are discussing an eventual repository reorganization for vdsm. In fact I'm sure that we all experienced at least once the discomfort of having several modules scattered around the tree. The main goal of the reorganization would be to place the modules in their proper location so that they can be used (imported) without any special change (or hack) even when the code is executed inside the development repository (e.g. tests). Recently there has been an initial proposal about moving some of these modules: http://gerrit.ovirt.org/#/c/11858/ That spawned an interesting discussion that must involve the entire community; in fact before starting any work we should try to converge on a decision for the final repository structure in order to minimize the discomfort for the contributors that will be forced to rebase their pending gerrit patches. Even if the full reorganization won't happen in a short time I think we should plan the entire structure now and then eventually move only few relevant modules to their final location. To start the discussion I'm attaching here a sketch of the vdsm repository structure that I envision: . |-- client | |-- [...] | `-- vdsClient.py |-- common | |-- [...] | |-- betterPopen | | `-- [...] | `-- vdsm | |-- [...] | `-- config.py |-- contrib | |-- [...] | |-- nfs-check.py | `-- sos |-- daemon | |-- [...] | |-- supervdsm.py | `-- vdsmd `-- tool |-- [...] `-- vdsm-tool Wondering if common should be named lib since you'll most likely want to install it to the python lib dir anyway. Minor bikeshedding. Why the distinction between client and tool? Do you expect other scripts to be added and give them all their own directory? I'd expect that most of their code would mostly live in vdsm and the actual executables are thin wrappers. This would match up with the scripts dir distutils has. Where do the hooks go? This would allow any component (daemon, client, tool, etc...) to run in place with the only addition of PYTHONPATH=$(top_srcdir)/common. The tests would need also the additional component path but that is a given since it's what they should be testing. +1 on fewer hacks to load the environment. Once the components are in the proper place we can eventually start using distutils inside the Makefile.am files in order to simplify the installation process (and also as verification that our repository structure is complying with the python standards). +1 IMHO use of auto* should be minimized when developing python. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
