[vdsm] About code segment in vdsm/storage/image.py
Hi, I am reading the code of merge() function in image.py and get one question about the code. I wonder why we should get the volume parameters from the parent of the destination volume if the parent is existing instead from the destination volume itself? 1101if dstParentUUID != sd.BLANK_UUID: volParams = vols[dstParentUUID].getVolumeParams() else: 1105 volParams = dstVol.getVolumeParams() -- Shu Ming shum...@linux.vnet.ibm.com IBM China Systems and Technology Laboratory ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] About code segment in vdsm/storage/image.py
On 2012-6-25 15:21, Shu Ming wrote: Hi, I am reading the code of merge() function in image.py and get one question about the code. I wonder why we should get the volume parameters from the parent of the destination volume if the parent is existing instead from the destination volume itself? 1101if dstParentUUID != sd.BLANK_UUID: volParams = vols[dstParentUUID].getVolumeParams() else: 1105 volParams = dstVol.getVolumeParams() In my understanding, the volParams is different in internal volume merge and base volume merge(merging to a top most base without parent). For internal volume merge, the volParams comes from the parent of the destination volume. While for base volume merge, the volParams comes from the destination volume. Is it correct? -- Shu Ming shum...@linux.vnet.ibm.com IBM China Systems and Technology Laboratory ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] [virt-node] VDSM as a general purpose virt host manager
On Fri, Jun 22, 2012 at 06:45:43PM -0400, Andrew Cathrow wrote: - Original Message - From: Ryan Harper ry...@us.ibm.com To: Adam Litke a...@us.ibm.com Cc: Anthony Liguori aligu...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Friday, June 22, 2012 12:45:42 PM Subject: Re: [vdsm] [virt-node] VDSM as a general purpose virt host manager * Adam Litke a...@us.ibm.com [2012-06-22 11:35]: On Thu, Jun 21, 2012 at 12:17:19PM +0300, Dor Laor wrote: On 06/19/2012 08:12 PM, Saggi Mizrahi wrote: - Original Message - From: Deepak C Shetty deepa...@linux.vnet.ibm.com To: Ryan Harper ry...@us.ibm.com Cc: Saggi Mizrahi smizr...@redhat.com, Anthony Liguori aligu...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Tuesday, June 19, 2012 10:58:47 AM Subject: Re: [vdsm] [virt-node] VDSM as a general purpose virt host manager On 06/19/2012 01:13 AM, Ryan Harper wrote: * Saggi Mizrahismizr...@redhat.com [2012-06-18 10:05]: I would like to put on to the table for descussion the growing need for a way to more easily reuse of the functionality of VDSM in order to service projects other than Ovirt-Engine. Originally VDSM was created as a proprietary agent for the sole purpose of serving the then proprietary version of what is known as ovirt-engine. Red Hat, after acquiring the technology, pressed on with it's commitment to open source ideals and released the code. But just releasing code into the wild doesn't build a community or makes a project successful. Further more when building open source software you should aspire to build reusable components instead of monolithic stacks. Saggi, Thanks for sending this out. I've been trying to pull together some thoughts on what else is needed for vdsm as a community. I know that for some time downstream has been the driving force for all of the work and now with a community there are challenges in finding our own way. While we certainly don't want to make downstream efforts harder, I think we need to develop and support our own vision for what vdsm can be come, some what independent of downstream and other exploiters. Revisiting the API is definitely a much needed endeavor and I think adding some use-cases or sample applications would be useful in demonstrating whether or not we're evolving the API into something easier to use for applications beyond engine. We would like to expose a stable, documented, well supported API. This gives us a chance to rethink the VDSM API from the ground up. There is already work in progress of making the internal logic of VDSM separate enough from the API layer so we could continue feature development and bug fixing while designing the API of the future. In order to achieve this though we need to do several things: 1. Declare API supportability guidelines 2. Decide on an API transport (e.g. REST, ZMQ, AMQP) 3. Make the API easily consumable (e.g. proper docs, example code, extending the API, etc) 4. Implement the API itself In the earlier we'd discussed working to have similarities in the modeling between the oVirt API and VDSM but that seems to have dropped off the radar. Yes, the current REST API has attempted to be compatible with the current ovirt-engine API. Going forward, I am not sure how easy this will be to maintain given than engine is built on Java and vdsm is built on Python. -- Adam Litke a...@us.ibm.com IBM Linux Technology Center ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
[vdsm] [RFC] An alternative way to provide a supported interface -- libvdsm
Hi, I've been reading through the API threads here and considering the options. To be honest, I worry a lot about the scope of these discussions and that there's a tremendous amount of work before we have a useful end result. I wonder if we can solve this problem by adding another layer of abstraction... As Adam is currently building a schema for VDSM's XML-RPC, we could use the QAPI code generators to build a libvdsm that provided a programmatic C interface for the XML-RPC interface. It would take some tweaking, but this could be made a supportable C interface. The rules for having a supportable C interface are basically: 1) Never change function signatures 2) Never remove functions 3) Always allocate structures in the library and/or pad 4) Only add to structures, never remove or reorder 5) Provide flags that default to zero to indicate that fields/features are not present. 6) Always zero-initialize structures Having a libvdsm would allow the transport to change over time w/o affecting end-users. There are lots of good tools for documenting C APIs and dealing with versioning of C APIs. While we can start out with a schema-generated API, over time, we can implement libvdsm in an open-coded fashion allowing old APIs to be reimplemented in terms of new APIs. From a compatibility perspective, libvdsm would be fully backwards compatible with old versions of VDSM (so it would keep XML-RPC support forever) but may require new versions of libvdsm to talk to new versions of VDSM. That would allow for APIs to be deprecated within VDSM without breaking old clients. I think this would be an incremental approach to building a supportable API today while still giving the flexibility to make changes in the long term. And it should be fairly easy to generate a JNI binding and also port ovirt-engine to use an interface like this (since it already uses the XML-RPC API). Regards, Anthony Liguori ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] [Engine-devel] RFC: Writeup on VDSM-libstoragemgmt integration
On 06/19/2012 01:45 AM, Saggi Mizrahi wrote: First of all I'd like to suggest not using the LSM acronym as it can also mean live-storage-migration and maybe other things. Sure, what do you suggest ? libSM ? Secondly I would like to avoid talking about what needs to be changed in VDSM before we figure out what exactly we want to accomplish. Also, there is no mention on credentials in any part of the process. How does VDSM or the host get access to actually modify the storage array? Who holds the creds for that and how? How does the user set this up? Per my original discussion on this with Ayal, this is what he had suggested... In addition, I'm assuming we will either need a new 'storage array' entity in engine to keep credentials, or, in case of storage array as storage domain, just keep this info as part of the domain at engine level. Either we can have the libstoragemgmt cred stored in the engine as part of engine-setup or have the user input them as part of Storage Prov and user clicks on remember cred button, so engine saves it and passes it to VDSM as needed ? In any way, the cred should come from the user/admin, no other way correct ? In the array as domain case. How are the luns being mapped to initiators. What about setting discovery credentials. In the array set up case. How will the hosts be represented in regards to credentials? How will the different schemes and capabilities in regard to authentication methods will be expressed. Not clear on what the concern here is. Can you pls provide more clarity on the problem here ? Maybe providing some examples will help. Rest of the comments inline - Original Message - From: Deepak C Shettydeepa...@linux.vnet.ibm.com To: VDSM Project Developmentvdsm-devel@lists.fedorahosted.org Cc: libstoragemgmt-de...@lists.sourceforge.net, engine-de...@ovirt.org Sent: Wednesday, May 30, 2012 5:38:46 AM Subject: [Engine-devel] RFC: Writeup on VDSM-libstoragemgmt integration Hello All, I have a draft write-up on the VDSM-libstoragemgmt integration. I wanted to run this thru' the mailing list(s) to help tune and crystallize it, before putting it on the ovirt wiki. I have run this once thru Ayal and Tony, so have some of their comments incorporated. I still have few doubts/questions, which I have posted below with lines ending with '?' Comments / Suggestions are welcome appreciated. thanx, deepak [Ccing engine-devel and libstoragemgmt lists as this stuff is relevant to them too] -- 1) Background: VDSM provides high level API for node virtualization management. It acts in response to the requests sent by oVirt Engine, which uses VDSM to do all node virtualization related tasks, including but not limited to storage management. libstoragemgmt aims to provide vendor agnostic API for managing external storage array. It should help system administrators utilizing open source solutions have a way to programmatically manage their storage hardware in a vendor neutral way. It also aims to facilitate management automation, ease of use and take advantage of storage vendor supported features which improve storage performance and space utilization. Home Page: http://sourceforge.net/apps/trac/libstoragemgmt/ libstoragemgmt (LSM) today supports C and python plugins for talking to external storage array using SMI-S as well as native interfaces (eg: netapp plugin ) Plan is to grow the SMI-S interface as needed over time and add more vendor specific plugins for exploiting features not possible via SMI-S or have better alternatives than using SMI-S. For eg: Many of the copy offload features require to use vendor specific commands, which justifies the need for a vendor specific plugin. 2) Goals: 2a) Ability to plugin external storage array into oVirt/VDSM virtualization stack, in a vendor neutral way. 2b) Ability to list features/capabilities and other statistical info of the array 2c) Ability to utilize the storage array offload capabilities from oVirt/VDSM. 3) Details: LSM will sit as a new repository engine in VDSM. VDSM Repository Engine WIP @ http://gerrit.ovirt.org/#change,192 Current plan is to have LSM co-exist with VDSM on the virtualization nodes. *Note : 'storage' used below is generic. It can be a file/nfs-export for NAS targets and LUN/logical-drive for SAN targets. VDSM can use LSM and do the following... - Provision storage - Consume storage 3.1) Provisioning Storage using LSM Typically this will be done by a Storage administrator. oVirt/VDSM should provide storage admin the - ability to list the different storage arrays along with their types (NAS/SAN), capabilities, free/used space. - ability to provision storage using any of the array capabilities (eg: thin provisioned lun or new NFS export ) - ability to manage the provisioned storage
Re: [vdsm] [Libstoragemgmt-devel] [Engine-devel] RFC: Writeup on VDSM-libstoragemgmt integration
On 06/25/2012 09:14 AM, Deepak C Shetty wrote: On 06/25/2012 07:47 AM, Shu Ming wrote: I think VDSM-libstoragemgmt will let the storage array itself to make the snapshot and handle the coordination of the various atomic functions. VDSM should be blocked on the following access to the specific luns which are under snapshotting. I kind of agree. If snapshot is being done at the array level, then the array takes care of quiesing the I/O, taking the snapshot and allowing the I/O, why does VDSM have to worry about anything here, it should all happen transparently for VDSM, isnt it ? The array can take a snapshot in flight, but the data may be in an inconsistent state. Only the end application/user of the storage knows when a point in time is consistent. Typically the application(s) are quiesced, the OS buffers flushed (outstanding tagged IO is allowed to complete) and then the storage is told to make a point in time copy. This is the only way to be sure of what you have on disk is coherent. A transactional database (two-phase commit) and logging file systems (meta data) are specifically written to handle these inconsistencies, but many applications are not. Regards, Tony ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] [Engine-devel] RFC: Writeup on VDSM-libstoragemgmt integration
On 06/25/2012 08:28 PM, Ryan Harper wrote: * Andrew Cathrowacath...@redhat.com [2012-06-24 21:11]: - Original Message - From: Andy Groveragro...@redhat.com To: Shu Mingshum...@linux.vnet.ibm.com Cc: libstoragemgmt-de...@lists.sourceforge.net, engine-de...@ovirt.org, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Sunday, June 24, 2012 10:05:45 PM Subject: Re: [vdsm] [Engine-devel] RFC: Writeup on VDSM-libstoragemgmt integration On 06/24/2012 07:28 AM, Shu Ming wrote: On 2012-6-23 20:40, Itamar Heim wrote: On 06/23/2012 03:09 AM, Andy Grover wrote: On 06/22/2012 04:46 PM, Itamar Heim wrote: On 06/23/2012 02:31 AM, Andy Grover wrote: On 06/18/2012 01:15 PM, Saggi Mizrahi wrote: Also, there is no mention on credentials in any part of the process. How does VDSM or the host get access to actually modify the storage array? Who holds the creds for that and how? How does the user set this up? It seems to me more natural to have the oVirt-engine use libstoragemgmt directly to allocate and export a volume on the storage array, and then pass this info to the vdsm on the node creating the vm. This answers Saggi's question about creds -- vdsm never needs array modification creds, it only gets handed the params needed to connect and use the new block device (ip, iqn, chap, lun). Is this usage model made difficult or impossible by the current software architecture? what about live snapshots? I'm not a virt guy, so extreme handwaving: vm X uses luns 1 2 engine - vdsm pause vm X that's pausing the VM. live snapshot isn't supposed to do so. Tough we don't expect to do a pausing operation to the VM when live snaphot is undergoing, the VM should be blocked on the access to specific luns for a while. The blocking time should be very short to avoid the storage IO time out in the VM. OK my mistake, we don't pause the VM during live snapshot, we block on access to the luns while snapshotting. Does this keep live snapshots working and mean ovirt-engine can use libsm to config the storage array instead of vdsm? Because that was really my main question, should we be talking about engine-libstoragemgmt integration rather than vdsm-libstoragemgmt integration. for snapshotting wouldn't we want VDSM to handle the coordination of the various atomic functions? Absolutely. Requiring every management application (engine, etc) to integrate with libstoragemanagement is a win here. We want to simplify working with KVM, storage, etc not require every mgmt application to know deep details about how to create a live VM snapshot. Sorry, but not clear to me. Are you saying engine-libstoragemgmt integration is a win here ? VDSM is the common factor here.. so integrating libstoragemgmt with VDSM helps anybody talkign with VDSM in future AFAIU. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] [Libstoragemgmt-devel] [Engine-devel] RFC: Writeup on VDSM-libstoragemgmt integration
On 06/25/2012 08:38 PM, Tony Asleson wrote: On 06/25/2012 09:14 AM, Deepak C Shetty wrote: On 06/25/2012 07:47 AM, Shu Ming wrote: I think VDSM-libstoragemgmt will let the storage array itself to make the snapshot and handle the coordination of the various atomic functions. VDSM should be blocked on the following access to the specific luns which are under snapshotting. I kind of agree. If snapshot is being done at the array level, then the array takes care of quiesing the I/O, taking the snapshot and allowing the I/O, why does VDSM have to worry about anything here, it should all happen transparently for VDSM, isnt it ? The array can take a snapshot in flight, but the data may be in an inconsistent state. Only the end application/user of the storage knows when a point in time is consistent. Typically the application(s) are quiesced, the OS buffers flushed (outstanding tagged IO is allowed to complete) and then the storage is told to make a point in time copy. This is the only way to be sure of what you have on disk is coherent. A transactional database (two-phase commit) and logging file systems (meta data) are specifically written to handle these inconsistencies, but many applications are not. Thanks for clarifying Tony. So that means we need to do whatever from VDSM to quiese the I/O and then VDSM should instruct the array to take the snapshot. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] [Engine-devel] RFC: Writeup on VDSM-libstoragemgmt integration
* Deepak C Shetty deepa...@linux.vnet.ibm.com [2012-06-25 10:14]: On 06/25/2012 08:28 PM, Ryan Harper wrote: * Andrew Cathrowacath...@redhat.com [2012-06-24 21:11]: - Original Message - From: Andy Groveragro...@redhat.com To: Shu Mingshum...@linux.vnet.ibm.com Cc: libstoragemgmt-de...@lists.sourceforge.net, engine-de...@ovirt.org, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Sunday, June 24, 2012 10:05:45 PM Subject: Re: [vdsm] [Engine-devel] RFC: Writeup on VDSM-libstoragemgmt integration On 06/24/2012 07:28 AM, Shu Ming wrote: On 2012-6-23 20:40, Itamar Heim wrote: On 06/23/2012 03:09 AM, Andy Grover wrote: On 06/22/2012 04:46 PM, Itamar Heim wrote: On 06/23/2012 02:31 AM, Andy Grover wrote: On 06/18/2012 01:15 PM, Saggi Mizrahi wrote: Also, there is no mention on credentials in any part of the process. How does VDSM or the host get access to actually modify the storage array? Who holds the creds for that and how? How does the user set this up? It seems to me more natural to have the oVirt-engine use libstoragemgmt directly to allocate and export a volume on the storage array, and then pass this info to the vdsm on the node creating the vm. This answers Saggi's question about creds -- vdsm never needs array modification creds, it only gets handed the params needed to connect and use the new block device (ip, iqn, chap, lun). Is this usage model made difficult or impossible by the current software architecture? what about live snapshots? I'm not a virt guy, so extreme handwaving: vm X uses luns 1 2 engine - vdsm pause vm X that's pausing the VM. live snapshot isn't supposed to do so. Tough we don't expect to do a pausing operation to the VM when live snaphot is undergoing, the VM should be blocked on the access to specific luns for a while. The blocking time should be very short to avoid the storage IO time out in the VM. OK my mistake, we don't pause the VM during live snapshot, we block on access to the luns while snapshotting. Does this keep live snapshots working and mean ovirt-engine can use libsm to config the storage array instead of vdsm? Because that was really my main question, should we be talking about engine-libstoragemgmt integration rather than vdsm-libstoragemgmt integration. for snapshotting wouldn't we want VDSM to handle the coordination of the various atomic functions? Absolutely. Requiring every management application (engine, etc) to integrate with libstoragemanagement is a win here. We want to simplify working with KVM, storage, etc not require every mgmt application to know deep details about how to create a live VM snapshot. Sorry, but not clear to me. Are you saying engine-libstoragemgmt integration is a win here ? Sorry if I wasn't clear. To answer your question: No. The mgmt app should *NOT* have to learn all of the ins and outs of the end-point storage and the management of it. VDSM is the common factor here.. so integrating libstoragemgmt with VDSM helps anybody talkign with VDSM in future AFAIU. Yes. 100% agree. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx ry...@us.ibm.com ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] [Engine-devel] RFC: Writeup on VDSM-libstoragemgmt integration
On 06/25/2012 08:17 AM, Ryan Harper wrote: * Deepak C Shetty deepa...@linux.vnet.ibm.com [2012-06-25 10:14]: On 06/25/2012 08:28 PM, Ryan Harper wrote: The mgmt app should *NOT* have to learn all of the ins and outs of the end-point storage and the management of it. VDSM is the common factor here.. so integrating libstoragemgmt with VDSM helps anybody talkign with VDSM in future AFAIU. Yes. 100% agree. Thanks, this has helped me understand vdsm's role much better. -- Andy ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] [Engine-devel] RFC: Writeup on VDSM-libstoragemgmt integration
On 06/25/2012 10:14 AM, Deepak C Shetty wrote: On 06/25/2012 07:47 AM, Shu Ming wrote: On 2012-6-25 10:10, Andrew Cathrow wrote: - Original Message - From: Andy Grover agro...@redhat.com To: Shu Ming shum...@linux.vnet.ibm.com Cc: libstoragemgmt-de...@lists.sourceforge.net, engine-de...@ovirt.org, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Sunday, June 24, 2012 10:05:45 PM Subject: Re: [vdsm] [Engine-devel] RFC: Writeup on VDSM-libstoragemgmt integration On 06/24/2012 07:28 AM, Shu Ming wrote: On 2012-6-23 20:40, Itamar Heim wrote: On 06/23/2012 03:09 AM, Andy Grover wrote: On 06/22/2012 04:46 PM, Itamar Heim wrote: On 06/23/2012 02:31 AM, Andy Grover wrote: On 06/18/2012 01:15 PM, Saggi Mizrahi wrote: Also, there is no mention on credentials in any part of the process. How does VDSM or the host get access to actually modify the storage array? Who holds the creds for that and how? How does the user set this up? It seems to me more natural to have the oVirt-engine use libstoragemgmt directly to allocate and export a volume on the storage array, and then pass this info to the vdsm on the node creating the vm. This answers Saggi's question about creds -- vdsm never needs array modification creds, it only gets handed the params needed to connect and use the new block device (ip, iqn, chap, lun). Is this usage model made difficult or impossible by the current software architecture? what about live snapshots? I'm not a virt guy, so extreme handwaving: vm X uses luns 1 2 engine - vdsm pause vm X that's pausing the VM. live snapshot isn't supposed to do so. Tough we don't expect to do a pausing operation to the VM when live snaphot is undergoing, the VM should be blocked on the access to specific luns for a while. The blocking time should be very short to avoid the storage IO time out in the VM. OK my mistake, we don't pause the VM during live snapshot, we block on access to the luns while snapshotting. Does this keep live snapshots working and mean ovirt-engine can use libsm to config the storage array instead of vdsm? Because that was really my main question, should we be talking about engine-libstoragemgmt integration rather than vdsm-libstoragemgmt integration. for snapshotting wouldn't we want VDSM to handle the coordination of the various atomic functions? I think VDSM-libstoragemgmt will let the storage array itself to make the snapshot and handle the coordination of the various atomic functions. VDSM should be blocked on the following access to the specific luns which are under snapshotting. I kind of agree. If snapshot is being done at the array level, then the array takes care of quiesing the I/O, taking the snapshot and allowing the I/O, why does VDSM have to worry about anything here, it should all happen transparently for VDSM, isnt it ? I may be misssing something, but afaiu you need to ask the guest to perform the quiesce, and i'm sure the storage array can't do that. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] [virt-node] VDSM as a general purpose virt host manager
On Mon, Jun 25, 2012 at 05:53:31PM +0300, Dan Kenigsberg wrote: On Mon, Jun 25, 2012 at 08:28:29AM -0500, Adam Litke wrote: On Fri, Jun 22, 2012 at 06:45:43PM -0400, Andrew Cathrow wrote: - Original Message - From: Ryan Harper ry...@us.ibm.com To: Adam Litke a...@us.ibm.com Cc: Anthony Liguori aligu...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Friday, June 22, 2012 12:45:42 PM Subject: Re: [vdsm] [virt-node] VDSM as a general purpose virt host manager * Adam Litke a...@us.ibm.com [2012-06-22 11:35]: On Thu, Jun 21, 2012 at 12:17:19PM +0300, Dor Laor wrote: On 06/19/2012 08:12 PM, Saggi Mizrahi wrote: - Original Message - From: Deepak C Shetty deepa...@linux.vnet.ibm.com To: Ryan Harper ry...@us.ibm.com Cc: Saggi Mizrahi smizr...@redhat.com, Anthony Liguori aligu...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Tuesday, June 19, 2012 10:58:47 AM Subject: Re: [vdsm] [virt-node] VDSM as a general purpose virt host manager On 06/19/2012 01:13 AM, Ryan Harper wrote: * Saggi Mizrahismizr...@redhat.com [2012-06-18 10:05]: I would like to put on to the table for descussion the growing need for a way to more easily reuse of the functionality of VDSM in order to service projects other than Ovirt-Engine. Originally VDSM was created as a proprietary agent for the sole purpose of serving the then proprietary version of what is known as ovirt-engine. Red Hat, after acquiring the technology, pressed on with it's commitment to open source ideals and released the code. But just releasing code into the wild doesn't build a community or makes a project successful. Further more when building open source software you should aspire to build reusable components instead of monolithic stacks. Saggi, Thanks for sending this out. I've been trying to pull together some thoughts on what else is needed for vdsm as a community. I know that for some time downstream has been the driving force for all of the work and now with a community there are challenges in finding our own way. While we certainly don't want to make downstream efforts harder, I think we need to develop and support our own vision for what vdsm can be come, some what independent of downstream and other exploiters. Revisiting the API is definitely a much needed endeavor and I think adding some use-cases or sample applications would be useful in demonstrating whether or not we're evolving the API into something easier to use for applications beyond engine. We would like to expose a stable, documented, well supported API. This gives us a chance to rethink the VDSM API from the ground up. There is already work in progress of making the internal logic of VDSM separate enough from the API layer so we could continue feature development and bug fixing while designing the API of the future. In order to achieve this though we need to do several things: 1. Declare API supportability guidelines 2. Decide on an API transport (e.g. REST, ZMQ, AMQP) 3. Make the API easily consumable (e.g. proper docs, example code, extending the API, etc) 4. Implement the API itself In the earlier we'd discussed working to have similarities in the modeling between the oVirt API and VDSM but that seems to have dropped off the radar. Yes, the current REST API has attempted to be compatible with the current ovirt-engine API. Going forward, I am not sure how easy this will be to maintain given than engine is built on Java and vdsm is built on Python. Could you elaborate why the language difference is an issue? Isn't this what APIs are supposed to solve? The main language issue is that ovirt-engine has built their API using a set of Java-specific frameworks (JAXB and its dependents). It's true, if you google for 'python jaxb' you will find some sourceforge projects that attempt to bring the jaxb interface to python but I don't think that's the right approach. If you're writing a java project, do things the java way. If you're writing a python project, do them the python way. Right now I am focused on defining the current API (API.py/xmlrpc) mechanically (creating a schema and API documentation). XSD is not the correct language for that task (which is why I forsee a divergence at least at first). I want to take a stab at defining the API in
Re: [vdsm] [virt-node] VDSM as a general purpose virt host manager
On 06/26/2012 05:19 AM, Adam Litke wrote: On Mon, Jun 25, 2012 at 05:53:31PM +0300, Dan Kenigsberg wrote: On Mon, Jun 25, 2012 at 08:28:29AM -0500, Adam Litke wrote: On Fri, Jun 22, 2012 at 06:45:43PM -0400, Andrew Cathrow wrote: - Original Message - From: Ryan Harper ry...@us.ibm.com To: Adam Litke a...@us.ibm.com Cc: Anthony Liguori aligu...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Friday, June 22, 2012 12:45:42 PM Subject: Re: [vdsm] [virt-node] VDSM as a general purpose virt host manager * Adam Litke a...@us.ibm.com [2012-06-22 11:35]: On Thu, Jun 21, 2012 at 12:17:19PM +0300, Dor Laor wrote: On 06/19/2012 08:12 PM, Saggi Mizrahi wrote: - Original Message - From: Deepak C Shetty deepa...@linux.vnet.ibm.com To: Ryan Harper ry...@us.ibm.com Cc: Saggi Mizrahi smizr...@redhat.com, Anthony Liguori aligu...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Tuesday, June 19, 2012 10:58:47 AM Subject: Re: [vdsm] [virt-node] VDSM as a general purpose virt host manager On 06/19/2012 01:13 AM, Ryan Harper wrote: * Saggi Mizrahismizr...@redhat.com [2012-06-18 10:05]: I would like to put on to the table for descussion the growing need for a way to more easily reuse of the functionality of VDSM in order to service projects other than Ovirt-Engine. Originally VDSM was created as a proprietary agent for the sole purpose of serving the then proprietary version of what is known as ovirt-engine. Red Hat, after acquiring the technology, pressed on with it's commitment to open source ideals and released the code. But just releasing code into the wild doesn't build a community or makes a project successful. Further more when building open source software you should aspire to build reusable components instead of monolithic stacks. Saggi, Thanks for sending this out. I've been trying to pull together some thoughts on what else is needed for vdsm as a community. I know that for some time downstream has been the driving force for all of the work and now with a community there are challenges in finding our own way. While we certainly don't want to make downstream efforts harder, I think we need to develop and support our own vision for what vdsm can be come, some what independent of downstream and other exploiters. Revisiting the API is definitely a much needed endeavor and I think adding some use-cases or sample applications would be useful in demonstrating whether or not we're evolving the API into something easier to use for applications beyond engine. We would like to expose a stable, documented, well supported API. This gives us a chance to rethink the VDSM API from the ground up. There is already work in progress of making the internal logic of VDSM separate enough from the API layer so we could continue feature development and bug fixing while designing the API of the future. In order to achieve this though we need to do several things: 1. Declare API supportability guidelines 2. Decide on an API transport (e.g. REST, ZMQ, AMQP) 3. Make the API easily consumable (e.g. proper docs, example code, extending the API, etc) 4. Implement the API itself In the earlier we'd discussed working to have similarities in the modeling between the oVirt API and VDSM but that seems to have dropped off the radar. Yes, the current REST API has attempted to be compatible with the current ovirt-engine API. Going forward, I am not sure how easy this will be to maintain given than engine is built on Java and vdsm is built on Python. Could you elaborate why the language difference is an issue? Isn't this what APIs are supposed to solve? The main language issue is that ovirt-engine has built their API using a set of Java-specific frameworks (JAXB and its dependents). It's true, if you google for 'python jaxb' you will find some sourceforge projects that attempt to bring the jaxb interface to python but I don't think that's the right approach. If you're writing a java project, do things the java way. If you're writing a python project, do them the python way. Right now I am focused on defining the current API (API.py/xmlrpc) mechanically (creating a schema and API documentation). XSD is not the correct language for that task (which is why I forsee a divergence at least at first). I want to take a stab at defining the API in a beneficial, long-term manner. 1) Completely define the current XMLRPC API including all functions, parameters, and return values. Complex data structures can be broken down into their basic types. These are: int, str, bool, list, dict, typed-dict, enum. I have already started this process and am using Qemu's QAPI schema language. You can see that here [1]. For an example of what that looks like describing the vdsm API see this snippet [2]. 2) Import the parser/generator code from qemu for the above schema. Vdsm will require a few extensions such as
