Re: [ovirt-devel] [vdsm] Engine XML: metadata and devices from XML

[Francesco Romani]

Please note that the approach described here is outdated, because
xmlpickle was too easy to use

and too generic: that lead to bloated XML and too high risk of misusing
metadata;

the new approach (https://gerrit.ovirt.org/#/c/74206/) tries to balance
things making usage

convenient but disallowing arbitrarily nesting


On 03/18/2017 03:27 PM, Nir Soffer wrote:
> On Wed, Mar 15, 2017 at 2:28 PM Francesco Romani  > wrote:
>
> Hi everyone,
>
> This is both a report of the current state of my Vdsm patches for
> Engine
> XML support, and a proposal to how move forward and solve
> the current open issues.
>
> TL;DR:
> 1. we can and IMO should reuse the current JSON schema to describe the
> structure (layout) and the types of the metadata section.
> 2. we don't need a priori validation of stuff in the metadata section.
> We will just raise in the creation flow if data is missing, or wrong,
> according to our schema.
> 2. we will add *few* items to the metadata section, only thing we
> can't
> express clearly-or at all in the libvirt XML. Redundancy and
> verbosiness
> will be thus kept at bay
> 3. I believe [3] is the best tool to do (de)serialize data to the
> metadata section. Existing tools fits poorly in our very specific
> use case
>
> Examples below
>
> +++
>
> Long(er) discussion:
>
>
> I have working code[1][2] to encode any custom, picklable, python
> object in the metadata section.
>
> We should decide which module will do the actual python<=>XML
> transformation.
> Please note that this actually also influences how the data in the
> medata section look like, so the two things are a bit coupled.
>
> I'm eager to reinvent another wheel, but after
> initial evaluation I honestly think that my pyxmlpickle[3] is the best
> tool for the job over the current alternatives: plistlib[4] and
> xmltodict[5].
>
> I added the initial rationale here:
> https://gerrit.ovirt.org/#/c/73790/4//COMMIT_MSG
>
> I have completed the initial draft of patches to make it possible to
> initialize devices from their XML representation [6]. This is the bare
> minimum we need to support the Engine XML, and we *need* this
> anyway to
> unlock the cleanup we planned and I outlined in my google doc.
>
> So we are progressing, but I'd like to speed up things. Those [6]
> patches are not yet complete, many flows are not covered or
> tested;  but
> they are good enough to demonstrate that there *are* pieces of
> information wen need to properly initialize the devices, but we can't
> easily extract from the XML.
>
> First examples that come to my mind are the storage.Drive UUIDs; there
> could also be some ambiguity I'm investigating right now for
> displayIp/displayNetwork in Graphics devices. In [6] there are various
> TODO to mark more of those cases. Most likely, few more cases will pop
> out as I cover all the flows we support.
>
> Long story short: it is hard to correctly rebuild the device conf from
> the XML. This is why in [6] I added the 'meta' argument to
> from_xml_tree
> classmethod in [7].
>
> 'meta' is supposed to be the device metadata: extra data related to a
> device which doesn't (yet) fit in the libvirt XML representation.
> For example, we can store 'displayIp' and 'displayNetwork' here and be
> done with that: using both per-device metadata and the XML
> representation of one graphic device, we will have everything we
> need to
> properly build one graphics.Graphics device.
> This example may (hopefully) be bogus, but I'm keeping it because
> it is
> one case easy to follow.
>
> The device metadata is going to be stored in the vm metadata for the
> short/mid term future. Even if the per-device metadata idea/RFE is
> accepted (no answer yet, but we are working on it), we will not
> have in
> 7.4, and unlikely in 7.5.
>
> As it stands today, I believe there are two open questions:
>
> 1. do we need a schema for the metadata section?
> 2. how do we bind the metadata to the devices? How do we know which
> metadata belongs to which metadata, if we don't have aliases nor
> addresses to match? (e.g. very first time the VM is created!)
>
> My current stance is the following
> 1. In general, one schema gives us two benefits: 1.a. we document how
> the layout of the data should be, including types; 1.b. we can
> validate
> the data we receive.
> So yes, we need a schema, but we don't need a *new* schema. I think we
> are in good enough shape with the current Vdsm schema: we can just
> translate the python object layout to a XML layout.
>
> One example is probably more explicative. Some actual data may look
> like, using my pyxmlpickle module:
>
> 
>   a0
>  

Re: [ovirt-devel] [vdsm] Engine XML: metadata and devices from XML

[Francesco Romani]

On 03/18/2017 01:14 PM, Nir Soffer wrote:
>
>
> On Fri, Mar 17, 2017 at 4:58 PM Francesco Romani  > wrote:
>
> On 03/16/2017 08:03 PM, Francesco Romani wrote:
> > On 03/16/2017 01:26 PM, Francesco Romani wrote:
> >> On 03/16/2017 11:47 AM, Michal Skrivanek wrote:
>  On 16 Mar 2017, at 09:45, Francesco Romani
> mailto:from...@redhat.com>> wrote:
> 
>  We talked about sending storage device purely on metadata,
> letting Vdsm
>  rebuild them and getting the XML like today.
> 
>  In the other direction, Vdsm will pass through the XML
> (perhaps only
>  parts of it, e.g. the devices subtree) like before.
> 
>  This way we can minimize the changes we are uncertain of, and
> more
>  importantly, we can minimize the risky changes.
> 
> 
>  The following is  a realistic example of how the XML could
> look like if
>  we send all but the storage devices. It is built using my
> pyxmlpickle
>  module (see [3] below).
> >>> That’s quite verbose. How much work would it need to actually
> minimize it and turn it into something more simple.
> >>> Most such stuff should go away and I believe it would be
> beneficial to make it difficult to use to discourage using
> metadata as a generic junkyard
> >> It is verbose because it is generic - indeed perhaps too generic.
> >> I can try something else based on a concept from Martin
> Polednik. Will
> >> follow up soon.
> > Early preview:
> >
> 
> https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:virt-metadata-compact
> >
> > still plenty of TODOs, I expect to be reviewable material worst case
> > monday morning.
>
> This is how typical XML could look like:
>
> 
> 
> 
> 
>
>
> Why do we need this nesting?

We use libvirt metadata for elements; so each direct child of metadata
is a separate metadata grop:

ovirt-tune:qos is one
ovirt-vm:vm is one

and so forth.  I don't want to mess up with the existing elements. So we
could be backward compatible at XML level (read: 4.1 XML works without
changes to 4.2)


>  
>
> 
>
>
> What is ovirt-instance?

Gone, merged with ovirt-vm

ovirt-vm will hold both per-vm metadata and per-device metadata.

>  
>
> 
> true
> true
> 192.168.1.51
> en-us
> DEFAULT
>
> 
> smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir
> ovirtmgmt
> 
> 
> 
> 
>
>
> Why do we need this nesting?

*Here* we had this nesting because the example took vm.conf and brutally
translated to XML. Is a worst case scenario.

Why is it relevant? There it was initial discussion about how to deal
with complex device; to minimize changes, we could
marshal the existing vm.conf into the device metadata, then unmarshal on
Vdsm side and just use it to rebuild the devices
with the very same code we have today (yes, this means
sneaking/embedding vm.conf into the XML)

Should we go that way, it could look like the above.


Let's talk in general now. There are three main use cases requiring nesting:

1. per-vm metadta. Attach key/value pairs. We need one level of nesting
to avoid to mess up with other data. So it could look like

 
  
1
2
  
 

this is simple and nice and I think is not bothering anyone (hopefully :))

2. per-device metadata: it has to fit into vm section, and we could
possibly have more than one device with metadata, so the simplest format
is something like

 
  
1
2

  
true
false
  
 
  
 

This is the minimal nesting level we need. We could gather the
per-device metadata in a dict and feed device with it, like with a new
"meta" argument to device constructor, much like "custom" and "specParams"

Would that look good?


3. QoS. we need to support the current layout for obvious backward
compatibility questions. We could postpone this and use existing code
for some more time, but ultimately this should handled by metadata
module, just because it is supposed to be
the process-wide metadata gateway.

>  
>
> 
> 
> 
> 
> 
>
> c578566d-bc61-420c-8f1e-8dfa0a18efd5
>
>
> We are using sd_id, img_id and vol_id now for these in new storage code.

Noted

>  
>
>
> 5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc
> 5890a292-0390-01d2-01ed-029a
>
> 66441539-f7ac-4946-8a25-75e422f939d4
>
>
> We need more info about disks, like diskType, shared, discard, etc.
>
> Can you show a comple

Re: [ovirt-devel] [vdsm] Engine XML: metadata and devices from XML

[Francesco Romani]

On 03/20/2017 09:05 AM, Francesco Romani wrote:
> On 03/17/2017 11:07 PM, Michal Skrivanek wrote:
>>> On 17 Mar 2017, at 15:57, Francesco Romani  wrote:
>>>
>>> On 03/16/2017 08:03 PM, Francesco Romani wrote:
 On 03/16/2017 01:26 PM, Francesco Romani wrote:
> On 03/16/2017 11:47 AM, Michal Skrivanek wrote:
>>> On 16 Mar 2017, at 09:45, Francesco Romani  wrote:
>>>
>>> We talked about sending storage device purely on metadata, letting Vdsm
>>> rebuild them and getting the XML like today.
>>>
>>> In the other direction, Vdsm will pass through the XML (perhaps only
>>> parts of it, e.g. the devices subtree) like before.
>>>
>>> This way we can minimize the changes we are uncertain of, and more
>>> importantly, we can minimize the risky changes.
>>>
>>>
>>> The following is  a realistic example of how the XML could look like if
>>> we send all but the storage devices. It is built using my pyxmlpickle
>>> module (see [3] below).
>> That’s quite verbose. How much work would it need to actually minimize 
>> it and turn it into something more simple.
>> Most such stuff should go away and I believe it would be beneficial to 
>> make it difficult to use to discourage using metadata as a generic 
>> junkyard
> It is verbose because it is generic - indeed perhaps too generic.
> I can try something else based on a concept from Martin Polednik. Will
> follow up soon.
 Early preview:
 https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:virt-metadata-compact

 still plenty of TODOs, I expect to be reviewable material worst case
 monday morning.
>>> This is how typical XML could look like:
>>>
>>>
>>>
>>>
>>>
>>>
>> not under the ?
>> any reason?
> No reason, I'll move under it
>

Unfortunately we need to have the prefix for all the elements, not just
for the top-level one. Updating.

-- 
Francesco Romani
Red Hat Engineering Virtualization R & D
IRC: fromani

___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [vdsm] Engine XML: metadata and devices from XML

[Francesco Romani]

On 03/17/2017 11:07 PM, Michal Skrivanek wrote:
>> On 17 Mar 2017, at 15:57, Francesco Romani  wrote:
>>
>> On 03/16/2017 08:03 PM, Francesco Romani wrote:
>>> On 03/16/2017 01:26 PM, Francesco Romani wrote:
 On 03/16/2017 11:47 AM, Michal Skrivanek wrote:
>> On 16 Mar 2017, at 09:45, Francesco Romani  wrote:
>>
>> We talked about sending storage device purely on metadata, letting Vdsm
>> rebuild them and getting the XML like today.
>>
>> In the other direction, Vdsm will pass through the XML (perhaps only
>> parts of it, e.g. the devices subtree) like before.
>>
>> This way we can minimize the changes we are uncertain of, and more
>> importantly, we can minimize the risky changes.
>>
>>
>> The following is  a realistic example of how the XML could look like if
>> we send all but the storage devices. It is built using my pyxmlpickle
>> module (see [3] below).
> That’s quite verbose. How much work would it need to actually minimize it 
> and turn it into something more simple.
> Most such stuff should go away and I believe it would be beneficial to 
> make it difficult to use to discourage using metadata as a generic 
> junkyard
 It is verbose because it is generic - indeed perhaps too generic.
 I can try something else based on a concept from Martin Polednik. Will
 follow up soon.
>>> Early preview:
>>> https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:virt-metadata-compact
>>>
>>> still plenty of TODOs, I expect to be reviewable material worst case
>>> monday morning.
>> This is how typical XML could look like:
>>
>>
>>
>>
>>
>>
> not under the ?
> any reason?

No reason, I'll move under it

Bests,

-- 
Francesco Romani
Red Hat Engineering Virtualization R & D
IRC: fromani

___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [vdsm] Engine XML: metadata and devices from XML

[Nir Soffer]

On Wed, Mar 15, 2017 at 2:28 PM Francesco Romani  wrote:

> Hi everyone,
>
> This is both a report of the current state of my Vdsm patches for Engine
> XML support, and a proposal to how move forward and solve
> the current open issues.
>
> TL;DR:
> 1. we can and IMO should reuse the current JSON schema to describe the
> structure (layout) and the types of the metadata section.
> 2. we don't need a priori validation of stuff in the metadata section.
> We will just raise in the creation flow if data is missing, or wrong,
> according to our schema.
> 2. we will add *few* items to the metadata section, only thing we can't
> express clearly-or at all in the libvirt XML. Redundancy and verbosiness
> will be thus kept at bay
> 3. I believe [3] is the best tool to do (de)serialize data to the
> metadata section. Existing tools fits poorly in our very specific use case
>
> Examples below
>
> +++
>
> Long(er) discussion:
>
>
> I have working code[1][2] to encode any custom, picklable, python
> object in the metadata section.
>
> We should decide which module will do the actual python<=>XML
> transformation.
> Please note that this actually also influences how the data in the
> medata section look like, so the two things are a bit coupled.
>
> I'm eager to reinvent another wheel, but after
> initial evaluation I honestly think that my pyxmlpickle[3] is the best
> tool for the job over the current alternatives: plistlib[4] and
> xmltodict[5].
>
> I added the initial rationale here:
> https://gerrit.ovirt.org/#/c/73790/4//COMMIT_MSG
>
> I have completed the initial draft of patches to make it possible to
> initialize devices from their XML representation [6]. This is the bare
> minimum we need to support the Engine XML, and we *need* this anyway to
> unlock the cleanup we planned and I outlined in my google doc.
>
> So we are progressing, but I'd like to speed up things. Those [6]
> patches are not yet complete, many flows are not covered or tested;  but
> they are good enough to demonstrate that there *are* pieces of
> information wen need to properly initialize the devices, but we can't
> easily extract from the XML.
>
> First examples that come to my mind are the storage.Drive UUIDs; there
> could also be some ambiguity I'm investigating right now for
> displayIp/displayNetwork in Graphics devices. In [6] there are various
> TODO to mark more of those cases. Most likely, few more cases will pop
> out as I cover all the flows we support.
>
> Long story short: it is hard to correctly rebuild the device conf from
> the XML. This is why in [6] I added the 'meta' argument to from_xml_tree
> classmethod in [7].
>
> 'meta' is supposed to be the device metadata: extra data related to a
> device which doesn't (yet) fit in the libvirt XML representation.
> For example, we can store 'displayIp' and 'displayNetwork' here and be
> done with that: using both per-device metadata and the XML
> representation of one graphic device, we will have everything we need to
> properly build one graphics.Graphics device.
> This example may (hopefully) be bogus, but I'm keeping it because it is
> one case easy to follow.
>
> The device metadata is going to be stored in the vm metadata for the
> short/mid term future. Even if the per-device metadata idea/RFE is
> accepted (no answer yet, but we are working on it), we will not have in
> 7.4, and unlikely in 7.5.
>
> As it stands today, I believe there are two open questions:
>
> 1. do we need a schema for the metadata section?
> 2. how do we bind the metadata to the devices? How do we know which
> metadata belongs to which metadata, if we don't have aliases nor
> addresses to match? (e.g. very first time the VM is created!)
>
> My current stance is the following
> 1. In general, one schema gives us two benefits: 1.a. we document how
> the layout of the data should be, including types; 1.b. we can validate
> the data we receive.
> So yes, we need a schema, but we don't need a *new* schema. I think we
> are in good enough shape with the current Vdsm schema: we can just
> translate the python object layout to a XML layout.
>
> One example is probably more explicative. Some actual data may look
> like, using my pyxmlpickle module:
>
> 
>   a0
>   ccd945c8-8069-4f31-8471-bbb58e9dd6ea
>   http://ovirt.org/vm/tune/1.0";
> xmlns:ovirt-vm="http://ovirt.org/vm/1.0";
> xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0";>
> 
> 
> 
>   
> 
>   
>  type="str">vnc
> 
>type="str">ovirtmgmt
>type="str">192.168.1.53
> 
>  type="str">graphics
>   
>   
>  type="str">spice
> 
>type="str">ovirtmgmt
>type="str">192.168.1.53
> 
>  type="str">graphics
>   
>   
>  type="str">5890a292-0390-01d2-01ed-029a
>  type="str">66441539-f7ac-4946-8a25-75e422f939d4
>  

Re: [ovirt-devel] [vdsm] Engine XML: metadata and devices from XML

[Nir Soffer]

On Wed, Mar 15, 2017 at 2:28 PM Francesco Romani  wrote:

Hi everyone,

This is both a report of the current state of my Vdsm patches for Engine
XML support, and a proposal to how move forward and solve
the current open issues.

TL;DR:
1. we can and IMO should reuse the current JSON schema to describe the
structure (layout) and the types of the metadata section.
2. we don't need a priori validation of stuff in the metadata section.
We will just raise in the creation flow if data is missing, or wrong,
according to our schema.
2. we will add *few* items to the metadata section, only thing we can't
express clearly-or at all in the libvirt XML. Redundancy and verbosiness
will be thus kept at bay
3. I believe [3] is the best tool to do (de)serialize data to the
metadata section. Existing tools fits poorly in our very specific use case

Examples below

+++

Long(er) discussion:


I have working code[1][2] to encode any custom, picklable, python
object in the metadata section.

We should decide which module will do the actual python<=>XML
transformation.
Please note that this actually also influences how the data in the
medata section look like, so the two things are a bit coupled.

I'm eager to reinvent another wheel, but after
initial evaluation I honestly think that my pyxmlpickle[3] is the best
tool for the job over the current alternatives: plistlib[4] and
xmltodict[5].

I added the initial rationale here:
https://gerrit.ovirt.org/#/c/73790/4//COMMIT_MSG

I have completed the initial draft of patches to make it possible to
initialize devices from their XML representation [6]. This is the bare
minimum we need to support the Engine XML, and we *need* this anyway to
unlock the cleanup we planned and I outlined in my google doc.

So we are progressing, but I'd like to speed up things. Those [6]
patches are not yet complete, many flows are not covered or tested;  but
they are good enough to demonstrate that there *are* pieces of
information wen need to properly initialize the devices, but we can't
easily extract from the XML.

First examples that come to my mind are the storage.Drive UUIDs; there
could also be some ambiguity I'm investigating right now for
displayIp/displayNetwork in Graphics devices. In [6] there are various
TODO to mark more of those cases. Most likely, few more cases will pop
out as I cover all the flows we support.

Long story short: it is hard to correctly rebuild the device conf from
the XML. This is why in [6] I added the 'meta' argument to from_xml_tree
classmethod in [7].

'meta' is supposed to be the device metadata: extra data related to a
device which doesn't (yet) fit in the libvirt XML representation.
For example, we can store 'displayIp' and 'displayNetwork' here and be
done with that: using both per-device metadata and the XML
representation of one graphic device, we will have everything we need to
properly build one graphics.Graphics device.
This example may (hopefully) be bogus, but I'm keeping it because it is
one case easy to follow.

The device metadata is going to be stored in the vm metadata for the
short/mid term future. Even if the per-device metadata idea/RFE is
accepted (no answer yet, but we are working on it), we will not have in
7.4, and unlikely in 7.5.

As it stands today, I believe there are two open questions:

1. do we need a schema for the metadata section?
2. how do we bind the metadata to the devices? How do we know which
metadata belongs to which metadata, if we don't have aliases nor
addresses to match? (e.g. very first time the VM is created!)

My current stance is the following
1. In general, one schema gives us two benefits: 1.a. we document how
the layout of the data should be, including types; 1.b. we can validate
the data we receive.
So yes, we need a schema, but we don't need a *new* schema. I think we
are in good enough shape with the current Vdsm schema: we can just
translate the python object layout to a XML layout.

One example is probably more explicative. Some actual data may look
like, using my pyxmlpickle module:


  a0
  ccd945c8-8069-4f31-8471-bbb58e9dd6ea
  http://ovirt.org/vm/tune/1.0";


The url is broken - do we have this?


xmlns:ovirt-vm="http://ovirt.org/vm/1.0";
xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0";>



What do we keep here? why does it need its own namespace?





What do we keep here? why does it need its own namespace?

Can we merge all namespaces into one generic namespace?





What is instance?


  



  


Isn't index redundant?


vnc

  ovirtmgmt
  192.168.1.53

graphics
  
  
spice

  ovirtmgmt
  192.168.1.53

graphics
  
  
5890a292-0390-01d2-01ed-029a
66441539-f7ac-4946-8a25-75e422f939d4
c578566d-bc61-420c-8f1e-8dfa0a18efd5
disk
/rh

Re: [ovirt-devel] [vdsm] Engine XML: metadata and devices from XML

[Nir Soffer]

On Fri, Mar 17, 2017 at 4:58 PM Francesco Romani  wrote:

> On 03/16/2017 08:03 PM, Francesco Romani wrote:
> > On 03/16/2017 01:26 PM, Francesco Romani wrote:
> >> On 03/16/2017 11:47 AM, Michal Skrivanek wrote:
>  On 16 Mar 2017, at 09:45, Francesco Romani 
> wrote:
> 
>  We talked about sending storage device purely on metadata, letting
> Vdsm
>  rebuild them and getting the XML like today.
> 
>  In the other direction, Vdsm will pass through the XML (perhaps only
>  parts of it, e.g. the devices subtree) like before.
> 
>  This way we can minimize the changes we are uncertain of, and more
>  importantly, we can minimize the risky changes.
> 
> 
>  The following is  a realistic example of how the XML could look like
> if
>  we send all but the storage devices. It is built using my pyxmlpickle
>  module (see [3] below).
> >>> That’s quite verbose. How much work would it need to actually minimize
> it and turn it into something more simple.
> >>> Most such stuff should go away and I believe it would be beneficial to
> make it difficult to use to discourage using metadata as a generic junkyard
> >> It is verbose because it is generic - indeed perhaps too generic.
> >> I can try something else based on a concept from Martin Polednik. Will
> >> follow up soon.
> > Early preview:
> >
> https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:virt-metadata-compact
> >
> > still plenty of TODOs, I expect to be reviewable material worst case
> > monday morning.
>
> This is how typical XML could look like:
>
> 
> 
> 
> 
>

Why do we need this nesting?


> 
>

What is ovirt-instance?

ovirt-tune and ovirt-vm make sense, I can guess what you mean, I don't have
any
idea what is ovirt-instance.


> 
> true
> true
> 192.168.1.51
> en-us
> DEFAULT
>
>
> smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir
> ovirtmgmt
> 
> 
> 
> 
>

Why do we need this nesting?


> 
> 
> 
> 
> 
> c578566d-bc61-420c-8f1e-8dfa0a18efd5
>

We are using sd_id, img_id and vol_id now for these in new storage code.


> 5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc
> 5890a292-0390-01d2-01ed-029a
> 66441539-f7ac-4946-8a25-75e422f939d4
>

We need more info about disks, like diskType, shared, discard, etc.

Can you show a complete vm xml with metadata?


> 
> 
> 
>
> still working on this
>
> --
> Francesco Romani
> Red Hat Engineering Virtualization R & D
> IRC: fromani
>
> ___
> Devel mailing list
> Devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [vdsm] Engine XML: metadata and devices from XML

[Francesco Romani]

On 03/16/2017 08:03 PM, Francesco Romani wrote:
> On 03/16/2017 01:26 PM, Francesco Romani wrote:
>> On 03/16/2017 11:47 AM, Michal Skrivanek wrote:
 On 16 Mar 2017, at 09:45, Francesco Romani  wrote:

 We talked about sending storage device purely on metadata, letting Vdsm
 rebuild them and getting the XML like today.

 In the other direction, Vdsm will pass through the XML (perhaps only
 parts of it, e.g. the devices subtree) like before.

 This way we can minimize the changes we are uncertain of, and more
 importantly, we can minimize the risky changes.


 The following is  a realistic example of how the XML could look like if
 we send all but the storage devices. It is built using my pyxmlpickle
 module (see [3] below).
>>> That’s quite verbose. How much work would it need to actually minimize it 
>>> and turn it into something more simple.
>>> Most such stuff should go away and I believe it would be beneficial to make 
>>> it difficult to use to discourage using metadata as a generic junkyard
>> It is verbose because it is generic - indeed perhaps too generic.
>> I can try something else based on a concept from Martin Polednik. Will
>> follow up soon.
> Early preview:
> https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:virt-metadata-compact
>
> still plenty of TODOs, I expect to be reviewable material worst case
> monday morning.

This is how typical XML could look like:







true
true
192.168.1.51
en-us
DEFAULT
   
smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir
ovirtmgmt









c578566d-bc61-420c-8f1e-8dfa0a18efd5
5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc
5890a292-0390-01d2-01ed-029a
66441539-f7ac-4946-8a25-75e422f939d4




still working on this

-- 
Francesco Romani
Red Hat Engineering Virtualization R & D
IRC: fromani

___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [vdsm] Engine XML: metadata and devices from XML

[Francesco Romani]

On 03/16/2017 01:26 PM, Francesco Romani wrote:
> On 03/16/2017 11:47 AM, Michal Skrivanek wrote:
>>> On 16 Mar 2017, at 09:45, Francesco Romani  wrote:
>>>
>>> We talked about sending storage device purely on metadata, letting Vdsm
>>> rebuild them and getting the XML like today.
>>>
>>> In the other direction, Vdsm will pass through the XML (perhaps only
>>> parts of it, e.g. the devices subtree) like before.
>>>
>>> This way we can minimize the changes we are uncertain of, and more
>>> importantly, we can minimize the risky changes.
>>>
>>>
>>> The following is  a realistic example of how the XML could look like if
>>> we send all but the storage devices. It is built using my pyxmlpickle
>>> module (see [3] below).
>> That’s quite verbose. How much work would it need to actually minimize it 
>> and turn it into something more simple.
>> Most such stuff should go away and I believe it would be beneficial to make 
>> it difficult to use to discourage using metadata as a generic junkyard
> It is verbose because it is generic - indeed perhaps too generic.
> I can try something else based on a concept from Martin Polednik. Will
> follow up soon.

Early preview:
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:virt-metadata-compact

still plenty of TODOs, I expect to be reviewable material worst case
monday morning.


-- 
Francesco Romani
Red Hat Engineering Virtualization R & D
IRC: fromani

___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [vdsm] Engine XML: metadata and devices from XML

[Francesco Romani]

On 03/16/2017 11:47 AM, Michal Skrivanek wrote:
>> On 16 Mar 2017, at 09:45, Francesco Romani  wrote:
>>
>> We talked about sending storage device purely on metadata, letting Vdsm
>> rebuild them and getting the XML like today.
>>
>> In the other direction, Vdsm will pass through the XML (perhaps only
>> parts of it, e.g. the devices subtree) like before.
>>
>> This way we can minimize the changes we are uncertain of, and more
>> importantly, we can minimize the risky changes.
>>
>>
>> The following is  a realistic example of how the XML could look like if
>> we send all but the storage devices. It is built using my pyxmlpickle
>> module (see [3] below).
> That’s quite verbose. How much work would it need to actually minimize it and 
> turn it into something more simple.
> Most such stuff should go away and I believe it would be beneficial to make 
> it difficult to use to discourage using metadata as a generic junkyard

It is verbose because it is generic - indeed perhaps too generic.
I can try something else based on a concept from Martin Polednik. Will
follow up soon.

Bests,

>
>>
>> 
>>  a0
>>  ccd945c8-8069-4f31-8471-bbb58e9dd6ea
>>  http://ovirt.org/vm/tune/1.0";
>> xmlns:ovirt-vm="http://ovirt.org/vm/1.0";
>> xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0";>
>>
>>
>>
>>  
>>
>>  
>>> type="str">2
>>> type="str">ide
>>
>>  
>>
>>> type="str">true
>>> type="str">e59c985c-46c2-4489-b355-a6f374125eb9
>>
>>  > type="str">1
>>  > type="str">0
>>  > type="str">drive
>>  > type="str">0
>>  > type="str">0
>>
>>> type="str">cdrom
>>> type="str">false
>>
>>> type="str">disk
>>  
>>  
>>> type="str">5890a292-0390-01d2-01ed-029a
>>
>>  > type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5
>>  > type="str">path
>>  > type="int">0
>>  > type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc
>>  > type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc
>>  > type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc.lease
>>  > type="str">66441539-f7ac-4946-8a25-75e422f939d4
>>
>>> type="int">0
>>> type="str">virtio
>>> type="str">8589934592
>>> type="str">66441539-f7ac-4946-8a25-75e422f939d4
>>> type="str">false
>>> type="str">false
>>> type="str">0
>>> type="str">disk
>>> type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5
>>> type="str">0
>>> type="str">raw
>>> type="str">66441539-f7ac-4946-8a25-75e422f939d4
>>
>>  > type="str">0x0
>>  > type="str">0x00
>>  > type="str">0x
>>  > type="str">pci
>>  > type="str">0x05
>>
>>> type="str">disk
>>> type="str">/rhev/data-center/5890a292-0390-01d2-01ed-029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc
>>> type="str">off
>>> type="str">false
>>> type="str">1
>>> type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc
>>
>>
>>  
>>> type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5
>>> type="str">path
>>> type="int">0
>>> type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc
>>> type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc
>>> type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc.lease
>>> type="str">66441539-f7ac-4946-8a25-75e422f939d4
>>  
>>
>>  
>>
>>  
>>
>>  
>>  4294967296
>>  4194304
>>  4194304
>>  16
>>  
>>1020
>>  
>>  
>>/machine
>>  
>>  
>>
>>  oVirt
>>  oVirt Node
>>  7-3.1611.el7.centos
>>  ccd79775-c888-4789-975a-fde1143dffc9
>>  ccd945c8-8069-4f31-8471-bbb58e9dd6ea
>>
>>  
>>  
>>hvm
>>
>>
>>  
>>  
>>
>>  
>>  
>>Haswell-noTSX
>>
>>
>>  
>>
>>  
>>  
>>
>>
>>
>>  
>>  destroy
>>  res

Re: [ovirt-devel] [vdsm] Engine XML: metadata and devices from XML

[Michal Skrivanek]

> On 16 Mar 2017, at 09:45, Francesco Romani  wrote:
> 
> We talked about sending storage device purely on metadata, letting Vdsm
> rebuild them and getting the XML like today.
> 
> In the other direction, Vdsm will pass through the XML (perhaps only
> parts of it, e.g. the devices subtree) like before.
> 
> This way we can minimize the changes we are uncertain of, and more
> importantly, we can minimize the risky changes.
> 
> 
> The following is  a realistic example of how the XML could look like if
> we send all but the storage devices. It is built using my pyxmlpickle
> module (see [3] below).

That’s quite verbose. How much work would it need to actually minimize it and 
turn it into something more simple.
Most such stuff should go away and I believe it would be beneficial to make it 
difficult to use to discourage using metadata as a generic junkyard

> 
> 
> 
>  a0
>  ccd945c8-8069-4f31-8471-bbb58e9dd6ea
>  http://ovirt.org/vm/tune/1.0";
> xmlns:ovirt-vm="http://ovirt.org/vm/1.0";
> xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0";>
>
>
>
>  
>
>  
> type="str">2
> type="str">ide
>
>  
>
> type="str">true
> type="str">e59c985c-46c2-4489-b355-a6f374125eb9
>
>   type="str">1
>   type="str">0
>   type="str">drive
>   type="str">0
>   type="str">0
>
> type="str">cdrom
> type="str">false
>
> type="str">disk
>  
>  
> type="str">5890a292-0390-01d2-01ed-029a
>
>   type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5
>   type="str">path
>   type="int">0
>   type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc
>   type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc
>   type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc.lease
>   type="str">66441539-f7ac-4946-8a25-75e422f939d4
>
> type="int">0
> type="str">virtio
> type="str">8589934592
> type="str">66441539-f7ac-4946-8a25-75e422f939d4
> type="str">false
> type="str">false
> type="str">0
> type="str">disk
> type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5
> type="str">0
> type="str">raw
> type="str">66441539-f7ac-4946-8a25-75e422f939d4
>
>   type="str">0x0
>   type="str">0x00
>   type="str">0x
>   type="str">pci
>   type="str">0x05
>
> type="str">disk
> type="str">/rhev/data-center/5890a292-0390-01d2-01ed-029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc
> type="str">off
> type="str">false
> type="str">1
> type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc
>
>
>  
> type="str">c578566d-bc61-420c-8f1e-8dfa0a18efd5
> type="str">path
> type="int">0
> type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc
> type="str">5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc
> type="str">/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc.lease
> type="str">66441539-f7ac-4946-8a25-75e422f939d4
>  
>
>  
>
>  
>
>  
>  4294967296
>  4194304
>  4194304
>  16
>  
>1020
>  
>  
>/machine
>  
>  
>
>  oVirt
>  oVirt Node
>  7-3.1611.el7.centos
>  ccd79775-c888-4789-975a-fde1143dffc9
>  ccd945c8-8069-4f31-8471-bbb58e9dd6ea
>
>  
>  
>hvm
>
>
>  
>  
>
>  
>  
>Haswell-noTSX
>
>
>  
>
>  
>  
>
>
>
>  
>  destroy
>  restart
>  destroy
>  
>/usr/libexec/qemu-kvm
>
>  
>   function='0x0'/>
>
>
>  
>   function='0x2'/>
>
>
>  
>
>
>  
>   function='0x1'/>
>
>
>  
>  
>  
>  
>  
>  
>  
>   function='0x0'/>
>
>
>   path='/var/run/ovirt-vmconsole-console/ccd945c8-8069-4f31-8471-bbb58e9dd6ea.sock'/>
>  

Re: [ovirt-devel] [vdsm] Engine XML: metadata and devices from XML

[Francesco Romani]

We talked about sending storage device purely on metadata, letting Vdsm
rebuild them and getting the XML like today.

In the other direction, Vdsm will pass through the XML (perhaps only
parts of it, e.g. the devices subtree) like before.

This way we can minimize the changes we are uncertain of, and more
importantly, we can minimize the risky changes.


The following is  a realistic example of how the XML could look like if
we send all but the storage devices. It is built using my pyxmlpickle
module (see [3] below).



  a0
  ccd945c8-8069-4f31-8471-bbb58e9dd6ea
  http://ovirt.org/vm/tune/1.0";
xmlns:ovirt-vm="http://ovirt.org/vm/1.0";
xmlns:ovirt-instance="http://ovirt.org/vm/instance/1.0";>



  

  
2
ide

  

true
e59c985c-46c2-4489-b355-a6f374125eb9

  1
  0
  drive
  0
  0

cdrom
false

disk
  
  
5890a292-0390-01d2-01ed-029a

  c578566d-bc61-420c-8f1e-8dfa0a18efd5
  path
  0
  /rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc
  5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc
  /rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc.lease
  66441539-f7ac-4946-8a25-75e422f939d4

0
virtio
8589934592
66441539-f7ac-4946-8a25-75e422f939d4
false
false
0
disk
c578566d-bc61-420c-8f1e-8dfa0a18efd5
0
raw
66441539-f7ac-4946-8a25-75e422f939d4

  0x0
  0x00
  0x
  pci
  0x05

disk
/rhev/data-center/5890a292-0390-01d2-01ed-029a/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc
off
false
1
5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc


  
c578566d-bc61-420c-8f1e-8dfa0a18efd5
path
0
/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc
5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc
/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_rel40x_data/c578566d-bc61-420c-8f1e-8dfa0a18efd5/images/66441539-f7ac-4946-8a25-75e422f939d4/5c4eeed4-f2a7-490a-ab57-a0d6f3a711cc.lease
66441539-f7ac-4946-8a25-75e422f939d4
  

  

  

  
  4294967296
  4194304
  4194304
  16
  
1020
  
  
/machine
  
  

  oVirt
  oVirt Node
  7-3.1611.el7.centos
  ccd79775-c888-4789-975a-fde1143dffc9
  ccd945c8-8069-4f31-8471-bbb58e9dd6ea

  
  
hvm


  
  

  
  
Haswell-noTSX


  

  
  



  
  destroy
  restart
  destroy
  
/usr/libexec/qemu-kvm

  
  


  
  


  


  
  


  
  
  
  
  
  
  
  


  
  
  


  
  
  


  
  
  
  


  
  
  
  


  
  
  


  


  


  


  
  
  
  
  
  
  
  
  


  
  


  
  
  


  
  

  
  
system_u:system_r:svirt_t:s0:c207,c629
system_u:object_r:svirt_image_t:s0:c207,c629
  
  
+107:+107
+107:+107
  



On 03/15/2017 01:28 PM, Francesco Romani wrote:
> Hi everyone,
>
> This is both a report of the current state of my Vdsm patches for Engine
> XML support, and a proposal to how move forward and solve
> the current open issues.
>
> TL;DR:
> 1. we can and IMO should reuse the current JSON schema to describe the
> structure (layout) and the types of the metadata section.
> 2. we don't need a priori validation of stuff in the metadata section.
> We will just raise in the creation flow if data is missing, or wrong,
> according to our schema.
> 2. we will add *few* items to the metadata section, only thing we can't
> express clearly-or at all in the libvirt XML. Redundancy and verbosiness
> will be thus kept at bay
> 3. I believe [3] is the best tool to do (de)serial