Re: [vdsm] vdsm sync meeting June 17th 2013
- Original Message - On 06/18/2013 09:29 AM, Arik Hadas wrote: - Original Message - VDSM sync meeting Monday June 17th 2013 === - Dan requests reviews for pending patches from Mei Liu and himself ( http://gerrit.ovirt.org/#/c/15356/8 ) for mom. - Toni to fix the problem with Mark's patch that Dan reviewed so that it can be merged together with the smaller following patches. - Federico has three high priority patches for features and two lower priority ones that need reviews in order to make it for the release deadline. They are related to disk extension and image upload/download (glance): + http://gerrit.ovirt.org/#/c/15442 - constants: unify the BLANK_UUID def... + http://gerrit.ovirt.org/#/c/14589 - volume: add the BLOCK_SIZE constant +1 + http://gerrit.ovirt.org/#/c/14590 - volume: add the extendVolumeSize method + http://gerrit.ovirt.org/#/c/15614 - vm: add the live diskSizeExtend method +1 + http://gerrit.ovirt.org/#/c/14955 - image: add support to upload/download images - Adam and others to review Gluster patches: + http://gerrit.ovirt.org/#/c/13785/ + http://gerrit.ovirt.org/#/c/11094/ - Is there a filing for Qemu image handling reported by virt team? Background about the problem: As part of the ram snapshots feature (http://wiki.ovirt.org/Features/RAM_Snapshots) we pass a path to file to libvirt, where the memory of the VM should be saved as part of the snapshot creation. the output file is identical to the one that is created to store the memory on hibernate command. When VM with snapshot(s) that has memory is being exported, we want the snapshot's memory file to be exported to the export domain as well. when trying to export such memory file, we saw that the file is being truncated to the closest size that is a multiple of block size (512). we found that the file is being truncated by qemu-img that is being used to copy the file to the export domain. We previously thought that it is a problem in qemu - not handling files with size that is not a multiple of block size, and a bug to qemu was opened (bz 970559). We now understand that the issue described above might be a problem in qemu that needs to be solved, but it is not the problem in our case: treating the memory file as qemu image is wrong - the created file is not a qemu image. the file has a special format: it contains xml part at the beginning and the memory dump as a binary data right after that. So basically it seems that we need the following things in vdsm: 1. The creation of the file that is expected to store the memory as qemu image seems to be redundant. instead, we need to have a way to create a file in a similar way to 'touch' command - just to verify that the file in the path we pass to libvirt is created with the right permissions. (it's probably true for the hibernation volume creation as well btw). how would this look for block devices? It is working as expected on block devices (tested). When dumping the memory the file, libvirt is doing O_TRUNC and the resultant file have a random lenght which can be odd. This is erronously treated by the qemu-img utility. 2. The more important issue is that we need to have a way to export files which are not qemu images to the export domain, which means without using the qemu-img process, just copy the file. qemu-img is capable to deal with raw images, as we are doing today, provided that the files are a block size multiples, as can be expected for a file which is used like a volume. Having two export paths can be very cumbersome, we should find a less disruptive solution. ayal - anything close to this today - i remember various qemu-img vs. dd patches in the past for copy operations? - Reminder to follow wiki feature page process closely to maximize changes to have features added to a release. - Multiple gateways are being worked on to fix dhcp races and comments by reviewers. - NetReloaded after multiple gateways. Try to get up to iproute2 configurator. - Cross-distro support: + Netreloaded with iproute2 plus persistence would help. + init scripts separation (proposed by Yaniv to make ready for 3.3). - Cloud-init integration by Greg Padgett. There is a vdsm patch that should be taken care of in terms of reviewing http://gerrit.ovirt.org/#/c/14347/ ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] vdsm sync meeting June 17th 2013
On Fri, Jun 21, 2013 at 02:27:39PM +0300, Itamar Heim wrote: On 06/18/2013 09:29 AM, Arik Hadas wrote: - Is there a filing for Qemu image handling reported by virt team? Background about the problem: As part of the ram snapshots feature (http://wiki.ovirt.org/Features/RAM_Snapshots) we pass a path to file to libvirt, where the memory of the VM should be saved as part of the snapshot creation. the output file is identical to the one that is created to store the memory on hibernate command. When VM with snapshot(s) that has memory is being exported, we want the snapshot's memory file to be exported to the export domain as well. when trying to export such memory file, we saw that the file is being truncated to the closest size that is a multiple of block size (512). we found that the file is being truncated by qemu-img that is being used to copy the file to the export domain. We previously thought that it is a problem in qemu - not handling files with size that is not a multiple of block size, and a bug to qemu was opened (bz 970559). We now understand that the issue described above might be a problem in qemu that needs to be solved, but it is not the problem in our case: treating the memory file as qemu image is wrong - the created file is not a qemu image. the file has a special format: it contains xml part at the beginning and the memory dump as a binary data right after that. So basically it seems that we need the following things in vdsm: 1. The creation of the file that is expected to store the memory as qemu image seems to be redundant. instead, we need to have a way to create a file in a similar way to 'touch' command - just to verify that the file in the path we pass to libvirt is created with the right permissions. (it's probably true for the hibernation volume creation as well btw). how would this look for block devices? 2. The more important issue is that we need to have a way to export files which are not qemu images to the export domain, which means without using the qemu-img process, just copy the file. ayal - anything close to this today - i remember various qemu-img vs. dd patches in the past for copy operations? Indeed, the problematic qemu-img call was introduced by http://gerrit.ovirt.org/#/c/10979/5/vdsm/storage/image.py In my opinion, replacing it with cp (that implies --sparse=auto) would do the right thing for all combinations of file/block qemu-disk/snapshot-ram-data. Regards, Dan. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] environment encoding, LC_ALL and vdsm tests
On Thu, Jun 20, 2013 at 12:39:22PM -0400, David Caro Estevez wrote:
- Original Message -
From: Dan Kenigsberg dan...@redhat.com
To: Martin Sivak msi...@redhat.com, dc...@redhat.com
Cc: vdsm-devel@lists.fedorahosted.org
Sent: Thursday, June 20, 2013 3:08:29 PM
Subject: Re: environment encoding, LC_ALL and vdsm tests
On Thu, Jun 20, 2013 at 05:50:16AM -0400, Martin Sivak wrote:
Hi,
recently I discovered an issue with our Jenkins test environment. It was
failing in testHooks.py because my Gerrit name contains diacritics and our
code tried to decode it as ascii.
Traceback (most recent call last):
File /usr/lib64/python2.6/unittest.py, line 278, in run
testMethod()
File /ephemeral0/vdsm_unit_tests_gerrit_el/tests/hooksTests.py, line
125, in test_deviceCustomProperties
params={'customProperty': ' rocks!'})
File /ephemeral0/vdsm_unit_tests_gerrit_el/vdsm/hooks.py, line 70, in
_runHooksDir
scriptenv[k] = unicode(v).encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 12:
ordinal not in range(128)
The relevant code is here:
hooks.py:
60scriptenv = os.environ.copy()
...
69for k, v in scriptenv.iteritems():
70scriptenv[k] = unicode(v).encode('utf-8')
My first instinct was to decode it using the proper encoding:
source_encoding = sys.stdin.encoding or locale.getpreferredencoding()
for k, v in scriptenv.iteritems():
scriptenv[k] = v.decode(source_encoding).encode('utf-8')
But it still did not work. So I tried to print out the environment and
encodings that are used when make check is being run and got this:
sys.stdin.encoding == None
locale.getpreferredencoding() - ANSI_X3.4-1968
os.environ['LC_ALL'] == 'C'
os.environ['LANG'] == 'en_US.UTF-8'
Please notice the encoding part, my system and terminal are using utf-8,
but vdsm reads the environment values using ANSI. That is obviously wrong
and can't work.
So i tried to investigate it further and found out we force LC_ALL to C in
vdsmd.init, run_tests.sh.in and run_tests_local.sh.in.
I also found the commit that introduced this -
107644dbad9af250c00e7f25fc51a92c6250d442 - and finally understood where
the issue was.
Although I understand the reasons for the patch, I do not agree with
it. If we are executing other tools and parse their output, we should
be preparing and passing the updated locale _only_ to those tools. We
should not be setting the locale we need for parsing stuff to the
whole vdsm daemon.
Since vdsm is not intended for direct human control, I actually like the
idea of turning off all locale noise by a global LC_ALL=C. The
alternative, of setting it to C before each application with parsed
output seems tedious and easily forgotten.
Our current practice of setting LC_ALL to C no matter on what terminal
or system we are starting vdsmd is causing us the above mentioned
issue, because the environment can (and does) contain data in the
system encoding. This essentially prevents anybody with utf-8 chars in
their names to submit anything to vdsm.
No doubt that we have to fix it. The easiest hack is to ask our Jenkins
job to clear the Jenkins env vars before calling `make check`. I'm sure
David (CCed) can do it quite easily.
Yes, that should be easy, if you decide to do that, it can be done in
30min (smallest fraction of time for a task).
Please do that, as a quick mitigation of the real problem.
It *is* important that people can use their real name when contributing
to vdsm code.
___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] vdsm sync meeting June 17th 2013
- Original Message - On Fri, Jun 21, 2013 at 02:27:39PM +0300, Itamar Heim wrote: On 06/18/2013 09:29 AM, Arik Hadas wrote: - Is there a filing for Qemu image handling reported by virt team? Background about the problem: As part of the ram snapshots feature (http://wiki.ovirt.org/Features/RAM_Snapshots) we pass a path to file to libvirt, where the memory of the VM should be saved as part of the snapshot creation. the output file is identical to the one that is created to store the memory on hibernate command. When VM with snapshot(s) that has memory is being exported, we want the snapshot's memory file to be exported to the export domain as well. when trying to export such memory file, we saw that the file is being truncated to the closest size that is a multiple of block size (512). we found that the file is being truncated by qemu-img that is being used to copy the file to the export domain. We previously thought that it is a problem in qemu - not handling files with size that is not a multiple of block size, and a bug to qemu was opened (bz 970559). We now understand that the issue described above might be a problem in qemu that needs to be solved, but it is not the problem in our case: treating the memory file as qemu image is wrong - the created file is not a qemu image. the file has a special format: it contains xml part at the beginning and the memory dump as a binary data right after that. So basically it seems that we need the following things in vdsm: 1. The creation of the file that is expected to store the memory as qemu image seems to be redundant. instead, we need to have a way to create a file in a similar way to 'touch' command - just to verify that the file in the path we pass to libvirt is created with the right permissions. (it's probably true for the hibernation volume creation as well btw). how would this look for block devices? 2. The more important issue is that we need to have a way to export files which are not qemu images to the export domain, which means without using the qemu-img process, just copy the file. ayal - anything close to this today - i remember various qemu-img vs. dd patches in the past for copy operations? Indeed, the problematic qemu-img call was introduced by http://gerrit.ovirt.org/#/c/10979/5/vdsm/storage/image.py In my opinion, replacing it with cp (that implies --sparse=auto) would do the right thing for all combinations of file/block qemu-disk/snapshot-ram-data. No, for qcow images it would not dedup zeros (and reclaim space where relevant) It would also not be able to collapse images. Regards, Dan. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] vdsm sync meeting June 17th 2013
On Sun, Jun 23, 2013 at 05:29:40AM -0400, Ayal Baron wrote: - Original Message - On Fri, Jun 21, 2013 at 02:27:39PM +0300, Itamar Heim wrote: On 06/18/2013 09:29 AM, Arik Hadas wrote: - Is there a filing for Qemu image handling reported by virt team? Background about the problem: As part of the ram snapshots feature (http://wiki.ovirt.org/Features/RAM_Snapshots) we pass a path to file to libvirt, where the memory of the VM should be saved as part of the snapshot creation. the output file is identical to the one that is created to store the memory on hibernate command. When VM with snapshot(s) that has memory is being exported, we want the snapshot's memory file to be exported to the export domain as well. when trying to export such memory file, we saw that the file is being truncated to the closest size that is a multiple of block size (512). we found that the file is being truncated by qemu-img that is being used to copy the file to the export domain. We previously thought that it is a problem in qemu - not handling files with size that is not a multiple of block size, and a bug to qemu was opened (bz 970559). We now understand that the issue described above might be a problem in qemu that needs to be solved, but it is not the problem in our case: treating the memory file as qemu image is wrong - the created file is not a qemu image. the file has a special format: it contains xml part at the beginning and the memory dump as a binary data right after that. So basically it seems that we need the following things in vdsm: 1. The creation of the file that is expected to store the memory as qemu image seems to be redundant. instead, we need to have a way to create a file in a similar way to 'touch' command - just to verify that the file in the path we pass to libvirt is created with the right permissions. (it's probably true for the hibernation volume creation as well btw). how would this look for block devices? 2. The more important issue is that we need to have a way to export files which are not qemu images to the export domain, which means without using the qemu-img process, just copy the file. ayal - anything close to this today - i remember various qemu-img vs. dd patches in the past for copy operations? Indeed, the problematic qemu-img call was introduced by http://gerrit.ovirt.org/#/c/10979/5/vdsm/storage/image.py In my opinion, replacing it with cp (that implies --sparse=auto) would do the right thing for all combinations of file/block qemu-disk/snapshot-ram-data. No, for qcow images it would not dedup zeros (and reclaim space where relevant) It would also not be able to collapse images. The relevant code runs for RAW file format. I.e., no qcow at the source and no chains. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
[vdsm] AUTO: Yih-Herng Chuang/Seattle/IBM is out of the office, (returning 06/24/2013)
I am out of the office until 06/24/2013. For FSM security tech issues, please contact project manager Mark Rayner. For others, please contact my manager John Aguiar. I will respond your email when I am back to work. Note: This is an automated response to your message vdsm-devel Digest, Vol 25, Issue 13 sent on 06/23/2013 2:44:22. This is the only notification you will receive while this person is away.___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] vdsm sync meeting June 17th 2013
- Original Message - On Sun, Jun 23, 2013 at 04:44:19AM -0400, Ayal Baron wrote: - Original Message - - Original Message - VDSM sync meeting Monday June 17th 2013 === - Dan requests reviews for pending patches from Mei Liu and himself ( http://gerrit.ovirt.org/#/c/15356/8 ) for mom. - Toni to fix the problem with Mark's patch that Dan reviewed so that it can be merged together with the smaller following patches. - Federico has three high priority patches for features and two lower priority ones that need reviews in order to make it for the release deadline. They are related to disk extension and image upload/download (glance): + http://gerrit.ovirt.org/#/c/15442 - constants: unify the BLANK_UUID def... + http://gerrit.ovirt.org/#/c/14589 - volume: add the BLOCK_SIZE constant +1 + http://gerrit.ovirt.org/#/c/14590 - volume: add the extendVolumeSize method + http://gerrit.ovirt.org/#/c/15614 - vm: add the live diskSizeExtend method +1 + http://gerrit.ovirt.org/#/c/14955 - image: add support to upload/download images - Adam and others to review Gluster patches: + http://gerrit.ovirt.org/#/c/13785/ + http://gerrit.ovirt.org/#/c/11094/ - Is there a filing for Qemu image handling reported by virt team? Background about the problem: As part of the ram snapshots feature (http://wiki.ovirt.org/Features/RAM_Snapshots) we pass a path to file to libvirt, where the memory of the VM should be saved as part of the snapshot creation. the output file is identical to the one that is created to store the memory on hibernate command. When VM with snapshot(s) that has memory is being exported, we want the snapshot's memory file to be exported to the export domain as well. when trying to export such memory file, we saw that the file is being truncated to the closest size that is a multiple of block size (512). we found that the file is being truncated by qemu-img that is being used to copy the file to the export domain. We previously thought that it is a problem in qemu - not handling files with size that is not a multiple of block size, and a bug to qemu was opened (bz 970559). We now understand that the issue described above might be a problem in qemu that needs to be solved, but it is not the problem in our case: treating the memory file as qemu image is wrong - the created file is not a qemu image. the file has a special format: it contains xml part at the beginning and the memory dump as a binary data right after that. This is not correct. qemu-img is a utility to manage images. And here, we are trying to use it to copy something that is not a qemu image. This is a file created by qemu/libvirt and represents part of the VM. If that is not a qemu image then I'm not sure what is. There is no format limitation on images, and the content of this file/device is no exception and no reason to treat it as one. This is a raw device/file, we don't and shouldn't care about whether it has xml, compression, encryption, qcow, whatever. This is an image we pass to libvirt to use in order to dump the memory. qemu-img manages raw files/devices and is used to keep sparseness, dedup zeros, etc. It should not be hard to fix qemu-img to work with binary files of any length. The only question is whether handling non-qemu-images is the job for qemu-img. Dan. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] vdsm sync meeting June 17th 2013
- Original Message - On Sun, Jun 23, 2013 at 06:24:50AM -0400, Ayal Baron wrote: - Original Message - On Sun, Jun 23, 2013 at 04:44:19AM -0400, Ayal Baron wrote: - Original Message - - Original Message - VDSM sync meeting Monday June 17th 2013 === - Dan requests reviews for pending patches from Mei Liu and himself ( http://gerrit.ovirt.org/#/c/15356/8 ) for mom. - Toni to fix the problem with Mark's patch that Dan reviewed so that it can be merged together with the smaller following patches. - Federico has three high priority patches for features and two lower priority ones that need reviews in order to make it for the release deadline. They are related to disk extension and image upload/download (glance): + http://gerrit.ovirt.org/#/c/15442 - constants: unify the BLANK_UUID def... + http://gerrit.ovirt.org/#/c/14589 - volume: add the BLOCK_SIZE constant +1 + http://gerrit.ovirt.org/#/c/14590 - volume: add the extendVolumeSize method + http://gerrit.ovirt.org/#/c/15614 - vm: add the live diskSizeExtend method +1 + http://gerrit.ovirt.org/#/c/14955 - image: add support to upload/download images - Adam and others to review Gluster patches: + http://gerrit.ovirt.org/#/c/13785/ + http://gerrit.ovirt.org/#/c/11094/ - Is there a filing for Qemu image handling reported by virt team? Background about the problem: As part of the ram snapshots feature (http://wiki.ovirt.org/Features/RAM_Snapshots) we pass a path to file to libvirt, where the memory of the VM should be saved as part of the snapshot creation. the output file is identical to the one that is created to store the memory on hibernate command. When VM with snapshot(s) that has memory is being exported, we want the snapshot's memory file to be exported to the export domain as well. when trying to export such memory file, we saw that the file is being truncated to the closest size that is a multiple of block size (512). we found that the file is being truncated by qemu-img that is being used to copy the file to the export domain. We previously thought that it is a problem in qemu - not handling files with size that is not a multiple of block size, and a bug to qemu was opened (bz 970559). We now understand that the issue described above might be a problem in qemu that needs to be solved, but it is not the problem in our case: treating the memory file as qemu image is wrong - the created file is not a qemu image. the file has a special format: it contains xml part at the beginning and the memory dump as a binary data right after that. This is not correct. qemu-img is a utility to manage images. And here, we are trying to use it to copy something that is not a qemu image. This is a file created by qemu/libvirt and represents part of the VM. If that is not a qemu image then I'm not sure what is. The file is created by libvirt, concatenating VM memory dump to its metadata. That something else to the qemu disk images which qemu-img was designed for (and named after). A raw file is a raw file, the data inside it is irrelevant. This is part of the VM. qemu-img was also not originally designed for block devices, nor was qemu... ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
