Re: [Openstack] [Nova] Got 'qemu-img: error while writing sector 634880: Input/output error' in _create_image
Actually, this is unlikely because there was locking for image fetching already. This patch only adds it for case when several compute processes share same $instances_dir. - disk full really might be the case. Simple check is to extend available space and try to reproduce it again. 2012/9/25 Wangpan > ** > Thanks for your clue. > yes, the occured condition of my issue is the same as BUG *1014227*, but > the difference is that my instance directory is NOT shared between compute > nodes, > the instances' images are saved in local disk on compute nodes. > I believe the bug Boris found may occur in NOT shared storage circumstance > as mine, and my issue may be resloved by this patch which adds a lock to > cache image method. > Thanks again. > > 2012-09-25 > -- > Wangpan > -- > *发件人：*Boris Filippov > *发送时间：*2012-09-24 19:59 > *主题：*Re: [Openstack] [Nova] Got 'qemu-img: error while writing sector > 634880: Input/output error' in _create_image > *收件人：*"Razique Mahroua" > *抄送：*"Wangpan","openstack"< > openstack@lists.launchpad.net> > > Your instance directory is shared between compute nodes? > Similar issue was fixed for folsom-rc1 > https://review.openstack.org/#/c/12024/ > https://bugs.launchpad.net/nova/+bug/1014227 > > > 2012/9/24 Razique Mahroua > >> Here is what I found >> http://comments.gmane.org/gmane.comp.emulators.qemu/153254 >> http://osdir.com/ml/qemu-devel/2012-05/msg04234.html >> >> What do you think ? >> >> *Nuage & Co - Razique Mahroua** * >> razique.mahr...@gmail.com >> >> >> Le 24 sept. 2012 à 10:43, Wangpan a écrit : >> >> qemu-img check -f qcow2 test.qcow2 >> No errors were found on the image. >> >> 2012-09-24 >> -- >> Wangpan >> -- >> *发件人：*Razique Mahroua >> *发送时间：*2012-09-24 16:30 >> *主题：*Re: Re: [Openstack] [Nova] Got 'qemu-img: error while writing >> sector 634880: Input/output error' in _create_image >> *收件人：*"Wangpan" >> *抄送：*"openstack" >> >> hi, >> what about a >> qemu-img check -f qcow2 test.qcow2 >> >> >> *Nuage & Co - Razique Mahroua** * >> razique.mahr...@gmail.com >> >> >> >> Le 24 sept. 2012 à 10:19, "Wangpan" a écrit >> : >> >> Thanks for your attention! >> >> - disk full # I believe there is enough space, because I have tried >> to boot serveral instances after this error with the same image, they are >> all booted successfully. >> - permission issues# Not this reason, too. explain as above. >> >> I still believe this is a race problem, if we convert to image from qcow2 >> to raw at a short interval, one of the convertion may fail, >> I have verified this by running two same commands continuously such as: >> qemu-img convert -f qcow2 -O raw test.qcow2 test.part & >> and one of them give me the same error: >> qemu-img: error while writing sector 1052672: Input/output error >> >> >> >> ___ >> Mailing list: https://launchpad.net/~openstack >> Post to : openstack@lists.launchpad.net >> Unsubscribe : https://launchpad.net/~openstack >> More help : https://help.launchpad.net/ListHelp >> >> > ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Nova] Got 'qemu-img: error while writing sector 634880: Input/output error' in _create_image
Your instance directory is shared between compute nodes? Similar issue was fixed for folsom-rc1 https://review.openstack.org/#/c/12024/ https://bugs.launchpad.net/nova/+bug/1014227 2012/9/24 Razique Mahroua > Here is what I found > http://comments.gmane.org/gmane.comp.emulators.qemu/153254 > http://osdir.com/ml/qemu-devel/2012-05/msg04234.html > > What do you think ? > > *Nuage & Co - Razique Mahroua** * > razique.mahr...@gmail.com > > > Le 24 sept. 2012 à 10:43, Wangpan a écrit : > > qemu-img check -f qcow2 test.qcow2 > No errors were found on the image. > > 2012-09-24 > -- > Wangpan > -- > *发件人：*Razique Mahroua > *发送时间：*2012-09-24 16:30 > *主题：*Re: Re: [Openstack] [Nova] Got 'qemu-img: error while writing sector > 634880: Input/output error' in _create_image > *收件人：*"Wangpan" > *抄送：*"openstack" > > hi, > what about a > qemu-img check -f qcow2 test.qcow2 > > > *Nuage & Co - Razique Mahroua** * > razique.mahr...@gmail.com > > > > Le 24 sept. 2012 à 10:19, "Wangpan" a écrit : > > Thanks for your attention! > > - disk full # I believe there is enough space, because I have tried > to boot serveral instances after this error with the same image, they are > all booted successfully. > - permission issues# Not this reason, too. explain as above. > > I still believe this is a race problem, if we convert to image from qcow2 > to raw at a short interval, one of the convertion may fail, > I have verified this by running two same commands continuously such as: > qemu-img convert -f qcow2 -O raw test.qcow2 test.part & > and one of them give me the same error: > qemu-img: error while writing sector 1052672: Input/output error > > > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > > ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] snapshot-temp-directory
This actually was fixed in upstream ( https://github.com/openstack/nova/commit/70129ed19db187cc90f74abd5c93c86098d29c27 ). Which OpenStack version you use? Maybe this patch can be backported there. There is workaround for this problem, but it's (not a bit) ugly: http://www.mail-archive.com/openstack@lists.launchpad.net/msg13730.html 2012/8/13 Wolfgang Hennerbichler : > hello fellow openstack users, > > I just tried to create a snapshot of a virtual machine, which was larger > than average. Bad news: The snapshot couldn't be saved, but what I see here > is some kind of design problem: > > 2012-08-13 13:44:41 DEBUG nova.utils > [req-a678e738-9c18-4ee6-8e2e-f1f176a4674b 9b4ad74ecb69492db1bd3818efe6a47f > 7edc3284f53f4e02b92d498db41b842d] Running cmd (subprocess): qemu-img convert > -f qcow2 -O raw -s 34dd12038e7b400b9f1d772de4e01 > 70a /var/lib/nova/instances/instance-005c/disk > /tmp/tmp1gRMY9/34dd12038e7b400b9f1d772de4e0170a from (pid=25365) execute > /usr/lib/python2.7/dist-packages/nova/utils.py:219 > > that's bad, I don't have so much disk space on /, but I have plenty in > /var/lib/nova/instances. > > df -h | grep nova > /dev/dm-8 1.4T 150G 1.3T 11% /var/lib/nova/instances > > is there a generic way to change this path? > > Wolfgang > > -- > DI (FH) Wolfgang Hennerbichler > Software Development > Unit Advanced Computing Technologies > RISC Software GmbH > A company of the Johannes Kepler University Linz > > IT-Center > Softwarepark 35 > 4232 Hagenberg > Austria > > Phone: +43 7236 3343 245 > Fax: +43 7236 3343 250 > wolfgang.hennerbich...@risc-software.at > http://www.risc-software.at > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Support for snapshot of LVM backend image
>> But qemu can also write the vm state outside of the backend image, which >> should be usable with all image backends. Use that instead managedSave? This should suffice. Save VM state, suspend domain - do snapshot - restore VM with previous state? ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Support for snapshot of LVM backend image
Well, according to http://libvirt.org/html/libvirt-libvirt.html#virDomainManagedSave domain will be suspended. After snapshotting there is new _create_domain call: if state == power_state.RUNNING: virt_dom.managedSave(0) ... snapshotting... finally: libvirt_utils.delete_snapshot(disk_path, snapshot_name) if state == power_state.RUNNING: self._create_domain(domain=virt_dom) So, Is this actually "live" snapshot right now? >> But qemu can also write the vm state outside of the backend image, which >> should be usable with all image backends. This can be exploited for saving VM states. There is blueprint for that: https://blueprints.launchpad.net/nova/+spec/live-snapshots ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Support for snapshot of LVM backend image
Current snapshot implementation for libvirt driver doesn't support snapshotting for everything except qcow2 (as i remember, because virDomainManagedSave will fail on everything except qcow2). So, maybe we can discuss it here. This for sure can be implemented for lvm using lvm snapshots but it will be unsafe doing snapshot on alive instance. LVM is much like raw right now and raw doesn't support snapshots with current logic too. 2012/7/16 Li Wang > hi, all > > As LVM backend image support was introduced in the trunk version > of nova, is there any plan to support snapshot of the LVM backend image? > > I am using libvirt and Xen, with lvm as the instance image backend. > > There are two issues when creating snapshot. > > 1. when calling managedSave, libvirt complained there's no > supported function by the driver. > > "libvirtError: this function is not supported by the > connection driver: virDomainManagedSave" > > 2. when actually creating snapshot, libvirt calls qemu-img, which > does not support lvm. > > As a result, snapshot status is always 'queue', and actually it's > failed. > > > For issue 1, it seems that libvirt does not support "save" or > similar functions for xen, some modifications to the libvirt Xen driver > maybe help? > > For issue 2, I think the snapshot action should call lvm related > command but not qemu-img when using lvm. > > For example:lvcreate --snapshot .. > > > Further discussions and replies are welcome. > > Thanks all > > > Regards > > > Wang Li > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > > ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] How do I stop image-create from using /tmp?
2012/7/3 John Garbutt : >> From: Daniel P. Berrange [mailto:berra...@redhat.com] >> Sent: 03 July 2012 11:09 >> This would suggest there's a potential use case for a new config parameter >> FLAGS.local_scratch_path, whose default value matches >> FLAGS.instances_path if not set. > > +1 > > Cheers, > John > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp e.g. should i return that config parameter? ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] How do I stop image-create from using /tmp?
2012/7/2 Daniel P. Berrange : > On Mon, Jul 02, 2012 at 10:24:02AM -0700, Matt Joyce wrote: >> I like the idea of making this a flagfile option. > > In the particular case of the qemu-img command described > in earlier in this thread, I'm not convinced we need a > new option. Instead of using /tmp when extracting a snapshot > from an existing disk image, it could just use the path > where the source image already resides. ie the existing > FLAGS.instances_path directory, which can be assumed to > be a suitably large persistent data store. > > Other uses of temporary files should be analysed ona case > by case basis to figure out a suitable storage location. > This might perhaps identify a need for a generic temp > file location for nova, such as /var/run/nova/ or > /var/cache/nova or both (depending on use case). > > Regards, > Daniel > -- > |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| > |: http://libvirt.org -o- http://virt-manager.org :| > |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| > |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp Sure, we can place snapshots inside instances_path. At least, just separate them from instances and place them in %instances_path%/snapshots. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] How do I stop image-create from using /tmp?
2012/7/1 Lars Kellogg-Stedman : >> So, maybe setting any of this environment variables for nova-compute >> to desired value sholuld help. > > Yeah, I was expecting that. > > Given that this could easily take out a compute host I'd like to see > it get an explicit configuration value (or default to instance_dir, I > guess). > > If I can sort out the corporate contributor agreement stuff I may try > to submit a patch... > > -- > Lars Kellogg-Stedman| > Senior Technologist| > http://ac.seas.harvard.edu/ > Academic Computing | > http://code.seas.harvard.edu/ > Harvard School of Engineering and Applied Sciences | https://review.openstack.org/#/c/9239/ Wrote patch for it. We had same issue with our OpenStack installation before and this behavior is quite surprise for people who believe that nobody will store Big Files in /tmp. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] How do I stop image-create from using /tmp?
As I remember, destination directory for snapshot is created by mkdtemp. What module documantation says about default temp directory: tempfile.tempdir When set to a value other than None, this variable defines the default value for the dir argument to all the functions defined in this module. If tempdir is unset or None at any call to any of the above functions, Python searches a standard list of directories and sets tempdir to the first one which the calling user can create files in. The list is: The directory named by the TMPDIR environment variable. The directory named by the TEMP environment variable. The directory named by the TMP environment variable. A platform-specific location:On RiscOS, the directory named by the Wimp$ScrapDir environment variable. On Windows, the directories C:\TEMP, C:\TMP, \TEMP, and \TMP, in that order. On all other platforms, the directories /tmp, /var/tmp, and /usr/tmp, in that order. As a last resort, the current working directory. (http://docs.python.org/library/tempfile.html) So, maybe setting any of this environment variables for nova-compute to desired value sholuld help. 2012/6/30 Lars Kellogg-Stedman > > I'm running "nova image-create" for the first time, and on the compute > node I see: > > qemu-img convert -f qcow2 -O qcow2 -s 98a8efe9ec114b489eb163c64661441a > /virt/pools/openstack_2/instance-0011/disk > /tmp/tmpH9KkUZ/98a8efe9ec114b489eb163c64661441a > > I am concerned to see this operation dropping a disk image into /tmp. > What if the disk image is larger than the root filesystem? Is there a > way to have OpenStack put these temporary images somewhere else? I > would prefer instance_dir by default, instead of /tmp, because > instance_dir has already been provisioned with enough space to meet > the needs of disk images, but an explicit parameter would probably be > a better option. > > -- > Lars Kellogg-Stedman| > Senior Technologist| > http://ac.seas.harvard.edu/ > Academic Computing | > http://code.seas.harvard.edu/ > Harvard School of Engineering and Applied Sciences | > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
