[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2015-01-14 Thread Chris J Arges
FWIW, just re-reproduced this with latest upstream kernel / qemu / fresh qcow2 image. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption in trusty

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-12-15 Thread Chris J Arges
** Changed in: qemu (Ubuntu) Status: Confirmed = In Progress -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption in trusty (qemu 1.7 and 2.0

Re: [Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-12-09 Thread Serge Hallyn
Excellent! Any chance you can start bisecting with http://people.canonical.com/~serge/binaries.{0..68}/{qemu-img,qemu- system-x86_64} ? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu.

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-12-09 Thread Chris J Arges
Serge, So I was able to just compile my own qemu and test with that. I did attempt a reverse bisect, and was able to reproduce as early as v1.1 and also reproduce on master HEAD. v1.0 was inconclusive because qcow2 format I made with the newer binary seemed to be incompatible with v1.0; however

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-12-01 Thread Ryan Harper
can we confirm what filesystems and options are enabled when reproducing (ie, ext4 +extent mapping)[1] ? Bug 1368815 sounds very much like this. If the reproducing systems have ext4 extents mapping enabled, one could create an ext4 fs without extent mapping[2] and see if this still reproduces.

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-12-01 Thread Chris J Arges
Ryan, The host's root filesystem is ext3/LVM (per Jamie's original configuration): sudo tune2fs -l /dev/disk/by-id/dm-name-ubuntu--vg-root | grep -i features Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file -- You received

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-12-01 Thread Jamie Strandboge
Actually, for me it is just ext3 without LVM. $ sudo tune2fs -l /dev/sda3 | grep -i features Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file -- You received this bug notification because you are a member of Ubuntu Server

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-12-01 Thread Chris J Arges
Attached is a reproducer for this issue, here is what needs to be done to setup the reproducer: 1) The host machine's filesystem needs to be ext3 2) Install a VM (via virsh) and use a qcow2 disk 3) Ensure you can ssh without a password and the VM has bonnie++ installed 4) Adjust the variables in

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-12-01 Thread Chris J Arges
Also I've been able to reproduce this with the latest master in qemu, and even with the latest daily 3.18-rcX kernel on the host. ** Also affects: qemu Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Server Team, which is

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-11-26 Thread Chris J Arges
Ok I think I can reproduce this; after running some disk operations (bonnie++ and split a 100MB file), if I shutdown and try to boot the VM the disk cannot be booted and I'm presented with the grub menu. However this reproducer is not yet 100% reliable. Next week I'll work on bisecting it down

Re: [Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-11-26 Thread Serge Hallyn
Awesome - thank you Chris. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate) To manage notifications about

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-11-25 Thread Chris J Arges
** Changed in: qemu (Ubuntu) Assignee: Serge Hallyn (serge-hallyn) = Chris J Arges (arges) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption in

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-11-24 Thread Chris J Arges
$ od -x -N 72 forhallyn-trusty-amd64.img.corrupted | grep '[1-9]*' refcount_table_cluster 0100 000 4651 fb49 0200 020 1000 0200 040 1000 0300 060 0100 0100 0100

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-11-20 Thread Jamie Strandboge
FYI, I was able to reproduce this last night and uploaded forhallyn- trusty-amd64.img.corrupted.gz to https://chinstrap.canonical.com/~jamie/lp1292234/ for comparison with forhallyn-trusty-amd64.img.gz. -- You received this bug notification because you are a member of Ubuntu Server Team, which

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-09-22 Thread Serge Hallyn
On my main server (3.13.0-32-generic with precise userspace) I installed a trusty container with ext3 (LVM) backing store. There I installed uvt and created 4 VMs, 2 precise amd64 and 2 precise i386. I several times did: ubuntu@uvttest:~$ cat list p-precise-server-amd64 p-precise-server-i386

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-09-17 Thread Jamie Strandboge
This happened again with an important VM. I still don't have a reproducer for testing the bisect packages :( -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-09-10 Thread Jamie Strandboge
I just had this happen to me with 2.1+dfsg-3ubuntu3 on utopic. I had a VM I had been using for a days, then did a 'uvt stop -rf ...' followed by 'uvt update sec-utopic-amd64' and I was dropped to a grub rescue. :\ I'll downgrade again and regenerate the VM. -- You received this bug notification

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-09-03 Thread Ryan Harper
I've been running the scripts from comment #10. I have two VMs each running simultaneously; I've completed 24 hours of this sequence, about 50 total cycles with zero errors in the qcow2 images. We're missing something; possibly hardware specific? Host machine is an Intel NUC on Trusty. Linux

Re: [Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-09-03 Thread Serge Hallyn
There are 69 commits to block/qcow* between 1.5.0 and 1.7.0. I have compiled binaries of qemu-system-x86_64 and qemu-img at each of those commits and pushed them to http://people.canonical.com/~serge/binaries.0 through http://people.canonical.com/~serge/binaries.68 Note that binaries.0 is the

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-09-03 Thread Ryan Harper
I'm also starting work on updating uvt to use external snapshots instead; this would be an alternative to use while chasing down the bug in internal snapshots. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu.

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-09-03 Thread Jamie Strandboge
I tried to reproduce this many different ways with 2.1+dfsg-3ubuntu3 over the weekend and could not trigger the issue (with ksm enabled too). I don't know what version I had in comment #12. 2.1+dfsg-3ubuntu2 is plausible based on the date of the comment and the publication of this version, though

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-08-29 Thread Ryan Harper
For the reproducers, something worth trying is to use to try is external snapshots (instead of internal which the snapshot-create-as does without flags). instead run: snapshot-create-as --disk-only which will basically do qemu-img create -b your_original_qcow2 -f qcow2 pristine And store the

Re: [Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-08-21 Thread Serge Hallyn
Hi Jamie, just to make sure, did you permanently disable ksm? Does cat /sys/kernel/mm/ksm/run still show 0? I've so far never seen a case where a reboot did not fix the issue, nor have I seen an issue (other than suspending the host sometimes causing the VM to hang so that I have to destroy

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-08-21 Thread Jamie Strandboge
I disabled KSM by setting /etc/default/qemu-kvm to have: KSM_ENABLED=0 and did 'sudo restart qemu-kvm'. I also rebooted before seeing the problem. Since then, I downgraded to saucy's qemu-kvm which reset KSM_ENABLED=1. I didn't specifically check /sys/kernel/mm/ksm/run and of course now this is

Re: [Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-08-21 Thread Serge Hallyn
Ok - thanks Jamie. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate) To manage notifications about this bug

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-08-19 Thread Jamie Strandboge
On utopic amd64, I tried the new qemu 2.1 packages and disabled KSM. They seemed to be ok for a while, but after using 'uvt update' today (which under the hood does what is decribed in the bug description), I lost 6 VMs to this bug. A reboot did not solve it. I've downgraded to saucy again.

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-08-11 Thread Serge Hallyn
As far as I know, everyone who has experienced this has been using a thinkpad. I've first experienced this myself last week, on a new thinkpad running utopic. Two curious things I noticed, beside this being a thinkpad: 1. I could not start the VM with the bad image at all. Until I rebooted.

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-08-07 Thread Serge Hallyn
** Tags added: qcow2 -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate) To manage notifications about this

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-05-22 Thread Serge Hallyn
I have a clean install of trusty on an intel laptop. I added the following upstart job in the forhallyn-trusty-amd64.img root partition: description update and shutdown author Serge Hallyn serge.hal...@ubuntu.com start on

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-05-19 Thread Serge Hallyn
** Changed in: qemu (Ubuntu) Assignee: (unassigned) = Serge Hallyn (serge-hallyn) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption in trusty

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-05-01 Thread Seth Arnold
I believe I just tripped this bug; I compressed some qcow2 images using this: for f in sec-{lucid,precise,quantal,saucy,trusty}-{amd64,i386} ; do echo $f ; qemu-img convert -s pristine -p -f qcow2 -O qcow2 $f.qcow2 reclaimed.qcow2 ; mv reclaimed.qcow2 $f.qcow2 ; virsh snapshot-delete $f

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-05-01 Thread Launchpad Bug Tracker
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: qemu (Ubuntu) Status: New = Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-05-01 Thread Jamie Strandboge
FYI, I periodically use and follow the same procedure that Seth described (in fact, I did it yesterday) and had no problems with qemu 1.5.0+dfsg-3ubuntu5.4 (which I've apt pinned since reporting this bug). ** Description changed: The security team uses a tool

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-03-25 Thread Serge Hallyn
I've not yet been able to definitively reproduce this. (On a bad nested qemu setup i had some issues which i think were unrelated). I've tried on a trusty laptop, and on a faster machine with a trusty container on a trusty kernel. Starting with the images you posted for me each time. -- You

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-03-21 Thread Jamie Strandboge
I don't, I just used the options that our uvt command uses. I downgraded to saucy's qemu in the meantime so I can do my work. Do you need me to try some new test? I'm not sure it makes any difference, but note that I am using a trusty host and kernel. -- You received this bug notification

Re: [Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-03-21 Thread Serge Hallyn
Quoting Jamie Strandboge (ja...@ubuntu.com): I don't, I just used the options that our uvt command uses. I downgraded to saucy's qemu in the meantime so I can do my work. Do you need me to try some new test? sigh, maybe. I will keep trying. I'm not sure it makes any difference, but note

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-03-21 Thread Jamie Strandboge
Did you try with the image on https://chinstrap.canonical.com/~jamie/lp1292234/? I was only able to trigger it by using an old image, creating the snapshot, starting it, apt-get dist-upgrading, cleanly shutting down, then deleting the snapshot and creating another with the same name. Using a fresh

Re: [Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-03-21 Thread Serge Hallyn
Quoting Jamie Strandboge (ja...@ubuntu.com): Did you try with the image on https://chinstrap.canonical.com/~jamie/lp1292234/? I was only able to Yup! I wget that, create the snapshot, upgrade, remove and create the snapshot, then start the vm. The upgrades take a long time so I've only tested

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-03-20 Thread Serge Hallyn
Have not yet been able to reproduce this. I'm considering adding an upstart job to your image which updates and shuts down, so I can test this in a loop. Do you know whether (a) the --children option to snapshot delete or (b) using the same name for the new snapshot as the one you just delete

[Bug 1292234] Re: qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate)

2014-03-13 Thread Serge Hallyn
** Changed in: qemu (Ubuntu) Importance: Undecided = High -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption in trusty (qemu 1.7 and 2.0