[Bug 1292234] Re: qcow2 image corruption on non-extent filesystems (ext3)
** Also affects: qemu (Ubuntu Vivid) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Vivid) Importance: Undecided Status: New ** Also affects: qemu (Ubuntu Trusty) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Trusty) Importance: Undecided Status: New ** No longer affects: qemu (Ubuntu) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption on non-extent filesystems (ext3) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1292234/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1292234] Re: qcow2 image corruption on non-extent filesystems (ext3)
Chris, please do, I just recreated the issue with the "uvt update -rf" recipe from earlier; four of six VMs couldn't boot to a login: prompt, presumably from this bug. Linux hunt 3.13.0-65-generic #106-Ubuntu SMP Fri Oct 2 22:08:27 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux (I know, it misses this week's update. I can't keep up on this treadmill...) Thanks -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption on non-extent filesystems (ext3) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1292234/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1292234] Re: qcow2 image corruption on non-extent filesystems (ext3)
The fix is the following: $ git describe --contains 6f30b7e37a8239f9d27db626a1d3427bc7951908 v4.0-rc1~1^2 I thought this was going to be queued up for stable, but doesn't look like that happened. If this still affects you in 3.13, 3.16, I can backport this patch. Let me know. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption on non-extent filesystems (ext3) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1292234/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1292234] Re: qcow2 image corruption on non-extent filesystems (ext3)
Is this still open against the 14.04.1 LTS kernel? Thanks -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption on non-extent filesystems (ext3) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1292234/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1292234] Re: qcow2 image corruption on non-extent filesystems (ext3)
** Changed in: linux (Ubuntu) Status: Confirmed = Fix Committed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption on non-extent filesystems (ext3) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1292234/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1292234] Re: qcow2 image corruption on non-extent filesystems (ext3)
This bug was fixed in the package linux - 3.19.0-12.12 --- linux (3.19.0-12.12) vivid; urgency=low [ Andy Whitcroft ] * [Packaging] do_common_tools should always be on * [Packaging] Provides: virtualbox-guest-modules when appropriate - LP: #1434579 [ Chris J Arges ] * Revert SAUCE: ext4: disable ext4_punch_hole for indirect filesystems - LP: #1292234 [ Leann Ogasawara ] * Release Tracking Bug - LP: #1439803 [ Timo Aaltonen ] * SAUCE: i915_bpo: Provide a backport driver for Skylake Cherryview graphics - LP: #1420774 * SAUCE: i915_bpo: Update intel_ips.h file location - LP: #1420774 * SAUCE: i915_bpo: Only support Skylake and Cherryview with the backport driver - LP: #1420774 * SAUCE: i915_bpo: Rename the backport driver to i915_bpo - LP: #1420774 * i915_bpo: [Config] Enable CONFIG_DRM_I915_BPO=m - LP: #1420774 * SAUCE: i915_bpo: Add i915_bpo_*() calls for ubuntu/i915 - LP: #1420774 * SAUCE: i915_bpo: Revert drm/i915: remove unused power_well/get_cdclk_freq api - LP: #1420774 * SAUCE: i915_bpo: Add i915_bpo specific power well calls - LP: #1420774 * SAUCE: Backport I915_PARAM_MMAP_VERSION and I915_MMAP_WC - LP: #1420774 * SAUCE: Partial backport of drm/i915: Add ioctl to set per-context parameters - LP: #1420774 * SAUCE: drm/i915: Specify bsd rings through exec flag - LP: #1420774 * SAUCE: drm/i915: add I915_PARAM_HAS_BSD2 to i915_getparam - LP: #1420774 * SAUCE: drm/i915: add component support - LP: #1420774 * SAUCE: drm/i915: Add tiled framebuffer modifiers - LP: #1420774 * SAUCE: Backport new displayable tiling formats - LP: #1420774 * SAUCE: Backport drm_crtc_vblank_reset() helper - LP: #1420774 * SAUCE: drm/i915: Add I915_PARAM_REVISION - LP: #1420774 * SAUCE: drm/i915: Export total subslice and EU counts - LP: #1420774 * SAUCE: i915_bpo: Revert drm/mm: Support 4 GiB and larger ranges - LP: #1420774 [ Upstream Kernel Changes ] * drm/i915/skl: Split the SKL PCI ids by GT - LP: #1420774 * drm: Reorganize probed mode validation - LP: #1420774 * drm: Perform basic sanity checks on probed modes - LP: #1420774 * drm: Do basic sanity checks for user modes - LP: #1420774 * drm/atomic-helper: Export both plane and modeset check helpers - LP: #1420774 * drm/atomic-helper: Again check modeset *before* plane states - LP: #1420774 * drm/atomic: Introduce state-obj backpointers - LP: #1420774 * drm: allow property validation for refcnted props - LP: #1420774 * drm: store property instead of id in obj attachment - LP: #1420774 * drm: get rid of direct property value access - LP: #1420774 * drm: add atomic_set_property wrappers - LP: #1420774 * drm: tweak getconnector locking - LP: #1420774 * drm: add atomic_get_property - LP: #1420774 * drm: Remove unneeded braces for single statement blocks - LP: #1420774 * drm: refactor getproperties/getconnector - LP: #1420774 * drm: add atomic properties - LP: #1420774 * drm/atomic: atomic_check functions - LP: #1420774 * drm: small property creation cleanup - LP: #1420774 * drm/atomic: atomic plane properties - LP: #1420774 * drm/atomic: atomic connector properties - LP: #1420774 * drm: Atomic modeset ioctl - LP: #1420774 * drm/atomic: Hide drm.ko internal interfaces - LP: #1420774 * drm: Ensure universal_planes is set for atomic - LP: #1420774 * drm: fix mismerge in drm_crtc.c - LP: #1420774 * drm: bit of spell-check / editorializing. - LP: #1420774 * drm: add support for tiled/compressed/etc modifier in addfb2 - LP: #1420774 * drm: Add rotation value to plane state - LP: #1420774 * drm: add helper to get crtc timings (v5) - LP: #1420774 * drm: Adding edp1.4 specific dpcd macros - LP: #1420774 * drm/plane-helper: Skip prepare_fb/cleanup_fb when newfb==oldfb - LP: #1420774 * next: drm/atomic: Use copy_from_user to copy 64 bit data from user space - LP: #1420774 * powerpc: Add PVR for POWER8NVL processor - LP: #1438938 * kernel/sched/clock.c: add another clock for use with the soft lockup watchdog - LP: #1427075 * powerpc: add running_clock for powerpc to prevent spurious softlockup warnings - LP: #1427075 * ext4: fix indirect punch hole corruption - LP: #1292234 * fm10k: Clean-up page reuse code - LP: #1397863 * net/fm10k: Avoid double setting of NETIF_F_SG for the HW encapsulation feature mask - LP: #1397863 * fm10k: Check tunnel header length in encap offload - LP: #1397863 * fm10k: Increase the timeout for the data path reset - LP: #1397863 * fm10k: Validate VLAN ID in fm10k_update_xc_addr_pf - LP: #1397863 * fm10k: Resolve compile warnings with W=1 - LP: #1397863 * UBUNTU Config: CONFIG_IXGBE_VXLAN=y - LP: #1397861 * ixgbe:
[Bug 1292234] Re: qcow2 image corruption on non-extent filesystems (ext3)
** Changed in: linux (Ubuntu) Status: Fix Released = Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption on non-extent filesystems (ext3) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1292234/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1292234] Re: qcow2 image corruption on non-extent filesystems (ext3)
** Description changed: [Impact] Users of non-extent ext4 filesystems (ext4 ^extents, or ext3 w/ CONFIG_EXT4_USE_FOR_EXT23=y) can encounter data corruption when using fallocate with FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE flags. [Test Case] 1) Setup ext4 ^extents, or ext3 filesystem with CONFIG_EXT4_USE_FOR_EXT23=y 2) Create and install a VM using a qcow2 image and store the file on the filesystem 3) Snapshot the image with qemu-img 4) Boot the image and do some disk operations (fio,etc) 5) Shutdown image and delete snapshot 6) Repeat 3-5 until VM no longer boots due to image corruption, generally this takes a few iterations depending on disk operations. [Fix] - This is being discussed upstream here: + commit 6f30b7e37a8239f9d27db626a1d3427bc7951908 upstream + + This has been discussed upstream here: http://marc.info/?l=linux-fsdevelm=142264422605440w=2 A temporary fix would be to disable punch_hole for non-extent filesystem. This is how the normal ext3 module handles this and it is up to userspace to handle the failure. I've run this with the test case and was able to run for 600 iterations over 3 days where most failures occur within the first 2-20 iterations. diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c index 5653fa4..e14cdfe 100644 --- a/fs/ext4/inode.c +++ b/fs/ext4/inode.c @@ -3367,6 +3367,10 @@ int ext4_punch_hole(struct inode *inode, loff_t offset, loff_t length) unsigned int credits; int ret = 0; + /* EXTENTS required */ + if (!(ext4_test_inode_flag(inode, EXT4_INODE_EXTENTS))) + return -EOPNOTSUPP; + if (!S_ISREG(inode-i_mode)) return -EOPNOTSUPP; -- The security team uses a tool (http://bazaar.launchpad.net/~ubuntu- bugcontrol/ubuntu-qa-tools/master/view/head:/vm-tools/uvt) that uses libvirt snapshots quite a bit. I noticed after upgrading to trusty some time ago that qemu 1.7 (and the qemu 2.0 in the candidate ppa) has had stability problems such that the disk/partition table seems to be corrupted after removing a libvirt snapshot and then creating another with the same name. I don't have a very simple reproducer, but had enough that hallyn suggested I file a bug. First off: qemu-kvm 2.0~git-20140307.4c288ac-0ubuntu2 $ cat /proc/version_signature Ubuntu 3.13.0-16.36-generic 3.13.5 $ qemu-img info ./forhallyn-trusty-amd64.img image: ./forhallyn-trusty-amd64.img file format: qcow2 virtual size: 8.0G (8589934592 bytes) disk size: 4.0G cluster_size: 65536 Format specific information: compat: 0.10 Steps to reproduce: 1. create a virtual machine. For a simplified reproducer, I used virt-manager with: OS type: Linux Version: Ubuntu 14.04 Memory: 768 CPUs: 1 Select managed or existing (Browse, new volume) Create a new storage volume: qcow2 Max capacity: 8192 Allocation: 0 Advanced: NAT kvm x86_64 firmware: default 2. install a VM. I used trusty-desktop-amd64.iso from Jan 23 since it seems like I can hit the bug more reliably if I have lots of updates in a dist-upgrade. I have seen this with lucid-trusty guests that are i386 and amd64. After the install, reboot and then cleanly shutdown 3. Backup the image file somewhere since steps 1 and 2 take a while :) 4. Execute the following commands which are based on what our uvt tool does: $ virsh snapshot-create-as forhallyn-trusty-amd64 pristine uvt snapshot $ virsh snapshot-current --name forhallyn-trusty-amd64 pristine $ virsh start forhallyn-trusty-amd64 $ virsh snapshot-list forhallyn-trusty-amd64 # this is showing as shutoff after start, this might be different with qemu 1.5 in guest: sudo apt-get update sudo apt-get dist-upgrade 780 upgraded... shutdown -h now $ virsh snapshot-delete forhallyn-trusty-amd64 pristine --children $ virsh snapshot-create-as forhallyn-trusty-amd64 pristine uvt snapshot $ virsh start forhallyn-trusty-amd64 # this command works, but there is often disk corruption The idea behind the above is to create a new VM with a pristine snapshot that we could revert later if we wanted. Instead, we boot the VM, run apt-get dist-upgrade, cleanly shutdown and then remove the old 'pristine' snapshot and create a new 'pristine' snapshot. The intention is to update the VM and the pristine snapshot so that when we boot the next time, we boot from the updated VM and can revert back to the updated VM. After running 'virsh start' after doing snapshot-delete/snapshot-create- as, the disk may be corrupted. This can be seen with grub failing to find .mod files, the kernel not booting, init failing, etc. This does not seem to be related to the machine type used. Ie, pc- i440fx-1.5, pc-i440fx-1.7 and pc-i440fx-2.0 all fail with qemu 2.0, pc- i440fx-1.5 and pc-i440fx-1.7 fail with qemu 1.7 and
[Bug 1292234] Re: qcow2 image corruption on non-extent filesystems (ext3)
Sent email to upstream stable to apply this bug to affected kernels. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption on non-extent filesystems (ext3) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1292234/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1292234] Re: qcow2 image corruption on non-extent filesystems (ext3)
Note there currently is a patch upstream: https://lkml.org/lkml/2015/2/10/520 This fixes the original bug correctly without having to disable ext4_punch_hole for indirect filesystems. Once this lands in Linus' tree, I'll file an SRU to get this fixed across the board. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption on non-extent filesystems (ext3) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1292234/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1292234] Re: qcow2 image corruption on non-extent filesystems (ext3)
Woohoo! *Huge* thanks. This was a tricky one :) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption on non-extent filesystems (ext3) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1292234/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1292234] Re: qcow2 image corruption on non-extent filesystems (ext3)
This bug was fixed in the package linux - 3.18.0-13.14 --- linux (3.18.0-13.14) vivid; urgency=low [ Andy Whitcroft ] * hyper-v -- fix comment handing in /etc/network/interfaces - LP: #1413020 [ Chris J Arges ] * [Config] Add ibmvfc to d-i - LP: #1416001 * SAUCE: ext4: disable ext4_punch_hole for indirect filesystems - LP: #1292234 [ Leann Ogasawara ] * rebase to v3.18.5 * [Config] CONFIG_X86_UP_APIC_MSI=y * Release Tracking Bug - LP: #1417475 -- Leann Ogasawara leann.ogasaw...@canonical.com Thu, 05 Feb 2015 09:58:20 +0200 ** Changed in: linux (Ubuntu) Status: In Progress = Fix Released -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption on non-extent filesystems (ext3) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1292234/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1292234] Re: qcow2 image corruption on non-extent filesystems (ext3)
@josep-m-perez Yes, this is an upstream bug. So it affects anyone using the right filesystem and CONFIGs. Once we fix this upstream, then it will be submitted as a stable kernel update and make its way into all stable kernels as applicable. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption on non-extent filesystems (ext3) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1292234/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1292234] Re: qcow2 image corruption on non-extent filesystems (ext3)
** Description changed: [Impact] Users of non-extent ext4 filesystems (ext4 ^extents, or ext3 w/ CONFIG_EXT4_USE_FOR_EXT23=y) can encounter data corruption when using fallocate with FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE flags. This seems to be a regression in ext4_ind_remove_space introduced in 4f579ae7, whereas commit 77ea2a4b passes the following test case. [Test Case] 1) Setup ext4 ^extents, or ext3 filesystem with CONFIG_EXT4_USE_FOR_EXT23=y 2) Create and install a VM using a qcow2 image and store the file on the filesystem 3) Snapshot the image with qemu-img 4) Boot the image and do some disk operations (fio,etc) 5) Shutdown image and delete snapshot 6) Repeat 3-5 until VM no longer boots due to image corruption, generally this takes a few iterations depending on disk operations. + + [Fix] + This is being discussed upstream here: + http://marc.info/?l=linux-fsdevelm=142264422605440w=2 + + A temporary fix would be to disable punch_hole for non-extent + filesystem. This is how the normal ext3 module handles this and it is up + to userspace to handle the failure. + + diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c + index 5653fa4..e14cdfe 100644 + --- a/fs/ext4/inode.c + +++ b/fs/ext4/inode.c + @@ -3367,6 +3367,10 @@ int ext4_punch_hole(struct inode *inode, loff_t + offset, loff_t length) + unsigned int credits; + int ret = 0; + + + /* EXTENTS required */ + + if (!(ext4_test_inode_flag(inode, EXT4_INODE_EXTENTS))) + + return -EOPNOTSUPP; + + + if (!S_ISREG(inode-i_mode)) + return -EOPNOTSUPP; -- The security team uses a tool (http://bazaar.launchpad.net/~ubuntu- bugcontrol/ubuntu-qa-tools/master/view/head:/vm-tools/uvt) that uses libvirt snapshots quite a bit. I noticed after upgrading to trusty some time ago that qemu 1.7 (and the qemu 2.0 in the candidate ppa) has had stability problems such that the disk/partition table seems to be corrupted after removing a libvirt snapshot and then creating another with the same name. I don't have a very simple reproducer, but had enough that hallyn suggested I file a bug. First off: qemu-kvm 2.0~git-20140307.4c288ac-0ubuntu2 $ cat /proc/version_signature Ubuntu 3.13.0-16.36-generic 3.13.5 $ qemu-img info ./forhallyn-trusty-amd64.img image: ./forhallyn-trusty-amd64.img file format: qcow2 virtual size: 8.0G (8589934592 bytes) disk size: 4.0G cluster_size: 65536 Format specific information: compat: 0.10 Steps to reproduce: 1. create a virtual machine. For a simplified reproducer, I used virt-manager with: OS type: Linux Version: Ubuntu 14.04 Memory: 768 CPUs: 1 Select managed or existing (Browse, new volume) Create a new storage volume: qcow2 Max capacity: 8192 Allocation: 0 Advanced: NAT kvm x86_64 firmware: default 2. install a VM. I used trusty-desktop-amd64.iso from Jan 23 since it seems like I can hit the bug more reliably if I have lots of updates in a dist-upgrade. I have seen this with lucid-trusty guests that are i386 and amd64. After the install, reboot and then cleanly shutdown 3. Backup the image file somewhere since steps 1 and 2 take a while :) 4. Execute the following commands which are based on what our uvt tool does: $ virsh snapshot-create-as forhallyn-trusty-amd64 pristine uvt snapshot $ virsh snapshot-current --name forhallyn-trusty-amd64 pristine $ virsh start forhallyn-trusty-amd64 $ virsh snapshot-list forhallyn-trusty-amd64 # this is showing as shutoff after start, this might be different with qemu 1.5 in guest: sudo apt-get update sudo apt-get dist-upgrade 780 upgraded... shutdown -h now $ virsh snapshot-delete forhallyn-trusty-amd64 pristine --children $ virsh snapshot-create-as forhallyn-trusty-amd64 pristine uvt snapshot $ virsh start forhallyn-trusty-amd64 # this command works, but there is often disk corruption The idea behind the above is to create a new VM with a pristine snapshot that we could revert later if we wanted. Instead, we boot the VM, run apt-get dist-upgrade, cleanly shutdown and then remove the old 'pristine' snapshot and create a new 'pristine' snapshot. The intention is to update the VM and the pristine snapshot so that when we boot the next time, we boot from the updated VM and can revert back to the updated VM. After running 'virsh start' after doing snapshot-delete/snapshot-create- as, the disk may be corrupted. This can be seen with grub failing to find .mod files, the kernel not booting, init failing, etc. This does not seem to be related to the machine type used. Ie, pc- i440fx-1.5, pc-i440fx-1.7 and pc-i440fx-2.0 all fail with qemu 2.0, pc- i440fx-1.5 and pc-i440fx-1.7 fail with qemu 1.7 and pc-i440fx-1.5 works fine with qemu 1.5. Only workaround I know if is to downgrade
[Bug 1292234] Re: qcow2 image corruption on non-extent filesystems (ext3)
Apparently this bug is also present in Debian. In my case the corrupted image was a windows one. When I run qemu-img check over it it will complain about lots of clusters, and if I pass it the repair flag, then it will end up crashing with the following message: $ qemu-img check -r all windows.img Repairing cluster 0 refcount=0 reference=1 Repairing cluster 1 refcount=0 reference=1 qcow2: Preventing invalid write on metadata (overlaps with active L1 table); image marked as corrupt. Repairing cluster 2 refcount=0 reference=1 qcow2: Preventing invalid write on metadata (overlaps with active L1 table); image marked as corrupt. qcow2: Preventing invalid write on metadata (overlaps with active L1 table); image marked as corrupt. Repairing cluster 3 refcount=0 reference=1 qcow2: Preventing invalid write on metadata (overlaps with active L1 table); image marked as corrupt. qcow2: Preventing invalid write on metadata (overlaps with active L1 table); image marked as corrupt. qcow2: Preventing invalid write on metadata (overlaps with active L1 table); image marked as corrupt. Repairing cluster 4 refcount=0 reference=1 qcow2: Preventing invalid write on metadata (overlaps with active L1 table); image marked as corrupt. qcow2: Preventing invalid write on metadata (overlaps with active L1 table); image marked as corrupt. qcow2: Preventing invalid write on metadata (overlaps with active L1 table); image marked as corrupt. qcow2: Preventing invalid write on metadata (overlaps with active L1 table); image marked as corrupt. Repairing cluster 5 refcount=0 reference=1 qcow2: Preventing invalid write on metadata (overlaps with active L1 table); image marked as corrupt. Repairing cluster 6 refcount=0 reference=1 ... Repairing OFLAG_COPIED data cluster: l2_entry=800397a59000 refcount=0 Repairing OFLAG_COPIED data cluster: l2_entry=800397a5a000 refcount=0 Repairing OFLAG_COPIED data cluster: l2_entry=8001b000 refcount=0 The following inconsistencies were found and repaired: 0 leaked clusters 97850 corruptions Double checking the fixed image now... [1]27716 segmentation fault (core dumped) qemu-img check -r all windows.img Has anyone else tried this over a copy of the corrupted image? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption on non-extent filesystems (ext3) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1292234/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1292234] Re: qcow2 image corruption on non-extent filesystems (ext3)
** Also affects: linux (Ubuntu) Importance: Undecided Status: New ** Changed in: linux (Ubuntu) Assignee: (unassigned) = Chris J Arges (arges) ** Changed in: linux (Ubuntu) Importance: Undecided = High ** Changed in: linux (Ubuntu) Status: New = In Progress ** Changed in: qemu (Ubuntu) Status: In Progress = Invalid ** Changed in: qemu (Ubuntu) Assignee: Chris J Arges (arges) = (unassigned) ** Changed in: qemu (Ubuntu) Importance: High = Undecided -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption on non-extent filesystems (ext3) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1292234/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1292234] Re: qcow2 image corruption on non-extent filesystems (ext3)
** Description changed: + [Impact] + Users of non-extent ext4 filesystems (ext4 ^extents, or ext3 w/ CONFIG_EXT4_USE_FOR_EXT23=y) can encounter data corruption when using fallocate with FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE flags. This seems to be a regression in ext4_ind_remove_space introduced in 4f579ae7, whereas commit 77ea2a4b passes the following test case. + + [Test Case] + 1) Setup ext4 ^extents, or ext3 filesystem with CONFIG_EXT4_USE_FOR_EXT23=y + 2) Create and install a VM using a qcow2 image and store the file on the filesystem + 3) Snapshot the image with qemu-img + 4) Boot the image and do some disk operations (fio,etc) + 5) Shutdown image and delete snapshot + 6) Repeat 3-5 until VM no longer boots due to image corruption, generally this takes a few iterations depending on disk operations. + + + -- + The security team uses a tool (http://bazaar.launchpad.net/~ubuntu- bugcontrol/ubuntu-qa-tools/master/view/head:/vm-tools/uvt) that uses libvirt snapshots quite a bit. I noticed after upgrading to trusty some time ago that qemu 1.7 (and the qemu 2.0 in the candidate ppa) has had stability problems such that the disk/partition table seems to be corrupted after removing a libvirt snapshot and then creating another with the same name. I don't have a very simple reproducer, but had enough that hallyn suggested I file a bug. First off: qemu-kvm 2.0~git-20140307.4c288ac-0ubuntu2 $ cat /proc/version_signature Ubuntu 3.13.0-16.36-generic 3.13.5 $ qemu-img info ./forhallyn-trusty-amd64.img image: ./forhallyn-trusty-amd64.img file format: qcow2 virtual size: 8.0G (8589934592 bytes) disk size: 4.0G cluster_size: 65536 Format specific information: compat: 0.10 Steps to reproduce: 1. create a virtual machine. For a simplified reproducer, I used virt-manager with: OS type: Linux Version: Ubuntu 14.04 Memory: 768 CPUs: 1 Select managed or existing (Browse, new volume) Create a new storage volume: qcow2 Max capacity: 8192 Allocation: 0 Advanced: NAT kvm x86_64 firmware: default 2. install a VM. I used trusty-desktop-amd64.iso from Jan 23 since it seems like I can hit the bug more reliably if I have lots of updates in a dist-upgrade. I have seen this with lucid-trusty guests that are i386 and amd64. After the install, reboot and then cleanly shutdown 3. Backup the image file somewhere since steps 1 and 2 take a while :) 4. Execute the following commands which are based on what our uvt tool does: $ virsh snapshot-create-as forhallyn-trusty-amd64 pristine uvt snapshot $ virsh snapshot-current --name forhallyn-trusty-amd64 pristine $ virsh start forhallyn-trusty-amd64 $ virsh snapshot-list forhallyn-trusty-amd64 # this is showing as shutoff after start, this might be different with qemu 1.5 in guest: sudo apt-get update sudo apt-get dist-upgrade 780 upgraded... shutdown -h now $ virsh snapshot-delete forhallyn-trusty-amd64 pristine --children $ virsh snapshot-create-as forhallyn-trusty-amd64 pristine uvt snapshot $ virsh start forhallyn-trusty-amd64 # this command works, but there is often disk corruption The idea behind the above is to create a new VM with a pristine snapshot that we could revert later if we wanted. Instead, we boot the VM, run apt-get dist-upgrade, cleanly shutdown and then remove the old 'pristine' snapshot and create a new 'pristine' snapshot. The intention is to update the VM and the pristine snapshot so that when we boot the next time, we boot from the updated VM and can revert back to the updated VM. After running 'virsh start' after doing snapshot-delete/snapshot-create- as, the disk may be corrupted. This can be seen with grub failing to find .mod files, the kernel not booting, init failing, etc. This does not seem to be related to the machine type used. Ie, pc- i440fx-1.5, pc-i440fx-1.7 and pc-i440fx-2.0 all fail with qemu 2.0, pc- i440fx-1.5 and pc-i440fx-1.7 fail with qemu 1.7 and pc-i440fx-1.5 works fine with qemu 1.5. Only workaround I know if is to downgrade qemu to 1.5.0+dfsg-3ubuntu5.4 from Ubuntu 13.10. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption on non-extent filesystems (ext3) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1292234/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1292234] Re: qcow2 image corruption on non-extent filesystems (ext3)
Sent e-mail upstream about this issue: http://marc.info/?l=linux- fsdevelm=142264422605440w=2 -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption on non-extent filesystems (ext3) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1292234/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1292234] Re: qcow2 image corruption on non-extent filesystems (ext3)
** No longer affects: qemu -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption on non-extent filesystems (ext3) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1292234/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1292234] Re: qcow2 image corruption on non-extent filesystems (ext3)
** Summary changed: - qcow2 image corruption in trusty (qemu 1.7 and 2.0 candidate) + qcow2 image corruption on non-extent filesystems (ext3) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1292234 Title: qcow2 image corruption on non-extent filesystems (ext3) To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1292234/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
