[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
According to comment #8 the fixes have been included in the upstream QEMU repository, so setting the status to "Fix released" now. ** Changed in: qemu Status: In Progress => Fix Released -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in Cinder: Won't Fix Status in OpenStack Compute (nova): Won't Fix Status in QEMU: Fix Released Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Released Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced "easily" on command line using cat $SRC_PATH > $TMP_PATH && $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH && cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
Closing based on the assumption that a working qemu-img is available now. ** Changed in: cinder Status: Triaged => Won't Fix -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in Cinder: Won't Fix Status in OpenStack Compute (nova): Won't Fix Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Released Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced "easily" on command line using cat $SRC_PATH > $TMP_PATH && $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH && cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
** Changed in: nova Milestone: kilo-3 = None -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in Cinder: Triaged Status in OpenStack Compute (Nova): Won't Fix Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Released Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
Marking as Wont-Fix. ** Changed in: nova Status: In Progress = Won't Fix -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in Cinder: Triaged Status in OpenStack Compute (Nova): Won't Fix Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Released Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
** Changed in: nova Milestone: kilo-2 = kilo-3 -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in Cinder: Triaged Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Released Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
** Changed in: nova Milestone: None = kilo-2 -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in Cinder: Triaged Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Released Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
** Changed in: cinder Status: In Progress = Triaged ** Changed in: cinder Assignee: Tony Breeds (o-tony) = (unassigned) -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in Cinder: Triaged Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Released Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
** Changed in: cinder Assignee: (unassigned) = Tony Breeds (o-tony) -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in Cinder: Triaged Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Released Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
Fix proposed to branch: master Review: https://review.openstack.org/143575 ** Changed in: cinder Assignee: John Griffith (john-griffith) = Tony Breeds (o-tony) -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in Cinder: In Progress Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Released Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
The second part of the fix (which does not reduce the performance) for qemu 2.0 (apparently uploading two patches at once is not so easy) ** Patch added: 0502-block-raw-posix-use-seek_hole-ahead-of-fiemap.patch https://bugs.launchpad.net/qemu/+bug/1368815/+attachment/4282119/+files/0502-block-raw-posix-use-seek_hole-ahead-of-fiemap.patch -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in Cinder: In Progress Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Released Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
- 501-block-raw-posix-fix-disk-corruption-in-try-fiemap - (note - 502-block-raw-posic-use-seek-hole-ahead-of-fiemap (which was also needed in utopic) appears to be unneeded here as the code being changed has not yet been switched to using try_fiemap) Actually such a enforces fsync and drastically reduces the performance of conversion. I propose to use seek_hole instead of FIEMAP (which is basically what 502-block-raw-posic-use-seek-hole-ahead-of-fiemap does). ** Patch added: 0500-block-raw-posix-Try-both-FIEMAP-and-SEEK_HOLE.patch https://bugs.launchpad.net/qemu/+bug/1368815/+attachment/4282118/+files/0500-block-raw-posix-Try-both-FIEMAP-and-SEEK_HOLE.patch -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in Cinder: In Progress Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Released Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
Patchg 0500-block-raw-posix-Try-both-FIEMAP-and-SEEK_HOLE.patch appears to be part of a bigger re-write of the related code. and is ON TOP of the patches already applied in this bug. No doubt the rewirtten code is better but backporting it contains more risk than the 2 simple fixes I already nominated. -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in Cinder: In Progress Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Released Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
Patch 0500-block-raw-posix-Try-both-FIEMAP-and-SEEK_HOLE.patch appears to be part of a bigger re-write of the related code. and is ON TOP of the patches already applied in this bug. Yep, sorry for not mentioning this. As far as I understand qemu-2.1 package contains this partially rewritten code too (without any recent changes like disabling FIEMAP completely and rewriting the code using SEEK_HOLE). No doubt the rewirtten code is better but backporting it contains more risk than the 2 simple fixes I already nominated. Can we completely disable the FIEMAP code and pretend that all blocks are allocated? I'm afraid fsync'ing 100+ GB files might be even slower than ignoring the sparseness. -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in Cinder: In Progress Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Released Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
** Changed in: cinder Importance: Undecided = High -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in Cinder: In Progress Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Released Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
This bug was fixed in the package qemu - 2.1+dfsg-4ubuntu6.2 --- qemu (2.1+dfsg-4ubuntu6.2) utopic-proposed; urgency=medium * Apply two patches to fix intermittent qemu-img corruption (LP: #1368815) - 501-block-raw-posix-fix-disk-corruption-in-try-fiemap - 502-block-raw-posic-use-seek-hole-ahead-of-fiemap -- Serge Hallyn serge.hal...@ubuntu.com Thu, 20 Nov 2014 16:33:09 -0600 ** Changed in: qemu (Ubuntu Utopic) Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Released Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
** Also affects: cinder Importance: Undecided Status: New -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in Cinder: New Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Released Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
I'd elevate this to high so it matches nova and ubuntu but I don't have permissions to do so. ** Changed in: cinder Status: New = In Progress -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in Cinder: In Progress Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Released Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
I'm happy to tackle to also fix cinder with a version of the nova fix (for consistency). I propose waiting until the nova fix lands ** Changed in: cinder Assignee: (unassigned) = Tony Breeds (o-tony) -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in Cinder: In Progress Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Released Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
Fix proposed to branch: master Review: https://review.openstack.org/141259 ** Changed in: cinder Assignee: Tony Breeds (o-tony) = John Griffith (john-griffith) -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in Cinder: In Progress Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Released Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
@Michael, by any chance would you be albe to test on utopic? -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Committed Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
I couldn't reproduce the bug on the old qemu myself, however Michael has verified the (same) fix on trusty, and the full qa-regression-test passed for me on utopic-proposed. So I would request that we call this verification-done. -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in Mirantis OpenStack: Triaged Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Committed Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/mos/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
** Also affects: mos Importance: Undecided Status: New ** Changed in: mos Status: New = Triaged ** Changed in: mos Importance: Undecided = Critical ** Changed in: mos Assignee: (unassigned) = MOS Linux (mos-linux) ** Changed in: mos Milestone: None = 6.0 -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in Mirantis OpenStack: Triaged Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Committed Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/mos/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
Looking at the fixes, I also see the following commits remove the above changes, which could mean we might encounter this again: c4875e5 raw-posix: SEEK_HOLE suffices, get rid of FIEMAP d1f06fe raw-posix: The SEEK_HOLE code is flawed, rewrite it Note there is also a related issue: bug 1292234 So far testing with the proposed qemu version or upstream I still encounter issues on ext4 w/ ^extent and ext3 filesystems. -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in Mirantis OpenStack: Triaged Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Committed Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/mos/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
** Tags removed: verification-needed-utopic ** Tags added: verification-done-utopic -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Committed Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
Filed a separate issue for MOS https://bugs.launchpad.net/mos/+bug/1401261 ** No longer affects: mos -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Committed Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
Hi Chris, Markus' rework will not reintroduce this bug as it completely removes all fiemap code. bug 129224 is a different issue, I'll comment on that bug. You say: you encounter issues with upstream with ^extent and ext3 filesystems. Just to be clear: Are you saying that *this* bug is still a problem for you? if it's a different bug then I write it up and I'll take a look. -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Committed Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
Tony, Yea, its a different bug. I tested with the above patched package and upstream qemu from git, and I can still hit bug 129224. I was hoping this also fixed my issue, but unfortunately it seems to be a different issue that occurs when using the same types of filesystems. I have a solid reproducer on my desk so let me know which experiments / areas of code / etc I should look at. -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Committed Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
Just to clarify it's bug 1292234 in the previous comment. -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Committed Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
Chris, I've read through 1292234 and I'll have a play with your reproducer locally and see if I can gain any insight. I'm sorry my fix didn't help 1292234, but glad you can't hit 1368815 with upstream, I was kinda having kittens here ;P -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Committed Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
This bug was fixed in the package qemu - 2.0.0+dfsg-2ubuntu1.8 --- qemu (2.0.0+dfsg-2ubuntu1.8) trusty-proposed; urgency=medium * debian/qemu-system-x86.qemu-kvm.upstart: create /dev/kvm in a container. (LP: #1370199) * Cherrypick upstream patch to fix intermittent qemu-img corruption (LP: #1368815) - 501-block-raw-posix-fix-disk-corruption-in-try-fiemap - (note - 502-block-raw-posic-use-seek-hole-ahead-of-fiemap (which was also needed in utopic) appears to be unneeded here as the code being changed has not yet been switched to using try_fiemap) -- Serge Hallyn serge.hal...@ubuntu.com Thu, 20 Nov 2014 11:24:51 -0600 ** Changed in: qemu (Ubuntu Trusty) Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Committed Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
** Tags added: verification-needed-utopic -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Committed Status in qemu source package in Utopic: Fix Committed Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
Tested qemu-utils 2.0.0+dfsg-2ubuntu1.8. Successful. -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Committed Status in qemu source package in Utopic: Fix Committed Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
** Tags removed: verification-needed ** Tags added: verification-done-trusty -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Committed Status in qemu source package in Utopic: Fix Committed Status in qemu source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
** Description changed: == - Impact: occasional qcow2 corruption + Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command - qemu-img convert -O raw inputimage.qcow2 outputimage.raw + qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using - cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH + cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in - http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html + http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in - http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html + http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change - f.fm.fm_flags = 0; + f.fm.fm_flags = 0; to - f.fm.fm_flags = FIEMAP_FLAG_SYNC; + f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also - https://bugs.launchpad.net/nova/+bug/1350766 + https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in “qemu” package in Ubuntu: Fix Released Status in “qemu” source package in Trusty: Triaged Status in “qemu” source package in Utopic: Triaged Status in “qemu” source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
Hello Michael, or anyone else affected, Accepted qemu into utopic-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/qemu/2.1+dfsg- 4ubuntu6.2 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: qemu (Ubuntu Utopic) Status: Triaged = Fix Committed ** Tags added: verification-needed ** Changed in: qemu (Ubuntu Trusty) Status: Triaged = Fix Committed -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in “qemu” package in Ubuntu: Fix Released Status in “qemu” source package in Trusty: Fix Committed Status in “qemu” source package in Utopic: Fix Committed Status in “qemu” source package in Vivid: Fix Released Bug description: == Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
** Description changed: + == + Impact: occasional qcow2 corruption + Test case: see the qemu-img command below + Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. + == + -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in “qemu” package in Ubuntu: Fix Released Status in “qemu” source package in Trusty: Triaged Status in “qemu” source package in Utopic: Triaged Status in “qemu” source package in Vivid: Fix Released Bug description: == Impact: occasional qcow2 corruption Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
Hi Serge, Is there any chance these fixes will go into trusty? -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in “qemu” package in Ubuntu: Fix Released Status in “qemu” source package in Trusty: Triaged Status in “qemu” source package in Utopic: Triaged Status in “qemu” source package in Vivid: Fix Released Bug description: == Impact: occasional qcow2 corruption Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
Hi Tony, yes, I've uploaded a proposed fix for trusty-proposed earlier today. It should be available for testing as soon as it is accepted. -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in “qemu” package in Ubuntu: Fix Released Status in “qemu” source package in Trusty: Triaged Status in “qemu” source package in Utopic: Triaged Status in “qemu” source package in Vivid: Fix Released Bug description: == Impact: occasional qcow2 corruption Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
Awesome. Thanks! -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in “qemu” package in Ubuntu: Fix Released Status in “qemu” source package in Trusty: Triaged Status in “qemu” source package in Utopic: Triaged Status in “qemu” source package in Vivid: Fix Released Bug description: == Impact: occasional qcow2 corruption Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-insignificantly older qemu source tree. While the cherrypick seems sane, it's possible that there are subtle interactions with the other delta. I'd really like for a full qa-regression-test qemu testcase to be run against this package. == -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
This bug was fixed in the package qemu - 2.1+dfsg-4ubuntu7 --- qemu (2.1+dfsg-4ubuntu7) vivid; urgency=medium * Apply two patches to fix intermittent qemu-img corruption (LP: #1368815) - 501-block-raw-posix-fix-disk-corruption-in-try-fiemap - 502-block-raw-posic-use-seek-hole-ahead-of-fiemap -- Serge Hallyn serge.hal...@ubuntu.com Wed, 29 Oct 2014 22:31:43 -0500 ** Changed in: qemu (Ubuntu Vivid) Status: Confirmed = Fix Released -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in “qemu” package in Ubuntu: Fix Released Status in “qemu” source package in Trusty: Triaged Status in “qemu” source package in Utopic: Triaged Status in “qemu” source package in Vivid: Fix Released Bug description: -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: qemu (Ubuntu Vivid) Status: New = Confirmed -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in “qemu” package in Ubuntu: Confirmed Status in “qemu” source package in Trusty: Triaged Status in “qemu” source package in Utopic: Triaged Status in “qemu” source package in Vivid: Confirmed Bug description: -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
Thanks for the information. Looks like we can apply these in debian too. ** Also affects: qemu (Ubuntu Utopic) Importance: High Status: Triaged ** Also affects: qemu (Ubuntu Vivid) Importance: Undecided Status: New -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in “qemu” package in Ubuntu: Triaged Status in “qemu” source package in Trusty: Triaged Status in “qemu” source package in Utopic: Triaged Status in “qemu” source package in Vivid: New Bug description: -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
FWIW the following 2 commits in qemu master resolve the issue for qemu- img. http://git.qemu.org/?p=qemu.git;a=commit;h=38c4d0aea3e1264c86e282d99560330adf2b6e25 http://git.qemu.org/?p=qemu.git;a=commit;h=7c15903789953ead14a417882657d52dc0c19a24 If possible they should be back ported to trusty and utopic. You'll also need something like: http://git.qemu.org/?p=qemu.git;a=commit;h=4f11aa8a40351b28c0e67c7276e0003b38cc46ac before my 2 patches. -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in “qemu” package in Ubuntu: Triaged Status in “qemu” source package in Trusty: Triaged Bug description: -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
openstack review at: https://review.openstack.org/#/c/123957/ Qemu patches at: http://patchwork.ozlabs.org/patch/393494/ ; and http://patchwork.ozlabs.org/patch/393495/ ** Changed in: nova Assignee: (unassigned) = Tony Breeds (o-tony) ** Changed in: qemu Status: New = In Progress ** Changed in: qemu Assignee: (unassigned) = Tony Breeds (o-tony) ** Changed in: nova Status: Triaged = In Progress -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): In Progress Status in QEMU: In Progress Status in “qemu” package in Ubuntu: Triaged Status in “qemu” source package in Trusty: Triaged Bug description: -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
It seems the dust settles a bit: Found the relevant difference between my various filesystems, and how to reproduce the failure: Susceptible filesystems don't have the extent feature of ext4 enabled. You can create such a filesystem using mke2fs -t ext4 -O ^extent /dev/... mount /mnt /dev/... Adapting the command line example provided above you can see rm -f /mnt/tmp.qcow2 cat $SRC_PATH /mnt/tmp.qcow2 qemu-img convert -O raw /mnt/tmp.qcow /mnt/tmp.qcow cksum /mnt/tmp.qcow creating corrupt (usually nullified) result images. By inserting a sleep of at least 33 seconds between the cat command and the qemu-img invocation I'm getting proper output. To me it's unclear now, where the actual defect is located. Creating ext4 filesystems with certain features disabled (such as the exetent tree) is apparently supported and ok. Is the fiemap ioctl supposed to handle this gracefully, for example by assuming FIEMAP_FLAG_SYNC in absence of an extent tree? Or are clients such as qemu-img supposed to always FIEMAP_FLAG_SYNC to be safe? -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): Triaged Status in QEMU: New Bug description: -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
The fallback cascade Pádraig mentions is already implemented in qemu-2.1.0, in function raw_co_get_block_status. Just swap ret = try_fiemap( ... ) and ret = try_seek_hole( ... ) to reverse the order. I can confirm that it works just fine on 3.13 kernel (all version since 3.1, according to lseek(2)), while older versions will fall back to fiemap, which needs to be protected with FIEMAP_FLAG_SYNC in try_fiemap, to be safe. This should work under all conditions, and avoid redundant syncs where possible, right? -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): Triaged Status in QEMU: New Bug description: -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
I see seek hole is supported in the latest qemu-img so I would reorder so that's tried first like: if lseek(SEEK_HOLE) == ENOTSUP use_that if fiemap(FIEMAP_FLAG_SYNC) use_that -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): Triaged Status in QEMU: New Bug description: -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
Marking as High since duplicate bug 1350766 was marked High. ** Changed in: nova Importance: Medium = High -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): Triaged Status in QEMU: New Bug description: -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
** Also affects: qemu (Ubuntu) Importance: Undecided Status: New ** Changed in: qemu (Ubuntu) Status: New = Triaged ** Changed in: qemu (Ubuntu) Importance: Undecided = High ** Also affects: qemu (Ubuntu Trusty) Importance: Undecided Status: New ** Changed in: qemu (Ubuntu Trusty) Status: New = Triaged ** Changed in: qemu (Ubuntu Trusty) Importance: Undecided = High -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): Triaged Status in QEMU: New Status in “qemu” package in Ubuntu: Triaged Status in “qemu” source package in Trusty: Triaged Bug description: -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
The affected code was introduced with version 1.2.0. However, due to https://bugs.launchpad.net/qemu/+bug/1193628 I can't build these old releases to verify whether they actually expose the same behaviour. -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): Triaged Status in QEMU: New Bug description: -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
Is there a minimum version of qemu that would be required to use the FIEMAP_FLAG_SYNC flag? ** Changed in: nova Status: New = Triaged ** Changed in: nova Importance: Undecided = Medium -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): Triaged Status in QEMU: New Bug description: -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images
** Also affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in OpenStack Compute (Nova): Triaged Status in QEMU: New Bug description: -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced easily on command line using cat $SRC_PATH $TMP_PATH $QEMU_IMG_PATH convert -O raw $TMP_PATH $DST_PATH cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html and it can be solved the same way as suggested in http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change f.fm.fm_flags = 0; to f.fm.fm_flags = FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions
