[Qemu-block] [PATCH v3 5/8] block/qcow2-refcount: check_refcounts_l2: don't count corrupted entries

2018-10-11 Thread Vladimir Sementsov-Ogievskiy
Misaligned entries will lead to fatal qcow2 driver corruption on read or write to corresponding offset, so there is no sense to take them into account. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/qcow2-refcount.c | 25 + 1 file changed, 13 insertions(+), 12

[Qemu-block] [PATCH v3 1/8] block/qcow2-refcount: fix check_oflag_copied

2018-10-11 Thread Vladimir Sementsov-Ogievskiy
Increase corruptions_fixed only after successful fix. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/qcow2-refcount.c | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index 3c539f02e5..b453d87a3f 100644 ---

Re: [Qemu-block] [PATCH] qcow2: Get the request alignment for encrypted images from QCryptoBlock

2018-10-11 Thread Kevin Wolf
Am 11.10.2018 um 12:58 hat Alberto Garcia geschrieben: > This doesn't have any practical effect at the moment because the > values of BDRV_SECTOR_SIZE, QCRYPTO_BLOCK_LUKS_SECTOR_SIZE and > QCRYPTO_BLOCK_QCOW_SECTOR_SIZE are all the same (512 bytes), but > future encryption methods could have

[Qemu-block] [PATCH v3 2/8] block/qcow2-refcount: avoid eating RAM

2018-10-11 Thread Vladimir Sementsov-Ogievskiy
qcow2_inc_refcounts_imrt() (through realloc_refcount_array()) can eat an unpredictable amount of memory on corrupted table entries, which are referencing regions far beyond the end of file. Prevent this, by skipping such regions from further processing. Signed-off-by: Vladimir

[Qemu-block] [PATCH v3 6/8] block/qcow2-refcount: refactor fixing L2 entry

2018-10-11 Thread Vladimir Sementsov-Ogievskiy
Split entry repairing to separate function, to be reused later in check_refcounts_l2, prepare the whole pipeline for adding more corruption types which may be repaired. Note: entry in in-memory l2 table (local variable in check_refcounts_l2) is not updated after this patch. Signed-off-by:

[Qemu-block] [PATCH v3 4/8] block/qcow2-refcount: check_refcounts_l2: reduce ignored overlaps

2018-10-11 Thread Vladimir Sementsov-Ogievskiy
Reduce number of structures ignored in overlap check: when checking active table ignore active tables, when checking inactive table ignore inactive ones. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz --- block/qcow2-refcount.c | 16 +--- 1 file changed, 9

[Qemu-block] [PATCH v3 7/8] block/qcow2-refcount: fix out-of-file L1 entries to be zero

2018-10-11 Thread Vladimir Sementsov-Ogievskiy
Zero out corrupted L1 table entry, which reference L2 table out of underlying file. Zero L1 table entry means that "the L2 table and all clusters described by this L2 table are unallocated." Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/qcow2-refcount.c | 26 ++

Re: [Qemu-block] [Qemu-devel] [PATCH v3] python: Use io.StringIO

2018-10-11 Thread Eduardo Habkost
On Thu, Oct 11, 2018 at 04:56:04PM +0200, Kevin Wolf wrote: > Am 11.10.2018 um 16:01 hat Markus Armbruster geschrieben: [...] > > I think we need to invest more resources in Python 2/3 compatiblity, or > > else we'll miss our hard deadline of January 1, 2020. > > Did we decide until when we want

[Qemu-block] [PATCH v3 3/8] block/qcow2-refcount: check_refcounts_l2: refactor compressed case

2018-10-11 Thread Vladimir Sementsov-Ogievskiy
Separate offset and size of compressed cluster. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/qcow2-refcount.c | 13 - 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index afaa1a1409..23b105b43b 100644 ---

[Qemu-block] [PATCH v3 0/8] qcow2 check improvements

2018-10-11 Thread Vladimir Sementsov-Ogievskiy
Hi all! v3: 01: s/l2_fixed_entries/l2_dirty/ 02: add comment, s/>/>= 03: s/BDRV_SECTOR_SIZE/512 04: add Max's r-b 05: new 06-08: mostly rewritten v2: 02, 06: check bdrv_getlength error return code v1: We've faced the following problem: after host fs corruption, vm images becomes invalid. And

[Qemu-block] [PATCH v3 8/8] block/qcow2-refcount: fix out-of-file L2 entries

2018-10-11 Thread Vladimir Sementsov-Ogievskiy
Rewrite corrupted L2 table entry, which reference space out of underlying file. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/qcow2-refcount.c | 66 +- 1 file changed, 65 insertions(+), 1 deletion(-) diff --git a/block/qcow2-refcount.c

[Qemu-block] ping Re: [PATCH v4 0/8] dirty-bitmap: rewrite bdrv_dirty_iter_next_area

2018-10-11 Thread Vladimir Sementsov-Ogievskiy
ping 17.09.2018 17:57, Vladimir Sementsov-Ogievskiy wrote: > Hi all. > > 1. bdrv_dirty_iter_next_area don't use hbitmap_next_zero and uses > inefficient loop instead. Let's improve it. > > 2. bdrv_dirty_iter_next_area don't handle unaligned offset and > max_offset correctly. I'm not sure that it

[Qemu-block] ping Re: [PATCH 0/2] replication: drop extra sync

2018-10-11 Thread Vladimir Sementsov-Ogievskiy
ping 19.09.2018 15:43, Vladimir Sementsov-Ogievskiy wrote: > Hi all! > > We finally implemented fleecing scheme, create a test case for it (222) > and even made it safe (f8d59dfb40). > > However, fleecing is used inside block-replication long ago, and to make > the scheme safe it uses direct

[Qemu-block] [PATCH v5 1/3] file-posix: Skip effectiveless OFD lock operations

2018-10-11 Thread Fam Zheng
If we know we've already locked the bytes, don't do it again; similarly don't unlock a byte if we haven't locked it. This doesn't change the behavior, but fixes a corner case explained below. Libvirt had an error handling bug that an image can get its (ownership, file mode, SELinux) permissions

[Qemu-block] [PATCH v5 0/3] file-posix: Simplifications on image locking

2018-10-11 Thread Fam Zheng
v5: Address Max's comments (Thanks for reviewing): - Clean up after test done. - Add rev-by to patch 1 and 2. v4: Fix test on systems without OFD. [Patchew] The first patch reduces chances of QEMU crash in unusual (but not unlikely) cases especially when used by Libvirt (see commit

[Qemu-block] [PATCH v5 2/3] file-posix: Drop s->lock_fd

2018-10-11 Thread Fam Zheng
The lock_fd field is not strictly necessary because transferring locked bytes from old fd to the new one shouldn't fail anyway. This spares the user one fd per image. Signed-off-by: Fam Zheng Reviewed-by: Max Reitz --- block/file-posix.c | 37 + 1 file

[Qemu-block] [PATCH v5 3/3] tests: Add unit tests for image locking

2018-10-11 Thread Fam Zheng
Signed-off-by: Fam Zheng --- tests/Makefile.include | 2 + tests/test-image-locking.c | 157 + 2 files changed, 159 insertions(+) create mode 100644 tests/test-image-locking.c diff --git a/tests/Makefile.include b/tests/Makefile.include index

Re: [Qemu-block] [PATCH 0/4] block: Add auto-read-only option

2018-10-11 Thread Peter Krempa
On Tue, Oct 09, 2018 at 21:35:20 +0200, Kevin Wolf wrote: > Peter, would this provide what libvirt urgently needs for backing files > vs. the commit block job? This looks fine for us with one exception. I'd prefer if the curl driver implemented this too so that we don't have to hardcode the

[Qemu-block] [PATCH] qcow2: Get the request alignment for encrypted images from QCryptoBlock

2018-10-11 Thread Alberto Garcia
This doesn't have any practical effect at the moment because the values of BDRV_SECTOR_SIZE, QCRYPTO_BLOCK_LUKS_SECTOR_SIZE and QCRYPTO_BLOCK_QCOW_SECTOR_SIZE are all the same (512 bytes), but future encryption methods could have different requirements. Signed-off-by: Alberto Garcia ---

Re: [Qemu-block] [PATCH] qcow2: Get the request alignment for encrypted images from QCryptoBlock

2018-10-11 Thread Daniel P . Berrangé
On Thu, Oct 11, 2018 at 01:58:02PM +0300, Alberto Garcia wrote: > This doesn't have any practical effect at the moment because the > values of BDRV_SECTOR_SIZE, QCRYPTO_BLOCK_LUKS_SECTOR_SIZE and > QCRYPTO_BLOCK_QCOW_SECTOR_SIZE are all the same (512 bytes), but > future encryption methods could

Re: [Qemu-block] [PATCH v3] python: Use io.StringIO

2018-10-11 Thread Kevin Wolf
Am 10.10.2018 um 23:31 hat Philippe Mathieu-Daudé geschrieben: > Both Python 2.7 and 3 support the same io.StringIO to > handle unicode strings. > > Use the common form to use indistinctly Python 2.7 or 3. > > http://python-future.org/compatible_idioms.html#stringio > > This fixes running tests

Re: [Qemu-block] [PATCH v3] python: Use io.StringIO

2018-10-11 Thread Paolo Bonzini
On 11/10/2018 13:33, Kevin Wolf wrote: > 045 1s ...[13:31:47] [13:31:47] [failed, exit status 1] - output > mismatch (see 045.out.bad) > --- /home/kwolf/source/qemu/tests/qemu-iotests/045.out 2017-01-24 > 14:49:48.0 +0100 > +++

Re: [Qemu-block] [PATCH v11 24/31] block: Purify .bdrv_refresh_filename()

2018-10-11 Thread Alberto Garcia
On Sat 06 Oct 2018 01:40:16 AM CEST, Max Reitz wrote: > Currently, BlockDriver.bdrv_refresh_filename() is supposed to both > refresh the filename (BDS.exact_filename) and set BDS.full_open_options. > Now that we have generic code in the central bdrv_refresh_filename() for > creating

Re: [Qemu-block] [Qemu-devel] [PATCH v3] python: Use io.StringIO

2018-10-11 Thread Markus Armbruster
Paolo Bonzini writes: > On 11/10/2018 13:33, Kevin Wolf wrote: >> 045 1s ...[13:31:47] [13:31:47] [failed, exit status 1] - output >> mismatch (see 045.out.bad) >> --- /home/kwolf/source/qemu/tests/qemu-iotests/045.out 2017-01-24 >> 14:49:48.0 +0100 >> +++

Re: [Qemu-block] [PATCH v11 31/31] iotests: Test json:{} filenames of internal BDSs

2018-10-11 Thread Alberto Garcia
On Sat 06 Oct 2018 01:40:23 AM CEST, Max Reitz wrote: > Signed-off-by: Max Reitz Reviewed-by: Alberto Garcia Berto

Re: [Qemu-block] [Qemu-devel] [PATCH v3] python: Use io.StringIO

2018-10-11 Thread Philippe Mathieu-Daudé
On 11/10/2018 16:01, Markus Armbruster wrote: > Paolo Bonzini writes: > >> On 11/10/2018 13:33, Kevin Wolf wrote: >>> 045 1s ...[13:31:47] [13:31:47] [failed, exit status 1] - output >>> mismatch (see 045.out.bad) >>> --- /home/kwolf/source/qemu/tests/qemu-iotests/045.out 2017-01-24

Re: [Qemu-block] [Qemu-devel] [PATCH v3] python: Use io.StringIO

2018-10-11 Thread Kevin Wolf
Am 11.10.2018 um 16:01 hat Markus Armbruster geschrieben: > Paolo Bonzini writes: > > > On 11/10/2018 13:33, Kevin Wolf wrote: > >> 045 1s ...[13:31:47] [13:31:47] [failed, exit status 1] - output > >> mismatch (see 045.out.bad) > >> ---