[PULL 2/3] block/nbd: nbd reconnect

2019-10-22 Thread Eric Blake
From: Vladimir Sementsov-Ogievskiy Implement reconnect. To achieve this: 1. add new modes: connecting-wait: means, that reconnecting is in progress, and there were small number of reconnect attempts, so all requests are waiting for the connection. connecting-nowait: reconnecting

[PULL 3/3] iotests: test nbd reconnect

2019-10-22 Thread Eric Blake
From: Vladimir Sementsov-Ogievskiy Add test, which starts backup to nbd target and restarts nbd server during backup. Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20191009084158.15614-4-vsement...@virtuozzo.com> Reviewed-by: Eric Blake Signed-off-by: Eric Blake --- tests/qemu-iot

Re: [PATCH v10 3/3] iotests: test nbd reconnect

2019-10-22 Thread Eric Blake
On 10/9/19 3:41 AM, Vladimir Sementsov-Ogievskiy wrote: Add test, which starts backup to nbd target and restarts nbd server during backup. Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/264| 95 +++ tests/qemu-iotests/264.out| 1

Re: [PATCH v3 0/6] Enable more iotests during "make check-block"

2019-10-22 Thread Alex Bennée
Thomas Huth writes: > On 22/10/2019 15.48, Alex Bennée wrote: >> >> Max Reitz writes: >> >>> On 22.10.19 15:11, Alex Bennée wrote: Thomas Huth writes: > As discussed here: > > https://lists.gnu.org/archive/html/qemu-devel/2019-10/msg00697.html > > and here:

Re: [PATCH 00/10] image-fuzzer: Port to Python 3

2019-10-22 Thread Eduardo Habkost
On Thu, Oct 17, 2019 at 06:29:27PM -0300, Eduardo Habkost wrote: > On Thu, Oct 17, 2019 at 05:11:29PM -0400, John Snow wrote: > > > > > > On 10/16/19 3:24 PM, Eduardo Habkost wrote: > > > This series ports image-fuzzer to Python 3. > > > > > > Eduardo Habkost (10): > > > image-fuzzer: Open ima

[PATCH v1 19/19] iotests: Remove 130 from the "auto" group

2019-10-22 Thread Alex Bennée
From: Thomas Huth Peter hit a "Could not open 'TEST_DIR/t.IMGFMT': Failed to get shared 'write' lock - Is another process using the image [TEST_DIR/t.IMGFMT]?" error with 130 already twice. Looks like this test is a little bit shaky, so for the time being, let's disable it from the "auto" group s

[PATCH v1 15/19] iotests: Test 041 only works on certain systems

2019-10-22 Thread Alex Bennée
From: Thomas Huth 041 works fine on Linux, FreeBSD, NetBSD and OpenBSD, but fails on macOS. Let's mark it as only supported on the systems where we know that it is working fine. Signed-off-by: Thomas Huth Message-Id: <20191022072135.11188-3-th...@redhat.com> --- tests/qemu-iotests/041 | 3 ++-

[PATCH v1 18/19] iotests: Enable more tests in the 'auto' group to improve test coverage

2019-10-22 Thread Alex Bennée
From: Thomas Huth According to Kevin, tests 030, 040 and 041 are among the most valuable tests that we have, so we should always run them if possible, even if they take a little bit longer. According to Max, it would be good to have a test for iothreads and migration. 127 and 256 seem to be good

[PATCH v1 16/19] iotests: Test 183 does not work on macOS and OpenBSD

2019-10-22 Thread Alex Bennée
From: Thomas Huth In the long term, we might want to add test 183 to the "auto" group (but it still fails occasionally, so we cannot do that yet). However, when running 183 in Cirrus-CI on macOS, or with our vm-build-openbsd target, it currently always fails with an "Timeout waiting for return on

[PATCH v1 14/19] iotests: remove 'linux' from default supported platforms

2019-10-22 Thread Alex Bennée
From: John Snow verify_platform will check an explicit whitelist and blacklist instead. The default will now be assumed to be allowed to run anywhere. For tests that do not specify their platforms explicitly, this has the effect of enabling these tests on non-linux platforms. For tests that alwa

Re: [PATCH v14 0/9] hw/m68k: add Apple Machintosh Quadra 800 machine

2019-10-22 Thread no-reply
Patchew URL: https://patchew.org/QEMU/20191022111738.20803-1-laur...@vivier.eu/ Hi, This series seems to have some coding style problems. See output below for more information: Subject: [PATCH v14 0/9] hw/m68k: add Apple Machintosh Quadra 800 machine Type: series Message-id: 20191022111738.208

Re: [PATCH v3 0/6] Enable more iotests during "make check-block"

2019-10-22 Thread Thomas Huth
On 22/10/2019 15.48, Alex Bennée wrote: > > Max Reitz writes: > >> On 22.10.19 15:11, Alex Bennée wrote: >>> >>> Thomas Huth writes: >>> As discussed here: https://lists.gnu.org/archive/html/qemu-devel/2019-10/msg00697.html and here: https://lists.gnu.org/ar

Re: [PATCH v3 07/16] libqos: enforce Device Initialization order

2019-10-22 Thread Thomas Huth
On 22/10/2019 17.48, Stefan Hajnoczi wrote: > On Mon, Oct 21, 2019 at 02:15:53PM +0200, Thomas Huth wrote: >> On 19/10/2019 08.38, Stefan Hajnoczi wrote: >>> According to VIRTIO 1.1 "3.1.1 Driver Requirements: Device >>> Initialization", configuration space and virtqueues cannot be accessed >>> bef

Re: [PATCH v3 09/16] libqos: access VIRTIO 1.0 vring in little-endian

2019-10-22 Thread Christophe de Dinechin
Stefan Hajnoczi writes: [...] > +static uint16_t qvirtio_readw(QVirtioDevice *d, QTestState *qts, uint64_t > addr) > +{ > +uint16_t val = qtest_readw(qts, addr); > + > +if (d->features & (1ull << VIRTIO_F_VERSION_1) && qtest_big_endian(qts)) > { For my education, I was wondering why te

Re: [PATCH v3 08/16] libqos: implement VIRTIO 1.0 FEATURES_OK step

2019-10-22 Thread Stefan Hajnoczi
On Mon, Oct 21, 2019 at 02:23:15PM +0200, Thomas Huth wrote: > On 19/10/2019 08.38, Stefan Hajnoczi wrote: > > Device initialization has an extra step in VIRTIO 1.0. The FEATURES_OK > > status bit is set to indicate that feature negotiation has completed. > > The driver then reads the status regis

Re: [PATCH v3 07/16] libqos: enforce Device Initialization order

2019-10-22 Thread Stefan Hajnoczi
On Mon, Oct 21, 2019 at 02:15:53PM +0200, Thomas Huth wrote: > On 19/10/2019 08.38, Stefan Hajnoczi wrote: > > According to VIRTIO 1.1 "3.1.1 Driver Requirements: Device > > Initialization", configuration space and virtqueues cannot be accessed > > before features have been negotiated. Enforce thi

Re: [PATCH v3 09/16] libqos: access VIRTIO 1.0 vring in little-endian

2019-10-22 Thread Stefan Hajnoczi
On Mon, Oct 21, 2019 at 02:48:08PM +0200, Thomas Huth wrote: > On 19/10/2019 08.38, Stefan Hajnoczi wrote: > > VIRTIO 1.0 uses little-endian for the vring. Legacy VIRTIO uses guest > > endianness. Adjust the code to handle both. > > > > Note that qvirtio_readq() is not defined because it has no

Re: [PATCH v3 04/16] virtio-scsi-test: add missing feature negotiation

2019-10-22 Thread Stefan Hajnoczi
On Mon, Oct 21, 2019 at 02:08:34PM +0200, Thomas Huth wrote: > On 19/10/2019 08.37, Stefan Hajnoczi wrote: > > VIRTIO Device Initialization requires feature negotiation. Currently > > virtio-scsi-test.c is non-compliant. > > > > Signed-off-by: Stefan Hajnoczi > > --- > > tests/virtio-scsi-test.

Re: [PATCH v3 0/6] Enable more iotests during "make check-block"

2019-10-22 Thread Alex Bennée
Max Reitz writes: > On 22.10.19 15:11, Alex Bennée wrote: >> >> Thomas Huth writes: >> >>> As discussed here: >>> >>> https://lists.gnu.org/archive/html/qemu-devel/2019-10/msg00697.html >>> >>> and here: >>> >>> https://lists.gnu.org/archive/html/qemu-devel/2019-10/msg01388.html >> >> Queued

Re: [PATCH v5 1/4] block: support compressed write at generic layer

2019-10-22 Thread Vladimir Sementsov-Ogievskiy
22.10.2019 15:56, Max Reitz wrote: > On 22.10.19 14:23, Vladimir Sementsov-Ogievskiy wrote: >> 22.10.2019 14:31, Max Reitz wrote: >>> On 22.10.19 12:46, Vladimir Sementsov-Ogievskiy wrote: 22.10.2019 13:21, Andrey Shinkevich wrote: > > On 22/10/2019 12:28, Max Reitz wrote: >> On 20

Re: [PATCH v5 1/4] block: support compressed write at generic layer

2019-10-22 Thread Andrey Shinkevich
On 22/10/2019 15:56, Max Reitz wrote: > On 22.10.19 14:23, Vladimir Sementsov-Ogievskiy wrote: >> 22.10.2019 14:31, Max Reitz wrote: >>> On 22.10.19 12:46, Vladimir Sementsov-Ogievskiy wrote: 22.10.2019 13:21, Andrey Shinkevich wrote: > > On 22/10/2019 12:28, Max Reitz wrote: >>

Re: [PATCH v3 0/6] Enable more iotests during "make check-block"

2019-10-22 Thread Max Reitz
On 22.10.19 15:11, Alex Bennée wrote: > > Thomas Huth writes: > >> As discussed here: >> >> https://lists.gnu.org/archive/html/qemu-devel/2019-10/msg00697.html >> >> and here: >> >> https://lists.gnu.org/archive/html/qemu-devel/2019-10/msg01388.html > > Queued to testing/next, thanks. It sho

Re: [PATCH] blockdev: modify blockdev-change-medium to change non-removable device

2019-10-22 Thread Denis Plotnikov
On 22.10.2019 16:18, Max Reitz wrote: > On 22.10.19 14:53, Denis Plotnikov wrote: >> On 22.10.2019 14:05, Max Reitz wrote: >>> On 21.10.19 08:50, Denis Plotnikov wrote: On 18.10.2019 18:02, Max Reitz wrote: > On 18.10.19 14:09, Denis Plotnikov wrote: >> The modification is useful to w

Re: [PATCH] blockdev: modify blockdev-change-medium to change non-removable device

2019-10-22 Thread Max Reitz
On 22.10.19 14:53, Denis Plotnikov wrote: > > On 22.10.2019 14:05, Max Reitz wrote: >> On 21.10.19 08:50, Denis Plotnikov wrote: >>> On 18.10.2019 18:02, Max Reitz wrote: On 18.10.19 14:09, Denis Plotnikov wrote: > The modification is useful to workaround exclusive file access > rest

Re: [PATCH v3 0/6] Enable more iotests during "make check-block"

2019-10-22 Thread Alex Bennée
Thomas Huth writes: > As discussed here: > > https://lists.gnu.org/archive/html/qemu-devel/2019-10/msg00697.html > > and here: > > https://lists.gnu.org/archive/html/qemu-devel/2019-10/msg01388.html Queued to testing/next, thanks. > > it would be good to have some more valuable iotests enab

Re: [PATCH v14 1/9] esp: add pseudo-DMA as used by Macintosh

2019-10-22 Thread Laurent Vivier
Le 22/10/2019 à 14:21, Philippe Mathieu-Daudé a écrit : > Hi Laurent, > > On 10/22/19 1:17 PM, Laurent Vivier wrote: >> There is no DMA in Quadra 800, so the CPU reads/writes the data from the >> PDMA register (offset 0x100, ESP_PDMA in hw/m68k/q800.c) and copies them >> to/from the memory. >> >>

Re: [PATCH v3 0/6] Enable more iotests during "make check-block"

2019-10-22 Thread Alex Bennée
Thomas Huth writes: > On 22/10/2019 13.39, Alex Bennée wrote: >> >> Thomas Huth writes: >> >>> On 22/10/2019 09.21, Thomas Huth wrote: As discussed here: https://lists.gnu.org/archive/html/qemu-devel/2019-10/msg00697.html and here: https://lists.gnu.org/arc

[PATCH v2 02/10] hbitmap: move hbitmap_iter_next_word to hbitmap.c

2019-10-22 Thread Vladimir Sementsov-Ogievskiy
The function is definitely internal (it's not used by third party and it has complicated interface). Move it to .c file. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/qemu/hbitmap.h | 30 -- util/hbitmap.c | 29 + 2 files

[PATCH v2 05/10] block/dirty-bitmap: switch _next_dirty_area and _next_zero to int64_t

2019-10-22 Thread Vladimir Sementsov-Ogievskiy
We are going to introduce bdrv_dirty_bitmap_next_dirty so that same variable may be used to store its return value and to be its parameter, so it would int64_t. Similarly, we are going to refactor hbitmap_next_dirty_area to use hbitmap_next_dirty together with hbitmap_next_zero, therefore we want

[PATCH v2 10/10] block/qcow2-bitmap: use bdrv_dirty_bitmap_next_dirty

2019-10-22 Thread Vladimir Sementsov-Ogievskiy
store_bitmap_data() loop does bdrv_set_dirty_iter() on each iteration, which means that we actually don't need iterator itself and we can use simpler bitmap API. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/qcow2-bitmap.c | 11 +-- 1 file changed, 5 insertions(+), 6 deletions(-)

[PATCH v2 04/10] hbitmap: drop meta bitmaps as they are unused

2019-10-22 Thread Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/qemu/hbitmap.h | 21 tests/test-hbitmap.c | 115 - util/hbitmap.c | 16 -- 3 files changed, 152 deletions(-) diff --git a/include/qemu/hbitmap.h b/include/qemu/hbitmap.h index

[PATCH v2 06/10] block/dirty-bitmap: add _next_dirty API

2019-10-22 Thread Vladimir Sementsov-Ogievskiy
We have bdrv_dirty_bitmap_next_zero, let's add corresponding bdrv_dirty_bitmap_next_dirty, which is more comfortable to use than bitmap iterators in some cases. For test modify test_hbitmap_next_zero_check_range to check both next_zero and next_dirty and add some new checks. Signed-off-by: Vladim

[PATCH v2 00/10] Further bitmaps improvements

2019-10-22 Thread Vladimir Sementsov-Ogievskiy
Hi! The main feature here is improvement of _next_dirty_area API, which I'm going to use then for backup / block-copy. v2: 01: just use INT64_MAX instead of adding new constant 08: add separate function nbd_extent_array_convert_to_be and converted state of NBDExtentArray, to make these things

[PATCH v2 03/10] hbitmap: unpublish hbitmap_iter_skip_words

2019-10-22 Thread Vladimir Sementsov-Ogievskiy
Function is internal and even commented as internal. Drop its definition from .h file. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/qemu/hbitmap.h | 7 --- util/hbitmap.c | 2 +- 2 files changed, 1 insertion(+), 8 deletions(-) diff --git a/include/qemu/hbitmap.h b/include

[PATCH v2 07/10] block/dirty-bitmap: improve _next_dirty_area API

2019-10-22 Thread Vladimir Sementsov-Ogievskiy
Firstly, _next_dirty_area is for scenarios when we may contiguously search for next dirty area inside some limited region, so it is more comfortable to specify "end" which should not be recalculated on each iteration. Secondly, mirror wants to limit resulting are, and for this thing it limits @cou

[PATCH v2 08/10] nbd/server: introduce NBDExtentArray

2019-10-22 Thread Vladimir Sementsov-Ogievskiy
Introduce NBDExtentArray class, to handle extents list creation in more controlled way and with less OUT parameters in functions. Signed-off-by: Vladimir Sementsov-Ogievskiy --- nbd/server.c | 201 --- 1 file changed, 109 insertions(+), 92 deletion

[PATCH v2 01/10] hbitmap: assert that we don't create bitmap larger than INT64_MAX

2019-10-22 Thread Vladimir Sementsov-Ogievskiy
We have APIs which returns signed int64_t, to be able to return error. Therefore we can't handle bitmaps with absolute size larger than (INT64_MAX+1). Still, keep maximum to be INT64_MAX which is a bit safer. Note, that bitmaps are used to represent disk images, which can't exceed INT64_MAX anyway

[PATCH v2 09/10] nbd/server: use bdrv_dirty_bitmap_next_dirty_area

2019-10-22 Thread Vladimir Sementsov-Ogievskiy
Use bdrv_dirty_bitmap_next_dirty_area for bitmap_to_extents. Since bdrv_dirty_bitmap_next_dirty_area is very accurate in its interface, we'll never exceed requested region with last chunk. So, we don't need dont_fragment, and bitmap_to_extents() interface becomes clean enough to not require any com

Re: [PATCH v5 1/4] block: support compressed write at generic layer

2019-10-22 Thread Max Reitz
On 22.10.19 14:23, Vladimir Sementsov-Ogievskiy wrote: > 22.10.2019 14:31, Max Reitz wrote: >> On 22.10.19 12:46, Vladimir Sementsov-Ogievskiy wrote: >>> 22.10.2019 13:21, Andrey Shinkevich wrote: On 22/10/2019 12:28, Max Reitz wrote: > On 20.10.19 22:37, Andrey Shinkevich wrote:

Re: [PATCH] blockdev: modify blockdev-change-medium to change non-removable device

2019-10-22 Thread Denis Plotnikov
On 22.10.2019 14:05, Max Reitz wrote: > On 21.10.19 08:50, Denis Plotnikov wrote: >> On 18.10.2019 18:02, Max Reitz wrote: >>> On 18.10.19 14:09, Denis Plotnikov wrote: The modification is useful to workaround exclusive file access restrictions, e.g. to implement VM migration with

Re: [PATCH v5 1/4] block: support compressed write at generic layer

2019-10-22 Thread Vladimir Sementsov-Ogievskiy
22.10.2019 14:31, Max Reitz wrote: > On 22.10.19 12:46, Vladimir Sementsov-Ogievskiy wrote: >> 22.10.2019 13:21, Andrey Shinkevich wrote: >>> >>> On 22/10/2019 12:28, Max Reitz wrote: On 20.10.19 22:37, Andrey Shinkevich wrote: > To inform the block layer about writing all the data compres

Re: [PATCH v14 1/9] esp: add pseudo-DMA as used by Macintosh

2019-10-22 Thread Philippe Mathieu-Daudé
Hi Laurent, On 10/22/19 1:17 PM, Laurent Vivier wrote: There is no DMA in Quadra 800, so the CPU reads/writes the data from the PDMA register (offset 0x100, ESP_PDMA in hw/m68k/q800.c) and copies them to/from the memory. There is a nice assembly loop in the kernel to do that, see linux/drivers/

Re: [PATCH v3 0/6] Enable more iotests during "make check-block"

2019-10-22 Thread Thomas Huth
On 22/10/2019 13.39, Alex Bennée wrote: > > Thomas Huth writes: > >> On 22/10/2019 09.21, Thomas Huth wrote: >>> As discussed here: >>> >>> https://lists.gnu.org/archive/html/qemu-devel/2019-10/msg00697.html >>> >>> and here: >>> >>> https://lists.gnu.org/archive/html/qemu-devel/2019-10/msg013

Re: [PATCH v3 0/6] Enable more iotests during "make check-block"

2019-10-22 Thread Alex Bennée
Thomas Huth writes: > On 22/10/2019 09.21, Thomas Huth wrote: >> As discussed here: >> >> https://lists.gnu.org/archive/html/qemu-devel/2019-10/msg00697.html >> >> and here: >> >> https://lists.gnu.org/archive/html/qemu-devel/2019-10/msg01388.html >> >> it would be good to have some more valu

[PATCH v14 3/9] hw/m68k: add VIA support

2019-10-22 Thread Laurent Vivier
Inside the 680x0 Macintosh, VIA (Versatile Interface Adapter) is used to interface the keyboard, Mouse, and real-time clock. It also provides control line for the floppy disk driver, video interface, sound circuitry and serial interface. This implementation is based on the MOS6522 object. Co-deve

[PATCH v14 2/9] dp8393x: manage big endian bus

2019-10-22 Thread Laurent Vivier
This is needed by Quadra 800, this card can run on little-endian or big-endian bus. Signed-off-by: Laurent Vivier Tested-by: Hervé Poussineau Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Hervé Poussineau --- hw/net/dp8393x.c | 88 +++- 1 file ch

Re: [PATCH v5 1/4] block: support compressed write at generic layer

2019-10-22 Thread Max Reitz
On 22.10.19 12:46, Vladimir Sementsov-Ogievskiy wrote: > 22.10.2019 13:21, Andrey Shinkevich wrote: >> >> On 22/10/2019 12:28, Max Reitz wrote: >>> On 20.10.19 22:37, Andrey Shinkevich wrote: To inform the block layer about writing all the data compressed, we introduce the 'compress' comm

[PATCH v14 8/9] hw/m68k: define Macintosh Quadra 800

2019-10-22 Thread Laurent Vivier
If you want to test the machine, it doesn't yet boot a MacROM, but you can boot a linux kernel from the command line. You can install your own disk using debian-installer with: ./qemu-system-m68k \ -M q800 \ -serial none -serial mon:stdio \ -m 1000M -drive file=m68k.qcow2,format=q

[PATCH v14 4/9] hw/m68k: implement ADB bus support for via

2019-10-22 Thread Laurent Vivier
VIA needs to be able to poll the ADB interface and to read/write data from/to the bus. This patch adds functions allowing that. Co-developed-by: Mark Cave-Ayland Signed-off-by: Mark Cave-Ayland Signed-off-by: Laurent Vivier Reviewed-by: Hervé Poussineau Reviewed-by: Thomas Huth --- include/

[PATCH v14 1/9] esp: add pseudo-DMA as used by Macintosh

2019-10-22 Thread Laurent Vivier
There is no DMA in Quadra 800, so the CPU reads/writes the data from the PDMA register (offset 0x100, ESP_PDMA in hw/m68k/q800.c) and copies them to/from the memory. There is a nice assembly loop in the kernel to do that, see linux/drivers/scsi/mac_esp.c:MAC_ESP_PDMA_LOOP(). The start of the tran

[PATCH v14 9/9] BootLinuxConsoleTest: Test the Quadra 800

2019-10-22 Thread Laurent Vivier
From: Philippe Mathieu-Daudé This test boots a Linux kernel on a Quadra 800 board and verify the serial is working. Example: $ avocado --show=app,console run -t machine:q800 tests/acceptance/boot_linux_console.py console: ABCFGHIJK console: Linux version 5.2.0-2-m68k (debian-ker...@lists

[PATCH v14 7/9] hw/m68k: add a dummy SWIM floppy controller

2019-10-22 Thread Laurent Vivier
SWIM (Sander-Wozniak Integrated Machine) is the floppy controller of the 680x0 Macintosh. This patch introduces only the basic support: it allows to switch from IWM (Integrated WOZ Machine) mode to the SWIM mode and makes the linux driver happy. It cannot read any floppy image. Co-developed-by:

[PATCH v14 5/9] hw/m68k: add Nubus support

2019-10-22 Thread Laurent Vivier
This patch adds basic support for the NuBus bus. This is used by 680x0 Macintosh. Co-developed-by: Mark Cave-Ayland Signed-off-by: Mark Cave-Ayland Signed-off-by: Laurent Vivier Reviewed-by: Thomas Huth --- include/hw/nubus/mac-nubus-bridge.h | 24 include/hw/nubus/nubus.h|

[PATCH v14 0/9] hw/m68k: add Apple Machintosh Quadra 800 machine

2019-10-22 Thread Laurent Vivier
I'm rebasing some of these patches for seven years now, too many years... if you want to test the machine, I'm sorry, it doesn't boot a MacROM, but you can boot a linux kernel from the command line. You can install your own disk using debian-installer, with: ... -M q800 \ -serial non

[PATCH v14 6/9] hw/m68k: add Nubus macfb video card

2019-10-22 Thread Laurent Vivier
This patch adds support for a graphic framebuffer device. This device can be added as a sysbus device or as a NuBus device. It is accessed as a framebuffer but the color palette can be set. Co-developed-by: Mark Cave-Ayland Signed-off-by: Mark Cave-Ayland Signed-off-by: Laurent Vivier Reviewed

[PATCH v3 6/6] block/block-copy: increase buffered copy request

2019-10-22 Thread Vladimir Sementsov-Ogievskiy
No reason to limit buffered copy to one cluster. Let's allow up to 1 MiB. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz --- include/block/block-copy.h | 2 +- block/block-copy.c | 48 +- 2 files changed, 33 insertions(+), 17 dele

[PATCH v3 1/6] block/block-copy: allocate buffer in block_copy_with_bounce_buffer

2019-10-22 Thread Vladimir Sementsov-Ogievskiy
Move bounce_buffer allocation block_copy_with_bounce_buffer. This commit simplifies further work on implementing copying by larger chunks (of different size) and further asynchronous handling of block_copy iterations (with help of block/aio_task API). Allocation works fast, a lot faster than disk

[PATCH v3 5/6] block/block-copy: add memory limit

2019-10-22 Thread Vladimir Sementsov-Ogievskiy
Currently total allocation for parallel requests to block-copy instance is unlimited. Let's limit it to 128 MiB. For now block-copy is used only in backup, so actually we limit total allocation for backup job. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz --- include/block

[PATCH v3 3/6] block/block-copy: refactor copying

2019-10-22 Thread Vladimir Sementsov-Ogievskiy
Merge copying code into one function block_copy_do_copy, which only calls bdrv_ io functions and don't do any synchronization (like dirty bitmap set/reset). Refactor block_copy() function so that it takes full decision about size of chunk to be copied and does all the synchronization (checking int

[PATCH v3 4/6] util: introduce SharedResource

2019-10-22 Thread Vladimir Sementsov-Ogievskiy
Introduce an API for some shared splittable resource, like memory. It's going to be used by backup. Backup uses both read/write io and copy_range. copy_range may consume memory implictly, so the new API is abstract: it doesn't allocate any real memory but only hands out tickets. The idea is that w

[PATCH v3 0/6] block-copy: memory limit

2019-10-22 Thread Vladimir Sementsov-Ogievskiy
I'm going to bring block-status driven, async copying process to block-copy, to make it fast. The first step is to limit memory usage of backup, here is it. v3: 03: add Max's r-b 04: fix commit message and include guards, add Max's r-b 05-06: add Max's r-b v2: [mostly by Max's comments] Now based

[PATCH v3 2/6] block/block-copy: limit copy_range_size to 16 MiB

2019-10-22 Thread Vladimir Sementsov-Ogievskiy
Large copy range may imply memory allocation and large io effort, so using 2G copy range request may be bad idea. Let's limit it to 16 MiB. It also helps the following patch to refactor copy-with-offload fallback to copy-with-bounce-buffer. Note, that total memory usage of backup is still not limi

Re: [PATCH] blockdev: modify blockdev-change-medium to change non-removable device

2019-10-22 Thread Max Reitz
On 21.10.19 08:50, Denis Plotnikov wrote: > > On 18.10.2019 18:02, Max Reitz wrote: >> On 18.10.19 14:09, Denis Plotnikov wrote: >>> The modification is useful to workaround exclusive file access restrictions, >>> e.g. to implement VM migration with shared disk stored on a storage with >>> the exc

Re: [PATCH v2 4/6] util: introduce SharedResource

2019-10-22 Thread Vladimir Sementsov-Ogievskiy
22.10.2019 13:23, Max Reitz wrote: > On 16.10.19 19:09, Vladimir Sementsov-Ogievskiy wrote: >> Introduce an API for some shared splittable resource, like memory. >> It's going to be used by backup. Backup uses both read/write io and >> copy_range. copy_range may consume memory implictly, so the new

Re: [PATCH v5 1/4] block: support compressed write at generic layer

2019-10-22 Thread Vladimir Sementsov-Ogievskiy
22.10.2019 13:21, Andrey Shinkevich wrote: > > On 22/10/2019 12:28, Max Reitz wrote: >> On 20.10.19 22:37, Andrey Shinkevich wrote: >>> To inform the block layer about writing all the data compressed, we >>> introduce the 'compress' command line option. Based on that option, the >>> written data w

Re: [PATCH v2 6/6] block/block-copy: increase buffered copy request

2019-10-22 Thread Max Reitz
On 16.10.19 19:09, Vladimir Sementsov-Ogievskiy wrote: > No reason to limit buffered copy to one cluster. Let's allow up to 1 > MiB. > > Signed-off-by: Vladimir Sementsov-Ogievskiy > --- > include/block/block-copy.h | 2 +- > block/block-copy.c | 48 +

Re: [PATCH v8 7/8] bootdevice: FW_CFG interface for LCHS values

2019-10-22 Thread Philippe Mathieu-Daudé
On 10/16/19 6:41 PM, Sam Eiderman wrote: From: Sam Eiderman Using fw_cfg, supply logical CHS values directly from QEMU to the BIOS. Non-standard logical geometries break under QEMU. A virtual disk which contains an operating system which depends on logical geometries (consistent values being

Re: [PATCH v8 6/8] bootdevice: Refactor get_boot_devices_list

2019-10-22 Thread Philippe Mathieu-Daudé
On 10/16/19 6:41 PM, Sam Eiderman wrote: From: Sam Eiderman Move device name construction to a separate function. We will reuse this function in the following commit to pass logical CHS parameters through fw_cfg much like we currently pass bootindex. Reviewed-by: Karl Heubaum Reviewed-by: Ar

Re: [PATCH v2 5/6] block/block-copy: add memory limit

2019-10-22 Thread Max Reitz
On 16.10.19 19:09, Vladimir Sementsov-Ogievskiy wrote: > Currently total allocation for parallel requests to block-copy instance > is unlimited. Let's limit it to 128 MiB. > > For now block-copy is used only in backup, so actually we limit total > allocation for backup job. > > Signed-off-by: Vla

Re: [PATCH v8 5/8] bootdevice: Gather LCHS from all relevant devices

2019-10-22 Thread Philippe Mathieu-Daudé
On 10/16/19 6:41 PM, Sam Eiderman wrote: From: Sam Eiderman Relevant devices are: * ide-hd (and ide-cd, ide-drive) * scsi-hd (and scsi-cd, scsi-disk, scsi-block) * virtio-blk-pci We do not call del_boot_device_lchs() for ide-* since we don't need to - IDE block devices do not su

Re: [PATCH v2 4/6] util: introduce SharedResource

2019-10-22 Thread Max Reitz
On 16.10.19 19:09, Vladimir Sementsov-Ogievskiy wrote: > Introduce an API for some shared splittable resource, like memory. > It's going to be used by backup. Backup uses both read/write io and > copy_range. copy_range may consume memory implictly, so the new API is > abstract: it doesn't allocate

Re: [PATCH v8 4/8] scsi: Propagate unrealize() callback to scsi-hd

2019-10-22 Thread Philippe Mathieu-Daudé
On 10/16/19 6:41 PM, Sam Eiderman wrote: From: Sam Eiderman We will need to add LCHS removal logic to scsi-hd's unrealize() in the next commit. Reviewed-by: Karl Heubaum Reviewed-by: Arbel Moshe Reviewed-by: Philippe Mathieu-Daudé Signed-off-by: Sam Eiderman Signed-off-by: Sam Eiderman --

Re: [PATCH v8 3/8] bootdevice: Add interface to gather LCHS

2019-10-22 Thread Philippe Mathieu-Daudé
On 10/16/19 6:41 PM, Sam Eiderman wrote: From: Sam Eiderman Add an interface to provide direct logical CHS values for boot devices. We will use this interface in the next commits. Reviewed-by: Karl Heubaum Reviewed-by: Arbel Moshe Signed-off-by: Sam Eiderman Signed-off-by: Sam Eiderman ---

Re: [PATCH v5 1/4] block: support compressed write at generic layer

2019-10-22 Thread Andrey Shinkevich
On 22/10/2019 12:28, Max Reitz wrote: > On 20.10.19 22:37, Andrey Shinkevich wrote: >> To inform the block layer about writing all the data compressed, we >> introduce the 'compress' command line option. Based on that option, the >> written data will be aligned by the cluster size at the generic l

Re: [PATCH v8 2/8] block: Support providing LCHS from user

2019-10-22 Thread Philippe Mathieu-Daudé
On 10/16/19 6:41 PM, Sam Eiderman wrote: From: Sam Eiderman Add logical geometry variables to BlockConf. A user can now supply "lcyls", "lheads" & "lsecs" for any HD device that supports CHS ("cyls", "heads", "secs"). These devices include: * ide-hd * scsi-hd * virtio-blk-pci

Re: [PATCH v2 3/6] block/block-copy: refactor copying

2019-10-22 Thread Max Reitz
On 16.10.19 19:09, Vladimir Sementsov-Ogievskiy wrote: > Merge copying code into one function block_copy_do_copy, which only > calls bdrv_ io functions and don't do any synchronization (like dirty > bitmap set/reset). > > Refactor block_copy() function so that it takes full decision about > size o

Re: [PATCH v5 4/4] tests/qemu-iotests: add case for block-stream compress

2019-10-22 Thread Max Reitz
On 20.10.19 22:37, Andrey Shinkevich wrote: > Add a case to the iotest #030 that tests the 'compress' option for a > block-stream job. > > Signed-off-by: Andrey Shinkevich > --- > tests/qemu-iotests/030 | 34 +- > tests/qemu-iotests/030.out | 4 ++-- > 2 file

Re: [PATCH v5 1/4] block: support compressed write at generic layer

2019-10-22 Thread Max Reitz
On 20.10.19 22:37, Andrey Shinkevich wrote: > To inform the block layer about writing all the data compressed, we > introduce the 'compress' command line option. Based on that option, the > written data will be aligned by the cluster size at the generic layer. > > Suggested-by: Roman Kagan > Sugg

Re: [PATCH v3 0/6] Enable more iotests during "make check-block"

2019-10-22 Thread Thomas Huth
On 22/10/2019 09.21, Thomas Huth wrote: > As discussed here: > > https://lists.gnu.org/archive/html/qemu-devel/2019-10/msg00697.html > > and here: > > https://lists.gnu.org/archive/html/qemu-devel/2019-10/msg01388.html > > it would be good to have some more valuable iotests enabled in the > "

[PATCH v3 6/6] iotests: Remove 130 from the "auto" group

2019-10-22 Thread Thomas Huth
Peter hit a "Could not open 'TEST_DIR/t.IMGFMT': Failed to get shared 'write' lock - Is another process using the image [TEST_DIR/t.IMGFMT]?" error with 130 already twice. Looks like this test is a little bit shaky, so for the time being, let's disable it from the "auto" group so that it does not g

[PATCH v3 5/6] iotests: Enable more tests in the 'auto' group to improve test coverage

2019-10-22 Thread Thomas Huth
According to Kevin, tests 030, 040 and 041 are among the most valuable tests that we have, so we should always run them if possible, even if they take a little bit longer. According to Max, it would be good to have a test for iothreads and migration. 127 and 256 seem to be good candidates for ioth

[PATCH v3 4/6] iotests: Skip "make check-block" if QEMU does not support virtio-blk

2019-10-22 Thread Thomas Huth
The next patch is going to add some python-based tests to the "auto" group, and these tests require virtio-blk to work properly. Running iotests without virtio-blk likely does not make too much sense anyway, so instead of adding a check for the availability of virtio-blk to each and every test (whi

[PATCH v3 3/6] iotests: Test 183 does not work on macOS and OpenBSD

2019-10-22 Thread Thomas Huth
In the long term, we might want to add test 183 to the "auto" group (but it still fails occasionally, so we cannot do that yet). However, when running 183 in Cirrus-CI on macOS, or with our vm-build-openbsd target, it currently always fails with an "Timeout waiting for return on handle 0" error. L

[PATCH v3 1/6] iotests: remove 'linux' from default supported platforms

2019-10-22 Thread Thomas Huth
From: John Snow verify_platform will check an explicit whitelist and blacklist instead. The default will now be assumed to be allowed to run anywhere. For tests that do not specify their platforms explicitly, this has the effect of enabling these tests on non-linux platforms. For tests that alwa

[PATCH v3 0/6] Enable more iotests during "make check-block"

2019-10-22 Thread Thomas Huth
As discussed here: https://lists.gnu.org/archive/html/qemu-devel/2019-10/msg00697.html and here: https://lists.gnu.org/archive/html/qemu-devel/2019-10/msg01388.html it would be good to have some more valuable iotests enabled in the "auto" group to get better iotest coverage during "make check

[PATCH v3 2/6] iotests: Test 041 only works on certain systems

2019-10-22 Thread Thomas Huth
041 works fine on Linux, FreeBSD, NetBSD and OpenBSD, but fails on macOS. Let's mark it as only supported on the systems where we know that it is working fine. Signed-off-by: Thomas Huth --- tests/qemu-iotests/041 | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/qemu-i