Re: bitmap migration bug with -drive while block mirror runs

2019-10-03 Thread John Snow
On 10/3/19 6:14 AM, Vladimir Sementsov-Ogievskiy wrote: > 03.10.2019 0:35, John Snow wrote: >> On 10/2/19 6:46 AM, Peter Krempa wrote: >> >> [ * poof * ] >> >>> >>> I'd like to re-iterate that the necessity to keep node names same on >>> both sides of migration is unexpected, undocumented and in

Re: [PULL v2 0/8] Ide patches

2019-10-03 Thread no-reply
Patchew URL: https://patchew.org/QEMU/20191003193245.8993-1-js...@redhat.com/ Hi, This series seems to have some coding style problems. See output below for more information: Type: series Message-id: 20191003193245.8993-1-js...@redhat.com Subject: [PULL v2 0/8] Ide patches === TEST SCRIPT BEG

[PULL v2 4/8] scsi: Propagate unrealize() callback to scsi-hd

2019-10-03 Thread John Snow
From: Sam Eiderman We will need to add LCHS removal logic to scsi-hd's unrealize() in the next commit. Signed-off-by: Sam Eiderman Reviewed-by: Karl Heubaum Reviewed-by: Arbel Moshe Signed-off-by: Sam Eiderman Message-id: 20190925110639.100699-5-sam...@google.com Signed-off-by: John Snow --

[PULL v2 5/8] bootdevice: Gather LCHS from all relevant devices

2019-10-03 Thread John Snow
From: Sam Eiderman Relevant devices are: * ide-hd (and ide-cd, ide-drive) * scsi-hd (and scsi-cd, scsi-disk, scsi-block) * virtio-blk-pci We do not call del_boot_device_lchs() for ide-* since we don't need to - IDE block devices do not support unplugging. Signed-off-by: Sam Eiderman

[PULL v2 8/8] hd-geo-test: Add tests for lchs override

2019-10-03 Thread John Snow
From: Sam Eiderman Add QTest tests to check the logical geometry override option. The tests in hd-geo-test are out of date - they only test IDE and do not test interesting MBRs. I added a few helper functions which will make adding more tests easier. QTest's fw_cfg helper functions support onl

[PULL v2 2/8] block: Support providing LCHS from user

2019-10-03 Thread John Snow
From: Sam Eiderman Add logical geometry variables to BlockConf. A user can now supply "lcyls", "lheads" & "lsecs" for any HD device that supports CHS ("cyls", "heads", "secs"). These devices include: * ide-hd * scsi-hd * virtio-blk-pci In future commits we will use the provided LCH

[PULL v2 6/8] bootdevice: Refactor get_boot_devices_list

2019-10-03 Thread John Snow
From: Sam Eiderman Move device name construction to a separate function. We will reuse this function in the following commit to pass logical CHS parameters through fw_cfg much like we currently pass bootindex. Reviewed-by: Karl Heubaum Reviewed-by: Arbel Moshe Signed-off-by: Sam Eiderman Mes

[PULL v2 7/8] bootdevice: FW_CFG interface for LCHS values

2019-10-03 Thread John Snow
From: Sam Eiderman Using fw_cfg, supply logical CHS values directly from QEMU to the BIOS. Non-standard logical geometries break under QEMU. A virtual disk which contains an operating system which depends on logical geometries (consistent values being reported from BIOS INT13 AH=08) will most l

[PULL v2 0/8] Ide patches

2019-10-03 Thread John Snow
The following changes since commit 7f21573c822805a8e6be379d9bcf3ad9effef3dc: Merge remote-tracking branch 'remotes/huth-gitlab/tags/pull-request-2019-10-01' into staging (2019-10-01 13:13:38 +0100) are available in the Git repository at: https://github.com/jnsnow/qemu.git tags/ide-pull-req

[PULL v2 3/8] bootdevice: Add interface to gather LCHS

2019-10-03 Thread John Snow
From: Sam Eiderman Add an interface to provide direct logical CHS values for boot devices. We will use this interface in the next commits. Reviewed-by: Karl Heubaum Reviewed-by: Arbel Moshe Signed-off-by: Sam Eiderman Message-id: 20190925110639.100699-4-sam...@google.com Signed-off-by: John S

[PULL v2 1/8] block: Refactor macros - fix tabbing

2019-10-03 Thread John Snow
From: Sam Eiderman Fixing tabbing in block related macros. Reviewed-by: Karl Heubaum Reviewed-by: Arbel Moshe Signed-off-by: Sam Eiderman Message-id: 20190925110639.100699-2-sam...@google.com Signed-off-by: John Snow --- include/hw/block/block.h | 16 hw/ide/qdev.c

Re: [PULL 0/8] Ide patches

2019-10-03 Thread John Snow
On 10/3/19 11:35 AM, Peter Maydell wrote: > On Wed, 2 Oct 2019 at 00:56, John Snow wrote: >> >> The following changes since commit 7f21573c822805a8e6be379d9bcf3ad9effef3dc: >> >> Merge remote-tracking branch >> 'remotes/huth-gitlab/tags/pull-request-2019-10-01' into staging (2019-10-01 >> 1

Re: [PULL 0/8] Ide patches

2019-10-03 Thread John Snow
On 10/3/19 11:35 AM, Peter Maydell wrote: > On Wed, 2 Oct 2019 at 00:56, John Snow wrote: >> >> The following changes since commit 7f21573c822805a8e6be379d9bcf3ad9effef3dc: >> >> Merge remote-tracking branch >> 'remotes/huth-gitlab/tags/pull-request-2019-10-01' into staging (2019-10-01 >> 1

[PATCH 2/6] block/block-copy: limit copy_range_size to 16 MiB

2019-10-03 Thread Vladimir Sementsov-Ogievskiy
Large copy range may imply memory allocation and large io effort, so using 2G copy range request may be bad idea. Let's limit it to 16 MiB. It also helps the following patch to refactor copy-with-offload fallback to copy-with-bounce-buffer. Note, that total memory usage of backup is still not limi

[PATCH 5/6] block/block-copy: add memory limit

2019-10-03 Thread Vladimir Sementsov-Ogievskiy
Currently total allocation for parallel requests to block-copy instance is unlimited. Let's limit it to 128 MiB. For now block-copy is used only in backup, so actually we limit total allocation for backup job. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/block-copy.h | 3 +++ b

[PATCH 4/6] util: introduce co-shared-amount

2019-10-03 Thread Vladimir Sementsov-Ogievskiy
Introduce an API for some shared splitable resource, like memory. It's going to be used by backup. Backup uses both read/write io and copy_range. copy_range may consume memory implictly, so new API is abstract: it don't allocate any real memory but only handling out tickets. The idea is that we ha

[PATCH 3/6] block/block-copy: refactor copying

2019-10-03 Thread Vladimir Sementsov-Ogievskiy
Merge copying code into one function block_copy_do_copy, which only calls bdrv_ io functions and don't do any synchronization (like dirty bitmap set/reset). Refactor block_copy() function so that it takes full decision about size of chunk to be copied and does all the synchronization (checking int

[PATCH 0/6] block-copy: memory limit

2019-10-03 Thread Vladimir Sementsov-Ogievskiy
Hi all! I'm going to bring block-status driven, async copying process to block-copy, to make it fast. The first step is to limit memory usage of backup, here is it. Based on my "[PATCH v15 0/5] backup-top filter driver for backup": Based-on: <20191001131409.14202-1-vsement...@virtuozzo.com> Vlad

[PATCH 6/6] block/block-copy: increase buffered copy request

2019-10-03 Thread Vladimir Sementsov-Ogievskiy
No reason to limit buffered copy to one cluster. Let's allow up to 1 MiB. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/block-copy.h | 2 +- block/block-copy.c | 44 +++--- 2 files changed, 32 insertions(+), 14 deletions(-) diff --git a/i

[PATCH 1/6] block/block-copy: allocate buffer in block_copy_with_bounce_buffer

2019-10-03 Thread Vladimir Sementsov-Ogievskiy
Move bounce_buffer allocation block_copy_with_bounce_buffer. This commit simplifies further work on implementing copying by larger chunks (of different size) and further asynchronous handling of block_copy iterations (with help of block/aio_task API). Allocation works fast, a lot faster than disk

Re: [PULL v2 0/4] Block layer patches

2019-10-03 Thread Peter Maydell
On Wed, 2 Oct 2019 at 17:18, Kevin Wolf wrote: > > The following changes since commit 7f21573c822805a8e6be379d9bcf3ad9effef3dc: > > Merge remote-tracking branch > 'remotes/huth-gitlab/tags/pull-request-2019-10-01' into staging (2019-10-01 > 13:13:38 +0100) > > are available in the Git reposito

Re: [PULL 0/8] Ide patches

2019-10-03 Thread Peter Maydell
On Wed, 2 Oct 2019 at 00:56, John Snow wrote: > > The following changes since commit 7f21573c822805a8e6be379d9bcf3ad9effef3dc: > > Merge remote-tracking branch > 'remotes/huth-gitlab/tags/pull-request-2019-10-01' into staging (2019-10-01 > 13:13:38 +0100) > > are available in the Git repositor

Re: [PATCH 03/67] iotests.py: Add unsupported_imgopts

2019-10-03 Thread Vladimir Sementsov-Ogievskiy
01.10.2019 22:46, Max Reitz wrote: > Signed-off-by: Max Reitz > --- > tests/qemu-iotests/iotests.py | 7 ++- > 1 file changed, 6 insertions(+), 1 deletion(-) > > diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py > index cdcb62c4ac..b5ea424de4 100644 > --- a/tests/q

Re: [PATCH 02/67] iotests.py: Add @skip_for_imgopts()

2019-10-03 Thread Vladimir Sementsov-Ogievskiy
01.10.2019 22:46, Max Reitz wrote: > Signed-off-by: Max Reitz > --- > tests/qemu-iotests/iotests.py | 13 + > 1 file changed, 13 insertions(+) > > diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py > index 7030900807..cdcb62c4ac 100644 > --- a/tests/qemu-iot

Re: [PATCH 01/67] iotests.py: Read $IMGOPTS

2019-10-03 Thread Vladimir Sementsov-Ogievskiy
01.10.2019 22:46, Max Reitz wrote: > We do not do anything with yet, but this is the first step. > > Signed-off-by: Max Reitz > --- > tests/qemu-iotests/iotests.py | 6 ++ > 1 file changed, 6 insertions(+) > > diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py > in

Re: [PATCH 01/67] iotests.py: Read $IMGOPTS

2019-10-03 Thread Vladimir Sementsov-Ogievskiy
02.10.2019 1:16, John Snow wrote: > > > On 10/1/19 3:46 PM, Max Reitz wrote: >> We do not do anything with yet, but this is the first step. >> >> Signed-off-by: Max Reitz >> --- >> tests/qemu-iotests/iotests.py | 6 ++ >> 1 file changed, 6 insertions(+) >> >> diff --git a/tests/qemu-iotes

Re: [PATCH v2 6/6] tests/qemu-iotests: add case for block-stream compress

2019-10-03 Thread Vladimir Sementsov-Ogievskiy
02.10.2019 17:22, Andrey Shinkevich wrote: > Add a test case to the iotest #030 that checks 'compress' option for a > block-stream job. > > Signed-off-by: Andrey Shinkevich > --- > tests/qemu-iotests/030 | 49 > +- > tests/qemu-iotests/030.out |

Re: [PATCH v2 5/6] block-stream: add compress option

2019-10-03 Thread Vladimir Sementsov-Ogievskiy
02.10.2019 17:22, Andrey Shinkevich wrote: > Allow data compression during block-stream job for backup backing chain. > > Signed-off-by: Anton Nefedov > Signed-off-by: Vladimir Sementsov-Ogievskiy > Signed-off-by: Denis V. Lunev > Signed-off-by: Andrey Shinkevich > --- > block/stream.c

Re: [PATCH v2 4/6] block: support compressed write for copy-on-read

2019-10-03 Thread Vladimir Sementsov-Ogievskiy
02.10.2019 17:22, Andrey Shinkevich wrote: > Support the data compression during block-stream job over a backup > backing chain implemented in the following patch 'block-stream: > add compress option'. > > Signed-off-by: Anton Nefedov > Signed-off-by: Denis V. Lunev > Signed-off-by: Andrey Shink

Re: [PATCH v2 3/6] qemu-nbd: add compression flag support

2019-10-03 Thread Vladimir Sementsov-Ogievskiy
02.10.2019 17:22, Andrey Shinkevich wrote: > Added possibility to write compressed data by using the > blk_write_compressed. This action has the limitations of the format > driver. For example we can't write compressed data over other. > > $ ./qemu-img create -f qcow2 -o size=10G ./image.qcow2 > $

Re: [PATCH v2 2/6] tests/qemu-iotests: add case of writing compressed data to multiple clusters

2019-10-03 Thread Vladimir Sementsov-Ogievskiy
02.10.2019 17:22, Andrey Shinkevich wrote: > Add the test case to the iotest #214 that checks possibility of writing > compressed data to more than one cluster. > > Signed-off-by: Andrey Shinkevich > --- > tests/qemu-iotests/214 | 9 + > tests/qemu-iotests/214.out | 6 ++ > 2

Re: [PATCH v2 1/6] qcow2: Allow writing compressed data to multiple clusters

2019-10-03 Thread Vladimir Sementsov-Ogievskiy
02.10.2019 17:22, Andrey Shinkevich wrote: > QEMU currently supports writing compressed data of size less than or > equal to one cluster. This patch allows writing QCOW2 compressed data > that exceed one cluster. The implementation is simple, we split buffered > data into separate clusters and writ

Re: bitmap migration bug with -drive while block mirror runs

2019-10-03 Thread Vladimir Sementsov-Ogievskiy
03.10.2019 0:35, John Snow wrote: > On 10/2/19 6:46 AM, Peter Krempa wrote: > > [ * poof * ] > >> >> I'd like to re-iterate that the necessity to keep node names same on >> both sides of migration is unexpected, undocumented and in some cases >> impossible. >> >> If you want to mandate that they

Re: [PATCH] test-bdrv-drain: fix iothread_join() hang

2019-10-03 Thread Paolo Bonzini
On 03/10/19 12:01, Stefan Hajnoczi wrote: > tests/test-bdrv-drain can hang in tests/iothread.c:iothread_run(): > > while (!atomic_read(&iothread->stopping)) { > aio_poll(iothread->ctx, true); > } > > The iothread_join() function works as follows: > > void iothread_join(IOThread *ioth

[PATCH] test-bdrv-drain: fix iothread_join() hang

2019-10-03 Thread Stefan Hajnoczi
tests/test-bdrv-drain can hang in tests/iothread.c:iothread_run(): while (!atomic_read(&iothread->stopping)) { aio_poll(iothread->ctx, true); } The iothread_join() function works as follows: void iothread_join(IOThread *iothread) { iothread->stopping = true; aio_notify(

Re: [PATCH 3/4] block/mirror: support unaligned write in active mirror

2019-10-03 Thread Vladimir Sementsov-Ogievskiy
02.10.2019 18:52, Max Reitz wrote: > On 02.10.19 17:06, Vladimir Sementsov-Ogievskiy wrote: >> 02.10.2019 18:03, Vladimir Sementsov-Ogievskiy wrote: >>> 02.10.2019 17:57, Max Reitz wrote: On 12.09.19 17:13, Vladimir Sementsov-Ogievskiy wrote: > Prior 9adc1cb49af8d do_sync_target_write had

Re: [Qemu-block] [PATCH v2 1/2] blockdev: release the AioContext at drive_backup_prepare

2019-10-03 Thread Sergio Lopez
Sergio Lopez writes: > Kevin Wolf writes: > >> Am 13.09.2019 um 21:54 hat John Snow geschrieben: >>> >>> >>> On 9/13/19 11:25 AM, Sergio Lopez wrote: >>> > do_drive_backup() already acquires the AioContext, so release it >>> > before the call. >>> > >>> > Signed-off-by: Sergio Lopez >>> > -