[PATCH v8 4/5] migration: Add migrate_use_tls() helper

2022-01-31 Thread Leonardo Bras
A lot of places check parameters.tls_creds in order to evaluate if TLS is in use, and sometimes call migrate_get_current() just for that test. Add new helper function migrate_use_tls() in order to simplify testing for TLS usage. Signed-off-by: Leonardo Bras Reviewed-by: Juan Quintela

[PATCH v8 5/5] multifd: Implement zero copy write in multifd migration (multifd-zero-copy)

2022-01-31 Thread Leonardo Bras
Implement zero copy send on nocomp_send_write(), by making use of QIOChannel writev + flags & flush interface. Change multifd_send_sync_main() so flush_zero_copy() can be called after each iteration in order to make sure all dirty pages are sent before a new iteration is started. It will also

[PATCH v8 3/5] migration: Add zero-copy-send parameter for QMP/HMP for Linux

2022-01-31 Thread Leonardo Bras
Add property that allows zero-copy migration of memory pages on the sending side, and also includes a helper function migrate_use_zero_copy_send() to check if it's enabled. No code is introduced to actually do the migration, but it allow future implementations to enable/disable this feature. On

[PATCH v8 1/5] QIOChannel: Add flags on io_writev and introduce io_flush callback

2022-01-31 Thread Leonardo Bras
Add flags to io_writev and introduce io_flush as optional callback to QIOChannelClass, allowing the implementation of zero copy writes by subclasses. How to use them: - Write data using qio_channel_writev*(...,QIO_CHANNEL_WRITE_FLAG_ZERO_COPY), - Wait write completion with qio_channel_flush().

[PATCH v8 2/5] QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX

2022-01-31 Thread Leonardo Bras
For CONFIG_LINUX, implement the new zero copy flag and the optional callback io_flush on QIOChannelSocket, but enables it only when MSG_ZEROCOPY feature is available in the host kernel, which is checked on qio_channel_socket_connect_sync() qio_channel_socket_flush() was implemented by counting

[PATCH v8 0/5] MSG_ZEROCOPY + multifd

2022-01-31 Thread Leonardo Bras
This patch series intends to enable MSG_ZEROCOPY in QIOChannel, and make use of it for multifd migration performance improvement, by reducing cpu usage. Patch #1 creates new callbacks for QIOChannel, allowing the implementation of zero copy writing. Patch #2 implements io_writev flags and

[PATCH v4 2/4] python/machine: raise VMLaunchFailure exception from launch()

2022-01-31 Thread John Snow
This allows us to pack in some extra information about the failure, which guarantees that if the caller did not *intentionally* cause a failure (by capturing this Exception), some pretty good clues will be printed at the bottom of the traceback information. This will help make failures in the

[PATCH v4 1/4] python/aqmp: Fix negotiation with pre-"oob" QEMU

2022-01-31 Thread John Snow
QEMU versions prior to the "oob" capability *also* can't accept the "enable" keyword argument at all. Fix the handshake process with older QEMU versions. Signed-off-by: John Snow Reviewed-by: Hanna Reitz --- python/qemu/aqmp/qmp_client.py | 4 ++-- 1 file changed, 2 insertions(+), 2

[PATCH v4 4/4] python/aqmp: add socket bind step to legacy.py

2022-01-31 Thread John Snow
The synchronous QMP library would bind to the server address during __init__(). The new library delays this to the accept() call, because binding occurs inside of the call to start_[unix_]server(), which is an async method -- so it cannot happen during __init__ anymore. Python 3.7+ adds the

[PATCH v4 3/4] python: upgrade mypy to 0.780

2022-01-31 Thread John Snow
We need a slightly newer version of mypy in order to use some features of the asyncio server functions in the next commit. (Note: pipenv is not really suited to upgrading individual packages; I need to replace this tool with something better for the task. For now, the miscellaneous updates not

[PATCH v4 0/4] Python: Improvements for iotest 040,041

2022-01-31 Thread John Snow
GitLab: https://gitlab.com/jsnow/qemu/-/commits/python-aqmp-fixes CI: https://gitlab.com/jsnow/qemu/-/pipelines/455146881 Fixes and improvements all relating to "iotest 040,041, intermittent failure in netbsd VM" https://lists.gnu.org/archive/html/qemu-devel/2022-01/msg01975.html See each patch

Re: [PATCH] block.h: remove outdated comment

2022-01-31 Thread Philippe Mathieu-Daudé via
On 31/1/22 13:56, Emanuele Giuseppe Esposito wrote: The comment "disk I/O throttling" doesn't make any sense at all any more. It was added in commit 0563e191516 to describe bdrv_io_limits_enable()/disable(), which were removed in commit 97148076, so the comment is just a forgotten leftover.

Re: [PATCH v3 2/4] python/machine: raise VMLaunchFailure exception from launch()

2022-01-31 Thread John Snow
On Thu, Jan 27, 2022 at 9:22 AM Hanna Reitz wrote: > > On 24.01.22 19:06, John Snow wrote: > > This allows us to pack in some extra information about the failure, > > which guarantees that if the caller did not *intentionally* cause a > > failure (by capturing this Exception), some pretty good

Re: [PATCH v2] qsd: Document fuse's allow-other option

2022-01-31 Thread Eric Blake
On Mon, Jan 31, 2022 at 11:31:24AM +0100, Hanna Reitz wrote: > We did not add documentation to the storage daemon's man page for fuse's > allow-other option when it was introduced, so do that now. > > Fixes: 8fc54f9428b9763f800 ("export/fuse: Add allow-other option") > Signed-off-by: Hanna Reitz

Re: [PATCH] qapi/block: Cosmetic change in BlockExportType schema

2022-01-31 Thread Eric Blake
On Sun, Jan 30, 2022 at 07:50:41PM +0100, Philippe Mathieu-Daudé wrote: > On 28/1/22 21:54, Eric Blake wrote: > > On Wed, Jan 19, 2022 at 01:14:39PM +0100, Philippe Mathieu-Daudé wrote: > > > From: Philippe Mathieu-Daude 'git am' used this line to insert the authorship... > > > > > > From:

Re: [PATCH 2/5] docs: Only mention iscsi in the man page when available

2022-01-31 Thread Peter Maydell
On Mon, 31 Jan 2022 at 17:33, Kevin Wolf wrote: > > If libiscsi is disabled in the build, the man page shouldn't contain > information on how to construct iscsi URLs etc. > > This patch is best viewed with whitespace changes ignored. > > Signed-off-by: Kevin Wolf > --- > -``iSCSI`` > - iSCSI

Re: [PATCH 1/5] docs: Pass disabled configure options to sphinx

2022-01-31 Thread Daniel P . Berrangé
On Mon, Jan 31, 2022 at 06:04:07PM +0100, Kevin Wolf wrote: > Man pages and other documentation should only contain things that are > actually available in this build. In order to conditionally build > documentation depending on the build configuration, pass the > configuration to sphinx as tags.

[PATCH 4/5] docs: Only mention curl in the man page when available

2022-01-31 Thread Kevin Wolf
If curl is disabled in the build, the man page shouldn't contain information on how to use the curl block drivers. This patch is best viewed with whitespace changes ignored. Signed-off-by: Kevin Wolf --- docs/system/device-url-syntax.rst.inc | 114 +- 1 file changed, 58

[PATCH 5/5] docs: Only mention gluster in the man page when available

2022-01-31 Thread Kevin Wolf
If gluster is disabled in the build, the man page shouldn't contain information on how to use the gluster block driver. This patch is best viewed with whitespace changes ignored. Signed-off-by: Kevin Wolf --- docs/system/device-url-syntax.rst.inc | 64 ++- 1 file

[PATCH 1/5] docs: Pass disabled configure options to sphinx

2022-01-31 Thread Kevin Wolf
Man pages and other documentation should only contain things that are actually available in this build. In order to conditionally build documentation depending on the build configuration, pass the configuration to sphinx as tags. The tags are negative so that building documentation from outside

[PATCH 2/5] docs: Only mention iscsi in the man page when available

2022-01-31 Thread Kevin Wolf
If libiscsi is disabled in the build, the man page shouldn't contain information on how to construct iscsi URLs etc. This patch is best viewed with whitespace changes ignored. Signed-off-by: Kevin Wolf --- docs/system/device-url-syntax.rst.inc | 60 ++- qemu-options.hx

[PATCH 0/5] docs: Don't document disabled features

2022-01-31 Thread Kevin Wolf
It was reported that it's confusing that the QEMU man page contains documentation of features that aren't even available in the build because they were disabled during configure. This series makes documentation for some block drivers conditional so that it is only included in the man page (and

[PATCH 3/5] docs: Only mention ssh in the man page when available

2022-01-31 Thread Kevin Wolf
If libssh is disabled in the build, the man page shouldn't contain information on how to use the ssh block driver. This patch is best viewed with whitespace changes ignored. Signed-off-by: Kevin Wolf --- docs/system/device-url-syntax.rst.inc | 18 ++ 1 file changed, 10

Re: [PATCH v6 20/33] block: rename bdrv_invalidate_cache_all, blk_invalidate_cache and test_sync_op_invalidate_cache

2022-01-31 Thread Paolo Bonzini
On 1/27/22 10:18, Emanuele Giuseppe Esposito wrote: /* Make sure all file formats flush their mutable metadata. * If we get an error here, just don't restart the VM yet. */ -bdrv_invalidate_cache_all(_err); +bdrv_activate_all(_err); I guess that we can

Re: [PATCH v6 02/33] include/block/block: split header into I/O and global state API

2022-01-31 Thread Paolo Bonzini
On 1/31/22 15:54, Kevin Wolf wrote: So I guess the decision depends on what you're going to use the categories in the future. I get the feeling that we have one more category than this series introduces: * Global State (must run from the main thread/hold the BQL) * I/O (can be called in any

Re: [PATCH v6 01/33] main-loop.h: introduce qemu_in_main_thread()

2022-01-31 Thread Paolo Bonzini
On 1/31/22 15:33, Kevin Wolf wrote: I feel "use this function if your code is going to be used by unit tests, but use qemu_mutex_iothread_locked() otherwise" is a very strange interface. I'm relatively sure that qemu_mutex_iothread_locked() isn't primarily used to make unit tests crash.

Re: [PATCH v6 02/33] include/block/block: split header into I/O and global state API

2022-01-31 Thread Kevin Wolf
Am 31.01.2022 um 14:40 hat Emanuele Giuseppe Esposito geschrieben: > On 27/01/2022 12:01, Kevin Wolf wrote: > >>> +BlockDriverState *bs_ = (bs); \ > >>> +AIO_WAIT_WHILE(bdrv_get_aio_context(bs_), \ > >>> + cond); }) > >> Hmm...

Re: [PATCH v6 01/33] main-loop.h: introduce qemu_in_main_thread()

2022-01-31 Thread Kevin Wolf
Am 31.01.2022 um 12:25 hat Emanuele Giuseppe Esposito geschrieben: > > > On 27/01/2022 11:56, Kevin Wolf wrote: > > Am 21.01.2022 um 18:05 hat Emanuele Giuseppe Esposito geschrieben: > >> When invoked from the main loop, this function is the same > >> as qemu_mutex_iothread_locked, and returns

[PATCH] qemu-img: Unify [-b [-F]] documentation

2022-01-31 Thread Hanna Reitz
qemu-img convert documents the backing file and backing format options as follows: [-B backing_file [-F backing_fmt]] whereas qemu-img create has this: [-b backing_file] [-F backing_fmt] That is, for convert, we document that -F cannot be given without -B, while for create, way say that

Re: [PATCH v6 02/33] include/block/block: split header into I/O and global state API

2022-01-31 Thread Emanuele Giuseppe Esposito
On 27/01/2022 12:01, Kevin Wolf wrote: >> +/* Common functions that are neither I/O nor Global State */ >> + >> +int bdrv_parse_aio(const char *mode, int *flags); > Makes sense to me to have this here, it is just a helper function that > parses stuff and doesn't touch any state. However, what is

[PATCH] block.h: remove outdated comment

2022-01-31 Thread Emanuele Giuseppe Esposito
The comment "disk I/O throttling" doesn't make any sense at all any more. It was added in commit 0563e191516 to describe bdrv_io_limits_enable()/disable(), which were removed in commit 97148076, so the comment is just a forgotten leftover. Suggested-by: Kevin Wolf Signed-off-by: Emanuele

Re: [PATCH v6 01/33] main-loop.h: introduce qemu_in_main_thread()

2022-01-31 Thread Emanuele Giuseppe Esposito
On 27/01/2022 11:56, Kevin Wolf wrote: > Am 21.01.2022 um 18:05 hat Emanuele Giuseppe Esposito geschrieben: >> When invoked from the main loop, this function is the same >> as qemu_mutex_iothread_locked, and returns true if the BQL is held. > > So its name is misleading because it doesn't

[PATCH v2] qsd: Document fuse's allow-other option

2022-01-31 Thread Hanna Reitz
We did not add documentation to the storage daemon's man page for fuse's allow-other option when it was introduced, so do that now. Fixes: 8fc54f9428b9763f800 ("export/fuse: Add allow-other option") Signed-off-by: Hanna Reitz --- v2: - Replaced instances of "QSD" by more generic descriptions, as

Re: [PATCH v4 16/19] job: detect change of aiocontext within job coroutine

2022-01-31 Thread Emanuele Giuseppe Esposito
> As a small improvement, we don't need aio_co_enter() in job_start(), > we can simplify it with qemu_coroutine_enter(). If the aio_context > where it is running is wrong, job_do_yield_locked() will automatically > reschedule the coroutine where it belongs, once the first > job_pause_point is

Re: [PULL 00/38] Migration 20220127 patches

2022-01-31 Thread Juan Quintela
Peter Maydell wrote: > On Thu, 27 Jan 2022 at 15:06, Juan Quintela wrote: >> >> The following changes since commit 48302d4eb628ff0bea4d7e92cbf6b726410eb4c3: >> >> Merge remote-tracking branch >> 'remotes/dgilbert-gitlab/tags/pull-virtiofs-20220126' into staging >> (2022-01-26 10:59:50 +)

Re: [PATCH v6 29/33] job.h: assertions in the callers of JobDriver funcion pointers

2022-01-31 Thread Hanna Reitz
On 28.01.22 16:19, Emanuele Giuseppe Esposito wrote: On 26/01/2022 15:13, Hanna Reitz wrote: On 21.01.22 18:05, Emanuele Giuseppe Esposito wrote: Signed-off-by: Emanuele Giuseppe Esposito ---   job.c | 9 +   1 file changed, 9 insertions(+) Just curious, why did you remove the