Re: [PATCH v2 3/5] iotests: Remove explicit checks for qemu_img() == 0

2022-03-04 Thread John Snow
On Fri, Mar 4, 2022 at 3:23 PM Eric Blake wrote: > > On Fri, Mar 04, 2022 at 02:47:44PM -0500, John Snow wrote: > > qemu_img() returning zero ought to be the rule, not the > > exception. Remove all explicit checks against the condition in > > preparation for making non-zero returns an Exception.

Re: [PATCH v2 5/9] meson.build: Don't misdetect posix_memalign() on Windows

2022-03-04 Thread Philippe Mathieu-Daudé
On 4/3/22 12:21, Peter Maydell wrote: Currently we incorrectly think that posix_memalign() exists on Windows. This is because of a combination of: * the msys2/mingw toolchain/libc claim to have a __builtin_posix_memalign when there isn't a builtin of that name * meson will assume that

Re: [PATCH] hw/block: m25p80: Add support for w25q01jvq

2022-03-04 Thread Michael Walle
Am 2022-03-04 20:30, schrieb Philippe Mathieu-Daudé: On 4/3/22 19:09, Patrick Williams wrote: The w25q01jvq is a 128MB part. Support is being added to the kernel[1] and the two have been tested together. 1. https://lore.kernel.org/lkml/2022022209.23108-1-potin@quantatw.com/

[PATCH] hw/block: m25p80: Add support for w25q01jvq

2022-03-04 Thread Patrick Williams
The w25q01jvq is a 128MB part. Support is being added to the kernel[1] and the two have been tested together. 1. https://lore.kernel.org/lkml/2022022209.23108-1-potin@quantatw.com/ Signed-off-by: Patrick Williams Cc: Potin Lai --- hw/block/m25p80.c | 1 + 1 file changed, 1

Re: [PATCH v2 12/12] tests/qemu-iotests: validate NBD TLS with UNIX sockets and PSK

2022-03-04 Thread Eric Blake
On Fri, Mar 04, 2022 at 07:36:10PM +, Daniel P. Berrangé wrote: > This validates that connections to an NBD server running on a UNIX > socket can use TLS with pre-shared keys (PSK). > > Reviewed-by: Eric Blake > Signed-off-by: Daniel P. Berrangé > --- > tests/qemu-iotests/233| 28

[PATCH v2 4/5] Replace qemu_gettimeofday() with g_get_real_time()

2022-03-04 Thread marcandre . lureau
From: Marc-André Lureau GLib g_get_real_time() is an alternative to gettimeofday() which allows to simplify our code. For semihosting, a few bits are lost on POSIX host, but this shouldn't be a big concern. Signed-off-by: Marc-André Lureau Reviewed-by: Laurent Vivier --- blockdev.c

[PATCH v2 3/5] qga: replace qemu_gettimeofday() with g_get_real_time()

2022-03-04 Thread marcandre . lureau
From: Marc-André Lureau GLib g_get_real_time() is an alternative to gettimeofday() which allows to simplify our code. Signed-off-by: Marc-André Lureau Reviewed-by: Laurent Vivier --- qga/commands-posix.c | 14 -- qga/commands-win32.c | 19 --- qga/commands.c

[PATCH v2 5/5] oslib: drop qemu_gettimeofday()

2022-03-04 Thread marcandre . lureau
From: Marc-André Lureau No longer used after the previous patches. Signed-off-by: Marc-André Lureau Reviewed-by: Laurent Vivier Reviewed-by: Stefan Weil Reviewed-by: Richard Henderson --- include/sysemu/os-posix.h | 3 --- include/sysemu/os-win32.h | 6 -- util/oslib-win32.c|

[PATCH v2 1/5] m68k/nios2-semi: fix gettimeofday() result check

2022-03-04 Thread marcandre . lureau
From: Marc-André Lureau gettimeofday() returns 0 for success. Signed-off-by: Marc-André Lureau Reviewed-by: Laurent Vivier Reviewed-by: Richard Henderson --- target/m68k/m68k-semi.c | 2 +- target/nios2/nios2-semi.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git

[PATCH v2 2/5] qtest: replace gettimeofday with GTimer

2022-03-04 Thread marcandre . lureau
From: Marc-André Lureau glib provides a convenience helper to measure elapsed time. It isn't subject to wall-clock time changes. Note that this changes the initial OPENED time, which used to print the current time. Signed-off-by: Marc-André Lureau Reviewed-by: Laurent Vivier ---

[PATCH v2 0/5] Remove qemu_gettimeofday()

2022-03-04 Thread marcandre . lureau
From: Marc-André Lureau Hi, Here is a few patches to replace qemu_gettimeofday() helper with functions provided by GLib. v2: - fix return of get-time in nanoseconds - qga: replace with a common get-time function for qga posix/win32 - split qga patch - add r-b tags, drop RFC Marc-André Lureau

Re: [PATCH v2 11/12] tests/qemu-iotests: validate NBD TLS with UNIX sockets

2022-03-04 Thread Eric Blake
On Fri, Mar 04, 2022 at 07:36:09PM +, Daniel P. Berrangé wrote: > This validates that connections to an NBD server running on a UNIX > socket can use TLS, and require a TLS hostname override to pass > certificate validation. > > Reviewed-by: Eric Blake > Signed-off-by: Daniel P. Berrangé >

Re: [PATCH 3/4] Replace qemu_gettimeofday() with g_get_real_time()

2022-03-04 Thread Marc-André Lureau
Hi On Sat, Mar 5, 2022 at 12:40 AM BALATON Zoltan wrote: > On Fri, 4 Mar 2022, marcandre.lur...@redhat.com wrote: > > From: Marc-André Lureau > > > > GLib g_get_real_time() is an alternative to gettimeofday(). > > > > For semihosting, a few bits are lost on POSIX host, but this shouldn't > >

Re: [PATCH 3/4] Replace qemu_gettimeofday() with g_get_real_time()

2022-03-04 Thread BALATON Zoltan
On Fri, 4 Mar 2022, marcandre.lur...@redhat.com wrote: From: Marc-André Lureau GLib g_get_real_time() is an alternative to gettimeofday(). For semihosting, a few bits are lost on POSIX host, but this shouldn't be a big concern. Signed-off-by: Marc-André Lureau --- blockdev.c

Re: [PATCH v2 08/12] tests/qemu-iotests: introduce filter for qemu-nbd export list

2022-03-04 Thread Eric Blake
On Fri, Mar 04, 2022 at 07:36:06PM +, Daniel P. Berrangé wrote: > Introduce a filter for the output of qemu-nbd export list so it can be > reused in multiple tests. > > The filter is a bit more permissive that what test 241 currently uses, > as its allows printing of the export count, along

Re: [PATCH v2 3/5] iotests: Remove explicit checks for qemu_img() == 0

2022-03-04 Thread Eric Blake
On Fri, Mar 04, 2022 at 02:47:44PM -0500, John Snow wrote: > qemu_img() returning zero ought to be the rule, not the > exception. Remove all explicit checks against the condition in > preparation for making non-zero returns an Exception. > > Signed-off-by: John Snow > --- > +++

Re: [PATCH v2 6/9] util: Share qemu_try_memalign() implementation between POSIX and Windows

2022-03-04 Thread Richard Henderson
On 3/4/22 01:21, Peter Maydell wrote: + * platform APIs (for instance _alligned_malloc() will Typo there with the 'll'. r~

Re: [PATCH v2 4/9] util: Return valid allocation for qemu_try_memalign() with zero size

2022-03-04 Thread Richard Henderson
On 3/4/22 01:21, Peter Maydell wrote: Currently qemu_try_memalign()'s behaviour if asked to allocate 0 bytes is rather variable: * on Windows, we will assert * on POSIX platforms, we get the underlying behaviour of the posix_memalign() or equivalent function, which may be either

[PATCH v2 2/5] python/utils: add VerboseProcessError

2022-03-04 Thread John Snow
This adds an Exception that extends the Python stdlib subprocess.CalledProcessError. The difference is that the str() method of this exception also adds the stdout/stderr logs. In effect, if this exception goes unhandled, Python will print the output in a visually distinct wrapper to the terminal

[PATCH v2 3/5] iotests: Remove explicit checks for qemu_img() == 0

2022-03-04 Thread John Snow
qemu_img() returning zero ought to be the rule, not the exception. Remove all explicit checks against the condition in preparation for making non-zero returns an Exception. Signed-off-by: John Snow --- tests/qemu-iotests/163 | 9 +++-- tests/qemu-iotests/216

[PATCH v2 1/5] python/utils: add add_visual_margin() text decoration utility

2022-03-04 Thread John Snow
>>> print(add_visual_margin(msg, width=72, name="Commit Message")) ┏━ Commit Message ━━ ┃ add_visual_margin() takes a chunk of text and wraps it in a visual ┃ container that force-wraps to a specified width. An optional title ┃ label may be

[PATCH v2 0/5] iotests: add enhanced debugging info to qemu-img failures

2022-03-04 Thread John Snow
V2: - Rebase on top of kwolf's latest PR. - Adjust tests/graph-changes-while-io in patch 3/5 - Drop eblake's r-b on 3/5. This is secretly V4ish of a series I started in response to Thomas Huth's encountering a failure in qemu-img because of missing zstd support. This series changes the

[PATCH v2 5/5] iotests: fortify compare_images() against crashes

2022-03-04 Thread John Snow
Fortify compare_images() to be more discerning about the status codes it receives. If qemu_img() returns an exit code that implies it didn't actually perform the comparison, treat that as an exceptional circumstance and force the caller to be aware of the peril. If a negative test is desired

[PATCH v2 4/5] iotests: make qemu_img raise on non-zero rc by default

2022-03-04 Thread John Snow
re-write qemu_img() as a function that will by default raise a VerboseProcessException (extended from CalledProcessException) on non-zero return codes. This will produce a stack trace that will show the command line arguments and return code from the failed process run. Users that want something

[PATCH v2 09/12] tests/qemu-iotests: convert NBD TLS test to use standard filters

2022-03-04 Thread Daniel P . Berrangé
Using standard filters is more future proof than rolling our own. Reviewed-by: Eric Blake Signed-off-by: Daniel P. Berrangé --- tests/qemu-iotests/233 | 29 - tests/qemu-iotests/233.out | 8 2 files changed, 16 insertions(+), 21 deletions(-) diff

[PATCH v2 06/12] tests/qemu-iotests: add QEMU_IOTESTS_REGEN=1 to update reference file

2022-03-04 Thread Daniel P . Berrangé
When developing an I/O test it is typical to add some logic to the test script, run it to view the output diff, and then apply the output diff to the reference file. This can be drastically simplified by letting the test runner update the reference file in place. By setting

[PATCH v2 08/12] tests/qemu-iotests: introduce filter for qemu-nbd export list

2022-03-04 Thread Daniel P . Berrangé
Introduce a filter for the output of qemu-nbd export list so it can be reused in multiple tests. The filter is a bit more permissive that what test 241 currently uses, as its allows printing of the export count, along with any possible error messages that might be emitted. Reviewed-by: Eric

[PATCH v2 04/12] qemu-nbd: add --tls-hostname option for TLS certificate validation

2022-03-04 Thread Daniel P . Berrangé
When using the --list option, qemu-nbd acts as an NBD client rather than a server. As such when using TLS, it has a need to validate the server certificate. This adds a --tls-hostname option which can be used to override the default hostname used for certificate validation. Reviewed-by: Eric

[PATCH v2 10/12] tests/qemu-iotests: validate NBD TLS with hostname mismatch

2022-03-04 Thread Daniel P . Berrangé
This validates that connections to an NBD server where the certificate hostname does not match will fail. It further validates that using the new 'tls-hostname' override option can solve the failure. Reviewed-by: Eric Blake Signed-off-by: Daniel P. Berrangé --- tests/qemu-iotests/233|

[PATCH v2 07/12] tests/qemu-iotests: expand _filter_nbd rules

2022-03-04 Thread Daniel P . Berrangé
Some tests will want to use 'localhost' instead of '127.0.0.1', and some will use the image options syntax rather than the classic URI syntax. Reviewed-by: Eric Blake Signed-off-by: Daniel P. Berrangé --- tests/qemu-iotests/common.filter | 4 1 file changed, 4 insertions(+) diff --git

[PATCH v2 12/12] tests/qemu-iotests: validate NBD TLS with UNIX sockets and PSK

2022-03-04 Thread Daniel P . Berrangé
This validates that connections to an NBD server running on a UNIX socket can use TLS with pre-shared keys (PSK). Reviewed-by: Eric Blake Signed-off-by: Daniel P. Berrangé --- tests/qemu-iotests/233| 28 tests/qemu-iotests/233.out| 17 +

[PATCH v2 01/12] crypto: mandate a hostname when checking x509 creds on a client

2022-03-04 Thread Daniel P . Berrangé
Currently the TLS session object assumes that the caller will always provide a hostname when using x509 creds on a client endpoint. This relies on the caller to detect and report an error if the user has configured QEMU with x509 credentials on a UNIX socket. The migration code has such a check,

[PATCH v2 11/12] tests/qemu-iotests: validate NBD TLS with UNIX sockets

2022-03-04 Thread Daniel P . Berrangé
This validates that connections to an NBD server running on a UNIX socket can use TLS, and require a TLS hostname override to pass certificate validation. Reviewed-by: Eric Blake Signed-off-by: Daniel P. Berrangé --- tests/qemu-iotests/233 | 24

[PATCH v2 02/12] block: pass desired TLS hostname through from block driver client

2022-03-04 Thread Daniel P . Berrangé
In commit a71d597b989fd701b923f09b3c20ac4fcaa55e81 Author: Vladimir Sementsov-Ogievskiy Date: Thu Jun 10 13:08:00 2021 +0300 block/nbd: reuse nbd_co_do_establish_connection() in nbd_open() the use of the 'hostname' field from the BDRVNBDState struct was lost, and 'nbd_connect' just

[PATCH v2 03/12] block/nbd: support override of hostname for TLS certificate validation

2022-03-04 Thread Daniel P . Berrangé
When connecting to an NBD server with TLS and x509 credentials, the client must validate the hostname it uses for the connection, against that published in the server's certificate. If the client is tunnelling its connection over some other channel, however, the hostname it uses may not match the

[PATCH v2 05/12] block/nbd: don't restrict TLS usage to IP sockets

2022-03-04 Thread Daniel P . Berrangé
The TLS usage for NBD was restricted to IP sockets because validating x509 certificates requires knowledge of the hostname that the client is connecting to. TLS does not have to use x509 certificates though, as PSK (pre-shared keys) provide an alternative credential option. These have no

Re: [PATCH 08/12] tests/qemu-iotests: introduce filter for qemu-nbd export list

2022-03-04 Thread Daniel P . Berrangé
On Fri, Mar 04, 2022 at 10:43:45AM -0600, Eric Blake wrote: > On Thu, Mar 03, 2022 at 04:03:26PM +, Daniel P. Berrangé wrote: > > Introduce a filter for the output of qemu-nbd export list so it can be > > reused in multiple tests. > > > > The filter is a bit more permissive that what test 241

[PATCH v2 00/12] nbd: enable use of TLS on non-TCP transports and other TLS improvements

2022-03-04 Thread Daniel P . Berrangé
This series was principally motivated by a desire to enabl use of TLS on non-TCP transports. For x509 certificates this means we need a way to set the hostname to use for validation. This also lets us override the hostname when connecting on a TCP transport that is tunnelled or port-forwarded. It

Re: [PATCH] hw/block: m25p80: Add support for w25q01jvq

2022-03-04 Thread Philippe Mathieu-Daudé
On 4/3/22 19:09, Patrick Williams wrote: The w25q01jvq is a 128MB part. Support is being added to the kernel[1] and the two have been tested together. 1. https://lore.kernel.org/lkml/2022022209.23108-1-potin@quantatw.com/ Signed-off-by: Patrick Williams Cc: Potin Lai ---

Re: [PATCH 02/12] block: pass desired TLS hostname through from block driver client

2022-03-04 Thread Daniel P . Berrangé
On Thu, Mar 03, 2022 at 02:14:34PM -0600, Eric Blake wrote: > On Thu, Mar 03, 2022 at 04:03:20PM +, Daniel P. Berrangé wrote: > > In > > > > commit a71d597b989fd701b923f09b3c20ac4fcaa55e81 > > Author: Vladimir Sementsov-Ogievskiy > > Date: Thu Jun 10 13:08:00 2021 +0300 > > > >

[PULL 50/50] block/amend: Keep strong reference to BDS

2022-03-04 Thread Kevin Wolf
From: Hanna Reitz Otherwise, the BDS might be freed while the job is running, which would cause a use-after-free. Signed-off-by: Hanna Reitz Message-Id: <20220304153729.711387-5-hre...@redhat.com> Signed-off-by: Kevin Wolf --- block/amend.c | 3 +++ 1 file changed, 3 insertions(+) diff

[PULL 41/50] block_int-common.h: assertions in the callers of BdrvChildClass function pointers

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito Signed-off-by: Emanuele Giuseppe Esposito Message-Id: <20220303151616.325444-29-eespo...@redhat.com> Signed-off-by: Kevin Wolf --- block.c | 10 +- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/block.c b/block.c index

[PULL 36/50] block/copy-before-write.h: global state API + assertions

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito copy-before-write functions always run under BQL. Signed-off-by: Emanuele Giuseppe Esposito Message-Id: <20220303151616.325444-24-eespo...@redhat.com> Signed-off-by: Kevin Wolf --- block/copy-before-write.h | 7 +++ block/copy-before-write.c | 2 ++ 2

[PULL 49/50] block/amend: Always call .bdrv_amend_clean()

2022-03-04 Thread Kevin Wolf
From: Hanna Reitz .bdrv_amend_clean() says block drivers can use it to clean up what was done in .bdrv_amend_pre_run(). Therefore, it should always be called after .bdrv_amend_pre_run(), which means we need it to call it in the JobDriver.free() callback, not in JobDriver.clean().

[PULL 32/50] assertions for blockjob.h global state API

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito Signed-off-by: Emanuele Giuseppe Esposito Message-Id: <20220303151616.325444-20-eespo...@redhat.com> Signed-off-by: Kevin Wolf --- blockjob.c | 11 +++ 1 file changed, 11 insertions(+) diff --git a/blockjob.c b/blockjob.c index d79a52d204..4868453d74

[PULL 33/50] include/sysemu/blockdev.h: global state API

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito blockdev functions run always under the BQL lock. Signed-off-by: Emanuele Giuseppe Esposito Message-Id: <20220303151616.325444-21-eespo...@redhat.com> Signed-off-by: Kevin Wolf --- include/sysemu/blockdev.h | 13 ++--- 1 file changed, 10

[PULL 43/50] job.h: split function pointers in JobDriver

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito The job API will be handled separately in another serie. Signed-off-by: Emanuele Giuseppe Esposito Message-Id: <20220303151616.325444-31-eespo...@redhat.com> Signed-off-by: Kevin Wolf --- include/qemu/job.h | 22 ++ 1 file changed, 22

[PULL 40/50] block_int-common.h: split function pointers in BdrvChildClass

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito Signed-off-by: Emanuele Giuseppe Esposito Message-Id: <20220303151616.325444-28-eespo...@redhat.com> Signed-off-by: Kevin Wolf --- include/block/block_int-common.h | 81 ++-- 1 file changed, 47 insertions(+), 34 deletions(-) diff

[PULL 42/50] block-backend-common.h: split function pointers in BlockDevOps

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito Assertions in the callers of the function pointrs are already added by previous patches. Signed-off-by: Emanuele Giuseppe Esposito Reviewed-by: Stefan Hajnoczi Reviewed-by: Philippe Mathieu-Daudé Message-Id: <20220303151616.325444-30-eespo...@redhat.com>

[PULL 37/50] block/coroutines: I/O and "I/O or GS" API

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito block coroutines functions run in different aiocontext, and are not protected by the BQL. Therefore are I/O. On the other side, generated_co_wrapper functions use BDRV_POLL_WHILE, meaning the caller can either be the main loop or a specific iothread.

[PULL 48/50] tests/qemu-iotests: Rework the checks and spots using GNU sed

2022-03-04 Thread Kevin Wolf
From: Thomas Huth Instead of failing the iotests if GNU sed is not available (or skipping them completely in the check-block.sh script), it would be better to simply skip the bash-based tests that rely on GNU sed, so that the other tests could still be run. Thus we now explicitely use "gsed"

[PULL 35/50] include/block/snapshot: global state API + assertions

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito Snapshots run also under the BQL, so they all are in the global state API. The aiocontext lock that they hold is currently an overkill and in future could be removed. Signed-off-by: Emanuele Giuseppe Esposito Message-Id:

[PULL 30/50] block.c: add assertions to static functions

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito Following the assertion derived from the API split, propagate the assertion also in the static functions. Signed-off-by: Emanuele Giuseppe Esposito Message-Id: <20220303151616.325444-18-eespo...@redhat.com> Signed-off-by: Kevin Wolf --- block.c

[PULL 31/50] include/block/blockjob.h: global state API

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito blockjob functions run always under the BQL lock. Signed-off-by: Emanuele Giuseppe Esposito Message-Id: <20220303151616.325444-19-eespo...@redhat.com> Signed-off-by: Kevin Wolf --- include/block/blockjob.h | 29 ++--- 1 file changed,

Re: [PATCH 12/12] tests/qemu-iotests: validate NBD TLS with UNIX sockets and PSK

2022-03-04 Thread Eric Blake
On Thu, Mar 03, 2022 at 04:03:30PM +, Daniel P. Berrangé wrote: > This validates that connections to an NBD server running on a UNIX > socket can use TLS with pre-shared keys (PSK). > > Signed-off-by: Daniel P. Berrangé > --- > tests/qemu-iotests/233| 28

Re: [PATCH 11/12] tests/qemu-iotests: validate NBD TLS with UNIX sockets

2022-03-04 Thread Eric Blake
On Thu, Mar 03, 2022 at 04:03:29PM +, Daniel P. Berrangé wrote: > This validates that connections to an NBD server running on a UNIX > socket can use TLS, and require a TLS hostname override to pass > certificate validation. > > Signed-off-by: Daniel P. Berrangé > --- >

Re: [PATCH 10/12] tests/qemu-iotests: validate NBD TLS with hostname mismatch

2022-03-04 Thread Eric Blake
On Thu, Mar 03, 2022 at 04:03:28PM +, Daniel P. Berrangé wrote: > This validates that connections to an NBD server where the certificate > hostname does not match will fail. It further validates that using the > new 'tls-hostname' override option can solve the failure. > > Signed-off-by:

[PULL 47/50] iotests/graph-changes-while-io: New test

2022-03-04 Thread Kevin Wolf
From: Hanna Reitz Test the following scenario: 1. Some block node (null-co) attached to a user (here: NBD server) that performs I/O and keeps the node in an I/O thread 2. Repeatedly run blockdev-add/blockdev-del to add/remove an overlay to/from that node Each blockdev-add triggers

[PULL 45/50] block: Make bdrv_refresh_limits() non-recursive

2022-03-04 Thread Kevin Wolf
From: Hanna Reitz bdrv_refresh_limits() recurses down to the node's children. That does not seem necessary: When we refresh limits on some node, and then recurse down and were to change one of its children's BlockLimits, then that would mean we noticed the changed limits by pure chance. The

[PULL 44/50] job.h: assertions in the callers of JobDriver function pointers

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito Signed-off-by: Emanuele Giuseppe Esposito Message-Id: <20220303151616.325444-32-eespo...@redhat.com> Signed-off-by: Kevin Wolf --- job.c | 10 ++ 1 file changed, 10 insertions(+) diff --git a/job.c b/job.c index 54db80df66..075c6f3a20 100644 ---

[PULL 27/50] block: introduce assert_bdrv_graph_writable

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito We want to be sure that the functions that write the child and parent list of a bs are under BQL and drain. BQL prevents from concurrent writings from the GS API, while drains protect from I/O. TODO: drains are missing in some functions using this assert.

[PULL 25/50] assertions for block_int global state API

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito Signed-off-by: Emanuele Giuseppe Esposito Message-Id: <20220303151616.325444-13-eespo...@redhat.com> Signed-off-by: Kevin Wolf --- block.c | 15 +++ block/backup.c | 1 + block/block-backend.c |

[PULL 19/50] block/export/fuse.c: allow writable exports to take RESIZE permission

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito Allow writable exports to get BLK_PERM_RESIZE permission from creation, in fuse_export_create(). In this way, there is no need to give the permission in fuse_do_truncate(), which might be run in an iothread. Permissions should be set only in the main thread, so

[PULL 46/50] iotests: Allow using QMP with the QSD

2022-03-04 Thread Kevin Wolf
From: Hanna Reitz Add a parameter to optionally open a QMP connection when creating a QemuStorageDaemon instance. Signed-off-by: Hanna Reitz Message-Id: <20220216105355.30729-3-hre...@redhat.com> Reviewed-by: Eric Blake Reviewed-by: Stefan Hajnoczi Signed-off-by: Kevin Wolf ---

[PULL 39/50] block_int-common.h: assertions in the callers of BlockDriver function pointers

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito Signed-off-by: Emanuele Giuseppe Esposito Message-Id: <20220303151616.325444-27-eespo...@redhat.com> Signed-off-by: Kevin Wolf --- block.c| 17 + block/create.c | 2 ++ 2 files changed, 19 insertions(+) diff --git a/block.c b/block.c

Re: [PATCH 3/4] Replace qemu_gettimeofday() with g_get_real_time()

2022-03-04 Thread Marc-André Lureau
Hi On Fri, Mar 4, 2022 at 8:10 PM Laurent Vivier wrote: > Le 04/03/2022 à 16:27, marcandre.lur...@redhat.com a écrit : > > From: Marc-André Lureau > > > > GLib g_get_real_time() is an alternative to gettimeofday(). > > > > For semihosting, a few bits are lost on POSIX host, but this shouldn't

[PULL 38/50] block_int-common.h: split function pointers in BlockDriver

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito Similar to the header split, also the function pointers in BlockDriver can be split in I/O and global state. Signed-off-by: Emanuele Giuseppe Esposito Message-Id: <20220303151616.325444-26-eespo...@redhat.com> Signed-off-by: Kevin Wolf ---

[PULL 03/50] block: introduce bdrv_activate

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito This function is currently just a wrapper for bdrv_invalidate_cache(), but in future will contain the code of bdrv_co_invalidate_cache() that has to always be protected by BQL, and leave the rest in the I/O coroutine. Replace all bdrv_invalidate_cache()

[PULL 34/50] assertions for blockdev.h global state API

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito Signed-off-by: Emanuele Giuseppe Esposito Message-Id: <20220303151616.325444-22-eespo...@redhat.com> Signed-off-by: Kevin Wolf --- block/block-backend.c | 3 +++ blockdev.c| 16 2 files changed, 19 insertions(+) diff --git

[PULL 23/50] block.c: assertions to the block layer permissions API

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito Now that we "covered" the three main cases where the permission API was being used under BQL (fuse, amend and invalidate_cache), we can safely assert for the permission functions implemented in block.c Signed-off-by: Emanuele Giuseppe Esposito Message-Id:

[PULL 24/50] include/block/block_int: split header into I/O and global state API

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito Similarly to the previous patch, split block_int.h in block_int-io.h and block_int-global-state.h block_int-common.h contains the structures shared between the two headers, and the functions that can't be categorized as I/O or global state. Assertions are added

[PULL 16/50] include/block/block: split header into I/O and global state API

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito block.h currently contains a mix of functions: some of them run under the BQL and modify the block layer graph, others are instead thread-safe and perform I/O in iothreads. Some others can only be called by either the main loop or the iothread running the

[PULL 07/50] util/async: replace __thread with QEMU TLS macros

2022-03-04 Thread Kevin Wolf
From: Stefan Hajnoczi QEMU TLS macros must be used to make TLS variables safe with coroutines. Signed-off-by: Stefan Hajnoczi Message-Id: <20220222140150.27240-3-stefa...@redhat.com> Signed-off-by: Kevin Wolf --- util/async.c | 12 +++- 1 file changed, 7 insertions(+), 5 deletions(-)

[PULL 29/50] GS and IO CODE macros for blockjob_int.h

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito Signed-off-by: Emanuele Giuseppe Esposito Message-Id: <20220303151616.325444-17-eespo...@redhat.com> Signed-off-by: Kevin Wolf --- blockjob.c | 5 + 1 file changed, 5 insertions(+) diff --git a/blockjob.c b/blockjob.c index 10815a89fe..d79a52d204 100644

[PULL 28/50] include/block/blockjob_int.h: split header into I/O and GS API

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito Since the I/O functions are not many, keep a single file. Also split the function pointers in BlockJobDriver. Signed-off-by: Emanuele Giuseppe Esposito Reviewed-by: Stefan Hajnoczi Message-Id: <20220303151616.325444-16-eespo...@redhat.com> Signed-off-by: Kevin

[PULL 17/50] assertions for block global state API

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito All the global state (GS) API functions will check that qemu_in_main_thread() returns true. If not, it means that the safety of BQL cannot be guaranteed, and they need to be moved to I/O. Signed-off-by: Emanuele Giuseppe Esposito Message-Id:

[PULL 26/50] IO_CODE and IO_OR_GS_CODE for block_int I/O API

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito Mark all I/O functions with IO_CODE, and all "I/O OR GS" with IO_OR_GS_CODE. Signed-off-by: Emanuele Giuseppe Esposito Message-Id: <20220303151616.325444-14-eespo...@redhat.com> Signed-off-by: Kevin Wolf --- include/block/block_int-io.h | 6 ++ block.c

[PULL 22/50] IO_CODE and IO_OR_GS_CODE for block-backend I/O API

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito Mark all I/O functions with IO_CODE, and all "I/O OR GS" with IO_OR_GS_CODE. Signed-off-by: Emanuele Giuseppe Esposito Message-Id: <20220303151616.325444-10-eespo...@redhat.com> Signed-off-by: Kevin Wolf --- include/sysemu/block-backend-io.h | 2 ++

[PULL 20/50] include/sysemu/block-backend: split header into I/O and global state (GS) API

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito Similarly to the previous patches, split block-backend.h in block-backend-io.h and block-backend-global-state.h In addition, remove "block/block.h" include as it seems it is not necessary anymore, together with "qemu/iov.h" block-backend-common.h contains the

[PULL 18/50] IO_CODE and IO_OR_GS_CODE for block I/O API

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito Mark all I/O functions with IO_CODE, and all "I/O OR GS" with IO_OR_GS_CODE. Signed-off-by: Emanuele Giuseppe Esposito Message-Id: <20220303151616.325444-6-eespo...@redhat.com> Signed-off-by: Kevin Wolf --- include/block/block-io.h | 1 + block.c

[PULL 21/50] block/block-backend.c: assertions for block-backend

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito All the global state (GS) API functions will check that qemu_in_main_thread() returns true. If not, it means that the safety of BQL cannot be guaranteed, and they need to be moved to I/O. Signed-off-by: Emanuele Giuseppe Esposito Message-Id:

[PULL 15/50] main loop: macros to mark GS and I/O functions

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito Righ now, IO_CODE and IO_OR_GS_CODE are nop, as there isn't really a way to check that a function is only called in I/O. On the other side, we can use qemu_in_main_thread() to check if we are in the main loop. The usage of macros makes easy to extend them in the

[PULL 11/50] qsd: Add pre-init argument parsing pass

2022-03-04 Thread Kevin Wolf
From: Hanna Reitz In contrast to qemu-nbd (where it is called --fork) and the system emulator, QSD does not have a --daemonize switch yet. Just like them, QSD allows setting up block devices and exports on the command line. When doing so, it is often necessary for whoever invoked the QSD to

[PULL 10/50] os-posix: Add os_set_daemonize()

2022-03-04 Thread Kevin Wolf
From: Hanna Reitz The daemonizing functions in os-posix (os_daemonize() and os_setup_post()) only daemonize the process if the static `daemonize` variable is set. Right now, it can only be set by os_parse_cmd_args(). In order to use os_daemonize() and os_setup_post() from the storage daemon to

Re: [PULL 0/6] hw/nvme updates

2022-03-04 Thread Peter Maydell
On Fri, 4 Mar 2022 at 06:24, Klaus Jensen wrote: > > From: Klaus Jensen > > Hi Peter, > > Last round of hw/nvme updates for v7.0. > > The following changes since commit 64ada298b98a51eb2512607f6e6180cb330c47b1: > > Merge remote-tracking branch 'remotes/legoater/tags/pull-ppc-20220302' into >

[PULL 02/50] crypto: distinguish between main loop and I/O in block_crypto_amend_options_generic_luks

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito block_crypto_amend_options_generic_luks uses the block layer permission API, therefore it should be called with the BQL held. However, the same function is being called by two BlockDriver callbacks: bdrv_amend_options (under BQL) and bdrv_co_amend (I/O). The

[PULL 14/50] main-loop.h: introduce qemu_in_main_thread()

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito When invoked from the main loop, this function is the same as qemu_mutex_iothread_locked, and returns true if the BQL is held. When invoked from iothreads or tests, it returns true only if the current AioContext is the Main Loop. This essentially just extends

[PULL 06/50] tls: add macros for coroutine-safe TLS variables

2022-03-04 Thread Kevin Wolf
From: Stefan Hajnoczi Compiler optimizations can cache TLS values across coroutine yield points, resulting in stale values from the previous thread when a coroutine is re-entered by a new thread. Serge Guelton developed an __attribute__((noinline)) wrapper and tested it with clang and gcc. I

[PULL 08/50] rcu: use coroutine TLS macros

2022-03-04 Thread Kevin Wolf
From: Stefan Hajnoczi RCU may be used from coroutines. Standard __thread variables cannot be used by coroutines. Use the coroutine TLS macros instead. Signed-off-by: Stefan Hajnoczi Message-Id: <20220222140150.27240-4-stefa...@redhat.com> Signed-off-by: Kevin Wolf --- include/qemu/rcu.h

[PULL 00/50] Block layer patches

2022-03-04 Thread Kevin Wolf
The following changes since commit 6629bf78aac7e53f83fd0bcbdbe322e2302dfd1f: Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20220302' into staging (2022-03-03 14:46:48 +) are available in the Git repository at: https://gitlab.com/kmwolf/qemu.git tags/for-upstream

[PULL 13/50] iotests/185: Add post-READY quit tests

2022-03-04 Thread Kevin Wolf
From: Hanna Reitz 185 tests quitting qemu while a block job is active. It does not specifically test quitting qemu while a mirror or active commit job is in its READY phase. Add two test cases for this, where we respectively mirror or commit to an external QSD instance, which provides a

[PULL 12/50] qsd: Add --daemonize

2022-03-04 Thread Kevin Wolf
From: Hanna Reitz To implement this, we reuse the existing daemonizing functions from the system emulator, which mainly do the following: - Fork off a child process, and set up a pipe between parent and child - The parent process waits until the child sends a status byte over the pipe (0 means

[PULL 04/50] block: rename bdrv_invalidate_cache_all, blk_invalidate_cache and test_sync_op_invalidate_cache

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito Following the bdrv_activate renaming, change also the name of the respective callers. bdrv_invalidate_cache_all -> bdrv_activate_all blk_invalidate_cache -> blk_activate test_sync_op_invalidate_cache -> test_sync_op_activate No functional change intended.

[PULL 09/50] cpus: use coroutine TLS macros for iothread_locked

2022-03-04 Thread Kevin Wolf
From: Stefan Hajnoczi qemu_mutex_iothread_locked() may be used from coroutines. Standard __thread variables cannot be used by coroutines. Use the coroutine TLS macros instead. Signed-off-by: Stefan Hajnoczi Message-Id: <20220222140150.27240-5-stefa...@redhat.com> Signed-off-by: Kevin Wolf ---

[PULL 05/50] block: move BQL logic of bdrv_co_invalidate_cache in bdrv_activate

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito Split bdrv_co_invalidate cache in two: the Global State (under BQL) code that takes care of permissions and running GS callbacks, and leave only the I/O code (->bdrv_co_invalidate_cache) running in the I/O coroutine. The only side effect is that

[PULL 01/50] crypto: perform permission checks under BQL

2022-03-04 Thread Kevin Wolf
From: Emanuele Giuseppe Esposito Move the permission API calls into driver-specific callbacks that always run under BQL. In this case, bdrv_crypto_luks needs to perform permission checks before and after qcrypto_block_amend_options(). The problem is that the caller,

Re: [PATCH 09/12] tests/qemu-iotests: convert NBD TLS test to use standard filters

2022-03-04 Thread Eric Blake
On Thu, Mar 03, 2022 at 04:03:27PM +, Daniel P. Berrangé wrote: > Using standard filters is more future proof than rolling our own. > > Signed-off-by: Daniel P. Berrangé > --- > tests/qemu-iotests/233 | 29 - > tests/qemu-iotests/233.out | 9 - > 2

Re: [PATCH 08/12] tests/qemu-iotests: introduce filter for qemu-nbd export list

2022-03-04 Thread Eric Blake
On Thu, Mar 03, 2022 at 04:03:26PM +, Daniel P. Berrangé wrote: > Introduce a filter for the output of qemu-nbd export list so it can be > reused in multiple tests. > > The filter is a bit more permissive that what test 241 currently uses, > as its allows printing of the export count, along

Re: [PATCH 07/12] tests/qemu-iotests: expand _filter_nbd rules

2022-03-04 Thread Eric Blake
On Thu, Mar 03, 2022 at 04:03:25PM +, Daniel P. Berrangé wrote: > Some tests will want to use 'localhost' instead of '127.0.0.1', and > some will use the image options syntax rather than the classic URI > syntax. > > Signed-off-by: Daniel P. Berrangé > --- > tests/qemu-iotests/common.filter

Re: [PATCH 06/12] tests/qemu-iotests: add QEMU_IOTESTS_REGEN=1 to update reference file

2022-03-04 Thread Eric Blake
On Thu, Mar 03, 2022 at 04:03:24PM +, Daniel P. Berrangé wrote: > When developing an I/O test it is typical to add some logic to the > test script, run it to view the output diff, and then apply the > output diff to the reference file. This can be drastically simplified > by letting the test

  1   2   >