Re: [Libguestfs] [nbdkit PATCH] data, memory: Optimize .zero > PAGE_SIZE

2019-08-15 Thread Eric Blake
On 8/15/19 5:25 AM, Richard W.M. Jones wrote: > On Wed, Aug 14, 2019 at 09:10:15PM -0500, Eric Blake wrote: >> When sparse_array_zero() is used for a range larger than a page, >> there's no need to waste time in memset() or is_zero() - we already >> know the page will be free()d. >> >>

Re: [Libguestfs] [PATCH libnbd v2 10/10] generator: Check requirements for BytesPersistIn/Out and completion callbacks.

2019-08-15 Thread Eric Blake
On 8/15/19 7:07 AM, Richard W.M. Jones wrote: > On Thu, Aug 15, 2019 at 07:01:49AM -0500, Eric Blake wrote: >> We still had another potential API change to squeeze into 0.9.8: >> Right now, the API is overloading 'command': >> nbd_connect_command/nbd_kill_command >> vs. >>

[Libguestfs] [nbdkit PATCH] nbd: Another libnbd version bump

2019-08-15 Thread Eric Blake
The 0.9.8 release breaks API, requiring a number of changes: - Use symbolic constants instead of magic numbers/open-coded strings (well, the string for "base:allocation" was present before this libnbd bump) - Change callbacks to drop the valid_flag parameter - Add _is to nbd_read_only call -

Re: [Libguestfs] [PATCH libnbd v2 04/10] lib: Permit .callback = NULL, .free != NULL.

2019-08-15 Thread Eric Blake
On 8/15/19 7:02 AM, Richard W.M. Jones wrote: >>> >>> /* Test if a callback is "null" or not, and set it to null. */ >>> -#define CALLBACK_IS_NULL(cb) ((cb).callback == NULL) >>> +#define CALLBACK_IS_NULL(cb) ((cb).callback == NULL && (cb).free == >>> NULL) >> >> Semantic change. In

Re: [Libguestfs] [libnbd PATCH] lib: Consolidate free callbacks to just happen at retire time

2019-08-15 Thread Eric Blake
On 8/15/19 5:22 AM, Richard W.M. Jones wrote: > On Wed, Aug 14, 2019 at 05:38:31PM -0500, Eric Blake wrote: >> When we introduced valid_flags, there was an incentive to do as few >> callbacks as possible, favoring cb(VALID|FREE) calls over the sequence >> cb(VALID);cb(FREE). To make it work, we

Re: [Libguestfs] [PATCH libnbd v2 10/10] generator: Check requirements for BytesPersistIn/Out and completion callbacks.

2019-08-15 Thread Richard W.M. Jones
On Thu, Aug 15, 2019 at 07:01:49AM -0500, Eric Blake wrote: > We still had another potential API change to squeeze into 0.9.8: > Right now, the API is overloading 'command': > nbd_connect_command/nbd_kill_command > vs. > nbd_aio_command_completed/nbd_aio_peek_command_completed > > Keeping

Re: [Libguestfs] [PATCH libnbd] api: Rename nbd_kill_command -> nbd_kill_subprocess.

2019-08-15 Thread Eric Blake
On 8/15/19 7:50 AM, Richard W.M. Jones wrote: > This is a simple renaming to avoid confusion over "command" when used > to mean an asynchronous request to the server and "command" meaning > the subprocess forked previously by libnbd. > --- > generator/generator | 4 ++-- > lib/handle.c

[Libguestfs] [libnbd PATCH] docs: Ensure .3 files get built

2019-08-15 Thread Eric Blake
Running 'make clean && make' loses all the .3 man pages produced from generated .pod files. Why? When Makefile.inc does not exist (on the first run), the value of $(api_built) is updated prior to re-reading the Makefile, so when the 'all-am: Makefile $(MANS)' dependency from automake is

Re: [Libguestfs] [PATCH libnbd v2 04/10] lib: Permit .callback = NULL, .free != NULL.

2019-08-15 Thread Richard W.M. Jones
On Thu, Aug 15, 2019 at 06:57:01AM -0500, Eric Blake wrote: > On 8/15/19 4:56 AM, Richard W.M. Jones wrote: > > Previously the .free function of a callback was not called if the > > .callback field was NULL, because the callback as a whole would be > > considered to be "null". > > > > This change

Re: [Libguestfs] [PATCH libnbd v2 10/10] generator: Check requirements for BytesPersistIn/Out and completion callbacks.

2019-08-15 Thread Eric Blake
On 8/15/19 4:56 AM, Richard W.M. Jones wrote: > --- > generator/generator | 25 + > 1 file changed, 25 insertions(+) > > diff --git a/generator/generator b/generator/generator > index 1252bdb..13cb0b9 100755 > --- a/generator/generator > +++ b/generator/generator > @@

[Libguestfs] [PATCH libnbd] api: Rename nbd_kill_command -> nbd_kill_subprocess.

2019-08-15 Thread Richard W.M. Jones
This is a simple renaming to avoid confusion over "command" when used to mean an asynchronous request to the server and "command" meaning the subprocess forked previously by libnbd. --- generator/generator | 4 ++-- lib/handle.c | 2 +- tests/closure-lifetimes.c | 4 ++--

[Libguestfs] [PATCH libnbd] docs: Change docs/Makefile.inc back to a regular include, readd to git.

2019-08-15 Thread Richard W.M. Jones
‘make clean && make’ was not rebuilding the docs/*.3 files. The reason is obscure: - docs/Makefile has rules: MANS = $(man_MANS) all: all-am all-am: Makefile $(MANS) - sinclude docs/Makefile.inc happened long after MANS is defined, so MANS held the earlier version of $(man_MANS)

Re: [Libguestfs] [PATCH libnbd v2 02/10] lib: Add macros to check if a callback is "null" or not, and set it to null.

2019-08-15 Thread Eric Blake
On 8/15/19 4:56 AM, Richard W.M. Jones wrote: > We have defined the concept of a "null callback" meaning one where the > .callback field = NULL. The first two new macros just test this > property, and the third one sets a callback to null. > > This change is neutral refactoring. > --- >

Re: [Libguestfs] [PATCH libnbd v2 04/10] lib: Permit .callback = NULL, .free != NULL.

2019-08-15 Thread Eric Blake
On 8/15/19 4:56 AM, Richard W.M. Jones wrote: > Previously the .free function of a callback was not called if the > .callback field was NULL, because the callback as a whole would be > considered to be "null". > > This change allows you to register callbacks where the .callback field > is NULL,

Re: [Libguestfs] [PATCH libnbd] docs: Change docs/Makefile.inc back to a regular include, readd to git.

2019-08-15 Thread Eric Blake
On 8/15/19 7:50 AM, Richard W.M. Jones wrote: > ‘make clean && make’ was not rebuilding the docs/*.3 files. The > reason is obscure: > > - docs/Makefile has rules: > >MANS = $(man_MANS) >all: all-am >all-am: Makefile $(MANS) > > - sinclude docs/Makefile.inc happened long after

[Libguestfs] [PATCH libnbd v2 10/10] generator: Check requirements for BytesPersistIn/Out and completion callbacks.

2019-08-15 Thread Richard W.M. Jones
--- generator/generator | 25 + 1 file changed, 25 insertions(+) diff --git a/generator/generator b/generator/generator index 1252bdb..13cb0b9 100755 --- a/generator/generator +++ b/generator/generator @@ -3121,6 +3121,31 @@ let () = failwithf "%s: first_version

[Libguestfs] [PATCH libnbd v2 09/10] python: Add test for doing asynch copy from one handle to another.

2019-08-15 Thread Richard W.M. Jones
--- python/t/590-aio-copy.py | 122 +++ 1 file changed, 122 insertions(+) diff --git a/python/t/590-aio-copy.py b/python/t/590-aio-copy.py new file mode 100644 index 000..129dde1 --- /dev/null +++ b/python/t/590-aio-copy.py @@ -0,0 +1,122 @@ +# libnbd

[Libguestfs] [PATCH libnbd v2 01/10] generator: Remove cases where we call the free callback early.

2019-08-15 Thread Richard W.M. Jones
Earlier on we had the ‘valid_flag’. It was possible to set this to VALID|FREE avoiding an additional function call. Since we got rid of this flag in favour of two separate functions there is no possible way to avoid an extra function call, and freeing the callbacks close to the point of last use

[Libguestfs] [PATCH libnbd v2 00/10] Callbacks and OCaml and Python persistent buffers.

2019-08-15 Thread Richard W.M. Jones
This is a combination of these two earlier series: https://www.redhat.com/archives/libguestfs/2019-August/msg00235.html https://www.redhat.com/archives/libguestfs/2019-August/msg00240.html plus changes to allow .callback = NULL / .free != NULL, and to reduce the complexity of freeing callbacks.

[Libguestfs] [PATCH libnbd v2 04/10] lib: Permit .callback = NULL, .free != NULL.

2019-08-15 Thread Richard W.M. Jones
Previously the .free function of a callback was not called if the .callback field was NULL, because the callback as a whole would be considered to be "null". This change allows you to register callbacks where the .callback field is NULL, but the .free field is != NULL, meaning that the callback

[Libguestfs] [PATCH libnbd v2 06/10] ocaml: Remove NBD.Buffer.free function, use the completion callback instead.

2019-08-15 Thread Richard W.M. Jones
By using the completion free callback function we don't need to manually free the buffer. This reverts part of commit fef1c281a65d061127bf178e5f8cfca0a2475c56. --- generator/generator | 49 ocaml/buffer.c | 22 --

Re: [Libguestfs] [libnbd PATCH] lib: Consolidate free callbacks to just happen at retire time

2019-08-15 Thread Richard W.M. Jones
On Wed, Aug 14, 2019 at 05:38:31PM -0500, Eric Blake wrote: > When we introduced valid_flags, there was an incentive to do as few > callbacks as possible, favoring cb(VALID|FREE) calls over the sequence > cb(VALID);cb(FREE). To make it work, we set .callback=NULL after an > early free, so that

[Libguestfs] [PATCH libnbd v2 03/10] lib: Remove unnecessary type punning in FREE_CALLBACK macro.

2019-08-15 Thread Richard W.M. Jones
Why did I need to cast cb to a particular type? I don't know. Fixes commit 9c8fccdf382c2c8483b557acc6b5d41a4e193990. --- lib/internal.h | 12 +++- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/lib/internal.h b/lib/internal.h index a18d581..1344d98 100644 ---

[Libguestfs] [PATCH libnbd v2 02/10] lib: Add macros to check if a callback is "null" or not, and set it to null.

2019-08-15 Thread Richard W.M. Jones
We have defined the concept of a "null callback" meaning one where the .callback field = NULL. The first two new macros just test this property, and the third one sets a callback to null. This change is neutral refactoring. --- generator/generator | 4 ++--

[Libguestfs] [PATCH libnbd v2 08/10] python: Hold a refcount to persistent AIO buffer until command completion.

2019-08-15 Thread Richard W.M. Jones
--- generator/generator | 22 -- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/generator/generator b/generator/generator index 8c055a9..1252bdb 100755 --- a/generator/generator +++ b/generator/generator @@ -4262,6 +4262,20 @@ let print_python_binding name {

[Libguestfs] [PATCH libnbd v2 07/10] python: Refactor user_data into a struct.

2019-08-15 Thread Richard W.M. Jones
Simple refactoring to use a struct to store the function pointer passed to the callback wrapper. --- generator/generator | 69 - 1 file changed, 50 insertions(+), 19 deletions(-) diff --git a/generator/generator b/generator/generator index

[Libguestfs] [PATCH libnbd v2 05/10] generator: ocaml: Define a struct user_data and common free function.

2019-08-15 Thread Richard W.M. Jones
Just a simple refactoring. --- generator/generator | 71 +++-- 1 file changed, 43 insertions(+), 28 deletions(-) diff --git a/generator/generator b/generator/generator index 98c99e0..35936c6 100755 --- a/generator/generator +++ b/generator/generator @@

Re: [Libguestfs] [nbdkit PATCH] data, memory: Optimize .zero > PAGE_SIZE

2019-08-15 Thread Richard W.M. Jones
On Wed, Aug 14, 2019 at 09:10:15PM -0500, Eric Blake wrote: > When sparse_array_zero() is used for a range larger than a page, > there's no need to waste time in memset() or is_zero() - we already > know the page will be free()d. > > Signed-off-by: Eric Blake > --- > > Here's a fun one :) > >

Re: [Libguestfs] [nbdkit PATCH] nbd: Another libnbd version bump

2019-08-15 Thread Richard W.M. Jones
Oh that was an ACK whatever you decide :-) Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring

Re: [Libguestfs] [nbdkit PATCH] nbd: Another libnbd version bump

2019-08-15 Thread Richard W.M. Jones
On Thu, Aug 15, 2019 at 09:27:14AM -0500, Eric Blake wrote: > The 0.9.8 release breaks API, requiring a number of changes: > - Use symbolic constants instead of magic numbers/open-coded strings > (well, the string for "base:allocation" was present before this > libnbd bump) > - Change

[Libguestfs] ANNOUNCE: libnbd 0.9.8 - prerelease of high performance NBD client library

2019-08-15 Thread Richard W.M. Jones
I'm pleased to announce a new high performance Network Block Device (NBD) client library called libnbd. It's written in C and there are also bindings available for Python, OCaml and (soon) Rust. 0.9.8 is the third pre-release before the stable 1.0 version where we freeze the API, so feedback on

Re: [Libguestfs] [nbdkit PATCH] nbd: Another libnbd version bump

2019-08-15 Thread Eric Blake
On 8/15/19 10:37 AM, Richard W.M. Jones wrote: > On Thu, Aug 15, 2019 at 09:27:14AM -0500, Eric Blake wrote: >> The 0.9.8 release breaks API, requiring a number of changes: >> - Use symbolic constants instead of magic numbers/open-coded strings >> (well, the string for "base:allocation" was

Re: [Libguestfs] [nbdkit PATCH] ocaml: Add support for dynamic .thread_model

2019-08-15 Thread Richard W.M. Jones
On Thu, Aug 15, 2019 at 04:52:00PM -0500, Eric Blake wrote: > We do not promise API stability for non-C languages; this is an API > break as follows: instead of calling 'NBDKit.register_plugin model > plugin' with a static model, you can now add .thread_model :(unit -> > thread_model) to plugin or

[Libguestfs] [nbdkit PATCH] ocaml: Add support for dynamic .thread_model

2019-08-15 Thread Eric Blake
We do not promise API stability for non-C languages; this is an API break as follows: instead of calling 'NBDKit.register_plugin model plugin' with a static model, you can now add .thread_model :(unit -> thread_model) to plugin or default to PARALLEL. Since all existing OCaml plugins will have