Re: [Libguestfs] [PATCH libnbd] golang: Add README.md and LICENCE files

2021-10-29 Thread Eric Blake
e (in the > +examples/ subdirectory) is distributed under a different license, see > +examples/LICENSE-FOR-EXAMPLES. Does the golang directory structure have an examples/ subdirectory, or is this wording only applicable when reading the file from the top level? -- Eric Blake, Princip

Re: [Libguestfs] [PATCH 2/9] qapi: Mark unstable QMP parts with feature 'unstable'

2021-10-29 Thread Eric Blake
'*x-iops-total': { 'type': 'int', > + 'features': [ 'unstable' ] }, This struct has been around since 381bd74 (v6.0); but was not listed as deprecated at the time. Do we still need it in 6.2, or have we gone

Re: [Libguestfs] [PATCH nbdkit v2 5/5] vddk: Implement parallel thread model

2021-10-29 Thread Eric Blake
: NBDKIT_FUA_NONE; > +} > + > +static int > +vddk_can_flush (void *handle) > +{ > + /* The Flush call was not available in VDDK < 6.0. */ > + return VixDiskLib_Flush != NULL; > +} Since we now require 6.5, can we be sure VixDiskLib_Flush is non-NULL? -- Eric Blake

Re: [Libguestfs] [PATCH v2 9/9] qapi: Extend -compat to set policy for unstable interfaces

2021-10-29 Thread Eric Blake
) > SRST > ``-compat > [deprecated-input=@var{input-policy}][,deprecated-output=@var{output-policy}]`` > @@ -3659,6 +3661,22 @@ SRST > Suppress deprecated command results and events > > Limitation: covers only syntactic aspects of QMP. > + > +``-comp

Re: [Libguestfs] [PATCH v2 5/9] qapi: Generalize struct member policy checking

2021-10-29 Thread Eric Blake
> } > -return visit_deprecated(ffv->target, name); > +return visit_policy_skip(ffv->target, name, special_features); > } > Otherwise, the rest of the logic changes for flipped sense look right. -- Eric Blake, Principal Software Engineer Red Hat, Inc.

Re: [Libguestfs] [PATCH v2 6/9] qapi: Generalize command policy checking

2021-10-29 Thread Eric Blake
(<< is higher than &); if writing it myself, I would probably have used explicit () to avoid reviewer confusion, but what you have is correct. (After grepping for ' & 1.*<<' and ' & (1.*<<', it looks like authors using explicit precedence happens

Re: [Libguestfs] [PATCH libnbd v2 1/2] golang: Changes test license to LPGL

2021-11-01 Thread Eric Blake
igned-off-by: Richard W.M. Jones > > However I'm not the only author here (by quite a lot!): > > $ git shortlog -s golang/src/libguestfs.org/libnbd/ > 13 Eric Blake > 2 Nir Soffer > 4 Richard W.M. Jones > > so could be a good idea to ask Eric spec

Re: [Libguestfs] [PATCH libnbd v2 3/8] common/utils/vector.c: Optimize vector append

2021-11-08 Thread Eric Blake
alloc) > +newalloc = reqalloc; > + > + newptr = realloc (v->ptr, newalloc * itemsize); >if (newptr == NULL) > return -1; but ENOMEM here. We probably want to guarantee it is ENOMEM on all error paths, rather than having to audit whether callers care. >

Re: [Libguestfs] [PATCH nbdkit 1/4] common/utils/test-vector.c: Add vector benchmarks

2021-11-08 Thread Eric Blake
rks, and we've not yet had success at fully porting our code base to be VPATH-friendly (bash completion and ocaml builds were the two that gave me the most grief last time I tried, but there may be others). -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Vi

Re: [Libguestfs] [PATCH nbdkit 2/4] common/urils/vector.c: Optimize vector append

2021-11-08 Thread Eric Blake
walloc < reqalloc) > +newalloc = reqalloc; > + > + newptr = realloc (v->ptr, newalloc * itemsize); Same potential for overflow as in the libnbd patches. Whatever we fix there, we'll have to port here as well. -- Eric Blake, Principal Software Engineer Red Hat, Inc.

Re: [Libguestfs] [PATCH libnbd 1/2] golang: Create distribution for a proxy server

2021-11-08 Thread Eric Blake
list that has too many (potentially duplicate) entries? > + > +cp go.mod $v_dir/$version.mod > +mv $version.zip $v_dir > + > +# Create tarball to upload and extract on the webserver. It should be > +# extracted in the directory pointed by the "go-import" meta tag. > +ta

[Libguestfs] [libnbd PATCH] common/utils/vector: Better overflow handling

2021-11-08 Thread Eric Blake
Check newcap * itemsize for overflow prior to calling realloc, so that we don't accidentally truncate an existing array. Set errno to ENOMEM on all failure paths, rather than leaving it indeterminate on overflow. The patch works by assuming a prerequisite that v->cap*itemsize is always smaller th

Re: [Libguestfs] [libnbd PATCH] common/utils/vector: Better overflow handling

2021-11-09 Thread Eric Blake
before, we can keep the old check > comparing caps. > > I can post a patch with this changes if you like. I hadn't pushed just yet, so I incorporated your suggestion, and the result is now commit 884052e054 in libnbd, and 0c342208d5 in nbdkit. -- Eric Blake, Principal Software En

Re: [Libguestfs] [PATCH nbdkit 1/2] common/utils/vector: Add comments to generic_vector_reserve

2021-11-09 Thread Eric Blake
letions(-) ACK. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] [PATCH nbdkit 0/2] common: Add checked-overflow macros

2021-11-09 Thread Eric Blake
fering with compilation on older distros with older compilers? I guess CI testing can help find that out quickly. > > I verified by disassembly that "jo" (jump overflow) / "jno" is used > where it was not used previously, by both compilers. Nice, and an argument in

Re: [Libguestfs] [PATCH nbdkit 2/2] common: Add checked-overflow macros and use for safe vector extension

2021-11-09 Thread Eric Blake
p /= 2; > if (ADD_SIZE_T_OVERFLOW (v->cap, extracap, &newcap)) > goto fallback; > > > + if (MUL_SIZE_T_OVERFLOW (newcap, itemsize, &newbytes)) > > +goto fallback; > > if (newbytes < reqbytes) { > > + fallback: > > Jumping i

Re: [Libguestfs] [PATCH libnbd] common/include/checked-overflow.h: Simplify

2021-11-10 Thread Eric Blake
gt; > Signed-off-by: Nir Soffer > --- > common/include/checked-overflow.h | 18 -- > common/utils/vector.c | 10 +- > 2 files changed, 9 insertions(+), 19 deletions(-) ACK. -- Eric Blake, Principal Software Engineer Red Hat, Inc.

Re: [Libguestfs] [PATCH nbdkit v2 2/2] ocaml: Simplify NBDKit.set_error

2021-11-17 Thread Eric Blake
t_set_error" > [@@noalloc] Yep, that's simpler. > +++ b/tests/test-cc-ocaml.sh > @@ -58,6 +58,6 @@ cleanup_fn rm -f $out > rm -f $out > > nbdkit -U - cc $script a=1 b=2 c=3 \ > - CC="$OCAMLOPT" CFLAGS="-output-obj -runtime-variant _pic -I > $

Re: [Libguestfs] [PATCH nbdkit 2/2] common/include/checked-overflow.h: Provide fallback

2021-11-18 Thread Eric Blake
t; ADD_UINT64_T_OVERFLOW, MUL_UINT64_T_OVERFLOW, ADD_SIZE_T_OVERFLOW, > MUL_SIZE_T_OVERFLOW function-like macros with actual functions. Just let > the compiler inline whatever it likes.) > > Comments? As long as we have good unit tests for the particular use cases we care about dete

Re: [Libguestfs] [PATCH nbdkit 2/2] common/include/checked-overflow.h: Provide fallback

2021-11-18 Thread Eric Blake
a function signature like: bool check_op_overflow (uint64_t a, uint64_t b, uint64_t max, uint64_t *r) > > - deduce the actual result limit from ((typeof (*r))-1), > > - assign the low-order bits in any case -- if either the uintmax_t > operation overflows or the actual result

Re: [Libguestfs] [PATCH nbdkit 2/2] common/include/checked-overflow.h: Provide fallback

2021-11-19 Thread Eric Blake
; whitespace, but I guess whatever works. However you definitely don't > need the parens around the entire return value, C doesn't require > that. I use extra () when using &, |, << and >> (as those are less frequent operators),

Re: [Libguestfs] [PATCH nbdkit 2/2] common/include/checked-overflow.h: Provide fallback

2021-11-23 Thread Eric Blake
#endif so that the testsuite can then #define NBDKIT_FALLBACK for testing the fallback code even on new enough compilers. I'm not sure how best to namespace the witness macro for forcing a fallback if the code is copied between nbdkit and libnbd. -- Eric Blake, Principal Software Engine

[Libguestfs] [libnbd PATCH] golang: Simplify nbd_block_status callback array

2021-11-24 Thread Eric Blake
Instead of copying from a C uint32_t[] to a golang []uint32, we can exploit the fact that their underlying memory has the same representation. An unsafe cast to more memory than necessary exposes all the more that Go then needs to give a well-bounded slice, with no copying necessary. https://newb

Re: [Libguestfs] [libnbd PATCH] golang: Simplify nbd_block_status callback array

2021-11-29 Thread Eric Blake
On Thu, Nov 25, 2021 at 10:50:03AM +0200, Nir Soffer wrote: > On Wed, Nov 24, 2021 at 10:03 PM Eric Blake wrote: > > > > Instead of copying from a C uint32_t[] to a golang []uint32, we can > > exploit the fact that their underlying memory has the same > > representati

[Libguestfs] [libnbd PATCH 1/2] api: Avoid memory leak on certain strict_mode failures

2021-11-29 Thread Eric Blake
Commit e57681fa ("generator: Free closures on failure", v1.5.2) makes sure that once we copy a callback out of the caller's variable, then we ensure it is cleaned up on all error paths. But I was developing two patch series in parallel, and due to botched rebasing on my part, shortly after, commit

[Libguestfs] [libnbd PATCH 0/2] Memory leak fixes

2021-11-29 Thread Eric Blake
1 is ready to go. Patch 2 helps, but is incomplete; I'm posting it now because it's my bedtime. Eric Blake (2): api: Avoid memory leak on certain strict_mode failures python: Fix memory leak on callback error generator/Python.ml | 4 ++-- lib/rw.c| 4 ++-- tests/err

[Libguestfs] [libnbd PATCH 2/2] python: Fix memory leak on callback error

2021-11-29 Thread Eric Blake
Commit 89af010d ("python: Fix more memory leaks", v1.5.2) tried to patch some python memory leaks on callback error paths. But it missed an obvious one: we are mistakenly calling Py_INCREF on the result of Py_BuildValue, which results in an object that is over-referenced and thus never gets freed.

Re: [Libguestfs] [libnbd PATCH 1/2] api: Avoid memory leak on certain strict_mode failures

2021-11-30 Thread Eric Blake
On Tue, Nov 30, 2021 at 01:08:24PM +0100, Laszlo Ersek wrote: > On 11/30/21 04:15, Eric Blake wrote: > > Commit e57681fa ("generator: Free closures on failure", v1.5.2) makes > > sure that once we copy a callback out of the caller's variable, then > > we ensure

[Libguestfs] [libnbd PATCH v2] python: Fix more callback memory leaks

2021-11-30 Thread Eric Blake
Commit 89af010d ("python: Fix more memory leaks", v1.5.2) tried to patch some python memory leaks on callback error paths. But it missed an obvious one: we are mistakenly calling Py_INCREF on the result of Py_BuildValue, which results in an object that is over-referenced and thus never gets freed,

Re: [Libguestfs] [PATCH libnbd] podwrapper.pl.in: Use short commit date

2021-11-30 Thread Eric Blake
=%cs outputs '-mm-dd\n', and the \n hurts. Reading 'git format --help', that is short for --format=tformat:%cs, but we need --format=format:%cs (that is, we do not want the trailing newline). I've reinstated your patch as bdcf0765 on nbdkit, and will shortly fo

Re: [Libguestfs] [libnbd PATCH v2] python: Fix more callback memory leaks

2021-11-30 Thread Eric Blake
, then I'll cut upstream 1.11.4 and 1.10.2. Once I've got that far, I'll ping you on IRC, and can let you build for RHEL 9 while I kick off Fedora 35 and rawhide builds. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-91

Re: [Libguestfs] [nbdkit PATCH v3 3/3] common/include/checked-overflow: provide fallback

2021-11-30 Thread Eric Blake
> > I think Eric said he would do it. > > OK, thanks! > Laszlo Done now; nbdkit commits 48c78415..bf6d609d -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org ___ Li

[Libguestfs] [libnbd PATCH] docs: Mention explicit and inherent limits on lengths

2021-12-02 Thread Eric Blake
Prior to implementing an NBD extension that will allow some commands to accept a 64-bit count, it is worth documenting places where there are 32-bit inherent limits, as well as where libnbd itself enforces a smaller arbitrary limit, in spite of the API having a size_t or uint64_t parameter. --- g

Re: [Libguestfs] [PATCH libnbd] golang: make-dist.sh: Use strict ISO 8601 format

2021-12-03 Thread Eric Blake
ot; > + \"Time\": \"$(git show -s --format=%cI)\" Interesting that this preserves the timezone information of the committer. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org ___

[Libguestfs] [libnbd PATCH] maint: Suggest better diff output for API.ml

2021-12-03 Thread Eric Blake
Git diff is able to customize the regex used to locate "function headers", or the text to output on @@ lines of a patch to make it easier to determine which portion of a file the patch touches. This is done by coupling .gitattributes contents with the user running the right 'git config' command.

Re: [Libguestfs] [libnbd PATCH] maint: Suggest better diff output for API.ml

2021-12-03 Thread Eric Blake
On Fri, Dec 03, 2021 at 08:35:18AM -0600, Eric Blake wrote: > Git diff is able to customize the regex used to locate "function > headers", or the text to output on @@ lines of a patch to make it > easier to determine which portion of a file the patch touches. This &g

Re: [Libguestfs] [libnbd PATCH] docs: Mention explicit and inherent limits on lengths

2021-12-03 Thread Eric Blake
he > following hunk headers: > > @@ -1822,6 +1822,12 @@ "pread", { > @@ -1898,6 +1904,12 @@ "pread_structured", { > @@ -1924,6 +1936,12 @@ "pwrite", { > @@ -2007,6 +2025,12 @@ "trim", { > @@ -2032,6 +2056,12

Re: [Libguestfs] [PATCH libnbd 0/3] generator: Reset line directive after included code in lib/states.c

2021-12-03 Thread Eric Blake
ntf("%d\n", i); (gdb) 6 15 return 0; (gdb) c Continuing. [Inferior 1 (process 3800760) exited normally] (gdb) q -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] [PATCH libnbd 0/3] generator: Reset line directive after included code in lib/states.c

2021-12-03 Thread Eric Blake
th that change, my gdb session single-stepping through the state machine lines up with the code being executed. ACK series, and thanks for tackling this! -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org __

Re: [Libguestfs] [libnbd PATCH] maint: Suggest better diff output for API.ml

2021-12-03 Thread Eric Blake
On Fri, Dec 03, 2021 at 07:08:26PM +, Richard W.M. Jones wrote: > On Fri, Dec 03, 2021 at 08:44:03AM -0600, Eric Blake wrote: > > On Fri, Dec 03, 2021 at 08:35:18AM -0600, Eric Blake wrote: > > > Git diff is able to customize the regex used to locate "function > &g

[Libguestfs] RFC for NBD protocol extension: extended headers

2021-12-03 Thread Eric Blake
may want to also make it easier to let servers advertise an actual maximum size they are willing to accept for the commands in question (for example, a server may be happy with a full 64-bit block status, but still want to limit non-fast zero and cache to a smaller limit to avoid denial of service).

[Libguestfs] [PATCH 05/14] nbd/server: Prepare for alternate-size headers

2021-12-03 Thread Eric Blake
the number of extents we are willing to report in NBD_CMD_BLOCK_STATUS). Where 64-bit fields really matter in the extension is in a later patch adding 64-bit support into a counterpart for REPLY_TYPE_BLOCK_STATUS. Signed-off-by: Eric Blake --- include/block/nbd.h | 10 +++ nbd/server.c

[Libguestfs] [PATCH 06/14] nbd: Prepare for 64-bit requests

2021-12-03 Thread Eric Blake
headers with the server, allowing the nbd driver to pass larger requests along where possible; although in this patch it always remains false for no semantic change yet. Signed-off-by: Eric Blake --- include/block/nbd.h | 19 +++ nbd/nbd-internal.h | 3 +-- block/nbd.c | 35

[Libguestfs] [PATCH 13/14] nbd/client: Request extended headers during negotiation

2021-12-03 Thread Eric Blake
make a difference to the information collected during 'qemu-nbd --list', but probing for it gives us one more piece of information in that output. Update the iotests affected by the new line of output. Signed-off-by: Eric Blake --- nbd/client-connection.c |

[Libguestfs] [PATCH 04/14] nbd/client: Add safety check on chunk payload length

2021-12-03 Thread Eric Blake
length, we should drop the connection right then rather than assuming the layer on top will be careful. This becomes more important when we permit 64-bit lengths which are even more likely to have the potential for attempted denial of service abuse. Signed-off-by: Eric Blake --- nbd/client.c

[Libguestfs] [PATCH 10/14] nbd/client: Initial support for extended headers

2021-12-03 Thread Eric Blake
insist that the server is compliant. The only caller to nbd_receive_reply() always passed NULL for errp; since we are changing the signature anyways, I decided to sink the decision to ignore errors one layer lower. Signed-off-by: Eric Blake --- include/block/nbd.h | 2 +- block/nbd.c

[Libguestfs] [PATCH 03/14] qemu-io: Allow larger write zeroes under no fallback

2021-12-03 Thread Eric Blake
zero can be done quickly (such as when it is already known that the device started with zero contents); in those cases, artificially capping things at 2G in qemu-io itself doesn't help us. Signed-off-by: Eric Blake --- qemu-io-cmds.c | 9 +++-- 1 file changed, 3 insertions(+), 6 dele

[Libguestfs] [libnbd PATCH 01/13] golang: Simplify nbd_block_status callback array copy

2021-12-03 Thread Eric Blake
In the block status callback glue code, we need to copy a C uint32_t[] into a golang []uint32. The copy is necessary since the lifetime of the C array is not guaranteed to outlive whatever the Go callback may have done with what it was handed; copying ensures that the user's Go code doesn't have t

[Libguestfs] [PATCH 00/14] qemu patches for NBD_OPT_EXTENDED_HEADERS

2021-12-03 Thread Eric Blake
Available at https://repo.or.cz/qemu/ericb.git/shortlog/refs/tags/exthdr-v1 Patch 14 is optional; I'm including it now because I tested with it, but I'm also okay with dropping it based on RFC discussion. Eric Blake (14): nbd/server: Minor cleanups qemu-io: Utilize 64-bit status

[Libguestfs] [PATCH 02/14] qemu-io: Utilize 64-bit status during map

2021-12-03 Thread Eric Blake
_client_co_block_status). Then in later patches, once I add an NBD extension for a 64-bit block status, the same map command completes with just one NBD_CMD_BLOCK_STATUS. Signed-off-by: Eric Blake --- qemu-io-cmds.c | 7 ++- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/qemu-io-cm

[Libguestfs] [PATCH 08/14] nbd/server: Initial support for extended headers

2021-12-03 Thread Eric Blake
siest approach is to truncate our answer back to 32 bits, letting us delay the effort of implementing NBD_REPLY_TYPE_BLOCK_STATUS_EXT to a separate patch. Signed-off-by: Eric Blake --- nbd/nbd-internal.h | 5 ++- nbd/server.c | 106 ++--- 2 files ch

[Libguestfs] [PATCH 14/14] do not apply: nbd/server: Send 64-bit hole chunk

2021-12-03 Thread Eric Blake
Since we cap NBD_CMD_READ requests to 32M, we never have a reason to send a 64-bit chunk type for a hole; but it is worth producing these for interoperability testing of clients that want extended headers. --- nbd/server.c | 18 ++ 1 file changed, 14 insertions(+), 4 deletions(-)

[Libguestfs] [PATCH 11/14] nbd/client: Accept 64-bit hole chunks

2021-12-03 Thread Eric Blake
Although our read requests are sized such that servers need not send an extended hole chunk, we still have to be prepared for it to be fully compliant if we request extended headers. We can also tolerate a non-compliant server sending the new chunk even when it should not. Signed-off-by: Eric

[Libguestfs] [PATCH 07/14] nbd: Add types for extended headers

2021-12-03 Thread Eric Blake
server use a packed-struct representation for the request header. Signed-off-by: Eric Blake --- docs/interop/nbd.txt | 1 + include/block/nbd.h | 67 +++- nbd/common.c | 10 +-- 3 files changed, 62 insertions(+), 16 deletions(-) diff --git a

[Libguestfs] [PATCH 09/14] nbd/server: Support 64-bit block status

2021-12-03 Thread Eric Blake
lizing 64-bits all the way through when the client understands that. Signed-off-by: Eric Blake --- nbd/server.c | 72 ++-- 1 file changed, 48 insertions(+), 24 deletions(-) diff --git a/nbd/server.c b/nbd/server.c index 0e496f60ffbd..7e6140350797 10064

[Libguestfs] [PATCH] spec: Add NBD_OPT_EXTENDED_HEADERS

2021-12-03 Thread Eric Blake
). Signed-off-by: Eric Blake --- Available at https://repo.or.cz/nbd/ericb.git/shortlog/refs/tags/exthdr-v1 doc/proto.md | 218 +-- 1 file changed, 177 insertions(+), 41 deletions(-) diff --git a/doc/proto.md b/doc/proto.md index 3a877a9..46560b6 100644

[Libguestfs] [PATCH 12/14] nbd/client: Accept 64-bit block status chunks

2021-12-03 Thread Eric Blake
esult, once a future patch enables the client requesting extended mode. We can also tolerate a non-compliant server sending the new chunk even when it should not. Signed-off-by: Eric Blake --- block/nbd.c | 38 +++--- 1 file changed, 27 insertions(+), 11 deletions(-)

[Libguestfs] [libnbd PATCH 05/13] protocol: Prepare to receive 64-bit replies

2021-12-03 Thread Eric Blake
Support receiving headers for 64-bit replies if extended headers were negotiated. We already insist that the server not send us too much payload in one reply, so we can exploit that and merge the 64-bit length back into a normalized 32-bit field for the rest of the payload length calculations. Th

[Libguestfs] [libnbd PATCH 04/13] protocol: Prepare to send 64-bit requests

2021-12-03 Thread Eric Blake
Support sending 64-bit requests if extended headers were negotiated. At this point, h->extended_headers is permanently false (we can't enable it until all other aspects of the protocol have likewise been converted). --- lib/internal.h | 12 --- generator/states-issue-

[Libguestfs] [libnbd PATCH 03/13] protocol: Add definitions for extended headers

2021-12-03 Thread Eric Blake
Add the magic numbers and new structs necessary to implement the NBD protocol extension of extended headers providing 64-bit lengths. --- lib/nbd-protocol.h | 61 ++ 1 file changed, 51 insertions(+), 10 deletions(-) diff --git a/lib/nbd-protocol.h b/lib

[Libguestfs] [libnbd PATCH 00/13] libnbd patches for NBD_OPT_EXTENDED_HEADERS

2021-12-03 Thread Eric Blake
Available here: https://repo.or.cz/libnbd/ericb.git/shortlog/refs/tags/exthdr-v1 I also want to do followup patches to teach 'nbdinfo --map' and 'nbdcopy' to utilize 64-bit extents. Eric Blake (13): golang: Simplify nbd_block_status callback array copy block_status: Re

[Libguestfs] [PATCH 01/14] nbd/server: Minor cleanups

2021-12-03 Thread Eric Blake
Spelling fixes, grammar improvements and consistent spacing, noticed while preparing other patches in this file. Signed-off-by: Eric Blake --- nbd/server.c | 13 ++--- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/nbd/server.c b/nbd/server.c index 4630dd732250

[Libguestfs] [libnbd PATCH 02/13] block_status: Refactor array storage

2021-12-03 Thread Eric Blake
For 32-bit block status, we were able to cheat and use an array with an odd number of elements, with array[0] holding the context id, and passing &array[1] to the user's callback. But once we have 64-bit extents, we can no longer abuse array element 0 like that. Split out a new state to receive t

[Libguestfs] [libnbd PATCH 07/13] generator: Add struct nbd_extent in prep for 64-bit extents

2021-12-03 Thread Eric Blake
The existing nbd_block_status() callback is permanently stuck with an array of uint32_t pairs (len/2 extents), and exposing 64-bit extents requires a new API. Before we get there, we first need a way to express a struct containing uint64_t length and uint32_t flags across the various language bind

[Libguestfs] [libnbd PATCH 09/13] block_status: Accept 64-bit extents during block status

2021-12-03 Thread Eric Blake
Support a server giving us a 64-bit extent. Note that the protocol says a server should not give a 64-bit answer when extended headers are not negotiated, but since the client's size is merely a hint, it is possible for a server to have a 64-bit answer even when the original query was 32 bits. At

[Libguestfs] [libnbd PATCH 08/13] block_status: Track 64-bit extents internally

2021-12-03 Thread Eric Blake
When extended headers are in use, the server can send us 64-bit extents, even for a 32-bit query (if the server knows the entire image is data, for example). For maximum flexibility, we are thus better off storing 64-bit lengths internally, even if we have to convert it back to 32-bit lengths when

[Libguestfs] [libnbd PATCH 10/13] api: Add [aio_]nbd_block_status_64

2021-12-03 Thread Eric Blake
Overcome the inherent 32-bit limitation of our existing nbd_block_status command by adding a 64-bit variant. The command sent to the server does not change, but the user's callback is now handed 64-bit information regardless of whether the server replies with 32- or 64-bit extents. Unit tests pro

[Libguestfs] [libnbd PATCH 06/13] protocol: Accept 64-bit holes during pread

2021-12-03 Thread Eric Blake
Even though we don't allow the user to request NBD_CMD_READ with more than 64M (and even if we did, our API signature caps us at SIZE_MAX, which is 32 bits on a 32-bit machine), the NBD extension to allow 64-bit requests implies that for symmetry we have to be able to support 64-bit holes over the

[Libguestfs] [libnbd PATCH 11/13] api: Add three functions for controlling extended headers

2021-12-03 Thread Eric Blake
The new NBD_OPT_EXTENDED_HEADERS feature is worth using by default, but there may be cases where the user explicitly wants to stick with the older 32-bit headers. nbd_set_request_extended_headers() will let the client override the default, nbd_get_request_extended_headers() determines the current

[Libguestfs] [libnbd PATCH 12/13] generator: Actually request extended headers

2021-12-03 Thread Eric Blake
This is the culmination of the previous patches preparation work for using extended headers when possible. The new states in the state machine are copied extensively from our handling of OPT_STRUCTURED_REPLY. At the same time I posted this patch, I had patches for qemu-nbd to support extended hea

[Libguestfs] [libnbd PATCH 13/13] interop: Add test of 64-bit block status

2021-12-03 Thread Eric Blake
Prove that we can round-trip a block status request larger than 4G through a new-enough qemu-nbd. Also serves as a unit test of our shim for converting internal 64-bit representation back to the older 32-bit nbd_block_status callback interface. --- interop/Makefile.am | 6 ++ interop/large-

Re: [Libguestfs] [PATCH] spec: Add NBD_OPT_EXTENDED_HEADERS

2021-12-06 Thread Eric Blake
x27;s whims, and the client must be prepared for either, regardless of the length the client used originally. ... > > Overall, seems good to me. Glad to hear it! > > 1. Could we move some fixes / rewordings to a preaparation patch? Yes, I'll do that for v2. > > 2. I

Re: [Libguestfs] [v2v PATCH 1/2] lib/utils: get_disk_allocated: assert that NBD block length is positive

2022-01-14 Thread Eric Blake
crash v2v by sending a server response that violates the protocol, that can only affect unencrypted connections (so it's no worse than any other MitM attack, and equally mitigated). I'm also inclined to ACK the patch. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] why do we have AI_ADDRCONFIG in nbdkit?

2022-01-18 Thread Eric Blake
r for us in the short term to not add an option (which is already difficult to test whether it works) if no one has expressed a desire for it. netcat has to be a lot more flexible than nbdkit, so having -4/-6 there but not in nbdkit is not a big loss from my point of view. -- Eric Blake, Pri

Re: [Libguestfs] [nbdkit PATCH 2/2] server/sockets: get rid of AI_ADDRCONFIG

2022-01-18 Thread Eric Blake
ect (when -i is not used) is good justification for having -4/-6 support. > + > +When both I<-4> and I<-6> options are present on the command line, the > +last one takes effect. Ah, so you documented the override as intentional. It is possibl

Re: [Libguestfs] [nbdkit PATCH 1/2] server/sockets: supply trailing newline for gai_strerror() format string

2022-01-18 Thread Eric Blake
On Tue, Jan 18, 2022 at 02:48:32PM +0100, Laszlo Ersek wrote: > Signed-off-by: Laszlo Ersek > --- May be worth adding: Fixes: 999797bad ("Implement nbdkit server.", v0.1.0) -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualiz

Re: [Libguestfs] [PATCH v2v 2/3] lib/qemuNBD.ml: Use qemu-nbd --shared=0 flag to allow multiple connections

2022-01-20 Thread Eric Blake
shared=0. > > Note this does not (in current qemu) enable multi-conn because we > aren't using the -r (read-only) flag. Yeah, I still need to work on that for qemu 7.0 (my attempt for 6.2 garnered some interesting review comments from Kevin, which I still need to address). -- Eric

Re: [Libguestfs] [PATCH libnbd 2/2] examples: copy-libev.c: Fix error handling

2022-01-27 Thread Eric Blake
n hoping to keep it under embargo if needed, but the cat's out of the bag now, so watch for similar patches to other affected clients.) -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org __

Re: [Libguestfs] Block alignment of qcow2 compress driver

2022-01-28 Thread Eric Blake
a/copy/nbd-ops.c > +++ b/copy/nbd-ops.c > @@ -59,6 +59,10 @@ open_one_nbd_handle (struct rw_nbd *rwn) > exit (EXIT_FAILURE); >} > > + uint32_t sm = nbd_get_strict_mode (nbd); > + sm &= ~LIBNBD_STRICT_ALIGN; > + nbd_set_strict_mode (nbd, sm); > + >

Re: [Libguestfs] [PATCH libnbd 3/9] golang: aio_buffer.go: Add missing documentation

2022-02-01 Thread Eric Blake
ointer to a byte in the underlying C array. The pointer can > +// be used to modify the underlying array. The pointer must not be used after > +// caling Free(). > func (b *AioBuffer) Get(i uint) *byte { > return (*byte)(unsafe.Pointer(uintptr(b.P) + uintptr(i))) >

Re: [Libguestfs] [PATCH libnbd 5/9] golang: aio_buffer.go: Add Slice()

2022-02-01 Thread Eric Blake
ufferSize) > + defer buf.Free() > + var r int > + > + b.ResetTimer() > + for i := 0; i < b.N; i++ { > + r += len(buf.Slice()) > + } > +} > -- > 2.34.1 > -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-30

Re: [Libguestfs] [PATCH libnbd 8/9] golang: aio_buffer.go: Benchmark copy flows

2022-02-01 Thread Eric Blake
ee() > var r int > > b.ResetTimer() Looks like you had a blank line with trailing whitespace in an earlier patch. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org

Re: [Libguestfs] [PATCH nbdkit] plugins: python: Make threading model explicit

2022-02-01 Thread Eric Blake
all requests so we can seek safely in pread and pwrite > +# and be compatible with python 3.6. In python 3.7 we can use > +# os.preadv and os.pwritev and use the parallel threading model. > +return nbdkit.THREAD_MODEL_SERIALIZE_ALL_REQUESTS ACK -- Eric Blake, Principa

[Libguestfs] [libnbd PATCH 3/4] examples: glib-main-loop: Fix error handling

2022-02-01 Thread Eric Blake
This example failed to check the *error parameter to the read and write completion callbacks. What's more, the buffers start life uninitialized, so a failed read can end up leaking heap contents to the corresponding write. Fortunately, the example program is hard-coded to two particular nbdkit in

[Libguestfs] [libnbd PATCH 1/4] examples: aio-connect-read.c: Fix error handling

2022-02-01 Thread Eric Blake
This example failed to check the *error parameter to the read completion callback. Although the buffers in this example start life zero-initialized due to living in .bss (so there is no leak of heap contents), later reads could repeat the decoding of a buffer returned from an earlier read rather t

[Libguestfs] [libnbd PATCH 4/4] copy: Ignore extents if we encountered earlier error

2022-02-01 Thread Eric Blake
In practice, the nbd extents callback will only be called once per nbd_aio_block_status, because we have only requested exactly one metacontext id, and because compliant servers send only one reply chunk per id. In theory, a server could reply first with an error chunk (perhaps meaning "I'm unable

[Libguestfs] [libnbd PATCH 0/4] Work towards fixing nbdcopy silent corruption

2022-02-01 Thread Eric Blake
and calls nbd_aio_command_completed later - in examples/strict-structured-reads.c, the completion callback properly checks *error I also checked that nbdkit's nbd plugin properly checks *error. Eric Blake (4): examples: aio-connect-read.c: Fix error handling examples: glib-main-loo

[Libguestfs] [libnbd PATCH 2/4] examples: glib-main-loop: don't strand commands when gsource disappears

2022-02-01 Thread Eric Blake
Returning 0 from a callback handler implies that we will later call nbd_aio_command_completed(), but we weren't doing that, and our finalize() handler instead asserts that no outstanding commands remain. Even when we are going to ignore a late completion (due to the gsource already being torn down

Re: [Libguestfs] [libnbd PATCH 4/4] copy: Ignore extents if we encountered earlier error

2022-02-02 Thread Eric Blake
On Wed, Feb 02, 2022 at 09:42:46AM +0100, Laszlo Ersek wrote: > On 02/01/22 20:44, Eric Blake wrote: > > In practice, the nbd extents callback will only be called once per > > nbd_aio_block_status, because we have only requested exactly one > > metacontext id, and because c

[Libguestfs] [libnbd PATCH] docs: Clarify how callbacks should handle errors

2022-02-02 Thread Eric Blake
Recent patches have demonstrated confusion on which order callbacks are reached, when it is safe or dangerous to ignore *error, and what a completion callback should do when auto-retirement is in use. Add wording to make it more obvious that: - mid-command callbacks are reached before the complet

[Libguestfs] [libnbd PATCH] copy: Fail nbdcopy if NBD read or write fails

2022-02-02 Thread Eric Blake
FIXME: This is CVE-2022-X (still awaiting assignment of the CVE number). nbdcopy has a nasty bug when performing multi-threaded copies using asynchronous nbd calls - it was blindly treating the completion of an asynchronous command as successful, rather than checking the *error parameter. Thi

Re: [Libguestfs] [libnbd PATCH] docs: Clarify how callbacks should handle errors

2022-02-03 Thread Eric Blake
On Thu, Feb 03, 2022 at 11:35:49AM +0100, Laszlo Ersek wrote: > On 02/02/22 17:58, Eric Blake wrote: > > Recent patches have demonstrated confusion on which order callbacks > > are reached, when it is safe or dangerous to ignore *error, and what a > > completion callbac

Re: [Libguestfs] [libnbd PATCH] docs: Clarify how callbacks should handle errors

2022-02-03 Thread Eric Blake
ck? This will eliminate the auto retire > concept - if you define a callback, commands always retires. Sadly, we've made a promise of C API stability, so no, we can't change the semantics. There are existing callers that use the completion callback but intentionally return 0 fo

Re: [Libguestfs] [libnbd PATCH] copy: Fail nbdcopy if NBD read or write fails

2022-02-03 Thread Eric Blake
On Thu, Feb 03, 2022 at 11:52:43AM +0100, Laszlo Ersek wrote: > On 02/03/22 02:50, Eric Blake wrote: > > FIXME: This is CVE-2022-X (still awaiting assignment of the CVE number). > > > > nbdcopy has a nasty bug when performing multi-threaded copies using > > asyn

Re: [Libguestfs] [libnbd PATCH] copy: Fail nbdcopy if NBD read or write fails

2022-02-03 Thread Eric Blake
On Thu, Feb 03, 2022 at 02:46:27PM +0200, Nir Soffer wrote: > On Thu, Feb 3, 2022 at 3:51 AM Eric Blake wrote: > > > > FIXME: This is CVE-2022-X (still awaiting assignment of the CVE number). > > > > nbdcopy has a nasty bug when performing multi-threaded copies

Re: [Libguestfs] [PATCH libnbd] golang: tests: Fix error handling

2022-02-03 Thread Eric Blake
deletions(-) ACK. python/t/590-aio-copy.py and ocaml/tests/test_490_aio_copy.ml need the same treatment. I'll post that with my v2 of the CVE fix. (It's nice that we've tried to share the same test numbers across language bindings - it makes it a bit easier where we copy the same cod

Re: [Libguestfs] [PATCH libnbd] golang: tests: Fix error handling

2022-02-03 Thread Eric Blake
On Thu, Feb 03, 2022 at 09:39:24PM +0200, Nir Soffer wrote: > On Thu, Feb 3, 2022 at 9:24 PM Eric Blake wrote: > > > > On Thu, Feb 03, 2022 at 08:07:52PM +0200, Nir Soffer wrote: > > > Like lot of the C examples, the aio copy test ignores read and write > > >

[Libguestfs] [libnbd PATCH v2 0/5] CVE-2022-0485 fix: nbdcopy silent corruption

2022-02-03 Thread Eric Blake
com/archives/libguestfs/2022-February/msg00039.html the rest are indeed new to this posting. Enough has changed that I didn't carry forward any positive reviews from Rich, Nir, or Laszlo. At the bottom of the patch, I'll include the full diff of how v2 improves over v1. Eric Blake (5): p

[Libguestfs] [libnbd PATCH v2 1/5] python: tests: Fix error handling

2022-02-03 Thread Eric Blake
Like a lot of the C examples, the aio copy test ignores read and write errors in the completion callback, which can cause silent data corruption. The failure in the test is not critical, but this is a bad example that may be copied by developers to a real application. The test dies with an asserti

[Libguestfs] [libnbd PATCH v2 2/5] ocaml: tests: Fix error handling

2022-02-03 Thread Eric Blake
Like a lot of the C examples, the aio copy test ignores read and write errors in the completion callback, which can cause silent data corruption. The failure in the test is not critical, but this is a bad example that may be copied by developers to a real application. The test dies with an asserti

<    1   2   3   4   5   6   7   8   9   10   >