Re: [Libguestfs] [nbdkit] Two POSIX questions for you ...
On 10/9/23 16:51, Richard W.M. Jones wrote: > > So one thing we could do for this test is to require (for the test) > that /bin/sh is bash, something like the patch below. I was only able > to test this in the positive case. > > diff --git a/tests/test-log-script-info.sh b/tests/test-log-script-info.sh > index fa9b2ed32..d65f6415d 100755 > --- a/tests/test-log-script-info.sh > +++ b/tests/test-log-script-info.sh > @@ -38,6 +38,10 @@ requires_run > requires_nbdinfo > requires_filter log > > +# This test implicitly assumes /bin/sh is bash, see: > +# https://listman.redhat.com/archives/libguestfs/2023-October/032767.html > +BASH_VERSION=no requires /bin/sh -c 'test "x$BASH_VERSION" != "xno"' > + > if ! nbdinfo --help | grep -- --map ; then > echo "$0: nbdinfo --map option required to run this test" > exit 77 > > Clever! Reviewed-by: Laszlo Ersek You can test it with "dash" BTW (it must be available in Fedora, because it is available in EPEL-9): BASH_VERSION=no /bin/bash -c 'test "x$BASH_VERSION" != "xno"; echo $?' BASH_VERSION=no /bin/dash -c 'test "x$BASH_VERSION" != "xno"; echo $?' Laszlo ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [nbdkit] Two POSIX questions for you ...
On 10/9/23 13:12, Richard W.M. Jones wrote:
> On Mon, Oct 09, 2023 at 12:57:14PM +0200, Laszlo Ersek wrote:
>> On 10/9/23 09:46, Richard W.M. Jones wrote:
>>> Hi Eric, a couple of POSIX questions for you from nbdkit.
>>>
>>> The first question is from an AUR comment on nbdkit:
>>>
>>> https://aur.archlinux.org/packages/nbdkit#comment-937381
>>>
>>> I think there's a bash-ism in the logscript parameter in this test:
>>>
>>>
>>> https://gitlab.com/nbdkit/nbdkit/-/blame/master/tests/test-log-script-info.sh#L51
>>>
>>> I believe it is happening in the $(( .. )) expression. How do we
>>> write that so it'll work in a posix shell?
>>
>> (I'm not Eric, but curious! :) )
>>
>> Here I think we should just explicitly insist on bash.
>
> Unfortunately we can't do that, not easily anyway.
>
> We use bash for writing the test scripts because it's the sane choice
> for shell programming, and we declare /bin/bash (or something with
> 'env') at the top of each of those scripts. This introduces a *build*
> dependency from nbdkit to /bin/bash. That's all fine.
>
> However when nbdkit is installed in production & runs an external
> script, it uses system(3) which uses /bin/sh. That might not be bash,
> and indeed bash might not even be installed on the same machine as
> nbdkit.
>
> [Note: I think the whole Debian dash-for-/bin/sh is the stupidest idea
> I ever heard, but (a) it's a thing and (b) there's BSD and Macs where
> they don't want to use bash for licensing reasons.]
>
> Arguably for the tests we could have some way to cause nbdkit to use
> /bin/bash instead of /bin/sh when running external scripts, but it's
> major complexity to implement.
>
> So we gotta use a lowest common denominator for these external
> scripts, even in our tests. Note only this one test is affected.
>
> [Aside: Windows is a whole separate kettle of fish. Currently the
> Windows port of nbdkit doesn't support --run for other reasons, but if
> it did it would probably run some Windows-ish thing (COMMAND.COM?
> Power Shell?). I'm not sure how Windows behaves for the other
> external commands we try to run like logscript.]
>
>> Shell arrays are
>> a bash-specific feature, and the extent array is deeply ingrained. See
>> especially commit df63b23b6280 ("log: Use strict shell quoting for every
>> parameter displayed in the log file.", 2021-01-04). In particular the
>> assignment
>>
>> extents=(0x0 0x8000 "hole,zero" 0x8000 0x8000 "")
>>
>> turns "extents" into an array variable.
>>
>> In the bug tracker, comment
>> <https://aur.archlinux.org/packages/nbdkit#comment-937375> says, "Thus
>> this is an upstream issue; their scripts are calling sh when they should
>> be calling bash". I think that's correct; for logscript=..., we should
>> require /bin/bash in the manual, and execute the script with /bin/bash
>> explicitly, not just system().
>
> This would create a runtime dependency from nbdkit to /bin/bash which
> I'd like to avoid.
The runtime dependency is already there in our logscript interface; the
name=(a b c ... z)
syntax is already bash-only, for defining a shell array.
So the question is basically how to best emulate an array in the POSIX
shell. Some (rough) options that occur to me:
- Use named variables such as name_0, name_1, name_2, ... and so on.
Requires eval tricks, and if the array is large, it creates many
variables, which some shells (?) may have issues with.
- Generate a shell function with a huge case statement; like "get_name
0" should print "a", "get_name 1" should print "b", etc. The caller
would then do
element=$(get_name $idx)
- write the elements of the array to a text file (one, quoted, element
per line), and then use a combination of "tail" and "head" for fetching
the right line. Incredibly slow, of course.
... I'm sure stackoverflow has further / better ideas for emulating
arrays in the POSIX shell.
And then, because keeping the current (fast, but nonportable) solution
would be nice, we should probably add a new argument for the log filter,
"compat" or "posix" or some such, which would select the more restricted
interface.
Laszlo
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [nbdkit] Two POSIX questions for you ...
On 10/9/23 09:46, Richard W.M. Jones wrote:
> Hi Eric, a couple of POSIX questions for you from nbdkit.
>
> The first question is from an AUR comment on nbdkit:
>
> https://aur.archlinux.org/packages/nbdkit#comment-937381
>
> I think there's a bash-ism in the logscript parameter in this test:
>
>
> https://gitlab.com/nbdkit/nbdkit/-/blame/master/tests/test-log-script-info.sh#L51
>
> I believe it is happening in the $(( .. )) expression. How do we
> write that so it'll work in a posix shell?
(I'm not Eric, but curious! :) )
Here I think we should just explicitly insist on bash. Shell arrays are
a bash-specific feature, and the extent array is deeply ingrained. See
especially commit df63b23b6280 ("log: Use strict shell quoting for every
parameter displayed in the log file.", 2021-01-04). In particular the
assignment
extents=(0x0 0x8000 "hole,zero" 0x8000 0x8000 "")
turns "extents" into an array variable.
In the bug tracker, comment
says, "Thus
this is an upstream issue; their scripts are calling sh when they should
be calling bash". I think that's correct; for logscript=..., we should
require /bin/bash in the manual, and execute the script with /bin/bash
explicitly, not just system().
>
> - - -
>
> Secondly while looking into this I was trying variations on:
>
> $ POSIXLY_CORRECT=1 ./nbdkit -fv data '1 2 3' --run 'nbdinfo $uri'
>
> This doesn't actually cause bash to emulate a posix shell, but it does
> uncover a different bug:
>
> nbdkit: error: raw|base64|data parameter must be specified exactly once
>
> This seems to be happening because getopt_long in wrapper.c behaves
> somehow differently parsing when POSIXLY_CORRECT is set. However I
> couldn't work out exactly why.
>
> https://gitlab.com/nbdkit/nbdkit/-/blob/master/wrapper.c?ref_type=heads#L278
>
> I guess the wrapper ought to work if POSIXLY_CORRECT is set (?)
IIRC, POSIXLY_CORRECT makes getopt() stop parsing options when the first
non-option argument (i.e., first operand) is reached. So "-f" and "-v"
are taken as options, then the two arguments "data" and '1 2 3' are
taken as operands, and then "--run" and the rest are taken as operands
as well.
I think the following loop is relevant:
/* Are there any non-option arguments? */
if (optind < argc) {
/* Ensure any further parameters can never be parsed as options by
* real nbdkit.
*/
passthru ("--");
/* The first non-option argument is the plugin name. If it is a
* short name then rewrite it.
*/
if (is_short_name (argv[optind])) {
const char *language;
/* Plugins written in scripting languages. */
if (is_script_plugin (argv[optind], )) {
passthru_format ("%s/plugins/%s/.libs/nbdkit-%s-plugin." SOEXT,
builddir, language, language);
passthru_format ("%s/plugins/%s/nbdkit-%s-plugin",
builddir, argv[optind], argv[optind]);
}
/* Otherwise normal plugins written in C or other languages that
* compile to .so files.
*/
else {
passthru_format ("%s/plugins/%s/.libs/nbdkit-%s-plugin." SOEXT,
builddir, argv[optind], argv[optind]);
}
++optind;
}
/* Everything else is passed through without rewriting. */
while (optind < argc) {
passthru (argv[optind]);
++optind;
}
}
With POSIXLY_CORRECT set, the "Everything else is passed through without
rewriting" logic extends to "--run" etc. I'm not sure how that would
lead to the specific error message, though.
Laszlo
>
> Rich.
>
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [libnbd PATCH 2/2] info: Show human sizes for block_size values
On 10/7/23 11:27, Richard W.M. Jones wrote:
> On Fri, Oct 06, 2023 at 10:18:09AM -0500, Eric Blake wrote:
>> Adding a human-readable size for block constraints is useful. For:
>>
>> $ ./run nbdinfo -- [ nbdkit memory \
>>--filter=blocksize-policy blocksize-preferred=32k 1M ] | grep pref
>>
>> this changes pre-patch:
>> block_size_preferred: 32768
>> to post-patch:
>> block_size_preferred: 32768 (32K)
>
> I think info/nbdinfo.pod needs to be updated.
Good catch!
With that:
series
Reviewed-by: Laszlo Ersek
>
> Rich.
>
>> Signed-off-by: Eric Blake
>> ---
>> info/show.c | 26 +++---
>> 1 file changed, 19 insertions(+), 7 deletions(-)
>>
>> diff --git a/info/show.c b/info/show.c
>> index 6aeffb54..ac483f34 100644
>> --- a/info/show.c
>> +++ b/info/show.c
>> @@ -35,6 +35,7 @@
>> #include "nbdinfo.h"
>>
>> static void show_boolean (const char *name, bool cond);
>> +static void show_size (const char *name, int64_t size);
>> static int collect_context (void *opaque, const char *name);
>> static char *get_content (struct nbd_handle *, int64_t size);
>>
>> @@ -181,13 +182,9 @@ show_one_export (struct nbd_handle *nbd, const char
>> *desc,
>>show_boolean ("can_trim", can_trim);
>> if (can_zero >= 0)
>>show_boolean ("can_zero", can_zero);
>> -if (block_minimum > 0)
>> - fprintf (fp, "\t%s: %" PRId64 "\n", "block_size_minimum",
>> block_minimum);
>> -if (block_preferred > 0)
>> - fprintf (fp, "\t%s: %" PRId64 "\n", "block_size_preferred",
>> - block_preferred);
>> -if (block_maximum > 0)
>> - fprintf (fp, "\t%s: %" PRId64 "\n", "block_size_maximum",
>> block_maximum);
>> +show_size ("block_size_minimum", block_minimum);
>> +show_size ("block_size_preferred", block_preferred);
>> +show_size ("block_size_maximum", block_maximum);
>>}
>>else {
>> if (first)
>> @@ -304,6 +301,21 @@ show_boolean (const char *name, bool cond)
>>ansi_restore (fp);
>> }
>>
>> +/* Used for displaying sizes in non-JSON output. */
>> +void show_size (const char *name, int64_t size)
>> +{
>> + char size_str[HUMAN_SIZE_LONGEST];
>> + bool human_size_flag = false;
>> +
>> + if (size > 0) {
>> +human_size (size_str, size, _size_flag);
>> +if (human_size_flag)
>> + fprintf (fp, "\t%s: %" PRId64 " (%s)\n", name, size, size_str);
>> +else
>> + fprintf (fp, "\t%s: %" PRId64 "\n", name, size);
>> + }
>> +}
>> +
>> static int
>> collect_context (void *opaque, const char *name)
>> {
>> --
>> 2.41.0
>>
>> ___
>> Libguestfs mailing list
>> Libguestfs@redhat.com
>> https://listman.redhat.com/mailman/listinfo/libguestfs
>
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH virt-v2v] -it ssh: Double quote ssh command which tests remote file exists
On 10/2/23 11:03, Richard W.M. Jones wrote:
> Double quoting was removed in
> commit e2af12ba69c4463bb73d30db63290a887cdd41eb ("input: -i vmx:
> Remove support for openssh scp < 8.8", Nov 2021). However it should
> only have been removed from scp commands, not for this ssh command
> where it is still required.
>
> See: https://github.com/libguestfs/virt-v2v/issues/35
> Thanks: Laszlo Ersek for diagnosis and suggesting the fix
> Reported-by: Bill Sanders
> ---
> input/parse_domain_from_vmx.ml | 6 +-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/input/parse_domain_from_vmx.ml b/input/parse_domain_from_vmx.ml
> index 15ee093f59..2e75e78506 100644
> --- a/input/parse_domain_from_vmx.ml
> +++ b/input/parse_domain_from_vmx.ml
> @@ -97,7 +97,11 @@ let remote_file_exists uri path =
> | None -> ""
> | Some user -> quote user ^ "@")
> (quote (server_of_uri uri))
> -(quote path) in
> +(* Double quoting is necessary for 'ssh', first to protect
> + * from the local shell, second to protect from the remote
> + * shell.
> https://github.com/libguestfs/virt-v2v/issues/35#issuecomment-1741730963
> + *)
> + (quote (quote path)) in
>if verbose () then
> eprintf "%s\n%!" cmd;
>Sys.command cmd = 0
Reviewed-by: Laszlo Ersek
Thanks!
Laszlo
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH v2v 5/5] convert: Find out if Windows guest is expected BIOS set to localtime
On 9/25/23 16:04, Richard W.M. Jones wrote: > Read HKLM\SYSTEM\CurrentControlSet\Control\TimeZoneInformation key > "RealTimeIsUniversal" to see if the Windows guest is expecting BIOS > set to localtime (not present) or UTC (present and set to 1). > > See: https://wiki.archlinux.org/title/System_time#UTC_in_Microsoft_Windows > See: > https://listman.redhat.com/archives/libguestfs/2023-September/thread.html#32556 > Reported-by: Lee Garrett > --- > convert/convert_windows.ml | 38 +- > tests/test-v2v-i-ova.xml | 2 +- > 2 files changed, 38 insertions(+), 2 deletions(-) > > diff --git a/convert/convert_windows.ml b/convert/convert_windows.ml > index f6e039be7e..84e8f7b7d3 100644 > --- a/convert/convert_windows.ml > +++ b/convert/convert_windows.ml > @@ -103,6 +103,42 @@ let convert (g : G.guestfs) _ inspect i_firmware > block_driver _ static_ips = >(* If the Windows guest has AV installed. *) >let has_antivirus = Windows.detect_antivirus inspect in > > + (* Does the guest expect the BIOS to be set to UTC or localtime? > + * See > https://wiki.archlinux.org/title/System_time#UTC_in_Microsoft_Windows > + * Note this might be a QWORD on 64 bit Windows instances. > + *) > + let bios_utc = > +Registry.with_hive_readonly g inspect.i_windows_system_hive > + (fun reg -> > + try > + let key_path = [ "Control"; "TimeZoneInformation" ] in > + let path = inspect.i_windows_current_control_set :: key_path in > + let node = > + match Registry.get_node reg path with > + | None -> raise Not_found > + | Some node -> node in > + let valueh = g#hivex_node_get_value node "RealTimeIsUniversal" in > + if valueh = 0L then raise Not_found; > + let t = g#hivex_value_type valueh in > + let data = g#hivex_value_value valueh in > + let is_utc = > + match t with > + | 0_L (* REG_NONE *) -> false (* localtime *) > + | 4_L (* REG_DWORD *) -> data = "\001\000\000\000" > + | 11_L (* REG_QWORD *) -> data = > "\001\000\000\000\000\000\000\000" > + | _ (* who knows ... *) -> > + warning (f_"unknown CurrentControlSet\\Control\\\ > + TimeZoneInformation key RealTimeIsUniversal \ > + type 0x%Lx, assuming BIOS set to UTC") t; > + true in [*] > + is_utc > + with Not_found -> > + (* If the key is not found then by default we assume > + * that Windows is expecting the BIOS to be set to localtime. > + *) > + false > + ) in > + >(* Open the software hive (readonly) and find the Xen PV uninstaller, > * if it exists. > *) [*] Just guessing, but I'm not sure if assuming UTC is right when the key is found but its type is unrecognized, in light of determining localtime when the key is absent or has type REG_NONE. Either way, the warning is honest about it! > @@ -275,7 +311,7 @@ let convert (g : G.guestfs) _ inspect i_firmware > block_driver _ static_ips = >gcaps_arch = Utils.kvm_arch inspect.i_arch; >gcaps_arch_min_version = 0; >gcaps_virtio_1_0 = virtio_win_installed.Inject_virtio_win.virtio_1_0; > - gcaps_bios_utc = true; > + gcaps_bios_utc = bios_utc; > } in > > guestcaps > diff --git a/tests/test-v2v-i-ova.xml b/tests/test-v2v-i-ova.xml > index a41827bfd5..fa7b4dbfc5 100644 > --- a/tests/test-v2v-i-ova.xml > +++ b/tests/test-v2v-i-ova.xml > @@ -18,7 +18,7 @@ > > hvm > > - > + >destroy >restart >restart With the BIOS -> RTC typo (?) fixed everywhere: series Reviewed-by: Laszlo Ersek ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH v2v 0/5] convert: Find out if Windows guest is expecting BIOS localtime or UTC
On 9/25/23 17:48, Laszlo Ersek wrote: > On 9/25/23 16:04, Richard W.M. Jones wrote: >> [Alice: See patch 2] >> >> [This patch is a bit rough, it could do with better commit messages >> and some tests. Please test it to see if it solves the Windows >> conversion issue described in the thread below.] >> >> We currently do not set any field in guest output. Most >> Windows guests expect the BIOS to be set to localtime, whereas almost >> all Linux guests would expect it to be set to UTC. It is also >> possible to configure a Windows guest to expect BIOS set to UTC. >> >> The default is usually BIOS set to UTC, so for many Windows guests >> this would be wrong. This specifically may cause problems when >> scheduling qemu-ga installation, see the thread here: >> >> https://listman.redhat.com/archives/libguestfs/2023-September/thread.html#32556 >> >> but could cause other general issues with time in the guest. >> >> One way to implement this would be to copy the source hypervisor >> information across; however I'm not confident this information is read >> correctly. A better way is to read out what the guest is expecting >> from the Windows registry. (For Linux we just assume BIOS is always >> UTC, since that's the default for almost any Linux guest which hasn't >> been dual-booted with Windows, which for VMs would be incredibly >> rare.) > > I think the word "BIOS" is incorrectly used all over the series; I'd > rather say "RTC" / "real time clock". ... meaning patch subject lines, commit message bodies, and code. ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH v2v 0/5] convert: Find out if Windows guest is expecting BIOS localtime or UTC
On 9/25/23 16:04, Richard W.M. Jones wrote: > [Alice: See patch 2] > > [This patch is a bit rough, it could do with better commit messages > and some tests. Please test it to see if it solves the Windows > conversion issue described in the thread below.] > > We currently do not set any field in guest output. Most > Windows guests expect the BIOS to be set to localtime, whereas almost > all Linux guests would expect it to be set to UTC. It is also > possible to configure a Windows guest to expect BIOS set to UTC. > > The default is usually BIOS set to UTC, so for many Windows guests > this would be wrong. This specifically may cause problems when > scheduling qemu-ga installation, see the thread here: > > https://listman.redhat.com/archives/libguestfs/2023-September/thread.html#32556 > > but could cause other general issues with time in the guest. > > One way to implement this would be to copy the source hypervisor > information across; however I'm not confident this information is read > correctly. A better way is to read out what the guest is expecting > from the Windows registry. (For Linux we just assume BIOS is always > UTC, since that's the default for almost any Linux guest which hasn't > been dual-booted with Windows, which for VMs would be incredibly > rare.) I think the word "BIOS" is incorrectly used all over the series; I'd rather say "RTC" / "real time clock". Laszlo ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH v2v 3/5] -o libvirt: Add to libvirt XML
On 9/25/23 16:04, Richard W.M. Jones wrote: > --- > output/create_libvirt_xml.ml | 10 -- > tests/test-v2v-i-ova.xml | 1 + > 2 files changed, 9 insertions(+), 2 deletions(-) > > diff --git a/output/create_libvirt_xml.ml b/output/create_libvirt_xml.ml > index 964acd25fd..f97272ca31 100644 > --- a/output/create_libvirt_xml.ml > +++ b/output/create_libvirt_xml.ml > @@ -292,10 +292,16 @@ let create_libvirt_xml ?pool source inspect > e "nvram" ["template", vars_template] [] ] in > > List.push_back_list os loader; > -!os in > +e "os" [] !os in > + > + (* The section. *) > + let clock_section = > +let offset = if guestcaps.gcaps_bios_utc then "utc" else "localtime" in > +e "clock" [ "offset", offset ] [] in > >List.push_back_list body [ > -e "os" [] os_section; > +os_section; > +clock_section; > > e "on_poweroff" [] [PCData "destroy"]; > e "on_reboot" [] [PCData "restart"]; I don't think we need to change what "os_section" is bound to, here; I understand the idea is to increase consistency, but for me it makes the patch harder to read -- ultimately it is a refactoring, and then adding in clock_section is the new thing. If you can split these apart, that's optimal; if not, this is viable too IMO. Thanks Laszlo > diff --git a/tests/test-v2v-i-ova.xml b/tests/test-v2v-i-ova.xml > index e5907ea1cc..a41827bfd5 100644 > --- a/tests/test-v2v-i-ova.xml > +++ b/tests/test-v2v-i-ova.xml > @@ -18,6 +18,7 @@ > > hvm > > + >destroy >restart >restart ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] nbdinfo default output [was: [libnbd PATCH v2 6/6] info: Tolerate missing size]
On 9/22/23 16:14, Eric Blake wrote: > On Fri, Sep 22, 2023 at 09:03:53AM -0500, Eric Blake wrote: $ ./run nbdinfo nbd://localhost protocol: newstyle-fixed without TLS, using extended packets ... block_size_maximum: 33554432 > > For the human-readable output, should we also be using human_size() on > the block_size_* values? I'd find: > > block_size_minimum: 1 > block_size_preferred: 4096 (4K) > block_size_maximum: 33554432 (32M) > > somewhat easier to read. Right, that looks very convenient! Thanks! Laszlo > I don't know if the JSON version would need > to output that extra information, though. But that's a question > independent of this patch series. > ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] Fwd: virt-v2v creating image that does not install guest agent on first boot
On 9/22/23 16:47, Lee Garrett wrote:
> On 22.09.23 14:54, Richard W.M. Jones wrote:
>> On Fri, Sep 22, 2023 at 11:40:03AM +0100, Richard W.M. Jones wrote:
>>> On Thu, Sep 21, 2023 at 07:47:52PM +0200, Lee Garrett wrote:
>>>> On 21.09.23 19:43, Richard W.M. Jones wrote:
>>>>> So this is probably another instance or variation of the timezone
>>>>> formatting problem (of schtasks). Which version of virt-v2v is this?
>>>>> I want to check that you have a version with all the latest patches in
>>>>> this area.
>>>>
>>>> It's 2.2.0-1 from Debian (12) bookworm. I've verified that it
>>>> doesn't have any distro-specific patches.
>>>>
>>>> (https://salsa.debian.org/libvirt-team/virt-v2v/-/tree/debian/master/debian
>>>> would have a patches/series file in this case)
>>>
>>> The timezone fixes are:
>>>
>>> commit 597d177567234c3a539098c423649781424eeb6f
>>> Author: Laszlo Ersek
>>> Date: Tue Mar 8 15:30:51 2022 +0100
>>>
>>> convert_windows: rewrite "configure_qemu_ga" script purely in
>>> PowerShell
>>>
>>> commit d9dc6c42ae64ba92993dbd9477f003ba73fcfa2f
>>> Author: Richard W.M. Jones
>>> Date: Fri Nov 12 08:47:55 2021 +
>>>
>>> convert/convert_windows.ml: Handle date formats with dots
>>> instead of /
>>>
>>> They are all included in >= 2.0
>>>
>>> I wonder if 597d177567 has a subtle flaw, or if we introduced a bug
>>> somewhere when refactoring this code later.
>>>
>>> Lee: Do you have a theory about exactly what is wrong with the
>>> schtasks date? Like what was it supposed to be, assuming it was 120
>>> seconds in the future from boot time, versus what it was set to:
>>>
>>>> Firstboot-qemu-ga 9/21/2023 4:04:00 PM Ready
>>>
>>> Could a date or time field have not been swapped or been corrupted
>>> in some predictable way?
>>
>> Or in even simpler terms, what is the time (and timezone) that
>> this ^^^ machine was booted?
>
> I believe I have it figured out.
> The guest local time is currently 7:08 AM (a few minutes after
> firstboot/provisioning), pacific daylight time (UTC-7, though Windows
> displays it as "UTC-08:00"). This is the timezone that the guest comes
> configured with at first boot. The task is scheduled for 2:01 PM,
> meaning it's scheduled to run ~7 hours in the future.
>
> So it seems like the task was meant to be scheduled for 2:01 PM UTC (=
> 7:01 AM PDT), but for some reason was scheduled for 2:01 PM *local time*.
>
> From what I can see, the host machine time zone is irrelevant (UTC+2).
>
> I don't know where the timezone mixup comes from, though. Running
> `(get-date)` in the powershell at this point correctly returns the local
> time (7:08 AM). I guess during injection the time is in UTC, and
> schtasks.exe has no awareness of timezones?
Right, I think there is a timezone disagreement between how we format the
timestamp and how schtasks.exe takes it.
What matters here is the /ST (start time) flag.
Today we have (in the common submodule):
add "$d = (get-date).AddSeconds(120)";
add "$dtfinfo = [System.Globalization.DateTimeFormatInfo]::CurrentInfo";
add "$sdp = $dtfinfo.ShortDatePattern";
add "$sdp = $sdp -replace 'y+', ''";
add "$sdp = $sdp -replace 'M+', 'MM'";
add "$sdp = $sdp -replace 'd+', 'dd'";
add "schtasks.exe /Create /SC ONCE `";
add " /ST $d.ToString('HH:mm') /SD $d.ToString($sdp) `";
add " /RU SYSTEM /TN Firstboot-qemu-ga `";
add (sprintf " /TR \"C:\\%s /forcerestart /qn /l+*vx C:\\%s.log\""
msi_path msi_path);
Note that for the /ST option's argument, we only perform the following steps:
$d = (get-date).AddSeconds(120)
/ST $d.ToString('HH:mm')
This actually goes back to commit dc66e78fa37d ("windows: delay installation of
qemu-ga MSI", 2020-03-10). The timestamp massaging we've since done only
targeted the /SD (start date) option, not the start time (/ST) one!
So the problem may be that
(get-date).AddSeconds(120).ToString('HH:mm')
formats the hour:minute timestamp in UTC (why though?), but the /ST option
takes hour:minute in local time.
Interestingly, DateTime objects seem to have a "Kind" property, which may be
Utc, Local, or Unspec.
https://learn.microsoft.com/en-us/dotnet/api/system.datetime.kind
It seems to be used when converting from UTC to local or vice versa, and it
prob
Re: [Libguestfs] [libnbd PATCH v2 6/6] info: Tolerate missing size
On 9/21/23 22:58, Eric Blake wrote:
> As previous patches showed, the NBD spec does not yet forbid a server
> sending us a size that does not fit in int64_t. We should gracefully
> handle this during nbdinfo, rather than giving up early.
>
> With the same one-line hack to qemu to set the most significant bit of
> the export size, output changes from pre-patch:
>
> $ ./run nbdinfo nbd://localhost
> /home/eblake/libnbd/info/.libs/nbdinfo: nbd_get_size: server claims size
> 9223372036854781440 which does not fit in signed result: Value too large for
> defined data type
> qemu-nbd: option negotiation failed: Failed to read opts magic: Unexpected
> end-of-file before all data were read
>
> to post-patch:
>
> $ ./run nbdinfo nbd://localhost
> protocol: newstyle-fixed without TLS, using extended packets
> ...
> block_size_maximum: 33554432
>
> or
>
> $ ./run nbdinfo nbd://localhost --json
> {
> "protocol": "newstyle-fixed",
> ...
> "block_size_maximum": 33554432,
> "export-size-str": "unavailable"
> } ]
> }
>
> Sadly, since writing a server with such large export sizes requires a
> one-off hack, I don't see the point in adding a unit test.
>
> Signed-off-by: Eric Blake
> ---
> info/show.c | 25 +
> 1 file changed, 13 insertions(+), 12 deletions(-)
>
> diff --git a/info/show.c b/info/show.c
> index a71d837e..3d80545e 100644
> --- a/info/show.c
> +++ b/info/show.c
> @@ -46,7 +46,7 @@ show_one_export (struct nbd_handle *nbd, const char *desc,
> bool first, bool last)
> {
>int64_t i, size;
> - char size_str[HUMAN_SIZE_LONGEST];
> + char size_str[HUMAN_SIZE_LONGEST] = "unavailable";
>bool human_size_flag;
>char *export_name = NULL;
>char *export_desc = NULL;
> @@ -89,13 +89,10 @@ show_one_export (struct nbd_handle *nbd, const char *desc,
> return false;
>}
>size = nbd_get_size (nbd);
> - if (size == -1) {
> -fprintf (stderr, "%s: %s\n", progname, nbd_get_error ());
> -exit (EXIT_FAILURE);
> + if (size >= 0) {
> +human_size (size_str, size, _size_flag);
>}
>
> - human_size (size_str, size, _size_flag);
> -
>if (uri_is_meaningful ())
> uri = nbd_get_uri (nbd);
>
> @@ -130,7 +127,8 @@ show_one_export (struct nbd_handle *nbd, const char *desc,
> show_context = true;
>
>/* Get content last, as it moves the connection out of negotiating */
> - content = get_content (nbd, size);
> + if (size >= 0)
> +content = get_content (nbd, size);
>
>if (!json_output) {
> ansi_colour (ANSI_FG_BOLD_BLACK, fp);
> @@ -140,10 +138,12 @@ show_one_export (struct nbd_handle *nbd, const char
> *desc,
> fprintf (fp, ":\n");
> if (desc && *desc)
>fprintf (fp, "\tdescription: %s\n", desc);
> -if (human_size_flag)
> - fprintf (fp, "\texport-size: %" PRIi64 " (%s)\n", size, size_str);
> -else
> - fprintf (fp, "\texport-size: %" PRIi64 "\n", size);
> +if (size >= 0) {
> + if (human_size_flag)
> +fprintf (fp, "\texport-size: %" PRIi64 " (%s)\n", size, size_str);
> + else
> +fprintf (fp, "\texport-size: %" PRIi64 "\n", size);
> +}
> if (content)
>fprintf (fp, "\tcontent: %s\n", content);
> if (uri)
> @@ -273,7 +273,8 @@ show_one_export (struct nbd_handle *nbd, const char *desc,
> block_maximum);
>
> /* Put this one at the end because of the stupid comma thing in JSON. */
> -fprintf (fp, "\t\"export-size\": %" PRIi64 ",\n", size);
> +if (size >= 0)
> + fprintf (fp, "\t\"export-size\": %" PRIi64 ",\n", size);
> fprintf (fp, "\t\"export-size-str\": \"%s\"\n", size_str);
>
> if (last)
Assuming the size is unavailable, the non-JSON output includes neither
the numeric nor the human-readable size, whereas the JSON output still
includes the human-readable size (as "unavailable").
Is this difference intentional? I'd have thought that the JSON output
should similarly exclude the human-readable size string in case the size
were not available -- consequently, there'd be no need for initializing
"size_str" to "unavailable", because "size_str" would never be printed
if the size were unavailable.
If OTOH the behavior is intentional, then
Reviewed-by: Laszlo Ersek
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [libnbd PATCH v2 5/6] block_status: Fix assertion on bad 64-bit block status reply
On 9/21/23 22:58, Eric Blake wrote:
> If a server replies to a block status command with an invalid count in
> NBD_REPLY_TYPE_BLOCK_STATUS_EXT, we correctly detect the server's
> error, but fail to mark that we've consumed enough data off the wire
> to resync back to the server's next reply. Rich's fuzzing run
> initially found this, but I was able to quickly write a one-byte patch
> on top of my pending qemu patches [1] to reproduce it:
>
> [1] https://lists.gnu.org/archive/html/qemu-devel/2023-08/msg05231.html
>
> | diff --git i/nbd/server.c w/nbd/server.c
> | index 898580a9b0b..bd8d46ba3c4 100644
> | --- i/nbd/server.c
> | +++ w/nbd/server.c
> | @@ -2326,7 +2326,7 @@ nbd_co_send_extents(NBDClient *client, NBDRequest
> *request, NBDExtentArray *ea,
> | iov[1].iov_base = _ext;
> | iov[1].iov_len = sizeof(meta_ext);
> | stl_be_p(_ext.context_id, context_id);
> | -stl_be_p(_ext.count, ea->count);
> | +stl_be_p(_ext.count, !ea->count);
> |
> | nbd_extent_array_convert_to_be(ea);
> | iov[2].iov_base = ea->extents;
>
> then with a just-built 'qemu-nbd -f raw -r file -t &' running, we have
> pre-patch:
>
> $ ./run nbdsh --base -u nbd://localhost -c - <<\EOF
>> def f(*k):
>> pass
>> try:
>> h.block_status(1,0,f)
>> except nbd.Error as ex:
>> print(ex.string)
>> h.shutdown()
>> EOF
> nbdsh: generator/states-reply-chunk.c:701:
> enter_STATE_REPLY_CHUNK_REPLY_FINISH: Assertion `h->payload_left == 0' failed.
> Aborted (core dumped)
>
> vs. post-patch:
>
> $ ./run nbdsh --base -u nbd://localhost -c - <<\EOF
>> def f(*k):
>> pass
>> try:
>> h.block_status(1,0,f)
>> except nbd.Error as ex:
>> print(ex.string)
>> h.shutdown()
>> EOF
> nbd_block_status: block-status: command failed: Protocol error
>
> Appears to be a casualty of rebasing: I added h->payload_left
> verification fairly late in the game, then floated it earlier in the
> series, and missed a spot where I added a state machine jump to RESYNC
> without having updated h->payload_left. An audit of h->hlen
> modification sites show that all other chunked reads updated
> h->payload_left appropriately (often in the next statement, but
> sometimes in a later state when that made logic easier).
>
> Requires a non-compliant server, and only possible when extended
> headers are negotiated, which does not affect any stable released
> libnbd. Thus, there is no reason to create a CVE, although since I
> will already be doing a security info email about previous patches
> also addressing fuzzer findings, I can mention this at the same time.
>
> Fixes: ab992766cd ("block_status: Accept 64-bit extents during block status")
> Thanks: Richard W.M. Jones
> Signed-off-by: Eric Blake
> ---
> generator/states-reply-chunk.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/generator/states-reply-chunk.c b/generator/states-reply-chunk.c
> index 20407d91..5a31c192 100644
> --- a/generator/states-reply-chunk.c
> +++ b/generator/states-reply-chunk.c
> @@ -476,6 +476,7 @@ REPLY.CHUNK_REPLY.RECV_BS_HEADER:
>if (h->bs_count != be32toh (h->sbuf.reply.payload.bs_hdr_64.count)) {
> h->rbuf = NULL;
> h->rlen = h->payload_left;
> +h->payload_left = 0;
> SET_NEXT_STATE (%RESYNC);
> return 0;
>}
Right, this seems consistent with other transitions to RESYNC (we zero
out payload_left in all those locations), and after we go from RESYNC to
FINISH, FINISH asserts the zero value.
Reviewed-by: Laszlo Ersek
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [libnbd PATCH v2 4/6] block_status: Fix assertion with large server size
* if treated as signed; treat this as an error now, rather
> - * than waiting for the comparison to cap later, to avoid
> - * arithmetic overflow.
> +if (len > INT64_MAX) {
> + /* Pick an aligned value rather than overflowing 64-bit
> + * callback; this does not require an error.
> */
>stop = true;
> - cmd->error = cmd->error ? : EPROTO;
> - len = h->exportsize;
> + len = INT64_MAX + 1ULL - MAX_REQUEST_SIZE;
> }
> if (len > UINT32_MAX && !cmd->cb.wide) {
>/* Pick an aligned value rather than overflowing 32-bit
> @@ -600,7 +602,13 @@ REPLY.CHUNK_REPLY.RECV_BS_ENTRIES:
> }
>}
>
> - total += len;
> + assert (total <= cap);
> + if (len > cap - total) {
> +/* Truncate and expose this extent as an error */
> +len = cap - total;
> +stop = true;
> +cmd->error = cmd->error ? : EPROTO;
> + }
>if (len == 0) {
> stop = true;
> if (i > 0)
> @@ -608,12 +616,7 @@ REPLY.CHUNK_REPLY.RECV_BS_ENTRIES:
> /* Expose this extent as an error; we made no progress */
> cmd->error = cmd->error ? : EPROTO;
>}
> - else if (total > cap) {
> -/* Expose this extent as an error, after truncating to make progress
> */
> -stop = true;
> -cmd->error = cmd->error ? : EPROTO;
> -len -= total - cap;
> - }
> + total += len;
>if (cmd->cb.wide) {
> h->bs_cooked.wide[i].length = len;
> h->bs_cooked.wide[i].flags = flags;
> diff --git a/generator/C.ml b/generator/C.ml
> index e5a2879b..ccaed116 100644
> --- a/generator/C.ml
> +++ b/generator/C.ml
> @@ -496,7 +496,7 @@ let
>pr "/* This is used in the callback for nbd_block_status_64.\n";
>pr " */\n";
>pr "typedef struct {\n";
> - pr " uint64_t length;\n";
> + pr " uint64_t length; /* Will not exceed INT64_MAX */\n";
>pr " uint64_t flags;\n";
>pr "} nbd_extent;\n";
>pr "\n";
I like that this patch handles (len == 0) uniformly between
(a) the server directly sending a 0-length extent, and
(b) len==0 resulting from a (series of) local truncation(s).
A consequence of that however is that -- I think -- the commit message
is inexact:
> As before, we never
> report a zero-length extent to the callback.
Suggest to append: "except when the zero-length is encountered in the
first extent, directly from the server, or as a result of client-side
truncations".
Because, both before and after the patch, we do seem to expose the
len==0 extent, in case i==0.
NB: I don't know *why* we report a zero-length extent if it is the very
first extent (both before and after the patch). But, that's not
particularly important; what's important is that the patch preserves,
and consistently extends, that behavior.
Reviewed-by: Laszlo Ersek
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [libnbd PATCH v2 3/6] api: Sanitize sizes larger than INT64_MAX
a different type rank despite being the same number
> of bits. The Python documentation states that "K" converts an
> arbitrary python integer value to a C uint64_t without overflow
> checking - so it is already possible to pass offset values larger than
> 2^63 in nbdsh; while values larger than 2^64 or negative values are
> effectively truncated as if with modulo math. Enhancing the language
> bindings to explicitly detect over/underflow is outside the scope of
> this patch (and could surprise users who were depending on the current
> truncation semantics).
>
> GoLang.ml generates UInt64 via native Go 'uint64' passed through
> 'C.uint64_t()', and Rust.ml generates UInt64 via native Rust 'u64'
> interpreted as C uint64_t. In both cases, while I am unsure whether
> those languages (which have tighter type rules than C) let you get
> away with directly assigning a negative value to the native type when
> you really want a positive value over 2^63; but since it is a direct
> map of an unsigned 64-bit value between the native type and C, there
> should be no surprises to people fluent in those languages.
>
> OCaml.ml is a bit different; as OCaml lacks a native unsigned 64-bit
> type, it generates UInt64 as native 'int64' converted to C via
> 'Int64_val()'. Thus, an OCaml client MUST pass a negative value if
> they want to access offsets beyond 2^63. But again, someone familiar
> with the language should be familiar with the limitations.
>
> Finally to demonstrate the difference in this patch, I temporarily
> applied this patch to qemu (here, on top of qemu commit 49076448):
>
> | diff --git i/nbd/server.c w/nbd/server.c
> | index b5f93a20c9c..228ce66ed2b 100644
> | --- i/nbd/server.c
> | +++ w/nbd/server.c
> | @@ -691,7 +691,7 @@ static int nbd_negotiate_handle_info(NBDClient *client,
> Error **errp)
> | myflags |= NBD_FLAG_SEND_DF;
> | }
> | trace_nbd_negotiate_new_style_size_flags(exp->size, myflags);
> | -stq_be_p(buf, exp->size);
> | +stq_be_p(buf, exp->size | 0x8000ULL);
> | stw_be_p(buf + 8, myflags);
> | rc = nbd_negotiate_send_info(client, NBD_INFO_EXPORT,
> | sizeof(buf), buf, errp);
>
> then with a just-built 'qemu-nbd -f raw -r file -t &' running, we have
> pre-patch:
>
> $ nbdsh --base -u nbd://localhost -c - <<\EOF
>> try:
>> print(h.get_size())
>> except nbd.Error as ex:
>> print(ex.string)
>> EOF
> -9223372036854770176
>
> vs. post-patch:
>
> $ ./run nbdsh --base -u nbd://localhost -c - <<\EOF
>> try:
>> print(h.get_size())
>> except nbd.Error as ex:
>> print(ex.string)
>> EOF
> nbd_get_size: server claims size 9223372036854781440 which does not fit in
> signed result: Value too large for defined data type
>
> A more complex patch to qemu to mask that bit back off from the offset
> parameter to NBD_CMD_READ/WRITE is also possible to explore behavior
> of passing large offsets over the wire, although I don't show it here.
>
> All stable releases of NBD have had this return value issue. We
> cannot guarantee whether clients may have their own arithmetic bugs
> (such as treating the size as signed, then entering an infinite loop
> when using a negative size as a loop bound), so we will be issuing a
> security notice in case client apps need to file their own CVEs.
> However, since all known production servers do not produces sizes that
s/do not produces/do not produce/
Reviewed-by: Laszlo Ersek
Thanks!
Laszlo
> large, and our audit shows that all stable releases of libnbd
> gracefully handle large offsets even when a client convers a negative
> int64_t result of nbd_get_size() back into large uint64_t offset
> values in subsequent API calls, we did not deem it high enough risk to
> issue a CVE against libnbd proper at this time, although we have
> reached out to Red Hat's secalert team to see if revisiting that
> decision might be warranted.
>
> Based on recent IRC chatter, there is also a slight possibility that
> some future extension to the NBD protocol could specifically allow
> clients to opt in to an extension where the server reports an export
> size of 2^64-1 (all ones) for a unidirectional connection where
> offsets are ignored (either a read-only export of indefinite length,
> or an append-only export data sink - either way, basically turning NBD
> into a cross-system FIFO rather than a seekable device); if such an
> extension materializes, we'd probably add a named constant negative
> sentinel value distinct from -1 for actual return from nbd_get_size()
> at that time.
>
> Fixes: 40881fce75 ("lib: Expose flags and export size
Re: [Libguestfs] [PATCH libguestfs] daemon: Omit 'file -S' option on older distros that lack support
e Free Software Foundation; either version 2 of the License, or > + * (at your option) any later version. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License along > + * with this program; if not, write to the Free Software Foundation, Inc., > + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. > + *) > + > +val file_has_S_option : unit -> bool > diff --git a/daemon/filearch.ml b/daemon/filearch.ml > index 7c858129db..cf784f18a2 100644 > --- a/daemon/filearch.ml > +++ b/daemon/filearch.ml > @@ -128,7 +128,10 @@ and cpio_arch magic orig_path path = > | bin :: bins -> > let bin_path = tmpdir // bin in > if is_regular_file bin_path then ( > - let out = command "file" ["-zSb"; bin_path] in > + let file_options = > + sprintf "-z%sb" > + (if File_helper.file_has_S_option () then "S" else "") in > + let out = command "file" [file_options; bin_path] in > file_architecture_of_magic out orig_path bin_path > ) > else Reviewed-by: Laszlo Ersek Thanks! Laszlo ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] regression: file does not understand the -S option
On 9/20/23 23:42, Olaf Hering wrote:
> Recently a commit was added to call 'file -zSb' instead of 'file -zb'.
>
> This causes a regression on Leap 15 (but not on Tumbleweed), because
> file 5.32 does not understand the -S option.
>
> How can this be fixed properly, to handle both cases either at runtime
> or at buildtime?
This is likely from commit 23986d3c4f4d ("file: Use -S option with -z",
2022-11-28).
Does your error output contain
file: invalid option -- 'S'
?
If it does, then I think we could modify "daemon/file.ml" and
"daemon/filearch.ml". Try "file" with the current options, and if
there's a failure, and stderr contains the above string, retry without -S.
Unfortunately, this is a bit messy. We'd probably want to cache the
availability of -S. Also, because this logic is used from multiple
places, we'd first have to factor out the current "file" invocation --
minimally, try to rebase the "file" invocation in "daemon/filearch.ml"
to the interface exposed by "daemon/file.mli".
Laszlo
>
>
> Thanks,
> Olaf
>
>
> ___
> Libguestfs mailing list
> Libguestfs@redhat.com
> https://listman.redhat.com/mailman/listinfo/libguestfs
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH nbdkit] server: Move size parsing code (nbdkit_parse_size) to common/include
On 9/5/23 16:57, Eric Blake wrote:
> On Tue, Sep 05, 2023 at 11:09:02AM +0100, Richard W.M. Jones wrote:
+static inline int64_t
+human_size_parse (const char *str,
+ const char **error, const char **pstr)
+{
+ int64_t size;
+ char *end;
+ uint64_t scale = 1;
+
+ /* XXX Should we also parse things like '1.5M'? */
+ /* XXX Should we allow hex? If so, hex cannot use scaling suffixes,
+ * because some of them are valid hex digits.
+ */
+ errno = 0;
+ size = strtoimax (str, , 10);
>>>
>>> (1) A further improvement here (likely best done in a separate patch)
>>> could be to change the type of "size" to "intmax_t", from "int64_t".
>>> That way, the assignment will be safe even theoretically, *and* the
>>> overflow check at the bottom of the function (with the division &
>>> comparison of the quotient against INT_MAX) will work just the same.
>>
>> I'm always very unsure how this all works. In particular I seem to
>> recall that intmax_t is no longer really the maximum possible int
>> (because of int128) and so it's always 64 bit on platforms we care
>> about. Can Eric comment?
>
> intmax_t was supposed to be whatever the compiler supports as its
> largest integer type; but you are right that when gcc added
> __int128_t, they did NOT change intmax_t at the time (arguably by
> using weasel-words that as an implementation-defined type, it was not
> an integer type merely because you can't write integer literals of
> that type, even though it behaves integral in every other aspect).
> We're kind of in a frozen state where making intmax_t larger than 64
> bits will break more programs than expected because it has ABI
> implications:
>
> https://stackoverflow.com/questions/21265462/why-in-g-stdintmax-t-is-not-a-int128-t
The fact (which I just learned today) that ABIs are unable to keep up
with intmax_t is a disaster, and entirely defeats the purpose of intmax_t.
>
> My personal preference is to avoid intmax_t, as it has too much
> baggage (the risk of future widening, vs. NOT being the widest type
> after all), similar to how size_t already has baggage. In short,
> having something that is not platform specific is easier to reason
> about (for the same way that using size_t gives me more grief than
> directly using int32_t or int64_t; even though size_t is such a
> naturally occuring type, the fact that it is not uniform width makes
> it trickier to work with).
Very painful disconnect between platforms (which prefer fixed size
integers) and the C standard (which defines all the rules with standard
integer types).
>
+
+ if (INT64_MAX / scale < size) {
+*error = "could not parse size: size * scale overflows";
+*pstr = str;
+return -1;
+ }
>
> And thus I prefer that this comparison stay pegged to INT64_MAX, and
> not INT_MAX.
>
Side comment: I never suggested INT64_MAX be replaced with INTMAX_MAX
here, and that was deliberate on my part. I only suggested changing the
type of "size" from "int64_t" to "intmax_t", so that "size"'s type would
literally match the retval type of strtoimax().
And after such a type change, the expression (INT64_MAX / scale < size)
would remain exactly right:
- All the participating values remain nonnegative, so whatever usual
arithmetic conversions are taken, the values will never change.
- Keeping the INT64_MAX limit continues to make sure that the final
(size * scale) product, although its type *might* change, will produce
the same safe value, for returning through an int64_t. (We'd not want
the retval type to change!)
... Ahhh! I now see where the confusion comes from. My mistake! I wrote:
> That way, the assignment will be safe even theoretically, *and* the
> overflow check at the bottom of the function (with the division &
> comparison of the quotient against INT_MAX) will work just the same.
That's a terrible typo! I meant to write INT64_MAX!
Laszlo
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH libnbd 5/5] copy: Allow human sizes for --queue-size, --request-size, --sparse
arse (optarg, , );
> + if (i64 == -1) {
> +fprintf (stderr, "%s: --sparse: %s: %s\n", prog, error, pstr);
> exit (EXIT_FAILURE);
>}
> - if (sparse_size != 0 &&
> - (sparse_size < 512 || !is_power_of_2 (sparse_size))) {
> -fprintf (stderr, "%s: --sparse: must be a power of 2 and >= 512\n",
> + if (i64 != 0 &&
> + (i64 < 512 || i64 > UINT_MAX || !is_power_of_2 (i64))) {
> +fprintf (stderr, "%s: --sparse: must be a power of 2, between 512
> and UINT_MAX\n",
> prog);
(4) For consistency with the pre-patch code, consider printing optarg or
i64 here as well.
(5) For consistency with (2), I'd suggest printing "within %u-%u" --
that does two things for us: clarifies that 512 precisely is permitted
("between" is a bit murky there), plus prints UINT_MAX numerically.
> exit (EXIT_FAILURE);
>}
> + sparse_size = i64;
>break;
>
> case 'T':
> diff --git a/copy/nbdcopy.h b/copy/nbdcopy.h
> index 465b7052e7..ade53d1a05 100644
> --- a/copy/nbdcopy.h
> +++ b/copy/nbdcopy.h
> @@ -28,7 +28,7 @@
> #include "vector.h"
>
> #define MIN_REQUEST_SIZE 4096
> -#define MAX_REQUEST_SIZE (32 * 1024 * 1024)
> +#define MAX_REQUEST_SIZE (32 * 1024 * 1024) /* must be <= UNSIGNED_MAX */
(6) Good update, but what about not touching this location, and adding a
STATIC_ASSERT at (3) instead? (I.e., just before assigning "request_size".)
>
> /* This must be a multiple of MAX_REQUEST_SIZE. Larger is better up
> * to a point, but it reduces the effectiveness of threads if the work
Address as many as you wish from the above;
Reviewed-by: Laszlo Ersek
Laszlo
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH libnbd 4/5] Revert "copy, info: Include common/utils/human-size.h"
On 9/3/23 17:23, Richard W.M. Jones wrote: > This reverts commit XXX FILL IN LATER XXX > --- > copy/main.c | 2 +- > info/show.c | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) Reviewed-by: Laszlo Ersek > > diff --git a/copy/main.c b/copy/main.c > index d2f415dc47..6928a4acde 100644 > --- a/copy/main.c > +++ b/copy/main.c > @@ -39,7 +39,7 @@ > #include > > #include "ispowerof2.h" > -#include "../utils/human-size.h" > +#include "human-size.h" > #include "minmax.h" > #include "version.h" > #include "nbdcopy.h" > diff --git a/info/show.c b/info/show.c > index 99b7b5b60a..920bbb0a27 100644 > --- a/info/show.c > +++ b/info/show.c > @@ -29,7 +29,7 @@ > #include > > #include "ansi-colours.h" > -#include "../utils/human-size.h" > +#include "human-size.h" > #include "string-vector.h" > > #include "nbdinfo.h" ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH libnbd 3/5] common: Combine human-size.h headers into one
On 9/3/23 17:23, Richard W.M. Jones wrote: > Copy the human_size() function from common/utils/ into the new > human-size.h header in common/include/. Remove human-size.c and > combine the tests into one. > --- > common/include/human-size.h | 51 ++ > common/include/test-human-size.c | 79 +--- > common/utils/Makefile.am | 10 +--- > common/utils/human-size.c| 56 > common/utils/human-size.h| 49 -- > common/utils/test-human-size.c | 89 > 6 files changed, 126 insertions(+), 208 deletions(-) Hopefully you won't get too many rebase conflicts here... Reviewed-by: Laszlo Ersek ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH libnbd 1/5] copy, info: Include common/utils/human-size.h
On 9/3/23 17:23, Richard W.M. Jones wrote: > The next commit will add a new "human-size.h" header copied in from > nbdkit. This breaks existing code that uses common/utils/human-size.h > (which we will combine later). > > As a temporary hack to maintain bisection, make sure we are using the > deprecated "human-size.h" header. This hack will be removed later. > --- > copy/main.c | 2 +- > info/show.c | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) > > diff --git a/copy/main.c b/copy/main.c > index 6928a4acde..d2f415dc47 100644 > --- a/copy/main.c > +++ b/copy/main.c > @@ -39,7 +39,7 @@ > #include > > #include "ispowerof2.h" > -#include "human-size.h" > +#include "../utils/human-size.h" > #include "minmax.h" > #include "version.h" > #include "nbdcopy.h" > diff --git a/info/show.c b/info/show.c > index 920bbb0a27..99b7b5b60a 100644 > --- a/info/show.c > +++ b/info/show.c > @@ -29,7 +29,7 @@ > #include > > #include "ansi-colours.h" > -#include "human-size.h" > +#include "../utils/human-size.h" > #include "string-vector.h" > > #include "nbdinfo.h" Reviewed-by: Laszlo Ersek ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH libnbd 2/5] common/include: Import the human_size_parse (nbdkit_parse_size) function
On 9/3/23 17:23, Richard W.M. Jones wrote: > This function is copied from nbdkit commit XXX [fill in after nbdkit > change is upstream] XXX Right, if you end up updating the nbdkit patch, please refresh this one. Reviewed-by: Laszlo Ersek Laszlo ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH nbdkit] server: Move size parsing code (nbdkit_parse_size) to common/include
/* Strings representing negative values */
> +{ "-1", -1 },
> +{ "-2", -1 },
> +{ "-9223372036854775809", -1 }, /* INT64_MIN - 1 */
> +{ "-9223372036854775808", -1 }, /* INT64_MIN */
> +{ "-9223372036854775807", -1 }, /* INT64_MIN + 1 */
> +{ "-18446744073709551616", -1 }, /* -UINT64_MAX - 1 */
> +{ "-18446744073709551615", -1 }, /* -UINT64_MAX */
> +{ "-18446744073709551614", -1 }, /* -UINT64_MAX + 1 */
> +
> +/* Strings we may want to support in the future */
> +{ "M", -1 },
> +{ "1MB", -1 },
> +{ "1MiB", -1 },
> +{ "1.5M", -1 },
> +
> +/* Valid strings */
> +{ "-0", 0 },
> +{ "0", 0 },
> +{ "+0", 0 },
> +{ " 08", 8 },
> +{ "1", 1 },
> +{ "+1", 1 },
> +{ "1234567890", 1234567890 },
> +{ "+1234567890", 1234567890 },
> +{ "9223372036854775807", INT64_MAX },
> +{ "1s", 512 },
> +{ "2S", 1024 },
> +{ "1b", 1 },
> +{ "1B", 1 },
> +{ "1k", 1024 },
> +{ "1K", 1024 },
> +{ "1m", 1024 * 1024 },
> +{ "1M", 1024 * 1024 },
> +{ "+1M", 1024 * 1024 },
> +{ "1g", 1024 * 1024 * 1024 },
> +{ "1G", 1024 * 1024 * 1024 },
> +{ "1t", 1024LL * 1024 * 1024 * 1024 },
> +{ "1T", 1024LL * 1024 * 1024 * 1024 },
> +{ "1p", 1024LL * 1024 * 1024 * 1024 * 1024 },
> +{ "1P", 1024LL * 1024 * 1024 * 1024 * 1024 },
> +{ "8191p", 1024LL * 1024 * 1024 * 1024 * 1024 * 8191 },
> +{ "1e", 1024LL * 1024 * 1024 * 1024 * 1024 * 1024 },
> +{ "1E", 1024LL * 1024 * 1024 * 1024 * 1024 * 1024 },
> + };
> +
> + for (i = 0; i < ARRAY_SIZE (pairs); i++) {
> +const char *error = NULL, *pstr = NULL;
> +int64_t r;
> +
> +r = human_size_parse (pairs[i].str, , );
> +if (r != pairs[i].res) {
> + fprintf (stderr,
> + "Wrong parse for %s, got %" PRId64 ", expected %" PRId64 "\n",
> + pairs[i].str, r, pairs[i].res);
> + pass = false;
> +}
> +if (r == -1) {
> + if (error == NULL || pstr == NULL) {
> +fprintf (stderr, "Wrong error message handling for %s\n",
> pairs[i].str);
> +pass = false;
> + }
> +}
> + }
> +
> + exit (pass ? EXIT_SUCCESS : EXIT_FAILURE);
> +}
(2) I don't like that we're repeating the test cases here, from
test_nbdkit_parse_size() [server/test-public.c].
Originally I intended to ask "why not just *move* that code as well",
but I think I see the point...
Namely, in test_nbdkit_parse_size(), we still need to test
nbdkit_error() -- via "error_flagged" --, and nbdkit_error() remains
unique to test_nbdkit_parse_size(), after factoring out
human_size_parse(). And so, for triggering the errors, we need to keep
the same test cases.
... Would it be possible to move the "pairs" array into a separate C
file under "common"? (Not necessarily under "common/include".) We'd need
a new header file (for defining the "pair" type, for declaring the
"pairs" array, and for declaring the "num_pairs" constant, which would
have to be a global variable then.)
If that's too difficult or intrusive, then please at least
cross-reference each source file from the other, in new comments, so
that whenever we update one of them, we don't forget the other.
(3) Calling "exit" at the end is a bit awkward to me. Correct, but
"return" would work just as fine.
With the cross-refs added:
Reviewed-by: Laszlo Ersek
Laszlo
> diff --git a/server/public.c b/server/public.c
> index 705ac3a47..a1ba603d4 100644
> --- a/server/public.c
> +++ b/server/public.c
> @@ -76,6 +76,7 @@
> #include "ascii-string.h"
> #include "get_current_dir_name.h"
> #include "getline.h"
> +#include "human-size.h"
> #include "poll.h"
> #include "realpath.h"
> #include "strndup.h"
> @@ -343,83 +344,16 @@ nbdkit_parse_uint64_t (const char *what, const char
> *str, uint64_t *rp)
> NBDKIT_DLL_PUBLIC int64_t
> nbdkit_parse_size (const char *str)
> {
> + const char *error, *pstr;
>int64_t size;
> - char *end;
> - uint64_t scale = 1;
>
> - /* Disk sizes cannot usefully exceed off_t (which is signed) and
> - * cannot be negative. */
> - /* XXX Should we also parse things
Re: [Libguestfs] [nbdkit PATCH] sh: Allow pwrite to not consume all data
On 8/31/23 11:47, Richard W.M. Jones wrote:
> On Thu, Aug 31, 2023 at 11:12:59AM +0200, Laszlo Ersek wrote:
>> On 8/31/23 10:02, Richard W.M. Jones wrote:
>>>
>>> On Wed, Aug 30, 2023 at 05:21:19PM -0500, Eric Blake wrote:
>>>> I hit another transient failure in libnbd CI when a poorly-written
>>>> eval script did not consume all of stdin during .pwrite. As behaving
>>>> as a data sink can be a somewhat reasonable feature of a
>>>> quickly-written sh or eval plugin, we should not be so insistent as
>>>> treating an EPIPE failure as an immediate return of EIO to the client.
>>>
>>> I was thinking about this over night, and came to the conclusion that
>>> it's always fine to ignore EPIPE errors.
>>
>> Interesting; I formed the opposite impression!
>>
>>> For example a script might
>>> be processing the input data gradually and then encounter an error and
>>> want to exit immediately. We also have plenty of plugins that discard
>>> some or all of the written data.
>>
>> But that would be associated with a nonzero exit status, right?
>
> In the error case, yes it would exit with a non-zero exit status.
> In the "don't care about data" case it would exit with 0.
>
> As a concrete example the code currently has to look like this:
>
> case "$1")
> can_write) echo 0 ;;
> pwrite)
> ...
> if [ there is an error ]; then
> cat >/dev/null # discard stdin
> echo 'EIO I/O error' >&2
> exit 1
> else
> cat >/dev/null # discard stdin
> exit 0
> fi
>
> where we're saying that the 'cat >/dev/null' commands are unnecessary
> complication.
OK, I'm *starting* to form an understanding, I think.
With a C-language plugin, we never have to worry about the plugin
*seeing* the input to the pwrite callback. So it is then up to the
plugin to do something with the data. If it wants to return success (0)
without processing even one byte from the input, that's on the plugin.
Furthermore, if the plugin crashes, then all of nbdkit crashes (I think
-- same process, right?), and the client (libnbd) notices that.
However, with the shell plugin, we have a more foundational problem. The
pwrite callback runs in a separate process, and so "parameter passing"
becomes risky in its own right.
I'd say:
- If a shell plugin pwrite exits normally, then ignoring EPIPE in nbdkit
should be OK. Given the explicit exit status (zero or nonzero), and the
errno stuff on stderr, we can trust the script to have made a conscious
decision, and accept that it didn't want to read all input. Fine.
- But if the same script crashes, possibly before it made up its mind
about reading all input or not, we don't have a normal exit status (zero
or nonzero) to rely upon. One way we learn about the script crash is
EPIPE. It's not fool-proof (the script might crash right after it reads
the last byte from the input), but when it happens, it indicates
something wrong.
Let me check the shell plugin for exit status handling...
Right, this is what I've been missing:
if (WIFSIGNALED (status)) {
nbdkit_error ("%s: script terminated by signal %d",
argv0, WTERMSIG (status));
goto error;
}
All good. With this, we can say that we're going to asssume that an
EPIPE combined with a normal exit (zero or nonzero status) means the
script didn't care for the rest of the input, and assume that EPIPE
combined with a child crash is just a child crash, which we're catching
anyway. Hence EPIPE makes no difference.
Thanks
Laszlo
>
> There have been cases where we have forgotten to discard stdin on
> every exit path and this has caused intermittent test failures in CI:
>
> https://gitlab.com/nbdkit/libnbd/-/commit/4df70870420d1be9348ac45f4aa467501eca5089
> https://gitlab.com/nbdkit/libnbd/-/commit/c713529e9fd0641b2d73f764517b5f9c21a767fd
>
> Rich.
>
>> And that way the nbd client would see the pwrite operation as failed.
>>
>> Laszlo
>>
>>>
>>> So my counter-proposal (coming soon) is going to simply turn the EPIPE
>>> error into a debug message and discard the rest of the write buffer.
>>>
>>> Rich.
>>>
>>>
>>>> Signed-off-by: Eric Blake
>>>> ---
>>>>
>>>> I probably need to add unit test coverage of this before committing
>>>> (although proving that I win the data race on a client process exiting
>>>> faster than the parent can write enough data to hit EPIPE is hard).
>>>>
>>>> plugins/sh/nbdkit-sh-plugin.pod | 8 +++
>>>> plugins/sh/call.c |
Re: [Libguestfs] [PATCH nbdkit] sh: In pwrite, allow scripts to ignore stdin
On 8/31/23 10:50, Richard W.M. Jones wrote:
> See comment for explanation and
> https://listman.redhat.com/archives/libguestfs/2023-August/032468.html
> ---
> tests/Makefile.am| 2 +
> plugins/sh/call.c| 33 +++-
> tests/test-sh-pwrite-ignore-stdin.sh | 77
> 3 files changed, 100 insertions(+), 12 deletions(-)
>
> diff --git a/tests/Makefile.am b/tests/Makefile.am
> index d57eb01b8..e69893e0d 100644
> --- a/tests/Makefile.am
> +++ b/tests/Makefile.am
> @@ -1325,11 +1325,13 @@ test-shell.img:
> TESTS += \
> test-sh-errors.sh \
> test-sh-extents.sh \
> + test-sh-pwrite-ignore-stdin.sh \
> test-sh-tmpdir-leak.sh \
> $(NULL)
> EXTRA_DIST += \
> test-sh-errors.sh \
> test-sh-extents.sh \
> + test-sh-pwrite-ignore-stdin.sh \
> test-sh-tmpdir-leak.sh \
> $(NULL)
>
> diff --git a/plugins/sh/call.c b/plugins/sh/call.c
> index 888c6459a..621465252 100644
> --- a/plugins/sh/call.c
> +++ b/plugins/sh/call.c
> @@ -275,22 +275,31 @@ call3 (const char *wbuf, size_t wbuflen, /* sent to
> stdin (can be NULL) */
>r = write (pfds[0].fd, wbuf, wbuflen);
>if (r == -1) {
> if (errno == EPIPE) {
> - /* We tried to write to the script but it didn't consume
> - * the data. Probably the script exited without reading
> - * from stdin. This is an error in the script.
> + /* In nbdkit <= 1.35.11 we gave an error here, arguing that
> + * scripts must always consume or discard their full input
> + * when 'pwrite' is called. Previously we had many cases
> + * where scripts forgot to discard the data on a path out of
> + * pwrite (such as an error or where the script is not
> + * interested in the data being written), resulting in
> + * intermittent test failures.
> + *
> + * It is valid for a script to ignore the written data
> + * (plenty of non-sh plugins do this), or for a script to be
> + * gradually processing the data, encounter an error and
> + * wish to exit immediately. Therefore ignore this error.
> */
I'm not reviewing the code changes in detail, just asking for a comment
extension here:
Can you highlight that, "if a script fails to consume all input due to a crash,
we're going to catch that with waitpid() / WIFSIGNALED() below"?
Basically I'd like us to show that we *know* that we cover for the case when an
EPIPE might not be a conscious decision from the child script.
... Hmmm, let me check something:
#!/bin/bash
ulimit -f 1
cat >f
$ ./script f
$ echo $?
153
$ kill -l 153
XFSZ
Good!
(This example simulates a subprocess of the pwrite script crashing with
SIGXFSZ, due to exceeding the permitted file size limit, *and* the shell
propagating that crashing exit status up to nbdkit's sh plugin.)
Acked-by: Laszlo Ersek
Laszlo
> - nbdkit_error ("%s: write to script failed because of a broken
> pipe: "
> -"this can happen if the script exits without "
> -"consuming stdin, which usually indicates a bug "
> -"in the script",
> -argv0);
> + nbdkit_debug ("%s: write: %m (ignored)", argv0);
> + wbuflen = 0; /* discard the rest */
> }
> -else
> +else {
>nbdkit_error ("%s: write: %m", argv0);
> -goto error;
> + goto error;
> +}
> + }
> + else {
> +wbuf += r;
> +wbuflen -= r;
>}
> - wbuf += r;
> - wbuflen -= r;
>/* After writing all the data we close the pipe so that
> * the reader on the other end doesn't wait for more.
> */
> diff --git a/tests/test-sh-pwrite-ignore-stdin.sh
> b/tests/test-sh-pwrite-ignore-stdin.sh
> new file mode 100755
> index 0..3448eca17
> --- /dev/null
> +++ b/tests/test-sh-pwrite-ignore-stdin.sh
> @@ -0,0 +1,77 @@
> +#!/usr/bin/env bash
> +# nbdkit
> +# Copyright Red Hat
> +#
> +# Redistribution and use in source and binary forms, with or without
> +# modification, are permitted provided that the following conditions are
> +# met:
> +#
> +# * Redistributions of source code must retain the above copyright
> +# notice, this list of conditions and the following disclaimer.
> +#
> +# * Redistributions in binary form must reproduce the above copyright
> +# notice, this list of conditions and the following disclaimer in the
> +# documentation a
Re: [Libguestfs] [nbdkit PATCH] sh: Allow pwrite to not consume all data
On 8/31/23 10:02, Richard W.M. Jones wrote:
>
> On Wed, Aug 30, 2023 at 05:21:19PM -0500, Eric Blake wrote:
>> I hit another transient failure in libnbd CI when a poorly-written
>> eval script did not consume all of stdin during .pwrite. As behaving
>> as a data sink can be a somewhat reasonable feature of a
>> quickly-written sh or eval plugin, we should not be so insistent as
>> treating an EPIPE failure as an immediate return of EIO to the client.
>
> I was thinking about this over night, and came to the conclusion that
> it's always fine to ignore EPIPE errors.
Interesting; I formed the opposite impression!
> For example a script might
> be processing the input data gradually and then encounter an error and
> want to exit immediately. We also have plenty of plugins that discard
> some or all of the written data.
But that would be associated with a nonzero exit status, right?
And that way the nbd client would see the pwrite operation as failed.
Laszlo
>
> So my counter-proposal (coming soon) is going to simply turn the EPIPE
> error into a debug message and discard the rest of the write buffer.
>
> Rich.
>
>
>> Signed-off-by: Eric Blake
>> ---
>>
>> I probably need to add unit test coverage of this before committing
>> (although proving that I win the data race on a client process exiting
>> faster than the parent can write enough data to hit EPIPE is hard).
>>
>> plugins/sh/nbdkit-sh-plugin.pod | 8 +++
>> plugins/sh/call.c | 38 ++---
>> 2 files changed, 34 insertions(+), 12 deletions(-)
>>
>> diff --git a/plugins/sh/nbdkit-sh-plugin.pod
>> b/plugins/sh/nbdkit-sh-plugin.pod
>> index b2c946a0..8b83a5b3 100644
>> --- a/plugins/sh/nbdkit-sh-plugin.pod
>> +++ b/plugins/sh/nbdkit-sh-plugin.pod
>> @@ -505,6 +505,14 @@ Unlike in other languages, if you provide a C
>> method you
>> B also provide a C method which exits with code C<0>
>> (true).
>>
>> +With nbdkit E 1.36, this method may return C<0> without consuming
>> +any data from stdin, and without producing any output, in order to
>> +behave as an intentional data sink. But in older versions, nbdkit
>> +would treat any C failure in writing to your script as an error
>> +condition even if your script returns success; to avoid unintended
>> +failures, you may want to include C<"cat >/dev/null"> in a script
>> +intending to ignore the client's write requests.
>> +
>> =item C
>>
>> /path/to/script flush
>> diff --git a/plugins/sh/call.c b/plugins/sh/call.c
>> index 888c6459..79c67a04 100644
>> --- a/plugins/sh/call.c
>> +++ b/plugins/sh/call.c
>> @@ -34,6 +34,7 @@
>>
>> #include
>> #include
>> +#include
>> #include
>> #include
>> #include
>> @@ -130,6 +131,7 @@ debug_call (const char **argv)
>> */
>> static int
>> call3 (const char *wbuf, size_t wbuflen, /* sent to stdin (can be NULL) */
>> + bool *pipe_full, /* set if wbuf not fully written */
>> string *rbuf, /* read from stdout */
>> string *ebuf, /* read from stderr */
>> const char **argv)/* script + parameters */
>> @@ -275,15 +277,8 @@ call3 (const char *wbuf, size_t wbuflen, /* sent to
>> stdin (can be NULL) */
>>r = write (pfds[0].fd, wbuf, wbuflen);
>>if (r == -1) {
>> if (errno == EPIPE) {
>> - /* We tried to write to the script but it didn't consume
>> - * the data. Probably the script exited without reading
>> - * from stdin. This is an error in the script.
>> - */
>> - nbdkit_error ("%s: write to script failed because of a broken
>> pipe: "
>> -"this can happen if the script exits without "
>> -"consuming stdin, which usually indicates a bug "
>> -"in the script",
>> -argv0);
>> + *pipe_full = true;
>> + r = wbuflen;
>> }
>> else
>>nbdkit_error ("%s: write: %m", argv0);
>> @@ -555,7 +550,7 @@ call (const char **argv)
>>CLEANUP_FREE_STRING string rbuf = empty_vector;
>>CLEANUP_FREE_STRING string ebuf = empty_vector;
>>
>> - r = call3 (NULL, 0, , , argv);
>> + r = call3 (NULL, 0, NULL, , , argv);
>>return handle_script_error (argv[0], , r);
>> }
>>
>> @@ -568,7 +563,7 @@ call_read (string *rbuf, const char **argv)
>>int r;
>>CLEANUP_FREE_STRING string ebuf = empty_vector;
>>
>> - r = call3 (NULL, 0, rbuf, , argv);
>> + r = call3 (NULL, 0, NULL, rbuf, , argv);
>>r = handle_script_error (argv[0], , r);
>>if (r == ERROR)
>> string_reset (rbuf);
>> @@ -584,7 +579,26 @@ call_write (const char *wbuf, size_t wbuflen, const
>> char **argv)
>>int r;
>>CLEANUP_FREE_STRING string rbuf = empty_vector;
>>CLEANUP_FREE_STRING string ebuf = empty_vector;
>> + bool pipe_full = false;
>>
>> - r = call3 (wbuf, wbuflen, , , argv);
>> + r = call3 (wbuf,
Re: [Libguestfs] [nbdkit PATCH] sh: Allow pwrite to not consume all data
On 8/31/23 10:55, Richard W.M. Jones wrote:
> On Thu, Aug 31, 2023 at 10:40:53AM +0200, Laszlo Ersek wrote:
>> On 8/31/23 00:21, Eric Blake wrote:
>>> I hit another transient failure in libnbd CI when a poorly-written
>>> eval script did not consume all of stdin during .pwrite. As behaving
>>> as a data sink can be a somewhat reasonable feature of a
>>> quickly-written sh or eval plugin, we should not be so insistent as
>>> treating an EPIPE failure as an immediate return of EIO to the client.
>>>
>>> Signed-off-by: Eric Blake
>>> ---
>>>
>>> I probably need to add unit test coverage of this before committing
>>> (although proving that I win the data race on a client process exiting
>>> faster than the parent can write enough data to hit EPIPE is hard).
>>>
>>> plugins/sh/nbdkit-sh-plugin.pod | 8 +++
>>> plugins/sh/call.c | 38 ++---
>>> 2 files changed, 34 insertions(+), 12 deletions(-)
>>>
>>> diff --git a/plugins/sh/nbdkit-sh-plugin.pod
>>> b/plugins/sh/nbdkit-sh-plugin.pod
>>> index b2c946a0..8b83a5b3 100644
>>> --- a/plugins/sh/nbdkit-sh-plugin.pod
>>> +++ b/plugins/sh/nbdkit-sh-plugin.pod
>>> @@ -505,6 +505,14 @@ Unlike in other languages, if you provide a C
>>> method you
>>> B also provide a C method which exits with code C<0>
>>> (true).
>>>
>>> +With nbdkit E 1.36, this method may return C<0> without consuming
>>> +any data from stdin, and without producing any output, in order to
>>> +behave as an intentional data sink. But in older versions, nbdkit
>>> +would treat any C failure in writing to your script as an error
>>> +condition even if your script returns success; to avoid unintended
>>> +failures, you may want to include C<"cat >/dev/null"> in a script
>>> +intending to ignore the client's write requests.
>>> +
>>> =item C
>>>
>>> /path/to/script flush
>>> diff --git a/plugins/sh/call.c b/plugins/sh/call.c
>>> index 888c6459..79c67a04 100644
>>> --- a/plugins/sh/call.c
>>> +++ b/plugins/sh/call.c
>>> @@ -34,6 +34,7 @@
>>>
>>> #include
>>> #include
>>> +#include
>>> #include
>>> #include
>>> #include
>>> @@ -130,6 +131,7 @@ debug_call (const char **argv)
>>> */
>>> static int
>>> call3 (const char *wbuf, size_t wbuflen, /* sent to stdin (can be NULL) */
>>> + bool *pipe_full, /* set if wbuf not fully written
>>> */
>>> string *rbuf, /* read from stdout */
>>> string *ebuf, /* read from stderr */
>>> const char **argv)/* script + parameters */
>>> @@ -275,15 +277,8 @@ call3 (const char *wbuf, size_t wbuflen, /* sent to
>>> stdin (can be NULL) */
>>>r = write (pfds[0].fd, wbuf, wbuflen);
>>>if (r == -1) {
>>> if (errno == EPIPE) {
>>> - /* We tried to write to the script but it didn't consume
>>> - * the data. Probably the script exited without reading
>>> - * from stdin. This is an error in the script.
>>> - */
>>> - nbdkit_error ("%s: write to script failed because of a broken
>>> pipe: "
>>> -"this can happen if the script exits without "
>>> -"consuming stdin, which usually indicates a bug "
>>> -"in the script",
>>> -argv0);
>>> + *pipe_full = true;
>>> + r = wbuflen;
>>> }
>>> else
>>>nbdkit_error ("%s: write: %m", argv0);
>>> @@ -555,7 +550,7 @@ call (const char **argv)
>>>CLEANUP_FREE_STRING string rbuf = empty_vector;
>>>CLEANUP_FREE_STRING string ebuf = empty_vector;
>>>
>>> - r = call3 (NULL, 0, , , argv);
>>> + r = call3 (NULL, 0, NULL, , , argv);
>>>return handle_script_error (argv[0], , r);
>>> }
>>>
>>> @@ -568,7 +563,7 @@ call_read (string *rbuf, const char **argv)
>>>int r;
>>>CLEANUP_FREE_STRING string ebuf = empty_vector;
>>>
>>> - r = call3 (NULL, 0, rbuf, , argv);
>>> + r = call3 (NULL, 0, NULL, rbuf, , argv);
>>>r = ha
Re: [Libguestfs] [nbdkit PATCH] sh: Allow pwrite to not consume all data
On 8/31/23 00:21, Eric Blake wrote:
> I hit another transient failure in libnbd CI when a poorly-written
> eval script did not consume all of stdin during .pwrite. As behaving
> as a data sink can be a somewhat reasonable feature of a
> quickly-written sh or eval plugin, we should not be so insistent as
> treating an EPIPE failure as an immediate return of EIO to the client.
>
> Signed-off-by: Eric Blake
> ---
>
> I probably need to add unit test coverage of this before committing
> (although proving that I win the data race on a client process exiting
> faster than the parent can write enough data to hit EPIPE is hard).
>
> plugins/sh/nbdkit-sh-plugin.pod | 8 +++
> plugins/sh/call.c | 38 ++---
> 2 files changed, 34 insertions(+), 12 deletions(-)
>
> diff --git a/plugins/sh/nbdkit-sh-plugin.pod b/plugins/sh/nbdkit-sh-plugin.pod
> index b2c946a0..8b83a5b3 100644
> --- a/plugins/sh/nbdkit-sh-plugin.pod
> +++ b/plugins/sh/nbdkit-sh-plugin.pod
> @@ -505,6 +505,14 @@ Unlike in other languages, if you provide a C
> method you
> B also provide a C method which exits with code C<0>
> (true).
>
> +With nbdkit E 1.36, this method may return C<0> without consuming
> +any data from stdin, and without producing any output, in order to
> +behave as an intentional data sink. But in older versions, nbdkit
> +would treat any C failure in writing to your script as an error
> +condition even if your script returns success; to avoid unintended
> +failures, you may want to include C<"cat >/dev/null"> in a script
> +intending to ignore the client's write requests.
> +
> =item C
>
> /path/to/script flush
> diff --git a/plugins/sh/call.c b/plugins/sh/call.c
> index 888c6459..79c67a04 100644
> --- a/plugins/sh/call.c
> +++ b/plugins/sh/call.c
> @@ -34,6 +34,7 @@
>
> #include
> #include
> +#include
> #include
> #include
> #include
> @@ -130,6 +131,7 @@ debug_call (const char **argv)
> */
> static int
> call3 (const char *wbuf, size_t wbuflen, /* sent to stdin (can be NULL) */
> + bool *pipe_full, /* set if wbuf not fully written */
> string *rbuf, /* read from stdout */
> string *ebuf, /* read from stderr */
> const char **argv)/* script + parameters */
> @@ -275,15 +277,8 @@ call3 (const char *wbuf, size_t wbuflen, /* sent to
> stdin (can be NULL) */
>r = write (pfds[0].fd, wbuf, wbuflen);
>if (r == -1) {
> if (errno == EPIPE) {
> - /* We tried to write to the script but it didn't consume
> - * the data. Probably the script exited without reading
> - * from stdin. This is an error in the script.
> - */
> - nbdkit_error ("%s: write to script failed because of a broken
> pipe: "
> -"this can happen if the script exits without "
> -"consuming stdin, which usually indicates a bug "
> -"in the script",
> -argv0);
> + *pipe_full = true;
> + r = wbuflen;
> }
> else
>nbdkit_error ("%s: write: %m", argv0);
> @@ -555,7 +550,7 @@ call (const char **argv)
>CLEANUP_FREE_STRING string rbuf = empty_vector;
>CLEANUP_FREE_STRING string ebuf = empty_vector;
>
> - r = call3 (NULL, 0, , , argv);
> + r = call3 (NULL, 0, NULL, , , argv);
>return handle_script_error (argv[0], , r);
> }
>
> @@ -568,7 +563,7 @@ call_read (string *rbuf, const char **argv)
>int r;
>CLEANUP_FREE_STRING string ebuf = empty_vector;
>
> - r = call3 (NULL, 0, rbuf, , argv);
> + r = call3 (NULL, 0, NULL, rbuf, , argv);
>r = handle_script_error (argv[0], , r);
>if (r == ERROR)
> string_reset (rbuf);
> @@ -584,7 +579,26 @@ call_write (const char *wbuf, size_t wbuflen, const char
> **argv)
>int r;
>CLEANUP_FREE_STRING string rbuf = empty_vector;
>CLEANUP_FREE_STRING string ebuf = empty_vector;
> + bool pipe_full = false;
>
> - r = call3 (wbuf, wbuflen, , , argv);
> + r = call3 (wbuf, wbuflen, _full, , , argv);
> + if (pipe_full && r == OK) {
> +/* We allow scripts to intentionally ignore data, but they must
> + * have no output when doing so.
> + */
> +if (rbuf.len > 0 || ebuf.len > 0) {
> + nbdkit_error ("%s: write to script failed because of a broken pipe: "
> +"this can happen if the script exits without "
> +"consuming stdin, which usually indicates a bug "
> +"in the script",
> +argv[0]);
> + r = ERROR;
> +}
> +else
> + nbdkit_debug ("%s: write to script failed because of a broken pipe; "
> +"assuming this was an intentional data sink, although it
> "
> +"may indicate a bug in the script",
> +argv[0]);
> + }
>return handle_script_error
Re: [Libguestfs] [libnbd PATCH v4 05/25] golang: Change logic of copy_uint32_array
On 8/8/23 13:36, Nir Soffer wrote:
> On Thu, Aug 3, 2023 at 4:57 AM Eric Blake wrote:
>>
>> Commit 6725fa0e12 changed copy_uint32_array() to utilize a Go hack
>> for accessing a C array as a Go slice in order to potentially benefit
>> from any optimizations in Go's copy() for bulk transfer of memory
>> over naive one-at-a-time iteration. But that commit also
>> acknowledged that no benchmark timings were performed, which would
>> have been useful to demonstrat an actual benefit for using hack in
>> the first place. And
>
> Why do you call this a hack? This is the documented way to create a Go
> slice from memory.
Just because it is documented, it's not less terrible.
<https://github.com/golang/go/wiki/cgo#turning-c-arrays-into-go-slices>:
% use a type conversion to a pointer to a very big array and then slice
% it to the length that you want
The "very big array" is the hack.
The documentation uses 1 << 28 for the element count, the libnbd code
uses 1 << 30 as the element count. In libnbd, the element type is
uint32, so we're banking on (a) the 4G size (byte count) being safely
expressible for the fake array, (b) the 1G elements in the fake array
sufficing to cover the actual C array we want.
In the documentation, the 1<<28 element count makes the same two
assumptions, which is equally bad for (b) (who knows if the actual C
array has no more than 256M elements), and *worse* for (a) (because
"C.YourType" could be a 128 byte large structure, and then the byte
count in the fake array would be 32G -- not representable in 32 bits.
Which may or may not matter).
In other words, the documented workaround is full of uncomfortable
assumptions that make anyone tired *if* the reader even bothers to
enumerate those assumptions.
>> since we are copying data anyways (rather than using the slice to
>> avoid a copy), and network transmission costs have a higher impact to
>> performance than in-memory copying speed, it's hard to justify
>> keeping the hack without hard data.
>
> Since this is not a hack we don't need to justify it :-)
It's a hack because it contains an unjustifiable and/or risky open-coded
constant, namely 1 << 28 (or 1 << 30, in libnbd).
>
>> What's more, while using Go's copy() on an array of C uint32_t makes
>> sense for 32-bit extents, our corresponding 64-bit code uses a struct
>> which does not map as nicely to Go's copy().
>
> If we return a slice of the C extent type, copy() can work, but it is
> probably not what we want to return.
>
>> Using a common style
>> between both list copying helpers is beneficial to mainenance.
>>
>> Additionally, at face value, converting C.size_t to int may truncate;
>> we could avoid that risk if we were to uniformly use uint64 instead
>> of int. But we can equally just panic if the count is oversized: our
>> state machine guarantees that the server's response fits within 64M
>> bytes (count will be smaller than that, since it is multiple bytes
>> per extent entry).
>
> Good to check this, but not related to changing the way we copy the
> array.
>
>> Suggested-by: Laszlo Ersek
>> CC: Nir Soffer
>> Signed-off-by: Eric Blake
>> ---
>>
>> v4: new patch to the series, but previously posted as part of the
>> golang cleanups. Since then: rework the commit message as it is no
>> longer a true revert, and add a panic() if count exceeds expected
>> bounds.
>> ---
>> generator/GoLang.ml | 13 +
>> 1 file changed, 9 insertions(+), 4 deletions(-)
>>
>> diff --git a/generator/GoLang.ml b/generator/GoLang.ml
>> index 73df5254..cc7d78b6 100644
>> --- a/generator/GoLang.ml
>> +++ b/generator/GoLang.ml
>> @@ -516,11 +516,16 @@ let
>> /* Closures. */
>>
>> func copy_uint32_array(entries *C.uint32_t, count C.size_t) []uint32 {
>> +if (uint64(count) > 64*1024*1024) {
>> +panic(\"violation of state machine guarantee\")
>
> This is unwanted in a library, it means the entire application will crash
> because of a bug in the library. Can we convert this to an error in the
> caller?
It's actually an assert().
As long as you agree that assertions are allowed in a library (for
expressing invariants ensured by other parts of the library), this panic
is not functionally wrong.
I'm unsure if "base Go" has something that is spelled "assert".
According to
<https://stackoverflow.com/questions/47558389/what-is-the-go-equivalent-to-assert-in-c>,
there isn't.
(BTW my original recommendation here was a loop that would not rely on
this invariant (instead it would use uint64 for "count"). But asserting
the invariant is
Re: [Libguestfs] [PATCH libnbd] copy/copy-file-to-qcow2-compressed.sh: Skip test for all zeroes disk
On 8/8/23 00:28, Richard W.M. Jones wrote: > Occasionally this test will choose a random seed which results in an > all-zeroes disk. The test tries to convert this to a compressed qcow2 > file, and fails because no compressed clusters are detected in the > resulting file. This happens because qcow2 stores zero clusters with > a special sparse representation, they are never stored compressed, so > a disk with only zeroes in it will never contain compressed clusters. > > To fix this, detect an all-zeroes disk and skip. > > Reported-by: Eric Blake > --- > copy/copy-file-to-qcow2-compressed.sh | 10 ++ > 1 file changed, 10 insertions(+) > > diff --git a/copy/copy-file-to-qcow2-compressed.sh > b/copy/copy-file-to-qcow2-compressed.sh > index 018c8bba2f..2706eadd66 100755 > --- a/copy/copy-file-to-qcow2-compressed.sh > +++ b/copy/copy-file-to-qcow2-compressed.sh > @@ -25,6 +25,7 @@ requires $QEMU_NBD --version > requires nbdkit --exit-with-parent --version > requires nbdkit sparse-random --dump-plugin > requires qemu-img --version > +requires nbdinfo --version > #requires stat --version > > # Check the compress driver is supported by this qemu-nbd. > @@ -45,6 +46,15 @@ cleanup_fn rm -f $file1 $file2 $out1 $out2 > size=1G > seed=$RANDOM > > +# Occasionally we will choose a seed which results in a completely > +# empty file. Skip this case. > +if nbdinfo --map --totals -- \ > +[ nbdkit --exit-with-parent sparse-random $size seed=$seed ] | > +grep -sq '100.0%.*hole,zero'; then > +echo "$0: bad seed chosen, skipping the test" > +exit 77 > +fi > + > # Create a compressed qcow2 file1. > # > # sparse-random files should compress easily because by default each Nbdkit uses its own PRNG from "common/include/random.h", not the one from libc. That means the seed=... parameter entirely determines the random series -- not only does it become time-independent, but also platform-independent. And based on that, could we just eliminate "seed=$RANDOM" from this test case, and use simply "seed=1" (for example)? What we want is a sparse-random disk image, generated both times with the same not-fully-zero contents. seed=1 satisfies that, and makes sure the test is always "armed" (and, not least, deterministic). Laszlo ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH nbdkit v2 8/9] curl: Use curl multi interface
On 8/7/23 13:08, Richard W.M. Jones wrote:
> On Mon, Aug 07, 2023 at 12:57:02PM +0200, Laszlo Ersek wrote:
>> On 8/4/23 20:04, Richard W.M. Jones wrote:
>>> On Fri, Aug 04, 2023 at 11:38:03AM -0500, Eric Blake wrote:
>>>> On Fri, Jul 28, 2023 at 06:17:52PM +0100, Richard W.M. Jones wrote:
>>>>> See the comment at the top of plugins/curl/pool.c for general
>>>>> information about how this works.
>>>>>
>>>>> This makes a very large difference to performance over the previous
>>>>> implementation. Note for the tests below I also applied the next
>>>>> commit changing the behaviour of the connections parameter.
>>>>>
>>>>> Using this test case:
>>>>>
>>>>> $
>>>>> uri=https://cloud-images.ubuntu.com/lunar/current/lunar-server-cloudimg-amd64.img
>>>>> $ nbdkit -r -U - curl $uri ipresolve=v4 --run 'nbdcopy -p $uri null'
>>>>>
>>>>> The times are as follows:
>>>>>
>>>>> multi, connections=64 21.5s
>>>>> multi, connections=32 30.2s
>>>>> multi, connections=16 56.0s
>>>>> before this commit 166s
>>>>
>>>> Awesome performance improvements! As painful as this series has been
>>>> for you to write and debug, it is showing its worth.
>>>>
>>>>> ---
>>>>> plugins/curl/curldefs.h | 35 ++--
>>>>> plugins/curl/config.c | 246 ---
>>>>> plugins/curl/curl.c | 366 +++-
>>>>> plugins/curl/pool.c | 346 -
>>>>> 4 files changed, 616 insertions(+), 377 deletions(-)
>>>>
>>>> Finally taking time to review this, even though it is already in-tree.
>>>>
>>>>> @@ -98,8 +88,30 @@ struct curl_handle {
>>>>>const char *read_buf;
>>>>>uint32_t read_count;
>>>>>
>>>>> + /* This field is used by curl_get_size. */
>>>>> + bool accept_range;
>>>>> +
>>>>>/* Used by scripts.c */
>>>>>struct curl_slist *headers_copy;
>>>>> +
>>>>> + /* Used by pool.c */
>>>>> + struct command *cmd;
>>>>> +};
>>>>> +
>>>>> +/* Asynchronous commands that can be sent to the pool thread. */
>>>>> +enum command_type { EASY_HANDLE, STOP };
>>>>> +struct command {
>>>>> + /* These fields are set by the caller. */
>>>>> + enum command_type type; /* command */
>>>>> + struct curl_handle *ch; /* for EASY_HANDLE, the easy handle */
>>>>> +
>>>>> + /* This field is set to a unique value by send_command_and_wait. */
>>>>> + uint64_t id; /* serial number */
>>>>> +
>>>>> + /* These fields are used to signal back that the command finished. */
>>>>> + pthread_mutex_t mutex;/* completion mutex */
>>>>> + pthread_cond_t cond; /* completion condition */
>>>>> + CURLcode status; /* status code (CURLE_OK = succeeded) */
>>>>> };
>>>>
>>>> Makes sense. The two types are mutually recursive (curl_handle
>>>> includes a struct command *; command includes a struct curl_handle *);
>>>> hopefully you have proper locking when altering multiple objects to
>>>> adjust how they point to one another.
>>>
>>> Actually locking is not needed. Let me document it through ...
>>>
>>> We create both the curl easy handle and the associated EASY_HANDLE
>>> command in the nbdkit thread that gets the request, eg. in the curl
>>> .pread method. That of course requires no locking.
>>>
>>> There is a single background worker thread.
>>>
>>> A "self pipe" passes pointers to 'struct command *' to this worker
>>> thread simple by writing the 8 byte pointer onto the pipe (hopefully
>>> atomic ...) The nbdkit request thread then blocks on the mutex/cond
>>> in the command handle.
>>
>> Right, that's exactly that I got curious about.
>>
>> In my opinion, writing to the self-pipe is mostly safe. Reading from the
>> self-pipe could be slightly polished.
>>
>> Here are the POSIX pages on write() and read():
>>
>> h
Re: [Libguestfs] [libnbd PATCH v4 05/25] golang: Change logic of copy_uint32_array
On 8/4/23 11:16, Richard W.M. Jones wrote:
> On Wed, Aug 02, 2023 at 08:50:25PM -0500, Eric Blake wrote:
>> Commit 6725fa0e12 changed copy_uint32_array() to utilize a Go hack for
>> accessing a C array as a Go slice in order to potentially benefit from
>> any optimizations in Go's copy() for bulk transfer of memory over
>> naive one-at-a-time iteration. But that commit also acknowledged that
>> no benchmark timings were performed, which would have been useful to
>> demonstrat an actual benefit for using hack in the first place. And
>
> "demonstrate"
>
>> since we are copying data anyways (rather than using the slice to
>> avoid a copy), and network transmission costs have a higher impact to
>> performance than in-memory copying speed, it's hard to justify keeping
>> the hack without hard data.
>>
>> What's more, while using Go's copy() on an array of C uint32_t makes
>> sense for 32-bit extents, our corresponding 64-bit code uses a struct
>> which does not map as nicely to Go's copy(). Using a common style
>> between both list copying helpers is beneficial to mainenance.
>>
>> Additionally, at face value, converting C.size_t to int may truncate;
>> we could avoid that risk if we were to uniformly use uint64 instead of
>> int. But we can equally just panic if the count is oversized: our
>> state machine guarantees that the server's response fits within 64M
>> bytes (count will be smaller than that, since it is multiple bytes per
>> extent entry).
>>
>> Suggested-by: Laszlo Ersek
>> CC: Nir Soffer
>> Signed-off-by: Eric Blake
>> ---
>>
>> v4: new patch to the series, but previously posted as part of the
>> golang cleanups. Since then: rework the commit message as it is no
>> longer a true revert, and add a panic() if count exceeds expected
>> bounds.
>> ---
>> generator/GoLang.ml | 13 +
>> 1 file changed, 9 insertions(+), 4 deletions(-)
>>
>> diff --git a/generator/GoLang.ml b/generator/GoLang.ml
>> index 73df5254..cc7d78b6 100644
>> --- a/generator/GoLang.ml
>> +++ b/generator/GoLang.ml
>> @@ -516,11 +516,16 @@ let
>> /* Closures. */
>>
>> func copy_uint32_array(entries *C.uint32_t, count C.size_t) []uint32 {
>> +if (uint64(count) > 64*1024*1024) {
>> +panic(\"violation of state machine guarantee\")
>> +}
>
> Not sure if golang includes the line number when it panics, but a
> brief bit of explanation might help here. Something like:
>
> panic(\"internal error: state machine should guarantee <= 64M of data \
> in the extents reply, but this was exceeded unexpectedly\")
Sounds helpful!
>
>> ret := make([]uint32, int(count))
>> -// See
>> https://github.com/golang/go/wiki/cgo#turning-c-arrays-into-go-slices
>> -// TODO: Use unsafe.Slice() when we require Go 1.17.
>> -s := (*[1 << 30]uint32)(unsafe.Pointer(entries))[:count:count]
>
> The whole magical 1 << 30 was another thing to dislike about golang,
> so glad to see it gone here.
>
>> -copy(ret, s)
>> +addr := uintptr(unsafe.Pointer(entries))
>> +for i := 0; i < int(count); i++ {
>> +ptr := (*C.uint32_t)(unsafe.Pointer(addr))
>> +ret[i] = uint32(*ptr)
>> +addr += unsafe.Sizeof(*ptr)
>> +}
>> return ret
>> }
>
> Reviewed-by: Richard W.M. Jones
Reviewed-by: Laszlo Ersek
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH nbdkit v2 8/9] curl: Use curl multi interface
On 8/4/23 20:04, Richard W.M. Jones wrote:
> On Fri, Aug 04, 2023 at 11:38:03AM -0500, Eric Blake wrote:
>> On Fri, Jul 28, 2023 at 06:17:52PM +0100, Richard W.M. Jones wrote:
>>> See the comment at the top of plugins/curl/pool.c for general
>>> information about how this works.
>>>
>>> This makes a very large difference to performance over the previous
>>> implementation. Note for the tests below I also applied the next
>>> commit changing the behaviour of the connections parameter.
>>>
>>> Using this test case:
>>>
>>> $
>>> uri=https://cloud-images.ubuntu.com/lunar/current/lunar-server-cloudimg-amd64.img
>>> $ nbdkit -r -U - curl $uri ipresolve=v4 --run 'nbdcopy -p $uri null'
>>>
>>> The times are as follows:
>>>
>>> multi, connections=64 21.5s
>>> multi, connections=32 30.2s
>>> multi, connections=16 56.0s
>>> before this commit 166s
>>
>> Awesome performance improvements! As painful as this series has been
>> for you to write and debug, it is showing its worth.
>>
>>> ---
>>> plugins/curl/curldefs.h | 35 ++--
>>> plugins/curl/config.c | 246 ---
>>> plugins/curl/curl.c | 366 +++-
>>> plugins/curl/pool.c | 346 -
>>> 4 files changed, 616 insertions(+), 377 deletions(-)
>>
>> Finally taking time to review this, even though it is already in-tree.
>>
>>> @@ -98,8 +88,30 @@ struct curl_handle {
>>>const char *read_buf;
>>>uint32_t read_count;
>>>
>>> + /* This field is used by curl_get_size. */
>>> + bool accept_range;
>>> +
>>>/* Used by scripts.c */
>>>struct curl_slist *headers_copy;
>>> +
>>> + /* Used by pool.c */
>>> + struct command *cmd;
>>> +};
>>> +
>>> +/* Asynchronous commands that can be sent to the pool thread. */
>>> +enum command_type { EASY_HANDLE, STOP };
>>> +struct command {
>>> + /* These fields are set by the caller. */
>>> + enum command_type type; /* command */
>>> + struct curl_handle *ch; /* for EASY_HANDLE, the easy handle */
>>> +
>>> + /* This field is set to a unique value by send_command_and_wait. */
>>> + uint64_t id; /* serial number */
>>> +
>>> + /* These fields are used to signal back that the command finished. */
>>> + pthread_mutex_t mutex;/* completion mutex */
>>> + pthread_cond_t cond; /* completion condition */
>>> + CURLcode status; /* status code (CURLE_OK = succeeded) */
>>> };
>>
>> Makes sense. The two types are mutually recursive (curl_handle
>> includes a struct command *; command includes a struct curl_handle *);
>> hopefully you have proper locking when altering multiple objects to
>> adjust how they point to one another.
>
> Actually locking is not needed. Let me document it through ...
>
> We create both the curl easy handle and the associated EASY_HANDLE
> command in the nbdkit thread that gets the request, eg. in the curl
> .pread method. That of course requires no locking.
>
> There is a single background worker thread.
>
> A "self pipe" passes pointers to 'struct command *' to this worker
> thread simple by writing the 8 byte pointer onto the pipe (hopefully
> atomic ...) The nbdkit request thread then blocks on the mutex/cond
> in the command handle.
Right, that's exactly that I got curious about.
In my opinion, writing to the self-pipe is mostly safe. Reading from the
self-pipe could be slightly polished.
Here are the POSIX pages on write() and read():
https://pubs.opengroup.org/onlinepubs/9699919799/functions/write.html
https://pubs.opengroup.org/onlinepubs/9699919799/functions/read.html
The first one (write) says that if you attempt to write at most PIPE_BUF
bytes, then writes will not be interleaved with other concurrent writes,
so in a sense the write will be atomic. (And in this case, O_NONBLOCK
only changes the behavior for the case when the buffer cannot be written
in entirety: with O_NONBLOCK clear, the writer thread will block, with
O_NONBLOCK set, the writer thread will see -1/EAGAIN.)
Now, PIPE_BUF is "variable" in a sense:
https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/limits.h.html
but it is greater than or equal to _POSIX_PIPE_BUF, which is 512. Our
pointers are 8 bytes in size, so the <=PIPE_BUF condition is surely
satisfied.
... The only case I see possible for a short write is the delivery of a
signal after transfer has started. *If* nbdkit catches some signal such
that the signal handler actually returns, then this could be a
(theoretical) problem, calling for xwrite() or similar.
>
> The background worker thread picks the 'struct command *' up from the
> pipe in the same event loop that it uses to process ongoing requests
> on the multi handle. It takes the easy handle and adds it to the
> multi handle.
The spec on read() does not seem to have the same kind of
non-interleaving / "atomicity" language that write() does. The rationale
section
Re: [Libguestfs] [libnbd PATCH 6/7] golang: Improve whitespace style in generated bindings
On 7/27/23 21:22, Eric Blake wrote:
> On Thu, Jul 27, 2023 at 08:50:31PM +0200, Laszlo Ersek wrote:
>>> File "GoLang.ml", line 186, characters 11-71:
>>> 186 | pr ("\t" ^ "/* %s field is ignored unless %sSet == true.
>>> */\n")
>>>
>>>
>>> Error: This expression has type string but an expression was expected of
>>> type
>>> ('weak2 -> 'weak3 -> 'weak4, unit, string, unit) format4 =
>>>('weak2 -> 'weak3 -> 'weak4, unit, string, string, string, unit)
>>>CamlinternalFormatBasics.format6
>>
>> Sigh. :/
>>
>> "pr" is declared like this [generator/utils.mli]:
>>
>> val pr : ('a, unit, string, unit) format4 -> 'a
>>
>> The format string would normally be converted to a "format4" type, which
>> is a polymorphic type, taking one type variable ('a).
>
> You dug even deeper than me. Thanks, even if it didn't yield a
> trivial solution.
>
>> Ultimately open-coding the \t escape sequences seems the least intrusive...
>>
>>>
>>> so those are non-starters. We can get more verbose with:
>>>
>>> +pr "\t";
>>> +pr "/* %s field is ignored unless %sSet == true. */\n"
>>>fname fname;
>>>
>>> but that may not scale as nicely. Go's use of TAB does not specify
>>> what tabstop it prefers;
>>
>> well that I find unfathomable; how can any coding style mandate TABs for
>> indentation if it doesn't explain how wide a TAB should be? For example,
>> that makes "maximum line width" mostly impossible to define.
>
> https://go.dev/doc/effective_go#formatting
>
> | Some formatting details remain. Very briefly:
> |
> | Indentation
> | We use tabs for indentation and gofmt emits them by default. Use spaces
> only if you must.
> | Line length
> | Go has no line length limit. Don't worry about overflowing a punched
> card. If a line feels too long, wrap it and indent with an extra tab.
>
> and no mention of tab width. The Go community really expects you to
> use gofmt without questions.
I may be overly sensitive, but I don't appreciate the "punched card"
reference. For work, I use a 22" monitor (landscape orientation), I like
to place two text windows (columns) side by side, and my eyesight isn't
the greatest, so I use relatively large fonts. My preferred source code
width is therefore 80 characters.
Other developers work with 100-120 chars per line; in edk2 we've seen
200+ chars even. When I complain, the answer is "just buy a larger
monitor, or use two monitors". Well, I don't feel *comfortable* using
two monitors (I've tried), *or* with a super wide screen (I've also got
a 24" monitor and that one feels too wide to me already -- but some
people use 49-50" monitors!).
It's strange that the go coding style tries to control the spacing
around operators like "+", but ignores the line length question (and
throws a quasi-insult at people that work with narrow source code). If a
codebase is consistently written with a 128 chars/line limit, it might
very well pass "gofmt", but I'd be mostly incapable of working with it.
>
>>
>>> our .editorconfig suggests using a tabstop of
>>> 4 (instead of the usual 8) when reading a .go file for less visual
>>> distraction, and which matches with GoLang.ml currently using 4-space
>>> indentation.
>>>
>>> Rich, do you have any ideas on any better approach to take here?
>>>
>>>>> -pr " %sSet bool\n" fname;
>>>>> -pr " %s " fname;
>>>>> +pr "\t%sSet bool\n" fname;
>>>>> +pr "\t%s" fname;
>>>
>>> I also debated about splitting this patch into two (yes, more
>>> gruntwork): one to address space before '(', and the other to address
>>> leading indentation. Lines like this would be touched by both
>>> patches if I split.
>>>
>>
>> I wouldn't insist on such a split. :)
>
> The benefit of such a split: changing "int (r)" to "int(r)" is
> non-controversial, while changing "line" to "\tline" could be reverted
> if we find a cleaner way to get pr to do indentation on our behalf.
>
> There's also the idea that if the main thing that gofmt still
> complains about is 4 spaces vs. TAB, we could use coreutils'
> unexpand(1) on platforms w
Re: [Libguestfs] [libnbd PATCH 4/7] Revert "generator/Go.ml: Simplify copy_uint32_array"
On 7/27/23 19:36, Eric Blake wrote:
> On Thu, Jul 27, 2023 at 01:23:24PM +0200, Laszlo Ersek wrote:
>> On 7/26/23 19:50, Eric Blake wrote:
>>> This reverts commit 6725fa0e129f9a60d7b89707ef8604e0aeeeaf43, although
>>> with a more modern style.
>>>
>>> Casting a C array to a Go slice just to benefit from potential
>>> optimizations in Go's copy(), is rather complex to understand,
>>> especially when we are still copying things (the main reason to treat
>>> a C array as a Go slice is when avoiding a copy has a benefit).
>>> Without a benchmark showing measurable difference in runtime speed,
>>> and considering that network transit time probably dominates the time
>>> spent on block status and its callback, it is not worth the
>>> complexity. Furthermore, an upcoming patch wants to add a similar
>>> helper function for converting between a list of C and Go structs,
>>> where the copy() trick will not work; and having the two helpers look
>>> alike is beneficial.
>>>
>>> Suggested-by: Laszlo Ersek
>>
>> I've needed to dig a bit, but indeed, bullet (8) in:
>>
>> 0e4ff751-88d6-837b-15a5-6f6c370a2f09@redhat.com">http://mid.mail-archive.com/0e4ff751-88d6-837b-15a5-6f6c370a2f09@redhat.com
>> https://listman.redhat.com/archives/libguestfs/2023-June/031736.html
>>
>>> CC: Nir Soffer
>>> Signed-off-by: Eric Blake
>>> ---
>>> generator/GoLang.ml | 14 --
>>> 1 file changed, 8 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/generator/GoLang.ml b/generator/GoLang.ml
>>> index 0aa83bdc..77dacadb 100644
>>> --- a/generator/GoLang.ml
>>> +++ b/generator/GoLang.ml
>>> @@ -509,12 +509,14 @@ let
>>>
>>> /* Closures. */
>>>
>>> -func copy_uint32_array (entries *C.uint32_t, count C.size_t) []uint32 {
>>> -ret := make([]uint32, int (count))
>>> -// See
>>> https://github.com/golang/go/wiki/cgo#turning-c-arrays-into-go-slices
>>> -// TODO: Use unsafe.Slice() when we require Go 1.17.
>>> -s := (*[1<<30]uint32)(unsafe.Pointer(entries))[:count:count]
>>> -copy(ret, s)
>>> +func copy_uint32_array(entries *C.uint32_t, count C.size_t) []uint32 {
>>> +ret := make([]uint32, int(count))
>>> +addr := uintptr(unsafe.Pointer(entries))
>>> +for i := 0; i < int(count); i++ {
>>> +ptr := (*C.uint32_t)(unsafe.Pointer(addr))
>>> +ret[i] = uint32(*ptr)
>>> +addr += unsafe.Sizeof(*ptr)
>>> +}
>>> return ret
>>> }
>>> ";
>>
>> This patch mixes four things:
>>
>> - whitespace changes (due to style modernization / gofmt, presumably),
>> - reverting commit 6725fa0e129f,
>> - changes proposed in my email,
>> - functional changes on top of my email.
>>
>> The "func" line matches my proposal (OK), with additional whitespace updates
>> (OK), but has nothing to do with reverting 6725fa0e129f, so calling the
>> patch a "revert" is misleading.
>
> Fair enough. It is undoing the effects of the earlier patch, but not
> in the same way as a straight revert (in part because enough else has
> changed in the meantime that a straight revert is no longer possible).
> Plus the decision on where to stage this in the series (at one point,
> I had it after 6/7 - and you already noted there how much rebase churn
> gets caused as we mix this series around).
>
>>
>> The initialization of "ret" undergoes a whitespace update (OK), but is
>> neither a revert (6725fa0e129f did not change the initialization of "ret"),
>> nor does it match my proposal. In my proposal, I had removed the "int" cast
>> (or conversion) intentionally. Casting a C size_t to a Go int seems wrong.
>> (IIRC I had verified the widths of the Go integer types from the Go
>> documentation.)
>
> I completely missed that point. You do make a valid point that a C
> size_t might be larger than a Go int (does Go even try to run on
> 32-bit platforms?) - but we DO have the additional knowledge that
> because our block status reply results cannot exceed 64M over the
> wire, any count passed to the callback function will fit in 32 bits
> regardless of the width of C's size_t. Maybe an assertion that count
> does not lose precision when cast to Go int is sufficient?
Sure, an assertion (with explanation) would be fine.
>
>>
>> The initialization of "addr" matches my proposal, with some whi
Re: [Libguestfs] [libnbd PATCH 6/7] golang: Improve whitespace style in generated bindings
On 7/27/23 19:27, Eric Blake wrote:
> On Thu, Jul 27, 2023 at 01:54:25PM +0200, Laszlo Ersek wrote:
>> On 7/26/23 19:50, Eric Blake wrote:
>>> Use Go-preferred whitespace: TAB for indent, and no space before
>>> opening '(', better use of blank lines. These changes reduce the size
>>> of a diff produced by gofmt, although there are still other places
>>> where we are not idiomatic in what we generate (mainly in lining up
>>> columns of = in const() blocks).
>>>
>>> Signed-off-by: Eric Blake
>>> ---
>>> generator/GoLang.ml | 244 ++--
>>> 1 file changed, 123 insertions(+), 121 deletions(-)
>>
>> Unfortunately:
>>
>> - This must have been an incredible amount of work for you.
>
> Some automation involved (emacs macros are nice), but yeah, it's
> grunt-work.
>
>>
>> - The many \t escape sequences make the generator source code harder to
>> read.
>
> Agreed. Maybe a better approach would be figuring out an optional
> argument to pr to insist on TAB instead of space indent? Or, as
> mentioned elsewhere, not bothering with TAB at all in OCaml but
> instead figuring out how to run gofmt itself as part of the generation
> pipeline.
>
>>
>> - This patch depends on patch#4 (minimally), so if you modify that one
>> like I've asked, it will create a conflict for this patch. :(
>
> Yep, I've already encountered a number of conflicts as I shuffle
> around patches, trying to pick out which changes are more than just
> whitespace (such as the RBool change). Not terribly difficult, but
> back in the grunt-work category.
>
>>
>> Acked-by: Laszlo Ersek
>>
>> Thanks
>> Laszlo
>>
>>>
>>> diff --git a/generator/GoLang.ml b/generator/GoLang.ml
>>> index 77dacadb..a38aa19f 100644
>>> --- a/generator/GoLang.ml
>>> +++ b/generator/GoLang.ml
>>> @@ -175,6 +175,7 @@ let
>>> let print_binding (name, { args; optargs; ret; shortdesc }) =
>>>let cname = camel_case name in
>>>
>>> + pr "\n";
>>>(* Tedious method of passing optional arguments in golang. *)
>>>if optargs <> [] then (
>>> pr "/* Struct carrying optional arguments for %s. */\n" cname;
>>> @@ -182,10 +183,10 @@ let
>>> List.iter (
>>>fun optarg ->
>>> let fname = go_name_of_optarg optarg in
>>> -pr " /* %s field is ignored unless %sSet == true. */\n"
>>> +pr "\t/* %s field is ignored unless %sSet == true. */\n"
>>>fname fname;
>
> I tried:
>
> +pr "\t" ^ "/* %s field is ignored unless %sSet == true. */\n"
>fname fname;
>
> File "GoLang.ml", line 186, characters 8-15:
> 186 | pr "\t" ^ "/* %s field is ignored unless %sSet == true. */\n"
> ^^^
> Error: This expression has type unit but an expression was expected of type
> string
>
> and:
>
> File "GoLang.ml", line 186, characters 11-71:
> 186 | pr ("\t" ^ "/* %s field is ignored unless %sSet == true. */\n")
>
> Error: This expression has type string but an expression was expected of type
> ('weak2 -> 'weak3 -> 'weak4, unit, string, unit) format4 =
>('weak2 -> 'weak3 -> 'weak4, unit, string, string, string, unit)
>CamlinternalFormatBasics.format6
Sigh. :/
"pr" is declared like this [generator/utils.mli]:
val pr : ('a, unit, string, unit) format4 -> 'a
The format string would normally be converted to a "format4" type, which
is a polymorphic type, taking one type variable ('a).
It seems that this automatic conversion/construction, from string to
format4, is performed by a mechanism that is similar to
"Stdlib.format_of_string":
https://v2.ocaml.org/api/Stdlib.html#1_Operationsonformatstrings
There are two catches however:
- Stdlib.format_of_string only works with string *literals*, according
to the documentation,
- Stdlib.format_of_string produces a format6, not a format4.
The documentation recommends "Scanf.format_from_string":
https://v2.ocaml.org/api/Scanf.html#VALformat_from_string
so you could *theoretically* do something like
printf (Scanf.format_from_string ("%d " ^ "%d\n")) 1 2
it still doesn't work though, because Scanf.format_from_string also
produces a format6 (like Stdlib.format_of_string does), but printf
e
Re: [Libguestfs] [libnbd PATCH 1/7] api: Expose "qemu:" meta-context constants
On 7/27/23 17:03, Laszlo Ersek wrote: > On 7/27/23 16:33, Eric Blake wrote: >> On Thu, Jul 27, 2023 at 01:41:03PM +0200, Laszlo Ersek wrote: >>> On 7/26/23 19:29, Eric Blake wrote: >>>> Qemu has a couple of documented meta-context definitions[1]; we might >>>> as well expose constants for these namespaces. >>>> [1] https://gitlab.com/qemu-project/qemu/-/blob/master/docs/interop/nbd.txt > FWIW, the qemu meta-context namespace being registered in the spec has > warmed me up to this patch! Can you perhaps add reference [2] to the commit message: [2] https://github.com/NetworkBlockDevice/nbd/commit/e96b311caf901 (The link introduced in that commit has been updated since, by commit 81eeeb55d8f1, "proto: update stale link to qemu's nbd.txt", 2023-03-14; but e96b311caf901 was the commit that performed the actual registration.) Thanks! Laszlo ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [libnbd PATCH 1/7] api: Expose "qemu:" meta-context constants
On 7/27/23 16:33, Eric Blake wrote: > On Thu, Jul 27, 2023 at 01:41:03PM +0200, Laszlo Ersek wrote: >> On 7/26/23 19:29, Eric Blake wrote: >>> Qemu has a couple of documented meta-context definitions[1]; we might >>> as well expose constants for these namespaces. >>> >>> "qemu:dirty-bitmap:NAME" is a set of namespaces for any arbitrary >>> dirty bitmap name; we can't define constants for every bitspace name, >>> but it is possible to do NBD_OPT_LIST_META_CONTEXT on >>> "qemu:dirty-bitmap:" to see which dirty bitmaps are available. When a >>> dirty bitmap is negotiated, only one bit is defined (an extent is >>> dirty or not). The presence of '-' and ':' in the context name beyond >>> the namespace requires a new helper function. >>> >>> "qemu:allocation-depth" returns an integer rather than a bitmap (0 for >>> unmapped, 1 for current image, 2 and beyond for number of files in the >>> backing chain before the data was supplied), so we can't really define >>> any constants for interpreting its values. >>> >>> [1] https://gitlab.com/qemu-project/qemu/-/blob/master/docs/interop/nbd.txt >>> >>> For libnbd.h, the generated diff is: >>> >>> | --- include/libnbd.h.bak 2023-07-26 11:01:45.401328604 -0500 >>> | +++ include/libnbd.h 2023-07-26 11:59:38.289021067 -0500 >>> | @@ -1083,6 +1083,16 @@ >>> | #define LIBNBD_STATE_HOLE 1 >>> | #define LIBNBD_STATE_ZERO 2 >>> | >>> | +/* "qemu" namespace */ >>> | +#define LIBNBD_NAMESPACE_QEMU "qemu:" >>> | + >>> | +/* "qemu" namespace contexts */ >>> | +#define LIBNBD_CONTEXT_QEMU_DIRTY_BITMAP "qemu:dirty-bitmap:" >>> | +#define LIBNBD_CONTEXT_QEMU_ALLOCATION_DEPTH "qemu:allocation-depth" >>> | + >>> | +/* "qemu:dirty-bitmap:" context related constants */ >>> | +#define LIBNBD_STATE_DIRTY1 >>> | + >>> | #ifdef __cplusplus >>> | } >>> | #endif >>> >>> Signed-off-by: Eric Blake >>> --- >>> generator/utils.mli| 1 + >>> generator/API.ml | 9 ++--- >>> generator/C.ml | 22 +++--- >>> generator/GoLang.ml| 3 ++- >>> generator/OCaml.ml | 4 ++-- >>> generator/Python.ml| 3 ++- >>> generator/utils.ml | 5 + >>> interop/dirty-bitmap.c | 6 -- >>> 8 files changed, 37 insertions(+), 16 deletions(-) >> >> I'm not really convinced this is helpful. >> >> - Libnbd is not supposed to be tied to any particular NBD server AIUI, >> so open-coding QEMU-specific constants in the library header (for which >> we promise stability, in C) seems unneeded and risky. > > I'm of the opinion that if any project declares their own namespace of > extensions, and then documents that declaration in the upstream NBD > spec (which qemu has done: [1]), then libnbd might as well try and > make it easier to probe for the existence of that extension. > > [1] > https://github.com/NetworkBlockDevice/nbd/blob/58b356bd19e63/doc/proto.md?plain=1#L963 Good point! I didn't realize the QEMU extensions were (effectively) standardized in the spec. That makes my first point moot. (More importantly -- I didn't realize there was a *registration process* in place! That's a great thing to have.) > >> >> (I do see the last hunk for the interop/ directory -- I'm unsure what >> that directory is for.) > > That provides interoperability tests of what libnbd does when talking > to various servers (some of the tests are repeated across nbdkit, > qemu, and nbd-server; some are specific to features of a given > server). Altering the test to actually use the constants defined by > this patch ensures that we have API stability for those constants. Yes. > >> >> - In the other direction, we don't implement the whole story (for >> "qemu:allocation-depth"). In theory, we could introduce symbolic >> constants for 0 and 1, such as LIBNBD_STATE_UNMAPPED and >> LIBNBD_STATE_CURRENT (and client code could use ">LIBNBD_STATE_CURRENT", >> i.e., ">1", equivalently to ">=2"). > > Yes, we could indeed add at least 0 and 1 as constants for > qemu:allocation-depth, if we wanted. I'm not sure how to go about > documenting them, though; the point is that base:allocation uses flags > as a bitmap, but qemu:allocation-depth uses flags as an integer
Re: [Libguestfs] [libnbd PATCH 7/7] golang: Enforce coding style during 'make check'
On 7/26/23 19:50, Eric Blake wrote:
> Now that I've finished tweaking the generator to output consistent Go
> style, add a test that runs gofmt to flag places where we introduce
> style regressions. As lining up columns in generated const() blocks
> is trickier, for now I am making the test skip that by default (export
> TEST_GOFMT_ALL=1 to see the difference).
>
> A later patch may figure out how to do it in OCaml (two passes: one to
> collect the maximum length of a name, the second to output columnar
> data), or to include gofmt as part of the generation process (when
> available), where a 'make dist' tarball will compile no matter what,
> but only have correct formatting if the developer building the tarball
> had gofmt installed.
>
> Signed-off-by: Eric Blake
> ---
> golang/Makefile.am| 2 +-
> golang/codestyle-tests.sh | 45 +++
> 2 files changed, 46 insertions(+), 1 deletion(-)
> create mode 100755 golang/codestyle-tests.sh
>
> diff --git a/golang/Makefile.am b/golang/Makefile.am
> index fac65248..9201ed8e 100644
> --- a/golang/Makefile.am
> +++ b/golang/Makefile.am
> @@ -98,7 +98,7 @@ TESTS_ENVIRONMENT = \
> abs_top_srcdir=$(abs_top_srcdir) \
> $(NULL)
> LOG_COMPILER = $(top_builddir)/run
> -TESTS = run-tests.sh
> +TESTS = run-tests.sh codestyle-tests.sh
>
> endif
>
> diff --git a/golang/codestyle-tests.sh b/golang/codestyle-tests.sh
> new file mode 100755
> index ..f4928fe5
> --- /dev/null
> +++ b/golang/codestyle-tests.sh
> @@ -0,0 +1,45 @@
> +#!/bin/bash -
> +# nbd client library in userspace
> +# Copyright Red Hat
> +#
> +# This library is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU Lesser General Public
> +# License as published by the Free Software Foundation; either
> +# version 2 of the License, or (at your option) any later version.
> +#
> +# This library is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> +# Lesser General Public License for more details.
> +#
> +# You should have received a copy of the GNU Lesser General Public
> +# License along with this library; if not, write to the Free Software
> +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301
> USA
> +
> +. ../tests/functions.sh
> +
> +set -e
> +set -x
> +
> +# Assume that 'gofmt' lives in the same place as 'go'
> +GOFMT=${GOLANG}fmt
> +
> +requires $GOFMT --help
> +
> +rm -f codestyle-tests.out
> +cleanup_fn rm -f codestyle-tests.out
> +
> +# Lining up generated = in const() in bindings.go is hard; to check
> +# that file, export TEST_GOFMT_ALL=1 while running this test.
> +if test x"$TEST_GOFMT_ALL" = x; then
> +exclude='-not -name bindings.go'
> +else
> +exclude=
> +fi
> +
> +$GOFMT -d $(find . -name "*.go" $exclude) > codestyle-tests.out
> +if test -s codestyle-tests.out; then
> +echo 'FAIL: fix the following style errors' >&2
> +cat codestyle-tests.out >&2
> +exit 1
> +fi
Looks good to me.
Reviewed-by: Laszlo Ersek
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [libnbd PATCH 5/7] golang: Use 'gofmt' style recommendations on manual files
_structured_test.go
> @@ -26,7 +26,7 @@
> var expected405 = make([]byte, 512)
>
> func psf(user_data int, buf2 []byte, offset uint64, status uint,
> - error *int) int {
> + error *int) int {
> if user_data != 42 {
> panic("expected user_data == 42")
> }
> @@ -68,10 +68,10 @@ func Test405PReadStructured(t *testing.T) {
>
> buf := make([]byte, 512)
> err = h.PreadStructured(buf, 0,
> - func(buf2 []byte, offset uint64, status uint,
> - error *int) int {
> - return psf(42, buf2, offset, status, error)
> - }, nil)
> + func(buf2 []byte, offset uint64, status uint,
> + error *int) int {
> + return psf(42, buf2, offset, status, error)
> + }, nil)
> if err != nil {
> t.Fatalf("%s", err)
> }
> diff --git a/golang/libnbd_590_aio_copy_test.go
> b/golang/libnbd_590_aio_copy_test.go
> index c22653a7..e8c32dc0 100644
> --- a/golang/libnbd_590_aio_copy_test.go
> +++ b/golang/libnbd_590_aio_copy_test.go
> @@ -28,8 +28,11 @@
> var bytes_read = uint(0)
> var bytes_written = uint(0)
>
> -/* Functions to handle FdSet.
> - XXX These probably only work on 64 bit platforms. */
> +/*
> +Functions to handle FdSet.
> +
> + XXX These probably only work on 64 bit platforms.
> +*/
> func fdset_set(set *syscall.FdSet, fd int) {
> (*set).Bits[fd/64] |= 1 << (uintptr(fd) % 64)
> }
> @@ -56,8 +59,11 @@ type wbuf struct {
>
> var writes []wbuf
>
> -/* Called whenever any asynchronous pread command from
> - the source has completed. */
> +/*
> +Called whenever any asynchronous pread command from
> +
> + the source has completed.
> +*/
> func read_completed(buf AioBuffer, offset uint64) int {
> bytes_read += buf.Size
> /* Move the AIO buffer to the write queue. */
> @@ -66,8 +72,11 @@ func read_completed(buf AioBuffer, offset uint64) int {
> return 1
> }
>
> -/* Called whenever any asynchronous pwrite command to the
> - destination has completed. */
> +/*
> +Called whenever any asynchronous pwrite command to the
> +
> + destination has completed.
> +*/
> func write_completed(buf AioBuffer) int {
> bytes_written += buf.Size
> /* Now we have to manually free the AIO buffer. */
> diff --git a/golang/libnbd_620_stats_test.go b/golang/libnbd_620_stats_test.go
> index 7b4ba009..63667160 100644
> --- a/golang/libnbd_620_stats_test.go
> +++ b/golang/libnbd_620_stats_test.go
> @@ -124,16 +124,16 @@ func Test620Stats(t *testing.T) {
> t.Fatalf("%s", err)
> }
>
> - if bs2 != bs1 + 28 {
> + if bs2 != bs1+28 {
> t.Fatalf("unexpected value for bs2")
> }
> - if cs2 != cs1 + 1 {
> + if cs2 != cs1+1 {
> t.Fatalf("unexpected value for cs2")
> }
> - if br2 != br1 + 16 { /* assumes nbdkit uses simple reply */
> + if br2 != br1+16 { /* assumes nbdkit uses simple reply */
> t.Fatalf("unexpected value for br2")
> }
> - if cr2 != cr1 + 1 {
> + if cr2 != cr1+1 {
> t.Fatalf("unexpected value for cr2")
> }
>
> @@ -169,13 +169,13 @@ func Test620Stats(t *testing.T) {
> if bs3 <= bs2 {
> t.Fatalf("unexpected value for bs3")
> }
> - if cs3 != cs2 + 1 {
> + if cs3 != cs2+1 {
> t.Fatalf("unexpected value for cs3")
> }
> if br3 < br2 {
> t.Fatalf("unexpected value for br3")
> }
> - if cr3 != cr2 + slop {
> + if cr3 != cr2+slop {
> t.Fatalf("unexpected value for cr3")
> }
> }
I've done nearly nothing in Go, and generally the updates look
justified... but the changes to "libnbd_620_stats_test.go" are hideous.
Who on Earth considers
if cr3 != cr2+slop
superior to
if cr3 != cr2 + slop
???
*shudder*
Question in passing: if we wrote
if cr3 != (cr2 + slop)
would gofmt contract that too to
if cr3 != (cr2+slop)
?
Asking because I suspect that gofmt's purpose with the whitespace
removal around "+" is to emphasize that "+" binds more strongly than
"!=". And, if we forced the binding with () around +, then gofmt might
not feel the need to emphasize the difference between the binding strengths.
Acked-by: Laszlo Ersek
Laszlo
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [libnbd PATCH 2/7] golang: Export meta-context constants
On 7/26/23 19:29, Eric Blake wrote:
> Go insists that a module's members are not public unless they start
> with a capital letter. Our attempt at defining constants
> 'namespace_base' and 'context_base_allocation' were thus only visible
> to our in-package tests, and not for external clients.
>
> Although this patch does not completely address gofmt complaints,
> adding some strategic grouping comments inside the const() block has
> an additional benefit of reducing the amount of whitespace that gofmt
> wants to inject into unrelated lines.
>
> The change to generated golang/bindigs.go is:
>
> | --- golang/bindings.go.bak 2023-07-26 12:06:47.618324437 -0500
> | +++ golang/bindings.go 2023-07-26 12:06:55.521385295 -0500
> | @@ -111,14 +111,18 @@
> | READ_DATA uint32 = 1
> | READ_HOLE uint32 = 2
> | READ_ERROR uint32 = 3
> | -namespace_base = "base:"
> | -context_base_allocation = "base:allocation"
> | +/* Meta-context namespace "base" */
> | +NAMESPACE_BASE = "base:"
> | +CONTEXT_BASE_ALLOCATION = "base:allocation"
> | +/* Defined bits in "base:allocation" */
> | STATE_HOLE uint32 = 1
> | STATE_ZERO uint32 = 2
> | -namespace_qemu = "qemu:"
> | -context_qemu_dirty_bitmap = "qemu:dirty-bitmap:"
> | +/* Meta-context namespace "qemu" */
> | +NAMESPACE_QEMU = "qemu:"
> | +CONTEXT_QEMU_DIRTY_BITMAP = "qemu:dirty-bitmap:"
> | +/* Defined bits in "qemu:dirty-bitmap:" */
> | STATE_DIRTY uint32 = 1
> | -context_qemu_allocation_depth = "qemu:allocation-depth"
> | +CONTEXT_QEMU_ALLOCATION_DEPTH = "qemu:allocation-depth"
> | )
> |
> | /* SetDebug: set or clear the debug flag */
>
> Signed-off-by: Eric Blake
>
> ---
>
> RFC: Do we want MixedCase rather than ALL_CAPS names? If so, this
> patch needs to be rethought a bit, as it would impact API for existing
> public constants as well as the ones changed here. But we don't
> promise API stability for Go.
I think all caps should be fine.
The "golang/bindigs.go" quote in the commit message would have to change
if you decided to drop patch#1.
Acked-by: Laszlo Ersek
Laszlo
> ---
> generator/GoLang.ml | 10 +---
> golang/libnbd_230_opt_info_test.go| 24 +--
> golang/libnbd_240_opt_list_meta_test.go | 14 +--
> .../libnbd_245_opt_list_meta_queries_test.go | 4 ++--
> golang/libnbd_250_opt_set_meta_test.go| 20
> .../libnbd_255_opt_set_meta_queries_test.go | 12 +-
> golang/libnbd_460_block_status_test.go| 4 ++--
> 7 files changed, 46 insertions(+), 42 deletions(-)
>
> diff --git a/generator/GoLang.ml b/generator/GoLang.ml
> index 7a7e7f4b..82d73ed6 100644
> --- a/generator/GoLang.ml
> +++ b/generator/GoLang.ml
> @@ -469,11 +469,15 @@ let
>) constants;
>List.iter (
> fun (ns, ctxts) ->
> - pr "namespace_%s = \"%s:\"\n" ns ns;
> + let ns_upper = String.uppercase_ascii ns in
> + pr "/* Meta-context namespace \"%s\" */\n" ns;
> + pr "NAMESPACE_%s = \"%s:\"\n" ns_upper ns;
>List.iter (
> fun (ctxt, consts) ->
> - let ctxt_macro = macro_name ctxt in
> - pr "context_%s_%s = \"%s:%s\"\n" ns ctxt_macro ns ctxt;
> + let ctxt_macro = String.uppercase_ascii (macro_name ctxt) in
> + pr "CONTEXT_%s_%s = \"%s:%s\"\n" ns_upper ctxt_macro ns ctxt;
> + if consts <> [] then
> +pr "/* Defined bits in \"%s:%s\" */\n" ns ctxt;
>List.iter (fun (n, v) ->
>pr "%s uint32 = %d\n" n v
>) consts
> diff --git a/golang/libnbd_230_opt_info_test.go
> b/golang/libnbd_230_opt_info_test.go
> index cd00be9e..cc7c2d91 100644
> --- a/golang/libnbd_230_opt_info_test.go
> +++ b/golang/libnbd_230_opt_info_test.go
> @@ -45,7 +45,7 @@ func Test230OptInfo(t *testing.T) {
> t.Fatalf("could not connect: %s", err)
> }
>
> - err = h.AddMetaContext(context_base_allocation)
> + err = h.AddMetaContext(CONTEXT_BASE_ALLOCATION)
> if err != nil {
> t.Fatalf("could not add meta context: %s", err)
> }
> @@ -59,7 +59,7 @@ func Test230OptInfo(t *testing.T) {
> if err == nil {
> t.Fatalf("expected error")
> }
> - _, err = h.CanMetaContext(co
Re: [Libguestfs] [libnbd PATCH v4 4/6] states: Prepare to send 64-bit requests
are currently limited by the 32 bit field in the
> - * command structure on the wire, but in future we hope to support
> - * 64 bit values here with a change to the NBD protocol which is
> - * being discussed upstream.
> +/* Other commands are limited by the 32 bit field in the command
> + * structure on the wire, unless extended headers were negotiated.
> */
>default:
> -if (count > UINT32_MAX) {
> +if (!h->extended_headers && count > UINT32_MAX) {
>set_error (ERANGE, "request too large: maximum request size is %"
> PRIu32,
> UINT32_MAX);
>goto err;
> @@ -358,6 +356,15 @@ nbd_unlocked_aio_pwrite (struct nbd_handle *h, const
> void *buf,
>return -1;
> }
>}
> + /* It is more convenient to manage PAYLOAD_LEN by what was negotiated
> + * than to require the user to have to set it correctly.
> + * TODO: Add new h->strict bit to allow intentional protocol violation
> + * for interoperability testing.
> + */
> + if (h->extended_headers)
> +flags |= LIBNBD_CMD_FLAG_PAYLOAD_LEN;
> + else
> +flags &= ~LIBNBD_CMD_FLAG_PAYLOAD_LEN;
>
>SET_CALLBACK_TO_NULL (*completion);
>return nbd_internal_command_common (h, flags, NBD_CMD_WRITE, offset, count,
> diff --git a/tests/Makefile.am b/tests/Makefile.am
> index 6eddcb7a..52fadd9c 100644
> --- a/tests/Makefile.am
> +++ b/tests/Makefile.am
> @@ -233,6 +233,7 @@ check_PROGRAMS += \
> closure-lifetimes \
> pread-initialize \
> socket-activation-name \
> + pwrite-extended \
> $(NULL)
>
> TESTS += \
> @@ -655,6 +656,9 @@ socket_activation_name_SOURCES = \
> requires.h
> socket_activation_name_LDADD = $(top_builddir)/lib/libnbd.la
>
> +pwrite_extended_SOURCES = pwrite-extended.c
> +pwrite_extended_LDADD = $(top_builddir)/lib/libnbd.la
> +
> #--
> # Testing TLS support.
>
> diff --git a/tests/pwrite-extended.c b/tests/pwrite-extended.c
> new file mode 100644
> index ..c6f4dd29
> --- /dev/null
> +++ b/tests/pwrite-extended.c
> @@ -0,0 +1,108 @@
> +/* NBD client library in userspace
> + * Copyright Red Hat
> + *
> + * This library is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2 of the License, or (at your option) any later version.
> + *
> + * This library is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with this library; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301
> USA
> + */
> +
> +/* Check behavior of pwrite with PAYLOAD_LEN flag for extended headers. */
> +
> +#include
> +
> +#include
> +#include
> +#include
> +#include
> +#include
> +#include
> +
> +#include
> +
> +static char *progname;
> +static char buf[512];
> +
> +static void
> +check (int experr, const char *prefix)
> +{
> + const char *msg = nbd_get_error ();
> + int errnum = nbd_get_errno ();
> +
> + fprintf (stderr, "error: \"%s\"\n", msg);
> + fprintf (stderr, "errno: %d (%s)\n", errnum, strerror (errnum));
> + if (strncmp (msg, prefix, strlen (prefix)) != 0) {
> +fprintf (stderr, "%s: test failed: missing context prefix: %s\n",
> + progname, msg);
> +exit (EXIT_FAILURE);
> + }
> + if (errnum != experr) {
> +fprintf (stderr, "%s: test failed: "
> + "expected errno = %d (%s), but got %d\n",
> + progname, experr, strerror (experr), errnum);
> +exit (EXIT_FAILURE);
> + }
> +}
> +
> +int
> +main (int argc, char *argv[])
> +{
> + struct nbd_handle *nbd;
> + const char *cmd[] = {
> +"nbdkit", "-s", "-v", "--exit-with-parent", "memory", "1048576", NULL
> + };
> + uint32_t strict;
> +
> + progname = argv[0];
> +
> + nbd = nbd_create ();
> + if (nbd == NULL) {
> +fprintf (stderr, "%s: %s\n", progname, nbd_get_error ());
> +exit (EXIT_FAILURE);
> + }
> +
> + /* Connect to the server. */
> + if (nbd_connect_command (nbd, (char **)cmd) == -1) {
> +fprintf (stderr, "%s: %s\n", progname, nbd_get_error ());
> +exit (EXIT_FAILURE);
> + }
> +
> + /* FIXME: future API addition to test if server negotiated extended mode.
> + * Until then, strict flags must ignore the PAYLOAD_LEN flag for pwrite,
> + * even though it is rejected for other commands.
> + */
> + strict = nbd_get_strict_mode (nbd);
> + if (!(strict & LIBNBD_STRICT_FLAGS)) {
> +fprintf (stderr, "%s: test failed: "
> + "nbd_get_strict_mode did not have expected flag set\n",
> + progname);
> +exit (EXIT_FAILURE);
> + }
> + if (nbd_aio_pread (nbd, buf, 512, 0, NBD_NULL_COMPLETION,
> + LIBNBD_CMD_FLAG_PAYLOAD_LEN) != -1) {
> +fprintf (stderr, "%s: test failed: "
> + "nbd_aio_pread did not fail with unexpected flag\n",
> + progname);
> +exit (EXIT_FAILURE);
> + }
> + check (EINVAL, "nbd_aio_pread: ");
> +
> + if (nbd_aio_pwrite (nbd, buf, 512, 0, NBD_NULL_COMPLETION,
> + LIBNBD_CMD_FLAG_PAYLOAD_LEN) == -1 ||
> + nbd_aio_pwrite (nbd, buf, 512, 0, NBD_NULL_COMPLETION, 0) == -1) {
> +fprintf (stderr, "%s: %s\n", progname, nbd_get_error ());
> +exit (EXIT_FAILURE);
> + }
> +
> + nbd_close (nbd);
> + exit (EXIT_SUCCESS);
> +}
> diff --git a/.gitignore b/.gitignore
> index b43e83c0..e5304f16 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -252,6 +252,7 @@ Makefile.in
> /tests/pki/
> /tests/pread-initialize
> /tests/private-data
> +/tests/pwrite-extended
> /tests/read-only-flag
> /tests/read-write-flag
> /tests/server-death
Acked-by: Laszlo Ersek
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [libnbd PATCH v4 6/6] states: Break deadlock if server goofs on extended replies
On 7/21/23 18:08, Eric Blake wrote:
> One of the benefits of extended replies is that we can do a
> fixed-length read for the entire header of every server reply, which
> is fewer syscalls than the split-read approach required by structured
> replies. But one of the drawbacks of doing a large read is that if
> the server is non-compliant (not a problem for normal servers, but
> something I hit rather more than I'd like to admit while developing
> extended header support in servers), nbd_pwrite() and friends will
> deadlock if the server replies with the wrong header. Add in some
> code to catch that failure mode and move the state machine to DEAD
> sooner, to make it easier to diagnose the fault in the server.
>
> Unlike in the case of an unexpected simple reply from a structured
> server (where, since b31e7bac, we can merely fail the command with
> EPROTO and successfully move on to the next server reply, because we
> didn't read too many bytes yet), in this case we really do have to
> move to DEAD: we cannot assume our short read was just the 16 (simple)
> or 20 (structured) bytes that the server sent for this command,
> because it is also possible that we can see a short read even while
> reading two back-to-back replies; if we have already read the initial
> bytes of the server's next reply, we have no way to push those extra
> bytes back onto our read stream for parsing on our next pass through
> the state machine.
>
> Signed-off-by: Eric Blake
> ---
>
> v4: rebase to master, improve comment accuracy [Laszlo], drop bogus #if 0
> ---
> generator/states-reply.c | 20 +++-
> 1 file changed, 19 insertions(+), 1 deletion(-)
>
> diff --git a/generator/states-reply.c b/generator/states-reply.c
> index 05301ae8..9f0918e2 100644
> --- a/generator/states-reply.c
> +++ b/generator/states-reply.c
> @@ -126,7 +126,25 @@ REPLY.START:
> REPLY.RECV_REPLY:
>switch (recv_into_rbuf (h)) {
>case -1: SET_NEXT_STATE (%.DEAD); return 0;
> - case 1: SET_NEXT_STATE (%.READY); return 0;
> + case 1: SET_NEXT_STATE (%.READY);
> +/* Special case: if we have a short read, but got at least far
> + * enough to decode the magic number, we can check if the server
> + * is matching our expectations. This lets us avoid deadlocking if
> + * we are blocked waiting for a 32-byte extended reply, while a
> + * buggy server only sent a shorter simple or structured reply.
> + * Magic number checks here must be repeated in CHECK_REPLY_MAGIC,
> + * since we do not always encounter a short read.
> + */
> +if (h->extended_headers &&
> +(char *)h->rbuf >=
> +(char *)>sbuf.reply.hdr + sizeof h->sbuf.reply.hdr.magic) {
> + uint32_t magic = be32toh (h->sbuf.reply.hdr.magic);
> + if (magic != NBD_EXTENDED_REPLY_MAGIC) {
> +SET_NEXT_STATE (%.DEAD); /* We've probably lost synchronization. */
> +set_error (0, "invalid or unexpected reply magic 0x%" PRIx32, magic);
> + }
> +}
> +return 0;
>case 0: SET_NEXT_STATE (%CHECK_REPLY_MAGIC);
>}
>return 0;
Acked-by: Laszlo Ersek
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [libnbd PATCH v4 5/6] states: Prepare to receive 64-bit replies
.offset_data) {
> -set_error (0, "invalid server reply length %" PRIu32, length);
> +set_error (0, "invalid server reply length %" PRIu64, length);
> SET_NEXT_STATE (%.DEAD);
> return 0;
>}
>
>/* Skip an unexpected structured reply, including to an unknown cookie. */
> - if (cmd == NULL || !h->structured_replies)
> + if (cmd == NULL || !h->structured_replies ||
> + (h->extended_headers && offset != cmd->offset))
> goto resync;
>h->payload_left = length;
>
> @@ -504,7 +511,8 @@ REPLY.CHUNK_REPLY.RECV_BS_ENTRIES:
> REPLY.CHUNK_REPLY.RESYNC:
>struct command *cmd = h->reply_cmd;
>uint16_t type;
> - uint32_t length;
> + uint64_t length;
> + uint64_t offset = -1;
>
>assert (h->rbuf == NULL);
>switch (recv_into_rbuf (h)) {
> @@ -524,11 +532,23 @@ REPLY.CHUNK_REPLY.RESYNC:
>return 0;
> }
> type = be16toh (h->sbuf.reply.hdr.structured.type);
> -length = be32toh (h->sbuf.reply.hdr.structured.length);
> -debug (h, "unexpected reply type %u or payload length %" PRIu32
> - " for cookie %" PRIu64 " and command %" PRIu32
> - ", this is probably a server bug",
> - type, length, cmd->cookie, cmd->type);
> +if (h->extended_headers) {
> + length = be64toh (h->sbuf.reply.hdr.extended.length);
> + offset = be64toh (h->sbuf.reply.hdr.extended.offset);
> + if (offset != cmd->offset)
> +debug (h, "unexpected reply offset %" PRIu64 " for cookie %" PRIu64
> + " and command %" PRIu32 ", this is probably a server bug",
> + length, cmd->cookie, cmd->type);
> + else
> +offset = -1;
> +}
> +else
> + length = be32toh (h->sbuf.reply.hdr.structured.length);
> +if (offset == -1)
> + debug (h, "unexpected reply type %u or payload length %" PRIu64
> + " for cookie %" PRIu64 " and command %" PRIu32
> + ", this is probably a server bug",
> + type, length, cmd->cookie, cmd->type);
> if (cmd->error == 0)
>cmd->error = EPROTO;
> SET_NEXT_STATE (%FINISH);
Acked-by: Laszlo Ersek
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [libnbd PATCH v4 1/6] internal: Track chunk payload length left
th >= 12);
> +assert (h->payload_left >= 12);
> assert (h->meta_valid);
>
> /* Need to byte-swap the entries returned, but apart from that we
> * don't validate them.
> */
> -for (i = 0; i < length/4; ++i)
> +for (i = 0; i < h->payload_left / sizeof *h->bs_entries; ++i)
>h->bs_entries[i] = be32toh (h->bs_entries[i]);
> +count = (h->payload_left / sizeof *h->bs_entries) - 1;
> +h->payload_left = 0;
>
> /* Look up the context ID. */
> context_id = h->bs_entries[0];
> @@ -443,8 +440,7 @@ REPLY.CHUNK_REPLY.RECV_BS_ENTRIES:
>
>if (CALL_CALLBACK (cmd->cb.fn.extent,
> h->meta_contexts.ptr[i].name, cmd->offset,
> - >bs_entries[1], (length-4) / 4,
> - ) == -1)
> + >bs_entries[1], count, ) == -1)
> if (cmd->error == 0)
>cmd->error = error ? error : EPROTO;
> }
> @@ -494,6 +490,7 @@ REPLY.CHUNK_REPLY.RESYNC:
> REPLY.CHUNK_REPLY.FINISH:
>uint16_t flags;
>
> + assert (h->payload_left == 0);
>flags = be16toh (h->sbuf.reply.hdr.structured.flags);
>if (flags & NBD_REPLY_FLAG_DONE) {
> SET_NEXT_STATE (%^FINISH_COMMAND);
Acked-by: Laszlo Ersek
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [libnbd PATCH v4 2/6] block_status: Refactor array storage
gt;payload_left / sizeof *h->bs_entries) - 1;
> -h->payload_left = 0;
> -
> /* Look up the context ID. */
> -context_id = h->bs_entries[0];
> +context_id = be32toh (h->sbuf.reply.payload.bs_hdr_32.context_id);
> for (i = 0; i < h->meta_contexts.len; ++i)
>if (context_id == h->meta_contexts.ptr[i].context_id)
> break;
>
> if (i < h->meta_contexts.len) {
> - /* Call the caller's extent function. */
>int error = cmd->error;
> + const char *name = h->meta_contexts.ptr[i].name;
>
> - if (CALL_CALLBACK (cmd->cb.fn.extent,
> - h->meta_contexts.ptr[i].name, cmd->offset,
> - >bs_entries[1], count, ) == -1)
> + /* Need to byte-swap the entries returned, but apart from that
> + * we don't validate them. Yes, our 32-bit public API foolishly
> + * tracks the number of uint32_t instead of block descriptors;
> + * see _block_desc_is_multiple_of_bs_entry above.
> + */
> + for (i = 0; i < h->bs_count * 2; ++i)
> +h->bs_entries[i] = be32toh (h->bs_entries[i]);
> +
> + /* Call the caller's extent function. */
> + if (CALL_CALLBACK (cmd->cb.fn.extent, name, cmd->offset,
> + h->bs_entries, h->bs_count * 2, ) == -1)
> if (cmd->error == 0)
>cmd->error = error ? error : EPROTO;
> }
Acked-by: Laszlo Ersek
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [libnbd PATCH v4 3/6] protocol: Add definitions for extended headers
On 7/21/23 18:08, Eric Blake wrote:
> Add the magic numbers and new structs necessary to implement the NBD
> protocol extension of extended headers providing 64-bit lengths. This
> corresponds to upstream nbd commits 36abf47d and a9384e2f on the
> extension-ext-header branch[1] (commit e6f3b94a for
> NBD_FLAG_BLOCK_STATUS_PAYLOAD is saved for a later patch).
>
> [1]
> https://github.com/NetworkBlockDevice/nbd/blob/extension-ext-header/doc/proto.md
>
> Signed-off-by: Eric Blake
> Reviewed-by: Laszlo Ersek
> ---
>
> v4: tweak comment on block_status_64.count [Laszlo], rebase to earlier
> struct renames (for example, nbd_block_descriptor_64 instead of
> nbd_structured_reply_block_status_hdr). Seemed obvious enough that I
> felt okay keeping Laszlo's R-b.
> ---
> lib/nbd-protocol.h | 55 ++
> 1 file changed, 46 insertions(+), 9 deletions(-)
Certainly, thanks!
Laszlo
>
> diff --git a/lib/nbd-protocol.h b/lib/nbd-protocol.h
> index 58583d1d..b5a28ae4 100644
> --- a/lib/nbd-protocol.h
> +++ b/lib/nbd-protocol.h
> @@ -124,6 +124,7 @@ struct nbd_fixed_new_option_reply {
> #define NBD_OPT_STRUCTURED_REPLY 8
> #define NBD_OPT_LIST_META_CONTEXT 9
> #define NBD_OPT_SET_META_CONTEXT 10
> +#define NBD_OPT_EXTENDED_HEADERS 11
>
> #define NBD_REP_ERR(val) (0x8000 | (val))
> #define NBD_REP_IS_ERR(val) (!!((val) & 0x8000))
> @@ -141,6 +142,7 @@ struct nbd_fixed_new_option_reply {
> #define NBD_REP_ERR_SHUTDOWN NBD_REP_ERR (7)
> #define NBD_REP_ERR_BLOCK_SIZE_REQD NBD_REP_ERR (8)
> #define NBD_REP_ERR_TOO_BIG NBD_REP_ERR (9)
> +#define NBD_REP_ERR_EXT_HEADER_REQD NBD_REP_ERR (10)
>
> #define NBD_INFO_EXPORT 0
> #define NBD_INFO_NAME1
> @@ -182,16 +184,26 @@ struct nbd_fixed_new_option_reply_meta_context {
>/* followed by a string */
> } NBD_ATTRIBUTE_PACKED;
>
> -/* Request (client -> server). */
> +/* Compact request (client -> server). */
> struct nbd_request {
>uint32_t magic; /* NBD_REQUEST_MAGIC. */
> - uint16_t flags; /* Request flags. */
> - uint16_t type;/* Request type. */
> + uint16_t flags; /* Request flags: NBD_CMD_FLAG_*. */
> + uint16_t type;/* Request type: NBD_CMD_*. */
>uint64_t cookie; /* Opaque handle. */
>uint64_t offset; /* Request offset. */
>uint32_t count; /* Request length. */
> } NBD_ATTRIBUTE_PACKED;
>
> +/* Extended request (client -> server). */
> +struct nbd_request_ext {
> + uint32_t magic; /* NBD_EXTENDED_REQUEST_MAGIC. */
> + uint16_t flags; /* Request flags: NBD_CMD_FLAG_*. */
> + uint16_t type;/* Request type: NBD_CMD_*. */
> + uint64_t cookie; /* Opaque handle. */
> + uint64_t offset; /* Request offset. */
> + uint64_t count; /* Request effect or payload length. */
> +} NBD_ATTRIBUTE_PACKED;
> +
> /* Simple reply (server -> client). */
> struct nbd_simple_reply {
>uint32_t magic; /* NBD_SIMPLE_REPLY_MAGIC. */
> @@ -208,6 +220,16 @@ struct nbd_structured_reply {
>uint32_t length; /* Length of following nbd_chunk_* payload.
> */
> } NBD_ATTRIBUTE_PACKED;
>
> +/* Extended reply (server -> client). */
> +struct nbd_extended_reply {
> + uint32_t magic; /* NBD_EXTENDED_REPLY_MAGIC. */
> + uint16_t flags; /* NBD_REPLY_FLAG_* */
> + uint16_t type;/* NBD_REPLY_TYPE_* */
> + uint64_t cookie; /* Opaque handle. */
> + uint64_t offset; /* Client's offset. */
> + uint64_t length; /* Length of following nbd_chunk_* payload.
> */
> +} NBD_ATTRIBUTE_PACKED;
> +
> struct nbd_chunk_offset_data {
>uint64_t offset; /* offset */
>/* Followed by data. */
> @@ -228,6 +250,17 @@ struct nbd_block_descriptor_32 {
>uint32_t status_flags;/* block type (hole etc) */
> } NBD_ATTRIBUTE_PACKED;
>
> +struct nbd_chunk_block_status_64 {
> + uint32_t context_id; /* metadata context ID */
> + uint32_t count; /* non-zero descriptor count */
> + /* followed by nbd_block_descriptor_64[count] extents */
> +} NBD_ATTRIBUTE_PACKED;
> +
> +struct nbd_block_descriptor_64 {
> + uint64_t length; /* length of block */
> + uint64_t status_flags;/* block type (hole etc) */
> +} NBD_ATTRIBUTE_PACKED;
> +
> struct nbd_chunk_error {
>uint32_t error; /* NBD_E* error number */
>uint16_t len; /* Length of human rea
Re: [Libguestfs] [libnbd PATCH v3 06/22] states: Break deadlock if server goofs on extended replies
On 7/21/23 14:50, Eric Blake wrote:
> On Thu, Jun 01, 2023 at 03:42:30PM +0200, Laszlo Ersek wrote:
>> On 5/25/23 15:00, Eric Blake wrote:
>>> One of the benefits of extended replies is that we can do a
>>> fixed-length read for the entire header of every server reply, which
>>> is fewer syscalls than the split-read approach required by structured
>>> replies.
>>
>> (Totally tangential comment: recvmsg() could scatter the incoming
>> traffic into non-contiguous fields of a non-packed structure. But I
>> don't know if TLS has anything similar. The current "linear receive"
>> approach is probably the least demanding of the underlying socket
>> abstractions. At the same time it requires us to use packed structs,
>> and/or multiple syscalls.)
>>
>>> But one of the drawbacks of doing a large read is that if
>>> the server is non-compliant (not a problem for normal servers, but
>>> something I hit rather more than I'd like to admit while developing
>>> extended header support in servers),
>>
>> Haha, so this is where it's coming from! :) I can totally relate.
>>
>>> nbd_pwrite() and friends will
>>> deadlock if the server replies with the wrong header. Add in some
>>> code to catch that failure mode and move the state machine to DEAD
>>> sooner, to make it easier to diagnose the fault in the server.
>>>
>>> Unlike in the case of an unexpected simply reply from a structured
>>
>> (1) s/simply/simple/
>
> Yep.
>
>>
>>> server (where we never over-read, and can therefore
>>
>> (2) At this point my English parser gets thrown off:
>>
>>> commit b31e7bac
>>> can merely fail the command with EPROTO and successfully move on to
>>> the next server reply),
>
> Fixing the parenthetical as follows:
>
> (where, since commit b31e7bac, we can merely fail the command with
> EPROTO and successfully move on to the next server reply, because we
> didn't read too many bytes yet)
>
>>
>> resync here:
>>
>>> in this case we really do have to move to
>>> DEAD: in addition to having already read the 16 or 20 bytes that the
>>> server sent in its (short) reply for this command, we may have already
>>> read the initial bytes of the server's next reply, but we have no way
>>> to push those extra bytes back onto our read stream for parsing on our
>>> next pass through the state machine.
>>>
>>> Signed-off-by: Eric Blake
>>> ---
>>> generator/states-reply.c | 23 ++-
>>> 1 file changed, 22 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/generator/states-reply.c b/generator/states-reply.c
>>> index 4e9f2dde..d4710d91 100644
>>> --- a/generator/states-reply.c
>>> +++ b/generator/states-reply.c
>>> @@ -109,7 +109,28 @@ REPLY.START:
>>> REPLY.RECV_REPLY:
>>>switch (recv_into_rbuf (h)) {
>>>case -1: SET_NEXT_STATE (%.DEAD); return 0;
>>> - case 1: SET_NEXT_STATE (%.READY); return 0;
>>> + case 1: SET_NEXT_STATE (%.READY);
>>> +/* Special case: if we have a short read, but got at least far
>>> + * enough to decode the magic number, we can check if the server
>>> + * is matching our expectations. This lets us avoid deadlocking if
>>> + * a buggy server sends only 16 bytes of a simple reply, and is
>>> + * waiting for our next command, while we are blocked waiting for
>>> + * the server to send 32 bytes of an extended reply.
>>> + */
>>
>> (4) Slight inconsistency between commit message and code comment: the
>> former mentions "20 bytes", but the latter doesn't mention "structured
>> reply".
>
> Did I miss (3)? But yes, I can improve this comment.
My mistake, probably due to over-editing! :)
Thanks
Laszlo
>
>>
>>> +if (h->extended_headers &&
>>> +(char *)h->rbuf >= (char *)>sbuf.reply.hdr.extended.flags) {
>>
>> (.) I wonder if (address-of-magic + size-of magic) might be more
>> readable / greppable. Just in case.
>>
>> Feel free to ignore.
>
> Actually, I agree that it is nicer (although a bit longer).
>
>>
>>> + uint32_t magic = be32toh (h->sbuf.reply.hdr.extended.magic);
>>> + if (magic != NBD_EXTENDED_REPLY_MAGIC) {
>>> +SET_NEXT_STATE (%.DEAD); /* We've probably lost synchronization. */
>>> +set_error (0, "invalid or unexpected reply magic 0x%" PRIx32,
>&
Re: [Libguestfs] [PATCH guestfs-tools] diff: Don't compare st_dev or st_ino fields
On 7/20/23 16:15, Richard W.M. Jones wrote:
> See comment for details.
>
> Link: https://listman.redhat.com/archives/libguestfs/2023-July/032061.html
> ---
> diff/diff.c | 16 ++--
> 1 file changed, 14 insertions(+), 2 deletions(-)
>
> diff --git a/diff/diff.c b/diff/diff.c
> index fb66a2bbd3..b6344b4ec2 100644
> --- a/diff/diff.c
> +++ b/diff/diff.c
> @@ -717,8 +717,20 @@ changed (guestfs_h *g1, struct file *file1,
> output_string ("changed:");
> #define COMPARE_STAT(n) \
> if (file1->stat->n != file2->stat->n) output_string (#n)
> -COMPARE_STAT (st_dev);
> -COMPARE_STAT (st_ino);
> +/* Comparing st_dev and st_ino is disabled for now, see the longer
> + * discussion here:
> + * https://listman.redhat.com/archives/libguestfs/2023-July/032061.html
> + * Even if we fixed the libguestfs API to do translation of this
> + * field correctly, it seems unlikely that there would ever be a
> + * meaningful difference in the st_dev or st_ino fields. We
> + * already know the fields refer to the same filename. Is it
> + * interesting that the file might have moved to a different disk?
> + * Everything else is comparing the content or direct metadata of
> + * the file, but st_dev and st_ino represent the metadata of the
> + * filesystem which is (arguably) different.
> + */
> +//COMPARE_STAT (st_dev);
> +//COMPARE_STAT (st_ino);
> COMPARE_STAT (st_mode);
> COMPARE_STAT (st_nlink);
> COMPARE_STAT (st_uid);
This seems to remain compatible with the existent virt-diff(1) manual,
which says:
https://libguestfs.org/virt-diff.1.html
> This tool will find differences in filenames, file sizes, checksums,
> extended attributes, file content and more from a virtual machine or
> disk image. However it does not look at the boot loader, unused space
> between partitions or within filesystems, "hidden" sectors and so on.
> In other words, it is not a security or forensics tool.
Acked-by: Laszlo Ersek
Laszlo
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [libguestfs PATCH v2 0/7] lib: support networking with passt
On 7/14/23 15:32, Richard W.M. Jones wrote:
> On Fri, Jul 14, 2023 at 03:22:06PM +0200, Laszlo Ersek wrote:
>> Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967
>> v1: https://listman.redhat.com/archives/libguestfs/2023-July/031984.html
>>
>> V2 implements small updates; the cumulative v1->v2 diff is just
>>
>>> diff --git a/lib/launch-direct.c b/lib/launch-direct.c
>>> index 8d6ad025a4e1..cdfd25a9afed 100644
>>> --- a/lib/launch-direct.c
>>> +++ b/lib/launch-direct.c
>>> @@ -338,9 +338,9 @@ add_drives (guestfs_h *g, struct backend_direct_data
>>> *data,
>>> /**
>>> * Launch passt such that it daemonizes.
>>> *
>>> - * On error, -1 is returned; C and C are not modified.
>>> + * On error, C<-1> is returned; C and C are not
>>> modified.
>>> *
>>> - * On success, 0 is returned. C contains the PID of the passt
>>> + * On success, C<0> is returned. C contains the PID of the
>>> passt
>>> * background process. C contains the pathname of the unix
>>> domain
>>> * socket where passt will accept a single connection.
>>> */
>>> @@ -394,7 +394,12 @@ launch_passt (guestfs_h *g, long *passt_pid, char
>>> (*sockpath)[UNIX_PATH_MAX])
>>> goto close_cmd;
>>>}
>>>
>>> - assert (WIFEXITED (passt_status));
>>> + if (!WIFEXITED (passt_status)) {
>>> +error (g, _("internal error: unexpected exit status from passt (%d)"),
>>> + passt_status);
>>> +goto close_cmd;
>>> + }
>>> +
>>>passt_exit = WEXITSTATUS (passt_status);
>>>if (passt_exit != 0) {
>>> error (g, _("passt exited with status %d"), passt_exit);
>>> diff --git a/lib/launch.c b/lib/launch.c
>>> index a0a8e1c45a51..b9b76e509162 100644
>>> --- a/lib/launch.c
>>> +++ b/lib/launch.c
>>> @@ -408,6 +408,9 @@ guestfs_int_passt_runnable (guestfs_h *g)
>>> return false;
>>>
>>>guestfs_int_cmd_add_string_unquoted (cmd, "passt --help");
>>> + if (!g->verbose)
>>> +guestfs_int_cmd_add_string_unquoted (cmd, " >/dev/null 2>&1");
>>> +
>>>r = guestfs_int_cmd_run (cmd);
>>>if (r == -1 || !WIFEXITED (r))
>>> return false;
>>
>> dispersed over patches #2 and #7.
>>
>> I lightly tested the updates with virt-rescue (direct & libvirt backends
>> with passt installed).
>
> For the series:
>
> Reviewed-by: Richard W.M. Jones
>
> If you can push it today then I can do a libguestfs release
> for Fedora.
Commit range 13c7052ff96d..02bbc9daa742.
Cheers
Laszlo
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [libnbd PATCH 1/2] api: Tighten rules on completion.callback
On 7/14/23 16:13, Eric Blake wrote:
> On Fri, Jul 14, 2023 at 09:13:42AM +0200, Laszlo Ersek wrote:
>> On 7/13/23 21:29, Eric Blake wrote:
>>> The documentation has claimed since commit 6f4dcdab that any
>>> completion callback will be called exactly once; but this is not
>>> consistent with the code: if nbd_aio_* itself returns an error, then
>>> nothing is queued and the user does not need to wait for a completion
>>> callback to know how the command failed. We could tweak the generator
>>> to call completion.callback no matter what, but since the
>>> completion.free callback already serves that role, it's easier to fix
>>> the documentation to match reality. After all, one only needs
>>> completion status if an aio command returned success (if it returned
>>> failure, we know that there is nothing that is going to complete
>>> later).
>>>
>>> However, there was one place where we indeed fail to call
>>> completion.callback, even though the corresponding aio call returned
>>> success, which can strand a user that was depending on the callback to
>>> know that the pending aio command failed after all. That's when a
>>> call to nbd_close() interrupts a connection while commands are in
>>> flight. This problem appears to have been around even before commit
>>> 52b9b492 (v0.9.8) when we finally settled on having .free callbacks in
>>> the first place.
>>>
>>> Beef up the closure-lifetimes unit test to more robustly check the
>>> various conditions guaranteed by the updated documentation, and to
>>> expose the previous skip of a completion callback during nbd_close.
>>>
>>> In summary, the behavior we want (where sequence is important) is:
>>>
>>> - aio command fails:
>>> mid-command .callback: 0 calls
>>> completion .callback: 0 calls
>>> mid-command .free: 1 call
>>> completion .free: 1 call
>>>
>>> - aio command succeeds:
>>> mid-command .callback: 0, 1, or multiple calls
>>> completion .callback: 1 call
>>> mid-command .free: 1 call
>>> completion .free: 1 call
>>>
>>> Reported-by: Tage Johansson
>>> Fixes: 6f4dcdab ("docs: Clarify how callbacks should handle errors",
>>> v1.11.8)
>>> Signed-off-by: Eric Blake
>>> ---
>>> docs/libnbd.pod | 26 --
>>> lib/handle.c | 3 +++
>>> tests/closure-lifetimes.c | 14 ++
>>> 3 files changed, 33 insertions(+), 10 deletions(-)
>>>
>>> diff --git a/docs/libnbd.pod b/docs/libnbd.pod
>>> index 72f74053..433479e6 100644
>>> --- a/docs/libnbd.pod
>>> +++ b/docs/libnbd.pod
>>> @@ -904,9 +904,12 @@ same nbd object, as it would cause deadlock.
>>> =head2 Completion callbacks
>>>
>>> All of the asychronous commands have an optional completion callback
>>> -function that is used right before the command is marked complete,
>>> -after any mid-command callbacks have finished, and before any free
>>> -functions.
>>> +function that is used if the asynchronous command succeeded, right
>>> +before the command is marked complete, after any mid-command callbacks
>>> +have finished, and before any free functions. The completion callback
>>> +is not reached if the asynchronous command itself fails, while free
>>> +functions are reached regardless of the initial result of the
>>> +asynchronous command.
>>>
>>> When the completion callback returns C<1>, the command is
>>> automatically retired (there is no need to call
>>
>> I agree with this approach (i.e., with adapting the documentation to
>> reality), but I find the language somewhat confusing. We have three terms:
>>
>> - "asynchronous command succeeds"
>> - "command is marked complete"
>> - "command is retired"
>>
>> The last two are mostly interchangeable in my view, and are also *not*
>> confusing in the documentation. But the first term is confusing, IMO; it
>> can easily be mistaken for meanings #2/#3. What we mean by #1 instead is
>> the successful queueing (or submission) of the command. The text below
>> does say "queued", but the text above doesn't. So I'd suggest replacing
>> term#1 above with "call to asynchronous API succeeds" or "asynchronous
>> command is successfully submitted" or something like that.
>>
>> (Side comment: I'd kinda prefer an all-
[Libguestfs] [libguestfs PATCH v2 7/7] lib/launch-direct: support networking with passt
On QEMU 7.2.0+, if "passt" is available, ask QEMU for passt ("stream")
rather than SLIRP ("user") networking.
For this, we need to run passt ourselves. Given that passt daemonizes by
default, start it with our traditional function guestfs_int_cmd_run(). Ask
passt to save its PID file, because in case something goes wrong before
we're completely sure the appliance (i.e. QEMU) is up and running, we'll
need to kill passt, the *grandchild*, ourselves.
Pass "--one-off" to passt (same as libvirt). This way, once we have proof
that QEMU has connected to passt (because the appliance shows signs of
life), we need not clean up passt ourselves -- once QEMU exits, passt will
see an EOF on the unix domain socket, and exit as well.
Passt is way more flexible than SLIRP, and passt normally intends to
imitate the host environment in the guest as much as possible. This means
that, when switching from SLIRP to passt, the guest would see changes to
the following:
- guest IP address,
- guest subnet mask,
- host (= gateway) IP address,
- host (= gateway) MAC address.
Extract the SLIRP defaults into the new macros NETWORK_GW_IP and
NETWORK_GW_MAC, and pass them explicitly to passt. In particular,
"tests/rsync/test-rsync.sh" fails without setting the host address
(NETWORK_GW_IP) properly.
(These artifacts can be verified in the appliance with "virt-rescue
--network", by running "ip addr", "ip route", and "ip neighbor" at the
virt-rescue prompt. There are four scenarios: two libguest backends, times
passt being installed or not installed.)
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967
Signed-off-by: Laszlo Ersek
---
Notes:
v2:
- document the constants returned by launch_passt() with C<> notation,
in the function's leading comment
- don't assert WIFEXITED just because of !WIFSIGNALED; check explicitly
[Rich]
lib/guestfs-internal.h | 26
lib/launch-direct.c| 152 +++-
2 files changed, 173 insertions(+), 5 deletions(-)
diff --git a/lib/guestfs-internal.h b/lib/guestfs-internal.h
index 9ba4d4ad46cf..9f4f800e6d6e 100644
--- a/lib/guestfs-internal.h
+++ b/lib/guestfs-internal.h
@@ -158,6 +158,32 @@ cleanup_mutex_unlock (pthread_mutex_t **ptr)
#define NETWORK_ADDRESS "169.254.2.15"
#define NETWORK_PREFIX "16"
+/* The IP address and the MAC address of the host, as seen from the appliance.
+ *
+ * NETWORK_GW_IP should be the same as NETWORK_ADDRESS, only replacing ".15" in
+ * the rightmost octet with ".2". NETWORK_GW_MAC cannot be changed. These
+ * restrictions are a consequence of the following landscape:
+ *
+ * libguestfs backend userspace network stack restrictions
+ * -- ---
+ * direct passtNone; both NETWORK_GW_IP and
+ * NETWORK_GW_MAC can be set on
the
+ * passt command line.
+ *
+ * direct SLIRPSLIRP hard-codes
NETWORK_GW_MAC.
+ *
+ * libvirt passtThe domain XML does not expose
+ * either knob (RHBZ#766),
even
+ * though passt could accept both.
+ *
+ * libvirt SLIRPThe domain XML does not expose
+ * either knob (RHBZ#766), and
+ * SLIRP hard-codes NETWORK_GW_MAC
+ * anyway.
+ */
+#define NETWORK_GW_IP "169.254.2.2"
+#define NETWORK_GW_MAC "52:56:00:00:00:02"
+
/* Guestfs handle and associated structures. */
/* State. */
diff --git a/lib/launch-direct.c b/lib/launch-direct.c
index 3f46f0509736..cdfd25a9afed 100644
--- a/lib/launch-direct.c
+++ b/lib/launch-direct.c
@@ -32,6 +32,7 @@
#include
#include
#include
+#include
#include
#include
#include
@@ -48,6 +49,7 @@
#include "guestfs-internal.h"
#include "guestfs_protocol.h"
#include "qemuopts.h"
+#include "ignore-value.h"
/* Per-handle data. */
struct backend_direct_data {
@@ -333,6 +335,115 @@ add_drives (guestfs_h *g, struct backend_direct_data
*data,
return 0;
}
+/**
+ * Launch passt such that it daemonizes.
+ *
+ * On error, C<-1> is returned; C and C are not modified.
+ *
+ * On success, C<0> is returned. C contains the PID of the passt
+ * background process. C contains the pathname of the unix domain
+ * socket where passt will accept a single connection.
+ */
+static int
+launch_passt (guestfs_h *g, long *passt_pid, char (*sockpath)[UNIX_PATH_MAX])
+{
+ int rc;
+ char sockpath_local[sizeof *sockpath];
+ char *p
[Libguestfs] [libguestfs PATCH v2 5/7] lib: move guestfs_int_create_socketname() from "launch.c" to "tmpdirs.c"
Consider the following inverted call tree (effectively a dependency tree
-- callees are at the top and near the left margin):
lazy_make_tmpdir() [lib/tmpdirs.c]
guestfs_int_lazy_make_tmpdir()[lib/tmpdirs.c]
guestfs_int_make_temp_path()[lib/tmpdirs.c]
guestfs_int_lazy_make_sockdir() [lib/tmpdirs.c]
guestfs_int_create_socketname() [lib/launch.c]
lazy_make_tmpdir() is our common workhorse / helper function that
centralizes the mkdtemp() function call.
guestfs_int_lazy_make_tmpdir() and guestfs_int_lazy_make_sockdir() are the
next level functions, both calling lazy_make_tmpdir(), just feeding it
different dirname generator functions, and different "is_runtime_dir"
qualifications. These functions create temp dirs for various, more
specific, purposes (see the manual and "lib/guestfs-internal.h" for more
details).
On a yet higher level are guestfs_int_make_temp_path() and
guestfs_int_create_socketname() -- they serve for creating *entries* in
those specific temp directories.
The discrepancy here is that, although all the other functions live in
"lib/tmpdirs.c", guestfs_int_create_socketname() is defined in
"lib/launch.c". That makes for a confusing code reading; move the function
to "lib/tmpdirs.c", just below its sibling function
guestfs_int_make_temp_path().
While at it, correct the leading comment on
guestfs_int_create_socketname() -- the socket pathname is created in the
socket directory, not in the temporary directory.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967
Signed-off-by: Laszlo Ersek
Reviewed-by: Richard W.M. Jones
---
Notes:
v2:
- pick up Rich's R-b
lib/guestfs-internal.h | 2 +-
lib/launch.c | 26
lib/tmpdirs.c | 26
3 files changed, 27 insertions(+), 27 deletions(-)
diff --git a/lib/guestfs-internal.h b/lib/guestfs-internal.h
index a6c4266ad3fe..2ee16ea1e75a 100644
--- a/lib/guestfs-internal.h
+++ b/lib/guestfs-internal.h
@@ -668,6 +668,7 @@ extern int guestfs_int_set_env_runtimedir (guestfs_h *g,
const char *envname, co
extern int guestfs_int_lazy_make_tmpdir (guestfs_h *g);
extern int guestfs_int_lazy_make_sockdir (guestfs_h *g);
extern char *guestfs_int_make_temp_path (guestfs_h *g, const char *name, const
char *extension);
+extern int guestfs_int_create_socketname (guestfs_h *g, const char *filename,
char (*sockname)[UNIX_PATH_MAX]);
extern char *guestfs_int_lazy_make_supermin_appliance_dir (guestfs_h *g);
extern void guestfs_int_remove_tmpdir (guestfs_h *g);
extern void guestfs_int_remove_sockdir (guestfs_h *g);
@@ -700,7 +701,6 @@ extern int guestfs_int_get_uefi (guestfs_h *g, char *const
*firmwares, const cha
extern int64_t guestfs_int_timeval_diff (const struct timeval *x, const struct
timeval *y);
extern void guestfs_int_launch_send_progress (guestfs_h *g, int perdozen);
extern void guestfs_int_unblock_sigterm (void);
-extern int guestfs_int_create_socketname (guestfs_h *g, const char *filename,
char (*sockname)[UNIX_PATH_MAX]);
extern void guestfs_int_register_backend (const char *name, const struct
backend_ops *);
extern int guestfs_int_set_backend (guestfs_h *g, const char *method);
extern bool guestfs_int_passt_runnable (guestfs_h *g);
diff --git a/lib/launch.c b/lib/launch.c
index 87fb10e75a6c..b9b76e509162 100644
--- a/lib/launch.c
+++ b/lib/launch.c
@@ -310,32 +310,6 @@ guestfs_impl_config (guestfs_h *g,
return 0;
}
-/**
- * Create the path for a socket with the selected filename in the
- * tmpdir.
- */
-int
-guestfs_int_create_socketname (guestfs_h *g, const char *filename,
- char (*sockpath)[UNIX_PATH_MAX])
-{
- int r;
-
- if (guestfs_int_lazy_make_sockdir (g) == -1)
-return -1;
-
- r = snprintf (*sockpath, UNIX_PATH_MAX, "%s/%s", g->sockdir, filename);
- if (r >= UNIX_PATH_MAX) {
-error (g, _("socket path too long: %s/%s"), g->sockdir, filename);
-return -1;
- }
- if (r < 0) {
-perrorf (g, _("%s"), g->sockdir);
-return -1;
- }
-
- return 0;
-}
-
/**
* When the library is loaded, each backend calls this function to
* register itself in a global list.
diff --git a/lib/tmpdirs.c b/lib/tmpdirs.c
index b8e19de2bf9e..24adf98daee0 100644
--- a/lib/tmpdirs.c
+++ b/lib/tmpdirs.c
@@ -253,6 +253,32 @@ guestfs_int_make_temp_path (guestfs_h *g,
extension ? extension : "");
}
+/**
+ * Create the path for a socket with the selected filename in the
+ * sockdir.
+ */
+int
+guestfs_int_create_socketname (guestfs_h *g, const char *filename,
+ char (*sockpath)[UNIX_PATH_MAX])
+{
+ int r;
+
+ if (guestfs_int_lazy_make_sockdir (g) == -1)
+return -1;
+
+ r = snprintf (*sockpath, UNIX_PATH_MAX, "%s/%s", g->sockdir, filename);
+ if (r >= UNIX_PATH_MAX) {
+error (g, _("s
[Libguestfs] [libguestfs PATCH v2 3/7] docs: fix broken link in the guestfs manual
Commit 55202a4d49a1 ("New API: get-sockdir", 2016-02-03) added identical
language to "fish/guestfish.pod" and "src/guestfs.pod", including an
internal link L. That's appropriate for
"fish/guestfish.pod", but the same API description is generated with a
different anchor for "src/guestfs.pod". Adapt the reference.
Fixes: 55202a4d49a101392148d79cb2e1591428db2681
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967
Signed-off-by: Laszlo Ersek
Reviewed-by: Richard W.M. Jones
---
Notes:
v2:
- pick up Rich's R-b
lib/guestfs.pod | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/guestfs.pod b/lib/guestfs.pod
index c6c8cb16860c..68688f31aa5f 100644
--- a/lib/guestfs.pod
+++ b/lib/guestfs.pod
@@ -3223,7 +3223,7 @@ non-essential runtime files.
If it is set, then is used to store temporary sockets. Otherwise,
F is used.
-See also L,
+See also L,
L<http://www.freedesktop.org/wiki/Specifications/basedir-spec/>.
=back
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
[Libguestfs] [libguestfs PATCH v2 4/7] docs: clarify sockdir's separation
There's another reason for separating sockdir from tmpdir, beyond "shorter pathnames needed": permissions. For example, passt drops privileges such that it cannot access "/tmp", and that restricts both the unix domain socket and the PID file of passt. Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967 Signed-off-by: Laszlo Ersek Reviewed-by: Richard W.M. Jones --- Notes: v2: - pick up Rich's R-b generator/actions_properties.ml | 8 ++-- fish/guestfish.pod | 4 ++-- lib/guestfs.pod | 4 ++-- 3 files changed, 10 insertions(+), 6 deletions(-) diff --git a/generator/actions_properties.ml b/generator/actions_properties.ml index f84afb10d674..42eaaa4d81e1 100644 --- a/generator/actions_properties.ml +++ b/generator/actions_properties.ml @@ -595,13 +595,17 @@ Get the handle identifier. See C." }; name = "get_sockdir"; added = (1, 33, 8); style = RString (RPlainString, "sockdir"), [], []; blocking = false; -shortdesc = "get the temporary directory for sockets"; +shortdesc = "get the temporary directory for sockets and PID files"; longdesc = "\ -Get the directory used by the handle to store temporary socket files. +Get the directory used by the handle to store temporary socket and PID +files. This is different from C, as we need shorter paths for sockets (due to the limited buffers of filenames for UNIX sockets), and C may be too long for them. +Furthermore, sockets and PID files must be accessible to such background +services started by libguestfs that may not have permission to access +the temporary directory returned by C. The environment variable C controls the default value: If C is set, then that is the default. diff --git a/fish/guestfish.pod b/fish/guestfish.pod index ccc0825b84a0..492aa7163fcb 100644 --- a/fish/guestfish.pod +++ b/fish/guestfish.pod @@ -1548,8 +1548,8 @@ See L, L. This directory represents a user-specific directory for storing non-essential runtime files. -If it is set, then is used to store temporary sockets. Otherwise, -F is used. +If it is set, then is used to store temporary sockets and PID files. +Otherwise, F is used. See also L, L<http://www.freedesktop.org/wiki/Specifications/basedir-spec/>. diff --git a/lib/guestfs.pod b/lib/guestfs.pod index 68688f31aa5f..e46dd81f9e0a 100644 --- a/lib/guestfs.pod +++ b/lib/guestfs.pod @@ -3220,8 +3220,8 @@ See L, L. This directory represents a user-specific directory for storing non-essential runtime files. -If it is set, then is used to store temporary sockets. Otherwise, -F is used. +If it is set, then is used to store temporary sockets and PID files. +Otherwise, F is used. See also L, L<http://www.freedesktop.org/wiki/Specifications/basedir-spec/>. ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
[Libguestfs] [libguestfs PATCH v2 6/7] lib: introduce guestfs_int_make_pid_path()
Introduce a small function for creating pathnames for PID files.
guestfs_int_make_pid_path() is something of an amalgamation of
guestfs_int_make_temp_path() [1] and guestfs_int_create_socketname() [2]:
- it creates a pathname under sockdir, like [2],
- it uses the handle's unique counter, like [1],
- it takes a name like both [1] and [2], but the name is not size-limited
like in [2], plus we hardcode the suffix from [1] as ".pid".
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967
Signed-off-by: Laszlo Ersek
Reviewed-by: Richard W.M. Jones
---
Notes:
v2:
- pick up Rich's R-b
lib/guestfs-internal.h | 1 +
lib/tmpdirs.c | 15 +++
2 files changed, 16 insertions(+)
diff --git a/lib/guestfs-internal.h b/lib/guestfs-internal.h
index 2ee16ea1e75a..9ba4d4ad46cf 100644
--- a/lib/guestfs-internal.h
+++ b/lib/guestfs-internal.h
@@ -669,6 +669,7 @@ extern int guestfs_int_lazy_make_tmpdir (guestfs_h *g);
extern int guestfs_int_lazy_make_sockdir (guestfs_h *g);
extern char *guestfs_int_make_temp_path (guestfs_h *g, const char *name, const
char *extension);
extern int guestfs_int_create_socketname (guestfs_h *g, const char *filename,
char (*sockname)[UNIX_PATH_MAX]);
+extern char *guestfs_int_make_pid_path (guestfs_h *g, const char *name);
extern char *guestfs_int_lazy_make_supermin_appliance_dir (guestfs_h *g);
extern void guestfs_int_remove_tmpdir (guestfs_h *g);
extern void guestfs_int_remove_sockdir (guestfs_h *g);
diff --git a/lib/tmpdirs.c b/lib/tmpdirs.c
index 24adf98daee0..22b8f54b0693 100644
--- a/lib/tmpdirs.c
+++ b/lib/tmpdirs.c
@@ -279,6 +279,21 @@ guestfs_int_create_socketname (guestfs_h *g, const char
*filename,
return 0;
}
+/**
+ * Generate unique paths for PID files.
+ *
+ * Returns a unique path or NULL on error. On success, the pathname points
+ * under sockdir and not tmpdir; daemons that write PID files after dropping
+ * privileges may not have access to tmpdir.
+ */
+char *
+guestfs_int_make_pid_path (guestfs_h *g, const char *name)
+{
+ if (guestfs_int_lazy_make_sockdir (g) < 0)
+return NULL;
+ return safe_asprintf (g, "%s/%s%d.pid", g->sockdir, name, ++g->unique);
+}
+
/**
* Create the supermin appliance directory under cachedir, if it does
* not exist.
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
[Libguestfs] [libguestfs PATCH v2 2/7] lib/launch-libvirt: support networking with passt
We generate the element on libvirt 3.8.0+ already.
For selecting passt rather than SLIRP, we only need to insert the child
element . Make that child element conditional on
libvirt 9.0.0+, plus "passt --help" being executable.
For the latter, place the new helper function guestfs_int_passt_runnable()
in "lib/launch.c" -- we're going to use the same function for the direct
backend as well.
This change exposes a number of (perceived) shortcomings in libvirt; I've
filed <https://bugzilla.redhat.com/show_bug.cgi?id=766> about those.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967
Signed-off-by: Laszlo Ersek
Reviewed-by: Richard W.M. Jones
---
Notes:
v2:
- redirect "passt --help" stdout+stderr to /dev/null unless verbose
[Rich]
- pick up Rich's R-b
lib/guestfs-internal.h | 1 +
lib/launch-libvirt.c | 11 +++
lib/launch.c | 31
3 files changed, 43 insertions(+)
diff --git a/lib/guestfs-internal.h b/lib/guestfs-internal.h
index 4be351e3d3cc..a6c4266ad3fe 100644
--- a/lib/guestfs-internal.h
+++ b/lib/guestfs-internal.h
@@ -703,6 +703,7 @@ extern void guestfs_int_unblock_sigterm (void);
extern int guestfs_int_create_socketname (guestfs_h *g, const char *filename,
char (*sockname)[UNIX_PATH_MAX]);
extern void guestfs_int_register_backend (const char *name, const struct
backend_ops *);
extern int guestfs_int_set_backend (guestfs_h *g, const char *method);
+extern bool guestfs_int_passt_runnable (guestfs_h *g);
/* Close all file descriptors matching the condition. */
#define close_file_descriptors(cond) do { \
diff --git a/lib/launch-libvirt.c b/lib/launch-libvirt.c
index d4bf1a8ff242..994909a35f2d 100644
--- a/lib/launch-libvirt.c
+++ b/lib/launch-libvirt.c
@@ -1414,6 +1414,17 @@ construct_libvirt_xml_devices (guestfs_h *g,
guestfs_int_version_ge (>data->libvirt_version, 3, 8, 0)) {
start_element ("interface") {
attribute ("type", "user");
+/* If libvirt is 9.0.0+ and "passt" is available, ask for passt rather
+ * than SLIRP (RHBZ#2184967). Note that this causes some
+ * appliance-visible changes (although network connectivity is
certainly
+ * functional); refer to RHBZ#766 about those.
+ */
+if (guestfs_int_version_ge (>data->libvirt_version, 9, 0, 0) &&
+guestfs_int_passt_runnable (g)) {
+ start_element ("backend") {
+attribute ("type", "passt");
+ } end_element ();
+}
start_element ("model") {
attribute ("type", "virtio");
} end_element ();
diff --git a/lib/launch.c b/lib/launch.c
index 6e08b12006da..87fb10e75a6c 100644
--- a/lib/launch.c
+++ b/lib/launch.c
@@ -36,6 +36,7 @@
#include
#include
#include
+#include
#include
#include
#include
@@ -414,6 +415,36 @@ guestfs_int_set_backend (guestfs_h *g, const char *method)
return 0;
}
+/**
+ * Return C if we can call S>, and it exits with status
+ * C<0> or C<1>.
+ *
+ * (At least C terminates with status
+ * C<1> in response to "--help", which is arguably wrong, and potentially
+ * subject to change, but it doesn't really matter.)
+ */
+bool
+guestfs_int_passt_runnable (guestfs_h *g)
+{
+ CLEANUP_CMD_CLOSE struct command *cmd = NULL;
+ int r, ex;
+
+ cmd = guestfs_int_new_command (g);
+ if (cmd == NULL)
+return false;
+
+ guestfs_int_cmd_add_string_unquoted (cmd, "passt --help");
+ if (!g->verbose)
+guestfs_int_cmd_add_string_unquoted (cmd, " >/dev/null 2>&1");
+
+ r = guestfs_int_cmd_run (cmd);
+ if (r == -1 || !WIFEXITED (r))
+return false;
+
+ ex = WEXITSTATUS (r);
+ return ex == 0 || ex == 1;
+}
+
/* This hack is only required to make static linking work. See:
*
https://stackoverflow.com/questions/1202494/why-doesnt-attribute-constructor-work-in-a-static-library
*/
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
[Libguestfs] [libguestfs PATCH v2 1/7] lib: fix NETWORK_ADDRESS: make it an actual IP address, not a subnet base
Currently we #define NETWORK_ADDRESS as "169.254.0.0". That's a bug in
libguestfs, but it's been invisible -- thus far it's been canceled out by
*independent* bugs in *both* QEMU and libvirt.
(1) With the direct backend, the current definition of NETWORK_ADDRESS
results in the following QEMU command line option:
-netdev user,id=usernet,net=169.254.0.0/16
According to the QEMU documentation, the "net" property here is supposed
to specify the *exact* IP address that the guest will receive. The guest
however receives 169.254.2.15 (I've checked that with virt-rescue).
In other words, libguestfs doesn't do what the QEMU documentation says,
and QEMU doesn't do what the QEMU documentation says either. The end
result has been good enough -- but only until now.
(2) With the libvirt backend, the current definition of NETWORK_ADDRESS
results in the following domain XML snippet:
which libvirt translates, in turn, to
-netdev {"type":"user","net":"169.254.0.0/16","id":"hostnet0"}
According to the domain XML documentation, the @address attribute here is
again supposed to specify the *exact* IP address that the guest will
receive. However, the guest receives 169.254.2.15 (I've checked that with
virt-rescue).
In other words, libguestfs doesn't do what the libvirt documentation says,
and libvirt doesn't do what the libvirt documentation says either. The end
result has been good enough -- but only until now.
Where things break down though is the subsequent passt enablement, in the
rest of this series. For example, when using the libvirt backend together
with passt, libvirt translates the @address attribute to passt's
"--address" option, but passt takes the address *verbatim*, and not as a
subnet base address. That difference is visible in the appliance; for
example, when running virt-rescue on a Fedora 38 image, and issuing "ip
addr". Namely, after enabling passt for the libvirt backend, the
guest-visible IP address changes from 169.254.2.15 to 169.254.0.0, which
is an IP address that makes no sense for an endpoint.
Fix the latent bug by specifying the actual guest IP address we want, in
NETWORK_ADDRESS.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967
Signed-off-by: Laszlo Ersek
Reviewed-by: Richard W.M. Jones
---
Notes:
v2:
- pick up Rich's R-b
lib/guestfs-internal.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/guestfs-internal.h b/lib/guestfs-internal.h
index c7ef32277e93..4be351e3d3cc 100644
--- a/lib/guestfs-internal.h
+++ b/lib/guestfs-internal.h
@@ -155,7 +155,7 @@ cleanup_mutex_unlock (pthread_mutex_t **ptr)
/* Network address and network mask (expressed as address prefix) that the
* appliance will see (if networking is enabled).
*/
-#define NETWORK_ADDRESS "169.254.0.0"
+#define NETWORK_ADDRESS "169.254.2.15"
#define NETWORK_PREFIX "16"
/* Guestfs handle and associated structures. */
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
[Libguestfs] [libguestfs PATCH v2 0/7] lib: support networking with passt
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967
v1: https://listman.redhat.com/archives/libguestfs/2023-July/031984.html
V2 implements small updates; the cumulative v1->v2 diff is just
> diff --git a/lib/launch-direct.c b/lib/launch-direct.c
> index 8d6ad025a4e1..cdfd25a9afed 100644
> --- a/lib/launch-direct.c
> +++ b/lib/launch-direct.c
> @@ -338,9 +338,9 @@ add_drives (guestfs_h *g, struct backend_direct_data
> *data,
> /**
> * Launch passt such that it daemonizes.
> *
> - * On error, -1 is returned; C and C are not modified.
> + * On error, C<-1> is returned; C and C are not
> modified.
> *
> - * On success, 0 is returned. C contains the PID of the passt
> + * On success, C<0> is returned. C contains the PID of the passt
> * background process. C contains the pathname of the unix domain
> * socket where passt will accept a single connection.
> */
> @@ -394,7 +394,12 @@ launch_passt (guestfs_h *g, long *passt_pid, char
> (*sockpath)[UNIX_PATH_MAX])
> goto close_cmd;
>}
>
> - assert (WIFEXITED (passt_status));
> + if (!WIFEXITED (passt_status)) {
> +error (g, _("internal error: unexpected exit status from passt (%d)"),
> + passt_status);
> +goto close_cmd;
> + }
> +
>passt_exit = WEXITSTATUS (passt_status);
>if (passt_exit != 0) {
> error (g, _("passt exited with status %d"), passt_exit);
> diff --git a/lib/launch.c b/lib/launch.c
> index a0a8e1c45a51..b9b76e509162 100644
> --- a/lib/launch.c
> +++ b/lib/launch.c
> @@ -408,6 +408,9 @@ guestfs_int_passt_runnable (guestfs_h *g)
> return false;
>
>guestfs_int_cmd_add_string_unquoted (cmd, "passt --help");
> + if (!g->verbose)
> +guestfs_int_cmd_add_string_unquoted (cmd, " >/dev/null 2>&1");
> +
>r = guestfs_int_cmd_run (cmd);
>if (r == -1 || !WIFEXITED (r))
> return false;
dispersed over patches #2 and #7.
I lightly tested the updates with virt-rescue (direct & libvirt backends
with passt installed).
Thanks
Laszlo
Laszlo Ersek (7):
lib: fix NETWORK_ADDRESS: make it an actual IP address, not a subnet
base
lib/launch-libvirt: support networking with passt
docs: fix broken link in the guestfs manual
docs: clarify sockdir's separation
lib: move guestfs_int_create_socketname() from "launch.c" to
"tmpdirs.c"
lib: introduce guestfs_int_make_pid_path()
lib/launch-direct: support networking with passt
fish/guestfish.pod | 4 +-
generator/actions_properties.ml | 8 +-
lib/guestfs-internal.h | 32 -
lib/guestfs.pod | 6 +-
lib/launch-direct.c | 152 +++-
lib/launch-libvirt.c| 11 ++
lib/launch.c| 57
lib/tmpdirs.c | 41 ++
8 files changed, 271 insertions(+), 40 deletions(-)
base-commit: 13c7052ff96d5ee99ec1b1252f1a3b4d7aed44d2
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [libguestfs PATCH 7/7] lib/launch-direct: support networking with passt
On 7/14/23 11:53, Richard W.M. Jones wrote:
> On Thu, Jul 13, 2023 at 07:10:52PM +0200, Laszlo Ersek wrote:
>> On QEMU 7.2.0+, if "passt" is available, ask QEMU for passt ("stream")
>> rather than SLIRP ("user") networking.
>>
>> For this, we need to run passt ourselves. Given that passt daemonizes by
>> default, start it with our traditional function guestfs_int_cmd_run(). Ask
>> passt to save its PID file, because in case something goes wrong before
>> we're completely sure the appliance (i.e. QEMU) is up and running, we'll
>> need to kill passt, the *grandchild*, ourselves.
>>
>> Pass "--one-off" to passt (same as libvirt). This way, once we have proof
>> that QEMU has connected to passt (because the appliance shows signs of
>> life), we need not clean up passt ourselves -- once QEMU exits, passt will
>> see an EOF on the unix domain socket, and exit as well.
>>
>> Passt is way more flexible than SLIRP, and passt normally intends to
>> imitate the host environment in the guest as much as possible. This means
>> that, when switching from SLIRP to passt, the guest would see changes to
>> the following:
>>
>> - guest IP address,
>>
>> - guest subnet mask,
>>
>> - host (= gateway) IP address,
>>
>> - host (= gateway) MAC address.
>>
>> Extract the SLIRP defaults into the new macros NETWORK_GW_IP and
>> NETWORK_GW_MAC, and pass them explicitly to passt. In particular,
>> "tests/rsync/test-rsync.sh" fails without setting the host address
>> (NETWORK_GW_IP) properly.
>
> What really matters is that programs like 'dnf' and 'apt' can be run.
> Nothing that uses libguestfs networking ought to depend on the exact
> IP address or other details of the current SLIRP implementation. So
> these changes are fine. (rsync is a weird feature that we should have
> pushed back on and not added ...)
OK, dnf does work (just tested it with "virt-builder --update" under the
cover letter).
>
>> (These artifacts can be verified in the appliance with "virt-rescue
>> --network", by running "ip addr", "ip route", and "ip neighbor" at the
>> virt-rescue prompt. There are four scenarios: two libguest backends, times
>> passt being installed or not installed.)
>>
>> Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967
>> Signed-off-by: Laszlo Ersek
>> ---
>> lib/guestfs-internal.h | 26
>> lib/launch-direct.c| 147 +++-
>> 2 files changed, 168 insertions(+), 5 deletions(-)
>>
>> diff --git a/lib/guestfs-internal.h b/lib/guestfs-internal.h
>> index 9ba4d4ad46cf..9f4f800e6d6e 100644
>> --- a/lib/guestfs-internal.h
>> +++ b/lib/guestfs-internal.h
>> @@ -158,6 +158,32 @@ cleanup_mutex_unlock (pthread_mutex_t **ptr)
>> #define NETWORK_ADDRESS "169.254.2.15"
>> #define NETWORK_PREFIX "16"
>>
>> +/* The IP address and the MAC address of the host, as seen from the
>> appliance.
>> + *
>> + * NETWORK_GW_IP should be the same as NETWORK_ADDRESS, only replacing
>> ".15" in
>> + * the rightmost octet with ".2". NETWORK_GW_MAC cannot be changed. These
>> + * restrictions are a consequence of the following landscape:
>> + *
>> + * libguestfs backend userspace network stack restrictions
>> + * -- ---
>>
>> + * direct passtNone; both NETWORK_GW_IP and
>> + * NETWORK_GW_MAC can be set
>> on the
>> + * passt command line.
>> + *
>> + * direct SLIRPSLIRP hard-codes
>> NETWORK_GW_MAC.
>> + *
>> + * libvirt passtThe domain XML does not
>> expose
>> + * either knob (RHBZ#766),
>> even
>> + * though passt could accept
>> both.
>> + *
>> + * libvirt SLIRPThe domain XML does not
>> expose
>> + * either knob (RHBZ#766),
>> and
>> + * SLIRP hard-codes
>> NETWORK_GW_MAC
>> + * anyway.
>> + */
>> +#define NETWORK_GW_IP "169.254.2.2"
>> +#define NETWORK_GW_MAC "52:56:00:00:00:02"
>
Re: [Libguestfs] [libguestfs PATCH 0/7] lib: support networking with passt
On 7/14/23 11:29, Richard W.M. Jones wrote:
> On Thu, Jul 13, 2023 at 07:10:45PM +0200, Laszlo Ersek wrote:
>> Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967
>>
>> This series makes both backends prefer passt over slirp for appliance
>> networking, if QEMU or libvirt (respectively) is recent enough, and
>> passt is installed.
>>
>> My test setup is:
>>
>> $ virt-builder fedora-38
>>
>> Then, each test run looks like this:
>>
>> Terminal#1:
>>
>> $ ./run rescue/virt-rescue --network --ro -a fedora-38.img
>>
>> Terminal#2:
>>
>> - check if "passt" is running (ps -ef, pgrep, ...), provided it *should*
>> be running
>>
>> Terminal#1:
>>
>>> ping -c 3 8.8.8.8
>>> ip neighbor
>>> ip addr
>>> ip route
>>
>> Expected results (in the above order), always on the "eth0" lines:
>> - all three pings succeed (get replies)
>> - 52:56:00:00:00:02
>> - 169.254.2.15/16
>> - default via 169.254.2.2
>
> Another good test (and the one that really matters) would be something
> like this on Terminal #1 ...
>
>> chroot /sysroot bash
>> dnf install emacs
>
> (or some other package). If dnf can see the Fedora repositories
> despite the network setup being slightly wonky, that will mean that
> virt-customize should work.
I've been actually thinking about another Clevis+Tang test, but my
original test environment for that is gone (gone with my RHEL7 laptop...) :/
But, given that I build all v2v projects from source, I can run my
"virt-builder" binary from guestfs-tools such that it pick up my
just-build libguestfs. And so I've now run:
$ rm -f fedora-37.img
$ LIBGUESTFS_BACKEND=libvirt virt-builder --update fedora-37
$ rm -f fedora-37.img
$ LIBGUESTFS_BACKEND=direct virt-builder --update fedora-37
I've verified that "passt" gets launched in both cases. And, the updates
do complete:
[ 4.4] Downloading: http://builder.libguestfs.org/fedora-37.xz
[ 5.2] Planning how to build this image
[ 5.2] Uncompressing
[ 10.8] Opening the new disk
[ 17.3] Setting a random seed
[ 17.3] Updating packages
[ 166.5] Setting passwords
virt-builder: Setting random password of root to VbTWjwUIVAVAXFLV
[ 167.3] SELinux relabelling
[ 176.7] Finishing off
Output file: fedora-37.img
Output size: 6.0G
Output format: raw
Total usable space: 5.9G
Free space: 4.0G (67%)
[ 3.4] Downloading: http://builder.libguestfs.org/fedora-37.xz
[ 4.2] Planning how to build this image
[ 4.2] Uncompressing
[ 9.9] Opening the new disk
[ 14.2] Setting a random seed
[ 14.2] Updating packages
[ 224.8] Setting passwords
virt-builder: Setting random password of root to tT0YGZ9adqTMXzYR
[ 225.6] SELinux relabelling
[ 234.6] Finishing off
Output file: fedora-37.img
Output size: 6.0G
Output format: raw
Total usable space: 5.9G
Free space: 4.0G (67%)
> It's good that this work has found a bunch of libvirt bugs!
Passt is meant to replace slirp, so its different mission statement
("make the guest networking env resemble the host one as much as
possible") is justified and welcome; but for the libguestfs appliance, I
figured we'd want to hide any slirp/passt differences, preferably with
both the direct and libvirt backends. (For example, the rsync test case,
which is already restricted to the direct backend, would break due to
those differences.) So that requires digging into a bunch of corner cases.
It's difficult to know where to draw the line: if a different host IP
address (as seen by the guest) breaks our rsync test case, then maybe a
different host MAC address, or different guest IP address, might break
something else, for someone else. They might have some custom shell
script they run in the appliance. So who can say how pedantically we
should imitate the slirp environment?... Close imitation was never the
intent in the libvirt enablement (and justifiedly so, again, I think),
but using just the defaults definitely broke our rsync test case at least.
Laszlo
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [libguestfs PATCH 4/7] docs: clarify sockdir's separation
On 7/14/23 11:42, Richard W.M. Jones wrote: > On Thu, Jul 13, 2023 at 07:10:49PM +0200, Laszlo Ersek wrote: >> There's another reason for separating sockdir from tmpdir, beyond "shorter >> pathnames needed": permissions. For example, passt drops privileges such >> that it cannot access "/tmp", and that restricts both the unix domain >> socket and the PID file of passt. >> >> Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967 >> Signed-off-by: Laszlo Ersek >> --- >> generator/actions_properties.ml | 8 ++-- >> fish/guestfish.pod | 4 ++-- >> lib/guestfs.pod | 4 ++-- >> 3 files changed, 10 insertions(+), 6 deletions(-) >> >> diff --git a/generator/actions_properties.ml >> b/generator/actions_properties.ml >> index f84afb10d674..42eaaa4d81e1 100644 >> --- a/generator/actions_properties.ml >> +++ b/generator/actions_properties.ml >> @@ -595,13 +595,17 @@ Get the handle identifier. See >> C." }; >> name = "get_sockdir"; added = (1, 33, 8); >> style = RString (RPlainString, "sockdir"), [], []; >> blocking = false; >> -shortdesc = "get the temporary directory for sockets"; >> +shortdesc = "get the temporary directory for sockets and PID files"; >> longdesc = "\ >> -Get the directory used by the handle to store temporary socket files. >> +Get the directory used by the handle to store temporary socket and PID >> +files. >> >> This is different from C, as we need shorter >> paths for sockets (due to the limited buffers of filenames for UNIX >> sockets), and C may be too long for them. >> +Furthermore, sockets and PID files must be accessible to such background >> +services started by libguestfs that may not have permission to access >> +the temporary directory returned by C. >> >> The environment variable C controls the default >> value: If C is set, then that is the default. >> diff --git a/fish/guestfish.pod b/fish/guestfish.pod >> index ccc0825b84a0..492aa7163fcb 100644 >> --- a/fish/guestfish.pod >> +++ b/fish/guestfish.pod >> @@ -1548,8 +1548,8 @@ See L, L. >> This directory represents a user-specific directory for storing >> non-essential runtime files. >> >> -If it is set, then is used to store temporary sockets. Otherwise, >> -F is used. >> +If it is set, then is used to store temporary sockets and PID files. >> +Otherwise, F is used. >> >> See also L, > > Although incidental to this change, shouldn't this link also be fixed > in the same way as patch 3? That's the thing: it shouldn't. The generator produces the individual API descriptions for both "guestfish.1" and "guestfs.3". The generator creates one style of anchor (effectively: link name) for "guestfish.1", and another style of anchor for "guestfs.3". In turn, the "ENVIRONMENT VARIABLES" section is open-coded (not generated), in both "guestfish.pod" and "guestfs.pod" files. So, whenever we insert a link to a generated anchor, in the "ENVIRONMENT VARIABLES" of one of these .pod files, we must use the anchor style that is appropriate for *that particular .pod file*. Commit 55202a4d49a1 had added the same link L to the "ENVIRONMENT VARIABLES" section of both .pod files. However, this link style is only good for "guestfish.1". The same link will not work in "guestfs.pod", because it will not match any generated anchor in "guestfs.3". That's why patch#3 only updates "guestfs.pod"; the same link in "guestfish.pod" is not broken. > >> L<http://www.freedesktop.org/wiki/Specifications/basedir-spec/>. >> diff --git a/lib/guestfs.pod b/lib/guestfs.pod >> index 68688f31aa5f..e46dd81f9e0a 100644 >> --- a/lib/guestfs.pod >> +++ b/lib/guestfs.pod >> @@ -3220,8 +3220,8 @@ See L, L. >> This directory represents a user-specific directory for storing >> non-essential runtime files. >> >> -If it is set, then is used to store temporary sockets. Otherwise, >> -F is used. >> +If it is set, then is used to store temporary sockets and PID files. >> +Otherwise, F is used. >> >> See also L, >> L<http://www.freedesktop.org/wiki/Specifications/basedir-spec/>. > > Reviewed-by: Richard W.M. Jones > > Rich. > Thanks! Laszlo ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [libguestfs PATCH 2/7] lib/launch-libvirt: support networking with passt
On 7/14/23 11:40, Richard W.M. Jones wrote:
> On Thu, Jul 13, 2023 at 07:10:47PM +0200, Laszlo Ersek wrote:
>> We generate the element on libvirt 3.8.0+ already.
>>
>> For selecting passt rather than SLIRP, we only need to insert the child
>> element . Make that child element conditional on
>> libvirt 9.0.0+, plus "passt --help" being executable.
>>
>> For the latter, place the new helper function guestfs_int_passt_runnable()
>> in "lib/launch.c" -- we're going to use the same function for the direct
>> backend as well.
>>
>> This change exposes a number of (perceived) shortcomings in libvirt; I've
>> filed <https://bugzilla.redhat.com/show_bug.cgi?id=766> about those.
>>
>> Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967
>> Signed-off-by: Laszlo Ersek
>> ---
>> lib/guestfs-internal.h | 1 +
>> lib/launch-libvirt.c | 11
>> lib/launch.c | 28
>> 3 files changed, 40 insertions(+)
>>
>> diff --git a/lib/guestfs-internal.h b/lib/guestfs-internal.h
>> index 4be351e3d3cc..a6c4266ad3fe 100644
>> --- a/lib/guestfs-internal.h
>> +++ b/lib/guestfs-internal.h
>> @@ -703,6 +703,7 @@ extern void guestfs_int_unblock_sigterm (void);
>> extern int guestfs_int_create_socketname (guestfs_h *g, const char
>> *filename, char (*sockname)[UNIX_PATH_MAX]);
>> extern void guestfs_int_register_backend (const char *name, const struct
>> backend_ops *);
>> extern int guestfs_int_set_backend (guestfs_h *g, const char *method);
>> +extern bool guestfs_int_passt_runnable (guestfs_h *g);
>>
>> /* Close all file descriptors matching the condition. */
>> #define close_file_descriptors(cond) do { \
>> diff --git a/lib/launch-libvirt.c b/lib/launch-libvirt.c
>> index d4bf1a8ff242..994909a35f2d 100644
>> --- a/lib/launch-libvirt.c
>> +++ b/lib/launch-libvirt.c
>> @@ -1414,6 +1414,17 @@ construct_libvirt_xml_devices (guestfs_h *g,
>> guestfs_int_version_ge (>data->libvirt_version, 3, 8, 0)) {
>>start_element ("interface") {
>> attribute ("type", "user");
>> +/* If libvirt is 9.0.0+ and "passt" is available, ask for passt
>> rather
>> + * than SLIRP (RHBZ#2184967). Note that this causes some
>> + * appliance-visible changes (although network connectivity is
>> certainly
>> + * functional); refer to RHBZ#766 about those.
>> + */
>> +if (guestfs_int_version_ge (>data->libvirt_version, 9, 0,
>> 0) &&
>> +guestfs_int_passt_runnable (g)) {
>> + start_element ("backend") {
>> +attribute ("type", "passt");
>> + } end_element ();
>> +}
>> start_element ("model") {
>>attribute ("type", "virtio");
>> } end_element ();
>> diff --git a/lib/launch.c b/lib/launch.c
>> index 6e08b12006da..94c8f676d8bd 100644
>> --- a/lib/launch.c
>> +++ b/lib/launch.c
>> @@ -36,6 +36,7 @@
>> #include
>> #include
>> #include
>> +#include
>> #include
>> #include
>> #include
>> @@ -414,6 +415,33 @@ guestfs_int_set_backend (guestfs_h *g, const char
>> *method)
>>return 0;
>> }
>>
>> +/**
>> + * Return C if we can call S>, and it exits with
>> status
>> + * C<0> or C<1>.
>> + *
>> + * (At least C terminates with
>> status
>> + * C<1> in response to "--help", which is arguably wrong, and potentially
>> + * subject to change, but it doesn't really matter.)
>> + */
>> +bool
>> +guestfs_int_passt_runnable (guestfs_h *g)
>> +{
>> + CLEANUP_CMD_CLOSE struct command *cmd = NULL;
>> + int r, ex;
>> +
>> + cmd = guestfs_int_new_command (g);
>> + if (cmd == NULL)
>> +return false;
>> +
>> + guestfs_int_cmd_add_string_unquoted (cmd, "passt --help");
>
> Do we need " >/dev/null 2>&1" here to avoid unnecessary messages being
> printed when libguestfs is not in verbose mode?
Yes. Thanks for pointing it out. I didn't quite know what to do with the
stdout / stderr of "passt --help".
So is the following pattern the right one?
guestfs_int_cmd_add_string_unquoted (cmd, "passt --help");
if (!g->verbose)
guestfs_int_cmd_add_string_unquoted (cmd, " >/dev/null 2>&1");
>
> Apart from that:
>
> Reviewed-by: Richard W.M. Jones
Thanks!
Laszlo
>
> Rich.
>
>> + r = guestfs_int_cmd_run (cmd);
>> + if (r == -1 || !WIFEXITED (r))
>> +return false;
>> +
>> + ex = WEXITSTATUS (r);
>> + return ex == 0 || ex == 1;
>> +}
>> +
>> /* This hack is only required to make static linking work. See:
>> *
>> https://stackoverflow.com/questions/1202494/why-doesnt-attribute-constructor-work-in-a-static-library
>> */
>>
>> ___
>> Libguestfs mailing list
>> Libguestfs@redhat.com
>> https://listman.redhat.com/mailman/listinfo/libguestfs
>
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [libnbd PATCH 2/2] tests: Add coverage of nbd_aio_opt_* in wrong state
On 7/13/23 21:29, Eric Blake wrote:
> Enhance the regression tests to prove that the completion callback is
> not reached if the aio call itself reports an error; only the .free
> callback is guaranteed.
>
> Also add some asserts to the library code that may aid future readers
> in seeing how we track transfer semantics of a callback.
>
> Goes hand in hand with the documentation cleanups made in the previous
> patch, but done separately for ease of backporting as nbd_aio_opt
> calls are not in as many releases.
>
> Reported-by: Tage Johansson
> Signed-off-by: Eric Blake
> ---
> lib/opt.c | 2 +
> tests/Makefile.am | 5 +
> tests/errors-not-negotiating-aio.c | 170 +
> .gitignore | 1 +
> 4 files changed, 178 insertions(+)
> create mode 100644 tests/errors-not-negotiating-aio.c
Acked-by: Laszlo Ersek
Laszlo
>
> diff --git a/lib/opt.c b/lib/opt.c
> index f58d5e19..1446eef3 100644
> --- a/lib/opt.c
> +++ b/lib/opt.c
> @@ -223,6 +223,7 @@ nbd_unlocked_opt_list (struct nbd_handle *h,
> nbd_list_callback *list)
>if (nbd_unlocked_aio_opt_list (h, , ) == -1)
> return -1;
>
> + assert (CALLBACK_IS_NULL (l));
>SET_CALLBACK_TO_NULL (*list);
>if (wait_for_option (h) == -1)
> return -1;
> @@ -269,6 +270,7 @@ opt_meta_context_queries (struct nbd_handle *h,
>if (aio_opt_meta_context_queries (h, opt, queries, , ) == -1)
> return -1;
>
> + assert (CALLBACK_IS_NULL (l));
>SET_CALLBACK_TO_NULL (*context);
>if (wait_for_option (h) == -1)
> return -1;
> diff --git a/tests/Makefile.am b/tests/Makefile.am
> index 3a93251e..6eddcb7a 100644
> --- a/tests/Makefile.am
> +++ b/tests/Makefile.am
> @@ -171,6 +171,7 @@ check_PROGRAMS += \
> errors-connect-null \
> errors-connect-twice \
> errors-not-negotiating \
> + errors-not-negotiating-aio \
> errors-notify-not-blocked \
> errors-bad-cookie \
> errors-pread-structured \
> @@ -243,6 +244,7 @@ TESTS += \
> errors-connect-null \
> errors-connect-twice \
> errors-not-negotiating \
> + errors-not-negotiating-aio \
> errors-notify-not-blocked \
> errors-bad-cookie \
> errors-pread-structured \
> @@ -332,6 +334,9 @@ errors_connect_twice_LDADD = $(top_builddir)/lib/libnbd.la
> errors_not_negotiating_SOURCES = errors-not-negotiating.c
> errors_not_negotiating_LDADD = $(top_builddir)/lib/libnbd.la
>
> +errors_not_negotiating_aio_SOURCES = errors-not-negotiating-aio.c
> +errors_not_negotiating_aio_LDADD = $(top_builddir)/lib/libnbd.la
> +
> errors_notify_not_blocked_SOURCES = errors-notify-not-blocked.c
> errors_notify_not_blocked_LDADD = $(top_builddir)/lib/libnbd.la
>
> diff --git a/tests/errors-not-negotiating-aio.c
> b/tests/errors-not-negotiating-aio.c
> new file mode 100644
> index ..b09cae82
> --- /dev/null
> +++ b/tests/errors-not-negotiating-aio.c
> @@ -0,0 +1,170 @@
> +/* NBD client library in userspace
> + * Copyright Red Hat
> + *
> + * This library is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2 of the License, or (at your option) any later version.
> + *
> + * This library is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with this library; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301
> USA
> + */
> +
> +/* Deliberately provoke some errors and check the error messages from
> + * nbd_get_error etc look reasonable.
> + */
> +
> +#include
> +
> +#include
> +#include
> +#include
> +#include
> +
> +#include
> +
> +static char *progname;
> +
> +static void
> +check (int experr, const char *prefix)
> +{
> + const char *msg = nbd_get_error ();
> + int errnum = nbd_get_errno ();
> +
> + fprintf (stderr, "error: \"%s\"\n", msg);
> + fprintf (stderr, "errno: %d (%s)\n", errnum, strerror (errnum));
> + if (strncmp (msg, prefix, strlen (prefix)) != 0) {
> +fprintf (stderr, "%s: test failed: missing context prefix: %s\n",
> + progname, msg);
> +exit (EXIT_FAILURE);
> + }
> + if (errnum != e
Re: [Libguestfs] [libnbd PATCH 1/2] api: Tighten rules on completion.callback
On 7/13/23 21:29, Eric Blake wrote:
> The documentation has claimed since commit 6f4dcdab that any
> completion callback will be called exactly once; but this is not
> consistent with the code: if nbd_aio_* itself returns an error, then
> nothing is queued and the user does not need to wait for a completion
> callback to know how the command failed. We could tweak the generator
> to call completion.callback no matter what, but since the
> completion.free callback already serves that role, it's easier to fix
> the documentation to match reality. After all, one only needs
> completion status if an aio command returned success (if it returned
> failure, we know that there is nothing that is going to complete
> later).
>
> However, there was one place where we indeed fail to call
> completion.callback, even though the corresponding aio call returned
> success, which can strand a user that was depending on the callback to
> know that the pending aio command failed after all. That's when a
> call to nbd_close() interrupts a connection while commands are in
> flight. This problem appears to have been around even before commit
> 52b9b492 (v0.9.8) when we finally settled on having .free callbacks in
> the first place.
>
> Beef up the closure-lifetimes unit test to more robustly check the
> various conditions guaranteed by the updated documentation, and to
> expose the previous skip of a completion callback during nbd_close.
>
> In summary, the behavior we want (where sequence is important) is:
>
> - aio command fails:
> mid-command .callback: 0 calls
> completion .callback: 0 calls
> mid-command .free: 1 call
> completion .free: 1 call
>
> - aio command succeeds:
> mid-command .callback: 0, 1, or multiple calls
> completion .callback: 1 call
> mid-command .free: 1 call
> completion .free: 1 call
>
> Reported-by: Tage Johansson
> Fixes: 6f4dcdab ("docs: Clarify how callbacks should handle errors", v1.11.8)
> Signed-off-by: Eric Blake
> ---
> docs/libnbd.pod | 26 --
> lib/handle.c | 3 +++
> tests/closure-lifetimes.c | 14 ++
> 3 files changed, 33 insertions(+), 10 deletions(-)
>
> diff --git a/docs/libnbd.pod b/docs/libnbd.pod
> index 72f74053..433479e6 100644
> --- a/docs/libnbd.pod
> +++ b/docs/libnbd.pod
> @@ -904,9 +904,12 @@ same nbd object, as it would cause deadlock.
> =head2 Completion callbacks
>
> All of the asychronous commands have an optional completion callback
> -function that is used right before the command is marked complete,
> -after any mid-command callbacks have finished, and before any free
> -functions.
> +function that is used if the asynchronous command succeeded, right
> +before the command is marked complete, after any mid-command callbacks
> +have finished, and before any free functions. The completion callback
> +is not reached if the asynchronous command itself fails, while free
> +functions are reached regardless of the initial result of the
> +asynchronous command.
>
> When the completion callback returns C<1>, the command is
> automatically retired (there is no need to call
I agree with this approach (i.e., with adapting the documentation to
reality), but I find the language somewhat confusing. We have three terms:
- "asynchronous command succeeds"
- "command is marked complete"
- "command is retired"
The last two are mostly interchangeable in my view, and are also *not*
confusing in the documentation. But the first term is confusing, IMO; it
can easily be mistaken for meanings #2/#3. What we mean by #1 instead is
the successful queueing (or submission) of the command. The text below
does say "queued", but the text above doesn't. So I'd suggest replacing
term#1 above with "call to asynchronous API succeeds" or "asynchronous
command is successfully submitted" or something like that.
(Side comment: I'd kinda prefer an all-or-nothing approach for async
APIs. If the API fails at once, it should not take ownership of
anything; i.e., it shouldn't call either completion or free callbacks.
And if it succeeds, then it should take complete ownership. I'm not
suggesting to rework callbacks whole-sale for this, though.)
With the language clarified a bit:
Acked-by: Laszlo Ersek
Thanks
Laszlo
> @@ -946,13 +949,16 @@ mid-command callback may be reached more times than
> expected if the
> server is non-compliant.
>
> On the other hand, if a completion callback is supplied (only possible
> -with asynchronous commands), it will always be reached exactly once,
> -and the completion callback must not ignore the value pointed to by
> -C. In particular, the content of a buffer passed to
> -L or L is und
Re: [Libguestfs] [libguestfs PATCH 7/7] lib/launch-direct: support networking with passt
On 7/13/23 19:10, Laszlo Ersek wrote:
> On QEMU 7.2.0+, if "passt" is available, ask QEMU for passt ("stream")
> rather than SLIRP ("user") networking.
>
> For this, we need to run passt ourselves. Given that passt daemonizes by
> default, start it with our traditional function guestfs_int_cmd_run(). Ask
> passt to save its PID file, because in case something goes wrong before
> we're completely sure the appliance (i.e. QEMU) is up and running, we'll
> need to kill passt, the *grandchild*, ourselves.
>
> Pass "--one-off" to passt (same as libvirt). This way, once we have proof
> that QEMU has connected to passt (because the appliance shows signs of
> life), we need not clean up passt ourselves -- once QEMU exits, passt will
> see an EOF on the unix domain socket, and exit as well.
>
> Passt is way more flexible than SLIRP, and passt normally intends to
> imitate the host environment in the guest as much as possible. This means
> that, when switching from SLIRP to passt, the guest would see changes to
> the following:
>
> - guest IP address,
>
> - guest subnet mask,
>
> - host (= gateway) IP address,
>
> - host (= gateway) MAC address.
>
> Extract the SLIRP defaults into the new macros NETWORK_GW_IP and
> NETWORK_GW_MAC, and pass them explicitly to passt. In particular,
> "tests/rsync/test-rsync.sh" fails without setting the host address
> (NETWORK_GW_IP) properly.
>
> (These artifacts can be verified in the appliance with "virt-rescue
> --network", by running "ip addr", "ip route", and "ip neighbor" at the
> virt-rescue prompt. There are four scenarios: two libguest backends, times
> passt being installed or not installed.)
>
> Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967
> Signed-off-by: Laszlo Ersek
> ---
> lib/guestfs-internal.h | 26
> lib/launch-direct.c| 147 +++-
> 2 files changed, 168 insertions(+), 5 deletions(-)
>
> diff --git a/lib/guestfs-internal.h b/lib/guestfs-internal.h
> index 9ba4d4ad46cf..9f4f800e6d6e 100644
> --- a/lib/guestfs-internal.h
> +++ b/lib/guestfs-internal.h
> @@ -158,6 +158,32 @@ cleanup_mutex_unlock (pthread_mutex_t **ptr)
> #define NETWORK_ADDRESS "169.254.2.15"
> #define NETWORK_PREFIX "16"
>
> +/* The IP address and the MAC address of the host, as seen from the
> appliance.
> + *
> + * NETWORK_GW_IP should be the same as NETWORK_ADDRESS, only replacing ".15"
> in
> + * the rightmost octet with ".2". NETWORK_GW_MAC cannot be changed. These
> + * restrictions are a consequence of the following landscape:
> + *
> + * libguestfs backend userspace network stack restrictions
> + * -- ---
>
> + * direct passtNone; both NETWORK_GW_IP and
> + * NETWORK_GW_MAC can be set on
> the
> + * passt command line.
> + *
> + * direct SLIRPSLIRP hard-codes
> NETWORK_GW_MAC.
> + *
> + * libvirt passtThe domain XML does not
> expose
> + * either knob (RHBZ#766),
> even
> + * though passt could accept
> both.
> + *
> + * libvirt SLIRPThe domain XML does not
> expose
> + * either knob (RHBZ#766),
> and
> + * SLIRP hard-codes
> NETWORK_GW_MAC
> + * anyway.
> + */
> +#define NETWORK_GW_IP "169.254.2.2"
> +#define NETWORK_GW_MAC "52:56:00:00:00:02"
> +
> /* Guestfs handle and associated structures. */
>
> /* State. */
> diff --git a/lib/launch-direct.c b/lib/launch-direct.c
> index 3f46f0509736..8d6ad025a4e1 100644
> --- a/lib/launch-direct.c
> +++ b/lib/launch-direct.c
> @@ -32,6 +32,7 @@
> #include
> #include
> #include
> +#include
> #include
> #include
> #include
> @@ -48,6 +49,7 @@
> #include "guestfs-internal.h"
> #include "guestfs_protocol.h"
> #include "qemuopts.h"
> +#include "ignore-value.h"
>
> /* Per-handle data. */
> struct backend_direct_data {
> @@ -333,6 +335,110 @@ add_drives (guestfs_h *g, struct backend_direct_data
> *data,
>return 0;
> }
>
> +/**
> + * Launch passt such that it daemonizes.
> + *
> + * On error
[Libguestfs] [libguestfs PATCH 5/7] lib: move guestfs_int_create_socketname() from "launch.c" to "tmpdirs.c"
Consider the following inverted call tree (effectively a dependency tree
-- callees are at the top and near the left margin):
lazy_make_tmpdir() [lib/tmpdirs.c]
guestfs_int_lazy_make_tmpdir()[lib/tmpdirs.c]
guestfs_int_make_temp_path()[lib/tmpdirs.c]
guestfs_int_lazy_make_sockdir() [lib/tmpdirs.c]
guestfs_int_create_socketname() [lib/launch.c]
lazy_make_tmpdir() is our common workhorse / helper function that
centralizes the mkdtemp() function call.
guestfs_int_lazy_make_tmpdir() and guestfs_int_lazy_make_sockdir() are the
next level functions, both calling lazy_make_tmpdir(), just feeding it
different dirname generator functions, and different "is_runtime_dir"
qualifications. These functions create temp dirs for various, more
specific, purposes (see the manual and "lib/guestfs-internal.h" for more
details).
On a yet higher level are guestfs_int_make_temp_path() and
guestfs_int_create_socketname() -- they serve for creating *entries* in
those specific temp directories.
The discrepancy here is that, although all the other functions live in
"lib/tmpdirs.c", guestfs_int_create_socketname() is defined in
"lib/launch.c". That makes for a confusing code reading; move the function
to "lib/tmpdirs.c", just below its sibling function
guestfs_int_make_temp_path().
While at it, correct the leading comment on
guestfs_int_create_socketname() -- the socket pathname is created in the
socket directory, not in the temporary directory.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967
Signed-off-by: Laszlo Ersek
---
lib/guestfs-internal.h | 2 +-
lib/launch.c | 26
lib/tmpdirs.c | 26
3 files changed, 27 insertions(+), 27 deletions(-)
diff --git a/lib/guestfs-internal.h b/lib/guestfs-internal.h
index a6c4266ad3fe..2ee16ea1e75a 100644
--- a/lib/guestfs-internal.h
+++ b/lib/guestfs-internal.h
@@ -668,6 +668,7 @@ extern int guestfs_int_set_env_runtimedir (guestfs_h *g,
const char *envname, co
extern int guestfs_int_lazy_make_tmpdir (guestfs_h *g);
extern int guestfs_int_lazy_make_sockdir (guestfs_h *g);
extern char *guestfs_int_make_temp_path (guestfs_h *g, const char *name, const
char *extension);
+extern int guestfs_int_create_socketname (guestfs_h *g, const char *filename,
char (*sockname)[UNIX_PATH_MAX]);
extern char *guestfs_int_lazy_make_supermin_appliance_dir (guestfs_h *g);
extern void guestfs_int_remove_tmpdir (guestfs_h *g);
extern void guestfs_int_remove_sockdir (guestfs_h *g);
@@ -700,7 +701,6 @@ extern int guestfs_int_get_uefi (guestfs_h *g, char *const
*firmwares, const cha
extern int64_t guestfs_int_timeval_diff (const struct timeval *x, const struct
timeval *y);
extern void guestfs_int_launch_send_progress (guestfs_h *g, int perdozen);
extern void guestfs_int_unblock_sigterm (void);
-extern int guestfs_int_create_socketname (guestfs_h *g, const char *filename,
char (*sockname)[UNIX_PATH_MAX]);
extern void guestfs_int_register_backend (const char *name, const struct
backend_ops *);
extern int guestfs_int_set_backend (guestfs_h *g, const char *method);
extern bool guestfs_int_passt_runnable (guestfs_h *g);
diff --git a/lib/launch.c b/lib/launch.c
index 94c8f676d8bd..a0a8e1c45a51 100644
--- a/lib/launch.c
+++ b/lib/launch.c
@@ -310,32 +310,6 @@ guestfs_impl_config (guestfs_h *g,
return 0;
}
-/**
- * Create the path for a socket with the selected filename in the
- * tmpdir.
- */
-int
-guestfs_int_create_socketname (guestfs_h *g, const char *filename,
- char (*sockpath)[UNIX_PATH_MAX])
-{
- int r;
-
- if (guestfs_int_lazy_make_sockdir (g) == -1)
-return -1;
-
- r = snprintf (*sockpath, UNIX_PATH_MAX, "%s/%s", g->sockdir, filename);
- if (r >= UNIX_PATH_MAX) {
-error (g, _("socket path too long: %s/%s"), g->sockdir, filename);
-return -1;
- }
- if (r < 0) {
-perrorf (g, _("%s"), g->sockdir);
-return -1;
- }
-
- return 0;
-}
-
/**
* When the library is loaded, each backend calls this function to
* register itself in a global list.
diff --git a/lib/tmpdirs.c b/lib/tmpdirs.c
index b8e19de2bf9e..24adf98daee0 100644
--- a/lib/tmpdirs.c
+++ b/lib/tmpdirs.c
@@ -253,6 +253,32 @@ guestfs_int_make_temp_path (guestfs_h *g,
extension ? extension : "");
}
+/**
+ * Create the path for a socket with the selected filename in the
+ * sockdir.
+ */
+int
+guestfs_int_create_socketname (guestfs_h *g, const char *filename,
+ char (*sockpath)[UNIX_PATH_MAX])
+{
+ int r;
+
+ if (guestfs_int_lazy_make_sockdir (g) == -1)
+return -1;
+
+ r = snprintf (*sockpath, UNIX_PATH_MAX, "%s/%s", g->sockdir, filename);
+ if (r >= UNIX_PATH_MAX) {
+error (g, _("socket path too long: %s/%s"), g->sockdir, filename);
+return -1;
+ }
[Libguestfs] [libguestfs PATCH 2/7] lib/launch-libvirt: support networking with passt
We generate the element on libvirt 3.8.0+ already.
For selecting passt rather than SLIRP, we only need to insert the child
element . Make that child element conditional on
libvirt 9.0.0+, plus "passt --help" being executable.
For the latter, place the new helper function guestfs_int_passt_runnable()
in "lib/launch.c" -- we're going to use the same function for the direct
backend as well.
This change exposes a number of (perceived) shortcomings in libvirt; I've
filed <https://bugzilla.redhat.com/show_bug.cgi?id=766> about those.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967
Signed-off-by: Laszlo Ersek
---
lib/guestfs-internal.h | 1 +
lib/launch-libvirt.c | 11
lib/launch.c | 28
3 files changed, 40 insertions(+)
diff --git a/lib/guestfs-internal.h b/lib/guestfs-internal.h
index 4be351e3d3cc..a6c4266ad3fe 100644
--- a/lib/guestfs-internal.h
+++ b/lib/guestfs-internal.h
@@ -703,6 +703,7 @@ extern void guestfs_int_unblock_sigterm (void);
extern int guestfs_int_create_socketname (guestfs_h *g, const char *filename,
char (*sockname)[UNIX_PATH_MAX]);
extern void guestfs_int_register_backend (const char *name, const struct
backend_ops *);
extern int guestfs_int_set_backend (guestfs_h *g, const char *method);
+extern bool guestfs_int_passt_runnable (guestfs_h *g);
/* Close all file descriptors matching the condition. */
#define close_file_descriptors(cond) do { \
diff --git a/lib/launch-libvirt.c b/lib/launch-libvirt.c
index d4bf1a8ff242..994909a35f2d 100644
--- a/lib/launch-libvirt.c
+++ b/lib/launch-libvirt.c
@@ -1414,6 +1414,17 @@ construct_libvirt_xml_devices (guestfs_h *g,
guestfs_int_version_ge (>data->libvirt_version, 3, 8, 0)) {
start_element ("interface") {
attribute ("type", "user");
+/* If libvirt is 9.0.0+ and "passt" is available, ask for passt rather
+ * than SLIRP (RHBZ#2184967). Note that this causes some
+ * appliance-visible changes (although network connectivity is
certainly
+ * functional); refer to RHBZ#766 about those.
+ */
+if (guestfs_int_version_ge (>data->libvirt_version, 9, 0, 0) &&
+guestfs_int_passt_runnable (g)) {
+ start_element ("backend") {
+attribute ("type", "passt");
+ } end_element ();
+}
start_element ("model") {
attribute ("type", "virtio");
} end_element ();
diff --git a/lib/launch.c b/lib/launch.c
index 6e08b12006da..94c8f676d8bd 100644
--- a/lib/launch.c
+++ b/lib/launch.c
@@ -36,6 +36,7 @@
#include
#include
#include
+#include
#include
#include
#include
@@ -414,6 +415,33 @@ guestfs_int_set_backend (guestfs_h *g, const char *method)
return 0;
}
+/**
+ * Return C if we can call S>, and it exits with status
+ * C<0> or C<1>.
+ *
+ * (At least C terminates with status
+ * C<1> in response to "--help", which is arguably wrong, and potentially
+ * subject to change, but it doesn't really matter.)
+ */
+bool
+guestfs_int_passt_runnable (guestfs_h *g)
+{
+ CLEANUP_CMD_CLOSE struct command *cmd = NULL;
+ int r, ex;
+
+ cmd = guestfs_int_new_command (g);
+ if (cmd == NULL)
+return false;
+
+ guestfs_int_cmd_add_string_unquoted (cmd, "passt --help");
+ r = guestfs_int_cmd_run (cmd);
+ if (r == -1 || !WIFEXITED (r))
+return false;
+
+ ex = WEXITSTATUS (r);
+ return ex == 0 || ex == 1;
+}
+
/* This hack is only required to make static linking work. See:
*
https://stackoverflow.com/questions/1202494/why-doesnt-attribute-constructor-work-in-a-static-library
*/
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
[Libguestfs] [libguestfs PATCH 4/7] docs: clarify sockdir's separation
There's another reason for separating sockdir from tmpdir, beyond "shorter pathnames needed": permissions. For example, passt drops privileges such that it cannot access "/tmp", and that restricts both the unix domain socket and the PID file of passt. Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967 Signed-off-by: Laszlo Ersek --- generator/actions_properties.ml | 8 ++-- fish/guestfish.pod | 4 ++-- lib/guestfs.pod | 4 ++-- 3 files changed, 10 insertions(+), 6 deletions(-) diff --git a/generator/actions_properties.ml b/generator/actions_properties.ml index f84afb10d674..42eaaa4d81e1 100644 --- a/generator/actions_properties.ml +++ b/generator/actions_properties.ml @@ -595,13 +595,17 @@ Get the handle identifier. See C." }; name = "get_sockdir"; added = (1, 33, 8); style = RString (RPlainString, "sockdir"), [], []; blocking = false; -shortdesc = "get the temporary directory for sockets"; +shortdesc = "get the temporary directory for sockets and PID files"; longdesc = "\ -Get the directory used by the handle to store temporary socket files. +Get the directory used by the handle to store temporary socket and PID +files. This is different from C, as we need shorter paths for sockets (due to the limited buffers of filenames for UNIX sockets), and C may be too long for them. +Furthermore, sockets and PID files must be accessible to such background +services started by libguestfs that may not have permission to access +the temporary directory returned by C. The environment variable C controls the default value: If C is set, then that is the default. diff --git a/fish/guestfish.pod b/fish/guestfish.pod index ccc0825b84a0..492aa7163fcb 100644 --- a/fish/guestfish.pod +++ b/fish/guestfish.pod @@ -1548,8 +1548,8 @@ See L, L. This directory represents a user-specific directory for storing non-essential runtime files. -If it is set, then is used to store temporary sockets. Otherwise, -F is used. +If it is set, then is used to store temporary sockets and PID files. +Otherwise, F is used. See also L, L<http://www.freedesktop.org/wiki/Specifications/basedir-spec/>. diff --git a/lib/guestfs.pod b/lib/guestfs.pod index 68688f31aa5f..e46dd81f9e0a 100644 --- a/lib/guestfs.pod +++ b/lib/guestfs.pod @@ -3220,8 +3220,8 @@ See L, L. This directory represents a user-specific directory for storing non-essential runtime files. -If it is set, then is used to store temporary sockets. Otherwise, -F is used. +If it is set, then is used to store temporary sockets and PID files. +Otherwise, F is used. See also L, L<http://www.freedesktop.org/wiki/Specifications/basedir-spec/>. ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
[Libguestfs] [libguestfs PATCH 6/7] lib: introduce guestfs_int_make_pid_path()
Introduce a small function for creating pathnames for PID files.
guestfs_int_make_pid_path() is something of an amalgamation of
guestfs_int_make_temp_path() [1] and guestfs_int_create_socketname() [2]:
- it creates a pathname under sockdir, like [2],
- it uses the handle's unique counter, like [1],
- it takes a name like both [1] and [2], but the name is not size-limited
like in [2], plus we hardcode the suffix from [1] as ".pid".
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967
Signed-off-by: Laszlo Ersek
---
lib/guestfs-internal.h | 1 +
lib/tmpdirs.c | 15 +++
2 files changed, 16 insertions(+)
diff --git a/lib/guestfs-internal.h b/lib/guestfs-internal.h
index 2ee16ea1e75a..9ba4d4ad46cf 100644
--- a/lib/guestfs-internal.h
+++ b/lib/guestfs-internal.h
@@ -669,6 +669,7 @@ extern int guestfs_int_lazy_make_tmpdir (guestfs_h *g);
extern int guestfs_int_lazy_make_sockdir (guestfs_h *g);
extern char *guestfs_int_make_temp_path (guestfs_h *g, const char *name, const
char *extension);
extern int guestfs_int_create_socketname (guestfs_h *g, const char *filename,
char (*sockname)[UNIX_PATH_MAX]);
+extern char *guestfs_int_make_pid_path (guestfs_h *g, const char *name);
extern char *guestfs_int_lazy_make_supermin_appliance_dir (guestfs_h *g);
extern void guestfs_int_remove_tmpdir (guestfs_h *g);
extern void guestfs_int_remove_sockdir (guestfs_h *g);
diff --git a/lib/tmpdirs.c b/lib/tmpdirs.c
index 24adf98daee0..22b8f54b0693 100644
--- a/lib/tmpdirs.c
+++ b/lib/tmpdirs.c
@@ -279,6 +279,21 @@ guestfs_int_create_socketname (guestfs_h *g, const char
*filename,
return 0;
}
+/**
+ * Generate unique paths for PID files.
+ *
+ * Returns a unique path or NULL on error. On success, the pathname points
+ * under sockdir and not tmpdir; daemons that write PID files after dropping
+ * privileges may not have access to tmpdir.
+ */
+char *
+guestfs_int_make_pid_path (guestfs_h *g, const char *name)
+{
+ if (guestfs_int_lazy_make_sockdir (g) < 0)
+return NULL;
+ return safe_asprintf (g, "%s/%s%d.pid", g->sockdir, name, ++g->unique);
+}
+
/**
* Create the supermin appliance directory under cachedir, if it does
* not exist.
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
[Libguestfs] [libguestfs PATCH 3/7] docs: fix broken link in the guestfs manual
Commit 55202a4d49a1 ("New API: get-sockdir", 2016-02-03) added identical
language to "fish/guestfish.pod" and "src/guestfs.pod", including an
internal link L. That's appropriate for
"fish/guestfish.pod", but the same API description is generated with a
different anchor for "src/guestfs.pod". Adapt the reference.
Fixes: 55202a4d49a101392148d79cb2e1591428db2681
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967
Signed-off-by: Laszlo Ersek
---
lib/guestfs.pod | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/guestfs.pod b/lib/guestfs.pod
index c6c8cb16860c..68688f31aa5f 100644
--- a/lib/guestfs.pod
+++ b/lib/guestfs.pod
@@ -3223,7 +3223,7 @@ non-essential runtime files.
If it is set, then is used to store temporary sockets. Otherwise,
F is used.
-See also L,
+See also L,
L<http://www.freedesktop.org/wiki/Specifications/basedir-spec/>.
=back
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
[Libguestfs] [libguestfs PATCH 1/7] lib: fix NETWORK_ADDRESS: make it an actual IP address, not a subnet base
Currently we #define NETWORK_ADDRESS as "169.254.0.0". That's a bug in
libguestfs, but it's been invisible -- thus far it's been canceled out by
*independent* bugs in *both* QEMU and libvirt.
(1) With the direct backend, the current definition of NETWORK_ADDRESS
results in the following QEMU command line option:
-netdev user,id=usernet,net=169.254.0.0/16
According to the QEMU documentation, the "net" property here is supposed
to specify the *exact* IP address that the guest will receive. The guest
however receives 169.254.2.15 (I've checked that with virt-rescue).
In other words, libguestfs doesn't do what the QEMU documentation says,
and QEMU doesn't do what the QEMU documentation says either. The end
result has been good enough -- but only until now.
(2) With the libvirt backend, the current definition of NETWORK_ADDRESS
results in the following domain XML snippet:
which libvirt translates, in turn, to
-netdev {"type":"user","net":"169.254.0.0/16","id":"hostnet0"}
According to the domain XML documentation, the @address attribute here is
again supposed to specify the *exact* IP address that the guest will
receive. However, the guest receives 169.254.2.15 (I've checked that with
virt-rescue).
In other words, libguestfs doesn't do what the libvirt documentation says,
and libvirt doesn't do what the libvirt documentation says either. The end
result has been good enough -- but only until now.
Where things break down though is the subsequent passt enablement, in the
rest of this series. For example, when using the libvirt backend together
with passt, libvirt translates the @address attribute to passt's
"--address" option, but passt takes the address *verbatim*, and not as a
subnet base address. That difference is visible in the appliance; for
example, when running virt-rescue on a Fedora 38 image, and issuing "ip
addr". Namely, after enabling passt for the libvirt backend, the
guest-visible IP address changes from 169.254.2.15 to 169.254.0.0, which
is an IP address that makes no sense for an endpoint.
Fix the latent bug by specifying the actual guest IP address we want, in
NETWORK_ADDRESS.
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967
Signed-off-by: Laszlo Ersek
---
lib/guestfs-internal.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/guestfs-internal.h b/lib/guestfs-internal.h
index c7ef32277e93..4be351e3d3cc 100644
--- a/lib/guestfs-internal.h
+++ b/lib/guestfs-internal.h
@@ -155,7 +155,7 @@ cleanup_mutex_unlock (pthread_mutex_t **ptr)
/* Network address and network mask (expressed as address prefix) that the
* appliance will see (if networking is enabled).
*/
-#define NETWORK_ADDRESS "169.254.0.0"
+#define NETWORK_ADDRESS "169.254.2.15"
#define NETWORK_PREFIX "16"
/* Guestfs handle and associated structures. */
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
[Libguestfs] [libguestfs PATCH 7/7] lib/launch-direct: support networking with passt
On QEMU 7.2.0+, if "passt" is available, ask QEMU for passt ("stream")
rather than SLIRP ("user") networking.
For this, we need to run passt ourselves. Given that passt daemonizes by
default, start it with our traditional function guestfs_int_cmd_run(). Ask
passt to save its PID file, because in case something goes wrong before
we're completely sure the appliance (i.e. QEMU) is up and running, we'll
need to kill passt, the *grandchild*, ourselves.
Pass "--one-off" to passt (same as libvirt). This way, once we have proof
that QEMU has connected to passt (because the appliance shows signs of
life), we need not clean up passt ourselves -- once QEMU exits, passt will
see an EOF on the unix domain socket, and exit as well.
Passt is way more flexible than SLIRP, and passt normally intends to
imitate the host environment in the guest as much as possible. This means
that, when switching from SLIRP to passt, the guest would see changes to
the following:
- guest IP address,
- guest subnet mask,
- host (= gateway) IP address,
- host (= gateway) MAC address.
Extract the SLIRP defaults into the new macros NETWORK_GW_IP and
NETWORK_GW_MAC, and pass them explicitly to passt. In particular,
"tests/rsync/test-rsync.sh" fails without setting the host address
(NETWORK_GW_IP) properly.
(These artifacts can be verified in the appliance with "virt-rescue
--network", by running "ip addr", "ip route", and "ip neighbor" at the
virt-rescue prompt. There are four scenarios: two libguest backends, times
passt being installed or not installed.)
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967
Signed-off-by: Laszlo Ersek
---
lib/guestfs-internal.h | 26
lib/launch-direct.c| 147 +++-
2 files changed, 168 insertions(+), 5 deletions(-)
diff --git a/lib/guestfs-internal.h b/lib/guestfs-internal.h
index 9ba4d4ad46cf..9f4f800e6d6e 100644
--- a/lib/guestfs-internal.h
+++ b/lib/guestfs-internal.h
@@ -158,6 +158,32 @@ cleanup_mutex_unlock (pthread_mutex_t **ptr)
#define NETWORK_ADDRESS "169.254.2.15"
#define NETWORK_PREFIX "16"
+/* The IP address and the MAC address of the host, as seen from the appliance.
+ *
+ * NETWORK_GW_IP should be the same as NETWORK_ADDRESS, only replacing ".15" in
+ * the rightmost octet with ".2". NETWORK_GW_MAC cannot be changed. These
+ * restrictions are a consequence of the following landscape:
+ *
+ * libguestfs backend userspace network stack restrictions
+ * -- ---
+ * direct passtNone; both NETWORK_GW_IP and
+ * NETWORK_GW_MAC can be set on
the
+ * passt command line.
+ *
+ * direct SLIRPSLIRP hard-codes
NETWORK_GW_MAC.
+ *
+ * libvirt passtThe domain XML does not expose
+ * either knob (RHBZ#766),
even
+ * though passt could accept both.
+ *
+ * libvirt SLIRPThe domain XML does not expose
+ * either knob (RHBZ#766), and
+ * SLIRP hard-codes NETWORK_GW_MAC
+ * anyway.
+ */
+#define NETWORK_GW_IP "169.254.2.2"
+#define NETWORK_GW_MAC "52:56:00:00:00:02"
+
/* Guestfs handle and associated structures. */
/* State. */
diff --git a/lib/launch-direct.c b/lib/launch-direct.c
index 3f46f0509736..8d6ad025a4e1 100644
--- a/lib/launch-direct.c
+++ b/lib/launch-direct.c
@@ -32,6 +32,7 @@
#include
#include
#include
+#include
#include
#include
#include
@@ -48,6 +49,7 @@
#include "guestfs-internal.h"
#include "guestfs_protocol.h"
#include "qemuopts.h"
+#include "ignore-value.h"
/* Per-handle data. */
struct backend_direct_data {
@@ -333,6 +335,110 @@ add_drives (guestfs_h *g, struct backend_direct_data
*data,
return 0;
}
+/**
+ * Launch passt such that it daemonizes.
+ *
+ * On error, -1 is returned; C and C are not modified.
+ *
+ * On success, 0 is returned. C contains the PID of the passt
+ * background process. C contains the pathname of the unix domain
+ * socket where passt will accept a single connection.
+ */
+static int
+launch_passt (guestfs_h *g, long *passt_pid, char (*sockpath)[UNIX_PATH_MAX])
+{
+ int rc;
+ char sockpath_local[sizeof *sockpath];
+ char *pid_path;
+ struct command *cmd;
+ int passt_status;
+ int passt_exit;
+ char *pid_str;
+ long passt_pid_local;
+ char *endptr;
+
+ rc = -1;
+ if (guestfs_int_create_socketname (g, "passt.sock", _local) == -1)
+return rc;
+
+ pid_p
[Libguestfs] [libguestfs PATCH 0/7] lib: support networking with passt
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2184967 This series makes both backends prefer passt over slirp for appliance networking, if QEMU or libvirt (respectively) is recent enough, and passt is installed. My test setup is: $ virt-builder fedora-38 Then, each test run looks like this: Terminal#1: $ ./run rescue/virt-rescue --network --ro -a fedora-38.img Terminal#2: - check if "passt" is running (ps -ef, pgrep, ...), provided it *should* be running Terminal#1: > ping -c 3 8.8.8.8 > ip neighbor > ip addr > ip route Expected results (in the above order), always on the "eth0" lines: - all three pings succeed (get replies) - 52:56:00:00:00:02 - 169.254.2.15/16 - default via 169.254.2.2 Actual results: (1) At current master (13c7052ff96d): (1.1) With "LIBGUESTFS_BACKEND=direct": Pass, this is the baseline. (1.2) With "LIBGUESTFS_BACKEND=libvirt": Pass, this is the baseline. (2) With this series applied: (2.1) With passt not installed: (2.1.1) With "LIBGUESTFS_BACKEND=direct": Pass, this is a regression test concerning the absence of "passt". (2.1.2) With "LIBGUESTFS_BACKEND=libvirt": Pass, this is a regression test concerning the absence of "passt". (2.2) With passt installed: (2.2.1) With "LIBGUESTFS_BACKEND=direct": Pass, this verifies the new feature. (2.2.2) With "LIBGUESTFS_BACKEND=libvirt": Basic connectivity works fine (i.e., the pings). The "ip neighbor" and "ip route" checks fail. In addition, the "ip addr" check *partially* fails. All that is due to libvirt bugs: (a) Libvirt specifies the *guest MAC* (virtio-net MAC) as the *host MAC* for passt, with a wrong "--mac-addr" option (from libvirt commit a56f0168d576, "qemu: hook up passt config to qemu domains", 2023-01-10). This breaks "ip neighbor". (b) Libvirt doesn't pass the "--gateway" option to passt. This breaks "ip route". Namely, passt (following its default behavior) sets the guest's gateway to 192.168.0.1, which is the gateway for my *host*. (c) "ip addr" also reports "169.254.2.15/1". The IP address is fine, but the netmask is wrong; it's not /16. This is most likely a consequence of (b) -- normally the gateway is supposed to be on the same Ethernet segment (subnet) as the endpoint! 192 decimal is 1100 binary, while 169 decimal is 10101001 binary, and their longest common initial bit sequence is just 1 bit -- hence the /1, most likely. I've filed <https://bugzilla.redhat.com/show_bug.cgi?id=766> about these. The upshot is that "appliance networking with passt" works with either backend, but with the libvirt backend, there are some appliance-visible differences from the SLIRP environment. Whether that's a problem in practice, I can't tell (probably not a problem) -- dependent on future decisions about RHBZ#766, we might want to update the libvirt backend code introduced here. Laszlo Laszlo Ersek (7): lib: fix NETWORK_ADDRESS: make it an actual IP address, not a subnet base lib/launch-libvirt: support networking with passt docs: fix broken link in the guestfs manual docs: clarify sockdir's separation lib: move guestfs_int_create_socketname() from "launch.c" to "tmpdirs.c" lib: introduce guestfs_int_make_pid_path() lib/launch-direct: support networking with passt fish/guestfish.pod | 4 +- generator/actions_properties.ml | 8 +- lib/guestfs-internal.h | 32 - lib/guestfs.pod | 6 +- lib/launch-direct.c | 147 +++- lib/launch-libvirt.c| 11 ++ lib/launch.c| 54 +++ lib/tmpdirs.c | 41 ++ 8 files changed, 263 insertions(+), 40 deletions(-) base-commit: 13c7052ff96d5ee99ec1b1252f1a3b4d7aed44d2 ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [libguestfs PATCH] lib: remove guestfs_int_cmd_clear_close_files()
On 7/11/23 15:10, Richard W.M. Jones wrote:
> On Tue, Jul 11, 2023 at 01:39:06PM +0200, Laszlo Ersek wrote:
>> The last (only?) caller of guestfs_int_cmd_clear_close_files() disappeared
>> in commit e4c396888056 ("lib/info: Remove /dev/fd hacking and pass a true
>> filename to qemu-img info.", 2018-01-23), part of v1.37.36.
>>
>> Simplify the code by removing guestfs_int_cmd_clear_close_files().
>>
>> Signed-off-by: Laszlo Ersek
>> ---
>> lib/guestfs-internal.h | 1 -
>> lib/command.c | 37 ++---
>> 2 files changed, 10 insertions(+), 28 deletions(-)
>>
>> diff --git a/lib/guestfs-internal.h b/lib/guestfs-internal.h
>> index fb55e02614f5..c7ef32277e93 100644
>> --- a/lib/guestfs-internal.h
>> +++ b/lib/guestfs-internal.h
>> @@ -751,7 +751,6 @@ extern void guestfs_int_cmd_set_stdout_callback (struct
>> command *, cmd_stdout_ca
>> extern void guestfs_int_cmd_set_stderr_to_stdout (struct command *);
>> extern void guestfs_int_cmd_set_child_rlimit (struct command *, int
>> resource, long limit);
>> extern void guestfs_int_cmd_clear_capture_errors (struct command *);
>> -extern void guestfs_int_cmd_clear_close_files (struct command *);
>> extern void guestfs_int_cmd_set_child_callback (struct command *,
>> cmd_child_callback child_callback, void *data);
>> extern int guestfs_int_cmd_run (struct command *);
>> extern void guestfs_int_cmd_close (struct command *);
>> diff --git a/lib/command.c b/lib/command.c
>> index 515ef624bd96..82a47bafa9e5 100644
>> --- a/lib/command.c
>> +++ b/lib/command.c
>> @@ -152,9 +152,6 @@ struct command
>>/* When using the pipe_* APIs, stderr is pointed to a temporary file. */
>>char *error_file;
>>
>> - /* Close file descriptors (defaults to true). */
>> - bool close_files;
>> -
>>/* Supply a callback to receive stdout. */
>>cmd_stdout_callback stdout_callback;
>>void *stdout_data;
>> @@ -186,7 +183,6 @@ guestfs_int_new_command (guestfs_h *g)
>>cmd = safe_calloc (g, 1, sizeof *cmd);
>>cmd->g = g;
>>cmd->capture_errors = true;
>> - cmd->close_files = true;
>>cmd->errorfd = -1;
>>cmd->outfd = -1;
>>return cmd;
>> @@ -358,17 +354,6 @@ guestfs_int_cmd_clear_capture_errors (struct command
>> *cmd)
>>cmd->capture_errors = false;
>> }
>>
>> -/**
>> - * Don't close file descriptors after the fork.
>> - *
>> - * XXX Should allow single fds to be sent to child process.
>> - */
>> -void
>> -guestfs_int_cmd_clear_close_files (struct command *cmd)
>> -{
>> - cmd->close_files = false;
>> -}
>> -
>> /**
>> * Set a function to be executed in the child, right before the
>> * execution. Can be used to setup the child, for example changing
>> @@ -564,18 +549,16 @@ run_child (struct command *cmd, char **env)
>>for (i = 1; i < NSIG; ++i)
>> sigaction (i, , NULL);
>>
>> - if (cmd->close_files) {
>> -/* Close all other file descriptors. This ensures that we don't
>> - * hold open (eg) pipes from the parent process.
>> - */
>> -max_fd = sysconf (_SC_OPEN_MAX);
>> -if (max_fd == -1)
>> - max_fd = 1024;
>> -if (max_fd > 65536)
>> - max_fd = 65536;/* bound the amount of work we do here */
>> -for (fd = 3; fd < max_fd; ++fd)
>> - close (fd);
>> - }
>> + /* Close all other file descriptors. This ensures that we don't
>> + * hold open (eg) pipes from the parent process.
>> + */
>> + max_fd = sysconf (_SC_OPEN_MAX);
>> + if (max_fd == -1)
>> +max_fd = 1024;
>> + if (max_fd > 65536)
>> +max_fd = 65536;/* bound the amount of work we do here */
>> + for (fd = 3; fd < max_fd; ++fd)
>> +close (fd);
>>
>>/* Set the umask for all subcommands to something sensible (RHBZ#610880).
>> */
>>umask (022);
>
> Reviewed-by: Richard W.M. Jones
>
Commit 13c7052ff96d.
Thanks!
Laszlo
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
[Libguestfs] [libguestfs PATCH] lib: remove guestfs_int_cmd_clear_close_files()
The last (only?) caller of guestfs_int_cmd_clear_close_files() disappeared
in commit e4c396888056 ("lib/info: Remove /dev/fd hacking and pass a true
filename to qemu-img info.", 2018-01-23), part of v1.37.36.
Simplify the code by removing guestfs_int_cmd_clear_close_files().
Signed-off-by: Laszlo Ersek
---
lib/guestfs-internal.h | 1 -
lib/command.c | 37 ++---
2 files changed, 10 insertions(+), 28 deletions(-)
diff --git a/lib/guestfs-internal.h b/lib/guestfs-internal.h
index fb55e02614f5..c7ef32277e93 100644
--- a/lib/guestfs-internal.h
+++ b/lib/guestfs-internal.h
@@ -751,7 +751,6 @@ extern void guestfs_int_cmd_set_stdout_callback (struct
command *, cmd_stdout_ca
extern void guestfs_int_cmd_set_stderr_to_stdout (struct command *);
extern void guestfs_int_cmd_set_child_rlimit (struct command *, int resource,
long limit);
extern void guestfs_int_cmd_clear_capture_errors (struct command *);
-extern void guestfs_int_cmd_clear_close_files (struct command *);
extern void guestfs_int_cmd_set_child_callback (struct command *,
cmd_child_callback child_callback, void *data);
extern int guestfs_int_cmd_run (struct command *);
extern void guestfs_int_cmd_close (struct command *);
diff --git a/lib/command.c b/lib/command.c
index 515ef624bd96..82a47bafa9e5 100644
--- a/lib/command.c
+++ b/lib/command.c
@@ -152,9 +152,6 @@ struct command
/* When using the pipe_* APIs, stderr is pointed to a temporary file. */
char *error_file;
- /* Close file descriptors (defaults to true). */
- bool close_files;
-
/* Supply a callback to receive stdout. */
cmd_stdout_callback stdout_callback;
void *stdout_data;
@@ -186,7 +183,6 @@ guestfs_int_new_command (guestfs_h *g)
cmd = safe_calloc (g, 1, sizeof *cmd);
cmd->g = g;
cmd->capture_errors = true;
- cmd->close_files = true;
cmd->errorfd = -1;
cmd->outfd = -1;
return cmd;
@@ -358,17 +354,6 @@ guestfs_int_cmd_clear_capture_errors (struct command *cmd)
cmd->capture_errors = false;
}
-/**
- * Don't close file descriptors after the fork.
- *
- * XXX Should allow single fds to be sent to child process.
- */
-void
-guestfs_int_cmd_clear_close_files (struct command *cmd)
-{
- cmd->close_files = false;
-}
-
/**
* Set a function to be executed in the child, right before the
* execution. Can be used to setup the child, for example changing
@@ -564,18 +549,16 @@ run_child (struct command *cmd, char **env)
for (i = 1; i < NSIG; ++i)
sigaction (i, , NULL);
- if (cmd->close_files) {
-/* Close all other file descriptors. This ensures that we don't
- * hold open (eg) pipes from the parent process.
- */
-max_fd = sysconf (_SC_OPEN_MAX);
-if (max_fd == -1)
- max_fd = 1024;
-if (max_fd > 65536)
- max_fd = 65536;/* bound the amount of work we do here */
-for (fd = 3; fd < max_fd; ++fd)
- close (fd);
- }
+ /* Close all other file descriptors. This ensures that we don't
+ * hold open (eg) pipes from the parent process.
+ */
+ max_fd = sysconf (_SC_OPEN_MAX);
+ if (max_fd == -1)
+max_fd = 1024;
+ if (max_fd > 65536)
+max_fd = 65536;/* bound the amount of work we do here */
+ for (fd = 3; fd < max_fd; ++fd)
+close (fd);
/* Set the umask for all subcommands to something sensible (RHBZ#610880). */
umask (022);
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [v2v PATCH v2 2/3] lib/utils: make "chown_for_libvirt_rhbz_1045069" fail hard
On 6/30/23 09:38, Richard W.M. Jones wrote: > On Fri, Jun 30, 2023 at 09:32:33AM +0200, Laszlo Ersek wrote: >> On 6/29/23 19:16, Richard W.M. Jones wrote: >>> On Thu, Jun 29, 2023 at 06:40:27PM +0200, Laszlo Ersek wrote: >> >>>> Either way, I can restore the parens. Do you want me to submit a v3? >>> >>> I think the parens are clearer. Don't need a v3, thanks! >> >> Can I add your R-b that way (with the parens reinstated) to this patch >> as well? > > Yes please! Thanks, commit range 97d8c28b7eb1..dcfea1b9b5d0. Laszlo ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [v2v PATCH v2 2/3] lib/utils: make "chown_for_libvirt_rhbz_1045069" fail hard
On 6/29/23 19:16, Richard W.M. Jones wrote: > On Thu, Jun 29, 2023 at 06:40:27PM +0200, Laszlo Ersek wrote: >> Either way, I can restore the parens. Do you want me to submit a v3? > > I think the parens are clearer. Don't need a v3, thanks! Can I add your R-b that way (with the parens reinstated) to this patch as well? Thanks! Laszlo ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [v2v PATCH v2 2/3] lib/utils: make "chown_for_libvirt_rhbz_1045069" fail hard
On 6/29/23 17:48, Richard W.M. Jones wrote: > On Thu, Jun 29, 2023 at 05:39:34PM +0200, Laszlo Ersek wrote: >> On 6/29/23 14:54, Richard W.M. Jones wrote: >>> On Thu, Jun 29, 2023 at 02:34:42PM +0200, Laszlo Ersek wrote: >>>> Currently "chown_for_libvirt_rhbz_1045069" is best effort; if it fails, we >>>> suppress the exception (we log it in verbose mode only, even). >>>> >>>> That's not proved helpful: it almost certainly leads to later errors, but >>>> those errors are less clear than the original (suppressed) exception. >>>> Namely, the user sees something like >>>> >>>>> Failed to connect to '/tmp/v2v.sKlulY/in0': Permission denied >>>> >>>> rather than >>>> >>>>> Failed to connect socket to '/var/run/libvirt/virtqemud-sock-ro': >>>>> Connection refused >>>> >>>> So just allow the exception to propagate outwards. >>>> >>>> And then, now that "chown_for_libvirt_rhbz_1045069" will be able to fail, >>>> hoist the call to "On_exit.rm_rf" before the call to >>>> "chown_for_libvirt_rhbz_1045069", after creating the v2v temporary >>>> directory. In the current order, if "chown_for_libvirt_rhbz_1045069" threw >>>> an exception, then we'd leak the temp dir in the filesystem. >>>> >>>> Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2182024 >>>> Signed-off-by: Laszlo Ersek >>>> --- >>>> >>>> Notes: >>>> v2: >>>> >>>> - new patch >>>> >>>> lib/utils.mli | 5 +--- >>>> lib/utils.ml | 26 >>>> 2 files changed, 11 insertions(+), 20 deletions(-) >>>> >>>> diff --git a/lib/utils.mli b/lib/utils.mli >>>> index cf88a467fd54..391a2a351ec7 100644 >>>> --- a/lib/utils.mli >>>> +++ b/lib/utils.mli >>>> @@ -67,10 +67,7 @@ val chown_for_libvirt_rhbz_1045069 : string -> unit >>>> to root-owned files and directories. To fix this, provide >>>> a function to chown things we might need to qemu:root so >>>> qemu can access them. Note that root normally ignores >>>> -permissions so can still access the resource. >>>> - >>>> -This is best-effort. If something fails then we carry >>>> -on and hope for the best. *) >>>> +permissions so can still access the resource. *) >>>> >>>> val error_if_no_ssh_agent : unit -> unit >>>> >>>> diff --git a/lib/utils.ml b/lib/utils.ml >>>> index 174c01b1e92f..7d69c9e0d177 100644 >>>> --- a/lib/utils.ml >>>> +++ b/lib/utils.ml >>>> @@ -149,21 +149,15 @@ let backend_is_libvirt () = >>>> >>>> let rec chown_for_libvirt_rhbz_1045069 file = >>>>let running_as_root = Unix.geteuid () = 0 in >>>> - if running_as_root && backend_is_libvirt () then ( >>>> -try >>>> - let user = Option.value ~default:"qemu" (libvirt_qemu_user ()) in >>>> - let uid = >>>> -if String.is_prefix user "+" then >>>> - int_of_string (String.sub user 1 (String.length user - 1)) >>>> -else >>>> - (Unix.getpwnam user).pw_uid in >>>> - debug "setting owner of %s to %d:root" file uid; >>>> - Unix.chown file uid 0 >>>> -with >>>> -| exn -> (* Print exception, but continue. *) >>>> - debug "could not set owner of %s: %s" >>>> - file (Printexc.to_string exn) >>>> - ) >>>> + if running_as_root && backend_is_libvirt () then >>>> +let user = Option.value ~default:"qemu" (libvirt_qemu_user ()) in >>>> +let uid = >>>> + if String.is_prefix user "+" then >>>> +int_of_string (String.sub user 1 (String.length user - 1)) >>>> + else >>>> +(Unix.getpwnam user).pw_uid in >>>> +debug "setting owner of %s to %d:root" file uid; >>>> +Unix.chown file uid 0 >>> >>> Hmm, doesn't this need parens around the 'then' clause? >> >> Ding ding ding, it does not :) >> >> I'm happy you asked. (I was so frustrated to discover this *once again* >> that I almost sen
Re: [Libguestfs] [PATCH libguestfs] generator: Add --chown option for virt-customize
On 6/29/23 14:49, Richard W.M. Jones wrote:
> Also this updates the common submodule to include the changes.
>
> Fixes: https://github.com/rwmjones/guestfs-tools/issues/12
> ---
> common | 2 +-
> generator/customize.ml | 28
> 2 files changed, 29 insertions(+), 1 deletion(-)
>
> diff --git a/common b/common
> index d61cd820b4..bbb54714ce 16
> --- a/common
> +++ b/common
> @@ -1 +1 @@
> -Subproject commit d61cd820b49e403848d15c5deaccbf8dd7045370
> +Subproject commit bbb54714ce24c76e5761d96a0227a753896dc4c4
> diff --git a/generator/customize.ml b/generator/customize.ml
> index aa7ac8e8af..8d3dec3e24 100644
> --- a/generator/customize.ml
> +++ b/generator/customize.ml
> @@ -95,6 +95,34 @@ I: C by default would be decimal,
> unless you prefix
> it with C<0> to get octal, ie. use C<0700> not C<700>.";
>};
>
> + { op_name = "chown";
> +op_type = StringPair "UID.GID:PATH";
> +op_discrim = "`Chown";
> +op_shortdesc = "Change the owner user and group ID of a file or
> directory";
> +op_pod_longdesc = "\
> +Change the owner user and group ID of a file or directory in the guest.
> +Note:
> +
> +=over 4
> +
> +=item *
> +
> +Only numeric UIDs and GIDs will work, and these may not be the same
> +inside the guest as on the host.
> +
> +=item *
> +
> +This will not work with Windows guests.
> +
> +=back
> +
> +For example:
> +
> + virt-customize --chown '0.0:/var/log/audit.log'
> +
> +See also: I<--upload>.";
> + };
> +
>{ op_name = "commands-from-file";
> op_type = StringFn ("FILENAME", "customize_read_from_file");
> op_discrim = "`CommandsFromFile";
Acked-by: Laszlo Ersek
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH guestfs-tools] customize: Implement --chown option
On 6/29/23 14:50, Richard W.M. Jones wrote: > This currently only works with numeric UID.GID. In theory in future > we could look up IDs from the guest password file (eg. using Augeas) > and do the right thing, but that's a bunch more work. > > For example: > > $ ./builder/virt-builder fedora-36 --chown 1.1:/var/tmp > [ 1.0] Downloading: http://builder.libguestfs.org/fedora-36.xz > [ 1.5] Planning how to build this image > [ 1.5] Uncompressing > [ 3.4] Opening the new disk > [ 7.3] Setting a random seed > [ 7.3] Changing owner of /var/tmp to 1.1 > [ 7.3] Setting passwords > virt-builder: Setting random password of root to x8fu6z7QNEdPeZHF > [ 7.8] SELinux relabelling > [ 12.0] Finishing off >Output file: fedora-36.img >Output size: 6.0G > Output format: raw > Total usable space: 6.0G > Free space: 4.7G (79%) > > $ guestfish -a fedora-36.img -i ll /var > total 8 > drwxr-xr-x. 18 root root 4096 May 12 2022 . > dr-xr-xr-x. 18 root root 235 May 12 2022 .. > drwxr-xr-x. 2 root root6 Jan 20 2022 adm > drwxr-xr-x. 9 root root 101 May 12 2022 cache > drwxr-xr-x. 3 root root 18 May 12 2022 db > drwxr-xr-x. 2 root root6 Jan 20 2022 empty > drwxr-xr-x. 2 root root6 Jan 20 2022 ftp > drwxr-xr-x. 2 root root6 Jan 20 2022 games > drwxr-xr-x. 3 root root 18 Apr 5 2022 kerberos > drwxr-xr-x. 24 root root 4096 May 12 2022 lib > drwxr-xr-x. 2 root root6 Jan 20 2022 local > lrwxrwxrwx. 1 root root 11 May 12 2022 lock -> ../run/lock > drwxr-xr-x. 8 root root 105 May 12 2022 log > lrwxrwxrwx. 1 root root 10 Jan 20 2022 mail -> spool/mail > drwxr-xr-x. 2 root root6 Jan 20 2022 nis > drwxr-xr-x. 2 root root6 Jan 20 2022 opt > drwxr-xr-x. 2 root root6 Jan 20 2022 preserve > lrwxrwxrwx. 1 root root6 May 12 2022 run -> ../run > drwxr-xr-x. 5 root root 45 May 12 2022 spool > drwxrwxrwt. 2 bin bin 6 May 12 2022 tmp > drwxr-xr-x. 2 root root6 Jan 20 2022 yp > > Fixes: https://github.com/rwmjones/guestfs-tools/issues/12 > --- > common | 2 +- > customize/customize_run.ml | 11 +++ > 2 files changed, 12 insertions(+), 1 deletion(-) > > diff --git a/common b/common > index 420892e660..bbb54714ce 16 > --- a/common > +++ b/common > @@ -1 +1 @@ > -Subproject commit 420892e660726c7184c000b9b86b11f491a5a126 > +Subproject commit bbb54714ce24c76e5761d96a0227a753896dc4c4 > diff --git a/customize/customize_run.ml b/customize/customize_run.ml > index f03774e003..409b46edc6 100644 > --- a/customize/customize_run.ml > +++ b/customize/customize_run.ml > @@ -153,6 +153,17 @@ let run (g : G.guestfs) root (ops : ops) = >let mode = if String.is_prefix mode "0" then "0o" ^ mode else mode in >g#chmod (int_of_string mode) path > > +| `Chown (uid_gid, path) -> > + let uid, gid = String.split "." uid_gid in > + let uid, gid = > + try int_of_string uid, int_of_string gid > + with Failure _ -> > + error (f_"--chown: could not parse numeric UID.GID from \ > + %s") uid_gid in > + > + message (f_"Changing owner of %s to %d.%d") path uid gid; > + g#chown uid gid path > + > | `Command cmd -> >message (f_"Running: %s") cmd; >do_run ~display:cmd cmd I've not looked at the context beyond this patch; the patch does look good. Acked-by: Laszlo Ersek ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [v2v PATCH v2 2/3] lib/utils: make "chown_for_libvirt_rhbz_1045069" fail hard
On 6/29/23 14:54, Richard W.M. Jones wrote: > On Thu, Jun 29, 2023 at 02:34:42PM +0200, Laszlo Ersek wrote: >> Currently "chown_for_libvirt_rhbz_1045069" is best effort; if it fails, we >> suppress the exception (we log it in verbose mode only, even). >> >> That's not proved helpful: it almost certainly leads to later errors, but >> those errors are less clear than the original (suppressed) exception. >> Namely, the user sees something like >> >>> Failed to connect to '/tmp/v2v.sKlulY/in0': Permission denied >> >> rather than >> >>> Failed to connect socket to '/var/run/libvirt/virtqemud-sock-ro': >>> Connection refused >> >> So just allow the exception to propagate outwards. >> >> And then, now that "chown_for_libvirt_rhbz_1045069" will be able to fail, >> hoist the call to "On_exit.rm_rf" before the call to >> "chown_for_libvirt_rhbz_1045069", after creating the v2v temporary >> directory. In the current order, if "chown_for_libvirt_rhbz_1045069" threw >> an exception, then we'd leak the temp dir in the filesystem. >> >> Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2182024 >> Signed-off-by: Laszlo Ersek >> --- >> >> Notes: >> v2: >> >> - new patch >> >> lib/utils.mli | 5 +--- >> lib/utils.ml | 26 >> 2 files changed, 11 insertions(+), 20 deletions(-) >> >> diff --git a/lib/utils.mli b/lib/utils.mli >> index cf88a467fd54..391a2a351ec7 100644 >> --- a/lib/utils.mli >> +++ b/lib/utils.mli >> @@ -67,10 +67,7 @@ val chown_for_libvirt_rhbz_1045069 : string -> unit >> to root-owned files and directories. To fix this, provide >> a function to chown things we might need to qemu:root so >> qemu can access them. Note that root normally ignores >> -permissions so can still access the resource. >> - >> -This is best-effort. If something fails then we carry >> -on and hope for the best. *) >> +permissions so can still access the resource. *) >> >> val error_if_no_ssh_agent : unit -> unit >> >> diff --git a/lib/utils.ml b/lib/utils.ml >> index 174c01b1e92f..7d69c9e0d177 100644 >> --- a/lib/utils.ml >> +++ b/lib/utils.ml >> @@ -149,21 +149,15 @@ let backend_is_libvirt () = >> >> let rec chown_for_libvirt_rhbz_1045069 file = >>let running_as_root = Unix.geteuid () = 0 in >> - if running_as_root && backend_is_libvirt () then ( >> -try >> - let user = Option.value ~default:"qemu" (libvirt_qemu_user ()) in >> - let uid = >> -if String.is_prefix user "+" then >> - int_of_string (String.sub user 1 (String.length user - 1)) >> -else >> - (Unix.getpwnam user).pw_uid in >> - debug "setting owner of %s to %d:root" file uid; >> - Unix.chown file uid 0 >> -with >> -| exn -> (* Print exception, but continue. *) >> - debug "could not set owner of %s: %s" >> - file (Printexc.to_string exn) >> - ) >> + if running_as_root && backend_is_libvirt () then >> +let user = Option.value ~default:"qemu" (libvirt_qemu_user ()) in >> +let uid = >> + if String.is_prefix user "+" then >> +int_of_string (String.sub user 1 (String.length user - 1)) >> + else >> +(Unix.getpwnam user).pw_uid in >> +debug "setting owner of %s to %d:root" file uid; >> +Unix.chown file uid 0 > > Hmm, doesn't this need parens around the 'then' clause? Ding ding ding, it does not :) I'm happy you asked. (I was so frustrated to discover this *once again* that I almost sent a separate tirade about it to the list! So I guess now I'll use the opportunity...) The short answer is that, if "Unix.chown" were out of the scope of the "then", then it would also be out of the scope of "uid", and so it wouldn't compile. But the long answer is more interesting: # open Printf;; # if false then printf "1\n"; printf "2\n";; 2 - : unit = () And then compare: # if false then let () = () in printf "1\n"; printf "2\n";; - : unit = () When we have "then" and a semiclon ";" but *no* "let/in", then the "then" binds more strongly than the semicolon. When we have a "then", a "let/in", and a semicolon ";", then the semicolon binds more strongly than the "in", *and* t
[Libguestfs] [v2v PATCH v2 3/3] docs/virt-v2v: document libvirt system instance startup
It has frequently tripped us up that on RHEL / Fedora, installing the right set of libvirt RPMs (such as the one pulled in by "libvirt-daemon-kvm") does not result in an immediately running libvirt system instance. Document the need, and the simplest method, for starting libvirt up manually. Thanks: Daniel Berrangé Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2182024 Signed-off-by: Laszlo Ersek --- Notes: v2: - rewrite using the language suggested by Rich context:-U12 docs/virt-v2v.pod | 23 +++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/docs/virt-v2v.pod b/docs/virt-v2v.pod index 4d2f241ad723..ac15108ae332 100644 --- a/docs/virt-v2v.pod +++ b/docs/virt-v2v.pod @@ -250,24 +250,26 @@ metadata. virt-v2v tries to guess the best default metadata. This is usually adequate but you can get finer control (eg. of memory and vCPUs) by using I<-i libvirtxml> instead. Only guests that use a single disk can be imported this way. =item B<-i> B Set the input method to I. This is the default. In this mode you have to specify a libvirt guest name or UUID on the command line. You may also specify a libvirt connection URI (see I<-ic>). +See L below. + =item B<-i> B Set the input method to I. In this mode you have to pass a libvirt XML file on the command line. This file is read in order to get metadata about the source guest (such as its name, amount of memory), and also to locate the input disks. See L below. =item B<-i> B This is the same as I<-i disk>. @@ -461,25 +463,26 @@ and guest metadata is created in the associated YAML file: /dir/name.yaml where C is the guest name. =item B<-o> B Set the output method to I. This is the default. In this mode, the converted guest is created as a libvirt guest. You may also specify a libvirt connection URI (see I<-oc>). -See L. +See L below, and +L. =item B<-o> B Set the output method to I. In this mode, the converted guest is written to a local directory specified by I<-os /dir> (the directory must exist). The converted guest’s disks are written as: /dir/name-sda /dir/name-sdb [etc] @@ -1373,24 +1376,26 @@ manually change ownership after virt-v2v has finished. =item Writing to libvirt When using I<-o libvirt>, you may need to run virt-v2v as root so that it can write to the libvirt system instance (ie. C) and to the default location for disk images (usually F). You can avoid this by setting up libvirt connection authentication, see L<http://libvirt.org/auth.html>. Alternatively, use I<-oc qemu:///session>, which will write to your per-user libvirt instance. +See also L. + =item Writing to Openstack Because of how Cinder volumes are presented as F block devices, using I<-o openstack> normally requires that virt-v2v is run as root. =item Writing to Glance This does I need root (in fact it probably won’t work), but may require either a special user and/or for you to source a script that sets authentication environment variables. Consult the Glance documentation. @@ -1521,24 +1526,40 @@ displayed to the user. The calling program should treat messages sent to stderr as error messages. In addition, virt-v2v exits with a non-zero status code if there was a fatal error. =back Virt-v2v E 0.9.1 did not support the I<--machine-readable> option at all. The option was added when virt-v2v was rewritten in 2014. It is possible to specify a format string for controlling the output; see L. +=head2 Starting the libvirt system instance + + Failed to connect socket to '/var/run/libvirt/virtqemud-sock': No such file or directory + Failed to connect socket to '/var/run/libvirt/virtqemud-sock-ro': Connection refused + +If you have just installed libvirt and virt-v2v, then you may see the +errors above. This is caused by libvirt daemons that provide various +services not running straight after installation. (This may depend on +your distribution and vendor presets). + +To fix this on systemd-based distributions, do: + + systemctl isolate multi-user.target + +See also L<https://bugzilla.redhat.com/2182024>. + =head1 FILES =over 4 =item F (Optional) If this directory is present, then virtio drivers for Windows guests will be found from this directory and installed in the guest during conversion. ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
[Libguestfs] [v2v PATCH v2 2/3] lib/utils: make "chown_for_libvirt_rhbz_1045069" fail hard
Currently "chown_for_libvirt_rhbz_1045069" is best effort; if it fails, we suppress the exception (we log it in verbose mode only, even). That's not proved helpful: it almost certainly leads to later errors, but those errors are less clear than the original (suppressed) exception. Namely, the user sees something like > Failed to connect to '/tmp/v2v.sKlulY/in0': Permission denied rather than > Failed to connect socket to '/var/run/libvirt/virtqemud-sock-ro': > Connection refused So just allow the exception to propagate outwards. And then, now that "chown_for_libvirt_rhbz_1045069" will be able to fail, hoist the call to "On_exit.rm_rf" before the call to "chown_for_libvirt_rhbz_1045069", after creating the v2v temporary directory. In the current order, if "chown_for_libvirt_rhbz_1045069" threw an exception, then we'd leak the temp dir in the filesystem. Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2182024 Signed-off-by: Laszlo Ersek --- Notes: v2: - new patch lib/utils.mli | 5 +--- lib/utils.ml | 26 2 files changed, 11 insertions(+), 20 deletions(-) diff --git a/lib/utils.mli b/lib/utils.mli index cf88a467fd54..391a2a351ec7 100644 --- a/lib/utils.mli +++ b/lib/utils.mli @@ -67,10 +67,7 @@ val chown_for_libvirt_rhbz_1045069 : string -> unit to root-owned files and directories. To fix this, provide a function to chown things we might need to qemu:root so qemu can access them. Note that root normally ignores -permissions so can still access the resource. - -This is best-effort. If something fails then we carry -on and hope for the best. *) +permissions so can still access the resource. *) val error_if_no_ssh_agent : unit -> unit diff --git a/lib/utils.ml b/lib/utils.ml index 174c01b1e92f..7d69c9e0d177 100644 --- a/lib/utils.ml +++ b/lib/utils.ml @@ -149,21 +149,15 @@ let backend_is_libvirt () = let rec chown_for_libvirt_rhbz_1045069 file = let running_as_root = Unix.geteuid () = 0 in - if running_as_root && backend_is_libvirt () then ( -try - let user = Option.value ~default:"qemu" (libvirt_qemu_user ()) in - let uid = -if String.is_prefix user "+" then - int_of_string (String.sub user 1 (String.length user - 1)) -else - (Unix.getpwnam user).pw_uid in - debug "setting owner of %s to %d:root" file uid; - Unix.chown file uid 0 -with -| exn -> (* Print exception, but continue. *) - debug "could not set owner of %s: %s" - file (Printexc.to_string exn) - ) + if running_as_root && backend_is_libvirt () then +let user = Option.value ~default:"qemu" (libvirt_qemu_user ()) in +let uid = + if String.is_prefix user "+" then +int_of_string (String.sub user 1 (String.length user - 1)) + else +(Unix.getpwnam user).pw_uid in +debug "setting owner of %s to %d:root" file uid; +Unix.chown file uid 0 (* Get the local user that libvirt uses to run qemu when we are * running as root. This is returned as an optional string @@ -205,8 +199,8 @@ let error_if_no_ssh_agent () = (* Create the directory containing inX and outX sockets. *) let create_v2v_directory () = let d = Mkdtemp.temp_dir "v2v." in - chown_for_libvirt_rhbz_1045069 d; On_exit.rm_rf d; + chown_for_libvirt_rhbz_1045069 d; d (* Wait for a file to appear until a timeout. *) ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
[Libguestfs] [v2v PATCH v2 1/3] lib/utils: fix typo
Fix a small comment typo from commit 4e7f20684373 ("lib: Improve security
of in/out sockets when running virt-v2v as root", 2022-03-23).
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2182024
Signed-off-by: Laszlo Ersek
---
Notes:
v2:
- new patch
lib/utils.mli | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/utils.mli b/lib/utils.mli
index 5687bf75867e..cf88a467fd54 100644
--- a/lib/utils.mli
+++ b/lib/utils.mli
@@ -62,7 +62,7 @@ val backend_is_libvirt : unit -> bool
(** Return true iff the current backend is libvirt. *)
val chown_for_libvirt_rhbz_1045069 : string -> unit
-(** If running and root, and if the backend is libvirt, libvirt
+(** If running as root, and if the backend is libvirt, libvirt
will run qemu as a non-root user. This prevents access
to root-owned files and directories. To fix this, provide
a function to chown things we might need to qemu:root so
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
[Libguestfs] [v2v PATCH v2 0/3] improve UX when running as root and we can't chown
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2182024 v1 was here: <https://listman.redhat.com/archives/libguestfs/2023-June/031910.html>. Make any "chown_for_libvirt_rhbz_1045069" failure a hard one; that way the user gets to see more direct and more uniform error messages (namely that we couldn't connect to libvirt). Document those connection problems, and a simple way to start up all libvirt daemons. Laszlo Ersek (3): lib/utils: fix typo lib/utils: make "chown_for_libvirt_rhbz_1045069" fail hard docs/virt-v2v: document libvirt system instance startup docs/virt-v2v.pod | 23 - lib/utils.ml | 26 lib/utils.mli | 7 ++ 3 files changed, 34 insertions(+), 22 deletions(-) base-commit: 4f47d6431cab97c09bd42279e29c378e6e65fc03 ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [v2v PATCH] docs/virt-v2v: document libvirt system instance startup
On 6/28/23 14:38, Richard W.M. Jones wrote: > On Wed, Jun 28, 2023 at 01:31:53PM +0200, Laszlo Ersek wrote: >> On 6/28/23 12:05, Richard W.M. Jones wrote: >>> On Tue, Jun 27, 2023 at 07:14:36PM +0200, Laszlo Ersek wrote: >>>> It has frequently tripped us up that on RHEL / Fedora, installing the >>>> right set of libvirt RPMs (such as the one pulled in by >>>> "libvirt-daemon-kvm") does not result in an immediately running libvirt >>>> system instance. Document the need, and the simplest method, for starting >>>> libvirt up manually. >>>> >>>> Thanks: Daniel Berrangé >>>> Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2182024 >>>> Signed-off-by: Laszlo Ersek >>>> --- >>>> >>>> Notes: >>>> context:-U12 >>>> >>>> docs/virt-v2v.pod | 20 +++- >>>> 1 file changed, 19 insertions(+), 1 deletion(-) >>>> >>>> diff --git a/docs/virt-v2v.pod b/docs/virt-v2v.pod >>>> index 4d2f241ad723..2bd0b4425d80 100644 >>>> --- a/docs/virt-v2v.pod >>>> +++ b/docs/virt-v2v.pod >>>> @@ -250,24 +250,26 @@ metadata. virt-v2v tries to guess the best default >>>> metadata. This is >>>> usually adequate but you can get finer control (eg. of memory and >>>> vCPUs) by using I<-i libvirtxml> instead. Only guests that use a single >>>> disk can be imported this way. >>>> >>>> =item B<-i> B >>>> >>>> Set the input method to I. This is the default. >>>> >>>> In this mode you have to specify a libvirt guest name or UUID on the >>>> command line. You may also specify a libvirt connection URI (see >>>> I<-ic>). >>>> >>>> +See L in addition. >>>> + >>> >>> I would just say "below" instead of "in addition", it seems a bit >>> more natural. >>> >>>> =item B<-i> B >>>> >>>> Set the input method to I. >>>> >>>> In this mode you have to pass a libvirt XML file on the command line. >>>> This file is read in order to get metadata about the source guest >>>> (such as its name, amount of memory), and also to locate the input >>>> disks. See L below. >>>> >>>> =item B<-i> B >>>> >>>> This is the same as I<-i disk>. >>>> @@ -461,25 +463,26 @@ and guest metadata is created in the associated YAML >>>> file: >>>> >>>> /dir/name.yaml >>>> >>>> where C is the guest name. >>>> >>>> =item B<-o> B >>>> >>>> Set the output method to I. This is the default. >>>> >>>> In this mode, the converted guest is created as a libvirt guest. You >>>> may also specify a libvirt connection URI (see I<-oc>). >>>> >>>> -See L. >>>> +See L and >>>> +L in addition. >>> >>> Same here. >>> >>>> =item B<-o> B >>>> >>>> Set the output method to I. >>>> >>>> In this mode, the converted guest is written to a local directory >>>> specified by I<-os /dir> (the directory must exist). The converted >>>> guest’s disks are written as: >>>> >>>> /dir/name-sda >>>> /dir/name-sdb >>>> [etc] >>>> @@ -1373,24 +1376,26 @@ manually change ownership after virt-v2v has >>>> finished. >>>> =item Writing to libvirt >>>> >>>> When using I<-o libvirt>, you may need to run virt-v2v as root so that >>>> it can write to the libvirt system instance (ie. C) >>>> and to the default location for disk images (usually >>>> F). >>>> >>>> You can avoid this by setting up libvirt connection authentication, >>>> see L<http://libvirt.org/auth.html>. Alternatively, use >>>> I<-oc qemu:///session>, which will write to your per-user libvirt >>>> instance. >>>> >>>> +See also L. >>>> + >>>> =item Writing to Openstack >>>> >>>> Because of how Cinder volumes are presented as F block devices, >>>> using I<-o openstack> normally requires that virt-v2v is ru
Re: [Libguestfs] [v2v PATCH] docs/virt-v2v: document libvirt system instance startup
On 6/28/23 12:05, Richard W.M. Jones wrote: > On Tue, Jun 27, 2023 at 07:14:36PM +0200, Laszlo Ersek wrote: >> It has frequently tripped us up that on RHEL / Fedora, installing the >> right set of libvirt RPMs (such as the one pulled in by >> "libvirt-daemon-kvm") does not result in an immediately running libvirt >> system instance. Document the need, and the simplest method, for starting >> libvirt up manually. >> >> Thanks: Daniel Berrangé >> Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2182024 >> Signed-off-by: Laszlo Ersek >> --- >> >> Notes: >> context:-U12 >> >> docs/virt-v2v.pod | 20 +++- >> 1 file changed, 19 insertions(+), 1 deletion(-) >> >> diff --git a/docs/virt-v2v.pod b/docs/virt-v2v.pod >> index 4d2f241ad723..2bd0b4425d80 100644 >> --- a/docs/virt-v2v.pod >> +++ b/docs/virt-v2v.pod >> @@ -250,24 +250,26 @@ metadata. virt-v2v tries to guess the best default >> metadata. This is >> usually adequate but you can get finer control (eg. of memory and >> vCPUs) by using I<-i libvirtxml> instead. Only guests that use a single >> disk can be imported this way. >> >> =item B<-i> B >> >> Set the input method to I. This is the default. >> >> In this mode you have to specify a libvirt guest name or UUID on the >> command line. You may also specify a libvirt connection URI (see >> I<-ic>). >> >> +See L in addition. >> + > > I would just say "below" instead of "in addition", it seems a bit > more natural. > >> =item B<-i> B >> >> Set the input method to I. >> >> In this mode you have to pass a libvirt XML file on the command line. >> This file is read in order to get metadata about the source guest >> (such as its name, amount of memory), and also to locate the input >> disks. See L below. >> >> =item B<-i> B >> >> This is the same as I<-i disk>. >> @@ -461,25 +463,26 @@ and guest metadata is created in the associated YAML >> file: >> >> /dir/name.yaml >> >> where C is the guest name. >> >> =item B<-o> B >> >> Set the output method to I. This is the default. >> >> In this mode, the converted guest is created as a libvirt guest. You >> may also specify a libvirt connection URI (see I<-oc>). >> >> -See L. >> +See L and >> +L in addition. > > Same here. > >> =item B<-o> B >> >> Set the output method to I. >> >> In this mode, the converted guest is written to a local directory >> specified by I<-os /dir> (the directory must exist). The converted >> guest’s disks are written as: >> >> /dir/name-sda >> /dir/name-sdb >> [etc] >> @@ -1373,24 +1376,26 @@ manually change ownership after virt-v2v has >> finished. >> =item Writing to libvirt >> >> When using I<-o libvirt>, you may need to run virt-v2v as root so that >> it can write to the libvirt system instance (ie. C) >> and to the default location for disk images (usually >> F). >> >> You can avoid this by setting up libvirt connection authentication, >> see L<http://libvirt.org/auth.html>. Alternatively, use >> I<-oc qemu:///session>, which will write to your per-user libvirt >> instance. >> >> +See also L. >> + >> =item Writing to Openstack >> >> Because of how Cinder volumes are presented as F block devices, >> using I<-o openstack> normally requires that virt-v2v is run as root. >> >> =item Writing to Glance >> >> This does I need root (in fact it probably won’t work), but may >> require either a special user and/or for you to source a script that >> sets authentication environment variables. Consult the Glance >> documentation. >> >> @@ -1521,24 +1526,37 @@ displayed to the user. >> The calling program should treat messages sent to stderr as error >> messages. In addition, virt-v2v exits with a non-zero status >> code if there was a fatal error. >> >> =back >> >> Virt-v2v E 0.9.1 did not support the I<--machine-readable> >> option at all. The option was added when virt-v2v was rewritten in 2014. >> >> It is possible to specify a format string for controlling the output; >> see L. >> >> +=head2 Starting the libvirt system instance >> + >>
Re: [Libguestfs] [PATCH libguestfs 4/4] ocaml/t/guestfs_065_implicit_close.ml: Skip this test on OCaml 5
On 6/27/23 14:33, Richard W.M. Jones wrote: > Link: > https://discuss.ocaml.org/t/ocaml-5-forcing-objects-to-be-collected-and-finalized/12492/2 > --- > ocaml/t/guestfs_065_implicit_close.ml | 8 > 1 file changed, 8 insertions(+) > > diff --git a/ocaml/t/guestfs_065_implicit_close.ml > b/ocaml/t/guestfs_065_implicit_close.ml > index f2dfecbd5c..04e511dd8a 100644 > --- a/ocaml/t/guestfs_065_implicit_close.ml > +++ b/ocaml/t/guestfs_065_implicit_close.ml > @@ -16,6 +16,14 @@ > * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 > USA. > *) > > +let () = > + (* In OCaml 5, Gc.full_major does not actually collect the handle > + * for unknown reasons. Skip the test until we can resolve this. > + * > https://discuss.ocaml.org/t/ocaml-5-forcing-objects-to-be-collected-and-finalized/12492/2 > + *) > + if Sys.ocaml_version >= "5" then > +exit 77 > + > let close_invoked = ref 0 > > let close _ _ _ _ = series looks great and even understandable, thanks! ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH libguestfs 1/4] ocaml: Replace old enter/leave_blocking_section calls
On 6/27/23 14:33, Richard W.M. Jones wrote: > Since OCaml 4 the old and confusing caml_enter_blocking_section and > caml_leave_blocking_section calls have been replaced with > caml_release_runtime_system and caml_acquire_runtime_system (in that > order). Use the new names. > --- > generator/OCaml.ml | 5 +++-- > ocaml/guestfs-c.c | 5 +++-- > 2 files changed, 6 insertions(+), 4 deletions(-) Highly welcome, the inverse meaning of the old names was terrible. :) Thanks! Laszlo > > diff --git a/generator/OCaml.ml b/generator/OCaml.ml > index 02d9ee2e91..07ccd26924 100644 > --- a/generator/OCaml.ml > +++ b/generator/OCaml.ml > @@ -429,6 +429,7 @@ and generate_ocaml_c () = > #include > #include > #include > +#include > > #include > #include \"guestfs-utils.h\" > @@ -689,12 +690,12 @@ copy_table (char * const * argv) >pr "\n"; > >if blocking then > -pr " caml_enter_blocking_section ();\n"; > +pr " caml_release_runtime_system ();\n"; >pr " r = %s " c_function; >generate_c_call_args ~handle:"g" style; >pr ";\n"; >if blocking then > -pr " caml_leave_blocking_section ();\n"; > +pr " caml_acquire_runtime_system ();\n"; > >(* Free strings if we copied them above. *) >List.iter ( > diff --git a/ocaml/guestfs-c.c b/ocaml/guestfs-c.c > index 3888c94564..8c8aa46096 100644 > --- a/ocaml/guestfs-c.c > +++ b/ocaml/guestfs-c.c > @@ -34,6 +34,7 @@ > #include > #include > #include > +#include > #include > > #include "guestfs-c.h" > @@ -395,12 +396,12 @@ event_callback_wrapper (guestfs_h *g, >/* Ensure we are holding the GC lock before any GC operations are > * possible. (RHBZ#725824) > */ > - caml_leave_blocking_section (); > + caml_acquire_runtime_system (); > >event_callback_wrapper_locked (g, data, event, event_handle, flags, > buf, buf_len, array, array_len); > > - caml_enter_blocking_section (); > + caml_release_runtime_system (); > } > > value ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
[Libguestfs] [v2v PATCH] docs/virt-v2v: document libvirt system instance startup
It has frequently tripped us up that on RHEL / Fedora, installing the right set of libvirt RPMs (such as the one pulled in by "libvirt-daemon-kvm") does not result in an immediately running libvirt system instance. Document the need, and the simplest method, for starting libvirt up manually. Thanks: Daniel Berrangé Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2182024 Signed-off-by: Laszlo Ersek --- Notes: context:-U12 docs/virt-v2v.pod | 20 +++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/docs/virt-v2v.pod b/docs/virt-v2v.pod index 4d2f241ad723..2bd0b4425d80 100644 --- a/docs/virt-v2v.pod +++ b/docs/virt-v2v.pod @@ -250,24 +250,26 @@ metadata. virt-v2v tries to guess the best default metadata. This is usually adequate but you can get finer control (eg. of memory and vCPUs) by using I<-i libvirtxml> instead. Only guests that use a single disk can be imported this way. =item B<-i> B Set the input method to I. This is the default. In this mode you have to specify a libvirt guest name or UUID on the command line. You may also specify a libvirt connection URI (see I<-ic>). +See L in addition. + =item B<-i> B Set the input method to I. In this mode you have to pass a libvirt XML file on the command line. This file is read in order to get metadata about the source guest (such as its name, amount of memory), and also to locate the input disks. See L below. =item B<-i> B This is the same as I<-i disk>. @@ -461,25 +463,26 @@ and guest metadata is created in the associated YAML file: /dir/name.yaml where C is the guest name. =item B<-o> B Set the output method to I. This is the default. In this mode, the converted guest is created as a libvirt guest. You may also specify a libvirt connection URI (see I<-oc>). -See L. +See L and +L in addition. =item B<-o> B Set the output method to I. In this mode, the converted guest is written to a local directory specified by I<-os /dir> (the directory must exist). The converted guest’s disks are written as: /dir/name-sda /dir/name-sdb [etc] @@ -1373,24 +1376,26 @@ manually change ownership after virt-v2v has finished. =item Writing to libvirt When using I<-o libvirt>, you may need to run virt-v2v as root so that it can write to the libvirt system instance (ie. C) and to the default location for disk images (usually F). You can avoid this by setting up libvirt connection authentication, see L<http://libvirt.org/auth.html>. Alternatively, use I<-oc qemu:///session>, which will write to your per-user libvirt instance. +See also L. + =item Writing to Openstack Because of how Cinder volumes are presented as F block devices, using I<-o openstack> normally requires that virt-v2v is run as root. =item Writing to Glance This does I need root (in fact it probably won’t work), but may require either a special user and/or for you to source a script that sets authentication environment variables. Consult the Glance documentation. @@ -1521,24 +1526,37 @@ displayed to the user. The calling program should treat messages sent to stderr as error messages. In addition, virt-v2v exits with a non-zero status code if there was a fatal error. =back Virt-v2v E 0.9.1 did not support the I<--machine-readable> option at all. The option was added when virt-v2v was rewritten in 2014. It is possible to specify a format string for controlling the output; see L. +=head2 Starting the libvirt system instance + +If you have just installed the libvirt distribution packages, then +dependent on your distribution and its vendor presets, the modular +libvirt daemons providing the various services of the libvirt system +instance may not be running yet. Therefore, if you intend to connect to +the libvirt system instance with virt-v2v (see S> / +I<-ic>, and/or S> / I<-oc>), first verify that the libvirt +services are running, before invoking virt-v2v. For example, on Fedora +and RHEL, you may have to start the related services individually with +C, or (recommended) start them all with S>. See L<https://bugzilla.redhat.com/2182024>. + =head1 FILES =over 4 =item F (Optional) If this directory is present, then virtio drivers for Windows guests will be found from this directory and installed in the guest during conversion. base-commit: 4f47d6431cab97c09bd42279e29c378e6e65fc03 ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH libnbd] ocaml: Release runtime lock around call to nbd_close
On 6/21/23 23:54, Richard W.M. Jones wrote:
> OCaml 5 is stricter than earlier versions about correct locking.
>
> We must release the OCaml runtime lock when calling nbd_close since it
> may do some long-running operations and we want to allow concurrent
> threads to run.
>
> However specifically if there are callbacks (eg. a debug callback)
> then we would end up trying to re-acquire the lock in the callback,
> resulting in a crash:
Drop the word "However"?
Paragraph #3 appears to support paragraph #2, not to oppose it.
Acked-by: Laszlo Ersek
Laszlo
>
> (gdb) bt
> #0 __pthread_kill_implementation (threadid=,
> signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> #1 0x7f47ffc8f773 in __pthread_kill_internal (signo=6,
> threadid=) at pthread_kill.c:78
> #2 0x7f47ffc3e71e in __GI_raise (sig=sig@entry=6)
> at ../sysdeps/posix/raise.c:26
> #3 0x7f47ffc2687f in __GI_abort () at abort.c:79
> #4 0x560c9eb62779 in caml_fatal_error ()
> #5 0x560c9eb63238 in caml_plat_fatal_error ()
> #6 0x560c9eb4ea77 in caml_acquire_domain_lock ()
> #7 0x560c9eb65cdc in caml_leave_blocking_section ()
> #8 0x560c9eaf8a87 in debug_wrapper (user_data=0x560ca0af2670,
> context=0x7f47fff8ca60 "nbd_close", msg=0x560ca0af28b0 "closing handle")
> at ../nbd-c.c:187
> #9 0x7f47fff7072f in nbd_internal_debug (h=h@entry=0x560ca0b57db0,
> context=0x7f47fff8ca60 "nbd_close", context@entry=0x0,
> fs=fs@entry=0x7f47fff8ca6a "closing handle")
> at /home/rjones/d/libnbd/lib/debug.c:90
> #10 0x7f47fff73f23 in nbd_close (h=0x560ca0b57db0)
> at /home/rjones/d/libnbd/lib/handle.c:127
> #11 0x560c9eae8dbe in nbd_internal_ocaml_handle_finalize (
> hv=) at ../handle.c:39
> #12 nbd_internal_ocaml_nbd_close (hv=) at ../handle.c:62
> #13
> #14 0x560c9eae84dc in camlTest_140_explicit_close__entry () at
> NBD.ml:148
> #15 0x560c9eae5c5b in caml_program ()
> #16
> #17 0x560c9eb6cb77 in caml_startup_common ()
> #18 0x560c9eb6cbef in caml_main ()
> #19 0x560c9eae5910 in main ()
> ---
> ocaml/handle.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/ocaml/handle.c b/ocaml/handle.c
> index fb45b30102..b3e5a0fc33 100644
> --- a/ocaml/handle.c
> +++ b/ocaml/handle.c
> @@ -26,6 +26,7 @@
> #include
> #include
> #include
> +#include
>
> #include
>
> @@ -36,7 +37,9 @@ nbd_internal_ocaml_handle_finalize (value hv)
> {
>struct nbd_handle *h = NBD_val (hv);
>
> + caml_enter_blocking_section ();
>nbd_close (h);
> + caml_leave_blocking_section ();
> }
>
> value
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [v2v PATCH] test-data/phony-guests: fix prerequisite list of "fedora-luks-on-lvm.img"
On 6/20/23 11:14, Laszlo Ersek wrote:
> On 6/20/23 09:41, Richard W.M. Jones wrote:
>> On Mon, Jun 19, 2023 at 06:27:29PM +0200, Laszlo Ersek wrote:
>>> In the virt-v2v repo, commit 1e75569aa074 ("test-data/phony-guests: Allow
>>> virt-v2v to work against phony Fedora") is an ancestor of commit
>>> e4efe4b7d240 ("tests: add LUKS-on-LVM test"). The latter created a state
>>> where "fedora-static-bin" and LUKS on LVM testing would coexist (i.e.,
>>> where "fedora-static-bin" would be uploaded to the LUKS-on-LVM disk image
>>> as well), but the commit didn't spell out the dependency in
>>> "test-data/phony-guests/Makefile.am".
>>>
>>> Do that now.
>>>
>>> The problem can be triggered with:
>>>
>>>> autoreconf -i
>>>> ./configure
>>>> make
>>>> make -C test-data/phony-guests fedora-luks-on-lvm.img
>>>
>>> where the last command fails with
>>>
>>>> make: Entering directory '.../test-data/phony-guests'
>>>> SRCDIR=. LAYOUT=luks-on-lvm ../../run --test ./make-fedora-img.pl
>>>> open: fedora-static-bin: No such file or directory at
>>>> .../test-data/phony-guests/make-fedora-img.pl line 373.
>>>
>>> (In the guestfs-tools repo, the relative order (the descendancy) between
>>> both commits is the opposite. There, commit 27da4b0c4991 ("inspector: add
>>> LUKS-on-LVM test") came first, and commit eb0ff1859eb6
>>> ("test-data/phony-guests: Allow virt-v2v to work against phony Fedora"),
>>> came second. The latter commit, in fact being a port of virt-v2v commit
>>> 1e75569aa074, brought together "fedora-static-bin" with "LUKS on LVM"
>>> testing, and it correctly added "fedora-static-bin" as a pre-requisite
>>> for building "fedora-luks-on-lvm.img".)
>>>
>>> Fixes: e4efe4b7d240b66b1d53fbe5a127f4f5966f6903
>>> Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2168506
>>> Signed-off-by: Laszlo Ersek
>>> ---
>>> test-data/phony-guests/Makefile.am | 3 ++-
>>> 1 file changed, 2 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/test-data/phony-guests/Makefile.am
>>> b/test-data/phony-guests/Makefile.am
>>> index 29dbd4d0f9f2..10c0241b7289 100644
>>> --- a/test-data/phony-guests/Makefile.am
>>> +++ b/test-data/phony-guests/Makefile.am
>>> @@ -103,7 +103,8 @@ fedora-btrfs.img: make-fedora-img.pl \
>>> # Make a (dummy) Fedora image with LUKS-on-LVM.
>>> fedora-luks-on-lvm.img: make-fedora-img.pl \
>>> fedora-journal.tar.xz \
>>> - fedora.db
>>> + fedora.db \
>>> + fedora-static-bin
>>> SRCDIR=$(srcdir) LAYOUT=luks-on-lvm $(top_builddir)/run --test ./$<
>>>
>>> # Make a (dummy) Fedora image with LVM-on-LUKS.
>>
>> Reviewed-by: Richard W.M. Jones
>>
>> Do we need this commit copied into the other projects as well?
>
> No; guestfs-tools is not affected (the dependency is correctly listed
> there -- see the parenthesized paragraph in the commit message).
>
> Libguestfs is not affected either, but that's for a different reason: in
> libguestfs, the perl script does not inject "fedora-static-bin" into the
> disk image (fedora-static-bin is not used in libguestfs at all, AFAICT).
>
> I'll push this later.
Commit 13a6f4b9686e.
Laszlo
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [libnbd PATCH v4 4/4] internal: Refactor layout of replies in sbuf
assert (cmd); /* guaranteed by CHECK */
> assert (cmd->type == NBD_CMD_BLOCK_STATUS);
> @@ -479,8 +479,8 @@ REPLY.CHUNK_REPLY.RESYNC:
>SET_NEXT_STATE (%^FINISH_COMMAND);
>return 0;
> }
> -type = be16toh (h->sbuf.sr.structured_reply.type);
> -length = be32toh (h->sbuf.sr.structured_reply.length);
> +type = be16toh (h->sbuf.reply.hdr.structured.type);
> +length = be32toh (h->sbuf.reply.hdr.structured.length);
> debug (h, "unexpected reply type %u or payload length %" PRIu32
> " for cookie %" PRIu64 " and command %" PRIu32
> ", this is probably a server bug",
> @@ -494,7 +494,7 @@ REPLY.CHUNK_REPLY.RESYNC:
> REPLY.CHUNK_REPLY.FINISH:
>uint16_t flags;
>
> - flags = be16toh (h->sbuf.sr.structured_reply.flags);
> + flags = be16toh (h->sbuf.reply.hdr.structured.flags);
>if (flags & NBD_REPLY_FLAG_DONE) {
> SET_NEXT_STATE (%^FINISH_COMMAND);
>}
> diff --git a/generator/states-reply-simple.c b/generator/states-reply-simple.c
> index 19be5418..898bb84e 100644
> --- a/generator/states-reply-simple.c
> +++ b/generator/states-reply-simple.c
> @@ -23,7 +23,7 @@ REPLY.SIMPLE_REPLY.START:
>struct command *cmd = h->reply_cmd;
>uint32_t error;
>
> - error = be32toh (h->sbuf.simple_reply.error);
> + error = be32toh (h->sbuf.reply.hdr.simple.error);
>
>if (cmd == NULL) {
> /* Unexpected reply. If error was set or we have structured
> @@ -39,7 +39,7 @@ REPLY.SIMPLE_REPLY.START:
> if (error || h->structured_replies)
>SET_NEXT_STATE (%^FINISH_COMMAND);
> else {
> - uint64_t cookie = be64toh (h->sbuf.simple_reply.cookie);
> + uint64_t cookie = be64toh (h->sbuf.reply.hdr.simple.cookie);
>SET_NEXT_STATE (%.DEAD);
>set_error (EPROTO,
> "no matching cookie %" PRIu64 " found for server reply, "
Acked-by: Laszlo Ersek
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [libnbd PATCH v4 3/4] generator: Rename states-reply-structured to states-reply-chunk
On 6/12/23 22:34, Eric Blake wrote:
> On Thu, Jun 08, 2023 at 09:17:37PM -0500, Eric Blake wrote:
>> Upcoming patches to add extended headers want to share the common
>> payload parser with structured replies. Renaming the file and the
>> associated states from "structured" to "chunk" makes it more obvious
>> that we will be sharing the code independent from the header style
>> parsed in the earlier REPLY portion of the state machine.
>>
>> Signed-off-by: Eric Blake
>> ---
>> generator/state_machine.ml| 26 +--
>> generator/states-reply.c | 2 +-
>> ...eply-structured.c => states-reply-chunk.c} | 22
>> generator/Makefile.am | 2 +-
>> 4 files changed, 26 insertions(+), 26 deletions(-)
>> rename generator/{states-reply-structured.c => states-reply-chunk.c} (97%)
>
> I'm thinking of squashing in:
>
> diff --git c/lib/internal.h w/lib/internal.h
> index 1e8d02f1..4b0043b3 100644
> --- c/lib/internal.h
> +++ w/lib/internal.h
> @@ -252,10 +252,10 @@ struct nbd_handle {
>} hdr;
>union {
> uint64_t align_; /* Start reply.payload on an 8-byte alignment */
> -struct nbd_structured_reply_offset_data offset_data;
> -struct nbd_structured_reply_offset_hole offset_hole;
> +struct nbd_chunk_offset_data offset_data;
> +struct nbd_chunk_offset_hole offset_hole;
> struct {
> - struct nbd_structured_reply_error error;
> + struct nbd_chunk_error error;
>char msg[NBD_MAX_STRING]; /* Common to all error types */
>uint64_t offset; /* Only used for NBD_REPLY_TYPE_ERROR_OFFSET */
> } NBD_ATTRIBUTE_PACKED error;
> diff --git c/lib/nbd-protocol.h w/lib/nbd-protocol.h
> index 50275dcd..fef19f2c 100644
> --- c/lib/nbd-protocol.h
> +++ w/lib/nbd-protocol.h
> @@ -211,20 +211,20 @@ struct nbd_structured_reply {
>uint16_t flags; /* NBD_REPLY_FLAG_* */
>uint16_t type;/* NBD_REPLY_TYPE_* */
>uint64_t cookie; /* Opaque handle. */
> - uint32_t length; /* Length of payload which follows. */
> + uint32_t length; /* Length of following nbd_chunk_* payload.
> */
> } NBD_ATTRIBUTE_PACKED;
>
> -struct nbd_structured_reply_offset_data {
> +struct nbd_chunk_offset_data {
>uint64_t offset; /* offset */
>/* Followed by data. */
> } NBD_ATTRIBUTE_PACKED;
>
> -struct nbd_structured_reply_offset_hole {
> +struct nbd_chunk_offset_hole {
>uint64_t offset;
>uint32_t length; /* Length of hole. */
> } NBD_ATTRIBUTE_PACKED;
>
> -struct nbd_structured_reply_error {
> +struct nbd_chunk_error {
> uint32_t error; /* NBD_E* error number */
>uint16_t len; /* Length of human readable error. */
>/* Followed by human readable error string, and possibly more structure. */
>
For both (squashed):
Acked-by: Laszlo Ersek
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH v3] ldmtool: fix NULL pointer dereference
On 6/20/23 10:08, Richard W.M. Jones wrote:
> On Tue, Jun 20, 2023 at 05:00:24PM +0900, Vincent Mailhol wrote:
>> If /sys/block can not be opened, get_devices() returns NULL.
>>
>> cmdline() does not check this result and below code snippet:
>>
>> scanned = get_devices();
>> devices = (gchar **) scanned->data;
>>
>> results in a segmentation fault.
>>
>> Add a check on scanned.
>>
>> Relevant logs:
>>
>> Unable to open /sys/block: No such file or directory
>> [0.777352] ldmtool[164]: segfault at 0 ip 563a225cd6a5 sp
>> 7ffe54965a60 error 4 in ldmtool[563a225cb000+3000]
>> [0.778278] Code: 18 64 48 33 1c 25 28 00 00 00 75 5e 48 83 c4 28 5b 5d
>> 41 5c 41 5d 41 5e 41 5f c3 66 2e 0f 1f 84 00 00 00 00 00 e8 db fd ff ff <4c>
>> 8b 20 48 89 44 24 08 4c 89 e7 e8 0b e1 ff ff 45 31 c0 4c 89 e1
>>
>> Fixes: 25d9635e4ee5 ("Add ldmtool")
>> Signed-off-by: Vincent Mailhol
>> ---
>>
>> * Changelog *
>>
>> v2 -> v3
>>
>> * Fix the From: tag (incorrect e-mail address, sorry for the noise).
>>
>> v1 -> v2
>>
>> * Directly return FALSE instead of goto error. Jumping to the error
>> label bypasses jb's declaration thus resulting in an undefined
>> behavior.
>>
>> ---
>> src/ldmtool.c | 2 ++
>> 1 file changed, 2 insertions(+)
>>
>> diff --git a/src/ldmtool.c b/src/ldmtool.c
>> index 6957c1a..dbe2c8c 100644
>> --- a/src/ldmtool.c
>> +++ b/src/ldmtool.c
>> @@ -746,6 +746,8 @@ cmdline(LDM * const ldm, gchar **devices,
>> GArray * scanned = NULL;
>> if (!devices) {
>> scanned = get_devices();
>> +if (!scanned)
>> + return FALSE;
>> devices = (gchar **) scanned->data;
>> }
>
> Seems fine, based on Laszlo's analysis of the first version, thus:
>
> Acked-by: Richard W.M. Jones
>
> I believe I will be able to push this patch (or if not, I'll ask Matt
> to do it later). Is this version OK Laszlo?
Reviewed-by: Laszlo Ersek
Thanks!
Laszlo
>
> Rich.
>
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [v2v PATCH] test-data/phony-guests: fix prerequisite list of "fedora-luks-on-lvm.img"
On 6/20/23 09:41, Richard W.M. Jones wrote:
> On Mon, Jun 19, 2023 at 06:27:29PM +0200, Laszlo Ersek wrote:
>> In the virt-v2v repo, commit 1e75569aa074 ("test-data/phony-guests: Allow
>> virt-v2v to work against phony Fedora") is an ancestor of commit
>> e4efe4b7d240 ("tests: add LUKS-on-LVM test"). The latter created a state
>> where "fedora-static-bin" and LUKS on LVM testing would coexist (i.e.,
>> where "fedora-static-bin" would be uploaded to the LUKS-on-LVM disk image
>> as well), but the commit didn't spell out the dependency in
>> "test-data/phony-guests/Makefile.am".
>>
>> Do that now.
>>
>> The problem can be triggered with:
>>
>>> autoreconf -i
>>> ./configure
>>> make
>>> make -C test-data/phony-guests fedora-luks-on-lvm.img
>>
>> where the last command fails with
>>
>>> make: Entering directory '.../test-data/phony-guests'
>>> SRCDIR=. LAYOUT=luks-on-lvm ../../run --test ./make-fedora-img.pl
>>> open: fedora-static-bin: No such file or directory at
>>> .../test-data/phony-guests/make-fedora-img.pl line 373.
>>
>> (In the guestfs-tools repo, the relative order (the descendancy) between
>> both commits is the opposite. There, commit 27da4b0c4991 ("inspector: add
>> LUKS-on-LVM test") came first, and commit eb0ff1859eb6
>> ("test-data/phony-guests: Allow virt-v2v to work against phony Fedora"),
>> came second. The latter commit, in fact being a port of virt-v2v commit
>> 1e75569aa074, brought together "fedora-static-bin" with "LUKS on LVM"
>> testing, and it correctly added "fedora-static-bin" as a pre-requisite
>> for building "fedora-luks-on-lvm.img".)
>>
>> Fixes: e4efe4b7d240b66b1d53fbe5a127f4f5966f6903
>> Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2168506
>> Signed-off-by: Laszlo Ersek
>> ---
>> test-data/phony-guests/Makefile.am | 3 ++-
>> 1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/test-data/phony-guests/Makefile.am
>> b/test-data/phony-guests/Makefile.am
>> index 29dbd4d0f9f2..10c0241b7289 100644
>> --- a/test-data/phony-guests/Makefile.am
>> +++ b/test-data/phony-guests/Makefile.am
>> @@ -103,7 +103,8 @@ fedora-btrfs.img: make-fedora-img.pl \
>> # Make a (dummy) Fedora image with LUKS-on-LVM.
>> fedora-luks-on-lvm.img: make-fedora-img.pl \
>> fedora-journal.tar.xz \
>> -fedora.db
>> +fedora.db \
>> +fedora-static-bin
>> SRCDIR=$(srcdir) LAYOUT=luks-on-lvm $(top_builddir)/run --test ./$<
>>
>> # Make a (dummy) Fedora image with LVM-on-LUKS.
>
> Reviewed-by: Richard W.M. Jones
>
> Do we need this commit copied into the other projects as well?
No; guestfs-tools is not affected (the dependency is correctly listed
there -- see the parenthesized paragraph in the commit message).
Libguestfs is not affected either, but that's for a different reason: in
libguestfs, the perl script does not inject "fedora-static-bin" into the
disk image (fedora-static-bin is not used in libguestfs at all, AFAICT).
I'll push this later.
Thanks
Laszlo
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH v1] ldmtool: fix NULL pointer dereference
On 6/19/23 16:36, Vincent Mailhol wrote:
> If /sys/block can not be opened, get_devices() returns NULL.
>
> cmdline() does not check this result and below code snippet:
>
> scanned = get_devices();
> devices = (gchar **) scanned->data;
>
> results in a segmentation fault.
>
> Add a check on scanned.
>
> Relevant logs:
>
> Unable to open /sys/block: No such file or directory
> [0.777352] ldmtool[164]: segfault at 0 ip 563a225cd6a5 sp
> 7ffe54965a60 error 4 in ldmtool[563a225cb000+3000]
> [0.778278] Code: 18 64 48 33 1c 25 28 00 00 00 75 5e 48 83 c4 28 5b 5d
> 41 5c 41 5d 41 5e 41 5f c3 66 2e 0f 1f 84 00 00 00 00 00 e8 db fd ff ff <4c>
> 8b 20 48 89 44 24 08 4c 89 e7 e8 0b e1 ff ff 45 31 c0 4c 89 e1
>
> Fixes: 25d9635e4ee5 ("Add ldmtool")
> Signed-off-by: Vincent Mailhol
> ---
> This thread did not yet show-up in
> https://listman.redhat.com/archives/libguestfs/2023-June/subject.html
> not sure why.
>
> For this reason, I couln't add a link reference.
> ---
> src/ldmtool.c | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/src/ldmtool.c b/src/ldmtool.c
> index 6957c1a..87aaccc 100644
> --- a/src/ldmtool.c
> +++ b/src/ldmtool.c
> @@ -746,6 +746,8 @@ cmdline(LDM * const ldm, gchar **devices,
> GArray * scanned = NULL;
> if (!devices) {
> scanned = get_devices();
> +if (!scanned)
> +goto error;
> devices = (gchar **) scanned->data;
> }
>
(I'm reviewing this against commit 5014da5b9071, in repository
.)
This fix is almost right, but not entirely. Consider what we have under
the "error" label:
error:
if (scanned) g_array_unref(scanned);
if (jb) g_object_unref(jb);
return FALSE;
When we jump to "error" on the new branch, "scanned" is is NULL, so that
is safe. However, the
JsonBuilder *jb = NULL;
line has not been reached yet, and that's a problem, because
g_object_unref() will receive an indeterminate value (undefined
behavior). (In fact, just *evaluating* "jb" as the controlling
expression of the "if" that is supposed to protect g_object_unref() is
*already* undefined behavior.)
According to C99 6.8 "Statements and blocks" paragraph 3,
A /block/ allows a set of declarations and statements to be grouped
into one syntactic unit. The initializers of objects that have
automatic storage duration [...] are evaluated and the values are
stored in the objects (including storing an indeterminate value in
objects without an initializer) each time the declaration is reached
in the order of execution, as if it were a statement, and within
each declaration in the order that declarators appear.
Furthermore, C99 6.8.4.2 "The switch statement" paragraph 7 provides the
following example:
In the artificial program fragment
switch (expr)
{
int i = 4;
f(i);
case 0:
i = 17;
/* falls through into default code */
default:
printf("%d\n", i);
}
the object whose identifier is *i* exists with automatic storage
duration (within the block) but is never initialized, and thus if
the controlling expression has a nonzero value, the call to the
*printf* function will access an indeterminate value. Similarly, the
call to the function *f* cannot be reached.
Thus, you can do one of two things (at least):
- *In addition* to the current contents of the patch, move the
declaration of "jb", together with its NULL initializer, to the top of
the function (just below that of "scanned"). This makes sure that the
new jump to "error" will still "pass" a well-defined "jb" value (NULL),
which the logic at the "error" label handles well.
- Alternatively: replace "goto error" with just "return FALSE" in your
patch. At that point, there is nothing to release yet.
Laszlo
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] libldm crashes in a linux-sandbox context
On 6/19/23 16:32, Vincent Mailhol wrote: > On Mon 19 June 2023 at 21:16, Laszlo Ersek wrote: >> On 6/19/23 13:18, Vincent MAILHOL wrote: >>> On Fri. 16 juin 2023 at 16:34, Richard W.M. Jones wrote: >>> (...) >>>>> Last thing, the segfault on ldmtool [1] still seems a valid issue. >>>>> Even if I now do have a workaround for my problem, that segfault might >>>>> be worth a bit more investigation. >>>> >>>> Yes that does look like a real problem. Does it crash if you just run >>>> ldmtool as a normal command, nothing to do with libguestfs? Might be >>>> a good idea to try to get a stack trace of the crash. >>> >>> The fact is that it only crashes with the UUID 65534 in the qemu VM. I >>> am not sure what command line is passed to ldmtool for this crash to >>> occur. >>> >>> I can help to gather information, but my biggest issue is that I do >>> not know how to interact with the VM under /tmp/.guestfs-1001/ >>> >>> [0.777352] ldmtool[164]: segfault at 0 ip 563a225cd6a5 sp >>> 7ffe54965a60 error 4 in ldmtool[563a225cb000+3000] >>> ^^^ >>> This smells like a NULL pointer dereference. >> >> ... Hey this is actually my line from an email I started writing earlier >> today :) , but I then decided not to send it. >> >> It certainly looks like a null pointer dereference, and if you >> disassemble the instruction byte stream dump (the "Code:" line from the >> kernel log) with (e.g.) ndisasm, that confirms it. You get something like >> >> 0025 E8DBFDcall 0xfe05 >> 002A 4C8B20mov r12,[rax] < crash >> 002D 4889442408mov [rsp+0x8],rax >> 0032 4C89E7mov rdi,r12 >> 0035 E80BE1call 0xe145 >> >> with the "mov r12,[rax]" instruction faulting (with the previously >> called function presumably having returned 0 in rax). See the "<4c> 8b >> 20" substring in the "Code:" line -- the angle brackets point at the >> first byte of the crashing instruction. >> >> I didn't send the email ultimately because your email included a link >> [1] pointing at a particular line number: >> >> https://github.com/mdbooth/libldm/blob/master/src/ldmtool.c#L164 >> >> and so I assumed you actually traced the crash to that line. >> >> Is that the case? >> >> Or did you perhaps mistake *PID* 164 (from the kernel log) for the line >> number? > > Yes, two messages back, I misinterpreted the PID (164) as a line > number. Because that particular line manipulate the result of a > g_array_index(), it looked coherent with the potential NULL pointer > dereference. Realizing my mistake, I then started to do a deeper > addr2line investigation in the previous message. Sorry. > >>> The instruction pointer >>> being 563a225cd6a5, I installed libguestfs-tools-dbgsym and tried a: >>> >>> addr2line -e /usr/bin/ldmtool 564a892506a5 > > > Reading my previous message, I do not know where this 564a892506a5 > comes from. I meant 563a225cd6a5 here (and below in gdb). > >>> Results: >>> >>> ??:0 >>> >>> Without conviction, I also tried in GDB: >>> >>> $ gdb /usr/bin/ldmtool >>> (...) >>> Reading symbols from /usr/bin/ldmtool... >>> Reading symbols from >>> /usr/lib/debug/.build-id/21/37b4a64903ebe427c242be08b8d496ba570583.debug... >>> (gdb) info line *0x564a892506a5 >>> No line number information available for address 0x564a892506a5 >>> >>> Debug symbols are correctly installed but impossible to convert that >>> instruction pointer into a line number. It is as if the ldmtool on my >>> host and the ldmtool in the qemu VM were from a different build. I >>> tried to mount /tmp/.guestfs-1001/appliance.d/root but that disk image >>> did not contain ldmtool. >>> >>> I am not sure how to generate a stack trace or a core dump within that >>> qemu VM. If you can tell me how to get an interactive prompt (or any >>> other guidance) I can try to collect more information. >> >> The IP where the crash occurs is 563a225cd6a5. The ldmtool binary >> (as opposed to a shared object / library) is mapped into the process's >> address space at 563a225cb000, for a length of 0x3000 by
