Re: [Libguestfs] Fwd: virt-v2v creating image that does not install guest agent on first boot

2023-09-28 Thread Richard W.M. Jones 
On Thu, Sep 28, 2023 at 07:41:55PM +0800, Ming Xie wrote:
> Hi Richard,
> 
> On Thu, Sep 28, 2023 at 6:14 PM Richard W.M. Jones  wrote:
> 
> Ming,
> 
> For some reason the build seems "stuck" at the moment, but
> if it finishes could you see if this fixes the RHEL 9 case?
> 
> https://kojihub.stream.centos.org/koji/taskinfo?taskID=2879547

It finished building now.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] [PATCH v7 01/12] nbd/server: Support a request payload

2023-09-28 Thread Vladimir Sementsov-Ogievskiy 

On 28.09.23 17:33, Eric Blake wrote:

On Thu, Sep 28, 2023 at 12:09:51PM +0300, Vladimir Sementsov-Ogievskiy wrote:

On 27.09.23 18:59, Eric Blake wrote:

We could also try to be a bit more complicated by peeking at the next
few bytes: if they look like a magic number of the next request,
assume the client set the bit accidentally but didn't send a payload
after all; for anything else, assume the client did pass a payload.
But adding in machinery to peek at a prefix is more complex than
either assuming a payload is always present (as done in this patch) or
assuming the bit was in error (and dropping the connection
unconditionally).  Preferences?



Ohh, you are right, thanks for comprehensive explanation. I really missed some things you 
are saying about. Yes, now I agree that "payload always exist when flag is set" 
is the best effort. Finally, that was our aim of the protocol design: make it more 
context independent. Probably, we may fix that in specification as preferable or at least 
possible server behavior about non-compliant client.


One other possibility I just thought of: have a heuristic where the
flag set with h->request_length less than 512 bytes is likely to
indicate an intentional payload (even if for a command where we
weren't expecting payload, so still a client error); while the flag
set wtih h->request_length >= 512 bytes is likely to be a mistaken
setting of the flag (but also still a client error).  NBD_CMD_WRITE is
probably the only command that will ever need to send a payload larger
than one sector, but that command already has handling to accept
payloads of all sizes because we know what to do with them and where
the client is not in error.



I'd prefer to avoid extra logic for optimizing handling of bad client, better 
keep code simpler.


--
Best regards,
Vladimir

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] [PATCH v7 01/12] nbd/server: Support a request payload

2023-09-28 Thread Eric Blake 
On Thu, Sep 28, 2023 at 12:09:51PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> On 27.09.23 18:59, Eric Blake wrote:
> > We could also try to be a bit more complicated by peeking at the next
> > few bytes: if they look like a magic number of the next request,
> > assume the client set the bit accidentally but didn't send a payload
> > after all; for anything else, assume the client did pass a payload.
> > But adding in machinery to peek at a prefix is more complex than
> > either assuming a payload is always present (as done in this patch) or
> > assuming the bit was in error (and dropping the connection
> > unconditionally).  Preferences?
> 
> 
> Ohh, you are right, thanks for comprehensive explanation. I really missed 
> some things you are saying about. Yes, now I agree that "payload always exist 
> when flag is set" is the best effort. Finally, that was our aim of the 
> protocol design: make it more context independent. Probably, we may fix that 
> in specification as preferable or at least possible server behavior about 
> non-compliant client.

One other possibility I just thought of: have a heuristic where the
flag set with h->request_length less than 512 bytes is likely to
indicate an intentional payload (even if for a command where we
weren't expecting payload, so still a client error); while the flag
set wtih h->request_length >= 512 bytes is likely to be a mistaken
setting of the flag (but also still a client error).  NBD_CMD_WRITE is
probably the only command that will ever need to send a payload larger
than one sector, but that command already has handling to accept
payloads of all sizes because we know what to do with them and where
the client is not in error.

> 
> r-b coming soon, I just need to take another look with corrected picture in 
> mind.
> 
> -- 
> Best regards,
> Vladimir
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] Fwd: virt-v2v creating image that does not install guest agent on first boot

2023-09-28 Thread Lee Garrett 

(resend, as it bounced on tyan-...@redhat.com)

Hi Ming,

On 27.09.23 14:46, Ming Xie wrote:

Hi Richard,

Sorry, I missed the email, I found out that the ova OS used by the customer is 
win11 uefi, so I tried to reproduce the error today using win11-uefi guest


Steps:
1. Convert win10-efi with UTC-8 timezone, win11-non-efi with UTC-8 timezone, 
win11-efi with UTC-8 timezone, win11-efi with UTC+8 timezone, win2022-efi with 
UTC-8 timezone from VMware by v2v, then check the installation status after 
finishing conversion



Summary the test result as below:

     qemu-ga

   Win10-efi-UTC-8  PASS
   Win11-non-efi UTC-8   PASS
   Win11-efi-UTC-8  FAIL(can't find qemu-ga log in c:\)
   Win11-efi-UTC+8 FAIL(can't find qemu-ga log in c:\)
   Win2022-efi-UTC-8  PASS

   Based on the above results, I think this problem is only caused by win11-efi 
and has nothing to do with the time zone "UTC-8"


hmm, where are you exactly setting the timezone? Within the guest, the scheduled 
task, or in the libvirt XML description?


The Windows 11 trial VM in question (downloaded from 
https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/ -> 
https://aka.ms/windev_VM_vmware) comes with the timezone within the VM set to 
US/Pacific.


If I also set the RTC clock for the VM in the libvirt XML description after 
conversion like this:

  




  

Then qemu-ga successfully gets installed after a 2 minute delay. If I leave the 
default libvirt XML description at:




then the issue of qemu-ga not installing remains (the task is scheduled to run 7 
hours in the future). So it is clearly an issue of the timezone of the guest and 
what is passed from libvirt to be out of sync.


Would be nice to compare how you test it so we're on the same page. (Also, to 
clarify, I'm running virt-v2v 2.2.0 without the proposed patches)




The following are existing bugs about qemu-ga:

*Bug 1820144*  - cannot 
install qemu-ga to some guests even if scheduled qemu-ga installation task exist
*Bug 2114809*  - Can't 
install qemu-ga because of network address error after converting MD-RAID1 win11 
host by virt-p2v
*Bug 1820152*  - Fail to 
execute installation-qemu-ga.msi for win2012r2 and win8.1-i386 guests



Thanks & Regards
Ming Xie


Greetings,
Lee

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] Fwd: virt-v2v creating image that does not install guest agent on first boot

2023-09-28 Thread Lee Garrett 

Hi Ming,

On 27.09.23 14:46, Ming Xie wrote:

Hi Richard,

Sorry, I missed the email, I found out that the ova OS used by the customer is 
win11 uefi, so I tried to reproduce the error today using win11-uefi guest


Steps:
1. Convert win10-efi with UTC-8 timezone, win11-non-efi with UTC-8 timezone, 
win11-efi with UTC-8 timezone, win11-efi with UTC+8 timezone, win2022-efi with 
UTC-8 timezone from VMware by v2v, then check the installation status after 
finishing conversion



Summary the test result as below:

     qemu-ga

   Win10-efi-UTC-8  PASS
   Win11-non-efi UTC-8   PASS
   Win11-efi-UTC-8  FAIL(can't find qemu-ga log in c:\)
   Win11-efi-UTC+8 FAIL(can't find qemu-ga log in c:\)
   Win2022-efi-UTC-8  PASS

   Based on the above results, I think this problem is only caused by win11-efi 
and has nothing to do with the time zone "UTC-8"


hmm, where are you exactly setting the timezone? Within the guest, the scheduled 
task, or in the libvirt XML description?


The Windows 11 trial VM in question (downloaded from 
https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/ -> 
https://aka.ms/windev_VM_vmware) comes with the timezone within the VM set to 
US/Pacific.


If I also set the RTC clock for the VM in the libvirt XML description after 
conversion like this:

  




  

Then qemu-ga successfully gets installed after a 2 minute delay. If I leave the 
default libvirt XML description at:




then the issue of qemu-ga not installing remains (the task is scheduled to run 7 
hours in the future). So it is clearly an issue of the timezone of the guest and 
what is passed from libvirt to be out of sync.


Would be nice to compare how you test it so we're on the same page. (Also, to 
clarify, I'm running virt-v2v 2.2.0 without the proposed patches)




The following are existing bugs about qemu-ga:

*Bug 1820144*  - cannot 
install qemu-ga to some guests even if scheduled qemu-ga installation task exist
*Bug 2114809*  - Can't 
install qemu-ga because of network address error after converting MD-RAID1 win11 
host by virt-p2v
*Bug 1820152*  - Fail to 
execute installation-qemu-ga.msi for win2012r2 and win8.1-i386 guests



Thanks & Regards
Ming Xie


Greetings,
Lee

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] Fwd: virt-v2v creating image that does not install guest agent on first boot

2023-09-28 Thread Richard W.M. Jones 
On Thu, Sep 28, 2023 at 07:41:55PM +0800, Ming Xie wrote:
> Hi Richard,
> 
> On Thu, Sep 28, 2023 at 6:14 PM Richard W.M. Jones  wrote:
> 
> Ming,
> 
> For some reason the build seems "stuck" at the moment, but
> if it finishes could you see if this fixes the RHEL 9 case?
> 
> https://kojihub.stream.centos.org/koji/taskinfo?taskID=2879547
> 
> 
> Sure, no problem, but I don't see the package available now, I will check 
> later.

The build seems stuck.  I asked on the RHEL mailing list
if someone can look at it.

> BTW, is there a bug for this issue?

Not at the moment.

Rich.


> Regards
> Ming Xie
> 

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] Fwd: virt-v2v creating image that does not install guest agent on first boot

2023-09-28 Thread Ming Xie 
Hi Richard,

On Thu, Sep 28, 2023 at 6:14 PM Richard W.M. Jones 
wrote:

> Ming,
>
> For some reason the build seems "stuck" at the moment, but
> if it finishes could you see if this fixes the RHEL 9 case?
>
> https://kojihub.stream.centos.org/koji/taskinfo?taskID=2879547
>

Sure, no problem, but I don't see the package available now, I will check
later. BTW, is there a bug for this issue?

Regards
Ming Xie
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] Fwd: virt-v2v creating image that does not install guest agent on first boot

2023-09-28 Thread Richard W.M. Jones 
On Thu, Sep 28, 2023 at 04:03:33PM +0800, Ming Xie wrote:
>    Yes, I reproduced the problem using rhel9 virt-v2v, because I saw that the
> customer was also using rhel9 virt-v2v (version: 2.2), I saw that you wanted 
> to
> solve the problem by fixing the  date of the schtask command, but I see that
> the schtask command used to install qemu-ga has been changed to the one shown
> in "qemu-ga-command-rhel9.png" on rhel9

The \windows\temp\install-qemu-ga-x86_64.msi.ps1 script will
contain the actual install commands.

I'll send you an updated v2v package anyway.

Rich.

>   Besides,  qemu-ga can be installed successfully after converting win11-efi
> -UTC-8 guest on rhel8.9, which means I cannot reproduce the issue on rhel8 and
> the schtask command used for installing qemu-ga is old one as ''
> qemu-ga-command-rhel8.png' ' shown on rhel8.9, so I'm not sure if the fix can
> be work with rhel9 virt-v2v  
> 
> 
> Regards
> Ming Xie




-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] Fwd: virt-v2v creating image that does not install guest agent on first boot

2023-09-28 Thread Richard W.M. Jones 
Ming,

For some reason the build seems "stuck" at the moment, but
if it finishes could you see if this fixes the RHEL 9 case?

https://kojihub.stream.centos.org/koji/taskinfo?taskID=2879547

Thanks,

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] [PATCH v7 01/12] nbd/server: Support a request payload

2023-09-28 Thread Vladimir Sementsov-Ogievskiy 

On 27.09.23 18:59, Eric Blake wrote:

We could also try to be a bit more complicated by peeking at the next
few bytes: if they look like a magic number of the next request,
assume the client set the bit accidentally but didn't send a payload
after all; for anything else, assume the client did pass a payload.
But adding in machinery to peek at a prefix is more complex than
either assuming a payload is always present (as done in this patch) or
assuming the bit was in error (and dropping the connection
unconditionally).  Preferences?



Ohh, you are right, thanks for comprehensive explanation. I really missed some things you 
are saying about. Yes, now I agree that "payload always exist when flag is set" 
is the best effort. Finally, that was our aim of the protocol design: make it more 
context independent. Probably, we may fix that in specification as preferable or at least 
possible server behavior about non-compliant client.

r-b coming soon, I just need to take another look with corrected picture in 
mind.

--
Best regards,
Vladimir

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] [libnbd PATCH] info: Try harder for graceful disconnect from server

2023-09-28 Thread Richard W.M. Jones 
On Wed, Sep 27, 2023 at 04:22:48PM -0500, Eric Blake wrote:
> On Fri, Sep 22, 2023 at 04:03:19PM -0500, Eric Blake wrote:
> > On Fri, Sep 22, 2023 at 09:47:55PM +0100, Richard W.M. Jones wrote:
> > > On Fri, Sep 22, 2023 at 12:33:36PM -0500, Eric Blake wrote:
> > > > There are a number of ways in which gathering information can fail.
> > > > But when possible, it makes sense to let the server know that we are
> > > > done talking, as it minimizes the likelihood that nbdinfo's error
> > > > message will be obscured by an accompanying error message by the
> > > > server complaining about an unclean disconnect.
> > > > 
> > > > For example, with a one-off qemu temporarily patched as mentioned in
> > > > commit 0f8ee8c6 to advertise sizes larger than 2^63, kicking off
> > > > 'qemu-nbd -f raw -r file &' produces:
> > > > 
> > > > pre-patch:
> > > > 
> > > > $ ./run nbdinfo --size nbd://localhost
> > > > /home/eblake/libnbd/info/.libs/nbdinfo: nbd_get_size: server claims 
> > > > size 9223372036854781440 which does not fit in signed result: Value too 
> > > > large for defined data type
> > > > qemu-nbd: option negotiation failed: Failed to read opts magic: 
> > > > Unexpected end-of-file before all data were read
> > > 
> > > This doesn't necessarily seem like a bug?
> > 
> > It's a quality of service thing - qemu is just being noisy that the
> > client closed the socket abruptly without giving any reason why.  It
> > doesn't change libnbd's behavior (nbdinfo has already reported its
> > error message), but may confuse people reading qemu-nbd logs.
> > 
> > It may also be a case where we could patch qemu-nbd to not output a
> > complaint if the client exited at a clean point in the back-and-forth
> > transactions.  We still want to be noisy if the socket closes after
> > the first byte has been read, but if there are zero bytes available,
> > announcing that the client no longer cares does not add much value.
> > 
> > > 
> > > This feels like quite a significant change in behaviour to me.  In
> > > particular I'm worried if it interacts in some subtle way with the
> > > forking done by the "[ ... ]" syntax for running servers on the
> > > command line (or any of the other ways that libnbd might fork/exec).
> > 
> > Interesting observation.  atexit() handlers are not preserved across
> > exec, and I think all our fork() paths end in either exec or _exit
> > (also where atexit handlers are ignored), so I don't think we are
> > risking calling the handler twice.

So there's a couple of other things I don't like about this patch
after further reflection:

(1) It causes nbdinfo to do extra I/O on the error path.  While I
think we're only writing (NBD_CMD_DISC?) and not expecting a reply,
there might be some situation where a server (especially a local one)
is able to block progress by refusing to read, causing nbdinfo to
hang on exit.  I'm not sure how realistic that is.

(2) It hints that the best way to use the API is to always issue a
shutdown before close, but in reality that's not required.

I think it's also trying to solve something which isn't a problem.

Rich.

> > > 
> > > Can we hold this patch until after 1.18 is released and then put it
> > > in?  Should only be a week or two.
> > 
> > Sure, being conservative on this one is fine by me.  I'll delay it
> > until after 1.18.
> > 
> > > 
> > > Provisionally ACKed for post-1.18
> 
> Now in as commit fd4f3fea, so we can start getting some soak time under CI.
> 
> -- 
> Eric Blake, Principal Software Engineer
> Red Hat, Inc.
> Virtualization:  qemu.org | libguestfs.org

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs