ANNOUNCE: Forthcoming changes to libvirt mailing list hosting

[Daniel P . Berrangé]

Hello Libvirt Community,

For the last 18(!) years, libvirt community mailing lists have been kindly
hosted by Red Hat Corporate IT on the redhat.com Mailman installation.
In retrospect this wasn't the ideal home for community mailing lists but
that decision is ancient history.

Unfortunately as a consequence of infrastructure changes, the redhat.com
Mailman service is scheduled to be decomissioned, and in the very near
future ($low weeks) will cease to be able to send/receive mail for
libvirt lists.

As a result, all the libvirt mailing lists will need to move to a new
Mailman installation, managed on our behalf by the Red Hat Open Source
Community Infrastructure team[1], who provide services for a variety
of OSS projects.

It is intended that this new service will be dedicated for our community
and thus located somewhere under the libvirt.org domain. It should go
without saying that all existing mail archives will be migrated over to
the new server. None of the historical record should be lost.

It is not considered appropriate netiquette, however, to bulk-resubscribe
all current list members under a new domain. Thus anyone who wants to
remain on the community lists will soon be requested to take action
to enroll on the new mailman service.

The new service is not quite ready for use yet.

This mail serves as a heads up that the changes will be coming in the
next couple of weeks that will require action. A further announcement
will be sent when the new mailman is ready to accept subscriptions,
as well as warnings before the old lists stop accepting messages.

With regards,
Daniel

[1] https://www.osci.io/
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Issue accidentally creating a '--type' defined pool

[Daniel P . Berrangé]

On Fri, Sep 29, 2023 at 09:40:26AM +, Joshua Blake wrote:
> Hello,
> 
> I seem to have accidentally created a pool called "--type", but it doesn't
> appear in 'virsh pool-list'. See below.
> 
> [root@vh01 /]# virsh pool-define-as --name --type dir --target
> /data/storage/
> Pool --type defined
> 
> [root@vh01 /]# virsh pool-list
>  Name StateAutostart
> --
>  isos active   yes
>  jblake   active   yes

You need

  virsh pool-list  --alll

to also seen inactive pools

> [root@vh01 /]# virsh pool-destroy '--type'
> error: command 'pool-destroy' doesn't support option --type

You would need

   virsh pool-destroy -- --type

the bare '--' tells it to stop looking for option flags even if following
strings start in --

> 
> I can't also create a new "default" type because the directory exists for
> this defined '--type'
> 
> [root@vh01 /]# virsh pool-define-as --name default --type dir --target
> /data/storage/
> error: Failed to define pool default
> error: operation failed: Storage source conflict with pool: '--type'
> 
> [root@vh01 /]# virsh pool-delete "--type"
> error: command 'pool-delete' doesn't support option --type
> 
> Any help would be greatly appreciated.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Help ! libvirt

[Daniel P . Berrangé]

On Thu, Sep 21, 2023 at 09:05:43AM +0100, Bhasker C V wrote:
> Adding libvirt mailing list
> apologies for cross-posting
> libvirt version: 9.7.0-1
> 
> On Thu, Sep 21, 2023 at 8:39 AM john doe  wrote:
> 
> > On 9/21/23 09:32, Bhasker C V wrote:
> > > I am getting an error with libivrt when I create a VM
> > >
> > > ```
> > >   $ sudo virsh create ./win11.xml

Please post the win11.xml file contents too

> > > error: Failed to create domain from ./win11.xml
> > > error: internal error: mishandled storage format 'none'
> > >
> > > ```
> > >
> > > This is after I have done a dist-upgrade (was working fine before)
> > > debian trixie.
> > >
> > > error message says
> > > qemuBlockStorageSourceGetBlockdevFormatProps:1227 : internal error:
> > > mishandled storage format 'none'
> > >

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: virtio-mem with virt-install to share memory between guest and host on-demand

[Daniel P . Berrangé]

On Fri, Sep 15, 2023 at 02:41:50AM +0100, Daniel Krajnik wrote:
> Hi
> 
> I'm trying to Passthrough a GPU to a VM and have it still share its free
> memory with the host (just like it would with virtio-balloon). Is this
> something virtio-mem is capable of? I've tried to run it, but it immediately
> took over 9GB of memory:
> 
> virt-install --name test --cpu
> cell0.cpus=0,cell0.memory=4194304,cell1.cpus=1,cell1.memory=4194304 --memory
> maxMemory=65536,maxMemory.slots=8 --memdev
> model=virtio-mem,target.node=0,target.block=2048,target.size=1000 --disk
> size=400 --network network=ai
> --location=/tmp/debian-12.1.0-amd64-netinst.iso --os-variant=debian11
> --graphics none --extra-args "console=ttyS0

When using PCI device assignment, the VFIO driver requires that libvit
pin guest memory into RAM

  
https://gitlab.com/libvirt/libvirt/-/blob/19484ccac5cb1586f9d10d3e6eb9b93ee82885c6/src/qemu/qemu_domain.c#L9567

so that's why you'll see the immediate memory usage. I dont' think there's
any way to workaround this limitation automatically. Closest you can get is
to actually hot-unplug the RAM your guest isn't using, but that needs manual
work to re-hot-plug it if the guest later needs more RAM, so that's not very
friendly.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: how to build libvirt-go-module statically

[Daniel P . Berrangé]

On Wed, Sep 06, 2023 at 03:25:48PM +0800, Jiatong Shen wrote:
> Hello community,
> 
>I want to know how to build libvirt-go-module statically? My go version
> is 1.17 and libvirt-go-module version is v1.9007.0. Thank you very much for
> the help.

There is not any way to build fully statically, because libvirt.so itself
does not support static builds. The default build of libvirt Go bindings
will dynamically link to libvirt.so. If you set the 'libvirt_dlopen'
build tag when compiling, then instead of directly dynamically linking,
the libvirt Go binding will dynamically load libvirt.so at runtime.
The latter approach means you DO NOT need libvirt installed in your
development or build environment, but you DO need libvirt.so installed
in your production deployment.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: SEV, SEV-ES, SEV-SNP

[Daniel P . Berrangé]

On Tue, Aug 29, 2023 at 09:54:40AM +0200, Erik Skultety wrote:
> On Mon, Aug 28, 2023 at 02:10:42PM -0700, Derek Lee wrote:
> > When SEV is enabled in domcapabilities does that just mean any of SEV,
> > SEV-ES, SEV-SNP is possible on the hardware?
> 
> No,  only means that the CPU has 'sev' in the flags. On
> its own it doesn't say anything about the ES/SNP features. It might be 
> possible
> to extract information whether these have been enabled on the HW via BIOS (or
> if they're enabled by default on new platforms) via the base64 encoded ID in
> the  XML subelement, but that's just my guess.

Even if the CPU had the "SEV-ES" feature, it doesn't mean you can use
it, because the firmware lets the hardware owner specify the split
between SEV & SEV-ES guests that are allowed on the machine. IOW even
if it supports SEV-ES, firmware could have been configured to have
zero SEV-ES guests.

For this reason, in the domain capabilities, the 
element has child elements which give the info about the number of
configured guests of each type: 

  
...snip...
10
500
  


Currently there is no SEV-SNP support merged in upstream Linux or
QEMU, and thus libvirt also does not support SEV-SNP. Once all the
pieces are merged in QEMU, we'll extend libvirt doman capabilities
to also reflect whether SEV-SNP is available.

NB, while SEV and SEV-ES can be thought of as just minor variations
of each other, I recommend that SEV-SNP be considered an entirely
separate confidential VM technology because architecturally SEV-SNP
is quite different from SEV/SEV-ES in the way that attestation is
driven from the guest, not the host.  Long term I expect SEV/SEV-ES
to fade into irrelevance and SEV-SNP will be the only thing people
care about on AMD, because SNP is so much better.

> > Similarly, does enabling SEV as a launchSecurity option in a domainXML mean
> > that whichever SEV is available will  be enabled? And if the guest policy
> > has the ES flag set, it will not be created unless ES is enabled?
> 
> Again, ES/SNP are just extra useful security features which sit on top of SEV
> (i.e. there's no such thing as "whichever" SEV), both serving a different
> purpose and both supposed to be transparent to the user, IOW if they're 
> enabled
> on the HW they'll be used.
> 
> Whether a VM is created if you explicitly request the ES flag via the guest
> policy on a non-ES enabled SEV CPU is totally dependent on AMD's FW behaviour
> as libvirt only passes through the values. Based on the SEV API spec, if you
> set ES in the guest policy SEV-ES must be enabled on the HW with no further
> information provided, so my assumption is that the FW will refuse to comply
> with such a request which in turn will result in QEMU returning a failure on
> creating such a VM. Again, totally based on my assumption it would be a design
> flaw of such a security feature if you requested the vCPU states to be 
> included
> in the measurement with the ES policy flag and the FW would silently ignore 
> the
> fact it's not enabled in HW and simply not include the data in the response.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: site certificate

[Daniel P . Berrangé]

On Mon, Jun 12, 2023 at 10:56:29AM +0200, Daan Hoogland wrote:
> LS,
> I am sure you have already noticed but I want to report it anyway;
> When compiling a dependend on
> "https://libvirt.org/maven2/net/juniper/tungsten/juniper-tungsten-api/2.0/juniper-tungsten-api-2.0.pom;,

I'm not sure what that path is for - AFAIK we've only ever hosted
maven stuff under the /org/libvirt namespace / sub-path.

> I get a certificate error. I get the a similar error when accessing
> https://libvirt.org btw
> 
> hope you can do something with this information,

Thanks for reporting this. We had an automation failure due to moving
one of our sub-sites which broke the auto-renewal process.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Cron jobs not working with virsh commands

[Daniel P . Berrangé]

On Fri, May 26, 2023 at 12:30:18PM +0200, Jeff Brown wrote:
> Recently, while testing backup scripts, I happened to notice that 'virsh
> snaphot-create-as . . .' has been failing with 'error: failed to get domain
> '$DOMAIN'' when run through crontab. So I've been unwittingly backing up the
> live, running, QCOW2 images for quite some time because scripts which run
> fine when testing from CLI, fail to work as expected when run as a cron job.
> 
> After much searching, I stumbled upon a related issue with the (obvious?)
> simple solution almost hidden in the answers. You have to explicitly connect
> to the local server with, e.g.:
> 
> virsh -c qemu:///system snapshot-create-as . . .
> 
> Humble apologies if I missed something I should have been aware of, but I
> just thought it worthwhile mentioning here for anyone else searching for an
> answer in the archives.

If you don't specify a URI, then libvirt attempts to guess one based
on what's installed, what's running and what user you're running the
command as.

Assuming libvirt qemu driver is installed & running, the latter point
will be where you hit the problem

Running as 'root' libvirt defaults to qemu:///system.

Running as any non-root account, it defaults to qemu:///session

Some more info here:

  https://libvirt.org/uri.html#default-uri-choice
  https://libvirt.org/drvqemu.html#connections-to-qemu-driver

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Windows Guest on KVM running "single core" after windows update

[Daniel P . Berrangé]

On Tue, May 09, 2023 at 02:00:25PM -0500, Benjammin2068 wrote:
> I will say one thing that I noticed is: when giving the guest 8 cores
> and not setting the topology, windows device manager shows 8 CPUs but
> the task manager only shows 2. Feh. Windows.

Historically, if you did NOT specify topology then QEMU would
defaults to sockets==nCPUs, cores==1, threads=1.

Since QEMU 6.2, new machine types for x86 will prefer cores,
IOW, if you don't specify topology then QEMU would give you
sockets=1, cores=nCPUs, threads=1

The rationale for this is that it matches common physical
hardware. It is rare to see an x86 machine with 32 sockets,
and 1 core, but "commonplace" to see 1 sockets and 32 cores.
Many OS have also had/have dubious licensing logic that would
refuse to use > 1 sockets, but happily use > 1 cores, so
defaulting to cores makes it more likely guest OS would avoid
hitting licensing caps.

Still none of it explains why a guest OS upgrade would suddenly
cease to use all CPUs that it previously had

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Upgrade machine type during migration

[Daniel P . Berrangé]

On Sat, Mar 11, 2023 at 06:35:20PM +0100, Michael Schwartzkopff wrote:
> Hi,
> 
> 
> I have an old system. The guest there is defined with:
> 
>   
>     hvm
>   
> 
> 
> When I try to migrate this guest to a new system I get the error:
> 
> error: internal error: unable to execute QEMU command 'blockdev-add': Failed
> to connect socket: Permission denied
> 
> On the new host I see the log entries:
> 
> libvirtd[22411]: Domain id=18 name='test02'
> uuid=9bad33a8-d18e-4c68-bdbe-dad34142dc22 is tainted: deprecated-config
> (machine type 'pc-q35-rhel8.2.0')
> systemd-machined[9980]: New machine qemu-18-test02.
> systemd[1]: Started Virtual Machine qemu-18-test02.
> systemd-networkd[3914]: vnet20: Link DOWN
> systemd-networkd[3914]: vnet20: Lost carrier
> kernel: br1: port 3(vnet20) entered disabled state
> kernel: device vnet20 left promiscuous mode
> kernel: br1: port 3(vnet20) entered disabled state
> systemd[1]: machine-qemu\x2d18\x2dtest02.scope: Deactivated successfully.
> systemd-machined[9980]: Machine qemu-18-test02 terminated.
> libvirtd[22411]: migration successfully aborted
> 
> 
> It seems there is no possible combination of machine types that is
> compatible for both hosts. The old host only can start rhel8.2 guests, the
> new one only rhel9.0.

RHEL-9 is capable of running all rhel8.x machine types, so there shold
be no compatibility problems here.

The message you see about deprecation is simply a warning that continuing
to use this rhel8.2 machine type is discouraged, because you won't be
able to live migrate it onto future RHEL-10 hosts.

> Is there a possibility to change the machine type during a migration? Is
> such a migration possible? Or do I have so shut down the guest?

Changing machine types during live migration is impossible as that's
a guest visible hardware change. That shouldn't matter foir your
case though, as RHEL-9 is capable fo running rhel8.2 machine types.

I'd suggeest checking the /var/log/libvirt/qemu/$GUEST.log file
for details of why it might have actually failed.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: virNetSocketReadWire

[Daniel P . Berrangé]

On Tue, Mar 07, 2023 at 01:57:24PM +, Simon Fairweather wrote:
> What are the standard reasons for this to fail?
> 
> virNetSocketReadWire:1791 : End of file while reading data: Input/output
> error

The socket connection to libvirtd / other daemons was abruptly
closed. Essentially you can almost always ignore this error
message as being mostly harmless.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Need help

[Daniel P . Berrangé]

On Wed, Dec 21, 2022 at 08:14:14PM +0530, Gk Gk wrote:
> Hi All,
> 
> I am trying to get guest VM's disk statistics using a libvirt python
> library as follows:
> ---
> raw_xml = domain.XMLDesc(0)
> for disk in doc.getElementsByTagName('disk'):
>   disk_data = disk.getElementsByTagName('target')
> ---
> 
> The above code is yielding the following stats for two disks inside the VM:
> --
> disk_name=sda
> read_req=34137,read_bytes=445358592,write_req=2405970,write_bytes=24943391744
> disk_name=sdb
> read_req=34357,read_bytes=450754048,write_req=2405984,write_bytes=24943465472
> ---
> 
> The write_bytes divided by 1024 yields 24358853 kB.
> But when I compare the above statistics from the iostat data shown from
> inside the VM, I see a stark difference between the two results as shown
> below:
> 
> ---
> root@sample:~# iostat
> Linux 4.15.0-101-generic (sample) 12/21/2022 _x86_64_ (1 CPU)
> 
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>0.270.000.030.020.01   99.68
> 
> Device tpskB_read/skB_wrtn/skB_readkB_wrtn
> sda   0.24 0.0411.87 400052  121844120
> sdb   0.00 0.00 1.19   5269   12223752
> 
> 
> Why is there a huge difference in the number of bytes read and written for
> the two disks as reported by libvirt ?  Please explain.

Note that libvirt reports data that starts from when the virtual
machine was created.

The guest OS reports data that starts from when the kernel
booted.

IOW, if you have a running guest, and reboot it several times,
the guest OS values will keep getting reset back to 0 on each
boot, but the libvirt values will accumulate, until a hard
poweroff.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: libvirt_connect_get_machine_types

[Daniel P . Berrangé]

On Wed, Dec 07, 2022 at 11:02:45AM +0100, Michal Prívozník wrote:
> On 12/6/22 20:13, Simon Fairweather wrote:
> > Thanks for the update. What does this mean?
> > 
> > The only problem is that QEMU feeds us PPM which is not wired to PHP
> > really.
> > 
> > We are using this patch to remove imageMagik as it is not required for
> > our needs.
> > Eric - I strip out the requirement for ImageMagick because of all the
> > dependancies
> > needed for that peice.  ImageMagick is only used for libvirt's
> > screenshot API calls
> > which we dont even take advantage of in the unRAID web UI.
> > --- a/configure.ac   2016-10-05 22:20:39.172968837
> > -0500
> > +++ b/configure.ac   2016-10-05 22:21:05.645301252
> > -0500
> > @@ -172,8 +172,6 @@
> >  LIBVIRT_CHECK_PHP_EXTENSIONDIR
> >  LIBVIRT_CHECK_PHP_CONFDIR
> >  
> > -LIBVIRT_CHECK_PHP_EXTENSION([imagick])
> > -
> 
> Yeah, pretty much every distro uses this patch. The problem is, without
> imagick the examples won't work:
> 
> libvirt-php.git/tests $ php -n -c php.ini -S localhost:8080
> ../examples/index.php
> 
> BUT, here's what I can do. I just found out that QEMU added support for
> taking screenshots in PNG format, so I can fix libvirt to prefer that
> (PPM is not very well supported anyway - I haven't found a browser that
> would render it) and drop imagick completely. Then I can do the release
> and distros can then just drop the patch.

FWIW, it should be  relatively easy to display a PPM using a bit of
javascript and a HTML  element, as a PPM is essentially just
raw bitmap data. Parsing the PPM heaer is trivial, and then at most
you have a simple pixel format conversion todo. Might not be the
fastest, but doable.


With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Predictable and consistent net interface naming in guests

[Daniel P . Berrangé]

On Wed, Nov 02, 2022 at 04:08:43PM +0100, Igor Mammedov wrote:
> On Wed, 2 Nov 2022 10:43:10 -0400
> Laine Stump  wrote:
> 
> > On 11/1/22 7:46 AM, Igor Mammedov wrote:
> > > On Mon, 31 Oct 2022 14:48:54 +
> > > Daniel P. Berrangé  wrote:
> > >   
> > >> On Mon, Oct 31, 2022 at 04:32:27PM +0200, Edward Haas wrote:  
> > >>> Hi Igor and Laine,
> > >>>
> > >>> I would like to revive a 2 years old discussion [1] about consistent 
> > >>> network
> > >>> interfaces in the guest.
> > >>>
> > >>> That discussion mentioned that a guest PCI address may change in two 
> > >>> cases:
> > >>> - The PCI topology changes.
> > >>> - The machine type changes.
> > >>>
> > >>> Usually, the machine type is not expected to change, especially if one
> > >>> wants to allow migrations between nodes.
> > >>> I would hope to argue this should not be problematic in practice, 
> > >>> because
> > >>> guest images would be made per a specific machine type.
> > >>>
> > >>> Regarding the PCI topology, I am not sure I understand what changes
> > >>> need to occur to the domxml for a defined guest PCI address to change.
> > >>> The only think that I can think of is a scenario where hotplug/unplug is
> > >>> used,
> > >>> but even then I would expect existing devices to preserve their PCI 
> > >>> address
> > >>> and the plug/unplug device to have a reserved address managed by the one
> > >>> acting on it (the management system).
> > >>>
> > >>> Could you please help clarify in which scenarios the PCI topology can 
> > >>> cause
> > >>> a mess to the naming of interfaces in the guest?
> > >>>
> > >>> Are there any plans to add the acpi_index support?  
> > >>
> > >> This was implemented a year & a half ago
> > >>
> > >>https://libvirt.org/formatdomain.html#network-interfaces
> > >>
> > >> though due to QEMU limitations this only works for the old
> > >> i440fx chipset, not Q35 yet.  
> > > 
> > > Q35 should work partially too. In its case acpi-index support
> > > is limited to hotplug enabled root-ports and PCIe-PCI bridges.
> > > One also has to enable ACPI PCI hotplug (it's enled by default
> > > on recent machine types) for it to work (i.e.it's not supported
> > > in native PCIe hotplug mode).
> > > 
> > > So if mgmt can put nics on root-ports/bridges, then acpi-index
> > > should just work on Q35 as well.  
> > 
> > With only a few exceptions (e.g. the first ich9 audio device, which is 
> > placed directly on the root bus at 00:1B.0 because that is where the 
> > ich9 audio device is located on actual Q35 hardware), libvirt will 
> > automatically put all PCI devices (including network interfaces) on a 
> > pcie-root-port.
> > 
> > After seeing reports that "acpi index doesn't work with Q35 
> > machinetypes" I just assumed that was correct and didn't try it. But 
> > after seeing the "should work partially" statement above, I tried it 
> > just now and an  of a Q35 guest that had its PCI address 
> > auto-assigned by libvirt (and so was placed on a pcie-root-port)m and 
> > had  was given the name "eno4". So what exactly is it 
> > that *doesn't* work?
> 
> From QEMU side:
> acpi-index requires: 
>  1. acpi pci hotplug enabled (which is default on relatively new q35 machine 
> types)
>  2. hotpluggble pci bus (root-port, various pci bridges)
>  3. NIC can be cold or hotplugged, guest should pick up acpi-index of the 
> device
> currently plugged into slot
> what doesn't work:
>  1. device attached to host-bridge directly  (work in progress)
>(q35)
>  2. devices attached to any PXB port and any hierarchy hanging of it (there 
> are not plans to make it work)
>(q35, pc)

I'd say this is still a relatively important, as the PXBs are needed
to create a NUMA placement aware topology for guests, and I'd say it
is undesirable to loose acpi-index if a guest is updated to be NUMA
aware, or if a guest image can be deployed in either normal or NUMA
aware setups.

>  3. devices plugged into hot-plugged bridges/root-ports
> (hotplugged bridge lacks ACPI description) (hard to fix, maybe not 
> possible)
>(q35, pc)

Not so bothered about that, since I think generally mgmt apps
pre-plug sufficient bridges to cope.

>  4. multifunction devices (it's undefined by spec, hence not supported)

Not a big deal.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Predictable and consistent net interface naming in guests

[Daniel P . Berrangé]

On Mon, Oct 31, 2022 at 04:32:27PM +0200, Edward Haas wrote:
> Hi Igor and Laine,
> 
> I would like to revive a 2 years old discussion [1] about consistent network
> interfaces in the guest.
> 
> That discussion mentioned that a guest PCI address may change in two cases:
> - The PCI topology changes.
> - The machine type changes.
> 
> Usually, the machine type is not expected to change, especially if one
> wants to allow migrations between nodes.
> I would hope to argue this should not be problematic in practice, because
> guest images would be made per a specific machine type.
> 
> Regarding the PCI topology, I am not sure I understand what changes
> need to occur to the domxml for a defined guest PCI address to change.
> The only think that I can think of is a scenario where hotplug/unplug is
> used,
> but even then I would expect existing devices to preserve their PCI address
> and the plug/unplug device to have a reserved address managed by the one
> acting on it (the management system).
> 
> Could you please help clarify in which scenarios the PCI topology can cause
> a mess to the naming of interfaces in the guest?
> 
> Are there any plans to add the acpi_index support?

This was implemented a year & a half ago

  https://libvirt.org/formatdomain.html#network-interfaces

though due to QEMU limitations this only works for the old
i440fx chipset, not Q35 yet.


With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Qeustion about how to use domsetlaunchsecstate command correctly.

[Daniel P . Berrangé]

On Fri, Oct 14, 2022 at 11:11:08AM +0800, 贺培轩 wrote:
> Hello,
>   I'm new to libvirt. I have tried to launch a sev vm with secret
> injection recently, and I found the command domsetlaunchsecstate is what I
> need. But I had some problem to make it work. Here is what I did to use
> this command.
> 1. run command: virsh create sev-guest.xml
> 2. create secret header file and secret file.
> 3. run command: virsh domsetlaunchsecstate sev-guest-1
> --secrethdr  --secret  .
> But it will report this error: SEV: not in correct state.
> I think it is because the vm is not in a paused state. So how can I launch
> a sev vm which is in a paused state? How should I revise my xml file?

Just pass the --paused flag, eg

  $ virsh create --paused sev-guest.xml

Note, that before injecting sectrets to the guest, you would want to
perform an attestation to validate the boot measurement is what is
expected.

https://listman.redhat.com/archives/libvir-list/2022-October/234729.html

The next release of libvirt is likley to include a script which
handles the attestation and can inject a secret when it succeeds:

  
https://gitlab.com/berrange/libvirt/-/blob/lgtm-vm/tools/virt-qemu-sev-validate

See docs showing usage here:

  
https://gitlab.com/berrange/libvirt/-/blob/lgtm-vm/docs/manpages/virt-qemu-sev-validate.rst#examples

This script is hardcoded to inject a LUKS disk secret, as defined
by the OVMF amdsev build flavour. I'm curious what kind of secret
you are wanting to inject, and whether our tool needs extending
to cope with other secrets besides the disk.

> The sev-guest.xml I use is as follows:
>  
> sev-guest-1
> d50a4205-40e0-4482-b0dc-f26bb4a1a9ff
> 
> http://libosinfo.org/xmlns/libvirt/domain/1.0;>
> http://ubuntu.com/ubuntu/16.04"/>
> 
> 
> 4194304
> 4194304
> 
> 4563402
> 
> 32
> 
> EPYC
> 
> 
> hvm
> /data01/OVMF.fd
>  template="/data01/OVMF.fd">/var/lib/libvirt/qemu/nvram/sev-guest-1_VARS.fd

For use with SEV, if you want to perform attestation prior
to injecting a disk secret, then use of a  stateless
firmware (ie no NVRAM) is strongly recommended, otherwise
the NVRAM can be used to undermine the integrity of the
guest from a malicious host.


With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Caching qemu capabilities and KubeVirt

[Daniel P . Berrangé]

On Tue, Sep 13, 2022 at 05:31:42PM +0200, Roman Mohr wrote:
> On Thu, Sep 8, 2022 at 4:32 PM Daniel P. Berrangé 
> wrote:
> 
> > On Thu, Sep 08, 2022 at 04:22:17PM +0200, Roman Mohr wrote:
> > > On Thu, Sep 8, 2022 at 4:04 PM Daniel P. Berrangé 
> > > wrote:
> > >
> > > > On Thu, Sep 08, 2022 at 02:23:31PM +0100, Daniel P. Berrangé wrote:
> > > > > On Thu, Sep 08, 2022 at 03:10:09PM +0200, Roman Mohr wrote:
> > > > > > On Thu, Sep 8, 2022 at 2:56 PM Daniel P. Berrangé <
> > berra...@redhat.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > On Thu, Sep 08, 2022 at 02:24:00PM +0200, Roman Mohr wrote:
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > >
> > > > > > > > I have a question regarding capability caching  in the context
> > of
> > > > > > > KubeVirt.
> > > > > > > > Since we start in KubeVirt one libvirt instance per VM, libvirt
> > > > has to
> > > > > > > > re-discover on every VM start the qemu capabilities which
> > leads to
> > > > a
> > > > > > > 1-2s+
> > > > > > > > delay in startup.
> > > > > > > >
> > > > > > > > We already discover the features in a dedicated KubeVirt pod on
> > > > each
> > > > > > > node.
> > > > > > > > Therefore I tried to copy the capabilities over to see if that
> > > > would
> > > > > > > work.
> > > > > > > >
> > > > > > > > It looks like in general it could work, but libvirt seems to
> > > > detect a
> > > > > > > > mismatch in the exposed KVM CPU ID in every pod. Therefore it
> > > > invalidates
> > > > > > > > the cache. The recreated capability cache looks esctly like the
> > > > original
> > > > > > > > one though ...
> > > > > > > >
> > > > > > > > The check responsible for the invalidation is this:
> > > > > > > >
> > > > > > > > ```
> > > > > > > > Outdated capabilities for '%s': host cpuid changed
> > > > > > > > ```
> > > > > > > >
> > > > > > > > So the KVM_GET_SUPPORTED_CPUID call seems to return
> > > > > > > > slightly different values in different containers.
> > > > > > > >
> > > > > > > > After trying out the attached golang scripts in different
> > > > containers, I
> > > > > > > > could indeed see differences.
> > > > > > > >
> > > > > > > > I can however not really judge what the differences in these
> > KVM
> > > > function
> > > > > > > > registers mean and I am curious if someone else knows. The
> > files
> > > > are
> > > > > > > > attached too (as json for easy diffing).
> > > > > > >
> > > > > > > Can you confirm whether the two attached data files were captured
> > > > > > > by containers running on the same physical host, or could each
> > > > > > > container have run on a different host.
> > > > > > >
> > > > > >
> > > > > > They are coming from the same host, that is the most surprising bit
> > > > for me.
> > > > > > I am also very sure that this is the case, because I only had one
> > k8s
> > > > node
> > > > > > from where I took these.
> > > > > > The containers however differ (obviously) on namespaces and on the
> > > > > > privilege level (less obvious). The handler dump is from a fully
> > > > privileged
> > > > > > container.
> > > > >
> > > > > The privilege level sounds like something that might be impactful,
> > > > > so I'll investigate that.  I'd be pretty surprised for namespaces
> > > > > to have any impact thnough.
> > > >
> > > > The privilege level is a red herring. Peter reminded me that we have
> > > > to filter out some parts of CPUID because the APIC IDs vary depending
> > > > on what host CPU the task execut

Re: Caching qemu capabilities and KubeVirt

[Daniel P . Berrangé]

On Thu, Sep 08, 2022 at 04:22:17PM +0200, Roman Mohr wrote:
> On Thu, Sep 8, 2022 at 4:04 PM Daniel P. Berrangé 
> wrote:
> 
> > On Thu, Sep 08, 2022 at 02:23:31PM +0100, Daniel P. Berrangé wrote:
> > > On Thu, Sep 08, 2022 at 03:10:09PM +0200, Roman Mohr wrote:
> > > > On Thu, Sep 8, 2022 at 2:56 PM Daniel P. Berrangé  > >
> > > > wrote:
> > > >
> > > > > On Thu, Sep 08, 2022 at 02:24:00PM +0200, Roman Mohr wrote:
> > > > > > Hi,
> > > > > >
> > > > > >
> > > > > > I have a question regarding capability caching  in the context of
> > > > > KubeVirt.
> > > > > > Since we start in KubeVirt one libvirt instance per VM, libvirt
> > has to
> > > > > > re-discover on every VM start the qemu capabilities which leads to
> > a
> > > > > 1-2s+
> > > > > > delay in startup.
> > > > > >
> > > > > > We already discover the features in a dedicated KubeVirt pod on
> > each
> > > > > node.
> > > > > > Therefore I tried to copy the capabilities over to see if that
> > would
> > > > > work.
> > > > > >
> > > > > > It looks like in general it could work, but libvirt seems to
> > detect a
> > > > > > mismatch in the exposed KVM CPU ID in every pod. Therefore it
> > invalidates
> > > > > > the cache. The recreated capability cache looks esctly like the
> > original
> > > > > > one though ...
> > > > > >
> > > > > > The check responsible for the invalidation is this:
> > > > > >
> > > > > > ```
> > > > > > Outdated capabilities for '%s': host cpuid changed
> > > > > > ```
> > > > > >
> > > > > > So the KVM_GET_SUPPORTED_CPUID call seems to return
> > > > > > slightly different values in different containers.
> > > > > >
> > > > > > After trying out the attached golang scripts in different
> > containers, I
> > > > > > could indeed see differences.
> > > > > >
> > > > > > I can however not really judge what the differences in these KVM
> > function
> > > > > > registers mean and I am curious if someone else knows. The files
> > are
> > > > > > attached too (as json for easy diffing).
> > > > >
> > > > > Can you confirm whether the two attached data files were captured
> > > > > by containers running on the same physical host, or could each
> > > > > container have run on a different host.
> > > > >
> > > >
> > > > They are coming from the same host, that is the most surprising bit
> > for me.
> > > > I am also very sure that this is the case, because I only had one k8s
> > node
> > > > from where I took these.
> > > > The containers however differ (obviously) on namespaces and on the
> > > > privilege level (less obvious). The handler dump is from a fully
> > privileged
> > > > container.
> > >
> > > The privilege level sounds like something that might be impactful,
> > > so I'll investigate that.  I'd be pretty surprised for namespaces
> > > to have any impact thnough.
> >
> > The privilege level is a red herring. Peter reminded me that we have
> > to filter out some parts of CPUID because the APIC IDs vary depending
> > on what host CPU the task executes on.
> >
> >
> > https://gitlab.com/libvirt/libvirt/-/blob/master/src/util/virhostcpu.c#L1346
> >
> > In the 2 jSON files you provide, the differences i see should already
> > be matched by
> >
> > /* filter out local apic id */
> > if (entry->function == 0x01 && entry->index == 0x00)
> > entry->ebx &= 0x00ff;
> > if (entry->function == 0x0b)
> > entry->edx &= 0xff00;
> >
> > so those differences ought not to be causing the cache to be
> > invalidated.
> >
> 
> Hm, maybe I misinterpreted the logs then. The snipped I looked at was this:
> 
> 
> ```
> {"component":"virt-launcher","level":"info","msg":"/dev/kvm has changed
> (1661786802 vs
> 0)","pos":"virQEMUCapsKVMUsable:4850","subcomponent":"libvirt&quo

Re: Caching qemu capabilities and KubeVirt

[Daniel P . Berrangé]

On Thu, Sep 08, 2022 at 02:23:31PM +0100, Daniel P. Berrangé wrote:
> On Thu, Sep 08, 2022 at 03:10:09PM +0200, Roman Mohr wrote:
> > On Thu, Sep 8, 2022 at 2:56 PM Daniel P. Berrangé 
> > wrote:
> > 
> > > On Thu, Sep 08, 2022 at 02:24:00PM +0200, Roman Mohr wrote:
> > > > Hi,
> > > >
> > > >
> > > > I have a question regarding capability caching  in the context of
> > > KubeVirt.
> > > > Since we start in KubeVirt one libvirt instance per VM, libvirt has to
> > > > re-discover on every VM start the qemu capabilities which leads to a
> > > 1-2s+
> > > > delay in startup.
> > > >
> > > > We already discover the features in a dedicated KubeVirt pod on each
> > > node.
> > > > Therefore I tried to copy the capabilities over to see if that would
> > > work.
> > > >
> > > > It looks like in general it could work, but libvirt seems to detect a
> > > > mismatch in the exposed KVM CPU ID in every pod. Therefore it 
> > > > invalidates
> > > > the cache. The recreated capability cache looks esctly like the original
> > > > one though ...
> > > >
> > > > The check responsible for the invalidation is this:
> > > >
> > > > ```
> > > > Outdated capabilities for '%s': host cpuid changed
> > > > ```
> > > >
> > > > So the KVM_GET_SUPPORTED_CPUID call seems to return
> > > > slightly different values in different containers.
> > > >
> > > > After trying out the attached golang scripts in different containers, I
> > > > could indeed see differences.
> > > >
> > > > I can however not really judge what the differences in these KVM 
> > > > function
> > > > registers mean and I am curious if someone else knows. The files are
> > > > attached too (as json for easy diffing).
> > >
> > > Can you confirm whether the two attached data files were captured
> > > by containers running on the same physical host, or could each
> > > container have run on a different host.
> > >
> > 
> > They are coming from the same host, that is the most surprising bit for me.
> > I am also very sure that this is the case, because I only had one k8s node
> > from where I took these.
> > The containers however differ (obviously) on namespaces and on the
> > privilege level (less obvious). The handler dump is from a fully privileged
> > container.
> 
> The privilege level sounds like something that might be impactful,
> so I'll investigate that.  I'd be pretty surprised for namespaces
> to have any impact thnough.

The privilege level is a red herring. Peter reminded me that we have
to filter out some parts of CPUID because the APIC IDs vary depending
on what host CPU the task executes on.

https://gitlab.com/libvirt/libvirt/-/blob/master/src/util/virhostcpu.c#L1346

In the 2 jSON files you provide, the differences i see should already
be matched by

/* filter out local apic id */
if (entry->function == 0x01 && entry->index == 0x00)
entry->ebx &= 0x00ff;
if (entry->function == 0x0b)
entry->edx &= 0xff00;

so those differences ought not to be causing the cache to be
invalidated.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Caching qemu capabilities and KubeVirt

[Daniel P . Berrangé]

On Thu, Sep 08, 2022 at 03:10:09PM +0200, Roman Mohr wrote:
> On Thu, Sep 8, 2022 at 2:56 PM Daniel P. Berrangé 
> wrote:
> 
> > On Thu, Sep 08, 2022 at 02:24:00PM +0200, Roman Mohr wrote:
> > > Hi,
> > >
> > >
> > > I have a question regarding capability caching  in the context of
> > KubeVirt.
> > > Since we start in KubeVirt one libvirt instance per VM, libvirt has to
> > > re-discover on every VM start the qemu capabilities which leads to a
> > 1-2s+
> > > delay in startup.
> > >
> > > We already discover the features in a dedicated KubeVirt pod on each
> > node.
> > > Therefore I tried to copy the capabilities over to see if that would
> > work.
> > >
> > > It looks like in general it could work, but libvirt seems to detect a
> > > mismatch in the exposed KVM CPU ID in every pod. Therefore it invalidates
> > > the cache. The recreated capability cache looks esctly like the original
> > > one though ...
> > >
> > > The check responsible for the invalidation is this:
> > >
> > > ```
> > > Outdated capabilities for '%s': host cpuid changed
> > > ```
> > >
> > > So the KVM_GET_SUPPORTED_CPUID call seems to return
> > > slightly different values in different containers.
> > >
> > > After trying out the attached golang scripts in different containers, I
> > > could indeed see differences.
> > >
> > > I can however not really judge what the differences in these KVM function
> > > registers mean and I am curious if someone else knows. The files are
> > > attached too (as json for easy diffing).
> >
> > Can you confirm whether the two attached data files were captured
> > by containers running on the same physical host, or could each
> > container have run on a different host.
> >
> 
> They are coming from the same host, that is the most surprising bit for me.
> I am also very sure that this is the case, because I only had one k8s node
> from where I took these.
> The containers however differ (obviously) on namespaces and on the
> privilege level (less obvious). The handler dump is from a fully privileged
> container.

The privilege level sounds like something that might be impactful,
so I'll investigate that.  I'd be pretty surprised for namespaces
to have any impact thnough.


With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Caching qemu capabilities and KubeVirt

[Daniel P . Berrangé]

On Thu, Sep 08, 2022 at 02:24:00PM +0200, Roman Mohr wrote:
> Hi,
> 
> 
> I have a question regarding capability caching  in the context of KubeVirt.
> Since we start in KubeVirt one libvirt instance per VM, libvirt has to
> re-discover on every VM start the qemu capabilities which leads to a 1-2s+
> delay in startup.
> 
> We already discover the features in a dedicated KubeVirt pod on each node.
> Therefore I tried to copy the capabilities over to see if that would work.
> 
> It looks like in general it could work, but libvirt seems to detect a
> mismatch in the exposed KVM CPU ID in every pod. Therefore it invalidates
> the cache. The recreated capability cache looks esctly like the original
> one though ...
> 
> The check responsible for the invalidation is this:
> 
> ```
> Outdated capabilities for '%s': host cpuid changed
> ```
> 
> So the KVM_GET_SUPPORTED_CPUID call seems to return
> slightly different values in different containers.
> 
> After trying out the attached golang scripts in different containers, I
> could indeed see differences.
> 
> I can however not really judge what the differences in these KVM function
> registers mean and I am curious if someone else knows. The files are
> attached too (as json for easy diffing).

Can you confirm whether the two attached data files were captured
by containers running on the same physical host, or could each
container have run on a different host.

My understanding is that KVM_GET_SUPPORTED_CPUID returns the intersection
of CPUID flags supported by the physical CPUs and CPUID flag supported by
the KVM kernel module.

IOW, I believe the results should only differe if run across hosts with
differing CPU models and/or kernel versions.

I've not tried to diagnose exactly which feature bits are different
in your dumps yet.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Libvirt web application

[Daniel P . Berrangé]

On Thu, Aug 25, 2022 at 05:13:37PM +0200, Erik Skultety wrote:
> On Thu, Aug 25, 2022 at 10:39:14AM -0400, Callie Martin wrote:
> > Please see below for the user's information:
> > 
> > Sender's Name: Chatnaut
> > Subject: Request for Libvirt's Web Application Listing
> > Sent From: supp...@chatnaut.com
> 
> I don't any such email on the list, so it could have been moderated. Dan, can
> you please check the moderation queue?

There's nothing pending, but there's no record of past decisions so
there's always a possibility it was mistakenly discarded as spam.

THey should just re-send any mail/patch.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: New functionality in virt-install

[Daniel P . Berrangé]

On Tue, Aug 09, 2022 at 12:16:13PM +, O'Reilly, Michael wrote:
> Hi libvirt-users,
> 
> I've been tracking the progress of this patch 
> https://listman.redhat.com/archives/libvir-list/2022-August/233328.html
> to enable SGX support in libvirt.
> 
> My question is this. Who is responsible for ensuring that new functionality 
> like SGX is also supported by virt-install?
> Is it part of the commit process to ensure virt-install can use the new 
> functionality in libvirt, or is it up to the virt-manager community to track 
> changes.
> 
> And maybe this is a question for the virt-tools-l...@redhat.com list also?

Yes, that's a question best directed to virt-tools-list really.

There's no requirement to implement stuff in libvirt & virt-install at
the same time. Anyone who is interested & has time, can contribute
virt-install support separately after libvirt stuff is merged.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Can RHEL7 VM run remote commands to Fedora36 host?

[Daniel P . Berrangé]

On Mon, Aug 01, 2022 at 08:41:02AM -0400, Carol Bouchard wrote:
> I believe I've found what has messed things up.  I've run beaker-in-a-box
> ansible scripts
> which includes the following role:
> https://github.com/beaker-project/beaker-in-a-box/blob/master/roles/libvirt/tasks/main.yml
> as well as a few other changes.

That needs to be changed to not touch the .services at all, and
instead deal with .sockets. And also not libvirtd, for RHEL >= 9
or Fedora >= 35

> 
> So I found these instructions for installing libvirt but it seems obsolete
> for fedora36
> https://computingforgeeks.com/how-to-install-kvm-on-fedora/
> Do you have a list of install steps for the various fedoras?

The install steps haven't changed. All that's different is what systemd
units are active by default. For a fresh install the right thing should
"just work" out of the box, due to systemd presets.

The problem is this ansible script is trying to manually start systemd
services, which ignores the fact they shouldn't be running, as they
are socket activated, and also doesnt' take into account libvirtd
is no longer used on new Fedora/RHEL

> I do have things working with info you and Martin Kletzander provided me
> but would like to see
> a formal list in case there is something else that comes up.



With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Can RHEL7 VM run remote commands to Fedora36 host?

[Daniel P . Berrangé]

On Mon, Aug 01, 2022 at 07:42:51AM -0400, Carol Bouchard wrote:
> These services do not exist
> 
> [cbouchar@ibm-p8-rhevm-17-fsp cbouchar-devel]$ systemctl status
> virtxend.service
> Unit virtxend.service could not be found.
> [cbouchar@ibm-p8-rhevm-17-fsp cbouchar-devel]$ systemctl status
> virtxcd.service
> Unit virtxcd.service could not be found.

That's fine if you've not installed that piece of functionality.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Can RHEL7 VM run remote commands to Fedora36 host?

[Daniel P . Berrangé]

On Fri, Jul 29, 2022 at 08:19:04AM -0400, Carol Bouchard wrote:
> The only reason I toggled libvirtd was because the remote virsh commands
> failed and
> I could see the socket didn't exist.  That suggests to me that virtproxyd
> wasn't up AND
> it was configured at disabled.

Note, that it is important to distinguish the service from the sockets.

In Fedora 36, not only are we using modular daemons by default, but we
also use socket activation.  In a fresh install, you should only see
the following services enabled

 virtqemud.service
 virtxend.service
 virtlxcd.service

Those are needed becuase of requiremenmt todo VM auto-start on host
bootup.

For everything else we only expect sockets enabled:

 virtproxyd.socket
 virtinterfaced.socket
 virtnetworkd.socket
 virtnodedevd.socket
 virtnwfilterd.socket
 virtsecretd.socket
 virtstoraged.socket


IOW,  virtproxyd.service should not be enabled - it is autostarted
when someone connects to virtproxyd.socket.

And nothing related to libvirtd.service or libvirtd.socket should
be enabled

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Can RHEL7 VM run remote commands to Fedora36 host?

[Daniel P . Berrangé]

On Fri, Jul 29, 2022 at 07:49:16AM -0400, Carol Bouchard wrote:
> TY VM!!!
> virtproxyd was disabled and I can assure you I didn't disable it.
> 
> /run/libvirt/libvirt-sock now exists AND
> the remote virsh actions were successful.
> 
> Background on my fedora36 install.  I did not do an upgrade. This
> was a fresh install on a new laptop.  I could see libvirt was
> running so I assumed it was intact.  I had enabled/disabled
> libvirtd only because my remote virsh commands were not working.

Enabling/disabling libvirtd probably interfered with virtproxyd, as
they both want the same sockets.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Can RHEL7 VM run remote commands to Fedora36 host?

[Daniel P . Berrangé]

On Wed, Jul 27, 2022 at 01:18:00PM -0400, Carol Bouchard wrote:
> I have a Fedora36 laptop which hosts VMs with RHEL7 using libvirt.  One of
> the RHEL7 VMs, runs remote commands (as root) to 'start' another VM by
> way of my laptop.  In other words,  the following command is run:
> virsh --connect 'qemu+ssh://192.168.120.1/system' start
> beaker-test-vm1.beaker
> If I run non-remote version of the command on the laptop, it is successful.
> For example,
> virsh --connect qemu:///system start beaker-test-vm1.beaker  <-- Successful
> on laptop.
> 
> If I do a query like the following *(notice socket use)*, it is successful.
> virsh -d0 --connect
> 'qemu+ssh://192.168.120.1/system?*socket*=/var/run/libvirt/libvirt-sock-ro'
> domstate beaker-test-vm1.beaker
> 
> Without socket, I get the following error:
> 
> *error: failed to connect to the hypervisor*
> 
> *error: End of file while reading data: Ncat: No such file or directory.:
> Input/output error*

This is peculiar, it suggests that

  /var/run/libvirt/libvirt-sock

does not exist, while /var/run/libvirt/libvirt-sock-ro does exist.

This ought to be an impossible situation in general. As Erik says,
In Fedora 36 we have the moduler daemons, so these two sockets are
provided by  'virtproxyd.socket' and 'virtproxyd-ro.socket' unit
files, so make sure both of those are running.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: how to get vm uptime through libvirt api

[Daniel P . Berrangé]

On Tue, Jul 26, 2022 at 10:27:19AM +0200, Michal Prívozník wrote:
> On 7/26/22 10:07, Jiatong Shen wrote:
> > thank you very much for reply. I have got another question which is off
> > topic.
> > Am I able to get vm's process id through libvirt api?
> 
> Unfortunately, no. We specifically try to avoid telling users this
> information so that they are not tempted to go behind libvirt's back and
> interact with QEMU directly (which can lead to a split brain scenario -
> libvirt thinks guest is in a certain state, but in fact it's in a
> different state).
> 
> But as I said, it should be fairly easy to extend stats to report when a
> domain was started. If you don't feel comfortable providing a patch for
> that I can try.

Note, the time since a guest was started on the host is NOT the same as
guest uptime. Guest uptime resets on every warm reboot the guest OS
does, while QEMU stays running.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: The default USB controller of pc machine type

[Daniel P . Berrangé]

On Fri, Jul 15, 2022 at 02:36:36PM +0800, Han Han wrote:
> On Wed, Jul 13, 2022 at 7:04 PM Daniel P. Berrangé 
> wrote:
> 
> > On Wed, Jul 13, 2022 at 07:00:17PM +0800, Han Han wrote:
> > > Hello,
> > > I found that the default USB controller for q35 machine type is qemu-xhci
> > > while for pc it is piix3-uhci. Here are the results(on qemu-kvm-7.0.0
> > > qemu-kvm-7.0.0):
> > > ➜  ~ cat /tmp/usb.xml
> > > 
> > >   USB
> > >   1a81d4ef-6537-4ae8-b257-8a15e0c0525f
> > >   2097152
> > >   2097152
> > >   1
> > >   
> > > hvm
> > > 
> > >   
> > >   
> > > /usr/libexec/qemu-kvm
> > >   
> > > 
> > >
> > > ➜  ~ virsh define /tmp/usb.xml && virsh dumpxml USB|grep usb
> > > Domain 'USB' defined from /tmp/usb.xml
> > >
> > > 
> > >
> > >
> > > ➜  ~ cat /tmp/usb-pc.xml
> > > 
> > >   USB
> > >   1a81d4ef-6537-4ae8-b257-8a15e0c0525f
> > >   2097152
> > >   2097152
> > >   1
> > >   
> > > hvm
> > > 
> > >   
> > >   
> > > /usr/libexec/qemu-kvm
> > >   
> > > 
> > >
> > > ➜  ~ virsh define /tmp/usb-pc.xml && virsh dumpxml USB|grep usb
> > > Domain 'USB' defined from /tmp/usb-pc.xml
> > >
> > > 
> > >
> > > And the default usb controller for q35 has been added since:
> > > ed2049ea19 qemu: auto-add generic xhci rather than NEC xhci to Q35
> > domains
> > >
> > > As I know, pc machine supports qemu-xhci as well.   Morever, the
> > > performance of qemu-xhci is better than piix3-uhci. So why not update the
> > > default usb controller of pc machine type to qemu-xhci? I think we can
> > > improve it here.
> >
> > A qemu-xhci controller can be added to any x86 machien type regardless
> > of the default, and most mgmt apps will do exactly that.
> >
> 
> Does that mean, the default USB controller for a specific x86 machine type
> does NOT matter?
> And leave the default USB controller to up-layer mgmt apps?

Yes, that's our general view for most defaults in fact. There's no
single perfect defaults, so we let mgmt apps select thigns explicitly

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: The default USB controller of pc machine type

[Daniel P . Berrangé]

On Wed, Jul 13, 2022 at 07:00:17PM +0800, Han Han wrote:
> Hello,
> I found that the default USB controller for q35 machine type is qemu-xhci
> while for pc it is piix3-uhci. Here are the results(on qemu-kvm-7.0.0
> qemu-kvm-7.0.0):
> ➜  ~ cat /tmp/usb.xml
> 
>   USB
>   1a81d4ef-6537-4ae8-b257-8a15e0c0525f
>   2097152
>   2097152
>   1
>   
> hvm
> 
>   
>   
> /usr/libexec/qemu-kvm
>   
> 
> 
> ➜  ~ virsh define /tmp/usb.xml && virsh dumpxml USB|grep usb
> Domain 'USB' defined from /tmp/usb.xml
> 
> 
> 
> 
> ➜  ~ cat /tmp/usb-pc.xml
> 
>   USB
>   1a81d4ef-6537-4ae8-b257-8a15e0c0525f
>   2097152
>   2097152
>   1
>   
> hvm
> 
>   
>   
> /usr/libexec/qemu-kvm
>   
> 
> 
> ➜  ~ virsh define /tmp/usb-pc.xml && virsh dumpxml USB|grep usb
> Domain 'USB' defined from /tmp/usb-pc.xml
> 
> 
> 
> And the default usb controller for q35 has been added since:
> ed2049ea19 qemu: auto-add generic xhci rather than NEC xhci to Q35 domains
> 
> As I know, pc machine supports qemu-xhci as well.   Morever, the
> performance of qemu-xhci is better than piix3-uhci. So why not update the
> default usb controller of pc machine type to qemu-xhci? I think we can
> improve it here.

A qemu-xhci controller can be added to any x86 machien type regardless
of the default, and most mgmt apps will do exactly that.


With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: systemtap stapvirt tool

[Daniel P . Berrangé]

On Wed, Jul 06, 2022 at 05:25:32PM -0400, Stan Cox wrote:
> stapvirt seems to have a quite similar layout to virsh-console.  All is fine
> for short packets:
> 
> 1. Send file command to remote, stdin_event gets event
> stdin_event:1000 fd 0 events 0x1 stbuf_off 0 stbuf 0x7ffd0e99f848
> termbuf_off 58 buf "file 926928 stap_1f18b4b"
> 2. stream_event gets event
> stream_event:1067 events 0x2 stbuf_off 0 stbuf 0x7ffd0e99f848 buf "file
> 926928 stap_1f18b4b"
> 
> but stream_event is not triggered when kernel module is being sent in chunks
> 
> 3. stdin_event reads first chunk of kernel module
>(which does virStreamEventUpdateCallback when it notices buffer contents)
> stdin_event:1000 fd 0 events 0x1 stbuf_off 0 stbuf 0x7ffd0e99f848
> termbuf_off 1024 buf "^?ELF^B^A^A"
> 4. but stream_event is never triggered
> 
> What triggers stream_event to be called such that the 2 case would be
> triggered but not the 4 case?

I don't know - as best I can tell all your logic matches what is in
virsh console.

What libvirt version are you testing with, and have you tried a differnt
version ?

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: systemtap stapvirt tool

[Daniel P . Berrangé]

On Wed, Jul 06, 2022 at 11:43:41AM -0400, Stan Cox wrote:
> I am having a quandary with libvirt and am wondering if someone can shed
> some light.
> 
> Systemtap converts a probing and instrumentation language into a kernel
> module.  The module can optionally be run remotely via ssh or libvirt. The
> libvirt mechanism is not working (not yet sure of timing of regression)  The
> systemtap stap command pipes to a stapvirt command that is the libvirt
> interface.  The remote is running a stapsh tool that receives the requests.
> The initial commands are sent and received okay (strace output on remote)
> but never receives the stap kernel module:
> 
> # local sent option data request
> [pid 11959] read(3, "option data\n", 4096) = 12
> # remote says OK
> [pid 11959] write(3, "OK\n", 3) = 3
> [pid 11959] poll([{fd=3, events=POLLIN}, {fd=0, events=0}, {fd=0,
> events=0}], 3, -1) = 1 ([{fd=3, revents=POLLIN}])
> [pid 11959] --- SIGIO {si_signo=SIGIO, si_code=SI_KERNEL} ---
> [pid 11959] poll([{fd=3, events=POLLHUP}], 1, 0) = 0 (Timeout)
> [pid 11959] rt_sigreturn({mask=[]}) = 1
> # local sent file ... request to download the module
> [pid 11959] read(3, "file 926928 stap_1f18b4b54e74602"..., 4096) = 58
> # remote opens the module
> [pid 11959] openat(AT_FDCWD,
> "stap_1f18b4b54e74602a0a1f0685f2f7333e_1006.ko", O_WRONLY|O_CREAT|O_TRUNC,
> 0666) = 4
> # but never receives it
> [pid 11959] read(3, "", 4096)   = 0
> 
> 
> The local sets up callbacks for stdin, stdout, stream:
> ctxt.stdin_w = virEventAddHandle(STDIN_FILENO,
> VIR_EVENT_HANDLE_READABLE,
> stdin_event, , NULL);
>  ...
> ctxt.stdout_w = virEventAddHandle(STDOUT_FILENO, 0,
>   stdout_event, , NULL);
>  ...
> virStreamEventAddCallback(ctxt.st, VIR_STREAM_EVENT_READABLE |
> VIR_EVENT_HANDLE_WRITABLE,
>   stream_event, , NULL) < 0)
> 
> Then starts an even loop
> while (!disconnect) {
> if (virEventRunDefaultImpl() != 0)
> break;
> }
> 
> 
> stream_event is notified for the command requests, but is not notified for
> the kernel module chunks:
> 
> stdin_event excerpt:
> bytes_read = read(fd, ctxt->termbuf + ctxt->termbuf_off,
>   sizeof(ctxt->termbuf) - ctxt->termbuf_off);
>  ...
> ctxt->termbuf_off += bytes_read;
>  ...
> if (ctxt->termbuf_off) { // we have stuff to write to the stream
> virStreamEventUpdateCallback(ctxt->st, VIR_STREAM_EVENT_READABLE
>  | VIR_STREAM_EVENT_WRITABLE);
> }
> 
> stream_event excerpt:
> 
> if ((events & VIR_STREAM_EVENT_READABLE)
> && (ctxt->stbuf_off < sizeof(ctxt->stbuf))) {
> int bytes_recv = virStreamRecv(st, ctxt->stbuf + ctxt->stbuf_off,
>sizeof(ctxt->stbuf) -
> ctxt->stbuf_off);
> 
> I notice
>  https://libvirt.org/html/libvirt-libvirt-stream.html
> has an example using virStreamSend.  Is that the preferred way to do the
> above?

The example is fairly simplistic and wouldn't suit your need where
you have interleaved bi-directional I/O

The virsh console code is likely the best matching example that we
know is pretty reliable

  https://gitlab.com/libvirt/libvirt/-/blob/master/tools/virsh-console.c

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Slow VM start/revert, when trying to start/revert dozens of VMs in parallel

[Daniel P . Berrangé]

On Mon, May 09, 2022 at 06:52:32PM +, Petr Beneš wrote:
> Hi,
> 
> my problem can be described simply: libvirt can't handle starting dozens of 
> VMs at the same time.
> 
> (technically, it can, but it's really slow.)
> 
> We have an AMD machine with 256 logical cores and 1.5T ram.
> On that machine there is roughly 200 VMs.
> Each VM is the same: 8GB of RAM, 4 VCPUs. Half of them is Win7 x86, the other 
> half is Win7 x64.
> VMs are using qcow2 as the disk image. These images reside in the ramdisk 
> (tmpfs).
> 
> We use these machines for automatic malware analysis, so our scenario 
> consists of this cycle:
> - reverting VM to a running state
> - execute sample inside of the VM for ~1-2 minutes
> - shutdown the VM
> 
> Of course, this results in multiple VMs trying to start at the same time.
> At first, reverts/starts are really fast - second or two.
> After about a minute, the "revertToSnapshot" suddenly takes 10-15 seconds, 
> which is really unacceptable.
> For comparison, we're running the same scenarion on Proxmox, where the 
> revertToSnapshot usually takes 2 seconds.

Can you share the XML configuration of one of your guests - assuming
they all have the same basic configuration.

As a gut feeling it sounds to me like it could be initially fast due to
utilization of host I/O cache, but then slows down due to having to
flush data to disk / read fresh from disk. This could be the case if
the disk configuration cache mode is set to certain values, so the XML
config will show us this info.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: macvtap with disconnected physical interface

[Daniel P . Berrangé]

On Mon, May 02, 2022 at 03:42:05AM +0200, Gionatan Danti wrote:
> Dear list,
> I just discovered the hard way that if the lower lever physical interface of
> a macvlan bridge is disconnected (ie: by unplugging the eth cable, resulting
> in no carrier), inter-guest network traffic from all virtual machines bound
> to the disconnected interface is dropped.
> 
> This behavior surprises me, as with the classic bridges I can disconnect the
> underlying physical interface without causing any harm to inter-guest
> traffic.
> 
> Am I doing something wrong, or this really is the expected behavior? If so,
> can I force the macvtap interfaces to bridge traffic even when the
> underlying physical interface is disconnected?

Can you share the  configuration for your guest NIC so we
can see how it is setup.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: When does the balloon-change or control-error event occur

[Daniel P . Berrangé]

On Thu, Apr 07, 2022 at 05:16:45PM +0800, Han Han wrote:
> Hi developers,
> I have questions about balloon-change or control-error event:
> 1. What's the meaning of these events
> 2. When do the events occur?
> 
> The comments of their callbacks don't mention that(
> https://gitlab.com/libvirt/libvirt/-/blob/master/include/libvirt/libvirt-domain.h#L4130
> https://gitlab.com/libvirt/libvirt/-/blob/master/include/libvirt/libvirt-domain.h#L3736

'balloon-change' is emitted any time the guest OS changes the ballon
inflation level.

eg if the host admin sets the balloon target to 1 GB and the guest is
currently using 2 GB, it might not be able to immediately drop down to
the 1 GB mark. The balloon-change events will emited as it make progress
towards teh 1 GB mark.

control-error is emitted when libvirt has some kind of problem controlling
the VM. The VM is still running, but libvirt may not be able to make changes
to its config. This can happen if libvirt has problems parsing JSON from QMP.
In practice it is highly unlikely for this to ever happen.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Libvirt vs QEMU benefits

[Daniel P . Berrangé]

On Wed, Apr 06, 2022 at 12:44:37PM +, M, Shivakumar wrote:
> Hello,
> 
> For one of our case validation, we were using direct QEMU commands before for 
> VM creation as it was easier to configure the VMs. Inside VM we do run the 
> real-time latency test.
> Recently we switched to libvirt for the VM creation and deletion. 
> Surprisingly, we do see a significant increase in the real-time latency 
> performance for the VMs launched through the libvirt.
> 
> W.r.t configuration wise both VMs are the same, we just converted the 
> existing QEMU commands into libvirt XMLs.

It would be useful to share your QEMU command line args seen both
with directly running QEMU and via libvirt. I'd be really suprised
if your direct config was exactly the same as libvirt's.

> I am wondering what and all the features which libvirt has,
> improving this performance.

If it isn't related to QEMU configuration, then most likely candidate
is the use of cgroups for placing VMs.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Debugging hanging libvirt

[Daniel P . Berrangé]

On Wed, Mar 30, 2022 at 01:11:50AM +, Tobias Hofmann (tohofman) wrote:
> Hello all,
> 
> I have a system with one VM running. After some time the VM needs to be
> temporarily stopped and started again. This start operation fails and
> from that point on any virsh command is hanging and does not execute.
> This issue is reproducible and I have already figured out that restarting
> libvirtd resolves this issue. However, I’m now trying to understand why
> it’s getting stuck in the first place.
> 
> I try not to get too much into detail because I think this would be more
> confusing than it would actually help to understand the problem. In
> general, I’m wondering what approach you should follow to debug why
> libvirt gets stuck.
> Online I’ve read that you should run this command: `# gdb -batch -p
> $(pidof libvirtd) -ex 't a a bt f'`. I’ve run that command and attached
> the output to this mail. However, I have to admit that I have no idea
> what to do with it.

That's perfect. In that trace we can see two threads of interest

One thread is attempting to start the QEMU instance:

Thread 18 (Thread 0x7f58bcf3e700 (LWP 19010)):
#0  0x7f58c9e3a9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x7f58ccc79e26 in virCondWait () from /lib64/libvirt.so.0
#2  0x7f58a046f70b in qemuMonitorSend () from 
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#3  0x7f58a04811d0 in qemuMonitorJSONCommandWithFd () from 
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x7f58a0482a01 in qemuMonitorJSONSetCapabilities () from 
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x7f58a044f567 in qemuConnectMonitor () from 
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x7f58a04506f8 in qemuProcessWaitForMonitor () from 
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x7f58a0456a52 in qemuProcessLaunch () from 
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x7f58a045a4b2 in qemuProcessStart () from 
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#9  0x7f58a04bd5c6 in qemuDomainObjStart.constprop.50 () from 
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#10 0x7f58a04bdd26 in qemuDomainCreateWithFlags () from 
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#11 0x7f58cce1524c in virDomainCreate () from /lib64/libvirt.so.0
#12 0x55e7499a3da3 in remoteDispatchDomainCreateHelper ()
#13 0x7f58ccd461d5 in virNetServerProgramDispatch () from 
/lib64/libvirt.so.0
#14 0x7f58ccd4c9ad in virNetServerHandleJob () from /lib64/libvirt.so.0
#15 0x7f58ccc7a831 in virThreadPoolWorker () from /lib64/libvirt.so.0
#16 0x7f58ccc79bb8 in virThreadHelper () from /lib64/libvirt.so.0
#17 0x7f58c9e36e65 in start_thread () from /lib64/libpthread.so.0
#18 0x7f58c9b5f92d in clone () from /lib64/libc.so.6

This shows we have started QEMU and are trying to connect to the monitor.
We have queued some data to send and are waiting for the event loop to
send it and get the reply back.


The next thread of interest is

Thread 1 (Thread 0x7f58cd74c8c0 (LWP 19008)):
#0  0x7f58c9e3a35e in pthread_rwlock_wrlock () from 
/lib64/libpthread.so.0
#1  0x7f58a0b779e7 in nwfilterStateReload () from 
/usr/lib64/libvirt/connection-driver/libvirt_driver_nwfilter.so
#2  0x7f58a0b77aa5 in nwfilterFirewalldDBusFilter () from 
/usr/lib64/libvirt/connection-driver/libvirt_driver_nwfilter.so
#3  0x7f58ca8f6c5e in dbus_connection_dispatch () from 
/lib64/libdbus-1.so.3
#4  0x7f58ccc0e421 in virDBusWatchCallback () from /lib64/libvirt.so.0
#5  0x7f58ccc193a7 in virEventPollRunOnce () from /lib64/libvirt.so.0
#6  0x7f58ccc17a72 in virEventRunDefaultImpl () from /lib64/libvirt.so.0
#7  0x7f58ccd4c22d in virNetDaemonRun () from /lib64/libvirt.so.0
#8  0x55e7499758b9 in main ()


This is the main event loop thread. Here we see that it is running a
reload operation for the nwfilter driver. This is triggered when something
restarts the firewalld daemon, or requests that it flushes & reloads its
rules.

This is where the problem is.  The thread starting the QEMU process is
likely holding a read lock on the nwfilter driver, and is waiting for
the event loop thread to finish monitor I/O.

The event loop thread though is blocked waiting for a write lock on
the nwfilter driver, and this will never come available because the
othre thread holds a read lock and is waiting for this thread.


So the root problem here is that we should not be running the reload
code for the nwfilter driver in the main event loop thread, because
it blocks the entire event loop preventing othre threads making
process, and is also dependant on those othuer threads releasing
locks.

I'm really surprised no one has reported this problem to us 

Re: "default" watchdog device - ?

[Daniel P . Berrangé]

On Tue, Mar 15, 2022 at 10:39:50AM +, lejeczek wrote:
> Hi guys.
> 
> Without explicitly, manually using watchdog device for a VM, the VM (centOS
> 8 Stream 4.18.0-365.el8.x86_64) shows '/dev/watchdog' exists.
> To double check - 'dumpxml' does not show any such device - what kind of a
> 'watchdog' that is?

The kernel can always provide a pure software watchdog IIRC. It can be
useful if a userspace app wants a watchdog. The limitation is that it
relies on the kernel remaining functional, as there's no hardware
backing it up.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: libvirtd file descriptors leaking

[Daniel P . Berrangé]

On Tue, Feb 22, 2022 at 06:23:13PM +0300, Дмитрий wrote:
> 
> Updated libvirt with:
> 
> # dnf install @virt
> 
> Not its 2.module_el8.6.0+1087+b42c8331
> 
> Testing:
> 
> # virt-df -d kvm3551
> 
> But still, issue remains. Here how it looks like in /proc (all these 
> socket/pipe symlinks are broken):
> 
> # ls -lah /proc/$(pgrep -f libvirtd)/fd/*
> 
> lrwx-- 1 root root 64 фев 22 18:17 /proc/3600693/fd/49 -> 
> 'socket:[99301766]'
> lrwx-- 1 root root 64 фев 22 18:13 /proc/3600693/fd/5 -> 
> 'socket:[99234506]'
> lrwx-- 1 root root 64 фев 22 18:15 /proc/3600693/fd/51 -> 
> 'socket:[99295594]'
> lrwx-- 1 root root 64 фев 22 18:13 /proc/3600693/fd/52 -> 
> 'socket:[99288645]'
> lrwx-- 1 root root 64 фев 22 18:15 /proc/3600693/fd/53 -> 
> 'socket:[99296279]'
> lrwx-- 1 root root 64 фев 22 18:15 /proc/3600693/fd/54 -> 
> 'socket:[99289772]'
> lrwx-- 1 root root 64 фев 22 18:17 /proc/3600693/fd/55 -> 
> 'socket:[99300744]'
> lrwx-- 1 root root 64 фев 22 18:15 /proc/3600693/fd/56 -> 
> 'socket:[99284626]'
> lrwx-- 1 root root 64 фев 22 18:15 /proc/3600693/fd/57 -> 
> 'socket:[99294777]'
> lrwx-- 1 root root 64 фев 22 18:17 /proc/3600693/fd/58 -> 
> 'socket:[99297768]'
> lrwx-- 1 root root 64 фев 22 18:15 /proc/3600693/fd/59 -> 
> 'socket:[99279867]'
> lrwx-- 1 root root 64 фев 22 18:13 /proc/3600693/fd/6 -> 
> 'socket:[99282298]'
> lrwx-- 1 root root 64 фев 22 18:13 /proc/3600693/fd/60 -> 
> 'socket:[99286467]'
> l-wx-- 1 root root 64 фев 22 18:13 /proc/3600693/fd/7 -> /run/libvirtd.pid
> lrwx-- 1 root root 64 фев 22 18:15 /proc/3600693/fd/70 -> 
> 'socket:[99286794]'
> lr-x-- 1 root root 64 фев 22 18:13 /proc/3600693/fd/8 -> 'pipe:[99282299]'
> lrwx-- 1 root root 64 фев 22 18:15 /proc/3600693/fd/81 -> 
> 'socket:[99281902]'
> lrwx-- 1 root root 64 фев 22 18:13 /proc/3600693/fd/82 -> 
> 'socket:[99288059]'
> lrwx-- 1 root root 64 фев 22 18:13 /proc/3600693/fd/84 -> 
> 'socket:[99285702]'
> lrwx-- 1 root root 64 фев 22 18:15 /proc/3600693/fd/85 -> 
> 'socket:[99300385]'
> lrwx-- 1 root root 64 фев 22 18:13 /proc/3600693/fd/88 -> 
> 'socket:[99291046]'
> lrwx-- 1 root root 64 фев 22 18:15 /proc/3600693/fd/89 -> 
> 'socket:[99299347]'
> l-wx-- 1 root root 64 фев 22 18:13 /proc/3600693/fd/9 -> 'pipe:[99282299]'
> lrwx-- 1 root root 64 фев 22 18:15 /proc/3600693/fd/92 -> 
> 'socket:[99284744]'

It is generally more useful to use 'lsof -p $LIBVIRT_PID', so you
get readable names for sockets.

Also compare before & after so you can identify which is the new
"leaked" file.

Also is that a one time leak, or is a new FD leaked for every
virt-df command that is run ?

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: libvirtd file descriptors leaking

[Daniel P . Berrangé]

On Tue, Feb 22, 2022 at 06:00:50PM +0300, Дмитрий wrote:
> Hello, 
> 
> ## SETUP 
> 
> CentOS 8 Stream 
> libguestfs-tools.noarch 1:1.40.2-24.el8.plesk 
> libvirt.x86_64 7.6.0-6.el8s

The current libvirt shipping in CentOS 8 Stream is

8.0.0-2.module_el8.6.0+1087+b42c8331

which is 6 months newer than what you have installed. So I'd recommend
ensuring you have latest CentOS stream content as a first step.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: qemu+ssh connections to a remote libvirt fail as ssh banner configured

[Daniel P . Berrangé]

On Thu, Feb 10, 2022 at 09:33:38AM +0100, Michal Prívozník wrote:
> On 2/10/22 09:02, Daniel P. Berrangé wrote:
> > On Thu, Feb 10, 2022 at 09:52:52AM +0800, Yalan Zhang wrote:
> >> Hi there,
> >>
> >> I have a system configured with ssh login banner like as below:
> >> # cat ~/.bashrc
> >> ...
> >> echo
> >> "="
> >> echo "== This machine is occupied by xxx for testing now. If you are
> >> about to use it, contact xxx first =="
> >> echo
> >> "="
> >>
> >> It works as expected that whenever someone logs into this system by ssh,
> >> he/she will see this warning message.
> >> But it seems such settings will impact a virsh client connection with ssh,
> >> when I try to connect the libvirt daemon on this system, it will error out 
> >> :
> >> # virsh -c qemu+ssh://${my_host}/system list --all
> >> root@${my_host}'s password:
> >> error: failed to connect to the hypervisor
> >> error: packet 1027423545 bytes received from server too large, want 
> >> 33554432
> > 
> > Libvirt is tunnelling an RPC protocol over the SSH connection.
> > Your bashrc is printing this text onto the SSH conmnection and
> > that corrupts the libvirt RPC protocol.
> > 
> > If you want to print something whjen people login use the
> > /etc/motd file which is designed for this pupose, don't
> > print stuff from a .bashrc.  Libvirt gives the options to
> > SSH that prevent display of /etc/motd contents, so that
> > its RPC protocol doesn't get corrupted.
> 
> One more thing, I wasn't able to reproduce when virt-ssh-helper was
> used. But maybe I wasn't trying hard enough.

That should be affected in exactly the same way. It still relies on
stdout/stdin being clean data channels.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: qemu+ssh connections to a remote libvirt fail as ssh banner configured

[Daniel P . Berrangé]

On Thu, Feb 10, 2022 at 09:52:52AM +0800, Yalan Zhang wrote:
> Hi there,
> 
> I have a system configured with ssh login banner like as below:
> # cat ~/.bashrc
> ...
> echo
> "="
> echo "== This machine is occupied by xxx for testing now. If you are
> about to use it, contact xxx first =="
> echo
> "="
> 
> It works as expected that whenever someone logs into this system by ssh,
> he/she will see this warning message.
> But it seems such settings will impact a virsh client connection with ssh,
> when I try to connect the libvirt daemon on this system, it will error out :
> # virsh -c qemu+ssh://${my_host}/system list --all
> root@${my_host}'s password:
> error: failed to connect to the hypervisor
> error: packet 1027423545 bytes received from server too large, want 33554432

Libvirt is tunnelling an RPC protocol over the SSH connection.
Your bashrc is printing this text onto the SSH conmnection and
that corrupts the libvirt RPC protocol.

If you want to print something whjen people login use the
/etc/motd file which is designed for this pupose, don't
print stuff from a .bashrc.  Libvirt gives the options to
SSH that prevent display of /etc/motd contents, so that
its RPC protocol doesn't get corrupted.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: udmabuf error with libvirt + QEMU

[Daniel P . Berrangé]

On Tue, Feb 01, 2022 at 02:27:55PM +, M, Shivakumar wrote:
> Hi,
> 
> We are seeing an issue with udambuf, where it says "open /dev/udmabuf: No 
> such file or directory " even if the device exits. This issue particularly we 
> are seeing with libvirt. When we run the QEMU args on the command line, 
> everything works as expected.
> It seems to be some permission issue when we use the Libvirt, please help us 
> on resolving this.

When libvirt launches QEMU it puts in place a number of strict
security protections. Libvirt will grant access on a per-file
basis to resources on the host that QEMU should be allowed to
access based on the device configuration in the XML.

In your case though you're using command line passthrough:

>   
> 
> 
> 
> 
> 
>   

This is totally opaque to libvirt and so libvirt won't be
granting access to any resources needed by these args. I'm
assuming /dev/udmabuf is needed by the GTK display for GL
support, or something along those lines.

For further information about your options please consult this page:

  https://libvirt.org/kbase/qemu-passthrough-security.html

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: DMARC & similar @admins

[Daniel P . Berrangé]

On Tue, Jan 11, 2022 at 06:58:29PM +, lejeczek wrote:
> 
> 
> On 11/01/2022 17:29, Daniel P. Berrangé wrote:
> > On Tue, Jan 11, 2022 at 05:18:53PM +, lejeczek wrote:
> > > Hi guys
> > > 
> > > My memory fools me? - I remembered this list had DMAR+other_bits sorted, 
> > > so
> > > users like myself, on Yahoo, did not loose their own emails.
> > Nope, it is currently unfixable. The mail server software handling
> > @redhat.com culls the DKIM signatures, so anyone sending from a
> > domain that requests strict DMARC policy will have their sent mail
> > dropped by any recipient that honours this.  This basically means
> > that most subscribers to this list won't see any mail sent by
> > @yahoo addresses.
> > 
> > I've had bug report escalated for over a year now and there's no
> > ETA to fix it.
> > 
> I don't know if it's the exact same Redhat(platform) but might be worth
> consulting freeIPA, oVirt, clusterlabs(who recently fixed it) - "it" works
> over there.

The mailing lists for all of those projects are hosted on different
servers - none are using the redhat.com mailman. 

I know what needs to be fixed at the technical level, but the SMTP
service for redhat.com doesn't allow us to do what is needed - ie
not strip the DKIM signature. The outdated version of mailman in
use also does not have the workaround feature for this that newer
mailman releases have

I would love to fix this but there's nothing more I can do at this
time. As soon as something else becomes possible I'll try it.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: 'migrate' says it worked but in reality it did not - centOS 9

[Daniel P . Berrangé]

On Tue, Jan 11, 2022 at 05:14:53PM +, lejeczek wrote:
> 
> 
> On 11/01/2022 16:36, Daniel P. Berrangé wrote:
> > On Tue, Jan 11, 2022 at 04:30:11PM +, lejeczek wrote:
> > > Hi guys.
> > > 
> > > I have a peculiar situation where between boxes:
> > > C->A
> > > -> $ virsh migrate --unsafe --live c8kubermaster1
> > > qemu+ssh://10.1.1.99/system
> > > -> $ echo $?
> > > 0
> > > but above does _not_ happen, instead!! VM was stopped in started, but 
> > > _not_
> > > migrated LIVE
> > > 
> > > A->C
> > > -> $ virsh migrate --unsafe --live c8kubermaster1
> > > qemu+ssh://10.1.1.100/system
> > > -> $ echo $?
> > > 0
> > > indeed VM migrates live.
> > > 
> > > box A & C have virtually identical OS stack,
> > > HW difference is:
> > > C = Ryzen 5 5600G
> > > A = Ryzen 5 3600
> > > 
> > > domain XML snippet where I think it matters:
> > > ...
> > >    
> > >    4
> > >    4
> > >    2
> > >    
> > >      /machine
> > >    
> > >    
> > >      hvm
> > >      
> > >    
> > >    
> > >      
> > >      
> > >    
> > >    
> > >      EPYC-IBPB
> > >      
> > >      
> > >      
> > >      
> > >      
> > >      
> > >      
> > >      
> > >    
> > >    
> > >      
> > >      
> > >      
> > >    
> > >    destroy
> > >    restart
> > >    destroy
> > >    
> > >      
> > >      
> > >    
> > >    
> > >      /usr/libexec/qemu-kvm
> > >      
> > > ...
> > > 
> > > Initially I submitted a BZ against 'PCS' but continued to filled with it 
> > > and
> > > I find 'libvirt' might be the culprit(also?) here.
> > > There is not much in logs, certainly nothing (with default verbosity) in
> > > virtqemud.service
> > > Is it that VM gets migrated but then is restarted on 'migrate_to' host? if
> > > so then why?
> > > How to start troubleshooting such 'monstrosity'? - all suggestions
> > > appreciated.
> > /var/log/libvirt/qemu/$GUEST.log  on both hosts should have more info
> > 
> What if there is not much there neither?
> migrate_to(host A) seems to show only config for qemu, no errors no
> warnings.
> migrate_from(host C) shows only:
> ...
> 2022-01-11 17:00:40.687+: initiating migration
> 2022-01-11 17:00:43.413+: shutting down, reason=migrated
> 2022-01-11T17:00:43.414063Z qemu-kvm: terminating on signal 15 from pid
> 24022 ()
> 
> no errors/warning but that 2nd line - ??
> 
> Again, migrating back between the same two hosts - where LIVE succeeds
> migrate_from(host A) also shows:
> ...
> 2022-01-11 17:10:27.921+: initiating migration
> 2022-01-11 17:10:30.459+: shutting down, reason=migrated
> 2022-01-11T17:10:30.460528Z qemu-kvm: terminating on signal 15 from pid
> 73193 (

Both those logs only show the state on the src QEMU during a
migration op. There should be corresponding log for the dst
QEMU at the same point in time.

All tehse messages show that migration was successful from libvirt and
QEMU's POV on the src.

So I expect what'ps happening is that QEMU is crashing on the target
host after migration has finished.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: DMARC & similar @admins

[Daniel P . Berrangé]

On Tue, Jan 11, 2022 at 05:18:53PM +, lejeczek wrote:
> Hi guys
> 
> My memory fools me? - I remembered this list had DMAR+other_bits sorted, so
> users like myself, on Yahoo, did not loose their own emails.

Nope, it is currently unfixable. The mail server software handling
@redhat.com culls the DKIM signatures, so anyone sending from a
domain that requests strict DMARC policy will have their sent mail
dropped by any recipient that honours this.  This basically means
that most subscribers to this list won't see any mail sent by
@yahoo addresses.

I've had bug report escalated for over a year now and there's no
ETA to fix it.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: 'migrate' says it worked but in reality it did not - centOS 9

[Daniel P . Berrangé]

On Tue, Jan 11, 2022 at 04:30:11PM +, lejeczek wrote:
> Hi guys.
> 
> I have a peculiar situation where between boxes:
> C->A
> -> $ virsh migrate --unsafe --live c8kubermaster1
> qemu+ssh://10.1.1.99/system
> -> $ echo $?
> 0
> but above does _not_ happen, instead!! VM was stopped in started, but _not_
> migrated LIVE
> 
> A->C
> -> $ virsh migrate --unsafe --live c8kubermaster1
> qemu+ssh://10.1.1.100/system
> -> $ echo $?
> 0
> indeed VM migrates live.
> 
> box A & C have virtually identical OS stack,
> HW difference is:
> C = Ryzen 5 5600G
> A = Ryzen 5 3600
> 
> domain XML snippet where I think it matters:
> ...
>   
>   4
>   4
>   2
>   
>     /machine
>   
>   
>     hvm
>     
>   
>   
>     
>     
>   
>   
>     EPYC-IBPB
>     
>     
>     
>     
>     
>     
>     
>     
>   
>   
>     
>     
>     
>   
>   destroy
>   restart
>   destroy
>   
>     
>     
>   
>   
>     /usr/libexec/qemu-kvm
>     
> ...
> 
> Initially I submitted a BZ against 'PCS' but continued to filled with it and
> I find 'libvirt' might be the culprit(also?) here.
> There is not much in logs, certainly nothing (with default verbosity) in
> virtqemud.service
> Is it that VM gets migrated but then is restarted on 'migrate_to' host? if
> so then why?
> How to start troubleshooting such 'monstrosity'? - all suggestions
> appreciated.

/var/log/libvirt/qemu/$GUEST.log  on both hosts should have more info


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Failed in using LD_PRELOAD to hook mmap system call in qemu(through libvirt)

[Daniel P . Berrangé]

On Thu, Jan 06, 2022 at 07:44:25PM +0800, Xiong。 wrote:
> Dear sir: 
> I want to hook mmap system call in qemu, and I use
> libvirt to passthrough the environment argument like this:
>  
>
>  
> But it failed. I can hardly find out what is the matter!
> The log in libvirt like this:ERROR: ld.so: object
> '/glibcHookMMAP.so' from LD_PRELOAD cannot be preloaded
> (cannot open shared object file): ignored.Firstly,
> I think it's the problem of authority. But the file
> glibcHookMMAP.so has been done with command `chmod 777`.
> My environment is as follows: Ubuntu 18.04.2 Linux
> 5.4.0-92 libvirt 4.0.0 qemu 2.11.1

Aside from normal file permissions, the other likely problem on Ubuntu
will be AppArmor policy. It likely won't allow QEMU to load files from
the / directory - if you put your .so in /usr/local/lib it might
work.



Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: simulating multiple hypervisors with the test driver

[Daniel P . Berrangé]

On Mon, Jan 03, 2022 at 09:06:35PM -0600, Tom Ammon wrote:
> Hello,
> 
> I'm working on a python application that will manage multiple remote
> libvirt hypervisors. I've been using the test:///default uri for
> single-hypervisor tests, and it works great.
> 
> I'd like to simulate connecting to two different remote hypervisors,
> however, in my testing so far it appears that multiple connections to the
> test:///default uri just look like different connections to the same
> hypervisor. Here's what I tried :

Yes, the test:///default URI is shared process-global state.

> What I would like is to be able to spin up two completely independent
> instances of the test driver so that it can simulate two different
> hypervisors/instances of libvirtd.

Pass in a path to a custom XML file for the connection

eg:

   test:///path/to/checkout/of/libvirt.git/examples/xml/test/testnode.xml

every instance of a file base URL will be unique. See this example
file for guidance on how to write the XML to pre-populate arbitrary
resources

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Best way to install guest when it is not listed in output of osinfo-query os

[Daniel P . Berrangé]

On Tue, Dec 21, 2021 at 02:19:50PM +0100, john doe wrote:
> On 12/21/2021 10:41 AM, Andrea Bolognani wrote:
> > On Mon, Dec 20, 2021 at 10:59:15PM +0100, Martin Kletzander wrote:
> > > Any reason for debian not having an -unknown version like lot of the
> > > other distros?
> > 
> > I don't think there's a specific reason for that, it's probably just
> > a matter of nobody thinking of it until now :)
> > 
> > In addition to that, considering that there already entries for
> > Debian testing and Fedora Rawhide, adding one for Debian unstable
> > might make sense too.
> > 
> 
> That would be lovely if 'debian-unknown' and 'debian11' could be
> available on Bullseye!!! :)
> 
> Is it intentional that the Debian URLs in the output of 'osinfo-query
> os' point to 'debian.org/debian/VERSION_ID' instead of
> 'debian.org/releases/VERSION_ID|VERSION_CODENAME'?

The URLs are not a pointer to any specific resource. They are just an
arbitrarily invented unique identifier & once released, we must never
change any URL. By convention we pick a short "product name" as the
first path component, because over time vendors have introduced new
or parallel products. Thus '/releases/' would not be future proof.

As an example, Fedora has both the traditional 'fedora' OS releases
and 'silverblue'.


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

[Daniel P . Berrangé]

On Tue, Dec 14, 2021 at 06:59:12PM +0800, Qu Wenruo wrote:
> 
> 
> On 2021/12/14 18:36, Daniel P. Berrangé wrote:
> > On Tue, Dec 14, 2021 at 09:34:18AM +, Marc Zyngier wrote:
> > > On Tue, 14 Dec 2021 00:41:01 +,
> > > Qu Wenruo  wrote:
> > > > 
> > > > 
> > > > 
> > > > On 2021/12/14 00:49, Marc Zyngier wrote:
> > > > > On Mon, 13 Dec 2021 16:06:14 +,
> > > > > Peter Maydell  wrote:
> > > > > > 
> > > > > > KVM on big.little setups is a kernel-level question really; I've
> > > > > > cc'd the kvmarm list.
> > > > > 
> > > > > Thanks Peter for throwing us under the big-little bus! ;-)
> > > > > 
> > > > > > 
> > > > > > On Mon, 13 Dec 2021 at 15:02, Qu Wenruo  
> > > > > > wrote:
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > On 2021/12/13 21:17, Michal Prívozník wrote:
> > > > > > > > On 12/11/21 02:58, Qu Wenruo wrote:
> > > > > > > > > Hi,
> > > > > > > > > 
> > > > > > > > > Recently I got my libvirt setup on both RK3399 (RockPro64) 
> > > > > > > > > and RPI CM4,
> > > > > > > > > with upstream kernels.
> > > > > > > > > 
> > > > > > > > > For RPI CM4 its mostly smooth sail, but on RK3399 due to its 
> > > > > > > > > little.BIG
> > > > > > > > > setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 
> > > > > > > > > cores), it
> > > > > > > > > brings quite some troubles for VMs.
> > > > > > > > > 
> > > > > > > > > In short, without proper cpuset to bind the VM to either all 
> > > > > > > > > A72 cores
> > > > > > > > > or all A55 cores, the VM will mostly fail to boot.
> > > > > 
> > > > > s/A55/A53/. There were thankfully no A72+A55 ever produced (just the
> > > > > though of it makes me sick).
> > > > > 
> > > > > > > > > 
> > > > > > > > > Currently the working xml is:
> > > > > > > > > 
> > > > > > > > >  2
> > > > > > > > >  
> > > > > > > > > 
> > > > > > > > > But even with vcpupin, pinning each vcpu to each physical 
> > > > > > > > > core, VM will
> > > > > > > > > mostly fail to start up due to vcpu initialization failed 
> > > > > > > > > with -EINVAL.
> > > > > 
> > > > > Disclaimer: I know nothing about libvirt (and no, I don't want to
> > > > > know! ;-).
> > > > > 
> > > > > However, for things to be reliable, you need to taskset the whole QEMU
> > > > > process to the CPU type you intend to use.
> > > > 
> > > > Yep, that's what I'm doing.
> > > 
> > > Are you sure? The xml directive above seem to only apply to the vcpus,
> > > and no other QEMU thread.
> > 
> > For historical reasons this XML element is a bit misleadingly named.
> > 
> > With the config
> > 
> > 2
> > 
> > the 'cpuset' applies to the QEMU process as a whole - its vCPUs,
> > I/O threads and any other emulator threads.
> > 
> > There is a separate config for setting per-VCPU binding which was
> > illustrated elsewhere in this thread.
> 
> Which also means, I can put the io threads to A53 cores freeing up the
> A72 cores more.
> 
> And is there any plan to deprecate the old "cpuset" key of vcpu element,
> and recommend to use "vcpupin" element?

No, they're complementary as they're operating at different levels
and not every scenario needs this fine grained level.
In fact if you just use 'vcpupin' and don't provide 'cpuset', then
internally treats it as if 'cpuset' was the union of all 'vcpupin'
bitsets.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

[Daniel P . Berrangé]

On Tue, Dec 14, 2021 at 09:34:18AM +, Marc Zyngier wrote:
> On Tue, 14 Dec 2021 00:41:01 +,
> Qu Wenruo  wrote:
> > 
> > 
> > 
> > On 2021/12/14 00:49, Marc Zyngier wrote:
> > > On Mon, 13 Dec 2021 16:06:14 +,
> > > Peter Maydell  wrote:
> > >> 
> > >> KVM on big.little setups is a kernel-level question really; I've
> > >> cc'd the kvmarm list.
> > > 
> > > Thanks Peter for throwing us under the big-little bus! ;-)
> > > 
> > >> 
> > >> On Mon, 13 Dec 2021 at 15:02, Qu Wenruo  wrote:
> > >>> 
> > >>> 
> > >>> 
> > >>> On 2021/12/13 21:17, Michal Prívozník wrote:
> >  On 12/11/21 02:58, Qu Wenruo wrote:
> > > Hi,
> > > 
> > > Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI 
> > > CM4,
> > > with upstream kernels.
> > > 
> > > For RPI CM4 its mostly smooth sail, but on RK3399 due to its 
> > > little.BIG
> > > setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it
> > > brings quite some troubles for VMs.
> > > 
> > > In short, without proper cpuset to bind the VM to either all A72 cores
> > > or all A55 cores, the VM will mostly fail to boot.
> > > 
> > > s/A55/A53/. There were thankfully no A72+A55 ever produced (just the
> > > though of it makes me sick).
> > > 
> > > 
> > > Currently the working xml is:
> > > 
> > > 2
> > > 
> > > 
> > > But even with vcpupin, pinning each vcpu to each physical core, VM 
> > > will
> > > mostly fail to start up due to vcpu initialization failed with 
> > > -EINVAL.
> > > 
> > > Disclaimer: I know nothing about libvirt (and no, I don't want to
> > > know! ;-).
> > > 
> > > However, for things to be reliable, you need to taskset the whole QEMU
> > > process to the CPU type you intend to use.
> > 
> > Yep, that's what I'm doing.
> 
> Are you sure? The xml directive above seem to only apply to the vcpus,
> and no other QEMU thread.

For historical reasons this XML element is a bit misleadingly named.

With the config

   2

the 'cpuset' applies to the QEMU process as a whole - its vCPUs,
I/O threads and any other emulator threads.

There is a separate config for setting per-VCPU binding which was
illustrated elsewhere in this thread.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Guest vm doesn't recover after the nfs connection resume

[Daniel P . Berrangé]

On Tue, Dec 14, 2021 at 03:35:42PM +0800, Liang Cong wrote:
> Hi Daniel,
> 
> Thanks for your reply. I tried the nfs hard mount, and got the same
> behavior of the soft mount.
> But in the /var/log/message, got nfs server recovery message which is not
> printed when mounting as soft mode.
> 
> Dec 14 02:12:47 test-1 kernel: nfs: server ip not responding, still trying
> Dec 14 02:13:39 test-1 kernel: nfs: server ip not responding, timed out
> *Dec 14 02:14:34 test-1 kernel: nfs: server ip OK*
> Dec 14 02:14:34 test-1 kernel: NFS: __nfs4_reclaim_open_state: Lock reclaim
> failed!
> 
> 
> According to my understanding the vm boot process will not recover(the vm
> is still in running state, never paused) to normal until restarting the vm
> guest.
> And it is not the issue of libvirt or qemu, it is just the correct behavior
> with the nfs connection timeout, right?

With 'hard' mount I would not expect QEMU/guest to see any errors
at all, though your messages here about 'Lock reclaim failed' are
a little concerning as it suggests something is not working right
wth NFS. This is beyond my knowledge of NFS though.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: internal error - invalid characters

[Daniel P . Berrangé]

On Mon, Dec 13, 2021 at 09:39:20AM +, lejeczek wrote:
> 
> 
> On 07/12/2021 11:28, Ján Tomko wrote:
> > On a Monday in 2021, lejeczek wrote:
> > > Hi guys.
> > > 
> > > Have you seen something like this below?
> > > ...
> > > internal error: The string resource has invalid characters in its
> > > value
> > > internal error: The provided value contains invalid characters:
> > > Solarstorm SFN5162F SFP+ Server Adapter
> > > 
> > 
> > This error message was only present in libvirt v7.9.0. Since v7.10.0
> > libvirt should ignore non-printable characters.
> > https://bugzilla.redhat.com/show_bug.cgi?id=2022589
> > 
> > Jano
> > 
> Would that "bug" be a reason why my libvirtd fails?

It shouldn't have a negative impact - it just means that VPD info
would not be reported against that particular host device. Everything
else should be unaffected.

> I have libvirtd.service started and see that in the logs, then suffices I
> do:
> -> $ virsh list --all
> (having no VMs started)
> I next thing in the logs is:
> ...
> internal error: The provided value contains invalid characters: Solarstorm
> SFN5162F SFP+ Server Adapter
> internal error: The string resource has invalid characters in its value
> Stopping Virtualization daemon...
> libvirtd.service: Deactivated successfully.
> Stopped Virtualization daemon.
> 
> I'm on Centos 9 with libvirt-daemon-7.9.0-1.el9.x86_64.
> With .service failed I still can start a VM.
> -> $ systemctl status -l libvirtd.service
> ○ libvirtd.service - Virtualization daemon
>  Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; disabled;
> vendor preset: dis>
>  Active: inactive (dead) since Mon 2021-12-13 09:32:04 GMT; 5min ago
> TriggeredBy: ○ libvirtd-tcp.socket
>  ● libvirtd-ro.socket
>  ● libvirtd-admin.socket
>  ● libvirtd.socket
>  ○ libvirtd-tls.socket
> 
> Is it that these versions and/or Centos 9 have introduced new (different
> defaults) ways to use/mange 'libvirtd', with .socket and no .service?

In RHEL-9 and therefore CentOS 9 we're switching to using modular
daemons, so libvirtd is no longer expected to be used.

  https://fedoraproject.org/wiki/Changes/LibvirtModularDaemons
  https://libvirt.org/daemons.html

a current fresh install should setup the modular daemons automatically
and leave libvirtd (and its .sockets) entirely disabled. Older installs 
would remain with libvirtd.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Libgfapi gone too?

[Daniel P . Berrangé]

On Fri, Dec 10, 2021 at 09:03:55PM +0400, Dmitry Melekhov wrote:
> 
> 10.12.2021 18:27, lejeczek пишет:
> > really??
> > That was one of the most useful & practical bits - having libvirt
> > to make use of GlusterFS without having to "expose" GF volumes to the
> > filesystem. (everybody who do virt+glusterfs I know, do have it that
> > way)
> > 
> > 
> I guess this will be available only in RHEV :-)

RHEV is based on RHEL-8, so changes in RHEL-9 don't impact it.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Libgfapi gone too?

[Daniel P . Berrangé]

On Fri, Dec 10, 2021 at 11:18:16AM +, lejeczek wrote:
> 
> 
> On 10/12/2021 11:08, Daniel P. Berrangé wrote:
> > On Thu, Dec 09, 2021 at 08:45:52PM +, lejeczek wrote:
> > > Hi guys.
> > > 
> > > Is glusterFS/libgfapi support removed (permanently) too? In CentOS 9 
> > > Stream.
> > Correct, the in-QEMU glusterfs client has been removed from builds
> > of the qemu-kvm RPM in CentOS-9 / RHEL-9.
> > 
> > If storing VM disk on gluster you will have to resort to using the
> > in-kernel glusterfs client to mount the disks locally, whereupon
> > QEMU can access them as plain files
> > 
> > 
> > 
> Guys, @redhat and all the others responsible - Official (but also blogs,
> changelogs, man pages, etc) docs have begun to form -
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9-beta
> Please put all those _critical_ changes everywhere and admin might go first
> to read for what's new & different, make it underscored + in-bold I'd
> suggest.

I would ultimately expect this kind of thing to be documented in the
release notes, under some kind of heading like "incompatible changes"
or similar.  We're only at 9.0-beta though, so docs are not all
fully fleshed out yet.


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Libgfapi gone too?

[Daniel P . Berrangé]

On Thu, Dec 09, 2021 at 08:45:52PM +, lejeczek wrote:
> Hi guys.
> 
> Is glusterFS/libgfapi support removed (permanently) too? In CentOS 9 Stream.

Correct, the in-QEMU glusterfs client has been removed from builds
of the qemu-kvm RPM in CentOS-9 / RHEL-9.

If storing VM disk on gluster you will have to resort to using the
in-kernel glusterfs client to mount the disks locally, whereupon
QEMU can access them as plain files


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Guest vm doesn't recover after the nfs connection resume

[Daniel P . Berrangé]

On Thu, Dec 09, 2021 at 05:54:15PM +0800, Liang Cong wrote:
> Dear developers：
> 
> I found one issue during regular test and I could not confirm whether it is
> a libvirt|qemu issue or it is a nfs client issue or it is not an issue, so
> could you help to check it?
> Below is the issue reproduce steps:
> 
> 1.there is a nfs server with exports file like:
> /nfs *(async,rw,no_root_squash)
> 2. host machine soft mount nfs:
> mount nfs_server_ip:/nfs /var/lib/libvirt/images/nfs -o v4,soft
> 3. start a guest vm with disk tag xml like below:
> 
> 
>  file='/var/lib/libvirt/images/nfs/RHEL-8.6.0-20211102.1-x86_64.qcow2'
> index='1'/>
> 
> 
> 
> 
> 4.Start the vm and during the guest vm boot, apply the iptables rule to
> drop the nfs connection to nfs server
> iptables -A OUTPUT -d nfs_server_ip -p tcp --dport 2049 -j DROP
> 5. Wait until the error log appear in /var/log/message
> kernel: nfs: server nfs_server_ip not responding, timed out
> 6. delete the iptables rule to retain the connection to nfs server
> iptables -D OUTPUT -d nfs_server_ip -p tcp --dport 2049 -j DROP
> 7. check the guest vm, found the boot process with error and can not
> recover.
> rror: ../../grub-core/disk/i386/pc/biosdisk.c:546:failure reading sector
> 
> 0x7ab8 from `hd0'.
> 
> error: ../../grub-core/disk/i386/pc/biosdisk.c:546:failure reading sector
> 
> 0x9190 from `hd0'.
> 
> error: ../../grub-core/disk/i386/pc/biosdisk.c:546:failure reading sector


So this shows that I/O errors have been sent from the host to the guest.

This means two things:

 - The host has reported I/O errors to QEMU
 - QEMU is confjigured to reporte I/O errors to the guest
   (rerror/werror attributes for disk config)

I expect the first point there is a result of you using 'soft' for
the NFS mount - try it again with 'hard'.

The alternative for 'rerror/werror' is to pause the guest, allowing
the host problem to be solved whereupon you unpause the guest.

Overall this behaviour just looks like a result of your config
choices.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Qemu monitor info tlb gives unable to encode message payload

[Daniel P . Berrangé]

On Wed, Dec 08, 2021 at 01:32:48PM +0100, Philipp Klocke wrote:
> On 12/8/21 12:09, Peter Krempa wrote:
> > CAUTION: This email originated from outside of the organization. Do not 
> > click links or open attachments unless you recognize the sender and know 
> > the content is safe.
> > 
> > 
> > On Wed, Dec 08, 2021 at 10:30:27 +, Philipp Klocke wrote:
> > > Hi,
> > > 
> > > the command
> > >  virsh qemu-monitor-command ubuntu_vm --hmp --cmd "info tlb"
> > > fails with "error: Unable to encode message payload".
> > > 
> > > I found a bugtracker entry for a similiar error [1], but I don't if this 
> > > is the same error (message too large). I also don't know how large an 
> > > info tlb message is.
> > > Preferably I would not have to recompile libvirt just to issue this 
> > > monitor command..
> > Libvirt unfortunately limits strings to 4MiB:
> > 
> > const REMOTE_STRING_MAX = 4194304;
> > 
> > And the reply from qemu-monitor-command is a single string. Now
> > internally we process JSON messages up to 10 MiB so one could argue that
> > we should increase the size for the 'qemu-monitor-command' reply up to
> > 10MiB. This could be straightforward but it's questionable whether it's
> > worth it.
> Thanks for the clarification! I will try to go for the 2nd monitor then.
> > > Then I thought about circumventing the error by connecting directly to 
> > > the qemu monitor via netcat, but I found a thread [2] that says I cannot 
> > > add my own "-monitor tcp:..." to the Qemu commandline arguments.
> > IIRC at that point qemu wasn't able to handle two monitor connections.
> > At this point it is possible to have two concurrent connections to the
> > montitor. Obviously things may break and you get to keep the pieces if
> > it breaks.
> > 
> > By adding:
> > 
> >
> >  
> >  
> >
> 
> When I add this to the config via virsh edit and then do a shutdown +
> reboot, I get a kernel panic.
> I put the corresponding dmesg log on gist:
> https://gist.github.com/klockeph/323f3e0d23a3254ef98a65fd7c8f5a1c

I can't see how that is related to adding an extra -qmp parameter,
since it doesn't affect the guest ABI

FWIW, exposing -qmp over a tcp socket is totally insecure - any
local process can connect to that.


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: No QXL in CentOS 9 Stream?

[Daniel P . Berrangé]

On Mon, Dec 06, 2021 at 06:50:36PM +, lejeczek wrote:
> Hi.
> 
> There seems to be no QXL (and Spice?) support in versions from CentOS 9
> repos.
> Would anybody know if that intended, because something else replaces qxl
> perhaps?

SPICE is discontinued in RHEL, and thus CentOS too.  It remains in Fedora
and other distros, but is unlikely to get further development upstream.

If you haved a Red Hat account there is more info here - sorry it is
annoyingly behind a login

  https://access.redhat.com/solutions/5414901

This page gives accurate recommendations (if you ignore QXL mentions):

  https://www.kraxel.org/blog/2019/09/display-devices-in-qemu/#tldr

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: qemu-img create with preallocation=full and compression_type=zlib

[Daniel P . Berrangé]

On Tue, Nov 30, 2021 at 02:51:22PM +0100, Richard Taubo wrote:
> Hi!
> 
> When running this command on Fedora 35 I get:
> [#] qemu-img create -o preallocation=full -f qcow2 /var/vm_mail/mytest.qcow2 
> 10GB
> => Formatting '/var/vm_mail//mytest.qcow2', fmt=qcow2 . . . 
> preallocation=full compression_type=zlib . . . ."
> 
> Question:
> When using 'preallocation=full’ I was surprised that the resulting qcow2 
> image includes the 'compression_type=zlib’ part.
> Why/when is this compression used when the qcow2 image is fully preallocated?

It doesn't mean the image is currently compressed.

Rather it just records the desired compression algorithm for this image,
if any future writes were to be told to use compression.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: migrate libvirt config between machines

[Daniel P . Berrangé]

On Tue, Nov 30, 2021 at 09:28:36AM +0100, Michal Prívozník wrote:
> On 11/29/21 17:37, daggs wrote:
> > Greetings,
> > 
> > I'd like to migrate my current libvirt config from one machine to another, 
> > what is the best way to do so?
> > 
> > Thanks
> > 
> 
> Do you mean domain/network/... XMLs or {libvirtd,qemu,...}.config files?
> 
> For the former I suggest 'virsh dumpxml $dom > dom.xml' and copying over
> the file onto new machine and defining the domain again 'virsh define
> dom.xml' (s/dumpxml/net-dumpxml/ and s/define/net-define/ for networks
> and so on).
> 
> If you run new enough libvirt there's even 'virsh migrate --offline'
> that encapsulates steps from above.

NB, i'm not sure this is clever enough to copy across the TPM state or
the EDK variables state, if you're using such feature.


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: virsh create fails with "Unable to find any firmware to satisfy 'efi'" for aarch64 guest on macOS

[Daniel P . Berrangé]

On Tue, Nov 30, 2021 at 02:40:07AM -0800, Andrea Bolognani wrote:
> On Tue, Nov 30, 2021 at 09:54:38AM +0100, Michal Prívozník wrote:
> > On 11/30/21 06:49, Nathan Vander Wilt wrote:
> > > Ah, but it looks like the arm64 -> VIR_ARCH_AARCH64 patch
> > > is still in limbo? AFAICT that's the main issue I'm hitting now.
> 
> Unfortunately that's the case.
> 
> > Looking into virArchFromHost() I can see uname() called which is then
> > passed to virArchFromString(). In here, uname machine (which is
> > equivalent to 'uname -m' from shell) is compared against virArchData array:
> >
> > https://gitlab.com/libvirt/libvirt/-/blob/master/src/util/virarch.c#L42
> >
> > So what you are saying is that 'uname -m' reports arm64 and not aarch64?
> > If that's the case then we should revisit the patch you mention.
> 
> Correct: on Apple Silicon Macs the architecture name is reported as
> "arm64", but our code expects it to be "aarch64" because that's what
> we get on Linux.
> 
> Michal, have you actually looked at the patch mentioned earlier? If
> not, you can perhaps do a clean room implementation of the fix based
> on the information provided above and get us out of this stalemate?
> 
> It's quite a simple change, but having seen the original patch I feel
> like I couldn't possibly submit it myself and still be in the clear.

We're over thinking things here. The way this change is implemented
is the only way anyone would write this code, and is a simple cut+paste
from the code pattern in the lines above so arguably already a derived
work. This is not a bit of code that meets the criteria to taint you
from a copyright POV.

I just implemented & posted the obvious fix in virArch and it is
identical to the patch referenced earlier in this thread, rather
proving my point.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Cannot access KVM using virsh console after KVM rebooted or shut off

[Daniel P . Berrangé]

On Mon, Nov 15, 2021 at 06:13:22PM -0500, Yifei Liu wrote:
> Dear all,
> 
> I used ssh with -X option to connect to an Ubuntu remote server where I 
> applied virsh 
> (libvirt) to manage my KVM virtual machines, and my local machine is MacOS. 
> Then, 
> I used virt-install to install a Tiny Core Linux VM, and I got a pop-up Virt 
> Viewer window 
> (via XQuartz) which is the console for my Tiny Core VM. However, if I 
> rebooted my VM, 
> then I checked the VM state by virsh list --all which was running. Then, I 
> tried 
> `virsh console my_vm_domain` but this command got stuck on:
> > Connected to domain my_vm_domain
> > Escape character is ^]
> 
> The `virsh console` command hung without any response. This issue also 
> happened when 
> I used virsh console after I shut down and later started the VM. Even the VM 
> was already 
> started, I cannot return to the VM console by `virsh console`.

This isn't a hang, this is simply indicating that your guest OS is
not running anything on the serial console.  If you configure
the guest to launch a login prompt on the serial port, then
you'll see something on virsh console.

> I can get the VM console in Virt Viewer at the first boot (using 
> virt-install). 
> But, if I shutdown the VM once, I can never access the VM console by virsh 
> console even 
> the VM state is running. Any idea about how to use virsh console?

You're comparing two different things. Virt Viewer connects to the
*graphical* console exposed by the VGA card (or whatever graphics
devce is configure).  virsh console connect to the primary serial
port.

Most OS will only display on the graphical console by default and
require extra steps to activate the serial console.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: is there a way to stop a domain in 'D' process state (uninterruptible) ?

[Daniel P . Berrangé]

On Thu, Oct 21, 2021 at 06:08:58PM +0200, Lentes, Bernd wrote:
> Hi,
> 
> how can i stop/shutdown a domain which is in process state 'D' ?
> 'D' means uninterruptible and a process in 'D' can't be terminated by kill, 
> even not with kill -9.

Yeah, this is tricky. Usually this is a result of an I/O problem of
some kind and so the approach would be to diagnose what that root
cause is and try to solve it, to then unblock the VM.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: snapshot creation broken recently?

[Daniel P . Berrangé]

On Thu, Sep 09, 2021 at 02:33:25PM +0200, Štěpán Němec wrote:
> 
>   Hello,
> 
> until very recently I had been happily creating snapshots of running VMs
> (created using virt-install) simply with
> 
>   virsh --connect qemu:///system snapshot-create-as  
> 
> 
> Now when I try to do that I get the following error:
> 
>   Error creating snapshot: operation failed: Failed to take snapshot: 
> pre-save failed: qxl
> 
> Same thing trying to create a snapshot from virt-manager (which also
> used to work). Here's the full backtrace (as displayed in virt-manager):
> 
> Error: Error while writing VM state: Unknown error -1
> 
> 
> Traceback (most recent call last):
>   File "/usr/share/virt-manager/virtManager/asyncjob.py", line 65, in 
> cb_wrapper
> callback(asyncjob, *args, **kwargs)
>   File "/usr/share/virt-manager/virtManager/details/snapshots.py", line 237, 
> in _do_create_snapshot
> self.vm.create_snapshot(xml)
>   File "/usr/share/virt-manager/virtManager/object/domain.py", line 1124, in 
> create_snapshot
> self._backend.snapshotCreateXML(xml, flags)
>   File "/usr/lib/python3.9/site-packages/libvirt.py", line 3059, in 
> snapshotCreateXML
> raise libvirtError('virDomainSnapshotCreateXML() failed')
> libvirt.libvirtError: operation failed: Failed to take snapshot: pre-save 
> failed: qxl
> Error: Error while writing VM state: Unknown error -1
> 
> I am on Arch Linux (x86_64), using the distro packages. Checking my
> pacman logs, might update of qemu-headless from 6.0.0 to 6.1.0 have
> something to do with that?

Yes, auite likely related to the QEMU update as

  "pre-save failed: qxl"

is an error message that libvirt is getting back from QEMU.

There was a change in this code in qemu in 6.1.0, but that was
replacing an assert() with this graceful error message. So if
anything your older QEMU ought to have been crashing !

Anyway, I think you need to report this to the QEMU community.


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: vm vcpu uses isolated cores

[Daniel P . Berrangé]

On Thu, Sep 02, 2021 at 10:24:08AM +0200, Martin Kletzander wrote:
> On Thu, Sep 02, 2021 at 10:44:06AM +0800, Jiatong Shen wrote:
> > Hello,
> > 
> >   I am trying to understand why qemu vm CPU threads uses isolated cpus.
> > 
> >  I have a host which isolates some cpus using isolcpu
> > like isolcpus=1,2,3,4,5,7,8,9,10,11. unfortunately, vcpupin does not mask
> > out these cpus (vcpupin is still something like ).
> 
> That is because there are use cases which need this.  They isolate cpus
> to be used by VM only (sometimes even moving kernel workload from these
> cpus) and automatically removing isolcpus from the set would break this
> valid behaviour.
> 
> > When I log in to the system, seems qemu cpu thread only runs on these
> > isolcpus. I do not quite understand this behavior, because I think by using
> > isolcpu, kernel schedule will exclude these cpus and thus vcpu thread
> > shouldn't use these cores unless taskset explicitly got called.. So my
> > question is how does cpu thread got scheduled on isolated cpus?
> > 
> 
> libvirt sets the affinity for VMs because libvirt itself might not be
> running on all cores and qemu being a child process would otherwise
> inherit the affinity.  We even have this in the documentation and if you
> want to limit the cpus it needs to be defined in the XML.
> 
> It begs the question whether we should somehow coordinate with the
> kernel based on isolcpus, but I am not sure under what circumstances we
> should do that and what is the proper way to do that.  I would suggest
> you file an issue to discuss this further unless someone comes up with a
> clear decision here.

Well if someone is using isolcpus, it is because they want to have some
pretty strict control over what runs on what CPUs. The notion of what
a good default placement would be, is then is quite ill-defined. Do you
want to avoid the isolcpus CPUs mask, because it is being used for non-VM
tasks, or do you want to use the isolcpus CPUs becuase it is intended
for VM tasks.  Further more, if we paid any attention to isolcpus mask,
then the VM XML configuration would no longer result in a reproducable
deployment - semantics would vary based on the invisible isolcpus
setting.

Given all that, if someone is using isolcpus, then I'd really expect
that they set explicit affinity for the VMs too. Note this does not
have to be done at the libvirt level at all. On systemd hosts all
VMs will be placed in /machine.slice, so even if the QEMU processes
have an all-1s affinity mask, the CPU affinity on /machine.slice
will take priority.

IOW, if setting isolcpus, you should also always set /machine.slice
CPU affinity.

Which leads into the final point - the need for isolcpus is widely
mis-understood. The only scenario isolcpus is generally required
is for hard real-time workloads, where you absolutely must stop
all kernel threads running on those CPUs. In any non-real-time
scenario, it is sufficient to "isolate" / "reserve" CPUs using
CPU affinity in cgroups alone. For systemd this can be done
globally using CPUAffinity in /etc/systemd/system.conf, to
restrict most OS services to some house keeping CPUs, and then
using /machine.slice to grant access to other CPUs for VMs.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: a test email (& dmarc) @admins

[Daniel P . Berrangé]

On Mon, Aug 16, 2021 at 07:23:25PM +0100, lejeczek wrote:
> 
> 
> On 16/08/2021 09:35, Daniel P. Berrangé wrote:
> > On Sat, Aug 14, 2021 at 09:38:50AM +0100, lejeczek wrote:
> > > hi guys.
> > > I sent email a while ago and I wonder, also having not received a single
> > > reply, because I did not get that own email - is there are problem I'm
> > > having here or DMARC(yahoo & others?) are not fixed in by this mailing 
> > > list
> > > - would an admin(anybody) know?
> > This mailing list (and all hosted on redhat.com) are seriously broken
> > when senders have strict DMARC settings. This hits everyone posting
> > from @yahoo.* addresses - mail still gets sent out by the list, but
> > a large number of receipients mail servers will throw it away due to
> > DMARC violation.  I've had tickets open with our mail provider for
> > a year & a half now but progress fixing it is painfully slow.
> > 
> > Regards,
> > Daniel
> not all, which fact is most intriguing to me - as if there were no
> communication on some fundamental level and each mailman(s) admin(s) lived
> in a separate closet.
> One such example where guys had it for very long (if not always) fixed in is
> freeIPA mailing list, but there is a few more I'm subscribed to "owned" by
> Redhat which work.

The freeIPA mailing lists are hosted on fedoraproject.org, which is
handled by different mail infrastructure than redhat.com. NB the
problem isn't in mailman, it is the backend mail infra that is
causing the trouble.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: a test email (& dmarc) @admins

[Daniel P . Berrangé]

On Sat, Aug 14, 2021 at 09:38:50AM +0100, lejeczek wrote:
> hi guys.
> I sent email a while ago and I wonder, also having not received a single
> reply, because I did not get that own email - is there are problem I'm
> having here or DMARC(yahoo & others?) are not fixed in by this mailing list
> - would an admin(anybody) know?

This mailing list (and all hosted on redhat.com) are seriously broken
when senders have strict DMARC settings. This hits everyone posting
from @yahoo.* addresses - mail still gets sent out by the list, but
a large number of receipients mail servers will throw it away due to
DMARC violation.  I've had tickets open with our mail provider for
a year & a half now but progress fixing it is painfully slow.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: RX throttling causes keep-alive timeout

[Daniel P . Berrangé]

On Wed, Aug 04, 2021 at 04:16:10PM +, Ivan Teterevkov wrote:
> Hello folks,
> 
> I recently discovered that the max_client_requests configuration option
> affects the keep-alive RPC and can cause the connection timeout, and I
> wanted to verify if my understanding is correct.
> 
> Let me outline the context. Under certain circumstances, one of the
> connected clients in my setup issued multiple concurrent long-standing
> live-migration requests and reached the default limit of five concurrent
> requests. Consequently, it triggered the RX throttling, so the server
> stopped reading the incoming data from this client's file descriptor.
> Meanwhile, the server issued the keepalive "ping" requests but ignored
> the "pong" responses from the client due to the RX throttling. As a result,
> the server concluded the client was dead and closed the connection with the
> "connection closed due to keepalive timeout" message after the default five
> "ping" attempts with five seconds timeout each.
> 
> The idea of throttling makes perfect sense: the server prevents hogging
> of the worker thread pool (or would prevent the unbounded growth of the
> memory footprint if the libvirtd server continued parsing the incoming
> data and queued the requests). What concerns me is that the server drops
> the connection for the alive clients when they're throttled.

Note the limiting happens before the parsing - we don't even read
the data off the socket when we are rate limited, as we don't want
our in memory pending queue growing unbounded.

> One approach to this problem is implementing the EAGAIN-like handling:
> parse the incoming data above the limit and respond with the error response,
> but handle the keep-alive RPCs gracefully. However, I see two problems here:
> either it's a backwards-compatibility concern if implemented unconditionally
> or polluting the configuration space if implemented conditionally.

There is no way to parse the keepalives, without also pulling all
the preceeding data off the socket, which defeats the purpose of
having the limit.

> What is the community's opinion on the above issue?

IMHO this is a tuning problem for your application. If you are expecting
to have 5 long running operations happening concurrently in normal usage,
then you should have increased the max_client_requests parameter to a
value greater than 5 to give yourself more headroom.

The right limit is hard to suggest without knowing more about your mgmt
application. As an example though, if your application can potentially
have 2 API calls pending per running VM and your host capacity allows
for 100 VMs, then you might plan for your max_client_requests value
to be 200. Having a big value for max_client_requests is not inherantly
a bad thing - we just want to prevent unbounded growth when things go
wrong.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: virsh qemu-monitor-command block_resize not working

[Daniel P . Berrangé]

On Wed, Jul 14, 2021 at 10:10:35AM +0200, arjen-libv...@gugu.be wrote:
> Hi List,
> 
> some version info:
> ]# /usr/libexec/qemu-kvm -version
> QEMU emulator version 4.2.0
> (qemu-kvm-4.2.0-48.module+el8.4.0+534+4680a14e) Copyright (c) 2003-2019
> Fabrice Bellard and the QEMU Project developers
> 
> ]# virsh version
> Compiled against library: libvirt 6.0.0
> Using library: libvirt 6.0.0
> Using API: QEMU 6.0.0
> Running hypervisor: QEMU 4.2.0
> 
> information about block devices for one specific guest:
> ]# virsh qemu-monitor-command uat-win201202.uat.gugu.be --hmp info block
> libvirt-2-format:
> /data/kvm/storage/diskimages01/uat-win201202_disk1_C.qcow2 (qcow2)
> Attached to:  /machine/peripheral/virtio-disk0/virtio-backend Cache
> mode:   writeback
> 
> ide0-0-0: [not inserted]
> Attached to:  ide0-0-0
> Removable device: not locked, tray closed
> 
> 
> But then, when I want to resize the block device, I just cannot seem to
> find the correct device. I tried all of these:
> 
> virsh qemu-monitor-command uat-win201202.uat.gugu.be --hmp block_resize
> libvirt-2-storage 48G
> Error: Cannot find device=libvirt-2-storage nor node_name=
> 
> virsh qemu-monitor-command uat-win201202.uat.gugu.be --hmp block_resize
> drive-libvirt-2-storage 48G
> Error: Cannot find device=drive-libvirt-2-storage nor node_name=
> 
> virsh qemu-monitor-command uat-win201202.uat.gugu.be --hmp block_resize
> virtio-disk0 48G
> Error: Cannot find device=virtio-disk0 nor node_name=
> 
> virsh qemu-monitor-command uat-win201202.uat.gugu.be --hmp block_resize
> drive-virtio-disk0 48G
> Error: Cannot find device=drive-virtio-disk0 nor node_name=
> 
> In a previous qemu version this worked (EL 7). Can someone help me out?

Libvirt has changed the way it configures disks with QEMU to use the
modern -blockdev argument. I expect this impacts your usage, but I
dont know what the correct naming should be.

Ignoring that though, why are you even using qemu-monitor-command in
the first place ?  Libvirt directly supports live block resizing
via the 'virsh blockresize' command


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: host identifier

[Daniel P . Berrangé]

On Mon, Jul 05, 2021 at 08:31:15AM +0800, Jiatong Shen wrote:
> Hello community,
> 
> I am using OpenStack nova and realize that nova now uses host fqdn from
> libvirt as a uniq identifier for a compute node.  The issue is that host
> might need to change its fqdn under some
> rare circumstances and lose some usage data due to the nova manages
> compute node resources.
> 
> My question is: does libvirt already export host uuid identifier to other
> applications? I skim through the code base and find there is host uuid
> related code which reads /etc/machine-id to obtain a uniq uuid for
> migration, but seems this uuid cannot be obtained through virsh interface.

Libvirt has long exposed a host UUID in the capabilities XML

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: issue when not using acpi indices in libvirt 7.4.0 and qemu 6.0.0

[Daniel P . Berrangé]

On Wed, Jun 23, 2021 at 06:49:12PM +0200, Riccardo Ravaioli wrote:
> Hi everyone,
> 
> We have an issue with how network interfaces are presented in the VM with
> the latest libvirt 7.4.0 and qemu 6.0.0.
> 
> Previously, we were on libvirt 7.0.0 and qemu 5.2.0, and we used increasing
> virtual PCI addresses for any type of network interface (virtio, PCI
> passthrough, SRIOV) in order to decide the interface order inside the VM.
> For instance the following snippet yields ens1, ens2 and ens3 in a Debian
> Buster VM:
> 
>   
>  
>  
>   type="pci"/>
>  
>  
> 
>  
>   
>   
>  
>  
>   type="pci"/>
>  
>  
> 
>  
>   
>   
>  
> 
>  
>   type="pci"/>
>   

So your config here does NOT list any ACPI indexes

> After upgrading to libvirt 7.4.0 and qemu 6.0.0, the XML snippet above
> yielded:
> - ens1 for the first virtio interface => OK
> - rename4 for the second virtio interface => **KO**
> - ens3 for the PCI passthrough interface  => OK

So from libvirt's POV, nothing should have changed upon upgrade,
as we wouldn't be setting any ACPI indexes by default.

Can you show the QEMU command line from /var/log/libvirt/qemu/$GUEST.log
both before and after the libvirt upgrade.


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: the SSL cert of libvirt.org is outdated

[Daniel P . Berrangé]

On Mon, Jun 14, 2021 at 09:21:48AM +0100, Daniel P. Berrangé wrote:
> On Mon, Jun 14, 2021 at 02:14:32PM +0800, 伍翀 wrote:
> > hi, there,
> > 
> > the SSL cert of libvirt.org is outdated, could anyone replace it as new one?
> 
> Unfortunately the auto-renew process has broken, and it appears to not
> be quickly fixable because LetsEncrypt have disabled the v1 ACME API
> that our script. I'll need to figure out and deploy a new tool for
> acquiring certs.

Wasn't as bad as first feared. Managed to find a new version of the
software we're using that actually works with v2 ACME APIs, so the
certs should be fine again now.


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: the SSL cert of libvirt.org is outdated

[Daniel P . Berrangé]

On Mon, Jun 14, 2021 at 02:14:32PM +0800, 伍翀 wrote:
> hi, there,
> 
> the SSL cert of libvirt.org is outdated, could anyone replace it as new one?

Unfortunately the auto-renew process has broken, and it appears to not
be quickly fixable because LetsEncrypt have disabled the v1 ACME API
that our script. I'll need to figure out and deploy a new tool for
acquiring certs.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [Virtio-fs] virtiofs mounted filesystems & SELinux

[Daniel P . Berrangé]

On Fri, Jun 04, 2021 at 02:59:29PM +0100, Daniel P. Berrangé wrote:
> On Fri, Jun 04, 2021 at 09:44:39AM -0400, Vivek Goyal wrote:
> > On Thu, Jun 03, 2021 at 10:14:24PM -0400, Link Dupont wrote:
> > > On Thu, Jun 3 2021 at 08:56:46 PM -0400, Link Dupont 
> > > wrote:
> > > >  reproducible scenarios
> > > 
> > > Alright. I reran my tests with a CentOS 8 guest. On CentOS 8 (with a
> > > virtiofs filesystem and with xattr on), the type of files in the mounted
> > > hierarchy are unlabeled_t. I can work around that by switching SELinux in
> > > the guest to permissive or disabled.
> > 
> > cc Dan Walsh. I was discussing this with Dan Walsh yesterday in general.
> > 
> > In general, if we want to enable SELinux both on host and guest, then
> > both host and guest should have same SELinux policy. Otherwise there
> > will be lot of different kind of conflicts because both host and
> > guest will try to work with same selinux label. I guess that in
> > practice this will be very hard to achieve as people will run
> > different host and guest flavors and these might have different
> > policies.
> 
> Yeah, I think there's little to no chance of people keeping the
> same SELinux policy in host/guest, except in very tightly controlled
> narrow use cases where the host admin exerts direct control over
> the precise guest config.
> 
> 
> > So another option is to rename selinux xattr in virtiofs so that
> > any selinux xattr coming from guest is saved as
> > user.virtiofs.security.selinux xattr on host. That way host and guest
> > can have their separate labels without interfering with each other.
> > David Gilbert already has added support for this. I can't remember
> > the exact syntax but you can figure it out from documentation here
> > in xattr remappig section.
> 
> For general purpose virt usage, I think remapping in some way is
> likely to be needed as the default strategy.
> 
> > https://github.com/qemu/qemu/blob/master/docs/tools/virtiofsd.rst
> > 
> > But I have question with selinux xattr remapping. What will happen
> > to initial labels when fs is exported. I mean until and unless
> > some process in guest labels all the exported files, they all
> > with either be unlabeled or pick some generic label for all the
> > files.
> 
> I'd say you need some mechanism to force a re-label inside the
> guest. Normally a relabel will be done in /.autorelabel file
> is present, or in certain other scenarios like selinux policy
> RPM updates.
> 
> We wouldn't want to force a relabel neccesarily for the entire
> FS if we're just hotplugging a new virtiofs export though. So
> perhaps there's scope for supporting usage of a per-mount
> point relabel trigger. eg Host creates $VIRTIOFS-ROOT/.autorelabel
> and whenever the guest sees a new virtiofs export arriving, it
> can look for $VIRTIOFS-MOUNT-POINT/.autorelabel
> 
> > Another option is, can we use a single label for whole of the
> > virtiofs (using context=) option in guest. That way nothing
> > is saved in files as such. But this means that processes in guest
> > can't have different selinux labels on different virtiofs dir/files.
> 
> Forcing a single label for the entire export is passable as a
> fallback plan. This is what people have done for years with
> NFS v3 mounts. It has annoying usage limitations though, so
> if at all possible remapping is a preferrable approach.

Another thing we should bear in mind is that it is entirely plausible for
users to want SELinux in their guest, wihle the host is not using SELinux
at all. eg running a RHEL guest, on a cloud running Ubuntu hosts.

In such a case SELinux essentially won't exist from the POV of the
host OS / admin using. Anything related to labelling will have to be
exclusively done by the guest. If we provide a general purpose way
to remap XAttrs, then this ought to be viable by having a generic
remap rule for anything the guest does.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [Virtio-fs] virtiofs mounted filesystems & SELinux

[Daniel P . Berrangé]

On Fri, Jun 04, 2021 at 10:43:13AM -0400, Vivek Goyal wrote:
> On Fri, Jun 04, 2021 at 02:59:29PM +0100, Daniel P. Berrangé wrote:
> > On Fri, Jun 04, 2021 at 09:44:39AM -0400, Vivek Goyal wrote:
> > > On Thu, Jun 03, 2021 at 10:14:24PM -0400, Link Dupont wrote:
> > > > On Thu, Jun 3 2021 at 08:56:46 PM -0400, Link Dupont 
> > > > wrote:
> > > > >  reproducible scenarios
> > > > 
> > > > Alright. I reran my tests with a CentOS 8 guest. On CentOS 8 (with a
> > > > virtiofs filesystem and with xattr on), the type of files in the mounted
> > > > hierarchy are unlabeled_t. I can work around that by switching SELinux 
> > > > in
> > > > the guest to permissive or disabled.
> > > 
> > > cc Dan Walsh. I was discussing this with Dan Walsh yesterday in general.
> > > 
> > > In general, if we want to enable SELinux both on host and guest, then
> > > both host and guest should have same SELinux policy. Otherwise there
> > > will be lot of different kind of conflicts because both host and
> > > guest will try to work with same selinux label. I guess that in
> > > practice this will be very hard to achieve as people will run
> > > different host and guest flavors and these might have different
> > > policies.
> > 
> > Yeah, I think there's little to no chance of people keeping the
> > same SELinux policy in host/guest, except in very tightly controlled
> > narrow use cases where the host admin exerts direct control over
> > the precise guest config.
> > 
> > 
> > > So another option is to rename selinux xattr in virtiofs so that
> > > any selinux xattr coming from guest is saved as
> > > user.virtiofs.security.selinux xattr on host. That way host and guest
> > > can have their separate labels without interfering with each other.
> > > David Gilbert already has added support for this. I can't remember
> > > the exact syntax but you can figure it out from documentation here
> > > in xattr remappig section.
> > 
> > For general purpose virt usage, I think remapping in some way is
> > likely to be needed as the default strategy.
> > 
> > > https://github.com/qemu/qemu/blob/master/docs/tools/virtiofsd.rst
> > > 
> > > But I have question with selinux xattr remapping. What will happen
> > > to initial labels when fs is exported. I mean until and unless
> > > some process in guest labels all the exported files, they all
> > > with either be unlabeled or pick some generic label for all the
> > > files.
> > 
> > I'd say you need some mechanism to force a re-label inside the
> > guest. Normally a relabel will be done in /.autorelabel file
> > is present, or in certain other scenarios like selinux policy
> > RPM updates.
> > 
> > We wouldn't want to force a relabel neccesarily for the entire
> > FS if we're just hotplugging a new virtiofs export though. So
> > perhaps there's scope for supporting usage of a per-mount
> > point relabel trigger. eg Host creates $VIRTIOFS-ROOT/.autorelabel
> > and whenever the guest sees a new virtiofs export arriving, it
> > can look for $VIRTIOFS-MOUNT-POINT/.autorelabel
> 
> Per mount point auto relabel seems interesting. Will it relabel
> everytime virtiofs export shows up. Or it will intelligence to
> figure out exported fs is already labeled (say from previous boot)
> and no need to relabel again.

Normal behaviour with the existing global /.autorelabel is that
selinux deletes the file once the relabel is complete, so it is
a one-time thing.

Essentially anytime you need to force[1] a one-time only relabel you
just do

   "touch /.autorelabel && reboot"


DanW does however recommend that you configure your system such
that the labels are always correct and thus don't need the global
relabel. https://danwalsh.livejournal.com/38157.html

None the less I think the virtiofs situation is slightly different
and justifies a way to force relabel, provided it can be scoped
to just that one filesystem mount.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [Virtio-fs] virtiofs mounted filesystems & SELinux

[Daniel P . Berrangé]

On Fri, Jun 04, 2021 at 09:44:39AM -0400, Vivek Goyal wrote:
> On Thu, Jun 03, 2021 at 10:14:24PM -0400, Link Dupont wrote:
> > On Thu, Jun 3 2021 at 08:56:46 PM -0400, Link Dupont 
> > wrote:
> > >  reproducible scenarios
> > 
> > Alright. I reran my tests with a CentOS 8 guest. On CentOS 8 (with a
> > virtiofs filesystem and with xattr on), the type of files in the mounted
> > hierarchy are unlabeled_t. I can work around that by switching SELinux in
> > the guest to permissive or disabled.
> 
> cc Dan Walsh. I was discussing this with Dan Walsh yesterday in general.
> 
> In general, if we want to enable SELinux both on host and guest, then
> both host and guest should have same SELinux policy. Otherwise there
> will be lot of different kind of conflicts because both host and
> guest will try to work with same selinux label. I guess that in
> practice this will be very hard to achieve as people will run
> different host and guest flavors and these might have different
> policies.

Yeah, I think there's little to no chance of people keeping the
same SELinux policy in host/guest, except in very tightly controlled
narrow use cases where the host admin exerts direct control over
the precise guest config.


> So another option is to rename selinux xattr in virtiofs so that
> any selinux xattr coming from guest is saved as
> user.virtiofs.security.selinux xattr on host. That way host and guest
> can have their separate labels without interfering with each other.
> David Gilbert already has added support for this. I can't remember
> the exact syntax but you can figure it out from documentation here
> in xattr remappig section.

For general purpose virt usage, I think remapping in some way is
likely to be needed as the default strategy.

> https://github.com/qemu/qemu/blob/master/docs/tools/virtiofsd.rst
> 
> But I have question with selinux xattr remapping. What will happen
> to initial labels when fs is exported. I mean until and unless
> some process in guest labels all the exported files, they all
> with either be unlabeled or pick some generic label for all the
> files.

I'd say you need some mechanism to force a re-label inside the
guest. Normally a relabel will be done in /.autorelabel file
is present, or in certain other scenarios like selinux policy
RPM updates.

We wouldn't want to force a relabel neccesarily for the entire
FS if we're just hotplugging a new virtiofs export though. So
perhaps there's scope for supporting usage of a per-mount
point relabel trigger. eg Host creates $VIRTIOFS-ROOT/.autorelabel
and whenever the guest sees a new virtiofs export arriving, it
can look for $VIRTIOFS-MOUNT-POINT/.autorelabel

> Another option is, can we use a single label for whole of the
> virtiofs (using context=) option in guest. That way nothing
> is saved in files as such. But this means that processes in guest
> can't have different selinux labels on different virtiofs dir/files.

Forcing a single label for the entire export is passable as a
fallback plan. This is what people have done for years with
NFS v3 mounts. It has annoying usage limitations though, so
if at all possible remapping is a preferrable approach.

> 
> Dan, what do you think?
> 
> Thanks
> Vivek
> 
> 
> > 
> > With a CentOS 7 guest, things get less usable. I digested this to a
> > reproducible scenario.
> > 
> > Build a disk image with `virt-builder`, configuring the CentOS Plus kernel
> > to get 9p support.
> > 
> > virt-builder centos-7.8 \
> > --root-password password:centos \
> > --output centos-7.8.qcow2 \
> > --install yum-utils \
> > --run-command 'yum-config-manager --enable centosplus' \
> > --run-command 'sed -ie "s/DEFAULTKERNEL=kernel/DEFAULTKERNEL=kernel-plus/"
> > /etc/sysconfig/kernel' \
> > --append-line '/etc/dracut.conf.d/virtio.conf:add_drivers+="virtio_scsi
> > virtio_pci virtio_console"' \
> > --append-line '/etc/modules-load.d/9pnet_virtio.conf:9pnet_virtio' \
> > --install kernel-plus \
> > --append-line '/etc/fstab:home /home 9p trans=virtio,version=9p2000.L 0 0'
> > 
> > Install the volume into the `default` pool.
> > 
> > sudo install -m644 centos-7.8.qcow2 /var/lib/libvirt/images
> > 
> > Next, define a domain using the disk image (using `virt-install` here for
> > "easy mode").
> > 
> > virt-install \
> > --import \
> > --os-variant centos7.0 \
> > --name centos \
> > --ram 2048 \
> > --disk path=/var/lib/libvirt/images/centos-7.8.qcow2 \
> > --memorybacking access.mode=shared \
> > --filesystem source=/home,target=home,accessmode=passthrough \
> > --autoconsole none
> > 
> > Now with SELinux enforcing, I cannot list the contents of the directories in
> > the mounted hierarchy.
> > 
> > [root@localhost ~]# ls -lZ /home/link
> > ls: cannot open directory /home/link: Permission denied
> > 
> > 
> > 
> > ___
> > Virtio-fs mailing list
> > virtio...@redhat.com
> > https://listman.redhat.com/mailman/listinfo/virtio-fs
> > 
> 

Regards,
Daniel
-- 
|: https://berrange.com   

Re: How to hot plugin a new vhost-user-blk-pci device to running VM?

[Daniel P . Berrangé]

On Mon, May 24, 2021 at 01:04:44PM +0200, Michal Prívozník wrote:
> On 5/21/21 5:28 PM, 梁朝军 wrote:
> > Thanks all of you for your help.
> > One more question regarding  vhost-user-blk-pci type device,  how to 
> > identify a vhost-blk disk in QEUM VM ?  for example, disk name looks like 
> > vda,vdb,..., but that some application in VM want to detect  that a certain 
> > entry is really the device it is waiting for. Specific for windows , they 
> > always show as disk 0, 1, 2….etc
> > Is there any way to identify those disk with each other in VM?
> 
> In general no. Usually disks will be enumerated sequentially - thus the
> first disk on a sata/scsi/usb/.. bus will be sda, the second will be
> sdb, and so on. But libvirt can't guarantee it - the same way you can't
> guarantee how a disk is going to be called with real HW.

You can set the 'serial' property in the disk in libvirt, and then match
that in the guest. For Linux guests that's used in /dev/disk/by-id
symlinks.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: l3 cache and cpu pinning

[Daniel P . Berrangé]

On Thu, Apr 22, 2021 at 01:34:18PM +0200, Roman Mohr wrote:
> On Thu, Apr 22, 2021 at 1:24 PM Roman Mohr  wrote:
> 
> >
> >
> > On Thu, Apr 22, 2021 at 1:19 PM Roman Mohr  wrote:
> >
> >>
> >>
> >> On Wed, Apr 21, 2021 at 1:09 PM Daniel P. Berrangé 
> >> wrote:
> >>
> >>> On Wed, Apr 21, 2021 at 12:53:49PM +0200, Roman Mohr wrote:
> >>> > Hi,
> >>> >
> >>> > I have a question regarding enabling l3 cache emulation on Domains. Can
> >>> > this also be enabled without cpu-pinning, or does it need cpu pinning
> >>> to
> >>> > emulate the l3 caches according to the cpus where the guest is pinned
> >>> to?
> >>>
> >>> I presume you're referring to
> >>>
> >>>   
> >>> 
> >>>   
> >>>
> >>> There is no hard restriction placed on usage of these modes by QEMU.
> >>>
> >>> Conceptually though, you only want to use "passthrough" mode if you
> >>> have configured the sockets/cores/threads topology to match the host
> >>> CPUs. In turn you only ever want to set sockets/cores/threads to
> >>> match the host if you have done CPU pinning such that the topology
> >>> actually matches the host CPUs that have been pinned to.
> >>>
> >>> As a rule of thumb
> >>>
> >>>  - If letting CPUs float
> >>>
> >>>  -> Always uses sockets=1, cores=num-vCPUs, threads=1
> >>>  -> cache==emulate
> >>>  -> Always use 1 guest NUMA node (ie the default)
> >>>
> >>>
> >> Is `emulate` also the default in libvirt? If not, would you see any
> >> reason, e.g. thinking about migrations, to not set it always if no cpu
> >> pinning is done?
> >>
> >
> > To answer my own question: I guess something like [1] is a good reason to
> > not enable l3-cache by default, since it seems to have an impact on VM
> > density on nodes.
> >
> 
> Hm, seems like this change got only merged for older machine types. So
> according to the libvirt doc (not setting it means hypervisor default), it
> is probably set to emulation?

Actually that patch didn't get merged at all afaict.

The support for l3-cache was introduced in QEMU 2.8.0, defaulting to
enabled. The code you see that disables it in older machine types dates
from this time, because we had to preserve ABI for machine tpyes < 2.8.0


So in practice today you'll be getting "emulate" mode already with any
non-ancient QEMU.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: l3 cache and cpu pinning

[Daniel P . Berrangé]

On Wed, Apr 21, 2021 at 12:09:42PM +0100, Daniel P. Berrangé wrote:
> On Wed, Apr 21, 2021 at 12:53:49PM +0200, Roman Mohr wrote:
> > Hi,
> > 
> > I have a question regarding enabling l3 cache emulation on Domains. Can
> > this also be enabled without cpu-pinning, or does it need cpu pinning to
> > emulate the l3 caches according to the cpus where the guest is pinned to?
> 
> I presume you're referring to
> 
>   
> 
>   
> 
> There is no hard restriction placed on usage of these modes by QEMU.
> 
> Conceptually though, you only want to use "passthrough" mode if you
> have configured the sockets/cores/threads topology to match the host
> CPUs. In turn you only ever want to set sockets/cores/threads to
> match the host if you have done CPU pinning such that the topology
> actually matches the host CPUs that have been pinned to.
> 
> As a rule of thumb
> 
>  - If letting CPUs float
>  
>  -> Always uses sockets=1, cores=num-vCPUs, threads=1
>  -> cache==emulate
>  -> Always use 1 guest NUMA node (ie the default)
> 
> 
>  - If strictly pinning CPUs 1:1
> 
>  -> Use sockets=N, cores=M, threads=0 to match the topology
> of the CPUs that have been pinned to

Opps, I meant  threads=P there, not 0 - ie match host threads.

With recentish libvirt+QEMU there is also a "dies=NNN" parameter for
topology which may be relevant for some host CPUs (very recent Intel
ones)

>  -> cache==passthrough
>  -> Configure virtual NUMA nodes if the CPU pinning or guest
> RAM needs cross host NUMA nodes.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: l3 cache and cpu pinning

[Daniel P . Berrangé]

On Wed, Apr 21, 2021 at 12:53:49PM +0200, Roman Mohr wrote:
> Hi,
> 
> I have a question regarding enabling l3 cache emulation on Domains. Can
> this also be enabled without cpu-pinning, or does it need cpu pinning to
> emulate the l3 caches according to the cpus where the guest is pinned to?

I presume you're referring to

  

  

There is no hard restriction placed on usage of these modes by QEMU.

Conceptually though, you only want to use "passthrough" mode if you
have configured the sockets/cores/threads topology to match the host
CPUs. In turn you only ever want to set sockets/cores/threads to
match the host if you have done CPU pinning such that the topology
actually matches the host CPUs that have been pinned to.

As a rule of thumb

 - If letting CPUs float
 
 -> Always uses sockets=1, cores=num-vCPUs, threads=1
 -> cache==emulate
 -> Always use 1 guest NUMA node (ie the default)


 - If strictly pinning CPUs 1:1

 -> Use sockets=N, cores=M, threads=0 to match the topology
of the CPUs that have been pinned to
 -> cache==passthrough
 -> Configure virtual NUMA nodes if the CPU pinning or guest
RAM needs cross host NUMA nodes.



Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Live migration with change to setmaxmem

[Daniel P . Berrangé]

On Wed, Apr 07, 2021 at 07:45:47AM -0700, Dariusz Smigiel wrote:
> Hi Team,
> 
> I need to migrate beefy instance between two different hosts which is
> running out of resources.
> 
> I was wondering, if there is a way to change maximum memory value during
> live migration, without shutting down the instance.

The maximum memory value controls the RAM mapping exposed to the guest,
and as such is part of the guest ABI.   Guest ABI is never permitted
to change during a live migration, as that will almost always have
seriously bad effects on the guest OS, and possibly the QEMU migration
data stream format.

IOW, no, you can't change max memory during migration, only a cold
boot.

Libvirt will enforces that preservation of ABI during migration by
validating the 2 XML configs for the src+dst host, and only permits
changes to the backend host OS integration, never frontend.


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: virsh capabilities does not include qemu

[Daniel P . Berrangé]

On Mon, Mar 22, 2021 at 01:57:24PM +0100, Michal Privoznik wrote:
> On 3/22/21 5:17 AM, shafnamol N wrote:
> > Hi,
> > I am new  to Libvirt and Qemu.I have installed Libvirt 7.1.0 and
> > qemu-kvm 4.2.0.
> > I configured and built libvirt based on instructions from
> > https://libvirt.org/compiling.html .
> > But when i tried to create a VM using virsh it shows the following error:
> > # virsh create /home/abc.xml
> > error: Failed to create domain from /home/abc.xml
> > error: invalid argument: could not find capabilities for arch=x86_64
> > domaintype=kvm
> > 
> > When i check the hypervisor capabilities ,it doest show qemu in guest
> > domain type.
> > # virsh capabilities
> > 
> > .
> >   
> > 
> >    
> >      exe
> >      
> >        64
> >        /usr/local/libexec/libvirt_lxc
> >        
> >      
> >    
> > 
> >    
> >      exe
> >      
> >        32
> >        /usr/local/libexec/libvirt_lxc
> 
> This smells suspicious. Perhaps you did not pass proper prefix (meson
> -Dsystem=true)?
> 
> Anyway, libvirt tries to find qemu-system-$arch in $PATH and falls back to
> /usr/libexec/qemu-kvm if no binary was found. So I suspect that maybe the
> directory you installed QEMU into is not in $PATH?

This capabiliities output is showing an lxc:///system driver connection
too, not QEMU. So I suspect libvirt was perhaps built without QEMU
support ?


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Question about migrate the vm with host-passthrough cpu

[Daniel P . Berrangé]

On Wed, Feb 24, 2021 at 09:42:37PM +0800, Guoyi Tu wrote:
> Hi there,
> 
> Sorry to bother you, but could I ask you a question about the live migration
> of virtual machine configured with host-passthough cpu, it confuse me a lot.
> 
> Here i have two hosts with different cpu, and the cpu feature set of old
> host is subset of the new one (by virsh cpu-compare). If the vm was first
> started on the old host,is it safe to migrate the vm between the two hosts
> back and forth with the vm always keep running?

The CPUID features are the biggest compatibility issue that is likely
to cause problems generally, but I believe there can be others.

For example if performance counters are exposed to the guest, these
are likely to vary between different CPU models. Even different BIOS
settings can change performance counters exposed by a CPU (HT vs no-HT)

There might also be problems with CPU TSC frequency differences between
the hosts.

> I've test the case in my environment, and the migration succeeds, the cpu
> family/model/stepping and features of lscpu in Guest OS is the same.

Right, the issue is that the migration appears to succeed from QEMU's
POV, but the guest may none the less see incompatibilities which
result in a crash or incorrect behaviour an arbitrary amount of time
later. It is really hard to guarantee that behaviour is going to be
correct in all scenarios.

Personally I would consider the use of host-passthrough to be unsupportable
in production environment, unless the CPUs are all homogeneous.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Some confusion about lsilogic controller

[Daniel P . Berrangé]

On Wed, Feb 24, 2021 at 09:11:03AM +, xingchaochao wrote:
> 
> Hello,
> I have been confused by such a phenomenon recently.
> Libvirt is the master branch , and the VM is centos8.2(kernel is 
> 4.18.0-193.el8.aarch64).
> When I hot-plug the scsi disk for a virtual machine without a virtio-scsi 
> controller, libvirt will automatically generate an lsilogic controller for 
> the scsi disk.

snip

> I am confused, why libvirt chooses to generate an lsilogic controller
> for the scsi disk when there is no scsi controller, instead of directly
> reporting an error and exiting the hot plug operation. After all, the
> scsi disk based on the lsilogic controller is not perceived inside
> the virtual machine, and lsilogic will remain in the XML file of the
> virtual machine.

libvirt has no knowledge of what devices the guest OS is able to
support. In this case RHEL8 doesn't support lsilogic, but libvirt
doesn't know this.

Generally a libvirt mgmt application will not rely on the defaults
for choosing devices, and instead use libosinfo to figure out
what devices are likely to be supported and explicitly use those.


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [ovirt-devel] Issue: Device path changed after adding disks to guest VM

[Daniel P . Berrangé]

On Wed, Jan 06, 2021 at 04:36:46PM +0100, Peter Krempa wrote:
> On Wed, Jan 06, 2021 at 17:16:24 +0200, Nir Soffer wrote:
> > On Wed, Dec 2, 2020 at 4:57 PM Joy Li  wrote:
> 
> [...]
> 
> > Comparing to state before reboot:
> > 
> > # virsh -r domblklist disk-mapping
> >  Target   Source
> > ---
> >  sdc  -
> >  sda  
> > /rhev/data-center/mnt/blockSD/84dc4e3c-00fd-4263-84e8-fc2466e9/images/40018b33-2b11-4d10-82e4-604a5b135fb2/40f455c4-8c92-4f8f-91c2-991b0ddfc2f5
> >  vda  /dev/mapper/3600140594af345ed76d42058f2b1a454
> >  vdb  /dev/mapper/360014050058f2f8a0474dc7a8a7cc6a5
> >  vdc  /dev/mapper/36001405b4d0c0b7544d47438b21296ef
> > 
> > # ls -lh /dev/disk/by-id/virtio-*
> > lrwxrwxrwx. 1 root root 9 Jan  6 09:42
> > /dev/disk/by-id/virtio-b97e68b2-87ea-45ca-9 -> ../../vda
> > lrwxrwxrwx. 1 root root 9 Jan  6 09:42
> > /dev/disk/by-id/virtio-d9a29187-f492-4a0d-a -> ../../vdb
> > lrwxrwxrwx. 1 root root 9 Jan  6 09:51
> > /dev/disk/by-id/virtio-e801c2e4-dc2e-4c53-b -> ../../vdc
> > 
> > In the guest disks are mapped to the same device name.
> > 
> > It looks like libivrt domblklist is not correct - vdb and vdc are switched.
> > Peter, this expected?
> 
> The names in 'virsh domblklist' are unfortunately and confusingly chosen
> to match the expected /dev/ device node name, but it's at kernel's
> discretion to name /dev/ nodes.
> 
> This means that it's not guaranteed that what you see in 'virsh
> domblklist' will match the state in the guest.

Essentially the only thing the disk device name is used for is
sorting the  elements within the XML document. This in
turn affects what order PCI addresses (virtio-blk) or SCSI
LUNS (virtio-scsi) are assigned in. This influences/hints as
to what order the guest OS *might* assign device names in.

The device name from the XML is not exposed to the guest
directly though.

Certainly when hotplugging/unplugging is involved all bets are
off wrt what disk names you'll see in the guest vs the XML. Dont
expect them to match except by luck.


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Migration via qemu+ssh using a given private ssh key possible ?

[Daniel P . Berrangé]

On Tue, Jan 05, 2021 at 07:39:49PM +0100, Oliver Dzombic wrote:
> Hi Peter,
> 
> thank you very much for this hint. Seems to work! :)
> 
> But i also tried another switch: no_verify=1
> 
> The whole call:
> 
> virsh -K0 -k0 migrate --copy-storage-inc --verbose --persistent --live
> testInstance
> qemu+ssh://testnode4:22/system?keyfile=/tmp/key-5ff4b02ca966c?no_verify=1?no_tty=1
> 
> So as you can see
> 
> no_verify=1
> no_tty=1
> 
> has been added.

You have incorrect URL syntax here. You need to use "&" to separate
parameters, not "?". The "?" is only valid to separate the path from
the start of the parameter list.

qemu+ssh://testnode4:22/system?keyfile=/tmp/key-5ff4b02ca966c_verify=1_tty=1


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: relation between "virbr0" / "virbr0-nic" network interfaces

[Daniel P . Berrangé]

On Fri, Jan 01, 2021 at 10:24:41AM +0100, vrms wrote:
> I am trying to understand KVM networking a little better and have noted
> that the "virbr0" network interface (the default KVM bridge) comes with
> another device named "virbr0-nic".
> The same kind of pair comes with each new bridge you may create via the
> "virtual machine manager" (and assumingly other KVM tools alike)
> Like "virbr1" comes in a pair with "virbr1-nic" (the *-nic interface is
> being created automatically (and I assume will disapear automatically if
> you remove the parent interface)
> 
> Can anybody kindly explain how these pairs are related to each other
> and/or work together?

The "$FOO-nic"  device is a tap device created as a hack to force a
stable MAC address on the main bridge device due to bad kernel impl
of bridge device MAC address assignment.

We stopped creating the "$FOO-nic" device in libvirt 6.8.0 since the
kernel was long ago fixed.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: live migration with authentication - how?

[Daniel P . Berrangé]

On Tue, Dec 15, 2020 at 12:38:39PM +, lejeczek wrote:
> Hi guys.
> 
> I'm trying I assumed something very crude, logic suggested I should try it
> first:
> 
> $ virsh migrate reactos
> qemu+ssh://dzien/system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf
> Please enter your authentication name: vdsm@ovirt
> Please enter your password:
> 
> As you can see I still get asked for credentials.
> Is it possible to live migrate that(or some) way, without being asked for
> those?

You are using an unqualified hostname, and I'd guess your authfile contains
a fully qualified hostname. We don't try to do hostname/domain expansion
when doing authfile looks, so you need an exact match.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Libvirt driver iothread property for virtio-scsi disks

[Daniel P . Berrangé]

On Wed, Nov 04, 2020 at 05:48:40PM +0200, Nir Soffer wrote:
> The docs[1] say:
> 
> - The optional iothread attribute assigns the disk to an IOThread as defined 
> by
>   the range for the domain iothreads value. Multiple disks may be assigned to
>   the same IOThread and are numbered from 1 to the domain iothreads value.
>   Available for a disk device target configured to use "virtio" bus and "pci"
>   or "ccw" address types. Since 1.2.8 (QEMU 2.1)
> 
> Does it mean that virtio-scsi disks do not use iothreads?
> 
> I'm experiencing a horrible performance using nested vms (up to 2 levels of
> nesting) when accessing NFS storage running on one of the VMs. The NFS
> server is using scsi disk.

When you say  2 levels of nesting do you definitely have KVM enabled at
all levels, or are you ending up using TCG emulation, because the latter
would certainly explain terrible performance.

> 
> My theory is:
> - Writing to NFS server is very slow (too much nesting, slow disk)
> - Not using iothreads (because we don't use virtio?)
> - Guest CPU is blocked by slow I/O

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Fail to build libvirt upstream rpm on fedora 33

[Daniel P . Berrangé]

On Fri, Oct 30, 2020 at 10:30:00AM +0800, Han Han wrote:
> Hello,
> I encountered errors when building rpm on fedora33:
> 
> Versions:
> libvirt: v6.9.0-rc2
> gcc-10.2.1-5.fc33.x86_64
> meson-0.55.3-1.fc33.noarch
> ninja-build-1.10.1-2.fc33.x86_64
> binutils-2.35-11.fc33.x86_64
> 
> Steps:
> 1. Archive libvirt source code to ~/rpmbuild/SOURCES/libvirt-6.9.0.tar.xz
> 2. Copy libvirt.spec.in to ~/rpmbuild/SPECS/libvirt.spec. Then `sed -i
> s/@VERSION@/6.9.0/g ~/rpmbuild/SPECS/libvirt.spec`
> 3. Build rpm:
> $ rpmbuild -ba ~/rpmbuild/SPECS/libvirt.spec


> FAILED: tests/cputest
> gcc  -o tests/cputest src/libvirt_probes.o tests/cputest.p/cputest.c.o
> -Wl,--as-needed -Wl,--no-undefined -Wl,-export-dynamic -pie
> -Wl,--whole-archive -Wl,--start-group tests/libtest_utils.a
> tests/libtest_utils_qemu.a -Wl,--no-whole-archive -O2 -flto=auto
> -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall
> -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
> -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong
> -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
> -fasynchronous-unwind-tables -fstack-clash-protection -Wl,-z,relro
> -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld
> src/libvirt.so.0.6009.0 tests/libtest_utils_qemu_monitor.a
> tests/libtest_qemu_driver.so -Wl,--no-copy-dt-needed-entries
> -Wl,-export-dynamic -ldl /usr/lib64/libglib-2.0.so /usr/lib64/
> libgobject-2.0.so /usr/lib64/libgio-2.0.so /usr/lib64/libgnutls.so
> /usr/lib64/libnl-3.so /usr/lib64/libnl-route-3.so /usr/lib64/libxml2.so
> /usr/lib64/libsasl2.so -lselinux /usr/lib64/libtirpc.so
> /usr/lib64/libyajl.so -Wl,-export-dynamic -lselinux -Wl,-export-dynamic
> -lselinux -Wl,-export-dynamic -lselinux -Wl,--end-group
> '-Wl,-rpath,$ORIGIN/../src:$ORIGIN/'
> -Wl,-rpath-link,/home/hhan/rpmbuild/BUILD/libvirt-6.9.0/x86_64-redhat-linux-gnu/src
> -Wl,-rpath-link,/home/hhan/rpmbuild/BUILD/libvirt-6.9.0/x86_64-redhat-linux-gnu/tests
> /usr/bin/ld: tests/libtest_qemu_driver.so: undefined reference to
> `__open_missing_mode'
> collect2: error: ld returned 1 exit status
> ninja: build stopped: subcommand failed.
> error: Bad exit status from /var/tmp/rpm-tmp.xEqkEF (%build)

Hmm, same binutils problem we're seeing on F34


   https://bugzilla.redhat.com/show_bug.cgi?id=1889763

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Encrypting boot partition Libvirt not showing the OS booting up

[Daniel P . Berrangé]

On Tue, Oct 13, 2020 at 10:40:11AM +0200, john doe wrote:
> On 10/13/2020 8:50 AM, john doe wrote:
> > On 10/13/2020 3:30 AM, Laine Stump wrote:
> > > On 10/12/20 1:10 PM, john doe wrote:
> > > > On 10/12/2020 5:14 PM, Michal Privoznik wrote:
> > > > > On 10/12/20 4:27 PM, john doe wrote:
> > > > > > On 10/12/2020 4:09 PM, Peter Krempa wrote:
> > > > > > > On Mon, Oct 12, 2020 at 16:05:43 +0200, Michal Privoznik wrote:
> > > > > > > > On 10/12/20 2:14 PM, john doe wrote:
> > > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > I sent privately the requested xml file to 'Peter Krempa
> > > > > > > > > '.
> > > > > > > > > Peter Krempa 's privately answered me back suggesting to add 
> > > > > > > > > the
> > > > > > > > > following in the domain xml file:
> > > > > > > > 
> > > > > > > > Solving things privately doesn't help the community.
> > > > > > > 
> > > > > > > Additionally it doesn't help solving the problem, since it's now
> > > > > > > opaque
> > > > > > > to others what the problem might be.
> > > > > > > 
> > > > > > > > > 
> > > > > > > > >  under 
> > > > > > > 
> > > > > > > I've suggested this as the outputs I've got privately hinted that 
> > > > > > > the
> > > > > > > console (as in virsh console) didn't get to asking for the 
> > > > > > > password,
> > > > > > > while the manually-started-qemu did.
> > > > > > > 
> > > > > > > Thus the problem actually doesn't have to do with encryption or
> > > > > > > wahatver, but the console doesn't plainly work.
> > > > > > > 
> > > > > > > > > 
> > > > > > > > > such as ...
> > > > > > > > > 
> > > > > > > > >      
> > > > > > > > >        hvm
> > > > > > > > >        
> > > > > > > > >        
> > > > > > > > >      
> > > > > > > > > 
> > > > > > > > 
> > > > > > > > Try adding:
> > > > > > > > 
> > > > > > > >  /usr/share/seabios/bios.bin
> > > > > 
> > > > > Darn, this should have been sgabios: /usr/share/sgabios/sgabios.bin
> > > > > but if your seabios is new enough (v1.11.0 and newer) then this is not
> > > > > needed as seabios itself is capable of serial interface. And looking 
> > > > > at
> > > > > earlier e-mails in the thread you have v1.12.0-1 you you're good and
> > > > > don't need to add  at all.
> > > > > 
> > > > > But honestly, I don't know why you are not getting the console.
> > > > > Could it
> > > > > be that you are getting the console and the qemu is waiting for your
> > > > > input, i.e. what happens if you type in the password?
> > > > > 
> > > > 
> > > > Nothing happened at all if I try to type the password.
> > > > Yes, so am I , I'm totaly lost on why it does not work.
> > > > 
> > > > How can I find the command libvirt is passing to qemu?
> > > 
> > > The qemu command issued by libvirt can be found at the end of
> > > /etc/libvirt/qemu/${guestname}.log
> > > 
> > 
> > Thank you, I have now isolated the command generated by libvirt.
> > Starting this command from a script, a vnc server is started.
> > 
> > Is libvirt internally using vnc connection?
> > 
> 
> It looks like the issue is that the libvirt command pass to qemu is
> using '-display none' where it should be '-nographic'.

No, using "-display none" is correct to disable any graphical display
in QEMU.

-nographic is an old discouraged syntax which has many side effects,
beyond disabling graphical display it mess with serial port config.


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Is it possible that "virsh destroy" does not stop a domain ?

[Daniel P . Berrangé]

On Thu, Oct 08, 2020 at 06:25:32PM +0200, Lentes, Bernd wrote:
> 
> 
> - On Oct 7, 2020, at 7:26 PM, Peter Crowther peter.crowt...@melandra.com 
> wrote:
> 
> > Bernd, another option would be a mismatch between the message that "virsh
> > destroy" issues and the message that force_stop() in the pacemaker agent
> > expects to receive. Pacemaker is trying to determine the success or failure 
> > of
> > the destroy based on the concatenation of the text of the exit code and the
> > text output by virsh; if either of those have changed between virsh 
> > versions,
> > and especially if virsh destroy ever exits with a status other than zero, 
> > then
> > you'll get that OCF error.
> 
> > Do you know what $VIRSH_OPTIONS ends up as in your Pacemaker config,
> > particularly whether --graceful is specified?
> 
> > Cheers,
> 
> > - Peter
> 
> that means in the end that with "virsh destroy" i can't be 100% sure
> that a domain is stopped.

Assuming you do *NOT* use the --graceful flag, then libvirt will end up
sending SIGKILL to QEMU if SIGTERM didn't cause it to quit.

It is possible that QEMU will not die immediately even with SIGKILL, but
you should get an error code back from virsh destroy in this scenario
at least.

On highly overcommitted hosts, the kernel may not reap the QEMU process
quickly enough, but libvirt will definitely have delivered SIGKILL by
the time the command returns.

The only reasons why SIGKILL won't work eventually is if the process is
stuck in an uninterruptable sleep in kernel space. This is typically
seen for example, when the VM is doing I/O to a disk on NFS, and the NFS
server is dead, and the NFS mount is set with  "hard,nointr".

There's nothing any app can do this in case really. If the host has a dead
NFS mount you really need to be fencing the entire host.


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: what does "virsh destroy" really ?

[Daniel P . Berrangé]

On Tue, Oct 06, 2020 at 09:15:57AM +0200, Martin Kletzander wrote:
> On Mon, Oct 05, 2020 at 07:12:34PM -0400, Digimer wrote:
> > On 2020-10-05 6:04 p.m., Lentes, Bernd wrote:
> > > Hi,
> > > 
> > > what does "virsh destroy" with the domain ? Send a kill -9 to the process 
> > > ?
> > > 
> > > Bernd
> > > 
> > 
> > It forces the guest off, like pulling the power on a hardware machine.
> > Not sure of the exact mechanism behind the scenes. It does leave the
> > server defined and you can reboot it again later (albeit like restoring
> > power to a normal machine, so it might need to replay journals, etc).
> > 
> 
> The technical details should not matter, but it tries to send "quit" to QEMU 
> and
> falls back to killing it IIRC (firsh with SIGTERM and then with SIGKILL).  
> Don't
> do that if you want to use the machine again ;-)

You don't need to be so scared of virsh destroy. It is simply akin to pulling
out the power plug, and modern OS with a decent journaling filesystem will
recover from that quite reasonably. Sure you'll loose state of what's running,
but 99% of the time your guest will boot up again just fine.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Help on Meson build Error

[Daniel P . Berrangé]

On Fri, Sep 25, 2020 at 05:31:09PM +0800, Wei Wang wrote:
> On Fri, Sep 25, 2020 at 5:24 PM Andrea Bolognani 
> wrote:
> 
> > On Fri, 2020-09-25 at 10:16 +0100, Daniel P. Berrangé wrote:
> > > On Fri, Sep 25, 2020 at 05:03:24PM +0800, Wei Wang wrote:
> > > > /usr/local/bin/rst2html5 --stylesheet= --strict docs/manpages/virsh.rst
> > > > docs/manpages/virsh.rst:41: (ERROR/3) Error in "code-block" directive:
> > > > 1 argument(s) required, 0 supplied.
> > > >
> > > > .. code-block::
> > > >
> > > >virsh [OPTION]...   [ARG]...
> > >
> > > This is a sign of having a bad version of "rst2html5" installed.
> > >
> > > We expect to be using  rst2html5 that comes with python docutils
> > > package.
> >
> > Yeah, the python3-docutils Debian package will contain a working
> > rst2html5 binary.
> >
> >
> That's strange, my rst2html5 comes from docutils:
> python3-docutils: /usr/share/docutils/scripts/python3/rst2html5
> 
> and
> /usr/local/bin/rst2html5 --version
> rst2html5 (Docutils 0.16 [release], Python 3.8.2, on linux)

I wouldn't expect this to have been in /usr/local/bin - that is usually
a sign of a local adminstrator install, via "pip" or some other means.
The distro provided version would be in /usr/bin

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Help on Meson build Error

[Daniel P . Berrangé]

On Fri, Sep 25, 2020 at 11:23:57AM +0200, Andrea Bolognani wrote:
> On Fri, 2020-09-25 at 10:16 +0100, Daniel P. Berrangé wrote:
> > On Fri, Sep 25, 2020 at 05:03:24PM +0800, Wei Wang wrote:
> > > /usr/local/bin/rst2html5 --stylesheet= --strict docs/manpages/virsh.rst
> > > docs/manpages/virsh.rst:41: (ERROR/3) Error in "code-block" directive:
> > > 1 argument(s) required, 0 supplied.
> > > 
> > > .. code-block::
> > > 
> > >virsh [OPTION]...   [ARG]...
> > 
> > This is a sign of having a bad version of "rst2html5" installed.
> > 
> > We expect to be using  rst2html5 that comes with python docutils
> > package.
> 
> Yeah, the python3-docutils Debian package will contain a working
> rst2html5 binary.
> 
> > There is a completely different rst2html5 that people might get
> > from "pip" if you did "pip install rst2html5". This version is
> > broken and unusable and so needs to be removed.
> 
> Can we do something to get that package off PyPi? It's not the first
> time the overlap has caused trouble for our users, and it certainly
> won't be the last...

Well it is not really our place to demand its removal. It isn't violating
any rules of PyPi and AFAICT, it existed before docutils provided the
same binary name.

The only good course of action is probably to submit a patch to actually
make it work.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Help on Meson build Error

[Daniel P . Berrangé]

On Fri, Sep 25, 2020 at 10:40:34AM +0200, Andrea Bolognani wrote:
> On Fri, 2020-09-25 at 16:21 +0800, Wei Wang wrote:
> > On Fri, Sep 25, 2020 at 3:56 PM Andrea Bolognani  
> > wrote:
> > > On Fri, 2020-09-25 at 10:33 +0800, Wei Wang wrote:
> > > > In file included from ../src/util/virfile.c:42:
> > > > /usr/include/libutil.h:43:10: fatal error: sys/_types.h: No such file 
> > > > or directory
> > > >43 | #include 
> > > >   |  ^~
> > > > compilation terminated.
> > > > [51/908] Compiling C object 
> > > > src/qemu/libvirt_driver_qemu_impl.a.p/qemu_driver.c.o
> > > > ninja: build stopped: subcommand failed
> > > >  
> > > > I have  /usr/include/sys/types.h, but no "_types.h"
> > > 
> > > Are you building on FreeBSD? Because that's where we expect
> > >  to be included. If you're not on FreeBSD, can you check
> > > which package /usr/include/libutil.h belongs to?
> >  
> > No, I'm building on Linux. It belongs to libutil-freebsd-dev, not sure why 
> > that's needed in my build.
> 
> Can you retry after uninstalling that package? It looks like it
> messes up our header detection, and I sort of doubt we care enough
> about this scenario to change our build system to cope with it.

Yes, we do need to fix our build system. The user can have valid reasons
for having such a package installed, and we shouldn't require them to
remove it.


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Help on Meson build Error

[Daniel P . Berrangé]

On Fri, Sep 25, 2020 at 05:03:24PM +0800, Wei Wang wrote:
> On Fri, Sep 25, 2020 at 4:54 PM Daniel P. Berrangé 
> wrote:
> 
> >
> > > Yes, I tried, but it fails with
> > > ../src/util/virfile.c:42:11: fatal error: libutil.h: No such file or
> > > directory
> >
> > You'd have to delete the build dir and re-run meson so that it
> > auto-detects again.
> >
> >
> OK, that issue seems gone, but ninja reports lots of new errors:
> 
> 
> ninja: Entering directory `build'
> [947/1250] Generating virsh.html.in with a meson_exe.py custom command
> FAILED: docs/manpages/virsh.html.in
> /usr/bin/meson --internal exe --capture docs/manpages/virsh.html.in --
> /usr/local/bin/rst2html5 --stylesheet= --strict docs/manpages/virsh.rst
> docs/manpages/virsh.rst:41: (ERROR/3) Error in "code-block" directive:
> 1 argument(s) required, 0 supplied.
> 
> .. code-block::
> 
>virsh [OPTION]...   [ARG]...

This is a sign of having a bad version of "rst2html5" installed.

We expect to be using  rst2html5 that comes with python docutils
package.

There is a completely different rst2html5 that people might get
from "pip" if you did "pip install rst2html5". This version is
broken and unusable and so needs to be removed.


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|