Re: Emulated TPM devices and snapshots of running VMs
Peter Krempa writes: > On Thu, Jul 09, 2020 at 17:54:23 +0200, Milan Zamazal wrote: >> Peter Krempa writes: >> > >> > On Thu, Jul 09, 2020 at 14:14:32 +0200, Milan Zamazal wrote: >> >> Milan Zamazal writes: >> >> >> > >> >> > Hi, >> >> > >> >> > I would like to clarify how to make snapshots of running VMs with >> >> > emulated TPM devices. As far as I understand QEMU documentation, it's >> >> > possible to make snapshots of running VMs with TPM, but it's important >> >> > to retain the state of swtpm. Does libvirt assist with that in any way >> >> > or is it completely user's responsibility? libvirt pauses the VM >> >> > internally when making a snapshot, which should be the right moment to >> >> > copy the swtpm data, but the user doesn't have control over it. Is >> >> > there a way to make a copy of swtpm data that is guaranteed to be >> >> > consistent with the snapshot? >> >> >> >> No idea? >> > >> > I can comment only on the fact that libvirt doesn't do anything >> > regarding snapshots on a VM with TPM. >> >> Thank you for the confirmation. >> >> Can anybody confirm there is no way to perform custom actions while a VM >> is frozen by libvirt when making a memory snapshot, before we start >> thinking about workarounds and/or filing a RFE? > > No, currently we don't support any custom actions at the point when the > external memory snapshot is finalized prior to continuing the VM. > > Please file a generic RFE for snapshoting including TPM rather than a > partial one where you'll request a way to do your hack. OK, thanks, done: https://bugzilla.redhat.com/1855367
Re: Emulated TPM devices and snapshots of running VMs
On Thu, Jul 09, 2020 at 17:54:23 +0200, Milan Zamazal wrote: > Peter Krempa writes: > > > On Thu, Jul 09, 2020 at 14:14:32 +0200, Milan Zamazal wrote: > >> Milan Zamazal writes: > >> > > > >> > Hi, > >> > > >> > I would like to clarify how to make snapshots of running VMs with > >> > emulated TPM devices. As far as I understand QEMU documentation, it's > >> > possible to make snapshots of running VMs with TPM, but it's important > >> > to retain the state of swtpm. Does libvirt assist with that in any way > >> > or is it completely user's responsibility? libvirt pauses the VM > >> > internally when making a snapshot, which should be the right moment to > >> > copy the swtpm data, but the user doesn't have control over it. Is > >> > there a way to make a copy of swtpm data that is guaranteed to be > >> > consistent with the snapshot? > >> > >> No idea? > > > > I can comment only on the fact that libvirt doesn't do anything > > regarding snapshots on a VM with TPM. > > Thank you for the confirmation. > > Can anybody confirm there is no way to perform custom actions while a VM > is frozen by libvirt when making a memory snapshot, before we start > thinking about workarounds and/or filing a RFE? No, currently we don't support any custom actions at the point when the external memory snapshot is finalized prior to continuing the VM. Please file a generic RFE for snapshoting including TPM rather than a partial one where you'll request a way to do your hack.
Re: Emulated TPM devices and snapshots of running VMs
Peter Krempa writes: > On Thu, Jul 09, 2020 at 14:14:32 +0200, Milan Zamazal wrote: >> Milan Zamazal writes: >> > >> > Hi, >> > >> > I would like to clarify how to make snapshots of running VMs with >> > emulated TPM devices. As far as I understand QEMU documentation, it's >> > possible to make snapshots of running VMs with TPM, but it's important >> > to retain the state of swtpm. Does libvirt assist with that in any way >> > or is it completely user's responsibility? libvirt pauses the VM >> > internally when making a snapshot, which should be the right moment to >> > copy the swtpm data, but the user doesn't have control over it. Is >> > there a way to make a copy of swtpm data that is guaranteed to be >> > consistent with the snapshot? >> >> No idea? > > I can comment only on the fact that libvirt doesn't do anything > regarding snapshots on a VM with TPM. Thank you for the confirmation. Can anybody confirm there is no way to perform custom actions while a VM is frozen by libvirt when making a memory snapshot, before we start thinking about workarounds and/or filing a RFE? Thanks, Milan
Re: Could you please help with questions about the net failover feature
On 7/8/20 12:53 PM, Laine Stump wrote: On 7/8/20 10:02 AM, Ken Cox wrote: On 7/8/20 1:30 AM, Stefan Assmann wrote: On 2020-07-06 10:01, Laine Stump wrote: On 7/6/20 5:10 AM, Yalan Zhang wrote: Hi Laine, For the feature testing before, I only test the linux bridge setting as in 2), it works. Now I tried 1), to use macvtap bridge mode connected to the PF, it can not work as the hostdev interface can not get dhcp ip address on the guest. Check on host, the /var/log/messages and dmesg both says: "Jul 6 04:54:45 dell-per730-xx kernel: ixgbe :82:00.1 enp130s0f1: 1 Spoofed packets detected .. Jul 6 04:56:17 dell-per730-xx kernel: ixgbe :82:00.1 enp130s0f1: 1 Spoofed packets detected Jul 6 04:56:54 dell-per730-xx kernel: ixgbe :82:00.1 enp130s0f1: 1 Spoofed packets detected " (enp130s0f1 is the PF's interface name, and :82:00.1 is the PF's pci address) # rpm -q kernel kernel-4.18.0-193.4.1.el8_2.x86_64 Could you please help to confirm if this is a kernel bug? Thank you very much! Interesting. I'm not sure if this is expected behavior, or if it's improper behavior and it just hasn't been tested before (obviously based on my earlier recommendation, I think it *should* be able to work like this, and I *thought* I had tried it, but maybe I just imagined it :-/). I'm Cc'ing Stefan Assmann to see if he has an opinion on whether or not this should work. For his convenience, here is a summary of the config: The setup is that there is a bridge-mode macvtap interface on the PF, and one of the VF's has been given the same MAC address as the macvtap. the macvtap interface is connected to an emulated NIC in the guest, and the VF is assigned to the guest with VFIO. IIUC, the problem is using the same mac address for macvtap and for a VF. This is what's causing the spoofed packets. How is it a spoof? Because one interface detects a packet not originating from itself that has its own MAC address? Is this check done by the kernel, or by the firmware on the card? Both interfaces are specifically and consciously configured with the same MAC address (since that is a requirement for the simplified bonding provided by the virtio-net "failover" feature). If DHCP isn't working, then I guess the guest is sending a DHCP discover packet out through the VF. How is this packet triggering anti-spoof protection, since it is the legitimate MAC address of that interface? Or am I misinterpreting what's going on? (the log message just says a spoofed packet was detected, so it could be some other packet triggering the log, and this is all just a red herring...) The spoof checking is done by the firmware in the NIC. You might try to disable spoofchecking and see if that helps. I'm going to try and set this up to look into it further. Could you give more detailed instructions to set it up? I'll try this today on my setup, which uses I350 cards (igb driver). You have two choices for the backup virtio interface: 1) it can be a macvtap device connected to the PF of the same SRIOV device. 2) it can be a standard tap device connected to a Linux host bridge (created outside libvirt in the host system network config) that is attached to the PF (or alternately one of the VFs that isn't being used for VMs, or to another physical ethernet adapter on the host that is connected to the same network. --- Best Regards, Yalan Zhang IRC: yalzhang On Sun, Mar 22, 2020 at 6:50 AM Laine Stump mailto:la...@redhat.com>> wrote: On 3/21/20 1:08 AM, Yalan Zhang wrote: > In my understanding, the standby and primary hostdev interface may be in > different subnet. There is only one hostdev device in the team pair (that will be the one with since it needs to be unplugged during migration). The other device must be a virtio device (the one with ). And no, they cannot be on different subnets. They must both connect into the same ethernet "collision domain", such that the guest could assign the same IP address to either of them and be able to communicate on the network. There is some explanation of the use case for this option. and some example config, here: https://www.libvirt.org/formatdomain.html#elementsTeaming > I'm not sure whether it is correct. Could you please help to explain? > Thank you in advance. > > For example, primary hostdev is connected to vf-pool with , > while the standby is connected to NAT network with " forward dev='eth0'". > The standby interface will get ip as 192.168.122.x, but after NAT, it > will be in the same subnet of the vf. > > So after the VF is unplugged, the packet will still broadcast in the > same subnet, and the vm will get the packet as the standby share the > same mac. Right? No, not right :-) The VF of an SRIOV network adapter is connected
Re: Emulated TPM devices and snapshots of running VMs
On Thu, Jul 09, 2020 at 14:14:32 +0200, Milan Zamazal wrote: > Milan Zamazal writes: > > > Hi, > > > > I would like to clarify how to make snapshots of running VMs with > > emulated TPM devices. As far as I understand QEMU documentation, it's > > possible to make snapshots of running VMs with TPM, but it's important > > to retain the state of swtpm. Does libvirt assist with that in any way > > or is it completely user's responsibility? libvirt pauses the VM > > internally when making a snapshot, which should be the right moment to > > copy the swtpm data, but the user doesn't have control over it. Is > > there a way to make a copy of swtpm data that is guaranteed to be > > consistent with the snapshot? > > No idea? I can comment only on the fact that libvirt doesn't do anything regarding snapshots on a VM with TPM.
Re: Emulated TPM devices and snapshots of running VMs
Milan Zamazal writes: > Hi, > > I would like to clarify how to make snapshots of running VMs with > emulated TPM devices. As far as I understand QEMU documentation, it's > possible to make snapshots of running VMs with TPM, but it's important > to retain the state of swtpm. Does libvirt assist with that in any way > or is it completely user's responsibility? libvirt pauses the VM > internally when making a snapshot, which should be the right moment to > copy the swtpm data, but the user doesn't have control over it. Is > there a way to make a copy of swtpm data that is guaranteed to be > consistent with the snapshot? No idea? > Thank you, > Milan
Re: NVDIMM in devdax mode and SELinux (was: Two questions about NVDIMM devices)
On Thu, Jul 09, 2020 at 11:53:19AM +0200, Milan Zamazal wrote:
> Milan Zamazal writes:
>
> > Daniel P. Berrangé writes:
> >
> >> On Thu, Jul 02, 2020 at 01:21:15PM +0200, Milan Zamazal wrote:
> >>> The second problem is that a VM fails to start with a backing NVDIMM in
> >>> devdax mode due to SELinux preventing access to the /dev/dax* device (it
> >>> doesn't happen with any other NVDIMM modes). Who should be responsible
> >>> for handling the SELinux label appropriately in that case? libvirt, the
> >>> system administrator, anybody else? Using in NVDIMM's source
> >>> doesn't seem to be accepted by the domain XML schema.
> >>
> >> The expectation is that out of the box SELinux will "just work". So
> >> anything that is broken is a bug in either libvirt or selinux policy.
> >>
> >> There is no expectation/requirement to use unless you want
> >> to setup non-default behaviour which isn't the case here.
> >>
> >> IOW this sounds like a genuine bug.
> >
> > OK, I'll try to find out what and where is the problem exactly.
>
> The problem apparently is that /dev/dax* is a character device rather
> than a block device (such as /dev/pmem*), which is not expected by
> SELinux policy rules.
>
> This is an NVDIMM in fsdax mode:
>
> # ls -lZ /dev/pmem0
> brw-rw. 1 root disk system_u:object_r:device_t:s0 259, 0 Jul 9 11:39
> /dev/pmem0
>
> This is the same NVDIMM reconfigured as devdax:
>
> # ls -lZ /dev/dax0.0
> crw---. 1 root root system_u:object_r:device_t:s0 252, 5 Jul 9 11:43
> /dev/dax0.0
>
> (Unix permissions are different, but when I change them to `disk' group
> and 660, the same problem still occurs.)
>
> audit.log reports the following when starting a VM with an NVDIMM device
> in devdax mode:
>
> type=AVC msg=audit(1594144691.758:913): avc: denied { map } for
> pid=21659 comm="qemu-kvm" path="/dev/dax0.0" dev="tmpfs" ino=1521557
> scontext=system_u:system_r:svirt_t:s0:c216,c981
> tcontext=system_u:object_r:svirt_image_t:s0:c216,c981 tclass=chr_file
> permissive=0
> type=AVC msg=audit(1594144691.758:914): avc: denied { map } for
> pid=21659 comm="qemu-kvm" path="/dev/dax0.0" dev="tmpfs" ino=1521557
> scontext=system_u:system_r:svirt_t:s0:c216,c981
> tcontext=system_u:object_r:svirt_image_t:s0:c216,c981 tclass=chr_file
> permissive=0
>
> Indeed, svirt_t map access to svirt_image_t is allowed only for files
> and block devices:
>
> # sesearch -A -p map -s svirt_t -t svirt_image_t
> ...
> allow svirt_t svirt_image_t:blk_file map;
> allow svirt_t svirt_image_t:file map;
>
> What to do about it? Do I handle the NVDIMM in a wrong way or should
> sVirt policies be fixed?
File a BZ against the policy as this looks like a valid use case
Regards,
Daniel
--
|: https://berrange.com -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
NVDIMM in devdax mode and SELinux (was: Two questions about NVDIMM devices)
Milan Zamazal writes:
> Daniel P. Berrangé writes:
>
>> On Thu, Jul 02, 2020 at 01:21:15PM +0200, Milan Zamazal wrote:
>>> The second problem is that a VM fails to start with a backing NVDIMM in
>>> devdax mode due to SELinux preventing access to the /dev/dax* device (it
>>> doesn't happen with any other NVDIMM modes). Who should be responsible
>>> for handling the SELinux label appropriately in that case? libvirt, the
>>> system administrator, anybody else? Using in NVDIMM's source
>>> doesn't seem to be accepted by the domain XML schema.
>>
>> The expectation is that out of the box SELinux will "just work". So
>> anything that is broken is a bug in either libvirt or selinux policy.
>>
>> There is no expectation/requirement to use unless you want
>> to setup non-default behaviour which isn't the case here.
>>
>> IOW this sounds like a genuine bug.
>
> OK, I'll try to find out what and where is the problem exactly.
The problem apparently is that /dev/dax* is a character device rather
than a block device (such as /dev/pmem*), which is not expected by
SELinux policy rules.
This is an NVDIMM in fsdax mode:
# ls -lZ /dev/pmem0
brw-rw. 1 root disk system_u:object_r:device_t:s0 259, 0 Jul 9 11:39
/dev/pmem0
This is the same NVDIMM reconfigured as devdax:
# ls -lZ /dev/dax0.0
crw---. 1 root root system_u:object_r:device_t:s0 252, 5 Jul 9 11:43
/dev/dax0.0
(Unix permissions are different, but when I change them to `disk' group
and 660, the same problem still occurs.)
audit.log reports the following when starting a VM with an NVDIMM device
in devdax mode:
type=AVC msg=audit(1594144691.758:913): avc: denied { map } for pid=21659
comm="qemu-kvm" path="/dev/dax0.0" dev="tmpfs" ino=1521557
scontext=system_u:system_r:svirt_t:s0:c216,c981
tcontext=system_u:object_r:svirt_image_t:s0:c216,c981 tclass=chr_file
permissive=0
type=AVC msg=audit(1594144691.758:914): avc: denied { map } for pid=21659
comm="qemu-kvm" path="/dev/dax0.0" dev="tmpfs" ino=1521557
scontext=system_u:system_r:svirt_t:s0:c216,c981
tcontext=system_u:object_r:svirt_image_t:s0:c216,c981 tclass=chr_file
permissive=0
Indeed, svirt_t map access to svirt_image_t is allowed only for files
and block devices:
# sesearch -A -p map -s svirt_t -t svirt_image_t
...
allow svirt_t svirt_image_t:blk_file map;
allow svirt_t svirt_image_t:file map;
What to do about it? Do I handle the NVDIMM in a wrong way or should
sVirt policies be fixed?
Thanks,
Milan
