Cannot pass secret id for backing file after taking external snapshot on encrypted qcow2 file
Hi，
I used 'virsh snapshot-create' create an encrypted external snapshot, when I
try to use 'qemu-img check' top file, found no entrance to pass backing-file's
secret-id
1、Version
centos-release-8.2-2.2004.0.1.el8.x86_64
libvirt.x86_64 6.0.0-17.el8
qemu-kvm.x86_64 15:4.2.0-19.el82、Reproduce Steps
1)Create an encrypted qcow2
qemu-img create --object secret,id=sec0,data=123456 -f qcow2 -o
encrypt.format=luks,encrypt.key-secret=sec0 first.qcow2 1G
2)Create external snapshot with 'encrypted' xml
# cat snap.xml
# virsh dumpxml test-vm | awk '/
3)try to qemu-img check top qcow2 file
Note: The secid of the backing file is not recorded, so when I use qemu-img
check/etc.. how to pass the secret to qemu of backing files
# qemu-img info -U first-snapshot.qcow2
image: first-snapshot.qcow2
file format: qcow2
virtual size: 1 GiB (1073741824 bytes)
disk size: 544 KiB
encrypted: yes
cluster_size: 65536
backing file: /root/first.qcow2 ### backing file: json:{"encrypt.format":
"luks", "encrypt.key-secret": "secrete-id"}
backing file format: luks
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
encrypt:
ivgen alg: plain64
hash alg: sha256
cipher alg: aes-256
uuid: e4158089-26e4-433f-990e-1d1d0723feee
format: luks
cipher mode: xts
slots:
[0]:
active: true
iters: 1257888
key offset: 4096
stripes: 4000
[1]:
active: false
key offset: 262144
[2]:
active: false
key offset: 520192
[3]:
active: false
key offset: 778240
[4]:
active: false
key offset: 1036288
[5]:
active: false
key offset: 1294336
[6]:
active: false
key offset: 1552384
[7]:
active: false
key offset: 1810432
payload offset: 2068480
master key iters: 300073
corrupt: false
# qemu-img check -U --object
secret,id=sec_1,file=/etc/libvirt/secrets/f52a81b2-424e-490c-823d-6bd4235bc572.base64,format=base64
--image-opts
encrypt.format=luks,encrypt.key-secret=sec_1,file.filename=first-snapshot.qcow2
--object
secret,id=sec_2,file=/etc/libvirt/secrets/f981dd17-143f-45bc-88e6-.base64,format=base64
qemu-img: Could not open
'encrypt.format=luks,encrypt.key-secret=sec_1,file.filename=first-snapshot.qcow2':
Could not open backing file: Parameter 'key-secret' is required for cipher
yaohua...@zstack.io
Re: SELinux labels change in libvirt
On Tue, Jul 14, 2020 at 6:03 PM Daniel P. Berrangé
wrote:
> On Tue, Jul 14, 2020 at 04:02:17PM +0300, Ram Lavi wrote:
> > On Tue, Jul 14, 2020 at 3:33 PM Daniel P. Berrangé
> > wrote:
> >
> > > On Tue, Jul 14, 2020 at 03:21:17PM +0300, Ram Lavi wrote:
> > > > Hello all,
> > > >
> > > > tl;dr, can you point me to the point in the libvirt repo where it's
> > > trying
> > > > to change a tap-device's SELinux label?
> > > >
> > > > I am trying to create a tap device with libvirt on a
> > > > super-privileged container, and then use it on another,
> > > > unprivileged container with libvirt.
> > > > User wise, I know I need the super-privileged container to open the
> tap
> > > > device with the user of the unprivileged one - that I already did and
> > > it's
> > > > not the issue.
> > > > But I have a problem when I open the tap device in the
> > > > non-privileged container: the tap device currently has the spc_t
> label
> > > > since the tun_socket inherited the selinux context from the
> > > > super-privileged container who creates it. then libvirt is trying to
> > > change
> > > > the SELinux labels, and since it's not privileged then it fails.
> > > > But I didn't find where and how libvirt is trying to change the tap
> > > > device's label.
> > > >
> > > > Can you point me to that specific code on libvirt?
> > >
> > > If the SELinux policy that libvirtd is running under prevents it from
> > > re-labelling, then TAP devices label failure is just going to be one
> > > out of 100's of labelling failures.
> > >
> > IIUC normally libvirtd would use devices created by itself, so there
> > shouldn't be relabeling failures, right?
>
> Whether or not there are relabeling failures is determined by what
> SELinux policy libvirtd is running under.
>
> > >
> > > Either the SELinux policy needs to be changed to allow libvirtd to
> > > relabel stuff in the normal manner, or you will have to turn off
> > > SELinux support in libvirtd. in /etc/libvirt/qemu.conf via the
> > > param security_driver = "none". If you turn off SELinux in
> > > libvirt, then you no longer have separation of QEMU processes
> > > which may be a security flaw depending on your deplyoment
> > > scenario.
> > >
> > turning SELinux in libvirtd off or allowing libvirt to relabel are
> tempting
> > options but it is not an option I'm afraid due to security concerns.
> > Our plan (or at least this particular effort) is to try to relabel the
> > tun-socket after created in the super-privileged container to be the same
> > one as the one used in the unprivileged one, then it won't have an issue
> > consuming it.
> >
> > btw the change I (or rather Migule's my teammate) made is in this PR,
> where
> > I want to add a tap device in virt-handler (i.e. the super privileged
> > container) to be further uses in virt-launcher (i.e. the non-privileged
> > container): https://github.com/kubevirt/kubevirt/pull/3290
>
> In normal host OS deployment, libvirtd runs under virtd_t, and when
> it spawns QEMU, it will relabel files to svirt_image_t:s0:$MCS, and
> spawn QEMU as svirt_t:s0:$MCS.
>
> My understanding is what in kubevirt, things work differently. Docker
> (or podman), launch the container as container_t:s0:$MCS. libvirtd
> *and* QEMU thus both run as container_t:s0:$MCS. ie All the labelling
> is setup when the container is launched and libvirtd should not do
> anything.
>
> So I'm really not sure why you have libvirtd configured to do relabelling
> at all ? I'd be expecting it to have security_driver=none in the qemu.conf
> file so that libvirtd doesn't do anything.
>
I checked the dumpxml of the virt-launcher pod (that runs the qemu in
kubevirt) - it has dynamic policy.
+107:+107
+107:+107
Are you saying this a wrong configuration for a kubevirt vmi?
>
> If libvirtd is doing relabelling, I'm not sure how it works for anything,
> let alone tap devices.
>
I want to emphasize that in this current situation the reason we fail is
that we are trying to give libvirt in a non-privileged pod (i.e.
virt-launcher) a tap device that was created externally (by a privileged
pod - virt-handler).
I think the reason it works so far is that libvirt was using only devices
which were self created so there was no issue to relabel.
Our thought was that if we know how libvirt is relabeling then we could
also do it so that the externally created tap's label will match the
virt-launcher's.
Is this were libvirt does the relabeling
https://github.com/libvirt/libvirt/blob/e71e13488dc1aa65456e54a4b41bc925821b4263/src/security/security_selinux.c#L1256
?
btw the error we get is (from audit)
type=AVC msg=audit(1586956552.265:513): avc: denied { relabelfrom }
for pid=27423 comm="libvirtd"
scontext=system_u:system_r:container_t:s0:c143,c582
tcontext=system_u:system_r:spc_t:s0 tclass=tun_socket permissive=0
> Regards,
> Daniel
> --
> |: https://berrange.com -o-
> https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o-
>
Re: SELinux labels change in libvirt
On Thu, Jul 16, 2020 at 11:22:26AM +0300, Ram Lavi wrote:
> On Tue, Jul 14, 2020 at 6:03 PM Daniel P. Berrangé
> wrote:
>
> I checked the dumpxml of the virt-launcher pod (that runs the qemu in
> kubevirt) - it has dynamic policy.
>
>
> +107:+107
> +107:+107
>
>
> Are you saying this a wrong configuration for a kubevirt vmi?
No, this is exactly what I'm expecting and matches the behaviour I
mentioned in my previous mail, where SELinux is not enabled in libvirtd.
This setting is completely unrelated to SELinux. This seclabel is for the
"dac" model, aka Discretionary Access Control, aka plain old POSIX users
and groups. It is used when libvirtd runs as root, and then launches QEMU
as a separate "qemu:qemu" user:group pair.
If selinux / sVirt was in use you would see a separate with
model="selinux".
If you don't see that, then libvirtd is *not* doing any SELinux labelling
at all.
This makes me wonder how you hit the relabelling AVC ? It feels very much
like your custom work to deal with passing in TAP devices has mistakenly
changed something else and thus incorrectly turned on SELinux/sVirt in
libvirtd.
> > If libvirtd is doing relabelling, I'm not sure how it works for anything,
> > let alone tap devices.
> >
> I want to emphasize that in this current situation the reason we fail is
> that we are trying to give libvirt in a non-privileged pod (i.e.
> virt-launcher) a tap device that was created externally (by a privileged
> pod - virt-handler).
> I think the reason it works so far is that libvirt was using only devices
> which were self created so there was no issue to relabel.
I believe the reason it works historically is that libvirtd is not using
SELinux, per your config above, and so was not attempting
todo any relabelling in the first place.
> Our thought was that if we know how libvirt is relabeling then we could
> also do it so that the externally created tap's label will match the
> virt-launcher's.
You almost certainly *will* need to relabel the tap device when passing
it from your privileged container into the unprivileged, but this is not
due to libvirtd. This is because podman is likely using a different
SELinux label for the two containers as they have different privilege
levels. The tap device needs to be given the SELinux label matching the
unprivileged container.
> Is this were libvirt does the relabeling
> https://github.com/libvirt/libvirt/blob/e71e13488dc1aa65456e54a4b41bc925821b4263/src/security/security_selinux.c#L1256
> ?
That's for setting contexts on files, for FDs you need to call
fsetfilecon_raw
> btw the error we get is (from audit)
>
> type=AVC msg=audit(1586956552.265:513): avc: denied { relabelfrom }
> for pid=27423 comm="libvirtd"
> scontext=system_u:system_r:container_t:s0:c143,c582
> tcontext=system_u:system_r:spc_t:s0 tclass=tun_socket permissive=0
This just suggests libvirtd has been misconfigured in some way that causes
SELinux to become enabled again.
You need to do relabelling of the TAP device, but that isn't going to
avoid this AVC - it will just change what you see as the tcontext in
the AVC. You still need to fix the configuration of libvirtd I believe.
If you run 'virsh capabilities' in your container that is showing this
problem, I expect it will say:
selinux
0
system_u:system_r:svirt_t:s0
system_u:system_r:svirt_tcg_t:s0
If you compare with the "virsh capabilities" from a correctly operating
existing KubeVirt install, then I expect selinux won't be reported at
all, which would match your example from earlier.
Regards,
Daniel
--
|: https://berrange.com -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: Unable to decode message length
Thank you for the help. But here I get exactly the same data as you. So I suppose the issue is on the server side only. I will try with gdb now. On Wed, Jul 15, 2020 at 16:02, Michal Privoznik wrote: On 7/14/20 7:42 PM, Valentin David wrote: Hello all, I have been trying to get libvirtd to work but when I connect to it with virsh, I get "error : virNetMessageDecodeLength:131 : Unable to decode message length" This happens with libvirt 6.1.0, libtirpc 1.2.6, rpcsvc-proto 1.4.1. I have tried with other versions, but I still get the same error. If anybody has any tip on what to try next, that would be helpful. Thank you in advance. Here is the debug log of libvirtd when I try to connect with virsh: 2020-07-14 16:29:58.220+: 5352: error : virNetMessageDecodeLength:131 : Unable to decode message length This is interesting. The error message is reported when xdr_u_int() fails. It is trying to decode first 4 bytes of incoming message to see how long it is. I can think of two possible reasons for this: 1) the client did not send 4 bytes 2) the rpcgen generated some wrong code that libtirpc is unable to decode. I suggest using wireshark to check what data is send (don't forget to use plain TCP transport in URI, for instance 'virsh -c qemu+tcp://localhost/system'). The first call that virsh does is AUTH_LIST procedure and it should look like this: 00 00 00 00 00 00 00 00 00 00 00 00 86 dd 60 04 0010 52 cc 00 3c 06 40 00 00 00 00 00 00 00 00 00 00 0020 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 0030 00 00 00 00 00 01 df 6c 40 7d fd e7 38 94 09 92 0040 bf 48 80 18 02 00 00 44 00 00 01 01 08 0a c9 4f 0050 f4 f2 c9 4f f4 f2 00 00 00 1c 20 00 80 86 00 00 0060 00 01 00 00 00 42 00 00 00 00 00 00 00 00 00 00 0070 00 00 bytes 0x56-0x59 contain 0x1c (= 28) which is the length of the packet (28 bytes). This is how our dissector decodes it: Libvirt length: 28 program: REMOTE (0x20008086) version: 1 procedure: AUTH_LIST (66) type: CALL (0) serial: 0 status: OK (0) Michal
Re: Unable to decode message length
I have found the issue. So I just reply to archive the solution in case others have the same issue. I had both portablexdr and litirpc available at build time. I removed portablexdr for it to link to libtirpc, and now it works. On Thu, Jul 16, 2020 at 11:48, Valentin David wrote: Thank you for the help. But here I get exactly the same data as you. So I suppose the issue is on the server side only. I will try with gdb now. On Wed, Jul 15, 2020 at 16:02, Michal Privoznik wrote: On 7/14/20 7:42 PM, Valentin David wrote: Hello all, I have been trying to get libvirtd to work but when I connect to it with virsh, I get "error : virNetMessageDecodeLength:131 : Unable to decode message length" This happens with libvirt 6.1.0, libtirpc 1.2.6, rpcsvc-proto 1.4.1. I have tried with other versions, but I still get the same error. If anybody has any tip on what to try next, that would be helpful. Thank you in advance. Here is the debug log of libvirtd when I try to connect with virsh: 2020-07-14 16:29:58.220+: 5352: error : virNetMessageDecodeLength:131 : Unable to decode message length This is interesting. The error message is reported when xdr_u_int() fails. It is trying to decode first 4 bytes of incoming message to see how long it is. I can think of two possible reasons for this: 1) the client did not send 4 bytes 2) the rpcgen generated some wrong code that libtirpc is unable to decode. I suggest using wireshark to check what data is send (don't forget to use plain TCP transport in URI, for instance 'virsh -c qemu+tcp://localhost/system'). The first call that virsh does is AUTH_LIST procedure and it should look like this: 00 00 00 00 00 00 00 00 00 00 00 00 86 dd 60 04 0010 52 cc 00 3c 06 40 00 00 00 00 00 00 00 00 00 00 0020 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 0030 00 00 00 00 00 01 df 6c 40 7d fd e7 38 94 09 92 0040 bf 48 80 18 02 00 00 44 00 00 01 01 08 0a c9 4f 0050 f4 f2 c9 4f f4 f2 00 00 00 1c 20 00 80 86 00 00 0060 00 01 00 00 00 42 00 00 00 00 00 00 00 00 00 00 0070 00 00 bytes 0x56-0x59 contain 0x1c (= 28) which is the length of the packet (28 bytes). This is how our dissector decodes it: Libvirt length: 28 program: REMOTE (0x20008086) version: 1 procedure: AUTH_LIST (66) type: CALL (0) serial: 0 status: OK (0) Michal
