Re: Triggering a checkpoint from inside the VM

[Leek, Jim]

I’ll try snapshot create, I didn’t know about that.

I understand your security concerns, but they don’t apply in this case. The 
target environment is air gapped and isolated. I can’t see any reason to fear 
an attack on the VM before an attack on the host.

2. Well, I was experimenting with getting libvirt out of the system to simplify 
things, but I didn’t have much luck yet.


---
Sent from Workspace ONE Boxer

On September 9, 2021 at 1:20:38 AM PDT, Daniel P. Berrangé 
 wrote:
On Wed, Sep 08, 2021 at 04:22:31AM +, Leek, Jim wrote:
> I'm on a RHEL 8 host, using virt-manager to run a CentOS 8 guest.  I need
> to be able to have a program on the guest trigger a checkpoint to save
> the guest.  I came up with a kludgy way to do this involving a script
> that ssh's to the host and runs 'virsh qemu-monitor-command --hmp
> centos8_1 "savevm savestate1"' and that works to some degree, but it
> takes a long time and sometimes I get an error.

This is a bad idea.

"savevm" completely stops execution of the guest for the duration
that it runs.so your ssh conenction is suspended. Depending
on how long this takes, your ssh connection may take some time to
recover, or in the worst case fail.

Using qemu-monitor-command is not neccessary because libvirt already
has support for savevm via its domain snapshot APIs epxosed in virsh
using snapshot-* commands. Using qemu-monitor-command in this case
is likely to confuse libvirt because it is resulting in unexpected
state changes in the guest.

Allowing the guest to ssh into the host and connect to libvirt
throws away any security isolation your host has from the guest.
So if your guest is compromised it'll easily take over the host
too.

> So, I'm trying to think of ways to simplify the system.  If anyone has any 
> ideas, I would love to have them.  All I can think of is:
>
>   1.  Connect to the qemu monitor with telnet from inside the VM.  (Therefore 
> skipping the whole ssh remote command thing.)

Definitely don't want todo that - access to the QEMU monitor
again allows guest to attack the host in various ways. If
libvirt is connected to the QEMU monitor, you can't have a
second connection anyway.

Regards,
Daniel
--
|: 
https://urldefense.us/v3/__https://berrange.com__;!!G2kpM7uM-TzIFchu!lFVFSs5C2w6Vt5mss2OePJAnR8QGxohw4OvKhWVxKNwxttCUfPD5f7tPKtVgtmY$
   -o-
https://urldefense.us/v3/__https://www.flickr.com/photos/dberrange__;!!G2kpM7uM-TzIFchu!lFVFSs5C2w6Vt5mss2OePJAnR8QGxohw4OvKhWVxKNwxttCUfPD5f7tPzRsfC3g$
  :|
|: 
https://urldefense.us/v3/__https://libvirt.org__;!!G2kpM7uM-TzIFchu!lFVFSs5C2w6Vt5mss2OePJAnR8QGxohw4OvKhWVxKNwxttCUfPD5f7tPsx80x1w$
  -o-
https://urldefense.us/v3/__https://fstop138.berrange.com__;!!G2kpM7uM-TzIFchu!lFVFSs5C2w6Vt5mss2OePJAnR8QGxohw4OvKhWVxKNwxttCUfPD5f7tPsGUnCCQ$
  :|
|: 
https://urldefense.us/v3/__https://entangle-photo.org__;!!G2kpM7uM-TzIFchu!lFVFSs5C2w6Vt5mss2OePJAnR8QGxohw4OvKhWVxKNwxttCUfPD5f7tPTFrwcL0$
 -o-
https://urldefense.us/v3/__https://www.instagram.com/dberrange__;!!G2kpM7uM-TzIFchu!lFVFSs5C2w6Vt5mss2OePJAnR8QGxohw4OvKhWVxKNwxttCUfPD5f7tPe1eBb0E$
  :|

Re: Triggering a checkpoint from inside the VM

[Daniel P . Berrangé]

On Thu, Sep 09, 2021 at 08:31:58AM +, Leek, Jim wrote:
> I’ll try snapshot create, I didn’t know about that.
> 
> I understand your security concerns, but they don’t apply in this case. The 
> target environment is air gapped and isolated. I can’t see any reason to fear 
> an attack on the VM before an attack on the host.
> 
> 2. Well, I was experimenting with getting libvirt out of the system to 
> simplify things, but I didn’t have much luck yet.

Taking libvirt out of the loop is likely to make your life harder
in the long term, especially as you'll no longer be insulated from
changes in QEMU's external management interfaces. QEMU may look
simple to manage directly at first glance, but it is actually quite
a complicated to do it well when you look closely.


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Triggering a checkpoint from inside the VM

[Daniel P . Berrangé]

On Thu, Sep 09, 2021 at 12:30:53AM +, Leek, Jim wrote:
> Here's the error I sometimes get on checkpoint/restore.  It just happened 
> again.  In this test I had at least a 40 second break between checkpoint and 
> restore, so extra sleep doesn't seem to help much.
> 
> Error unpausing domain: Timed out during operation: cannot acquire state 
> change lock (held by monitor=qemuDispatchDomainMonitorCommand)
> 
> Traceback (most recent call last):
>   File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in 
> cb_wrapper
> callback(asyncjob, *args, **kwargs)
>   File "/usr/share/virt-manager/virtManager/asyncjob.py", line 111, in tmpcb
> callback(*args, **kwargs)
>   File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 
> 66, in newfn
> ret = fn(self, *args, **kwargs)
>   File "/usr/share/virt-manager/virtManager/object/domain.py", line 1312, in 
> resume
> self._backend.resume()
>   File "/usr/lib64/python3.6/site-packages/libvirt.py", line 2174, in resume
> if ret == -1: raise libvirtError ('virDomainResume() failed', dom=self)
> libvirt.libvirtError: Timed out during operation: cannot acquire state change 
> lock (held by monitor=qemuDispatchDomainMonitorCommand)

This suggests that the previously executed 'qemu-montor-command' has not
finished running yet.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Triggering a checkpoint from inside the VM

[Leek, Jim]

Here's the error I sometimes get on checkpoint/restore.  It just happened 
again.  In this test I had at least a 40 second break between checkpoint and 
restore, so extra sleep doesn't seem to help much.

Error unpausing domain: Timed out during operation: cannot acquire state change 
lock (held by monitor=qemuDispatchDomainMonitorCommand)

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 111, in tmpcb
callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 66, 
in newfn
ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/domain.py", line 1312, in 
resume
self._backend.resume()
  File "/usr/lib64/python3.6/site-packages/libvirt.py", line 2174, in resume
if ret == -1: raise libvirtError ('virDomainResume() failed', dom=self)
libvirt.libvirtError: Timed out during operation: cannot acquire state change 
lock (held by monitor=qemuDispatchDomainMonitorCommand)

From: Tony Brian Albers 
Sent: Wednesday, September 8, 2021 12:06 AM
To: Leek, Jim ; virt-tools-list@redhat.com 

Subject: Re: Triggering a checkpoint from inside the VM

On 08/09/2021 08.31, Leek, Jim wrote:
> 1. I guess you mean by Using a shared disk?  That could work, I had some 
> trouble with the shared disk before, I don’t think I ever got it to work. 
> I’ll have to check my notes. IT has the host machine locked down very 
> securely in some ways, and sometimes that causes trouble.

On my host I've set up a local network that only the VM's and the host
has access to. That makes it possible to do smth like:

user@host$  ssh user@vm ls -l somefile

and then compare that output to something you've defined, and trigger
the savevm based on that.

>
> 2. I did specify the settings in virt-manager and I can use virsh. But 
> sometimes I get an error from libvirt when doing a series of savevm and 
> loadvm in a row. I have a hope that using qemu-kvm without libvirt might make 
> it more reliable.  But as I said, I’m having trouble getting the networking 
> and display working properly without libvirt.
>

I think libvirt is necessary to make networking and display work(not
entirely sure). But the errors you get might be because the processes
are scheduled too close to one another, see if you can put something
like a sleep() in between them. I think 10 seconds will do.

/tony

>
> ---
> Sent from Workspace ONE 
> Boxer<https://urldefense.us/v3/__https://whatisworkspaceone.com/boxer__;!!G2kpM7uM-TzIFchu!hbhAdjLTl6sGjvB4cfYJDpIObK0BhNLmWmmi0NEBWX7ls4h84l2slu6DAIhiKh0$
>  >
>
> On September 7, 2021 at 10:58:03 PM PDT, Tony Brian Albers  wrote:
> On 08/09/2021 06.22, Leek, Jim wrote:
>
>>
>> 1.  Connect to the qemu monitor with telnet from inside the VM.  
>> (Therefore skipping the whole ssh remote command thing.)
>
> I'd go the other way around, make the guest touch a file somewhere and
> let the host check if the file is there/has been updated, and based on
> that, run the savevm process.
>
>
>> 2.  Run the VM without virt-manager (perhaps that would be simpler?)
>>
>
> I think you can specify the settings for the VM through virt-manager and
> then just use virsh to manage it afterwards.
>
> Also, check out github for kvm backup scripts, I know there are some
> that you could use.
>
>
> HTH
>
> /tony
>
>
> --
> Tony Albers - Systems Architect - Data Department, Royal Danish Library,
> Victor Albecks Vej 1, 8000 Aarhus C, Denmark
> Tel: +45 2566 2383 - CVR/SE: 2898 8842 - EAN: 5798000792142
>


--
Tony Albers - Systems Architect - Data Department, Royal Danish Library,
Victor Albecks Vej 1, 8000 Aarhus C, Denmark
Tel: +45 2566 2383 - CVR/SE: 2898 8842 - EAN: 5798000792142

Re: Triggering a checkpoint from inside the VM

[Leek, Jim]

1. I guess you mean by Using a shared disk?  That could work, I had some 
trouble with the shared disk before, I don’t think I ever got it to work. I’ll 
have to check my notes. IT has the host machine locked down very securely in 
some ways, and sometimes that causes trouble.

2. I did specify the settings in virt-manager and I can use virsh. But 
sometimes I get an error from libvirt when doing a series of savevm and loadvm 
in a row. I have a hope that using qemu-kvm without libvirt might make it more 
reliable.  But as I said, I’m having trouble getting the networking and display 
working properly without libvirt.


---
Sent from Workspace ONE Boxer

On September 7, 2021 at 10:58:03 PM PDT, Tony Brian Albers  wrote:
On 08/09/2021 06.22, Leek, Jim wrote:

>
>1.  Connect to the qemu monitor with telnet from inside the VM.  
> (Therefore skipping the whole ssh remote command thing.)

I'd go the other way around, make the guest touch a file somewhere and
let the host check if the file is there/has been updated, and based on
that, run the savevm process.


>2.  Run the VM without virt-manager (perhaps that would be simpler?)
>

I think you can specify the settings for the VM through virt-manager and
then just use virsh to manage it afterwards.

Also, check out github for kvm backup scripts, I know there are some
that you could use.


HTH

/tony


--
Tony Albers - Systems Architect - Data Department, Royal Danish Library,
Victor Albecks Vej 1, 8000 Aarhus C, Denmark
Tel: +45 2566 2383 - CVR/SE: 2898 8842 - EAN: 5798000792142

Triggering a checkpoint from inside the VM

[Leek, Jim]

I'm on a RHEL 8 host, using virt-manager to run a CentOS 8 guest.  I need to be 
able to have a program on the guest trigger a checkpoint to save the guest.  I 
came up with a kludgy way to do this involving a script that ssh's to the host 
and runs 'virsh qemu-monitor-command --hmp centos8_1 "savevm savestate1"' and 
that works to some degree, but it takes a long time and sometimes I get an 
error.

So, I'm trying to think of ways to simplify the system.  If anyone has any 
ideas, I would love to have them.  All I can think of is:

  1.  Connect to the qemu monitor with telnet from inside the VM.  (Therefore 
skipping the whole ssh remote command thing.)
  2.  Run the VM without virt-manager (perhaps that would be simpler?)

I actually don't know how to do #1 without doing #2.  When I ran qemu from the 
command line there was a command line argument to define the port the monitor 
should open on.  I don't know how to access the monitor via telnet without that 
command line argument.

As for #2, I'm having trouble getting that to work as well.  I found this page 
and followed the instructions: 
https://developers.redhat.com/blog/2020/03/06/configure-and-run-a-qemu-based-vm-outside-of-libvirt
 (although he talks about qemu-system-x86_64 and I'm using qemu-kvm, but I hope 
that doesn't make any difference.)

But I can't get it to work.  I've tried a number of modifications, and the 
command line below launches the VM, but I can't access it.  No screen pops up, 
and I can't get there via ssh.  If I take off the last lines about the display 
it will say I can use VNC ::1:5900, but when I connect there it just says 
"guest has not initialized display yet."

Anybody have any ideas on any of this?
Here's the command line I've tried.  The removing the last 2 arguments does 
change the display behavior, but not to any particular benefit:
#! /bin/sh
export LC_ALL=C
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
export HOME=/var/lib/libvirt/qemu/domain-6-centos8_3
export XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-6-centos8_3/.local/share
export XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-6-centos8_3/.cache
export XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-6-centos8_3/.config
export QEMU_AUDIO_DRV=spice
/usr/libexec/qemu-kvm \
-name guest=centos8_3,debug-threads=on \
-S \
-enable-fips \
-machine pc-q35-rhel8.2.0,accel=kvm,usb=off,vmport=off,dump-guest-core=off \
-cpu 
Skylake-Server-IBRS,ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,clflushopt=on,umip=on,pku=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaves=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on
 \
-m 24000 \
-overcommit mem-lock=off \
-smp 2,sockets=2,cores=1,threads=1 \
-uuid 8dd20f24-3e31-464c-956e-67d3d9f2a83c \
-no-user-config \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device 
pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2
 \
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 \
-device pcie-root-port,port=0x16,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6 \
-device pcie-root-port,port=0x17,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x7 \
-device pcie-pci-bridge,id=pci.9,bus=pci.1,addr=0x0 \
-device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.2,addr=0x0 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 \
-blockdev 
'{"driver":"file","filename":"/home/leek2/qemu/rhel8_1-clone-1.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}'
 \
-blockdev 
'{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null}'
 \
-device 
virtio-blk-pci,scsi=off,bus=pci.4,addr=0x0,drive=libvirt-1-format,id=virtio-disk0,bootindex=1
 \
-device 
virtio-net-pci,netdev=hostnet0,id=net0,mac=6c:2b:59:e9:44:49,bus=pci.7,addr=0x0 
\
-netdev bridge,id=hostnet0,br=virbr0 \
-chardev spicevmc,id=charchannel0,name=vdagent \
-device 
virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
 \
-device virtio-balloon-pci,id=balloon0,bus=pci.5,addr=0x0 \
-object rng-random,id=objrng0,filename=/dev/urandom \
-device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.6,addr=0x0 \
-msg timestamp=on \
-device 
qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1
 \
-spice 
port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on

Re: Triggering a checkpoint from inside the VM

[Tony Brian Albers]

On 08/09/2021 08.31, Leek, Jim wrote:
> 1. I guess you mean by Using a shared disk?  That could work, I had some 
> trouble with the shared disk before, I don’t think I ever got it to work. 
> I’ll have to check my notes. IT has the host machine locked down very 
> securely in some ways, and sometimes that causes trouble.

On my host I've set up a local network that only the VM's and the host 
has access to. That makes it possible to do smth like:

user@host$  ssh user@vm ls -l somefile

and then compare that output to something you've defined, and trigger 
the savevm based on that.

> 
> 2. I did specify the settings in virt-manager and I can use virsh. But 
> sometimes I get an error from libvirt when doing a series of savevm and 
> loadvm in a row. I have a hope that using qemu-kvm without libvirt might make 
> it more reliable.  But as I said, I’m having trouble getting the networking 
> and display working properly without libvirt.
> 

I think libvirt is necessary to make networking and display work(not 
entirely sure). But the errors you get might be because the processes 
are scheduled too close to one another, see if you can put something 
like a sleep() in between them. I think 10 seconds will do.

/tony

> 
> ---
> Sent from Workspace ONE Boxer
> 
> On September 7, 2021 at 10:58:03 PM PDT, Tony Brian Albers  wrote:
> On 08/09/2021 06.22, Leek, Jim wrote:
> 
>>
>> 1.  Connect to the qemu monitor with telnet from inside the VM.  
>> (Therefore skipping the whole ssh remote command thing.)
> 
> I'd go the other way around, make the guest touch a file somewhere and
> let the host check if the file is there/has been updated, and based on
> that, run the savevm process.
> 
> 
>> 2.  Run the VM without virt-manager (perhaps that would be simpler?)
>>
> 
> I think you can specify the settings for the VM through virt-manager and
> then just use virsh to manage it afterwards.
> 
> Also, check out github for kvm backup scripts, I know there are some
> that you could use.
> 
> 
> HTH
> 
> /tony
> 
> 
> --
> Tony Albers - Systems Architect - Data Department, Royal Danish Library,
> Victor Albecks Vej 1, 8000 Aarhus C, Denmark
> Tel: +45 2566 2383 - CVR/SE: 2898 8842 - EAN: 5798000792142
> 


-- 
Tony Albers - Systems Architect - Data Department, Royal Danish Library, 
Victor Albecks Vej 1, 8000 Aarhus C, Denmark
Tel: +45 2566 2383 - CVR/SE: 2898 8842 - EAN: 5798000792142

Re: Triggering a checkpoint from inside the VM

[Tony Brian Albers]

On 08/09/2021 06.22, Leek, Jim wrote:

> 
>1.  Connect to the qemu monitor with telnet from inside the VM.  
> (Therefore skipping the whole ssh remote command thing.)

I'd go the other way around, make the guest touch a file somewhere and 
let the host check if the file is there/has been updated, and based on 
that, run the savevm process.


>2.  Run the VM without virt-manager (perhaps that would be simpler?)
> 

I think you can specify the settings for the VM through virt-manager and 
then just use virsh to manage it afterwards.

Also, check out github for kvm backup scripts, I know there are some 
that you could use.


HTH

/tony


-- 
Tony Albers - Systems Architect - Data Department, Royal Danish Library, 
Victor Albecks Vej 1, 8000 Aarhus C, Denmark
Tel: +45 2566 2383 - CVR/SE: 2898 8842 - EAN: 5798000792142