kiranchavala commented on issue #11956: URL: https://github.com/apache/cloudstack/issues/11956#issuecomment-3484097745
> oh, good, so the issue is with the live volume migration between arm64 hosts. thanks [@kiranchavala](https://github.com/kiranchavala) for the testing > > any errors in /var/log/libvirt/qemu/i-xx-xxx.log on both source and dest hosts ? Please find the logs Tried migrating s-77 from raspi4 to raspi5 cslogs ``` 2025-11-04 06:21:52,419 ERROR [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-5:[]) (logid:2c65c047) Can't migrate domain [s-77-VM] due to: [org.libvirt.LibvirtException: operation failed: job 'migration out' failed: Unable to read from socket: Connection reset by peer]. java.util.concurrent.ExecutionException: org.libvirt.LibvirtException: operation failed: job 'migration out' failed: Unable to read from socket: Connection reset by peer at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.execute(LibvirtMigrateCommandWrapper.java:326) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.execute(LibvirtMigrateCommandWrapper.java:91) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2280) at com.cloud.agent.Agent.processRequest(Agent.java:813) at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295) at com.cloud.utils.nio.Task.call(Task.java:83) at com.cloud.utils.nio.Task.call(Task.java:29) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: org.libvirt.LibvirtException: operation failed: job 'migration out' failed: Unable to read from socket: Connection reset by peer at org.libvirt.ErrorHandler.processError(Unknown Source) at org.libvirt.ErrorHandler.processError(Unknown Source) at org.libvirt.Domain.migrate(Unknown Source) at com.cloud.hypervisor.kvm.resource.MigrateKVMAsync.call(MigrateKVMAsync.java:142) at com.cloud.hypervisor.kvm.resource.MigrateKVMAsync.call(MigrateKVMAsync.java:34) ... 4 more ``` qemu log from raspi4 ``` root@raspberrypi4:/var/log/libvirt/qemu# cat s-77-VM.log 2025-11-04 06:16:26.490+0000: starting up libvirt version: 10.0.0, package: 10.0.0-2ubuntu8.9 (Ubuntu), qemu version: 8.2.2Debian 1:8.2.2+ds-0ubuntu1.10, kernel: 6.8.0-1040-raspi, hostname: raspberrypi4 LC_ALL=C \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/snap/bin \ USER=root \ HOME=/var/lib/libvirt/qemu/domain-2-s-77-VM \ XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-2-s-77-VM/.local/share \ XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-2-s-77-VM/.cache \ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-2-s-77-VM/.config \ /usr/bin/qemu-system-aarch64 \ -name guest=s-77-VM,debug-threads=on \ -S \ -object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-2-s-77-VM/master-key.aes"}' \ -blockdev '{"driver":"file","filename":"/usr/share/AAVMF/AAVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \ -blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/s-77-VM_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \ -machine virt-8.2,usb=off,gic-version=2,dump-guest-core=off,memory-backend=mach-virt.ram,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,acpi=on \ -accel kvm \ -cpu host \ -m size=524288k \ -object '{"qom-type":"memory-backend-ram","id":"mach-virt.ram","size":536870912}' \ -overcommit mem-lock=off \ -smp 1,sockets=1,dies=1,cores=1,threads=1 \ -uuid 8ff51165-a0c8-4482-9e85-cb54e8656c2b \ -smbios 'type=1,manufacturer=Apache Software Foundation,product=CloudStack KVM Hypervisor,serial=8ff51165-a0c8-4482-9e85-cb54e8656c2b,uuid=8ff51165-a0c8-4482-9e85-cb54e8656c2b' \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=36,server=on,wait=off \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=utc \ -no-shutdown \ -boot strict=on \ -device '{"driver":"pcie-root-port","port":8,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x1"}' \ -device '{"driver":"pcie-root-port","port":9,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x1.0x1"}' \ -device '{"driver":"pcie-root-port","port":10,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x1.0x2"}' \ -device '{"driver":"pcie-root-port","port":11,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x1.0x3"}' \ -device '{"driver":"pcie-root-port","port":12,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x1.0x4"}' \ -device '{"driver":"pcie-root-port","port":13,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x1.0x5"}' \ -device '{"driver":"pcie-pci-bridge","id":"pci.7","bus":"pci.1","addr":"0x0"}' \ -device '{"driver":"pcie-root-port","port":14,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x1.0x6"}' \ -device '{"driver":"pcie-root-port","port":15,"chassis":9,"id":"pci.9","bus":"pcie.0","addr":"0x1.0x7"}' \ -device '{"driver":"pcie-root-port","port":16,"chassis":10,"id":"pci.10","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \ -device '{"driver":"pcie-root-port","port":17,"chassis":11,"id":"pci.11","bus":"pcie.0","addr":"0x2.0x1"}' \ -device '{"driver":"qemu-xhci","id":"usb","bus":"pci.5","addr":"0x0"}' \ -device '{"driver":"lsi","id":"scsi0","bus":"pci.7","addr":"0x1"}' \ -device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.6","addr":"0x0"}' \ -blockdev '{"driver":"file","filename":"/mnt/c7d94500-acf4-3ee0-9410-56cd2aa3fda9/bb80dd0f-05a8-4eb9-a525-59b15f0a77ac","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap","cache":{"direct":true,"no-flush":false}}' \ -blockdev '{"node-name":"libvirt-3-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-3-storage","backing":null}' \ -blockdev '{"driver":"file","filename":"/mnt/c7d94500-acf4-3ee0-9410-56cd2aa3fda9/36d04d51-90cb-4248-9d15-fbb27f9bc0e0","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap","cache":{"direct":true,"no-flush":false}}' \ -blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage","backing":"libvirt-3-format"}' \ -device '{"driver":"virtio-blk-pci","bus":"pci.8","addr":"0x0","drive":"libvirt-2-format","id":"virtio-disk0","bootindex":2,"write-cache":"on","serial":"36d04d5190cb42489d15"}' \ -device '{"driver":"scsi-cd","bus":"scsi0.0","scsi-id":2,"device_id":"drive-scsi0-0-2","id":"scsi0-0-2","bootindex":1}' \ -netdev '{"type":"tap","fd":"33","vhost":true,"vhostfd":"34","id":"hostnet0"}' \ -device '{"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"0e:00:a9:fe:88:39","bus":"pci.2","addr":"0x0","rombar":0,"romfile":""}' \ -netdev '{"type":"tap","fd":"41","vhost":true,"vhostfd":"46","id":"hostnet1"}' \ -device '{"driver":"virtio-net-pci","netdev":"hostnet1","id":"net1","mac":"1e:00:89:00:00:0c","bus":"pci.3","addr":"0x0","rombar":0,"romfile":""}' \ -netdev '{"type":"tap","fd":"48","vhost":true,"vhostfd":"50","id":"hostnet2"}' \ -device '{"driver":"virtio-net-pci","netdev":"hostnet2","id":"net2","mac":"1e:00:6d:00:00:17","bus":"pci.4","addr":"0x0","rombar":0,"romfile":""}' \ -chardev pty,id=charserial0 \ -serial chardev:charserial0 \ -chardev socket,id=charchannel0,fd=35,server=on,wait=off \ -device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \ -device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \ -device '{"driver":"usb-kbd","id":"input1","bus":"usb.0","port":"2"}' \ -device '{"driver":"usb-mouse","id":"input2","bus":"usb.0","port":"3"}' \ -audiodev '{"id":"audio1","driver":"none"}' \ -object '{"qom-type":"tls-creds-x509","id":"vnc-tls-creds0","dir":"/etc/pki/libvirt-vnc","endpoint":"server","verify-peer":true}' \ -vnc 192.168.55.211:1,password=on,tls-creds=vnc-tls-creds0,audiodev=audio1 \ -device '{"driver":"virtio-gpu-pci","id":"video0","max_outputs":1,"bus":"pci.10","addr":"0x0"}' \ -device '{"driver":"i6300esb","id":"watchdog0","bus":"pci.7","addr":"0x2"}' \ -watchdog-action none \ -device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.9","addr":"0x0"}' \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on 2025-11-04 06:16:26.491+0000: Domain id=2 is tainted: high-privileges char device redirected to /dev/pts/1 (label charserial0) 2025-11-04 06:17:11.931+0000: Domain id=2 is tainted: custom-ga-command 2025-11-04 06:21:47.508+0000: initiating migration 2025-11-04T06:21:52.011225Z qemu-system-aarch64: Unable to read from socket: Connection reset by peer warning: old compression is deprecated; use multifd compression methods instead warning: old compression is deprecated; use multifd compression methods instead warning: old compression is deprecated; use multifd compression methods instead warning: block migration is deprecated; use blockdev-mirror with NBD instead ``` qemu log from raspi4 ``` root@raspberrypi5:/var/log/libvirt/qemu# cat s-77-VM.log 2025-11-04 06:21:45.774+0000: starting up libvirt version: 10.0.0, package: 10.0.0-2ubuntu8.9 (Ubuntu), qemu version: 8.2.2Debian 1:8.2.2+ds-0ubuntu1.10, kernel: 6.8.0-1040-raspi, hostname: raspberrypi5 LC_ALL=C \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/snap/bin \ USER=root \ HOME=/var/lib/libvirt/qemu/domain-2-s-77-VM \ XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-2-s-77-VM/.local/share \ XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-2-s-77-VM/.cache \ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-2-s-77-VM/.config \ /usr/bin/qemu-system-aarch64 \ -name guest=s-77-VM,debug-threads=on \ -S \ -object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-2-s-77-VM/master-key.aes"}' \ -blockdev '{"driver":"file","filename":"/usr/share/AAVMF/AAVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \ -blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/s-77-VM_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \ -machine virt-8.2,usb=off,gic-version=2,dump-guest-core=off,memory-backend=mach-virt.ram,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,acpi=on \ -accel kvm \ -cpu host \ -m size=524288k \ -object '{"qom-type":"memory-backend-ram","id":"mach-virt.ram","size":536870912}' \ -overcommit mem-lock=off \ -smp 1,sockets=1,dies=1,cores=1,threads=1 \ -uuid 8ff51165-a0c8-4482-9e85-cb54e8656c2b \ -smbios 'type=1,manufacturer=Apache Software Foundation,product=CloudStack KVM Hypervisor,serial=8ff51165-a0c8-4482-9e85-cb54e8656c2b,uuid=8ff51165-a0c8-4482-9e85-cb54e8656c2b' \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=39,server=on,wait=off \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=utc \ -no-shutdown \ -boot strict=on \ -device '{"driver":"pcie-root-port","port":8,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x1"}' \ -device '{"driver":"pcie-root-port","port":9,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x1.0x1"}' \ -device '{"driver":"pcie-root-port","port":10,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x1.0x2"}' \ -device '{"driver":"pcie-root-port","port":11,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x1.0x3"}' \ -device '{"driver":"pcie-root-port","port":12,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x1.0x4"}' \ -device '{"driver":"pcie-root-port","port":13,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x1.0x5"}' \ -device '{"driver":"pcie-pci-bridge","id":"pci.7","bus":"pci.1","addr":"0x0"}' \ -device '{"driver":"pcie-root-port","port":14,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x1.0x6"}' \ -device '{"driver":"pcie-root-port","port":15,"chassis":9,"id":"pci.9","bus":"pcie.0","addr":"0x1.0x7"}' \ -device '{"driver":"pcie-root-port","port":16,"chassis":10,"id":"pci.10","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \ -device '{"driver":"pcie-root-port","port":17,"chassis":11,"id":"pci.11","bus":"pcie.0","addr":"0x2.0x1"}' \ -device '{"driver":"qemu-xhci","id":"usb","bus":"pci.5","addr":"0x0"}' \ -device '{"driver":"lsi","id":"scsi0","bus":"pci.7","addr":"0x1"}' \ -device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.6","addr":"0x0"}' \ -blockdev '{"driver":"file","filename":"/mnt/c7d94500-acf4-3ee0-9410-56cd2aa3fda9/bb80dd0f-05a8-4eb9-a525-59b15f0a77ac","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap","cache":{"direct":true,"no-flush":false}}' \ -blockdev '{"node-name":"libvirt-3-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-3-storage","backing":null}' \ -blockdev '{"driver":"file","filename":"/mnt/c7d94500-acf4-3ee0-9410-56cd2aa3fda9/36d04d51-90cb-4248-9d15-fbb27f9bc0e0","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap","cache":{"direct":true,"no-flush":false}}' \ -blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage","backing":"libvirt-3-format"}' \ -device '{"driver":"virtio-blk-pci","bus":"pci.8","addr":"0x0","drive":"libvirt-2-format","id":"virtio-disk0","bootindex":2,"write-cache":"on","serial":"36d04d5190cb42489d15"}' \ -device '{"driver":"scsi-cd","bus":"scsi0.0","scsi-id":2,"device_id":"drive-scsi0-0-2","id":"scsi0-0-2","bootindex":1}' \ -netdev '{"type":"tap","fd":"40","vhost":true,"vhostfd":"42","id":"hostnet0"}' \ -device '{"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"0e:00:a9:fe:88:39","bus":"pci.2","addr":"0x0","rombar":0,"romfile":""}' \ -netdev '{"type":"tap","fd":"43","vhost":true,"vhostfd":"44","id":"hostnet1"}' \ -device '{"driver":"virtio-net-pci","netdev":"hostnet1","id":"net1","mac":"1e:00:89:00:00:0c","bus":"pci.3","addr":"0x0","rombar":0,"romfile":""}' \ -netdev '{"type":"tap","fd":"45","vhost":true,"vhostfd":"46","id":"hostnet2"}' \ -device '{"driver":"virtio-net-pci","netdev":"hostnet2","id":"net2","mac":"1e:00:6d:00:00:17","bus":"pci.4","addr":"0x0","rombar":0,"romfile":""}' \ -chardev pty,id=charserial0 \ -serial chardev:charserial0 \ -chardev socket,id=charchannel0,fd=38,server=on,wait=off \ -device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \ -device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \ -device '{"driver":"usb-kbd","id":"input1","bus":"usb.0","port":"2"}' \ -device '{"driver":"usb-mouse","id":"input2","bus":"usb.0","port":"3"}' \ -audiodev '{"id":"audio1","driver":"none"}' \ -object '{"qom-type":"tls-creds-x509","id":"vnc-tls-creds0","dir":"/etc/pki/libvirt-vnc","endpoint":"server","verify-peer":true}' \ -vnc 192.168.55.210:1,password=on,tls-creds=vnc-tls-creds0,audiodev=audio1 \ -device '{"driver":"virtio-gpu-pci","id":"video0","max_outputs":1,"bus":"pci.10","addr":"0x0"}' \ -device '{"driver":"i6300esb","id":"watchdog0","bus":"pci.7","addr":"0x2"}' \ -watchdog-action none \ -incoming defer \ -device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.9","addr":"0x0"}' \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on 2025-11-04 06:21:45.775+0000: Domain id=2 is tainted: high-privileges 2025-11-04 06:21:45.775+0000: Domain id=2 is tainted: hook-script 2025-11-04 06:21:45.775+0000: Domain id=2 is tainted: host-cpu char device redirected to /dev/pts/2 (label charserial0) 2025-11-04T06:21:51.802223Z qemu-system-aarch64: error while loading state for instance 0x0 of device 'cpu' 2025-11-04T06:21:51.803803Z qemu-system-aarch64: Failed to put registers after init: Invalid argument 2025-11-04 06:21:52.090+0000: shutting down, reason=crashed ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
