Public bug reported: == Comment: #0 - Balamuruhan S <[email protected]> - 2019-03-12 08:04:56 == ---Problem Description--- VM4 crashed during MultiVM postcopy migration and went to shutoff state,
2019-03-12 07:39:29.467+0000: starting up libvirt version: 4.0.0, package: 1ubuntu8.6 (Christian Ehrhardt <[email protected]> Fri, 09 Nov 2018 07:42:01 +0100), qemu version: 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.8-1ibm3), hostname: pok1-qz1-sr1-rk005-s03.pok.stglabs.ibm.com LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-system-ppc64 -name guest=vm4,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-92-vm4/master-key.aes -machine pseries-2.11,accel=kvm,usb=off,dump-guest-core=off -m 32768 -realtime mlock=off -smp 4,sockets=1,cores=1,threads=4 -uuid 781be17d-66e0-4741-a230-45f56b9e0231 -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-92-vm4/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device qemu-xhci,id=usb,bus=pci.0,addr=0x3 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x2 -drive file=/home/bala/sharing/ubuntu-genesis-18.04-ppc64le_vm4.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0,cache=none -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -netdev tap,fd=56,id=hostnet0,vhost=on,vhostfd=58 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:33:34:35,bus=pci.0,addr=0x1 -netdev tap,fd=59,id=hostnet1,vhost=on,vhostfd=60 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=9a:49:81:1e:91:af,bus=pci.0,addr=0x5 -netdev tap,fd=61,id=hostnet2,vhost=on,vhostfd=62 -device virtio-net-pci,netdev=hostnet2,id=net2,mac=9a:02:0c:b0:27:73,bus=pci.0,addr=0x6 -chardev pty,id=charserial0 -device spapr-vty,chardev=charserial0,id=serial0,reg=0x30000000 -incoming defer -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on 2019-03-12 07:39:29.479+0000: 82411: info : libvirt version: 4.0.0, package: 1ubuntu8.6 (Christian Ehrhardt <[email protected]> Fri, 09 Nov 2018 07:42:01 +0100) 2019-03-12 07:39:29.479+0000: 82411: info : hostname: pok1-qz1-sr1-rk005-s03.pok.stglabs.ibm.com 2019-03-12 07:39:29.479+0000: 82411: info : virObjectUnref:350 : OBJECT_UNREF: obj=0x76a33c158c10 2019-03-12T07:39:29.497052Z qemu-system-ppc64: -chardev pty,id=charserial0: char device redirected to /dev/pts/16 (label charserial0) 2019-03-12 09:02:35.698+0000: initiating migration 2019-03-12 09:19:42.074+0000: initiating migration 2019-03-12T09:20:13.709960Z qemu-system-ppc64: RP: Received invalid message 0x0000 length 0x0000 2019-03-12T09:20:13.715306Z qemu-system-ppc64: postcopy_start: Migration stream errored qemu-system-ppc64: /usr/local/src/qemu/block/io.c:1626: bdrv_co_pwritev: Assertion `!(bs->open_flags & BDRV_O_INACTIVE)' failed. 2019-03-12 10:04:29.298+0000: shutting down, reason=crashed ---Steps to Reproduce--- 1. Place the guest image in NFS shared location and mount the same path in source and target that can be accessed by guest. [step2 to step5 can be achieved by defining Guest xmls that are attached] 2. 28 VMs with 32G memory, 4 vcpus (1 socket 1 core and 4 threads) [Host with 1TB memory] 3. Each vcpu pinned to each hostcpu from 0 to 111 4. Emulator pinned with 4 vms with 1 host cpu from 112 to 118 5. create 28 VFs from available 2 PFs available in the host and give it to guest 6. create VFs by echo 28 to PFs available # echo 28 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs # echo 28 > /sys/bus/pci/devices/0000:01:00.1/sriov_numvfs 7. stress-ng with 90% work load inside guest VM ``` # stress-ng --all 4 --vm-bytes 90% --aggressive --maximize --oomable --timeout 86400 --verify --syslog --metrics --times ``` 8. Perform MultiVM postcopy migration in parallel all 28 guest from source to target. 9. It is observed that VM4 got crashed and went to shutoff state during migration Attachment: 1. Guest XMLs 2. libvirtd debug logs for source 3. sosreport for source and target Contact Information = Balamuruhan S / [email protected] ---System configuration--- Kernel Version: Target: # uname -a Linux pok1-qz1-sr1-rk005-s02.pok.stglabs.ibm.com 4.15.0-1016.18-fix1-ibm-gt #fix1 SMP Wed Feb 20 15:48:47 CST 2019 ppc64le ppc64le ppc64le GNU/Linux Source: # uname -a Linux pok1-qz1-sr1-rk005-s03.pok.stglabs.ibm.com 4.15.0-1016.18-fix1-ibm-gt #fix1 SMP Wed Feb 20 15:48:47 CST 2019 ppc64le ppc64le ppc64le GNU/Linux Guest: # uname -a Linux ubuntu 4.15.0-1016.18-fix1-ibm-gt #fix1 SMP Wed Feb 20 15:48:47 CST 2019 ppc64le ppc64le ppc64le GNU/Linux Qemu version in source and target hosts: # dpkg -l | grep qemu ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2.2 all PXE boot firmware - ROM images for qemu ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu ii qemu 1:2.11+dfsg-1ubuntu7.8-1ibm3 ppc64el fast processor emulator ii qemu-block-extra:ppc64el 1:2.11+dfsg-1ubuntu7.8-1ibm3 ppc64el extra block backend modules for qemu-system and qemu-utils ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version ii qemu-system 1:2.11+dfsg-1ubuntu7.8-1ibm3 ppc64el QEMU full system emulation binaries ii qemu-system-common 1:2.11+dfsg-1ubuntu7.8-1ibm3 ppc64el QEMU full system emulation binaries (common files) ii qemu-system-misc 1:2.11+dfsg-1ubuntu7.8-1ibm3 ppc64el QEMU full system emulation binaries (miscellaneous) ii qemu-system-ppc 1:2.11+dfsg-1ubuntu7.8-1ibm3 ppc64el QEMU full system emulation binaries (ppc) ii qemu-system-x86 1:2.11+dfsg-1ubuntu7.10 ppc64el QEMU full system emulation binaries (x86) ii qemu-user 1:2.11+dfsg-1ubuntu7.8-1ibm3 ppc64el QEMU user mode emulation binaries ii qemu-user-binfmt 1:2.11+dfsg-1ubuntu7.8-1ibm3 ppc64el QEMU user mode binfmt registration for qemu-user ii qemu-utils 1:2.11+dfsg-1ubuntu7.8-1ibm3 ppc64el QEMU utilities Libvirt version in source and target hosts: # dpkg -l | grep libvirt ii gir1.2-libvirt-glib-1.0:ppc64el 1.0.0-1 ppc64el GObject introspection files for the libvirt-glib library ii gir1.2-libvirt-sandbox-1.0 0.5.1+git20160404-1 ppc64el GObject introspection files for the libvirt-sandbox library ii libvirt-bin 4.0.0-1ubuntu8.6 ppc64el programs for the libvirt library ii libvirt-clients 4.0.0-1ubuntu8.6 ppc64el Programs for the libvirt library ii libvirt-daemon 4.0.0-1ubuntu8.6 ppc64el Virtualization daemon ii libvirt-daemon-driver-storage-gluster 4.0.0-1ubuntu8.6 ppc64el Virtualization daemon glusterfs storage driver ii libvirt-daemon-driver-storage-rbd 4.0.0-1ubuntu8.6 ppc64el Virtualization daemon RBD storage driver ii libvirt-daemon-driver-storage-sheepdog 4.0.0-1ubuntu8.6 ppc64el Virtualization daemon Sheedog storage driver ii libvirt-daemon-driver-storage-zfs 4.0.0-1ubuntu8.6 ppc64el Virtualization daemon ZFS storage driver ii libvirt-daemon-system 4.0.0-1ubuntu8.6 ppc64el Libvirt daemon configuration files ii libvirt-dev:ppc64el 4.0.0-1ubuntu8.6 ppc64el development files for the libvirt library ii libvirt-doc 4.0.0-1ubuntu8.6 all documentation for the libvirt library ii libvirt-glib-1.0-0:ppc64el 1.0.0-1 ppc64el libvirt GLib and GObject mapping library ii libvirt-glib-1.0-dev:ppc64el 1.0.0-1 ppc64el Development files for the libvirt-glib library ii libvirt-ocaml 0.6.1.4-2build1 ppc64el OCaml bindings for libvirt (runtime) ii libvirt-ocaml-dev 0.6.1.4-2build1 ppc64el OCaml bindings for libvirt (development files) ii libvirt-sandbox-1.0-5 0.5.1+git20160404-1 ppc64el Application sandbox toolkit shared library ii libvirt-sandbox-1.0-dev 0.5.1+git20160404-1 ppc64el Development files for libvirt-sandbox library ii libvirt-sanlock 4.0.0-1ubuntu8.6 ppc64el Sanlock plugin for virtlockd ii libvirt-wireshark 4.0.0-1ubuntu8.6 ppc64el Wireshark dissector for the libvirt protocol ii libvirt0:ppc64el 4.0.0-1ubuntu8.6 ppc64el library for interfacing with different virtualization systems ii python-libvirt 4.0.0-1 ppc64el libvirt Python bindings ---uname output--- 4.15.0-1016.18-fix1-ibm-gt Machine Type = Witherspoon ---Debugger--- A debugger is not configured Userspace tool common name: Qemu The userspace tool has the following bit modes: ppc64le Userspace rpm: ii qemu-system-ppc 1:2.11 +dfsg-1ubuntu7.8-1ibm3 ppc64el QEMU full system emulation binaries (ppc) Userspace tool obtained from project website: na *Additional Instructions for Balamuruhan S / [email protected]: -Post a private note with access information to the machine that the bug is occuring on. -Attach ltrace and strace of userspace application. == Comment: #1 - Balamuruhan S <[email protected]> - 2019-03-12 08:06:38 == == Comment: #2 - Balamuruhan S <[email protected]> - 2019-03-12 08:08:43 == == Comment: #3 - Balamuruhan S <[email protected]> - 2019-03-12 08:09:18 == == Comment: #4 - Balamuruhan S <[email protected]> - 2019-03-12 08:10:26 == == Comment: #5 - Balamuruhan S <[email protected]> - 2019-03-12 08:12:10 == == Comment: #6 - Balamuruhan S <[email protected]> - 2019-03-12 08:19:15 == ** Affects: ubuntu-power-systems Importance: Undecided Status: Invalid ** Affects: qemu (Ubuntu) Importance: Undecided Assignee: Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) Status: Invalid ** Tags: architecture-ppc64le bugnameltc-176084 severity-high targetmilestone-inin--- ** Tags added: architecture-ppc64le bugnameltc-176084 severity-high targetmilestone-inin--- ** Changed in: ubuntu Assignee: (unassigned) => Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) ** Package changed: ubuntu => qemu (Ubuntu) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1820255 Title: VM crashed and went to shutoff state during MultiVM postcopy migration To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1820255/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
