Public bug reported:
== Comment: #0 - Balamuruhan S <[email protected]> - 2019-03-12 07:21:48
==
---Problem Description---
MultiVM postcopy migration fails with "operation failed: migration job:
unexpectedly failed" for some of the VMs.
---Steps to Reproduce---
1. Place the guest image in NFS shared location and mount the same path in
source and target that can be accessed by guest.
[step2 to step5 can be achieved by defining Guest xmls that are attached]
2. 28 VMs with 32G memory, 4 vcpus (1 socket 1 core and 4 threads) [Host with
1TB memory]
3. Each vcpu pinned to each hostcpu from 0 to 111
4. Emulator pinned with 4 vms with 1 host cpu from 112 to 118
5. create 28 VFs from available 2 PFs available in the host and give it to guest
6. create VFs by echo 28 to PFs available
# echo 28 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs
# echo 28 > /sys/bus/pci/devices/0000:01:00.1/sriov_numvfs
7. stress-ng with 90% work load inside guest VM
```
# stress-ng --all 4 --vm-bytes 90% --aggressive --maximize --oomable --timeout
86400 --verify --syslog --metrics --times
```
8. perform MultiVM postcopy migration in parallel all 28 guest from source to
target.
9. It is observed that migration fails with "operation failed: migration job:
unexpectedly failed" for some of the VMs
Attachment:
1. Guest XMLs
2. libvirtd debug logs for source and target
3. sosreport for source and target
Contact Information = Balamuruhan S / [email protected]
---System configuration---
Kernel Version:
Target:
# uname -a
Linux pok1-qz1-sr1-rk005-s02.pok.stglabs.ibm.com 4.15.0-1016.18-fix1-ibm-gt
#fix1 SMP Wed Feb 20 15:48:47 CST 2019 ppc64le ppc64le ppc64le GNU/Linux
Source:
# uname -a
Linux pok1-qz1-sr1-rk005-s03.pok.stglabs.ibm.com 4.15.0-1016.18-fix1-ibm-gt
#fix1 SMP Wed Feb 20 15:48:47 CST 2019 ppc64le ppc64le ppc64le GNU/Linux
Guest:
# uname -a
Linux ubuntu 4.15.0-1016.18-fix1-ibm-gt #fix1 SMP Wed Feb 20 15:48:47 CST 2019
ppc64le ppc64le ppc64le GNU/Linux
Qemu version in source and target hosts:
# dpkg -l | grep qemu
ii ipxe-qemu
1.0.0+git-20180124.fbe8c52d-0ubuntu2.2 all PXE boot firmware - ROM
images for qemu
ii ipxe-qemu-256k-compat-efi-roms
1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat
EFI ROM images for qemu
ii qemu 1:2.11+dfsg-1ubuntu7.8-1ibm3
ppc64el fast processor emulator
ii qemu-block-extra:ppc64el 1:2.11+dfsg-1ubuntu7.8-1ibm3
ppc64el extra block backend modules for qemu-system and
qemu-utils
ii qemu-slof 20170724+dfsg-1ubuntu1
all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system 1:2.11+dfsg-1ubuntu7.8-1ibm3
ppc64el QEMU full system emulation binaries
ii qemu-system-common 1:2.11+dfsg-1ubuntu7.8-1ibm3
ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-misc 1:2.11+dfsg-1ubuntu7.8-1ibm3
ppc64el QEMU full system emulation binaries (miscellaneous)
ii qemu-system-ppc 1:2.11+dfsg-1ubuntu7.8-1ibm3
ppc64el QEMU full system emulation binaries (ppc)
ii qemu-system-x86 1:2.11+dfsg-1ubuntu7.10
ppc64el QEMU full system emulation binaries (x86)
ii qemu-user 1:2.11+dfsg-1ubuntu7.8-1ibm3
ppc64el QEMU user mode emulation binaries
ii qemu-user-binfmt 1:2.11+dfsg-1ubuntu7.8-1ibm3
ppc64el QEMU user mode binfmt registration for qemu-user
ii qemu-utils 1:2.11+dfsg-1ubuntu7.8-1ibm3
ppc64el QEMU utilities
Libvirt version in source and target hosts:
# dpkg -l | grep libvirt
ii gir1.2-libvirt-glib-1.0:ppc64el 1.0.0-1
ppc64el GObject introspection files for the libvirt-glib library
ii gir1.2-libvirt-sandbox-1.0 0.5.1+git20160404-1
ppc64el GObject introspection files for the libvirt-sandbox
library
ii libvirt-bin 4.0.0-1ubuntu8.6
ppc64el programs for the libvirt library
ii libvirt-clients 4.0.0-1ubuntu8.6
ppc64el Programs for the libvirt library
ii libvirt-daemon 4.0.0-1ubuntu8.6
ppc64el Virtualization daemon
ii libvirt-daemon-driver-storage-gluster 4.0.0-1ubuntu8.6
ppc64el Virtualization daemon glusterfs storage driver
ii libvirt-daemon-driver-storage-rbd 4.0.0-1ubuntu8.6
ppc64el Virtualization daemon RBD storage driver
ii libvirt-daemon-driver-storage-sheepdog 4.0.0-1ubuntu8.6
ppc64el Virtualization daemon Sheedog storage driver
ii libvirt-daemon-driver-storage-zfs 4.0.0-1ubuntu8.6
ppc64el Virtualization daemon ZFS storage driver
ii libvirt-daemon-system 4.0.0-1ubuntu8.6
ppc64el Libvirt daemon configuration files
ii libvirt-dev:ppc64el 4.0.0-1ubuntu8.6
ppc64el development files for the libvirt library
ii libvirt-doc 4.0.0-1ubuntu8.6
all documentation for the libvirt library
ii libvirt-glib-1.0-0:ppc64el 1.0.0-1
ppc64el libvirt GLib and GObject mapping library
ii libvirt-glib-1.0-dev:ppc64el 1.0.0-1
ppc64el Development files for the libvirt-glib library
ii libvirt-ocaml 0.6.1.4-2build1
ppc64el OCaml bindings for libvirt (runtime)
ii libvirt-ocaml-dev 0.6.1.4-2build1
ppc64el OCaml bindings for libvirt (development files)
ii libvirt-sandbox-1.0-5 0.5.1+git20160404-1
ppc64el Application sandbox toolkit shared library
ii libvirt-sandbox-1.0-dev 0.5.1+git20160404-1
ppc64el Development files for libvirt-sandbox library
ii libvirt-sanlock 4.0.0-1ubuntu8.6
ppc64el Sanlock plugin for virtlockd
ii libvirt-wireshark 4.0.0-1ubuntu8.6
ppc64el Wireshark dissector for the libvirt protocol
ii libvirt0:ppc64el 4.0.0-1ubuntu8.6
ppc64el library for interfacing with different virtualization
systems
ii python-libvirt 4.0.0-1
ppc64el libvirt Python bindings
---uname output---
4.15.0-1016.18-fix1-ibm-gt
Machine Type = Witherspoon
---Debugger---
A debugger is not configured
Userspace tool common name: Libvirt / Qemu
The userspace tool has the following bit modes: ppc64le
Userspace rpm:
ii qemu-system-ppc 1:2.11+dfsg-1ubuntu7.8-1ibm3
ppc64el QEMU full system emulation binaries (ppc)
ii libvirt-bin 4.0.0-1ubuntu8.6
ppc64el programs for the libvirt library
Userspace tool obtained from project website: na
*Additional Instructions for Balamuruhan S / [email protected]:
-Post a private note with access information to the machine that the bug is
occuring on.
-Attach ltrace and strace of userspace application.
== Comment: #1 - Balamuruhan S <[email protected]> - 2019-03-12
07:22:39 ==
== Comment: #2 - Balamuruhan S <[email protected]> - 2019-03-12 07:25:02
==
== Comment: #3 - Balamuruhan S <[email protected]> - 2019-03-12 07:25:47
==
== Comment: #4 - Balamuruhan S <[email protected]> - 2019-03-12 07:26:14
==
== Comment: #5 - Balamuruhan S <[email protected]> - 2019-03-12 07:26:36
==
** Affects: libvirt (Ubuntu)
Importance: Undecided
Assignee: Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage)
Status: New
** Tags: architecture-ppc64le bugnameltc-176082 severity-high
targetmilestone-inin---
** Tags added: architecture-ppc64le bugnameltc-176082 severity-high
targetmilestone-inin---
** Changed in: ubuntu
Assignee: (unassigned) => Ubuntu on IBM Power Systems Bug Triage
(ubuntu-power-triage)
** Package changed: ubuntu => libvirt (Ubuntu)
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1820264
Title:
MultiVM postcopy migration fails with "operation failed: migration
job: unexpectedly failed"
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1820264/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
