[Bug 1192499] Re: virsh migration copy-storage-all fails with "Unable to read from monitor: Connection reset by peer"
Launchpad has imported 15 comments from the remote bug at https://bugzilla.redhat.com/show_bug.cgi?id=979411. If you reply to an imported comment from within Launchpad, your comment will be sent to the remote bug automatically. Read more about Launchpad's inter-bugtracker facilities at https://help.launchpad.net/InterBugTracking. On 2013-06-28T13:15:38+00:00 chandrashekar wrote: Created attachment 766573 Source-migrate-logs Description of problem: virsh migration copy-storage-all fails with "Unable to read from monitor: Connection reset by peer" and shut downs the guest on the source host. Kernel Version: 3.10.0-rc5+ Libvirt Version: 1.0.6 Qemu Version: 1.5.50 Steps to Reproduce: 1. Created the qemu-img create -f qcow2 vm.qcow2 11G on the destination host which is same as the source. 2. Started the guest on the source 3. Started the vncdisplay to monitor the guest 4. Initiated the migration "virsh migrate --live VM1 qemu+ssh://host-ip/system tcp://host-ip --verbose --copy-storage-all" 5. It started the copying the storage from souce to destination (conitinously monitored it was growing) 6. Guest on the destination was paused and was running on the source 7. At some point the VM on the source got shutdown and migration failed with "Unable to read from monitor: Connection reset by peer" Attached the libvirt debug logs. The debug logs shows : 2013-06-19 08:43:12.253+: 4026: debug : virEventPollInterruptLocked:716 : Interrupting 2013-06-19 08:43:12.253+: 4026: debug : virEventPollAddTimeout:248 : EVENT_POLL_ADD_TIMEOUT: timer=1 frequency=0 cb=0x7fe930baa960 opaque=(nil) ff=(nil) Note: The virsh live migration works fine with nfs storage from source to destination and vice versa. With libvirt 1.0.5 and qemu 1.5 also we were facing the same issue, but with that even "Live migration with nfs also was not working". Guest XML: VM1 47feb0e1-0c23-9be9-da12-2ead34864de2 4096000 2048000 1 hvm destroy restart restart /usr/local/bin/qemu-system-x86_64 Reply at: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/comments/2 On 2013-06-28T13:27:20+00:00 Michal wrote: Can you provide the destination debug logs? Esp. content of /var/log/libvirt/qemu/VM1.log as there's supposed to be the reason why qemu died. Reply at: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/comments/3 On 2013-06-28T13:59:18+00:00 chandrashekar wrote: Created attachment 766578 Source Logs Reply at: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/comments/4 On 2013-06-28T13:59:51+00:00 chandrashekar wrote: Created attachment 766579 Destination Logs Reply at: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/comments/5 On 2013-06-28T14:20:28+00:00 Michal wrote: >From the VM1_dest.log: ... Completed 97 % Completed 98 % Completed 99 % qemu: warning: error while loading state section id 1 load of migration failed So the qemu is unable to migrate itself. Therefore I think this is actually a qemu bug. On the other hand, I wonder why the guest on the source is shut down. There's no sign of that in the logs. Reply at: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/comments/6 On 2013-07-01T10:02:08+00:00 chandrashekar wrote: When the migration fails the guest gets shutdown on the source. [root@9 images]# virsh list --all IdName State 5 VM1 running [root@9 images]# virsh migrate --live rhel64-64 qemu+ssh:/IP/system tcp://IP --verbose --copy-storage-all Migration: [ 93 %]error: Unable to read from monitor: Connection reset by peer At the destination: [root@9 images]# virsh list --all IdName State - VM1 shut off [root@destination]# virsh list --all IdName State 16VM1 paused [root@destination]# virsh list --all IdName State Reply at:
[Bug 1192499] Re: virsh migration copy-storage-all fails with "Unable to read from monitor: Connection reset by peer"
There hasn't been a reply to my question in the last comment within months, so I assume nobody cares about this anymore. So I'm closing this ticket now... ** Changed in: qemu Status: Incomplete => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1192499 Title: virsh migration copy-storage-all fails with "Unable to read from monitor: Connection reset by peer" To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1192499/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1192499] Re: virsh migration copy-storage-all fails with "Unable to read from monitor: Connection reset by peer"
Is there still something left to do here, or could we close this ticket nowadays? ** Changed in: qemu Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1192499 Title: virsh migration copy-storage-all fails with "Unable to read from monitor: Connection reset by peer" To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1192499/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1192499] Re: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer
migrate -b tcp :destination-ip:1234 There should be no space between tcp and the rest of the connection details: migrate -b tcp:destination-ip:1234 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1192499 Title: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1192499/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1192499] Re: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer
On 07/05/2013 01:52 PM, Stefan Hajnoczi wrote: The destination VM's log says: qemu: warning: error while loading state section id 1 This indicates that either there was an error on the destination while loading state or the migration stream got out of sync. Please check that QEMU on source and destination are identical. If you are running different versions of QEMU on source and destination this could be the cause. Try with qemu.git/master and a simple QEMU command-line (without libvirt): qemu-system-x86_64 -machine pc-i440fx-1.5,accel=kvm -m 4000 \ -drive file=/home/images/rhel64-64.qcow2,if=ide,format=qcow2,cache=none Use the same command-line on the destination but also add -incoming tcp::1234. To start the migrate on the source, run migrate -b tcp :destination-ip:1234 in the QEMU monitor. I tried this migrate -b tcp :destination-ip:1234 comes out and gives me the qemu prompt [ I mean it doesn't wait till the migration completes] and also on the destination the image is not growing, the status shows paused. ~Shastri If the failure can be reproduce on qemu.git/master in this way it will be easier to debug. I will be away next week and therefore unable to look into this more. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1192499 Title: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1192499/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1192499] Re: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer
** Attachment added: source guest logs https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/+attachment/3725328/+files/source.log.tar.bz2 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1192499 Title: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1192499/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1192499] Re: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer
** Attachment added: Source Libvirtd logs https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/+attachment/3725330/+files/source_libvirtd.logs -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1192499 Title: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1192499/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1192499] Re: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer
** Attachment added: Destination guest logs https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/+attachment/3725329/+files/dest.tar.bz2 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1192499 Title: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1192499/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1192499] Re: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer
** Attachment added: Destination Libvirtd logs https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/+attachment/3725331/+files/destination_libvirtd.logs -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1192499 Title: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1192499/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1192499] Re: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer
The destination VM's log says: qemu: warning: error while loading state section id 1 This indicates that either there was an error on the destination while loading state or the migration stream got out of sync. Please check that QEMU on source and destination are identical. If you are running different versions of QEMU on source and destination this could be the cause. Try with qemu.git/master and a simple QEMU command-line (without libvirt): qemu-system-x86_64 -machine pc-i440fx-1.5,accel=kvm -m 4000 \ -drive file=/home/images/rhel64-64.qcow2,if=ide,format=qcow2,cache=none Use the same command-line on the destination but also add -incoming tcp::1234. To start the migrate on the source, run migrate -b tcp :destination-ip:1234 in the QEMU monitor. If the failure can be reproduce on qemu.git/master in this way it will be easier to debug. I will be away next week and therefore unable to look into this more. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1192499 Title: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1192499/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1192499] Re: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer
Thanks, I ll work on this and update it. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1192499 Title: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1192499/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1192499] Re: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer
Moving to qemu component as qemu is crashing based on the inputs from Michal Privoznik Bugzilla : Bug 979411 - virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer ** Project changed: libvirt = qemu ** Bug watch added: Red Hat Bugzilla #979411 https://bugzilla.redhat.com/show_bug.cgi?id=979411 ** Also affects: libvirt (Fedora) via https://bugzilla.redhat.com/show_bug.cgi?id=979411 Importance: Unknown Status: Unknown -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1192499 Title: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1192499/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1192499] Re: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer
I am sorry. Host Kernel : 3.10.0-rc5+ [upstream kernel fedora base], both the source and destination host are the same. We test the upstream kernel, qemu and libvirt. We don't have ubuntu. Oh, I see - then please see http://libvirt.org/bugs.html (specifically General libvirt bug reports) for where to file upstream bug reports. ** Also affects: libvirt Importance: Undecided Status: New ** Changed in: libvirt (Ubuntu) Status: Incomplete = Invalid -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/1192499 Title: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer To manage notifications about this bug go to: https://bugs.launchpad.net/libvirt/+bug/1192499/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1192499] Re: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer
** Attachment added: Source sosreport https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/+attachment/3720183/+files/sosreport-source.1192499-20130701151939-cc68.tar.xz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1192499 Title: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1192499] Re: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer
We tried with emulator/usr/bin/qemu-system-x86_64/emulator and we are facing the same issue. For both source and destination the fs is ext4. We are using fedora base, hence I have attached the sosreport of both source and destination. ** Attachment added: Destination sosreport https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/+attachment/3720182/+files/sosreport-destination.1192499-20130701054850-8f87.tar.xz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1192499 Title: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1192499] Re: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer
On 06/29/2013 12:15 AM, Serge Hallyn wrote: Thanks for submitting this bug. I can' t reproduce it with an empty image (sitting at cd boot menu). What is the underlying fs (/home/images) on both source and destination? For both source and destination the fs is ext4. Could you run 'apport-collect 1192499' on the destination host? We are using fedora base, hence I have attached the sosreport of both source and destination in the bug. Oh, actually I notice you have /usr/local/bin/qemu-system-x86_64 listed as the emulator in the .xml. That is not the qemu-system-x86_64 shipped with the qemu-system-x86 package. Does switching that for emulator/usr/bin/qemu-system-x86_64/emulator We tried with emulator/usr/bin/qemu-system-x86_64/emulator and we are facing the same issue. and see if that helps matters? ** Changed in: libvirt (Ubuntu) Status: New = Incomplete Thanks, Shastri -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1192499 Title: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1192499] Re: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer
Hi, I'm confused - what do you mean by you are using a fedora base? Are you talking about the VM, or the destination host? If the destination host, then (a) is it identical to the source host, and either way (b) exactly how did you set up the hosts with ubuntu packages? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1192499 Title: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1192499] Re: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer
I am sorry. Host Kernel : 3.10.0-rc5+ [upstream kernel fedora base], both the source and destination host are the same. We test the upstream kernel, qemu and libvirt. We don't have ubuntu. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1192499 Title: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1192499] Re: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer
I am sorry. Host Kernel : 3.10.0-rc5+ [upstream kernel fedora base], both the source and destination host are the same. We test the upstream kernel, qemu and libvirt. We don't have ubuntu. Oh, I see - then please see http://libvirt.org/bugs.html (specifically General libvirt bug reports) for where to file upstream bug reports. ** Also affects: libvirt Importance: Undecided Status: New ** Changed in: libvirt (Ubuntu) Status: Incomplete = Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1192499 Title: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer To manage notifications about this bug go to: https://bugs.launchpad.net/libvirt/+bug/1192499/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1192499] Re: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer
Thanks for submitting this bug. I can' t reproduce it with an empty image (sitting at cd boot menu). What is the underlying fs (/home/images) on both source and destination? Could you run 'apport-collect 1192499' on the destination host? Oh, actually I notice you have /usr/local/bin/qemu-system-x86_64 listed as the emulator in the .xml. That is not the qemu-system-x86_64 shipped with the qemu-system-x86 package. Does switching that for emulator/usr/bin/qemu-system-x86_64/emulator and see if that helps matters? ** Changed in: libvirt (Ubuntu) Status: New = Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1192499 Title: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1192499] Re: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer
** Changed in: libvirt (Ubuntu) Importance: Undecided = Medium -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1192499 Title: virsh migration copy-storage-all fails with Unable to read from monitor: Connection reset by peer To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
