Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work
Hi, i now was able to catch the error. It is: Length mismatch: :00:12.0/virtio-net-pci.rom: 4 in != 1 qemu: warning: error while loading state for instance 0x0 of device 'ram' load of migration failed Stefan Am 09.05.2014 19:05, schrieb Paolo Bonzini: Il 09/05/2014 15:13, Stefan Priebe - Profihost AG ha scritto: I see no output at the monitor of Qemu 2.0. # migrate -d tcp:a.b.c.d:6000 # info migrate capabilities: xbzrle: off x-rdma-pin-all: off auto-converge: on zero-blocks: off Migration status: failed total time: 0 milliseconds The target machine is still running at this point with no output. Anything on its stdout? Another test you could do, in addition to changing the devices, is this: 1) try with commit 6141f3bd6904df7cf9519c6444a14a608b9874c4 on the destination (the next one caused a migration problem that was fixed later). If it passes, go to step 1a. If it fails, go to step 2. 1a) try with commit c01a71c1a56fa27f43449ff59e5d03b2483658a2. If it passes, go to step 1b. If it fails, report it here. 1b) bisect between v2.0.0 (known-bad) and c01a71c1a56fa27f43449ff59e5d03b2483658a2 (known-good) to find the culprit. Report results. 2) change the source to v1.7.0 and bisect between v1.7.0 and 6141f3bd6904df7cf9519c6444a14a608b9874c4. Report results. Thanks! Paolo
Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work
Il 14/05/2014 09:17, Stefan Priebe - Profihost AG ha scritto: i now was able to catch the error. It is: Length mismatch: :00:12.0/virtio-net-pci.rom: 4 in != 1 qemu: warning: error while loading state for instance 0x0 of device 'ram' load of migration failed This is a bug of your distribution. The file sizes for ROMs should never change. In particular, if you round the sizes up to the next power of 2 you should always have: 128k for bios.bin 256k for bios-256k.bin 64k for pxe-*.rom 256k for efi-*.rom 64k for vgabios-* Unfortunately, most distribution get pxe-*.rom sizes wrong, because at some point iPXE grew more features and didn't fit in 64k anymore with the default configuration. I know at least Fedora does. The solution is to copy the binaries from the QEMU git repository (directory pc-bios/) to /usr/share/qemu. Paolo
Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work
Am 14.05.2014 10:11, schrieb Paolo Bonzini: Il 14/05/2014 09:17, Stefan Priebe - Profihost AG ha scritto: i now was able to catch the error. It is: Length mismatch: :00:12.0/virtio-net-pci.rom: 4 in != 1 qemu: warning: error while loading state for instance 0x0 of device 'ram' load of migration failed This is a bug of your distribution. The file sizes for ROMs should never change. In particular, if you round the sizes up to the next power of 2 you should always have: 128k for bios.bin 256k for bios-256k.bin 64k for pxe-*.rom 256k for efi-*.rom 64k for vgabios-* Unfortunately, most distribution get pxe-*.rom sizes wrong, because at some point iPXE grew more features and didn't fit in 64k anymore with the default configuration. I know at least Fedora does. The solution is to copy the binaries from the QEMU git repository (directory pc-bios/) to /usr/share/qemu. Hi, i compile qemu on my own. I have the rom files under /usr/share/kvm and they look like this: ls -la /usr/share/kvm/*.rom -rw-r--r-- 1 root root 173568 May 14 09:39 /usr/share/kvm/efi-e1000.rom -rw-r--r-- 1 root root 174592 May 14 09:39 /usr/share/kvm/efi-eepro100.rom -rw-r--r-- 1 root root 173056 May 14 09:39 /usr/share/kvm/efi-ne2k_pci.rom -rw-r--r-- 1 root root 173056 May 14 09:39 /usr/share/kvm/efi-pcnet.rom -rw-r--r-- 1 root root 176640 May 14 09:39 /usr/share/kvm/efi-rtl8139.rom -rw-r--r-- 1 root root 171008 May 14 09:39 /usr/share/kvm/efi-virtio.rom -rw-r--r-- 1 root root 67072 May 14 09:39 /usr/share/kvm/pxe-e1000.rom -rw-r--r-- 1 root root 61440 May 14 09:39 /usr/share/kvm/pxe-eepro100.rom -rw-r--r-- 1 root root 61440 May 14 09:39 /usr/share/kvm/pxe-ne2k_pci.rom -rw-r--r-- 1 root root 61440 May 14 09:39 /usr/share/kvm/pxe-pcnet.rom -rw-r--r-- 1 root root 61440 May 14 09:39 /usr/share/kvm/pxe-rtl8139.rom -rw-r--r-- 1 root root 60416 May 14 09:39 /usr/share/kvm/pxe-virtio.rom currently i don't know what's wrong. Stefan
Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work
Il 14/05/2014 10:29, Stefan Priebe - Profihost AG ha scritto: Hi, i compile qemu on my own. I have the rom files under /usr/share/kvm and they look like this: ls -la /usr/share/kvm/*.rom -rw-r--r-- 1 root root 173568 May 14 09:39 /usr/share/kvm/efi-e1000.rom -rw-r--r-- 1 root root 174592 May 14 09:39 /usr/share/kvm/efi-eepro100.rom -rw-r--r-- 1 root root 173056 May 14 09:39 /usr/share/kvm/efi-ne2k_pci.rom -rw-r--r-- 1 root root 173056 May 14 09:39 /usr/share/kvm/efi-pcnet.rom -rw-r--r-- 1 root root 176640 May 14 09:39 /usr/share/kvm/efi-rtl8139.rom -rw-r--r-- 1 root root 171008 May 14 09:39 /usr/share/kvm/efi-virtio.rom -rw-r--r-- 1 root root 67072 May 14 09:39 /usr/share/kvm/pxe-e1000.rom -rw-r--r-- 1 root root 61440 May 14 09:39 /usr/share/kvm/pxe-eepro100.rom -rw-r--r-- 1 root root 61440 May 14 09:39 /usr/share/kvm/pxe-ne2k_pci.rom -rw-r--r-- 1 root root 61440 May 14 09:39 /usr/share/kvm/pxe-pcnet.rom -rw-r--r-- 1 root root 61440 May 14 09:39 /usr/share/kvm/pxe-rtl8139.rom -rw-r--r-- 1 root root 60416 May 14 09:39 /usr/share/kvm/pxe-virtio.rom currently i don't know what's wrong. What about the source machine? Paolo
Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work
Am 14.05.2014 10:36, schrieb Paolo Bonzini: Il 14/05/2014 10:29, Stefan Priebe - Profihost AG ha scritto: Hi, i compile qemu on my own. I have the rom files under /usr/share/kvm and they look like this: ls -la /usr/share/kvm/*.rom -rw-r--r-- 1 root root 173568 May 14 09:39 /usr/share/kvm/efi-e1000.rom -rw-r--r-- 1 root root 174592 May 14 09:39 /usr/share/kvm/efi-eepro100.rom -rw-r--r-- 1 root root 173056 May 14 09:39 /usr/share/kvm/efi-ne2k_pci.rom -rw-r--r-- 1 root root 173056 May 14 09:39 /usr/share/kvm/efi-pcnet.rom -rw-r--r-- 1 root root 176640 May 14 09:39 /usr/share/kvm/efi-rtl8139.rom -rw-r--r-- 1 root root 171008 May 14 09:39 /usr/share/kvm/efi-virtio.rom -rw-r--r-- 1 root root 67072 May 14 09:39 /usr/share/kvm/pxe-e1000.rom -rw-r--r-- 1 root root 61440 May 14 09:39 /usr/share/kvm/pxe-eepro100.rom -rw-r--r-- 1 root root 61440 May 14 09:39 /usr/share/kvm/pxe-ne2k_pci.rom -rw-r--r-- 1 root root 61440 May 14 09:39 /usr/share/kvm/pxe-pcnet.rom -rw-r--r-- 1 root root 61440 May 14 09:39 /usr/share/kvm/pxe-rtl8139.rom -rw-r--r-- 1 root root 60416 May 14 09:39 /usr/share/kvm/pxe-virtio.rom currently i don't know what's wrong. What about the source machine? Currently it has the same as i already updated the package there too. So you mean i had done a mistake compiling the old package - so it had wrong sizes? Greets. Stefan
Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work
Il 14/05/2014 10:38, Stefan Priebe - Profihost AG ha scritto: Currently it has the same as i already updated the package there too. So you mean i had done a mistake compiling the old package - so it had wrong sizes? Yes, probably. Can you do an info mtree for a machine that's running on the source? Paolo
Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work
Am 14.05.2014 11:00, schrieb Paolo Bonzini: Il 14/05/2014 10:38, Stefan Priebe - Profihost AG ha scritto: Currently it has the same as i already updated the package there too. So you mean i had done a mistake compiling the old package - so it had wrong sizes? Yes, probably. Can you do an info mtree for a machine that's running on the source? Paolo here it is: # info mtree memory -7ffe (prio 0, RW): system -dfff (prio 0, RW): alias ram-below-4g @pc.ram -dfff 000a-000b (prio 1, RW): alias smram-region @pci 000a-000b 000c-000c3fff (prio 1, R-): alias pam-rom @pc.ram 000c-000c3fff 000c4000-000c7fff (prio 1, R-): alias pam-rom @pc.ram 000c4000-000c7fff 000c8000-000cbfff (prio 1, R-): alias pam-rom @pc.ram 000c8000-000cbfff 000ca000-000ccfff (prio 1000, RW): alias kvmvapic-rom @pc.ram 000ca000-000ccfff 000cc000-000c (prio 1, R-): alias pam-rom @pc.ram 000cc000-000c 000d-000d3fff (prio 1, RW): alias pam-ram @pc.ram 000d-000d3fff 000d4000-000d7fff (prio 1, RW): alias pam-ram @pc.ram 000d4000-000d7fff 000d8000-000dbfff (prio 1, RW): alias pam-ram @pc.ram 000d8000-000dbfff 000dc000-000d (prio 1, RW): alias pam-ram @pc.ram 000dc000-000d 000e-000e3fff (prio 1, RW): alias pam-ram @pc.ram 000e-000e3fff 000e4000-000e7fff (prio 1, RW): alias pam-ram @pc.ram 000e4000-000e7fff 000e8000-000ebfff (prio 1, RW): alias pam-ram @pc.ram 000e8000-000ebfff 000ec000-000e (prio 1, RW): alias pam-ram @pc.ram 000ec000-000e 000f-000f (prio 1, R-): alias pam-rom @pc.ram 000f-000f e000- (prio 0, RW): alias pci-hole @pci e000- fec0-fec00fff (prio 0, RW): kvm-ioapic fed0-fed003ff (prio 0, RW): hpet fee0-feef (prio 4096, RW): icc-apic-container fee0-feef (prio 0, RW): kvm-apic-msi 0001-00021fff (prio 0, RW): alias ram-above-4g @pc.ram e000-0001 00022000-40021fff (prio 0, RW): alias pci-hole64 @pci 00022000-40021fff I/O - (prio 0, RW): io -0007 (prio 0, RW): dma-chan 0008-000f (prio 0, RW): dma-cont 0020-0021 (prio 0, RW): kvm-pic 0040-0043 (prio 0, RW): kvm-pit 0060-0060 (prio 0, RW): i8042-data 0061-0061 (prio 0, RW): elcr 0064-0064 (prio 0, RW): i8042-cmd 0070-0071 (prio 0, RW): rtc 007e-007f (prio 0, RW): kvmvapic 0080-0080 (prio 0, RW): ioport80 0081-0083 (prio 0, RW): alias dma-page @dma-page 0081-0083 0087-0087 (prio 0, RW): alias dma-page @dma-page 0087-0087 0089-008b (prio 0, RW): alias dma-page @dma-page 0089-008b 008f-008f (prio 0, RW): alias dma-page @dma-page 008f-008f 0092-0092 (prio 0, RW): port92 00a0-00a1 (prio 0, RW): kvm-pic 00b2-00b3 (prio 0, RW): apm-io 00c0-00cf (prio 0, RW): dma-chan 00d0-00df (prio 0, RW): dma-cont 00f0-00f0 (prio 0, RW): ioportF0 0170-0177 (prio 0, RW): alias ide @ide 0170-0177 01f0-01f7 (prio 0, RW): alias ide @ide 01f0-01f7 0376-0376 (prio 0, RW): alias ide @ide 0376-0376 03b0-03df (prio 0, RW): cirrus-io 03f1-03f5 (prio 0, RW): alias fdc @fdc 03f1-03f5 03f6-03f6 (prio 0, RW): alias ide @ide 03f6-03f6 03f7-03f7 (prio 0, RW): alias fdc @fdc 03f7-03f7 04d0-04d0 (prio 0, RW): kvm-elcr 04d1-04d1 (prio 0, RW): kvm-elcr 0505-0505 (prio 0, RW): pvpanic
Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work
Il 14/05/2014 11:25, Stefan Priebe - Profihost AG ha scritto: Currently it has the same as i already updated the package there too. So you mean i had done a mistake compiling the old package - so it had wrong sizes? Yes, probably. Can you do an info mtree for a machine that's running on the source? Paolo here it is: # info mtree Sorry, unfortunately it doesn't help. Paolo
Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work
* Stefan Priebe - Profihost AG (s.pri...@profihost.ag) wrote: Hello list, i was trying to migrate older Qemu (1.5 and 1.7.2) to a machine running Qemu 2.0. I started the target machine with: -machine type=pc-i440fx-1.5 / -machine type=pc-i440fx-1.7 I'd expect you to have to run with the same machine type on both sides. I ran some simple virt-test migrate tests (just the basic one) and got v1.5.3-v1.6.2 v1.5.3-v1.7.1 v1.5.3-v2.0.0-rc1 working for most machine types, pc-i440fx-1.5 passed with all of those. Note that's only the simplest test. Dave -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work
Am 09.05.2014 um 15:41 schrieb Dr. David Alan Gilbert dgilb...@redhat.com: * Stefan Priebe - Profihost AG (s.pri...@profihost.ag) wrote: Hello list, i was trying to migrate older Qemu (1.5 and 1.7.2) to a machine running Qemu 2.0. I started the target machine with: -machine type=pc-i440fx-1.5 / -machine type=pc-i440fx-1.7 I'd expect you to have to run with the same machine type on both sides. I ran some simple virt-test migrate tests (just the basic one) and got v1.5.3-v1.6.2 v1.5.3-v1.7.1 v1.5.3-v2.0.0-rc1 working for most machine types, pc-i440fx-1.5 passed with all of those. Note that's only the simplest test. My test involved USB Controller virtio Network cards and rbd virtio-scsi drives. Can you give me your simple start line so I could test if this works for me while adding some more arguments. Thanks! Stefan Dave -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work
* Stefan Priebe - Profihost AG (s.pri...@profihost.ag) wrote: Am 09.05.2014 um 15:41 schrieb Dr. David Alan Gilbert dgilb...@redhat.com: * Stefan Priebe - Profihost AG (s.pri...@profihost.ag) wrote: Hello list, i was trying to migrate older Qemu (1.5 and 1.7.2) to a machine running Qemu 2.0. I started the target machine with: -machine type=pc-i440fx-1.5 / -machine type=pc-i440fx-1.7 I'd expect you to have to run with the same machine type on both sides. I ran some simple virt-test migrate tests (just the basic one) and got v1.5.3-v1.6.2 v1.5.3-v1.7.1 v1.5.3-v2.0.0-rc1 working for most machine types, pc-i440fx-1.5 passed with all of those. Note that's only the simplest test. My test involved USB Controller virtio Network cards and rbd virtio-scsi drives. That probably makes things more interesting :-) I'd start with the simplest config you can find and add each one of those until it breaks. Can you give me your simple start line so I could test if this works for me while adding some more arguments. I've got a slightly hacked up libvirt, but I'm doing ./run -t qemu --machine-type=$MACTYPE -g Linux.JeOS.19.x86_64.i440fx --qemu-bin=$SRC --qemu-dst-bin=$DST --qemu_sandbox=off --tests migrate.default.tcp --disk-bus=ide and looking at the logs we have: /opt/qemu/v1.5.3/bin/qemu-system-x86_64 \ -S \ -name 'virt-tests-vm1' \ -sandbox off \ -M pc-i440fx-1.5 \ -nodefaults \ -vga std \ -chardev socket,id=hmp_id_hmp1,path=/tmp/monitor-hmp1-20140415-104517-rAfzfDef,server,nowait \ -mon chardev=hmp_id_hmp1,mode=readline \ -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20140415-104517-rAfzfDef,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -chardev socket,id=seabioslog_id_20140415-104517-rAfzfDef,path=/tmp/seabios-20140415-104517-rAfzfDef,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20140415-104517-rAfzfDef,iobase=0x402 \ -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=03 \ -drive id=drive_image1,if=none,file=/home/dgilbert/virt-test/shared/data/images/jeos-19-64.qcow2 \ -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=04 \ -device virtio-net-pci,mac=9a:18:19:1a:1b:1c,id=idz0uWFP,vectors=4,netdev=idMwmdMc,bus=pci.0,addr=05 \ -netdev user,id=idMwmdMc,hostfwd=tcp::5000-:22 \ -m 1024 \ -smp 2,maxcpus=2,cores=1,threads=1,sockets=2 \ -cpu 'SandyBridge' \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :1 \ -rtc base=utc,clock=host,driftfix=none \ -boot order=cdn,once=c,menu=off \ -enable-kvm Dave Thanks! Stefan Dave -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work
Il 09/05/2014 15:13, Stefan Priebe - Profihost AG ha scritto: I see no output at the monitor of Qemu 2.0. # migrate -d tcp:a.b.c.d:6000 # info migrate capabilities: xbzrle: off x-rdma-pin-all: off auto-converge: on zero-blocks: off Migration status: failed total time: 0 milliseconds The target machine is still running at this point with no output. Anything on its stdout? Another test you could do, in addition to changing the devices, is this: 1) try with commit 6141f3bd6904df7cf9519c6444a14a608b9874c4 on the destination (the next one caused a migration problem that was fixed later). If it passes, go to step 1a. If it fails, go to step 2. 1a) try with commit c01a71c1a56fa27f43449ff59e5d03b2483658a2. If it passes, go to step 1b. If it fails, report it here. 1b) bisect between v2.0.0 (known-bad) and c01a71c1a56fa27f43449ff59e5d03b2483658a2 (known-good) to find the culprit. Report results. 2) change the source to v1.7.0 and bisect between v1.7.0 and 6141f3bd6904df7cf9519c6444a14a608b9874c4. Report results. Thanks! Paolo
Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work
Am 09.05.2014 18:29, schrieb Dr. David Alan Gilbert: * Stefan Priebe - Profihost AG (s.pri...@profihost.ag) wrote: Am 09.05.2014 um 15:41 schrieb Dr. David Alan Gilbert dgilb...@redhat.com: * Stefan Priebe - Profihost AG (s.pri...@profihost.ag) wrote: Hello list, i was trying to migrate older Qemu (1.5 and 1.7.2) to a machine running Qemu 2.0. I started the target machine with: -machine type=pc-i440fx-1.5 / -machine type=pc-i440fx-1.7 I'd expect you to have to run with the same machine type on both sides. I ran some simple virt-test migrate tests (just the basic one) and got v1.5.3-v1.6.2 v1.5.3-v1.7.1 v1.5.3-v2.0.0-rc1 working for most machine types, pc-i440fx-1.5 passed with all of those. Note that's only the simplest test. My test involved USB Controller virtio Network cards and rbd virtio-scsi drives. That probably makes things more interesting :-) I'd start with the simplest config you can find and add each one of those until it breaks. Can you give me your simple start line so I could test if this works for me while adding some more arguments. I've got a slightly hacked up libvirt, but I'm doing ./run -t qemu --machine-type=$MACTYPE -g Linux.JeOS.19.x86_64.i440fx --qemu-bin=$SRC --qemu-dst-bin=$DST --qemu_sandbox=off --tests migrate.default.tcp --disk-bus=ide and looking at the logs we have: /opt/qemu/v1.5.3/bin/qemu-system-x86_64 \ -S \ -name 'virt-tests-vm1' \ -sandbox off \ -M pc-i440fx-1.5 \ -nodefaults \ -vga std \ -chardev socket,id=hmp_id_hmp1,path=/tmp/monitor-hmp1-20140415-104517-rAfzfDef,server,nowait \ -mon chardev=hmp_id_hmp1,mode=readline \ -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20140415-104517-rAfzfDef,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -chardev socket,id=seabioslog_id_20140415-104517-rAfzfDef,path=/tmp/seabios-20140415-104517-rAfzfDef,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20140415-104517-rAfzfDef,iobase=0x402 \ -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=03 \ -drive id=drive_image1,if=none,file=/home/dgilbert/virt-test/shared/data/images/jeos-19-64.qcow2 \ -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=04 \ -device virtio-net-pci,mac=9a:18:19:1a:1b:1c,id=idz0uWFP,vectors=4,netdev=idMwmdMc,bus=pci.0,addr=05 \ -netdev user,id=idMwmdMc,hostfwd=tcp::5000-:22 \ -m 1024 \ -smp 2,maxcpus=2,cores=1,threads=1,sockets=2 \ -cpu 'SandyBridge' \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :1 \ -rtc base=utc,clock=host,driftfix=none \ -boot order=cdn,once=c,menu=off \ -enable-kvm Dave OK biggest bummer it works fine with all freshly started VMs. No idea why. So i'll try with paolos idea migrating old VMs. Thanks! Stefan Dave -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work
) Am 09.05.2014 19:05, schrieb Paolo Bonzini: Il 09/05/2014 15:13, Stefan Priebe - Profihost AG ha scritto: I see no output at the monitor of Qemu 2.0. # migrate -d tcp:a.b.c.d:6000 # info migrate capabilities: xbzrle: off x-rdma-pin-all: off auto-converge: on zero-blocks: off Migration status: failed total time: 0 milliseconds The target machine is still running at this point with no output. Anything on its stdout? Sadly no. But far more complex than i thought. I'm seeing this problem only on machines originally started under qemu 1.5 ten migrated to qemu 1.7 with machine type=pc440fc-1.5 . Only those do not work. But they also do not work migrating them to qemu 1.7.1. So let's ignore the whole fact and thanks for your bisect ideas. I will just shut them down all and start them with a fresh qemu 2.0. Another test you could do, in addition to changing the devices, is this: 1) try with commit 6141f3bd6904df7cf9519c6444a14a608b9874c4 on the destination (the next one caused a migration problem that was fixed later). If it passes, go to step 1a. If it fails, go to step 2. 1a) try with commit c01a71c1a56fa27f43449ff59e5d03b2483658a2. If it passes, go to step 1b. If it fails, report it here. 1b) bisect between v2.0.0 (known-bad) and c01a71c1a56fa27f43449ff59e5d03b2483658a2 (known-good) to find the culprit. Report results. 2) change the source to v1.7.0 and bisect between v1.7.0 and 6141f3bd6904df7cf9519c6444a14a608b9874c4. Report results. Thanks! Paolo
