Bug#964793: odd qemu/xen crashes + toolchain rings a bell
On Wed, 15 Jul 2020 at 14:41, 小太 wrote: > > On Tue, 14 Jul 2020 at 19:21, 小太 wrote: > > > > On Tue, 14 Jul 2020 at 00:19, Hans van Kranenburg wrote: > > > 小太, can you do... > > > > > > xl create -vvv > > > > > > ...which should show how qemu is invoked. Can you show that command? > > > > > > I can provide you with some test packages with the mentioned upstream > > > patch applied (on top of 4.11.4+24-gddaaccbbab-1), so you can test if > > > your domU starts with them. > > > > > > If so, we can request the backport upstream and/or maybe pick it for > > > Debian 4.11 into the patch queue, whatever happens earlier. > > > > I've updated Xen to 4.11.4+24-gddaaccbbab-1 now, but qemu is still at > > 1:5.0-5. > > I'll update qemu again to 1:5.0-6 tomorrow and run a test > > I've tried qemu 1:5.0-6 with xen 4.11.4+24-gddaaccbbab-1 today, and > attached is the full output of: xl -vvv create windows.xen.cfg -F > It's a bit too long to directly include in this email :/ > > The contents on windows.xen.cfg for this run were: > > name = "windows" > builder = "hvm" > vcpus = "4" > memory = "1024" > boot = "c" And attached is the same command and config output, but with qemu 1:5.0-5 There appears to be no practical difference between the two logs (at least before the crash) Parsing config from windows.xen.cfg libxl: debug: libxl_create.c:1671:do_domain_create: Domain 0:ao 0x5649af511270: create: how=(nil) callback=(nil) poller=0x5649af511300 libxl: debug: libxl_create.c:1007:initiate_domain_create: Domain 1:running bootloader libxl: debug: libxl_bootloader.c:328:libxl__bootloader_run: Domain 1:not a PV/PVH domain, skipping bootloader libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=0x5649af5165f0: deregister unregistered domainbuilder: detail: xc_dom_allocate: cmdline="", features="" domainbuilder: detail: xc_dom_kernel_file: filename="/usr/lib/xen-4.11/boot/hvmloader" domainbuilder: detail: xc_dom_malloc_filemap: 174 kB libxl: debug: libxl_dom.c:973:libxl__load_hvm_firmware_module: Loading BIOS: /usr/share/seabios/bios-256k.bin domainbuilder: detail: xc_dom_boot_xen_init: ver 4.11, caps xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 domainbuilder: detail: xc_dom_parse_image: called domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ... domainbuilder: detail: loader probe failed domainbuilder: detail: xc_dom_find_loader: trying HVM-generic loader ... domainbuilder: detail: loader probe OK xc: detail: ELF: phdr: paddr=0x10 memsz=0x35104 xc: detail: ELF: memory: 0x10 -> 0x135104 domainbuilder: detail: xc_dom_mem_init: mem 1016 MB, pages 0x3f800 pages, 4k each domainbuilder: detail: xc_dom_mem_init: 0x3f800 pages domainbuilder: detail: xc_dom_boot_mem_init: called domainbuilder: detail: range: start=0x0 end=0x3f80 domainbuilder: detail: xc_dom_malloc: 2032 kB xc: detail: PHYSICAL MEMORY ALLOCATION: xc: detail: 4KB PAGES: 0x0400 xc: detail: 2MB PAGES: 0x01fa xc: detail: 1GB PAGES: 0x domainbuilder: detail: xc_dom_build_image: called domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x100+0x36 at 0x7f39b5a7a000 domainbuilder: detail: xc_dom_alloc_segment: kernel : 0x10 -> 0x136000 (pfn 0x100 + 0x36 pages) xc: detail: ELF: phdr 0 at 0x7f39b5a44000 -> 0x7f39b5a6f680 domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x136+0x40 at 0x7f39b5a3a000 domainbuilder: detail: xc_dom_alloc_segment: System Firmware module : 0x136000 -> 0x176000 (pfn 0x136 + 0x40 pages) domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x176+0x1 at 0x7f39b60b1000 domainbuilder: detail: xc_dom_alloc_segment: HVM start info : 0x176000 -> 0x177000 (pfn 0x176 + 0x1 pages) domainbuilder: detail: alloc_pgtables_hvm: doing nothing domainbuilder: detail: xc_dom_build_image : virt_alloc_end : 0x177000 domainbuilder: detail: xc_dom_build_image : virt_pgtab_end : 0x0 domainbuilder: detail: xc_dom_boot_image: called domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_64 domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_32p domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32 <= matches domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32p domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_64 domainbuilder: detail: domain builder memory footprint domainbuilder: detail:allocated domainbuilder: detail: malloc : 2037 kB domainbuilder: detail: anon mmap : 0 bytes domainbuilder: detail:mapped domainbuilder: detail: file mmap : 174 kB domainbuilder: detail: domU mmap : 476 kB domainbuilder: detail: vcpu_hvm: called domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff000 domainbuilder: detail: xc_dom_gnttab_hvm_seed: called,
Bug#964793: odd qemu/xen crashes + toolchain rings a bell
On Tue, 14 Jul 2020 at 19:21, 小太 wrote: > > On Tue, 14 Jul 2020 at 00:19, Hans van Kranenburg wrote: > > 小太, can you do... > > > > xl create -vvv > > > > ...which should show how qemu is invoked. Can you show that command? > > > > I can provide you with some test packages with the mentioned upstream > > patch applied (on top of 4.11.4+24-gddaaccbbab-1), so you can test if > > your domU starts with them. > > > > If so, we can request the backport upstream and/or maybe pick it for > > Debian 4.11 into the patch queue, whatever happens earlier. > > I've updated Xen to 4.11.4+24-gddaaccbbab-1 now, but qemu is still at 1:5.0-5. > I'll update qemu again to 1:5.0-6 tomorrow and run a test I've tried qemu 1:5.0-6 with xen 4.11.4+24-gddaaccbbab-1 today, and attached is the full output of: xl -vvv create windows.xen.cfg -F It's a bit too long to directly include in this email :/ The contents on windows.xen.cfg for this run were: name = "windows" builder = "hvm" vcpus = "4" memory = "1024" boot = "c" Parsing config from windows.xen.cfg libxl: debug: libxl_create.c:1671:do_domain_create: Domain 0:ao 0x55aad1cfc270: create: how=(nil) callback=(nil) poller=0x55aad1cfc300 libxl: debug: libxl_create.c:1007:initiate_domain_create: Domain 2:running bootloader libxl: debug: libxl_bootloader.c:328:libxl__bootloader_run: Domain 2:not a PV/PVH domain, skipping bootloader libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=0x55aad1d015f0: deregister unregistered domainbuilder: detail: xc_dom_allocate: cmdline="", features="" domainbuilder: detail: xc_dom_kernel_file: filename="/usr/lib/xen-4.11/boot/hvmloader" domainbuilder: detail: xc_dom_malloc_filemap: 174 kB libxl: debug: libxl_dom.c:973:libxl__load_hvm_firmware_module: Loading BIOS: /usr/share/seabios/bios-256k.bin domainbuilder: detail: xc_dom_boot_xen_init: ver 4.11, caps xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 domainbuilder: detail: xc_dom_parse_image: called domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ... domainbuilder: detail: loader probe failed domainbuilder: detail: xc_dom_find_loader: trying HVM-generic loader ... domainbuilder: detail: loader probe OK xc: detail: ELF: phdr: paddr=0x10 memsz=0x35104 xc: detail: ELF: memory: 0x10 -> 0x135104 domainbuilder: detail: xc_dom_mem_init: mem 1016 MB, pages 0x3f800 pages, 4k each domainbuilder: detail: xc_dom_mem_init: 0x3f800 pages domainbuilder: detail: xc_dom_boot_mem_init: called domainbuilder: detail: range: start=0x0 end=0x3f80 domainbuilder: detail: xc_dom_malloc: 2032 kB xc: detail: PHYSICAL MEMORY ALLOCATION: xc: detail: 4KB PAGES: 0x0400 xc: detail: 2MB PAGES: 0x01fa xc: detail: 1GB PAGES: 0x domainbuilder: detail: xc_dom_build_image: called domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x100+0x36 at 0x7f3680f56000 domainbuilder: detail: xc_dom_alloc_segment: kernel : 0x10 -> 0x136000 (pfn 0x100 + 0x36 pages) xc: detail: ELF: phdr 0 at 0x7f3680f2 -> 0x7f3680f4b680 domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x136+0x40 at 0x7f3680f16000 domainbuilder: detail: xc_dom_alloc_segment: System Firmware module : 0x136000 -> 0x176000 (pfn 0x136 + 0x40 pages) domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x176+0x1 at 0x7f368158d000 domainbuilder: detail: xc_dom_alloc_segment: HVM start info : 0x176000 -> 0x177000 (pfn 0x176 + 0x1 pages) domainbuilder: detail: alloc_pgtables_hvm: doing nothing domainbuilder: detail: xc_dom_build_image : virt_alloc_end : 0x177000 domainbuilder: detail: xc_dom_build_image : virt_pgtab_end : 0x0 domainbuilder: detail: xc_dom_boot_image: called domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_64 domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_32p domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32 <= matches domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32p domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_64 domainbuilder: detail: domain builder memory footprint domainbuilder: detail:allocated domainbuilder: detail: malloc : 2037 kB domainbuilder: detail: anon mmap : 0 bytes domainbuilder: detail:mapped domainbuilder: detail: file mmap : 174 kB domainbuilder: detail: domU mmap : 476 kB domainbuilder: detail: vcpu_hvm: called domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff000 domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff001 domainbuilder: detail: xc_dom_release: called libxl: debug: libxl_dm.c:1655:libxl__build_device_model_args_new: Domain 2:dm_restrict disabled, starting QEMU as root libxl: debug: libxl_dm.c:2331:libxl__spawn_local_dm: Domain 2:Spawning device-model
Bug#964793: odd qemu/xen crashes + toolchain rings a bell
On Tue, 14 Jul 2020 at 00:19, Hans van Kranenburg wrote: > 小太, can you do... > > xl create -vvv > > ...which should show how qemu is invoked. Can you show that command? > > I can provide you with some test packages with the mentioned upstream > patch applied (on top of 4.11.4+24-gddaaccbbab-1), so you can test if > your domU starts with them. > > If so, we can request the backport upstream and/or maybe pick it for > Debian 4.11 into the patch queue, whatever happens earlier. I've updated Xen to 4.11.4+24-gddaaccbbab-1 now, but qemu is still at 1:5.0-5. I'll update qemu again to 1:5.0-6 tomorrow and run a test, but for now, here's the relevant output with 1:5.0-5 I'd be happy to run any custom version of qemu you have in mind libxl: debug: libxl_dm.c:2331:libxl__spawn_local_dm: Domain 2:Spawning device-model /usr/bin/qemu-system-i386 with arguments: libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: /usr/bin/qemu-system-i386 libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: -xen-domid libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: 2 libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: -chardev libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-2,server,nowait libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: -no-shutdown libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: -mon libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: chardev=libxl-cmd,mode=control libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: -chardev libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: socket,id=libxenstat-cmd,path=/var/run/xen/qmp-libxenstat-2,server,nowait libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: -mon libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: chardev=libxenstat-cmd,mode=control libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: -nodefaults libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: -no-user-config libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: -name libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: windows2 libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: -vnc libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: 127.0.0.1:0,to=99 libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: -display libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: none libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: -device libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: cirrus-vga,vgamem_mb=8 libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: -boot libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: order=c libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: -smp libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: 4,maxcpus=4 libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: -net libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: none libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: -machine libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: xenfv libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: -m libxl: debug: libxl_dm.c:2333:libxl__spawn_local_dm: Domain 2: 504 libxl: debug: libxl_dm.c:2335:libxl__spawn_local_dm: Domain 2:Spawning device-model /usr/bin/qemu-system-i386 with additional environment: libxl: debug: libxl_dm.c:2337:libxl__spawn_local_dm: Domain 2: XEN_QEMU_CONSOLE_LIMIT=1048576
Bug#964793: odd qemu/xen crashes + toolchain rings a bell
However, On 7/13/20 4:19 PM, Hans van Kranenburg wrote: > (Adding more To:; Note that mailing the bug number does not make it end > up at the submitter automatically, only the package maintainer). > > Hi Christian, > > thanks for the hints! > > On Mon, 13 Jul 2020 09:01:18 +0200 Christian Ehrhardt > wrote: >> Hi, >> I was seeing the bug updates flying by and just wanted to mention that we >> have seen something similar in Ubuntu - but back then things weren't >> replicable on Debian so we couldn't contribute things back. >> It seemed to be due to the newer and different-defaults toolchain that we >> had in Ubuntu at the time. >> >> But here qemu/xen crashes + new toolchain come together again which >> reminded me. >> >> So without any promises that it really is related I wanted to FYI you to >> these two fixes we needed for Xen: >> https://git.launchpad.net/ubuntu/+source/xen/tree/debian/patches/1001-strip-note-gnu-property.patch?h=ubuntu/groovy-devel > > I guess this first one would be one needed? "Force fcf-protection off > when using -mindirect-branch". > > In that case want this one, it's not backported to 4.11-stable: > > "x86/build: Unilaterally disable -fcf-protection" > > https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=3a218961b16f1f4feb1147f56338faf1ac8f5703 However, this is a workaround for a gcc bug that is fixed in: https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=a03efb266f This fix is included in gcc-9 in Debian since 9.3.0-12: https://salsa.debian.org/toolchain-team/gcc/-/blob/gcc-9-debian/debian/changelog#L55 (it's the PR target/93654 (x86)) Reporter says the 4.11.4-1 package is used, which is built using gcc 9.3.0-13: https://buildd.debian.org/status/fetch.php?pkg=xen=all=4.11.4-1=1590602099=0 >> https://git.launchpad.net/ubuntu/+source/xen/tree/debian/patches/1000-flags-fcs-protect-none.patch?h=ubuntu/groovy-devel > > This one is about the build failing. > >> This would seem more applicable if the new toolchain would have recently >> rebuilt xen and not qemu as in this case. But as an FYI it is still worth a >> ping. > > 小太, can you do... > > xl create -vvv > > ...which should show how qemu is invoked. Can you show that command? > > I can provide you with some test packages with the mentioned upstream > patch applied (on top of 4.11.4+24-gddaaccbbab-1), so you can test if > your domU starts with them. > > If so, we can request the backport upstream and/or maybe pick it for > Debian 4.11 into the patch queue, whatever happens earlier. So, the above info tells us that this probably is not the issue that we're looking at. (I'm fine with still making some test packages for reporter to test with to 100% check this.) Then, let's see what shows up in the xl -vvv output and if there's anything that can be debugged when starting the qemu process with those args? > Thanks, > Hans (Debian Xen Team) >
Bug#964793: odd qemu/xen crashes + toolchain rings a bell
(Adding more To:; Note that mailing the bug number does not make it end up at the submitter automatically, only the package maintainer). Hi Christian, thanks for the hints! On Mon, 13 Jul 2020 09:01:18 +0200 Christian Ehrhardt wrote: > Hi, > I was seeing the bug updates flying by and just wanted to mention that we > have seen something similar in Ubuntu - but back then things weren't > replicable on Debian so we couldn't contribute things back. > It seemed to be due to the newer and different-defaults toolchain that we > had in Ubuntu at the time. > > But here qemu/xen crashes + new toolchain come together again which > reminded me. > > So without any promises that it really is related I wanted to FYI you to > these two fixes we needed for Xen: > https://git.launchpad.net/ubuntu/+source/xen/tree/debian/patches/1001-strip-note-gnu-property.patch?h=ubuntu/groovy-devel I guess this first one would be one needed? "Force fcf-protection off when using -mindirect-branch". In that case want this one, it's not backported to 4.11-stable: "x86/build: Unilaterally disable -fcf-protection" https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=3a218961b16f1f4feb1147f56338faf1ac8f5703 > https://git.launchpad.net/ubuntu/+source/xen/tree/debian/patches/1000-flags-fcs-protect-none.patch?h=ubuntu/groovy-devel This one is about the build failing. > This would seem more applicable if the new toolchain would have recently > rebuilt xen and not qemu as in this case. But as an FYI it is still worth a > ping. 小太, can you do... xl create -vvv ...which should show how qemu is invoked. Can you show that command? I can provide you with some test packages with the mentioned upstream patch applied (on top of 4.11.4+24-gddaaccbbab-1), so you can test if your domU starts with them. If so, we can request the backport upstream and/or maybe pick it for Debian 4.11 into the patch queue, whatever happens earlier. Thanks, Hans (Debian Xen Team)
Bug#964793: odd qemu/xen crashes + toolchain rings a bell
Hi, I was seeing the bug updates flying by and just wanted to mention that we have seen something similar in Ubuntu - but back then things weren't replicable on Debian so we couldn't contribute things back. It seemed to be due to the newer and different-defaults toolchain that we had in Ubuntu at the time. But here qemu/xen crashes + new toolchain come together again which reminded me. So without any promises that it really is related I wanted to FYI you to these two fixes we needed for Xen: https://git.launchpad.net/ubuntu/+source/xen/tree/debian/patches/1001-strip-note-gnu-property.patch?h=ubuntu/groovy-devel https://git.launchpad.net/ubuntu/+source/xen/tree/debian/patches/1000-flags-fcs-protect-none.patch?h=ubuntu/groovy-devel This would seem more applicable if the new toolchain would have recently rebuilt xen and not qemu as in this case. But as an FYI it is still worth a ping. -- Christian Ehrhardt Staff Engineer, Ubuntu Server Canonical Ltd
