Re: [Xen-devel] [xen-unstable bisection] complete test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm
On Thu, Mar 02, 2017 at 10:42:57AM +, Andrew Cooper wrote: > On 02/03/17 10:41, Daniel Kiper wrote: > > On Wed, Mar 01, 2017 at 11:53:52PM +, osstest service owner wrote: > >> branch xen-unstable > >> xenbranch xen-unstable > >> job test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm > >> testid xen-boot > >> > >> Tree: linux git://xenbits.xen.org/linux-pvops.git > >> Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git > >> Tree: qemu git://xenbits.xen.org/qemu-xen-traditional.git > >> Tree: qemuu git://xenbits.xen.org/qemu-xen.git > >> Tree: xen git://xenbits.xen.org/xen.git > >> > >> *** Found and reproduced problem changeset *** > >> > >> Bug is in tree: xen git://xenbits.xen.org/xen.git > >> Bug introduced: c5b9805bc1f79319ae342c65fcc201a15a47 > >> Bug not present: b199c44afa3a0d18d0e968e78a590eb9e69e20ad > >> Last fail repro: http://logs.test-lab.xenproject.org/osstest/logs/106324/ > >> > >> > >> commit c5b9805bc1f79319ae342c65fcc201a15a47 > >> Author: Daniel Kiper> >> Date: Wed Feb 22 14:38:06 2017 +0100 > >> > >> efi: create new early memory allocator > > Does anybody know why this still fails? Andrew applied a fix (workaround) > > for this yesterday. > > Bisection runs from before will still be running. Also, the fix hasn't This is what I expected. > passed staging yet. Yep, I saw that. Thanks for update. Daniel ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] [xen-unstable bisection] complete test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm
On 02/03/17 10:41, Daniel Kiper wrote: > On Wed, Mar 01, 2017 at 11:53:52PM +, osstest service owner wrote: >> branch xen-unstable >> xenbranch xen-unstable >> job test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm >> testid xen-boot >> >> Tree: linux git://xenbits.xen.org/linux-pvops.git >> Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git >> Tree: qemu git://xenbits.xen.org/qemu-xen-traditional.git >> Tree: qemuu git://xenbits.xen.org/qemu-xen.git >> Tree: xen git://xenbits.xen.org/xen.git >> >> *** Found and reproduced problem changeset *** >> >> Bug is in tree: xen git://xenbits.xen.org/xen.git >> Bug introduced: c5b9805bc1f79319ae342c65fcc201a15a47 >> Bug not present: b199c44afa3a0d18d0e968e78a590eb9e69e20ad >> Last fail repro: http://logs.test-lab.xenproject.org/osstest/logs/106324/ >> >> >> commit c5b9805bc1f79319ae342c65fcc201a15a47 >> Author: Daniel Kiper>> Date: Wed Feb 22 14:38:06 2017 +0100 >> >> efi: create new early memory allocator > Does anybody know why this still fails? Andrew applied a fix (workaround) for > this yesterday. Bisection runs from before will still be running. Also, the fix hasn't passed staging yet. ~Andrew ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] [xen-unstable bisection] complete test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm
On Wed, Mar 01, 2017 at 11:53:52PM +, osstest service owner wrote: > branch xen-unstable > xenbranch xen-unstable > job test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm > testid xen-boot > > Tree: linux git://xenbits.xen.org/linux-pvops.git > Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git > Tree: qemu git://xenbits.xen.org/qemu-xen-traditional.git > Tree: qemuu git://xenbits.xen.org/qemu-xen.git > Tree: xen git://xenbits.xen.org/xen.git > > *** Found and reproduced problem changeset *** > > Bug is in tree: xen git://xenbits.xen.org/xen.git > Bug introduced: c5b9805bc1f79319ae342c65fcc201a15a47 > Bug not present: b199c44afa3a0d18d0e968e78a590eb9e69e20ad > Last fail repro: http://logs.test-lab.xenproject.org/osstest/logs/106324/ > > > commit c5b9805bc1f79319ae342c65fcc201a15a47 > Author: Daniel Kiper> Date: Wed Feb 22 14:38:06 2017 +0100 > > efi: create new early memory allocator Does anybody know why this still fails? Andrew applied a fix (workaround) for this yesterday. Daniel ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] [xen-unstable bisection] complete test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm
On Sun, Oct 30, 2016 at 04:53:33PM +, Andrew Cooper wrote: > On 30/10/16 04:29, osstest service owner wrote: > > branch xen-unstable > > xenbranch xen-unstable > > job test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm > > testid debian-hvm-install > > > > Tree: linux git://xenbits.xen.org/linux-pvops.git > > Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git > > Tree: qemu git://xenbits.xen.org/qemu-xen-traditional.git > > Tree: qemuu git://xenbits.xen.org/qemu-xen.git > > Tree: xen git://xenbits.xen.org/xen.git > > > > *** Found and reproduced problem changeset *** > > > > Bug is in tree: xen git://xenbits.xen.org/xen.git > > Bug introduced: 0897514b4b376a167f968f79c6ea0dee1061458e > > Bug not present: 4000a7c7d7b0e01837abd3918e393f289c07d68c > > Last fail repro: http://logs.test-lab.xenproject.org/osstest/logs/101803/ > > > > > > commit 0897514b4b376a167f968f79c6ea0dee1061458e > > Author: Andrew Cooper> > Date: Wed Oct 26 10:34:21 2016 +0100 > > > > tools/oxenstored: Avoid allocating invalid transaction ids > > I have to admit that I am staring at this report in belief, but it is > apparently deterministic. It is very strange that only this job is > affected; if there was actually a problem with xenstore transactions, I > would have thought that there to be collateral damage everywhere. > > Looking through the logs, there are several concerning things happening > even in the success cases. > > First: > > (XEN) HVM1 restore: CPU 0 > (XEN) avc: denied { getvcpucontext } for domid=0 target=2 > scontext=system_u:system_r:dom0_t tcontext=system_u:system_r:dm_dom_t > tclass=domain > > The toolstack calls getvcpucontext as part of domain creation, and the > XSM policy disallows this on dm_dom_t's. Interestingly, this failure > doesn't appear to be fatal to domain creation, and it really ought to > be. I expect there is also another bug lurking in the lower levels of > the toolstack. > No idea about this at the moment. > Second: > > (XEN) Assertion '!in_irq()' failed at xmalloc_tlsf.c:577 > (XEN) [ Xen-4.8.0-rc x86_64 debug=y Not tainted ] > > (XEN) Xen call trace: > (XEN)[] _xmalloc+0x2f/0x313 > (XEN)[] services.c#context_struct_to_string+0x98/0x16d > (XEN)[] security_sid_to_context+0xd3/0xe7 > (XEN)[] hooks.c#flask_show_security_evtchn+0x6f/0x87 > (XEN)[] event_channel.c#dump_evtchn_info+0x246/0x2cb > (XEN)[] handle_keypress+0x8c/0xac > (XEN)[] console.c#__serial_rx+0x38/0x73 > (XEN)[] console.c#serial_rx+0x8a/0x8f > (XEN)[] serial_rx_interrupt+0x90/0xac > (XEN)[] ns16550.c#ns16550_interrupt+0x57/0x71 > (XEN)[] do_IRQ+0x56e/0x60f > (XEN)[] common_interrupt+0x67/0x70 > (XEN)[] mwait-idle.c#mwait_idle+0x2af/0x2f9 > > The 'e' debugkey isn't safe to use when XSM is compiled in, as > security_sid_to_context() allocates memory. > This can not be easily fixed without reworking the XSM framework API. I propose we just disable the offending snippet. > Furthermore, any unexpected host crashes should cause a failure of the > test. This appears to have gone unnoticed because it happens in the > capture-logs phase, with presumably sufficient timeouts that OSSTest > doesn't notice that the host rebooted in the middle of log collection. > > Third: > > (d2) ** > (d2) blk_open(/local/domain/2/device/vbd/5632) -> 6 > (d2) xs_watch(device-model/1/logdirty/cmd, logdirty) > (d2) xs_watch(device-model/1/command, dm-command) > (d2) xs_watch(/local/domain/1/cpu, vcpu-set) > (d2) xs_read(/local/domain/0/backend/pci/1/0/msitranslate): ENOENT > (d2) xs_read(/local/domain/0/backend/pci/1/0/power_mgmt): ENOENT > (d2) open(/var/log/dm-serial.log) -> 7 > (d2) fcntl(-1, 3, 3ffbc8/17775710) > (d2) fcntl(-1, 4, /377) > (d2) fcntl(7, 3, /377) > (d2) fcntl(7, 4, /377) > (d2) xs_watch(/local/domain/0/backend/console/1, be:0x14b1a7:1:0x186800) > (d2) xs_directory(/local/domain/0/backend/console/1): EACCES > (d2) xs_watch(/local/domain/0/backend/vkbd/1, be:0x1479ff:1:0x1867c0) > (d2) xs_directory(/local/domain/0/backend/vkbd/1): EACCES > (d2) xs_read(device-model/1/disable_pf): ENOENT > (d2) xs_watch(/local/domain/1/log-throttling, > /local/domain/1/log-throttling) > (d2) Thread "kbdfront": pointer: 0x0xb0182570, stack: 0x0xaa > (d2) *** FBFRONT for /local/domain/2/device/vfb/0 ** > > The stub qemu attempts to read d1's backends. It probably shouldn't be > doing that. > This is (supposedly) harmless, so a low priority bug. Wei. > > Comparing the xenstored-access logs, between the success and failure > cases, it does appear that in the failing case, all transactions have > the id 1. I am trying to debug why. > > ~Andrew ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] [xen-unstable bisection] complete test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm
On 31/10/16 10:31, Ian Jackson wrote: > Andrew Cooper writes ("Re: [Xen-devel] [xen-unstable bisection] complete > test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm"): >> On 30/10/16 04:29, osstest service owner wrote: >>> Last fail repro: http://logs.test-lab.xenproject.org/osstest/logs/101803/ > ... >> Where can I find the build logs for the bisection runs? I am fairly >> sure I have a newer version of Ocaml than comes by default in Debian, >> and I wonder whether I have hit some version-dependent behaviour. > Let me walk you through this. > > We'll follow links starting with, say, the "Last fail repro" url, > above: > > http://logs.test-lab.xenproject.org/osstest/logs/101803/ > > Click on the test column heading to get to > > http://logs.test-lab.xenproject.org/osstest/logs/101803/test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm/info.html > > Scroll down and you'll see `Test control variables'. Select the build > of interest, which I think is `buildjob'. (There are also > `xenbuildjob' which is the hypervisor build, and `kernbuildjob' which > is, as might be expected, the kernel.) So click through to > 101797.build-i386-xsm: > > http://logs.test-lab.xenproject.org/osstest/logs/101797/build-i386-xsm/info.html > > The main build log is that for the `xen-build' step: > > http://logs.test-lab.xenproject.org/osstest/logs/101797/build-i386-xsm/4.ts-xen-build-prep.log > > Finding out which ocaml was actually installed can sometimes be a > little harder, because build host sharing means that the run of > `host-build-prep' shown in that job might be a no-op, and there isn't > an easy way to click through to the job that actually did the install > (sorry). > > However, it's easy enough to know what ocaml was probably installed: > looking at `Test control variables' in the build job, you see > `all_host_suite' with value `jessie'. Then go here > > https://www.debian.org/distrib/packages > > and in the `search package directories' enter `ocaml' (searching > within `stable', since jessie is currently stable); > > https://packages.debian.org/search?keywords=ocaml=names=stable=all > > Results: 4.01.0-5. Thankyou. It turns out that I am using the same version of Ocaml. However, looking at the build log, I think the fact that this is a 32bit build of the binary might be the salient point. Let me see if I can rebuild oxenstored as 32bit. ~Andrew ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] [xen-unstable bisection] complete test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm
Andrew Cooper writes ("Re: [Xen-devel] [xen-unstable bisection] complete test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm"): > On 30/10/16 04:29, osstest service owner wrote: > > Last fail repro: http://logs.test-lab.xenproject.org/osstest/logs/101803/ ... > Where can I find the build logs for the bisection runs? I am fairly > sure I have a newer version of Ocaml than comes by default in Debian, > and I wonder whether I have hit some version-dependent behaviour. Let me walk you through this. We'll follow links starting with, say, the "Last fail repro" url, above: http://logs.test-lab.xenproject.org/osstest/logs/101803/ Click on the test column heading to get to http://logs.test-lab.xenproject.org/osstest/logs/101803/test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm/info.html Scroll down and you'll see `Test control variables'. Select the build of interest, which I think is `buildjob'. (There are also `xenbuildjob' which is the hypervisor build, and `kernbuildjob' which is, as might be expected, the kernel.) So click through to 101797.build-i386-xsm: http://logs.test-lab.xenproject.org/osstest/logs/101797/build-i386-xsm/info.html The main build log is that for the `xen-build' step: http://logs.test-lab.xenproject.org/osstest/logs/101797/build-i386-xsm/4.ts-xen-build-prep.log Finding out which ocaml was actually installed can sometimes be a little harder, because build host sharing means that the run of `host-build-prep' shown in that job might be a no-op, and there isn't an easy way to click through to the job that actually did the install (sorry). However, it's easy enough to know what ocaml was probably installed: looking at `Test control variables' in the build job, you see `all_host_suite' with value `jessie'. Then go here https://www.debian.org/distrib/packages and in the `search package directories' enter `ocaml' (searching within `stable', since jessie is currently stable); https://packages.debian.org/search?keywords=ocaml=names=stable=all Results: 4.01.0-5. Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] [xen-unstable bisection] complete test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm
On 30/10/16 04:29, osstest service owner wrote: > branch xen-unstable > xenbranch xen-unstable > job test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm > testid debian-hvm-install > > Tree: linux git://xenbits.xen.org/linux-pvops.git > Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git > Tree: qemu git://xenbits.xen.org/qemu-xen-traditional.git > Tree: qemuu git://xenbits.xen.org/qemu-xen.git > Tree: xen git://xenbits.xen.org/xen.git > > *** Found and reproduced problem changeset *** > > Bug is in tree: xen git://xenbits.xen.org/xen.git > Bug introduced: 0897514b4b376a167f968f79c6ea0dee1061458e > Bug not present: 4000a7c7d7b0e01837abd3918e393f289c07d68c > Last fail repro: http://logs.test-lab.xenproject.org/osstest/logs/101803/ > > > commit 0897514b4b376a167f968f79c6ea0dee1061458e > Author: Andrew Cooper> Date: Wed Oct 26 10:34:21 2016 +0100 > > tools/oxenstored: Avoid allocating invalid transaction ids Looking at xenstored-access.log before: A8.1 write /vm/6fc3ae48-3d32-4710-aaac-55801d273773/uuid 6fc3ae48-3d32-4710-aaac-55801d273773 A8.1 write /vm/6fc3ae48-3d32-4710-aaac-55801d273773/name debianhvm.guest.osstest A8.1 write /local/domain/1/control/platform-feature-multiprocessor-suspend 1 A8.1 write /local/domain/1/control/platform-feature-xs_reset_watches 1 A8.1 commit A8 write /libxl/1/dm-version qemu_xen_traditional A8.2 write /local/domain/1/memory/static-max 512 A8.2 write /local/domain/1/memory/target 5115904 A8.2 write /local/domain/1/memory/videoram 4096 A8.2 write /local/domain/1/domid 1 And after: A8.1 write /vm/0a3c1b35-33f8-432d-aab7-b98ea16d490f/uuid 0a3c1b35-33f8-432d-aab7-b98ea16d490f A8.1 write /vm/0a3c1b35-33f8-432d-aab7-b98ea16d490f/name debianhvm.guest.osstest A8.1 write /local/domain/1/control/platform-feature-multiprocessor-suspend 1 A8.1 write /local/domain/1/control/platform-feature-xs_reset_watches 1 A8.1 commit A8 write /libxl/1/dm-version qemu_xen_traditional A8.1 write /local/domain/1/memory/static-max 512 A8.1 write /local/domain/1/memory/target 5115904 A8.1 write /local/domain/1/memory/videoram 4096 A8.1 write /local/domain/1/domid 1 the logging shows that almost all details are identical, other than the transaction ids on successive transactions. Compiling oxenstored locally on my dev box (and in the XenServer build system) confirms that transaction ids behave as intended, i.e. ids are allocated incrementally, even with the identified changeset in place. Where can I find the build logs for the bisection runs? I am fairly sure I have a newer version of Ocaml than comes by default in Debian, and I wonder whether I have hit some version-dependent behaviour. ~Andrew ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] [xen-unstable bisection] complete test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm
On 30/10/16 04:29, osstest service owner wrote: > branch xen-unstable > xenbranch xen-unstable > job test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm > testid debian-hvm-install > > Tree: linux git://xenbits.xen.org/linux-pvops.git > Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git > Tree: qemu git://xenbits.xen.org/qemu-xen-traditional.git > Tree: qemuu git://xenbits.xen.org/qemu-xen.git > Tree: xen git://xenbits.xen.org/xen.git > > *** Found and reproduced problem changeset *** > > Bug is in tree: xen git://xenbits.xen.org/xen.git > Bug introduced: 0897514b4b376a167f968f79c6ea0dee1061458e > Bug not present: 4000a7c7d7b0e01837abd3918e393f289c07d68c > Last fail repro: http://logs.test-lab.xenproject.org/osstest/logs/101803/ > > > commit 0897514b4b376a167f968f79c6ea0dee1061458e > Author: Andrew Cooper> Date: Wed Oct 26 10:34:21 2016 +0100 > > tools/oxenstored: Avoid allocating invalid transaction ids I have to admit that I am staring at this report in belief, but it is apparently deterministic. It is very strange that only this job is affected; if there was actually a problem with xenstore transactions, I would have thought that there to be collateral damage everywhere. Looking through the logs, there are several concerning things happening even in the success cases. First: (XEN) HVM1 restore: CPU 0 (XEN) avc: denied { getvcpucontext } for domid=0 target=2 scontext=system_u:system_r:dom0_t tcontext=system_u:system_r:dm_dom_t tclass=domain The toolstack calls getvcpucontext as part of domain creation, and the XSM policy disallows this on dm_dom_t's. Interestingly, this failure doesn't appear to be fatal to domain creation, and it really ought to be. I expect there is also another bug lurking in the lower levels of the toolstack. Second: (XEN) Assertion '!in_irq()' failed at xmalloc_tlsf.c:577 (XEN) [ Xen-4.8.0-rc x86_64 debug=y Not tainted ] (XEN) Xen call trace: (XEN)[] _xmalloc+0x2f/0x313 (XEN)[] services.c#context_struct_to_string+0x98/0x16d (XEN)[] security_sid_to_context+0xd3/0xe7 (XEN)[] hooks.c#flask_show_security_evtchn+0x6f/0x87 (XEN)[] event_channel.c#dump_evtchn_info+0x246/0x2cb (XEN)[] handle_keypress+0x8c/0xac (XEN)[] console.c#__serial_rx+0x38/0x73 (XEN)[] console.c#serial_rx+0x8a/0x8f (XEN)[] serial_rx_interrupt+0x90/0xac (XEN)[] ns16550.c#ns16550_interrupt+0x57/0x71 (XEN)[] do_IRQ+0x56e/0x60f (XEN)[] common_interrupt+0x67/0x70 (XEN)[] mwait-idle.c#mwait_idle+0x2af/0x2f9 The 'e' debugkey isn't safe to use when XSM is compiled in, as security_sid_to_context() allocates memory. Furthermore, any unexpected host crashes should cause a failure of the test. This appears to have gone unnoticed because it happens in the capture-logs phase, with presumably sufficient timeouts that OSSTest doesn't notice that the host rebooted in the middle of log collection. Third: (d2) ** (d2) blk_open(/local/domain/2/device/vbd/5632) -> 6 (d2) xs_watch(device-model/1/logdirty/cmd, logdirty) (d2) xs_watch(device-model/1/command, dm-command) (d2) xs_watch(/local/domain/1/cpu, vcpu-set) (d2) xs_read(/local/domain/0/backend/pci/1/0/msitranslate): ENOENT (d2) xs_read(/local/domain/0/backend/pci/1/0/power_mgmt): ENOENT (d2) open(/var/log/dm-serial.log) -> 7 (d2) fcntl(-1, 3, 3ffbc8/17775710) (d2) fcntl(-1, 4, /377) (d2) fcntl(7, 3, /377) (d2) fcntl(7, 4, /377) (d2) xs_watch(/local/domain/0/backend/console/1, be:0x14b1a7:1:0x186800) (d2) xs_directory(/local/domain/0/backend/console/1): EACCES (d2) xs_watch(/local/domain/0/backend/vkbd/1, be:0x1479ff:1:0x1867c0) (d2) xs_directory(/local/domain/0/backend/vkbd/1): EACCES (d2) xs_read(device-model/1/disable_pf): ENOENT (d2) xs_watch(/local/domain/1/log-throttling, /local/domain/1/log-throttling) (d2) Thread "kbdfront": pointer: 0x0xb0182570, stack: 0x0xaa (d2) *** FBFRONT for /local/domain/2/device/vfb/0 ** The stub qemu attempts to read d1's backends. It probably shouldn't be doing that. Comparing the xenstored-access logs, between the success and failure cases, it does appear that in the failing case, all transactions have the id 1. I am trying to debug why. ~Andrew ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel