Re: [Xen-devel] Regression between Xen 4.6.0 and 4.7.0, Direct kernel boot on a qemu-xen and seabios HVM guest doesn't work anymore.
On Tue, Oct 25, 2016 at 07:25:06PM +0200, Sander Eikelenboom wrote: > On 2016-10-25 16:49, Wei Liu wrote: > >On Tue, Oct 25, 2016 at 01:37:45PM +0200, Sander Eikelenboom wrote: > >> > >>Tuesday, October 25, 2016, 1:24:12 PM, you wrote: > >> > >>> On Tue, Oct 18, 2016 at 01:48:23PM +0100, Wei Liu wrote: > On Mon, Oct 17, 2016 at 05:28:17PM +0200, Sander Eikelenboom wrote: > > Thursday, October 13, 2016, 4:43:31 PM, you wrote: > > > > > Hi Jan / Wei, > > > > > Took a while before i had the chance to fiddle some more to find the > > > actual culprit. > > > After analyzing the output of xl -v create somewhat more i came > > > to the > > > insight it was probably Qemu and not Xen causing the fault. > > > > > As a test I just used a qemu-xen binary build with xen-4.6.0 booting > > > up a guest with > > > direct kernel boot mode on xen-unstable. And that old qemu binary > > > works fine. > > > > > After testing i can conclude, Jan was right, the bisection was a red > > > herring, > > > the problem is caused by some change in Qemu and not by something in > > > the Xen tree. > > > (strange thing is that for as far as i know i did a "make distclean" > > > between > > > every build (taking a lot of time), which should have pulled a fresh > > > qemu-xen > > > tree and therefor the bisection should have lead to a commit with a > > > Config.mk > > > hash change for qemu-xen version.) > > > > > Will see if i can find some more time and bisect qemu and find the > > > culprit. > > > > > -- > > > Sander > > > > > > Unfortunately i have to give up on this issue, for me it's impossible > > to bisect this > > issue with my present git-foo. > > > > The first try with bisection of the whole xen-tree seems to have hit > > the issue that the > > qemu-revision that gets pulled on a fresh build is "master" during the > > whole > > dev period. That creates havoc when trying to bisect, since you are > > testing > > combinations that were never developed (nor auto tested) in that > > combination > > (especially when a xen-tree and qemu-tree change have a dependency > > like Roger's > > "xen: fix usage of xc_domain_create in domain builder") > > > > While trying to bisect only qemu (keeping xen itself on RELEASE-4.6.0 > > and > > seabios on rel-1.8.2) it get stuck on issues with that tree. > > Between 4.6.0 and 4.7.0 the qemu tree switched from > > git://xenbits.xen.org/qemu-upstream-4.6-testing.git > > to git://xenbits.xen.org/qemu-xen.git),after that there seem to have > > been a lot of merges going back and forth and to me it seems a mess > > (but as i > > said it could also be a lack of git-foo). I tried by manual bisecting, > > removing > > and cloning trees again etc. but that doesn't suffice, it's all going > > no-where. > > (while the known good build (plain RELEASE-4.6.0) always works, so it > > doesn't > > seem to be some random problem) > > > > Thanks for trying. > > > So perhaps some dev can at least verify that the issue is there (since > > 4.7.0) > > and put it on the "known broken" list of things. > > > > I will put this into the list of things I need to look at. > > >> > >>> I investigated this a bit. The root cause is the memory accounting is > >>> wrong in QEMU. It would try to allocate more ram than allowed. I haven't > >>> tried to figure out exactly what is wrong, though. > >> > >>That confirms what i was thinking in the end, but bisection the > >>qemu-tree > >>changes between the xen-4.6.0 and xen-4.7.0 release proved to be pretty > >>difficult as i explained. So i you have a hunch as to in what code it > >>should > >>reside debugging instead of bisecting would probably be better. > >>(so one of the questions is what changes in the memory accounting when > >>you > >>supply the kernel from the host instead of the guest, since booting a > >>kernel > >>with grub from within the guest doesn't give any memory accounting > >>issues.) > >> > >>Thanks for investigating ! > > > >I think I hunted down the offending function. > > > >Mind trying this patch for me? > > Hi Wei, > > This seems to help :) > > With a linux 4.8 kernel the HVM guest now boots fine with direct kernel boot > ! > > But there seems to be a gotcha which i think is not in the Xen docs/wiki: > when trying a linux 4.3 kernel the guest still didn't boot and i got a: > "qemu: linux kernel too old to load a ram disk" in the qemu log. > I don't know what qemu regards as "old" in this case. > QEMU checks for a signature / version in kernel header or whatnot. I can't tell why that specific number is chosen, though. > Another considiration: would it be
Re: [Xen-devel] Regression between Xen 4.6.0 and 4.7.0, Direct kernel boot on a qemu-xen and seabios HVM guest doesn't work anymore.
On 2016-10-25 16:49, Wei Liu wrote: On Tue, Oct 25, 2016 at 01:37:45PM +0200, Sander Eikelenboom wrote: Tuesday, October 25, 2016, 1:24:12 PM, you wrote: > On Tue, Oct 18, 2016 at 01:48:23PM +0100, Wei Liu wrote: >> On Mon, Oct 17, 2016 at 05:28:17PM +0200, Sander Eikelenboom wrote: >> > Thursday, October 13, 2016, 4:43:31 PM, you wrote: >> > >> > > Hi Jan / Wei, >> > >> > > Took a while before i had the chance to fiddle some more to find the actual culprit. >> > > After analyzing the output of xl -v create somewhat more i came to the >> > > insight it was probably Qemu and not Xen causing the fault. >> > >> > > As a test I just used a qemu-xen binary build with xen-4.6.0 booting up a guest with >> > > direct kernel boot mode on xen-unstable. And that old qemu binary works fine. >> > >> > > After testing i can conclude, Jan was right, the bisection was a red herring, >> > > the problem is caused by some change in Qemu and not by something in the Xen tree. >> > > (strange thing is that for as far as i know i did a "make distclean" between >> > > every build (taking a lot of time), which should have pulled a fresh qemu-xen >> > > tree and therefor the bisection should have lead to a commit with a Config.mk >> > > hash change for qemu-xen version.) >> > >> > > Will see if i can find some more time and bisect qemu and find the culprit. >> > >> > > -- >> > > Sander >> > >> > >> > Unfortunately i have to give up on this issue, for me it's impossible to bisect this >> > issue with my present git-foo. >> > >> > The first try with bisection of the whole xen-tree seems to have hit the issue that the >> > qemu-revision that gets pulled on a fresh build is "master" during the whole >> > dev period. That creates havoc when trying to bisect, since you are testing >> > combinations that were never developed (nor auto tested) in that combination >> > (especially when a xen-tree and qemu-tree change have a dependency like Roger's >> > "xen: fix usage of xc_domain_create in domain builder") >> > >> > While trying to bisect only qemu (keeping xen itself on RELEASE-4.6.0 and >> > seabios on rel-1.8.2) it get stuck on issues with that tree. >> > Between 4.6.0 and 4.7.0 the qemu tree switched from git://xenbits.xen.org/qemu-upstream-4.6-testing.git >> > to git://xenbits.xen.org/qemu-xen.git),after that there seem to have >> > been a lot of merges going back and forth and to me it seems a mess (but as i >> > said it could also be a lack of git-foo). I tried by manual bisecting, removing >> > and cloning trees again etc. but that doesn't suffice, it's all going no-where. >> > (while the known good build (plain RELEASE-4.6.0) always works, so it doesn't >> > seem to be some random problem) >> > >> >> Thanks for trying. >> >> > So perhaps some dev can at least verify that the issue is there (since 4.7.0) >> > and put it on the "known broken" list of things. >> > >> >> I will put this into the list of things I need to look at. >> > I investigated this a bit. The root cause is the memory accounting is > wrong in QEMU. It would try to allocate more ram than allowed. I haven't > tried to figure out exactly what is wrong, though. That confirms what i was thinking in the end, but bisection the qemu-tree changes between the xen-4.6.0 and xen-4.7.0 release proved to be pretty difficult as i explained. So i you have a hunch as to in what code it should reside debugging instead of bisecting would probably be better. (so one of the questions is what changes in the memory accounting when you supply the kernel from the host instead of the guest, since booting a kernel with grub from within the guest doesn't give any memory accounting issues.) Thanks for investigating ! I think I hunted down the offending function. Mind trying this patch for me? Hi Wei, This seems to help :) With a linux 4.8 kernel the HVM guest now boots fine with direct kernel boot ! But there seems to be a gotcha which i think is not in the Xen docs/wiki: when trying a linux 4.3 kernel the guest still didn't boot and i got a: "qemu: linux kernel too old to load a ram disk" in the qemu log. I don't know what qemu regards as "old" in this case. Another considiration: would it be worthwhile to add an OSStest for direct kernel boot ? (under the assumption that the host kernel that gets build can also boot on HVM guest it's probably a very cheap test not requiring any additional builds.) Thanks again ! -- Sander ---8<--- From 3c7f8b55109959cf470d452f452f7c0ade51 Mon Sep 17 00:00:00 2001 From: Wei Liu Date: Tue, 25 Oct 2016 15:45:04 +0100 Subject: [PATCH] acpi: don't build acpi tables for xen guests Xen's toolstack is in charge of building ACPI tables. Skip acpi table building if running on Xen. This issue is discovered due to direct kernel boot on Xen doesn't boot anymore, because the new ACPI tables cause the guest to exceed its memory allocation limit. Reported-by: Sander Eikelenboom Signed-off-by: Wei Liu
Re: [Xen-devel] Regression between Xen 4.6.0 and 4.7.0, Direct kernel boot on a qemu-xen and seabios HVM guest doesn't work anymore.
On Tue, Oct 25, 2016 at 03:49:59PM +0100, Wei Liu wrote: > On Tue, Oct 25, 2016 at 01:37:45PM +0200, Sander Eikelenboom wrote: > > > > Tuesday, October 25, 2016, 1:24:12 PM, you wrote: > > > > > On Tue, Oct 18, 2016 at 01:48:23PM +0100, Wei Liu wrote: > > >> On Mon, Oct 17, 2016 at 05:28:17PM +0200, Sander Eikelenboom wrote: > > >> > Thursday, October 13, 2016, 4:43:31 PM, you wrote: > > >> > > > >> > > Hi Jan / Wei, > > >> > > > >> > > Took a while before i had the chance to fiddle some more to find the > > >> > > actual culprit. > > >> > > After analyzing the output of xl -v create somewhat more i came > > >> > > to the > > >> > > insight it was probably Qemu and not Xen causing the fault. > > >> > > > >> > > As a test I just used a qemu-xen binary build with xen-4.6.0 booting > > >> > > up a guest with > > >> > > direct kernel boot mode on xen-unstable. And that old qemu binary > > >> > > works fine. > > >> > > > >> > > After testing i can conclude, Jan was right, the bisection was a red > > >> > > herring, > > >> > > the problem is caused by some change in Qemu and not by something in > > >> > > the Xen tree. > > >> > > (strange thing is that for as far as i know i did a "make distclean" > > >> > > between > > >> > > every build (taking a lot of time), which should have pulled a fresh > > >> > > qemu-xen > > >> > > tree and therefor the bisection should have lead to a commit with a > > >> > > Config.mk > > >> > > hash change for qemu-xen version.) > > >> > > > >> > > Will see if i can find some more time and bisect qemu and find the > > >> > > culprit. > > >> > > > >> > > -- > > >> > > Sander > > >> > > > >> > > > >> > Unfortunately i have to give up on this issue, for me it's impossible > > >> > to bisect this > > >> > issue with my present git-foo. > > >> > > > >> > The first try with bisection of the whole xen-tree seems to have hit > > >> > the issue that the > > >> > qemu-revision that gets pulled on a fresh build is "master" during the > > >> > whole > > >> > dev period. That creates havoc when trying to bisect, since you are > > >> > testing > > >> > combinations that were never developed (nor auto tested) in that > > >> > combination > > >> > (especially when a xen-tree and qemu-tree change have a dependency > > >> > like Roger's > > >> > "xen: fix usage of xc_domain_create in domain builder") > > >> > > > >> > While trying to bisect only qemu (keeping xen itself on RELEASE-4.6.0 > > >> > and > > >> > seabios on rel-1.8.2) it get stuck on issues with that tree. > > >> > Between 4.6.0 and 4.7.0 the qemu tree switched from > > >> > git://xenbits.xen.org/qemu-upstream-4.6-testing.git > > >> > to git://xenbits.xen.org/qemu-xen.git),after that there seem to have > > >> > been a lot of merges going back and forth and to me it seems a mess > > >> > (but as i > > >> > said it could also be a lack of git-foo). I tried by manual bisecting, > > >> > removing > > >> > and cloning trees again etc. but that doesn't suffice, it's all going > > >> > no-where. > > >> > (while the known good build (plain RELEASE-4.6.0) always works, so it > > >> > doesn't > > >> > seem to be some random problem) > > >> > > > >> > > >> Thanks for trying. > > >> > > >> > So perhaps some dev can at least verify that the issue is there (since > > >> > 4.7.0) > > >> > and put it on the "known broken" list of things. > > >> > > > >> > > >> I will put this into the list of things I need to look at. > > >> > > > > > I investigated this a bit. The root cause is the memory accounting is > > > wrong in QEMU. It would try to allocate more ram than allowed. I haven't > > > tried to figure out exactly what is wrong, though. > > > > That confirms what i was thinking in the end, but bisection the qemu-tree > > changes between the xen-4.6.0 and xen-4.7.0 release proved to be pretty > > difficult as i explained. So i you have a hunch as to in what code it > > should > > reside debugging instead of bisecting would probably be better. > > (so one of the questions is what changes in the memory accounting when you > > supply the kernel from the host instead of the guest, since booting a kernel > > with grub from within the guest doesn't give any memory accounting issues.) > > > > Thanks for investigating ! > > I think I hunted down the offending function. > > Mind trying this patch for me? > > ---8<--- > From 3c7f8b55109959cf470d452f452f7c0ade51 Mon Sep 17 00:00:00 2001 > From: Wei Liu > Date: Tue, 25 Oct 2016 15:45:04 +0100 > Subject: [PATCH] acpi: don't build acpi tables for xen guests > > Xen's toolstack is in charge of building ACPI tables. Skip acpi table > building if running on Xen. > > This issue is discovered due to direct kernel boot on Xen doesn't boot > anymore, because the new ACPI tables cause the guest to exceed its > memory allocation limit. > > Reported-by: Sander Eikelenboom > Signed-off-by: Wei Liu > --- > Cc: Anthony PERARD > Cc: Stefano Stabellini >
Re: [Xen-devel] Regression between Xen 4.6.0 and 4.7.0, Direct kernel boot on a qemu-xen and seabios HVM guest doesn't work anymore.
On Tue, Oct 25, 2016 at 01:37:45PM +0200, Sander Eikelenboom wrote: > > Tuesday, October 25, 2016, 1:24:12 PM, you wrote: > > > On Tue, Oct 18, 2016 at 01:48:23PM +0100, Wei Liu wrote: > >> On Mon, Oct 17, 2016 at 05:28:17PM +0200, Sander Eikelenboom wrote: > >> > Thursday, October 13, 2016, 4:43:31 PM, you wrote: > >> > > >> > > Hi Jan / Wei, > >> > > >> > > Took a while before i had the chance to fiddle some more to find the > >> > > actual culprit. > >> > > After analyzing the output of xl -v create somewhat more i came to > >> > > the > >> > > insight it was probably Qemu and not Xen causing the fault. > >> > > >> > > As a test I just used a qemu-xen binary build with xen-4.6.0 booting > >> > > up a guest with > >> > > direct kernel boot mode on xen-unstable. And that old qemu binary > >> > > works fine. > >> > > >> > > After testing i can conclude, Jan was right, the bisection was a red > >> > > herring, > >> > > the problem is caused by some change in Qemu and not by something in > >> > > the Xen tree. > >> > > (strange thing is that for as far as i know i did a "make distclean" > >> > > between > >> > > every build (taking a lot of time), which should have pulled a fresh > >> > > qemu-xen > >> > > tree and therefor the bisection should have lead to a commit with a > >> > > Config.mk > >> > > hash change for qemu-xen version.) > >> > > >> > > Will see if i can find some more time and bisect qemu and find the > >> > > culprit. > >> > > >> > > -- > >> > > Sander > >> > > >> > > >> > Unfortunately i have to give up on this issue, for me it's impossible to > >> > bisect this > >> > issue with my present git-foo. > >> > > >> > The first try with bisection of the whole xen-tree seems to have hit the > >> > issue that the > >> > qemu-revision that gets pulled on a fresh build is "master" during the > >> > whole > >> > dev period. That creates havoc when trying to bisect, since you are > >> > testing > >> > combinations that were never developed (nor auto tested) in that > >> > combination > >> > (especially when a xen-tree and qemu-tree change have a dependency like > >> > Roger's > >> > "xen: fix usage of xc_domain_create in domain builder") > >> > > >> > While trying to bisect only qemu (keeping xen itself on RELEASE-4.6.0 > >> > and > >> > seabios on rel-1.8.2) it get stuck on issues with that tree. > >> > Between 4.6.0 and 4.7.0 the qemu tree switched from > >> > git://xenbits.xen.org/qemu-upstream-4.6-testing.git > >> > to git://xenbits.xen.org/qemu-xen.git),after that there seem to have > >> > been a lot of merges going back and forth and to me it seems a mess (but > >> > as i > >> > said it could also be a lack of git-foo). I tried by manual bisecting, > >> > removing > >> > and cloning trees again etc. but that doesn't suffice, it's all going > >> > no-where. > >> > (while the known good build (plain RELEASE-4.6.0) always works, so it > >> > doesn't > >> > seem to be some random problem) > >> > > >> > >> Thanks for trying. > >> > >> > So perhaps some dev can at least verify that the issue is there (since > >> > 4.7.0) > >> > and put it on the "known broken" list of things. > >> > > >> > >> I will put this into the list of things I need to look at. > >> > > > I investigated this a bit. The root cause is the memory accounting is > > wrong in QEMU. It would try to allocate more ram than allowed. I haven't > > tried to figure out exactly what is wrong, though. > > That confirms what i was thinking in the end, but bisection the qemu-tree > changes between the xen-4.6.0 and xen-4.7.0 release proved to be pretty > difficult as i explained. So i you have a hunch as to in what code it should > reside debugging instead of bisecting would probably be better. > (so one of the questions is what changes in the memory accounting when you > supply the kernel from the host instead of the guest, since booting a kernel > with grub from within the guest doesn't give any memory accounting issues.) > > Thanks for investigating ! I think I hunted down the offending function. Mind trying this patch for me? ---8<--- From 3c7f8b55109959cf470d452f452f7c0ade51 Mon Sep 17 00:00:00 2001 From: Wei Liu Date: Tue, 25 Oct 2016 15:45:04 +0100 Subject: [PATCH] acpi: don't build acpi tables for xen guests Xen's toolstack is in charge of building ACPI tables. Skip acpi table building if running on Xen. This issue is discovered due to direct kernel boot on Xen doesn't boot anymore, because the new ACPI tables cause the guest to exceed its memory allocation limit. Reported-by: Sander Eikelenboom Signed-off-by: Wei Liu --- Cc: Anthony PERARD Cc: Stefano Stabellini RFC because I'm not sure this is the best way to fix it. --- hw/i386/acpi-build.c | 7 +++ 1 file changed, 7 insertions(+) diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c index a26a4bb..6ba5031 100644 --- a/hw/i386/acpi-build.c +++ b/hw/i386/acpi-build.c @@ -45,6 +45,7 @@ #include "sy
Re: [Xen-devel] Regression between Xen 4.6.0 and 4.7.0, Direct kernel boot on a qemu-xen and seabios HVM guest doesn't work anymore.
Tuesday, October 25, 2016, 1:24:12 PM, you wrote: > On Tue, Oct 18, 2016 at 01:48:23PM +0100, Wei Liu wrote: >> On Mon, Oct 17, 2016 at 05:28:17PM +0200, Sander Eikelenboom wrote: >> > Thursday, October 13, 2016, 4:43:31 PM, you wrote: >> > >> > > Hi Jan / Wei, >> > >> > > Took a while before i had the chance to fiddle some more to find the >> > > actual culprit. >> > > After analyzing the output of xl -v create somewhat more i came to >> > > the >> > > insight it was probably Qemu and not Xen causing the fault. >> > >> > > As a test I just used a qemu-xen binary build with xen-4.6.0 booting up >> > > a guest with >> > > direct kernel boot mode on xen-unstable. And that old qemu binary works >> > > fine. >> > >> > > After testing i can conclude, Jan was right, the bisection was a red >> > > herring, >> > > the problem is caused by some change in Qemu and not by something in the >> > > Xen tree. >> > > (strange thing is that for as far as i know i did a "make distclean" >> > > between >> > > every build (taking a lot of time), which should have pulled a fresh >> > > qemu-xen >> > > tree and therefor the bisection should have lead to a commit with a >> > > Config.mk >> > > hash change for qemu-xen version.) >> > >> > > Will see if i can find some more time and bisect qemu and find the >> > > culprit. >> > >> > > -- >> > > Sander >> > >> > >> > Unfortunately i have to give up on this issue, for me it's impossible to >> > bisect this >> > issue with my present git-foo. >> > >> > The first try with bisection of the whole xen-tree seems to have hit the >> > issue that the >> > qemu-revision that gets pulled on a fresh build is "master" during the >> > whole >> > dev period. That creates havoc when trying to bisect, since you are >> > testing >> > combinations that were never developed (nor auto tested) in that >> > combination >> > (especially when a xen-tree and qemu-tree change have a dependency like >> > Roger's >> > "xen: fix usage of xc_domain_create in domain builder") >> > >> > While trying to bisect only qemu (keeping xen itself on RELEASE-4.6.0 and >> > seabios on rel-1.8.2) it get stuck on issues with that tree. >> > Between 4.6.0 and 4.7.0 the qemu tree switched from >> > git://xenbits.xen.org/qemu-upstream-4.6-testing.git >> > to git://xenbits.xen.org/qemu-xen.git),after that there seem to have >> > been a lot of merges going back and forth and to me it seems a mess (but >> > as i >> > said it could also be a lack of git-foo). I tried by manual bisecting, >> > removing >> > and cloning trees again etc. but that doesn't suffice, it's all going >> > no-where. >> > (while the known good build (plain RELEASE-4.6.0) always works, so it >> > doesn't >> > seem to be some random problem) >> > >> >> Thanks for trying. >> >> > So perhaps some dev can at least verify that the issue is there (since >> > 4.7.0) >> > and put it on the "known broken" list of things. >> > >> >> I will put this into the list of things I need to look at. >> > I investigated this a bit. The root cause is the memory accounting is > wrong in QEMU. It would try to allocate more ram than allowed. I haven't > tried to figure out exactly what is wrong, though. That confirms what i was thinking in the end, but bisection the qemu-tree changes between the xen-4.6.0 and xen-4.7.0 release proved to be pretty difficult as i explained. So i you have a hunch as to in what code it should reside debugging instead of bisecting would probably be better. (so one of the questions is what changes in the memory accounting when you supply the kernel from the host instead of the guest, since booting a kernel with grub from within the guest doesn't give any memory accounting issues.) Thanks for investigating ! -- Sander > Wei. ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Regression between Xen 4.6.0 and 4.7.0, Direct kernel boot on a qemu-xen and seabios HVM guest doesn't work anymore.
On Tue, Oct 18, 2016 at 01:48:23PM +0100, Wei Liu wrote: > On Mon, Oct 17, 2016 at 05:28:17PM +0200, Sander Eikelenboom wrote: > > Thursday, October 13, 2016, 4:43:31 PM, you wrote: > > > > > Hi Jan / Wei, > > > > > Took a while before i had the chance to fiddle some more to find the > > > actual culprit. > > > After analyzing the output of xl -v create somewhat more i came to > > > the > > > insight it was probably Qemu and not Xen causing the fault. > > > > > As a test I just used a qemu-xen binary build with xen-4.6.0 booting up a > > > guest with > > > direct kernel boot mode on xen-unstable. And that old qemu binary works > > > fine. > > > > > After testing i can conclude, Jan was right, the bisection was a red > > > herring, > > > the problem is caused by some change in Qemu and not by something in the > > > Xen tree. > > > (strange thing is that for as far as i know i did a "make distclean" > > > between > > > every build (taking a lot of time), which should have pulled a fresh > > > qemu-xen > > > tree and therefor the bisection should have lead to a commit with a > > > Config.mk > > > hash change for qemu-xen version.) > > > > > Will see if i can find some more time and bisect qemu and find the > > > culprit. > > > > > -- > > > Sander > > > > > > Unfortunately i have to give up on this issue, for me it's impossible to > > bisect this > > issue with my present git-foo. > > > > The first try with bisection of the whole xen-tree seems to have hit the > > issue that the > > qemu-revision that gets pulled on a fresh build is "master" during the whole > > dev period. That creates havoc when trying to bisect, since you are testing > > combinations that were never developed (nor auto tested) in that combination > > (especially when a xen-tree and qemu-tree change have a dependency like > > Roger's > > "xen: fix usage of xc_domain_create in domain builder") > > > > While trying to bisect only qemu (keeping xen itself on RELEASE-4.6.0 and > > seabios on rel-1.8.2) it get stuck on issues with that tree. > > Between 4.6.0 and 4.7.0 the qemu tree switched from > > git://xenbits.xen.org/qemu-upstream-4.6-testing.git > > to git://xenbits.xen.org/qemu-xen.git),after that there seem to have > > been a lot of merges going back and forth and to me it seems a mess (but as > > i > > said it could also be a lack of git-foo). I tried by manual bisecting, > > removing > > and cloning trees again etc. but that doesn't suffice, it's all going > > no-where. > > (while the known good build (plain RELEASE-4.6.0) always works, so it > > doesn't > > seem to be some random problem) > > > > Thanks for trying. > > > So perhaps some dev can at least verify that the issue is there (since > > 4.7.0) > > and put it on the "known broken" list of things. > > > > I will put this into the list of things I need to look at. > I investigated this a bit. The root cause is the memory accounting is wrong in QEMU. It would try to allocate more ram than allowed. I haven't tried to figure out exactly what is wrong, though. Wei. ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Regression between Xen 4.6.0 and 4.7.0, Direct kernel boot on a qemu-xen and seabios HVM guest doesn't work anymore.
Den 18. okt. 2016 14:48, skrev Wei Liu: > On Mon, Oct 17, 2016 at 05:28:17PM +0200, Sander Eikelenboom wrote: >> Thursday, October 13, 2016, 4:43:31 PM, you wrote: >> >>> Hi Jan / Wei, >>> Took a while before i had the chance to fiddle some more to find the actual >>> culprit. >>> After analyzing the output of xl -v create somewhat more i came to the >>> insight it was probably Qemu and not Xen causing the fault. >>> As a test I just used a qemu-xen binary build with xen-4.6.0 booting up a >>> guest with >>> direct kernel boot mode on xen-unstable. And that old qemu binary works >>> fine. >>> After testing i can conclude, Jan was right, the bisection was a red >>> herring, >>> the problem is caused by some change in Qemu and not by something in the >>> Xen tree. >>> (strange thing is that for as far as i know i did a "make distclean" >>> between >>> every build (taking a lot of time), which should have pulled a fresh >>> qemu-xen >>> tree and therefor the bisection should have lead to a commit with a >>> Config.mk >>> hash change for qemu-xen version.) >>> Will see if i can find some more time and bisect qemu and find the culprit. >>> -- >>> Sander >> >> Unfortunately i have to give up on this issue, for me it's impossible to >> bisect this >> issue with my present git-foo. >> >> The first try with bisection of the whole xen-tree seems to have hit the >> issue that the >> qemu-revision that gets pulled on a fresh build is "master" during the whole >> dev period. That creates havoc when trying to bisect, since you are testing >> combinations that were never developed (nor auto tested) in that combination >> (especially when a xen-tree and qemu-tree change have a dependency like >> Roger's >> "xen: fix usage of xc_domain_create in domain builder") >> >> While trying to bisect only qemu (keeping xen itself on RELEASE-4.6.0 and >> seabios on rel-1.8.2) it get stuck on issues with that tree. >> Between 4.6.0 and 4.7.0 the qemu tree switched from >> git://xenbits.xen.org/qemu-upstream-4.6-testing.git >> to git://xenbits.xen.org/qemu-xen.git),after that there seem to have >> been a lot of merges going back and forth and to me it seems a mess (but as >> i >> said it could also be a lack of git-foo). I tried by manual bisecting, >> removing >> and cloning trees again etc. but that doesn't suffice, it's all going >> no-where. >> (while the known good build (plain RELEASE-4.6.0) always works, so it >> doesn't >> seem to be some random problem) >> > Thanks for trying. > >> So perhaps some dev can at least verify that the issue is there (since 4.7.0) >> and put it on the "known broken" list of things. >> > I will put this into the list of things I need to look at. > > Wei. > In the mean time, a viable work-around is to use pxe boot if one needs external boot for hvm under xen-4.7. Still, the effort is appreciated :-). Regards, Håkon. P.S: I had some difficulty with pxe-boot and serial console, feel free to email me direct if anyone wants to compare notes. ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Regression between Xen 4.6.0 and 4.7.0, Direct kernel boot on a qemu-xen and seabios HVM guest doesn't work anymore.
On Mon, Oct 17, 2016 at 05:28:17PM +0200, Sander Eikelenboom wrote: > Thursday, October 13, 2016, 4:43:31 PM, you wrote: > > > Hi Jan / Wei, > > > Took a while before i had the chance to fiddle some more to find the actual > > culprit. > > After analyzing the output of xl -v create somewhat more i came to the > > insight it was probably Qemu and not Xen causing the fault. > > > As a test I just used a qemu-xen binary build with xen-4.6.0 booting up a > > guest with > > direct kernel boot mode on xen-unstable. And that old qemu binary works > > fine. > > > After testing i can conclude, Jan was right, the bisection was a red > > herring, > > the problem is caused by some change in Qemu and not by something in the > > Xen tree. > > (strange thing is that for as far as i know i did a "make distclean" > > between > > every build (taking a lot of time), which should have pulled a fresh > > qemu-xen > > tree and therefor the bisection should have lead to a commit with a > > Config.mk > > hash change for qemu-xen version.) > > > Will see if i can find some more time and bisect qemu and find the culprit. > > > -- > > Sander > > > Unfortunately i have to give up on this issue, for me it's impossible to > bisect this > issue with my present git-foo. > > The first try with bisection of the whole xen-tree seems to have hit the > issue that the > qemu-revision that gets pulled on a fresh build is "master" during the whole > dev period. That creates havoc when trying to bisect, since you are testing > combinations that were never developed (nor auto tested) in that combination > (especially when a xen-tree and qemu-tree change have a dependency like > Roger's > "xen: fix usage of xc_domain_create in domain builder") > > While trying to bisect only qemu (keeping xen itself on RELEASE-4.6.0 and > seabios on rel-1.8.2) it get stuck on issues with that tree. > Between 4.6.0 and 4.7.0 the qemu tree switched from > git://xenbits.xen.org/qemu-upstream-4.6-testing.git > to git://xenbits.xen.org/qemu-xen.git),after that there seem to have > been a lot of merges going back and forth and to me it seems a mess (but as i > said it could also be a lack of git-foo). I tried by manual bisecting, > removing > and cloning trees again etc. but that doesn't suffice, it's all going > no-where. > (while the known good build (plain RELEASE-4.6.0) always works, so it doesn't > seem to be some random problem) > Thanks for trying. > So perhaps some dev can at least verify that the issue is there (since 4.7.0) > and put it on the "known broken" list of things. > I will put this into the list of things I need to look at. Wei. > -- > Sander > ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Regression between Xen 4.6.0 and 4.7.0, Direct kernel boot on a qemu-xen and seabios HVM guest doesn't work anymore.
Thursday, October 13, 2016, 4:43:31 PM, you wrote: > Hi Jan / Wei, > Took a while before i had the chance to fiddle some more to find the actual > culprit. > After analyzing the output of xl -v create somewhat more i came to the > insight it was probably Qemu and not Xen causing the fault. > As a test I just used a qemu-xen binary build with xen-4.6.0 booting up a > guest with > direct kernel boot mode on xen-unstable. And that old qemu binary works fine. > After testing i can conclude, Jan was right, the bisection was a red herring, > the problem is caused by some change in Qemu and not by something in the Xen > tree. > (strange thing is that for as far as i know i did a "make distclean" between > every build (taking a lot of time), which should have pulled a fresh qemu-xen > tree and therefor the bisection should have lead to a commit with a Config.mk > hash change for qemu-xen version.) > Will see if i can find some more time and bisect qemu and find the culprit. > -- > Sander Unfortunately i have to give up on this issue, for me it's impossible to bisect this issue with my present git-foo. The first try with bisection of the whole xen-tree seems to have hit the issue that the qemu-revision that gets pulled on a fresh build is "master" during the whole dev period. That creates havoc when trying to bisect, since you are testing combinations that were never developed (nor auto tested) in that combination (especially when a xen-tree and qemu-tree change have a dependency like Roger's "xen: fix usage of xc_domain_create in domain builder") While trying to bisect only qemu (keeping xen itself on RELEASE-4.6.0 and seabios on rel-1.8.2) it get stuck on issues with that tree. Between 4.6.0 and 4.7.0 the qemu tree switched from git://xenbits.xen.org/qemu-upstream-4.6-testing.git to git://xenbits.xen.org/qemu-xen.git),after that there seem to have been a lot of merges going back and forth and to me it seems a mess (but as i said it could also be a lack of git-foo). I tried by manual bisecting, removing and cloning trees again etc. but that doesn't suffice, it's all going no-where. (while the known good build (plain RELEASE-4.6.0) always works, so it doesn't seem to be some random problem) So perhaps some dev can at least verify that the issue is there (since 4.7.0) and put it on the "known broken" list of things. -- Sander ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Regression between Xen 4.6.0 and 4.7.0, Direct kernel boot on a qemu-xen and seabios HVM guest doesn't work anymore.
Hi Jan / Wei, Took a while before i had the chance to fiddle some more to find the actual culprit. After analyzing the output of xl -v create somewhat more i came to the insight it was probably Qemu and not Xen causing the fault. As a test I just used a qemu-xen binary build with xen-4.6.0 booting up a guest with direct kernel boot mode on xen-unstable. And that old qemu binary works fine. After testing i can conclude, Jan was right, the bisection was a red herring, the problem is caused by some change in Qemu and not by something in the Xen tree. (strange thing is that for as far as i know i did a "make distclean" between every build (taking a lot of time), which should have pulled a fresh qemu-xen tree and therefor the bisection should have lead to a commit with a Config.mk hash change for qemu-xen version.) Will see if i can find some more time and bisect qemu and find the culprit. -- Sander ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Regression between Xen 4.6.0 and 4.7.0, Direct kernel boot on a qemu-xen and seabios HVM guest doesn't work anymore.
On 2016-09-05 13:43, Jan Beulich wrote: On 05.09.16 at 13:19, wrote: On 2016-09-05 12:25, Jan Beulich wrote: Anyway - with you quite clearly having used HAP before, I can't see how this commit would matter for you at all. In case you want to double check you could try with a hypervisor built without shadow paging code (which we've been allowing for quite a while). I just tried that and without shadow paging code the guest boots fine, so that's interesting. Indeed. Was that try with plain staging/master, or with much of the reverts in place (from the bisection)? It seems to me that investigating this odd difference would perhaps be a better route than trying to guess what's wrong with said commit. Jan It was a try with a tree at the culprit commit and editted xen/arch/x86/Rules.mk to disable the shadow paging code from being build. Now just tried with unstable and using Kconfig, but with that build the guest doesn't boot. So or the KConfig option doesn't work or the reliability isn't 100% afterall (but i should have noticed that earlier on i would say) or there is something else (semantics around the disabling changed ?) *sigh*, seems it's not going to be an easy one :-\ My /boot/xen-4.8-unstable.config gives: # # Architecture Features # CONFIG_NR_CPUS=256 # CONFIG_SHADOW_PAGING is not set # CONFIG_BIGMEM is not set CONFIG_HVM_FEP=y CONFIG_TBOOT=y So it should be off i guess. -- Sander ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Regression between Xen 4.6.0 and 4.7.0, Direct kernel boot on a qemu-xen and seabios HVM guest doesn't work anymore.
>>> On 05.09.16 at 13:19, wrote: > On 2016-09-05 12:25, Jan Beulich wrote: >> Anyway - with you quite clearly having used HAP before, I can't >> see how this commit would matter for you at all. In case you want >> to double check you could try with a hypervisor built without >> shadow paging code (which we've been allowing for quite a >> while). > > I just tried that and without shadow paging code the guest boots fine, > so that's interesting. Indeed. Was that try with plain staging/master, or with much of the reverts in place (from the bisection)? It seems to me that investigating this odd difference would perhaps be a better route than trying to guess what's wrong with said commit. Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Regression between Xen 4.6.0 and 4.7.0, Direct kernel boot on a qemu-xen and seabios HVM guest doesn't work anymore.
On 2016-09-05 12:25, Jan Beulich wrote: On 05.09.16 at 12:02, wrote: On 2016-09-05 11:46, Jan Beulich wrote: On 05.09.16 at 11:20, wrote: Hmm it seems my thread was kind of hijacked and i was dropped from the CC. I had some time and bisected the issue and it resulted in: 5a3ce8f85e7e7bdd339d259daa19f6bc5cb4735f is the first bad commit commit 5a3ce8f85e7e7bdd339d259daa19f6bc5cb4735f Author: Jan Beulich Date: Wed Oct 21 10:56:31 2015 +0200 x86/shadow: drop stray name tags from sh_{guest_get,map}_eff_l1e() Hmm, as Wei already indicated - that's rather odd. The commit isn't really supposed to have any effect on functionality (and going through it again I also can't spot any now). And are you indeed using shadow mode, and if so does your problem not occur when you use HAP instead? In any event, if there was some hidden (and unintended) change in functionality here, then the most likely result would seem to be a crash, yet from the log fragment you posted it doesn't look like there's _any_ relevant hypervisor output. Hmm i was already afraid of that. Attached is the output of xl dmesg, HAP is supported and should be enabled by default (and i didn't disable it explicitly in my guest.cfg). I just tried the opposite and specified hap=0 in my guest.cfg and this case leads to 2 lines of additional output: XEN) [2016-09-05 09:58:22.201] sh error: sh_remove_all_mappings(): can't find all mappings of mfn 471b69: c=8003 t=7401 (XEN) [2016-09-05 09:58:22.201] sh error: sh_remove_all_mappings(): can't find all mappings of mfn 471b68: c=8003 t=7401 And these two messages are relevant here? I.e. do they go away when you use a commit ahead of the one your bisect spotted? Just double checked with a build one commit ahead of the culprit the bisection reported and hap=0, and those messages are there as well and the guest boots fine now. So they don't seem to be relevant. Anyway - with you quite clearly having used HAP before, I can't see how this commit would matter for you at all. In case you want to double check you could try with a hypervisor built without shadow paging code (which we've been allowing for quite a while). I just tried that and without shadow paging code the guest boots fine, so that's interesting. Is it possible that the reproduction of the issue isn't 100% reliable? Nope it seems 100% reliable. I.e. did you verify with a couple of runs each that it really is this commit, and not just some spurious effect? If it is, then from all I know so far I'd suspect an effect from code / data arrangement rather than the commit itself to be the actual culprit. Well at least there is one other independent user running into the same issue, so it doesn't seem specifically related to my machine or my builds. It also happens when running all my guests (and this is the last to start) and with only this guest. Which reminds me of another possible way of double checking: If said commit reverts reasonably cleanly at the tip of staging or master, maybe you could try with just this change reverted, instead of with everything subsequent to it reverted too? Nope it tried that already and it didn't revert cleanly (and i didn't see how to correctly fix it up). -- Sander Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Regression between Xen 4.6.0 and 4.7.0, Direct kernel boot on a qemu-xen and seabios HVM guest doesn't work anymore.
>>> On 05.09.16 at 12:02, wrote: > On 2016-09-05 11:46, Jan Beulich wrote: > On 05.09.16 at 11:20, wrote: >>> Hmm it seems my thread was kind of hijacked and i was dropped from the >>> CC. >>> >>> I had some time and bisected the issue and it resulted in: >>> >>> 5a3ce8f85e7e7bdd339d259daa19f6bc5cb4735f is the first bad commit >>> commit 5a3ce8f85e7e7bdd339d259daa19f6bc5cb4735f >>> Author: Jan Beulich >>> Date: Wed Oct 21 10:56:31 2015 +0200 >>> >>> x86/shadow: drop stray name tags from >>> sh_{guest_get,map}_eff_l1e() >> >> Hmm, as Wei already indicated - that's rather odd. The commit isn't >> really supposed to have any effect on functionality (and going >> through it again I also can't spot any now). And are you indeed >> using shadow mode, and if so does your problem not occur when >> you use HAP instead? >> >> In any event, if there was some hidden (and unintended) change >> in functionality here, then the most likely result would seem to be >> a crash, yet from the log fragment you posted it doesn't look like >> there's _any_ relevant hypervisor output. > > Hmm i was already afraid of that. > Attached is the output of xl dmesg, HAP is supported and should be > enabled by default (and i didn't disable it explicitly in my guest.cfg). > > I just tried the opposite and specified hap=0 in my guest.cfg and this > case leads to 2 lines of additional output: > > XEN) [2016-09-05 09:58:22.201] sh error: sh_remove_all_mappings(): can't > find all mappings of mfn 471b69: c=8003 t=7401 > (XEN) [2016-09-05 09:58:22.201] sh error: sh_remove_all_mappings(): > can't find all mappings of mfn 471b68: c=8003 > t=7401 And these two messages are relevant here? I.e. do they go away when you use a commit ahead of the one your bisect spotted? Anyway - with you quite clearly having used HAP before, I can't see how this commit would matter for you at all. In case you want to double check you could try with a hypervisor built without shadow paging code (which we've been allowing for quite a while). Is it possible that the reproduction of the issue isn't 100% reliable? I.e. did you verify with a couple of runs each that it really is this commit, and not just some spurious effect? If it is, then from all I know so far I'd suspect an effect from code / data arrangement rather than the commit itself to be the actual culprit. Which reminds me of another possible way of double checking: If said commit reverts reasonably cleanly at the tip of staging or master, maybe you could try with just this change reverted, instead of with everything subsequent to it reverted too? Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Regression between Xen 4.6.0 and 4.7.0, Direct kernel boot on a qemu-xen and seabios HVM guest doesn't work anymore.
On 2016-09-05 11:46, Jan Beulich wrote: On 05.09.16 at 11:20, wrote: Hmm it seems my thread was kind of hijacked and i was dropped from the CC. I had some time and bisected the issue and it resulted in: 5a3ce8f85e7e7bdd339d259daa19f6bc5cb4735f is the first bad commit commit 5a3ce8f85e7e7bdd339d259daa19f6bc5cb4735f Author: Jan Beulich Date: Wed Oct 21 10:56:31 2015 +0200 x86/shadow: drop stray name tags from sh_{guest_get,map}_eff_l1e() Hmm, as Wei already indicated - that's rather odd. The commit isn't really supposed to have any effect on functionality (and going through it again I also can't spot any now). And are you indeed using shadow mode, and if so does your problem not occur when you use HAP instead? In any event, if there was some hidden (and unintended) change in functionality here, then the most likely result would seem to be a crash, yet from the log fragment you posted it doesn't look like there's _any_ relevant hypervisor output. Jan Hmm i was already afraid of that. Attached is the output of xl dmesg, HAP is supported and should be enabled by default (and i didn't disable it explicitly in my guest.cfg). I just tried the opposite and specified hap=0 in my guest.cfg and this case leads to 2 lines of additional output: XEN) [2016-09-05 09:58:22.201] sh error: sh_remove_all_mappings(): can't find all mappings of mfn 471b69: c=8003 t=7401 (XEN) [2016-09-05 09:58:22.201] sh error: sh_remove_all_mappings(): can't find all mappings of mfn 471b68: c=8003 t=7401 (XEN) [2016-09-05 09:58:22.334] d0v5 Over-allocation for domain 3: 262401 > 262400 (XEN) [2016-09-05 09:58:22.334] memory.c:163:d0v5 Could not allocate order=0 extent: id=3 memflags=0 (192 of 512) -- Sander __ ___ _ _ __ _ \ \/ /___ _ __ | || | |___ | _ _ _ __ ___| |_ __ _| |__ | | ___ \ // _ \ '_ \ | || |_ / /_| | | | '_ \/ __| __/ _` | '_ \| |/ _ \ / \ __/ | | | |__ _| / /__| |_| | | | \__ \ || (_| | |_) | | __/ /_/\_\___|_| |_||_|(_)_/\__,_|_| |_|___/\__\__,_|_.__/|_|\___| (XEN) Xen version 4.7-unstable (r...@dyndns.org) (gcc-4.9.real (Debian 4.9.2-10) 4.9.2) debug=y Mon Sep 5 11:03:14 CEST 2016 (XEN) Latest ChangeSet: Wed Oct 21 10:56:31 2015 +0200 git:5a3ce8f (XEN) Bootloader: GRUB 2.02~beta2-22+deb8u1 (XEN) Command line: dom0_mem=1536M,max:1536M loglvl=all loglvl_guest=all console_timestamps=datems vga=gfx-1280x1024x32 no-cpuidle cpufreq=xen com1=38400,8n1 console=vga,com1 ivrs_ioapic[6]=00:14.0 iommu=on,verbose,debug,amd-iommu-debug conring_size=128k sched=credit2 ucode=-1 (XEN) Video information: (XEN) VGA is graphics mode 1280x1024, 32 bpp (XEN) VBE/DDC methods: V2; EDID transfer time: 1 seconds (XEN) EDID info not retrieved because of reasons unknown (XEN) Disc information: (XEN) Found 2 MBR signatures (XEN) Found 2 EDD information structures (XEN) Xen-e820 RAM map: (XEN) - 00099400 (usable) (XEN) 00099400 - 000a (reserved) (XEN) 000e4000 - 0010 (reserved) (XEN) 0010 - 9ff9 (usable) (XEN) 9ff9 - 9ff9e000 (ACPI data) (XEN) 9ff9e000 - 9ffe (ACPI NVS) (XEN) 9ffe - a000 (reserved) (XEN) ffe0 - 0001 (reserved) (XEN) 0001 - 00056000 (usable) (XEN) ACPI: RSDP 000FB100, 0014 (r0 ACPIAM) (XEN) ACPI: RSDT 9FF9, 0048 (r1 MSIOEMSLIC 20100913 MSFT 97) (XEN) ACPI: FACP 9FF90200, 0084 (r1 7640MS A7640100 20100913 MSFT 97) (XEN) ACPI: DSDT 9FF905E0, 9427 (r1 A7640 A7640100 100 INTL 20051117) (XEN) ACPI: FACS 9FF9E000, 0040 (XEN) ACPI: APIC 9FF90390, 0088 (r1 7640MS A7640100 20100913 MSFT 97) (XEN) ACPI: MCFG 9FF90420, 003C (r1 7640MS OEMMCFG 20100913 MSFT 97) (XEN) ACPI: SLIC 9FF90460, 0176 (r1 MSIOEMSLIC 20100913 MSFT 97) (XEN) ACPI: OEMB 9FF9E040, 0072 (r1 7640MS A7640100 20100913 MSFT 97) (XEN) ACPI: SRAT 9FF9A5E0, 0108 (r3 AMDFAM_F_102 AMD 1) (XEN) ACPI: HPET 9FF9A6F0, 0038 (r1 7640MS OEMHPET 20100913 MSFT 97) (XEN) ACPI: IVRS 9FF9A730, 0110 (r1 AMD RD890S 202031 AMD 0) (XEN) ACPI: SSDT 9FF9A840, 0DA4 (r1 A M I POWERNOW1 AMD 1) (XEN) System RAM: 20479MB (20970660kB) (XEN) SRAT: PXM 0 -> APIC 00 -> Node 0 (XEN) SRAT: PXM 0 -> APIC 01 -> Node 0 (XEN) SRAT: PXM 0 -> APIC 02 -> Node 0 (XEN) SRAT: PXM 0 -> APIC 03 -> Node 0 (XEN) SRAT: PXM 0 -> APIC 04 -> Node 0 (XEN) SRAT: PXM 0 -> APIC 05 -> Node 0 (XEN) SRAT: Node 0 PXM 0 0-a (XEN) SRAT: Node 0 PXM 0 10-a000 (XEN) SRAT: Node 0 PXM 0 1-56000 (XEN) NUMA: Allocated memnodemap from 55c797000 - 55c79d000 (XEN) NUMA: Using 8 for the hash shift. (XEN) Domain heap initialised (XEN) Allocated console ring of 128 KiB. (XE
Re: [Xen-devel] Regression between Xen 4.6.0 and 4.7.0, Direct kernel boot on a qemu-xen and seabios HVM guest doesn't work anymore.
>>> On 05.09.16 at 11:20, wrote: > Hmm it seems my thread was kind of hijacked and i was dropped from the > CC. > > I had some time and bisected the issue and it resulted in: > > 5a3ce8f85e7e7bdd339d259daa19f6bc5cb4735f is the first bad commit > commit 5a3ce8f85e7e7bdd339d259daa19f6bc5cb4735f > Author: Jan Beulich > Date: Wed Oct 21 10:56:31 2015 +0200 > > x86/shadow: drop stray name tags from sh_{guest_get,map}_eff_l1e() Hmm, as Wei already indicated - that's rather odd. The commit isn't really supposed to have any effect on functionality (and going through it again I also can't spot any now). And are you indeed using shadow mode, and if so does your problem not occur when you use HAP instead? In any event, if there was some hidden (and unintended) change in functionality here, then the most likely result would seem to be a crash, yet from the log fragment you posted it doesn't look like there's _any_ relevant hypervisor output. Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Regression between Xen 4.6.0 and 4.7.0, Direct kernel boot on a qemu-xen and seabios HVM guest doesn't work anymore.
On Mon, Sep 05, 2016 at 11:20:30AM +0200, li...@eikelenboom.it wrote: > On 2016-08-25 23:18, li...@eikelenboom.it wrote: > >On 2016-08-25 22:34, Doug Goldstein wrote: > >>On 8/25/16 4:21 PM, li...@eikelenboom.it wrote: > >>>Today i tried to switch some of my HVM guests (qemu-xen) from booting > >>>of > >>>a kernel *inside* the guest, to a dom0 supplied kernel, which is > >>>described as "Direct Kernel Boot" here: > >>>https://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html : > >>> > >>>Direct Kernel Boot > >>> > >>>Direct kernel boot allows booting directly from a kernel and > >>>initrd > >>>stored in the host physical > >>>machine OS, allowing command line arguments to be passed directly. > >>>PV guest direct kernel boot > >>>is supported. HVM guest direct kernel boot is supported with > >>>limitation (it's supported when > >>>using qemu-xen and default BIOS 'seabios'; not supported in case > >>>of > >>>stubdom-dm and old rombios.) > >>> > >>>kernel="PATHNAME"Load the specified file as the kernel image. > >>>ramdisk="PATHNAME" Load the specified file as the ramdisk. > >>> > >>>But qemu fails to start, output appended below. > >>> > >>>I tested with: > >>>- current Xen-unstable, which fails. > >>>- xen-stable-4.7.0 release, which fails. > >>>- xen-stable-4.6.0 release, works fine. > >> > >>Can you include the logs from xl dmesg around that time frame as well? > > > >Ah i thought there wasn't any, but didn't check thoroughly or wasn't there > >since the release builds are non-debug by default. > > > >However, back on xen-unstable: > >(XEN) [2016-08-25 21:09:15.172] HVM19 save: CPU > >(XEN) [2016-08-25 21:09:15.172] HVM19 save: PIC > >(XEN) [2016-08-25 21:09:15.172] HVM19 save: IOAPIC > >(XEN) [2016-08-25 21:09:15.172] HVM19 save: LAPIC > >(XEN) [2016-08-25 21:09:15.172] HVM19 save: LAPIC_REGS > >(XEN) [2016-08-25 21:09:15.172] HVM19 save: PCI_IRQ > >(XEN) [2016-08-25 21:09:15.172] HVM19 save: ISA_IRQ > >(XEN) [2016-08-25 21:09:15.172] HVM19 save: PCI_LINK > >(XEN) [2016-08-25 21:09:15.172] HVM19 save: PIT > >(XEN) [2016-08-25 21:09:15.172] HVM19 save: RTC > >(XEN) [2016-08-25 21:09:15.172] HVM19 save: HPET > >(XEN) [2016-08-25 21:09:15.172] HVM19 save: PMTIMER > >(XEN) [2016-08-25 21:09:15.172] HVM19 save: MTRR > >(XEN) [2016-08-25 21:09:15.172] HVM19 save: VIRIDIAN_DOMAIN > >(XEN) [2016-08-25 21:09:15.172] HVM19 save: CPU_XSAVE > >(XEN) [2016-08-25 21:09:15.172] HVM19 save: VIRIDIAN_VCPU > >(XEN) [2016-08-25 21:09:15.172] HVM19 save: VMCE_VCPU > >(XEN) [2016-08-25 21:09:15.172] HVM19 save: TSC_ADJUST > >(XEN) [2016-08-25 21:09:15.172] HVM19 restore: CPU 0 > >(XEN) [2016-08-25 21:09:16.126] d0v1 Over-allocation for domain 19: > >262401 > 262400 > >(XEN) [2016-08-25 21:09:16.126] memory.c:213:d0v1 Could not allocate > >order=0 extent: id=19 memflags=0 (192 of 512) > > > >Hmm some off by one issue ? > > > > > >>Just wondering how much RAM you're domain is defined with as well? > > > >1024 Mb, there is more than enough unallocated memory for xen to start > >the guest (and dom0 is fixed with dom0_mem=1536M,max:1536M and > >ballooning is off) > > > Hmm it seems my thread was kind of hijacked and i was dropped from the CC. > Oops, I thought you were CC'ed. Sorry. > I had some time and bisected the issue and it resulted in: > > 5a3ce8f85e7e7bdd339d259daa19f6bc5cb4735f is the first bad commit > commit 5a3ce8f85e7e7bdd339d259daa19f6bc5cb4735f > Author: Jan Beulich > Date: Wed Oct 21 10:56:31 2015 +0200 > > x86/shadow: drop stray name tags from sh_{guest_get,map}_eff_l1e() > > They (as a now being removed comment validly says) depend only on Xen's > number of page table levels, and hence their tags didn't serve any > useful purpose (there could only ever be one instance in a single > binary, even back in the x86-32 days). > > Further conditionalize the inclusion of PV-specific hook pointers, at > once making sure that PV guests can't ever get other than 4-level mode > enabled for them. > > For consistency reasons shadow_{write,cmpxchg}_guest_entry() also get > moved next to the other PV-only actors, allowing them to become static > just like the $subject ones do. > > Signed-off-by: Jan Beulich > Acked-by: Tim Deegan > > :04 04 0c2e3475f81547f934a5960d9f1ac4849707d4ed > f17f5ff17ca50d6ab908afe9a2d8555d954d3d0a M xen > Unfortunately I can't see immediately why this would affect QEMU direct boot. It also suggests that it only affects shadow code -- what kind of hardware are you using? Wei. > > -- > Sander > > > > > >-- > >Sander ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Regression between Xen 4.6.0 and 4.7.0, Direct kernel boot on a qemu-xen and seabios HVM guest doesn't work anymore.
On 2016-08-25 23:18, li...@eikelenboom.it wrote: On 2016-08-25 22:34, Doug Goldstein wrote: On 8/25/16 4:21 PM, li...@eikelenboom.it wrote: Today i tried to switch some of my HVM guests (qemu-xen) from booting of a kernel *inside* the guest, to a dom0 supplied kernel, which is described as "Direct Kernel Boot" here: https://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html : Direct Kernel Boot Direct kernel boot allows booting directly from a kernel and initrd stored in the host physical machine OS, allowing command line arguments to be passed directly. PV guest direct kernel boot is supported. HVM guest direct kernel boot is supported with limitation (it's supported when using qemu-xen and default BIOS 'seabios'; not supported in case of stubdom-dm and old rombios.) kernel="PATHNAME"Load the specified file as the kernel image. ramdisk="PATHNAME" Load the specified file as the ramdisk. But qemu fails to start, output appended below. I tested with: - current Xen-unstable, which fails. - xen-stable-4.7.0 release, which fails. - xen-stable-4.6.0 release, works fine. Can you include the logs from xl dmesg around that time frame as well? Ah i thought there wasn't any, but didn't check thoroughly or wasn't there since the release builds are non-debug by default. However, back on xen-unstable: (XEN) [2016-08-25 21:09:15.172] HVM19 save: CPU (XEN) [2016-08-25 21:09:15.172] HVM19 save: PIC (XEN) [2016-08-25 21:09:15.172] HVM19 save: IOAPIC (XEN) [2016-08-25 21:09:15.172] HVM19 save: LAPIC (XEN) [2016-08-25 21:09:15.172] HVM19 save: LAPIC_REGS (XEN) [2016-08-25 21:09:15.172] HVM19 save: PCI_IRQ (XEN) [2016-08-25 21:09:15.172] HVM19 save: ISA_IRQ (XEN) [2016-08-25 21:09:15.172] HVM19 save: PCI_LINK (XEN) [2016-08-25 21:09:15.172] HVM19 save: PIT (XEN) [2016-08-25 21:09:15.172] HVM19 save: RTC (XEN) [2016-08-25 21:09:15.172] HVM19 save: HPET (XEN) [2016-08-25 21:09:15.172] HVM19 save: PMTIMER (XEN) [2016-08-25 21:09:15.172] HVM19 save: MTRR (XEN) [2016-08-25 21:09:15.172] HVM19 save: VIRIDIAN_DOMAIN (XEN) [2016-08-25 21:09:15.172] HVM19 save: CPU_XSAVE (XEN) [2016-08-25 21:09:15.172] HVM19 save: VIRIDIAN_VCPU (XEN) [2016-08-25 21:09:15.172] HVM19 save: VMCE_VCPU (XEN) [2016-08-25 21:09:15.172] HVM19 save: TSC_ADJUST (XEN) [2016-08-25 21:09:15.172] HVM19 restore: CPU 0 (XEN) [2016-08-25 21:09:16.126] d0v1 Over-allocation for domain 19: 262401 > 262400 (XEN) [2016-08-25 21:09:16.126] memory.c:213:d0v1 Could not allocate order=0 extent: id=19 memflags=0 (192 of 512) Hmm some off by one issue ? Just wondering how much RAM you're domain is defined with as well? 1024 Mb, there is more than enough unallocated memory for xen to start the guest (and dom0 is fixed with dom0_mem=1536M,max:1536M and ballooning is off) Hmm it seems my thread was kind of hijacked and i was dropped from the CC. I had some time and bisected the issue and it resulted in: 5a3ce8f85e7e7bdd339d259daa19f6bc5cb4735f is the first bad commit commit 5a3ce8f85e7e7bdd339d259daa19f6bc5cb4735f Author: Jan Beulich Date: Wed Oct 21 10:56:31 2015 +0200 x86/shadow: drop stray name tags from sh_{guest_get,map}_eff_l1e() They (as a now being removed comment validly says) depend only on Xen's number of page table levels, and hence their tags didn't serve any useful purpose (there could only ever be one instance in a single binary, even back in the x86-32 days). Further conditionalize the inclusion of PV-specific hook pointers, at once making sure that PV guests can't ever get other than 4-level mode enabled for them. For consistency reasons shadow_{write,cmpxchg}_guest_entry() also get moved next to the other PV-only actors, allowing them to become static just like the $subject ones do. Signed-off-by: Jan Beulich Acked-by: Tim Deegan :04 04 0c2e3475f81547f934a5960d9f1ac4849707d4ed f17f5ff17ca50d6ab908afe9a2d8555d954d3d0a M xen -- Sander -- Sander ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Regression between Xen 4.6.0 and 4.7.0, Direct kernel boot on a qemu-xen and seabios HVM guest doesn't work anymore.
Could you please use xl -vvv create to create the guest and collect the output? Wei. ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Regression between Xen 4.6.0 and 4.7.0, Direct kernel boot on a qemu-xen and seabios HVM guest doesn't work anymore.
Den 25. aug. 2016 23:18, skrev li...@eikelenboom.it: > On 2016-08-25 22:34, Doug Goldstein wrote: >> On 8/25/16 4:21 PM, li...@eikelenboom.it wrote: >>> Today i tried to switch some of my HVM guests (qemu-xen) from >>> booting of >>> a kernel *inside* the guest, to a dom0 supplied kernel, which is >>> described as "Direct Kernel Boot" here: >>> https://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html : >>> >>> Direct Kernel Boot >>> >>> Direct kernel boot allows booting directly from a kernel and initrd >>> stored in the host physical >>> machine OS, allowing command line arguments to be passed directly. >>> PV guest direct kernel boot >>> is supported. HVM guest direct kernel boot is supported with >>> limitation (it's supported when >>> using qemu-xen and default BIOS 'seabios'; not supported in case of >>> stubdom-dm and old rombios.) >>> >>> kernel="PATHNAME"Load the specified file as the kernel image. >>> ramdisk="PATHNAME" Load the specified file as the ramdisk. >>> >>> But qemu fails to start, output appended below. >>> >>> I tested with: >>> - current Xen-unstable, which fails. >>> - xen-stable-4.7.0 release, which fails. >>> - xen-stable-4.6.0 release, works fine. >> >> Can you include the logs from xl dmesg around that time frame as well? > > Ah i thought there wasn't any, but didn't check thoroughly or wasn't > there > since the release builds are non-debug by default. > > However, back on xen-unstable: > (XEN) [2016-08-25 21:09:15.172] HVM19 save: CPU > (XEN) [2016-08-25 21:09:15.172] HVM19 save: PIC > (XEN) [2016-08-25 21:09:15.172] HVM19 save: IOAPIC > (XEN) [2016-08-25 21:09:15.172] HVM19 save: LAPIC > (XEN) [2016-08-25 21:09:15.172] HVM19 save: LAPIC_REGS > (XEN) [2016-08-25 21:09:15.172] HVM19 save: PCI_IRQ > (XEN) [2016-08-25 21:09:15.172] HVM19 save: ISA_IRQ > (XEN) [2016-08-25 21:09:15.172] HVM19 save: PCI_LINK > (XEN) [2016-08-25 21:09:15.172] HVM19 save: PIT > (XEN) [2016-08-25 21:09:15.172] HVM19 save: RTC > (XEN) [2016-08-25 21:09:15.172] HVM19 save: HPET > (XEN) [2016-08-25 21:09:15.172] HVM19 save: PMTIMER > (XEN) [2016-08-25 21:09:15.172] HVM19 save: MTRR > (XEN) [2016-08-25 21:09:15.172] HVM19 save: VIRIDIAN_DOMAIN > (XEN) [2016-08-25 21:09:15.172] HVM19 save: CPU_XSAVE > (XEN) [2016-08-25 21:09:15.172] HVM19 save: VIRIDIAN_VCPU > (XEN) [2016-08-25 21:09:15.172] HVM19 save: VMCE_VCPU > (XEN) [2016-08-25 21:09:15.172] HVM19 save: TSC_ADJUST > (XEN) [2016-08-25 21:09:15.172] HVM19 restore: CPU 0 > (XEN) [2016-08-25 21:09:16.126] d0v1 Over-allocation for domain 19: > 262401 > 262400 > (XEN) [2016-08-25 21:09:16.126] memory.c:213:d0v1 Could not allocate > order=0 extent: id=19 memflags=0 (192 of 512) > > Hmm some off by one issue ? > > >> Just wondering how much RAM you're domain is defined with as well? > > 1024 Mb, there is more than enough unallocated memory for xen to start > the guest (and dom0 is fixed with dom0_mem=1536M,max:1536M and > ballooning is off) > > -- > Sander > > > I've got the same issue, reported it in xen-users som time ago. I never caught on that internal/external kernel would trigger it. I'll just paste my entire message from xen-users below: -- I have been trying for some time now to upgrade from Xen 4.6.* to 4.7. Trying several different dom0 kernel versions, and jiggling the xl.cfg files. All to no avail. I am unable to launch most of my guests under 4.7, though they run fine under 4.6 (except for some usb/pci-pass-though -related issues) . As seen from the device-model log below, qemu claims it is unable to allocate ram: "qemu: hardware error: xen: failed to populate ram at 28005", but I have plenty ram available, and this same VM (and many more) launch fine under 4.6.* I admit I am a rank amateur at this, so my config is probably pretty weird, possibly leading to a set-up that nobody knowledgeable would run. If somebody can give me a hint on how to work around this issue I'll happily test patches and provide logs. Example VM which does not start under 4.7. : --xl.cfg for media.hvm (i pass pci-pass-through for usb-card on command-line. Works OK)name = "media.hvm" builder = "hvm" xen_platform_pci = '1' pvh=1 memory = 7168 mmio_hole=3072 vcpus = 6 cap=600 cpus_soft="node:0" cpu_weight=6144 device_model_version="qemu-xen" serial = 'pty' disk = [ 'vdev=xvda, format=raw, target=/dev/system/media-backend' ,'vdev=xvdb, format=raw, target=/dev/system/media-backend-swap' ,'vdev=xvdd, format=raw, target=/dev/system/apub' ,'vdev=xvde, format=raw, target=/dev/system/apub1' ,'vdev=xvdf, format=raw, target=/dev/system/apub2' ,'vdev=xvdg, format=raw, target=/dev/system/apub3' ,'vdev=xvdh, format=raw, target=/dev/system/apub4' ,'vdev=xvdi, format=raw, target=/dev/system/apub5' ,'vdev=xvdj, format=raw, target=/dev/system/apub6' ,'vdev=xvdk, format=raw, target=/dev/system/apub7' ] kernel = "/etc/xen/media-boot/vmlinuz-4.1.12-gentoo" extra = "root=/dev/xv
Re: [Xen-devel] Regression between Xen 4.6.0 and 4.7.0, Direct kernel boot on a qemu-xen and seabios HVM guest doesn't work anymore.
On 2016-08-25 22:34, Doug Goldstein wrote: On 8/25/16 4:21 PM, li...@eikelenboom.it wrote: Today i tried to switch some of my HVM guests (qemu-xen) from booting of a kernel *inside* the guest, to a dom0 supplied kernel, which is described as "Direct Kernel Boot" here: https://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html : Direct Kernel Boot Direct kernel boot allows booting directly from a kernel and initrd stored in the host physical machine OS, allowing command line arguments to be passed directly. PV guest direct kernel boot is supported. HVM guest direct kernel boot is supported with limitation (it's supported when using qemu-xen and default BIOS 'seabios'; not supported in case of stubdom-dm and old rombios.) kernel="PATHNAME"Load the specified file as the kernel image. ramdisk="PATHNAME" Load the specified file as the ramdisk. But qemu fails to start, output appended below. I tested with: - current Xen-unstable, which fails. - xen-stable-4.7.0 release, which fails. - xen-stable-4.6.0 release, works fine. Can you include the logs from xl dmesg around that time frame as well? Ah i thought there wasn't any, but didn't check thoroughly or wasn't there since the release builds are non-debug by default. However, back on xen-unstable: (XEN) [2016-08-25 21:09:15.172] HVM19 save: CPU (XEN) [2016-08-25 21:09:15.172] HVM19 save: PIC (XEN) [2016-08-25 21:09:15.172] HVM19 save: IOAPIC (XEN) [2016-08-25 21:09:15.172] HVM19 save: LAPIC (XEN) [2016-08-25 21:09:15.172] HVM19 save: LAPIC_REGS (XEN) [2016-08-25 21:09:15.172] HVM19 save: PCI_IRQ (XEN) [2016-08-25 21:09:15.172] HVM19 save: ISA_IRQ (XEN) [2016-08-25 21:09:15.172] HVM19 save: PCI_LINK (XEN) [2016-08-25 21:09:15.172] HVM19 save: PIT (XEN) [2016-08-25 21:09:15.172] HVM19 save: RTC (XEN) [2016-08-25 21:09:15.172] HVM19 save: HPET (XEN) [2016-08-25 21:09:15.172] HVM19 save: PMTIMER (XEN) [2016-08-25 21:09:15.172] HVM19 save: MTRR (XEN) [2016-08-25 21:09:15.172] HVM19 save: VIRIDIAN_DOMAIN (XEN) [2016-08-25 21:09:15.172] HVM19 save: CPU_XSAVE (XEN) [2016-08-25 21:09:15.172] HVM19 save: VIRIDIAN_VCPU (XEN) [2016-08-25 21:09:15.172] HVM19 save: VMCE_VCPU (XEN) [2016-08-25 21:09:15.172] HVM19 save: TSC_ADJUST (XEN) [2016-08-25 21:09:15.172] HVM19 restore: CPU 0 (XEN) [2016-08-25 21:09:16.126] d0v1 Over-allocation for domain 19: 262401 > 262400 (XEN) [2016-08-25 21:09:16.126] memory.c:213:d0v1 Could not allocate order=0 extent: id=19 memflags=0 (192 of 512) Hmm some off by one issue ? Just wondering how much RAM you're domain is defined with as well? 1024 Mb, there is more than enough unallocated memory for xen to start the guest (and dom0 is fixed with dom0_mem=1536M,max:1536M and ballooning is off) -- Sander ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Regression between Xen 4.6.0 and 4.7.0, Direct kernel boot on a qemu-xen and seabios HVM guest doesn't work anymore.
On 8/25/16 4:21 PM, li...@eikelenboom.it wrote: > Today i tried to switch some of my HVM guests (qemu-xen) from booting of > a kernel *inside* the guest, to a dom0 supplied kernel, which is > described as "Direct Kernel Boot" here: > https://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html : > > Direct Kernel Boot > > Direct kernel boot allows booting directly from a kernel and initrd > stored in the host physical > machine OS, allowing command line arguments to be passed directly. > PV guest direct kernel boot > is supported. HVM guest direct kernel boot is supported with > limitation (it's supported when > using qemu-xen and default BIOS 'seabios'; not supported in case of > stubdom-dm and old rombios.) > > kernel="PATHNAME"Load the specified file as the kernel image. > ramdisk="PATHNAME" Load the specified file as the ramdisk. > > But qemu fails to start, output appended below. > > I tested with: > - current Xen-unstable, which fails. > - xen-stable-4.7.0 release, which fails. > - xen-stable-4.6.0 release, works fine. Can you include the logs from xl dmesg around that time frame as well? Just wondering how much RAM you're domain is defined with as well? -- Doug Goldstein signature.asc Description: OpenPGP digital signature ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel