Re: [Xen-devel] GRUB2 missing multiboot2 patches?Re: Only 1 CPU was detected

[Stefan Bader]

On 28.09.2017 16:03, Konrad Rzeszutek Wilk wrote:
> On Thu, Sep 28, 2017 at 02:45:38PM +, Hongjiang Zhang wrote:
 (XEN) ACPI Error (tbxfroot-0218): A valid RSDP was not found
 [20070126]
>>>
>>> Uuh, that is rather bad, I guess.
> 
> I am going to assume this is due to not having:
> 
> b4d709b6e Use grub-file to figure out whether multiboot2 should be used for 
> Xen.gz
> a8e0f1adf Fix util/grub.d/20_linux_xen.in: Add xen_boot command support for 
> aarch64
> 
> In the grub that he is using (Ubuntu?)
> 
> In other words he is using 'multiboot' instead of 'multiboot2'
> 
If this is Ubuntu, my expectation is that this would require Xen 4.9 (which is
part of 17.10 but not yet released) and work on grub2 (which I  will very
unlikely have the time for). Debian has not yet moved to Xen 4.9, so I would
doubt that it would work there either.

-Stefan



signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen 4.6.5 released

[Stefan Bader]

On 13.03.2017 11:29, Andrew Cooper wrote:
> On 13/03/17 09:24, Jan Beulich wrote:
> On 10.03.17 at 18:22,  wrote:
>>> On 08.03.2017 13:54, Jan Beulich wrote:
 All,

 I am pleased to announce the release of Xen 4.6.5. This is
 available immediately from its git repository
 http://xenbits.xen.org/gitweb/?p=xen.git;a=shortlog;h=refs/heads/stable-4.6
  
 (tag RELEASE-4.6.5) or from the XenProject download page
 http://www.xenproject.org/downloads/xen-archives/xen-46-series/xen-465.html
  
 (where a list of changes can also be found).

 We recommend all users of the 4.6 stable series to update to this
 latest point release.
>>> This does not seem to compile for me (x86_64) without the attached 
>>> (admittedly
>>> brutish) change.
>> I guess it's the emulator test code which has a problem here (I
>> did notice this myself), but that doesn't get built by default (and
>> I see no reason why anyone would want to build it when putting
>> together packages for people to consume - this is purely a dev
>> tool). Please clarify.
> 
> These tools are all built automatically.  Therefore, build fixes should
> be backported.
> 
> To avoid building them, you need override CONFIG_TESTS := n in the root
> .config file to override the default in Config.mk

Thanks Andrew,

I was not sure but I did not do anything special except replacing the orig
tarballs. The rest of the build is as we share it with Debian. So for a minor
release / stable release update I would rather not change the environment.

For the patch I just copied the definition from lib.h because gcc seems to be
called without access to hypervisor includes (probably adapting the Makefile
plus adding an include would be the better path but it was late'ish on a Friday
and I wanted something compiling quickly).

-Stefan





signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen 4.6.5 released

[Stefan Bader]

On 08.03.2017 13:54, Jan Beulich wrote:
> All,
> 
> I am pleased to announce the release of Xen 4.6.5. This is
> available immediately from its git repository
> http://xenbits.xen.org/gitweb/?p=xen.git;a=shortlog;h=refs/heads/stable-4.6 
> (tag RELEASE-4.6.5) or from the XenProject download page
> http://www.xenproject.org/downloads/xen-archives/xen-46-series/xen-465.html 
> (where a list of changes can also be found).
> 
> We recommend all users of the 4.6 stable series to update to this
> latest point release.

This does not seem to compile for me (x86_64) without the attached (admittedly
brutish) change.

-Stefan

> 
> Regards, Jan
> 
> 
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
> 

Index: xen-4.6.5/xen/arch/x86/x86_emulate/x86_emulate.c
===
--- xen-4.6.5.orig/xen/arch/x86/x86_emulate/x86_emulate.c
+++ xen-4.6.5/xen/arch/x86/x86_emulate/x86_emulate.c
@@ -20,6 +20,8 @@
  * along with this program; If not, see .
  */
 
+#define MASK_EXTR(v, m) (((v) & (m)) / ((m) & -(m)))
+
 /* Operand sizes: 8-bit operands or specified/overridden size. */
 #define ByteOp  (1<<0) /* 8-bit operands. */
 /* Destination operand type. */


signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] xen-4.7 regression when saving a pv guest

[Stefan Bader]

On 26.08.2016 13:53, Juergen Gross wrote:
> On 26/08/16 12:52, Stefan Bader wrote:
>> On 25.08.2016 19:31, Juergen Gross wrote:
>>> On 25/08/16 17:48, Stefan Bader wrote:
>>>> When I try to save a PV guest with 4G of memory using xen-4.7 I get the
>>>> following error:
>>>>
>>>> II: Guest memory 4096 MB
>>>> II: Saving guest state to file...
>>>> Saving to /tmp/pvguest.save new xl format (info 0x3/0x0/1131)
>>>> xc: info: Saving domain 23, type x86 PV
>>>> xc: error: Bad mfn in p2m_frame_list[0]: Internal error
>>>
>>> So the first mfn of the memory containing the p2m information is bogus.
>>> Weird.
>>
>> Hm, not sure how bogus. From below the first mfn is 0x4eb1c8 and points to
>> pfn=0xff7c8 which is above the current max of 0xb. But then the dmesg 
>> inside
>> the guest said: "last_pfn = 0x10" which would be larger than the pfn 
>> causing
>> the error.
>>
>>>
>>>> xc: error: mfn 0x4eb1c8, max 0x82: Internal error
>>>> xc: error:   m2p[0x4eb1c8] = 0xff7c8, max_pfn 0xb: Internal error
>>>> xc: error: Save failed (34 = Numerical result out of range): Internal error
>>>> libxl: error: libxl_stream_write.c:355:libxl__xc_domain_save_done: saving
>>>> domain: domain did not respond to suspend request: Numerical result out of 
>>>> range
>>>> Failed to save domain, resuming domain
>>>> xc: error: Dom 23 not suspended: (shutdown 0, reason 255): Internal error
>>>> libxl: error: libxl_dom_suspend.c:460:libxl__domain_resume: 
>>>> xc_domain_resume
>>>> failed for domain 23: Invalid argument
>>>> EE: Guest not off after save!
>>>> FAIL
>>>>
>>>> From dmesg inside the guest:
>>>> [0.00] e820: last_pfn = 0x10 max_arch_pfn = 0x4
>>>>
>>>> Somehow I am slightly suspicious about
>>>>
>>>> commit 91e204d37f44913913776d0a89279721694f8b32
>>>>   libxc: try to find last used pfn when migrating
>>>>
>>>> since that seems to potentially lower ctx->x86_pv.max_pfn which is checked
>>>> against in mfn_in_pseudophysmap(). Is that a known problem?
>>>> With xen-4.6 and the same dom0/guest kernel version combination this does 
>>>> work.
>>>
>>> Can you please share some more information? Especially:
>>>
>>> - guest kernel version?
>> Hm, apparently 4.4 and 4.6 with stable updates. I just tried a much older 
>> guest
>> kernel (3.2) environment and that works. So it is the combination of 
>> switching
>> from xen-4.6 to 4.7 and guest kernels running 4.4 and later. And while the 
>> exact
>> mfn/pfn which gets dumped varies a little, the offending mapping always 
>> points
>> to 0xffxxx which would be below last_pfn.
> 
> Aah, okay. The problem seems to be specific to the linear p2m list
> handling.
> 
> Trying on my system... Yep, seeing your problem, too.
> 
> Weird that nobody else stumbled over it.
> Ian, don't we have any test in OSSTEST which should catch this problem?
> A 4GB 64-bit pv-domain with Linux kernel 4.3 or newer can't be saved
> currently.
> 
> Following upstream patch fixes it for me:

Ah! :) Thanks. I applied the below locally, too. And save works with a 4.6 guest
kernel.

-Stefan

> 
> diff --git a/tools/libxc/xc_sr_save_x86_pv.c
> b/tools/libxc/xc_sr_save_x86_pv.c
> index 4a29460..7043409 100644
> --- a/tools/libxc/xc_sr_save_x86_pv.c
> +++ b/tools/libxc/xc_sr_save_x86_pv.c
> @@ -430,6 +430,8 @@ static int map_p2m_list(struct xc_sr_context *ctx,
> uint64_t p2m_cr3)
> 
>  if ( level == 2 )
>  {
> +if ( saved_idx == idx_end )
> +saved_idx++;
>  max_pfn = ((xen_pfn_t)saved_idx << 9) * fpp - 1;
>  if ( max_pfn < ctx->x86_pv.max_pfn )
>  {
> 
> 
> Juergen
> 




signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] xen-4.7 regression when saving a pv guest

[Stefan Bader]

On 25.08.2016 19:31, Juergen Gross wrote:
> On 25/08/16 17:48, Stefan Bader wrote:
>> When I try to save a PV guest with 4G of memory using xen-4.7 I get the
>> following error:
>>
>> II: Guest memory 4096 MB
>> II: Saving guest state to file...
>> Saving to /tmp/pvguest.save new xl format (info 0x3/0x0/1131)
>> xc: info: Saving domain 23, type x86 PV
>> xc: error: Bad mfn in p2m_frame_list[0]: Internal error
> 
> So the first mfn of the memory containing the p2m information is bogus.
> Weird.

Hm, not sure how bogus. From below the first mfn is 0x4eb1c8 and points to
pfn=0xff7c8 which is above the current max of 0xb. But then the dmesg inside
the guest said: "last_pfn = 0x10" which would be larger than the pfn causing
the error.

> 
>> xc: error: mfn 0x4eb1c8, max 0x82: Internal error
>> xc: error:   m2p[0x4eb1c8] = 0xff7c8, max_pfn 0xb: Internal error
>> xc: error: Save failed (34 = Numerical result out of range): Internal error
>> libxl: error: libxl_stream_write.c:355:libxl__xc_domain_save_done: saving
>> domain: domain did not respond to suspend request: Numerical result out of 
>> range
>> Failed to save domain, resuming domain
>> xc: error: Dom 23 not suspended: (shutdown 0, reason 255): Internal error
>> libxl: error: libxl_dom_suspend.c:460:libxl__domain_resume: xc_domain_resume
>> failed for domain 23: Invalid argument
>> EE: Guest not off after save!
>> FAIL
>>
>> From dmesg inside the guest:
>> [0.00] e820: last_pfn = 0x10 max_arch_pfn = 0x4
>>
>> Somehow I am slightly suspicious about
>>
>> commit 91e204d37f44913913776d0a89279721694f8b32
>>   libxc: try to find last used pfn when migrating
>>
>> since that seems to potentially lower ctx->x86_pv.max_pfn which is checked
>> against in mfn_in_pseudophysmap(). Is that a known problem?
>> With xen-4.6 and the same dom0/guest kernel version combination this does 
>> work.
> 
> Can you please share some more information? Especially:
> 
> - guest kernel version?
Hm, apparently 4.4 and 4.6 with stable updates. I just tried a much older guest
kernel (3.2) environment and that works. So it is the combination of switching
from xen-4.6 to 4.7 and guest kernels running 4.4 and later. And while the exact
mfn/pfn which gets dumped varies a little, the offending mapping always points
to 0xffxxx which would be below last_pfn.

Xen version 4.6 4.7
Guest Kernel
3.13.x  ok  ok
4.2.x   ok  ok
4.4.15  ok  fail
4.6.7   ok  fail

I will try 4.7 and 4.8 based guest kernels with xen-4.7 in a bit, too.

> - any patches in kernel not being upstream, especially in Xen-specific
None I know of.

>   boot path?
With affected kernels both direct kernel load and pvgrub.

> - dmesg from guest with E820 map?

From 4.4.x kernel:
[0.00] e820: BIOS-provided physical RAM map:
[0.00] e820: update [mem 0x-0x0fff] usable ==> reserved
[0.00] e820: remove [mem 0x000a-0x000f] usable
[0.00] e820: last_pfn = 0x10 max_arch_pfn = 0x4
[0.00] e820: cannot find a gap in the 32bit address range
   e820: PCI devices with unassigned 32bit BARs may break!
[0.00] e820: [mem 0x10010-0x1004f] available for PCI devices

Old 3.13 kernel (I see nothing different here):
[0.00] e820: BIOS-provided physical RAM map:
[0.00] e820: update [mem 0x-0x0fff] usable ==> reserved
[0.00] e820: remove [mem 0x000a-0x000f] usable
[0.00] e820: last_pfn = 0x10 max_arch_pfn = 0x4
[0.00] e820: cannot find a gap in the 32bit address range
[0.00] e820: PCI devices with unassigned 32bit BARs may break!
[0.00] e820: [mem 0x10010-0x1004f] available for PCI devices

> - guest configuration?

Rather simple (some of it ls /for historic reasons, I also tried externally
supplied kernel and initrd):

name = "testpv"
kernel   = "/root/boot/pv-grub-hd0--x86_64.gz"
memory   = 4096
vcpus= 4
disk = [
'file:/root/img/testpv.img,xvda1,w'
]
vif  = [ 'mac=xx:xx:xx:xx:xx:xx, bridge=br0' ]
on_crash = "coredump-destroy"

> 
> The same error would occur when trying to live migrate the guest. And
> this has been tested a lot since above commit, so I suspect something
> is very special in your case.
> 
> 
> Juergen
> 




signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] xen-4.7 regression when saving a PV guest

[Stefan Bader]

Sorry for the incomplete subject. Got interrupted while writing the email and
then forgot to complete it... :/

On 25.08.2016 17:48, Stefan Bader wrote:
> When I try to save a PV guest with 4G of memory using xen-4.7 I get the
> following error:
> 
> II: Guest memory 4096 MB
> II: Saving guest state to file...
> Saving to /tmp/pvguest.save new xl format (info 0x3/0x0/1131)
> xc: info: Saving domain 23, type x86 PV
> xc: error: Bad mfn in p2m_frame_list[0]: Internal error
> xc: error: mfn 0x4eb1c8, max 0x82: Internal error
> xc: error:   m2p[0x4eb1c8] = 0xff7c8, max_pfn 0xb: Internal error
> xc: error: Save failed (34 = Numerical result out of range): Internal error
> libxl: error: libxl_stream_write.c:355:libxl__xc_domain_save_done: saving
> domain: domain did not respond to suspend request: Numerical result out of 
> range
> Failed to save domain, resuming domain
> xc: error: Dom 23 not suspended: (shutdown 0, reason 255): Internal error
> libxl: error: libxl_dom_suspend.c:460:libxl__domain_resume: xc_domain_resume
> failed for domain 23: Invalid argument
> EE: Guest not off after save!
> FAIL
> 
> From dmesg inside the guest:
> [0.00] e820: last_pfn = 0x10 max_arch_pfn = 0x4
> 
> Somehow I am slightly suspicious about
> 
> commit 91e204d37f44913913776d0a89279721694f8b32
>   libxc: try to find last used pfn when migrating
> 
> since that seems to potentially lower ctx->x86_pv.max_pfn which is checked
> against in mfn_in_pseudophysmap(). Is that a known problem?
> With xen-4.6 and the same dom0/guest kernel version combination this does 
> work.
> 
> -Stefan
> 
> 
> 
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
> 




signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] xen-4.7

[Stefan Bader]

When I try to save a PV guest with 4G of memory using xen-4.7 I get the
following error:

II: Guest memory 4096 MB
II: Saving guest state to file...
Saving to /tmp/pvguest.save new xl format (info 0x3/0x0/1131)
xc: info: Saving domain 23, type x86 PV
xc: error: Bad mfn in p2m_frame_list[0]: Internal error
xc: error: mfn 0x4eb1c8, max 0x82: Internal error
xc: error:   m2p[0x4eb1c8] = 0xff7c8, max_pfn 0xb: Internal error
xc: error: Save failed (34 = Numerical result out of range): Internal error
libxl: error: libxl_stream_write.c:355:libxl__xc_domain_save_done: saving
domain: domain did not respond to suspend request: Numerical result out of range
Failed to save domain, resuming domain
xc: error: Dom 23 not suspended: (shutdown 0, reason 255): Internal error
libxl: error: libxl_dom_suspend.c:460:libxl__domain_resume: xc_domain_resume
failed for domain 23: Invalid argument
EE: Guest not off after save!
FAIL

From dmesg inside the guest:
[0.00] e820: last_pfn = 0x10 max_arch_pfn = 0x4

Somehow I am slightly suspicious about

commit 91e204d37f44913913776d0a89279721694f8b32
  libxc: try to find last used pfn when migrating

since that seems to potentially lower ctx->x86_pv.max_pfn which is checked
against in mfn_in_pseudophysmap(). Is that a known problem?
With xen-4.6 and the same dom0/guest kernel version combination this does work.

-Stefan



signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen 4.1 maintenance? (Was Re: [xen-4.1.6.1] SIGSEGV libxc/xc_save_domain.c: p2m_size >> configured_ram_size)

[Stefan Bader]

On 17.06.2016 11:31, George Dunlap wrote:
> On 13/06/16 12:14, Philipp Hahn wrote:
>> Am 13.06.2016 um 12:15 schrieb George Dunlap:
>>> On Fri, Jun 10, 2016 at 4:22 PM, Philipp Hahn <h...@univention.de> wrote:
>>>> while trying to live migrate some VMs from an xen-4.1.6.1 host "xc_save"
>>>> crashes with a segmentation fault in tools/libxc/xc_domain_save.c:1141
>>>>> /*
>>>>>  * Quick belt and braces sanity check.
>>>>>  */
>>>>> for ( i = 0; i < dinfo->p2m_size; i++ )
>>>>> {
>>>>> mfn = pfn_to_mfn(i);
>>>>> if( (mfn != INVALID_P2M_ENTRY) && (mfn_to_pfn(mfn) != i) )
>>>>  ^^^
>>>> due to a de-reference through
>>>>> #define pfn_to_mfn(_pfn)\
>>>>>   ((xen_pfn_t) ((dinfo->guest_width==8)   \
>>>>> ? (((uint64_t *)ctx->live_p2m)[(_pfn)])  \
>>>>> : uint32_t *)ctx->live_p2m)[(_pfn)]) == 0xU  \
>>>>>? (-1UL) : (((uint32_t *)ctx->live_p2m)[(_pfn)]
>> ...
>>> Given that 4.1 is long out of support, we won't be making a proper fix
>>> in-tree (since it will never be released).
>>
>> I know that 4.1 is EOL.
>> I'm aware of Ubuntu still having xen-4.1 in one of their LTS versions
>> (Precise) and its also in Debian-oldstable, which a lot people (us
>> included) still use. I would prefer to update, but I can for reasons
>> outside my direct control.
>>
>> I'm already working with Stefan Bader from Canonical to backport most of
>> the XSAs to 4.1, so there already exists a "better" version outside of
>> the official Xen repositories.
> 
> Philipp / Stefan -- if there really is a large following of people still
> using 4.1, would it make sense to have one or both of you step up and
> maintain an official branch on xenbits?

Hi George,

[sorry for the late answer, I was busy/on vacation the last two weeks]

I can only speak for myself. So for me 4,1 will be relevant only until April
next year. So not a full year anymore. Also my take would be that if taking up
an official stable branch it should be done properly which takes time. And the
pool of spare time is rather dry these days. So I'd rather decline the honour.

Though one thing to improve as a compromise would be to do a better job of
submitting hard(er) backports more consistently to xen-devel for review (usually
I am doing my batches after lifting embargoes anyway).

-Stefan
> 
>  -George
> 




signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] bad page flags booting 32bit dom0 on 64bit hypervisor using dom0_mem (kernel >=4.2)

[Stefan Bader]

On 02.05.2016 16:24, Stefan Bader wrote:
> On 02.05.2016 13:41, Juergen Gross wrote:
>> On 02/05/16 12:47, Stefan Bader wrote:
>>> I recently tried to boot 32bit dom0 on 64bit Xen host which I configured to 
>>> run
>>> with a limited, fix amount of memory for dom0. It seems that somewhere 
>>> between
>>> kernel versions 3.19 and 4.2 (sorry that is still a wide range) the Linux 
>>> kernel
>>> would report bad page flags for a range of pages (which seem to be around 
>>> the
>>> end of the guest pfn range). For a 4.2 kernel that was easily missed as the 
>>> boot
>>> finished ok and dom0 was accessible. However starting with 4.4 (tested 4.5 
>>> and a
>>> 4.6-rc) the serial console output freezes after some of those bad page flag
>>> messages and then (unfortunately without any further helpful output) the 
>>> host
>>> reboots (I assume there is a panic that triggers a reset).
>>>
>>> I suspect the problem is more a kernel side one. It is just possible to
>>> influence things by variation of dom0_mem=#,max:#. 512M seems ok, 1024M, 
>>> 2048M,
>>> and 3072M cause bad page flags starting around kernel 4.2 and reboots around
>>> 4.4. Then 4096M and not clamping dom0 memory seem to be ok again (though not
>>> limiting dom0 memory seems to cause trouble on 32bit dom0 later when a domU
>>> tries to balloon memory, but I think that is a different problem).
>>>
>>> I have not seen this on a 64bit dom0. Below is an example of those bad page
>>> errors. Somehow it looks to be a page marked as reserved. Initially I 
>>> wondered
>>> whether this could be a problem of not clearing page flags when moving 
>>> mappings
>>> to match the e820. But I never looked into i386 memory setup in that 
>>> detail. So
>>> I am posting this, hoping that someone may have an idea from the detail 
>>> about
>>> where to look next. PAE is enabled there. Usually its bpf init that gets 
>>> hit but
>>> that likely is just because that is doing the first vmallocs.
>>
>> Could you please post the kernel config, Xen and dom0 boot parameters?
>> I'm quite sure this is no common problem as there are standard tests
>> running for each kernel version including 32 bit dom0 with limited
>> memory size.
> 
> Hi Jürgen,
> 
> sure. Though by doing that I realized where I actually messed the whole thing
> up. I got the max limit syntax completely wrong. :( Instead of the correct
> "dom0_mem=1024M,max:1024M" I am using "dom0_mem=1024M:max=1024M" which I guess
> is like not having max set at all. Not sure whether that is a valid use case.
> 
> When I actually do the dom0_mem argument right, there are no bad page flag
> errors even in 4.4 with 1024M limit. I was at least consistent in my
> mis-configuration, so doing the same stupid thing on 64bit seems to be handled
> more gracefully.
> 
> Likely false alarm. But at least cut the config into mail made me spot
> the problem...
> 

Ok, thinking that "dom0_mem=x" (without a max or min) still is a valid case, I
went ahead and did a bisect for when the bad page flag issue started. I ended 
up at:

  92923ca "mm: meminit: only set page reserved in the memblock region"

And with a few more printks in the new functions I finally realized why this
goes wrong. The new reserve_bootmem_region is using unsigned long for start and
end addresses which just isn't working too well for 32bit.
For Xen dom0 the problem with that can just be more easily triggered. When dom0
memory is limited to a small size but allowed to balloon for more, the
additional system memory is put into reserved regions.
In my case a host with 8G memory and say 1G initial dom0 memory this created
(apart from other) one reserved region which started at 4GB and covered the
remaining 4G of host memory. Which reserve_bootmem_region() got as 0-4G due to
the unsigned long conversion. This basically marked *all* memory below 4G as
reserved.
The fix is relatively simple, just use phys_addr_t for start and end. I tested
this on 4.2 and 4.4 kernels. Both now boot without errors and neither does the
4.4 kernel crash. Maybe still not 100% safe when running on very large memory
systems (if I did not get the math wrong 16T) but at least some improvement...

-Stefan


From 1588a8b3983f63f8e690b91e99fe631902e38805 Mon Sep 17 00:00:00 2001
From: Stefan Bader <stefan.ba...@canonical.com>
Date: Tue, 10 May 2016 19:05:16 +0200
Subject: [PATCH] mm: Use phys_addr_t for reserve_bootmem_region arguments

Since 92923ca the reserved bit is set on reserved memblock regions.
However start and end address

[Xen-devel] bad page flags booting 32bit dom0 on 64bit hypervisor using dom0_mem (kernel >=4.2)

[Stefan Bader]

I recently tried to boot 32bit dom0 on 64bit Xen host which I configured to run
with a limited, fix amount of memory for dom0. It seems that somewhere between
kernel versions 3.19 and 4.2 (sorry that is still a wide range) the Linux kernel
would report bad page flags for a range of pages (which seem to be around the
end of the guest pfn range). For a 4.2 kernel that was easily missed as the boot
finished ok and dom0 was accessible. However starting with 4.4 (tested 4.5 and a
4.6-rc) the serial console output freezes after some of those bad page flag
messages and then (unfortunately without any further helpful output) the host
reboots (I assume there is a panic that triggers a reset).

I suspect the problem is more a kernel side one. It is just possible to
influence things by variation of dom0_mem=#,max:#. 512M seems ok, 1024M, 2048M,
and 3072M cause bad page flags starting around kernel 4.2 and reboots around
4.4. Then 4096M and not clamping dom0 memory seem to be ok again (though not
limiting dom0 memory seems to cause trouble on 32bit dom0 later when a domU
tries to balloon memory, but I think that is a different problem).

I have not seen this on a 64bit dom0. Below is an example of those bad page
errors. Somehow it looks to be a page marked as reserved. Initially I wondered
whether this could be a problem of not clearing page flags when moving mappings
to match the e820. But I never looked into i386 memory setup in that detail. So
I am posting this, hoping that someone may have an idea from the detail about
where to look next. PAE is enabled there. Usually its bpf init that gets hit but
that likely is just because that is doing the first vmallocs.

-Stefan


[4.748815] BUG: Bad page state in process swapper/0  pfn:3fc1e
[4.748861] page:f675a4b0 count:0 mapcount:0 mapping:  (null) index:0x0
[4.748908] flags: 0x3000400(reserved)
[4.748984] page dumped because: PAGE_FLAGS_CHECK_AT_PREP flag set
[4.749030] bad because of flags:
[4.749069] flags: 0x400(reserved)
[4.749143] Modules linked in:
[4.749201] CPU: 0 PID: 1 Comm: swapper/0 Tainted: GB
4.2.0-10-generic #12-Ubuntu
[4.749303] Hardware name: Intel Corporation ...
[4.749379]    f0cffcfc c1730710 f675a4b0 f0cffd20 c115be27
c194692c
[4.749584]  f0d503ec 0003fc1e 007f c1946eb8 c194a70e 0001 f0cffd8c
c115f4c3
[4.749790]  0002 0141 9069 c1ac5384  f11d7ce4 f11d7ce4
c1ac4dc0
[4.749993] Call Trace:
[4.750034]  [] dump_stack+0x41/0x52
[4.750078]  [] bad_page+0xb7/0x110
[4.750121]  [] get_page_from_freelist+0x2d3/0x610
[4.750168]  [] __alloc_pages_nodemask+0x146/0x8f0
[4.750215]  [] ? find_entry.isra.13+0x52/0x90
[4.750260]  [] ? kmem_cache_alloc_trace+0x175/0x1e0
[4.750308]  [] ? 
__raw_callee_save___pv_queued_spin_unlock+0x6/0x10
[4.750373]  [] ? __kmalloc+0x21d/0x240
[4.750417]  [] __vmalloc_node_range+0x10e/0x210
[4.750464]  [] ? bpf_prog_alloc+0x37/0xa0
[4.750509]  [] __vmalloc_node+0x66/0x70
[4.750553]  [] ? bpf_prog_alloc+0x37/0xa0
[4.750598]  [] __vmalloc+0x34/0x40
[4.750642]  [] ? bpf_prog_alloc+0x37/0xa0
[4.750687]  [] bpf_prog_alloc+0x37/0xa0
[4.750732]  [] bpf_prog_create+0x2c/0x90
[4.750776]  [] ? bsp_pm_check_init+0x11/0x11
[4.750821]  [] ptp_classifier_init+0x20/0x28
[4.750866]  [] ?
[4.750933]  [] sock_init+0x7c/0x83
[4.750977]  [] do_one_initcall+0xaa/0x200
[4.751021]  [] ? bsp_pm_check_init+0x11/0x11
[4.751067]  [] ? parse_args+0x2ad/0x540
[4.751112]  [] kernel_init_freeable+0x13a/0x1bc
[4.751158]  [] kernel_init+0x10/0xe0
[4.751203]  [] ? schedule_tail+0x11/0x50
[4.751251]  [] ret_from_kernel_thread+0x21/0x30
[4.751297]  [] ? rest_init+0x70/0x70

For reference some memory info from an Intel box with 8G physical memory, booted
with 1024M dom0 memory on a 4.2 kernel. The range of bad page pfn from syslog
was 3fc1e to 3fc3b. The reported pages (f675a4b0 to f675a938) all with the same
line about no mappings or references.

(XEN) Xen-e820 RAM map:
(XEN)  - 0009a400 (usable)
(XEN) 0009a400 - 000a (reserved)
(XEN) 000e - 0010 (reserved)
(XEN) 0010 - 30a48000 (usable)
(XEN) 30a48000 - 30a49000 (reserved)
(XEN) 30a49000 - a27f4000 (usable)
(XEN) a27f4000 - a2ab4000 (reserved)
(XEN) a2ab4000 - a2fb4000 (ACPI NVS)
(XEN) a2fb4000 - a2feb000 (ACPI data)
(XEN) a2feb000 - a300 (usable)
(XEN) a300 - afa0 (reserved)
(XEN) e000 - f000 (reserved)
(XEN) fec0 - fec01000 (reserved)
(XEN) fed0 - fed04000 (reserved)
(XEN) fed1 - fed1a000 (reserved)
(XEN) fed1c000 - fed2 (reserved)
(XEN) fed84000 - fed85000 (reserved)
(XEN) 

Re: [Xen-devel] xen-4.6: xenstored crashes during domain->interface access

[Stefan Bader]

On 26.01.2016 11:58, Stefan Bader wrote:
> Hi,
> 
> while playing around with xen-4.6 I stumbled over an odd problem and am
> wondering whether anybody has seen the same. A method to relatively quickly
> reproduce this for me seems to:
> 
> - Start one domU (PV or HVM does not seem to matter)
> - Repeatedly call xenstore-ls a few times
> 
> I think I never got beyond 10 repeats when the xenstore-ls call suddenly locks
> up and xenstored crashes with a SIGBUS error. In the majority of cases (I 
> think
> I saw one different), the crash happens while accessing 
> conn->domain->interface
> in tools/xenstore/xenstored_domain.c:domain_can_read().
> Looking at the corefile produced by xenstored I now got at least one case 
> where
> the pointer still matches the previously mapped value. Though I think I had 
> also
> at least one run (with less debugging added) where it seemed to be really 
> wrong.
> There is more info at [1] in case someone is interested.
> 
> I need to repeat a few more times to see how consistent the whole thing is. 
> Does
> this happen for anybody else? Any advice what I should look at (in the sense 
> of
> gathering better data)?

Just as an update and confirmation for Ian and Bastian: Debian testing is fine.
I have not dug into the specifics but its not the Xen package side at all.
Something in our 4.3 kernel causes this. Unfortunately without any hint in
dmesg. But since we move to 4.4 soon and I cannot reproduce it with the pending
4.4 build it seems good enough to me.

-Stefan
> 
> Thanks,
> Stefan
> 
> [1] https://bugs.launchpad.net/ubuntu/+source/xen/+bug/1538049
> 




signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] xen-4.6: xenstored crashes during domain->interface access

[Stefan Bader]

On 28.01.2016 10:39, Ian Campbell wrote:
> On Thu, 2016-01-28 at 09:50 +0100, Stefan Bader wrote:
>> On 26.01.2016 11:58, Stefan Bader wrote:
>>> Hi,
>>>
>>> while playing around with xen-4.6 I stumbled over an odd problem and am
>>> wondering whether anybody has seen the same. A method to relatively
>>> quickly
>>> reproduce this for me seems to:
>>>
>>> - Start one domU (PV or HVM does not seem to matter)
>>> - Repeatedly call xenstore-ls a few times
>>>
>>> I think I never got beyond 10 repeats when the xenstore-ls call
>>> suddenly locks
>>> up and xenstored crashes with a SIGBUS error. In the majority of cases
>>> (I think
>>> I saw one different), the crash happens while accessing conn->domain-
>>>> interface
>>> in tools/xenstore/xenstored_domain.c:domain_can_read().
>>> Looking at the corefile produced by xenstored I now got at least one
>>> case where
>>> the pointer still matches the previously mapped value. Though I think I
>>> had also
>>> at least one run (with less debugging added) where it seemed to be
>>> really wrong.
>>> There is more info at [1] in case someone is interested.
>>>
>>> I need to repeat a few more times to see how consistent the whole thing
>>> is. Does
>>> this happen for anybody else? Any advice what I should look at (in the
>>> sense of
>>> gathering better data)?
>>
>> Just as an update and confirmation for Ian and Bastian: Debian testing is 
>> fine.
>> I have not dug into the specifics but its not the Xen package side at all.
>> Something in our 4.3 kernel causes this. Unfortunately without any hint in
>> dmesg. But since we move to 4.4 soon and I cannot reproduce it with the 
>> pending
>> 4.4 build it seems good enough to me.
> 
> Ah, this is probably fixed by 9c17d96500f78 "xen/gntdev: Grant maps should
> not be subject to NUMA balancing" then.

Oh right. That sounds very possible. Maybe paired with balancing done even on a
non-NUMA system (because I saw the same happen on a non-NUMA host, too). And I
cannot remember anytime having this with 4.2, so 4.3 seems to have introduced
the additional (or maybe more aggressive) balancing.
But the result pretty much was what I saw. That from one second to the next the
grant-table page of xenstored for the running domU was invalid. Without the
daemon having done any unmap. So yeah, likely the balancing got rid of it.

-Stefan
> 
> Ian.
> 




signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] xen-4.6: xenstored crashes during domain->interface access

[Stefan Bader]

Hi,

while playing around with xen-4.6 I stumbled over an odd problem and am
wondering whether anybody has seen the same. A method to relatively quickly
reproduce this for me seems to:

- Start one domU (PV or HVM does not seem to matter)
- Repeatedly call xenstore-ls a few times

I think I never got beyond 10 repeats when the xenstore-ls call suddenly locks
up and xenstored crashes with a SIGBUS error. In the majority of cases (I think
I saw one different), the crash happens while accessing conn->domain->interface
in tools/xenstore/xenstored_domain.c:domain_can_read().
Looking at the corefile produced by xenstored I now got at least one case where
the pointer still matches the previously mapped value. Though I think I had also
at least one run (with less debugging added) where it seemed to be really wrong.
There is more info at [1] in case someone is interested.

I need to repeat a few more times to see how consistent the whole thing is. Does
this happen for anybody else? Any advice what I should look at (in the sense of
gathering better data)?

Thanks,
Stefan

[1] https://bugs.launchpad.net/ubuntu/+source/xen/+bug/1538049



signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Fwd: Xen-4.1.6.1 backport for XSA156

[Stefan Bader]

On 23.11.2015 08:51, Jan Beulich wrote:
 On 23.11.15 at 08:37,  wrote:
>> Actually there's no problem with ICEBP - just like INTnn it isn't itself
>> interceptable (and the injection of vector 0x01 from the x86
>> emulator path can't fully distinguish between ICEBP and INT $1  in
>> these old versions anyway). So what you have should be good
>> enough, albeit I think I'll code it slightly differently (keeping the fall-
>> through in place).
> 
> Like this:
> 
> @@ -1364,7 +1358,6 @@ void vmx_inject_hw_exception(int trap, i
>  switch ( trap )
>  {
>  case TRAP_debug:
> -type = X86_EVENTTYPE_SW_EXCEPTION;
>  if ( guest_cpu_user_regs()->eflags & X86_EFLAGS_TF )
>  {
>  __restore_debug_registers(curr);
> @@ -1379,9 +1372,11 @@ void vmx_inject_hw_exception(int trap, i
>  domain_pause_for_debugger();
>  return;
>  }
> -
> -type = X86_EVENTTYPE_SW_EXCEPTION;
> -__vmwrite(VM_ENTRY_INSTRUCTION_LEN, 1); /* int3 */
> +if ( trap == TRAP_int3 )
> +{
> +type = X86_EVENTTYPE_SW_EXCEPTION;
> +__vmwrite(VM_ENTRY_INSTRUCTION_LEN, 1);
> +}
>  }
>  
>  if ( unlikely(intr_info & INTR_INFO_VALID_MASK) &&
> 

Yeah, for my side I think I stick with what I had because I already have now run
that variant through testing. But I will include both variants when talking to
the Debian guys.

-Stefan
> Jan
> 




signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] Fwd: Xen-4.1.6.1 backport for XSA156

[Stefan Bader]

Hi Jan, hi Andrew,

I am currently trying to backport the changes of XSA156 back to Xen-4.1.x and I
am struggling with the VMX side. I did see the backports made for 4.2 and 3.4 on
the security mailing list but I am not sure the 3.4 backport is not having the
same issues (or similar ones).

Trying to write down my understanding of the changes: For the 3.4 backport there
are only changes to the toggles for debugging and the general trap mask. So if I
understand this right, before the change, TRAP_debug and TRAP_int3 were only
handled in vmexit when a debugger was attached to the domain. Now, only
TRAP_int3 will be toggled and TRAP_debug is always handled.

My testing does (beside other things) involve some verification of ptrace
handling. Which on 4.1.x with the changes, now causes a crash of the HVM guest
in vm_resume (vm_resume_fail error code 7).

I think this is caused by TRAP_debug being handled in vmexit. I don't have the
3.4 code so not sure whether there is anything handling it. In the 4.1.x case
and without changing the vmexit code in xen/arch/x86/hvm/vmx/vmx.c it would be a
certain crash as no domain debugging is done. The problem seems to be that I do
inject (as the 4.2 patch does) an exception. Though 4.1.x does not, yet, have
the changes from "xen: Define new struct hvm_trap and cleanup vmx exception", so
I only have either hvm_inject_exception or vmx_inject_hw_exception. The former
ends up calling the latter. What I think is the problem (which svm does not
have) is the debug/int3 handling in the function below. This seems to convert
the exception unconditionally into a software exception that has an opcode
associated. Would you also think this is the issue? And if yes, is there any
sane way you can think of to prevent this without having to resort to pulling in
large hunks of rewrite?

Regards,
Stefan

vmx_inject_hw_exception(
  ...
switch ( trap )
{
case TRAP_debug:
type = X86_EVENTTYPE_SW_EXCEPTION;
if ( guest_cpu_user_regs()->eflags & X86_EFLAGS_TF )
{
__restore_debug_registers(curr);
write_debugreg(6, read_debugreg(6) | 0x4000);
}
if ( cpu_has_monitor_trap_flag )
break;
case TRAP_int3:
if ( curr->domain->debugger_attached )
{
/* Debug/Int3: Trap to debugger. */
domain_pause_for_debugger();
return;
}

type = X86_EVENTTYPE_SW_EXCEPTION;
__vmwrite(VM_ENTRY_INSTRUCTION_LEN, 1); /* int3 */
  ...
}







signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Fwd: Xen-4.1.6.1 backport for XSA156

[Stefan Bader]

On 20.11.2015 17:10, Stefan Bader wrote:
> On 20.11.2015 16:59, Jan Beulich wrote:
>>>>> On 20.11.15 at 16:03, <stefan.ba...@canonical.com> wrote:
>>> I am currently trying to backport the changes of XSA156 back to Xen-4.1.x 
>>> and I
>>> am struggling with the VMX side. I did see the backports made for 4.2 and 
>>> 3.4 on
>>> the security mailing list but I am not sure the 3.4 backport is not having 
>>> the
>>> same issues (or similar ones).
>>>
>>> Trying to write down my understanding of the changes: For the 3.4 backport 
>>> there
>>> are only changes to the toggles for debugging and the general trap mask. So 
>>> if I
>>> understand this right, before the change, TRAP_debug and TRAP_int3 were only
>>> handled in vmexit when a debugger was attached to the domain. Now, only
>>> TRAP_int3 will be toggled and TRAP_debug is always handled.
>>
>> I've never looked at that 3.4 backport, but not changing the VMEXIT
>> handling certainly sounds wrong. I'll attach what I have done for 4.1.
>> Please report back any problems you encounter.
> 
> If I am not missing any detail your 4.1 patch looks exactly the same as the
> version I ended up with (basically dropping some trace).
> Have you tested the resulting HV on an Intel/VMX box and tried to use ptrace
> inside the HVM guest?
> 
> This is where my problems come from. Or potentially your 
> vmx_inject_hw_exception
> has been modified since stable-4.1.6.1?

So this is a quick hack I just tried and that keeps the HVM alive:

@@ -1294,7 +1288,6 @@ void vmx_inject_hw_exception(int trap, i
 switch ( trap )
 {
 case TRAP_debug:
-type = X86_EVENTTYPE_SW_EXCEPTION;
 if ( guest_cpu_user_regs()->eflags & X86_EFLAGS_TF )
 {
 __restore_debug_registers(curr);
@@ -1302,6 +1295,13 @@ void vmx_inject_hw_exception(int trap, i
 }
 if ( cpu_has_monitor_trap_flag )
 break;
+if ( curr->domain->debugger_attached )
+{
+/* Debug/Int3: Trap to debugger. */
+domain_pause_for_debugger();
+return;
+}
+break;
 case TRAP_int3:
 if ( curr->domain->debugger_attached )
 {

Though this looks like an ugly hack and probably is wrong in the other case of
TRAP_debug caused by an opcode...

-Stefan

> 
> -Stefan
>>
>> Jan
>>
> 
> 
> 
> 
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 




signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Fwd: Xen-4.1.6.1 backport for XSA156

[Stefan Bader]

On 20.11.2015 16:59, Jan Beulich wrote:
 On 20.11.15 at 16:03,  wrote:
>> I am currently trying to backport the changes of XSA156 back to Xen-4.1.x 
>> and I
>> am struggling with the VMX side. I did see the backports made for 4.2 and 
>> 3.4 on
>> the security mailing list but I am not sure the 3.4 backport is not having 
>> the
>> same issues (or similar ones).
>>
>> Trying to write down my understanding of the changes: For the 3.4 backport 
>> there
>> are only changes to the toggles for debugging and the general trap mask. So 
>> if I
>> understand this right, before the change, TRAP_debug and TRAP_int3 were only
>> handled in vmexit when a debugger was attached to the domain. Now, only
>> TRAP_int3 will be toggled and TRAP_debug is always handled.
> 
> I've never looked at that 3.4 backport, but not changing the VMEXIT
> handling certainly sounds wrong. I'll attach what I have done for 4.1.
> Please report back any problems you encounter.

If I am not missing any detail your 4.1 patch looks exactly the same as the
version I ended up with (basically dropping some trace).
Have you tested the resulting HV on an Intel/VMX box and tried to use ptrace
inside the HVM guest?

This is where my problems come from. Or potentially your vmx_inject_hw_exception
has been modified since stable-4.1.6.1?

-Stefan
> 
> Jan
> 




signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Fwd: Xen-4.1.6.1 backport for XSA156

[Stefan Bader]

On 20.11.2015 17:54, Jan Beulich wrote:
 On 20.11.15 at 17:15,  wrote:
>> So this is a quick hack I just tried and that keeps the HVM alive:
>>
>> @@ -1294,7 +1288,6 @@ void vmx_inject_hw_exception(int trap, i
>>  switch ( trap )
>>  {
>>  case TRAP_debug:
>> -type = X86_EVENTTYPE_SW_EXCEPTION;
>>  if ( guest_cpu_user_regs()->eflags & X86_EFLAGS_TF )
>>  {
>>  __restore_debug_registers(curr);
>> @@ -1302,6 +1295,13 @@ void vmx_inject_hw_exception(int trap, i
>>  }
>>  if ( cpu_has_monitor_trap_flag )
>>  break;
>> +if ( curr->domain->debugger_attached )
>> +{
>> +/* Debug/Int3: Trap to debugger. */
>> +domain_pause_for_debugger();
>> +return;
>> +}
>> +break;
>>  case TRAP_int3:
>>  if ( curr->domain->debugger_attached )
>>  {
>>
>> Though this looks like an ugly hack and probably is wrong in the other case 
>> of
>> TRAP_debug caused by an opcode...
> 
> Right, and I'm afraid this case doesn't get handled correctly even on
> -unstable now. But apart from that aspect I think the change above
> is okay.

Oh, ok. Thanks for the review. I guess then I go with that. At least this does
no longer crash.

-Stefan

> 
> Jan
> 




signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] ARM64: XEN Domu not booting with the qemu qcow AARCH64 Ubuntu 15.04 disk

[Stefan Bader]

On 02.06.2015 09:40, Sanjeev Pandita wrote:
 All,
 
 I am pretty new to xen . I am trying to boot DOMU with qemu qcow AARCH64
 Ubuntu 15.04 disk on Xen but I am getting the errors which link to
 /usr/local/lib/xen/bin/qemu-system-i386.
 Since I am working on aarch64 system the
 /usr/local/lib/xen/bin/qemu-system-i386 bin might not be present or might
 not work as expected.

Because I am lacking hardware and feedback, the arm64 packaging is a rather
theoretical exercise. At least for armhf I thought qemu-system-x86 was a
dependency. That binary should provide x86 emulation on arm64, the same as one
could install qemu for other arches on x86.
Have you tried to install qemu-system-x86 manually?

-Stefan

 
 Please let me know how to make the Qemu qcow image work on Xen.
 Attached are the DomU boot log and config file.
 
 Thanks,
 San
 
 
 
 ___
 Xen-devel mailing list
 Xen-devel@lists.xen.org
 http://lists.xen.org/xen-devel
 




signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] ARM64: XEN Domu not booting with the qemu qcow AARCH64 Ubuntu 15.04 disk

[Stefan Bader]

On 02.06.2015 12:35, Stefano Stabellini wrote:
 On Tue, 2 Jun 2015, Stefano Stabellini wrote:
 On Tue, 2 Jun 2015, Stefan Bader wrote:
 On 02.06.2015 09:40, Sanjeev Pandita wrote:
 All,

 I am pretty new to xen . I am trying to boot DOMU with qemu qcow AARCH64
 Ubuntu 15.04 disk on Xen but I am getting the errors which link to
 /usr/local/lib/xen/bin/qemu-system-i386.
 Since I am working on aarch64 system the
 /usr/local/lib/xen/bin/qemu-system-i386 bin might not be present or might
 not work as expected.

 Because I am lacking hardware and feedback, the arm64 packaging is a rather
 theoretical exercise. At least for armhf I thought qemu-system-x86 was a
 dependency. That binary should provide x86 emulation on arm64, the same as 
 one
 could install qemu for other arches on x86.
 Have you tried to install qemu-system-x86 manually?

 Hi Stefan,

 On arm and arm64 Xen still needs a qemu-system-i386 binary, just to
 provide the PV backends in userspace (disk, console, etc.).
 Unfortunately the output binary is still named qemu-system-i386. I
 know that the name is misleading, but fixing it is not trivial: it
 requires disentangling code in QEMU in non trivial ways.
 
 Just to be clear, qemu-system-i386 for ARM is the output of a QEMU build
 on ARM with ./configure --enable-xen --target-list=i386-softmmu. It
 could do x86 emulation, but it does not when used on Xen.
 

Hi Stefano,

so for Debian and Ubuntu we moved to use the standard qemu binary which is build
with xen enabled. This works on x86, but I could not verify correctness for any
arm port (due to lack of hw).

-Stefan



signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] ARM64: XEN Domu not booting with the qemu qcow AARCH64 Ubuntu 15.04 disk

[Stefan Bader]

On 02.06.2015 14:09, Sanjeev Pandita wrote:
 Hi,
 
 -Original Message-
 From: Stefan Bader [mailto:stefan.ba...@canonical.com]
 Sent: Tuesday, June 02, 2015 1:52 PM
 To: Sanjeev Pandita; xen-devel@lists.xen.org
 Cc: ian.campb...@citrix.com; Pranavkumar Sawargaonkar;
 stefano.stabell...@eu.citrix.com
 Subject: Re: [Xen-devel] ARM64: XEN Domu not booting with the qemu qcow
 AARCH64 Ubuntu 15.04 disk
 
 On 02.06.2015 09:40, Sanjeev Pandita wrote:
 All,

 I am pretty new to xen . I am trying to boot DOMU with qemu qcow
 AARCH64 Ubuntu 15.04 disk on Xen but I am getting the errors which
 link to /usr/local/lib/xen/bin/qemu-system-i386.
 Since I am working on aarch64 system the
 /usr/local/lib/xen/bin/qemu-system-i386 bin might not be present or
 might not work as expected.
 
 Because I am lacking hardware and feedback, the arm64 packaging is a
 rather theoretical exercise. At least for armhf I thought qemu-system-x86
 was a dependency. That binary should provide x86 emulation on arm64, the
 same as one could install qemu for other arches on x86.
 Have you tried to install qemu-system-x86 manually?
 
 -Stefan
 

 Please let me know how to make the Qemu qcow image work on Xen.
 Attached are the DomU boot log and config file.

 Thanks,
 San
 
 Thanks for your inputs, I have installed the qemu-system-i386 but my DomU
 booting is still crashing with following short logs. Am I missing anything
 ?
 
 Kernel Crash logs:
 
 xenbus_probe_frontend: Waiting for devices to initialise:
 25s...20s...15s...10s...5s...0s...
 235s...230s...225s...220s...215s...210s...205s...200s...195s...190s...185s
 ...180s...175s...170s...165s...160s...155s...150s...145s...140s...135s...1
 30s...125s...120s...115s...110s...105s...100s...95s...90s...85s...80s...75
 s...70s...65s...60s...55s...50s...45s...40s...35s...30s...25s...20s...15s.
 ..10s...5s...0s...
 
 xenbus_probe_frontend: Timeout connecting to device: device/vbd/51712
 (local state 3, remote state 2)
 console [netcon0] enabled
 netconsole: network logging started
 drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
 VFS: Cannot open root device xvda or unknown-block(0,0): error -6
 Please append a correct root= boot option; here are the available
 partitions:
 Kernel panic - not syncing: VFS: Unable to mount root fs on
 unknown-block(0,0)
 CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.19.8 #5
 Hardware name: XENVM-4.6 (DT)
 Call trace:
 [ffc8a0dc] dump_backtrace+0x0/0x124
 [ffc8a210] show_stack+0x10/0x1c
 [ffc000657d88] dump_stack+0x80/0xc4
 [ffc000656f04] panic+0xe0/0x220
 [ffc00087eea8] mount_block_root+0x1a4/0x24c
 [ffc00087f19c] mount_root+0x110/0x130
 [ffc00087f328] prepare_namespace+0x16c/0x1b8
 [ffc00087eb44] kernel_init_freeable+0x1c4/0x1ec
 [ffc00065481c] kernel_init+0xc/0xd8
 ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on
 unknown-block(0,0)
 
 
 My config file  :
 
 linux:/mnt/xen # cat vm1
 name = vm1
 uuid = 3fb78ba6-8182-484c-acf7-8faba9773f68
 disk = [ 'tap:qcow:/mnt/xen/vivid-server-cloudimg-arm64-disk1.img,xvda,w'

We do not have blktap support. Maybe dropping that prefix helps...

 ]
 memory = 512
 kernel = ./Image
 extra = root=/dev/xvda rw console=hvc0 earlyprintk=xen
 
 
 Regards,
 San
 
 



 ___
 Xen-devel mailing list
 Xen-devel@lists.xen.org
 http://lists.xen.org/xen-devel

 




signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] Using kexec-crashdump with recent Xen and Linux HVM

[Stefan Bader]

After being asked about this I started to play around with Xen-4.4.1/4.5
together with HVM Linux guest running 3.13/3.16/3.19. With mixed success.
Usually rather failing.

From a bit of research most activity to enable things were back in 2011. There
was a bit of a throwback around Linux 3.2[1] but it appears [2] restored this in
a backwards compatible way. Around 3.17 xen_nopv was introduced but I have not
figured out a helpful usage of this.

The failure exhibits no visible messages in the guest after the crash stacktrace
caused by sysreq-trigger while 1vcpu seems to be in a spinning loop. On the host
side I noticed changing messages (TX queue drain or EVCHNOP errors).

Command-line is a mix of nomodeset (to keep cirrusdrm away) and some defaults
from the kdump-tools (maxcpus=1 irqpoll nousb).

The closest thing to success I can get to is using xen_emul_unplug=never for the
normal boot (which propagates into the kexec command). This of course stops
usage of the pv drivers. I also tried some variation of blacklisting the
emulated drivers and using xen_emul_unplug=unnecessary but that did not seem to
work for me. The crash-kexec boot without unplugging still fails to bring up the
NIC but at least finds the root disk to store the dump there. But not using the
pv drivers is not a setup one would want to have running just in case.

So I was wondering whether I still miss something.

-Stefan



[1] commit 12275dd4b747f5d87fa36229774d76bca8e63068
Revert xen/pv-on-hvm kexec: add xs_reset_watches to shutdown watches...
[2] commit cb6b6df111e46b9d0f79eb971575fd50555f43f4
xen/pv-on-hvm kexec: add quirk for Xen 3.4 and shutdown watches.



signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Crash in acpi_ps_peek_opcode when booting kernel 3.19 as Xen dom0

[Stefan Bader]

On 05.02.2015 15:33, Stefan Bader wrote:
 While experimenting/testing various kernel versions I discovered that trying 
 to
 boot a Haswell based hosts will always crash when booting as Xen dom0
 (Xen-4.4.1). The same crash happens since v3.19-rc1 and still does happen with
 v3.19-rc7. A bare metal boot is having no issues and also an Opteron based 
 host
 is having no issues (dom0 and bare metal).
 Could be a table that the other host does not have and since its only 
 happening
 in dom0 maybe some cpu capability that needs to be passed on?

I think I may have some more data here. I tried some patches which Juergen sent
me, but those were not changing much. I found that the problem is related on
that host to the use of dom0_mem= and may be a crash like below or a hang or
weird state in general.
When not using dom0_mem, I can boot with a 3.19 kernel, otherwise (trying 512M
and 1G) there is trouble. What is special about this host is that is has more
holes than the other machine I usually use.

(XEN) Xen-e820 RAM map:
(XEN)   - 0009a400 (usable)
(XEN)  0009a400 - 000a (reserved)
(XEN)  000e - 0010 (reserved)
   The first hole is common
(XEN)  0010 - 30a48000 (usable)
(XEN)  30a48000 - 30a49000 (reserved)
(XEN)  30a49000 - a27f4000 (usable)
   But then normally there is only one usable area up to
   around ACPI_NVS
(XEN)  a27f4000 - a2ab4000 (reserved)
(XEN)  a2ab4000 - a2fb4000 (ACPI NVS)
(XEN)  a2fb4000 - a2feb000 (ACPI data)
(XEN)  a2feb000 - a300 (usable)
(XEN)  a300 - afa0 (reserved)
(XEN)  e000 - f000 (reserved)
(XEN)  fec0 - fec01000 (reserved)
(XEN)  fed0 - fed04000 (reserved)
(XEN)  fed1 - fed1a000 (reserved)
(XEN)  fed1c000 - fed2 (reserved)
(XEN)  fed84000 - fed85000 (reserved)
(XEN)  fee0 - fee01000 (reserved)
(XEN)  ffc0 - 0001 (reserved)
(XEN)  0001 - 00024e60 (usable)
   Also after ACPI data there is some usable, and then another
   hole (area) which is unuasual.

So I added a bit more debug printk's: Here a boot with dom0_mem=512M:max=512M:

[0.00] SMB: remap 154(0x9A)-256(0x100) - 131072(0x2)
   == 0x09A000-0x10 - 0x2000 (@512M+)
   == 0x09A000-0x09A3FF was usable but partial

The first hole is supposed to be remapped as it is below the 512M which are in
the initial MFN list. I suppose this works but Juergen, I really would love to
understand how and I am not sure I grasp things. To me it looks like the remap
info is stored in the memory area to be mapped... which is reserved(?!)

I think the problem comes from these other holes (which are beyond 512M). When
not using dom0_mem those are remapped (like the first one), while with the clamp
they supposedly should be identity mapped...

[0.00] SMB: prange id 199240(0x30A48) - 199241(0x30A49)
   == 0x30A48000(~778M)
[0.00] SMB: prange id 665588(0xA27F4) - 667627(0xA2FEB)
   == 0xA27F4000(~2599M)
[0.00] SMB: prange id 667648(0xA3000) - 1048576(0x10)
   == 0xA300(~2608M)-0x1(=4G) id mapped
[0.00] Released 0 page(s)
[0.00] Remapped 102 page(s)

So here is xen_set_identity_and_remap_chunk():

...
while (i  n) {
  ...
  /* Do not remap pages beyond the current allocation */
  if (cur_pfn = nr_pages) {
/* Identity map remaining pages */
set_phys_range_identity(cur_pfn, cur_pfn + size);
break;
  }
  ...

Now, I think the call to set_phys_range_identity() is really doing nothing
because nr_pages really is the same (or mostly beside of an 512 alignment) as
xen_p2m_size, so it just returns 0.

  ...
  /*
   * If the PFNs are currently mapped, the VA mapping also needs
   * to be updated to be 1:1.
   */
  for (pfn = start_pfn; pfn = max_pfn_mapped  pfn  end_pfn; pfn++)
  (void)HYPERVISOR_update_va_mapping(
  (unsigned long)__va(pfn  PAGE_SHIFT),
  mfn_pte(pfn, PAGE_KERNEL_IO), 0);

I cannot make my head up about this one. Before this all changed, there was code
that resembled this loop but was rather clearing the mapping (except for a range
below 1M). Ok, that was done then in a different order which set identity
mapping after...

My feeling is that the problem comes from assuming identity mapping for holes
after the initial mapping. I might miss something but I cannot really see where
this could be recovered.

-Stefan
 
 [2.108038] ACPI: Core revision 20141107
 [2.108153] ACPI Warning: Unsupported module-level executable opcode 0x91 
 at
 table offset 0x002B (20141107/psloop-225)
 [2.108264] ACPI Warning: Unsupported module-level executable opcode

Re: [Xen-devel] Booting dom0, various issues since v3.16 kernels

[Stefan Bader]

On 05.02.2015 15:47, Sander Eikelenboom wrote:
 
 Thursday, February 5, 2015, 3:22:49 PM, you wrote:
 
 Hey David,
 
 after just being in that pain, I thought I might as well give a summary to
 you/the list. Maybe helpful to not forget which piece should go to which 
 stable...
 
 So:
 v3.16...v3.17.8: Somewhen in between those, the acpi irq seems to have 
 broken.
  I have not yet verified that, but at least three changes in
  3.19-rc6 seem to look related:
 
  * x86/xen: Treat SCI interrupt as normal GSI interrupt,
  * ACPI: pci: Do not clear pci_dev-irq in acpi_pci_irq_disable(), and
  * x86/xen: Override ACPI IRQ management callback __acpi_unregister_gsi
 
 Yes Jiang Liu fixed those and said he would backport the required fixes once
 they where accepted in mainline, put perhaps a polite ping is necessary there.
 
 v3.17.8...v3.18.4: Beside the acpi interrupt, no USB devices (beyond the
hubs) get initialized. Not sure what fixed it, but it
looks ok in v3.19-rc7.
Beside that, there also was a regression in swiotlb
that I think was passed on to some stable maintainers:
 
 Probably the same issue as above (for me it fixed a powerbutton issue and some
 pci-passthrough problems).
 
 And there seems to be more refactoring coming for 3.20 .. so fingerscrossed.
 
 --
 Sander

Hi Sander,

sorry, I know you did the ping somewhere in another thread. The one which I
cannot find again right now. :/ Maybe you can forward the following two patches
which would be my backport to 3.18. I hope things are correct. I dropped the
middle patch as it did not seem to be needed and mushed around the other two.
At least things seemed to be fixed up on a quick test-boot.

-Stefan


 
  * Revert swiotlb-xen: pass dev_addr to swiotlb_tbl_unmap_single
 
 v3.18.4..v3.19-rc7: The issues above look to be fixed. Only some Haswell
 based box now crashed on boot as dom0 while parsing
 some ACPI tables (will send more detail seperately).
 This happens only on that host and only when running
 as dom0. Bare-metal is ok and an Opteron based different
 host is also fine.
 
 -Stefan
 
 
 
 
 ___
 Xen-devel mailing list
 Xen-devel@lists.xen.org
 http://lists.xen.org/xen-devel
 

From 8b5b328b62248d95743ca9af7aa71c06dd808dfe Mon Sep 17 00:00:00 2001
From: Jiang Liu jiang@linux.intel.com
Date: Tue, 20 Jan 2015 10:21:05 +0800
Subject: [PATCH 1/2] x86/xen: Treat SCI interrupt as normal GSI interrupt

Currently Xen Domain0 has special treatment for ACPI SCI interrupt,
that is initialize irq for ACPI SCI at early stage in a special way as:
xen_init_IRQ()
	-pci_xen_initial_domain()
		-xen_setup_acpi_sci()
			Allocate and initialize irq for ACPI SCI

Function xen_setup_acpi_sci() calls acpi_gsi_to_irq() to get an irq
number for ACPI SCI. But unfortunately acpi_gsi_to_irq() depends on
IOAPIC irqdomains through following path
acpi_gsi_to_irq()
	-mp_map_gsi_to_irq()
		-mp_map_pin_to_irq()
			-check IOAPIC irqdomain

For PV domains, it uses Xen event based interrupt manangement and
doesn't make uses of native IOAPIC, so no irqdomains created for IOAPIC.
This causes Xen domain0 fail to install interrupt handler for ACPI SCI
and all ACPI events will be lost. Please refer to:
https://lkml.org/lkml/2014/12/19/178

So the fix is to get rid of special treatment for ACPI SCI, just treat
ACPI SCI as normal GSI interrupt as:
acpi_gsi_to_irq()
	-acpi_register_gsi()
		-acpi_register_gsi_xen()
			-xen_register_gsi()

With above change, there's no need for xen_setup_acpi_sci() anymore.
The above change also works with bare metal kernel too.

Signed-off-by: Jiang Liu jiang@linux.intel.com
Tested-by: Sander Eikelenboom li...@eikelenboom.it
Cc: Tony Luck tony.l...@intel.com
Cc: xen-de...@lists.xenproject.org
Cc: Konrad Rzeszutek Wilk konrad.w...@oracle.com
Cc: David Vrabel david.vra...@citrix.com
Cc: Rafael J. Wysocki r...@rjwysocki.net
Cc: Len Brown len.br...@intel.com
Cc: Pavel Machek pa...@ucw.cz
Cc: Bjorn Helgaas bhelg...@google.com
Link: http://lkml.kernel.org/r/1421720467-7709-2-git-send-email-jiang@linux.intel.com
Signed-off-by: Thomas Gleixner t...@linutronix.de
Signed-off-by: Stefan Bader stefan.ba...@canonical.com
---
 arch/x86/kernel/acpi/boot.c | 23 +++---
 arch/x86/pci/xen.c  | 47 -
 2 files changed, 12 insertions(+), 58 deletions(-)

diff --git a/arch/x86/kernel/acpi/boot.c b/arch/x86/kernel/acpi/boot.c
index a142e77..460f498 100644
--- a/arch/x86/kernel/acpi/boot.c
+++ b/arch/x86/kernel/acpi/boot.c
@@ -604,18 +604,19 @@ void __init acpi_pic_sci_set_trigger(unsigned int irq, u16 trigger)
 
 int acpi_gsi_to_irq(u32 gsi, unsigned int *irqp)
 {
-	int irq;
-
-	if (acpi_irq_model == ACPI_IRQ_MODEL_PIC) {
-		*irqp = gsi;
-	} else

Re: [Xen-devel] Crash in acpi_ps_peek_opcode when booting kernel 3.19 as Xen dom0

[Stefan Bader]

On 09.02.2015 14:07, Stefan Bader wrote:
 On 05.02.2015 20:36, Konrad Rzeszutek Wilk wrote:
 On Thu, Feb 05, 2015 at 03:33:02PM +0100, Stefan Bader wrote:
 While experimenting/testing various kernel versions I discovered that 
 trying to
 boot a Haswell based hosts will always crash when booting as Xen dom0
 (Xen-4.4.1). The same crash happens since v3.19-rc1 and still does happen 
 with
 v3.19-rc7. A bare metal boot is having no issues and also an Opteron based 
 host
 is having no issues (dom0 and bare metal).
 Could be a table that the other host does not have and since its only 
 happening
 in dom0 maybe some cpu capability that needs to be passed on?

 Usually it means that the ACPI AML code is trying to do something with
 the IOAPIC or something wihch is not accessible.

 But this on the other hand looks to be trying to execute some AML code
 that is unknown. Any chance you cna disassemble it and perhaps also
 run with acpi debug options on to figure out where it blows up?
 
 The weird thing here is that bare-metal on the same machine does work. And
 previous kernels did work as well. So I think we can assume the ACPI tables 
 are
 ok. It could even be a red-herring. Well, likely is as booting with acpi=off
 does hang instead of crashing.
 
 Since I got no clue, I did what we always do when we are dumbfound, I went 
 ahead
 and bisected 3.18..3.19-rc1. Unfortunately the very last kernel I build was
 something in between good and bad. Good as it did not crash exactly but bad as
 it did not come up in a usable state. So I would not be sure the claimed to be
 offending commit is right. Could be one in the range of:
 
 G  * xen: use common page allocation function in p2m.c
* xen: Delay remapping memory of pv-domain
 g  * xen: Delay m2p_override initialization
 - * xen: Delay invalidating extra memory
 B  * x86: Introduce function to get pmd entry pointer
 
 (G) really good, (g) somewhat not bad, (B) bad, (-) claimed first broken.

Oh, since that all sounds related to E820 in some way:

(XEN) Xen-e820 RAM map:
(XEN)   - 0009a400 (usable)
(XEN)  0009a400 - 000a (reserved)
(XEN)  000e - 0010 (reserved)
(XEN)  0010 - 30a48000 (usable)
(XEN)  30a48000 - 30a49000 (reserved)
(XEN)  30a49000 - a27f4000 (usable)
(XEN)  a27f4000 - a2ab4000 (reserved)
(XEN)  a2ab4000 - a2fb4000 (ACPI NVS)
(XEN)  a2fb4000 - a2feb000 (ACPI data)
(XEN)  a2feb000 - a300 (usable)
(XEN)  a300 - afa0 (reserved)
(XEN)  e000 - f000 (reserved)
(XEN)  fec0 - fec01000 (reserved)
(XEN)  fed0 - fed04000 (reserved)
(XEN)  fed1 - fed1a000 (reserved)
(XEN)  fed1c000 - fed2 (reserved)
(XEN)  fed84000 - fed85000 (reserved)
(XEN)  fee0 - fee01000 (reserved)
(XEN)  ffc0 - 0001 (reserved)
(XEN)  0001 - 00024e60 (usable)

and how it looks with a 3.18 boot:

[0.00] e820: BIOS-provided physical RAM map:
[0.00] Xen: [mem 0x-0x00099fff] usable
[0.00] Xen: [mem 0x0009a400-0x000f] reserved
[0.00] Xen: [mem 0x0010-0x30a47fff] usable
[0.00] Xen: [mem 0x30a48000-0x30a48fff] reserved
[0.00] Xen: [mem 0x30a49000-0xa27f3fff] usable
[0.00] Xen: [mem 0xa27f4000-0xa2ab3fff] reserved
[0.00] Xen: [mem 0xa2ab4000-0xa2fb3fff] ACPI NVS
[0.00] Xen: [mem 0xa2fb4000-0xa2feafff] ACPI data
[0.00] Xen: [mem 0xa2feb000-0xa2ff] usable
[0.00] Xen: [mem 0xa300-0xaf9f] reserved
[0.00] Xen: [mem 0xe000-0xefff] reserved
[0.00] Xen: [mem 0xfec0-0xfec00fff] reserved
[0.00] Xen: [mem 0xfed0-0xfed03fff] reserved
[0.00] Xen: [mem 0xfed1-0xfed19fff] reserved
[0.00] Xen: [mem 0xfed1c000-0xfed1] reserved
[0.00] Xen: [mem 0xfed84000-0xfed84fff] reserved
[0.00] Xen: [mem 0xfee0-0xfeef] reserved
[0.00] Xen: [mem 0xffc0-0x] reserved
[0.00] Xen: [mem 0x0001-0x0001bdc59fff] usable
[0.00] Xen: [mem 0x0001bdc5a000-0x00024e5f] unusable

Not sure that helps much. I probably have to try comparing later output. But
that will need a bit of time.

-Stefan

 
 So it seems one of the delaying changes has a very bad effect on that 
 Sharkbay.
 A bit odd since none of those sounds Intel/AMD geared. Could only be a 
 different
 usage of memory (my AMD box has considerably

Re: [Xen-devel] Crash in acpi_ps_peek_opcode when booting kernel 3.19 as Xen dom0

[Stefan Bader]

On 05.02.2015 20:36, Konrad Rzeszutek Wilk wrote:
 On Thu, Feb 05, 2015 at 03:33:02PM +0100, Stefan Bader wrote:
 While experimenting/testing various kernel versions I discovered that trying 
 to
 boot a Haswell based hosts will always crash when booting as Xen dom0
 (Xen-4.4.1). The same crash happens since v3.19-rc1 and still does happen 
 with
 v3.19-rc7. A bare metal boot is having no issues and also an Opteron based 
 host
 is having no issues (dom0 and bare metal).
 Could be a table that the other host does not have and since its only 
 happening
 in dom0 maybe some cpu capability that needs to be passed on?
 
 Usually it means that the ACPI AML code is trying to do something with
 the IOAPIC or something wihch is not accessible.
 
 But this on the other hand looks to be trying to execute some AML code
 that is unknown. Any chance you cna disassemble it and perhaps also
 run with acpi debug options on to figure out where it blows up?

The weird thing here is that bare-metal on the same machine does work. And
previous kernels did work as well. So I think we can assume the ACPI tables are
ok. It could even be a red-herring. Well, likely is as booting with acpi=off
does hang instead of crashing.

Since I got no clue, I did what we always do when we are dumbfound, I went ahead
and bisected 3.18..3.19-rc1. Unfortunately the very last kernel I build was
something in between good and bad. Good as it did not crash exactly but bad as
it did not come up in a usable state. So I would not be sure the claimed to be
offending commit is right. Could be one in the range of:

G  * xen: use common page allocation function in p2m.c
   * xen: Delay remapping memory of pv-domain
g  * xen: Delay m2p_override initialization
- * xen: Delay invalidating extra memory
B  * x86: Introduce function to get pmd entry pointer

(G) really good, (g) somewhat not bad, (B) bad, (-) claimed first broken.

So it seems one of the delaying changes has a very bad effect on that Sharkbay.
A bit odd since none of those sounds Intel/AMD geared. Could only be a different
usage of memory (my AMD box has considerably more memory and also no CPU with
GPU functionality as the Haswell).

Jürgen, maybe some description that might trigger an idea for you...?

-Stefan

---

git bisect start
# good: [b2776bf7149bddd1f4161f14f79520f17fc1d71d] Linux 3.18
git bisect good b2776bf7149bddd1f4161f14f79520f17fc1d71d
# bad: [97bf6af1f928216fd6c5a66e8a57bfa95a659672] Linux 3.19-rc1
git bisect bad 97bf6af1f928216fd6c5a66e8a57bfa95a659672
# good: [70e71ca0af244f48a5dcf56dc435243792e3a495] Merge
git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next
git bisect good 70e71ca0af244f48a5dcf56dc435243792e3a495
# good: [988adfdffdd43cfd841df734664727993076d7cb] Merge branch 'drm-next' of
git://people.freedesktop.org/~airlied/linux
git bisect good 988adfdffdd43cfd841df734664727993076d7cb
# good: [b024793188002b9eed452b5f6a04d45003ed5772] staging: rtl8723au:
phy_SsPwrSwitch92CU() was never called with bRegSSPwrLvl != 1
git bisect good b024793188002b9eed452b5f6a04d45003ed5772
# bad: [66dcff86ba40eebb5133cccf450878f2bba102ef] Merge tag 'for-linus' of
git://git.kernel.org/pub/scm/virt/kvm/kvm
git bisect bad 66dcff86ba40eebb5133cccf450878f2bba102ef
# bad: [dbe6f0c43efb9475d1d35fbef9f8be61b7b1] Merge tag 'for-linus-20141215'
of git://git.infradead.org/linux-mtd
git bisect bad dbe6f0c43efb9475d1d35fbef9f8be61b7b1
# bad: [94bbdb63d7ed5ca56b788e43d0ca4a8f9494c9e7] Merge tag 'fixes-for-linus' of
git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
git bisect bad 94bbdb63d7ed5ca56b788e43d0ca4a8f9494c9e7
# good: [2dbfca5a181973558277b28b1f4c36362291f5e0] Merge branch 'for-next' of
git://git.kernel.org/pub/scm/linux/kernel/git/cooloney/linux-leds
git bisect good 2dbfca5a181973558277b28b1f4c36362291f5e0
# bad: [0db2812a5240f2663b92d8d4b761122dd2e0c6c3] Merge
git://git.kernel.org/pub/scm/linux/kernel/git/cmetcalf/linux-tile
git bisect bad 0db2812a5240f2663b92d8d4b761122dd2e0c6c3
# bad: [f1d04b23b2015b4c3c0a8419677179b133afea08] Merge branch
'devel/for-linus-3.19' into stable/for-linus-3.19
git bisect bad f1d04b23b2015b4c3c0a8419677179b133afea08
# bad: [792230c3a66b3d17d6dcca712866d24f2283d4a6] x86: Introduce function to get
pmd entry pointer
git bisect bad 792230c3a66b3d17d6dcca712866d24f2283d4a6
# good: [7108c9ce8f6e59f775b0c8250dba52b569b6cba2] xen: use common page
allocation function in p2m.c
# NOTE: This was the last really good
git bisect good 7108c9ce8f6e59f775b0c8250dba52b569b6cba2
# good: [97f4533a60ce5d0cb35ff44a190111f81a987620] xen: Delay m2p_override
initialization
# NOTE: This revision did not crash the usual way but was not useable either
# NOTE: Use of wrong bits in page-tables.
git bisect good 97f4533a60ce5d0cb35ff44a190111f81a987620
 

 [2.108038] ACPI: Core revision 20141107
 [2.108153] ACPI Warning: Unsupported module-level executable opcode 0x91 
 at
 table offset 0x002B (20141107/psloop-225)
 [2.108264] ACPI Warning: Unsupported module

[Xen-devel] Crash in acpi_ps_peek_opcode when booting kernel 3.19 as Xen dom0

[Stefan Bader]

While experimenting/testing various kernel versions I discovered that trying to
boot a Haswell based hosts will always crash when booting as Xen dom0
(Xen-4.4.1). The same crash happens since v3.19-rc1 and still does happen with
v3.19-rc7. A bare metal boot is having no issues and also an Opteron based host
is having no issues (dom0 and bare metal).
Could be a table that the other host does not have and since its only happening
in dom0 maybe some cpu capability that needs to be passed on?

[2.108038] ACPI: Core revision 20141107
[2.108153] ACPI Warning: Unsupported module-level executable opcode 0x91 at
table offset 0x002B (20141107/psloop-225)
[2.108264] ACPI Warning: Unsupported module-level executable opcode 0x91 at
table offset 0x0033 (20141107/psloop-225)
[2.108375] ACPI Warning: Unsupported module-level executable opcode 0x95 at
table offset 0x0038 (20141107/psloop-225)
[2.108489] ACPI Warning: Unsupported module-level executable opcode 0x95 at
table offset 0x0041 (20141107/psloop-225)
[2.108613] ACPI Warning: Unsupported module-level executable opcode 0x7D at
table offset 0x040D (20141107/psloop-225)
[2.108751] BUG: unable to handle kernel paging request at c9ee74e0
[2.108835] IP: [814573db] acpi_ps_peek_opcode+0xd/0x1f
[2.108902] PGD 1f4be067 PUD 1f4bd067 PMD 1488f067 PTE 0
[2.109018] Oops:  [#1] SMP
[2.109094] Modules linked in:
[2.109153] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.19.0-031900rc7-generi
c #201502020035
[2.109220] Hardware name: Intel Corporation Shark Bay Client platform/Flathe
ad Creek Crb, BIOS HSWLPTU1.86C.0109.R03.1301282055 01/28/2013
[2.109295] task: 81c1c500 ti: 81c0 task.ti: 81c0

[2.109360] RIP: e030:[814573db]  [814573db] acpi_ps_peek
_opcode+0xd/0x1f
[2.109445] RSP: e02b:81c03ce8  EFLAGS: 00010283
[2.109490] RAX: 000c RBX: 880014887000 RCX: 81c03d50
[2.109539] RDX: c9ee74e0 RSI: 880014887030 RDI: 880014887030
[2.109587] RBP: 81c03ce8 R08: ea522600 R09: 81432c4f
[2.109635] R10: 880014899090 R11: 00ba R12: 880014887030
[2.109684] R13: 880014887000 R14: 81c03d50 R15: 000d
[2.109735] FS:  () GS:880018c0() knlGS:0
000
[2.109836] CS:  e033 DS:  ES:  CR0: 80050033
[2.109881] CR2: c9ee74e0 CR3: 01c15000 CR4: 00042660
[2.109930] Stack:
[2.109968]  81c03d38 81456537 81c03d28 81457
a40
[2.110104]  880014887000 880014887000 8800148990c0 c9ee7
4e0
[2.110238]  880014887030  81c03d78 81456
760
[2.110373] Call Trace:
[2.110413]  [81456537] acpi_ps_get_next_arg+0x114/0x1f9
[2.110461]  [81457a40] ? acpi_ps_pop_scope+0x54/0x72
[2.110508]  [81456760] acpi_ps_get_arguments+0x91/0x228
[2.110555]  [81456ad2] acpi_ps_parse_loop+0x1db/0x311
[2.110602]  [81457705] acpi_ps_parse_aml+0x96/0x275
[2.110649]  [8145322f] acpi_ns_one_complete_parse+0xf7/0x114
[2.110698]  [817d149a] ? _raw_spin_lock_irqsave+0x1a/0x60
[2.110746]  [8145326c] acpi_ns_parse_table+0x20/0x38
[2.110792]  [81452c20] acpi_ns_load_table+0x4c/0x90
[2.110840]  [817c50b5] acpi_tb_load_namespace+0xa6/0x14a
[2.110889]  [81d83269] acpi_load_tables+0xc/0x35
[2.110935]  [81454bf6] ? acpi_ns_get_node+0xb7/0xc9
[2.110982]  [81d825cf] acpi_early_init+0x73/0x105
[2.111029]  [81d3b083] start_kernel+0x348/0x3f0
[2.111075]  [81d3abcd] ? set_init_arg+0x56/0x56
[2.21]  [81d3a5f8] x86_64_start_reservations+0x2a/0x2c
[2.69]  [81d3e88c] xen_start_kernel+0x4f5/0x4f7
[2.111215] Code: 8a 87 60 05 87 81 5d c3 e8 73 cc 37 00 55 81 ff 00 01 00 00
 19 c0 48 89 e5 83 c0 02 5d c3 e8 5d cc 3



signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] Booting dom0, various issues since v3.16 kernels

[Stefan Bader]

Hey David,

after just being in that pain, I thought I might as well give a summary to
you/the list. Maybe helpful to not forget which piece should go to which 
stable...

So:
v3.16...v3.17.8: Somewhen in between those, the acpi irq seems to have broken.
 I have not yet verified that, but at least three changes in
 3.19-rc6 seem to look related:

 * x86/xen: Treat SCI interrupt as normal GSI interrupt,
 * ACPI: pci: Do not clear pci_dev-irq in acpi_pci_irq_disable(), and
 * x86/xen: Override ACPI IRQ management callback __acpi_unregister_gsi

v3.17.8...v3.18.4: Beside the acpi interrupt, no USB devices (beyond the
   hubs) get initialized. Not sure what fixed it, but it
   looks ok in v3.19-rc7.
   Beside that, there also was a regression in swiotlb
   that I think was passed on to some stable maintainers:

 * Revert swiotlb-xen: pass dev_addr to swiotlb_tbl_unmap_single

v3.18.4..v3.19-rc7: The issues above look to be fixed. Only some Haswell
based box now crashed on boot as dom0 while parsing
some ACPI tables (will send more detail seperately).
This happens only on that host and only when running
as dom0. Bare-metal is ok and an Opteron based different
host is also fine.

-Stefan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] xen-netfront: Fix handling packets on compound pages with skb_linearize

[Stefan Bader]

On 08.12.2014 12:11, David Vrabel wrote:
 On 08/12/14 10:19, Luis Henriques wrote:
 On Mon, Dec 01, 2014 at 09:55:24AM +0100, Stefan Bader wrote:
 On 11.08.2014 19:32, Zoltan Kiss wrote:
 There is a long known problem with the netfront/netback interface: if the 
 guest
 tries to send a packet which constitues more than MAX_SKB_FRAGS + 1 ring 
 slots,
 it gets dropped. The reason is that netback maps these slots to a frag in 
 the
 frags array, which is limited by size. Having so many slots can occur since
 compound pages were introduced, as the ring protocol slice them up into
 individual (non-compound) page aligned slots. The theoretical worst case
 scenario looks like this (note, skbs are limited to 64 Kb here):
 linear buffer: at most PAGE_SIZE - 17 * 2 bytes, overlapping page boundary,
 using 2 slots
 first 15 frags: 1 + PAGE_SIZE + 1 bytes long, first and last bytes are at 
 the
 end and the beginning of a page, therefore they use 3 * 15 = 45 slots
 last 2 frags: 1 + 1 bytes, overlapping page boundary, 2 * 2 = 4 slots
 Although I don't think this 51 slots skb can really happen, we need a 
 solution
 which can deal with every scenario. In real life there is only a few slots
 overdue, but usually it causes the TCP stream to be blocked, as the retry 
 will
 most likely have the same buffer layout.
 This patch solves this problem by linearizing the packet. This is not the
 fastest way, and it can fail much easier as it tries to allocate a big 
 linear
 area for the whole packet, but probably easier by an order of magnitude 
 than
 anything else. Probably this code path is not touched very frequently 
 anyway.

 Signed-off-by: Zoltan Kiss zoltan.k...@citrix.com
 Cc: Wei Liu wei.l...@citrix.com
 Cc: Ian Campbell ian.campb...@citrix.com
 Cc: Paul Durrant paul.durr...@citrix.com
 Cc: net...@vger.kernel.org
 Cc: linux-ker...@vger.kernel.org
 Cc: xen-de...@lists.xenproject.org

 This does not seem to be marked explicitly as stable. Has someone already 
 asked
 David Miller to put it on his stable queue? IMO it qualifies quite well and 
 the
 actual change should be simple to pick/backport.


 Thank you Stefan, I'm queuing this for the next 3.16 kernel release.
 
 Don't backport this yes.  It's broken.  It produces malformed requests
 and netback will report a fatal error and stop all traffic on the VIF.

Thanks David. Did this just come up? I don't remember seeing any report of the
regression. :/

-Stefan

 
 David
 
 ___
 Xen-devel mailing list
 Xen-devel@lists.xen.org
 http://lists.xen.org/xen-devel
 




signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] xen-netfront: Fix handling packets on compound pages with skb_linearize

[Stefan Bader]

On 08.12.2014 12:31, David Vrabel wrote:
 On 08/12/14 11:21, Stefan Bader wrote:
 On 08.12.2014 12:11, David Vrabel wrote:
 On 08/12/14 10:19, Luis Henriques wrote:
 On Mon, Dec 01, 2014 at 09:55:24AM +0100, Stefan Bader wrote:
 On 11.08.2014 19:32, Zoltan Kiss wrote:
 There is a long known problem with the netfront/netback interface: if 
 the guest
 tries to send a packet which constitues more than MAX_SKB_FRAGS + 1 ring 
 slots,
 it gets dropped. The reason is that netback maps these slots to a frag 
 in the
 frags array, which is limited by size. Having so many slots can occur 
 since
 compound pages were introduced, as the ring protocol slice them up into
 individual (non-compound) page aligned slots. The theoretical worst case
 scenario looks like this (note, skbs are limited to 64 Kb here):
 linear buffer: at most PAGE_SIZE - 17 * 2 bytes, overlapping page 
 boundary,
 using 2 slots
 first 15 frags: 1 + PAGE_SIZE + 1 bytes long, first and last bytes are 
 at the
 end and the beginning of a page, therefore they use 3 * 15 = 45 slots
 last 2 frags: 1 + 1 bytes, overlapping page boundary, 2 * 2 = 4 slots
 Although I don't think this 51 slots skb can really happen, we need a 
 solution
 which can deal with every scenario. In real life there is only a few 
 slots
 overdue, but usually it causes the TCP stream to be blocked, as the 
 retry will
 most likely have the same buffer layout.
 This patch solves this problem by linearizing the packet. This is not the
 fastest way, and it can fail much easier as it tries to allocate a big 
 linear
 area for the whole packet, but probably easier by an order of magnitude 
 than
 anything else. Probably this code path is not touched very frequently 
 anyway.

 Signed-off-by: Zoltan Kiss zoltan.k...@citrix.com
 Cc: Wei Liu wei.l...@citrix.com
 Cc: Ian Campbell ian.campb...@citrix.com
 Cc: Paul Durrant paul.durr...@citrix.com
 Cc: net...@vger.kernel.org
 Cc: linux-ker...@vger.kernel.org
 Cc: xen-de...@lists.xenproject.org

 This does not seem to be marked explicitly as stable. Has someone already 
 asked
 David Miller to put it on his stable queue? IMO it qualifies quite well 
 and the
 actual change should be simple to pick/backport.


 Thank you Stefan, I'm queuing this for the next 3.16 kernel release.

 Don't backport this yes.  It's broken.  It produces malformed requests
 and netback will report a fatal error and stop all traffic on the VIF.

 Thanks David. Did this just come up? I don't remember seeing any report of 
 the
 regression. :/
 
 There's been a couple of reports on xen-devel recently with 3.17
 frontends and I've just repro'd it (by always forcing a skb_linearize()
 in netfront).

Ah ok. Found at least one now (plus your fixup proposal). Too long ago to
remember for sure but I thought the change was tested by reporters. Could be
that I only had tried the approach I was working on back then (which was more
complicated trying to replace individual frags by aligned copies). :( And on our
devel we have not yet switched to past 3.16. Hope I can squeeze in some testing
there.

-Stefan
 
 David
 




signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] xen-netfront: Fix handling packets on compound pages with skb_linearize

[Stefan Bader]

On 11.08.2014 19:32, Zoltan Kiss wrote:
 There is a long known problem with the netfront/netback interface: if the 
 guest
 tries to send a packet which constitues more than MAX_SKB_FRAGS + 1 ring 
 slots,
 it gets dropped. The reason is that netback maps these slots to a frag in the
 frags array, which is limited by size. Having so many slots can occur since
 compound pages were introduced, as the ring protocol slice them up into
 individual (non-compound) page aligned slots. The theoretical worst case
 scenario looks like this (note, skbs are limited to 64 Kb here):
 linear buffer: at most PAGE_SIZE - 17 * 2 bytes, overlapping page boundary,
 using 2 slots
 first 15 frags: 1 + PAGE_SIZE + 1 bytes long, first and last bytes are at the
 end and the beginning of a page, therefore they use 3 * 15 = 45 slots
 last 2 frags: 1 + 1 bytes, overlapping page boundary, 2 * 2 = 4 slots
 Although I don't think this 51 slots skb can really happen, we need a solution
 which can deal with every scenario. In real life there is only a few slots
 overdue, but usually it causes the TCP stream to be blocked, as the retry will
 most likely have the same buffer layout.
 This patch solves this problem by linearizing the packet. This is not the
 fastest way, and it can fail much easier as it tries to allocate a big linear
 area for the whole packet, but probably easier by an order of magnitude than
 anything else. Probably this code path is not touched very frequently anyway.
 
 Signed-off-by: Zoltan Kiss zoltan.k...@citrix.com
 Cc: Wei Liu wei.l...@citrix.com
 Cc: Ian Campbell ian.campb...@citrix.com
 Cc: Paul Durrant paul.durr...@citrix.com
 Cc: net...@vger.kernel.org
 Cc: linux-ker...@vger.kernel.org
 Cc: xen-de...@lists.xenproject.org

This does not seem to be marked explicitly as stable. Has someone already asked
David Miller to put it on his stable queue? IMO it qualifies quite well and the
actual change should be simple to pick/backport.

-Stefan

 
 diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
 index 055222b..23359ae 100644
 --- a/drivers/net/xen-netfront.c
 +++ b/drivers/net/xen-netfront.c
 @@ -628,9 +628,10 @@ static int xennet_start_xmit(struct sk_buff *skb, struct 
 net_device *dev)
   slots = DIV_ROUND_UP(offset + len, PAGE_SIZE) +
   xennet_count_skb_frag_slots(skb);
   if (unlikely(slots  MAX_SKB_FRAGS + 1)) {
 - net_alert_ratelimited(
 - xennet: skb rides the rocket: %d slots\n, slots);
 - goto drop;
 + net_dbg_ratelimited(xennet: skb rides the rocket: %d slots, %d 
 bytes\n,
 + slots, skb-len);
 + if (skb_linearize(skb))
 + goto drop;
   }
  
   spin_lock_irqsave(queue-tx_lock, flags);
 
 ___
 Xen-devel mailing list
 Xen-devel@lists.xen.org
 http://lists.xen.org/xen-devel
 




signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel