Re: [Xen-devel] Error booting Xen

[Dario Faggioli]

Hey Harmandeep,

On Fri, 2016-01-29 at 03:13 -0700, Jan Beulich wrote:
> > > > On 28.01.16 at 20:17,  wrote:
> > Latest set looks good. No boot issues. No resets.
> > Full log at http://paste2.org/NxHNW4vn 
> 
> > Sorry I don't know much about source of last few
> > lines. I was just tracing in xen when these came.
> > I was unable to create them again. I will inform
> > you if I get these again.
> 
> The WRMSR ones are of no concern. What I'm curious about are the
> five instances of
> 
> (XEN) traps.c:3290: GPF (): 82d08019cb87 -> 82d080242e15
> 
> Could you check what these addresses correspond to in source?
> 
I guess you just happened to forgot to follow up on this question from
Jan?

> So far you've said you don't start any, but
> the logs clearly show otherwise. Are there perhaps any that
> get auto-started? In which case I'd be interested in you
> confirming that these come up fine.
> 
And this too?

Can you please do that? :-)

Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)



signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Dario Faggioli]

On Wed, 2016-02-03 at 13:10 +, Andrew Cooper wrote:
> On 03/02/16 12:55, Dario Faggioli wrote:
> > 
> > tbox login: (d1) mapping kernel into physical memory
> > (d1) about to get started...
> > (XEN) traps.c:2684:d1v0 Domain attempted WRMSR c081
> > from 0xe023e008 to 0x00230010.
> > (XEN) traps.c:2684:d1v0 Domain attempted WRMSR c082
> > from 0x82d0bfffe080 to 0x817ef990.
> > (XEN) traps.c:2684:d1v0 Domain attempted WRMSR c083
> > from 0x82d0bfffe0a0 to 0x817f1f60.
> > (XEN) traps.c:2684:d1v0 Domain attempted WRMSR 0174
> > from 0xe008 to 0x0010.
> > (XEN) traps.c:2684:d1v0 Domain attempted WRMSR 0175
> > from 0x83026d40ffc0 to 0x.
> > (XEN) traps.c:2684:d1v0 Domain attempted WRMSR 0176
> > from 0x82d08023eaf0 to 0x817f1d60.
> > (XEN) traps.c:2684:d1v0 Domain attempted WRMSR c084
> > from 0x00074700 to 0x00047700.
> > (XEN) traps.c:2684:d1v1 Domain attempted WRMSR c081
> > from 0xe023e008 to 0x00230010.
> > (XEN) traps.c:2684:d1v1 Domain attempted WRMSR c082
> > from 0x82d0b000 to 0x817ef990.
> > (XEN) traps.c:2684:d1v1 Domain attempted WRMSR c083
> > from 0x82d0b020 to 0x817f1f60.
> > (XEN) traps.c:2684:d1v1 Domain attempted WRMSR 0174
> > from 0xe008 to 0x0010.
> > (XEN) traps.c:2684:d1v1 Domain attempted WRMSR 0175
> > from 0x8300866fffc0 to 0x.
> > (XEN) traps.c:2684:d1v1 Domain attempted WRMSR 0176
> > from 0x82d08023eaf0 to 0x817f1d60.
> > (XEN) traps.c:2684:d1v1 Domain attempted WRMSR c084
> > from 0x00074700 to 0x00047700.
> > 
> > Because this is what you just said above, and that's... well...
> > just
> > impossible?!?! :-O
> 
> This is a Linux PV trying to set up the SYSCALL/SYSENTER MSRs when it
> shouldn't.  There is a fix upstream, but this specifically is
> harmless
> noise.
> 
That's ok... What we're arguing is that is happens as a consequence of
a domain being created, not just "by itself, after boot, without
starting any domain"  :-)

Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)



signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Dario Faggioli]

On Wed, 2016-02-03 at 17:05 +0530, Harmandeep Kaur wrote:
> On Wed, Feb 3, 2016 at 1:53 PM, Dario Faggioli
>  wrote:
> > 
> Maybe I failed to shutdown a guest which was showing up
> on next boot. But there are no auto-starting guests.
> Following link goes to latest serial log
> http://paste2.org/5PDz9bP1 
>
No, sorry, I probably am not getting. Are you saying that, just booting
Xen and *not* doing anything else --either from SSH, from the keyboard
of that box, from serial connection... anything at all-- and without
any guest configured to auto start itself, results in these lines to
appear on your serial console?

tbox login: (d1) mapping kernel into physical memory
(d1) about to get started...
(XEN) traps.c:2684:d1v0 Domain attempted WRMSR c081 from 
0xe023e008 to 0x00230010.
(XEN) traps.c:2684:d1v0 Domain attempted WRMSR c082 from 
0x82d0bfffe080 to 0x817ef990.
(XEN) traps.c:2684:d1v0 Domain attempted WRMSR c083 from 
0x82d0bfffe0a0 to 0x817f1f60.
(XEN) traps.c:2684:d1v0 Domain attempted WRMSR 0174 from 
0xe008 to 0x0010.
(XEN) traps.c:2684:d1v0 Domain attempted WRMSR 0175 from 
0x83026d40ffc0 to 0x.
(XEN) traps.c:2684:d1v0 Domain attempted WRMSR 0176 from 
0x82d08023eaf0 to 0x817f1d60.
(XEN) traps.c:2684:d1v0 Domain attempted WRMSR c084 from 
0x00074700 to 0x00047700.
(XEN) traps.c:2684:d1v1 Domain attempted WRMSR c081 from 
0xe023e008 to 0x00230010.
(XEN) traps.c:2684:d1v1 Domain attempted WRMSR c082 from 
0x82d0b000 to 0x817ef990.
(XEN) traps.c:2684:d1v1 Domain attempted WRMSR c083 from 
0x82d0b020 to 0x817f1f60.
(XEN) traps.c:2684:d1v1 Domain attempted WRMSR 0174 from 
0xe008 to 0x0010.
(XEN) traps.c:2684:d1v1 Domain attempted WRMSR 0175 from 
0x8300866fffc0 to 0x.
(XEN) traps.c:2684:d1v1 Domain attempted WRMSR 0176 from 
0x82d08023eaf0 to 0x817f1d60.
(XEN) traps.c:2684:d1v1 Domain attempted WRMSR c084 from 
0x00074700 to 0x00047700.

Because this is what you just said above, and that's... well... just
impossible?!?! :-O

What's the output then, if you login (either via serial, ssh or regular
keyboad+monitor) and run `xl list' and `xl vcpu-list' as the very first
thing?  What's inside `ls -l /var/log/xen/' (or `ls -l
/var/local/log/xen')?

> and then I created a guest with
> config (http://paste2.org/azvj25Hg) and
> http://paste2.org/cgKna0j1 log was presented at terminal.
> 
Well, it looks like the guest boot did not indeed go well, but that is
probably because of other thing... can you try uncommenting the line
'extra = "root=/dev/xvda1"' from the config?

Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)



signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Harmandeep Kaur]

On Wed, Feb 3, 2016 at 6:25 PM, Dario Faggioli
 wrote:
> On Wed, 2016-02-03 at 17:05 +0530, Harmandeep Kaur wrote:
>> On Wed, Feb 3, 2016 at 1:53 PM, Dario Faggioli
>>  wrote:
>> >
>> Maybe I failed to shutdown a guest which was showing up
>> on next boot. But there are no auto-starting guests.
>> Following link goes to latest serial log
>> http://paste2.org/5PDz9bP1
>>
> No, sorry, I probably am not getting. Are you saying that, just booting
> Xen and *not* doing anything else --either from SSH, from the keyboard
> of that box, from serial connection... anything at all-- and without
> any guest configured to auto start itself, results in these lines to
> appear on your serial console?
>
> tbox login: (d1) mapping kernel into physical memory
> (d1) about to get started...
> (XEN) traps.c:2684:d1v0 Domain attempted WRMSR c081 from 
> 0xe023e008 to 0x00230010.
> (XEN) traps.c:2684:d1v0 Domain attempted WRMSR c082 from 
> 0x82d0bfffe080 to 0x817ef990.
> (XEN) traps.c:2684:d1v0 Domain attempted WRMSR c083 from 
> 0x82d0bfffe0a0 to 0x817f1f60.
> (XEN) traps.c:2684:d1v0 Domain attempted WRMSR 0174 from 
> 0xe008 to 0x0010.
> (XEN) traps.c:2684:d1v0 Domain attempted WRMSR 0175 from 
> 0x83026d40ffc0 to 0x.
> (XEN) traps.c:2684:d1v0 Domain attempted WRMSR 0176 from 
> 0x82d08023eaf0 to 0x817f1d60.
> (XEN) traps.c:2684:d1v0 Domain attempted WRMSR c084 from 
> 0x00074700 to 0x00047700.
> (XEN) traps.c:2684:d1v1 Domain attempted WRMSR c081 from 
> 0xe023e008 to 0x00230010.
> (XEN) traps.c:2684:d1v1 Domain attempted WRMSR c082 from 
> 0x82d0b000 to 0x817ef990.
> (XEN) traps.c:2684:d1v1 Domain attempted WRMSR c083 from 
> 0x82d0b020 to 0x817f1f60.
> (XEN) traps.c:2684:d1v1 Domain attempted WRMSR 0174 from 
> 0xe008 to 0x0010.
> (XEN) traps.c:2684:d1v1 Domain attempted WRMSR 0175 from 
> 0x8300866fffc0 to 0x.
> (XEN) traps.c:2684:d1v1 Domain attempted WRMSR 0176 from 
> 0x82d08023eaf0 to 0x817f1d60.
> (XEN) traps.c:2684:d1v1 Domain attempted WRMSR c084 from 
> 0x00074700 to 0x00047700.
>
> Because this is what you just said above, and that's... well... just
> impossible?!?! :-O
>
> What's the output then, if you login (either via serial, ssh or regular
> keyboad+monitor) and run `xl list'

$ sudo xl list
NameID   Mem VCPUsStateTime(s)
Domain-0 0  4096 4 r-  37.6

> and `xl vcpu-list' as the very first
> thing?

$ sudo xl vcpu-list
NameID  VCPU   CPU State   Time(s)
Affinity (Hard / Soft)
Domain-0 0 03   r--  13.3  all / all
Domain-0 0 10   -b-   7.0  all / all
Domain-0 0 21   -b-   5.4  all / all
Domain-0 0 32   r--  13.5  all / all


> What's inside `ls -l /var/log/xen/' (or `ls -l
> /var/local/log/xen')?

$ ls -l /var/log/xen/
total 48
-rw-r--r-- 1 root adm  28 Jan 23 00:14 xen-hotplug.log
-rw-r--r-- 1 root adm  91 Feb  3 17:14 xl-ubuntu.pvlinux.log
-rw-r--r-- 1 root adm  91 Jan 29 00:54 xl-ubuntu.pvlinux.log.1
-rw-r--r-- 1 root adm 144 Jan 26 23:50 xl-ubuntu.pvlinux.log.10
-rw-r--r-- 1 root adm  91 Jan 29 00:41 xl-ubuntu.pvlinux.log.2
-rw-r--r-- 1 root adm  91 Jan 29 00:32 xl-ubuntu.pvlinux.log.3
-rw-r--r-- 1 root adm  91 Jan 29 00:24 xl-ubuntu.pvlinux.log.4
-rw-r--r-- 1 root adm  91 Jan 28 22:22 xl-ubuntu.pvlinux.log.5
-rw-r--r-- 1 root adm  62 Jan 27 19:09 xl-ubuntu.pvlinux.log.6
-rw-r--r-- 1 root adm  91 Jan 27 19:07 xl-ubuntu.pvlinux.log.7
-rw-r--r-- 1 root adm 222 Jan 27 18:41 xl-ubuntu.pvlinux.log.8
-rw-r--r-- 1 root adm  62 Jan 27 15:03 xl-ubuntu.pvlinux.log.9

>
>> and then I created a guest with
>> config (http://paste2.org/azvj25Hg) and
>> http://paste2.org/cgKna0j1 log was presented at terminal.
>>
> Well, it looks like the guest boot did not indeed go well, but that is
> probably because of other thing... can you try uncommenting the line
> 'extra = "root=/dev/xvda1"' from the config?

Trying this suggestion.

>
> Regards,
> Dario
> --
> <> (Raistlin Majere)
> -
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)
>

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Andrew Cooper]

On 03/02/16 12:55, Dario Faggioli wrote:
> On Wed, 2016-02-03 at 17:05 +0530, Harmandeep Kaur wrote:
>> On Wed, Feb 3, 2016 at 1:53 PM, Dario Faggioli
>>  wrote:
>>>  
>> Maybe I failed to shutdown a guest which was showing up
>> on next boot. But there are no auto-starting guests.
>> Following link goes to latest serial log
>> http://paste2.org/5PDz9bP1 
>>
> No, sorry, I probably am not getting. Are you saying that, just booting
> Xen and *not* doing anything else --either from SSH, from the keyboard
> of that box, from serial connection... anything at all-- and without
> any guest configured to auto start itself, results in these lines to
> appear on your serial console?
>
> tbox login: (d1) mapping kernel into physical memory
> (d1) about to get started...
> (XEN) traps.c:2684:d1v0 Domain attempted WRMSR c081 from 
> 0xe023e008 to 0x00230010.
> (XEN) traps.c:2684:d1v0 Domain attempted WRMSR c082 from 
> 0x82d0bfffe080 to 0x817ef990.
> (XEN) traps.c:2684:d1v0 Domain attempted WRMSR c083 from 
> 0x82d0bfffe0a0 to 0x817f1f60.
> (XEN) traps.c:2684:d1v0 Domain attempted WRMSR 0174 from 
> 0xe008 to 0x0010.
> (XEN) traps.c:2684:d1v0 Domain attempted WRMSR 0175 from 
> 0x83026d40ffc0 to 0x.
> (XEN) traps.c:2684:d1v0 Domain attempted WRMSR 0176 from 
> 0x82d08023eaf0 to 0x817f1d60.
> (XEN) traps.c:2684:d1v0 Domain attempted WRMSR c084 from 
> 0x00074700 to 0x00047700.
> (XEN) traps.c:2684:d1v1 Domain attempted WRMSR c081 from 
> 0xe023e008 to 0x00230010.
> (XEN) traps.c:2684:d1v1 Domain attempted WRMSR c082 from 
> 0x82d0b000 to 0x817ef990.
> (XEN) traps.c:2684:d1v1 Domain attempted WRMSR c083 from 
> 0x82d0b020 to 0x817f1f60.
> (XEN) traps.c:2684:d1v1 Domain attempted WRMSR 0174 from 
> 0xe008 to 0x0010.
> (XEN) traps.c:2684:d1v1 Domain attempted WRMSR 0175 from 
> 0x8300866fffc0 to 0x.
> (XEN) traps.c:2684:d1v1 Domain attempted WRMSR 0176 from 
> 0x82d08023eaf0 to 0x817f1d60.
> (XEN) traps.c:2684:d1v1 Domain attempted WRMSR c084 from 
> 0x00074700 to 0x00047700.
>
> Because this is what you just said above, and that's... well... just
> impossible?!?! :-O

This is a Linux PV trying to set up the SYSCALL/SYSENTER MSRs when it
shouldn't.  There is a fix upstream, but this specifically is harmless
noise.

~Andrew

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Jan Beulich]

>>> On 03.02.16 at 14:40,  wrote:
> $ ls -l /var/log/xen/
> total 48
> -rw-r--r-- 1 root adm  28 Jan 23 00:14 xen-hotplug.log
> -rw-r--r-- 1 root adm  91 Feb  3 17:14 xl-ubuntu.pvlinux.log

This date looks suspiciously new.

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Harmandeep Kaur]

On Wed, Feb 3, 2016 at 1:53 PM, Dario Faggioli
 wrote:
> Hey Harmandeep,
>
> On Fri, 2016-01-29 at 03:13 -0700, Jan Beulich wrote:
>> > > > On 28.01.16 at 20:17,  wrote:
>> > Latest set looks good. No boot issues. No resets.
>> > Full log at http://paste2.org/NxHNW4vn
>>
>> > Sorry I don't know much about source of last few
>> > lines. I was just tracing in xen when these came.
>> > I was unable to create them again. I will inform
>> > you if I get these again.
>>
>> The WRMSR ones are of no concern. What I'm curious about are the
>> five instances of
>>
>> (XEN) traps.c:3290: GPF (): 82d08019cb87 -> 82d080242e15
>>
>> Could you check what these addresses correspond to in source?
>>
> I guess you just happened to forgot to follow up on this question from
> Jan?

Sorry for responding so late.

$ addr2line -e xen-syms 82d08019cb87
xen/xen/arch/x86/traps.c:2820

>> So far you've said you don't start any, but
>> the logs clearly show otherwise. Are there perhaps any that
>> get auto-started? In which case I'd be interested in you
>> confirming that these come up fine.
>>

Maybe I failed to shutdown a guest which was showing up
on next boot. But there are no auto-starting guests.
Following link goes to latest serial log
http://paste2.org/5PDz9bP1 and then I created a guest with
config (http://paste2.org/azvj25Hg) and
http://paste2.org/cgKna0j1 log was presented at terminal.

I hope this helps.

Thanks and regards,
Harmandeep Kaur

> And this too?
>
> Can you please do that? :-)
>
> Regards,
> Dario
> --
> <> (Raistlin Majere)
> -
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)
>

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Jan Beulich]

>>> On 03.02.16 at 12:35,  wrote:
> On Wed, Feb 3, 2016 at 1:53 PM, Dario Faggioli
>  wrote:
>> Hey Harmandeep,
>>
>> On Fri, 2016-01-29 at 03:13 -0700, Jan Beulich wrote:
>>> > > > On 28.01.16 at 20:17,  wrote:
>>> > Latest set looks good. No boot issues. No resets.
>>> > Full log at http://paste2.org/NxHNW4vn 
>>>
>>> > Sorry I don't know much about source of last few
>>> > lines. I was just tracing in xen when these came.
>>> > I was unable to create them again. I will inform
>>> > you if I get these again.
>>>
>>> The WRMSR ones are of no concern. What I'm curious about are the
>>> five instances of
>>>
>>> (XEN) traps.c:3290: GPF (): 82d08019cb87 -> 82d080242e15
>>>
>>> Could you check what these addresses correspond to in source?
>>>
>> I guess you just happened to forgot to follow up on this question from
>> Jan?
> 
> Sorry for responding so late.

Thanks for getting back.

> $ addr2line -e xen-syms 82d08019cb87
> xen/xen/arch/x86/traps.c:2820

That's the RDMSR one, so (luckily) not of interest here.

>>> So far you've said you don't start any, but
>>> the logs clearly show otherwise. Are there perhaps any that
>>> get auto-started? In which case I'd be interested in you
>>> confirming that these come up fine.
>>>
> 
> Maybe I failed to shutdown a guest which was showing up
> on next boot. But there are no auto-starting guests.
> Following link goes to latest serial log
> http://paste2.org/5PDz9bP1 and then I created a guest with

But this _again_ shows the presence of a (running) Dom1, ...

> config (http://paste2.org/azvj25Hg) and
> http://paste2.org/cgKna0j1 log was presented at terminal.

... and you leave us guessing whether the first of these logs was
indeed captured before you started the guest (as one may imply
from your wording) or after (as one may imply from your claim of
there not being any auto-started guests). Please try to be more
precise in your statements.

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Konrad Rzeszutek Wilk]

On Wed, Feb 03, 2016 at 02:17:26PM +0100, Dario Faggioli wrote:
> On Wed, 2016-02-03 at 13:10 +, Andrew Cooper wrote:
> > On 03/02/16 12:55, Dario Faggioli wrote:
> > > 
> > > tbox login: (d1) mapping kernel into physical memory
> > > (d1) about to get started...
> > > (XEN) traps.c:2684:d1v0 Domain attempted WRMSR c081
> > > from 0xe023e008 to 0x00230010.
> > > (XEN) traps.c:2684:d1v0 Domain attempted WRMSR c082
> > > from 0x82d0bfffe080 to 0x817ef990.
> > > (XEN) traps.c:2684:d1v0 Domain attempted WRMSR c083
> > > from 0x82d0bfffe0a0 to 0x817f1f60.
> > > (XEN) traps.c:2684:d1v0 Domain attempted WRMSR 0174
> > > from 0xe008 to 0x0010.
> > > (XEN) traps.c:2684:d1v0 Domain attempted WRMSR 0175
> > > from 0x83026d40ffc0 to 0x.
> > > (XEN) traps.c:2684:d1v0 Domain attempted WRMSR 0176
> > > from 0x82d08023eaf0 to 0x817f1d60.
> > > (XEN) traps.c:2684:d1v0 Domain attempted WRMSR c084
> > > from 0x00074700 to 0x00047700.
> > > (XEN) traps.c:2684:d1v1 Domain attempted WRMSR c081
> > > from 0xe023e008 to 0x00230010.
> > > (XEN) traps.c:2684:d1v1 Domain attempted WRMSR c082
> > > from 0x82d0b000 to 0x817ef990.
> > > (XEN) traps.c:2684:d1v1 Domain attempted WRMSR c083
> > > from 0x82d0b020 to 0x817f1f60.
> > > (XEN) traps.c:2684:d1v1 Domain attempted WRMSR 0174
> > > from 0xe008 to 0x0010.
> > > (XEN) traps.c:2684:d1v1 Domain attempted WRMSR 0175
> > > from 0x8300866fffc0 to 0x.
> > > (XEN) traps.c:2684:d1v1 Domain attempted WRMSR 0176
> > > from 0x82d08023eaf0 to 0x817f1d60.
> > > (XEN) traps.c:2684:d1v1 Domain attempted WRMSR c084
> > > from 0x00074700 to 0x00047700.
> > > 
> > > Because this is what you just said above, and that's... well...
> > > just
> > > impossible?!?! :-O
> > 
> > This is a Linux PV trying to set up the SYSCALL/SYSENTER MSRs when it
> > shouldn't.  There is a fix upstream, but this specifically is
> > harmless
> > noise.
> > 
> That's ok... What we're arguing is that is happens as a consequence of
> a domain being created, not just "by itself, after boot, without
> starting any domain"  :-)

It has booted. It says 'about to get started...' which is in the early
code of the Linux pvops.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Shuai Ruan]

On Tue, Feb 02, 2016 at 10:39:09AM +, Andrew Cooper wrote:
> On 02/02/16 04:43, Shuai Ruan wrote:
> >
> >>> I tried booting staging branch but results were identical. Following
> >>> line repeats endlessly.
> >>> (XEN) traps.c:3290: GPF (): 82d0801c1cce -> 82d080252e5c
> >>>
> >>> $ 'addr2line -e xen-syms 82d0801c1cce' returns
> >>> 'xen/xen/arch/x86/xstate.c:387' which again points to
> >>> xsave. Also, adding 'xsave=0' makes it boot just fine.
> >> Ah, I think I see the issue: We're zeroing the entire save area in
> >> the fixup code, which will make XRSTORS fault unconditionally.
> >> Shuai, would you please look into possible ways of fixing this?
> > Ok. I will look into this problem.
> 
> Fixes for this have been committed already.  changesets 104a409^..d570d82
> 
> It would be useful if you could review them to see if there are any
> issues Jan and myself missed.
Ok. I will review them and test them. 

Thanks
> 
> ~Andrew
> 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Andrew Cooper]

On 02/02/16 04:43, Shuai Ruan wrote:
>
>>> I tried booting staging branch but results were identical. Following
>>> line repeats endlessly.
>>> (XEN) traps.c:3290: GPF (): 82d0801c1cce -> 82d080252e5c
>>>
>>> $ 'addr2line -e xen-syms 82d0801c1cce' returns
>>> 'xen/xen/arch/x86/xstate.c:387' which again points to
>>> xsave. Also, adding 'xsave=0' makes it boot just fine.
>> Ah, I think I see the issue: We're zeroing the entire save area in
>> the fixup code, which will make XRSTORS fault unconditionally.
>> Shuai, would you please look into possible ways of fixing this?
> Ok. I will look into this problem.

Fixes for this have been committed already.  changesets 104a409^..d570d82

It would be useful if you could review them to see if there are any
issues Jan and myself missed.

~Andrew

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Shuai Ruan]

On Wed, Jan 20, 2016 at 03:06:09AM -0700, Jan Beulich wrote:
> >>> On 19.01.16 at 20:55,  wrote:
Sorry for late response. I am away from the mail list for a couple
weeks.

> > I tried booting staging branch but results were identical. Following
> > line repeats endlessly.
> > (XEN) traps.c:3290: GPF (): 82d0801c1cce -> 82d080252e5c
> > 
> > $ 'addr2line -e xen-syms 82d0801c1cce' returns
> > 'xen/xen/arch/x86/xstate.c:387' which again points to
> > xsave. Also, adding 'xsave=0' makes it boot just fine.
> 
> Ah, I think I see the issue: We're zeroing the entire save area in
> the fixup code, which will make XRSTORS fault unconditionally.
> Shuai, would you please look into possible ways of fixing this?
Ok. I will look into this problem.
> Just setting the compressed flag may not be enough, and falling
> back to plain XRSTOR doesn't seem to be an option either - I'm in
> particular worried that the current model of zeroing everything
> is bogus with the growing number of different components which
> get loaded here.
> 
> But of course another question then is why the XRSTORS faults
> in the first place. I guess we'll need a debugging patch to dump
> the full state to understand that.
> 
> Jan
> 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Jan Beulich]

>>> On 28.01.16 at 20:17,  wrote:
> Latest set looks good. No boot issues. No resets.
> Full log at http://paste2.org/NxHNW4vn 

Thanks!

> Sorry I don't know much about source of last few
> lines. I was just tracing in xen when these came.
> I was unable to create them again. I will inform
> you if I get these again.

The WRMSR ones are of no concern. What I'm curious about are the
five instances of

(XEN) traps.c:3290: GPF (): 82d08019cb87 -> 82d080242e15

Could you check what these addresses correspond to in source?

Also did you meanwhile clarify for yourself what domain(s) are
being started here? So far you've said you don't start any, but
the logs clearly show otherwise. Are there perhaps any that
get auto-started? In which case I'd be interested in you
confirming that these come up fine.

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Harmandeep Kaur]

Latest set looks good. No boot issues. No resets.
Full log at http://paste2.org/NxHNW4vn

Sorry I don't know much about source of last few
lines. I was just tracing in xen when these came.
I was unable to create them again. I will inform
you if I get these again.
Regards,
Harmandeep

On Wed, Jan 27, 2016 at 7:58 PM, Jan Beulich  wrote:
 On 27.01.16 at 14:28,  wrote:
>> I tried to apply your patches but it seems
>> to have some merge conflicts with latest
>> staging branch.
>>
>> ~/xen$ git apply ~/Downloads/x86-xsaves-init.patch
>> error: patch failed: xen/arch/x86/hvm/hvm.c:2094
>> error: xen/arch/x86/hvm/hvm.c: patch does not apply
>
> Oh, indeed. Here's a better set.
>
> Jan
>

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Jan Beulich]

>>> On 26.01.16 at 19:02,  wrote:
> Last time, I did absolutely nothing. System was idle
> and it crashed just after the login. Now, I booted the
> system again and this time, there is no reset. But,
> performance of the system is very slow. Browser
> (Mozilla Firefox) freezes a lot. Also, before applying
> patches, when I used to disabe xsave it resulted in
> same kind of performance issues. And the following
> is still present in the log.
> 
> (XEN) traps.c:3290: GPF (): 82d0801c1cea -> 82d080252e5c
> (XEN) d1v1 fault#1: mxcsr=1f80
> (XEN) d1v1 xs=0003 xc=8000
> (XEN) d1v1 r0= r1=
> (XEN) d1v1 r2= r3=
> (XEN) d1v1 r4= r5=
> (XEN) traps.c:3290: GPF (): 82d0801c1cea -> 82d080252e5c
> (XEN) d1v1 fault#2: mxcsr=1f80
> (XEN) d1v1 xs= xc=
> (XEN) d1v1 r0= r1=
> (XEN) d1v1 r2= r3=
> (XEN) d1v1 r4= r5=
> 
> Full log here: http://paste2.org/C8WpyKOg 

This is not _still_ present, but that's another, different instance,
now during guest creation. Please try to be precise in your
wording. Namely, in this case, you again fail to mention what
kind of guest you're now creating that causes this to re-appear.

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Jan Beulich]

>>> On 27.01.16 at 14:28,  wrote:
> I tried to apply your patches but it seems
> to have some merge conflicts with latest
> staging branch.
> 
> ~/xen$ git apply ~/Downloads/x86-xsaves-init.patch
> error: patch failed: xen/arch/x86/hvm/hvm.c:2094
> error: xen/arch/x86/hvm/hvm.c: patch does not apply

Oh, indeed. Here's a better set.

Jan

x86/xstate: fix fault behavior on XRSTORS

XRSTORS unconditionally faults when xcomp_bv has bit 63 clear. Instead
of just fixing this issue, overhaul the fault recovery code, which -
one of the many mistakes made when xstate support got introduced - was
blindly mirroring that accompanying FXRSTOR, neglecting the fact that
XRSTOR{,S} aren't all-or-nothing instructions. The new code, first of
all, does all the recovery actions in C, simplifying the inline
assembly used. And it does its work in a multi-stage fashion: Upon
first seeing a fault, state fixups get applied strictly based on what
architecturally may cause #GP. When seeing another fault despite the
fixups done, state gets fully reset. A third fault would then lead to
crashing the domain (instead of hanging the hypervisor in an infinite
loop of recurring faults).

Reported-by: Harmandeep Kaur 
Signed-off-by: Jan Beulich 

--- unstable.orig/xen/arch/x86/xstate.c 2016-01-25 09:35:12.0 +0100
+++ unstable/xen/arch/x86/xstate.c  2016-01-27 10:23:06.0 +0100
@@ -29,6 +29,8 @@ unsigned int *__read_mostly xstate_sizes
 static unsigned int __read_mostly xstate_features;
 static unsigned int __read_mostly xstate_comp_offsets[sizeof(xfeature_mask)*8];
 
+static uint32_t __read_mostly mxcsr_mask = MXCSR_DEFAULT;
+
 /* Cached xcr0 for fast read */
 static DEFINE_PER_CPU(uint64_t, xcr0);
 
@@ -342,6 +344,7 @@ void xrstor(struct vcpu *v, uint64_t mas
 uint32_t hmask = mask >> 32;
 uint32_t lmask = mask;
 struct xsave_struct *ptr = v->arch.xsave_area;
+unsigned int faults, prev_faults;
 
 /*
  * AMD CPUs don't save/restore FDP/FIP/FOP unless an exception
@@ -361,35 +364,85 @@ void xrstor(struct vcpu *v, uint64_t mas
 /*
  * XRSTOR can fault if passed a corrupted data block. We handle this
  * possibility, which may occur if the block was passed to us by control
- * tools or through VCPUOP_initialise, by silently clearing the block.
+ * tools or through VCPUOP_initialise, by silently adjusting state.
  */
-switch ( __builtin_expect(ptr->fpu_sse.x[FPU_WORD_SIZE_OFFSET], 8) )
+for ( prev_faults = faults = 0; ; prev_faults = faults )
 {
+switch ( __builtin_expect(ptr->fpu_sse.x[FPU_WORD_SIZE_OFFSET], 8) )
+{
 #define XRSTOR(pfx) \
 alternative_io("1: .byte " pfx "0x0f,0xae,0x2f\n" \
+   "3:\n" \
"   .section .fixup,\"ax\"\n" \
-   "2: mov %[size],%%ecx\n" \
-   "   xor %[lmask_out],%[lmask_out]\n" \
-   "   rep stosb\n" \
-   "   lea %[mem],%[ptr]\n" \
-   "   mov %[lmask_in],%[lmask_out]\n" \
-   "   jmp 1b\n" \
+   "2: inc%z[faults] %[faults]\n" \
+   "   jmp 3b\n" \
"   .previous\n" \
_ASM_EXTABLE(1b, 2b), \
".byte " pfx "0x0f,0xc7,0x1f\n", \
X86_FEATURE_XSAVES, \
-   ASM_OUTPUT2([ptr] "+" (ptr), [lmask_out] "+" 
(lmask)), \
-   [mem] "m" (*ptr), [lmask_in] "g" (lmask), \
-   [hmask] "d" (hmask), [size] "m" (xsave_cntxt_size) \
-   : "ecx")
-
-default:
-XRSTOR("0x48,");
-break;
-case 4: case 2:
-XRSTOR("");
-break;
+   ASM_OUTPUT2([mem] "+m" (*ptr), [faults] "+g" (faults)), 
\
+   [lmask] "a" (lmask), [hmask] "d" (hmask), \
+   [ptr] "D" (ptr))
+
+default:
+XRSTOR("0x48,");
+break;
+case 4: case 2:
+XRSTOR("");
+break;
 #undef XRSTOR
+}
+if ( likely(faults == prev_faults) )
+break;
+#ifndef NDEBUG
+gprintk(XENLOG_WARNING, "fault#%u: mxcsr=%08x\n",
+faults, ptr->fpu_sse.mxcsr);
+gprintk(XENLOG_WARNING, "xs=%016lx xc=%016lx\n",
+ptr->xsave_hdr.xstate_bv, ptr->xsave_hdr.xcomp_bv);
+gprintk(XENLOG_WARNING, "r0=%016lx r1=%016lx\n",
+ptr->xsave_hdr.reserved[0], ptr->xsave_hdr.reserved[1]);
+gprintk(XENLOG_WARNING, "r2=%016lx r3=%016lx\n",
+ptr->xsave_hdr.reserved[2], ptr->xsave_hdr.reserved[3]);
+gprintk(XENLOG_WARNING, "r4=%016lx r5=%016lx\n",
+ptr->xsave_hdr.reserved[4], ptr->xsave_hdr.reserved[5]);
+#endif
+switch ( faults )
+{
+case 1:
+/* Stage 

Re: [Xen-devel] Error booting Xen

[Harmandeep Kaur]

I tried to apply your patches but it seems
to have some merge conflicts with latest
staging branch.

~/xen$ git apply ~/Downloads/x86-xsaves-init.patch
error: patch failed: xen/arch/x86/hvm/hvm.c:2094
error: xen/arch/x86/hvm/hvm.c: patch does not apply

Do you mind having a look ?

Regards,
Harmandeep

On Wed, Jan 27, 2016 at 6:42 PM, Jan Beulich  wrote:
 On 26.01.16 at 19:02,  wrote:
>> Last time, I did absolutely nothing. System was idle
>> and it crashed just after the login. Now, I booted the
>> system again and this time, there is no reset. But,
>> performance of the system is very slow. Browser
>> (Mozilla Firefox) freezes a lot. Also, before applying
>> patches, when I used to disabe xsave it resulted in
>> same kind of performance issues. And the following
>> is still present in the log.
>>
>> (XEN) traps.c:3290: GPF (): 82d0801c1cea -> 82d080252e5c
>> (XEN) d1v1 fault#1: mxcsr=1f80
>> (XEN) d1v1 xs=0003 xc=8000
>> (XEN) d1v1 r0= r1=
>> (XEN) d1v1 r2= r3=
>> (XEN) d1v1 r4= r5=
>> (XEN) traps.c:3290: GPF (): 82d0801c1cea -> 82d080252e5c
>> (XEN) d1v1 fault#2: mxcsr=1f80
>> (XEN) d1v1 xs= xc=
>> (XEN) d1v1 r0= r1=
>> (XEN) d1v1 r2= r3=
>> (XEN) d1v1 r4= r5=
>>
>> Full log here: http://paste2.org/C8WpyKOg
>
> This together with ...
>
>> On Tue, Jan 26, 2016 at 10:53 PM, Jan Beulich  wrote:
>> On 26.01.16 at 18:01,  wrote:
 I tried 3rd patch together with earlier two. I'm
 afraid the problem is not solved completely.
 Full log goes here, http://paste2.org/KEAetMHb
>
> ... this, and both being apparently the same build makes me suspect
> uninitialized data to get passed in from the tool stack. But that's a
> secondary issue for now. For the immediate problem here are four
> patches replacing the three earlier ones (I think only one of them is
> unchanged, so be sure to remove the old ones first).
>
> Their intended ordering is:
> x86-xsaves-init.patch
> x86-xstate-align.patch
> x86-xrstors-fault.patch
> x86-xstate-validate.patch
>
> Jan
>

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Jan Beulich]

>>> On 26.01.16 at 19:02,  wrote:
> Last time, I did absolutely nothing. System was idle
> and it crashed just after the login. Now, I booted the
> system again and this time, there is no reset. But,
> performance of the system is very slow. Browser
> (Mozilla Firefox) freezes a lot. Also, before applying
> patches, when I used to disabe xsave it resulted in
> same kind of performance issues. And the following
> is still present in the log.
> 
> (XEN) traps.c:3290: GPF (): 82d0801c1cea -> 82d080252e5c
> (XEN) d1v1 fault#1: mxcsr=1f80
> (XEN) d1v1 xs=0003 xc=8000
> (XEN) d1v1 r0= r1=
> (XEN) d1v1 r2= r3=
> (XEN) d1v1 r4= r5=
> (XEN) traps.c:3290: GPF (): 82d0801c1cea -> 82d080252e5c
> (XEN) d1v1 fault#2: mxcsr=1f80
> (XEN) d1v1 xs= xc=
> (XEN) d1v1 r0= r1=
> (XEN) d1v1 r2= r3=
> (XEN) d1v1 r4= r5=
> 
> Full log here: http://paste2.org/C8WpyKOg 

This together with ...

> On Tue, Jan 26, 2016 at 10:53 PM, Jan Beulich  wrote:
> On 26.01.16 at 18:01,  wrote:
>>> I tried 3rd patch together with earlier two. I'm
>>> afraid the problem is not solved completely.
>>> Full log goes here, http://paste2.org/KEAetMHb 

... this, and both being apparently the same build makes me suspect
uninitialized data to get passed in from the tool stack. But that's a
secondary issue for now. For the immediate problem here are four
patches replacing the three earlier ones (I think only one of them is
unchanged, so be sure to remove the old ones first).

Their intended ordering is:
x86-xsaves-init.patch
x86-xstate-align.patch
x86-xrstors-fault.patch
x86-xstate-validate.patch

Jan

x86/xstate: fix fault behavior on XRSTORS

XRSTORS unconditionally faults when xcomp_bv has bit 63 clear. Instead
of just fixing this issue, overhaul the fault recovery code, which -
one of the many mistakes made when xstate support got introduced - was
blindly mirroring that accompanying FXRSTOR, neglecting the fact that
XRSTOR{,S} aren't all-or-nothing instructions. The new code, first of
all, does all the recovery actions in C, simplifying the inline
assembly used. And it does its work in a multi-stage fashion: Upon
first seeing a fault, state fixups get applied strictly based on what
architecturally may cause #GP. When seeing another fault despite the
fixups done, state gets fully reset. A third fault would then lead to
crashing the domain (instead of hanging the hypervisor in an infinite
loop of recurring faults).

Reported-by: Harmandeep Kaur 
Signed-off-by: Jan Beulich 

--- unstable.orig/xen/arch/x86/xstate.c 2016-01-25 09:35:12.0 +0100
+++ unstable/xen/arch/x86/xstate.c  2016-01-27 10:23:06.0 +0100
@@ -29,6 +29,8 @@ unsigned int *__read_mostly xstate_sizes
 static unsigned int __read_mostly xstate_features;
 static unsigned int __read_mostly xstate_comp_offsets[sizeof(xfeature_mask)*8];
 
+static uint32_t __read_mostly mxcsr_mask = MXCSR_DEFAULT;
+
 /* Cached xcr0 for fast read */
 static DEFINE_PER_CPU(uint64_t, xcr0);
 
@@ -342,6 +344,7 @@ void xrstor(struct vcpu *v, uint64_t mas
 uint32_t hmask = mask >> 32;
 uint32_t lmask = mask;
 struct xsave_struct *ptr = v->arch.xsave_area;
+unsigned int faults, prev_faults;
 
 /*
  * AMD CPUs don't save/restore FDP/FIP/FOP unless an exception
@@ -361,35 +364,85 @@ void xrstor(struct vcpu *v, uint64_t mas
 /*
  * XRSTOR can fault if passed a corrupted data block. We handle this
  * possibility, which may occur if the block was passed to us by control
- * tools or through VCPUOP_initialise, by silently clearing the block.
+ * tools or through VCPUOP_initialise, by silently adjusting state.
  */
-switch ( __builtin_expect(ptr->fpu_sse.x[FPU_WORD_SIZE_OFFSET], 8) )
+for ( prev_faults = faults = 0; ; prev_faults = faults )
 {
+switch ( __builtin_expect(ptr->fpu_sse.x[FPU_WORD_SIZE_OFFSET], 8) )
+{
 #define XRSTOR(pfx) \
 alternative_io("1: .byte " pfx "0x0f,0xae,0x2f\n" \
+   "3:\n" \
"   .section .fixup,\"ax\"\n" \
-   "2: mov %[size],%%ecx\n" \
-   "   xor %[lmask_out],%[lmask_out]\n" \
-   "   rep stosb\n" \
-   "   lea %[mem],%[ptr]\n" \
-   "   mov %[lmask_in],%[lmask_out]\n" \
-   "   jmp 1b\n" \
+   "2: inc%z[faults] %[faults]\n" \
+   "   jmp 3b\n" \
"   .previous\n" \
_ASM_EXTABLE(1b, 2b), \
".byte " pfx "0x0f,0xc7,0x1f\n", \
 

Re: [Xen-devel] Error booting Xen

[Dario Faggioli]

On Mon, 2016-01-25 at 06:42 -0700, Jan Beulich wrote:
> > > > On 21.01.16 at 16:14,  wrote:
> > On Wed, 2016-01-20 at 03:06 -0700, Jan Beulich wrote:

> > > But of course another question then is why the XRSTORS faults
> > > in the first place. I guess we'll need a debugging patch to dump
> > > the full state to understand that.
> > > 
> > If someone can produce and send such patch, I'm sure Harmandeep
> > will be
> > happy to give it a try on her hardware.
> 
> So here you go. Instead of a debugging one, I hope I have at
> once fixed the issue in a suitable way. Whether we'd like to keep
> the debugging output we can decide later on.
> 
Great, thanks!

> Both patches need to be applied; while the order shouldn't matter,
> the alignment one is a prereq to the actual change.
> 
Ok. Harmandeep, can you give these patches a try when you get a chance?
(contact me, by email or on IRC, if you need help doing so).

Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)



signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Jan Beulich]

>>> On 26.01.16 at 13:51,  wrote:
> I tried the patch and I am very happy to inform you
> all that the patch has solved my problem. Now I am
> able to boot Xen without disabling XSAVE. I have
> full log of boot at http://paste2.org/gVW0Z9nm (if
> any one is interested. also "XXX Hello, this is my
> first mod :)" is printed by my mod, so ignore that
> one).

Thanks for trying it out, but while I'm glad it helped I'm afraid
we're not done here. With (for every vCPU)

(XEN) traps.c:3290: GPF (): 82d0801c1d08 -> 82d080252e5c
(XEN) d0v1 fault#1: mxcsr=1f80
(XEN) d0v1 xs= xc=
(XEN) d0v1 r0= r1=
(XEN) d0v1 r2= r3=
(XEN) d0v1 r4= r5=
(XEN) traps.c:3290: GPF (): 82d0801c1d08 -> 82d080252e5c
(XEN) d0v1 fault#2: mxcsr=1f80
(XEN) d0v1 xs= xc=
(XEN) d0v1 r0= r1=
(XEN) d0v1 r2= r3=
(XEN) d0v1 r4= r5=

it continues to be unclear why bit 63 in the value printed as
xc= isn't set from the beginning. Or wait, I think I see where
the problem is. Here's a 3rd patch, to try together with the
other two. The expectation would be for the above log
messages to then disappear altogether. (And then the patch
should also work on its own, i.e. with the other two removed
again.) Please let us know.

Thanks, Jan

x86/xstate: don't unintentionally clear compaction bit

When the VGCF_I387_VALID flag is clear in arch_set_info_guest()'s input
we must not clear the compaction bit when using XSAVES/XRSTORS. Split
initialization of xcomp_bv from the other FPU/SSE/AVX related state
setup in this function.

Signed-off-by: Jan Beulich 

--- unstable.orig/xen/arch/x86/domain.c
+++ unstable/xen/arch/x86/domain.c
@@ -922,15 +922,10 @@ int arch_set_info_guest(
 {
 memcpy(v->arch.fpu_ctxt, >fpu_ctxt, sizeof(c.nat->fpu_ctxt));
 if ( v->arch.xsave_area )
-{
 v->arch.xsave_area->xsave_hdr.xstate_bv = XSTATE_FP_SSE;
-v->arch.xsave_area->xsave_hdr.xcomp_bv =
-cpu_has_xsaves ? XSTATE_COMPACTION_ENABLED : 0;
-}
 }
 else if ( v->arch.xsave_area )
-memset(>arch.xsave_area->xsave_hdr, 0,
-   sizeof(v->arch.xsave_area->xsave_hdr));
+v->arch.xsave_area->xsave_hdr.xstate_bv = 0;
 else
 {
 typeof(v->arch.xsave_area->fpu_sse) *fpu_sse = v->arch.fpu_ctxt;
@@ -939,6 +934,11 @@ int arch_set_info_guest(
 fpu_sse->fcw = FCW_DEFAULT;
 fpu_sse->mxcsr = MXCSR_DEFAULT;
 }
+if ( cpu_has_xsaves )
+{
+ASSERT(v->arch.xsave_area);
+v->arch.xsave_area->xsave_hdr.xcomp_bv = XSTATE_COMPACTION_ENABLED;
+}
 
 if ( !compat )
 {




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Jan Beulich]

>>> On 26.01.16 at 14:13,  wrote:
> The patch as I already said is letting me boot
> into the Xen, but the system is now resetting
> stating XSAVE as the cause. I have attached
> links to two cases where system was reset as
> the result. I don't think that problem is fully
> solved yet.
> http://paste2.org/Ky56Z92g 
> http://paste2.org/3hcbG6L7 

I guess this would also get resolved by the 3rd patch just sent.

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Harmandeep Kaur]

I tried 3rd patch together with earlier two. I'm
afraid the problem is not solved completely.
Full log goes here, http://paste2.org/KEAetMHb

Regards,
Harmandeep

On Tue, Jan 26, 2016 at 6:53 PM, Jan Beulich  wrote:
 On 26.01.16 at 14:13,  wrote:
>> The patch as I already said is letting me boot
>> into the Xen, but the system is now resetting
>> stating XSAVE as the cause. I have attached
>> links to two cases where system was reset as
>> the result. I don't think that problem is fully
>> solved yet.
>> http://paste2.org/Ky56Z92g
>> http://paste2.org/3hcbG6L7
>
> I guess this would also get resolved by the 3rd patch just sent.
>
> Jan
>

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Jan Beulich]

>>> On 26.01.16 at 18:01,  wrote:
> I tried 3rd patch together with earlier two. I'm
> afraid the problem is not solved completely.
> Full log goes here, http://paste2.org/KEAetMHb 

Well, wait - we can't solve all problems at once. The crash here
is in the context of do_domctl(), i.e. makes me assume that the
system booted up fine (and the problematic log messages are
gone). So the original issue is fixed. Now for this second issue,
would you mind first of all telling us what exactly it is you're
doing when the crash occurs? Because, as you hopefully
understand, such information often helps understanding where
and/or why things are going wrong.

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Harmandeep Kaur]

Last time, I did absolutely nothing. System was idle
and it crashed just after the login. Now, I booted the
system again and this time, there is no reset. But,
performance of the system is very slow. Browser
(Mozilla Firefox) freezes a lot. Also, before applying
patches, when I used to disabe xsave it resulted in
same kind of performance issues. And the following
is still present in the log.

(XEN) traps.c:3290: GPF (): 82d0801c1cea -> 82d080252e5c
(XEN) d1v1 fault#1: mxcsr=1f80
(XEN) d1v1 xs=0003 xc=8000
(XEN) d1v1 r0= r1=
(XEN) d1v1 r2= r3=
(XEN) d1v1 r4= r5=
(XEN) traps.c:3290: GPF (): 82d0801c1cea -> 82d080252e5c
(XEN) d1v1 fault#2: mxcsr=1f80
(XEN) d1v1 xs= xc=
(XEN) d1v1 r0= r1=
(XEN) d1v1 r2= r3=
(XEN) d1v1 r4= r5=

Full log here: http://paste2.org/C8WpyKOg

Regards,
Harmandeep

On Tue, Jan 26, 2016 at 10:53 PM, Jan Beulich  wrote:
 On 26.01.16 at 18:01,  wrote:
>> I tried 3rd patch together with earlier two. I'm
>> afraid the problem is not solved completely.
>> Full log goes here, http://paste2.org/KEAetMHb
>
> Well, wait - we can't solve all problems at once. The crash here
> is in the context of do_domctl(), i.e. makes me assume that the
> system booted up fine (and the problematic log messages are
> gone). So the original issue is fixed. Now for this second issue,
> would you mind first of all telling us what exactly it is you're
> doing when the crash occurs? Because, as you hopefully
> understand, such information often helps understanding where
> and/or why things are going wrong.
>
> Jan
>

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Dario Faggioli]

On Tue, 2016-01-26 at 23:32 +0530, Harmandeep Kaur wrote:
> Last time, I did absolutely nothing. System was idle
> and it crashed just after the login. Now, I booted the
> system again and this time, there is no reset. But,
> performance of the system is very slow. Browser
> (Mozilla Firefox) freezes a lot. Also, before applying
> patches, when I used to disabe xsave it resulted in
> same kind of performance issues. 
>
Mmm... are you sure the performance is actually affected by "xsave=0",
and/or by Jan's patch? It's hard to check, as without at least one of
them the box does not boot, but I don't think the things (e.g., Firefox
freezing or not starting) are necessarily related.

In particular, you have in your Xen boot parameters list, this item:

 "dom0_mem=1024M,max:1024M"

This means that, in dom0, you will "only" have 1GB of RAM available.
And if you just login after boot and start Firefox, dom0 is where
Firefox is going to be running... and 1G, that Firefox will have to
share with the rest of Linux running as dom0, may be too few RAM for
it. And in fact, in your last log, we see this (from dom0, not from
Xen!):

[  851.63] Out of memory: Kill process 1945 (firefox) score 325 or 
sacrifice child
[  851.644461] Killed process 1945 (firefox) total-vm:1228008kB, 
anon-rss:305536kB, file-rss:0kB

If you want to run a graphical environment on that test box, and browse
with Firefox, then you should increase the amount of RAM you allow dom0
to use. When I suggested you to use 1024, I was assuming (given how
your work environment is setup) you were not going to do any such
thing.

> And the following
> is still present in the log.
> 
> (XEN) traps.c:3290: GPF (): 82d0801c1cea -> 82d080252e5c
> (XEN) d1v1 fault#1: mxcsr=1f80
> (XEN) d1v1 xs=0003 xc=8000
> (XEN) d1v1 r0= r1=
> (XEN) d1v1 r2= r3=
> (XEN) d1v1 r4= r5=
> (XEN) traps.c:3290: GPF (): 82d0801c1cea -> 82d080252e5c
> (XEN) d1v1 fault#2: mxcsr=1f80
> (XEN) d1v1 xs= xc=
> (XEN) d1v1 r0= r1=
> (XEN) d1v1 r2= r3=
> (XEN) d1v1 r4= r5=
> 
Mmm... and this is with all Jan's patches applied?

So, just to make sure we understand each others, you're saying that,
again with all patches applied, and with you not doing anything
significantly different between a) and b) below, the system either:

 a) crashes right after login, like this: http://paste2.org/KEAetMHb

 b) does not crash (you're even able to try starting Firefox), but Xen
    produces the following output: http://paste2.org/C8WpyKOg

Is this correct?

Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)



signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Harmandeep Kaur]

On Tue, Jan 26, 2016 at 11:58 PM, Dario Faggioli
 wrote:
> On Tue, 2016-01-26 at 23:32 +0530, Harmandeep Kaur wrote:
>> Last time, I did absolutely nothing. System was idle
>> and it crashed just after the login. Now, I booted the
>> system again and this time, there is no reset. But,
>> performance of the system is very slow. Browser
>> (Mozilla Firefox) freezes a lot. Also, before applying
>> patches, when I used to disabe xsave it resulted in
>> same kind of performance issues.
>>
> Mmm... are you sure the performance is actually affected by "xsave=0",
> and/or by Jan's patch? It's hard to check, as without at least one of
> them the box does not boot, but I don't think the things (e.g., Firefox
> freezing or not starting) are necessarily related.
>
> In particular, you have in your Xen boot parameters list, this item:
>
>  "dom0_mem=1024M,max:1024M"
>
> This means that, in dom0, you will "only" have 1GB of RAM available.
> And if you just login after boot and start Firefox, dom0 is where
> Firefox is going to be running... and 1G, that Firefox will have to
> share with the rest of Linux running as dom0, may be too few RAM for
> it. And in fact, in your last log, we see this (from dom0, not from
> Xen!):
>
> [  851.63] Out of memory: Kill process 1945 (firefox) score 325 or 
> sacrifice child
> [  851.644461] Killed process 1945 (firefox) total-vm:1228008kB, 
> anon-rss:305536kB, file-rss:0kB
>
> If you want to run a graphical environment on that test box, and browse
> with Firefox, then you should increase the amount of RAM you allow dom0
> to use. When I suggested you to use 1024, I was assuming (given how
> your work environment is setup) you were not going to do any such
> thing.
>
Actually I didn't notice this performance issue before this xsave
bug, maybe we added this line later on. Anyways I can now
check this by increasing the memory.

>> And the following
>> is still present in the log.
>>
>> (XEN) traps.c:3290: GPF (): 82d0801c1cea -> 82d080252e5c
>> (XEN) d1v1 fault#1: mxcsr=1f80
>> (XEN) d1v1 xs=0003 xc=8000
>> (XEN) d1v1 r0= r1=
>> (XEN) d1v1 r2= r3=
>> (XEN) d1v1 r4= r5=
>> (XEN) traps.c:3290: GPF (): 82d0801c1cea -> 82d080252e5c
>> (XEN) d1v1 fault#2: mxcsr=1f80
>> (XEN) d1v1 xs= xc=
>> (XEN) d1v1 r0= r1=
>> (XEN) d1v1 r2= r3=
>> (XEN) d1v1 r4= r5=
>>
> Mmm... and this is with all Jan's patches applied?

Yes, all three patches applied.

> So, just to make sure we understand each others, you're saying that,
> again with all patches applied, and with you not doing anything
> significantly different between a) and b) below, the system either:
>
>  a) crashes right after login, like this: http://paste2.org/KEAetMHb
>
>  b) does not crash (you're even able to try starting Firefox), but Xen
> produces the following output: http://paste2.org/C8WpyKOg
>
> Is this correct?

Yes. And I tried to boot several times and around 70-80% of
times system crashes right after the login as in case 'a'.

Regards,
Harmandeep

> Regards,
> Dario
> --
> <> (Raistlin Majere)
> -
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)
>

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Harmandeep Kaur]

The patch as I already said is letting me boot
into the Xen, but the system is now resetting
stating XSAVE as the cause. I have attached
links to two cases where system was reset as
the result. I don't think that problem is fully
solved yet.
http://paste2.org/Ky56Z92g
http://paste2.org/3hcbG6L7

Regards,
Harman

On Tue, Jan 26, 2016 at 6:21 PM, Harmandeep Kaur
 wrote:
> Hi guys,
>
> I tried the patch and I am very happy to inform you
> all that the patch has solved my problem. Now I am
> able to boot Xen without disabling XSAVE. I have
> full log of boot at http://paste2.org/gVW0Z9nm (if
> any one is interested. also "XXX Hello, this is my
> first mod :)" is printed by my mod, so ignore that
> one).
>
> Thank you guys for your help.
> Regards,
> Harman
>
> On Mon, Jan 25, 2016 at 7:12 PM, Jan Beulich  wrote:
> On 21.01.16 at 16:14,  wrote:
>>> On Wed, 2016-01-20 at 03:06 -0700, Jan Beulich wrote:
 > > > On 19.01.16 at 20:55,  wrote:
 >
 > $ 'addr2line -e xen-syms 82d0801c1cce' returns
 > 'xen/xen/arch/x86/xstate.c:387' which again points to
 > xsave. Also, adding 'xsave=0' makes it boot just fine.

 Ah, I think I see the issue: We're zeroing the entire save area in
 the fixup code, which will make XRSTORS fault unconditionally.
 Shuai, would you please look into possible ways of fixing this?
 Just setting the compressed flag may not be enough, and falling
 back to plain XRSTOR doesn't seem to be an option either - I'm in
 particular worried that the current model of zeroing everything
 is bogus with the growing number of different components which
 get loaded here.

 But of course another question then is why the XRSTORS faults
 in the first place. I guess we'll need a debugging patch to dump
 the full state to understand that.

>>> If someone can produce and send such patch, I'm sure Harmandeep will be
>>> happy to give it a try on her hardware.
>>
>> So here you go. Instead of a debugging one, I hope I have at
>> once fixed the issue in a suitable way. Whether we'd like to keep
>> the debugging output we can decide later on.
>>
>> Both patches need to be applied; while the order shouldn't matter,
>> the alignment one is a prereq to the actual change.
>>
>> Jan
>>

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Harmandeep Kaur]

Hi guys,

I tried the patch and I am very happy to inform you
all that the patch has solved my problem. Now I am
able to boot Xen without disabling XSAVE. I have
full log of boot at http://paste2.org/gVW0Z9nm (if
any one is interested. also "XXX Hello, this is my
first mod :)" is printed by my mod, so ignore that
one).

Thank you guys for your help.
Regards,
Harman

On Mon, Jan 25, 2016 at 7:12 PM, Jan Beulich  wrote:
 On 21.01.16 at 16:14,  wrote:
>> On Wed, 2016-01-20 at 03:06 -0700, Jan Beulich wrote:
>>> > > > On 19.01.16 at 20:55,  wrote:
>>> >
>>> > $ 'addr2line -e xen-syms 82d0801c1cce' returns
>>> > 'xen/xen/arch/x86/xstate.c:387' which again points to
>>> > xsave. Also, adding 'xsave=0' makes it boot just fine.
>>>
>>> Ah, I think I see the issue: We're zeroing the entire save area in
>>> the fixup code, which will make XRSTORS fault unconditionally.
>>> Shuai, would you please look into possible ways of fixing this?
>>> Just setting the compressed flag may not be enough, and falling
>>> back to plain XRSTOR doesn't seem to be an option either - I'm in
>>> particular worried that the current model of zeroing everything
>>> is bogus with the growing number of different components which
>>> get loaded here.
>>>
>>> But of course another question then is why the XRSTORS faults
>>> in the first place. I guess we'll need a debugging patch to dump
>>> the full state to understand that.
>>>
>> If someone can produce and send such patch, I'm sure Harmandeep will be
>> happy to give it a try on her hardware.
>
> So here you go. Instead of a debugging one, I hope I have at
> once fixed the issue in a suitable way. Whether we'd like to keep
> the debugging output we can decide later on.
>
> Both patches need to be applied; while the order shouldn't matter,
> the alignment one is a prereq to the actual change.
>
> Jan
>

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Jan Beulich]

>>> On 21.01.16 at 16:14,  wrote:
> On Wed, 2016-01-20 at 03:06 -0700, Jan Beulich wrote:
>> > > > On 19.01.16 at 20:55,  wrote:
>> > 
>> > $ 'addr2line -e xen-syms 82d0801c1cce' returns
>> > 'xen/xen/arch/x86/xstate.c:387' which again points to
>> > xsave. Also, adding 'xsave=0' makes it boot just fine.
>> 
>> Ah, I think I see the issue: We're zeroing the entire save area in
>> the fixup code, which will make XRSTORS fault unconditionally.
>> Shuai, would you please look into possible ways of fixing this?
>> Just setting the compressed flag may not be enough, and falling
>> back to plain XRSTOR doesn't seem to be an option either - I'm in
>> particular worried that the current model of zeroing everything
>> is bogus with the growing number of different components which
>> get loaded here.
>> 
>> But of course another question then is why the XRSTORS faults
>> in the first place. I guess we'll need a debugging patch to dump
>> the full state to understand that.
>> 
> If someone can produce and send such patch, I'm sure Harmandeep will be
> happy to give it a try on her hardware.

So here you go. Instead of a debugging one, I hope I have at
once fixed the issue in a suitable way. Whether we'd like to keep
the debugging output we can decide later on.

Both patches need to be applied; while the order shouldn't matter,
the alignment one is a prereq to the actual change.

Jan

x86/xstate: fix fault behavior on XRSTORS

XRSTORS unconditionally faults when xcomp_bv has bit 63 clear. Instead
of just fixing this issue, overhaul the fault recovery code, which -
one of the many mistakes made when xstate support got introduced - was
blindly mirroring that accompanying FXRSTOR, neglecting the fact that
XRSTOR{,S} aren't all-or-nothing instructions. The new code, first of
all, does all the recovery actions in C, simplifying the inline
assembly used. And it does its work in a multi-stage fashion: Upon
first seeing a fault, state fixups get applied strictly based on what
architecturally may cause #GP. When seeing another fault despite the
fixups done, state gets fully reset. A third fault would then lead to
crashing the domain (instead of hanging the hypervisor in an infinite
loop of recurring faults).

Reported-by: Harmandeep Kaur 
Signed-off-by: Jan Beulich 

--- unstable.orig/xen/arch/x86/xstate.c 2016-01-25 09:35:12.0 +0100
+++ unstable/xen/arch/x86/xstate.c  2016-01-25 12:34:48.0 +0100
@@ -29,6 +29,8 @@ unsigned int *__read_mostly xstate_sizes
 static unsigned int __read_mostly xstate_features;
 static unsigned int __read_mostly xstate_comp_offsets[sizeof(xfeature_mask)*8];
 
+static uint32_t __read_mostly mxcsr_mask = MXCSR_DEFAULT;
+
 /* Cached xcr0 for fast read */
 static DEFINE_PER_CPU(uint64_t, xcr0);
 
@@ -342,6 +344,7 @@ void xrstor(struct vcpu *v, uint64_t mas
 uint32_t hmask = mask >> 32;
 uint32_t lmask = mask;
 struct xsave_struct *ptr = v->arch.xsave_area;
+unsigned int faults, prev_faults;
 
 /*
  * AMD CPUs don't save/restore FDP/FIP/FOP unless an exception
@@ -361,35 +364,79 @@ void xrstor(struct vcpu *v, uint64_t mas
 /*
  * XRSTOR can fault if passed a corrupted data block. We handle this
  * possibility, which may occur if the block was passed to us by control
- * tools or through VCPUOP_initialise, by silently clearing the block.
+ * tools or through VCPUOP_initialise, by silently adjusting state.
  */
-switch ( __builtin_expect(ptr->fpu_sse.x[FPU_WORD_SIZE_OFFSET], 8) )
+for ( prev_faults = faults = 0; ; prev_faults = faults )
 {
+switch ( __builtin_expect(ptr->fpu_sse.x[FPU_WORD_SIZE_OFFSET], 8) )
+{
 #define XRSTOR(pfx) \
 alternative_io("1: .byte " pfx "0x0f,0xae,0x2f\n" \
+   "3:\n" \
"   .section .fixup,\"ax\"\n" \
-   "2: mov %[size],%%ecx\n" \
-   "   xor %[lmask_out],%[lmask_out]\n" \
-   "   rep stosb\n" \
-   "   lea %[mem],%[ptr]\n" \
-   "   mov %[lmask_in],%[lmask_out]\n" \
-   "   jmp 1b\n" \
+   "2: inc%z[faults] %[faults]\n" \
+   "   jmp 3b\n" \
"   .previous\n" \
_ASM_EXTABLE(1b, 2b), \
".byte " pfx "0x0f,0xc7,0x1f\n", \
X86_FEATURE_XSAVES, \
-   ASM_OUTPUT2([ptr] "+" (ptr), [lmask_out] "+" 
(lmask)), \
-   [mem] "m" (*ptr), [lmask_in] "g" (lmask), \
-   [hmask] "d" (hmask), [size] "m" (xsave_cntxt_size) \
-   : "ecx")
-
-default:
-XRSTOR("0x48,");
-break;
-case 4: case 2:
-XRSTOR("");
-break;
+   ASM_OUTPUT2([mem] "+m" (*ptr), [faults] 

Re: [Xen-devel] Error booting Xen

[Dario Faggioli]

On Wed, 2016-01-20 at 03:06 -0700, Jan Beulich wrote:
> > > > On 19.01.16 at 20:55,  wrote:
> > 
> > $ 'addr2line -e xen-syms 82d0801c1cce' returns
> > 'xen/xen/arch/x86/xstate.c:387' which again points to
> > xsave. Also, adding 'xsave=0' makes it boot just fine.
> 
> Ah, I think I see the issue: We're zeroing the entire save area in
> the fixup code, which will make XRSTORS fault unconditionally.
> Shuai, would you please look into possible ways of fixing this?
> Just setting the compressed flag may not be enough, and falling
> back to plain XRSTOR doesn't seem to be an option either - I'm in
> particular worried that the current model of zeroing everything
> is bogus with the growing number of different components which
> get loaded here.
> 
> But of course another question then is why the XRSTORS faults
> in the first place. I guess we'll need a debugging patch to dump
> the full state to understand that.
> 
If someone can produce and send such patch, I'm sure Harmandeep will be
happy to give it a try on her hardware.

Thanks and Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)



signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Harmandeep Kaur]

On Thu, Jan 21, 2016 at 8:44 PM, Dario Faggioli
 wrote:
> On Wed, 2016-01-20 at 03:06 -0700, Jan Beulich wrote:
>> > > > On 19.01.16 at 20:55,  wrote:
>> >
>> > $ 'addr2line -e xen-syms 82d0801c1cce' returns
>> > 'xen/xen/arch/x86/xstate.c:387' which again points to
>> > xsave. Also, adding 'xsave=0' makes it boot just fine.
>>
>> Ah, I think I see the issue: We're zeroing the entire save area in
>> the fixup code, which will make XRSTORS fault unconditionally.
>> Shuai, would you please look into possible ways of fixing this?
>> Just setting the compressed flag may not be enough, and falling
>> back to plain XRSTOR doesn't seem to be an option either - I'm in
>> particular worried that the current model of zeroing everything
>> is bogus with the growing number of different components which
>> get loaded here.
>>
>> But of course another question then is why the XRSTORS faults
>> in the first place. I guess we'll need a debugging patch to dump
>> the full state to understand that.
>>
> If someone can produce and send such patch, I'm sure Harmandeep will be
> happy to give it a try on her hardware.
>

Yes, sure. I will be more than happy to do this :)
Thank you Dario.

Regards,
Harman
> Thanks and Regards,
> Dario
> --
> <> (Raistlin Majere)
> -
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)
>

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Jan Beulich]

>>> On 19.01.16 at 20:55,  wrote:
> I tried booting staging branch but results were identical. Following
> line repeats endlessly.
> (XEN) traps.c:3290: GPF (): 82d0801c1cce -> 82d080252e5c
> 
> $ 'addr2line -e xen-syms 82d0801c1cce' returns
> 'xen/xen/arch/x86/xstate.c:387' which again points to
> xsave. Also, adding 'xsave=0' makes it boot just fine.

Ah, I think I see the issue: We're zeroing the entire save area in
the fixup code, which will make XRSTORS fault unconditionally.
Shuai, would you please look into possible ways of fixing this?
Just setting the compressed flag may not be enough, and falling
back to plain XRSTOR doesn't seem to be an option either - I'm in
particular worried that the current model of zeroing everything
is bogus with the growing number of different components which
get loaded here.

But of course another question then is why the XRSTORS faults
in the first place. I guess we'll need a debugging patch to dump
the full state to understand that.

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Harmandeep Kaur]

On Tue, Jan 19, 2016 at 10:38 PM, Dario Faggioli
 wrote:
> On Tue, 2016-01-19 at 17:02 +, Andrew Cooper wrote:
>> On 19/01/16 16:47, Jan Beulich wrote:
>> > > > > On 19.01.16 at 17:27,  wrote:
>> > > Adding 'xsave=0' is working for now. Thank you
>> > > all for your help :)
>> > But that means we actually should get to the bottom of your
>> > problem!
>>
>> There was some discussion on IRC.  `xrstror` was repeatedly taking
>> the
>> same fault; i.e. the fixup code wasn't fixing up suitably.
>>
> Yes, indeed.
>
>> As a first candidate, I expect 83ae0bb2 is a likely candidate.
>> Harmandeep is using a Skylake processor.
>>
> Yes, she is on Skylake. But she's also using master, so,
> AFAICT, 83ae0bb2 is not there.
>
> She will be trying to use staging (and kill xsave=0) soon, and will let
> us know.

I tried booting staging branch but results were identical. Following
line repeats endlessly.
(XEN) traps.c:3290: GPF (): 82d0801c1cce -> 82d080252e5c

$ 'addr2line -e xen-syms 82d0801c1cce' returns
'xen/xen/arch/x86/xstate.c:387' which again points to
xsave. Also, adding 'xsave=0' makes it boot just fine.

Full boot log here, http://paste2.org/1DCge9Fb

Thanks and regards,
Harmandeep

> Thanks and Regards,
> Dario
> --
> <> (Raistlin Majere)
> -
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)
>

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Harmandeep Kaur]

> On Mon, 2016-01-18 at 18:09 +, Andrew Cooper wrote:
> > On 18/01/16 18:03, Harmandeep Kaur wrote:
> > > Hi,
> > >
> > > I tried to boot Xen (cloned from git clone
> > > git://xenbits.xen.org/xen.git), but it stucks
> > > in a infinite loop. I got the log via serial.
> > > You can inspect it at
> > > http://paste2.org/YCpkzbpG
> > >
> > > If anybody has encountered this issue
> > > before, I will appreciate any help.
> >
> > Does that last line "(XEN) traps.c:3290: GPF (): 82d0801af585
> > ->
> > 82d080240e5a" repeat forever?
> >
> It does, as far as Harmandeep told me.
>
> Also, that same Linux kernel binary boots as baremetal.
>
> ISTR (but, Harmandeep, correct me if I'm wrong) that it was booting
> fine as Dom0, if the Ubuntu packaged version of Xen was used (which I
> think is 4.5).
>

Yes, you're right Dario. Even 4.7-unstable used to run (around till 2
weeks ago).

Maybe kernel has been changed since then.

> > If so, it is a dom0 kernel bug not a Xen bug.
> >
> Yeah, but again, it was booting as dom0 with another Xen... doesn't
> that mean that Xen at least has a role in exposing it?
>
> Also, what would you think it's better to try next... try another
> kernel?
>
> Thanks and Regards,
> Dario

On Mon, Jan 18, 2016 at 11:33 PM, Harmandeep Kaur
 wrote:
> Hi,
>
> I tried to boot Xen (cloned from git clone
> git://xenbits.xen.org/xen.git), but it stucks
> in a infinite loop. I got the log via serial.
> You can inspect it at
> http://paste2.org/YCpkzbpG
>
> If anybody has encountered this issue
> before, I will appreciate any help.
>
> Regards,
> Harmandeep Kaur
> Outreachy Intern

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Jan Beulich]

>>> On 18.01.16 at 19:29,  wrote:
> On Mon, 2016-01-18 at 18:09 +, Andrew Cooper wrote:
>> On 18/01/16 18:03, Harmandeep Kaur wrote:
>> > I tried to boot Xen (cloned from git clone
>> > git://xenbits.xen.org/xen.git), but it stucks
>> > in a infinite loop. I got the log via serial.
>> > You can inspect it at
>> > http://paste2.org/YCpkzbpG 
>> > 
>> > If anybody has encountered this issue
>> > before, I will appreciate any help.
>> 
>> Does that last line "(XEN) traps.c:3290: GPF (): 82d0801af585
>> ->
>> 82d080240e5a" repeat forever?
>> 
> It does, as far as Harmandeep told me.
> 
> Also, that same Linux kernel binary boots as baremetal.
> 
> ISTR (but, Harmandeep, correct me if I'm wrong) that it was booting
> fine as Dom0, if the Ubuntu packaged version of Xen was used (which I
> think is 4.5).
> 
>> If so, it is a dom0 kernel bug not a Xen bug.
>> 
> Yeah, but again, it was booting as dom0 with another Xen... doesn't
> that mean that Xen at least has a role in exposing it?
> 
> Also, what would you think it's better to try next... try another
> kernel?

Figure out what the GP fault that gets recovered from is being
caused by. Then - if indeed it worked on another hypervisor
build - figure out the difference(s) in behavior.

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Dario Faggioli]

On Tue, 2016-01-19 at 06:36 -0700, Jan Beulich wrote:
> > > > On 18.01.16 at 19:29,  wrote:
> > 
> > Yeah, but again, it was booting as dom0 with another Xen... doesn't
> > that mean that Xen at least has a role in exposing it?
> > 
> > Also, what would you think it's better to try next... try another
> > kernel?
> 
> Figure out what the GP fault that gets recovered from is being
> caused by. 
>
Yep, thanks.

Harmandeep is looking at (learning how to do) this already. If you feel
like sharing any knowledge/giving any advice on how to best achieve
that, I'm sure she'll appreciate (but if you can't or don't have time,
no problem :-D).

> Then - if indeed it worked on another hypervisor
> build - figure out the difference(s) in behavior.
> 
Right. Harmandeep, you can try this rather easily!.

You have a clone of the xen.git already. I think you just have to:
 - `make uninstall' (I'm not sure it will work, but it at least should 
    not harm!)
 - clean the sources
 - checkout, e.g., in a branch, one of the tags of a previous release, 
   say RELEASE-4.6.0 or RELEASE-4.5.0
 - build again
 - install
 - update the bootloader
 - reboot and report the result.

Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)



signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Andrew Cooper]

On 19/01/16 14:51, Dario Faggioli wrote:
> On Tue, 2016-01-19 at 06:36 -0700, Jan Beulich wrote:
> On 18.01.16 at 19:29,  wrote:
>>>  
>>> Yeah, but again, it was booting as dom0 with another Xen... doesn't
>>> that mean that Xen at least has a role in exposing it?
>>>
>>> Also, what would you think it's better to try next... try another
>>> kernel?
>> Figure out what the GP fault that gets recovered from is being
>> caused by. 
>>
> Yep, thanks.
>
> Harmandeep is looking at (learning how to do) this already. If you feel
> like sharing any knowledge/giving any advice on how to best achieve
> that, I'm sure she'll appreciate (but if you can't or don't have time,
> no problem :-D).

First, locate exactly where the fault is

`addr2line -e xen-syms 82d0801af585`

will give you a file/line reference.  xen-syms must be from the same
build of the hypervisor that you booted.

~Andrew

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Harmandeep Kaur]

Hi,

Adding 'xsave=0' is working for now. Thank you
all for your help :)

Regards,
Harman

On Tue, Jan 19, 2016 at 8:37 PM, Andrew Cooper
 wrote:
> On 19/01/16 14:51, Dario Faggioli wrote:
>> On Tue, 2016-01-19 at 06:36 -0700, Jan Beulich wrote:
>> On 18.01.16 at 19:29,  wrote:

 Yeah, but again, it was booting as dom0 with another Xen... doesn't
 that mean that Xen at least has a role in exposing it?

 Also, what would you think it's better to try next... try another
 kernel?
>>> Figure out what the GP fault that gets recovered from is being
>>> caused by.
>>>
>> Yep, thanks.
>>
>> Harmandeep is looking at (learning how to do) this already. If you feel
>> like sharing any knowledge/giving any advice on how to best achieve
>> that, I'm sure she'll appreciate (but if you can't or don't have time,
>> no problem :-D).
>
> First, locate exactly where the fault is
>
> `addr2line -e xen-syms 82d0801af585`
>
> will give you a file/line reference.  xen-syms must be from the same
> build of the hypervisor that you booted.
>
> ~Andrew

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Jan Beulich]

>>> On 19.01.16 at 17:27,  wrote:
> Adding 'xsave=0' is working for now. Thank you
> all for your help :)

But that means we actually should get to the bottom of your problem!

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Andrew Cooper]

On 19/01/16 16:47, Jan Beulich wrote:
 On 19.01.16 at 17:27,  wrote:
>> Adding 'xsave=0' is working for now. Thank you
>> all for your help :)
> But that means we actually should get to the bottom of your problem!

There was some discussion on IRC.  `xrstror` was repeatedly taking the
same fault; i.e. the fixup code wasn't fixing up suitably.

As a first candidate, I expect 83ae0bb2 is a likely candidate. 
Harmandeep is using a Skylake processor.

~Andrew

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Dario Faggioli]

On Tue, 2016-01-19 at 17:02 +, Andrew Cooper wrote:
> On 19/01/16 16:47, Jan Beulich wrote:
> > > > > On 19.01.16 at 17:27,  wrote:
> > > Adding 'xsave=0' is working for now. Thank you
> > > all for your help :)
> > But that means we actually should get to the bottom of your
> > problem!
> 
> There was some discussion on IRC.  `xrstror` was repeatedly taking
> the
> same fault; i.e. the fixup code wasn't fixing up suitably.
> 
Yes, indeed.
 
> As a first candidate, I expect 83ae0bb2 is a likely candidate. 
> Harmandeep is using a Skylake processor.
> 
Yes, she is on Skylake. But she's also using master, so,
AFAICT, 83ae0bb2 is not there.

She will be trying to use staging (and kill xsave=0) soon, and will let
us know.

Thanks and Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)



signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Andrew Cooper]

On 19/01/16 17:08, Dario Faggioli wrote:
> On Tue, 2016-01-19 at 17:02 +, Andrew Cooper wrote:
>> On 19/01/16 16:47, Jan Beulich wrote:
>> On 19.01.16 at 17:27,  wrote:
 Adding 'xsave=0' is working for now. Thank you
 all for your help :)
>>> But that means we actually should get to the bottom of your
>>> problem!
>> There was some discussion on IRC.  `xrstror` was repeatedly taking
>> the
>> same fault; i.e. the fixup code wasn't fixing up suitably.
>>
> Yes, indeed.
>  
>> As a first candidate, I expect 83ae0bb2 is a likely candidate. 
>> Harmandeep is using a Skylake processor.
>>
> Yes, she is on Skylake. But she's also using master, so,
> AFAICT, 83ae0bb2 is not there.
>
> She will be trying to use staging (and kill xsave=0) soon, and will let
> us know.

We have some Skylake kit in XenServer.  I will investigate when it
becomes available.

~Andrew

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] Error booting Xen

[Harmandeep Kaur]

Hi,

I tried to boot Xen (cloned from git clone
git://xenbits.xen.org/xen.git), but it stucks
in a infinite loop. I got the log via serial.
You can inspect it at
http://paste2.org/YCpkzbpG

If anybody has encountered this issue
before, I will appreciate any help.

Regards,
Harmandeep Kaur
Outreachy Intern

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error booting Xen

[Andrew Cooper]

On 18/01/16 18:03, Harmandeep Kaur wrote:
> Hi,
>
> I tried to boot Xen (cloned from git clone
> git://xenbits.xen.org/xen.git), but it stucks
> in a infinite loop. I got the log via serial.
> You can inspect it at
> http://paste2.org/YCpkzbpG
>
> If anybody has encountered this issue
> before, I will appreciate any help.

Does that last line "(XEN) traps.c:3290: GPF (): 82d0801af585 ->
82d080240e5a" repeat forever?

If so, it is a dom0 kernel bug not a Xen bug.

~Andrew

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel