Re: [Xen-devel] [xen-4.8-testing test] 114505: regressions - FAIL

[Jan Beulich]

>>> Andrew Cooper  10/16/17 6:39 PM >>>
>On 16/10/17 17:12, Jan Beulich wrote:
> On 16.10.17 at 11:14,  wrote:
>>> Jan: This highlights that TSX/VMEntry failure fixes probably want
>>> backporting to before Xen 4.9.  IIRC, the 6 patches needed are:
>> So I'm mildly confused by this request:
>>
>>> e3eb84e33c36 (only as a functional prerequisite)
>>> 9b93c6b3695b: x86/vmx: introduce vmx_find_msr()
>>> 7f11aa4b2b1f: x86/vmx: optimize vmx_read/write_guest_msr()
>>> d6e9f8d4f35d: x86/vmx: fix vmentry failure with TSX bits in LBR
>>> f97838bbd980: x86: Move microcode loading earlier
>> Up to here, everything is in 4.9 already afaict. Considering the
>> context here is a 4.8 test report, did you perhaps mean to ask
>> for this on 4.8 (and possibly also 4.7)?
>
>Well - I did ask for "backporting to before Xen 4.9".

Oh, I'm sorry - I had read something you didn't write (to do the backporting
before 4.9 has its first stable release go out).

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [xen-4.8-testing test] 114505: regressions - FAIL

[Andrew Cooper]

On 16/10/17 16:16, Ian Jackson wrote:
> Andrew Cooper writes ("Re: [Xen-devel] [xen-4.8-testing test] 114505: 
> regressions - FAIL"):
>> On 15/10/17 20:45, osstest service owner wrote:
>>> flight 114505 xen-4.8-testing real [real]
>>> http://logs.test-lab.xenproject.org/osstest/logs/114505/
>>>
>>> Regressions :-(
>>>
>>> Tests which did not succeed and are blocking,
>>> including tests which could not be run:
>>>  test-xtf-amd64-amd64-2 48 xtf/test-hvm64-lbr-tsx-vmentry fail REGR. vs. 
>>> 114173
> ...
>> Ian: These tests exercise something very machine specific, and the XTF
>> tests really do need tying to specific hardware when making regression
>> considerations.
> Is this test new enough that it might have never run on that
> hardware ?  If so then a force push might be justified.

andrewcoop@andrewcoop:/local/xen-test-framework.git$ git show --format=fuller 
36d926fe
commit 36d926fe0e9b7db39965f430cdb4c5f1daf4eef3
Author: Andrew Cooper <andrew.coop...@citrix.com>
AuthorDate: Wed Oct 12 18:23:42 2016
Commit: Andrew Cooper <andrew.coop...@citrix.com>
CommitDate: Tue Apr 25 13:55:42 2017

LBR/TSX VMentry failure test

Signed-off-by: Andrew Cooper <andrew.coop...@citrix.com>

It has been running in OSSTest for a fair while now.

The test will only fail on versions of Xen before the fixes went in
(Currently Xen 4.9), on Haswell and Broadwell hardware.

Its also possible

> It is difficult to tie the tests to specific hardware without
> insisting that every run uses every host.

How hard would it be to tag each flight with which host it ran on, and
filter for host == current when determining whether a regression has
occurred?

~Andrew

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [xen-4.8-testing test] 114505: regressions - FAIL

[Andrew Cooper]

On 16/10/17 17:12, Jan Beulich wrote:
 On 16.10.17 at 11:14,  wrote:
>> On 15/10/17 20:45, osstest service owner wrote:
>>> flight 114505 xen-4.8-testing real [real]
>>> http://logs.test-lab.xenproject.org/osstest/logs/114505/ 
>>>
>>> Regressions :-(
>>>
>>> Tests which did not succeed and are blocking,
>>> including tests which could not be run:
>>>  test-xtf-amd64-amd64-2 48 xtf/test-hvm64-lbr-tsx-vmentry fail REGR. vs. 
>> 114173
>>> Tests which are failing intermittently (not blocking):
>>>  test-xtf-amd64-amd64-5 48 xtf/test-hvm64-lbr-tsx-vmentry fail in 114454 
>> pass in 114505
>>
>> Ian: These tests exercise something very machine specific, and the XTF
>> tests really do need tying to specific hardware when making regression
>> considerations.
>>
>> Jan: This highlights that TSX/VMEntry failure fixes probably want
>> backporting to before Xen 4.9.  IIRC, the 6 patches needed are:
> So I'm mildly confused by this request:
>
>> e3eb84e33c36 (only as a functional prerequisite)
>> 9b93c6b3695b: x86/vmx: introduce vmx_find_msr()
>> 7f11aa4b2b1f: x86/vmx: optimize vmx_read/write_guest_msr()
>> d6e9f8d4f35d: x86/vmx: fix vmentry failure with TSX bits in LBR
>> f97838bbd980: x86: Move microcode loading earlier
> Up to here, everything is in 4.9 already afaict. Considering the
> context here is a 4.8 test report, did you perhaps mean to ask
> for this on 4.8 (and possibly also 4.7)?

Well - I did ask for "backporting to before Xen 4.9".

> If so, I'm not really sure -
> these changes taken together look a little large for the gain
> they provide.

We have had several xen-devel reports of this problem, starting against
Xen 4.6 iirc.  If you really thing its more risk than its worth then fine.

>
>> 20f1976b4419: x86/vmx: Fix vmentry failure because of invalid LER on 
>> Broadwell
> I'll see to pull this one in for 4.9.1.

Oops - I'd not spotted that that change was missing in Xen 4.9.  Yes -
please backport that one.

~Andrew

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [xen-4.8-testing test] 114505: regressions - FAIL

[Jan Beulich]

>>> On 16.10.17 at 11:14,  wrote:
> On 15/10/17 20:45, osstest service owner wrote:
>> flight 114505 xen-4.8-testing real [real]
>> http://logs.test-lab.xenproject.org/osstest/logs/114505/ 
>>
>> Regressions :-(
>>
>> Tests which did not succeed and are blocking,
>> including tests which could not be run:
>>  test-xtf-amd64-amd64-2 48 xtf/test-hvm64-lbr-tsx-vmentry fail REGR. vs. 
> 114173
>>
>> Tests which are failing intermittently (not blocking):
>>  test-xtf-amd64-amd64-5 48 xtf/test-hvm64-lbr-tsx-vmentry fail in 114454 
> pass in 114505
> 
> Ian: These tests exercise something very machine specific, and the XTF
> tests really do need tying to specific hardware when making regression
> considerations.
> 
> Jan: This highlights that TSX/VMEntry failure fixes probably want
> backporting to before Xen 4.9.  IIRC, the 6 patches needed are:

So I'm mildly confused by this request:

> e3eb84e33c36 (only as a functional prerequisite)
> 9b93c6b3695b: x86/vmx: introduce vmx_find_msr()
> 7f11aa4b2b1f: x86/vmx: optimize vmx_read/write_guest_msr()
> d6e9f8d4f35d: x86/vmx: fix vmentry failure with TSX bits in LBR
> f97838bbd980: x86: Move microcode loading earlier

Up to here, everything is in 4.9 already afaict. Considering the
context here is a 4.8 test report, did you perhaps mean to ask
for this on 4.8 (and possibly also 4.7)? If so, I'm not really sure -
these changes taken together look a little large for the gain
they provide.

> 20f1976b4419: x86/vmx: Fix vmentry failure because of invalid LER on 
> Broadwell

I'll see to pull this one in for 4.9.1.

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [xen-4.8-testing test] 114505: regressions - FAIL

[Ian Jackson]

Andrew Cooper writes ("Re: [Xen-devel] [xen-4.8-testing test] 114505: 
regressions - FAIL"):
> On 15/10/17 20:45, osstest service owner wrote:
> > flight 114505 xen-4.8-testing real [real]
> > http://logs.test-lab.xenproject.org/osstest/logs/114505/
> >
> > Regressions :-(
> >
> > Tests which did not succeed and are blocking,
> > including tests which could not be run:
> >  test-xtf-amd64-amd64-2 48 xtf/test-hvm64-lbr-tsx-vmentry fail REGR. vs. 
> > 114173
...
> Ian: These tests exercise something very machine specific, and the XTF
> tests really do need tying to specific hardware when making regression
> considerations.

Is this test new enough that it might have never run on that
hardware ?  If so then a force push might be justified.

It is difficult to tie the tests to specific hardware without
insisting that every run uses every host.

Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [xen-4.8-testing test] 114505: regressions - FAIL

[Andrew Cooper]

On 15/10/17 20:45, osstest service owner wrote:
> flight 114505 xen-4.8-testing real [real]
> http://logs.test-lab.xenproject.org/osstest/logs/114505/
>
> Regressions :-(
>
> Tests which did not succeed and are blocking,
> including tests which could not be run:
>  test-xtf-amd64-amd64-2 48 xtf/test-hvm64-lbr-tsx-vmentry fail REGR. vs. 
> 114173
>
> Tests which are failing intermittently (not blocking):
>  test-xtf-amd64-amd64-5 48 xtf/test-hvm64-lbr-tsx-vmentry fail in 114454 pass 
> in 114505

Ian: These tests exercise something very machine specific, and the XTF
tests really do need tying to specific hardware when making regression
considerations.

Jan: This highlights that TSX/VMEntry failure fixes probably want
backporting to before Xen 4.9.  IIRC, the 6 patches needed are:

e3eb84e33c36 (only as a functional prerequisite)
9b93c6b3695b: x86/vmx: introduce vmx_find_msr()
7f11aa4b2b1f: x86/vmx: optimize vmx_read/write_guest_msr()
d6e9f8d4f35d: x86/vmx: fix vmentry failure with TSX bits in LBR
f97838bbd980: x86: Move microcode loading earlier
20f1976b4419: x86/vmx: Fix vmentry failure because of invalid LER on Broadwell

~Andrew

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [xen-4.8-testing test] 114505: regressions - FAIL

[osstest service owner]

flight 114505 xen-4.8-testing real [real]
http://logs.test-lab.xenproject.org/osstest/logs/114505/

Regressions :-(

Tests which did not succeed and are blocking,
including tests which could not be run:
 test-xtf-amd64-amd64-2 48 xtf/test-hvm64-lbr-tsx-vmentry fail REGR. vs. 114173

Tests which are failing intermittently (not blocking):
 test-xtf-amd64-amd64-5 48 xtf/test-hvm64-lbr-tsx-vmentry fail in 114454 pass 
in 114505
 test-armhf-armhf-xl-rtds 12 guest-startfail pass in 114454

Tests which did not succeed, but are not blocking:
 test-xtf-amd64-amd64-3 48 xtf/test-hvm64-lbr-tsx-vmentry fail in 114454 like 
114173
 test-armhf-armhf-xl-rtds 16 guest-start/debian.repeat fail in 114454 like 
114173
 test-armhf-armhf-xl-rtds13 migrate-support-check fail in 114454 never pass
 test-armhf-armhf-xl-rtds 14 saverestore-support-check fail in 114454 never pass
 test-amd64-i386-xl-qemuu-win7-amd64 17 guest-stop fail like 114173
 test-amd64-i386-xl-qemut-win7-amd64 17 guest-stop fail like 114173
 test-amd64-amd64-xl-qemut-win7-amd64 17 guest-stopfail like 114173
 test-amd64-amd64-xl-rtds 10 debian-install   fail  like 114173
 test-amd64-amd64-xl-qemut-ws16-amd64 10 windows-installfail never pass
 test-amd64-amd64-libvirt-xsm 13 migrate-support-checkfail   never pass
 test-amd64-i386-libvirt  13 migrate-support-checkfail   never pass
 test-amd64-amd64-libvirt 13 migrate-support-checkfail   never pass
 test-amd64-i386-libvirt-xsm  13 migrate-support-checkfail   never pass
 test-amd64-amd64-xl-qemuu-ws16-amd64 10 windows-installfail never pass
 test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm 11 migrate-support-check 
fail never pass
 build-i386-prev   7 xen-build/dist-test  fail   never pass
 test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 11 migrate-support-check 
fail never pass
 test-amd64-i386-libvirt-qcow2 12 migrate-support-checkfail  never pass
 build-amd64-prev  7 xen-build/dist-test  fail   never pass
 test-armhf-armhf-xl-arndale  13 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-arndale  14 saverestore-support-checkfail   never pass
 test-amd64-amd64-libvirt-vhd 12 migrate-support-checkfail   never pass
 test-amd64-i386-xl-qemut-ws16-amd64 13 guest-saverestore   fail never pass
 test-amd64-amd64-qemuu-nested-amd 17 debian-hvm-install/l1/l2  fail never pass
 test-armhf-armhf-xl-multivcpu 13 migrate-support-checkfail  never pass
 test-armhf-armhf-xl-multivcpu 14 saverestore-support-checkfail  never pass
 test-armhf-armhf-libvirt 13 migrate-support-checkfail   never pass
 test-armhf-armhf-libvirt 14 saverestore-support-checkfail   never pass
 test-armhf-armhf-xl  13 migrate-support-checkfail   never pass
 test-armhf-armhf-xl  14 saverestore-support-checkfail   never pass
 test-armhf-armhf-xl-credit2  13 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-credit2  14 saverestore-support-checkfail   never pass
 test-amd64-i386-xl-qemuu-ws16-amd64 13 guest-saverestore   fail never pass
 test-armhf-armhf-libvirt-xsm 13 migrate-support-checkfail   never pass
 test-armhf-armhf-libvirt-xsm 14 saverestore-support-checkfail   never pass
 test-armhf-armhf-xl-cubietruck 13 migrate-support-checkfail never pass
 test-armhf-armhf-xl-cubietruck 14 saverestore-support-checkfail never pass
 test-armhf-armhf-xl-vhd  12 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-vhd  13 saverestore-support-checkfail   never pass
 test-armhf-armhf-libvirt-raw 12 migrate-support-checkfail   never pass
 test-armhf-armhf-libvirt-raw 13 saverestore-support-checkfail   never pass
 test-amd64-amd64-xl-qemut-win10-i386 10 windows-installfail never pass
 test-amd64-amd64-xl-qemuu-win10-i386 10 windows-installfail never pass
 test-amd64-i386-xl-qemut-win10-i386 10 windows-install fail never pass
 test-amd64-i386-xl-qemuu-win10-i386 10 windows-install fail never pass
 test-armhf-armhf-xl-xsm  13 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-xsm  14 saverestore-support-checkfail   never pass

version targeted for testing:
 xen  bdc2ae68e2ecba1c3f55ad953189fe33362d1c51
baseline version:
 xen  667f70e658c4c382672056ebaf1505b4c5cdb0aa

Last test of basis   114173  2017-10-09 03:27:38 Z6 days
Failing since114313  2017-10-11 00:46:14 Z4 days4 attempts
Testing same since   114454  2017-10-13 06:48:53 Z2 days2 attempts


People who touched revisions under test:
  Andrew Cooper 
  George Dunlap 
  Jan Beulich 
  Julien Grall