Re: [Xen-devel] Guest Testing in OSSTEST - What distros and versions should we test against

[Roger Pau Monné]

On Fri, May 10, 2019 at 01:48:58AM -0600, Jan Beulich wrote:
> >>> On 10.05.19 at 03:28,  wrote:
> > Hi all,
> > 
> > following a discussion with committers about Guest testing in OSSTEST, it 
> > surfaced that we have not updated what distros we test in OSSTEST for a 
> > very 
> > long time. All agreed that we should regularly review what we test against: 
> > maybe at the beginning of a release cycle
> > 
> > In any case, currently we test against
> > 
> > x86 HVM guests:
> >   debian-9.4.0-{i386,amd64}-CD-1.iso
> >   rhel-server-6.1-i386-dvd.iso
> >   win10v1703-x86.iso
> >   win7-x64.iso
> >   ws16-x64.iso
> >   FreeBSD-10.1-CUSTOM-{i386,amd64}-20150525.raw.xz
> 
> Testing fixed snapshots (commonly the original release afaict) instead
> of latest updates is pretty meaningless imo. While I would like to
> recommend testing all the (active) distros of parties actively involved
> in Xen development, I have to admite that I have no idea at all how
> this would work for e.g. our SLE versions (such that latest bits would
> get tested).

FWIW, this is what osstest does in order to test FreeBSD HEAD. I'm not
sure how much of this would apply to SLE.

There's a dedicated FreeBSD flight, that does the following:

 - Install a bare-metal box using the latest known to work install
   media.
 - Fetch FreeBSD sources and build new install media from scratch.
 - Use the newly built media to install a box from scratch.
 - Build FreeBSD again to make sure the new version is capable of
   building itself at least.
 - Build Xen using the new FreeBSD version.

If all of those steps succeed the new version is anointed, and it will
be used as the install media for new flights.

Note that this is a time consuming process, so making this part of a
regular Xen flight is not feasible, hence the anointment in order to
store the binary installer so other flights can re-use it.

Also note that this process is circular, and needs an initial seed
(binary installer) to be manually anointed for all this to work.

Roger.

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Guest Testing in OSSTEST - What distros and versions should we test against

[Lars Kurth]

I was hoping we'd get a list of distros + versions together, but have not had 
any input on specific distros

To make this easier, I created 
https://cryptpad.fr/pad/#/2/pad/edit/9MppkbU36LDcWT12uC6Xk3SQ/ 
Trying something else, as some people have trouble with google docs

Regards
Lars

> On 9 May 2019, at 19:28, Lars Kurth  wrote:
> 
> Hi all,
> 
> following a discussion with committers about Guest testing in OSSTEST, it 
> surfaced that we have not updated what distros we test in OSSTEST for a very 
> long time. All agreed that we should regularly review what we test against: 
> maybe at the beginning of a release cycle
> 
> In any case, currently we test against
> 
> x86 HVM guests:
>  debian-9.4.0-{i386,amd64}-CD-1.iso
>  rhel-server-6.1-i386-dvd.iso
>  win10v1703-x86.iso
>  win7-x64.iso
>  ws16-x64.iso
>  FreeBSD-10.1-CUSTOM-{i386,amd64}-20150525.raw.xz
>  Debian HVM {i386,amd64} via debian-installer netinst [1]
> 
> x86 PV guests:
>  Debian PV {i386,amd64} via debian-installer netinst [1]
> 
> ARM guests:
>  Debian PV via debian-installer netinst [1]
> 
> [1] whatever Debian release osstest itself mostly runs
> 
> So I am opening the floor to suggestions.
> 
> With regards to Windows testing we have some restrictions. We have tried 
> several times to buy additional test licenses, but this never went anywhere 
> (some of the VM licenses are not available for our environment, unless you 
> bulk buy, which is very expensive). The only approach that would allow us to 
> test against different windows versions would be to require everyone who may 
> touch OSSTEST which is not doable.
> 
> I can bring this up with the MS open source office, if there are strong 
> feelings about this and try again
> 
> Lars
> 


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Guest Testing in OSSTEST - What distros and versions should we test against

[Lars Kurth]

> On 9 May 2019, at 19:43, Rich Persaud  wrote:
> 
> On May 9, 2019, at 21:28, Lars Kurth  > wrote:
> 
>> With regards to Windows testing we have some restrictions. We have tried 
>> several times to buy additional test licenses, but this never went anywhere 
>> (some of the VM licenses are not available for our environment, unless you 
>> bulk buy, which is very expensive). The only approach that would allow us to 
>> test against different windows versions would be to require everyone who may 
>> touch OSSTEST which is not doable.
> 
> Are the 90-day test/eval versions of Windows incompatible with OSSTEST?
> 
>   https://www.microsoft.com/en-us/evalcenter/evaluate-windows-10-enterprise 
> 
>   https://www.microsoft.com/en-us/evalcenter/ 
> 
> 

Actually, that's a possibility. Our use may not be compatible with the T's of 
the eval license though. It wasn't when we checked last time. But that can be 
checked.
Lars

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Guest Testing in OSSTEST - What distros and versions should we test against

[Lars Kurth]

> On 10 May 2019, at 01:48, Jan Beulich  wrote:
>> 
>> With regards to Windows testing we have some restrictions. We have tried 
>> several times to buy additional test licenses, but this never went anywhere 
>> (some of the VM licenses are not available for our environment, unless you 
>> bulk buy, which is very expensive). The only approach that would allow us to 
>> test against different windows versions would be to require everyone who may 
>> touch OSSTEST which is not doable.
>> 
>> I can bring this up with the MS open source office, if there are strong 
>> feelings about this and try again
> 
> If there's at least a (not overly) small chance of succeeding, I think this
> may be worth it, unless Rich's suggestion already helps.
> 

I will try again. Let's work on the basis that this is possible and see where 
it goes
Lars___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Guest Testing in OSSTEST - What distros and versions should we test against

[Jan Beulich]

>>> On 10.05.19 at 03:28,  wrote:
> Hi all,
> 
> following a discussion with committers about Guest testing in OSSTEST, it 
> surfaced that we have not updated what distros we test in OSSTEST for a very 
> long time. All agreed that we should regularly review what we test against: 
> maybe at the beginning of a release cycle
> 
> In any case, currently we test against
> 
> x86 HVM guests:
>   debian-9.4.0-{i386,amd64}-CD-1.iso
>   rhel-server-6.1-i386-dvd.iso
>   win10v1703-x86.iso
>   win7-x64.iso
>   ws16-x64.iso
>   FreeBSD-10.1-CUSTOM-{i386,amd64}-20150525.raw.xz

Testing fixed snapshots (commonly the original release afaict) instead
of latest updates is pretty meaningless imo. While I would like to
recommend testing all the (active) distros of parties actively involved
in Xen development, I have to admite that I have no idea at all how
this would work for e.g. our SLE versions (such that latest bits would
get tested).

>   Debian HVM {i386,amd64} via debian-installer netinst [1]
> 
> x86 PV guests:
>   Debian PV {i386,amd64} via debian-installer netinst [1]
> 
> ARM guests:
>   Debian PV via debian-installer netinst [1]
> 
> [1] whatever Debian release osstest itself mostly runs
> 
> So I am opening the floor to suggestions.
> 
> With regards to Windows testing we have some restrictions. We have tried 
> several times to buy additional test licenses, but this never went anywhere 
> (some of the VM licenses are not available for our environment, unless you 
> bulk buy, which is very expensive). The only approach that would allow us to 
> test against different windows versions would be to require everyone who may 
> touch OSSTEST which is not doable.
> 
> I can bring this up with the MS open source office, if there are strong 
> feelings about this and try again

If there's at least a (not overly) small chance of succeeding, I think this
may be worth it, unless Rich's suggestion already helps.

Jan



___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Guest Testing in OSSTEST - What distros and versions should we test against

[Rich Persaud]

> On May 9, 2019, at 21:28, Lars Kurth  wrote:
> 
> Hi all,
> 
> following a discussion with committers about Guest testing in OSSTEST, it 
> surfaced that we have not updated what distros we test in OSSTEST for a very 
> long time. All agreed that we should regularly review what we test against: 
> maybe at the beginning of a release cycle
> 
> In any case, currently we test against
> 
> x86 HVM guests:
>  debian-9.4.0-{i386,amd64}-CD-1.iso
>  rhel-server-6.1-i386-dvd.iso
>  win10v1703-x86.iso
>  win7-x64.iso
>  ws16-x64.iso
>  FreeBSD-10.1-CUSTOM-{i386,amd64}-20150525.raw.xz
>  Debian HVM {i386,amd64} via debian-installer netinst [1]
> 
> x86 PV guests:
>  Debian PV {i386,amd64} via debian-installer netinst [1]
> 
> ARM guests:
>  Debian PV via debian-installer netinst [1]
> 
> [1] whatever Debian release osstest itself mostly runs
> 
> So I am opening the floor to suggestions.
> 
> With regards to Windows testing we have some restrictions. We have tried 
> several times to buy additional test licenses, but this never went anywhere 
> (some of the VM licenses are not available for our environment, unless you 
> bulk buy, which is very expensive). The only approach that would allow us to 
> test against different windows versions would be to require everyone who may 
> touch OSSTEST which is not doable.

Are the 90-day test/eval versions of Windows incompatible with OSSTEST?

  https://www.microsoft.com/en-us/evalcenter/evaluate-windows-10-enterprise
  https://www.microsoft.com/en-us/evalcenter/

Rich___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] Guest Testing in OSSTEST - What distros and versions should we test against

[Lars Kurth]

Hi all,

following a discussion with committers about Guest testing in OSSTEST, it 
surfaced that we have not updated what distros we test in OSSTEST for a very 
long time. All agreed that we should regularly review what we test against: 
maybe at the beginning of a release cycle

In any case, currently we test against

x86 HVM guests:
  debian-9.4.0-{i386,amd64}-CD-1.iso
  rhel-server-6.1-i386-dvd.iso
  win10v1703-x86.iso
  win7-x64.iso
  ws16-x64.iso
  FreeBSD-10.1-CUSTOM-{i386,amd64}-20150525.raw.xz
  Debian HVM {i386,amd64} via debian-installer netinst [1]

x86 PV guests:
  Debian PV {i386,amd64} via debian-installer netinst [1]

ARM guests:
  Debian PV via debian-installer netinst [1]

[1] whatever Debian release osstest itself mostly runs

So I am opening the floor to suggestions.

With regards to Windows testing we have some restrictions. We have tried 
several times to buy additional test licenses, but this never went anywhere 
(some of the VM licenses are not available for our environment, unless you bulk 
buy, which is very expensive). The only approach that would allow us to test 
against different windows versions would be to require everyone who may touch 
OSSTEST which is not doable.

I can bring this up with the MS open source office, if there are strong 
feelings about this and try again

Lars

  
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel