Re: Validation test proposal: mediawriter testing table

[Kamil Paral]

On Sat, Apr 22, 2017 at 8:22 AM, Adam Williamson  wrote:

> The #expr stuff is basically a reinvention of the FedoraVersionNumber
> template; that template transcludes CurrentFedoraVersion, so if we just
> substitute FedoraVersionNumber (or FedoraVersion) it doesn't solve the
> problem, as the underlying CurrentFedoraVersion usage is still a
> *transclusion*, so the number would still change in result pages. Which
> is what we're trying to avoid.
>

Eh. Thanks! That's exactly what I wanted to achieve without studying that
crazy syntax.


>
> If you're interested in all the details, ask and I'll rewrite the long
> answer. :)
>

No, no, I'm good, thanks :)
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Validation test proposal: mediawriter testing table

[Kamil Paral]

On Fri, Apr 21, 2017 at 3:46 PM, Thomas Gilliard 
wrote:

> I added download links and test results to this
> https://fedoraproject.org/w/index.php?title=Template:
> Installation_test_matrix=next=491518#Fedora_Media_Writer
> Please edit these changes if this is in error
>
> satellit
>


Adam reverted those (thanks Adam). This is a template page, individual
results are not supposed to be written there. Results are written into
standard test matrices, which are generated from the matrix template. If
you find this difficult to distinguish, it's easy - only fill out matrices
which are linked in test-announce list :) Thanks.
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Validation test proposal: mediawriter testing table

[Adam Williamson]

On Fri, 2017-04-21 at 14:16 +0200, Kamil Paral wrote:
> On Wed, Apr 19, 2017 at 2:25 PM, Kamil Paral  wrote:
> 
> > 
> > That's the question though - is it required? I thought the test case would
> > be marked as optional. We already require FMW in "Default boot and install"
> > matrix, so Fedora is covered (we just don't know *what* Fedora they used
> > for that). And are any FMW crashes on Win/macOS blocking? I don't think we
> > have a criterion for that (the question is whether we should).
> > 
> > So as long as we keep that table optional, I think it's useful to split
> > out the Fedora version, so that we clearly see what people tried and what
> > they didn't. Also, for the same reason it would probably help to see
> > Windows 7 and Windows 10 separated.
> > 
> > But if we want to mark it as required, I wouldn't want yet another
> > exploded matrix, and I agree Fedora 26 could be left out (and Windows
> > merged together).
> > 
> 
> I figured it wouldn't hurt if I add the table marked as optional:
> https://fedoraproject.org/w/index.php?title=Template%3AInstallation_test_matrix=491515=491270
> 
> We can now continue discussing how we want it to look like, but at least
> it's there already.
> 
> As a footnote, I hardcoded Fedora 24/25/26 numbers, because I wasn't sure
> how to do that better. I can use a template, but I don't want the numbers
> to change once it is converted to a standard install matrix, and I'm not
> sure what method relval uses to do so. (So this is mainly a question for
> Adam, whether we can use templates in there or not).

That weird empty mail was a long reply to this which something ate.
*sob*

So here's the short version: yeah, you can do it. It involves advanced-
level wiki magic, though. I did it:

https://fedoraproject.org/w/index.php?title=Template%3AInstallation_test_matrix=491546=491533

What's going on there is documented at




https://www.mediawi
ki.org/wiki/Help:Substitution , it's basically all about setting up
template invocations within a template so that a substitution occurs
when the parent template is being substituted, but doesn't happen in
any other circumstance (when the parent template is viewed or saved, or
when it's transcluded). (That page explains the difference between
transclusion and substitution).

The #expr stuff is basically a reinvention of the FedoraVersionNumber
template; that template transcludes CurrentFedoraVersion, so if we just
substitute FedoraVersionNumber (or FedoraVersion) it doesn't solve the
problem, as the underlying CurrentFedoraVersion usage is still a
*transclusion*, so the number would still change in result pages. Which
is what we're trying to avoid.

If you're interested in all the details, ask and I'll rewrite the long
answer. :)
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Validation test proposal: mediawriter testing table

[Adam Williamson]

On Fri, 2017-04-21 at 14:16 +0200, Kamil Paral wrote:

> but I don't want the numbers
> to change once it is converted to a standard install matrix, and I'm not
> sure what method relval uses to do so.

just for the record, relval / python-wikitcms per se don't do anything
magic in this regard, it's all done in the wiki templates. you can
(still) create the validation pages correctly by hand, just by using
the correct template invocation. the text that relval actually uses to
create a validation page is just something like this:

{{subst:Validation_results|testtype=Installation|release=26|milestone=Beta|compose=1.1}}

or:

{{subst:Validation_results|testtype=Installation|release=26|milestone=Beta|date=20170420.n.0}}

and thanks to a heavy dose of wiki template magic, that 'seed text' (as
I call it) produces the entire clean validation page. relval's job
(some of which is done in python-wikitcms) is knowing what pages to
create for a given 'validation event', and what to call them, and what
categories they should be in.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Validation test proposal: mediawriter testing table

[Adam Williamson]

___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Validation test proposal: mediawriter testing table

[Adam Williamson]

On Fri, 2017-04-21 at 06:46 -0700, Thomas Gilliard wrote:
> 
> I added download links and test results to this 
> https://fedoraproject.org/w/index.php?title=Template:Installation_test_matrix=next=491518#Fedora_Media_Writer
> 
> Please edit these changes if this is in error

Sorry, but yes, it is. As the note on the page says, please don't edit
it unless you're sure you understand how it works. I will revert it
once the site's up again.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Validation test proposal: mediawriter testing table

[Thomas Gilliard]

\
On 04/21/2017 05:16 AM, Kamil Paral wrote:
On Wed, Apr 19, 2017 at 2:25 PM, Kamil Paral > wrote:



That's the question though - is it required? I thought the test
case would be marked as optional. We already require FMW in
"Default boot and install" matrix, so Fedora is covered (we just
don't know *what* Fedora they used for that). And are any FMW
crashes on Win/macOS blocking? I don't think we have a criterion
for that (the question is whether we should).

So as long as we keep that table optional, I think it's useful to
split out the Fedora version, so that we clearly see what people
tried and what they didn't. Also, for the same reason it would
probably help to see Windows 7 and Windows 10 separated.

But if we want to mark it as required, I wouldn't want yet another
exploded matrix, and I agree Fedora 26 could be left out (and
Windows merged together).


I figured it wouldn't hurt if I add the table marked as optional:
https://fedoraproject.org/w/index.php?title=Template%3AInstallation_test_matrix=491515=491270

We can now continue discussing how we want it to look like, but at 
least it's there already.


As a footnote, I hardcoded Fedora 24/25/26 numbers, because I wasn't 
sure how to do that better. I can use a template, but I don't want the 
numbers to change once it is converted to a standard install matrix, 
and I'm not sure what method relval uses to do so. (So this is mainly 
a question for Adam, whether we can use templates in there or not).




___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org


I added download links and test results to this 
https://fedoraproject.org/w/index.php?title=Template:Installation_test_matrix=next=491518#Fedora_Media_Writer


Please edit these changes if this is in error

satellit
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Validation test proposal: mediawriter testing table

[Kamil Paral]

On Wed, Apr 19, 2017 at 2:25 PM, Kamil Paral  wrote:

>
> That's the question though - is it required? I thought the test case would
> be marked as optional. We already require FMW in "Default boot and install"
> matrix, so Fedora is covered (we just don't know *what* Fedora they used
> for that). And are any FMW crashes on Win/macOS blocking? I don't think we
> have a criterion for that (the question is whether we should).
>
> So as long as we keep that table optional, I think it's useful to split
> out the Fedora version, so that we clearly see what people tried and what
> they didn't. Also, for the same reason it would probably help to see
> Windows 7 and Windows 10 separated.
>
> But if we want to mark it as required, I wouldn't want yet another
> exploded matrix, and I agree Fedora 26 could be left out (and Windows
> merged together).
>

I figured it wouldn't hurt if I add the table marked as optional:
https://fedoraproject.org/w/index.php?title=Template%3AInstallation_test_matrix=491515=491270

We can now continue discussing how we want it to look like, but at least
it's there already.

As a footnote, I hardcoded Fedora 24/25/26 numbers, because I wasn't sure
how to do that better. I can use a template, but I don't want the numbers
to change once it is converted to a standard install matrix, and I'm not
sure what method relval uses to do so. (So this is mainly a question for
Adam, whether we can use templates in there or not).
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Validation test proposal: mediawriter testing table

[Kamil Paral]

On Tue, Apr 18, 2017 at 6:13 PM, Adam Williamson  wrote:

> > We'll also probably want "Fedora 26" (Branched at that time) column.
>
> Eh, I intentionally left it out because we've always held that being
> able to write images from the new release is much less important than
> being able to write images from the existing stable releases. The point
> was to try and keep a lid on the required testing, remember? :)
>

That's the question though - is it required? I thought the test case would
be marked as optional. We already require FMW in "Default boot and install"
matrix, so Fedora is covered (we just don't know *what* Fedora they used
for that). And are any FMW crashes on Win/macOS blocking? I don't think we
have a criterion for that (the question is whether we should).

So as long as we keep that table optional, I think it's useful to split out
the Fedora version, so that we clearly see what people tried and what they
didn't. Also, for the same reason it would probably help to see Windows 7
and Windows 10 separated.

But if we want to mark it as required, I wouldn't want yet another exploded
matrix, and I agree Fedora 26 could be left out (and Windows merged
together).
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Validation test proposal: mediawriter testing table

[Martin Bříza]

On Wed, 19 Apr 2017 06:51:00 +0200, Allan Mwenda   
wrote:


Definitely split windows. I've had writer crash on 10 but not 7, trying  
to get a fedora spin (lxde)


On 18 April 2017 15:50:06 GMT+03:00, Lukas Brabec   
wrote:

On Fri, Jan 13, 2017 at 12:32 AM, Adam Williamson
 wrote:

Hi folks! So I finally got around to that 'think about USB test
coverage' item that's been on my todo list forever.

I propose we add a table to the Installation Validation page. The
purpose is simply to check that writing images is working with
mediawriter in the major supported environments: Windows, macOS , and
the supported stable Fedora releases. So it could just look like

this:



WindowsmacOSFedora 24Fedora 25
QA:Testcase_USB_fmw

with the intent being that we at least check that writing any one
release blocking image with mediawriter in each environment.

We could split Windows into 7, 8 and 10 or something, but not sure if
it's really necessary...

Thoughts?
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin .

net

http://www.happyassassin.net
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org


Current media writer supports ARM, I would propose to add a row for
it, so it could look like this:

WindowsmacOSFedora 24Fedora 25
x86_64 iso
arm image

where both "x86_64 iso" and "arm image" links to QA:Testcase_USB_fmw
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org




Yeah I mentioned this already, Win10 is a bit different story than  
anything else when testing FMW.

___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Validation test proposal: mediawriter testing table

[Allan Mwenda]

Definitely split windows. I've had writer crash on 10 but not 7, trying to get 
a fedora spin (lxde)

On 18 April 2017 15:50:06 GMT+03:00, Lukas Brabec  wrote:
>On Fri, Jan 13, 2017 at 12:32 AM, Adam Williamson
> wrote:
>> Hi folks! So I finally got around to that 'think about USB test
>> coverage' item that's been on my todo list forever.
>>
>> I propose we add a table to the Installation Validation page. The
>> purpose is simply to check that writing images is working with
>> mediawriter in the major supported environments: Windows, macOS , and
>> the supported stable Fedora releases. So it could just look like
>this:
>>
>>
>> WindowsmacOSFedora 24Fedora 25
>> QA:Testcase_USB_fmw
>>
>> with the intent being that we at least check that writing any one
>> release blocking image with mediawriter in each environment.
>>
>> We could split Windows into 7, 8 and 10 or something, but not sure if
>> it's really necessary...
>>
>> Thoughts?
>> --
>> Adam Williamson
>> Fedora QA Community Monkey
>> IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin .
>net
>> http://www.happyassassin.net
>> ___
>> test mailing list -- test@lists.fedoraproject.org
>> To unsubscribe send an email to test-le...@lists.fedoraproject.org
>
>Current media writer supports ARM, I would propose to add a row for
>it, so it could look like this:
>
> WindowsmacOSFedora 24Fedora 25
>x86_64 iso
>arm image
>
>where both "x86_64 iso" and "arm image" links to QA:Testcase_USB_fmw
>___
>test mailing list -- test@lists.fedoraproject.org
>To unsubscribe send an email to test-le...@lists.fedoraproject.org

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Validation test proposal: mediawriter testing table

[Adam Williamson]

On Tue, 2017-04-18 at 14:55 +0200, Kamil Paral wrote:
> On Tue, Apr 18, 2017 at 2:50 PM, Lukas Brabec  wrote:
> 
> > 
> > Current media writer supports ARM, I would propose to add a row for
> > it, so it could look like this:
> > 
> >  WindowsmacOSFedora 24Fedora 25
> > x86_64 iso
> > arm image
> > 
> 
> 
> We'll also probably want "Fedora 26" (Branched at that time) column.

Eh, I intentionally left it out because we've always held that being
able to write images from the new release is much less important than
being able to write images from the existing stable releases. The point
was to try and keep a lid on the required testing, remember? :)
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Validation test proposal: mediawriter testing table

[Kamil Paral]

On Tue, Apr 18, 2017 at 2:50 PM, Lukas Brabec  wrote:

>
> Current media writer supports ARM, I would propose to add a row for
> it, so it could look like this:
>
>  WindowsmacOSFedora 24Fedora 25
> x86_64 iso
> arm image
>


We'll also probably want "Fedora 26" (Branched at that time) column.
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Validation test proposal: mediawriter testing table

[Lukas Brabec]

On Fri, Jan 13, 2017 at 12:32 AM, Adam Williamson
 wrote:
> Hi folks! So I finally got around to that 'think about USB test
> coverage' item that's been on my todo list forever.
>
> I propose we add a table to the Installation Validation page. The
> purpose is simply to check that writing images is working with
> mediawriter in the major supported environments: Windows, macOS , and
> the supported stable Fedora releases. So it could just look like this:
>
>
> WindowsmacOSFedora 24Fedora 25
> QA:Testcase_USB_fmw
>
> with the intent being that we at least check that writing any one
> release blocking image with mediawriter in each environment.
>
> We could split Windows into 7, 8 and 10 or something, but not sure if
> it's really necessary...
>
> Thoughts?
> --
> Adam Williamson
> Fedora QA Community Monkey
> IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
> http://www.happyassassin.net
> ___
> test mailing list -- test@lists.fedoraproject.org
> To unsubscribe send an email to test-le...@lists.fedoraproject.org

Current media writer supports ARM, I would propose to add a row for
it, so it could look like this:

 WindowsmacOSFedora 24Fedora 25
x86_64 iso
arm image

where both "x86_64 iso" and "arm image" links to QA:Testcase_USB_fmw
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Fedora Media Writer release schedule (was Re: Validation test proposal: mediawriter testing table)

[Matthew Miller]

On Mon, Jan 16, 2017 at 11:28:50AM +0100, Martin Bříza wrote:
> new non-Fedora releases. There is no rigid process of getting the
> releases to websites. The process now is I put up a new release and
> ping dgilmore to compile it. He then, when he has time, compiles it
> and puts it somewhere and notifies websites to pull it or he puts it
> there himself.

I think changing "ping dgilmore" to "file a ticket with rel-eng" would
be an improvement. Not that Dennis doesn't do a bunch of stuff -- in
fact, exactly because he does, a ticket will help keep it from getting
lost.

Ideally, we'd have something so you could trigger the build
automatically, but I can see the work/reward of that being low if this
is infrequent.

> For example for 4.0.8 which was released about a week ago there has
> been no activity yet - meanwhile, f25 package is already stable.

A week and waiting for the F25 package to go stable seems fine in most
cases - but we'd want it to be much faster in the event of a serious
bugfix.


> All I can do is ping the people over and over and hope one day the
> version get pushed there. There is very limited feedback in regards
> to what the status is now, if it compiles and there is no testing
> besides mine.

A ticket will help, yeah.

And there needs to be a step in between "build made and available for
download $somewhere" and "build pushed to web server" where QA can do
the QAing.

-- 
Matthew Miller

Fedora Project Leader
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Fedora Media Writer release schedule (was Re: Validation test proposal: mediawriter testing table)

[Martin Bříza]

On Mon, 16 Jan 2017 10:52:28 +0100, Kamil Paral  wrote:


This is related to validation tests, but different enough that I wanted
a new thread. While we *can* release new versions of FMW on
getfedora.org at Fedora GA, there's no reason to tie the two together —
and really, some good ones to *not* double up:

* we can fix bugs without waiting
* we can add new functionality whenever
* and we don't have to worry about surprise failures due to regressions
  on GA release day affecting the rush of people


Sure, makes sense.



I have a feature request (use a distinct user-agent for ISO downloads
so we can tell when FMW is being used) which I'd love to not wait for
June/July to get live.

Do we want to go through validation and update the website whenever
Martin cuts a new release upstream? Or do we want to have
regularly-scheduled times to do it? Or whenever there's a particular
reason to do it and leave it alone otherwise? Or something else?


Martin should be probably the one to tell websites team when to pull new  
version (once it is tested, not every one needs to be pulled).


We'll probably not be able to test every each upstream release he does,  
especially on all platforms. During Branched release cycle, it will be  
easier, since we're testing FMW all the time anyway. Outside of the  
release cycle, he can ping us if we have time to do that, or he can wait  
until some feedback is available in Bodhi and act based on that  
(however, that does not include feedback for non-Fedora systems, but I  
think he said he performs a few smoke tests every time on his test  
machines, so that should be probably fine).


I'm testing the releases myself, yes. Usually I focus on the stuff I  
changed and then just try writing and erasing a drive, so nothing too  
complicated.


I see another problem with the release process now though: pulling new  
non-Fedora releases. There is no rigid process of getting the releases to  
websites. The process now is I put up a new release and ping dgilmore to  
compile it. He then, when he has time, compiles it and puts it somewhere  
and notifies websites to pull it or he puts it there himself.


For example for 4.0.8 which was released about a week ago there has been  
no activity yet - meanwhile, f25 package is already stable.


All I can do is ping the people over and over and hope one day the version  
get pushed there. There is very limited feedback in regards to what the  
status is now, if it compiles and there is no testing besides mine.

___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Fedora Media Writer release schedule (was Re: Validation test proposal: mediawriter testing table)

[Kamil Paral]

> This is related to validation tests, but different enough that I wanted
> a new thread. While we *can* release new versions of FMW on
> getfedora.org at Fedora GA, there's no reason to tie the two together —
> and really, some good ones to *not* double up:
> 
> * we can fix bugs without waiting
> * we can add new functionality whenever
> * and we don't have to worry about surprise failures due to regressions
>   on GA release day affecting the rush of people

Sure, makes sense.

> 
> I have a feature request (use a distinct user-agent for ISO downloads
> so we can tell when FMW is being used) which I'd love to not wait for
> June/July to get live.
> 
> Do we want to go through validation and update the website whenever
> Martin cuts a new release upstream? Or do we want to have
> regularly-scheduled times to do it? Or whenever there's a particular
> reason to do it and leave it alone otherwise? Or something else?

Martin should be probably the one to tell websites team when to pull new 
version (once it is tested, not every one needs to be pulled).

We'll probably not be able to test every each upstream release he does, 
especially on all platforms. During Branched release cycle, it will be easier, 
since we're testing FMW all the time anyway. Outside of the release cycle, he 
can ping us if we have time to do that, or he can wait until some feedback is 
available in Bodhi and act based on that (however, that does not include 
feedback for non-Fedora systems, but I think he said he performs a few smoke 
tests every time on his test machines, so that should be probably fine).
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Validation test proposal: mediawriter testing table

[Chris Murphy]

On Fri, Jan 13, 2017 at 2:36 AM, Kamil Paral  wrote:
>> > Well I don't really want to eat a hat, because I've had my fill for a
>> > lifetime, but I'd eat my hat if FMW works on Windows 10 but does not
>> > work on Windows 7. Or vice versa.
>> > ...
>>
>> It does not work on Windows 7 for some folks. On my laptop with Win 7, it
>> works. I've got a virtual machine where it doesn't work for testing
>> though. I suspect there's a problem with ANGLE (which is a library that
>> translates OpenGL calls to DirectX on Windows) combined with hardware
>> drivers not supporting recent enough OpenGL.
>
> Chris, a picture or didn't happen ;-)

I am not going to eat another goddamn hat just because of some Windows
edge cases :-P

Besides, we live in an era now of merely saying things makes them
true/untrue, and I wouldn't want to impune that by suggesting a photo
enhances my hat eating claim. But it should be completely believable
because I've switched to hat eating due to my former target of self
punishment for being wrong was having noticeable effects on billy goat
herd population.


-- 
Chris Murphy
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Fedora Media Writer release schedule (was Re: Validation test proposal: mediawriter testing table)

[Chris Murphy]

On Fri, Jan 13, 2017 at 8:12 AM, Matthew Miller
 wrote:
> This is related to validation tests, but different enough that I wanted
> a new thread. While we *can* release new versions of FMW on
> getfedora.org at Fedora GA, there's no reason to tie the two together —
> and really, some good ones to *not* double up:
>
> * we can fix bugs without waiting
> * we can add new functionality whenever
> * and we don't have to worry about surprise failures due to regressions
>   on GA release day affecting the rush of people
>
> I have a feature request (use a distinct user-agent for ISO downloads
> so we can tell when FMW is being used) which I'd love to not wait for
> June/July to get live.
>
> Do we want to go through validation and update the website whenever
> Martin cuts a new release upstream? Or do we want to have
> regularly-scheduled times to do it? Or whenever there's a particular
> reason to do it and leave it alone otherwise? Or something else?

I like the idea of decoupling this from the release schedule.

Maybe mbriza "nominates" a release for validation (not every release
needs to go to the web site ASAP perhaps?) and schedules something
like a Test Day with QA. And that test day can act as validation, out
of band. And then after all the major bugs are found to be fixed, it
can be handed off to web folks for updating.


-- 
Chris Murphy
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Fedora Media Writer release schedule (was Re: Validation test proposal: mediawriter testing table)

[Matthew Miller]

This is related to validation tests, but different enough that I wanted
a new thread. While we *can* release new versions of FMW on
getfedora.org at Fedora GA, there's no reason to tie the two together —
and really, some good ones to *not* double up:

* we can fix bugs without waiting
* we can add new functionality whenever
* and we don't have to worry about surprise failures due to regressions
  on GA release day affecting the rush of people

I have a feature request (use a distinct user-agent for ISO downloads
so we can tell when FMW is being used) which I'd love to not wait for
June/July to get live.

Do we want to go through validation and update the website whenever
Martin cuts a new release upstream? Or do we want to have
regularly-scheduled times to do it? Or whenever there's a particular
reason to do it and leave it alone otherwise? Or something else?

-- 
Matthew Miller

Fedora Project Leader
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Validation test proposal: mediawriter testing table

[Matthew Miller]

On Thu, Jan 12, 2017 at 08:30:38PM -0700, Chris Murphy wrote:
> API/ABI stability on Windows is pretty extreme.  In order of market
> share though, by a long shot it's Windows 7, then 10, and a distant
> third is 8.1 and then 8(.0) barely registers. So weirdly enough,
> chances are it'll work on 8/8.1 if it works on 7 and 10. So a thorough
> test would be testing both 7 and 10. Less thorough, but sane, is
> testing just on 10.

From
http://stackoverflow.com/research/developer-survey-2016#technology-desktop-operating-system

22.5% Windows 7
20.8% Windows 10
 8.4% Windows 8
 0.4% Windows XP
 0.1% Windows Vista

and 26.2% OS X, FWIW.

(Fedora, at 1.4%, beats XP and Vista *combined*. Whoo!)

This backs up what you're saying with numbers, and specifically from
our target userbase for Workstation. 

To my deep sadness, they did not ask this question on the 2017 survey
(which is open now).


-- 
Matthew Miller

Fedora Project Leader
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Validation test proposal: mediawriter testing table

[Martin Bříza]

On Fri, 13 Jan 2017 11:07:13 +0100, Kamil Paral  wrote:

I'd consider focusing on using checkisomd5 as the main way to test if  
FMW
works ok. First the offline one and then online one running from the  
flash

drive before bootup.


That's actually a very good advice for the test case. As Martin showed  
me recently, running simply "checkisomd5 --verbose /dev/sdX" after  
you've written the image to /dev/sdX is actually a fully satisfactory  
verification that the image was written properly (or you could also do  
"cmp /dev/sdX image.iso"). Of course booting the thumb stick and letting  
it verify consistency from the syslinux menu is a bonus point.


There's a potential issue here. Sometimes it can happen that the written  
partitions are automounted and some filesystem metadata are changed  
before unplugging (at least that's my guess). This happened to me  
several times during the last cycle, and the checksum verification then  
fails (but the installation proceeds completely OK). This happens also  
on Windows, according to Martin, and it's quite difficult to prevent it.  
We should mention it in the test case, but I'm not sure what advice we  
can give. We don't want to of course handwaive such issues completely,  
because they can also mean some data writing corruption (I found one  
such bug a few months back, and it was FMW's fault).




If the integrity check passes, all other errors are either in the  
compose
process or the packages. Anything above this I would consider testing  
the

actual written image, not FMW.


Yes. In our particular case, in the "Default boot and install" section,  
we also test default installation (anaconda), so I guess we can't  
optimize there. But for that particular test case, the offline/online  
verification should be enough, I believe.


I think I have fixed the issue on Windows. However, I think there's still  
a possibility the OS will mess with your drive if you reinsert it to the  
system. Especially on Mac, where I have worked around this by blocking all  
media mounting while FMW is running.


To be sure this doesn't happen and to included the case in our test, I  
proposed using the online media check too.

___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Validation test proposal: mediawriter testing table

[Kamil Paral]

> I'd consider focusing on using checkisomd5 as the main way to test if FMW
> works ok. First the offline one and then online one running from the flash
> drive before bootup.

That's actually a very good advice for the test case. As Martin showed me 
recently, running simply "checkisomd5 --verbose /dev/sdX" after you've written 
the image to /dev/sdX is actually a fully satisfactory verification that the 
image was written properly (or you could also do "cmp /dev/sdX image.iso"). Of 
course booting the thumb stick and letting it verify consistency from the 
syslinux menu is a bonus point.

There's a potential issue here. Sometimes it can happen that the written 
partitions are automounted and some filesystem metadata are changed before 
unplugging (at least that's my guess). This happened to me several times during 
the last cycle, and the checksum verification then fails (but the installation 
proceeds completely OK). This happens also on Windows, according to Martin, and 
it's quite difficult to prevent it. We should mention it in the test case, but 
I'm not sure what advice we can give. We don't want to of course handwaive such 
issues completely, because they can also mean some data writing corruption (I 
found one such bug a few months back, and it was FMW's fault).

> 
> If the integrity check passes, all other errors are either in the compose
> process or the packages. Anything above this I would consider testing the
> actual written image, not FMW.

Yes. In our particular case, in the "Default boot and install" section, we also 
test default installation (anaconda), so I guess we can't optimize there. But 
for that particular test case, the offline/online verification should be 
enough, I believe.
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Validation test proposal: mediawriter testing table

[Martin Bříza]

On Fri, 13 Jan 2017 00:32:06 +0100, Adam Williamson  
 wrote:



Hi folks! So I finally got around to that 'think about USB test
coverage' item that's been on my todo list forever.

I propose we add a table to the Installation Validation page. The
purpose is simply to check that writing images is working with
mediawriter in the major supported environments: Windows, macOS , and
the supported stable Fedora releases. So it could just look like this:


WindowsmacOSFedora 24Fedora 25
QA:Testcase_USB_fmw

with the intent being that we at least check that writing any one
release blocking image with mediawriter in each environment.

We could split Windows into 7, 8 and 10 or something, but not sure if
it's really necessary...

Thoughts?


I'd consider focusing on using checkisomd5 as the main way to test if FMW  
works ok. First the offline one and then online one running from the flash  
drive before bootup.


If the integrity check passes, all other errors are either in the compose  
process or the packages. Anything above this I would consider testing the  
actual written image, not FMW.

___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Validation test proposal: mediawriter testing table

[Kamil Paral]

> >> > Well I don't really want to eat a hat, because I've had my fill for a
> >> > lifetime, but I'd eat my hat if FMW works on Windows 10 but does not
> >> > work on Windows 7. Or vice versa.
> >> > ...
> >>
> >> It does not work on Windows 7 for some folks. On my laptop with Win 7,
> >> it
> >> works. I've got a virtual machine where it doesn't work for testing
> >> though. I suspect there's a problem with ANGLE (which is a library that
> >> translates OpenGL calls to DirectX on Windows) combined with hardware
> >> drivers not supporting recent enough OpenGL.
> >
> > Chris, a picture or didn't happen ;-)
> 
> Don't need a picture or anything, I know there's a problem. A complex one
> at that. Let's see what happens when Qt 5.8 comes out (in a few days
> actually, I may as well start packaging it for mingw already).
> 

I was talking about *the hat*.
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Validation test proposal: mediawriter testing table

[Kamil Paral]

> Hi folks! So I finally got around to that 'think about USB test
> coverage' item that's been on my todo list forever.
> 
> I propose we add a table to the Installation Validation page. The
> purpose is simply to check that writing images is working with
> mediawriter in the major supported environments: Windows, macOS , and
> the supported stable Fedora releases. So it could just look like this:
> 
> 
> WindowsmacOSFedora 24Fedora 25
> QA:Testcase_USB_fmw
> 
> with the intent being that we at least check that writing any one
> release blocking image with mediawriter in each environment.
> 
> We could split Windows into 7, 8 and 10 or something, but not sure if
> it's really necessary...

After reading Martin Briza's reply, we could have a Win 7 and Win 10 column, 
yeah. But e.g. for dual-boot, I always test just with Win 10 and hope the 
community does the rest :-) I'd probably do it the same here - it should be OK 
if at least one of those fields is populated, not require both. Alternatively, 
we can have a single column and recommend to primarily test on Win 7 and Win 10 
in the test case description (and hope that people will submit multiple results 
for this from time to time, and add a note to indicate what they tested with). 
I don't have a real preference here.

What do we do if one or both of the Windows fields fail? Or the macOS one? Our 
release criteria don't specify a host system at the moment.

Also, should we split Fedora systems per arch (x86_64, armhfp)? I don't want to 
have a too much exploded matrix here, I'd keep it architecture-less I guess.
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Validation test proposal: mediawriter testing table

[Martin Bříza]

On Fri, 13 Jan 2017 10:36:37 +0100, Kamil Paral  wrote:


> Well I don't really want to eat a hat, because I've had my fill for a
> lifetime, but I'd eat my hat if FMW works on Windows 10 but does not
> work on Windows 7. Or vice versa.
> ...

It does not work on Windows 7 for some folks. On my laptop with Win 7,  
it

works. I've got a virtual machine where it doesn't work for testing
though. I suspect there's a problem with ANGLE (which is a library that
translates OpenGL calls to DirectX on Windows) combined with hardware
drivers not supporting recent enough OpenGL.


Chris, a picture or didn't happen ;-)


Don't need a picture or anything, I know there's a problem. A complex one  
at that. Let's see what happens when Qt 5.8 comes out (in a few days  
actually, I may as well start packaging it for mingw already).

___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Validation test proposal: mediawriter testing table

[Kamil Paral]

> > Well I don't really want to eat a hat, because I've had my fill for a
> > lifetime, but I'd eat my hat if FMW works on Windows 10 but does not
> > work on Windows 7. Or vice versa.
> > ...
> 
> It does not work on Windows 7 for some folks. On my laptop with Win 7, it
> works. I've got a virtual machine where it doesn't work for testing
> though. I suspect there's a problem with ANGLE (which is a library that
> translates OpenGL calls to DirectX on Windows) combined with hardware
> drivers not supporting recent enough OpenGL. 

Chris, a picture or didn't happen ;-)
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Validation test proposal: mediawriter testing table

[Martin Bříza]

On Fri, 13 Jan 2017 04:30:38 +0100, Chris Murphy   
wrote:



On Thu, Jan 12, 2017 at 4:32 PM, Adam Williamson
 wrote:

Hi folks! So I finally got around to that 'think about USB test
coverage' item that's been on my todo list forever.

I propose we add a table to the Installation Validation page. The
purpose is simply to check that writing images is working with
mediawriter in the major supported environments: Windows, macOS , and
the supported stable Fedora releases. So it could just look like this:


WindowsmacOSFedora 24Fedora 25
QA:Testcase_USB_fmw

with the intent being that we at least check that writing any one
release blocking image with mediawriter in each environment.

We could split Windows into 7, 8 and 10 or something, but not sure if
it's really necessary...

Thoughts?


Well I don't really want to eat a hat, because I've had my fill for a
lifetime, but I'd eat my hat if FMW works on Windows 10 but does not
work on Windows 7. Or vice versa. So may be ask mbriza which one to
test on, and hope the numbers game fills in the rest on its own. The
API/ABI stability on Windows is pretty extreme.  In order of market
share though, by a long shot it's Windows 7, then 10, and a distant
third is 8.1 and then 8(.0) barely registers. So weirdly enough,
chances are it'll work on 8/8.1 if it works on 7 and 10. So a thorough
test would be testing both 7 and 10. Less thorough, but sane, is
testing just on 10.

(For those who don't know, it's possible to get a free copy of Windows
Enterprise, the installer iSO will install a copy of Windows that'll
work for 90 days, the timer starts from the time of installation; i.e.
it's not the download ISO that's time limited.)

FWIW at the moment on Fedora 25 I'm running into these two bugs and
can't write images at all with FMW.
https://bugzilla.redhat.com/show_bug.cgi?id=1412063
https://bugzilla.redhat.com/show_bug.cgi?id=1412057




It does not work on Windows 7 for some folks. On my laptop with Win 7, it  
works. I've got a virtual machine where it doesn't work for testing  
though. I suspect there's a problem with ANGLE (which is a library that  
translates OpenGL calls to DirectX on Windows) combined with hardware  
drivers not supporting recent enough OpenGL. This is probably a problem  
with Qt packaging on Fedora but as of now, I don't know how to fix that.  
There's a bug [1] open for this specific issue for a while now.


I'm currently waiting for the release of Qt 5.8 where there is a more  
advanced level of deciding which rendering backend to use for QML and it  
can even fallback to a software renderer now which could be sufficient for  
FMW. However, I need to test that first.


So in the light of this, I agree with Chris that testing on Windows 7 and  
10 would probably be what we should do.


Chris, regarding the issues you mentioned:
 #1412063 should be fixed in 4.0.8 (now in updates-testing)
 #1412057 I have a fix for this using a different method in the UDisks2  
API - now FMW will seem to be "benchmarking" a drive instead of writing  
(especially in polkit dialogs). OpenForRestore doesn't restore writing AND  
reading the written data, hence I needed to use a method that allows me to  
do so. The source of your problem was using OpenForRestore and then  
OpenForBackup - in case of large images or slow flash drives, a timeout  
somewhere in polkit ran out and it requested authentication again.



[1] https://bugzilla.redhat.com/show_bug.cgi?id=1319946
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Re: Validation test proposal: mediawriter testing table

[Chris Murphy]

On Thu, Jan 12, 2017 at 4:32 PM, Adam Williamson
 wrote:
> Hi folks! So I finally got around to that 'think about USB test
> coverage' item that's been on my todo list forever.
>
> I propose we add a table to the Installation Validation page. The
> purpose is simply to check that writing images is working with
> mediawriter in the major supported environments: Windows, macOS , and
> the supported stable Fedora releases. So it could just look like this:
>
>
> WindowsmacOSFedora 24Fedora 25
> QA:Testcase_USB_fmw
>
> with the intent being that we at least check that writing any one
> release blocking image with mediawriter in each environment.
>
> We could split Windows into 7, 8 and 10 or something, but not sure if
> it's really necessary...
>
> Thoughts?

Well I don't really want to eat a hat, because I've had my fill for a
lifetime, but I'd eat my hat if FMW works on Windows 10 but does not
work on Windows 7. Or vice versa. So may be ask mbriza which one to
test on, and hope the numbers game fills in the rest on its own. The
API/ABI stability on Windows is pretty extreme.  In order of market
share though, by a long shot it's Windows 7, then 10, and a distant
third is 8.1 and then 8(.0) barely registers. So weirdly enough,
chances are it'll work on 8/8.1 if it works on 7 and 10. So a thorough
test would be testing both 7 and 10. Less thorough, but sane, is
testing just on 10.

(For those who don't know, it's possible to get a free copy of Windows
Enterprise, the installer iSO will install a copy of Windows that'll
work for 90 days, the timer starts from the time of installation; i.e.
it's not the download ISO that's time limited.)

FWIW at the moment on Fedora 25 I'm running into these two bugs and
can't write images at all with FMW.
https://bugzilla.redhat.com/show_bug.cgi?id=1412063
https://bugzilla.redhat.com/show_bug.cgi?id=1412057




-- 
Chris Murphy
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Validation test proposal: mediawriter testing table

[Adam Williamson]

Hi folks! So I finally got around to that 'think about USB test
coverage' item that's been on my todo list forever.

I propose we add a table to the Installation Validation page. The
purpose is simply to check that writing images is working with
mediawriter in the major supported environments: Windows, macOS , and
the supported stable Fedora releases. So it could just look like this:


WindowsmacOSFedora 24Fedora 25
QA:Testcase_USB_fmw

with the intent being that we at least check that writing any one
release blocking image with mediawriter in each environment.

We could split Windows into 7, 8 and 10 or something, but not sure if
it's really necessary...

Thoughts?
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
___
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org