Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Matthew N. 

On 2016-09-02 2:29 AM, Gijs Kruitbosch wrote:

On 02/09/2016 00:15, Matthew N. wrote:

On 2016-09-01 9:24 AM, Gijs Kruitbosch wrote:

On 01/09/2016 16:37, Henrik Skupin wrote:

Do those locations sound good? I have heard at least once that
"firefox_ui" might not be the best choice as folder name, but that's
how
the harness is called, and corresponds to what we have for other
harnesses too.


As I did over IRC, I would like to strongly object to the continued use
of per-test-type subfolders in our test directories. You can already use
a specific mach command per test type, and the tests are listed in
different manifests, *and* there's all the different filename
conventions (browser_, test_html, test_xul, .js) that
further point out what type of test you're looking at. The subfolders
add nothing useful.


As someone who has been adding the directory levels to
toolkit/components/passwordmgr/test/ recently, I disagree with this
since they add a grouping of relevant files making it more obvious which
files go with which test suites.


But in general you don't need to know this? When does it matter and is
it an unknown? You can just pass a filename to ./mach test and it'll do
the right thing.


* refactoring tests and related helper functions (e.g. for e10s). e.g. 
pwmgr_common.js or notification_common.js could belong to any of the 
four suites used by pwmgr. Same for a head.js file.
* when working on a test and the directory doesn't take a super long 
time to run I often want to run everything in the directory for the 
suite of the I'm working on, especially as I'm putting the final touches 
on the test. This is mostly to make sure I'm doing proper cleanup in the 
test.
* knowing which of the ini files to add a new test to (and actually 
noticing the ini files if there are lots of files in the directory).
* when writing a new test I want to look at what helpers relevant to the 
component are available for that suite



* Chrome mochitests, plain mochitests and xpcshell share the same prefix
(test_) so they are interleaved together in directory listings.


Again, not sure why that's a problem. If there's too many files in a
directory, that's a problem, but I'd rather we split them out by
function/subject than by test suite.


See above


* Files that accompany tests have no prefix convention.


Yes, but if that is a problem (which I'm not sure about) it continues to
be a problem when you don't mix test types into one directory.


If you're not mixing then you know that accompanying files are relevant 
to the tests in that directory.



* head.js files have no indication of what suite they're for (i.e. no
prefix)


But there's no need for the files to be called head.js. All of this is
also already true without adding the firefox-ui tests.


Right (for xpcshell) but that's the standard name that is commonly used, 
second is to have head_(name of thing being tested).js which would still 
conflict with another head_(name of thing being tested).js for a 
different suite in the same directory if they're testing the same 
module/component.



* `mach test` doesn't support specifying a flavor of mochitest.


I'm sure that could be added (in fact, I thought there were plans for
this already) - in the meantime, ./mach mochitest works.


I would really like this but it's still a reason to not flatten test 
directories at the moment.



* Changing the subdirectory of my `mach mochitest` command is usually
faster than adding the flavor argument since the path is usually at the
end of the command. Since `mach test` doesn't support the flavor
argument I don't have to remember whether to use the argument or change
the path as I can always change the path when directories are used.


Again, we should just fix ./mach test. But also, this isn't just about
the mach command, but also about editing, and running individual tests,
where the subdirectory "system" forces me to do useless extra typing to
run/edit "browser/browser_foo.js". It gets even worse when you write a
new test and realize halfway through that you need to switch from
mochitest-plain to mochitest-browser because mochitest-plain doesn't
have enough privileges to determine whatever you need to determine.


In the rare cases where you need to move a test between suites I don't 
see this a big deal.



Out of curiosity: if you're not running a single test, why isn't just
running all the mochitests sufficient? Why are you wanting to run a
specific suite but not the others? And is that really the majority case?


My most common experience is with password manager which uses three 
flavors of mochitest (it really should only need 2 but the last "chrome" 
test hasn't been rewritten yet). When I'm developing a new test which 
may involve refactoring shared suite helpers for the directory and I 
already know that my application code change is passing all existing 
test suites I don't want to have to wait for the unrelated mochitest 
flavors to run while developing.

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Andrew McCreight 
On Fri, Sep 2, 2016 at 12:11 PM, Ryan VanderMeulen  wrote:

> On 9/2/2016 2:56 PM, Andrew McCreight wrote:
>
>> In DOM triage, we've just set up our list to not include bugs with the
>> keyword intermittent failure. Ideally, we'd see intermittent failures that
>> have a high rate, but given that sheriffs keep an eye on oranges, they can
>> look for somebody to fix frequent ones as needed.
>>
>
> I guess that explains why trunk's OrangeFactor is pushing 30 these days.


No, that is unrelated. We have only recently started systematically
triaging DOM bugs at all, so it isn't like we stopped triaging intermittent
oranges.


>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

RE: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Wes Kocher 
I don't believe that's a workable situation. At the moment, the policy is that 
every new intermittent failure gets a bug filed for the purpose of tracking it. 
There's talk (and has been for at least a year or two, now) that work will 
begin on OrangeFactor version 2, where intermittent failures could get tracked 
within OrangeFactor's own database, and we only file bugs for the failures that 
get too noisy, but no one has had time to work on that in the last two years, 
so would need prioritization from higher ups for someone(s) to dedicate some 
time to work on it.

KW

From: e...@mozilla.com
Date: Fri, 2 Sep 2016 11:35:26 -0700
Subject: Re: Reorganization of Firefox-UI tests in mozilla-central
To: m...@hskupin.info
CC: dev-platform@lists.mozilla.org; kwie...@gmail.com

Henrik, 

It's not clear to me if this is just moving the location of existing tests, or 
if you're bringing in new tests. 

If it's the later, I'm concerned that we're going to get more intermittents 
filed in Bugzilla, which adds noise to the triage process. 

If you are adding more tests, then would it be possible to back off from filing 
intermittent bugs unless they become problematic, failing tests more than a 
quarter to half the time?

Thanks!

Emma

On Thu, Sep 1, 2016 at 8:37 AM, Henrik Skupin  wrote:
Hello,



Via bug 1272145 we want to move our existing Firefox-UI tests from

/testing/firefox-ui/ closer to the code which these are testing, so that

the former location only contains the test harness and appropriate unit

tests.



Link to the current set of tests:

https://dxr.mozilla.org/mozilla-central/source/testing/firefox-ui/tests



For those of you who haven't heard about these tests yet, they are

basically Marionette tests with an additional layer (firefox-puppeteer)

on top to ease the test creation for ui specific checks. Benefits we

have are restarting and quitting the browser, and running the tests in

any localized build of Firefox.



As listed below you can find the current type of tests and a proposed

location:



* locationbar tests:/browser/base/content/test/urlbar/firefox_ui/

* private browsing tests:

/browser/components/privatebrowsing/test/firefox_ui/

* safe browsing tests:

/browser/components/safebrowsing/content/test/firefox_ui/

* session store tests:  
/browser/components/sessionstore/test/firefox_ui/

* update tests: /toolkit/mozapps/update/tests/firefox_ui/



Do those locations sound good? I have heard at least once that

"firefox_ui" might not be the best choice as folder name, but that's how

the harness is called, and corresponds to what we have for other

harnesses too.



Locations for the following tests are not clear yet:



* l10n tests (shortcuts):   not clear yet (maybe under

/browser/base/content/test/)

* security tests:   not clear yet



L10n related tests mostly cover shortcuts and that the correct command

is invoked to find failures like bug 1173735.



Nearly all of the security tests are running checks against a real

server with various SSL certificates (DV, OV, EV) and protocol versions.

We will have a meeting soon to determine which of those tests are needed

and which ones can be removed. So we might also be able to find the

correct location for the tests.



For now I would like to know if the proposal is fine or if we should go

a completely different route.



Thanks



--

Henrik

___

dev-platform mailing list

dev-platform@lists.mozilla.org

https://lists.mozilla.org/listinfo/dev-platform


  
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Emma Humphries 
Henrik,

It's not clear to me if this is just moving the location of existing tests,
or if you're bringing in new tests.

If it's the later, I'm concerned that we're going to get more intermittents
filed in Bugzilla, which adds noise to the triage process.

If you are adding more tests, then would it be possible to back off from
filing intermittent bugs unless they become problematic, failing tests more
than a quarter to half the time?

Thanks!

Emma

On Thu, Sep 1, 2016 at 8:37 AM, Henrik Skupin  wrote:

> Hello,
>
> Via bug 1272145 we want to move our existing Firefox-UI tests from
> /testing/firefox-ui/ closer to the code which these are testing, so that
> the former location only contains the test harness and appropriate unit
> tests.
>
> Link to the current set of tests:
> https://dxr.mozilla.org/mozilla-central/source/testing/firefox-ui/tests
>
> For those of you who haven't heard about these tests yet, they are
> basically Marionette tests with an additional layer (firefox-puppeteer)
> on top to ease the test creation for ui specific checks. Benefits we
> have are restarting and quitting the browser, and running the tests in
> any localized build of Firefox.
>
> As listed below you can find the current type of tests and a proposed
> location:
>
> * locationbar tests:/browser/base/content/test/
> urlbar/firefox_ui/
> * private browsing tests:
> /browser/components/privatebrowsing/test/firefox_ui/
> * safe browsing tests:
> /browser/components/safebrowsing/content/test/firefox_ui/
> * session store tests:  /browser/components/
> sessionstore/test/firefox_ui/
> * update tests: /toolkit/mozapps/update/tests/firefox_ui/
>
> Do those locations sound good? I have heard at least once that
> "firefox_ui" might not be the best choice as folder name, but that's how
> the harness is called, and corresponds to what we have for other
> harnesses too.
>
> Locations for the following tests are not clear yet:
>
> * l10n tests (shortcuts):   not clear yet (maybe under
> /browser/base/content/test/)
> * security tests:   not clear yet
>
> L10n related tests mostly cover shortcuts and that the correct command
> is invoked to find failures like bug 1173735.
>
> Nearly all of the security tests are running checks against a real
> server with various SSL certificates (DV, OV, EV) and protocol versions.
> We will have a meeting soon to determine which of those tests are needed
> and which ones can be removed. So we might also be able to find the
> correct location for the tests.
>
> For now I would like to know if the proposal is fine or if we should go
> a completely different route.
>
> Thanks
>
> --
> Henrik
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

What if you could reinvent Firefox theming?

2016-09-02 Thread Jared Wein 
(cc'ing dev-platform, please reply to firefox-dev)

What if you could reinvent Firefox theming? What would it look like, what
would its capabilities be?

We want users to have fun customizing Firefox and make it feel like their
own. We hope to make it easier to create the type of themes that people
have always wanted to make.

Today Firefox has both "complete themes" and "themes". "Complete themes"
are harder to make but provide unlimited theming power, whereas "themes"
are easier to make but limit the theme author to just setting a background
image and some text colors. We would like to merge these into a single
system that provides the right amount of balance while also easier to use
than what we already have.

Can you help us out by filling out the following survey?

https://goo.gl/forms/qUqQ4cAJ3oJueD5c2

Thanks,
Mike de Boer and Jared Wein on behalf of the Firefox engineering team
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: All profile settings lost in new version

2016-09-02 Thread ISHIKAWA,chiaki 

On 2016/09/02 20:15, Wayne wrote:

I'm wondering, under what circumstances would a user have a prefs.js-1
and prefs.js be missing?

ref https://support.mozilla.org/en-US/questions/1136992
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform




I am curious if the affected user(s) use remotely mounted file system to 
store pref.js.


This is because I have  noticed in so many places the return value of 
close() and its wrapper functions are not checked in both C-C and even M-C.



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Gijs Kruitbosch 

On 02/09/2016 13:49, Henrik Skupin wrote:

Gijs Kruitbosch wrote on 09/02/2016 11:37 AM:

We should be doing the same for firefox-ui tests. Not just to avoid
duplication of files in archives, but because otherwise, if we want to
add new tests somewhere else, we both have to add the manifests to the
build system and then modify this build system python file to make sure
they get included in the test archive. That would be wrong.


No, this is not what we want to do. Anything related to the obj dir
causes a lot of operations and cpu cycles to build those archives (see
bug 1283919 comment 1).


This comment has no information at all about operations or cpu cycles, 
so I don't follow.



With having the manifests in place we can also
run the tests directly from the source tree.


I don't understand what this sentence actually means. We have manifests 
for all our test suites - it doesn't seem to have any direct bearing on 
whether they can be run from the source tree or not (mochitests have 
manifests and can't be run from the source tree today, AIUI). What point 
are you trying to make?



It's true that the
situation with a master manifest is not ideal right now, and especially
for marionette we have it located in
testing/marionette/harness/marionette/test/. Anyway for firefox-ui tests
I want to have testing/firefox-ui/tests/(functional|update)-tests.ini.

From here we can reference everything.


So what's the proposal? This is no longer clear to me. You want to keep 
a single manifest somewhere under testing/, and then have the actual 
test files spread around the tree?


It would seem to make more sense to have individual manifests for the 
different bits. That's certainly more scalable and maintainable (ie have 
it be obvious why and when which tests run, and not have a motley 
collection in a single manifest file) and helps preserve infra resources 
as well, because we make decisions about what builds/tests to run based 
on the paths of files modified. It's not sensible that whenever I would 
want to write a patch and accompanying test, I have to modify files in 
testing/marionette/harness or testing/firefox-ui/tests/ .


It's not clear to me why marionette and firefox-ui don't use the same 
moz.build infrastructure to collect test manifests and thereby test 
files as mochitest/xpcshell/reftests, and why we couldn't use exactly 
that infrastructure to determine what to stick in the test archives as 
far as the tests themselves are concerned. The mixing of information 
storage here (in both directory naming/structure and manifest files) 
only leads to duplication and confusion/bugs.


~ Gijs
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Henrik Skupin 
Gijs Kruitbosch wrote on 09/02/2016 01:22 PM:
> AIUI, if we can run tests directly from a checkout, we no longer need 
> the common.tests.zip archive to exist (or at least not to have the tests 
> in them), so the problem is moot, no?

Those files are used by test machines for all of our test suites. That
is so they do not have to clone the full tree. I'm not aware of any
planned changes here. But build people might know more.

-- 
Henrik
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Henrik Skupin 
Gijs Kruitbosch wrote on 09/02/2016 01:22 PM:

> AIUI, if we can run tests directly from a checkout, we no longer need 
> the common.tests.zip archive to exist (or at least not to have the tests 
> in them), so the problem is moot, no?

I think so. There should not be a need to have to run "mach build
package-tests" before running any test from the source tree. mach itself
takes care about copying necessary bits to the obj dir if necessary.

-- 
Henrik
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Henrik Skupin 
Gijs Kruitbosch wrote on 09/02/2016 11:37 AM:
> I am not familiar with this bit of our build architecture, but as far as 
> I can tell from a quick look it builds bits of the zipfile off the 
> objdir. So it collects mochitests from $OBJDIR/_tests/testing/mochitest, 
> where (again, AFAICT) things get installed via manifests.

This is only happening for old test suites but not for more recent ones.
But also for reftests we do no longer have to copy all the files to the
obj dir first. We package them directly from the source tree into the
various *.tests.zip files. But this is all related to directories and
not individual files.

> We should be doing the same for firefox-ui tests. Not just to avoid 
> duplication of files in archives, but because otherwise, if we want to 
> add new tests somewhere else, we both have to add the manifests to the 
> build system and then modify this build system python file to make sure 
> they get included in the test archive. That would be wrong.

No, this is not what we want to do. Anything related to the obj dir
causes a lot of operations and cpu cycles to build those archives (see
bug 1283919 comment 1). With having the manifests in place we can also
run the tests directly from the source tree. It's true that the
situation with a master manifest is not ideal right now, and especially
for marionette we have it located in
testing/marionette/harness/marionette/test/. Anyway for firefox-ui tests
I want to have testing/firefox-ui/tests/(functional|update)-tests.ini.
>From here we can reference everything.

-- 
Henrik
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Gijs Kruitbosch 

On 02/09/2016 10:54, James Graham wrote:

On 02/09/16 10:37, Gijs Kruitbosch wrote:

On 02/09/2016 08:08, Henrik Skupin wrote:



The problematic piece here will be the package-tests step which
currently picks complete subfolders. It would mean if we mix-up tests
for firefox-ui-tests and eg. mochitests all would end-up twice in the
common.tests.zip archive. If we want to get away from using subfolders
we would have to improve the test archiver
(https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/action/test_archive.py)


first to not only collect the directories of referenced manifests, but
also only pick those tests which are referenced and leave all others
behind. This would apply to all test suites currently covered by this
mozbuild action.


I am not familiar with this bit of our build architecture, but as far as
I can tell from a quick look it builds bits of the zipfile off the
objdir. So it collects mochitests from $OBJDIR/_tests/testing/mochitest,
where (again, AFAICT) things get installed via manifests.

We should be doing the same for firefox-ui tests. Not just to avoid
duplication of files in archives, but because otherwise, if we want to
add new tests somewhere else, we both have to add the manifests to the
build system and then modify this build system python file to make sure
they get included in the test archive. That would be wrong.


In the medium term we are trying to move away from requiring the
package-tests step in favour of being able to run tests directly from a
checkout. Therefore I suggest we avoid adding unnecessary dependencies
on the objdir.


AIUI, if we can run tests directly from a checkout, we no longer need 
the common.tests.zip archive to exist (or at least not to have the tests 
in them), so the problem is moot, no?


~ Gijs
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

All profile settings lost in new version

2016-09-02 Thread Wayne 
I'm wondering, under what circumstances would a user have a prefs.js-1 
and prefs.js be missing?


ref https://support.mozilla.org/en-US/questions/1136992
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Build System Project - Latest Update

2016-09-02 Thread David Burns 
Below is a highlight of all work the build peers have done since the last
report[1].

The build peers have been working to get faster builds in automation as
well as for local developers. We have landed changes to stop generating
XPIDL sources in artifact builds[2], which is a performance and correctness
improvement.

We have also been working hard at removing the legacy m4+shell autoconf
mess. We are now below 70% of the size when we started this project. Since
the last update we have removed over 3000 lines[3].

Build tasks in taskcluster are now using c4.4xlarge instances instead of
c3.2xlarge[4] - builds are faster by around 20%[5].

As part of the work to allow us to use an alternate build backend we have
managed to get Firefox building NSS with GYP on Linux[6]. Other platforms
are to follow before we land this change. We have also been reviewing
changes to build XPIDL sources with TUP[7].

Support for building Rust sources via Cargo has been landed in
mozilla-central[8]. For more information read the Dev.Platform post[9]

Last but not least, our intern project to use msys2 has reached its goal.
The new msys2 environment now builds on mozilla-central. We will be sending
out a separate email about this for those who would like to try it out.

[1]
https://groups.google.com/d/msg/mozilla.dev.platform/i0rRsYgnXDM/O5DxU-PqBQAJ

[2] https://bugzilla.mozilla.org/show_bug.cgi?id=1240134

[3] https://bugzilla.mozilla.org/show_bug.cgi?id=1292463

[4] https://bugzilla.mozilla.org/show_bug.cgi?id=1290282

[5] https://mzl.la/2c5bjSR

[6] https://bugzilla.mozilla.org/show_bug.cgi?id=1237872

[7] https://bugzilla.mozilla.org/show_bug.cgi?id=1293448

[8] https://bugzilla.mozilla.org/show_bug.cgi?id=1231764

[9]
https://groups.google.com/d/msg/mozilla.dev.platform/ZgkBLZb6dtc/oWBE_nUxAQAJ
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread James Graham 

On 02/09/16 10:37, Gijs Kruitbosch wrote:

On 02/09/2016 08:08, Henrik Skupin wrote:



The problematic piece here will be the package-tests step which
currently picks complete subfolders. It would mean if we mix-up tests
for firefox-ui-tests and eg. mochitests all would end-up twice in the
common.tests.zip archive. If we want to get away from using subfolders
we would have to improve the test archiver
(https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/action/test_archive.py)

first to not only collect the directories of referenced manifests, but
also only pick those tests which are referenced and leave all others
behind. This would apply to all test suites currently covered by this
mozbuild action.


I am not familiar with this bit of our build architecture, but as far as
I can tell from a quick look it builds bits of the zipfile off the
objdir. So it collects mochitests from $OBJDIR/_tests/testing/mochitest,
where (again, AFAICT) things get installed via manifests.

We should be doing the same for firefox-ui tests. Not just to avoid
duplication of files in archives, but because otherwise, if we want to
add new tests somewhere else, we both have to add the manifests to the
build system and then modify this build system python file to make sure
they get included in the test archive. That would be wrong.


In the medium term we are trying to move away from requiring the 
package-tests step in favour of being able to run tests directly from a 
checkout. Therefore I suggest we avoid adding unnecessary dependencies 
on the objdir.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Gijs Kruitbosch 

On 02/09/2016 08:08, Henrik Skupin wrote:

Gijs Kruitbosch wrote on 09/01/2016 06:24 PM:


As I did over IRC, I would like to strongly object to the continued use
of per-test-type subfolders in our test directories. You can already use
a specific mach command per test type, and the tests are listed in
different manifests, *and* there's all the different filename
conventions (browser_, test_html, test_xul, .js) that
further point out what type of test you're looking at. The subfolders
add nothing useful.


The problematic piece here will be the package-tests step which
currently picks complete subfolders. It would mean if we mix-up tests
for firefox-ui-tests and eg. mochitests all would end-up twice in the
common.tests.zip archive. If we want to get away from using subfolders
we would have to improve the test archiver
(https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/action/test_archive.py)
first to not only collect the directories of referenced manifests, but
also only pick those tests which are referenced and leave all others
behind. This would apply to all test suites currently covered by this
mozbuild action.


I am not familiar with this bit of our build architecture, but as far as 
I can tell from a quick look it builds bits of the zipfile off the 
objdir. So it collects mochitests from $OBJDIR/_tests/testing/mochitest, 
where (again, AFAICT) things get installed via manifests.


We should be doing the same for firefox-ui tests. Not just to avoid 
duplication of files in archives, but because otherwise, if we want to 
add new tests somewhere else, we both have to add the manifests to the 
build system and then modify this build system python file to make sure 
they get included in the test archive. That would be wrong.


In case I'm wrong (always possible!): As I noted in my other email, 
there are other cases where we mix tests, so if the archiver is really 
that dumb then maybe we should fix the archiver. :-)


~ Gijs
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Gijs Kruitbosch 

On 02/09/2016 00:15, Matthew N. wrote:

On 2016-09-01 9:24 AM, Gijs Kruitbosch wrote:

On 01/09/2016 16:37, Henrik Skupin wrote:

Do those locations sound good? I have heard at least once that
"firefox_ui" might not be the best choice as folder name, but that's how
the harness is called, and corresponds to what we have for other
harnesses too.


As I did over IRC, I would like to strongly object to the continued use
of per-test-type subfolders in our test directories. You can already use
a specific mach command per test type, and the tests are listed in
different manifests, *and* there's all the different filename
conventions (browser_, test_html, test_xul, .js) that
further point out what type of test you're looking at. The subfolders
add nothing useful.


As someone who has been adding the directory levels to
toolkit/components/passwordmgr/test/ recently, I disagree with this
since they add a grouping of relevant files making it more obvious which
files go with which test suites.


But in general you don't need to know this? When does it matter and is 
it an unknown? You can just pass a filename to ./mach test and it'll do 
the right thing.



* Chrome mochitests, plain mochitests and xpcshell share the same prefix
(test_) so they are interleaved together in directory listings.


Again, not sure why that's a problem. If there's too many files in a 
directory, that's a problem, but I'd rather we split them out by 
function/subject than by test suite.



* Files that accompany tests have no prefix convention.


Yes, but if that is a problem (which I'm not sure about) it continues to 
be a problem when you don't mix test types into one directory.



* head.js files have no indication of what suite they're for (i.e. no
prefix)


But there's no need for the files to be called head.js. All of this is 
also already true without adding the firefox-ui tests.



* `mach test` doesn't support specifying a flavor of mochitest.


I'm sure that could be added (in fact, I thought there were plans for 
this already) - in the meantime, ./mach mochitest works.



* Changing the subdirectory of my `mach mochitest` command is usually
faster than adding the flavor argument since the path is usually at the
end of the command. Since `mach test` doesn't support the flavor
argument I don't have to remember whether to use the argument or change
the path as I can always change the path when directories are used.


Again, we should just fix ./mach test. But also, this isn't just about 
the mach command, but also about editing, and running individual tests, 
where the subdirectory "system" forces me to do useless extra typing to 
run/edit "browser/browser_foo.js". It gets even worse when you write a 
new test and realize halfway through that you need to switch from 
mochitest-plain to mochitest-browser because mochitest-plain doesn't 
have enough privileges to determine whatever you need to determine.


Out of curiosity: if you're not running a single test, why isn't just 
running all the mochitests sufficient? Why are you wanting to run a 
specific suite but not the others? And is that really the majority case?


Personally, I would prefer to have per-subject directories, and to have 
"mach test" and friends allow specifying particular suites if we're 
hunting down other-test-dependent intermittents in a particular suite, 
or something. The most common situation right now is that you want to 
run either a single test or want to ensure that the code changes you 
just made didn't break any of the relevant tests. In the latter case 
"relevant tests" is related to *what the tests are testing*, not to what 
framework the tests use. So ultimately, the grouping by test framework 
makes it hard to run only the tests you care about. I know that some 
test frameworks (last I checked, not including marionette or fx-ui) 
support running tests by "tags", but most tests aren't annotated with 
them, and in any case I feel like it ends up being a hack for the fact 
that our directory layout is not well-suited for that usecase.



* xpcshell and browser-chrome both use "head.js" as the filename for
helpers though they run in very different contexts.


xpcshell lets you use whatever you like, I think, with the head/tail 
annotations in the .ini file, for cases where this would actually be 
confusing.



For a new contributor, having each suite in their own directory is much
less confusing/overwhelming for the above reasons.


I will buy that it is a more obvious signal (than filename and content) 
if you don't know much about our testing infrastructure and need to know 
what kind of test you're looking at while you're looking at the 
directory view. But again, that doesn't feel like it's the most common 
case. Wanting to write a new test (where you'd need advice anyway, 
especially given that our naming (chrome/browser-chrome) is so 
confusing) or editing a specific existing test (e.g. one that fails on 
try) don't actually fall in this

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Henrik Skupin 
Matthew N. wrote on 09/02/2016 01:15 AM:

>>> /browser/components/sessionstore/test/firefox_ui/
>>> * update tests:/toolkit/mozapps/update/tests/firefox_ui/
> 
> Is there a plan to merge those with 
> /toolkit/mozapps/update/tests/marionette? It seems unusual to use both 
> when this new test suite is basically a layer on top of Marionette as 
> you said.

Those marionette tests have been created specifically for B2G:
https://dxr.mozilla.org/mozilla-central/source/testing/marionette/harness/marionette/tests/update-tests.ini

But the firefox-ui harness can only be used for Firefox. Reason is that
we have our own testcase classes and several enhancements to the
harness. An integration of those into Marionette base was denied to not
make Marionette that bloated. As result we have to keep our layering for
now.

> * Chrome mochitests, plain mochitests and xpcshell share the same prefix 
> (test_) so they are interleaved together in directory listings.

And exactly this makes it hard to only collect the relevant files
(tests, support files) via test_archive.py as mentioned in my other email.

> "firefox_ui", "ui", and "integration" all overlap with what 
> mochitest-browser-chrome is about IMO and I think naming the suite 
> "Firefox-UI" was confusing from the beginning. If I was a new 
> contributor working on a UI feature and decided I wanted to write tests, 
> I wouldn't want to be misled into thinking I should write a "Firefox-UI" 
> test when mochitest-browser-chrome is actually the desired suite. I 
> would suggest "puppeteer" or "marionette" for directory names to avoid 
> confusion.

As of now we only have puppeteer available for Firefox. And it will not
be a hard requirement to use hopefully soon once we decoupled the
harness from it. Also we just got Marionette support for Fennec. It
means beside testing/puppeteer/firefox it might be good to get shared
helper code for Fennec specific tests into testing/puppeteer/fennec.

Using marionette here would also be confusing because those tests won't
work with the marionette command.

Maybe at the end we should try again to get our testcase classes and
harness extensions into Marionette again. But depending on the future
that we may want to use Selenium for our tests directly, I would wait
and not spend my time with such a refactoring.

-- 
Henrik
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Henrik Skupin 
Gijs Kruitbosch wrote on 09/01/2016 06:24 PM:

> As I did over IRC, I would like to strongly object to the continued use 
> of per-test-type subfolders in our test directories. You can already use 
> a specific mach command per test type, and the tests are listed in 
> different manifests, *and* there's all the different filename 
> conventions (browser_, test_html, test_xul, .js) that 
> further point out what type of test you're looking at. The subfolders 
> add nothing useful.

The problematic piece here will be the package-tests step which
currently picks complete subfolders. It would mean if we mix-up tests
for firefox-ui-tests and eg. mochitests all would end-up twice in the
common.tests.zip archive. If we want to get away from using subfolders
we would have to improve the test archiver
(https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/action/test_archive.py)
first to not only collect the directories of referenced manifests, but
also only pick those tests which are referenced and leave all others
behind. This would apply to all test suites currently covered by this
mozbuild action.

> Finally, "firefox_ui" (as well as "ui") as a name for a directory is 
> going to cause all kinds of confusion for people exploring the repo 
> without detailed knowledge of what's going on. Additionally, it's not 
> like most of the mochitest-browser tests aren't tests of the Firefox 
> UI... If we absolutely must have some kind of subdirectory because of 
> reasons I have yet to hear, I think "integration" would be a better 
> choice of name as far as subdirs of "test" go (as juxtaposed to "unit" 
> for our xpcshell tests).

I would totally be fine with integration as name, unless we don't hit
the above mentioned problem that tests for other harnesses end-up in the
same folder.

-- 
Henrik
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

NS_WARN_IF_FALSE has been renamed NS_WARNING_ASSERTION

2016-09-02 Thread Nicholas Nethercote 
Greetings,

NS_WARN_IF_FALSE has been renamed NS_WARNING_ASSERTION.

See https://bugzilla.mozilla.org/show_bug.cgi?id=1299727 for details,
including the justification.

See https://bugzilla.mozilla.org/show_bug.cgi?id=137 for a bug
that this change exposed, thus adding weight to the justification.

See https://bugzilla.mozilla.org/show_bug.cgi?id=132 for a
comm-central patch.

Nick
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform