Re: disabled non-e10s tests on trunk

2017-09-11 Thread Randell Jesup
>you could edit the taskcluster/ci/test/tests.yml file and add
>|linux64/debug: both| in the places we have |windows7-32/debug: both|, as
>seen here:
>http://searchfox.org/mozilla-central/source/taskcluster/ci/test/tests.yml#138
>
>If there are concerns with windows build times, please ask for those to be
>faster- it will make it better for all of us :)

Where's my magic wand...? if it was easy to make them faster I presume
that would have been done long ago.

I frequently do (only) linux Trys unless I suspect platform-specific
issues.  Saves resources...  Can't "mochitest-media" (and other non-e10s
versions) be not-run by default, unless you specify it explicitly in -u ?

-- 
Randell Jesup, Mozilla Corp
remove "news" for personal email
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-09-11 Thread Joel Maher
you could edit the taskcluster/ci/test/tests.yml file and add
|linux64/debug: both| in the places we have |windows7-32/debug: both|, as
seen here:
http://searchfox.org/mozilla-central/source/taskcluster/ci/test/tests.yml#138

If there are concerns with windows build times, please ask for those to be
faster- it will make it better for all of us :)



On Fri, Sep 8, 2017 at 5:40 PM, Ben Kelly  wrote:

> Joel,
>
> Is there an easy way for me to run non-e10s tests on linux?  We often use
> "t-style" try pushes where we only run tests on one platform.  Restricting
> non-e10s to win7-debug means I either need to run tests on multiple
> platforms or use windows for the "t-style".  I don't want to use windows
> because it builds/runs much slower than linux.
>
> Thanks.
>
> Ben
>
> On Fri, Aug 18, 2017 at 4:15 PM,  wrote:
>
>> Yesterday I landed bug 1391371 which enabled non-e10s unittests on
>> windows 7 debug.  A few tests had to be disabled in order to do this.  To
>> keep track of what we did and each of the test suites to evaluate, I have
>> filed bug 1391350.
>>
>> As a note we now have the following non-e10s tests running:
>> windows7-debug: all trunk branches
>> android opt/debug: all trunk branches (existing media, plain, reftest)
>> linux64-jsdcov: mozilla-central (mochitest-plain/browser-chrome/devtools)
>> ** this is a linux64 opt build and we use the jsdebugger to collect code
>> coverage metrics- but we specifically run this in non-e10s mode.
>>
>> Please let me know if there are large areas that we have overlooked.
>>
>> Thanks!
>> ___
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>>
>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-09-08 Thread Ben Kelly
Joel,

Is there an easy way for me to run non-e10s tests on linux?  We often use
"t-style" try pushes where we only run tests on one platform.  Restricting
non-e10s to win7-debug means I either need to run tests on multiple
platforms or use windows for the "t-style".  I don't want to use windows
because it builds/runs much slower than linux.

Thanks.

Ben

On Fri, Aug 18, 2017 at 4:15 PM,  wrote:

> Yesterday I landed bug 1391371 which enabled non-e10s unittests on windows
> 7 debug.  A few tests had to be disabled in order to do this.  To keep
> track of what we did and each of the test suites to evaluate, I have filed
> bug 1391350.
>
> As a note we now have the following non-e10s tests running:
> windows7-debug: all trunk branches
> android opt/debug: all trunk branches (existing media, plain, reftest)
> linux64-jsdcov: mozilla-central (mochitest-plain/browser-chrome/devtools)
> ** this is a linux64 opt build and we use the jsdebugger to collect code
> coverage metrics- but we specifically run this in non-e10s mode.
>
> Please let me know if there are large areas that we have overlooked.
>
> Thanks!
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-18 Thread jmaher
Yesterday I landed bug 1391371 which enabled non-e10s unittests on windows 7 
debug.  A few tests had to be disabled in order to do this.  To keep track of 
what we did and each of the test suites to evaluate, I have filed bug 1391350.

As a note we now have the following non-e10s tests running:
windows7-debug: all trunk branches
android opt/debug: all trunk branches (existing media, plain, reftest)
linux64-jsdcov: mozilla-central (mochitest-plain/browser-chrome/devtools)
** this is a linux64 opt build and we use the jsdebugger to collect code 
coverage metrics- but we specifically run this in non-e10s mode.

Please let me know if there are large areas that we have overlooked.

Thanks!
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-16 Thread jmaher
On Wednesday, August 16, 2017 at 4:03:20 PM UTC-4, Nils Ohlmeier wrote:
> > On Aug 16, 2017, at 07:23, James Graham  wrote:
> > 
> > On 16/08/17 01:26, Nils Ohlmeier wrote:
> >> I guess not a lot of people are aware of it, but for WebRTC we still have 
> >> two distinct implementations for the networking code.
> >> So if I understand the impact here right we just lost test coverage for 
> >> probably a couple of thousand lines of code.
> > […]
> > 
> >> I’m not sure how others do it, but our low level C++ unit tests don’t have 
> >> an e10s mode at all.
> >> Therefore we can’t simply delete the non-e10s WebRTC networking code 
> >> either (without loosing a ton of test coverage).
> > 
> > If the networking code is only covered by C++ unit tests, there is separate 
> > code for non-e10s vs e10s,  and the unit tests don't work in e10s mode 
> > doesn't that mean we currently don't have any test coverage for our 
> > shipping configuration on desktop? What am I missing?
> 
> So we have mochitest-media which works as kind of integration test on a 
> higher level. They execute high level JS API tests, but also try to ensure 
> that the lower level networking pieces (the once which are exposed through 
> JS) match the expectations.
> The mochitest-media got executed for e10s and non-e10s and therefore covered 
> both implementations.
> 
> And then we have C++ unit tests, which cover a lot more corner cases of 
> different scenarios for networking. And yes these only work with non-e10s 
> right now. It would be a lot of work to create the same amount of tests with 
> a higher level test suite like mochitest to get the e10s coverage. Plus these 
> tests would probably take a lot execution time.
> 
> Technically that leaves us with a somewhat blind spot for the coverage of 
> networking corner cases under e10s. I guess if there is a high demand for 
> turning off all non-e10s tests we need to look at how to get our C++ unit 
> tests working with something like e10s.
> But until we can get to that I think we really should keep running 
> mochitest-media with e10s and without it.
> 
> Best
>   Nils Ohlmeier

As a note, C++ tests will continue to run in non-e10s mode (cppunit, gtest)- 
there are no plans to run these as e10s; this is mostly referring to: 
mochitest*, web-platform-test*, reftest*, marionette*
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-16 Thread Nils Ohlmeier

> On Aug 16, 2017, at 07:23, James Graham  wrote:
> 
> On 16/08/17 01:26, Nils Ohlmeier wrote:
>> I guess not a lot of people are aware of it, but for WebRTC we still have 
>> two distinct implementations for the networking code.
>> So if I understand the impact here right we just lost test coverage for 
>> probably a couple of thousand lines of code.
> […]
> 
>> I’m not sure how others do it, but our low level C++ unit tests don’t have 
>> an e10s mode at all.
>> Therefore we can’t simply delete the non-e10s WebRTC networking code either 
>> (without loosing a ton of test coverage).
> 
> If the networking code is only covered by C++ unit tests, there is separate 
> code for non-e10s vs e10s,  and the unit tests don't work in e10s mode 
> doesn't that mean we currently don't have any test coverage for our shipping 
> configuration on desktop? What am I missing?

So we have mochitest-media which works as kind of integration test on a higher 
level. They execute high level JS API tests, but also try to ensure that the 
lower level networking pieces (the once which are exposed through JS) match the 
expectations.
The mochitest-media got executed for e10s and non-e10s and therefore covered 
both implementations.

And then we have C++ unit tests, which cover a lot more corner cases of 
different scenarios for networking. And yes these only work with non-e10s right 
now. It would be a lot of work to create the same amount of tests with a higher 
level test suite like mochitest to get the e10s coverage. Plus these tests 
would probably take a lot execution time.

Technically that leaves us with a somewhat blind spot for the coverage of 
networking corner cases under e10s. I guess if there is a high demand for 
turning off all non-e10s tests we need to look at how to get our C++ unit tests 
working with something like e10s.
But until we can get to that I think we really should keep running 
mochitest-media with e10s and without it.

Best
  Nils Ohlmeier




signature.asc
Description: Message signed with OpenPGP
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-16 Thread James Graham

On 16/08/17 19:36, Ben Kelly wrote:

My only thought about windows7-debug is that android is a variant of
linux.  Running a linux platform might be closer to android behavior.  But
I don't have a known specific difference in mind.


Right it seems like there are two use cases here:

1) Tests that are really aimed at Desktop or are cross-product but don't 
run on e10s for (reasons).


2) Tests for features that are run in e10s on Desktop, but exercise 
functional differences in non-e10s and don't run in Android.


For 1) running on Windows makes some sense because that's where most 
users are. For 2) it makes no sense because it's the most different from 
Android. For those cases running on Linux(64) makes more sense (and is 
also usually where we have most capacity, so that helps with infra issues).

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-16 Thread Ben Kelly
On Wed, Aug 16, 2017 at 2:32 PM, Joel Maher  wrote:

> Thanks everyone for chiming in here.  I see this isn't as simple as a
> binary decision and to simplify things, I think turning on all non-e10s
> tests that were running for windows7-debug would give us reasonable
> coverage and ensure that users on our most popular OS (and focus for 57)
> have a stable experience if they choose to go without e10s.
>
> Instead of a date to turn things off, I would like to just focus on each
> framework one at a time and ideally in a few months we would have a clear
> set of requirements (in the form of bugs) to resolve before turning off the
> specific non-e10s tests.
>
> If there is a different configuration other than windows7-debug I would
> like to hear about it.
>

Thank you Joel.

My only thought about windows7-debug is that android is a variant of
linux.  Running a linux platform might be closer to android behavior.  But
I don't have a known specific difference in mind.

It would also be nice to opt and debug since things do differ between them,
but I'll take what I can get.

Thanks.

Ben
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-16 Thread Joel Maher
Thanks everyone for chiming in here.  I see this isn't as simple as a
binary decision and to simplify things, I think turning on all non-e10s
tests that were running for windows7-debug would give us reasonable
coverage and ensure that users on our most popular OS (and focus for 57)
have a stable experience if they choose to go without e10s.

Instead of a date to turn things off, I would like to just focus on each
framework one at a time and ideally in a few months we would have a clear
set of requirements (in the form of bugs) to resolve before turning off the
specific non-e10s tests.

If there is a different configuration other than windows7-debug I would
like to hear about it.

Joel


On Wed, Aug 16, 2017 at 12:53 PM, Ehsan Akhgari 
wrote:

> On 08/15/2017 04:29 PM, jma...@mozilla.com wrote:
>
>> I would propose running these above tests on windows7-opt (we don't have
>> these running yet in windows 10, although we are close), and only running
>> specific tests which are not run in e10s mode, turning them off December
>> 29th, 2017.
>>
>> Keep in mind we have had many years to get all our tests running in e10s
>> mode and we have known since last year that Firefox 57 would be the first
>> release that we ship e10s by default to our users- my proposal is a 4 month
>> temporary measure to let test owners finish ensuring their tests are
>> running in e10s mode.
>>
> Hi Joel,
>
> I think Ben is right on here.  Please note that it's not just service
> workers that have a different implementation in e10s vs. non-e10s, many
> other super important parts of the browser are in the same boat.  For
> example, our entire HTTP(S) stack has different code paths in e10s vs.
> non-e10s.  Furthermore, even in places where we share a lot of code in the
> two modes, there are often subtle differences in the behavior in between
> the two modes.  I hope this is convincing in terms of the necessity of
> maintaining the test coverage across the two modes.
>
> To slightly readjust what you stated above from the perspective of the
> Firefox developers, we have known since last years that Firefox 57 would be
> the first release that we ship e10s by default for our *desktop* users, and
> we have also known that Firefox 57 for Android will be shipped without e10s
> for our *mobile* users, so it shouldn't be any surprise why no large scale
> effort has been put into porting all of our tests to run into e10s mode.
> Also please note that some of the test frameworks you have listed above
> like browser-chrome aren't relevant to Android, and some like jsreftests
> aren't relevant to e10s, so we should probably have a more detailed
> conversation about the remaining ones.
>
> I think a better way to think about this problem is perhaps how to get to
> a point where we never end up losing test coverage on code that we ship on
> our tier 1 platforms.  Thinking about this in terms of date-based deadlines
> where we don't have the human power to do the work necessary isn't
> particularly helpful, and would ultimately result us in losing invaluable
> test coverage.  If past history is any lesson for us, regressions will
> creep in as a result, and especially due to the fact that this is mostly
> affecting Firefox for Android where we have the lowest pre-release testing
> population among our tier1 products (AFAIK) makes that extremely risky.
>
> Therefore, I think we should:
>
>  * start running all of the non-e10s tests that can affect Android again
> immediately.
>  * work out a realistic plan between engineering and the automation team
> to address the existing gaps that prevent us from turning off non-e10s
> tests without losing coverage on Android, and execute that plan.
>  * turn non-e10s tests off when the above has been done.
>
> Please let me know what you think!
>
> Thanks,
> Ehsan
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-16 Thread Ehsan Akhgari

On 08/15/2017 04:29 PM, jma...@mozilla.com wrote:

I would propose running these above tests on windows7-opt (we don't have these 
running yet in windows 10, although we are close), and only running specific 
tests which are not run in e10s mode, turning them off December 29th, 2017.

Keep in mind we have had many years to get all our tests running in e10s mode 
and we have known since last year that Firefox 57 would be the first release 
that we ship e10s by default to our users- my proposal is a 4 month temporary 
measure to let test owners finish ensuring their tests are running in e10s mode.

Hi Joel,

I think Ben is right on here.  Please note that it's not just service 
workers that have a different implementation in e10s vs. non-e10s, many 
other super important parts of the browser are in the same boat.  For 
example, our entire HTTP(S) stack has different code paths in e10s vs. 
non-e10s.  Furthermore, even in places where we share a lot of code in 
the two modes, there are often subtle differences in the behavior in 
between the two modes.  I hope this is convincing in terms of the 
necessity of maintaining the test coverage across the two modes.


To slightly readjust what you stated above from the perspective of the 
Firefox developers, we have known since last years that Firefox 57 would 
be the first release that we ship e10s by default for our *desktop* 
users, and we have also known that Firefox 57 for Android will be 
shipped without e10s for our *mobile* users, so it shouldn't be any 
surprise why no large scale effort has been put into porting all of our 
tests to run into e10s mode.  Also please note that some of the test 
frameworks you have listed above like browser-chrome aren't relevant to 
Android, and some like jsreftests aren't relevant to e10s, so we should 
probably have a more detailed conversation about the remaining ones.


I think a better way to think about this problem is perhaps how to get 
to a point where we never end up losing test coverage on code that we 
ship on our tier 1 platforms.  Thinking about this in terms of 
date-based deadlines where we don't have the human power to do the work 
necessary isn't particularly helpful, and would ultimately result us in 
losing invaluable test coverage.  If past history is any lesson for us, 
regressions will creep in as a result, and especially due to the fact 
that this is mostly affecting Firefox for Android where we have the 
lowest pre-release testing population among our tier1 products (AFAIK) 
makes that extremely risky.


Therefore, I think we should:

 * start running all of the non-e10s tests that can affect Android 
again immediately.
 * work out a realistic plan between engineering and the automation 
team to address the existing gaps that prevent us from turning off 
non-e10s tests without losing coverage on Android, and execute that plan.

 * turn non-e10s tests off when the above has been done.

Please let me know what you think!

Thanks,
Ehsan
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-16 Thread James Graham

On 15/08/17 21:39, Ben Kelly wrote:

On Tue, Aug 15, 2017 at 4:37 PM, Joel Maher  wrote:


All of the above mentioned tests are not run on Android (well
mochitest-media is to some degree).  Is 4 months unreasonable to fix the
related tests that do not run in e10s?  Is there another time frame that
seems more reasonable?



Last I checked it was your team that told me WPT on android was not an
immediate priority.  The WPT harness itself does not run there.


FWIW the story with wpt/Android is that originally Android didn't 
support the kind of remote control required to run wpt tests (i.e. 
marionette). That has subsequently been fixed, and it's believed to be 
possible incorporate Android into the wpt harness without a significant 
refactoring, but after that there is substantial, time consuming, work 
required to get from "it runs" to "this is a thing we can run in 
production".


I doubt that the work required to implement an Android backend, sort out 
issues with the relative slowness of the android emulator, and green up 
the tests, would be less than one person's work for a quarter. Even once 
this is done there would likely be ongoing problems updating the 
metadata on android for syncs from upstream simply from the additional 
slowness of try runs and probable additional intermittency.


So far there haven't been enough people working on wpt to make this a 
priority relative to other goals. Given the situation with e10s, it 
seems reasonable to reassess this when planning future work, but I 
wouldn't bet on such work being complete by Dec. 29th this year.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-16 Thread James Graham

On 16/08/17 01:26, Nils Ohlmeier wrote:

I guess not a lot of people are aware of it, but for WebRTC we still have two 
distinct implementations for the networking code.
So if I understand the impact here right we just lost test coverage for 
probably a couple of thousand lines of code.

[…]


I’m not sure how others do it, but our low level C++ unit tests don’t have an 
e10s mode at all.
Therefore we can’t simply delete the non-e10s WebRTC networking code either 
(without loosing a ton of test coverage).


If the networking code is only covered by C++ unit tests, there is 
separate code for non-e10s vs e10s,  and the unit tests don't work in 
e10s mode doesn't that mean we currently don't have any test coverage 
for our shipping configuration on desktop? What am I missing?

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-15 Thread Nils Ohlmeier
I guess not a lot of people are aware of it, but for WebRTC we still have two 
distinct implementations for the networking code.
So if I understand the impact here right we just lost test coverage for 
probably a couple of thousand lines of code.

And yes you can potentially blame it on WebRTC networking for not having 
unified it’s networking code for e10s and non-e10s.
But when e10s got turned on I asked around if we could delete the non-e10s code 
soon, I was told no.
So I assumed both technologies would stick around for some more time.

I’m not sure how others do it, but our low level C++ unit tests don’t have an 
e10s mode at all.
Therefore we can’t simply delete the non-e10s WebRTC networking code either 
(without loosing a ton of test coverage).

With this being said I think the right thing here would be to turn the non-e10s 
mochitests back on for WebRTC until we have a better solution.

  Nils

> On Aug 15, 2017, at 13:54, J. Ryan Stinnett  wrote:
> 
> I think Ben's argument has merit:
> 
> 1. Even after Firefox 57, we will still be shipping a product in non-e10s
> mode: Firefox for Android
> 2. If WPT (and potentially other suites) aren't being run in non-e10s mode,
> it increases risk because we are shipping untested code paths to our users
> 
> Someone might add code that assumes e10s is the only mode and land it
> successfully, only later to hear that it fails or crashes on Android, since
> e10s doesn't exist there today.
> 
> - Ryan
> 
> On Tue, Aug 15, 2017 at 3:43 PM, Joel Maher  wrote:
> 
>> This is a discussion about tests in e10s mode, not WPT on Android.
>> 
>> What specific coverage are we missing by not running WPT in non-e10s mode
>> on desktop?  When can we ensure we have that coverage in e10s mode?  I
>> assume this is just making sure the tests that we have disabled on e10s
>> mode need to get fixed.
>> 
>> On Tue, Aug 15, 2017 at 4:39 PM, Ben Kelly  wrote:
>> 
>>> On Tue, Aug 15, 2017 at 4:37 PM, Joel Maher  wrote:
>>> 
 All of the above mentioned tests are not run on Android (well
 mochitest-media is to some degree).  Is 4 months unreasonable to fix the
 related tests that do not run in e10s?  Is there another time frame that
 seems more reasonable?
 
>>> 
>>> Last I checked it was your team that told me WPT on android was not an
>>> immediate priority.  The WPT harness itself does not run there.
>>> 
>> ___
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>> 
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform



signature.asc
Description: Message signed with OpenPGP
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-15 Thread J. Ryan Stinnett
I think Ben's argument has merit:

1. Even after Firefox 57, we will still be shipping a product in non-e10s
mode: Firefox for Android
2. If WPT (and potentially other suites) aren't being run in non-e10s mode,
it increases risk because we are shipping untested code paths to our users

Someone might add code that assumes e10s is the only mode and land it
successfully, only later to hear that it fails or crashes on Android, since
e10s doesn't exist there today.

- Ryan

On Tue, Aug 15, 2017 at 3:43 PM, Joel Maher  wrote:

> This is a discussion about tests in e10s mode, not WPT on Android.
>
> What specific coverage are we missing by not running WPT in non-e10s mode
> on desktop?  When can we ensure we have that coverage in e10s mode?  I
> assume this is just making sure the tests that we have disabled on e10s
> mode need to get fixed.
>
> On Tue, Aug 15, 2017 at 4:39 PM, Ben Kelly  wrote:
>
> > On Tue, Aug 15, 2017 at 4:37 PM, Joel Maher  wrote:
> >
> >> All of the above mentioned tests are not run on Android (well
> >> mochitest-media is to some degree).  Is 4 months unreasonable to fix the
> >> related tests that do not run in e10s?  Is there another time frame that
> >> seems more reasonable?
> >>
> >
> > Last I checked it was your team that told me WPT on android was not an
> > immediate priority.  The WPT harness itself does not run there.
> >
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-15 Thread Ben Kelly
On Tue, Aug 15, 2017 at 4:43 PM, Joel Maher  wrote:

> This is a discussion about tests in e10s mode, not WPT on Android.
>

Yes.  And android is our last tier 1 platform that requires non-e10s.  I
think that makes it relevant to the discussion.


>
> What specific coverage are we missing by not running WPT in non-e10s mode
> on desktop?  When can we ensure we have that coverage in e10s mode?  I
> assume this is just making sure the tests that we have disabled on e10s
> mode need to get fixed.
>

Some features are implemented completely differently in e10s vs non-e10s
mode.  The main one I am aware of is service workers.  If you turn on
non-e10s WPT tests there will be regressions.

Don't get me wrong.  I'd prefer not to deal with non-e10s either.  But its
*absolutely false* to say we don't need to support it right now because its
required for a tier 1 platform.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-15 Thread Joel Maher
This is a discussion about tests in e10s mode, not WPT on Android.

What specific coverage are we missing by not running WPT in non-e10s mode
on desktop?  When can we ensure we have that coverage in e10s mode?  I
assume this is just making sure the tests that we have disabled on e10s
mode need to get fixed.

On Tue, Aug 15, 2017 at 4:39 PM, Ben Kelly  wrote:

> On Tue, Aug 15, 2017 at 4:37 PM, Joel Maher  wrote:
>
>> All of the above mentioned tests are not run on Android (well
>> mochitest-media is to some degree).  Is 4 months unreasonable to fix the
>> related tests that do not run in e10s?  Is there another time frame that
>> seems more reasonable?
>>
>
> Last I checked it was your team that told me WPT on android was not an
> immediate priority.  The WPT harness itself does not run there.
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-15 Thread Ben Kelly
On Tue, Aug 15, 2017 at 4:37 PM, Joel Maher  wrote:

> All of the above mentioned tests are not run on Android (well
> mochitest-media is to some degree).  Is 4 months unreasonable to fix the
> related tests that do not run in e10s?  Is there another time frame that
> seems more reasonable?
>

Last I checked it was your team that told me WPT on android was not an
immediate priority.  The WPT harness itself does not run there.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-15 Thread Joel Maher
All of the above mentioned tests are not run on Android (well
mochitest-media is to some degree).  Is 4 months unreasonable to fix the
related tests that do not run in e10s?  Is there another time frame that
seems more reasonable?

On Tue, Aug 15, 2017 at 4:34 PM, Ben Kelly  wrote:

> On Tue, Aug 15, 2017 at 4:29 PM,  wrote:
>
>> While there are many tests which individually are disabled or lacking
>> coverage, these test suites have no non-e10s coverage:
>> * web-platform-tests
>> * browser-chrome
>> * devtools
>> * jsreftests
>> * mochitest-webgl, mochitest-gpu, mochitest-media
>> * reftest un-accel
>>
>> I would propose running these above tests on windows7-opt (we don't have
>> these running yet in windows 10, although we are close), and only running
>> specific tests which are not run in e10s mode, turning them off December
>> 29th, 2017.
>>
>> Keep in mind we have had many years to get all our tests running in e10s
>> mode and we have known since last year that Firefox 57 would be the first
>> release that we ship e10s by default to our users- my proposal is a 4 month
>> temporary measure to let test owners finish ensuring their tests are
>> running in e10s mode.
>>
>
> WPT tests do not run on android.  Unless we can get WPT tests running on
> android, I don't think dropping all non-e10s coverage for them is
> reasonable.  I don't know of any plan to move android to e10s by Dec 29.
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-15 Thread Ben Kelly
On Tue, Aug 15, 2017 at 4:29 PM,  wrote:

> While there are many tests which individually are disabled or lacking
> coverage, these test suites have no non-e10s coverage:
> * web-platform-tests
> * browser-chrome
> * devtools
> * jsreftests
> * mochitest-webgl, mochitest-gpu, mochitest-media
> * reftest un-accel
>
> I would propose running these above tests on windows7-opt (we don't have
> these running yet in windows 10, although we are close), and only running
> specific tests which are not run in e10s mode, turning them off December
> 29th, 2017.
>
> Keep in mind we have had many years to get all our tests running in e10s
> mode and we have known since last year that Firefox 57 would be the first
> release that we ship e10s by default to our users- my proposal is a 4 month
> temporary measure to let test owners finish ensuring their tests are
> running in e10s mode.
>

WPT tests do not run on android.  Unless we can get WPT tests running on
android, I don't think dropping all non-e10s coverage for them is
reasonable.  I don't know of any plan to move android to e10s by Dec 29.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-15 Thread jmaher
Thanks everyone for commenting on this thread.  As a note, we run many tests in 
non-e10s mode:
* android mochitest, reftest, crashtest, marionette,
* mochitest-chrome
* xpcshell
* gtest/cpptest/jittest
* mochitest-a11y
* mochitest-jetpack (very few tests remain)

While there are many tests which individually are disabled or lacking coverage, 
these test suites have no non-e10s coverage:
* web-platform-tests
* browser-chrome
* devtools
* jsreftests
* mochitest-webgl, mochitest-gpu, mochitest-media
* reftest un-accel

I would propose running these above tests on windows7-opt (we don't have these 
running yet in windows 10, although we are close), and only running specific 
tests which are not run in e10s mode, turning them off December 29th, 2017.

Keep in mind we have had many years to get all our tests running in e10s mode 
and we have known since last year that Firefox 57 would be the first release 
that we ship e10s by default to our users- my proposal is a 4 month temporary 
measure to let test owners finish ensuring their tests are running in e10s mode.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-10 Thread Ben Kelly
On Tue, Aug 8, 2017 at 5:18 PM, Ben Kelly  wrote:

> On Tue, Aug 8, 2017 at 5:12 PM,  wrote:
>
>> In bug 1386689, we have turned them off.  There was some surprise in
>> doing this and some valid concerns expressed in comments in the bug.  Given
>> that, I thought we should bring this information to a wider audience on
>> dev.platform so more developers are aware of the change.
>>
>> While we get some advantages to not running duplicated tests (faster try
>> results, less backlogs, fewer intermittent failures) there might be
>> compelling reasons to run some tests in e10s based on specific coverage.
>> With that said, I would like to say that any tests we turn on as non-e10s
>> must have a clearly marked end date- as in this is only a temporary measure
>> while we schedule work in to gain this coverage fully with e10s tests.
>>
>
> If we have an android test disabled (a lot I think), then we should
> consider continuing to run the test in non-e10s on desktop linux.  Having
> more real devices to run android tests on would probably reduce the number
> of tests that are disabled.  The emulator is extremely slow and not
> representative of real hardware.
>

BTW, we have a large corpus of tests that don't run on android at all:
WPT.  Increasingly over time features are only covered by WPT tests.

I think we should keep at least non-e10s running on linux or linux64 for
now until we can improve our android test coverage or move android to e10s.

Ben
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-09 Thread Felipe G
I ran some scripts that I had to find out tests that are *fully* disabled
on e10s, and posted the results to
https://bugzilla.mozilla.org/show_bug.cgi?id=1376934

In summary:
mochitest-plain: 49 tests
browser-chrome: 15 tests
devtools: 86 tests

Note that the script evaluates the skip-if condition for each test, so it's
able to not count tests such as "skip=if e10s && debug" as fully disabled.

On Wed, Aug 9, 2017 at 7:35 AM, Gabor Krizsanits 
wrote:

> On Wed, Aug 9, 2017 at 3:36 AM, Boris Zbarsky  wrote:
>
> >
> > Hmm.  Do we load about:blank from the url bar in a content process?
> >
> >
> Yes.
>
> I agree, I find it annoying too that we have to rely on
> MOZ_DEBUG_CHILD_PROCESS
> or MOZ_DEBUG_CHILD_PAUSE and that I have to be clever all the time about
> how to hit the right process at the right time with the debugger. I never
> switched back to non-e10s though since I don't trust that everything will
> work the same and I don't think that should be the solution. Switching back
> to single content process for debugging should come with less side effects
> though... Also, this is not just an e10s/e10s-multi related issues we're
> adding all kind of processes (extension/gpu/plugin/etc.).
>
> I didn't file a bug about this but I've been trying to find a decent
> solution for it, but it seems like it's not trivial in any debugger (msvc,
> gdb, lldb). Or maybe I was just hoping to find something better than what
> seems to be achievable. Anyway, let's start with the bug: Bug 1388693.
>
> Gabor
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-09 Thread Gabor Krizsanits
On Wed, Aug 9, 2017 at 3:36 AM, Boris Zbarsky  wrote:

>
> Hmm.  Do we load about:blank from the url bar in a content process?
>
>
Yes.

I agree, I find it annoying too that we have to rely on MOZ_DEBUG_CHILD_PROCESS
or MOZ_DEBUG_CHILD_PAUSE and that I have to be clever all the time about
how to hit the right process at the right time with the debugger. I never
switched back to non-e10s though since I don't trust that everything will
work the same and I don't think that should be the solution. Switching back
to single content process for debugging should come with less side effects
though... Also, this is not just an e10s/e10s-multi related issues we're
adding all kind of processes (extension/gpu/plugin/etc.).

I didn't file a bug about this but I've been trying to find a decent
solution for it, but it seems like it's not trivial in any debugger (msvc,
gdb, lldb). Or maybe I was just hoping to find something better than what
seems to be achievable. Anyway, let's start with the bug: Bug 1388693.

Gabor
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-08 Thread Boris Zbarsky

On 8/8/17 6:40 PM, Blake Kaplan wrote:

What part of the current set-up is rocket science?


Debugging pageload.  Especially pageload with no session history.


In multi, there's
definitely a problem figuring out which process is the active one (though
tooltips when hovering over tabs can help).


That doesn't help when you want a no-history load.  You might get an 
entirely new process.  Or one of your existing processes, who knows.



remote tab, you can load pages in that tab and we'll stay in the same process.


Right, but then you have to worry about the interactions of the unload 
and new load...



That means that loading about:blank explicitly, attaching to that tab's
process, setting breakpoints, and loading the page should do the right thing.


Hmm.  Do we load about:blank from the url bar in a content process?

-Boris

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-08 Thread Ehsan Akhgari

On 08/08/2017 06:51 PM, Blake Kaplan wrote:

L. David Baron  wrote:

Has there been any additional effort to deal with tests that have
been disabled under e10s?  This change means we've essentially
turned off a decent number of tests.

IIRC, the last time we talked about this, there was interest in either running
tests explicitly disabled in e10s in a non-e10s browser or doing another
e10s-style run-through and sign-off of the remaining tests to move them to
suites that won't be turned off or to verify that we don't lose test coverage
of things we care about. Felipe was going to use the scripts that we used in
the original e10s rollout to generate a new list of tests to verify.
Was this done before the tests were turned off though?  From what I read 
from the bug, it seems like the answer is no.  It seems to me that the 
above should have been a prerequisite for turning these tests off, 
especially at a time like this leading to the release of Firefox 57 when 
we should all we can to reduce the risk of the release...  Losing test 
coverage on many tests here and there, and also on Android for the tests 
that we run on desktop in lieu seems to me as potential factors to 
increase risk.  :-(

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-08 Thread Blake Kaplan
L. David Baron  wrote:
> Has there been any additional effort to deal with tests that have
> been disabled under e10s?  This change means we've essentially
> turned off a decent number of tests.

IIRC, the last time we talked about this, there was interest in either running
tests explicitly disabled in e10s in a non-e10s browser or doing another
e10s-style run-through and sign-off of the remaining tests to move them to
suites that won't be turned off or to verify that we don't lose test coverage
of things we care about. Felipe was going to use the scripts that we used in
the original e10s rollout to generate a new list of tests to verify.
-- 
Blake Kaplan
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-08 Thread Blake Kaplan
Boris Zbarsky  wrote:
> On 8/8/17 5:12 PM, jma...@mozilla.com wrote:
>> In bug 1386689, we have turned them off.  There was some surprise in doing 
>> this and some valid concerns expressed in comments in the bug.
> Something as simple as "debug something that happens during pageload" is 
> currently fairly rocket science to do in e10s mode; doubly so in 
> e10s-multi.  I haven't seen any concrete proposals for improving this

What part of the current set-up is rocket science? In multi, there's
definitely a problem figuring out which process is the active one (though
tooltips when hovering over tabs can help). That being said, once you have a
remote tab, you can load pages in that tab and we'll stay in the same process.
That means that loading about:blank explicitly, attaching to that tab's
process, setting breakpoints, and loading the page should do the right thing.
-- 
Blake Kaplan
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-08 Thread Robert O'Callahan
On Wed, Aug 9, 2017 at 9:31 AM, Boris Zbarsky  wrote:

Something as simple as "debug something that happens during pageload" is
> currently fairly rocket science to do in e10s mode; doubly so in
> e10s-multi.  I haven't seen any concrete proposals for improving this


rr makes it fairly easy on Linux. On Mac and Windows ... yeah.

Rob
-- 
lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf
toD
selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t
rdsme,aoreseoouoto
o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea
lurpr
.a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr
esn
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-08 Thread L. David Baron
On Tuesday 2017-08-08 14:12 -0700, jma...@mozilla.com wrote:
> As Firefox 57 is on trunk, we are shipping e10s by default.  This means that 
> our primary support is for e10s.  As part of this, there is little to no need 
> to run duplicated tests in non-e10s and e10s mode.  
> 
> In bug 1386689, we have turned them off.  There was some surprise in doing 
> this and some valid concerns expressed in comments in the bug.  Given that, I 
> thought we should bring this information to a wider audience on dev.platform 
> so more developers are aware of the change.

Has there been any additional effort to deal with tests that have
been disabled under e10s?  This change means we've essentially
turned off a decent number of tests.

Some of these tests show up in these searches:
https://searchfox.org/mozilla-central/search?q=skip-if+%3D+e10s
https://searchfox.org/mozilla-central/search?q=skip-if%28browserIsRemote
although that search isn't exhaustive, and contains some tests that
we still run under some conditions.

-David

-- 
턞   L. David Baron http://dbaron.org/   턂
턢   Mozilla  https://www.mozilla.org/   턂
 Before I built a wall I'd ask to know
 What I was walling in or walling out,
 And to whom I was like to give offense.
   - Robert Frost, Mending Wall (1914)


signature.asc
Description: PGP signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-08 Thread Boris Zbarsky

On 8/8/17 5:12 PM, jma...@mozilla.com wrote:

In bug 1386689, we have turned them off.  There was some surprise in doing this 
and some valid concerns expressed in comments in the bug.


Indeed.  Given how often non-e10s mode needs to get used for debugging, 
it's a little concerning to see the "yeah, it's fine if we just totally 
break non-e10s mode" comments in that bug.


Something as simple as "debug something that happens during pageload" is 
currently fairly rocket science to do in e10s mode; doubly so in 
e10s-multi.  I haven't seen any concrete proposals for improving this


-Boris
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-08 Thread Ben Kelly
On Tue, Aug 8, 2017 at 5:18 PM, Ben Kelly  wrote:

> On Tue, Aug 8, 2017 at 5:12 PM,  wrote:
>
>> While we get some advantages to not running duplicated tests (faster try
>> results, less backlogs, fewer intermittent failures) there might be
>> compelling reasons to run some tests in e10s based on specific coverage.
>> With that said, I would like to say that any tests we turn on as non-e10s
>> must have a clearly marked end date- as in this is only a temporary measure
>> while we schedule work in to gain this coverage fully with e10s tests.
>>
>
> If we have an android test disabled (a lot I think), then we should
> consider continuing to run the test in non-e10s on desktop linux.  Having
> more real devices to run android tests on would probably reduce the number
> of tests that are disabled.  The emulator is extremely slow and not
> representative of real hardware.
>

Sorry to self-reply, but the best solution would of course be to move
android to a single content process e10s model.  Then we could truly remove
non-e10s.  Obviously that is probably not easy to do, though...

Ben
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: disabled non-e10s tests on trunk

2017-08-08 Thread Ben Kelly
On Tue, Aug 8, 2017 at 5:12 PM,  wrote:

> As Firefox 57 is on trunk, we are shipping e10s by default.  This means
> that our primary support is for e10s.  As part of this, there is little to
> no need to run duplicated tests in non-e10s and e10s mode.
>

We still run android in non-e10s mode, right?


> In bug 1386689, we have turned them off.  There was some surprise in doing
> this and some valid concerns expressed in comments in the bug.  Given that,
> I thought we should bring this information to a wider audience on
> dev.platform so more developers are aware of the change.
>
> While we get some advantages to not running duplicated tests (faster try
> results, less backlogs, fewer intermittent failures) there might be
> compelling reasons to run some tests in e10s based on specific coverage.
> With that said, I would like to say that any tests we turn on as non-e10s
> must have a clearly marked end date- as in this is only a temporary measure
> while we schedule work in to gain this coverage fully with e10s tests.
>

If we have an android test disabled (a lot I think), then we should
consider continuing to run the test in non-e10s on desktop linux.  Having
more real devices to run android tests on would probably reduce the number
of tests that are disabled.  The emulator is extremely slow and not
representative of real hardware.

Ben
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform