Re: Intent to deprecate - linux32 tests starting with Firefox 69

[Joel Maher]

On Thursday, April 11, 2019 at 10:08:27 AM UTC-4, William Lachance wrote:
> On 2019-04-09 11:00 a.m., Gian-Carlo Pascutto wrote:
> > On 5/04/19 15:35,jma...@mozilla.com  wrote:
> >> Currently linux32 makes up about .77% of our total users.  This is
> >> still 1M+ users on any given week.
> > I asked jmaher what percentage of our Linux users this is. It's 21%.
> > This doesn't seem small.
> > 
> > On top of that, we know that not all distros have telemetry enabled and
> > so we won't be counting those either (Debian is the largest).
> > 
> > I don't know how many users we need to keep supporting a platform, but
> > this seems like quite a few people to throw under the bus.
> 
> I'm not sure where :jmaher got his original figure, but I just did up a 
> crude analysis counting the number of clients_daily entries on Linux 
> day-by-day over the past year reporting x86 (32-bit) vs. x86-64 (64-bit):
> 
> https://sql.telemetry.mozilla.org/queries/62267
> 
> The 21% number looks like it was accurate about a year ago. At this 
> point, we're hovering around 15% of reported usage day-to-day.
> 
> Obviously this isn't a complete analysis and doesn't validate that each 
> client ping represents a real user (or the distinction between e.g. a 
> ping representing a user who uses Firefox once a week vs. once a day), 
> but it does suggest (as intuition would tell us) that linux32 is of 
> diminishing interest. If you want a more scientific answer here, I'd 
> recommend engaging Firefox data science:
> 
> https://mana.mozilla.org/wiki/display/PM/Firefox+Data+Science#FirefoxDataScience-initiatingaproject
> 
> Will

Thanks everyone for your responses, It sounds like making a decision to 
deprecate linux32 builds and tests need to happen at the same time and that 
decision will need to come from a variety of people who are higher up in the 
decision making process of the Firefox products.

Until we get that, I propose when trunk switches to Firefox 69 (May 13th - and 
68 is on track to be the new ESR), we make these changes on trunk for linux32:
* demote the builds/tests to tier-2 (only on mozilla-central/try)
* stop building/testing linux32-debug (we don't ship it and it hasn't found any 
unique regressions resulting in a product fix in 9+ months of data I have 
available)
* reduce the tests we run to be web-platform-tests; this ensures we maintain 
compatibility with web standards.


Based on all the feedback on this thread, I don't see any concerns with this 
path forward.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to deprecate - linux32 tests starting with Firefox 69

[William Lachance]

On 2019-04-09 11:00 a.m., Gian-Carlo Pascutto wrote:

On 5/04/19 15:35,jma...@mozilla.com  wrote:

Currently linux32 makes up about .77% of our total users.  This is
still 1M+ users on any given week.

I asked jmaher what percentage of our Linux users this is. It's 21%.
This doesn't seem small.

On top of that, we know that not all distros have telemetry enabled and
so we won't be counting those either (Debian is the largest).

I don't know how many users we need to keep supporting a platform, but
this seems like quite a few people to throw under the bus.


I'm not sure where :jmaher got his original figure, but I just did up a 
crude analysis counting the number of clients_daily entries on Linux 
day-by-day over the past year reporting x86 (32-bit) vs. x86-64 (64-bit):


https://sql.telemetry.mozilla.org/queries/62267

The 21% number looks like it was accurate about a year ago. At this 
point, we're hovering around 15% of reported usage day-to-day.


Obviously this isn't a complete analysis and doesn't validate that each 
client ping represents a real user (or the distinction between e.g. a 
ping representing a user who uses Firefox once a week vs. once a day), 
but it does suggest (as intuition would tell us) that linux32 is of 
diminishing interest. If you want a more scientific answer here, I'd 
recommend engaging Firefox data science:


https://mana.mozilla.org/wiki/display/PM/Firefox+Data+Science#FirefoxDataScience-initiatingaproject

Will
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to deprecate - linux32 tests starting with Firefox 69

[Joel Maher]

Thanks everyone for your comments.

If we were to run linux32 tests in whole on mozilla-central only that would
result in about half of the load we see from linux32 today and about 1
backfill a week (given that most unique linux32 regressions result in test
disabling).  That alone would be a good savings of time, money, and
reduction of intermittents.

I would like to see what others think about keeping the platform supported
on trunk vs letting it ride the trains to beta where it would live until
near the end of 2020.  I assume a discussion about what unique features and
code will help determine what level of support we want for tests/builds of
linux32.


On Wed, Apr 10, 2019 at 12:53 PM Bobby Holley  wrote:

> I'd like to refocus this thread a bit around Jed's question, because it
> gets to the core of the issue.
>
> The proposal argues that test results for linux32 closely track those for
> linux64, and that this duplication is expensive. If that's the only
> problem, we could solve it by keeping linux32 as a tier-1 platform, but
> just running the tests much less frequently. This would require
> occasionally backfilling jobs when something does break, but that should be
> rare.
>
> There is a separate question of whether it is expensive to _engineer_ for
> linux32, and whether we should drop support on those grounds (a la Windows
> XP and non-SSE2). So far, the only linux32-specific engineering cost that
> has been raised is linux32-specific sandboxing code. Is this a significant
> maintenance burden? Are there other large areas of the code that are
> linux32-specific?
>
>
>
> On Mon, Apr 8, 2019 at 4:20 PM Jed Davis  wrote:
>
> > jma...@mozilla.com writes:
> >
> > > As our next ESR is upcoming, I would like to turn off linux32 on
> > > Firefox 69 and let it ride the trains and stay on 68 ESR.  This will
> > > allow builds/tests to be supported with security updates into 2021.
> >
> > Does this mean that Linux on 32-bit x86 is being demoted to Tier 3, like
> > (non-Android) Linux on other uncommon architectures?  Have we identified
> > a maintainer to take responsibility for testing and working with us to
> > fix regressions?  I don't see how it can remain Tier 1 without CI
> coverage.
> >
> > Also, will it still be possible to explicitly request linux32 on Try
> > runs?  Our cross-building story is not good, and being able to use Try
> > for this helpful for avoiding regressions and testing changes to
> > arch-dependent code.
> >
> > For context, I'm the module owner for Linux sandboxing, and we have to
> > deal with low-level details of the system call ABI, and as a result
> > there is nontrivial code used only for linux32 that I'm responsible for.
> >
> > --Jed
> > ___
> > dev-platform mailing list
> > dev-platform@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-platform
> >
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to deprecate - linux32 tests starting with Firefox 69

[Bobby Holley]

I'd like to refocus this thread a bit around Jed's question, because it
gets to the core of the issue.

The proposal argues that test results for linux32 closely track those for
linux64, and that this duplication is expensive. If that's the only
problem, we could solve it by keeping linux32 as a tier-1 platform, but
just running the tests much less frequently. This would require
occasionally backfilling jobs when something does break, but that should be
rare.

There is a separate question of whether it is expensive to _engineer_ for
linux32, and whether we should drop support on those grounds (a la Windows
XP and non-SSE2). So far, the only linux32-specific engineering cost that
has been raised is linux32-specific sandboxing code. Is this a significant
maintenance burden? Are there other large areas of the code that are
linux32-specific?



On Mon, Apr 8, 2019 at 4:20 PM Jed Davis  wrote:

> jma...@mozilla.com writes:
>
> > As our next ESR is upcoming, I would like to turn off linux32 on
> > Firefox 69 and let it ride the trains and stay on 68 ESR.  This will
> > allow builds/tests to be supported with security updates into 2021.
>
> Does this mean that Linux on 32-bit x86 is being demoted to Tier 3, like
> (non-Android) Linux on other uncommon architectures?  Have we identified
> a maintainer to take responsibility for testing and working with us to
> fix regressions?  I don't see how it can remain Tier 1 without CI coverage.
>
> Also, will it still be possible to explicitly request linux32 on Try
> runs?  Our cross-building story is not good, and being able to use Try
> for this helpful for avoiding regressions and testing changes to
> arch-dependent code.
>
> For context, I'm the module owner for Linux sandboxing, and we have to
> deal with low-level details of the system call ABI, and as a result
> there is nontrivial code used only for linux32 that I'm responsible for.
>
> --Jed
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to deprecate - linux32 tests starting with Firefox 69

[Henri Sivonen]

On Tue, Apr 9, 2019 at 6:05 PM Gian-Carlo Pascutto  wrote:

> On top of that, we know that not all distros have telemetry enabled and
> so we won't be counting those either (Debian is the largest).

We get telemetry from Ubuntu, Fedora, and Arch (subject to choices
made by the user). I'm not particularly worried about Debian, because
x86_64 (as amd64) has had prominent (unlike Ubuntu; see below)
availability from Debian pretty much for as long as x86_64 hardware
has been available. But in any case, not enabling telemetry means not
having representation in data-driven decision making.

> At least current Ubuntu and Ubuntu LTS are still available in 32-bit:
> https://www.ubuntu.com/download/alternative-downloads
>
> Ubuntu is our largest userbase (with telemetry...)

The latest and the latest LTS are offered for 32-bit x86 only as
netboot installers, and the latest LTS also as an upgrade from an
earlier version, so Ubuntu now positions 32-bit x86 as more obscure
than s390x, ppc64el, aarch64, and armv7hf. Ubuntu has already disabled
automatic updates to 18.10 for 32-bit x86, because the expectation is
that 18.04 will be the longest-supported release for 32-bit x86.

I think the main problem with Ubuntu is that Ubuntu promoted 32-bit
x86 downloads to users who didn't have the expertise to seek the
x86_64 version for way longer than they, in my opinion, should have
(though, in fairness, at that time, Mozilla also was promoting 32-bit
x86 downloads over 64-bit). Users who installed Ubuntu at that time
may have gotten a 32-bit distro even if the hardware would work just
fine with the 64-bit version, and there are no upgrades from 32-bit to
64-bit other than reinstall. Probably more Canonical's job than ours
to communicate to those users that they should do a 64-bit reinstall.

(In terms of security support from the Ubuntu repos, I find the
situation for architectures that aren't tier-1 for us a bit
concerning. Compare
https://packages.ubuntu.com/search?suite=disco=names=firefox
with 
https://packages.ubuntu.com/search?suite=cosmic=names=firefox
. Despite 32-bit x86 having been demoted below s390x, ppc64el,
aarch64, and armv7hf in terms of distribution images, 32-bit x86 gets
better security support in the Ubuntu repos than s390x, ppc64el,
aarch64, and armv7hf. So does security support in Ubuntu repos depend
on our tier positioning or to Ubuntu's own positioning?)

-- 
Henri Sivonen
hsivo...@mozilla.com
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re[2]: Intent to deprecate - linux32 tests starting with Firefox 69

[mhoye]

-- Original Message --
From: "Gian-Carlo Pascutto" 
To: dev-platform@lists.mozilla.org
Sent: 2019-04-09 11:00:14 AM
Subject: Re: Intent to deprecate - linux32 tests starting with Firefox 
69



On 5/04/19 15:35, jma...@mozilla.com wrote:

Currently linux32 makes up about .77% of our total users.  This is
still 1M+ users on any given week.


I asked jmaher what percentage of our Linux users this is. It's 21%.
This doesn't seem small.


It's roughly comparable to the numbers involved for other support 
decisions we've made.  Ending support for non-SSE2 CPUs was was 
comparable, IIRC. Ending OSX support (10.6-10.8) impacted roughly 1.2% 
of our userbase. When we moved XP users to ESR before ending support for 
XP entirely, the numbers were _a lot_ bigger than that.




On top of that, we know that not all distros have telemetry enabled and
so we won't be counting those either (Debian is the largest).
Distros that have opted out of telemetry have opted out of informing our 
planning and decision-making.


- mhoye

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to deprecate - linux32 tests starting with Firefox 69

[Gian-Carlo Pascutto]

On 5/04/19 15:35, jma...@mozilla.com wrote:
> Currently linux32 makes up about .77% of our total users.  This is
> still 1M+ users on any given week.

I asked jmaher what percentage of our Linux users this is. It's 21%.
This doesn't seem small.

On top of that, we know that not all distros have telemetry enabled and
so we won't be counting those either (Debian is the largest).

I don't know how many users we need to keep supporting a platform, but
this seems like quite a few people to throw under the bus.

> There is not a lot of support in the industry for 32 bit linux, all
> the major vendors are only distributing 64 bit versions now and other
> browser vendors only distribute 64 bit versions officially.

At least current Ubuntu and Ubuntu LTS are still available in 32-bit:
https://www.ubuntu.com/download/alternative-downloads

Ubuntu is our largest userbase (with telemetry...)

> Currently we run a full suite of unit tests on linux32 per commit
> (integration, mozilla-central, try, mozilla-beta) and overall this is
> about 11% of our total CPU usage (which is about $416/day).

This is indeed a significant cost.

> Linux32 runs many tests in both e10s and non-e10s (see
> https://bugzilla.mozilla.org/show_bug.cgi?id=1433276 ), for a few
> reasons: 1) fennec ships non-e10s, our code base should have more
> coverage 2) not all tests run on fennec 3) users can disable non-e10s
> locally

3) doesn't seem like a very good reason. At least going forward I don't
think it's something we want to keep supporting or really worry about
right now. So on the Linux side just disabling those tests should be
acceptable.

The problem seems to be that linux32 and in particular non-e10s is used
as a proxy to test other platforms. Fennec was mentioned (and might be
fixable), but are there some other non-e10s use cases like Windows with
accessibility? Or is that no longer an issue?

So indeed then you'd have to move those tests first.

-- 
GCP
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to deprecate - linux32 tests starting with Firefox 69

[Jed Davis]

jma...@mozilla.com writes:

> As our next ESR is upcoming, I would like to turn off linux32 on
> Firefox 69 and let it ride the trains and stay on 68 ESR.  This will
> allow builds/tests to be supported with security updates into 2021.

Does this mean that Linux on 32-bit x86 is being demoted to Tier 3, like
(non-Android) Linux on other uncommon architectures?  Have we identified
a maintainer to take responsibility for testing and working with us to
fix regressions?  I don't see how it can remain Tier 1 without CI coverage.

Also, will it still be possible to explicitly request linux32 on Try
runs?  Our cross-building story is not good, and being able to use Try
for this helpful for avoiding regressions and testing changes to
arch-dependent code.

For context, I'm the module owner for Linux sandboxing, and we have to
deal with low-level details of the system call ABI, and as a result
there is nontrivial code used only for linux32 that I'm responsible for.

--Jed
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to deprecate - linux32 tests starting with Firefox 69

[Kartikaya Gupta]

On Fri, Apr 5, 2019 at 10:45 AM Boris Zbarsky  wrote:
>
> On 4/5/19 10:20 AM, Kartikaya Gupta wrote:
> > If we can drop that then we can remove a bunch of complexity
> > and code paths.
>
> Are we talking about just dropping linux32 _tests_, or dropping linux32
> _support_?

Tests.

> Because if we're keeping support, and linux32 is the only non-APZ
> confiration,

linux32 itself ships with e10s and APZ enabled like all our other
desktop platforms. the linux32 _tests_ also exercise non-e10s
codepaths (mostly to help out fennec), as Joel said in his original
email. But Fennec itself has APZ enabled even though it doesn't have
e10s. So from an APZ point of view those tests are not testing
anything useful. Dropping these tests is not a blocker for dropping
the non-APZ codepaths but it just makes it easier since we don't have
to figure out what to do with the tests that might break as a result.

kats
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to deprecate - linux32 tests starting with Firefox 69

[Boris Zbarsky]

On 4/5/19 10:20 AM, Kartikaya Gupta wrote:

If we can drop that then we can remove a bunch of complexity
and code paths.


Are we talking about just dropping linux32 _tests_, or dropping linux32 
_support_?


Because if we're keeping support, and linux32 is the only non-APZ 
confiration, then not only do we need to keep the non-APZ code but we 
need to keep the tests that might depend on APZ vs non-APZ, no?


-Boris
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to deprecate - linux32 tests starting with Firefox 69

[Kartikaya Gupta]

I would be happy to see Linux32 (and in particular, desktop non-e10s) tests
decommissioned, as that is pretty much the only configuration with APZ
disabled now. If we can drop that then we can remove a bunch of complexity
and code paths.

On Fri., Apr. 5, 2019, 10:05 Boris Zbarsky,  wrote:

> On 4/5/19 9:35 AM, jma...@mozilla.com wrote:
> > As our next ESR is upcoming, I would like to turn off linux32 on Firefox
> 69 and let it ride the trains and stay on 68 ESR.  This will allow
> builds/tests to be supported with security updates into 2021.
>
> The only thing I've used the linux32 tests for recently was on try to
> get tests that:
>
> 1) Compile and run in finite time (unlike Windows and Mac)
> 2) Have useful crash stacks (unlike linux64; see
> ).
>
> When drop linux32, we should make sure we treat stacks in logs on
> linux64 as a priority...
>
> -Boris
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to deprecate - linux32 tests starting with Firefox 69

[Boris Zbarsky]

On 4/5/19 9:35 AM, jma...@mozilla.com wrote:

As our next ESR is upcoming, I would like to turn off linux32 on Firefox 69 and 
let it ride the trains and stay on 68 ESR.  This will allow builds/tests to be 
supported with security updates into 2021.


The only thing I've used the linux32 tests for recently was on try to 
get tests that:


1) Compile and run in finite time (unlike Windows and Mac)
2) Have useful crash stacks (unlike linux64; see 
).


When drop linux32, we should make sure we treat stacks in logs on 
linux64 as a priority...


-Boris
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Intent to deprecate - linux32 tests starting with Firefox 69

[jmaher]

Currently linux32 makes up about .77% of our total users.  This is still 1M+ 
users on any given week.

There is not a lot of support in the industry for 32 bit linux, all the major 
vendors are only distributing 64 bit versions now and other browser vendors 
only distribute 64 bit versions officially.

Currently we run a full suite of unit tests on linux32 per commit (integration, 
mozilla-central, try, mozilla-beta) and overall this is about 11% of our total 
CPU usage (which is about $416/day).

Linux32 has a similar number of intermittent failures (2.06% of tasks fail) and 
regressions as other platforms (such as Android, Windows, OSX, Linux64) 
indicating that this functions as an equal in many ways.  The cost associated 
with intermittents and failures will be a savings for everyone looking at 
results of a push by reducing 10% of intermittents allowing focus other 
failures for faster decision making.

I have looked at regressions that only appear on linux32, in fact looking 
deeper into the changes, there are 3 regressions since July which caused a 
backout and fix to the product (bug 571074, bug 1497580, bug 1499426).  For 
reference, in 2019, osx, windows7, and android 4.3 all have ~5 unique 
regressions found and all other configs have 0 or 1.  The larger volume of 
regressions we find are seen on multiple platforms or didn't result in fixing 
the browser.

Linux32 runs many tests in both e10s and non-e10s (see 
https://bugzilla.mozilla.org/show_bug.cgi?id=1433276 ), for a few reasons:
1) fennec ships non-e10s, our code base should have more coverage
2) not all tests run on fennec
3) users can disable non-e10s locally

I looked at Firefox non-e10s users, and it is 0.2% of our users for anyone that 
has upgraded the browser in the last 6 months.  I focused on that because if we 
are supporting modern versions and users never see it, then our efforts to 
build/develop non-e10s are having little effect.  In fact 96.4% of our users 
that use non-e10s are on Windows, so if we determine there is a need for 
testing this, I would encourage us to test on windows instead of Linux.  As for 
running non-e10s tests in place of coverage on Android, we have enabled more 
tests on android emulators (primarily due to web-platform-tests) which has 
reduced the need to rely on Linux32 as a means for getting close enough 
coverage for Android.

Earlier I mentioned 3 regressions in the browser that we fixed, 2 of those were 
for non-e10s specific cases, and one was for our default browser in e10s mode.  
While there have been many other regressions found on linux32, they are either 
fixed by hacking tests or disabling the test, or we see them on other platforms 
and would catch them in the absence of linux32 tests.

As our next ESR is upcoming, I would like to turn off linux32 on Firefox 69 and 
let it ride the trains and stay on 68 ESR.  This will allow builds/tests to be 
supported with security updates into 2021.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform