Re: 64-bit Firefox progress report: 2017-07-18

[Mike Hommey]

On Wed, Jul 19, 2017 at 09:40:12PM -0400, Ben Kelly wrote:
> On Jul 19, 2017 8:57 PM, "Mike Hommey"  wrote:
> 
> On Wed, Jul 19, 2017 at 08:48:45PM -0400, Ben Kelly wrote:
> > On Jul 19, 2017 6:20 PM, "Mike Hommey"  wrote:
> >
> > > What would be the rationale behind this choice?
> >
> > Smaller memory footprint, which, you'll admit, when you're on a machine
> > with (less than) 2GB RAM, makes a difference.
> >
> >
> > I thought we had data that showed OOM (small) due to VM fragmentation
> still
> > outweighed OOM (large) on these machines.  If that is the case 64-bit is
> > strictly better.
> 
> I don't believe you can exhaust the address space and have VM
> fragmentation be an actual problem when you have less than 2GB RAM. Of
> if you do, everything is slow to a crawl before that happens.
> 
> 
> I don't understand why that would be, but if so it should show in
> crashstats as fewer small OOMs on these devices.  Does the data actually
> show that?

I don't know. Can that be filtered?

> Also, that tells nothing about people that never hit OOM (a lot of
> people even close their browser well before that could happen).
> 
> 
> You're saying the size difference between 32-bit and 64-bit is so great
> users will start manually shutting down?  I find that hard to believe.  If
> they are shutting down apps to save memory they will probably do it with
> both 32-bit and 64-bit.  Again, though, in theory we have usage hour
> telemetry we could look at to confirm (although maybe only for beta?).

I'm not saying they shut down because of memory. I'm saying a surprising
lot of people have browser sessions that don't last more than 5 minutes
(I've heard multiple times in the past years that we found that out from
our data).
Those people are not going to see OOM.

Mike

PS: your mailer likes to not put quotation marks, making your messages
hard to follow. I thought that was because it's sending both a text and
an html version, and the text version sucks, but it appears it's not
sending an html version...
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: 64-bit Firefox progress report: 2017-07-18

[Ben Kelly]

On Jul 19, 2017 8:57 PM, "Mike Hommey"  wrote:

On Wed, Jul 19, 2017 at 08:48:45PM -0400, Ben Kelly wrote:
> On Jul 19, 2017 6:20 PM, "Mike Hommey"  wrote:
>
> > What would be the rationale behind this choice?
>
> Smaller memory footprint, which, you'll admit, when you're on a machine
> with (less than) 2GB RAM, makes a difference.
>
>
> I thought we had data that showed OOM (small) due to VM fragmentation
still
> outweighed OOM (large) on these machines.  If that is the case 64-bit is
> strictly better.

I don't believe you can exhaust the address space and have VM
fragmentation be an actual problem when you have less than 2GB RAM. Of
if you do, everything is slow to a crawl before that happens.


I don't understand why that would be, but if so it should show in
crashstats as fewer small OOMs on these devices.  Does the data actually
show that?

Also, that tells nothing about people that never hit OOM (a lot of
people even close their browser well before that could happen).


You're saying the size difference between 32-bit and 64-bit is so great
users will start manually shutting down?  I find that hard to believe.  If
they are shutting down apps to save memory they will probably do it with
both 32-bit and 64-bit.  Again, though, in theory we have usage hour
telemetry we could look at to confirm (although maybe only for beta?).
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: 64-bit Firefox progress report: 2017-07-18

[Mike Hommey]

On Wed, Jul 19, 2017 at 08:48:45PM -0400, Ben Kelly wrote:
> On Jul 19, 2017 6:20 PM, "Mike Hommey"  wrote:
> 
> > What would be the rationale behind this choice?
> 
> Smaller memory footprint, which, you'll admit, when you're on a machine
> with (less than) 2GB RAM, makes a difference.
> 
> 
> I thought we had data that showed OOM (small) due to VM fragmentation still
> outweighed OOM (large) on these machines.  If that is the case 64-bit is
> strictly better.

I don't believe you can exhaust the address space and have VM
fragmentation be an actual problem when you have less than 2GB RAM. Of
if you do, everything is slow to a crawl before that happens.

Also, that tells nothing about people that never hit OOM (a lot of
people even close their browser well before that could happen).

Mike
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: 64-bit Firefox progress report: 2017-07-18

[Ben Kelly]

On Jul 19, 2017 6:20 PM, "Mike Hommey"  wrote:

> What would be the rationale behind this choice?

Smaller memory footprint, which, you'll admit, when you're on a machine
with (less than) 2GB RAM, makes a difference.


I thought we had data that showed OOM (small) due to VM fragmentation still
outweighed OOM (large) on these machines.  If that is the case 64-bit is
strictly better.

Ben
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: removing "the old way" of signing add-ons

[Andrew Swan]

On Wed, Jul 19, 2017 at 5:02 PM, R Kent James  wrote:

> On 7/19/2017 4:06 PM, David Keeler wrote:
>
>> [dev-apps-thunderbird and dev-apps-seamonkey cc'd, but please discuss on
>> dev-platform]
>>
> ...
>
>> Given all this, the question is do we still need this second API? Does
>> Thunderbird or SeaMonkey use it for any reason, or can we simplify the
>> code-base, reduce build size, etc.?
>>
>
> Thunderbird never implemented required code signing for addons, so I can't
> imagine we would need the API that you want to remove. But there is a lot
> of past history that I do not know, but I'm not sure who else would know
> better.
>

"Required code signing" was not the point of the original question.
David's email distinguishes between the method used to enforce extension
signing in Firefox (which requires that extensions be signed by Mozilla)
and the "second API" that verifies signatures against anything in the app's
certificate store.  As he says, that second verification method isn't used
in Firefox but in other Gecko applications that use the old add-on install
confirmation dialog, that dialog  includes a note about the certificate
(kind of like what's shown in the location bar when browsing with https).
The question is about removing the code that powers that.

-Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: removing "the old way" of signing add-ons

[Kris Maglione]

On Wed, Jul 19, 2017 at 05:02:23PM -0700, R Kent James wrote:

On 7/19/2017 4:06 PM, David Keeler wrote:

[dev-apps-thunderbird and dev-apps-seamonkey cc'd, but please discuss on
dev-platform]

...

Given all this, the question is do we still need this second API? Does
Thunderbird or SeaMonkey use it for any reason, or can we simplify the
code-base, reduce build size, etc.?


Thunderbird never implemented required code signing for addons, so I 
can't imagine we would need the API that you want to remove. But there 
is a lot of past history that I do not know, but I'm not sure who else 
would know better.


Thunderbird never implemented the new model for mandatory add-on 
signing, but it used to, and probably still does, implement the 
signature verification approach used by the old add-on install 
flow, based on object signing certs in the root store.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: removing "the old way" of signing add-ons

[R Kent James]

On 7/19/2017 4:06 PM, David Keeler wrote:

[dev-apps-thunderbird and dev-apps-seamonkey cc'd, but please discuss on
dev-platform]

...

Given all this, the question is do we still need this second API? Does
Thunderbird or SeaMonkey use it for any reason, or can we simplify the
code-base, reduce build size, etc.?


Thunderbird never implemented required code signing for addons, so I 
can't imagine we would need the API that you want to remove. But there 
is a lot of past history that I do not know, but I'm not sure who else 
would know better.


:rkent
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

removing "the old way" of signing add-ons

[David Keeler]

[dev-apps-thunderbird and dev-apps-seamonkey cc'd, but please discuss on
dev-platform]

Hello Everyone,

You may or may not be surprised to learn that Gecko contains two
different ways to verify that an add-on has been signed. The primary
method is nsIX509CertDB.openSignedAppFileAsync. This is what Gecko-based
products that require add-on signing use. However, there is also
nsIZipRreader.getSigningCert (plus some additional glue code).

The only place where these two implementations share code is in the
actual signature verification. That is, the logic to ensure that every
file in the archive has a corresponding valid entry in the manifest (and
that every entry in the manifest has a corresponding file in the
archive) and so on appears in Gecko twice.

From what I can tell, the actual functionality provided by the second
API (which is only applicable in builds that do not require add-on
signing) is a small text label in the install dialog that identifies the
certificate that signed the add-on. Note that this isn't even the dialog
that Firefox uses by default - you have to flip
"xpinstall.customConfirmationUI" to false to see it. In the default
dialog in Firefox, there is no difference between an unsigned add-on and
an add-on signed by a non-Mozilla root certificate that has been trusted
for code signing (and note that soon no certificate in Mozilla's root CA
program will have the code signing trust bit enabled by default [0])
(and again, this only applies to builds where add-on signing isn't
required - for builds where it is required, this API is not used at all).

Given all this, the question is do we still need this second API? Does
Thunderbird or SeaMonkey use it for any reason, or can we simplify the
code-base, reduce build size, etc.?

Cheers,
David

[0] https://bugzilla.mozilla.org/show_bug.cgi?id=1366243



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: 64-bit Firefox progress report: 2017-07-18

[Mike Hommey]

On Thu, Jul 20, 2017 at 12:01:04AM +0200, Jean-Yves Avenard wrote:
> Hi
> 
> On Wed, Jul 19, 2017 at 9:01 AM, Mike Hommey  wrote:
> > What's the plan for eligible people that still want to keep 32-bit
> > Firefox? Are they going to have to stop auto upgrades, which would get
> > them automatically on 64-bits and upgrade manually? This is especially
> > going to be a problem for users with less than 2GB RAM that do still
> > want 32-bit Firefox if we decide against the minimum memory requirement.
> 
> Just curious.
> 
> What would be the rationale behind this choice?

Smaller memory footprint, which, you'll admit, when you're on a machine
with (less than) 2GB RAM, makes a difference.

Mike
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Phabricator Update, July 2017

[Randell Jesup]

>On 2017-07-14 1:31 AM, Jim Blandy wrote:
>> Many people seem to be asking, essentially: What will happen to old bugs?
>> I'm trying to follow the discussion, and I'm not clear on this myself.
>>
>> For example, "Splinter will be turned off." For commenting and reviewing,
>> okay, understood. What about viewing patches on old bugs?
>
>Migration-specific issues are still in flux, since we have still have
>plenty of time to sort them out.  I'll be clarifying the migration plan as
>we get closer to turning off old tools.  So far we've agreed that keeping
>Splinter in read-only mode to view old patches is totally fine, and Dylan
>(BMO lead) mentioned that there are even nicer diff viewers that we can
>integrate with BMO, like  https://diff2html.xyz/, which is currently in use
>by Red Hat's Bugzilla installation.

Good! your recent posting made me believe you'd gone back to "splinter
must be totally turned off".  Thanks.  I still have significant
questions about how security-bug access and review (and security of such
items) will work; is there a design?  What has changed since our
discussion in May, and have you met with the security engineers and
major security reviewers/triagers?

Thanks

-- 
Randell Jesup, Mozilla Corp
remove "news" for personal email
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Phabricator Update, July 2017

[Randell Jesup]

>On 7/13/17 9:04 PM, Mark Côté wrote:
>> It is also what newer systems
>> do today (e.g. GitHub and the full Phabricator suite)
>
>I should note that with GitHub what this means is that you get discussion
>on the PR that should have gone in the issue, with the result that people
>following the issue don't see half the relevant discussion. In particular,
>it's common to go off from "reviewing this line of code" to "raising
>questions about what the desired behavior is", which is squarely back in
>issue-land, not code-review land.

Yes, exactly.  Once the data is in two places, there's no way to avoid
some discussions occuring in the "wrong" place.  And Github shows it
clearly happens all the time, so you still have to flip back and forth
between looking at N review-streams of comments, and the bug, and maybe
looking at the dates of comments, in order to understand everything.

>Unfortunately, I don't know how to solve that problem without designating a
>"central point where all discussion happens" and ensuring that anything
>outside it is mirrored there...

Ditto - I think it's inherent if you can reply to review comments in the
separate tool.  Making the review-tool *worse* for commenting
(features/ease-wise) can actually kinda help, at the cost of hurting
responding to review comments.  Googe's Issue via codereview tooling
kinda takes this approach, from what I've seen, with *very*
non-integrated patches vs bugs.

GPS's comments about the pain of syncing also makes total sense, in that
MozReview tried to mostly attach to existing API points of bugzilla and
do things noone anticipated would be done.

I don't think limiting comments in phabricator is a viable option, nor
making it less-featured, so we may have to eat the pain and lash people
with wet noodles if they post in the "wrong" place. 

-- 
Randell Jesup, Mozilla Corp
remove ".news" for personal email
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Firefox data platform & tools update, Q2 2017

[Georg Fritzsche]

The data platform and tools teams are working on our core Telemetry system,
the data pipeline, providing core datasets and maintaining some central
data viewing tools.

To make new work more visible, we provide quarterly updates.
What’s new in the last few months?

A lot of work in the last months was on reducing latency, supporting
experimentation and providing a more reliable experience of the data
platform.




On the data collection side, we have significantly improved reporting
latency from Firefox 55, with preliminary results from Beta showing we receive
95% of the "main" ping within 8 hours

(compared to previously over 90 hours). Curious for more detail? #1

and #2

should have you covered.

We also added a "new-profile" ping
,
which gives a clear and timely signal for new clients.

There is a new API to record active experiments in Firefox
.
This allows annotating experiments or interesting populations in a standard
way.

The record_in_processes is now required for all histograms. This removes
ambiguity about which process they are recorded in.

The data documentation moved to a new home: docs.telemetry.mozilla.org. Are
there gaps in the documentation you want to see filled? Let us know by filing
a bug

.

For datasets, we added telemetry_new_profile_parquet, which makes the data
from the "new-profile" ping available.

Additionally, the main_summary dataset now includes all scalars and uses a
whitelist for histograms, making it easy to add them. Important fields like
active_ticks and Quantum release criteria were also added and backfilled.

For custom analysis on ATMO ,
cluster lifetimes can now be extended self-serve in the UI. The stability
of scheduled job stability also saw major improvements.

There were first steps towards supporting Zeppelin notebooks better; they
can now be rendered as Markdown
 in Python.

The data tools work is focused on making our data available in a more
accessible way. Here, our main tool re:dash
 saw multiple improvements.

Large queries should no longer show the slow script dialog
. A new Athena data source was
introduced, which contains a subset of our Telemetry-based derived
datasets. This brings huge performance and stability improvements over
Presto.

Finally, scheduled queries can now have an expiration date.
What is up next?

For the next few months, interesting projects in the pipeline include:

   -

   The experiments viewer & pipeline, which will make it much easier to run
   pref-flipping experiments in Firefox.
   -

   Recording new probes from add-ons into the main ping (events, scalars,
   histograms).
   -

   We will define and monitor basic guarantees for the Telemetry client
   data (like reporting latency ranges).
   -

   A re-design of about:telemetry is currently on-going, with more
   improvements on the way.
   -

   A first version of Mission Control will be available, a tool for more
   real-time release monitoring.
   -

   Analyzing the results of the Telemetry survey, (thanks everyone!) to
   inform our planning.
   -

   Extending the main_summary dataset to include all histograms.
   -

   Adding a pre-release longitudinal dataset, which will include all
   measures on those channels.
   -

   Looking into additional options to decrease the Firefox data reporting
   latency.

How to contact us.

Please reach out to us with any questions or concerns.

   -

   You can find us on IRC in #telemetry and #datapipeline.
   -

   We are available on slack in #fx-metrics.
   -

   The main mailing list for data topics is fx-data-dev
   .
   -

   Bugs can be filed in one of these components
   .
   -

   You can also find us on Twitter as @MozTelemetry
   .
___
dev-platform 

Re: 64-bit Firefox progress report: 2017-07-18

[Chris Peterson]

On 2017-07-19 10:18 AM, Mike Hoye wrote:

On 2017-07-19 3:58 AM, Chris Peterson wrote:

On 2017-07-19 12:01 AM, Mike Hommey wrote:

What's the plan for eligible people that still want to keep 32-bit
Firefox? 
Outside of our QA team (or others orgs, I guess?) do we have a set of 
use cases that would motivate people to flip that switch?


64-bit code is slightly larger, so there is some memory overheard. Users 
with less than 4 GB RAM might feel that 32-bit is faster on their 
machine, but our testing on Windows 7 and 10 machines with just 2 GB RAM 
doesn't show any measurable performance differences. We don't expect 
64-bit Firefox to have any performance improvements over 32-bit. The 
benefits of 64-bit are primarily ASLR and fewer OOM crashes, which are 
somewhat intangible to end users.




Are they going to have to stop auto upgrades, which would get
them automatically on 64-bits and upgrade manually? This is especially
going to be a problem for users with less than 2GB RAM that do still
want 32-bit Firefox if we decide against the minimum memory 
requirement.


We have two options in mind: 
Is ESR on the radar while you're planning this, or is it unrelated? 


When the next ESR comes around (59?), we will announce to the ESR 
mailing list that 64-bit is considered stable and the preferred version, 
but we do not plan to auto migrate 32-bit ESR users to 64-bit. We figure 
that enterprises will want to test and control their deployments.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: 64-bit Firefox progress report: 2017-07-18

[Mike Hoye]

On 2017-07-19 3:58 AM, Chris Peterson wrote:


On 2017-07-19 12:01 AM, Mike Hommey wrote:

What's the plan for eligible people that still want to keep 32-bit
Firefox? 
Outside of our QA team (or others orgs, I guess?) do we have a set of 
use cases that would motivate people to flip that switch?




Are they going to have to stop auto upgrades, which would get
them automatically on 64-bits and upgrade manually? This is especially
going to be a problem for users with less than 2GB RAM that do still
want 32-bit Firefox if we decide against the minimum memory requirement.


We have two options in mind: 

Is ESR on the radar while you're planning this, or is it unrelated?

- mhoye
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: 64-bit Firefox progress report: 2017-07-18

[Chris Peterson]

On 2017-07-19 12:01 AM, Mike Hommey wrote:

What's the plan for eligible people that still want to keep 32-bit
Firefox? Are they going to have to stop auto upgrades, which would get
them automatically on 64-bits and upgrade manually? This is especially
going to be a problem for users with less than 2GB RAM that do still
want 32-bit Firefox if we decide against the minimum memory requirement.


We have two options in mind:

1. An opt-out registry key (piggybacking on an updater feature mhowell 
or rstrong is developing). We would document this registry key before 
the 56 release so 32-bit users would have time to set it and prevent 
migration.


2. 56 will be a watershed release (for multiple reasons), so all users 
updating from old versions will get 56.0 before then updating to any 
future updates. We will limit the migration to exactly one version 
(probably a 56.0.1) so 32-bit users who were migrated but don't want 
64-bit can manually re-install 32-bit 56.0.1 and not be re-migrated in a 
later update.




Other than end users, I can imagine it being a problem for developers or
QA at some point.

Also, what about beta, developer edition and nightly?


Our current plan is to not migrate 32-bit Nightly users to 64-bit. We 
haven't made a decision about Beta or Developer Edition users. About 60% 
of Beta users don't actually know they are on the Beta channel, so they 
are not typical pre-release testers and would benefit from 64-bit. With 
the opt-out registry key and the 56.0.1 watershed migration, there are 
options for developers and testers who don't want to be migrated. Given 
the options, is there any strong reason to not migrate Beta and/or 
Developer Edition users?

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: 64-bit Firefox progress report: 2017-07-18

[Mike Hommey]

On Tue, Jul 18, 2017 at 11:38:56PM -0700, Chris Peterson wrote:
> We are on track to make 64-bit Firefox the default build for Win64 OS,
> bringing improved ASLR and fewer OOM crashes to the 70% of Windows Firefox
> users running Win64.
> 
> PLANS:
> 
> * In Firefox 55 (August 8), the Windows stub installer will default to
> 64-bit Firefox for eligible users (Win64 and 2+ GB RAM).
> 
> * In Firefox 56 (September 26), we will migrate existing eligible 32-bit
> Firefox users to 64-bit. About 70% of Windows Firefox users currently run
> 32-bit Firefox build on Win64. Nearly all of these users can be migrated to
> 64-bit Firefox.

What's the plan for eligible people that still want to keep 32-bit
Firefox? Are they going to have to stop auto upgrades, which would get
them automatically on 64-bits and upgrade manually? This is especially
going to be a problem for users with less than 2GB RAM that do still
want 32-bit Firefox if we decide against the minimum memory requirement.
Other than end users, I can imagine it being a problem for developers or
QA at some point.

Also, what about beta, developer edition and nightly?

Mike
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

64-bit Firefox progress report: 2017-07-18

[Chris Peterson]

We are on track to make 64-bit Firefox the default build for Win64 OS, 
bringing improved ASLR and fewer OOM crashes to the 70% of Windows 
Firefox users running Win64.


PLANS:

* In Firefox 55 (August 8), the Windows stub installer will default to 
64-bit Firefox for eligible users (Win64 and 2+ GB RAM).


* In Firefox 56 (September 26), we will migrate existing eligible 32-bit 
Firefox users to 64-bit. About 70% of Windows Firefox users currently 
run 32-bit Firefox build on Win64. Nearly all of these users can be 
migrated to 64-bit Firefox.


Our Win64 project wiki has more details about the long road to making 
64-bit Firefox the default:


https://wiki.mozilla.org/Firefox/Win64

PROGRESS:

* David Parks fixed the insidious Flash video streaming bug affecting 
Xfinity and MLB! (bug 1334803)


* Bob Owen fixed the sandbox issue that prevented 64-bit Firefox from 
being run from a network-mounted drive! (bug 1377555)


* Some highlights from week 2 of our Firefox 54 experiment comparing 32- 
and 64-bit Firefox users:


- About 8% of Windows users have <= 2 GB RAM!

- User retention and engagement metrics (session length, # of pages, # 
of tabs) are about the same for 32- and 64-bit Firefox users, regardless 
of memory size.


- 64-bit content process crash rate is about the same as 32-bit for 
users with 2 GB and about 20% less with 2+ GB!


- 64-bit browser process crash rate is about the same as 32-bit for 
users with 2 GB and about 20% less with 2+ GB!


- 64-bit plugin process crash rate is about 50% less than 32-bit for 
users with 2 GB and 80% less with 2+ GB!


We are still considering whether 64-bit Firefox should have a default 
minimum memory requirement. As a placeholder value, Firefox 55 currently 
requires 2+ GB, but based on the results of our experiment, we may 
remove the minimum memory requirement. Microsoft's minimum memory 
require for Win64 itself is 2 GB, so anyone running Win64 with less than 
2 GB is having a bad time.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform