Quantum Flow Engineering Newsletter #16

[Ehsan Akhgari]

Hi everyone,

It has been almost a month and a half since the last time

that I talked about our progress in fighting sync IPC issues.  So I figured
it's time to prepare another Sync IPC Analysis

report.  Again, unfortunately only the latest data is available in the
spreadsheet.  But here are screenshot of the C++ and JS IPC message pie
charts:




As you can see, as we have made even more progress in fixing more sync IPC
issues, now the document.cookie issue
 is even a larger
relative share of the pie, at 60%.  That is followed by some JS IPC,
PAPZCTreeManager::Msg_ReceiveMouseInputEvent (which is a fast sync IPC
message used by the async pan zoom component which would be hard to
replace), followed by more JS IPC, followed by
PContent::Msg_GetBlocklistState
 which is recently
fixed, followed by PBrowser::Msg_NotifyIMEFocus
, followed by more JS
IPC and CPOW overhead before we get to the longer tail.  If you look at the
JS sync IPC chart, you will see that almost all the overhead there is due
to add-ons.  Hopefully none of this will be an issue after Firefox 57 with
the new out of process WebExtensions for Windows users.  The only message
in this chart stemming from our code that shows up in the data is
contextmenu .

The rate of progress here has been really great to see, and this is thanks
to the hard work of many people across many different teams.  Some of these
issues have required heroic efforts to fix, and it's really great to see
this much progress made in so little time.

The development of Firefox 56 in coming to a close rapidly.  Firefox 57
branches off on Aug 2, and we have about 9 weeks from now until Firefox 57
rides the trains to beta.  So far, according to our burn-down chart
, we have closed
around 224 [qf:p1] bugs and have yet 110 more to fix.  Fortunately Quantum
Flow is not one of those projects that needs all of those bugs to be fixed,
because we may not end up having enough time to fix these bugs for the
final release, especially since we usually keep adding new bugs to the list
in our weekly triage sessions.  Soon we will probably need to reassess the
priority of some of these bugs as the eventual deadline approaches.

It is now time for me to acknowledge the great work of everyone who helped
by contributing performance improvements over the past two weeks.  As
usual, I hope I'm not forgetting any names!

   - Perry Jiang prevented leaving the preferences window open to cause
   expensive periodic checks for the default browser to run off of a timer in
   the background .
   - Kris Maglione cached the add-on blocklist state property in the
   add-ons database for faster retrieval
   .  Additionally,
   he switched away from using IndexedDB to using a simple compressed
   binary flat file for storing the WebExtensions startup cache
   .  He also ensured
   that the Extension.jsm’s promiseLocales() method doesn’t perform
   main-thread I/O .
   Last but not least, he turned on out of process Web Extensions for
   Windows .  This is
   a huge improvement to the responsiveness of the main process by moving the
   code for all the user’s extensions to run out of the main process
   eventually in Firefox 57.  Support for Mac and Linux
    is going to
   follow after 57.
   - Stephen Pohl added support for remote layer trees in popups
   .  This is one of
   the dependencies for out of process WebExtensions
   .
   - Mohammed Yaseen Khan removed support for the -webide command line
   argument , and
   therefore removed one XPCOM component from the critical path of first paint.
   - Olli Pettay made us avoid registering the visited callback when we
   have a pending link update
   , and also made
   our IME support be less eager
    to flush layout.
   - Doug Thayer removed a call to _tzset() on 

Re: wpt CSS tests now running on Linux

[Gregory Szorc]

On Thu, Jul 20, 2017 at 9:33 AM, James Graham 
wrote:

> Bug 1341078 and dependencies just landed on inbound, which means we now
> have the W3C/web-platform-tests CSS tests in-tree and running in
> automation. This adds about 12,000 reftests for CSS features to the
> web-platform-tests suite. They are currently enabled in CI, but only on
> linux*, due to limited capacity on OSX, and issues with the harness on
> Windows. The tests will be enabled on other platforms once these problems
> are resolved.
>
> Servo was already running many of these tests in their automation, so this
> landing plugs a gap in the stylo testing vs upstream.
>
> Note that, as usual with wpt landings, these tests have been vetted for
> stability, but not for the actual results.
>
> Changes to the css tests in this directory will be upstreamed to
> web-platform-tests in the same way as for any other web-platform-test. Note
> that the reftest harness versions of the Mozilla submitted tests are still
> running, so if you want to edit or add to those it is recommended to use
> the copy in layout/reftests/w3c-css/submitted/ since that will be
> correctly synchronised and currently runs on more platforms in CI.
>
> The number of tests and nature of reftests means that this change added a
> large number of files to the repository (around 37,000). Apologies for any
> inconvenience caused by such a large changeset. I'm told that narrow clones
> are just around the corner and may make this kind of thing more tolerable
> in the future.
>

"Around the corner" is a bit optimistic. Mercurial 4.3 has experimental
support for sparse checkouts via an extension. (The feature has been part
of a Facebook extension for years.) I plan to leverage this in automation
to speed up VCS interactions. But end-user support will likely be
experimental until the feature stabilizes upstream, hopefully in 4.4 in 3
months.

In the mean time, be sure to install Watchman (https://facebook.github.io/
watchman/) and the fsmonitor Mercurial extension to make operations faster.
`mach mercurial-setup` should help with this.

Git doesn't yet have filesystem watcher integration, so you are pretty much
at the mercy of your filesystem and I/O subsystem. However, an engineer at
Twitter is currently trying to land Watchman support into the upstream Git
project! The first version of the patches was emailed in March. The 14th
iteration was emailed last week. But there still appears to be active
discussion on some high-level details of the patch series. So I'm not
holding my breath waiting for it to land.

For reference, the best timings I could obtain are as follows. These are
under ideal conditions with as much as possible already in filesystem
caches. If you actually need to perform I/O due to a cache miss (which is
pretty common because common build system activity touches tons of files
and tends to cause cache eviction), the differences with Watchman are even
more pronounced. So it isn't uncommon for Watchman to save a second or two
for random operations.


Linux
$ hg status
no watchman: ~1.00s
w/ watchman: ~0.11s

$ git status
~0.28s

MacOS
$ hg status
no watchman: ~1.78s
w/ watchman: ~0.23s

Windows
$ hg status
no watchman: ~2.31s
w/ watchman: ~0.35s


Linux
$ hg status
no watchman: ~1.16s
w/ watchman: ~0.12s

$ git status
~0.33s

MacOS
$ hg status
no watchman: ~2.04s
w/ watchman: ~0.26s

Windows
$ hg status
no watchman: ~2.56s
w/ watchman: ~0.40s
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Only build the Gecko-dev html parser

[peter279k]

Hi everyone, as title, I want to use the C/C++ to research the Gecko-dev html 
parser.
Is it possible to build the Gecko-dev html parser?

https://github.com/mozilla/gecko-dev/tree/e9fa5c772abe4426c5e33ffe61c438f75f990aca/parser

Thanks.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: wpt CSS tests now running on Linux

[James Graham]

On 20/07/17 18:26, Emilio Cobos Álvarez wrote:

Thanks for this James! \o/

One question, do we run the CSS test linter on automation, or are there
any plans for it?


Yes, that should be run as part of the W lint job (e.g. [1]), which is 
run on pushes (including to try) that change files under 
testing/web-platform/tests/. We don't run it for changes under 
layout/reftests/w3c-css/submitted/ and it's not clear how easy that 
would be, since the lint is rather tied to the structure of the wpt 
repository.


Note that there are other reasons that a push might be blocked upstream 
but land in m-c (e.g. unstable tests). We are working to create an 
upstream PR earlier in the cycle, and improve the communication with 
test authors about problems upstreaming their changes (as well as 
corresponding improvements to downstreaming that should allow us to 
notify relevant people when "interesting" test changes are going to land).


[1]https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound=60fa00c73b05bd2bc0e7485826a86ffed47627c9=testfailed=busted=exception=runnable=pending=running=success=lint=115937895
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: wpt CSS tests now running on Linux

[Emilio Cobos Álvarez]

Thanks for this James! \o/

One question, do we run the CSS test linter on automation, or are there
any plans for it?

We probably should, because otherwise we may only notice when trying to
upstream, like in [1], which is more work for everyone.

[1]:
https://github.com/w3c/web-platform-tests/pull/6357#issuecomment-311778265

On 07/20/2017 06:33 PM, James Graham wrote:
> Bug 1341078 and dependencies just landed on inbound, which means we now
> have the W3C/web-platform-tests CSS tests in-tree and running in
> automation. This adds about 12,000 reftests for CSS features to the
> web-platform-tests suite. They are currently enabled in CI, but only on
> linux*, due to limited capacity on OSX, and issues with the harness on
> Windows. The tests will be enabled on other platforms once these
> problems are resolved.
> 
> Servo was already running many of these tests in their automation, so
> this landing plugs a gap in the stylo testing vs upstream.
> 
> Note that, as usual with wpt landings, these tests have been vetted for
> stability, but not for the actual results.
> 
> Changes to the css tests in this directory will be upstreamed to
> web-platform-tests in the same way as for any other web-platform-test.
> Note that the reftest harness versions of the Mozilla submitted tests
> are still running, so if you want to edit or add to those it is
> recommended to use the copy in layout/reftests/w3c-css/submitted/ since
> that will be correctly synchronised and currently runs on more platforms
> in CI.
> 
> The number of tests and nature of reftests means that this change added
> a large number of files to the repository (around 37,000). Apologies for
> any inconvenience caused by such a large changeset. I'm told that narrow
> clones are just around the corner and may make this kind of thing more
> tolerable in the future.
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

wpt CSS tests now running on Linux

[James Graham]

Bug 1341078 and dependencies just landed on inbound, which means we now 
have the W3C/web-platform-tests CSS tests in-tree and running in 
automation. This adds about 12,000 reftests for CSS features to the 
web-platform-tests suite. They are currently enabled in CI, but only on 
linux*, due to limited capacity on OSX, and issues with the harness on 
Windows. The tests will be enabled on other platforms once these 
problems are resolved.


Servo was already running many of these tests in their automation, so 
this landing plugs a gap in the stylo testing vs upstream.


Note that, as usual with wpt landings, these tests have been vetted for 
stability, but not for the actual results.


Changes to the css tests in this directory will be upstreamed to 
web-platform-tests in the same way as for any other web-platform-test. 
Note that the reftest harness versions of the Mozilla submitted tests 
are still running, so if you want to edit or add to those it is 
recommended to use the copy in layout/reftests/w3c-css/submitted/ since 
that will be correctly synchronised and currently runs on more platforms 
in CI.


The number of tests and nature of reftests means that this change added 
a large number of files to the repository (around 37,000). Apologies for 
any inconvenience caused by such a large changeset. I'm told that narrow 
clones are just around the corner and may make this kind of thing more 
tolerable in the future.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: removing "the old way" of signing add-ons

[Frank-Rainer Grahl]

I know but I don't see any comm-central code which calls the apis in 
question directly or uses the return values. I just searched via dxr but 
would probably take a deeper look at the code to understand it. Might 
not matter anyway in the short time. With the destruction of the classic 
add-on infrastructure in Gecko 57 we need to port web extension support 
first anyway which will take some time. So just another task imho.


FRG

Kris Maglione wrote:

On Thu, Jul 20, 2017 at 10:32:27AM +0200, Frank-Rainer Grahl wrote:

Given all this, the question is do we still need this second API? Does
Thunderbird or SeaMonkey use it for any reason, or can we simplify the
code-base, reduce build size, etc.?


I looked at the code and don't see any use outside of toolkit. Same 
for the PREF_CUSTOM_CONFIRMATION_UI/"xpinstall.customConfirmationUI" 
pref. Seems to be only used in browser.


Please see replies elsewhere in this thread. Seamonkey and Thunderbird 
do use the old signature code that we're talking about removing.


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: removing "the old way" of signing add-ons

[Kris Maglione]

On Thu, Jul 20, 2017 at 10:32:27AM +0200, Frank-Rainer Grahl wrote:

Given all this, the question is do we still need this second API? Does
Thunderbird or SeaMonkey use it for any reason, or can we simplify the
code-base, reduce build size, etc.?


I looked at the code and don't see any use outside of toolkit. Same 
for the PREF_CUSTOM_CONFIRMATION_UI/"xpinstall.customConfirmationUI" 
pref. Seems to be only used in browser.


Please see replies elsewhere in this thread. Seamonkey and 
Thunderbird do use the old signature code that we're talking 
about removing.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Intent to Remove: Insecure use of WebCrypto

[Tim Taubert]

Summary: The WebCrypto spec [1] states that window.crypto.subtle
should only be usable from a secure origin (i.e. on a domain being
served over HTTPS). Currently Gecko's implementation works on insecure
origins (i.e. sites served over unencrypted HTTP). To bring our
implementation in line with the spec, we're going to remove access to
crypto.subtle on non-secure origins.

Sites using the WebCrypto API's crypto.subtle interface on a
non-secure origin should switch to HTTPS as soon as possible.

Chrome too is planning to remove access to crypto.subtle on non-secure
origins [2]. Edge seems positive about implementing those restrictions
as well [3].

Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1333140

Standard: https://w3c.github.io/webcrypto/Overview.html

Platform coverage: This will affect Windows, MacOS, Linux, and Android.

Estimated target date: This could land as early as Firefox 56, but
should in Firefox 57. We probably want to coordinate with Chrome, they
seem as ready as we are.

Our telemetry [4] indicates that about 9% of crypto.subtle use is
still on insecure origins. This was at around 1-2% - that's not the
percentage of all users, but only of those that visit pages that use
crypto.subtle - but became inflated around two weeks after we started
measuring. The blink-dev thread [2] has a good summary and indicates
that this is caused by Twitter launching AMP support serving from
origins which may be insecure. AMP has a fallback that is lazy-loaded
in case crypto.subtle isn't available, so no one's Twitter would break
when we ship this.


[1] https://w3c.github.io/webcrypto/Overview.html#crypto-interface
[2] 
https://groups.google.com/a/chromium.org/forum/#!topic/blink-dev/ZD3NWqkk-bo/discussion
[3] https://github.com/w3c/webcrypto/issues/28#issuecomment-243243989
[4] https://mzl.la/2ttx8aV
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Proposed W3C Charter: Web Commerce Interest Group

[Marcos Caceres]

Hi Tantek,

bcc: Ian Jacobs, who is the W3C Team contact and activity lead.

On July 18, 2017 at 9:58:40 AM, Tantek Çelik (tan...@cs.stanford.edu) wrote:
> I'd like to hear feedback from Marcos (cc'd) on how/why this group
> will complement or help our work in the Web Payments WG (does not seem
> to be addressed by the proposed Commerce IG charter).

I think it's great if that group goes off to collect use cases and
requirements - we have a good 2-5 year gap between seeing adoption of
just "basic-card" and possibly some custom payment solutions (via the
Payment Handler API). So, the IG can collect requirements and
eventually bring them back to the WG.

It seems like an oversight that the charter doesn't say they would
collaborate with the Web Payment WG - as eventually, whatever they do
find, the WG will potentially standardize. We could simply point that
out (hi Ian!), but I honestly just think it was an oversight because
there is so much overlap with the folks from both groups.

HTH!
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: 64-bit Firefox progress report: 2017-07-18

[Jean-Yves Avenard]

Hi

On Wed, Jul 19, 2017 at 9:01 AM, Mike Hommey  wrote:
> What's the plan for eligible people that still want to keep 32-bit
> Firefox? Are they going to have to stop auto upgrades, which would get
> them automatically on 64-bits and upgrade manually? This is especially
> going to be a problem for users with less than 2GB RAM that do still
> want 32-bit Firefox if we decide against the minimum memory requirement.

Just curious.

What would be the rationale behind this choice?

JY
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: removing "the old way" of signing add-ons

[Frank-Rainer Grahl]

Given all this, the question is do we still need this second API? Does
Thunderbird or SeaMonkey use it for any reason, or can we simplify the
code-base, reduce build size, etc.?


I looked at the code and don't see any use outside of toolkit. Same for 
the PREF_CUSTOM_CONFIRMATION_UI/"xpinstall.customConfirmationUI" pref. 
Seems to be only used in browser.


FRG


Frank-Rainer Grahl wrote:
SeaMonkey does not use signing and does not plan to do so. It is 
disabled in the mozconfigs during build time in all trees with:


 > # Disable checking that add-ons are signed by the trusted root
 > MOZ_ADDON_SIGNING=0
 > # Disable enforcing that add-ons are signed by the trusted root
 > MOZ_REQUIRE_SIGNING=0

So I think we don't have a problem here if something gets changed and 
the toolkit code still honors these.


As far as I know Thunderbird does the same.

FRG

David Keeler wrote:

[dev-apps-thunderbird and dev-apps-seamonkey cc'd, but please discuss on
dev-platform]

Hello Everyone,

You may or may not be surprised to learn that Gecko contains two
different ways to verify that an add-on has been signed. The primary
method is nsIX509CertDB.openSignedAppFileAsync. This is what Gecko-based
products that require add-on signing use. However, there is also
nsIZipRreader.getSigningCert (plus some additional glue code).

The only place where these two implementations share code is in the
actual signature verification. That is, the logic to ensure that every
file in the archive has a corresponding valid entry in the manifest (and
that every entry in the manifest has a corresponding file in the
archive) and so on appears in Gecko twice.

 From what I can tell, the actual functionality provided by the second
API (which is only applicable in builds that do not require add-on
signing) is a small text label in the install dialog that identifies the
certificate that signed the add-on. Note that this isn't even the dialog
that Firefox uses by default - you have to flip
"xpinstall.customConfirmationUI" to false to see it. In the default
dialog in Firefox, there is no difference between an unsigned add-on and
an add-on signed by a non-Mozilla root certificate that has been trusted
for code signing (and note that soon no certificate in Mozilla's root CA
program will have the code signing trust bit enabled by default [0])
(and again, this only applies to builds where add-on signing isn't
required - for builds where it is required, this API is not used at all).

Given all this, the question is do we still need this second API? Does
Thunderbird or SeaMonkey use it for any reason, or can we simplify the
code-base, reduce build size, etc.?

Cheers,
David

[0] https://bugzilla.mozilla.org/show_bug.cgi?id=1366243





___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: 64-bit Firefox progress report: 2017-07-18

[Chris Peterson]

On 2017-07-19 6:58 PM, Mike Hommey wrote:

I don't understand why that would be, but if so it should show in
crashstats as fewer small OOMs on these devices.  Does the data actually
show that?


I don't know. Can that be filtered?


I'm not sure I'm answering the right question, but searching through 
crash reports from Firefox 53/54 users over the last 30 days, I see 
small OOMs from both 32- and 64-bit users, regardless of available 
physical or virtual memory or browser uptime. But small OOMs are always 
the top crash (10% or more) for 32-bit users. Small OOMs are about 6% 
for 64-bit users with <= 2 GB, 3% with <= 4 GB, and 1% for > 4 GB.


Here is a search for 32-bit crash reports with <= 2 GB physical memory:
https://is.gd/205aNb

And 64-bit crash reports with <= 2 GB physical memory:
https://is.gd/AezNaZ

But I see a lot of strange crash reports, such as small OOMs from 64-bit 
users within 10 seconds of starting Firefox. Or small OOMs from 64-bit 
users with 9 GB of available physical memory and 128 TB of available 
virtual memory.




I'm not saying they shut down because of memory. I'm saying a surprising
lot of people have browser sessions that don't last more than 5 minutes
(I've heard multiple times in the past years that we found that out from
our data).
Those people are not going to see OOM.


Users with only 2 GB and 5 minute browser sessions would probably have a 
faster user experience with 32-bit Firefox than with 64-bit, but how do 
we weigh that experience versus the security benefits of ASLR?

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: removing "the old way" of signing add-ons

[Frank-Rainer Grahl]

SeaMonkey does not use signing and does not plan to do so. It is 
disabled in the mozconfigs during build time in all trees with:


> # Disable checking that add-ons are signed by the trusted root
> MOZ_ADDON_SIGNING=0
> # Disable enforcing that add-ons are signed by the trusted root
> MOZ_REQUIRE_SIGNING=0

So I think we don't have a problem here if something gets changed and 
the toolkit code still honors these.


As far as I know Thunderbird does the same.

FRG

David Keeler wrote:

[dev-apps-thunderbird and dev-apps-seamonkey cc'd, but please discuss on
dev-platform]

Hello Everyone,

You may or may not be surprised to learn that Gecko contains two
different ways to verify that an add-on has been signed. The primary
method is nsIX509CertDB.openSignedAppFileAsync. This is what Gecko-based
products that require add-on signing use. However, there is also
nsIZipRreader.getSigningCert (plus some additional glue code).

The only place where these two implementations share code is in the
actual signature verification. That is, the logic to ensure that every
file in the archive has a corresponding valid entry in the manifest (and
that every entry in the manifest has a corresponding file in the
archive) and so on appears in Gecko twice.

 From what I can tell, the actual functionality provided by the second
API (which is only applicable in builds that do not require add-on
signing) is a small text label in the install dialog that identifies the
certificate that signed the add-on. Note that this isn't even the dialog
that Firefox uses by default - you have to flip
"xpinstall.customConfirmationUI" to false to see it. In the default
dialog in Firefox, there is no difference between an unsigned add-on and
an add-on signed by a non-Mozilla root certificate that has been trusted
for code signing (and note that soon no certificate in Mozilla's root CA
program will have the code signing trust bit enabled by default [0])
(and again, this only applies to builds where add-on signing isn't
required - for builds where it is required, this API is not used at all).

Given all this, the question is do we still need this second API? Does
Thunderbird or SeaMonkey use it for any reason, or can we simplify the
code-base, reduce build size, etc.?

Cheers,
David

[0] https://bugzilla.mozilla.org/show_bug.cgi?id=1366243



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform