Welcome to the February & March edition of the Engineering Effectiveness &
OS integration Newsletter! The Engineering Effectiveness org makes it easy
to develop, test and release Mozilla software at scale. See below for some
highlights, then read on for more detailed info!
Highlights
-
PDF.js will ship in Fenix 111! Say goodbye to downloading PDFs on
Android and enjoy reading them in Firefox with maximum security!
-
We are also one of the few readers on Android to support filling
forms, JS in PDF and soon editing
-
A huge performance issue in Windows Defender’s Real-time Protection
feature has been identified
<https://bugzilla.mozilla.org/show_bug.cgi?id=1441918#c87> and reported
<https://bugzilla.mozilla.org/show_bug.cgi?id=1441918#c82>. The patch
from Microsoft will be released soon and appears to reduce CPU usage
from MsMpEng.exe by 75%
<https://bugzilla.mozilla.org/show_bug.cgi?id=1441918#c91> when Firefox
is running. There were also additional fixes
<https://bugzilla.mozilla.org/show_bug.cgi?id=1788004> and workarounds
<https://bugzilla.mozilla.org/show_bug.cgi?id=1733532> for various
crashes and performance issues
<https://bugzilla.mozilla.org/show_bug.cgi?id=1823412> caused by
third-party software on Windows.
-
Code to provide Glean Crash Ping Telemetry in Fenix
<https://bugzilla.mozilla.org/show_bug.cgi?id=1810951> has landed! This
means Fenix now has reliable stability metrics.
-
The minimum Python version for Mach and all commands has been bumped to
3.7 <https://bugzilla.mozilla.org/show_bug.cgi?id=1734402>! This
unblocks a ton of pain points and bugs that were held up due to dependency
conflicts.
-
Windows 11 is now a top tier test platform, running the full gamut of
tests in CI <https://bugzilla.mozilla.org/show_bug.cgi?id=1794900>.
Contributors
-
John Pangas (jpangas)
-
Xuanwo
-
Yogesh Singla (singla007)
-
Anurag Bhandari (WhiteWolf47)
-
Elena McLeod (ElusiveEllie)
-
Joshua Hassan (skynette)
-
Prerna Dabi (prernadabi23)
-
Srishti Gupta (srishtig2412)
Detailed Project UpdatesBugzilla and Bugbug
-
Suhaib Mujahid reduce the swinging on topcrash bugs by making autonag
more restrictive <https://github.com/mozilla/relman-auto-nag/pull/1852>
when re-adding topcrash keywords.
-
Suhaib Mujahid refactored autonag to use the Firefox Trains
<https://whattrainisitnow.com/> API instead of parsing a wiki page
<https://wiki.mozilla.org/index.php?title=Release_Management/Calendar>,
thanks to Pascal Chevrel for building the API!
-
Marco Castelluccio presented @ FOSDEM 2023 - Teaching machines to handle
bugs and test Firefox more efficiently
<https://fosdem.org/2023/schedule/event/firefox_testing/>.
-
John Pangas (first contribution🌟) prepared Bugbug to upgrade to a newer
version of scikit-learn <https://github.com/mozilla/bugbug/pull/3300>.
-
Suhaib Mujahid gave a talk @ Montreal Software Test & Automation Meetup
- Automate tedious tasks in the bug management process at Mozilla
<https://www.meetup.com/software-test-automation/events/291587479/>.
-
glob updated the Bugzilla home page <https://bugzilla.mozilla.org/home>,
adding links to download Firefox Beta and Nightly and refreshing the design.
-
Dave Lawrence added an endpoint to Bugzilla
<https://bugzilla.mozilla.org/show_bug.cgi?id=1802052> that
automatically comments on bugs when an associated pull request on GitHub
lands
-
Suhaib Mujahid implemented a new feature
<https://github.com/mozilla/relman-auto-nag/pull/1916> for autonag that
requests missing information when moving a bug to the Core::Performance
component.
-
We kicked off an effort to reduce open S2 platform bugs by 75%, closing
out around 80 bugs so far.
Build System and Mach Environment
-
Alex Hochheiden added caching to configure compiler checks (15-50%
speed-up on subsequent runs of configure with the same compiler version).
-
Alex Hochheiden bumped the minimum Python version
<https://bugzilla.mozilla.org/show_bug.cgi?id=1734402> for Mach to 3.7
-
Mike Hommey updated the Rust compiler to 1.68
<https://bugzilla.mozilla.org/show_bug.cgi?id=1805805>
-
Mike updated the build system
<https://bugzilla.mozilla.org/show_bug.cgi?id=1821221> to build
libunwind & compiler-rt for Android and upgraded the Android NDK to
r23c <https://bugzilla.mozilla.org/show_bug.cgi?id=1820876>.
-
Serge Guelton improved null build run time
<https://bugzilla.mozilla.org/show_bug.cgi?id=1806617>
-
Alex Hochheiden landed work that restores the ability to vendor Python
code into mozilla-central
CI and Treeherder
-
Sebastian Hengst, Joel Maher and Suhaib Mujahid identified one of the
main reasons for Treeherder performance issues
<https://github.com/mozilla/ci-and-quality-tools/issues/100> (you
know…those pesky times when it closes the trees) landed some fixes and
planned some work.
-
Joel Maher has been reducing the volume of errors
<https://bugzilla.mozilla.org/show_bug.cgi?id=1815963> Treeherder parses
(helps with TH database perf) by >50% already
-
Geoff Brown, Joel Maher and Jonathan Moss migrated Windows 7 from AWS ->
Azure <https://bugzilla.mozilla.org/show_bug.cgi?id=1804790>.
-
Joel Maher finished the pixel2 -> pixel5 migration
<https://bugzilla.mozilla.org/show_bug.cgi?id=1794895> (thanks to Jamie
Nicol for the fix to wrench
<https://bugzilla.mozilla.org/show_bug.cgi?id=1795592> jobs!)
-
Joel Maher and Marco Castelluccio got consensus to turn off Windows
10/Aarch64 tests <https://bugzilla.mozilla.org/show_bug.cgi?id=1813991>
in CI.
-
Joel Maher and Jonathan Moss have turned on win11 tests
<https://bugzilla.mozilla.org/show_bug.cgi?id=1794900> in CI, we will
continue to run a subset on win10
<https://bugzilla.mozilla.org/show_bug.cgi?id=1817884>.
-
Suhaib Mujahid and Joel Maher had the first cycle of variant expiration
<https://docs.google.com/document/d/1U17fEx9D1lRjJMfjNrGPjjbQCOy1qQjjuj9gufsXW9c/edit#heading=h.efozttw5p6fn>.
In Bug 1816141 <https://bugzilla.mozilla.org/show_bug.cgi?id=1816141>,
we tracked the first expiration/renewal cycle:
-
17 Renewed for 6 months
-
3 Removed
-
100% response rate!
-
Sylvestre Ledru created a sccache github action
<https://github.com/marketplace/actions/sccache-action>.
-
It allows very simple usage of sccache with GitHub storage for faster
builds
-
For example, servo is now using sccache actions
<https://github.com/servo/servo/pull/29572> for their builds.
-
Sylvestre Ledru released version 0.4.0 of sccache
<https://github.com/mozilla/sccache/releases/tag/v0.4.0>. This version
brings many more error checks, leverages OpenDAL
<https://github.com/apache/incubator-opendal> for storage access and
other improvements. Thanks to Xuanwo for all the hard work.
-
Marco Castelluccio made some CI artifacts expire sooner, saving dozens
of TBs of cloud storage
-
Glob updated the Build Telemetry Dashboard
<https://sql.telemetry.mozilla.org/dashboard/build> to include “mach
try” latency
<https://sql.telemetry.mozilla.org/queries/90584?p_date=d_last_30_days#224317>
-
Andrew Halberstadt implemented
<https://github.com/taskcluster/taskgraph/commit/5df1c816e24f5ae2f6b36e13de3d2108d147a628>
a taskgraph init subcommand to help projects get bootstrapped
<https://taskcluster-taskgraph.readthedocs.io/en/latest/#getting-started>
with Taskgraph more quickly. He also created a firefox-ci-playground
<https://github.com/mozilla-releng/firefox-ci-playground> repository for
anyone who wants to try Taskgraph out.
-
Andrew Halberstadt created a generic “mozilla” trust domain and pools.
This will allow new projects to get set up much quicker by using
pre-defined resources instead of blocking on project specific ones.
Crash Management
-
Alex Franchuk landed the code to provide Glean Crash Ping Telemetry in
Fenix <https://bugzilla.mozilla.org/show_bug.cgi?id=1810951>. This means
Fenix now will also have reliable stability metrics
<https://mozilla.cloud.looker.com/dashboards/1114>.
Lint, Static Analysis and Code Coverage
-
Valentin Rigal has been making steady progress towards before/after
analysis in the code review bot
-
Michelle Goossens has migrated the code review and the code coverage
services from AWS to GCP
-
Andrew Halberstadt replaced our Python linters
<https://bugzilla.mozilla.org/show_bug.cgi?id=1811850> (flake8, isort,
pylint) with Ruff <https://github.com/charliermarsh/ruff>, a
comprehensive linter written in Rust. Python linting now takes one or two
seconds instead of minutes!
OS Integration and Security
-
Bob Owen landed the code to enable Low Privileged Application Container
support <https://bugzilla.mozilla.org/show_bug.cgi?id=1793972> which we
will use to sandbox some media decoders.
-
Yannis Juglaret identified
<https://bugzilla.mozilla.org/show_bug.cgi?id=1441918#c87> and reported
<https://bugzilla.mozilla.org/show_bug.cgi?id=1441918#c82> a huge
performance issue in Windows Defender’s Real-time Protection feature. The
patch from Microsoft will be released soon and appears to reduce CPU
usage from MsMpEng.exe by 75%
<https://bugzilla.mozilla.org/show_bug.cgi?id=1441918#c91> when Firefox
is running. He also landed a series of fixes
<https://bugzilla.mozilla.org/show_bug.cgi?id=1788004> and workarounds
<https://bugzilla.mozilla.org/show_bug.cgi?id=1733532> for various
crashes and performance issues
<https://bugzilla.mozilla.org/show_bug.cgi?id=1823412> caused by
third-party software on Windows.
-
Stephen A Pohl landed support for macOS session resume
<https://bugzilla.mozilla.org/show_bug.cgi?id=639707> after restarts of
macOS, for example as a result of OS updates. We plan to use telemetry to
measure how this affects users getting back into Firefox after macOS OS
updates. Currently, we are observing that roughly 1% of Firefox starts are
occurring after a macOS restart vs. users manually starting Firefox:
-
Greg Stoll wrote a blog post on hacks about third-party DLLs: Letting
users block injected third-party DLLs in Firefox - Mozilla Hacks - the Web
developer blog
<https://hacks.mozilla.org/2023/03/letting-users-block-injected-third-party-dlls-in-firefox/>
(reddit discussion 1
<https://www.reddit.com/r/firefox/comments/127mpu1/letting_users_block_injected_thirdparty_dlls_in/>,
reddit discussion 2
<https://www.reddit.com/r/programming/comments/126vpe1/letting_users_block_injected_thirdparty_dlls_in/>
)
PDF.js
-
PDF.js will ship in Fenix 111, say goodbye to downloading PDFs on
Android, enjoy reading them in Firefox with maximum security!
-
We are also one of the few readers on Android to support filling
forms, JS in PDF and soon editing
Power use
-
Florian Quèze talked at FOSDEM to explain the work we are doing to
understand
and reduce Firefox’s power use
<https://fosdem.org/2023/schedule/event/mozilla_firefox_energy_use/>. He
also explained how power profiling works
<https://fosdem.org/2023/schedule/event/energy_power_profiling_firefox/>
in the Firefox Profiler.
Phabricator , moz-phab, and Lando
-
Zeid Zabaneh significantly reduced the time it takes Lando to load large
stacks.
-
Zeid Zabaneh implemented various changes in Lando in preparation for
revision worker, including fixes to merge conflict detection and
improvements to how Phabricator data is cached
Release Engineering and Management
-
Ben Hearsum fixed mach try
<https://bugzilla.mozilla.org/show_bug.cgi?id=1816440>, so that –artifact
implies –disable-pgo which makes it easier to get tests or other jobs
that depend on shippable builds to run quickly.
-
Geoff Brown and Johan Lorenzo migrated Fenix (Android) to the new
Android Monorepo (bug 1803130
<https://bugzilla.mozilla.org/show_bug.cgi?id=1803130>). Now
android-components, Focus, and Fenix all reside and are all built in the
same repo.
-
RelMan identified a major regression
<https://bugzilla.mozilla.org/1819096> with the Denmark Digital ID
system via the Local Firefox project started last quarter. One day later,
we shipped a planned dot release with an additional fix for this regression.
-
RelEng, Release SRE and RelMan handled a Chain of Trust rotation
incident immediately prior to shipping Firefox 110. These rotations
normally take months, but we were able to get it done in two days and avoid
delaying the release.
-
Johan Lorenzo ensured Geckoview nightly builds twice daily
<https://bugzilla.mozilla.org/show_bug.cgi?id=1817052>. He also made
Android Fenix/Focus Nightlies and Betas follow the same cadence as desktop
Firefox.
-
Johan Lorenzo gave a talk (in French) at PyCon France - Can a bunch of
Python make Firefox less prone to supply chain attacks?
<https://www.pycon.fr/2023/en/talks/30m.html#une-bonne-quantite-de-python-p>.
It details our use of Taskcluster, Taskgraph and Chain of Trust.
-
Pascal Chevre refreshed and migrated the Release Management Calendar
page from https://wiki.mozilla.org/Release_Management/Calendar (manually
edited) to https://whattrainisitnow.com/calendar/ (automated). The
redirect from the wiki will be set up soon.
-
Pascal Chevre wrote a proof of concept called BzKarma
<https://github.com/pascalchevrel/bzkarma>. It creates an impact field
for bugs that gives a value score for uplifting patches based on a bug
metadata. Early testing proves interesting and seems to highlight some
potential valuable uplifts.
Version Control
-
Connor Sheehan, Zeid Zabaneh, and Frida Kiriakos performed a security
audit of pash
<https://github.com/mozilla/version-control-tools/tree/master/hgserver/pash>
(the code that runs when you ssh into hg.mozilla.org). While no new
issues were found, a number of code quality and test coverage issues were
identified and resolved by Connor Sheehan.
-
Connor Sheehan and Christopher Knowles added rate limiting to
hg.mozilla.org, which will reduce the load impact of spam and improper
service use.
mozregression
-
Zeid Zabaneh <[email protected]> fixed a codesign issue
<https://bugzilla.mozilla.org/show_bug.cgi?id=1817001> when using
mozregression with Thunderbird.
Other
-
Rob Lemley added markdown support
<https://bugzilla.mozilla.org/show_bug.cgi?id=1820780> when exporting
mots.yaml.
-
Zeid Zabaneh improved support for subcommands
<https://bugzilla.mozilla.org/show_bug.cgi?id=1814407> in mots when
using them via mach.
-
Zeid Zabaneh fixed an issue
<https://github.com/mozilla/pyo3-parsepatch/pull/54> with rs-parserpatch
builds that were failing when using the latest rust toolchain.
Org Changes
-
The fuzzing team moved to the Security organization
-
Johan Lorenzo is now the manager of the Release Engineering team
-
Andrew Halberstadt has been promoted to P5 (Sr Staff Software Engineer)
-
Gabriele Svelto has been promoted to P5 (Sr Staff Software Engineer)
-
Stephen Pohl has been promoted to P5 (Sr Staff Software Engineer)
-
Ray Kraesig has been promoted to P3 (Sr Software Engineer)
-
Alex Hochheiden has been promoted to P3 (Sr Software Engineer)
Thanks for reading and see you next month!
--
You received this message because you are subscribed to the Google Groups
"[email protected]" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/CAAJAz%2B4LK4dqKdAW0PmW7navM2dAjnM8PLTnifPisZiGA632ag%40mail.gmail.com.
