Re: Bug Handling in Bugzilla over the End of the Year

[Kim Moir]

Thank you Emma.  As a further note, you can indicate in Phabricator that
you aren't accepting reviews using the calendar feature.  See this link for
more details.

https://wiki.mozilla.org/Phabricator/FAQ#How_do_I_mark_myself_.27Out_Of_Office.27_to_stop_review_requests.3F

Kim

On Fri, Dec 20, 2019 at 2:31 AM Emma Humphries  wrote:

> As you're winding down work on Firefox at the end of the year, there's a
> few things to remember.
>
> First, there will be coverage for Bugzilla. Staff will be monitoring #bmo
> on Slack and cloud operations will be watching over Bugzilla.
>
> Second, if you see something, tag something. If you run across comments or
> attachments that are spam or violate our Community Participation Guidlines,
> please tag them as 'spam' or 'admin' accordingly, and we will review them.
>
> Last, if you are taking time away from Bugzilla over the holidays:
>
> * Update your display name in preferences with the days you are out
> * Set yourself to decline needinfo and review/feedback requests while you
> are away
>
> If you are celebrating this time of year, I wish you a wonderful time with
> your friends and families.
>
> -- Emma Humphries, Bugmaster
>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

deprecating mozilla-inbound

[Kim Moir]

Effective Thursday November 28, 2019, mozilla-inbound will be deprecated
for active development.  The repository will be changed to read-only for
historical reference.

You can refer to this bug for updates
https://bugzilla.mozilla.org/show_bug.cgi?id=inbound-outbound

Please reach out in the #conduit or #firefox-ci channels on Slack if you
have questions.

Thank you mozilla-inbound for your many years of service to the Mozilla
community.

Kim
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Upcoming changes to hg.mozilla.org access

[Kim Moir]

Just a note to let everyone know the hg.mozilla.org permissions change was
implemented this morning.  Thanks to everyone for the work that allowed us
to make our code repositories more secure.

Kim

On Fri, Nov 1, 2019 at 4:39 PM Kim Moir  wrote:

> The Engineering Workflow team enabled a hook in July which asked people to
> provide a reason for directly pushing to hg.mozilla.org.  Since it was
> enabled, we have seen the number of direct pushes decrease to a few per
> week.
>
> Enabling developers to use standard tools to land reviewed code through a
> secure pipeline also allows us to enable new workflow capabilities such as
> running static analyzers, linters, and code formatting tools, while making
> hg.mozilla.org more secure by reducing the number of people who can
> access it directly. It also paves the way for decommissioning
> mozilla-inbound, which will simplify our tree management and reduce
> infrastructure costs.
>
> On Nov 14, 2019, we intend to change the permissions associated with Level
> 3 access to revoke direct push access to hg.mozilla.org on
> mozilla-inbound, mozilla-central, mozilla-beta, mozilla-release and esr
> repos. If you do need a patch landed outside the Phabricator/Lando pipeline
> such as in the case of bustage, please use the #checkin-needed process
> https://wiki.mozilla.org/Sheriffing/How_To/Landing_checkin-needed_patches
> and contact the sheriffs in #sheriffs in Slack or IRC to land your patch.
>
> Specific teams will retain direct access to hg.mozilla.org (now called
> Level 4) such as Sheriffs and Release Management. We expect everyone else
> to use the Phabricator/Lando pipeline, but exceptions may be granted for
> special situations with director-level approval. If this applies to you,
> please follow up with your manager.
>
> The Engineering Workflow team is committed to supporting and improving the
> productivity of developers working on Firefox. If you have questions or
> need help with tooling, please reach out to us in the #phabricator Slack
> channel.
>
> Kim
>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Upcoming changes to hg.mozilla.org access

[Kim Moir]

Officially decommissioning m-i will take place after we change the
permissions.  It will remain a read-only repo for historical purposes. No I
don't see a need to run things in CI on m-i beyond that date.
deprecate mozilla-inbound after Lando is used for most mozilla-central
landing
https://bugzilla.mozilla.org/show_bug.cgi?id=1530401

Kim

On Fri, Nov 1, 2019 at 4:56 PM Andrew Halberstadt  wrote:

> Very nice!
>
> Does this mean mozilla-inbound is effectively decommissioned at this
> point? Are there any circumstances it will need to run things in CI beyond
> November 14th?
>
> -Andrew
>
>
> On Fri, Nov 1, 2019 at 4:39 PM Kim Moir  wrote:
>
>> The Engineering Workflow team enabled a hook in July which asked people to
>> provide a reason for directly pushing to hg.mozilla.org.  Since it was
>> enabled, we have seen the number of direct pushes decrease to a few per
>> week.
>>
>> Enabling developers to use standard tools to land reviewed code through a
>> secure pipeline also allows us to enable new workflow capabilities such as
>> running static analyzers, linters, and code formatting tools, while making
>> hg.mozilla.org more secure by reducing the number of people who can
>> access
>> it directly. It also paves the way for decommissioning mozilla-inbound,
>> which will simplify our tree management and reduce infrastructure costs.
>>
>> On Nov 14, 2019, we intend to change the permissions associated with Level
>> 3 access to revoke direct push access to hg.mozilla.org on
>> mozilla-inbound,
>> mozilla-central, mozilla-beta, mozilla-release and esr repos. If you do
>> need a patch landed outside the Phabricator/Lando pipeline such as in the
>> case of bustage, please use the #checkin-needed process
>> https://wiki.mozilla.org/Sheriffing/How_To/Landing_checkin-needed_patches
>> and contact the sheriffs in #sheriffs in Slack or IRC to land your patch.
>>
>> Specific teams will retain direct access to hg.mozilla.org (now called
>> Level 4) such as Sheriffs and Release Management. We expect everyone else
>> to use the Phabricator/Lando pipeline, but exceptions may be granted for
>> special situations with director-level approval. If this applies to you,
>> please follow up with your manager.
>>
>> The Engineering Workflow team is committed to supporting and improving the
>> productivity of developers working on Firefox. If you have questions or
>> need help with tooling, please reach out to us in the #phabricator Slack
>> channel.
>>
>> Kim
>> ___
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Upcoming changes to hg.mozilla.org access

[Kim Moir]

The Engineering Workflow team enabled a hook in July which asked people to
provide a reason for directly pushing to hg.mozilla.org.  Since it was
enabled, we have seen the number of direct pushes decrease to a few per
week.

Enabling developers to use standard tools to land reviewed code through a
secure pipeline also allows us to enable new workflow capabilities such as
running static analyzers, linters, and code formatting tools, while making
hg.mozilla.org more secure by reducing the number of people who can access
it directly. It also paves the way for decommissioning mozilla-inbound,
which will simplify our tree management and reduce infrastructure costs.

On Nov 14, 2019, we intend to change the permissions associated with Level
3 access to revoke direct push access to hg.mozilla.org on mozilla-inbound,
mozilla-central, mozilla-beta, mozilla-release and esr repos. If you do
need a patch landed outside the Phabricator/Lando pipeline such as in the
case of bustage, please use the #checkin-needed process
https://wiki.mozilla.org/Sheriffing/How_To/Landing_checkin-needed_patches
and contact the sheriffs in #sheriffs in Slack or IRC to land your patch.

Specific teams will retain direct access to hg.mozilla.org (now called
Level 4) such as Sheriffs and Release Management. We expect everyone else
to use the Phabricator/Lando pipeline, but exceptions may be granted for
special situations with director-level approval. If this applies to you,
please follow up with your manager.

The Engineering Workflow team is committed to supporting and improving the
productivity of developers working on Firefox. If you have questions or
need help with tooling, please reach out to us in the #phabricator Slack
channel.

Kim
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Lando service outage

[Kim Moir]

The Lando service has been restored and patches are landing on trees.  A
big thanks to Cloud Ops and other folks who helped to debug and resolve
this issue. A post-mortem will be conducted to investigate steps to
mitigate the issue going forward.

Kim


On Thu, Sep 19, 2019 at 11:37 AM Kim Moir  wrote:

> The Lando service is currently down due to Google Cloud Platform related
> issues.  See https://bugzilla.mozilla.org/show_bug.cgi?id=1582383
> <https://slack-redir.net/link?url=https%3A%2F%2Fbugzilla.mozilla.org%2Fshow_bug.cgi%3Fid%3D1582383=3>
>  for
> status.
>
> Cloud Services Operations engineers have escalated the issue with Google
> are working with their engineers to resolve the issue.  Patches that have
> been reviewed in Phabricator will be queued and not land on the trees until
> this issue is resolved.
>
> Please reach out in #lando or #conduit in slack if you have questions.
>
> Kim
>
>
>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Lando service outage

[Kim Moir]

The Lando service is currently down due to Google Cloud Platform related
issues.  See https://bugzilla.mozilla.org/show_bug.cgi?id=1582383

for
status.

Cloud Services Operations engineers have escalated the issue with Google
are working with their engineers to resolve the issue.  Patches that have
been reviewed in Phabricator will be queued and not land on the trees until
this issue is resolved.

Please reach out in #lando or #conduit in slack if you have questions.

Kim
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: logging direct hg.m.o pushes

[Kim Moir]

Just a reminder that this hook will be enabled tomorrow Thursday July 25.

As an update to the previous announcement, this line should have read

"(To override, in your head commit, include the literal string, "MANUAL
PUSH:" followed by a sentence of justification.)"

head commit vs top commit in the previous announcement.

Kim

On Wed, Jul 17, 2019 at 10:12 AM Kim Moir  wrote:

> On July 25, 2019 the Engineering Workflow team will enable a hook to
> require an explanation string for direct (non-Lando) pushes to autoland,
> mozilla-inbound and mozilla-central repositories. The intent of this change
> is to shift usage to our automated infrastructure, and better understand
> any remaining barriers to its adoption.
>
> On average, 85% of mozilla-central commits reviewed in Phabricator are
> landed by Lando. That indicates that Lando is working well for most
> engineers, and we’d like to drive this number closer to 100% to unlock new
> workflow capabilities such as running static analyzers, linters, and code
> formatting tools through this pipeline. This will also reduce operational
> risk by landing reviewed code automatically. However, we also want to avoid
> creating roadblocks in any edge-case situations where the tooling fails -
> so we’re keeping a manual fallback in place for use in exceptional
> circumstances.
>
> What is an acceptable reason to push directly to autoland, mozilla-inbound
> or mozilla-central?
>
> In brief, if something in Lando is actively preventing you from getting
> work done, you may push directly to hg.mozilla.org. This includes
> prolonged outages, specific breakage or a time-sensitive emergency. The
> other possible reason for pushing directly is that the patches you are
> pushing aren’t supported by Lando (for instance a large amount of vendored
> code).  You will be asked to provide a brief explanation of why you are
> pushing directly as part of the “{hg,git} push” command. (To override, in
> your top commit, include the literal string, "MANUAL PUSH:" followed by a
> sentence of justification.)
>
> Note that we will be logging all manual pushes both to ensure this
> functionality is reserved for exceptional circumstances, and to identify
> problems that are blocking people from using Lando.  Six weeks after the
> hook is enabled, we’ll provide a summary of the issues identified and
> relevant bug reports to this list.
>
> What about checkin-needed?
>
> We will be using a very similar process to Bugzilla, in which users can
> use a #checkin-needed tag in Phabricator to request that a sheriff or other
> user with appropriate privileges land their patches. Follow bug 1514807
> <https://bugzilla.mozilla.org/show_bug.cgi?id=1514807> for updates on
> this process or read the checkin-needed documentation
> <https://wiki.mozilla.org/Sheriffing/How_To/Landing_checkin-needed_patches>
> .
>
>
> If you have questions, please reach out in the #phabricator channel in
> slack or irc.
>
> Kim
>
>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

logging direct hg.m.o pushes

[Kim Moir]

On July 25, 2019 the Engineering Workflow team will enable a hook to
require an explanation string for direct (non-Lando) pushes to autoland,
mozilla-inbound and mozilla-central repositories. The intent of this change
is to shift usage to our automated infrastructure, and better understand
any remaining barriers to its adoption.

On average, 85% of mozilla-central commits reviewed in Phabricator are
landed by Lando. That indicates that Lando is working well for most
engineers, and we’d like to drive this number closer to 100% to unlock new
workflow capabilities such as running static analyzers, linters, and code
formatting tools through this pipeline. This will also reduce operational
risk by landing reviewed code automatically. However, we also want to avoid
creating roadblocks in any edge-case situations where the tooling fails -
so we’re keeping a manual fallback in place for use in exceptional
circumstances.

What is an acceptable reason to push directly to autoland, mozilla-inbound
or mozilla-central?

In brief, if something in Lando is actively preventing you from getting
work done, you may push directly to hg.mozilla.org. This includes prolonged
outages, specific breakage or a time-sensitive emergency. The other
possible reason for pushing directly is that the patches you are pushing
aren’t supported by Lando (for instance a large amount of vendored code).
You will be asked to provide a brief explanation of why you are pushing
directly as part of the “{hg,git} push” command. (To override, in your top
commit, include the literal string, "MANUAL PUSH:" followed by a sentence
of justification.)

Note that we will be logging all manual pushes both to ensure this
functionality is reserved for exceptional circumstances, and to identify
problems that are blocking people from using Lando.  Six weeks after the
hook is enabled, we’ll provide a summary of the issues identified and
relevant bug reports to this list.

What about checkin-needed?

We will be using a very similar process to Bugzilla, in which users can use
a #checkin-needed tag in Phabricator to request that a sheriff or other
user with appropriate privileges land their patches. Follow bug 1514807
 for updates on this
process or read the checkin-needed documentation
.


If you have questions, please reach out in the #phabricator channel in
slack or irc.

Kim
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Changes to autoland repository configuration

[Kim Moir]

If you don’t pull from the autoland repository, you can stop reading.

The autoland repository has a “special” configuration: Until its changesets
are merged into mozilla-central, they are in the mercurial ‘draft’
phase[1][2].  On Thursday May 9, 2019 we will change the repository to be
configured like mozilla-inbound. After this change is implemented all
changesets landed by Lando to autoland will be in the public state. Also,
running `hg wip` after pulling autoland locally will no longer cause issues.

If you have questions about this change, please reach out in the #lando
slack channel. If you find an issue related to this change, file a
regressed by bug against
https://bugzilla.mozilla.org/show_bug.cgi?id=1538695

[1] https://hg.mozilla.org/integration/autoland/

[2] https://www.mercurial-scm.org/wiki/Phases


Kim
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Kim Moir]

The tracking bug for sccache being deployed to offices is here

https://bugzilla.mozilla.org/show_bug.cgi?id=1524533

Kim

On Tue, Apr 2, 2019 at 3:22 AM Frederik Braun  wrote:

> Am 01.04.19 um 22:16 schrieb Chris M.:
> > Hi Emilio, if you're interested you're encouraged to try it out, however
> > we're shipping machines to offices now to run the schedulers, the first
> of
> > which I'll be testing in the SF office this week, so we're planning an
> > officially supported setup in the near future.
> >
>
> That's great news. Can we follow progress or detailed plans somewhere?
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Moving reviews to Phabricator

[Kim Moir]

Just another reminder,

On February 28 Bugzilla review flags will be disabled for Firefox and other
mozilla-central products and components. From this point forward, all
reviews of code changes to mozilla-central should be conducted in
Phabricator.

Please reach out in the #phabricator channel in irc or Slack if you have
questions.

Kim

On Wed, Feb 6, 2019 at 3:42 PM Kim Moir  wrote:

> On February 28 Bugzilla review flags will be disabled for Firefox and
> other mozilla-central products and components. From this point forward, all
> reviews of code changes to mozilla-central should be conducted in
> Phabricator. Tasks that have been identified as crucial to this transition
> have been set as blocker bugs to https://bugzil.la/1514775.
>
> 
>
> FAQ
>
> Phabricator transition
>
> Are we ready for this?
>
> The weekly average number of commits to mozilla-central that were reviewed
> in Phabricator has been hovering around 80% for many weeks now.  While the
> system can always be improved, this number indicates that most engineers
> are able to use Phabricator effectively. The Engineering Workflow team has
> discussed this change with the remaining high-volume users of the old
> workflow to ensure there are no showstoppers with Phabricator. If you think
> there may be something we’ve missed, please get in touch with us directly. .
>
> What products and components will have the review flag disabled?
>
> The list includes the Core, Firefox, Firefox Build System, NSS, Geckoview
> and Toolkit products. It will also include some components of the Release
> Engineering and Testing products. It is entirely possible that we will miss
> some products/components, but users should not treat such omissions as
> invitations to continue to use Bugzilla for mozilla-central reviews.
> Adjustments will be made over time as necessary. Any code that will be
> landed to mozilla-central should be reviewed in Phabricator.
>
> Why are we making this change?
>
> Requiring Phabricator for code reviews will allow us to improve code
> quality by running linters and static analysis tools automatically on
> patches.  It will also allow us to simplify and standardize our engineering
> workflow by reducing the number of request queues that developers are
> expected to monitor.  The previous post (
> https://groups.google.com/forum/#!topic/mozilla.dev.platform/JKbDxHSaVfM) 
> announcing
> the general availability of Phabricator has further details and links to
> documentation.
>
> What about uplifts?
>
> Initially, only code landing in mozilla-central will be required to be
> reviewed in Phabricator. Once we have implemented more processes in
> Phabricator, we will move reviews of all of the mozilla-central “branches”
> to Phabricator.
>
> I have an unusual use case using patches and Bugzilla, how do I use
> Phabricator to do this?
>
>
> We’ve put together documentation
> https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html on how
> to accomplish various common tasks in Phabricator. If you’re doing
> something unusual that truly can’t be done with Phabricator, you can of
> course still attach a file to Bugzilla, and use a needinfo or IRC ping to
> get sign-off from the reviewer. We expect these cases to be exceedingly
> rare, and ask that developers use Phabricator wherever possible to unlock
> the benefits described above.  Please also reference the Bugzilla actions
> and their equivalents on Phabricator
> https://wiki.mozilla.org/Phabricator/Bugzilla_Equivalents documentation.
>
> Where can I reach out if I have questions?
>
> Please reach out in the #phabricator channel in irc or Slack.
>
>
>
> Kim
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Moving reviews to Phabricator

[Kim Moir]

On February 28 Bugzilla review flags will be disabled for Firefox and other
mozilla-central products and components. From this point forward, all
reviews of code changes to mozilla-central should be conducted in
Phabricator. Tasks that have been identified as crucial to this transition
have been set as blocker bugs to https://bugzil.la/1514775.



FAQ

Phabricator transition

Are we ready for this?

The weekly average number of commits to mozilla-central that were reviewed
in Phabricator has been hovering around 80% for many weeks now.  While the
system can always be improved, this number indicates that most engineers
are able to use Phabricator effectively. The Engineering Workflow team has
discussed this change with the remaining high-volume users of the old
workflow to ensure there are no showstoppers with Phabricator. If you think
there may be something we’ve missed, please get in touch with us directly. .

What products and components will have the review flag disabled?

The list includes the Core, Firefox, Firefox Build System, NSS, Geckoview
and Toolkit products. It will also include some components of the Release
Engineering and Testing products. It is entirely possible that we will miss
some products/components, but users should not treat such omissions as
invitations to continue to use Bugzilla for mozilla-central reviews.
Adjustments will be made over time as necessary. Any code that will be
landed to mozilla-central should be reviewed in Phabricator.

Why are we making this change?

Requiring Phabricator for code reviews will allow us to improve code
quality by running linters and static analysis tools automatically on
patches.  It will also allow us to simplify and standardize our engineering
workflow by reducing the number of request queues that developers are
expected to monitor.  The previous post (
https://groups.google.com/forum/#!topic/mozilla.dev.platform/JKbDxHSaVfM)
announcing
the general availability of Phabricator has further details and links to
documentation.

What about uplifts?

Initially, only code landing in mozilla-central will be required to be
reviewed in Phabricator. Once we have implemented more processes in
Phabricator, we will move reviews of all of the mozilla-central “branches”
to Phabricator.

I have an unusual use case using patches and Bugzilla, how do I use
Phabricator to do this?


We’ve put together documentation
https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html on how
to accomplish various common tasks in Phabricator. If you’re doing
something unusual that truly can’t be done with Phabricator, you can of
course still attach a file to Bugzilla, and use a needinfo or IRC ping to
get sign-off from the reviewer. We expect these cases to be exceedingly
rare, and ask that developers use Phabricator wherever possible to unlock
the benefits described above.  Please also reference the Bugzilla actions
and their equivalents on Phabricator
https://wiki.mozilla.org/Phabricator/Bugzilla_Equivalents documentation.

Where can I reach out if I have questions?

Please reach out in the #phabricator channel in irc or Slack.



Kim
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: arm64 windows nightlies now available

[Kim Moir]

This is fantastic news!  Congratulations on the launch of arm64 windows
nightlies.

Kim

On Wed, Dec 19, 2018 at 7:55 PM Nathan Froyd  wrote:

> I'm excited to announce that we have bona fide arm64 windows nightlies
> available for download!
>
> https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/
>
> featuring full updater and installer support; see the
> firefox-66.0a1.en-US.win64-aarch64* files in that directory.  Thanks
> to Tom Prince for the updater work and Rob Strong and Matt Howell for
> the installer work.
>
> Please note that these builds are even nightlier than our normal
> nightlies on other platforms: they have *not* gone through our usual
> automated testing process, bugs are almost certain to crop up, etc.
> etc.  That being said, I have been using builds off automation
> (manually updating them) for several weeks now and have had a pleasant
> experience.
>
> There are still a few areas that we know need work: the Gecko profiler
> is not functional, but should be by the end of the week.  The
> crashreporter does not work.  Our top-tier JS JIT (IonMonkey) is not
> turned on.  WebRTC is not turned on.  EME (Netflix, etc.) does not
> work yet.  And so forth...Did I mention that these are nightlies?
>
> If you use these builds and you find issues, please file bugs blocking:
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=arm64-windows-bugs
>
> so we can start to triage and prioritize what needs to be fixed.
>
> Happy dogfooding!
> -Nathan
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Workflow Apropos!

[Kim Moir]

For a richer version of this newsletter, please see <
https://docs.google.com/document/d/1uJECrrYitWbuyLJl_W_QHaBJtYcabJsAv2aQdFSmr3E/
>.

Hello from the Engineering Workflow team! This is our first newsletter.
Exciting! But first, who are we and what do we do? The Engineering Workflow
team[1] exists to improve the quality, clarity, and efficiency of Firefox
development through the integration and development of tools and
automation. This includes such things as reporting and querying bugs,
reviewing and committing code, and optimizing our repositories and build
processes. We’re publishing a newsletter so the rest of engineering can get
some insights into what we’re working on and why. Drawing inspiration from
tl;dr, we will present a few short stories highlighting some of our work
each month(ish).



Speeding up moz-phab

moz-phab[2] is Engineering Workflow’s officially supported custom
command-line interface to Phabricator, built in order to better support the
“stacked commits” workflow that is common in Firefox engineering.
Unfortunately some of the design decisions we made early on, such as
wrapping Arcanist in order to reduce long-term maintenance burden, have
made moz-phab painfully slow in some circumstances. We’ve spent some time
doing performance analysis and have put together a plan for
improvements[3], taking some inspiration from phlay[4] (which is Git only)
and phabsend[5] (which is Mercurial only). Phase 0 was completed last
week[6] and released yesterday[7]!



Lando confidentially

Lando[8], can now land confidential revisions[9]. You’ll be able to see a
confidential revision in Lando to which you have access in Phabricator by
providing a Phabricator API key. This user experience isn’t quite as smooth
as we would have preferred, since Phabricator is tied to Bugzilla’s login
(so we can use Bugzilla’s security controls), but Lando is currently tied
to Auth0 (so we can query SCM permissions).

A better future is in sight, however, as Bugzilla is just about to get
OAuth2 support[10], which we’ll be able to use to connect Bugzilla and
Auth0 accounts. We’ll also be implementing some special features around
confidential revisions to help developers follow the proper process[11].



Digging into builds

We've done a lot of work over the years to try to improve Firefox local
builds, but we have always relied on anecdotes from individual developers
to know when a problem exists and when things have improved. Recently we've
finished landing some work[12] to enable the build system to submit
telemetry[13], and we're excited to start getting data that we can use to
prioritize future work and validate our results! The schema of data we're
collecting is available in the build system documentation[14]. We have some
preliminary dashboards[15] of the data we have collected.



Do you have comments on this newsletter? Do you want to know more about us
and the projects we’re working on? You can find us in #engworkflow on IRC
and in project-specific channels including #build, #vcs, #phabricator,
#lando, and #bmo.


1. https://wiki.mozilla.org/Engineering_Workflow
2. https://github.com/mozilla-conduit/review
3.
https://docs.google.com/document/d/14v33dMrAvs9KlhYVdbN7yYqJpubs8XFpnrDlsZTPBW0/
4. https://github.com/mystor/phlay
5. https://www.mercurial-scm.org/repo/hg/file/tip/hgext/phabricator.py
6. https://bugzilla.mozilla.org/show_bug.cgi?id=1504965
7. https://github.com/mozilla-conduit/review/releases/tag/1.8
8. https://lando.services.mozilla.com
9.
https://moz-conduit.readthedocs.io/en/latest/lando-user.html#viewing-a-confidential-revision
10. https://bugzilla.mozilla.org/show_bug.cgi?id=1354589
11. https://wiki.mozilla.org/Security/Bug_Approval_Process
12. https://bugzilla.mozilla.org/show_bug.cgi?id=1237610
13.
https://groups.google.com/forum/%23!topic/mozilla.dev.platform/yfnVeUSIYR0
14. https://firefox-source-docs.mozilla.org/build/buildsystem/telemetry.html
15. https://sql.telemetry.mozilla.org/dashboard/sheehan-build-telemetry-dash
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Scheduled - [TCW] Scheduled Tree Closing Maintenance Window July 14th

[Kim Moir]

As a fyi there will be a tree closing window tomorrow July 14 06:00-15:00PDT

tldr: Trees will be closed for the duration of this maintenance,
bugzilla and mercurial will also be unavailable.




>From https://mozilla2.statuspage.io/incidents/980f0kzk533g

Impact to Users:
Trees will be closed for the duration of this maintenance, bugzilla and
mercurial will also be unavailable. See complete list of work below.

List of work to be completed:

06:00 - ~14:00 PDT:
Bug 1471612 - Bugzilla DB changes (~8 hours)

09:00 - ~13:00 PDT:
Bug 1471319 - hgmo migration from SCL3 to MDC1 (~4 Hours)

~14:00 PST - 15:00 PDT:
Bug 1468437 / CHG0013025 - OpenVPN maintenance (~10 minutes)

Tracker Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1472833

If you have questions about this issue, please email m...@mozilla.com or
visit #moc in IRC. We appreciate your patience while we perform this
maintenance work.

Thank you,
Mozilla Operations Center
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Localized Repacks now visible on (most) pushes

[Kim Moir]

Congratulations Callek!  I know this was a tremendous amount of work on
your side to implement and coordinate. Thank you for all your efforts to
drive this forward, it has unblocked a lot of future l10n work.

Kim

On Wed, May 30, 2018 at 1:57 PM, Gregory Szorc  wrote:

> On Wed, May 30, 2018 at 10:08 AM, Justin Wood  wrote:
>
> > Hello Everyone,
> >
> > tl;dr You should now see "L10n" jobs on treeherder with many pushes,
> these
> > are tier 1 and if they break they would also be breaking Nightly so your
> > patch would need to be backed out.
> >
> > As many of you know, especially the old guard [1] here, Localized Repacks
> > have frequently been known to fail in weird and interesting ways on
> Nightly
> > and Beta builds.
> >
> > Throughout the movement to taskcluster we have been reducing the
> > differences in automation to make what we ship to release users happen
> with
> > the same process as what we ship to nightly users. We have recently
> > achieved that parity now that we have finished our migration to
> taskcluster
> > [2]
> >
> > One straggler was on our implementation of L10n builds on try [3][4]
> which
> > had begun to frequently fail when users add/remove any localized file
> (.dtd
> > or .ftl). And similarly we have always lacked the ability to easily vet a
> > change to central/inbound/autoland as "will this break l10n".
> >
> > With the work I've now done we have aligned this "try" l10n job with what
> > we perform in the Nightly and Release Promotion process, as well as
> allowed
> > ourselves the ability to run these on every push.
> >
> > Implementation details:
> > * For now these still run only when a subset of files change [5] but this
> > list can be expanded easily, or we can rip it out and instead *always*
> run
> > these jobs.
> > * These jobs are performed using live L10n repositories, but just a small
> > set of our total localization, specifically: en-CA, he, it, ja, ja-JP-mac
> > [6]
> > * As part of doing this work, we needed to specify the STUB Installer
> > differently, if we need it on any new channels/builds we need to specify
> it
> > in the build taskcluster kind, like [7]. We have a check in configure to
> > error if its not set correctly [8]
> >
> > If you have any questions, feel free to reach out to me/releng.
> > ~Justin Wood (Callek)
> >
>
> Thank you, Justin and everyone else who worked on this! l10n packaging has
> historically suffered from a lack of visibility in CI and lack of
> understanding outside its small circle of maintainers. Moving the l10n
> automation to Taskcluster and giving it visibility in Treeherder as part of
> regular jobs that normal people see will go a very long way to increasing
> understanding of l10n packaging. It also paves the road for overhauling the
> technical underpinnings of l10n packaging. For those not aware, l10n
> packaging has historically been a major burden for build system maintainers
> because of the convoluted ways it interacts with the build system. Let's
> just say that we have actively avoided touching code related to l10n out of
> fear that it will break a convoluted system. Now that the l10n CI can more
> easily be toggled and tested, it is substantially easier to iterate on and
> we now have the confidence that our changes won't break things. This is a
> game changer and will directly enable us to perform some long-overdue
> refactoring of l10n code. Thank you!
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: 57 beta cycle: 57beta/devedition published earlier + soft freeze

[Kim Moir]

Several releng folks, RyanVM and jcristau had a meeting today discussing
tomorrow's merge

We wanted to ensure that we would be able to have the correct branding for
Fennec beta after the merge.  In order to do this, we will have a relbranch
created for Fennec on Sept 14 named FIREFOX_56b13_RELBRANCH.  We'll use
this branch to build Fennec 56.0b13 on Sept 18. This process will look like
this:

Sept 13th:

   -

   GTB: Desktop and DevEdition 56.0b12

Sept 14:

   -

   Releng Migration
   -

  RyanVM: Create FIREFOX_56b13_RELBRANCH on m-b out of
  FIREFOX_BETA_56_END and manually bump version number to 56.0b13.
  -

 Approved patches for 56 will be landed on both mozilla-release
 (default branch) and mozilla-beta (FIREFOX_56b13_RELBRANCH).
 -

  Merge m-b->m-r normally. All tags will be made. Version numbers will
  be bumped. Branding will be release.
  -

  Merge m-c->m-b


   -

   No action on m-c.
   -

  Prior to m-c->m-b script run, comment out: https://dxr.mozilla.org/
  mozilla-central/source/testing/mozharness/scripts/
  merge_day/gecko_migration.py#329-336
  

  -

 Or undo those steps after the run


Between sept 14->21

   -

   GTB: DevEdition 57.0b1, sometimes this week, off mozilla-beta default
   -

   Uplifts: Regular merges will happen between still 57 m-c, to m-b.
   Preferably once a day.
   -

   Uplifts: Regular uplifts will happen to 56 m-r (and need to be
   backported to fennec 56 branch on m-b)


Sept 18th

   -

   GTB: Desktop 56.0rc1 off m-r default
   -

   GTB: Fennec 56.0b13 off m-b FIREFOX_56b13_RELBRANCH


The discussion is summarized in this document at the end under "Releng
actions"

https://docs.google.com/document/d/1SRjdBNjj4LM0IV4vs1UMrPnp
yhyNTYpJLZhLHg2TZI8

Kim

On Mon, Sep 11, 2017 at 12:28 PM, Ryan VanderMeulen <
rvandermeu...@mozilla.com> wrote:

> In general, I would say your best bet is to have patches landed by mid-day
> PDT the day before if you want to be reasonably confident that it'll get
> merged over to Beta in time. Predicting when merges will happen is
> difficult due circumstances like build/test failures, infra issues, etc.
> It's possible that there will be a final set of autoland/inbound merges to
> m-c on those days ahead of the merge to Beta, but I wouldn't assume they'll
> happen either.
>
> -Ryan
>
> On Sat, Sep 9, 2017 at 6:39 PM, Ed Lee  wrote:
>
>> Is there any guidance on when commits should make it to autoland and
>> inbound for both Merge Days (14th and 21st)? I suppose people should just
>> get things in 24 hours before midnight 00:00 Pacific Time to be "safe" in
>> case there's any bustages (hopefully reduced with the soft freeze).
>>
>> For some context from 56 Merge Day (August 2nd), the last autoland commit
>> that made it to Beta 56 was from August 1st 06:49 Pacific, and the last
>> inbound commit was from 14:12. mozilla-central was tagged at August 2nd
>> 00:35. In this case, autoland had an ~18 hour earlier cutoff and inbound
>> had a ~10 hour earlier cutoff to "make it" without needing
>> approval-beta-56. There were 176 bugs that landed on autoland [1] or
>> inbound [2] while it was still "August 2nd Pacific", and 44 (25%) of those
>> were uplifted. (I suppose not auto-uplifting the other 75% might be worth
>> the overhead of additionally requesting+approving 10% of the 445 bugs
>> uplifted to 56.)
>>
>> Ed Lee
>>
>> [1] https://hg.mozilla.org/integration/autoland/pushloghtml?
>> startdate=2017-08-01+13%3A50=2017-08-03+07
>> [2] https://hg.mozilla.org/integration/mozilla-inbound/pushloght
>> ml?startdate=2017-08-01+21%3A13=2017-08-03+07
>>
>> On Thu, Sep 7, 2017 at 6:07 AM, Sylvestre Ledru  wrote:
>>
>>> Hello,
>>>
>>> tldr: we are going to do a first merge of m-c => m-b and use the
>>> devedition population one week earlier to push 57.0b1 & 57.0b2.
>>>
>>> In order to maximize beta testing, we are going to take advantage of the
>>> fact that we have a different set of users with the developer edition
>>> (which is, since the completion of the dawn project, based on
>>> mozilla-beta).
>>> September 14th, we will merge mozilla-central => mozilla-beta and
>>> mozilla-beta => mozilla-release. September 19th, we will push to the
>>> devedition population the first beta of 57. Later, we will be pushing a
>>> second beta to this channel.
>>> In parallel, to minimize the impact on the developer, mozilla-central
>>> will remain 57.0a1 and will be merged to mozilla-beta transparently.
>>> Nightly will become 58.0a1 on September 21th.
>>> The regular beta population will remain on 56 and will be provided
>>> updates to the RC builds.
>>>
>>> The milestones are detailed in this Google doc:
>>> https://docs.google.com/document/d/1jeypuqBqEyIh-4qxXT0UnE2a
>>> VjetN7uVD8W7L7TbWKg/edit
>>>
>>> In parallel, as the 

reducing high try macosx pending counts

[Kim Moir]

You may have noticed that the time to wait for macosx test results on try
has been very long (>1day) this week.

[tracking] macosx test load is unsustainable
https://bugzilla.mozilla.org/show_bug.cgi?id=1386625

There are several factors that contributed to this situation.
1) Additional macosx stylo tests were enabled on trunk which added
significant test load to a system already running at capacity
2) These tests were enabled before some optimizations were in place to
reduce the number of tests run per push
3) Since jobs on try have a lower priority than jobs on trunk, the trunk
jobs continue to consume the the mac test machine capacity before it can be
used by try jobs

We have taken the following steps to address this problem
1) I have disabled the mac stylo tests on m-i to reduce load (
https://bugzilla.mozilla.org/show_bug.cgi?id=1386625)
2)  The optimizations have been enabled for autoland and m-i branches so
all the tests are not run on every push for the new stylo tests (
https://bugzilla.mozilla.org/show_bug.cgi?id=1386405)
3) Patches are in progress to disable some non-e10s tests to further reduce
load (https://bugzilla.mozilla.org/show_bug.cgi?id=1386689)

We will continue to monitor the situation and implement additional measures
to allow macosx test runs on try to complete in a more reasonable interval.

Please ping us in #releng or follow the tracking bug if you have questions
https://bugzilla.mozilla.org/show_bug.cgi?id=1386625

Kim
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

nightly updates for mac are currently disabled

[Kim Moir]

We were alerted to a problem with nightly updates to the macosx cross
compiled builds on Friday afternoon and updates were disabled that
evening.  Essentially the app is not signed after an update is applied.  If
you download the dmg it is signed, the issue only arises after an update is
applied.

Any builds downloaded as a dmg are signed as expected
https://www.mozilla.org/en-US/firefox/channel/desktop/

This is as result of the switch to taskcluster nightly builds last week.
This issue does not impact taskcluster opt and debug builds on trunk.

We are investigating a possible solution

Nightly on macOS loses code signature after update
https://bugzilla.mozilla.org/show_bug.cgi?id=1375904

Kim Moir
Mozilla Release Engineering
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: switch to macosx cross-compiled builds on taskcluster on trunk

[Kim Moir]

After successful testing this morning, we have now enabled macosx nightly
updates again.

The migration is complete, enjoy your cross compiled builds and more
scalable CI infrastructure.

Kim

On Wed, Jun 21, 2017 at 10:04 PM, Kim Moir <km...@mozilla.com> wrote:

> Status update:
>
> We have successful cross compiled macosx opt builds running on trunk.  We
> also have run a successful nightly cross compiled nightly on m-c.  We are
> going to leave the rule in place that blocks updates for macosx nightlies
> so we can test and update from tonight's nightly to tomorrow's nightly on
> the test channel, before enabling updates for all nightly users.
>
> Kim
>
> On Wed, Jun 21, 2017 at 12:35 PM, Kim Moir <km...@mozilla.com> wrote:
>
>> At 11:00PT today, we will be landing patches to run mac opt builds on
>> trunk as cross compiled builds on Linux machines on taskcluster.  As this
>> change is uplifted to m-c, nightly builds for mac will also switch to run
>> on taskcluster on Linux.  We will be testing to ensure that updates work as
>> expected.  We don’t expect any impact to developers as this has been tested
>> extensively on a project branch as well as by manual testing by PI.
>>
>>
>>
>> If you have questions, please contact us in #releng
>>
>>
>>
>> Enable OSX cross-compile builds as tier1 on mozilla-central
>>
>> https://bugzilla.mozilla.org/show_bug.cgi?id=1374422
>>
>>
>>
>> Land completed OSX cross compile Nightly code to mozilla-central
>>
>> https://bugzilla.mozilla.org/show_bug.cgi?id=1357867
>>
>>
>>
>> Land more Nightly OSX support to central
>>
>> https://bugzilla.mozilla.org/show_bug.cgi?id=1373326
>> <https://bugzilla.mozilla.org/show_bug.cgi?id=1374422>
>>
>>
>>
>> Kim Moir
>>
>> Mozilla Release Engineering
>>
>>
>>
>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: switch to macosx cross-compiled builds on taskcluster on trunk

[Kim Moir]

Status update:

We have successful cross compiled macosx opt builds running on trunk.  We
also have run a successful nightly cross compiled nightly on m-c.  We are
going to leave the rule in place that blocks updates for macosx nightlies
so we can test and update from tonight's nightly to tomorrow's nightly on
the test channel, before enabling updates for all nightly users.

Kim

On Wed, Jun 21, 2017 at 12:35 PM, Kim Moir <km...@mozilla.com> wrote:

> At 11:00PT today, we will be landing patches to run mac opt builds on
> trunk as cross compiled builds on Linux machines on taskcluster.  As this
> change is uplifted to m-c, nightly builds for mac will also switch to run
> on taskcluster on Linux.  We will be testing to ensure that updates work as
> expected.  We don’t expect any impact to developers as this has been tested
> extensively on a project branch as well as by manual testing by PI.
>
>
>
> If you have questions, please contact us in #releng
>
>
>
> Enable OSX cross-compile builds as tier1 on mozilla-central
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=1374422
>
>
>
> Land completed OSX cross compile Nightly code to mozilla-central
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=1357867
>
>
>
> Land more Nightly OSX support to central
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=1373326
> <https://bugzilla.mozilla.org/show_bug.cgi?id=1374422>
>
>
>
> Kim Moir
>
> Mozilla Release Engineering
>
>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

switch to macosx cross-compiled builds on taskcluster on trunk

[Kim Moir]

(I apologize for the late notice on this, I forgot to include dev-platform
in the original earlier email)


At 11:00PT today, we will be landing patches to run mac opt builds on trunk
as cross compiled builds on Linux machines on taskcluster.  As this change
is uplifted to m-c, nightly builds for mac will also switch to run on
taskcluster on Linux.  We will be testing to ensure that updates work as
expected.  We don’t expect any impact to developers as this has been tested
extensively on a project branch as well as by manual testing by PI.



If you have questions, please contact us in #releng



Enable OSX cross-compile builds as tier1 on mozilla-central

https://bugzilla.mozilla.org/show_bug.cgi?id=1374422



Land completed OSX cross compile Nightly code to mozilla-central

https://bugzilla.mozilla.org/show_bug.cgi?id=1357867



Land more Nightly OSX support to central

https://bugzilla.mozilla.org/show_bug.cgi?id=1373326
<https://bugzilla.mozilla.org/show_bug.cgi?id=1374422>



Kim Moir

Mozilla Release Engineering
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform