Re: Workflow Apropos!
James Graham wrote on 2018-11-28 13:20: On 28/11/2018 20:15, Mark Côté wrote: My team has pretty much nothing to do with the gecko GitHub clone; we need to keep our focus on the "standard" workflow. Sure, the problem is that it's an attractive nuisance for new contributors who find it and go "It's a GitHub repo! I know this" without realising it's largely unsupported. I've updated the description of the repo to emphasize that it's unsupported and direct readers to its website for further details. -myk ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Workflow Apropos!
On 11/28/2018 01:20 PM, James Graham wrote: On 28/11/2018 20:15, Mark Côté wrote: * It's not obvious to people that patches can't go up for review without a preexisting bug, and won't actually be reviewed unless they specify a reviewer in the commit message (or go into Phabricator and add a reviewer after the fact). Part of this problem has always existed (knowing to pick a reviewer and who); we've got plans to introduce suggested reviewers into the flow in an even better way than it's done in Bugzilla. Timeline here is a bit uncertain in part because there are some prerequisites. Some system for auto-assigning reviewers where none are provided would be a big win; even as a regular contributor I sometimes make changes to parts of the tree where I have to guess a possible reviewer from the VCS logs. PSA for anyone unaware: on IRC, you can /msg mrgiggles who has reviewed CodeGen.py? and the bot will do that scan for you, looking at a bunch of past revisions, grabbing out the reviewers and weighting them by recency, then giving you the top few to choose from. Give the full path for non-unique filenames. Please be aware that using this blindly will tend to overload the most active reviewers. It also doesn't take authors into account, only reviewers (the logs have the full name of authors and the nicks of reviewers, with no mechanism implemented to correlate them.) bugzilla also has suggested reviewers, and the poorly-named mqext mercurial extension has a reviewers command that will do the above process. Or look at an entire patch and take all of the modified files into account. We could also make moz-phab more helpful when it comes to bugs. And of course there's still the controversial idea of not requiring bugs for all patches that comes up now and again, but that's a (big) policy question. Yeah, I don't have a specific solution to suggest for the bugs thing, but it's a real issue that people have. Maybe there's some compromise where if you send commits for review without a bug the tooling can offer to file one for you using the changed files to guess at the product/component using the metadata in moz.buid files and the commit message to set the bug summary/description. (Not really relevant to anyone.) I never quite got around to landing the code in the soon-to-be-obsolete bzexport extension to do this, though I've been running with it in my local version for several years. It's really handy to be able to make a commit, then file a bug with no additional information and have it guess at something sensible for the bug title, product, and component. (I also never switched it to the moz.build metadata; it looks through the bugs associated with past changes to the modified files instead.) ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Workflow Apropos!
On Wed, Nov 28, 2018 at 09:20:17PM +, James Graham wrote: > On 28/11/2018 20:15, Mark Côté wrote: > > We're still working through a longer-term vision that we'll share early > > next year, but I can answer some questions now. > > Thanks, this is helpful! > > > * Have to make a choice early on about whether to learn a relatively > > unfamiliar (to the majority of developers) VCS (mercurial), use a > > slightly unorthodox git setup with slow initial clone (cinnabar), or > > use > > the largely unsupported (?) GitHub clone. > > > > > > This is a very difficult problem. I can't see this problem going away > > entirely without some sort of executive decision to require everyone use > > a particular VCS. That said, Mercurial should still be seen as the > > default VCS, especially as we get partial-clone support > > (https://bugzilla.mozilla.org/show_bug.cgi?id=150). Git-cinnabar > > should be treated as an "advanced" option. Perhaps docs could be > > clarified as to this. > > I understand that mercurial is not going away :) Having said that it's a > pretty hard sell to get someone to make an initial contribution if it > involves learning a whole new VCS; that's a considerable investment of time > on its own. I wonder if there's something we could do to help people here > like run a taskcluster task on m-c to produce an artifact containing a > cinnabar clone archived using git-bundle, and then set up the bootstrap.py > script to give a choice between mercurial and git using cinnabar, and in the > git case, do the initial clone by downloading a recent bundle and then > fetching missing commits. There's probably a good reason that won't work, > but I'm not sure what it is yet :) https://bugzilla.mozilla.org/show_bug.cgi?id=1459200 is one step in that direction. And git-cinnabar version 0.5.0b4 and newer support cloning from a git reference repo[1]. There is even a mercurial extension[2] to make this more automated, but git-cinnabar won't try to use it by default for now. I want to add support for git bundles, or at least future-proofing for git bundles, first. The reference repo would also need to be updated frequently. Mike 1. For example, you can do: git -c cinnabar.clone=https://github.com/glandium/gecko clone hg::https://hg.mozilla.org/mozilla-unified 2. https://github.com/glandium/git-cinnabar/blob/master/mercurial/cinnabarclone.py ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Workflow Apropos!
On 28/11/2018 20:15, Mark Côté wrote: We're still working through a longer-term vision that we'll share early next year, but I can answer some questions now. Thanks, this is helpful! * Have to make a choice early on about whether to learn a relatively unfamiliar (to the majority of developers) VCS (mercurial), use a slightly unorthodox git setup with slow initial clone (cinnabar), or use the largely unsupported (?) GitHub clone. This is a very difficult problem. I can't see this problem going away entirely without some sort of executive decision to require everyone use a particular VCS. That said, Mercurial should still be seen as the default VCS, especially as we get partial-clone support (https://bugzilla.mozilla.org/show_bug.cgi?id=150). Git-cinnabar should be treated as an "advanced" option. Perhaps docs could be clarified as to this. I understand that mercurial is not going away :) Having said that it's a pretty hard sell to get someone to make an initial contribution if it involves learning a whole new VCS; that's a considerable investment of time on its own. I wonder if there's something we could do to help people here like run a taskcluster task on m-c to produce an artifact containing a cinnabar clone archived using git-bundle, and then set up the bootstrap.py script to give a choice between mercurial and git using cinnabar, and in the git case, do the initial clone by downloading a recent bundle and then fetching missing commits. There's probably a good reason that won't work, but I'm not sure what it is yet :) My team has pretty much nothing to do with the gecko GitHub clone; we need to keep our focus on the "standard" workflow. Sure, the problem is that it's an attractive nuisance for new contributors who find it and go "It's a GitHub repo! I know this" without realising it's largely unsupported. * Cloning the repository doesn't provide you with the right tooling to actually request review on a patch. You have to download something else and — particularly if you wrote the patch as a series of commits — there's a choice of tools at various levels of completeness. If you use something backed by arcanist this probably involves installing system-level dependencies that aren't handled by mach bootstrap. Yes, this is an issue we'll be addressing. The first step is to stop using Arcanist in moz-phab; not only does it introduce other dependencies (PHP) but it is causing some performance issues in moz-phab as well. After that, we can see about installing it via mach bootstrap or such. Sounds good. It would be great if we can get to a place where submitting/updating a review is just `mach review [commits]`, or similar. * It's not obvious to people that patches can't go up for review without a preexisting bug, and won't actually be reviewed unless they specify a reviewer in the commit message (or go into Phabricator and add a reviewer after the fact). Part of this problem has always existed (knowing to pick a reviewer and who); we've got plans to introduce suggested reviewers into the flow in an even better way than it's done in Bugzilla. Timeline here is a bit uncertain in part because there are some prerequisites. Some system for auto-assigning reviewers where none are provided would be a big win; even as a regular contributor I sometimes make changes to parts of the tree where I have to guess a possible reviewer from the VCS logs. We could also make moz-phab more helpful when it comes to bugs. And of course there's still the controversial idea of not requiring bugs for all patches that comes up now and again, but that's a (big) policy question. Yeah, I don't have a specific solution to suggest for the bugs thing, but it's a real issue that people have. Maybe there's some compromise where if you send commits for review without a bug the tooling can offer to file one for you using the changed files to guess at the product/component using the metadata in moz.buid files and the commit message to set the bug summary/description. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Workflow Apropos!
On Wed, Nov 28, 2018 at 1:34 PM James Graham wrote: > Can you share with us the long term vision for what the workflow is > going to look like here? I've recently seen a few cases where > experienced develoeprs who have either never contributed to gecko before > or contribute infrequently tried to get things set up for a patch to get > into review, and it seems like there was a lot of frustration caused by > accidental complexity that's mostly hidden from people who are already > up and running. Some of the issues encountered seemed to be: > We're still working through a longer-term vision that we'll share early next year, but I can answer some questions now. > * Have to make a choice early on about whether to learn a relatively > unfamiliar (to the majority of developers) VCS (mercurial), use a > slightly unorthodox git setup with slow initial clone (cinnabar), or use > the largely unsupported (?) GitHub clone. > This is a very difficult problem. I can't see this problem going away entirely without some sort of executive decision to require everyone use a particular VCS. That said, Mercurial should still be seen as the default VCS, especially as we get partial-clone support ( https://bugzilla.mozilla.org/show_bug.cgi?id=150). Git-cinnabar should be treated as an "advanced" option. Perhaps docs could be clarified as to this. My team has pretty much nothing to do with the gecko GitHub clone; we need to keep our focus on the "standard" workflow. > * Cloning the repository doesn't provide you with the right tooling to > actually request review on a patch. You have to download something else > and — particularly if you wrote the patch as a series of commits — > there's a choice of tools at various levels of completeness. If you use > something backed by arcanist this probably involves installing > system-level dependencies that aren't handled by mach bootstrap. > Yes, this is an issue we'll be addressing. The first step is to stop using Arcanist in moz-phab; not only does it introduce other dependencies (PHP) but it is causing some performance issues in moz-phab as well. After that, we can see about installing it via mach bootstrap or such. > * It's not obvious to people that patches can't go up for review without > a preexisting bug, and won't actually be reviewed unless they specify a > reviewer in the commit message (or go into Phabricator and add a > reviewer after the fact). > Part of this problem has always existed (knowing to pick a reviewer and who); we've got plans to introduce suggested reviewers into the flow in an even better way than it's done in Bugzilla. Timeline here is a bit uncertain in part because there are some prerequisites. We could also make moz-phab more helpful when it comes to bugs. And of course there's still the controversial idea of not requiring bugs for all patches that comes up now and again, but that's a (big) policy question. > I appreciate that moving to new tooling is a tricky process and that's > why there are rough edges at the moment. But it would be really useful > to be able to tell people that the issues they are facing are understood > to be pain points and are going to go away in the future :) > Agreed, stay tuned for more details in 2019Q1. :) Mark ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Workflow Apropos!
On 27/11/2018 17:46, Kim Moir wrote: Speeding up moz-phab moz-phab[2] is Engineering Workflow’s officially supported custom command-line interface to Phabricator, built in order to better support the “stacked commits” workflow that is common in Firefox engineering. Unfortunately some of the design decisions we made early on, such as wrapping Arcanist in order to reduce long-term maintenance burden, have made moz-phab painfully slow in some circumstances. We’ve spent some time doing performance analysis and have put together a plan for improvements[3], taking some inspiration from phlay[4] (which is Git only) and phabsend[5] (which is Mercurial only). Phase 0 was completed last week[6] and released yesterday[7]! Can you share with us the long term vision for what the workflow is going to look like here? I've recently seen a few cases where experienced develoeprs who have either never contributed to gecko before or contribute infrequently tried to get things set up for a patch to get into review, and it seems like there was a lot of frustration caused by accidental complexity that's mostly hidden from people who are already up and running. Some of the issues encountered seemed to be: * Have to make a choice early on about whether to learn a relatively unfamiliar (to the majority of developers) VCS (mercurial), use a slightly unorthodox git setup with slow initial clone (cinnabar), or use the largely unsupported (?) GitHub clone. * Cloning the repository doesn't provide you with the right tooling to actually request review on a patch. You have to download something else and — particularly if you wrote the patch as a series of commits — there's a choice of tools at various levels of completeness. If you use something backed by arcanist this probably involves installing system-level dependencies that aren't handled by mach bootstrap. * It's not obvious to people that patches can't go up for review without a preexisting bug, and won't actually be reviewed unless they specify a reviewer in the commit message (or go into Phabricator and add a reviewer after the fact). I appreciate that moving to new tooling is a tricky process and that's why there are rough edges at the moment. But it would be really useful to be able to tell people that the issues they are facing are understood to be pain points and are going to go away in the future :) ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Workflow Apropos!
For a richer version of this newsletter, please see < https://docs.google.com/document/d/1uJECrrYitWbuyLJl_W_QHaBJtYcabJsAv2aQdFSmr3E/ >. Hello from the Engineering Workflow team! This is our first newsletter. Exciting! But first, who are we and what do we do? The Engineering Workflow team[1] exists to improve the quality, clarity, and efficiency of Firefox development through the integration and development of tools and automation. This includes such things as reporting and querying bugs, reviewing and committing code, and optimizing our repositories and build processes. We’re publishing a newsletter so the rest of engineering can get some insights into what we’re working on and why. Drawing inspiration from tl;dr, we will present a few short stories highlighting some of our work each month(ish). Speeding up moz-phab moz-phab[2] is Engineering Workflow’s officially supported custom command-line interface to Phabricator, built in order to better support the “stacked commits” workflow that is common in Firefox engineering. Unfortunately some of the design decisions we made early on, such as wrapping Arcanist in order to reduce long-term maintenance burden, have made moz-phab painfully slow in some circumstances. We’ve spent some time doing performance analysis and have put together a plan for improvements[3], taking some inspiration from phlay[4] (which is Git only) and phabsend[5] (which is Mercurial only). Phase 0 was completed last week[6] and released yesterday[7]! Lando confidentially Lando[8], can now land confidential revisions[9]. You’ll be able to see a confidential revision in Lando to which you have access in Phabricator by providing a Phabricator API key. This user experience isn’t quite as smooth as we would have preferred, since Phabricator is tied to Bugzilla’s login (so we can use Bugzilla’s security controls), but Lando is currently tied to Auth0 (so we can query SCM permissions). A better future is in sight, however, as Bugzilla is just about to get OAuth2 support[10], which we’ll be able to use to connect Bugzilla and Auth0 accounts. We’ll also be implementing some special features around confidential revisions to help developers follow the proper process[11]. Digging into builds We've done a lot of work over the years to try to improve Firefox local builds, but we have always relied on anecdotes from individual developers to know when a problem exists and when things have improved. Recently we've finished landing some work[12] to enable the build system to submit telemetry[13], and we're excited to start getting data that we can use to prioritize future work and validate our results! The schema of data we're collecting is available in the build system documentation[14]. We have some preliminary dashboards[15] of the data we have collected. Do you have comments on this newsletter? Do you want to know more about us and the projects we’re working on? You can find us in #engworkflow on IRC and in project-specific channels including #build, #vcs, #phabricator, #lando, and #bmo. 1. https://wiki.mozilla.org/Engineering_Workflow 2. https://github.com/mozilla-conduit/review 3. https://docs.google.com/document/d/14v33dMrAvs9KlhYVdbN7yYqJpubs8XFpnrDlsZTPBW0/ 4. https://github.com/mystor/phlay 5. https://www.mercurial-scm.org/repo/hg/file/tip/hgext/phabricator.py 6. https://bugzilla.mozilla.org/show_bug.cgi?id=1504965 7. https://github.com/mozilla-conduit/review/releases/tag/1.8 8. https://lando.services.mozilla.com 9. https://moz-conduit.readthedocs.io/en/latest/lando-user.html#viewing-a-confidential-revision 10. https://bugzilla.mozilla.org/show_bug.cgi?id=1354589 11. https://wiki.mozilla.org/Security/Bug_Approval_Process 12. https://bugzilla.mozilla.org/show_bug.cgi?id=1237610 13. https://groups.google.com/forum/%23!topic/mozilla.dev.platform/yfnVeUSIYR0 14. https://firefox-source-docs.mozilla.org/build/buildsystem/telemetry.html 15. https://sql.telemetry.mozilla.org/dashboard/sheehan-build-telemetry-dash ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
