Re: Intent To Require Manifests For Vendored Code In mozilla-central
thanks to everyone who provided feedback for this proposal. looks like general consensus is to proceed with the plan. i've filed bug 1454867 to track the work. -glob glob wrote: The plan is to create a YAML file for each library containing metadata such as the homepage url, vendored version, bugzilla component, etc. See https://goo.gl/QZyz4xfor the full specification. this should be: https://goo.gl/QZyz4x for the full specification. -- glob — engineering workflow — moz://a ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent To Require Manifests For Vendored Code In mozilla-central
Mark Banner wrote: There are some directories where we only import a file from a third-party and it is alongside other files in that same directory, e.g. testing/modules has "ajv-4.1.1.js" and "sinon-2.3.2.js" imported from elsewhere. How do they fit into this scheme? they would have to be moved into their own directories. One thing we do need is a way for the build system to give us a list of files and directories that are from third-parties, imported, etc. Can that be provided as a result of this work? yes; that list would be useful to multiple consumers. scoping vendoring to a directory level makes generation of that list easier and quicker. -- glob — engineering workflow — moz://a ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent To Require Manifests For Vendored Code In mozilla-central
Axel Hecht wrote: One thing I'm missing is the ability to do mono-repo imports. Say we want to vendor in https://github.com/projectfluent/fluent.js/tree/master/fluent-gecko. i'm not sure i understand the specific concerns around mono-repos; you can use exclude/include to perform a selective import. since fluent.js is a node project i suspect the best option right now is to manually vendor the externally built jsm artefacts, with a minimal moz.yaml documenting the source. once we support node in-tree we can revisit how best to deal with automatic vendoring of the source. For js libraries, we might also want to pay attention to .npmignore (others already mentioned hg, so also .hgignore). i'm not sure how this is relevant. why would there be files in the source repo that have been excluded from source control? There's no spec what happens with patches that fail to apply, or failed run_after scripts. vendoring would require all steps to be successful. Do we intend to do something if the LICENSE changes? Also, what are we supposed to do if the vendored code doesn't have a LICENSE file? currently you can't just vendor anything into mozilla-central; there's a list of compatible licenses. moz.yaml will make it possible to audit and later enforce that vendored code honours our licensing requirements. this already happens for rust crates. licens...@mozilla.org is the point of contact for these sorts of issues. -- glob — engineering workflow — moz://a ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent To Require Manifests For Vendored Code In mozilla-central
On Tue, Apr 10, 2018 at 6:00 AM, Mark Banner wrote: > There are some directories where we only import a file from a third-party > and it is alongside other files in that same directory, e.g. > testing/modules has "ajv-4.1.1.js" and "sinon-2.3.2.js" imported from > elsewhere. > > How do they fit into this scheme? > Just quick comment: I was the one who added our vendored version of Ajv (in bug 1267919), and the fact that it's a single file is just because it wasn't clear to me how else to use it. I listed how to update it here https://searchfox.org/mozilla-central/rev/6bfadf95b4a6aaa8bb3b2a166d6c3545983e179a/testing/modules/ajv-4.1.1.js#34-36 but it's clearly an awkward process, IMO preserving that process would be worse than finding a better way to do it (and presumably that's where this discussion is headed). I suspect something similar is true for sinon, but can't speak for it. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent To Require Manifests For Vendored Code In mozilla-central
On Tue, Apr 10, 2018 at 6:41 AM, glob wrote: >> You don't permit the use of a tag for vendoring, is that intentional? > > to echo gps and mike's responses use of a sha to is preferred over tags. Maybe. We currently use tags. Think about the usage model. If the process is to author the YAML, then run a tool to vendor the identified code, the opportunity for mischief is small. It depends on whether you consider this to be defense against attack, or a user interface. I was thinking the latter. Presumably every change to the YAML would be reviewed and tested. I'm sure that users can be trained to run `git ls-remote`, but it would be better to consider the UX trade-offs at least a little. Simple fix: have the vendoring tool add the hash if a tag is specified. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent To Require Manifests For Vendored Code In mozilla-central
A couple of comments: One thing I'm missing is the ability to do mono-repo imports. Say we want to vendor in https://github.com/projectfluent/fluent.js/tree/master/fluent-gecko. For js libraries, we might also want to pay attention to .npmignore (others already mentioned hg, so also .hgignore). There's no spec what happens with patches that fail to apply, or failed run_after scripts. Do we intend to do something if the LICENSE changes? Also, what are we supposed to do if the vendored code doesn't have a LICENSE file? Axel Am 10.04.18 um 06:25 schrieb glob: mozilla-central contains code vendored from external sources. Currently there is no standard way to document and update this code. In order to facilitate automation around auditing, vendoring, and linting we intend to require all vendored code to be annotated with an in-tree YAML file, and for the vendoring process to be standardised and automated. The plan is to create a YAML file for each library containing metadata such as the homepage url, vendored version, bugzilla component, etc. See https://goo.gl/QZyz4xfor the full specification. We will work with teams to add moz.yaml files where required, as well as adding the capability for push-button vendoring of new revisions. Please address comments to the dev-platform list. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent To Require Manifests For Vendored Code In mozilla-central
Hi all, On Mon, Apr 9, 2018 at 9:25 PM, glob wrote: > mozilla-central contains code vendored from external sources. Currently > there is no standard way to document and update this code. In order to > facilitate automation around auditing, vendoring, and linting we intend to > require all vendored code to be annotated with an in-tree YAML file, and > for the vendoring process to be standardised and automated. > > The plan is to create a YAML file for each library containing metadata > such as the homepage url, vendored version, bugzilla component, etc. See > https://goo.gl/QZyz4x for the full specification. > Generally, I think this is a great plan, and I'm pleased to see it moving forward. A few notes: 1) for the Node.js work I'm actively pursuing, I expect a |mach vendor rust|-like solution rather than a mozvendor.yaml-like solution (at least at first). This is because we need to integrate with Node-y workflows, where things are captured in package.json files -- a situation parallel to Rust, where things are captured in Cargo.toml files. 2) Firefox for Android vendored many dependencies into mobile/android/third_party, modified them, and has more-or-less not updated them since. That's seen as a problem, and some folks are pushing to upgrade those dependencies (see https://bugzilla.mozilla.org/show_bug.cgi?id=1438716). However, those upgrades will be captured as regular Android Maven dependencies and not as mozvendor.yaml dependencies. I don't think either of those things are controversial, just adding to the list of "things a little outside the system" on day one. Best, Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent To Require Manifests For Vendored Code In mozilla-central
Tom Tromey wrote: this should be: https://goo.gl/QZyz4x for the full specification. Some code in DevTools is vendored by dropping webpack bundles into the tree. The bundles are created by running a yarn command in the source repository; this also copies the bundle into an M-C tree. If these directories are going to have a moz.yaml, then either this process will have to be changed upstream, or the moz.yaml spec will have to change to account for this. for now we're focusing on places where we're vendoring in source. manifests for projects where we vendor in artefacts (eg. devtools, activity stream, screenshots, etc) will need to take a different approach, which will be addressed later (this is the primary reason for the manifest having a 'schema' field). -- glob — engineering workflow — moz://a ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent To Require Manifests For Vendored Code In mozilla-central
I've spoken to glob about this offline; but just wanted to note: Our fledgling 'Third Party Library Audit' project is planning on using this metadata (even if the library itself isn't completely vendored) for checking for security issues in upstream and auto-filing bugs. -tom On Mon, Apr 9, 2018 at 11:25 PM, glob wrote: > mozilla-central contains code vendored from external sources. Currently > there is no standard way to document and update this code. In order to > facilitate automation around auditing, vendoring, and linting we intend to > require all vendored code to be annotated with an in-tree YAML file, and for > the vendoring process to be standardised and automated. > > > The plan is to create a YAML file for each library containing metadata such > as the homepage url, vendored version, bugzilla component, etc. See > https://goo.gl/QZyz4x for the full specification. > > > We will work with teams to add moz.yaml files where required, as well as > adding the capability for push-button vendoring of new revisions. > > > Please address comments to the dev-platform list. > > -- > glob — engineering workflow — moz://a > > > ___ > firefox-dev mailing list > firefox-...@mozilla.org > https://mail.mozilla.org/listinfo/firefox-dev > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent To Require Manifests For Vendored Code In mozilla-central
>> this should be: https://goo.gl/QZyz4x for the full specification. Some code in DevTools is vendored by dropping webpack bundles into the tree. The bundles are created by running a yarn command in the source repository; this also copies the bundle into an M-C tree. If these directories are going to have a moz.yaml, then either this process will have to be changed upstream, or the moz.yaml spec will have to change to account for this. An example is devtools/client/shared/source-map Tom ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent To Require Manifests For Vendored Code In mozilla-central
On 10/04/2018 14:34, Ted Mielczarek wrote: On Tue, Apr 10, 2018, at 9:23 AM, James Graham wrote: On 10/04/2018 05:25, glob wrote: mozilla-central contains code vendored from external sources. Currently there is no standard way to document and update this code. In order to facilitate automation around auditing, vendoring, and linting we intend to require all vendored code to be annotated with an in-tree YAML file, and for the vendoring process to be standardised and automated. The plan is to create a YAML file for each library containing metadata such as the homepage url, vendored version, bugzilla component, etc. See https://goo.gl/QZyz4xfor the full specification. So we now have moz.build that in addition to build instructions, contains metadata for mozilla-authored code (e.g. bugzilla components) and moz.yaml that will contain similar metadata but only for non-mozilla-authored code, as well as Cargo.toml that will contain (some of) that metadata but only for code written in Rust. As someone who ended up having to write code to update moz.build files programatically, the situation where we have similar metadata spread over three different kinds of files, one of them Turing complete, doesn't make me happy. Rust may be unsolvable, but it would be good if we didn't have two mozilla-specific formats for specifying metadata about source files. It would be especially good if updating this metadata didn't require pattern matching a Python AST. We are in fact rethinking the decision to put file metadata in moz.build files for these very reasons. I floated the idea of having it live in these same YAML files that glob is proposing for vendoring info since it feels very similar. I don't want to block his initial work on tangentially-related concerns, but I think we should definitely look into this once he gets a first version of his vendoring proposal working. I don't know if there's anything useful we can do about Cargo.toml--we obviously want to continue using existing Rust practices there. If there are specific things you need to do that are hard because of that I'd be interested to hear about them to see if there's anything we can improve. That's great to hear! The main thing I currently have to do is automatically update bug component metadata when files move around during wpt imports. However one can certainly imagine having to script similar metadata updates For example, I assume that wpt is not "third party" code according to the terms of this discussion, since it's also edited in-tree, and whatever tooling we have to support generic third party repos won't apply. But it would make sense to store the upstream revision of wpt in there rather than in one-off custom file like we do currently. So reusing the same moz.yaml format everywhere rather than having one case for "local" code and one for "remote" would make sense to me as someone maintaining what amounts to an edge case. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent To Require Manifests For Vendored Code In mozilla-central
On Tue, Apr 10, 2018, at 9:23 AM, James Graham wrote: > On 10/04/2018 05:25, glob wrote: > > mozilla-central contains code vendored from external sources. Currently > > there is no standard way to document and update this code. In order to > > facilitate automation around auditing, vendoring, and linting we intend > > to require all vendored code to be annotated with an in-tree YAML file, > > and for the vendoring process to be standardised and automated. > > > > > > The plan is to create a YAML file for each library containing metadata > > such as the homepage url, vendored version, bugzilla component, etc. See > > https://goo.gl/QZyz4xfor the full specification. > > So we now have moz.build that in addition to build instructions, > contains metadata for mozilla-authored code (e.g. bugzilla components) > and moz.yaml that will contain similar metadata but only for > non-mozilla-authored code, as well as Cargo.toml that will contain (some > of) that metadata but only for code written in Rust. > > As someone who ended up having to write code to update moz.build files > programatically, the situation where we have similar metadata spread > over three different kinds of files, one of them Turing complete, > doesn't make me happy. Rust may be unsolvable, but it would be good if > we didn't have two mozilla-specific formats for specifying metadata > about source files. It would be especially good if updating this > metadata didn't require pattern matching a Python AST. We are in fact rethinking the decision to put file metadata in moz.build files for these very reasons. I floated the idea of having it live in these same YAML files that glob is proposing for vendoring info since it feels very similar. I don't want to block his initial work on tangentially-related concerns, but I think we should definitely look into this once he gets a first version of his vendoring proposal working. I don't know if there's anything useful we can do about Cargo.toml--we obviously want to continue using existing Rust practices there. If there are specific things you need to do that are hard because of that I'd be interested to hear about them to see if there's anything we can improve. FWIW, moz.build files being Turing complete is a thing that makes me sad very often. If I had a better solution that didn't involve a huge amount of engineering work I'd be all for it! -Ted ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent To Require Manifests For Vendored Code In mozilla-central
James Graham wrote: So we now have moz.build that in addition to build instructions, contains metadata for mozilla-authored code (e.g. bugzilla components) and moz.yaml that will contain similar metadata but only for non-mozilla-authored code, as well as Cargo.toml that will contain (some of) that metadata but only for code written in Rust. As someone who ended up having to write code to update moz.build files programatically, the situation where we have similar metadata spread over three different kinds of files, one of them Turing complete, doesn't make me happy. Rust may be unsolvable, but it would be good if we didn't have two mozilla-specific formats for specifying metadata about source files. It would be especially good if updating this metadata didn't require pattern matching a Python AST. the build team recently decided to transition away from using moz.build files for declaring static metadata (such as bug components); see https://groups.google.com/forum/#!topic/mozilla.dev.builds/X5hbK6a4VQc -- glob — engineering workflow — moz://a ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent To Require Manifests For Vendored Code In mozilla-central
On 10/04/2018 05:25, glob wrote: mozilla-central contains code vendored from external sources. Currently there is no standard way to document and update this code. In order to facilitate automation around auditing, vendoring, and linting we intend to require all vendored code to be annotated with an in-tree YAML file, and for the vendoring process to be standardised and automated. The plan is to create a YAML file for each library containing metadata such as the homepage url, vendored version, bugzilla component, etc. See https://goo.gl/QZyz4xfor the full specification. So we now have moz.build that in addition to build instructions, contains metadata for mozilla-authored code (e.g. bugzilla components) and moz.yaml that will contain similar metadata but only for non-mozilla-authored code, as well as Cargo.toml that will contain (some of) that metadata but only for code written in Rust. As someone who ended up having to write code to update moz.build files programatically, the situation where we have similar metadata spread over three different kinds of files, one of them Turing complete, doesn't make me happy. Rust may be unsolvable, but it would be good if we didn't have two mozilla-specific formats for specifying metadata about source files. It would be especially good if updating this metadata didn't require pattern matching a Python AST. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent To Require Manifests For Vendored Code In mozilla-central
On 10/04/2018 05:25, glob wrote: The plan is to create a YAML file for each library containing metadata such as the homepage url, vendored version, bugzilla component, etc. See https://goo.gl/QZyz4xfor the full specification. There are some directories where we only import a file from a third-party and it is alongside other files in that same directory, e.g. testing/modules has "ajv-4.1.1.js" and "sinon-2.3.2.js" imported from elsewhere. How do they fit into this scheme? One thing we do need is a way for the build system to give us a list of files and directories that are from third-parties, imported, etc. Can that be provided as a result of this work? Mark ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent To Require Manifests For Vendored Code In mozilla-central
Henri Sivonen wrote: This proposal makes sense to me when it comes to libraries that are not vendored from crates.io. However, this seems very heavyweight and only adds the Bugzilla metadata for crates.io crates. It seems to me that declaring the Bugzilla component isn't worth the trouble of having another metadata file in addition to Cargo.toml. i agree; excluding third-party/rust from this makes sense. sorry didn't explicitly call that out initially. Additonally, the examples suggest that this invents new ad hoc license identifiers. I suggest we not do that but instead use https://spdx.org/licenses/ and have a script to enforce that bogus values don't creep in. thanks; updated. -- glob — engineering workflow — moz://a ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent To Require Manifests For Vendored Code In mozilla-central
On Tue, Apr 10, 2018 at 7:33 AM, Byron Jones wrote: > glob wrote: >> >> The plan is to create a YAML file for each library containing metadata >> such as the homepage url, vendored version, bugzilla component, etc. See >> https://goo.gl/QZyz4xfor the full specification. > > > this should be: https://goo.gl/QZyz4x for the full specification. This proposal makes sense to me when it comes to libraries that are not vendored from crates.io. However, this seems very heavyweight and only adds the Bugzilla metadata for crates.io crates. It seems to me that declaring the Bugzilla component isn't worth the trouble of having another metadata file in addition to Cargo.toml. Additonally, the examples suggest that this invents new ad hoc license identifiers. I suggest we not do that but instead use https://spdx.org/licenses/ and have a script to enforce that bogus values don't creep in. -- Henri Sivonen hsivo...@hsivonen.fi https://hsivonen.fi/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent To Require Manifests For Vendored Code In mozilla-central
this should be: https://goo.gl/QZyz4x for the full specification. This format is essentially assuming the vendored code comes from a VCS repository. We have plenty of third party code that is imported through upstream tarballs, so this should probably be accounted for. we can certainly support tarballs at a later stage. i've expanded the comment for the url field to indicate it can point any location, with the limitations applying to automated vendoring. How does this fit with rust vendoring? Most if not all the information from your proposed moz.yaml is available in Cargo.toml. Do we need to duplicate this? Does mach vendor rust need to create moz.yaml files from Cargo.toml when it does the vendoring? i don't think there's a need to duplicate the information that's in Cargo.toml. if this pans out to be problematic moz.yaml generation during rust vendoring sounds viable. -- glob — engineering workflow — moz://a ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent To Require Manifests For Vendored Code In mozilla-central
this should be: https://goo.gl/QZyz4x for the full specification. This format is essentially assuming the vendored code comes from a VCS repository. We have plenty of third party code that is imported through upstream tarballs, so this should probably be accounted for. we can certainly support tarballs at a later stage. i've expanded the comment for the url field to indicate it can point any location, with the limitations applying to automated vendoring. How does this fit with rust vendoring? Most if not all the information from your proposed moz.yaml is available in Cargo.toml. Do we need to duplicate this? Does mach vendor rust need to create moz.yaml files from Cargo.toml when it does the vendoring? i don't think there's a need to duplicate the information that's in Cargo.toml. if this pans out to be problematic moz.yaml generation during rust vendoring sounds viable. -- glob — engineering workflow — moz://a ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent To Require Manifests For Vendored Code In mozilla-central
thanks for the feedback martin, Please consider adding hg.mozilla.org to your list of things you will clone from. adding support to vendor from hg.m.o is a great suggestion, and shouldn't be problematic once the work has been proven with github. You don't permit the use of a tag for vendoring, is that intentional? to echo gps and mike's responses use of a sha to is preferred over tags. -- glob — engineering workflow — moz://a ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent To Require Manifests For Vendored Code In mozilla-central
On Mon, Apr 9, 2018 at 9:52 PM, Mike Hommey wrote: > On Tue, Apr 10, 2018 at 02:46:40PM +1000, Martin Thomson wrote: > > This seems like a good idea. > > > > Please consider adding hg.mozilla.org to your list of things you will > > clone from. That will allow us to remove some ugly hacks from the > > tree for vendoring NSS and NSPR. (libffi uses the same script, but it > > seems to be on GitHub now, so that seems like an easy win assuming > > even that libffi still uses that script.) We could use the NSS GitHub > > mirror, but NSPR doesn't have one yet and those can't be the only > > projects that we have using mercurial. Of course, if that is the > > case, then don't worry, we'll just have to talk more seriously about > > moving NSS to GitHub. > > > > You don't permit the use of a tag for vendoring, is that intentional? > > Tags for releases are so commonplace that you should consider it. You > > shouldn't need a branch AND a tag either. More so because branches > > move, which makes them not reliable here. > > > > Having a version in addition to a tag is redundant and therefore > > likely to result in mismatches. I'd say that the tag is enough. > > I'd say a revision should be mandatory. Branches and tags can change. > Revisions can't. I'll reaffirm this. The full hash of a revision - at least in Mercurial and Git - means more than a branch or a tag. Both the branch name and a tag are symbolic and can move. Even if you GPG sign a tag, someone could steal your signing key and re-sign a new tag for a different revision and a VCS `verify` command would happily accept it because the signature is valid! A revision hash, however, is constant for all of time and a revision hash cannot change out from under you to refer to different content. Or at least that's the theory. Hash collisions /could/ occur. However, the actual probability of a hash collision is currently far less than the probability that someone moves a pointer on the symbolic branch or tag name because a SHA-1 preimage attack is currently far more difficult than running VCS operations [that Mercurial or Git let you run out of the box] which let you update a symbolic name. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent To Require Manifests For Vendored Code In mozilla-central
On Tue, Apr 10, 2018 at 02:46:40PM +1000, Martin Thomson wrote: > This seems like a good idea. > > Please consider adding hg.mozilla.org to your list of things you will > clone from. That will allow us to remove some ugly hacks from the > tree for vendoring NSS and NSPR. (libffi uses the same script, but it > seems to be on GitHub now, so that seems like an easy win assuming > even that libffi still uses that script.) We could use the NSS GitHub > mirror, but NSPR doesn't have one yet and those can't be the only > projects that we have using mercurial. Of course, if that is the > case, then don't worry, we'll just have to talk more seriously about > moving NSS to GitHub. > > You don't permit the use of a tag for vendoring, is that intentional? > Tags for releases are so commonplace that you should consider it. You > shouldn't need a branch AND a tag either. More so because branches > move, which makes them not reliable here. > > Having a version in addition to a tag is redundant and therefore > likely to result in mismatches. I'd say that the tag is enough. I'd say a revision should be mandatory. Branches and tags can change. Revisions can't. Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent To Require Manifests For Vendored Code In mozilla-central
On Tue, Apr 10, 2018 at 12:33:27PM +0800, Byron Jones wrote: > glob wrote: > > The plan is to create a YAML file for each library containing metadata > > such as the homepage url, vendored version, bugzilla component, etc. See > > https://goo.gl/QZyz4xfor the full specification. > > this should be: https://goo.gl/QZyz4x for the full specification. This format is essentially assuming the vendored code comes from a VCS repository. We have plenty of third party code that is imported through upstream tarballs, so this should probably be accounted for. How does this fit with rust vendoring? Most if not all the information from your proposed moz.yaml is available in Cargo.toml. Do we need to duplicate this? Does mach vendor rust need to create moz.yaml files from Cargo.toml when it does the vendoring? Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent To Require Manifests For Vendored Code In mozilla-central
This seems like a good idea. Please consider adding hg.mozilla.org to your list of things you will clone from. That will allow us to remove some ugly hacks from the tree for vendoring NSS and NSPR. (libffi uses the same script, but it seems to be on GitHub now, so that seems like an easy win assuming even that libffi still uses that script.) We could use the NSS GitHub mirror, but NSPR doesn't have one yet and those can't be the only projects that we have using mercurial. Of course, if that is the case, then don't worry, we'll just have to talk more seriously about moving NSS to GitHub. You don't permit the use of a tag for vendoring, is that intentional? Tags for releases are so commonplace that you should consider it. You shouldn't need a branch AND a tag either. More so because branches move, which makes them not reliable here. Having a version in addition to a tag is redundant and therefore likely to result in mismatches. I'd say that the tag is enough. (FWIW, NSS does some of its own vendoring for test runs, so maybe you could look at https://searchfox.org/nss/source/fuzz/config/git-copy.sh - vendoring git repos without pulling history turned out to have a few gotchas.) On Tue, Apr 10, 2018 at 2:25 PM, glob wrote: > mozilla-central contains code vendored from external sources. Currently > there is no standard way to document and update this code. In order to > facilitate automation around auditing, vendoring, and linting we intend to > require all vendored code to be annotated with an in-tree YAML file, and for > the vendoring process to be standardised and automated. > > > The plan is to create a YAML file for each library containing metadata such > as the homepage url, vendored version, bugzilla component, etc. See > https://goo.gl/QZyz4x for the full specification. > > > We will work with teams to add moz.yaml files where required, as well as > adding the capability for push-button vendoring of new revisions. > > > Please address comments to the dev-platform list. > > -- > glob — engineering workflow — moz://a > > > ___ > firefox-dev mailing list > firefox-...@mozilla.org > https://mail.mozilla.org/listinfo/firefox-dev > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent To Require Manifests For Vendored Code In mozilla-central
glob wrote: The plan is to create a YAML file for each library containing metadata such as the homepage url, vendored version, bugzilla component, etc. See https://goo.gl/QZyz4xfor the full specification. this should be: https://goo.gl/QZyz4x for the full specification. -- glob — engineering workflow — moz://a ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Intent To Require Manifests For Vendored Code In mozilla-central
mozilla-central contains code vendored from external sources. Currently there is no standard way to document and update this code. In order to facilitate automation around auditing, vendoring, and linting we intend to require all vendored code to be annotated with an in-tree YAML file, and for the vendoring process to be standardised and automated. The plan is to create a YAML file for each library containing metadata such as the homepage url, vendored version, bugzilla component, etc. See https://goo.gl/QZyz4xfor the full specification. We will work with teams to add moz.yaml files where required, as well as adding the capability for push-button vendoring of new revisions. Please address comments to the dev-platform list. -- glob — engineering workflow — moz://a ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
