Re: Duplicate dependency policy for Rust in mozilla-central?
On 15/03/2019 14:31, Xidorn Quan wrote: Servo has a policy banning duplicate dependencies with a whitelist, and such list currently has: This exact allow-list is not part of Servo’s policy, but is constantly evolving. If you can reduce it (typically by updating some intermediate dependencies to versions that use e.g. log 0.4 instead of 0.3), this is great and we love you. If you add a new exception to the allow-list, in review we will ask that you make some effort to avoid doing so. If the effort turns out to be disproportionate (for example: many intermediate dependencies are affected, and they in turn would affect other crates in the graph) or if we want to avoid waiting too long on upstream (because a patch is at risk of bitrotting, or blocks other work, or…) then we may accept growing the list. The important part is that machine-verification avoids accidentally adding new duplications. On 15/03/2019 15:38, Andreas Tolfsen wrote: It is my experience that far too many dependencies are defined on exact version numbers, e.g. "log = 0.3.9", which effectively forces us to vendor that exact version in-tree. It does not force that. Specifying `log = "0.3.9"` in Cargo.toml’s [dependencies] section is equivalent to `log = "^0.3.9"` which is equivalent to `log = ">=0.3.9 < 0.4.0"`. So if a project uses crates A with the above and crate B with `log = "0.3.12"`, then version 0.3.15 is acceptable to satisfy both dependencies. What would force an exact version is `log = "=0.3.9"`. (Note that the first equal sign is TOML syntax for key/value pairs, while the second one is part of the version specification string, inside the quotes.) See https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html -- Simon Sapin ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Duplicate dependency policy for Rust in mozilla-central?
On Fri, Mar 15, 2019, 15:39 Andreas Tolfsen wrote: > However, I want to talk a little bit about _why_ we see so many > duplicate crates and what causes it. It is my experience that far > too many dependencies are defined on exact version numbers, e.g. > "log = 0.3.9", which effectively forces us to vendor that exact > version in-tree. > While I initially thought the same thing, this is not how Cargo's dependencies work: 0.3.9 in this context means "everything semver-compatible with 0.3.9", so 0.3.10 would be accepted in this instance. However, 0.4.0 by definition is semver-incompatible with 0.3.x (that is, the normal incompatibility between 1.x and 2.y is moved one component to the right if the first component is 0). > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Duplicate dependency policy for Rust in mozilla-central?
Also sprach Xidorn Quan: > Should we have some kind of policy to address duplicate dependencies > in Gecko as well? Maybe I'm missing something but I don't think I'm > aware of any previous discussion about this. Deduplicating crate dependencies has so far been a mostly heroic undertaking done by individuals such as eijebong, so having a policy built in to "./mach vendor rust" that prevents duplication is probably a good idea! However, I want to talk a little bit about _why_ we see so many duplicate crates and what causes it. It is my experience that far too many dependencies are defined on exact version numbers, e.g. "log = 0.3.9", which effectively forces us to vendor that exact version in-tree. In some cases we see this reproduce throughout a crate’s dependency chain, where different crates depend on minutely different versions of crate D (v0.4.0, v0.3.9, v0.3.7) causing us to have to vendor and compile all of them, even as part of a single build: A |-- B v1.2.0 | |-- C v2.0.1 | |-- D v0.4.0 | |-- C v2.0.0 | |-- D v0.3.9 | |-- E v0.2 | |-- D v0.3.7 As kats points out, unravelling this spider web is not trivial. You often run into cases where some crate depends on a particular version of something, but you can’t upgrade that because something else depends on a too specific version, and you can’t upgrade that again because it would cause an API change or behaviour change that requires specialist peer review in an unrelated component. In conclusion, my plead to Rust crate maintainers is to be generous with what version range you tolerate for dependencies. As most crates respect semantic versioning you can be somewhat confident that relying on v1.2 rather than v1.2.1 will keep your program working when v1.2.2 is released. The most common argument I hear against tolerating a wider dependency range for dependencies is “reproducible builds”. Well, for binaries it is recommended you check in the Cargo.lock file, and for Gecko we do that for all components, including libraries. We additionally vendor the source code, allowing us to do hermetically sealed builds. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Duplicate dependency policy for Rust in mozilla-central?
On Fri, Mar 15, 2019 at 9:32 AM Xidorn Quan wrote: > Should we have some kind of policy to address duplicate dependencies in Gecko > as well? Maybe I'm missing something but I don't think I'm aware of any > previous discussion about this. I remember IRC discussions about this, but there were some concerns that banning duplicates would result in slowing people down. I understand the concern, but I'm not sure how much of a problem it would be in practice. I personally am in favor of banning duplicates. We would need a Servo-like list of crates that can be duplicated because there are several crates that are difficult to move forward. -Nathan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Duplicate dependency policy for Rust in mozilla-central?
FWIW there often ad-hoc attempts to deduplicate Rust dependencies, and they often run into nontrivial blockers. Currently there's a few things blocked on bug 1530448. On Fri, Mar 15, 2019 at 9:32 AM Xidorn Quan wrote: > > Hi all, > > Recently I tried to build Firefox on cloud, and noticed that building Rust > dependencies is now a significant part of it. Another thing I noticed is > that, we have lots of duplicate Rust dependencies. > > I scanned Cargo.lock in the root and found that we have: > * 2 versions of block-buffer (0.3.3, 0.7.0) > * 2 versions of byte-tools (0.2.0, 0.3.0) > * 2 versions of crossbeam-deque (0.2.0, 0.3.1) > * 2 versions of crossbeam-epoch (0.3.1, 0.4.3) > * 3 versions of crossbeam-utils (0.2.2, 0.3.2, 0.6.3) > * 2 versions of digest (0.7.6, 0.8.0) > * 2 versions of generic-array (0.9.0, 0.12.0) > * 2 versions of lazycell (0.4.0, 0.6.0) > * 2 versions of log (0.3.9, 0.4.6) > * 2 versions of memchr (1.0.2, 2.0.1) > * 2 versions of memmap (0.5.2, 0.6.2) > * 2 versions of nom (3.2.1, 4.1.1) > * 2 versions of num-traits (0.1.43, 0.2.6) > * 2 versions of proc-macro2 (0.3.5, 0.4.24) > * 2 versions of quote (0.5.2, 0.6.10) > * 2 versions of rand (0.3.22, 0.4.3) > * 2 versions of regex (0.2.2, 1.0.0) > * 2 versions of regex-syntax (0.4.1, 0.6.0) > * 2 versions of semver (0.6.0, 0.9.0) > * 2 versions of sha2 (0.7.1, 0.8.0) > * 2 versions of slab (0.3.0, 0.4.1) > * 2 versions of strsim (0.6.0, 0.7.0) > * 3 versions of syn (0.13.1, 0.14.6, 0.15.24) > * 2 versions of uuid (0.6.5, 0.7.1) > * 2 versions of winapi (0.2.8, 0.3.6) > > Although these duplicate dependencies may not go into the final binary, they > are still things which can slow down the build. In this list, I would suspect > crates like nom, regex, and syn could take significant time to build for each > version. > > Servo has a policy banning duplicate dependencies with a whitelist, and such > list currently has: > * base64 > * block-buffer > * byte-tools > * crossbeam-deque > * crossbeam-epoch > * crossbeam-utils > * digest > * generic-array > * rand > * unicase > * winapi > and they had a tradition (in my opinion) to help pushing the ecosystem > forward using new versions of dependencies. > > Should we have some kind of policy to address duplicate dependencies in Gecko > as well? Maybe I'm missing something but I don't think I'm aware of any > previous discussion about this. > > - Xidorn > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
