Re: Why does Guix duplicate dependency versions from Cargo.toml?
i may not understand this well enough, but with that in mind... the nix crowd allows something that they call vendoring: they use the native tools of the language ecosystem to fetch the transitive closure of the dependencies, as specified by their own package management descriptions. then they compute a hash on the entire directory, and record it in the leaf package's definition. i think this vendoring dir/archive then even gets cached by their substitute servers (for prosperity). IIUC, this method is rejected by guix on principle. if someone wants to test their mailing list search-fu, then there was a similar discussion about golang in the past. -- • attila lendvai • PGP: 963F 5D5F 45C7 DFCD 0A39 -- “As children go, so go nations. It's that simple.” — Carol Bellamy
Re: Why does Guix duplicate dependency versions from Cargo.toml?
Am Fri, Aug 25, 2023 at 03:56:56PM +0100 schrieb (: > Zhu Zihao writes: > > and AFIAK, Maxime Devos is working on new build system called > > "Antioxidant", which can build rust application without cargo (Yes, > > invoke rustc directly!), The new build system will cache the rlib > > intermediate result of crate and share between different builds. > Sadly, I think that's been abandoned :( My impression is that Maxime has left the Guix project, so that the work was naturally abandoned, but there was no conscious decision that it should be dropped. So it would be nice if someone knowledgeable about the topic could pick it up and finish the build system. Andreas
Re: Why does Guix duplicate dependency versions from Cargo.toml?
Zhu Zihao writes: > and AFIAK, Maxime Devos is working on new build system called > "Antioxidant", which can build rust application without cargo (Yes, > invoke rustc directly!), The new build system will cache the rlib > intermediate result of crate and share between different builds. Sadly, I think that's been abandoned :( -- (
Re: Why does Guix duplicate dependency versions from Cargo.toml?
Jonas Møller writes: > Hi Guix! Why does cargo-build-system need #:cargo-inputs specified in the > package definition? This seems like a big > mistake for a couple of reasons. Just like the nice people in mail list explained, when building a package, Guix builders are not allowed to connect to network, so the crates should be prefetched. and AFIAK, Maxime Devos is working on new build system called "Antioxidant", which can build rust application without cargo (Yes, invoke rustc directly!), The new build system will cache the rlib intermediate result of crate and share between different builds. > 1 It is completely redundant, it should match what is in Cargo.toml. I know > `guix import crate` exists to automate > this process, but I don't understand the rationale for duplicating > this information. Not only Guix do this, Debian also package Rust crates [1]. > 2 It is bad practice for Guix to override Cargo.lock if it exists, this means > that Guix is building a different binary to the > one the developers of the packaged Rust application are seeing on their end, > this is a much bigger problem. OK, IMO almost all software packaged by Linux distro are "different" from upstream at binary level. A notable example is, software developer may prefeer to add bundled 3rd party dependencies to ease the build of package. But distro maintainers want to ensure every software use the library provided by distro[2]. And there maybe distro specific patches (Redhat backports security fixes to the old version of package for their RHEL) > This can and will cause spurious build failures, or bugs that are unknown to > the developers of the Rust programs that > Guix packages. The Rust crate in Guix is used to package rust application, if user have problem with the rust application on Guix (e.g. ripgrep, fd, xxd, bat ...) They should report to Guix first, so it's Guix developer's responsibility to smooth out those differences. If user want to develop a Rust crate/application, they can still use "cargo" command and fetch crates from crates.io registry. [1]: https://packages.debian.org/sid/librust-bytecount-dev [2]: https://wiki.gentoo.org/wiki/Why_not_bundle_dependencies -- Retrieve my PGP public key: gpg --recv-keys B3EBC086AB0EBC0F45E0B4D433DB374BCEE4D9DC Zihao signature.asc Description: PGP signature
Re: Why does Guix duplicate dependency versions from Cargo.toml?
Hi, Sorry if I came off a bit harsh in the initial reply :) I didn't intend for it to read as a "ugh, how don't you understand this" sort of thing but that's what it appeared to be looking at it later. (Communication: It's Hard™) Aaanyway Jonas Møller writes: > Interesting, Guix already has git/url-fetch, what is keeping Guix from simply > fetching a cargo project and then running `cargo build` in the fetched source > directory? Okay, this will require a bit of explanation about how Guix's (and Nix's) derivations work. As I understand it, there are broadly two types of derivations: - fixed-output, used for things like - whatever-the-opposite-is-called, used for normal things like packages The reason fixed-output derivations are called that is because their hashes are *known before the derivation is built*, or at least their expected hashes; so Guix will download the file/repo, and if it doesn't match the given hash, it'll throw an error. Now, this means there's no reproducibility issue with internet access. If produced outputs O1 and and O2 are different, then either one or both will fail the hash check, and thus the output will never be built. There can never *be* a reproducibility issue because if there was one on the server side Guix would catch a hash-mismatch before the consequences of that irreproducibility were ever felt. Thus, *it is safe to allow internet access in a fixed-output build*, because reproducibility issues become null and void. Or, at least, that's how I understand it. On the flip side, of course, regular build scripts are not allowed to access anything (other than stuff we can't seem to figure out how to block, like system time) that could affect reproducibility. > If the problem is that the build daemon is sandboxed and doesn't have internet > access, it is also feasible to have one stage of the build process download > all > the resources specified in Cargo.lock (and cache this in /gnu/store) and > rewrite > `version = "x.y.z"` to `path = "x/y/z"` before everything is passed to the > build > daemon. This is actually *extraordinarily* close to what we already do. The cargo-build-system, when building a library, copies its entire source into the output directory (I know, I know... But without writing our own Rust build system, there's no alternative.) We use this source when building packages that depend on it; the sources of the #:CARGO-INPUTS of a package in the process of being built are copied into a 'guix-vendor' directory, and then we pass this flag to Cargo which makes it treat the vendor directory like a local package registry that takes precedence over crates.io. Hopefully that clears things up :) -- (
Re: Why does Guix duplicate dependency versions from Cargo.toml?
Interesting, Guix already has git/url-fetch, what is keeping Guix from simply fetching a cargo project and then running `cargo build` in the fetched source directory? If the problem is that the build daemon is sandboxed and doesn't have internet access, it is also feasible to have one stage of the build process download all the resources specified in Cargo.lock (and cache this in /gnu/store) and rewrite `version = "x.y.z"` to `path = "x/y/z"` before everything is passed to the build daemon. I just don't see why this can't be ad-hoc. --- Original Message --- On Thursday, August 24th, 2023 at 09:05, ( wrote: > > > Jonas Møller jonas@moller.systems writes: > > > Hi Guix! Why does cargo-build-system need #:cargo-inputs specified in the > > package definition? This seems like a > > big mistake for a couple of reasons. > > > > 1 It is completely redundant, it should match what is in Cargo.toml. I know > > `guix import crate` exists to > > automate this process, but I don't understand the rationale for duplicating > > this information. > > > Because it'd be literally impossible to do otherwise. You'd have to > contort Guix into some pretty weird shapes to change a package > derivation based on what was downloaded as the source. > > -- (
Re: Why does Guix duplicate dependency versions from Cargo.toml?
Jonas Møller writes: > Hi Guix! Why does cargo-build-system need #:cargo-inputs specified in the > package definition? This seems like a > big mistake for a couple of reasons. > > 1 It is completely redundant, it should match what is in Cargo.toml. I know > `guix import crate` exists to > automate this process, but I don't understand the rationale for duplicating > this information. Because it'd be literally impossible to do otherwise. You'd have to contort Guix into some pretty weird shapes to change a package derivation based on what was downloaded as the source. -- (
Why does Guix duplicate dependency versions from Cargo.toml?
Hi Guix! Why does cargo-build-system need #:cargo-inputs specified in the package definition? This seems like a big mistake for a couple of reasons. - It is completely redundant, it should match what is in Cargo.toml. I know `guix import crate` exists to automate this process, but I don't understand the rationale for duplicating this information. - It is bad practice for Guix to override Cargo.lock if it exists, this means that Guix is building a different binary to the one the developers of the packaged Rust application are seeing on their end, this is a much bigger problem. This can and will cause spurious build failures, or bugs that are unknown to the developers of the Rust programs that Guix packages.
