Re: rust-team branch merged
Efraim Flashner writes: [[PGP Signed Part:Undecided]] On Wed, Apr 24, 2024 at 11:58:22AM -0400, Jason Conroy wrote: Efraim Flashner writes: > On the other hand, by generating it during the build of each > package we > make sure to pull in all the crates which exist in the build, > so we > could add into a profile/manifest just the crates listed in a > Cargo.toml > and then each crate would pull in its own dependencies, and > then the > profile hook could combine them all together. Thanks. Just to make sure I understand: it sounds like you're saying that by creating the JSON index files up front, we'd be preserving some knowledge about a package's dependency graph that isn't easily recovered later by recursively walking through inputs and cargo-inputs for the package specs in a manifest/profile? If we create it upfront it may be easier to keep track of the cargo-dependencies, but I suppose it would mostly depend on the implementation. I'm not sure that'd actually be the case, so don't worry about it too much. Whatever we end up with will be better than what we have now, and we can always make it better later if it needs to change. Hi Efraim, I spent some time on this and have a few thoughts below. In short, I made progress but also hit some roadblocks that give me second thoughts about the approach we discussed up-thread. As you suggested, I kept the metadata generation code in cargo-build-system and used the cargo-registry profile hook to merge all of those package artifacts into index files. I also modified the hook to lift the full closure of source dependencies into the profile for any rust packages in the manifest, borrowing logic from `(guix build-system cargo)`. However, testing revealed some issues with this approach: 1) Formerly, for any cargo package with `#:skip-build? #t` the crate was treated as an opaque file: it was copied to a location where cargo could access it, but otherwise unprocessed. But to generate the index file we need to unpack the crate and call `cargo metadata`, which performs some sanity checks on the package structure. The implication is that if #:skip-build? is used for reasons other than performance - e.g., for a package that works fine as a dependency but somehow fails during a standalone build - then we might have trouble including such a package in Cargo's local registry. I don't know how big an issue this is in practice, but I think I encountered one package that fits this scenario: custom_derive-0.1.7. (We can discuss the details in a separate thread if you're interested.) 2) When building against a local-registry, it seems like Cargo expects to find crates that it wouldn't for a vendor tree. As mentioned above, my current patch pulls source crates into the profile using the same logic as cargo-build-system currently does: for each manifest package and its cargo-development-inputs, recursively walk through all of the cargo-inputs. But after building a profile this way, I encountered the following: $ cargo build error: no matching package named `automod` found location searched: registry `crates-io` required by package `serde_json v1.0.111` ... which satisfies dependency `serde_json = "^1.0"` of package `rand_chacha v0.3.1` ... which satisfies dependency `rand_chacha = "^0.3.0"` of package `rand v0.8.5` ... which satisfies dependency `rand = "^0.8.5"` of package `mytest v0.1.0 (/home/guix/mytest)` Note that serde_json is actually a dev-dependency of rand_chacha, and automod is a dev-dependency of serde_json, but our algorithm above assumes that dev-dependencies of dependencies can be omitted. So in practice, it seems like building a local-registry that cargo will accept is even more expensive than building a vendor tree. 3) The build error above is interesting for another reason: cargo simply gives up rather than fetching the missing dependencies. In our earlier discussion, the motivation for using a local-registry over a vendor tree in a profile was to support a sort of hybrid workflow: the profile seeds the registry with some packages, but cargo can still download others on its own during `cargo build`. But in this test and several others, I haven't observed a case where cargo falls back to crates.io for missing packages. Efraim, could you share details on how you accomplished this in the past? I wonder if our discussion might have conflated cargo's registry cache ($CARGO_HOME/registry) with a true local-registry; I haven't found any good documentation on the former, but it has a different directory layout and index file format from the local-registry's, so it wouldn't surprise me if it had different semantics as well. Cheers, Jason
Re: rust-team branch merged
Efraim Flashner writes: On the other hand, by generating it during the build of each package we make sure to pull in all the crates which exist in the build, so we could add into a profile/manifest just the crates listed in a Cargo.toml and then each crate would pull in its own dependencies, and then the profile hook could combine them all together. Thanks. Just to make sure I understand: it sounds like you're saying that by creating the JSON index files up front, we'd be preserving some knowledge about a package's dependency graph that isn't easily recovered later by recursively walking through inputs and cargo-inputs for the package specs in a manifest/profile? Jason
Re: rust-team branch merged
Efraim Flashner writes: Currently if you were to pull in rust-rand-0.8 and rust-rand-0.7 then you'd have both rand-0.*.crate files in the registry but only one of them would be listed in share/cargo/registry/index/ra/nd/rand. I need to adjust the generation of that file to combine multiple sources if they exist, and sort them (I'm not sure it's necessary, but wouldn't be surprised if we hit undefined behaviour if they were listed multiple times or out of order). Hi Efraim, I'm currently investigating this limitation of your proposed patch. Did you have a strategy in mind for how to fix it? I see that the index files are currently generated during a phase of cargo-build-system, rather than as a profile hook. So, to build an index that properly reflects the contents of a profile, it would seem that the two simplest options are: a) keep your existing index-generation logic during the build, and merge these per-package index files when building the profile; or b) move your patch's index-generating code out of the build phase and into a profile hook, so that we build each index file in a single pass (for all versions of a package) rather than merging the files from each package output. On the surface option (b) seems cleaner, but maybe you had a reason for generating the index contents during the build? Jason
Re: rust-team branch merged
On Tue, Mar 12, 2024 at 11:12 AM Efraim Flashner wrote: > On Thu, Mar 07, 2024 at 11:48:53AM -0500, Jason Conroy wrote: > > On Thu, Mar 7, 2024 at 3:08 AM Efraim Flashner > > wrote: > > > > > Until then > > > I've been experimenting by manually listing the other crates I've > needed > > > but in theory we could try to make `guix shell --development` pull in > > > the needed crates. > > > > > > > I was considering another option that falls somewhere in between: since > I'm > > already building shells from manifest files, it should suffice to have a > > Scheme utility function that calculates the transitive dependencies for a > > given list of library packages. Similar logic seems to exist already in > > `(guix build-system cargo)` but it's not exposed publicly. As interim > > solutions go, what do you think about this one versus modifying `guix > > shell`? > > Once you have something like that working it shouldn't be too hard to > merge that into `guix shell`, assuming we go that route. > Sounds good, I'll follow up if my exercise produces anything useful. Thanks for the input!
Re: rust-team branch merged
On Thu, Mar 7, 2024 at 3:08 AM Efraim Flashner wrote: > The transitive dependencies getting pulled in automatically should work > automatically if we ever finish the antioxidant-build-system. Since you bring up antioxidant, I'm kind of curious whether that stalled mainly due to shifts in contributor priorities, or due to significant technical issues. > Until then > I've been experimenting by manually listing the other crates I've needed > but in theory we could try to make `guix shell --development` pull in > the needed crates. > I was considering another option that falls somewhere in between: since I'm already building shells from manifest files, it should suffice to have a Scheme utility function that calculates the transitive dependencies for a given list of library packages. Similar logic seems to exist already in `(guix build-system cargo)` but it's not exposed publicly. As interim solutions go, what do you think about this one versus modifying `guix shell`? > > I misread what you wrote as it was working. It's definitely something I > want but wasn't ready to work on yet. > > > I took rust-rand as an example only because it does have some > dependencies: > > > > $ cd $CARGO_PROJECT > > $ cat Cargo.toml > > [package] > > name = "test_prog" > > ... > > [dependencies] > > rand = "0.8.5" > > > > $ cargo build > > Updating crates.io index > > Downloaded cfg-if v1.0.0 > > Downloaded rand_chacha v0.3.1 > > Downloaded rand v0.8.5 > > Downloaded ppv-lite86 v0.2.17 > > Downloaded rand_core v0.6.4 > > Downloaded getrandom v0.2.12 > > Downloaded libc v0.2.153 > > Downloaded 7 crates (932.0 KB) in 0.48s > >Compiling libc v0.2.153 > >Compiling cfg-if v1.0.0 > >Compiling ppv-lite86 v0.2.17 > >Compiling getrandom v0.2.12 > >Compiling rand_core v0.6.4 > >Compiling rand_chacha v0.3.1 > >Compiling rand v0.8.5 > >Compiling test_prog v0.1.0 (/home/...) > > > > > > > Currently if you were to pull in rust-rand-0.8 and rust-rand-0.7 then > > > you'd have both rand-0.*.crate files in the registry but only one of > > > them would be listed in share/cargo/registry/index/ra/nd/rand. I need > to > > > adjust the generation of that file to combine multiple sources if they > > > exist, and sort them (I'm not sure it's necessary, but wouldn't be > > > surprised if we hit undefined behaviour if they were listed multiple > > > times or out of order). > > > > > > I'm somewhat new to rust, but it appears that outside of Guix, the > > local-only development workflow looks like this: > > > > $ cd $CARGO_PROJECT > > $ mkdir $VENDOR > > $ cargo vendor $VENDOR > > > > After downloading and unpacking all of the crates into $VENDOR, this last > > command instructs me to add the following in ~/cargo/config.toml. > > Then, after opening a new guix shell without network access, I can > confirm > > that `cargo build` works fine with the vendored crates. > > > > [source.crates-io] > > replace-with = "vendored-sources" > > > > [source.vendored-sources] > > directory = "" > > I wanted local-registry over replace-with because IIRC replace-with > won't fall back to downloading from crates.io if there's missing crates, > while local-registry will check there first and then download any > missing crates. The use-case I was looking at for that was adding a new > dependency to a project and then not needing to re-create a shell or > package the new crates before continuing on. > I see, thanks. My preferred workflow is different but I acknowledge that use case. The link below claims that one can update the vendor directory in a similar way by re-running `cargo vendor` after adding a dependency to Cargo.toml, but for your use case I agree that it's nicer if `cargo build` can pull to the registry automatically. https://old.reddit.com/r/rust/comments/uvvmjy/how_to_include_vendored_crates_into_a_project/ > The registry as setup in the patches is actually mostly correct, minus > the multiple versions of a crate, it just needs to be fed the crates. > > > Getting back to your patch set: would it make sense to emulate this > vendor > > workflow instead of trying to construct a registry directly? Even > assuming > > that all details of the registry structure are stable and documented, the > > layout of the vendor directory appears much simpler. And IIUC the code > for > > setting up vendored libraries already exists in cargo-build-system. > > > > I also need to figure out something with a > > > config.toml to see if it's possible to generate one that could be > > > included from another one, since you can't add 'local-registry = > > > $GUIX_PROFILE/...' in a toml file. > > > > > > > You've probably researched this more than I have, but it seems that this > > use case is explicitly unsupported in the TOML language spec: > > https://github.com/toml-lang/toml/issues/397 > > > > With that option off the table, I can't think of any elegant solutions. > > Maybe a wrapper for the cargo binary that
Re: rust-team branch merged
On Wed, Mar 6, 2024 at 4:06 AM Efraim Flashner wrote: > On Mon, Feb 26, 2024 at 09:24:29PM -0500, Jason Conroy wrote: > > Hello Efraim, > > > > Thanks for investigating this - a Rust development workflow using only > > Guix-native crates is something I've been waiting for! > > > > I was experimenting with your patches and it seems that they do pull in > the > > source crates for requested packages, but not their dependencies (example > > below). Is there something I'm missing? > > When you say they pull in the source crates do you mean the sources of > the other rust packages needed by rust-rand? I didn't test that, but I > assumed it wouldn't. > That's right, I'm interested in having `guix shell` populate the full set of transitive dependencies in the registry, similar to what happens when using `cargo build` in the conventional way (but instead using Guix-packaged crates). I interpreted your message up-thread to mean that you were trying to accomplish the same, but please correct me if not. I took rust-rand as an example only because it does have some dependencies: $ cd $CARGO_PROJECT $ cat Cargo.toml [package] name = "test_prog" ... [dependencies] rand = "0.8.5" $ cargo build Updating crates.io index Downloaded cfg-if v1.0.0 Downloaded rand_chacha v0.3.1 Downloaded rand v0.8.5 Downloaded ppv-lite86 v0.2.17 Downloaded rand_core v0.6.4 Downloaded getrandom v0.2.12 Downloaded libc v0.2.153 Downloaded 7 crates (932.0 KB) in 0.48s Compiling libc v0.2.153 Compiling cfg-if v1.0.0 Compiling ppv-lite86 v0.2.17 Compiling getrandom v0.2.12 Compiling rand_core v0.6.4 Compiling rand_chacha v0.3.1 Compiling rand v0.8.5 Compiling test_prog v0.1.0 (/home/...) > Currently if you were to pull in rust-rand-0.8 and rust-rand-0.7 then > you'd have both rand-0.*.crate files in the registry but only one of > them would be listed in share/cargo/registry/index/ra/nd/rand. I need to > adjust the generation of that file to combine multiple sources if they > exist, and sort them (I'm not sure it's necessary, but wouldn't be > surprised if we hit undefined behaviour if they were listed multiple > times or out of order). I'm somewhat new to rust, but it appears that outside of Guix, the local-only development workflow looks like this: $ cd $CARGO_PROJECT $ mkdir $VENDOR $ cargo vendor $VENDOR After downloading and unpacking all of the crates into $VENDOR, this last command instructs me to add the following in ~/cargo/config.toml. Then, after opening a new guix shell without network access, I can confirm that `cargo build` works fine with the vendored crates. [source.crates-io] replace-with = "vendored-sources" [source.vendored-sources] directory = "" Getting back to your patch set: would it make sense to emulate this vendor workflow instead of trying to construct a registry directly? Even assuming that all details of the registry structure are stable and documented, the layout of the vendor directory appears much simpler. And IIUC the code for setting up vendored libraries already exists in cargo-build-system. I also need to figure out something with a > config.toml to see if it's possible to generate one that could be > included from another one, since you can't add 'local-registry = > $GUIX_PROFILE/...' in a toml file. > You've probably researched this more than I have, but it seems that this use case is explicitly unsupported in the TOML language spec: https://github.com/toml-lang/toml/issues/397 With that option off the table, I can't think of any elegant solutions. Maybe a wrapper for the cargo binary that pre-processes cargo.toml and then calls the real cargo?
Re: rust-team branch merged
Hello Efraim, Thanks for investigating this - a Rust development workflow using only Guix-native crates is something I've been waiting for! I was experimenting with your patches and it seems that they do pull in the source crates for requested packages, but not their dependencies (example below). Is there something I'm missing? Cheers, Jason $ guix shell --pure bash findutils rust-rand -- bash -c 'find -L $GUIX_ENVIRONMENT/share/cargo' /gnu/store/zf88v65rbg2di4qhgdbvhfcjf31rdzby-profile/share/cargo /gnu/store/zf88v65rbg2di4qhgdbvhfcjf31rdzby-profile/share/cargo/registry /gnu/store/zf88v65rbg2di4qhgdbvhfcjf31rdzby-profile/share/cargo/registry/index /gnu/store/zf88v65rbg2di4qhgdbvhfcjf31rdzby-profile/share/cargo/registry/index/ra /gnu/store/zf88v65rbg2di4qhgdbvhfcjf31rdzby-profile/share/cargo/registry/index/ra/nd /gnu/store/zf88v65rbg2di4qhgdbvhfcjf31rdzby-profile/share/cargo/registry/index/ra/nd/rand /gnu/store/zf88v65rbg2di4qhgdbvhfcjf31rdzby-profile/share/cargo/registry/rand-0.8.5.crate /gnu/store/zf88v65rbg2di4qhgdbvhfcjf31rdzby-profile/share/cargo/registry/config.json On Thu, Dec 14, 2023 at 10:10 AM Efraim Flashner wrote: > On Wed, Dec 13, 2023 at 10:34:11AM +0200, Efraim Flashner wrote: > > * Compiled rust packages currently have a 'package' phase, which runs > > the command used to crate a 'crate tarball', and is installed in > > %output/share/cargo/registry, with unpacked sources in > > %output/share/cargo/src. In theory it should be possible to use these > > for local rust development. The benefits include everything that comes > > with being a guix package, including pre-patched shebangs. Currently no > > index file is created in $GUIX_ENVIRONMENT/share/cargo/registry/index, > > which is likely necessary to actually make use of this. Additionally, I > > am unsure how to use '$GUIX_ENVIRONMENT' in ~/.cargo/config so that it > > is expanded and not taken as a literal string. > > In the Guix London meetup someone mentioned that they were interested in > playing around with using Guix for rust development. I've adjusted the > cargo-build-system to produce the registry index files and I added a > profile hook to generate the config.json to locate the packaged crates. > > toml files can't process environment variables (which is probably a good > thing ...) but that means its a little harder to test out. > > with the two patches applied create an environment with the crates you > want and get the location of GUIX_ENVIRONMENT: > `env | grep GUIX_ENVIRONMENT | cut -f2 -d=` > > in ~/.cargo/config: > [source.crates-io] > local-registry = '/share/cargo/registry' > > 'cargo build' should pull from the local crates in the GUIX_ENVIRONMENT. > I'm not sure what happens if it doesn't have those crates available and > would need to get them from crates.io. > > > -- > Efraim Flashner רנשלפ םירפא > GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 > Confidentiality cannot be guaranteed on emails sent or received unencrypted >
