Re: Golang go-updates feature branch?
Hi, On Thu, 16 Feb 2023 at 11:54, Leo Famulari wrote: > On Tue, Feb 14, 2023 at 10:22:07PM +0100, Josselin Poiret wrote: >> The inferior that's used in 'build-aux/cuirass/evaluate.scm' is built >> from the current check-out by first adding the checkout to the store and >> then building it the usual way. However, that copy only selects files >> that belong to the git tree using the git-predicate. If you add the >> manifest to the git checkout by first committing it, it should proceed >> happily (at least it did over here). [...] > I wonder where it should go: a blog post, the cookbook, the Guix manual, > etc? Maybe it could start with the Cookbook, waiting there is enough coherent materials. Cheers, simon
Re: Golang go-updates feature branch?
On Thu, Feb 16, 2023 at 09:05:42PM +0100, Josselin Poiret wrote: > What's the reason behind branch-specific manifests? I'd imagine we'd > want to test that Guix as a whole still works, even when upgrading just > specific parts. Otherwise, I guess this shouldn't qualify for a blog > post, maybe the cookbook but I'd even just add this as a comment at the > very top of `build-aux/cuirass/evaluate.scm`, since that's where people > will go looking for it. If it does end up becoming widespread, perhaps > a section of the manual/cookbook dedicated to how feature branches work > could be a nice home for these bits of info. Well, now that you ask, I guess there's not a good reason to use manifests here. After creating the kernel-updates CI job based on a manifest, I figured we'd use the same pattern for this, but I agree it's a bit clunky. On the other hand, Cuirass does tend to obscure the salient information when testing a branch that doesn't rebuild the world (i.e. core-updates). For example, for kernel updates, I want to know 1) if the kernel packages built and 2) do the system tests still pass? If I simply asked Cuirass to build everything on the kernel-updates branch, it might end up showing me the result of a bunch of unrelated package builds just because they happened to be selected as part of this jobset rather than 'master', due to (un)lucky timing. Does that make sense? Zooming out, our tooling still has a very long way to go. There are so many crucial features of Nix's Hydra (which we used previously) that we are still missing from Cuirass, and it's made it *much* more difficult to efficiently and carefully perform big updates to Guix, even though the build farm hardware is more capable than when we used to run Hydra. This was all so much easier with Hydra. Sorry for the rambling response that went off-topic. This has been on my mind.
Re: Golang go-updates feature branch?
Hi Leo, Leo Famulari writes: > Thank you for your help Josselin! It's much appreciated. Happy to help! > As part of our effort to move towards a "feature branch" development > workflow, it will be useful to collect these tips so that everyone can > create and test their own manifests and Cuirass specs. > > I wonder where it should go: a blog post, the cookbook, the Guix manual, > etc? What's the reason behind branch-specific manifests? I'd imagine we'd want to test that Guix as a whole still works, even when upgrading just specific parts. Otherwise, I guess this shouldn't qualify for a blog post, maybe the cookbook but I'd even just add this as a comment at the very top of `build-aux/cuirass/evaluate.scm`, since that's where people will go looking for it. If it does end up becoming widespread, perhaps a section of the manual/cookbook dedicated to how feature branches work could be a nice home for these bits of info. Best, -- Josselin Poiret signature.asc Description: PGP signature
Re: Golang go-updates feature branch?
On Tue, Feb 14, 2023 at 10:22:07PM +0100, Josselin Poiret wrote: > The inferior that's used in 'build-aux/cuirass/evaluate.scm' is built > from the current check-out by first adding the checkout to the store and > then building it the usual way. However, that copy only selects files > that belong to the git tree using the git-predicate. If you add the > manifest to the git checkout by first committing it, it should proceed > happily (at least it did over here). Thank you for your help Josselin! It's much appreciated. As part of our effort to move towards a "feature branch" development workflow, it will be useful to collect these tips so that everyone can create and test their own manifests and Cuirass specs. I wonder where it should go: a blog post, the cookbook, the Guix manual, etc?
Re: Golang go-updates feature branch?
Hi Leo, Leo Famulari writes: > And there is a diff for 'build-aux/cuirass/evaluate.scm', so that I can > test it with `make cuirass-jobs`. > > But, `make cuirass-jobs` crashes with this manifest and I don't know > why: The inferior that's used in 'build-aux/cuirass/evaluate.scm' is built from the current check-out by first adding the checkout to the store and then building it the usual way. However, that copy only selects files that belong to the git tree using the git-predicate. If you add the manifest to the git checkout by first committing it, it should proceed happily (at least it did over here). Best, -- Josselin Poiret signature.asc Description: PGP signature
Re: Golang go-updates feature branch?
On Wed, Jan 11, 2023 at 10:11:09AM -0700, Katherine Cox-Buday wrote: > I also think this is a fantastic idea, and like John, I think mesa is another > good candidate (and not only because I recently would have benefited from > this). Yeah, mesa is *the* archetypal package for the 'staging' branch, since its reverse dependency graph is too big to rebuild on the 'master' branch but it's not a "core" package. If we do go down that road, we will still need to be judicious about how often we do it. Mesa has ~4000 dependents and we don't want to boil the ocean. Before we can continue, I / we have to debug the patches I sent a few messages ago: https://lists.gnu.org/archive/html/guix-devel/2023-01/msg00131.html Paging guix-sysadmin for assistance...
Re: Golang go-updates feature branch?
Leo Famulari writes: > Now that our build farm is running smoothly, I propose we revive the > practice of feature branches, when appropriate. It was on my TODO list to bring this up on the mailing list! Thank you, Leo! I also think this is a fantastic idea, and like John, I think mesa is another good candidate (and not only because I recently would have benefited from this). > The first one that I propose is a go-updates branch, where we update > the Go compilers. This will affect ~500 packages. > > If I understand correctly, Go is a relatively self-contained reverse > dependency graph within Guix and thus a good candidate for this > approach. It contains the Go libaries, and then some end-user > applications, and they don't really affect other packages. So, it > should be easy to understand when the update is ready to push. > > I can set up a jobset on ci.guix.gnu.org if people like the idea. This was also on my TODO list for this week, and I would love to coordinate! I'm still pretty unfamiliar with the email based workflow and managing our CI, so this could be a great mentoring opportunity! The things I wanted to do: * Dust off 55210 to support a `go-next` package. There are two uses for compilers: (1) Building Guix packages (2) Installing in profiles for use. For (2), in addition to feature branches, it would be nice to be able to quickly release the latest Go package without having to immediately worry about testing all of Guix. The idea with 55210 was to try and do this. I think this is the first thing we should do, and we should land Go 1.19 as the first `go-next`. We should also try and * Bring the default Go in Guix up to Go 1.9 * Update/create some packages I have these in my channel I use for staging upstream changes. You can see them from this line down: https://github.com/kat-co/guix-channels/blob/5c17bdd6c4ae801effdf1069df69ec4e2ce0a0dd/upstream/packages/golang.scm#L429 - go-golang-org-x-vuln - go-mvdan-cc-unparam - go-github-com-google-go-cmdtest - go-github-com-client9-misspell - go-mvdan-cc-xurls-v2 - go-mvdan-cc-gofumpt - go-github-com-google-go-cmp-cmp - go-github-com-jba-templatecheck - go-github-com-google-safehtml - go-github-com-jba-printsrc - go-golang-org-x-exp - go-golang-org-x-mod - go-golang-org-x-sync There are more packages there I need to upstream, but my goal was to get the following bullet-points into master: * Create a package for `gopls` (Go's Language Server Protocol daemon) I've already done the work here: https://github.com/kat-co/guix-channels/blob/5c17bdd6c4ae801effdf1069df69ec4e2ce0a0dd/upstream/packages/golang.scm#L817-L860 * Create a home service for running gopls I've already done the work here: https://github.com/kat-co/guix-channels/blob/upstream-staging/upstream/home/services/gopls.scm Go 1.20 is supposed to be released in February, so we'd have a chance to exercise our new Go feature-branch for 1.19 and then quickly 1.20. I'll reach out on IRC to coordinate! -- Katherine
Re: Golang go-updates feature branch?
On Sun, Jan 08, 2023 at 10:32:36PM +, John Kehayias wrote:
> Heartily agree here. This has come up a few times on #guix and generally with
> support (don't let me speak for everyone though). I think the idea of smaller
> and more frequent feature branches is a great idea: less code changes coming
> to master to review at once (compared to big core-updates merges),
> substitutes and closer to master to be easier for people to pull the branch
> and test, and some areas despite the number of builds would work nicely as a
> branch than just pushed (and lingering) into core-updates for much later.
I agree with all your points. We used to work this way back in the day
when there weren't very many Guix packages. There was always a
'core-updates' job, but if it was limited to packages that truly are
"core". Then, for several years, the build farm was underpowered
relative to the number of packages and it became impractical. I think we
are past that now.
I've attached a couple patches to get started but I need help getting
them to work.
There is 'etc/go-manifest.scm', which does work with `guix package -m
etc/go-manifest.scm --dry-run`.
And there is a diff for 'build-aux/cuirass/evaluate.scm', so that I can
test it with `make cuirass-jobs`.
But, `make cuirass-jobs` crashes with this manifest and I don't know
why:
--
$ make cuirass-jobs
Compiling Scheme modules...
Compiling Scheme modules...
Compiling Scheme modules...
Compiling Scheme modules...
Compiling Scheme modules...
rm -rf "cuirass-jobs"
GEN cuirass-jobs
Computing Guix derivation for 'x86_64-linux'... /
In thread:
uncaught throw to %exception: (#< arguments: (wrong-type-arg
"primitive-load" "Wrong type argument in position ~A (expecting ~A): ~S" (1
"string" #f) (#f)) inferior: # stack: ((#f
("ice-9/boot-9.scm" 1779 13)) (raise-exception ("ice-9/boot-9.scm" 1682 16))
(raise-exception ("ice-9/boot-9.scm" 1684 16)) (primitive-load (#f #f #f))
(save-module-excursion ("ice-9/boot-9.scm" 2835 4)) (#f (#f #f #f)) (map1
("srfi/srfi-1.scm" 585 17)) (append-map ("srfi/srfi-1.scm" 672 15)) (#f
("gnu/ci.scm" 449 8)) (map1 ("srfi/srfi-1.scm" 585 17)) (append-map
("srfi/srfi-1.scm" 672 15)) (cuirass-jobs ("gnu/ci.scm" 489 4)) (#f
("ice-9/eval.scm" 158 9)) (with-exception-handler ("ice-9/boot-9.scm" 1751 10))
(call-with-prompt ("ice-9/boot-9.scm" 723 2)) (#f (#f #f #f)) (#f
("guix/repl.scm" 98 21)) (with-exception-handler ("ice-9/boot-9.scm" 1751 10))
(with-exception-handler ("ice-9/boot-9.scm" 1746 15)) (#f ("guix/repl.scm" 125
7)))>)
In thread:
uncaught throw to %exception: (#< arguments: (wrong-type-arg
"primitive-load" "Wrong type argument in position ~A (expecting ~A): ~S" (1
"string" #f) (#f)) inferior: # stack: ((#f
("ice-9/boot-9.scm" 1779 13)) (raise-exception ("ice-9/boot-9.scm" 1682 16))
(raise-exception ("ice-9/boot-9.scm" 1684 16)) (primitive-load (#f #f #f))
(save-module-excursion ("ice-9/boot-9.scm" 2835 4)) (#f (#f #f #f)) (map1
("srfi/srfi-1.scm" 585 17)) (append-map ("srfi/srfi-1.scm" 672 15)) (#f
("gnu/ci.scm" 449 8)) (map1 ("srfi/srfi-1.scm" 585 17)) (append-map
("srfi/srfi-1.scm" 672 15)) (cuirass-jobs ("gnu/ci.scm" 489 4)) (#f
("ice-9/eval.scm" 158 9)) (with-exception-handler ("ice-9/boot-9.scm" 1751 10))
(call-with-prompt ("ice-9/boot-9.scm" 723 2)) (#f (#f #f #f)) (#f
("guix/repl.scm" 98 21)) (with-exception-handler ("ice-9/boot-9.scm" 1751 10))
(with-exception-handler ("ice-9/boot-9.scm" 1746 15)) (#f ("guix/repl.scm" 125
7)))>)
In thread:
uncaught throw to %exception: (#< arguments: (wrong-type-arg
"primitive-load" "Wrong type argument in position ~A (expecting ~A): ~S" (1
"string" #f) (#f)) inferior: # stack: ((#f
("ice-9/boot-9.scm" 1779 13)) (raise-exception ("ice-9/boot-9.scm" 1682 16))
(raise-exception ("ice-9/boot-9.scm" 1684 16)) (primitive-load (#f #f #f))
(save-module-excursion ("ice-9/boot-9.scm" 2835 4)) (#f (#f #f #f)) (map1
("srfi/srfi-1.scm" 585 17)) (append-map ("srfi/srfi-1.scm" 672 15)) (#f
("gnu/ci.scm" 449 8)) (map1 ("srfi/srfi-1.scm" 585 17)) (append-map
("srfi/srfi-1.scm" 672 15)) (cuirass-jobs ("gnu/ci.scm" 489 4)) (#f
("ice-9/eval.scm" 158 9)) (with-exception-handler ("ice-9/boot-9.scm" 1751 10))
(call-with-prompt ("ice-9/boot-9.scm" 723 2)) (#f (#f #f #f)) (#f
("guix/repl.scm" 98 21)) (with-exception-handler ("ice-9/boot-9.scm" 1751 10))
(with-exception-handler ("ice-9/boot-9.scm" 1746 15)) (#f ("guix/repl.scm" 125
7)))>)
In thread:
uncaught throw to %exception: (#< arguments: (wrong-type-arg
"primitive-load" "Wrong type argument in position ~A (expecting ~A): ~S" (1
"string" #f) (#f)) inferior: # stack: ((#f
("ice-9/boot-9.scm" 1779 13)) (raise-exception ("ice-9/boot-9.scm" 1682 16))
(raise-exception ("ice-9/boot-9.scm" 1684 16)) (primitive-load (#f #f #f))
(save-module-excursion ("ice-9/boot-9.scm" 2835 4)) (#f (#f #f #f)) (map1
("srfi/srfi-1.scm" 585 17)) (append-map ("srfi/srfi-1.scm" 672 15)) (#f
("gnu/ci.scm" 449 8)) (map1 ("srfi/srfi-1.scm" 585
Re: Golang go-updates feature branch?
Hi Leo! and hi Guixers, On Sun, Jan 08, 2023 at 02:22 PM, Leo Famulari wrote: > Hello! > > Now that our build farm is running smoothly, I propose we revive the > practice of feature branches, when appropriate. > Heartily agree here. This has come up a few times on #guix and generally with support (don't let me speak for everyone though). I think the idea of smaller and more frequent feature branches is a great idea: less code changes coming to master to review at once (compared to big core-updates merges), substitutes and closer to master to be easier for people to pull the branch and test, and some areas despite the number of builds would work nicely as a branch than just pushed (and lingering) into core-updates for much later. > The first one that I propose is a go-updates branch, where we update the > Go compilers. This will affect ~500 packages. > > If I understand correctly, Go is a relatively self-contained reverse > dependency graph within Guix and thus a good candidate for this > approach. It contains the Go libaries, and then some end-user > applications, and they don't really affect other packages. So, it should > be easy to understand when the update is ready to push. > I can't comment on the Go ecosystem, but that sounds good to me. Another one I would personally like to see is Mesa: they move quickly, newer versions are needed for recent hardware, and despite the number of rebuilds it causes Mesa is careful with removing older support. This is also an area where getting people to test by running their desktop system on a newer version would be helpful, and I think the feature branch approach makes it much easier (i.e. in core-updates there is a ton to test and look for). Anyway, perhaps that is a separate discussion and part of a larger discussion on number of rebuilds and how we classify changes. > I can set up a jobset on ci.guix.gnu.org if people like the idea. > > Leo So yes, I'm all for this general approach and think it is one that will make Guix better for the end-user and on the development side. Thanks for bringing this up Leo, something I've also been wanting to move towards! John
Golang go-updates feature branch?
Hello! Now that our build farm is running smoothly, I propose we revive the practice of feature branches, when appropriate. The first one that I propose is a go-updates branch, where we update the Go compilers. This will affect ~500 packages. If I understand correctly, Go is a relatively self-contained reverse dependency graph within Guix and thus a good candidate for this approach. It contains the Go libaries, and then some end-user applications, and they don't really affect other packages. So, it should be easy to understand when the update is ready to push. I can set up a jobset on ci.guix.gnu.org if people like the idea. Leo
