Estimated overhead of building an orthogonal Musl-based LFS within Guix build system
Hi, I'm wondering what the overall estimated work or effort might look like to leverage Guix to build a co-existing family of packages that are in some sense "orthogonal" to the rest of Guix, based upon different package versions and perhaps musl libc - similar to https://github.com/dslm4515/CMLFS for example. Could a series of such packages be built up in the same way that these LFS type builds bootstrap themselves? For example, starting with the most primitive dependencies and going on upward. For this to work, different package versions for the same kind of package would need to coexist - which I don't believe is inherently a problem. But also, these builds would need to refer exclusively to paths and prefixes that are wholly self-contained and orthogonal to the rest of Guix. The overall aim here is to consider building some select packages for example with musl libc, or perhaps building a "stable track" of software that is unaffected by the rest of Guix evolving packages. The measurement of effort can be subjective. Perhaps it involves modifying existing recipes and adapting these to point to different packages/versions. Maybe there is a similar precedent somewhere. Any thoughts are appreciated.
Re: Merging core-updates?
Hi guix-devel! On Sun, Feb 12, 2023 at 03:49 PM, Josselin Poiret wrote: > Hi everyone, > > Andreas Enge writes: > >> I volunteer to follow your lead, but also have no clue what is actually >> expected. > > I would also like to give a hand! > Count me in as well! I only did some spot fixes the last round, but at least I have some familiarity with what happened. I don't have any particular expertise but I'm happy to help coordinate overall, test, and generally do some cat herding. And now that I have commit access I hope I can really help move things through with review and pushing ready patches. Unfortunately had some other stuff come up for the last few months since I got access, but I should have more time starting in a week or so. On that front, I think looking through relevant core-updates patches (the Mesa ones in particular, for me) is a good first step for review and I'll try building on my end to see how far I get. >> [...] >> Actually I am wondering whether the first step of killing these untamed >> non-feature branches would not be to build and merge staging? It is based >> on master, supposed to contain only medium sized changes, but which I >> suspect end up being a world-rebuilding cluster of changes. > > You're right, for this to smoothly transition into the "feature branch > workflow" (if that's actually what we want to do) I guess we also need > to lay out a plan first, and prepare for the post-c.u-merge > world. Getting rid of staging first could be an easier exercise, > followed by c-u. I was planning on taking your notes of the Guix days > and opening a discussion about how we could concretely transition to > this new workflow, I can maybe do that this evening. > I need to catch up on the thread about feature branches and I'm looking forward to that as well. It has something that I have long wanted and will gladly help in that transition and doing what I can to make that go smoothly. John
Re: Moving forward with teams and feature branches (was: Discussion notes on releases and branches)
Hello Josselin, Am Sun, Feb 12, 2023 at 10:13:35PM +0100 schrieb Josselin Poiret: > For a potential roadmap (doesn't have to be sequential): > 1. Document this workflow in the manual, in a dedicated node, with a >rationale as well. One thing worth mentioning would be how to handle >grafting/ungrafting now. Also remove the staging/core-updates >criterion. > 2. Teams start organizing around the features they are currently working >on, and document them accordingly. They can also draft how they work, >what they do and their roadmap if they wish. > 3. CI/QA gets examined and updated to support the new workflow. We test >it out with a sample feature. > 4. staging merge happens, and the branch gets deleted. > 5. core-updates merge happens, and the branch gets deleted or >repurposed (up to the core team). This (and the other points you wrote) sound good to me. I would start with 4. and 5., and in parallel consider 1. to 3. Andreas
Moving forward with teams and feature branches (was: Discussion notes on releases and branches)
Hi Andreas and thanks for taking the notes during the discussion! I think what came out of that discussion was very interesting and that it would greatly help the scaling problems Guix is facing as well as maintainer/contributor burnout, but that also means that we need to create some actionnable plan out of it. The window before the eventual core-updates merge would be the best time for it, IMO! For that plan to materialize, the best would be to agree on 1) what we want the new workflow to be, and 2) how to get there. To start the discussion, here's what I got out of the discussion (and is probably opinionated): The envisioned goal state would be to stop relying on staging/core-updates to manage non-trivial changes to Guix, but rather have specific branches that could be merged way faster, because they are focused on specific features that people can feel responsible for, instead of a hodgepodge of unrelated patches that no one wants to debug. These feature branches would be managed by teams, with one person from the team being responsible for the feature and it getting merged into master. Those features could be ephemeral or long-running and each team would be free to organize around them as they see fit, since e.g. a Stack upgrade is very different from a Mesa upgrade, or a gnu-build-system change. With the improved CI/QA we have now, we shouldn't have to worry too much since substitutes will be available right away (we should weigh what's acceptable for people using --no-substitutes vs. keeping software more up-to-date though, but my opinion is that in favor of the latter). Teams should thus better document how they work and what they do, either in the Guix documentation or a specific wiki/manual that would be maintained by team members themselves. Having all information in one place would help outsiders find their way around the inner workings of the project. Maintaining roadmaps for each team, with who's working on them and where that work can be found would also lure unsuspecting bystanders into working on Guix features, as well as structure the future of Guix; it shouldn't be too codified though, since we don't want to further burden precious team members. One nice thing about documenting all of this using a manual is that the changes go through the guix-patches ML, so that they can be discussed and recorded for the public to see. Note that this change would also mean that contributors are no longer responsible for gauging whether a patch belongs on master/staging/c-u, but rather the reviewer/committer. Also, as a nice side-effect of the change, I can see people getting interested in joining teams because they structure longer-term effort that's put into Guix, not just because they've been maintaining python-foo-for-bar for 5 years against their will. Regarding releases: a release team would have to keep in touch with what the other teams are doing, make some sort of periodic status report, and set deadlines such as feature freezes before preparing a new release. For a potential roadmap (doesn't have to be sequential): 1. Document this workflow in the manual, in a dedicated node, with a rationale as well. One thing worth mentioning would be how to handle grafting/ungrafting now. Also remove the staging/core-updates criterion. 2. Teams start organizing around the features they are currently working on, and document them accordingly. They can also draft how they work, what they do and their roadmap if they wish. 3. CI/QA gets examined and updated to support the new workflow. We test it out with a sample feature. 4. staging merge happens, and the branch gets deleted. 5. core-updates merge happens, and the branch gets deleted or repurposed (up to the core team). I bet I forgot a bunch of things, but at least we get the ball rolling! WDYT? NB: Just noticed there is no tooling/infrastructure team, this would probably be a good idea, to publicize the incredible work that is being put into all of it (mumi/cuirass/qa front page/the data service), as well as bring in some new souls and document how they all fit together. Best, -- Josselin Poiret signature.asc Description: PGP signature
Re: branch core-updates updated: gnu: coreutils-boot0: Fix building on arm architectures.
On Thu, Dec 15, 2022 at 03:56:24PM +0100, Ludovic Courtès wrote: > Hi, Sorry for taking a while to get back to answer this. > Efraim Flashner skribis: > > > On December 12, 2022 1:47:29 PM UTC, "Ludovic Courtès" wrote: > > [...] > > >>> + ,@(if (target-arm?) > >>> + ;; Some binaries fail to build. > >>> + `(#:configure-flags '(,(string-append > >>> +"--enable-no-install-program=" > >>> +;; the defaults > >>> +"arch,coreutils,hostname" > >>> +;; fails on aarch64 > >>> +",timeout,sort"))) > >> > >>Isn’t there a risk that things will break down the road if ‘sort’, > >>‘hostname’, etc. are missing? How hard would it be to address the build > >>failure instead? > >> > >>Thanks, > >>Ludo’. > > > > I built all the way out to hello without any problems. Also %final-inputs > > has coreutils-final, so it really shouldn't be noticable. > > That’s odd though. Isn’t there some upstream patch we could take? > Surely ‘sort’ has no reason to contain arch-specific code? > > If there’s no such patch, we can go with the patch above, but then there > should be a comment linking to bug reports and reassuring the reader > that yes, some packages do build even without these commands. :-) In file included from src/timeout.c:53:0: /gnu/store/4fy2658sxphy1kgclxrrmcka6lwiwap0-glibc-bootstrap-0/include/sys/prctl.h:22:66: fatal error: linux/prctl.h: No such file or directory #include /* The magic values come from here */ I'm not sure if it's because armhf and aarch64 are using %bootstrap-glibc vs glibc-2.16, but I didn't see this problem with glibc-mesboot or with the %bootstrap-glibc from architectures using 2.31. Other workarounds I thought of were adding in a glibc-boot0 here to replace libc for everybody or using an older version (the last one in the 8 series), but this seemed like the least invasive option for now. I've also added a comment so it'll be clearer what's happening there. > (There’s no “coreutils” command BTW, unless enabling the > everything-in-one-binary trick, no?) It turns out coreutils does allow for that. > Thanks, > Ludo’. -- Efraim Flashner אפרים פלשנר GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted signature.asc Description: PGP signature
Re: Merging core-updates?
Andreas Enge writes: > Am Sun, Feb 12, 2023 at 12:58:06PM +0100 schrieb Julien Lepiller: >> And I was able to rebuild (with --check) patch-mesboot. The error looks >> a lot like https://issues.guix.gnu.org/49985. We should fix that indeed >> :) > > Ah indeed, that looks like deal breaking; maybe someone from MES can have > a look? I have backported DRAFT lib: stat: Use SYS_stat64 for 32bit platforms.wip to `wip' https://git.savannah.gnu.org/cgit/mes.git/commit/?id=05d10877410b48d0a4f8960fd62190d70329eea7 that was slated to become Mes 0.25, after adding initial risc-v support and some thorough testing. If this works, maybe we should postpone initial risc-v support to 0.26. > What is the magic incantation with double "@@" to build this package? > Ah, here we go, for reference to self: >guix build -e '(@@ (gnu packages commencement) patch-mesboot)' Thanks. It would be nice, however, if there was a way for me to reproduce this bug. Greetings, Janneke -- Janneke Nieuwenhuizen | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com
Re: Merging core-updates?
Hi, --- Original Message --- On Sunday, February 12th, 2023 at 5:08 PM, Andreas Enge wrote: > > > Am Sun, Feb 12, 2023 at 12:58:06PM +0100 schrieb Julien Lepiller: > > > And I was able to rebuild (with --check) patch-mesboot. The error looks > > a lot like https://issues.guix.gnu.org/49985. We should fix that indeed > > :) > > > Ah indeed, that looks like deal breaking; maybe someone from MES can have > a look? > > What is the magic incantation with double "@@" to build this package? > Ah, here we go, for reference to self: > guix build -e '(@@ (gnu packages commencement) patch-mesboot)' > > Andreas While not directly related to the patch-mesboot error, I want to mention that there is also https://issues.guix.gnu.org/58719 blocking i686 builds on core-updates (and x86_64 builds of certain packages like wine64, which has i686 dependencies) since the update to glibc 2.35. It may also need assistance from the MES folks to fix, since the error message is about an undefined symbol in glibc-mesboot's libpthread.so.0: make[2]: Entering directory '/tmp/guix-build-file-5.44.drv-0/file-5.44/magic' ../src/file -C -m magic /tmp/guix-build-file-5.44.drv-0/file-5.44/src/.libs/file: symbol lookup error: /gnu/store/s4yd6ibxsh5q1j9ipygb9vpjj4g00wc9-glibc-mesboot-2.16.0/lib/libpthread.so.0: undefined symbol: h_errno, version GLIBC_PRIVATE make[2]: *** [Makefile:863: magic.mgc] Error 127 Cheers, Kaelyn P.S. I also have https://issues.guix.gnu.org/59453 open for few months to fix the Vulkan layer paths in mesa on core-updates, which I'd really like to see land before a core-updates merge happens (it had originally been part of a mesa version update patch I sent in back in October, but which had been obsoleted by a newer patch release of mesa landing in core-updates; mesa in core-updates is also a few releases behind at this point, but that's a separate matter).
Re: Licence of the Guix blog posts
Csepp writes: > I thought I agreed when I sent my part, but in case I haven't: > I agree to licensing your contribution under CC-BY-SA 4.0 and GFDL > version 1.3 or later, with no Invariant Sections, no Front-Cover Texts, > and no Back-Cover Texts. s/your/mine/ I copied the text from the email I was sent and apparently didn't edit it properly.
Re: Merging core-updates?
Am Sun, Feb 12, 2023 at 12:58:06PM +0100 schrieb Julien Lepiller: > And I was able to rebuild (with --check) patch-mesboot. The error looks > a lot like https://issues.guix.gnu.org/49985. We should fix that indeed > :) Ah indeed, that looks like deal breaking; maybe someone from MES can have a look? What is the magic incantation with double "@@" to build this package? Ah, here we go, for reference to self: guix build -e '(@@ (gnu packages commencement) patch-mesboot)' Andreas
Re: Merging core-updates?
On Sun, Feb 12, 2023 at 10:05:40AM +0100, Julien Lepiller wrote: > Hi Guix! > > As discussed at Guix Days before Fosdem, we haven't merged core-updates > in a very long time. I'd volunteer to lead this effort, but I don't > know what steps I should follow. Do we have some documentation about > that? I think we normally have a 2 week last-chance window to get all sorts of last minute packages bumped and then we freeze it and try to build "everything". -- Efraim Flashner אפרים פלשנר GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted signature.asc Description: PGP signature
Re: adding motif to guix
Hi! Not a review, but how is *tif ‘not an X toolkit’...? Why not just say it ‘uses *tif’, instead? Kind regards, T G-R Sent on the go. Excuse or enjoy my brevity.
Re: Merging core-updates?
Hi everyone, Andreas Enge writes: > I volunteer to follow your lead, but also have no clue what is actually > expected. I would also like to give a hand! > [...] > Actually I am wondering whether the first step of killing these untamed > non-feature branches would not be to build and merge staging? It is based > on master, supposed to contain only medium sized changes, but which I > suspect end up being a world-rebuilding cluster of changes. You're right, for this to smoothly transition into the "feature branch workflow" (if that's actually what we want to do) I guess we also need to lay out a plan first, and prepare for the post-c.u-merge world. Getting rid of staging first could be an easier exercise, followed by c-u. I was planning on taking your notes of the Guix days and opening a discussion about how we could concretely transition to this new workflow, I can maybe do that this evening. Best, -- Josselin Poiret signature.asc Description: PGP signature
Re: Merging core-updates?
Julien Lepiller writes: > As discussed at Guix Days before Fosdem, we haven't merged core-updates > in a very long time. I'd volunteer to lead this effort, but I don't > know what steps I should follow. Do we have some documentation about > that? I can try and help with this, at least in terms of helping get bordeaux.guix.gnu.org substitute availability to a good level. I'd also like to get the kind of comparison that the qa-frontpage can do for patches working for branches, but that'll probably take a bit of work in the data service/qa-frontpage to get working smoothly. signature.asc Description: PGP signature
Re: Merging core-updates?
Julien Lepiller writes: > Le Sun, 12 Feb 2023 12:52:51 +0100, > Julien Lepiller a écrit : > >> Le Sun, 12 Feb 2023 12:06:14 +0100, >> Andreas Enge a écrit : >> >> I just tried to build mpc on my machine, from core-updates. I get the >> same derivation as the one shown on the data service, and it built >> fine. Maybe there's something wrong with the machines behind bordeaux? >> I probably got substitutes from berlin, for gcc and friends (though I >> built gmp, mpfr and mpc). > > And I was able to rebuild (with --check) patch-mesboot. The error looks > a lot like https://issues.guix.gnu.org/49985. We should fix that indeed > :) It could be something "wrong" with a build machine, although my suspicion for a while is that various things are broken/don't work on btrfs, which milano-guix-1 (the most important x86_64-linux) build machine uses. I've filed a few issues that probably relate to this: https://issues.guix.gnu.org/53415 https://issues.guix.gnu.org/53416 https://issues.guix.gnu.org/60202 There are other build machines that don't use btrfs, so it should be possible to get these things built anyway. signature.asc Description: PGP signature
Re: Merging core-updates?
On Sun, Feb 12, 2023 at 10:05:40AM +0100, Julien Lepiller wrote: > As discussed at Guix Days before Fosdem, we haven't merged core-updates > in a very long time. I'd volunteer to lead this effort, but I don't > know what steps I should follow. Do we have some documentation about > that? The process could be considered "free form". Basically, you can try to reconfigure your system on core-updates. It will fail, many build failures will have to be fixed, and eventually it will work. Then I would run the Guix test suite and the system tests, and also try to make use of QA. It's likely that different architectures will have some specific problems to be fixed.
Re: Merging core-updates?
Le Sun, 12 Feb 2023 12:52:51 +0100, Julien Lepiller a écrit : > Le Sun, 12 Feb 2023 12:06:14 +0100, > Andreas Enge a écrit : > > I just tried to build mpc on my machine, from core-updates. I get the > same derivation as the one shown on the data service, and it built > fine. Maybe there's something wrong with the machines behind bordeaux? > I probably got substitutes from berlin, for gcc and friends (though I > built gmp, mpfr and mpc). > And I was able to rebuild (with --check) patch-mesboot. The error looks a lot like https://issues.guix.gnu.org/49985. We should fix that indeed :)
Re: Merging core-updates?
Le Sun, 12 Feb 2023 12:06:14 +0100, Andreas Enge a écrit : > Hello, > > Am Sun, Feb 12, 2023 at 10:05:40AM +0100 schrieb Julien Lepiller: > > As discussed at Guix Days before Fosdem, we haven't merged > > core-updates in a very long time. I'd volunteer to lead this > > effort, but I don't know what steps I should follow. Do we have > > some documentation about that? > > I volunteer to follow your lead, but also have no clue what is > actually expected. > > With Chris's help, I scheduled an evaluation on the build coordinator, > which has not finished yet, so there is not yet anything to see at > > http://data.qa.guix.gnu.org/gnu/store/bjyfp7bhb9j25sw62nrkqjiivnzjxija-mpc-1.3.1.drv > But if you click down the dependency tree, some of them have been > built. And the first one already failed: > > http://data.qa.guix.gnu.org/gnu/store/0p0zs4gv8vg7rny25ki1szv4c7yk9zzb-patch-mesboot-2.5.9.drv > With a strange error: > starting phase `build' > make: stat:Makefile: sterror: unknown error > make: *** No targets specified and no makefile found. Stop. > error: in phase 'build': uncaught exception: > srfi-34 # exit-status: 2 term-signal: #f stop-signal: #f] 1439cc0> phase > `build' failed after 0.0 seconds command "make" failed with status 2 > Having failed three times in a row, this does not look like a > transient error. > > I also reconfigured bayfront to do more builds itself. > > Actually I am wondering whether the first step of killing these > untamed non-feature branches would not be to build and merge staging? > It is based on master, supposed to contain only medium sized changes, > but which I suspect end up being a world-rebuilding cluster of > changes. > > Andreas > I just tried to build mpc on my machine, from core-updates. I get the same derivation as the one shown on the data service, and it built fine. Maybe there's something wrong with the machines behind bordeaux? I probably got substitutes from berlin, for gcc and friends (though I built gmp, mpfr and mpc).
Re: Merging core-updates?
Hello, Am Sun, Feb 12, 2023 at 10:05:40AM +0100 schrieb Julien Lepiller: > As discussed at Guix Days before Fosdem, we haven't merged core-updates > in a very long time. I'd volunteer to lead this effort, but I don't > know what steps I should follow. Do we have some documentation about > that? I volunteer to follow your lead, but also have no clue what is actually expected. With Chris's help, I scheduled an evaluation on the build coordinator, which has not finished yet, so there is not yet anything to see at http://data.qa.guix.gnu.org/gnu/store/bjyfp7bhb9j25sw62nrkqjiivnzjxija-mpc-1.3.1.drv But if you click down the dependency tree, some of them have been built. And the first one already failed: http://data.qa.guix.gnu.org/gnu/store/0p0zs4gv8vg7rny25ki1szv4c7yk9zzb-patch-mesboot-2.5.9.drv With a strange error: starting phase `build' make: stat:Makefile: sterror: unknown error make: *** No targets specified and no makefile found. Stop. error: in phase 'build': uncaught exception: srfi-34 # phase `build' failed after 0.0 seconds command "make" failed with status 2 Having failed three times in a row, this does not look like a transient error. I also reconfigured bayfront to do more builds itself. Actually I am wondering whether the first step of killing these untamed non-feature branches would not be to build and merge staging? It is based on master, supposed to contain only medium sized changes, but which I suspect end up being a world-rebuilding cluster of changes. Andreas
Re: Licence of the Guix blog posts
Ludovic Courtès writes: Hello! Thanks Simon for pinging me! > You might remember that I started long ago asking people who had > contributed to the blog whether they would agree to licensing their work > under CC-BY-SA 4.0 and GFDL version 1.3 or later, with no Invariant > Sections, no Front-Cover Texts, and no Back-Cover Texts¹. [..] > Simon, what do you think about emailing the authors of the “10 years of > stories” post asking if they agree with the licensing? :-) No rush, > though the sooner the more likely we are to get an answer. Sure, I'm happy for my blog contributions to be licensed under these terms. Greetings, Janneke -- Janneke Nieuwenhuizen | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com
Merging core-updates?
Hi Guix! As discussed at Guix Days before Fosdem, we haven't merged core-updates in a very long time. I'd volunteer to lead this effort, but I don't know what steps I should follow. Do we have some documentation about that?
