Re: Guix pull: avoiding "Computing Guix derivation"
Hi ! I may be a bit dense, but there's one part I don't understand: Ludovic Courtès writes: > Hi, > > Edouard Klein skribis: > >> - Why is this step not substitutable ? The inputs are known, a hash can >> be derived, a substitute server could be queried for an output of that >> hash ? What am I missing ? Does the guix derivation not end up in the >> store ? What makes it so special that it can't be served by a substitute >> server ? > > It’s not substitutable because it’s not a derivation. It’s not a > derivation because it needs to access the store to “compute the Guix > derivation”. > Doesn't it have hashable inputs ? Like the different git hashes of the different channels, plus the hash of whatever guix+guile revision is actually doing the work ? I think my misunderstanding boils down to a bad mental model of how guix works. What I imagine the "Computing guix derivation" step to be is no different than building a package whose inputs are the channels' sources, guile, and the current guix, and whose output is the new guix. Simon and Richard pointed out that the combinatorial combination of all possible input combination makes this hard to substitute, but I don't understand how its result, once in the store, can't lead to a simple linking from the profile to the store, instead of re-running the "Computing guix derivation" step when guix pull is called with the same inputs. Surely my mental model is wrong somehow ? > The latter could be addressed with “nested” or “recursive” derivations > as Nix calls them, but it’s quite a beast. > > Anther option would be to have a built-in derivation builder that would > do that process on behalf of users. That would turn the process into a > derivation, which is thus cachable, substitutable, etc. > >> - Is there a way (even a very dirty one, like hand copying stuff accross >> /var/guix/profiles/per-user/*/current-guix) I can stop paying this 4 >> minutes per user price ? As I said, this is downtime on my server, as >> I need to stop all other services to let guix pull finish. > > I don’t know of any quick hack for that. > > Thanks, > Ludo’.
Re: Guix pull: avoiding "Computing Guix derivation"
Thank you both for this detailed explanations. I think I have a better grasp at the problem. Keep in mind that I never not dive into guix's internals. I just - had previous experience with Gentoo and Source Mage (anyone remember this distro ?) who did some interesting steps towards reproductibility back in the day - knew what a Merkle dag is - read the manual and dived in with an overconfidence in my understanding of how this all works. To simplify the problem somewhat, bear in mind that a very good first step would be to avoid the "Computing Guix derivation" step, once it's been done already once and its result is "nothing to do". If that could happen, I could just run it as root once (4 minutes of downtime a week is manageable) and let the other users enjoy my cache. Simon Tournier writes: > Hi Richard, > > On lun., 13 mai 2024 at 20:52, Richard Sent > wrote: > >> You're correct. This solution wouldn't be sufficient to avoid "Computing >> Guix Derivation" for every possible A or B. To my understanding it could >> reduce the frequency this occurs. > > [...] > >> Assuming D changes significantly less frequently than A, B, C..., I >> would think this should be something we could feasibly substitute (at >> least for recent D and Z). > > Well, the package ’guix’ has changed 14 times over the past year. > Therefore, this D cannot be this package ’guix’, IMHO. > > $ git log --format="%cd %s" --since="1 year ago" | grep 'gnu: guix: Update' > Mon May 13 18:22:53 2024 +0200 gnu: guix: Update to 7ca9809. > Tue Mar 12 14:27:01 2024 +0100 gnu: guix: Update to 4c94b9e. > Mon Mar 11 23:14:37 2024 +0100 gnu: guix: Update to 8f4ffb3. > Sat Dec 2 15:37:44 2023 +0100 gnu: guix: Update to 1.4.0-16.aeb494322c. > Thu Nov 30 07:15:36 2023 +0100 gnu: guix: Update to 1.4.0-15.e0885fcfbb. > Thu Nov 9 10:42:55 2023 +0200 gnu: guix: Update to a60ff46. > Fri Oct 6 12:26:44 2023 +0200 gnu: guix: Update to e863274. > Thu Sep 28 11:44:08 2023 +0200 gnu: guix: Update to d0438fc. > Mon Sep 18 12:31:52 2023 +0200 Revert "gnu: guix: Update to > 1.4.0-11.658de25e99." > Mon Sep 18 06:49:46 2023 +0200 gnu: guix: Update to 1.4.0-11.658de25e99. > Tue Aug 22 21:30:49 2023 +0200 gnu: guix: Update to 1.4.0-10.4dfdd82210. > Tue Aug 22 11:17:52 2023 +0200 gnu: guix: Update to 30355c1. > Mon Oct 2 09:28:02 2023 +0200 gnu: guix: Update to 1.4.0-12.b9fae146d6. > Mon Aug 21 18:44:49 2023 +0200 gnu: guix: Update to 0e6215a. > Fri Jun 9 22:11:14 2023 +0200 gnu: guix: Update to 44bbfc2. > > > Maybe I have a bad practise but here my “guix pull” history: > > $ guix pull -l | grep Generation > Generation 1 nov. 17 2023 13:18:58 > Generation 2 déc. 11 2023 10:55:51 > Generation 3 févr. 02 2024 01:56:52 > Generation 4 mars 25 2024 18:22:25 > Generation 5 mai 13 2024 19:28:31(current) > > Therefore, I am not convinced that replacing "Computing Guix derivation" > (build-aux/build-self.scm) by the package ’guix’ would be robust enough. > > (Assuming another package ’guix’, lighter e.g., without requiring the > test suite, etc.) > > All that said, any experiment – even if it appears at first clunky – is > very welcome! This part will be improved only if there is a collective > effort / discussion / try, IMHO, i.e., by challenging the status quo. :-) > > Cheers, > simon
Re: Guix pull: avoiding "Computing Guix derivation"
Hi, Good advice, thank you ! I just began playing with btrfs, and I see that you can convert an existing ext4 to a btrfs, so I may try that (I'll just run some backups first). I was not aware that guix deploy does the computation on the local computer rather than on the target. I'll look more into it, thanks ! As for the user profiles, those exist because the users are real human users. The server in question is https://the-dam.org, a GNU Guix public access server. I'll keep this thread posted on how it went :) Cheers, Edouard. Ryan Sundberg writes: > Hi Edouard, I have couple of ideas for you, which may help with your > deployment process: > > 1) if possible, mount /gnu on a btrfs filesystem with zstd compression > enabled. This will let you fill up at least 3x of your physical disk space > because the store compresses very well. > > 2) for production servers you should try using `guix deploy` to push rather > than pull profiles. This will offload most of the process to the machine > pushing the deployment. > > With regards to multi user profiles, if this is a typical server (not a > terminal/shell server) I wonder why you need to have per user profiles at > all. I > only say this because I am not sure about running `guix deploy` on an > individual > user basis. > > Sincerely, > Ryan Sundberg > > May 13, 2024 8:08:35 AM Edouard Klein : > >> Hi Guix, >> >> First, I'd like to apologize for not having taken the time to answer >> those who helped me on a previous guix performance issue (with >> containers), the reason is tied to the topic of this email: the store >> has eaten all the space of my server, and solving that takes precedence >> over everything else, because no space == no services. >> >> So, I need to clear some space, and to do that I need to have every user >> run guix pull (and by that I mean root will sudo -u $user guix pull), >> then update all of their profiles, and then guix gc >> --delete-generations. >> >> This ought to turn deduplication up to 11, and enjoy a reduced store >> size. >> >> I've already solved the cache size problem: >> mount -t overlay overlay -o \ >> lowerdir="/root/.cache/guix",upperdir="/home/$user/.cache/guix-overlay",workdir="/home/$user/.cache/guix-workdir" >> \ >> "/home/$user/.cache/guix" >> >> Then >> bindfs --mirror=$user /home/$user/.cache/guix /home/$user/.cache/guix >> >> This lets root (who just ran guix pull) share its cache with every user, >> and avoid blasting away 700MB of disk space in every $HOME to reproduce >> the cache. >> >> However, now, I'm facing the previously addressed problem of guix pull >> being slow and hungry: >> https://www.mail-archive.com/guix-devel@gnu.org/msg66442.html (Guix pull >> speed) >> https://yhetil.org/guix/87h6mwf4u3@lapenas.dev/T/ (guix pull >> performance) >> >> On my server, in order to run guix pull, I have to stop all other >> services otherwise I run out of ram. >> >> Then, one root has pulled, I need to wait 4 minutes /per user/ for guix >> pull to finish its "Computing Guix derivation" step. >> >> I would like to know two things, one is for the sake of knowledge, and >> the other is to solve the problem at hand: >> >> - Why is this step not substitutable ? The inputs are known, a hash can >> be derived, a substitute server could be queried for an output of that >> hash ? What am I missing ? Does the guix derivation not end up in the >> store ? What makes it so special that it can't be served by a substitute >> server ? >> >> - Is there a way (even a very dirty one, like hand copying stuff accross >> /var/guix/profiles/per-user/*/current-guix) I can stop paying this 4 >> minutes per user price ? As I said, this is downtime on my server, as >> I need to stop all other services to let guix pull finish. >> >> Thanks in advance. >> >> Sorry for beating a dead horse, it's just that I can't scale anything up >> until I solve these performance issues. Sure I could rent a bigger >> server, but that's just kicking the can down the road. >> >> >> Cheers, >> >> Edouard.
Guix pull: avoiding "Computing Guix derivation"
Hi Guix, First, I'd like to apologize for not having taken the time to answer those who helped me on a previous guix performance issue (with containers), the reason is tied to the topic of this email: the store has eaten all the space of my server, and solving that takes precedence over everything else, because no space == no services. So, I need to clear some space, and to do that I need to have every user run guix pull (and by that I mean root will sudo -u $user guix pull), then update all of their profiles, and then guix gc --delete-generations. This ought to turn deduplication up to 11, and enjoy a reduced store size. I've already solved the cache size problem: mount -t overlay overlay -o \ lowerdir="/root/.cache/guix",upperdir="/home/$user/.cache/guix-overlay",workdir="/home/$user/.cache/guix-workdir" \ "/home/$user/.cache/guix" Then bindfs --mirror=$user /home/$user/.cache/guix /home/$user/.cache/guix This lets root (who just ran guix pull) share its cache with every user, and avoid blasting away 700MB of disk space in every $HOME to reproduce the cache. However, now, I'm facing the previously addressed problem of guix pull being slow and hungry: https://www.mail-archive.com/guix-devel@gnu.org/msg66442.html (Guix pull speed) https://yhetil.org/guix/87h6mwf4u3@lapenas.dev/T/ (guix pull performance) On my server, in order to run guix pull, I have to stop all other services otherwise I run out of ram. Then, one root has pulled, I need to wait 4 minutes /per user/ for guix pull to finish its "Computing Guix derivation" step. I would like to know two things, one is for the sake of knowledge, and the other is to solve the problem at hand: - Why is this step not substitutable ? The inputs are known, a hash can be derived, a substitute server could be queried for an output of that hash ? What am I missing ? Does the guix derivation not end up in the store ? What makes it so special that it can't be served by a substitute server ? - Is there a way (even a very dirty one, like hand copying stuff accross /var/guix/profiles/per-user/*/current-guix) I can stop paying this 4 minutes per user price ? As I said, this is downtime on my server, as I need to stop all other services to let guix pull finish. Thanks in advance. Sorry for beating a dead horse, it's just that I can't scale anything up until I solve these performance issues. Sure I could rent a bigger server, but that's just kicking the can down the road. Cheers, Edouard.
A paper about Plan 9 and Guix
Dear Guix developers, A paper of mine has been accepted as a Work in Progress at the next International Workshop on Plan 9 (http://iwp9.org/). I'll be presenting it not next week end, but the one after (12-14 April 2024). I'd be happy if some of you would be so kind as to read it with their extensive knowledge of Guix, in case I've made a mistake somewhere. https://the-dam.org/docs/explanations/Plan9ListenOnLinux.html Thanks in advance, Cheers, Edouard.
Re: guix --container is RAM hungry
Maxim Cournoyer writes: > Hi Ludovic, > > Ludovic Courtès writes: > >> Hi Edouard, >> >> Edouard Klein skribis: >> >>> I'm a huge fan of guix --container, and I created a system to use those >>> by default for network services. But the VPS these services run on has >>> only 2GB of RAM, and I just realized that a container, by default, >>> requires at least 200MB. >> >> Ouch, confirmed: >> >> $ \time -v guix shell -C coreutils -- uname 2>&1 |grep 'Maximum resident' >> Maximum resident set size (kbytes): 283048 >> $ \time -v guix shell coreutils -- uname 2>&1 |grep 'Maximum resident' >> Maximum resident set size (kbytes): 56588 >> $ guix describe >> Generation 297 Mar 24 2024 23:12:25(current) >> guix 28bc0e8 >> repository URL: https://git.savannah.gnu.org/git/guix.git >> branch: master >> commit: 28bc0e870b4d48b8e3e773382bb0e999df2e3611 >> >> >> As raingloom and Ricardo wrote, there’s a Guile process that keeps >> waiting. > > Is there a technical reason for this? Couldn't we replace the > current Guix process with 'exec', as hinted by Edouard? > Raingloom did, I just reported the problem without investigating the cause. > > If possible, that'd be the most direct way to avoid any of the memory > cost incurred by Guile/Guix. I stand ready to test any WIP patch, I'll take a look as well to see if I can guess where to replace a fork with an exec. Ricardo's breakdown of the calls will be helpful. Thank you all for having taken the report into account.
guix --container is RAM hungry
Dear Guixers, I'm a huge fan of guix --container, and I created a system to use those by default for network services. But the VPS these services run on has only 2GB of RAM, and I just realized that a container, by default, requires at least 200MB. Try it: guix shell time which -- bash -c "$(which time) -v guix shell --container" Then Ctrl-D, and look for this line: Maximum resident set size (kbytes): 291300 291MB of RAM to run bash. By contrast, removing the --container option from above yields: Maximum resident set size (kbytes): 64496 64MB, still a lot, but I can live with that. I tried various calls to unshare, but got no significant increase of RAM. >From an outsider's point of view, the --container option is a wrapper for unshare, and the dependency resolution is done by guix shell with or without the --container option, so I don't understand where the RAM explosion come from. I mean no implication that '--container' is simple or trivial to implement, I just wrestled with namespaces for a few weeks and I know they're a pain, to stay polite. I'm thankful for the tool and would like to use it more, but I can't in its current state. Any ideas ? Thanks in advance, Edouard.
Re: Introducing Guix "Features"! (Was: Syntactic Diabetes)
Weighting in here, as small as my weight may be: - re-using 'service' is IMHO a bad idea, it is a loaded term and the expectation of a new reader is that a service is a SysV-init service: it can be started, status-ed, stopped, restarted, and that's it. It maps to a daemon running in the background. Basically a shepherd service. - The fact that systemd service maps in functionality to guix services should be a huge redflag that the name is not good. Systemd sucks for many reasons, and using confusing and incosistent language is one of them. - I agree that 'features' is a worse name, way too generic. - Descriptive linguistics would consider the broader and historical use of the term in the UNIX crowd from which guix users are drawn, instead of the in-group use, which, despite being admitedly well documented and rooted in the history of the project, is overloading a close-but-not-exactly-matching term. I can not emphasize enough how much those subtle unexpected problems make adopting guix very hard. I pushed through because I understand that the project is fundamentally sound and worthwhile, but for a few years I have made interns and colleague work with guix and saw their motivation die by a thousand cuts because of confusing stuff like this. Finding a better term would be a worthwhile endeavour, for ease of adoption. Maybe we should dedicate a session for it in the next guix days ? - At os-declaration time, ALL guix services can be added, extended, modified, and removed. Discovering this and the difficult-to-write syntax that goes along with notably the extension, led me to the syntactic sugar we were initially discussing. - At os-reconfiguration time, SOME guix services will do their thing - At os-boot time, SOME guix services will do their thing - At os-running time, SOME (the shepherd kind) guix services can be stopped, started, statused, doced, custom-actioned, and restared - Understanding these behaviours may help use find an naming ontology that would foster understanding and adoption. Note that I don't disagree that service is a good name in *isolation*, I just want to emphasize, as Attila did, that ignoring beginner's expecations (however obsolete they may seem to be) is detrimental to the project's adoption. Cheers, Edouard. Felix Lechner writes: > On Thu, Nov 30 2023, Attila Lendvai wrote: > >> the use of 'service' to describe two rather different abstractions: a >> component of an OS vs. a deamon process run by shepherd. > > Indeed, the use of 'service' in much of Guix appears to be a grand > misnomer. It probably occurred because the meaning expanded over time. > > It's like we are looking back in time at the Big Bang. Our "services" > are the microwave echoes of Guix's initial, creative spark! > > Please consider a recent, helpful reply to help-guix. [1] Carlo > mentioned the term "service" eleven times, but none of them referred to > what I believe most readers of this message would call a service in > other contexts. What's a newbie on help-guix to think? > > Should Guix services instead be called "features"? > > Those "features" are central to any operating system definition. Other > choices like "provider" may not fully capture our collective uses > throughout the code and the documentation. I am especially thinking > about 'modify-features' and '%base-features'. > > Kind regards > Felix > > [1] https://lists.gnu.org/archive/html/help-guix/2024-01/msg00213.html
Re: Guix Days: Patch flow discussion
Skyler Ferris writes: > On 2/6/24 05:39, Steve George wrote: >> I agreed to organise some 'patch review' online sessions in the next couple >> of >> weeks. >> >> Organising a basic process is a good topic for that online session. For >> example, elsewhere in the thread someone mentions some tags we could use >> consistently so maintainers can find patches that have been reviewed easily. >> It >> would be great to agree those - try them for a bit - and document them in a >> 'howto' so that everyone uses the same process. > Have these been announced somewhere yet (eg, with url & exact time)? If > not will being subscribed to the help-guix list and/or checking the Guix > blog be sufficient to receive notification? As someone who has sent > patches in the past and intends to continue sending more in the future, > I'd like to do my part to keep the project in a good state. However, I > am new to interacting with large FLOSS projects so I'm nervous about > causing more problems than I solve if I just start doing things. Same here.
Re: Guix Days: Patch flow discussion
I, for one, would be willing to review patches, hoping that in turn my patches would be reviewed instead of staying in limbo forever, which is a drag on me submitting more patches. Is there a procedure to follow, or do I just start replying "LGTM" to patch email threads ? Cheers, Edouard. Steve George writes: > Hi, > > Our goal for the discussion: > > How do we double the number of patches that are *reviewed* and > *applied* to Guix in the next six months? > > Patch flow is a pipeline, to change it we could: > > a. Increase the number of committers - more people to do the > work > b. Increase the efficiency of existing committers > c. Open the gates by decreasing the quality expected from patches > > We essentially decided to focus our discussion on (b). We looked at > things that 'hinder' and 'help' patch review: > > > Hinders > > > - All our patch reviewers are volunteers doing it in their spare time. > > - For a volunteer reviewing someone else's work is not very rewarding, most > would prefer to use that precious time to scratch their own itch. > > - Can feel like an Sisyphean task: no matter how many patches someone reviews > there are more, exacerbated by the number of Guix packages. > > - Sense of responsibility: the minute that a reviewer looks at the patch they > are now stuck with it > > - Repetitive and boring: often patches have minor issues, but it's the same > sorts of issues time and time again. > > - Risk of negative social interaction: having to tell someone that their patch > is incorrect, or that their contribution cannot be used is difficult and > draining. Some people felt it was better to say nothing, rather than to > respond to a patch. > > > Helps > == > > This led us to the focus on the fact that **reviewing and applying > patches can be different people** > > We looked for ideas to create more reviewers, make reviewing easier and > more fun: > > > - Share in the work > > > 1. encourage new reviewers to step forward - making it more known that > reviewing > patches helps to get them applied. Anyone can review patches. > > 2. create directed 'how-to' documentation for reviewing and connect it to QA > so > that 'new reviewers' know what to do > > 3. create documentation about 'when' and 'how' it's appropriate to send a 'v2' > version of a patch so that the QA system builds and accepts it. Sometimes, > patches rot because non-committers don't want to be seen as 'stealing' > someone's > work with a v2 patch - but making the small changes and resubmitting to QA is > what is required. > > 4. Pay someone else to do it. Noted but out of scope. > 5. Remove old packages overhead. Old untouched packages create mental > overhead, > and make the task of maintaining the repository in a good state more > difficult. > We could remove old 'untouched' packages and ones that no-longer compile. We > have methods to hide and notify. > > > - Make it more fun > --- > > 1. do online sessions around reviews, some sprints or pairing - both social > and > a way to spread skills > 2. find ways to recognise and appreciate reviewers - 'reviewer of the month' > 3. make it a game - we could have a 'Guix London' vs 'Guix Paris' leader board > for reviews. Make it a group goal 'can we beat januarys reviews number' > 4. create some graphs / leaderboard so we know how many patches are being > reviewed and we can recognise the contribution > > > - Automate it away > --- > > 1. Chris is continuing to try and automate away the boring work - general > agreement in the group that QA has made a lot of difference. > > 2. general discussion about create a 'guix review' command (Nix has one) which > would download a branch with the appropriate patch and build it locally. This > is > for instances where some adjustment is needed or to check a build. While this > can be done today, it's a number of steps and quite involved. > > > Agreed Actions > == > > * [Chris]: continuing his work to improve QA automation. Implication was we'll > need some reports / graphs - but these were not discussed in detail. > > * [Futurile]: organise a **patch review online sessions**. To run every 13 > days > (so it rotates through the week) - for 3 months to see if it has any > traction. > Co-ordinate with maintainers so that patches that are reviewed can be > committed > > > Actions looking for someone - you? > > > * Carry forward the 'guix review' command idea > > * Write an RFC and discuss the idea of removing older 'bit-rot' packages > > * write 'how-to' documentation for reviews and when it's socially > acceptable to do a v2 patch. A checklist-like approach. > > > If you were in the discussion and I've misrepresented your point, or forgotten > an important aspect please please reply and correct me. > > Also, if you would like to help on any of the tasks please email back to the > group so
Re: Moment de convivialité Guix@Paris en nov… euh… décembre
Merci d'avoir organisé tout ça :) Tanguy LE CARROUR writes: > Bonjour Guix, > > Merci à ceux qui ont bravé les intempéries et les transports parisiens > pour venir hier soir ! Et, surtout merci à ceux qui, présents ou à > distance, on partagé leur travail, leurs découvertes ou leurs problèmes, > et ont contribué à faire de cette soirée un moment très… convivial ! > > Rendez-vous début janvier (sûrement le 11, à confirmer) pour la > prochaine ! > > En attendant, je vous souhaite à tou·tes une excellente journée, fin de > semaine et… fin d'année !
Re: Syntactic Diabetes (was Re: A friendlier API for operating-system declarations)
Liliana Marie Prikler writes:
>> (instantiate nginx)
> I do wish you spelled out service. Also, instantiate takes as much
> characters to type as add-service.
>
Done, see below. I was worried about the paronymy between add-service
and add-services which already exists. I defer to you and your
experience on this, naming things is hard.
>> To see the value of this syntactic sugar, try to replicate this MWE
>> with the standard syntax. It's not horrendous, but it *is* off-
>> putting to many newcomers to git, whereas this sugary piece is more
>> readable for them (sample size of 1, p=0.0005).
> Well, that'd be
> ...
I tried:
(let ((base (minimal-ovh "osef")))
(operating-system
(inherit base)
(services
(cons*
(service nginx-service-type)
(service mumble-server-service-type
(mumble-server-configuration
(welcome-text "couocu")
(port 64738)))
(service openssh-service-type
(openssh-configuration
(password-authentication? #f)
(allow-empty-passwords? #t)
(authorized-keys `(("alice" ,(local-file
"/home/edouard/.ssh/id_rsa.pub"))
(operating-system-user-services base)
I admit that this is as readable as the sweet version, but:
- Openssh is already defined in (minimal-ovh "osef"), so the build
fails with: 'guix system: error: service 'ssh-daemon' provided more
than once'
- you forgot the removal of guix-service, which admitedly is not used much in
practice
anyway
The problem with those two is that they break the nice structure of just
adding services, and now one has to first remove an element from
(operating-system-user-services base), then edit one of the element of
the resulting list, then add to elements to it. It is probably possible
to write it in a readable way, maybe less so than the sweet version, but
not so much as to justify adding the new macros to guix.
However the readability is not the main selling point, the writeability
is. Please do not discount how hard it is to write that kind of stuff
when scheme's not your main language. It is possible people here forgot
how hard this is for beginners, especially those like me whose brain is
deformed from years using algol-derived syntaxes.
I think we are losing a lot of mindshare because of the hard step of
learning both guile and guix, and the kind of syntactic sugar I
suggest may help bridge the gap long enough for newcomers to ease into
the syntax, after they get a taste of guix' capabilities.
Another experiment: with the sweet syntax, you can easily extend
services, without having to define a -record-type, etc. Just define a
function. I can write more at length about that if you want.
> On that note, we also have extend-openssh-authorized-keys for the use
> with modify-services.
>
I see it now in the source, but I was unaware of its existence.
Thanks for the swift again review :)
Cheers,
Edouard.
-mwe.scm
(use-modules
(beaver system)
(beaver functional-services)
(gnu packages version-control)
(gnu services web)
(gnu services telephony)
(gnu services ssh)
(gnu services base)
(guix gexp))
(-> (minimal-ovh "osef")
(add-service nginx)
(add-service mumble-server
(welcome-text "coucou")
(port 64738))
(extend-service openssh `(("alice" ,(local-file
"/home/edouard/.ssh/id_rsa.pub"
(modify-service openssh
(password-authentication? #f)
(allow-empty-passwords? #t))
(remove-service guix))
--functional-services.scm
(define-module (beaver functional-services)
#:use-module (gnu system)
#:use-module (gnu services)
#:export (add-service extend-service modify-service remove-service))
(define syntax->string (compose symbol->string syntax->datum))
(define (service-configuration stx service)
"Return the syntax one can use to refer to xxx-configuration for the given
service"
(datum->syntax stx (string->symbol
(string-append
(syntax->string service)
"-configuration"
(define (service-type stx service)
"Return the syntax one can use to refer to xxx-service-type for the given
service"
(datum->syntax stx (string->symbol
(string-append
(syntax->string service)
"-service-type"
(define-syntax add-service
(lambda (stx)
(syntax-case stx ()
[(_ os service-name)
(with-syntax
([service-type (service-type stx #'service-name)])
#'(begin
((lambda (x) ;; It is wrapped in a lamba to make sure os is
;; evaluated once only. It it wasn't in a labmda, whatever
;; form os is in the calling code would be repeated
;; multiple times, and so if the form was e.g. (some-func
;; os), then
Re: Syntactic Diabetes (was Re: A friendlier API for operating-system declarations)
Thank you Liliana and Attila for the swift and actionable feedback :)
Below is a revised proposition.
Here is a minimal working example of an os declaration:
--mwe.scm---
(use-modules
(beaver system)
(beaver functional-services)
(gnu packages version-control)
(gnu services web)
(gnu services telephony)
(gnu services ssh)
(gnu services base)
(guix gexp))
(-> (minimal-ovh "osef")
(instantiate nginx)
(instantiate mumble-server
(welcome-text "coucou")
(port 64738))
(extend openssh `(("alice" ,(local-file "/home/edouard/.ssh/id_rsa.pub"
(modify openssh
(password-authentication? #f)
(allow-empty-passwords? #t))
(remove guix))
---
To see the value of this syntactic sugar, try to replicate this MWE with
the standard syntax. It's not horrendous, but it *is* off-putting to
many newcomers to git, whereas this sugary piece is more readable for
them (sample size of 1, p=0.0005).
Here is the revised functional-services.scm, not yet commited and
pushed, and only lightly tested in local containers, but not in
production:
Advice and comments welcome :)
functional-services.scm--
(define-module (beaver functional-services)
#:use-module (gnu system)
#:use-module (gnu services)
#:export (instantiate extend modify remove))
(define syntax->string (compose symbol->string syntax->datum))
(define (service-configuration stx service)
"Return the syntax one can use to refer to xxx-configuration for the given
service"
(datum->syntax stx (string->symbol
(string-append
(syntax->string service)
"-configuration"
(define (service-type stx service)
"Return the syntax one can use to refer to xxx-service-type for the given
service"
(datum->syntax stx (string->symbol
(string-append
(syntax->string service)
"-service-type"
(define-syntax instantiate
(lambda (stx)
(syntax-case stx ()
[(_ os service-name)
(with-syntax
([service-type (service-type stx #'service-name)])
#'(begin
((lambda (x) ;; It is wrapped in a lamba to make sure os is
;; evaluated once only. It it wasn't in a labmda, whatever
;; form os is in the calling code would be repeated
;; multiple times, and so if the form was e.g. (some-func
;; os), then some-func would be called multiple times,
;; which may not be desirable.
(operating-system
(inherit x)
(services
(cons
(service service-type)
(operating-system-user-services x)
os)))]
[(_ os service-name forms ...)
(with-syntax
([service-type (service-type stx #'service-name)]
[service-configuration (service-configuration stx #'service-name)])
#'(begin
((lambda (x) ;; Wrapping in a lambda for the same reasons as above
(operating-system
(inherit x)
(services
(cons
(service service-type
(service-configuration forms ...))
(operating-system-user-services x)
os)))])))
(define-syntax extend
(lambda (stx)
(syntax-case stx ()
[(_ os service-name forms ...)
(with-syntax
([service-type (service-type stx #'service-name)])
#'(begin
((lambda (x)
(operating-system
(inherit x)
(services
(cons
(simple-service (format #f "A ~a extension" (syntax->string
#'service-name))
service-type
forms ...)
(operating-system-user-services x)
os)))])))
(define-syntax modify
(lambda (stx)
(syntax-case stx ()
[(_ os service-name forms ...)
(with-syntax
([service-type (service-type stx #'service-name)]
[service-configuration (service-configuration stx #'service-name)])
#'(begin
((lambda (x)
(operating-system
(inherit x)
(services
(modify-services (operating-system-user-services x)
(service-type
config =>
(service-configuration
(inherit config)
forms ...))
os)))])))
(define-syntax remove
(lambda (stx)
(syntax-case stx ()
[(_ os service-name forms ...)
(with-syntax
([service-type (service-type stx #'service-name)])
#'(begin
((lambda (x)
Syntactic Diabetes (was Re: A friendlier API for operating-system declarations)
web)) (define-os-macros-for tailon) (use-modules (gnu services web)) (define-os-macros-for varnish) (use-modules (gnu services xorg)) (define-os-macros-for dconf) (use-modules (gnu services xorg)) (define-os-macros-for gdm) (use-modules (gnu services xorg)) (define-os-macros-for localed) (use-modules (gnu services xorg)) (define-os-macros-for screen-locker) (use-modules (gnu services xorg)) (define-os-macros-for slim) (use-modules (gnu services xorg)) (define-os-macros-for xorg-server) (use-modules (gnu system install)) (define-os-macros-for configuration-template) (use-modules (gnu system install)) (define-os-macros-for cow-store) (use-modules (gnu system install)) (define-os-macros-for documentation) (use-modules (gnu system install)) (define-os-macros-for uvesafb) (use-modules (gnu system linux-container)) (define-os-macros-for dummy-networking) (use-modules (gnu system mapped-devices)) (define-os-macros-for device-mapping) (use-modules (gnu system pam)) (define-os-macros-for pam-root) (use-modules (gnu system pam)) (define-os-macros-for session-environment) (use-modules (gnu system shadow)) (define-os-macros-for account) (use-modules (gnu tests)) (define-os-macros-for marionette) (use-modules (guix scripts publish)) (define-os-macros-for publish) Edouard Klein writes: > Dear Guixers, > > For my clients and my own use, I use a layer on top of operating-system > declarations in which I create functions that take an os as a first > argument, and return an os. > > With the help of the handy -> macro, I can chain them, thus allowing an > easy stacking of multiple "roles" or "functionalities" on the same > machine, and an easy transfer of said functionality to another machine: > just cut and paste one line from one declaration to the other. > > I have written a tutorial for my clients here: > > https://guix-hosting.com/docs/tuto/getting-started.html > > that gives an example of what an os configuration may look like: > > #+begin_src scheme > (-> > (minimal-ovh "ssh-rsa AAASomethingSomething== root@minimal-ovh") > (http-static-content "sub2.example.com" #:to-dir "/srv/sub2") > (http-static-content "sub1.example.com" #:to-dir "/srv/sub1/") > (add-services my-db)) > #+end_src > > The code of the function is on my channel: > https://gitlab.com/edouardklein/guix/-/blob/beaverlabs/beaver/system.scm > > After a few months of experience, and positive feedback from my clients, > my question to you guys is: would you be interested in mainlining this, > or should I keep my development efforts separate in my channel ? > > I do think this API is easier than manipulating services, and although > extendable services are awesome and a very nifty piece of engineering, > they require quite a good knowledge of scheme and take a while to be > used to, while this new API, while way less powerful, lowers the barrier > to entry for newcomers. > > They are an easy way to maintain a declarative whole operating system > configuration, with a syntax similar enough to docker and ansible that > sysadmins familiar with it can quickly get up and running, thus exposing > more people to Guix. > > What do you think ? > > Cheers, > > Edouard.
Re: Moment de convivialité Guix@Paris en octobre
Je souhaite venir également :) J'avais raté l'annonce pour octobre. J'aurais deux trucs à présenter: - Du sucre syntaxique pour instancier, étendre, modifier, ou supprimer des services, - Comment monter des systèmes de fichier 9P au sein d'un guix container. C'est pas super formel, ce sont des recherches en cours, mais si je peux rétroprojeter mon écran et blablater pendant environ 20 minutes (en tout), ça serait super. Il faut apporter à manger/boire pour partager ? ++ Edouard. Tanguy LE CARROUR writes: > Quoting Tanguy LE CARROUR (2023-10-27 09:04:26) >> Vu que l'on a passé une bonne soirée, on va remettre ça ! À priori, >> le jeudi 1er décembre. La date sera confirmée sur ces listes une ou deux >> semaines avant. > > Erreur ! Il fallait lire « le jeudi 7 décembre ». > > Bonne journée,
Re: Can we provide another UI for patches than just email ?
Hi Simon, Saku Thank you for your answers. First, I should have slept on my email instead of sending it right away. I was still dealing with the frustration of having made a fool of myself with debbugs, which compounded with wrestling earlier with golang build system. My tone was unnecessarily harsh, please accept my apologies for that. I'd like to be part of the solution, so Saku, I'd like to help write the putative scripts you talk about. The main difficulty I see is that they need to be configured to be able to send and receive email on the user's behalf. I have some more patches to send over, I'll use that as an excuse to see what I can come up with. Second, I don't know what the general feeling is here towards debbugs, but if moving away from it is something that may happen, my suggestion is not to use anything with Pull Requests, as Simon you seem to have understood. I worded my suggestion poorly. What I had in mind was a (semi-)public-writable repo: code is pushed by developers, pulled and reviewed by maintainers, and if satisfactory, rebased on to master. If communication about the patch is needed, an email thread can be started on the mailing list, CCing the commit author(s) whose email address is known from the commits. A bash one liner starting with git log ... | sed ... can create the email automatically from the proposed branch. There is no need for any active code on the server, just SSH access or maybe public git:// access, and a place to store the repos. See https://gitlab.com/edouardklein/guix/-/blob/beaverlabs/beaver/system.scm?ref_type=heads#L526 the os/git function for an example on how to set that up on a guix server. See https://the-dam.org/docs/tutorials/git-hosting.html for how to set it up and https://the-dam.org/docs/tutorials/private-git.html for how to create repos once it is set up. Details need to be ironed out, such as whether to make it world-writable or to have a vetting process, whether to have a public-writable repo segregated from the official one, etc. but I think a reasonable configuration can be found. It seems simpler than git email with or without debbugs, for both maintainers and reviewers, but I'll admit I only have experience working with small teams, and not running a large and successful open source project, so I can't be very assertive there. Lastly, I think the lack of reviewers stems from the lack of contributors. I for one feel like I could give good reviews on trivial patches like new packages and package updates. Just check the repo is the official one, check the licence is good, and check that it, and its descendents, builds. I would gladly review a few packages a week, if I did not dread interacting with git email. Again, not insurmontable, but friction is a powerful thing. If we could onboard contributors more easily, we would have a larger pool of maintainers to pick from. Guix is freaking awesome, but very hard to pick up. It took me a few years to get up to speed. Anyway, thanks for the great software and for the time and guidance y'all give on help-guix and here. Sorry for the ranty email. Cheers, Edouard. Simon Tournier writes: > Hi Edouard, > > It is very important to speak up, although that makes me sad to read > such poor feedback experience. > > Well, the friction is about Debbugs. Maybe I repeat myself: Debbugs is > initially thought to be a bug tracker system and not a patch track system. > > Do not take me wrong, I am not trying to convince you. Instead, I am > just trying to explain that email workflow is not so much different and > the main annoyances you point come from Debbugs and not emails, > somehow. :-) > > On Thu, 14 Sep 2023 at 10:51, Edouard Klein wrote: > >> Before anybody tries to explain to me that git send-email is easier than >> I think, what you have to beat is: > > Well, the complete workflow you have in mind is: > > 1. $ git remote add guix-patches WHATEVER #only once > 2. $ git push -u guix-patches master:some-unique-name > 3. … send a Pull Request … #only once > > Instead, the email workflow somehow reads: > > a. … send an Email to guix-patc...@gnu.org #only once > b. $ git config sendemail.to 12...@debbugs.gnu.org #only once > c. $ git send-email --base=auto -v origin > > and the order is flipped: > > a == 3 #only once > b == 1 #only once > c == 2 > > Yeah, the order appears awkward. But that’s because of Debbugs; else it > would not be very different. > > The advantage is that contributing does not require from you to have a > public Git repository, it requires from you only an email address. > > The drawback (plural!), bah you already know them. :-) > > Again, thank you for your feedback. > > Cheers, > simon
Can we provide another UI for patches than just email ?
Dear Guix, I've just blown up Guix's debbugs by opening (and then closing) 33 bugs instead of sending a 33-commits patch series. I've also seen that there is a huge discussion on the cognitive overhead of contributing. I will read it as soon as I can spare more time on this. I find git email's workflow extremely frustrating and confusing. I know it works for most of you who actually commit, but I would like to point out that I'm one more person to be tripped by its complexity, and who cares enough to say so here. There are probably even more people who don't speak up. Therefore, there is a huge cost in keeping up using it: less people will be able to provide patches, and software will get stale. For example, ou current Go version is 1.17 ! From 2021 ! The current is 1.21. I suspect that if contributing was easier, we wouldn't have a 2 years delay in our versions. I do not want to open a flamewar about the merits of git email. I want to point out that I personnaly find it difficult and that I'm not the only one. Now, do we want to address this as a community, and welcome developers who may have good patches to contribute, but who can't spare at least ten hours to get confortable with git email ? An alternative solution would be to provide a public or semi-public git remote, and just let people git push their branches there. I suspect a git hook can then provide integration with debbugs if necessary, and I would be willing to assist writing it. In the current state, the UX is just unbearable, and I have in the past (and likely will again) just not bothered sending patches because the experience is so time consuming and infuriating. Before anybody tries to explain to me that git send-email is easier than I think, what you have to beat is: git remote add guix-patches WHATEVER #only once git push -u guix-patches master:some-unique-name You can then push, pull, rebase, whatever, from the command line (or magit in my case), without leaving your dev context. Can't be easier than that. Sorry about blowing up the debbugs. I'll do my best to avoid it in the future. Cheers,
Re: A friendlier API for operating-system declarations
Hi ! Ludovic Courtès writes: > Hi Edouard, > > Edouard Klein skribis: > >> For my clients and my own use, I use a layer on top of operating-system >> declarations in which I create functions that take an os as a first >> argument, and return an os. >> >> With the help of the handy -> macro, I can chain them, thus allowing an >> easy stacking of multiple "roles" or "functionalities" on the same >> machine, and an easy transfer of said functionality to another machine: >> just cut and paste one line from one declaration to the other. >> >> I have written a tutorial for my clients here: >> >> https://guix-hosting.com/docs/tuto/getting-started.html > > Very nice! > :) >> After a few months of experience, and positive feedback from my clients, >> my question to you guys is: would you be interested in mainlining this, >> or should I keep my development efforts separate in my channel ? > > I think it’s worth considering, taking into account maintenance > considerations as Liliana wrote. > Indeed. This was started as a quick and dirty overlay over guix to make it easier for people used to Ansible to deal with declarative OS configuration with guix, but it now need to be easy to maintain. >> I do think this API is easier than manipulating services, and although >> extendable services are awesome and a very nifty piece of engineering, >> they require quite a good knowledge of scheme and take a while to be >> used to, while this new API, while way less powerful, lowers the barrier >> to entry for newcomers. > > Normally, adding a service amounts to adding a line in the ‘services’ > field: > > (operating-system > ;; … > (services (append (list (service nginx-service-type) …) > %base-services))) > > It is more verbose that what you have, but it’s still roughly one line > (or one entry) per service. > You wouldn't believe the irrational fear of nested parentheses some people have ;-) I do agree a purely functional interface to OS declaration wouldn't bring much over the existing one. My goal is to then compose those primitive functions into usually-used-together bundles, such as e.g. nginx+gunicorn+flask in a function called, maybe, 'python-webapp'. This is kind of subjective, though. I have my own usage and some customer feedback to guide me, but we can keep it on my channel until we feel confident that the additional value is worth the maintenance cost. > Another interesting bit in your channel are templates though: an OS > template for hosting at OVH, etc. These are very valuable because they > take care of all the boilerplate. > I'd be happy to mainline it. I'll submit a patch when I get back home. > Thanks for sharing! > Well thanks to you for starting the whole schmilblick. > Ludo’. Edouard.
Re: A friendlier API for operating-system declarations
Hi Josselin, Josselin Poiret writes: > [[PGP Signed Part:Undecided]] > Hi Edouard, > > Edouard Klein writes: > >> #+begin_src scheme >> (-> >> (minimal-ovh "ssh-rsa AAASomethingSomething== root@minimal-ovh") >> (http-static-content "sub2.example.com" #:to-dir "/srv/sub2") >> (http-static-content "sub1.example.com" #:to-dir "/srv/sub1/") >> (add-services my-db)) >> #+end_src >> >> The code of the function is on my channel: >> https://gitlab.com/edouardklein/guix/-/blob/beaverlabs/beaver/system.scm >> >> After a few months of experience, and positive feedback from my clients, >> my question to you guys is: would you be interested in mainlining this, >> or should I keep my development efforts separate in my channel ? > > I am quite in favor of using operating-system transformations rather > than inheritance, because they're composable! This could be leveraged > to get a nice CLI API as well. > I hadn't thought of that but now that you mention it, starting a basic web server or database, isolated in a container, directly from the command line may come in handy sometimes. This is a good idea ! I'll think about how to achieve this :) In the meantime something like: #+begin_src bash guix system container -e "(begin (use-modules (beaver system)) (http-static-content minimal-container))" --share=somedir=/var/www #+end_src Will spin up such a server. Kind of like python3 -m http.server, but without the security flaws (you get a full blown nginx), and isolated in a way that even if your process gets owned, it's only an unpriviledged user in a container. >> I do think this API is easier than manipulating services, and although >> extendable services are awesome and a very nifty piece of engineering, >> they require quite a good knowledge of scheme and take a while to be >> used to, while this new API, while way less powerful, lowers the barrier >> to entry for newcomers. > > By this, do you mean that there's no way with your syntax to modify a > given service? Is there a reason for this? It does seem to me that it > could probably be done. > I'm not sure I understand your question. What I meant is that my proposal is only syntactic sugar over the existing system, and not meant to replace it. The existing system is actually very good and the way you can define how to collate the instances of a given service type, even if they are instanciated far away from one another, likely by different authors is really clever and well made. I'm talking here about things like how you can easily request that a new user be made for your service, and account-service-type will collate all the users that need to exist, and actually get the job done. Basically what fold-services does. I just want to hide all of this complexity for people who just want to activate one or more instances of a service coded up by somebody else. If by modifying a service you mean the parameters like the port ones listen to, or the directory the data should be put in etc. the functions just take keyword parameters and pass them to the underlying *-configuration scheme record. They are quite easy to edit and compose. Maybe we could even devise a semi-automated way of generating one function for every *-configuration structure that exists. See https://gitlab.com/edouardklein/guix/-/blob/beaverlabs/beaver/system.scm#L236 how http-static-content is basically just a wrapper over nginx-server-configuration. If you meant editing the service after the function has run, you then need to inspect the operating-system record and fall back to the original API, by finding the service in the record and editing its members, but that seems complex to me. I'm probably misunderstanding the use case. >> They are an easy way to maintain a declarative whole operating system >> configuration, with a syntax similar enough to docker and ansible that >> sysadmins familiar with it can quickly get up and running, thus exposing >> more people to Guix. >> >> What do you think ? > > You've got me interested :) especially since you already have customer > feedback! > Thanks :) Cheers, Edouard. > Best,
A friendlier API for operating-system declarations
Dear Guixers, For my clients and my own use, I use a layer on top of operating-system declarations in which I create functions that take an os as a first argument, and return an os. With the help of the handy -> macro, I can chain them, thus allowing an easy stacking of multiple "roles" or "functionalities" on the same machine, and an easy transfer of said functionality to another machine: just cut and paste one line from one declaration to the other. I have written a tutorial for my clients here: https://guix-hosting.com/docs/tuto/getting-started.html that gives an example of what an os configuration may look like: #+begin_src scheme (-> (minimal-ovh "ssh-rsa AAASomethingSomething== root@minimal-ovh") (http-static-content "sub2.example.com" #:to-dir "/srv/sub2") (http-static-content "sub1.example.com" #:to-dir "/srv/sub1/") (add-services my-db)) #+end_src The code of the function is on my channel: https://gitlab.com/edouardklein/guix/-/blob/beaverlabs/beaver/system.scm After a few months of experience, and positive feedback from my clients, my question to you guys is: would you be interested in mainlining this, or should I keep my development efforts separate in my channel ? I do think this API is easier than manipulating services, and although extendable services are awesome and a very nifty piece of engineering, they require quite a good knowledge of scheme and take a while to be used to, while this new API, while way less powerful, lowers the barrier to entry for newcomers. They are an easy way to maintain a declarative whole operating system configuration, with a syntax similar enough to docker and ansible that sysadmins familiar with it can quickly get up and running, thus exposing more people to Guix. What do you think ? Cheers, Edouard.
Re: GNU Guix maintainer rotation
Thank you Ludovic and Marius :) And good luck to Efraim ! Maxim Cournoyer writes: > Hello Guix! > > I'd like to bring your attention to a change to the current Guix > maintainers collective; in a nutshell, Ludovic and Marius are stepping > down from maintainer-ship while Efraim is joining. > > I won't write more as you can find all the details in this blog post: > https://guix.gnu.org/en/blog/2022/gnu-guix-maintainer-rotation/. > > Many thanks to Ludovic and Marius for their past roles as Guix > co-maintainers, and a warm welcome to Efraim! > > Happy hacking! > > Maxim
Re: generate commit messages for package updates
Thank you Ricardo !
Writing commit messages by hand is a major pain, especially for
newcommers like me. For simple commits it took me more time to write the
message than to write the code. The mistakes I made writing the commit
messages made the reviewers lose time correcting them. Automating this
task away is exactly what we need :)
Ricardo Wurmus writes:
> Hi Guix,
>
> I’m currently working on upgrading all Bioconductor packages to the 3.11
> release. The hardest work here is to write commit messages for the 200+
> packages that have changed. Even with the “update” yasnippet and magit
> this takes a very long time.
>
> I wrote a little tool to reduce the amount of time needed to create
> commits. It is not fully automatic yet as it requires my input to stage
> hunks, but the commit messages it produces take all input changes into
> account, which is something the “update” yasnippet does not do.
>
> After “guix refresh -t bioconductor -u”, manually verifying and
> implementing the suggested changes, and building all packages, I run
> “git add -p” to stage hunks that belong to the same package definition.
> Then I run the attached tool to make a commit:
>
> ./committer.scm | git commit -F -
>
> The tool works by looking at the unified diff in the staging area and
> generating two S-expressions corresponding to the original package
> definition and the changed package definition, respectively. It then
> looks at the inputs, propagated-inputs, and native-inputs fields and
> generates a GNU ChangeLog-style commit message describing the changes.
>
> Here’s an example diff:
>
> --8<---cut here---start->8---
> modified gnu/packages/bioconductor.scm
> @@ -2701,29 +2701,32 @@ gene and isoform level using RNA-seq data")
> (define-public r-karyoploter
>(package
> (name "r-karyoploter")
> -(version "1.12.4")
> +(version "1.14.0")
> (source (origin
>(method url-fetch)
>(uri (bioconductor-uri "karyoploteR" version))
>(sha256
> (base32
> -"03jmfgmw35hrgn3pc5lq6pblzhfx9fp4l6dx50rp303lr7kjxp9v"
> +"0h0gk4xd95k5phy6qcsv7j931d7gk3p24i2fg4mz5dsk110lpifs"
> (build-system r-build-system)
> (propagated-inputs
> - `(("r-regioner" ,r-regioner)
> + `(("r-annotationdbi" ,r-annotationdbi)
> + ("r-bamsignals" ,r-bamsignals)
> + ("r-bezier" ,r-bezier)
> + ("r-biovizbase" ,r-biovizbase)
> + ("r-digest" ,r-digest)
> + ("r-genomeinfodb" ,r-genomeinfodb)
> + ("r-genomicfeatures" ,r-genomicfeatures)
> ("r-genomicranges" ,r-genomicranges)
> ("r-iranges" ,r-iranges)
> - ("r-rsamtools" ,r-rsamtools)
> ("r-memoise" ,r-memoise)
> + ("r-regioner" ,r-regioner)
> + ("r-rsamtools" ,r-rsamtools)
> ("r-rtracklayer" ,r-rtracklayer)
> - ("r-genomeinfodb" ,r-genomeinfodb)
> ("r-s4vectors" ,r-s4vectors)
> - ("r-biovizbase" ,r-biovizbase)
> - ("r-digest" ,r-digest)
> - ("r-bezier" ,r-bezier)
> - ("r-bamsignals" ,r-bamsignals)
> - ("r-annotationdbi" ,r-annotationdbi)
> ("r-variantannotation" ,r-variantannotation)))
> +(native-inputs
> + `(("r-knitr" ,r-knitr)))
> (home-page "https://bioconductor.org/packages/karyoploteR/;)
> (synopsis "Plot customizable linear genomes displaying arbitrary data")
> (description "This package creates karyotype plots of arbitrary genomes
> and
> --8<---cut here---end--->8---
>
> …and this is the generated commit message:
>
> --8<---cut here---start->8---
> gnu: r-karyoploter: Update to 1.14.0.
>
> * gnu/packages/bioconductor.scm (r-karyoploter): Update to 1.14.0.
> [propagated-inputs]: Add r-genomicfeatures.
> [native-inputs]: Add r-knitr.
> --8<---cut here---end--->8---
>
> Obviously, this can be improved by avoiding the staging area and
> operating on all hunks in all selected files, so that more than one
> commit can be made at a time. But I thought I’d share this hack anyway,
> crude as it is.
Re: A script to check an edit does not break anything
An now with the attachment.
#!/bin/bash
# Put this script in a subfolder (e.g. gpwc)
# of the guix source
# And install sharness in it
# https://github.com/chriscool/sharness
# Install prove as well
# https://linux.die.net/man/1/prove
# To run the tests, invoke
# prove gpwc.t
# Fill in the names of the packages you've been meddling with
# In my case it was python-prompt-toolkit
PACKAGES=(python-prompt-toolkit python-ipywidgets)
test_description="Checking my meddling with ${PACKAGES[*]} did not break anything"
. ./sharness.sh
# Check that the new packages themselves are OK
for package in "${PACKAGES[@]}"
do
test_expect_success "Build $package" "
guix environment guix --pure -- ${SHARNESS_BUILD_DIRECTORY}/pre-inst-env guix build $package
"
test_expect_success "Lint $package" "
guix environment guix --pure -- ${SHARNESS_BUILD_DIRECTORY}/pre-inst-env guix lint $package
"
test_expect_success "See if $package is reproducible" "
guix environment guix --pure -- ${SHARNESS_BUILD_DIRECTORY}/pre-inst-env guix build --rounds=2 python-websockets
"
test_expect_success "Install $package" "
guix environment guix --pure -- ${SHARNESS_BUILD_DIRECTORY}/pre-inst-env \
guix install --profile=./${package}-tmp-profile ${package}
"
done
# Check the consequences of having meddled with the new packages
dependents(){
guix environment guix --pure -- "${SHARNESS_BUILD_DIRECTORY}"/pre-inst-env \
guix graph --type=reverse-bag "$1" | \
grep @ | sed 's/.*\"\(.*\)@.*/\1/'
}
# Union of all the dependents
for package in "${PACKAGES[@]}"
do
dependents "$package"
done | sort -u > dependents.txt
for package in $(cat dependents.txt)
do
test_expect_success "Build dependent $package" "
guix environment guix --pure -- \"${SHARNESS_BUILD_DIRECTORY}\"/pre-inst-env guix build $package
"
profile="$SHARNESS_TRASH_DIRECTORY/$(echo $package | sed 's/\(.*\)@/\1'/)-tmp-profile"
test_expect_success "Install dependent $package" "
guix environment guix --pure -- ${SHARNESS_BUILD_DIRECTORY}/pre-inst-env \
guix install --profile=\"${profile}\" ${package}
"
done
test_done
Edouard Klein writes:
> Dear Guixers,
>
> I recently broke jupyter while updating python-prompt-toolkit (a mistake
> I'm still trying to fix...) despite doing my best to follow the
> instructions of the manual:
> https://guix.gnu.org/manual/en/html_node/Packaging-Guidelines.html
> https://guix.gnu.org/manual/en/html_node/Submitting-Patches.html#Submitting-Patches
>
> Because of the subtle consequences of editing a package are hard to keep
> track of, I wrote the attached script (you need sharness to run it, but
> with slight modification it can become a standalone script).
>
> This script goes beyond the instructions of the manual on two fronts:
> - First, it not only tries to build the packages, but also to install
> them,
> - Secondly, it does not limit itself to the dependents (as listed by
> guix refresh --list-dependents) of the packages one is meddling with,
> but to the whole reverse bags (as listed by guix graph
> --type=reverse-bag).
>
> Both these extensions are necessary to discover that my update broke the
> jupyter package:
> - jupyter builds, so if you don't try to install it you won't discover
> it's broken
> - the leafs of the dependency graph that depend on jupyter fail to build
> for reasons unrelated to python-prompt-toolkit, so by just building the
> leafs (which are what is being returned by guix refresh
> --list-dependents) you can't discover jupyter's broken status.
>
> I think an update of the manual is in order (I've added it to my own
> TODO queue, but I have no idea when I'll be able to get to it) but I
> first wanted to share my script in case it may be useful to someone
> else, and gather some feedback as to whether other developers have
> encountered these kind of errors, and the way seasoned developers make
> sure they don't push breaking changes.
>
> Cheers,
>
> Edouard.
A script to check an edit does not break anything
Dear Guixers, I recently broke jupyter while updating python-prompt-toolkit (a mistake I'm still trying to fix...) despite doing my best to follow the instructions of the manual: https://guix.gnu.org/manual/en/html_node/Packaging-Guidelines.html https://guix.gnu.org/manual/en/html_node/Submitting-Patches.html#Submitting-Patches Because of the subtle consequences of editing a package are hard to keep track of, I wrote the attached script (you need sharness to run it, but with slight modification it can become a standalone script). This script goes beyond the instructions of the manual on two fronts: - First, it not only tries to build the packages, but also to install them, - Secondly, it does not limit itself to the dependents (as listed by guix refresh --list-dependents) of the packages one is meddling with, but to the whole reverse bags (as listed by guix graph --type=reverse-bag). Both these extensions are necessary to discover that my update broke the jupyter package: - jupyter builds, so if you don't try to install it you won't discover it's broken - the leafs of the dependency graph that depend on jupyter fail to build for reasons unrelated to python-prompt-toolkit, so by just building the leafs (which are what is being returned by guix refresh --list-dependents) you can't discover jupyter's broken status. I think an update of the manual is in order (I've added it to my own TODO queue, but I have no idea when I'll be able to get to it) but I first wanted to share my script in case it may be useful to someone else, and gather some feedback as to whether other developers have encountered these kind of errors, and the way seasoned developers make sure they don't push breaking changes. Cheers, Edouard.
Re: What is guix-package-cache ?
Hi Tobias,
Tobias Geerinckx-Rice writes:
> Afternoon Edouard,
>
> Edouard Klein 写道：
>> I don't understand why guix feels the need to try to build it.
>
> So Guix isn't trying to build any packages (yet), just complaining that you're
> still using python2-prompt-toolkit-1 somewhere. I'm inclined to believe it:
>
>> The log says:
>> (repl-version 0 1 1)
>> Generating package cache for
>> '/gnu/store/zdv0j5x5dfwr4b6bh8jc9vp818lvd38g-profile'...
>> (exception unbound-variable (value #f) (value "Unbound variable: ~S") (value
>> (python2-prompt-toolkit-1)) (value #f))
>
> Does ‘find -name \*.scm -exec grep python2-prompt-toolkit-1 {} +’ in your
> channel's directory return anything useful?
>
It does, it turns out that two packages are still using
python2-prompt-toolkit-1. I was certain that I grepped that to make sure
but I obviously messed up somehow.
Thank you very much for your help, and my apologies for the noise.
> Are there any uncommitted changes to your channel(s) that ‘guix pull’ can't
> see
> yet?
>
>> I grepped for 'guix-package-cache' and tried to read the code in
>> guix/channels.scm, but I don't understand.
>
> It's a simple cache of package metadata from your channel(s) to speed up
> future
> Guix operations. I doubt it's related to your problem; it's just what
> happened
> to fail first.
>
> Kind regards,
>
> T G-R
What is guix-package-cache ?
Hi all ! I'm working on packaging some software. Because I'm not done submitting my patches to the official guix, but I still need to use my new packages I've created a git repo where I push my edits. I've created a channel to use this git repo instead of the official guix repo. When I tried to run git pull, I got: The following derivation will be built: /gnu/store/saxml7kzw0ndy0s76483gwxhsxm27ybh-profile.drv The following profile hook will be built: /gnu/store/ggkmzkjw6bj75kq45k7p9bv33si8k66f-guix-package-cache.drv building package cache... /builder for `/gnu/store/ggkmzkjw6bj75kq45k7p9bv33si8k66f-guix-package-cache.drv' failed to produce output path `/gnu/store/ljadsvwvqfq7hv6kxr02d3fihgzi8wr6-guix-package-cache' build of /gnu/store/ggkmzkjw6bj75kq45k7p9bv33si8k66f-guix-package-cache.drv failed View build log at '/var/log/guix/drvs/gg/kmzkjw6bj75kq45k7p9bv33si8k66f-guix-package-cache.drv.bz2'. cannot build derivation `/gnu/store/saxml7kzw0ndy0s76483gwxhsxm27ybh-profile.drv': 1 dependencies couldn't be built guix pull: error: build of `/gnu/store/saxml7kzw0ndy0s76483gwxhsxm27ybh-profile.drv' failed The log says: (repl-version 0 1 1) Generating package cache for '/gnu/store/zdv0j5x5dfwr4b6bh8jc9vp818lvd38g-profile'... (exception unbound-variable (value #f) (value "Unbound variable: ~S") (value (python2-prompt-toolkit-1)) (value #f)) python2-prompt-toolkit-1 is a package I removed in the last push in my channel because no other package uses it. I did not install it, nor anything that depend on it (because nothing depends on it). I don't understand why guix feels the need to try to build it. I grepped for 'guix-package-cache' and tried to read the code in guix/channels.scm, but I don't understand. Does anybody have any pointers ? Cheers, Edouard.
Re: What to do when udpating a package ?
zimoun writes: > Dear Edouard, > > In complement to what Julien said. :-) > > > On Fri, 15 May 2020 at 13:36, Edouard Klein wrote: > >> - Some packages would not build, for reasons unrelated to the update of >> python-prompt-toolkit. Namely: >> - fdroidserver@1.1.1 will not build because of a failure of >> python-apache-libcloud@2.4.0 >> - python-matplotlib-documentation@3.1.2 and >> python-ipython-documentation@7.9.0 will not build because of a failure >> of texlive-union-51265 >> - python-rpy2@3.0.4-1.19868a8 would not build >> All those failure also happen on origin/master on my machine, and are >> therefore unrelated to the changes I made. >> --> Is any failure to build expected on origin/master ? > > No, it is not expected. > As Forrest Gump says: "It happens, sometimes." :-) > I was worried I had misconfigured my local version of guix. > >> --> If so, where can I check the official build success/fail status of >> a package ? > > Well, the link is ci.guix.gnu.org then top right, for example the query: > >spec:guix-master system:x86_64-linux python-apache-libcloud > > But personally I do not always understand what Cuirass reports. > Therefore, I prefer to use data.guix.gnu.org then for example: > > http://data.guix.gnu.org/repository/1/branch/master/package/fdroidserver/output-history > > I do not know the correct way to reach this webpage so I type directly > the URL -- well to be exact, I have an Emacs interactive function so I > type M-x my/guix-data fdroidserver RET and it opens the above webpage. > > Then I click to one "Failed dependency" and I arrive to: > > http://data.guix.gnu.org/build-server/1/build?derivation_file_name=/gnu/store/ccrk10g4vpzf6nk7x8j1a36s0b4z0w2l-fdroidserver-1.1.1.drv > > and I can click to the failing derivation and then I arrive to > ci.guix.gnu.org and I can give a look to the raw log. > I could not find the link to the raw log, but having access to the "official" build status is a huge relief, as I can stop worrying that the build failure is my fault. This is exactly what I was looking for. Thank you ! > > Well, I do not know if it is the correct way, but it is how I am doing > time to time to see what happens on the build farm. Be careful of > your current Guix version and the Guix version on ci you are > examining. > > >> --> If not, how can I check that my master version of guix is correct >>and is building everything it should build ? > > I do not understand what you mean. > "guix weather" to see if the substitute is available. > "guix challenge" to compare your own builds to the builds on substitutes. > > I was asking how to make sure that my local guix is OK (I had trouble getting it to work). >> --> In any case, how can I check that my change does not negatively >>affect these packages ? > > Which packages? The ones which do not currently build? Yeah, I'm worried maybe once the current build failure cause is addressed, my change will prevent them from building. > > >> - Some of the failing-in-master packages do not depend directly on >> python-prompt toolkit, but they depend on a package that depends on >> etc. >> e.g. fdroidserver@1.1.1 depends on python-androguard@3.2.1 which >> depends on python-ipython@7.9.0. >> --> fdroidserver@1.1.1 fails to build for reasons unrelated to >> python-prompt-toolkit, but both python-androguard@3.2.1 and >> python-ipython@7.9.0. both build with the new version of >> python-prompt-toolkit. Would I be correct in assuming that >> frdoidserver would build with the new version ? I assume not, but I >> would like to be sure > > I miss something about what depends on what. :-) > I do not have really the graph under my eyes. ;-) > > >> - python-iml@0.6.2 does not build with prompt-toolkit in version 3 (the >> version is explicitly stated as >=2.0 and <2.1 in the Python files), but >> when I add prompt-toolkit-2 to its propagated-inputs, it does. >> Given that python-ipython has prompt-toolkit (implicitly in version >> 3) installed in its propagated inputs, this means that any environment >> with python-iml will have prompt-toolkit in both its version 2 and 3 >> installed at the same time. I feel uneasy about this. >> --> Is this a problem ? >> --> Should I just trust that if it builds, then everything is >> allright ? > > No, it should pass the tests too. If there is tests. :-) > If not, you should install it locally and try it. Yeah, installing does not
Re: What to do when udpating a package ?
Hi Julien, Thank you for your answer. Julien Lepiller writes: > Le 15 mai 2020 03:20:06 GMT-04:00, Edouard Klein a > écrit : >>Dear Guix Developers, >> >>I have a few beginner questions. >>Attached to this email you will find the "reverse-package" graph of >>python-prompt-toolkit. > > Hi, > > In general, some packages on master may fail to build. We try to fix these, > but it's not always easy. > > When updating a package, you should make sure that its dependents all > build or did not build before your changes. Fix those who now fail to > build. OK, got it. I'll avoid touching those that fail to build on the build farm. > > I don't think having both versions is a good thing because it will create > conflicts when installing (have you tried to install a package that > propagates both to a profile?). It would be ok if they had no file in common > but I doubt it is the case. For any package that requires version 2, make > sure its dependencies only use version 2, or update the package if the newer > version can support version 3. It's not always easy to find the right order > of upgrades, as you should make sure nothing is broken in between patches. > I did not try to install them, and you were correct, here is what I get when I try to install python-iml: guix install: error: profile contains conflicting entries for python-prompt-toolkit guix install: error: first entry: python-prompt-toolkit@3.0.5 /gnu/store/80lzvbzvfp4226ic7czhch4p0mlsdwlv-python-prompt-toolkit-3.0.5 guix install: error:... propagated from python-ipython@7.9.0 guix install: error:... propagated from python-iml@0.6.2 guix install: error: second entry: python-prompt-toolkit@2.0.7 /gnu/store/0k7a0yp3b2sqqj8jhl7vp3cabb0x2mwd-python-prompt-toolkit-2.0.7 guix install: error:... propagated from python-iml@0.6.2 hint: You cannot have two different versions or variants of `python-iml' in the same profile. The problem is that python-iml depends on both python-ipython (which accepts python-prompt-toolkit 3) and python-prompt-toolkit-2. Looking at the github repo, the last update was in 2018, I don't think we'll see an update soon. I could pin python-ipython to python-prompt-toolkit-2, but that would just delay the problem and put it on somebody else's lap to let python-ipython move to python-prompt-toolkit 3. As I was typing a question I ctrl-Fed for 'variants' in the doc and ended up learning about package-input-rewriting. I will try to make python-iml depend on python-ipython, but with the prompt-toolkit input replaced with its version 2 on the fly. I think it makes sense. Is it the correct way to do what needs to be done ? > Relying on propagated inputs to provide a dependency is going to > simplify the graphs, but not the work of other maintainers who will > have to investigate how the dependency is provided, so I don't think > it's a good idea. I understood your sentence as saying that relying on propagated inputs of propagated inputs is not a good idea and that dependencies are better explicitly stated in the guix package. Is that correct ? > > You should rebuild every dependent, even those who only depend on the > package for native-inputs, since there can be an error an any point > (though less likely). OK, I'll try that next when I'll have gotten python-iml to build. > > I hope I answered your questions. Your message was split into two > multipart sections and my client wasn't able to cite the interesting > part, which makes it hard for me to check what your questions were > while typing my answer. You have indeed answered a lot of them, thank you very much :) Sorry about the multipart stuff, I don't know how to configure my client (mu4e) not to do that. I'll look into it. Cheers, Edouard.
What to do when udpating a package ?
Dear Guix Developers, I have a few beginner questions. Attached to this email you will find the "reverse-package" graph of python-prompt-toolkit. reverse.pdf Description: Adobe PDF document I am updating it from version 2.0.7 to version 3.0.5 because python-questionary (which I'm trying to add to guix) depends on version 3. After writing the new package definition for version 3.0.5, I tried building the packages listed by ./pre-inst-env guix refresh --list-dependent python-prompt-toolkit which if I understand correctly are the leafs of the attached DAG. This is where I ran into different problems. Here are the questions I have: - Some packages would not build, for reasons unrelated to the update of python-prompt-toolkit. Namely: - fdroidserver@1.1.1 will not build because of a failure of python-apache-libcloud@2.4.0 - python-matplotlib-documentation@3.1.2 and python-ipython-documentation@7.9.0 will not build because of a failure of texlive-union-51265 - python-rpy2@3.0.4-1.19868a8 would not build All those failure also happen on origin/master on my machine, and are therefore unrelated to the changes I made. --> Is any failure to build expected on origin/master ? --> If so, where can I check the official build success/fail status of a package ? --> If not, how can I check that my master version of guix is correct and is building everything it should build ? --> In any case, how can I check that my change does not negatively affect these packages ? - Some of the failing-in-master packages do not depend directly on python-prompt toolkit, but they depend on a package that depends on etc. e.g. fdroidserver@1.1.1 depends on python-androguard@3.2.1 which depends on python-ipython@7.9.0. --> fdroidserver@1.1.1 fails to build for reasons unrelated to python-prompt-toolkit, but both python-androguard@3.2.1 and python-ipython@7.9.0. both build with the new version of python-prompt-toolkit. Would I be correct in assuming that frdoidserver would build with the new version ? I assume not, but I would like to be sure - python-iml@0.6.2 does not build with prompt-toolkit in version 3 (the version is explicitly stated as >=2.0 and <2.1 in the Python files), but when I add prompt-toolkit-2 to its propagated-inputs, it does. Given that python-ipython has prompt-toolkit (implicitly in version 3) installed in its propagated inputs, this means that any environment with python-iml will have prompt-toolkit in both its version 2 and 3 installed at the same time. I feel uneasy about this. --> Is this a problem ? --> Should I just trust that if it builds, then everything is allright ? - Same question on a larger scale with r-irkernel@1.1: - it depends on python-jupyter-console (failing) which depends on python-prompt-toolkit. If I pin python-jupyter-console's dependency on python-prompt-toolkit to version 2, python-jupyter-console builds. The next dependency to fail is then python-widgetsnbextension@3.4.2, whose Python files explicitly requires 2.0 <= python-prompt-toolkit <2.1, but whose guix package does not (it relies on python-prompt-toolkit being a propagated input of a propagated input of a propagated etc.). Same for python-ipywidgets@5.2.2. Those packages build when I explicitly add python-prompt-toolkit-2 as a propagated input. Now I feel uneasy about two things: --> Is the concurrent presence in the environment of python-prompt-toolkit both in version 2 and 3 a problem ? --> Is the fact that the graph is now much more connected (2 extra edges) a problem ? --> Is there a way to check the graph to make the edges as sparse as possible (i.e. remove as many edges as possible without changing the reachability) ? Would this be something we want ? According to me it would because it would make the packages definitions shorter and the computations on the graph faster, but I'm not sure. I'm sorry if this is addressed in the documentation, I did not find the relevant parts. guix graph is awesome, it makes working with the dependencies much more intuitive. This email is a mess to anyone not looking at the graph at the same time, but I hope is quite understandable to somebody following along on the graph at the same time. Last question: --> Should I also check the build status of the packages for which python-prompt-toolkit or its dependents are native-inputs ? Why, why not ? Thanks in advance, Cheers, Edouard
Re: Not sure my patch reached guix-patches
Well thanks for spam-checking, that's an arduous job :) ! Cheers, Edouard. Tobias Geerinckx-Rice writes: > Edouard, > > Thanks for patching Guix! > > Edouard Klein 写道： >> Never mind, I just got the answer from the automated system with the bug >> number and I see it in the archive now. It just takes a little more time >> than I expected. > > The automated system isn't entirely automated ;-) > > Posts by new addresses are spam-checked by a human. This applies to all Guix > mailing lists, which is why your ‘hmm what's up’ and ‘oh never mind’ mails > will > hit the list at exactly the same time. > > Kind regards, > > T G-R
Not sure my patch reached guix-patches
Hi all, I've sent my first patch (adding a Python package) to the guix-patches mailing list as instructed in the documentation, but I don't see my message in the list archive https://lists.gnu.org/archive/html/guix-patches/2020-05/index.html and I don't have any acknowledgment of the message. I just wanted to make sure it's not lost in the ether. Cheers, Edouard.
Re: Not sure my patch reached guix-patches
Never mind, I just got the answer from the automated system with the bug number and I see it in the archive now. It just takes a little more time than I expected. Sorry for the noise. Edouard Klein writes: > Hi all, > > I've sent my first patch (adding a Python package) to the guix-patches > mailing list as instructed in the documentation, but I don't see my > message in the list archive > > https://lists.gnu.org/archive/html/guix-patches/2020-05/index.html > > and I don't have any acknowledgment of the message. I just wanted to > make sure it's not lost in the ether. > > Cheers, > > Edouard.
