Workflow ideas [Was: branch master updated: gnu: eudev: Use new package style.]
Hi Attila, On Tue, May 30, 2023 at 12:49 PM Attila Lendvai wrote: > > maybe with the addition of team branches a core-updates branch would be much > more manageable? and it'd be easier to merge it more frequently? I think even fewer people will feel responsible for that unloved branch. Instead, I would change the workflow: 1. Only folks who provide substitutes or their delegates can push to master. That would ensure the highest availability of substitutes the project can muster. I would rename the master branch to 'ready' or 'installable'. The people I would call "the farmers". 2. All team branches have to go through core-updates. That would strike a balance between individual and project responsibility. Like the "Three Musketeers," it would be "all for one, and one for all" at that point. The pressure to make the branch clean will be felt from all parts of the project. I would also rename the core-updates branch to 'testing'. 3. When fixes do not introduce new features, like security or documentation patches, the authors may appeal to "the farmers" for direct adoption, thereby sidestepping 'testing'. The farmers can delegate that work. The delegates I would call "the polishers". 4. No more pushes to core-updates (other than reversals) after the second Wednesday of each month. With that workflow, I think we would merge core-updates every one to three weeks. People can revert. Just another goofy idea from an odd fellow. Kind regards Felix
Re: How many bytes do we add (closure of guix) when adding one new package?
On Tue, 30 May 2023, Simon Tournier wrote: Just to put a figure on what means “big”: currently the .go files are 5 times bigger than their associated .scm. Somehow, it’s the trap of DSL. :-) Packages are declarative and the information they declare is not dense. However, because they are bytecompiled to a general programming language, their specificity is not exploited. In an ideal world, the compiled binary representation of the packages should be smaller than their human-readable text-file counterpart. The mentioned improvement is nice. And it’s visible: --8<---cut here---start->8--- 145M /gnu/store/nqrb3g4l59wd74w8mr9v0b992bj2sd1w-guix-d62c9b267-modules/lib/guile/3.0/site-ccache/gnu 117M /gnu/store/s6rqlhqr750k44ynkqqj5mwjj2cs2yln-guix-a09968565-modules/lib/guile/3.0/site-ccache/gnu 127M /gnu/store/ndii4bpyzh2rc05ya61s89rig9hdrl4k-guix-a0178d34f-modules/lib/guile/3.0/site-ccache/gnu 164M /gnu/store/ni63a203jf61dwxlv8kr9b8x3vb1pdsp-guix-8e2f32cee-modules/lib/guile/3.0/site-ccache/gnu --8<---cut here---end--->8--- This is probably a tagent, sorry, but I was curious how well the .go files compressed. It seems quite well, actually: jackhill@mimolette ~/.config/guix/current/lib/guile/3.0/site-ccache/gnu [env]$ sudo compsize . Processed 595 files, 1659 regular extents (1659 refs), 0 inline. Type Perc Disk Usage Uncompressed Referenced TOTAL 21% 36M 173M 173M none 100% 16K 16K 16K zstd21% 36M 173M 173M Best, Jack
Re: branch master updated: gnu: eudev: Use new package style.
> > As for reverting it, I'm somewhat indifferent. I'm more interested in > > the longer term cost of making changes like this than the temporary > > drops in substitute availability. > > You mean as a precedent for similar commits in the future or as a way > of involuntarily breaking other packages? As already stated, I only > pushed the commit because I was quite sure that all rebuilds would > succeed. a less frequent drop in substitute availability on master would considerably elevate my satisfaction with guix as a user. and in my reading that is the cost that Christopher talks about here. it's rather disruptive on my workflow when i want to deploy something from my channel to my servers, but due to missing substitutes i need to edit the channels.scm file, pick a commit in the past... and make sure i pick it in sync with That-Other-Channel... then hope that the past commit i picked is covered by substitutes, because currently it's not predictable which commits are fully built by the build farm... then issue a `guix pull --allow-downgrades` to revert the guix channel... and then repeat it all if i picked a commit that is also not covered well... my 0.02 is that it shouldn't be an either-or choice between 1) teams and topic branches, and 2) a core-updates-like staging branch. the latter should be employed, if for nothing else, to keep substitute availability high on master. maybe with the addition of team branches a core-updates branch would be much more manageable? and it'd be easier to merge it more frequently? then it could be a home for commits that are semantically simple, but expensive to build. -- • attila lendvai • PGP: 963F 5D5F 45C7 DFCD 0A39 -- “If he controls your ideas he will soon control your actions, because every action is preceded by an idea.” — Silvano Arieti (1914–1981), 'The Will to Be Human'
Re: How many bytes do we add (closure of guix) when adding one new package?
Simon Tournier writes: > Hi, > > On ven., 26 mai 2023 at 18:21, Ludovic Courtès wrote: > >> I agree that .go files are quite big (.scm files as well, but we’ve >> improved information density somewhat by removing input labels :-)). >> >> The size of .go files went down when we switch to the baseline compiler >> (aka. -O1): >> >> https://lists.gnu.org/archive/html/guix-devel/2020-06/msg00071.html >> >> That thread has ideas of things to do to further reduce .go size. > > Just to put a figure on what means “big”: currently the .go files are 5 > times bigger than their associated .scm. > > Somehow, it’s the trap of DSL. :-) Packages are declarative and the > information they declare is not dense. However, because they are > bytecompiled to a general programming language, their specificity is not > exploited. In an ideal world, the compiled binary representation of the > packages should be smaller than their human-readable text-file > counterpart. > > The mentioned improvement is nice. And it’s visible: > > --8<---cut here---start->8--- > 145M > /gnu/store/nqrb3g4l59wd74w8mr9v0b992bj2sd1w-guix-d62c9b267-modules/lib/guile/3.0/site-ccache/gnu > 117M > /gnu/store/s6rqlhqr750k44ynkqqj5mwjj2cs2yln-guix-a09968565-modules/lib/guile/3.0/site-ccache/gnu > 127M > /gnu/store/ndii4bpyzh2rc05ya61s89rig9hdrl4k-guix-a0178d34f-modules/lib/guile/3.0/site-ccache/gnu > 164M > /gnu/store/ni63a203jf61dwxlv8kr9b8x3vb1pdsp-guix-8e2f32cee-modules/lib/guile/3.0/site-ccache/gnu > --8<---cut here---end--->8--- > > However, it has almost no impact on the whole size; scaled by the number > of packages. > >> Download size has to be treated separately though. For example, ‘git >> pull’ doesn’t redownload all of the repo or directory, and it uses >> compression heavily. Thus, a few hundred bytes of additional .scm text >> translate in less than that. >> >> As for the rest, download size can be reduced for example by choosing a >> content-address transport, like something based on ERIS. >> >> I think we must look precisely at what we want to optimize—on-disk size, >> or bandwidth requirement, in particular—and look at the whole solution >> space. > > I think one direction is to tackle the way *package-modules* is built. > Because of that, Guix is building too much and the design is not optimal > – whatever technical solutions we implement for improving after that. > > On my poor laptop, Guix is becoming unusable because many operations are > becoming so slow – when it’s still acceptable with APT of Debian. For > instance, it’s something like 20 minutes for running “guix pull” without > substitutes. And when I am traveling without a fast Internet > connection, it’s often too much for the network at hand. > > Currently, “guix pull” is either building too much and downloading too > much; by design. > > > Cheers, > simon Something I've been considering is if Guix could make use of database optimizations on its packages. Having access to Scheme for everything is nice, but using it as a storage solution is kind of silly when we are mostly just storing structs. Some kind of struct-of-arrays optimization could definitely reduce their size by a lot, might even speed up some operations. It makes zero sense to load full package definitions from disk for most queries, such as guix search, with an SoA representation we could load only the fields that we care about. ps.: Now I'm even more glad that I'm using a file system with transparent compression on all my Guix systems.
Re: branch master updated: gnu: eudev: Use new package style.
Am Montag, dem 29.05.2023 um 20:29 +0100 schrieb Christopher Baines: > > Liliana Marie Prikler writes: > > > Am Montag, dem 29.05.2023 um 19:28 +0100 schrieb Christopher > > Baines: > > > > > > guix-comm...@gnu.org writes: > > > > > > > This is an automated email from the git hooks/post-receive > > > > script. > > > > > > > > lilyp pushed a commit to branch master > > > > in repository guix. > > > > > > > > The following commit(s) were added to refs/heads/master by this > > > > push: > > > > new 7ff003bcbf gnu: eudev: Use new package style. > > > > 7ff003bcbf is described below > > > > > > > > commit 7ff003bcbf388677c7c85b1709c58f41f84b9947 > > > > Author: Felix Lechner > > > > AuthorDate: Sun May 28 16:28:20 2023 -0700 > > > > > > > > gnu: eudev: Use new package style. > > > > > > > > * gnu/packages/linux.scm (eudev)[arguments]: Convert to > > > > list of > > > > G-Expressions. > > > > [native-inputs]: Drop labels. > > > > > > > > Signed-off-by: Liliana Marie Prikler > > > > > > > > --- > > > > gnu/packages/linux.scm | 94 -- > > > > > > > > > > > > 1 file changed, 45 insertions(+), 49 deletions(-) > > > > > > These changes do look to affect the derivation for eudev, and > > > eudev also has too many dependents to update on the master > > > branch. > > > > > > → guix refresh -l eudev > > > Building the following 1913 packages would ensure 4138 dependent > > > packages are rebuilt: ... > > Should I revert it? Even with the mass rebuild, my intuition was > > that it'd be okay since it's no functional change and core-updates > > was dropped in favour of teams (with no particular team being > > responsible for udev afaik). > > I think there is disagreement about this (including at least some > maintainers thinking differently), but I'm of the opinion that while > there has been discussion about stopping using core-updates, people > should follow the currently documented process, and for now that's > still using staging/core-updates [1]. Fair enough, I will refrain from pushing mass rebuilds directly to master. > 1: > https://guix.gnu.org/manual/devel/en/html_node/Submitting-Patches.html#index-rebuild-scheduling-strategy > > As for reverting it, I'm somewhat indifferent. I'm more interested in > the longer term cost of making changes like this than the temporary > drops in substitute availability. You mean as a precedent for similar commits in the future or as a way of involuntarily breaking other packages? As already stated, I only pushed the commit because I was quite sure that all rebuilds would succeed. Cheers
ping on a build fix for a build failure (main branch)
Hi, following previous comments (thanks) I have submitted a patch to correctly fix a build failure due to compiler warnings, instead of avoiding not building tests, on this Guix bug issue: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=63526 Please review the patch (which shall be a simple and low risk fix). Thanks!
Re: [PATCH] web: Include blocked-by in graphql .
Arun Isaac writes: > Thanks for the patch! I have pushed it after renaming blocked-by to > blocked_by. Underscores are a bit cringey, but unfortunately GraphQL > doesn't like hyphens. > > To see the effects of this change, we need to reconfigure berlin to run > the latest mumi. I will figure that out and do it soon. Awesome, thanks for fixing my rough untested changes! Chris signature.asc Description: PGP signature
Re: pam_ssh_agent_auth on a Guix System?
Hi Giovanni, On Tue, May 30, 2023 at 9:59 AM Giovanni Biscuolo wrote: > > AFAIU pam_ssh_agent_auth https://pamsshagentauth.sourceforge.net/ is not > already packaged in Guix, or am I missing something? I was not able to find it, either. > I'd like to execute sudo without having to set and enter a password [1] > and that PAM module is needed You could also add a line like this to your /etc/sudoers (but I don't recommend it) user_name ALL=(ALL) NOPASSWD:ALL > is someone already using such a configuration in a Guix System? Not quite. I added my public ssh key to root's authorized_keys. It's different from what you are looking for but gives you a root prompt with 'ssh root@localhost`. I did it because it's required for 'guix deploy'. Personally, I have not used the SSH agent, but it's an interesting avenue. I use Kerberos instead, which is probably the gold standard for distributed authentication. You are doing the right thing by thinking about your options. When playing with PAM, please remember that PAM can never elevate privileges of its own process. It is a shared library that runs as part of a privileged executable (often setuid root). PAM decides whether someone hoping to use the executable is authorized to do so. Kind regards Felix
pam_ssh_agent_auth on a Guix System?
Hello, AFAIU pam_ssh_agent_auth https://pamsshagentauth.sourceforge.net/ is not already packaged in Guix, or am I missing something? I'd like to execute sudo without having to set and enter a password [1] and that PAM module is needed ...then also a service to properly setup /etc/pam.d/sudo and /etc/sudoers is someone already using such a configuration in a Guix System? Thanks, Gio' [1] is it safer or more efficient to have users authentication without password but only with a SSH key? -- Giovanni Biscuolo Xelera IT Infrastructures signature.asc Description: PGP signature
Re: How many bytes do we add (closure of guix) when adding one new package?
Hi, On ven., 26 mai 2023 at 18:21, Ludovic Courtès wrote: > I agree that .go files are quite big (.scm files as well, but we’ve > improved information density somewhat by removing input labels :-)). > > The size of .go files went down when we switch to the baseline compiler > (aka. -O1): > > https://lists.gnu.org/archive/html/guix-devel/2020-06/msg00071.html > > That thread has ideas of things to do to further reduce .go size. Just to put a figure on what means “big”: currently the .go files are 5 times bigger than their associated .scm. Somehow, it’s the trap of DSL. :-) Packages are declarative and the information they declare is not dense. However, because they are bytecompiled to a general programming language, their specificity is not exploited. In an ideal world, the compiled binary representation of the packages should be smaller than their human-readable text-file counterpart. The mentioned improvement is nice. And it’s visible: --8<---cut here---start->8--- 145M /gnu/store/nqrb3g4l59wd74w8mr9v0b992bj2sd1w-guix-d62c9b267-modules/lib/guile/3.0/site-ccache/gnu 117M /gnu/store/s6rqlhqr750k44ynkqqj5mwjj2cs2yln-guix-a09968565-modules/lib/guile/3.0/site-ccache/gnu 127M /gnu/store/ndii4bpyzh2rc05ya61s89rig9hdrl4k-guix-a0178d34f-modules/lib/guile/3.0/site-ccache/gnu 164M /gnu/store/ni63a203jf61dwxlv8kr9b8x3vb1pdsp-guix-8e2f32cee-modules/lib/guile/3.0/site-ccache/gnu --8<---cut here---end--->8--- However, it has almost no impact on the whole size; scaled by the number of packages. > Download size has to be treated separately though. For example, ‘git > pull’ doesn’t redownload all of the repo or directory, and it uses > compression heavily. Thus, a few hundred bytes of additional .scm text > translate in less than that. > > As for the rest, download size can be reduced for example by choosing a > content-address transport, like something based on ERIS. > > I think we must look precisely at what we want to optimize—on-disk size, > or bandwidth requirement, in particular—and look at the whole solution > space. I think one direction is to tackle the way *package-modules* is built. Because of that, Guix is building too much and the design is not optimal – whatever technical solutions we implement for improving after that. On my poor laptop, Guix is becoming unusable because many operations are becoming so slow – when it’s still acceptable with APT of Debian. For instance, it’s something like 20 minutes for running “guix pull” without substitutes. And when I am traveling without a fast Internet connection, it’s often too much for the network at hand. Currently, “guix pull” is either building too much and downloading too much; by design. Cheers, simon
Re: Guidelines for pre-trained ML model weight binaries (Was re: Where should we put machine learning model parameters?)
Hi Ludo, On ven., 26 mai 2023 at 17:37, Ludovic Courtès wrote: >> Well, I do not know if we have reached a conclusion. From my point of >> view, both can be included *if* their licenses are compatible with Free >> Software – included the weights (pre-trained model) as licensed data. > > We discussed it in 2019: > > https://issues.guix.gnu.org/36071 Your concern in this thread was: My point is about whether these trained neural network data are something that we could distribute per the FSDG. https://issues.guix.gnu.org/36071#3-lineno21 and we discussed this specific concern for the package leela-zero. Quoting 3 messages: Perhaps we could do the same, but I’d like to hear what others think. Back to this patch: I think it’s fine to accept it as long as the software necessary for training is included. The whole link is worth a click since there seems to be a ‘server component’ involved as well. https://issues.guix.gnu.org/36071#3-lineno31 https://issues.guix.gnu.org/36071#5-lineno52 https://issues.guix.gnu.org/36071#6-lineno18 And somehow I am rising the same concern for packages using weights. We could discuss case-by-case, instead I find important to sketch guidelines about the weights because it would help to decide what to do with neuronal networks; as “Leela Chess Zero” [1] or others (see below). 1: https://issues.guix.gnu.org/63088 > This LWN article on the debate that then took place in Debian is > insightful: > > https://lwn.net/Articles/760142/ As pointed in #36071 mentioned above, this LWN article is a digest of some Debian discussion, and it is also worth to give a look to the raw material (arguments): https://lists.debian.org/debian-devel/2018/07/msg00153.html > To me, there is no doubt that neural networks are a threat to user > autonomy: hard to train by yourself without very expensive hardware, > next to impossible without proprietary software, plus you need that huge > amount of data available to begin with. About the “others” from above, please note that GNU Backgamon, already packaged in Guix with the name ’gnubg’, asks similar questions. :-) Quoting the webpage [2]: Tournament match and money session cube handling and cubeful play. All governed by underlying cubeless money game based neural networks. As Russ Allbery is pointing [3] – similarly as I tried to do in this thread – it seems hard to distinguish the data resulting from a pre-processing as some training to the data just resulting from good fitted parameters. 2: https://www.gnu.org/software/gnubg/ 3: https://lwn.net/Articles/760199/ > As a project, we don’t have guidelines about this though. I don’t know > if we can come up with general guidelines or if we should, at least as a > start, look at things on a case-by-case basis. Somehow, if we do not have guidelines for helping in deciding, it makes harder the review of #63088 [1] asking the inclusion of lc0 or it makes hard to know what to do about GNU Backgamon. On these specific cases, what do we do? :-) Cheers, simon
Re: [PATCH] web: Include blocked-by in graphql .
Hi Chris, Thanks for the patch! I have pushed it after renaming blocked-by to blocked_by. Underscores are a bit cringey, but unfortunately GraphQL doesn't like hyphens. To see the effects of this change, we need to reconfigure berlin to run the latest mumi. I will figure that out and do it soon. Regards, Arun
Re: [PATCH] web: Include blocked-by in graphql .
Christopher Baines writes: > * mumi/web/graphql.scm (): Include blocked-by. > --- > mumi/web/graphql.scm | 5 - > 1 file changed, 4 insertions(+), 1 deletion(-) > > diff --git a/mumi/web/graphql.scm b/mumi/web/graphql.scm > index aa4f38c..3cc19cd 100644 > --- a/mumi/web/graphql.scm > +++ b/mumi/web/graphql.scm > @@ -63,7 +63,10 @@ >(list >(messages (non-nullable-type (list-type )) > (lambda (parent . _) > - (issue-messages (bug-num parent) > + (issue-messages (bug-num parent > + (blocked-by (non-nullable-type (list-type )) > + (lambda (parent . _) > +(map bug-status (bug-blockedby parent) > > (define-object-type >(name (lambda (parent . _) I'm looking at this so that the qa-frontpage can see what issues are blocked. I'm not sure how to test this though. signature.asc Description: PGP signature
[PATCH] web: Include blocked-by in graphql .
* mumi/web/graphql.scm (): Include blocked-by. --- mumi/web/graphql.scm | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mumi/web/graphql.scm b/mumi/web/graphql.scm index aa4f38c..3cc19cd 100644 --- a/mumi/web/graphql.scm +++ b/mumi/web/graphql.scm @@ -63,7 +63,10 @@ (list (messages (non-nullable-type (list-type )) (lambda (parent . _) - (issue-messages (bug-num parent) + (issue-messages (bug-num parent + (blocked-by (non-nullable-type (list-type )) + (lambda (parent . _) +(map bug-status (bug-blockedby parent) (define-object-type (name (lambda (parent . _) -- 2.40.1
