Re: [bug#41694] [PATCH] doc: cookbook: Add entry about getting substitutes through Tor.
Hello Brice,
I think it would be useful to warn users that when pulling there is
a direct connection to guix git repos, so to route it through Tor,
one needs to use torsocks. It wont make the configuration foolproof,
but it will reduce the leaks to clearnet.
From 6a73b1b1129d3d636d7a0559dffa19e5d40aaf0d Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Andr=C3=A9=20Batista?=
Date: Tue, 16 Jun 2020 23:13:03 -0300
Subject: [PATCH] doc: cookbook: Add info on the need of using torsocks when
pulling.
To: guix-devel@gnu.org
* doc/guix-cookbook.texi (Getting substitutes from Tor): Add note at
the end on using torsocks when pulling.
---
doc/guix-cookbook.texi | 8 +++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/doc/guix-cookbook.texi b/doc/guix-cookbook.texi
index 1342826c97..1852ce6c3a 100644
--- a/doc/guix-cookbook.texi
+++ b/doc/guix-cookbook.texi
@@ -15,6 +15,7 @@ Copyright @copyright{} 2020 Oleg Pykhalov@*
Copyright @copyright{} 2020 Matthew Brooks@*
Copyright @copyright{} 2020 Marcin Karpezo@*
Copyright @copyright{} 2020 Brice Waegeneire@*
+Copyright @copyright{} 2020 André Batista@*
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3 or
@@ -1802,7 +1803,7 @@ at your own risk.
@end quotation
Guix's substitute server is available as a Onion service, if you want
-to use it to get your substitutes from Tor configure your system as
+to use it to get your substitutes through Tor configure your system as
follow:
@lisp
@@ -1843,6 +1844,11 @@ sudo herd set-http-proxy guix-daemon
http://localhost:9250
guix build --substitute-urls=https://bp7o7ckwlewr4slm.onion …
@end example
+Note that the procedure described above applies only to package substitution.
+When you update your guix distribution with @command{guix pull}, you should
+use @command{torsocks} if you want to route the connection to guix git
+repository servers through Tor.
+
@c *
@node Advanced package management
@chapter Advanced package management
--
2.26.2
signature.asc
Description: PGP signature
Re: Using --manfistest with /manifest files
Thank you for the direct reply, zimoun. > [The problem] is a pragmatic one. As any good Zen says: "Now is better than > never. Although never is often better than *right* now." Okay. If that is the case, then I very much empathize with the problem. > Going from imperative/sequential "install, pull, remove" way to the > declarative manifest.scm way, in the general case, needs to change the > format of /manifest (or add another file). Which means > transition plan, etc.. Otherwise, on the technical level, all the > material is there. That said. This makes me wonder that we may be thinking of different things altogether. The discussion seems to have congealed around smoothing the transition to declarative profile management for users. However, the proposal I have in mind is *first class profiles*. I am thinking of a file that can be fed to the --manifest (or some potential future equivalent) option of various guix commands. This hypothetical file would let users operate directly on profiles as needed. My current specific use case for this is packing the *development environments* produced by `guix environment'. > I do not see why it is straightforward for some cases. guix environment --ad-hoc can be indirectly packed via guix pack unless I am missing something. Otherwise, users are at an impasse. > Because your use case -- pack an existing profile for sharing -- is not > really related to transform /manifest to a valid manifest.scm, > if I understand correctly. And I agree with you that it should be > possible to pack an existing profile (created by any mean). > > Does "pack --profile-name" fit your needs? I am not quite sure this works. From the manual: > ‘--localstatedir’ > ‘--profile-name=NAME’ > Include the “local state directory”, ‘/var/guix’, in the resulting > pack, and notably the ‘/var/guix/profiles/per-user/root/NAME’ > profile—by default NAME is ‘guix-profile’, which corresponds to > ‘~root/.guix-profile’. > > ‘/var/guix’ contains the store database (*note The Store::) as well > as garbage-collector roots (*note Invoking guix gc::). Providing > it in the pack means that the store is “complete” and manageable by > Guix; not providing it pack means that the store is “dead”: items > cannot be added to it or removed from it after extraction of the > pack. I do not see how to use this with the transitory /gnu/store/-profile produced by a `guix environment' invocation. Also, my intention is simply to provide the profile's environment, not include the local state directory. Put more simply, I want to be able to produce a tarball/container capable of reproducing `guix environment --container '. I think this would be very useful. Am I just failing to grok something fundamental? Thoughts? More generally, I think first class profiles could be both a powerful feature and an important future-proofing against extra maintenance burden. Profiles are a central concept to guix usage. They form the atomic unit with which users interact. Wanting to tarball a profile is just one use case, but future guix commands (guix merge, anyone?) or future --manifest options (guix archive, anyone?) seem likely to directly benefit from an existing infrastructure that supports store profiles being created, recreated and munged. zimoun wrote: > Dear, > > On Tue, 16 Jun 2020 at 20:33, elaexuo...@wilsonb.com wrote: > >> 1. We can only approximate that actual profile content; storing > >>an approximate ‘manifest.scm’ along with the profile would IMO be > >>deceptive. > > > > Is this a technical barrier or a pragmatic one? > > [...] > > > If the problem is of pragmatics, then at the very least I would be > > interested > > in hearing a delineation of the challenges. I think this could be helpful > > for > > the discussion though. > > It is a pragmatic one. As any good Zen says: "Now is better than never. > Although never is often better than *right* now." > > Going from imperative/sequential "install, pull, remove" way to the > declarative manifest.scm way, in the general case, needs to change the > format of /manifest (or add another file). Which means > transition plan, etc.. Otherwise, on the technical level, all the > material is there. > > So it is some work and it is not clear that it will pay off. > > > >> Yeah, I think our goal is just to provide a tool to migrate from the > >> “imperative” way to the declarative way. Once people have gotten > >> started with manifests, they no longer need that migration tool. > > > > Would you mind commenting on the use case that I started this thread with? > > Specifically, I was trying to `guix pack' a `guix environment'. The > > equivalent > > is straightforward for purely --ad-hoc environments but not otherwise. > > I do not see why it is straightforward for some cases. > > > > Personally, I have already encountered several instances where this would > > have > > been useful. I
Re: [PATCH] Add Tor client only package definition
From ac47ba538dd5cf628b26cce05e3b15b24ca03077 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Andr=C3=A9=20Batista?=
Date: Tue, 16 Jun 2020 19:20:57 -0300
Subject: [PATCH] gnu: Add tor-client.
To: guix-devel@gnu.org
* gnu/packages/tor.scm (tor-client): New variable.
---
gnu/packages/tor.scm | 31 +--
1 file changed, 29 insertions(+), 2 deletions(-)
diff --git a/gnu/packages/tor.scm b/gnu/packages/tor.scm
index 2f2623b0e6..06debfed07 100644
--- a/gnu/packages/tor.scm
+++ b/gnu/packages/tor.scm
@@ -8,6 +8,7 @@
;;; Copyright © 2017 Rutger Helling
;;; Copyright © 2018 Ricardo Wurmus
;;; Copyright © 2020 Vincent Legoll
+;;; Copyright © 2020 André Batista
;;;
;;; This file is part of GNU Guix.
;;;
@@ -27,6 +28,7 @@
(define-module (gnu packages tor)
#:use-module ((guix licenses) #:prefix license:)
#:use-module (guix packages)
+ #:use-module (guix utils)
#:use-module (guix download)
#:use-module (guix git-download)
#:use-module (guix build-system gnu)
@@ -85,11 +87,36 @@ location. Tor works with many of your existing
applications, including
web browsers, instant messaging clients, remote login, and other
applications based on the TCP protocol.
+This package is the full featured @code{tor} which is needed for running
+relays, bridges or directory authorities. If you just want to access the Tor
+network or to setup an onion service you may install @code{tor-client}
+instead.")
+(license license:bsd-3)))
+
+(define-public tor-client
+ (package
+ (inherit tor)
+ (name "tor-client")
+ (arguments
+(substitute-keyword-arguments (package-arguments tor)
+ ((#:configure-flags flags)
+ (append flags
+ '("--disable-module-relay")
+ (synopsis "Client to the anonymous Tor network")
+ (description
+"Tor protects you by bouncing your communications around a distributed
+network of relays run by volunteers all around the world: it prevents
+somebody watching your Internet connection from learning what sites you
+visit, and it prevents the sites you visit from learning your physical
+location. Tor works with many of your existing applications, including
+web browsers, instant messaging clients, remote login, and other
+applications based on the TCP protocol.
+
To @code{torify} applications (to take measures to ensure that an application,
which has not been designed for use with Tor such as ssh, will use only Tor for
internet connectivity, and also ensures that there are no leaks from DNS, UDP
or
-the application layer) you need to install @code{torsocks}.")
-(license license:bsd-3)))
+the application layer) you need to install @code{torsocks}. This package only
+provides a client to the Tor Network.")))
(define-public torsocks
(package
--
2.26.2
signature.asc
Description: PGP signature
Re: K of N trust in substitutes (related to reproducible builds)
Ludovic Courtès writes: >> 3: http://theworld.com/~cme/spki.txt >> >> Using the above ACL, you'd trust a substitute for a path with a specific >> hash if you can find 2 narinfos for that path and hash if they're signed >> with keys in that entry. Multiple entries would still be supported, and >> you wouldn't need to specify the k-of-n bit if you don't want to. >> >> I'm not quite sure how expressive this is, or if there are some policies >> that would be good to support that either can't be expressed, or can't >> be expressed easily. There's probably other approaches, and how to >> support trusting substitutes is an important part to consider. > > I would be tempted to not bake it into /etc/guix/acl. You would still > authorize all the servers, but instead of choosing a policy that accepts > anything signed by one of them, as is currently the case, you would > choose a policy that only accepts something signed by two of them. > > The policy would be implemented in (guix scripts substitute). I haven’t > put much thought into it but it could be something akin to > ‘lookup-narinfos/diverse’, roughly. > > Thoughts? I think that could work, do you have any suggestions on how that "two" would be configured? I guess it could be a boolean on/off, but it would be probably more extensible to just allow providing a minimum number of substitiute servers to agree. Thanks, Chris signature.asc Description: PGP signature
Re: (propagated) 'inputs' depends on 'outputs'?
Hi Ricardo, On Tue, 16 Jun 2020 at 17:34, Ricardo Wurmus wrote: > That’s correct. Outputs are buckets into which we drop build > artifacts. We will build everything with all inputs in one derivation > and then move stuff into output buckets. This means that you can > download independent outputs individually, but they are still all the > result of that *single* derivation. So to build any of the outputs you > will need to build that derivation, even if it also results in other > outputs that you don’t care about. Thank you for the explanations. I understand even if time to time I feel frustrated to download and/or build a lot of unrelated packages that I am really interested in. But hey, it is how the thing works. :-) Now all is clear. Cheers, simon
Re: (propagated) 'inputs' depends on 'outputs'?
On Tue, 16 Jun 2020 at 10:28, Julien Lepiller wrote:
>>If I run "guix install foo:out --no-substitutes" then I potentially
>>build any other "outputs"" of foo, e.g., "doc" i.e., potentially
>>download a lot of TeX stuff, or in the case of Git, all the Subversion
>>stuff. Right?
>
> Yes, because building foo:out doesn't make sense. You build foo and
> guix is nice enough to understand that's wgat you mean :)
I understand. I do not know if it does not make sense but I
understand. :-)
Well, in this lockdown period, my bandwith and computing resources were
limited and building e.g. all Subversion (or any heavy doc packages)
when I wanted only small output e.g. Git:out (without the svn support)
appeared to me frustrating.
Now it is clear for me, it is by design. Thank you for the
explanations.
>>> It would make sense to only propagate for some outputs: suppose at
>>> runtime only foo:bin requires the propagation of bar. Since foo and
>>> bar are already built, it should be possible to restrict the
>>> propagation behaviour to that output. Foo:out would not bring in bar
>>> anymore, reducing the closure size.
>>
>>Yes, it seems making sense to only propagated if the output needs it.
>>Well, if it is not implemented yet maybe it is because it is not really
>>necessary. :-)
>
> That cannot be automated because usually we use propagation when there
> is no direct reference. It wouldn't be useful otherwise.
I am not sure to get the point. From my understanding, it could be
possible to add information to native-inputs, inputs and
propagated-inputs, e.g.,
;; For 'git-svn'.
("subversion" ,subversion "svn")
("perl-term-readkey" ,perl-term-readkey "svn")
or whatever other mean. Then it becomes possible to only build e.g.,
git:svn and/or propagate specific inputs depending on the outputs.
In the light of your explanation, it does not make sense for
native-inputs and inputs because Guix is building only one thing
(package object, derivation). But it could make sense for the
propagated-inputs. Well, the same way that "outputs" are not automatic
but specified somehow by "arguments".
However, it is not implemented and now after the explanations, I do not
know it is is worth. Well, it is another way to see package
parameters. :-)
Thank you again for the explanations.
Cheers,
simon
Re: (propagated) 'inputs' depends on 'outputs'?
zimoun writes: > On Sat, 13 Jun 2020 at 12:16, Christopher Baines wrote: > >> Looking at different outputs, the references are different. If you're >> just using the "out" output, then you don't need subversion in your >> store, but if you're using the "svn" output, then you do, as that output >> references an output for subversion. >> >> The references for an output isn't something specified, but something >> decided by what references that output actually contains. > > I understand. > > Back to the Git's example, there is still something I miss: I cannot > build Git from source (e.g. no substitutes) using "guix build git" > without downloading -- and possibly building too -- all the Subversion > stuff. That’s correct. Outputs are buckets into which we drop build artifacts. We will build everything with all inputs in one derivation and then move stuff into output buckets. This means that you can download independent outputs individually, but they are still all the result of that *single* derivation. So to build any of the outputs you will need to build that derivation, even if it also results in other outputs that you don’t care about. -- Ricardo
Re: Makefile logic to create Guix documentation
Hi,
Speaking about Makefile and Guix documentation, I have remarked things
(maybe I am doing wrong) that annoys me time to time.
1. "make info" is not self consistent.
--8<---cut here---start->8---
./doc/guix.texi:11297: @include: could not find os-config-bare-bones.texi
./doc/guix.texi:11438: @include: could not find os-config-desktop.texi
./doc/guix.texi:11445: @include: could not find
os-config-lightweight-desktop.texi
make[1]: *** [Makefile:4078: doc/guix.info] Error 1
--8<---cut here---end--->8---
2. I am always confused how to generate the EPS/PNG of "doc/images" and
generally I end up with "make ... wait C-c".
3. The modified PO files under "po/{guix,packages}/" hangs Magit and
then Emacs so generally I have to restore them before going back to
Magit.
I am not enough annoyed to unwrap the logic of Makefile and propose a
patch but if someone is visiting the topic... Or maybe a tips for
improving my workflow. :-)
All the best,
simon
Re: Using --manfistest with /manifest files
Dear, On Tue, 16 Jun 2020 at 10:03, George Clemmer wrote: > "manifest" occurs ~600 times in the ./guix directory. I am guessing its > use is deeply embedded with developers. If so, renaming it internally > seems like a bad idea. And if we write our internal manifest into the > profile and call it "specifications" it will only add the confusion. > > OTOH, "manifest" occurs only ~50 times in guix.info and the "user API" > seems limited to the --manifest option and the functions: > specifications->manifest and packages->manifest. Furthermore it is a new > concept for new users. So I don't think users care what we call it. Thank your for checking. You convince me. :-) All the best, simon
Re: (propagated) 'inputs' depends on 'outputs'?
Le 16 juin 2020 08:41:58 GMT-04:00, zimoun a écrit : >Bonjour Julien, > >On Sat, 13 Jun 2020 at 07:38, Julien Lepiller >wrote: > >> Exactly, no. You cannot separate inputs from outputs, because they >are >> part of the same derivation. When you build an output, you actually >> build the complete derivation and there's no way to separate that in >> "this part builds out" and "this part builds doc", etc. > >So it means that I need to build all the outputs even if I am >interested >in only one, right? Absolutely, but you shouldn't thenk of it as building multiple outputs, because you are really just building one thing: there is only one package object, one derivation. It happens to create multiple directories in the store, but there's only one thing you build. > >If I run "guix install foo:out --no-substitutes" then I potentially >build any other "outputs"" of foo, e.g., "doc" i.e., potentially >download a lot of TeX stuff, or in the case of Git, all the Subversion >stuff. Right? Yes, because building foo:out doesn't make sense. You build foo and guix is nice enough to understand that's wgat you mean :) > >Even if at the end, only the references used by "foo:out" will be >tracked and all the others potentially garbage collected. Right? > > >> It would make sense to only propagate for some outputs: suppose at >> runtime only foo:bin requires the propagation of bar. Since foo and >> bar are already built, it should be possible to restrict the >> propagation behaviour to that output. Foo:out would not bring in bar >> anymore, reducing the closure size. > >Yes, it seems making sense to only propagated if the output needs it. >Well, if it is not implemented yet maybe it is because it is not really >necessary. :-) That cannot be automated because usually we use propagation when there is no direct reference. It wouldn't be useful otherwise. > > >Thank you for explaining. It is clearer for me now. > >Cheers, >simon
Re: Using --manfistest with /manifest files
Hi zimoun, zimoun writes: > In contradiction with what I wrote above, I agree. :-) > > /manifest should be renamed /specifications or > something like that. > > And a comment could be inserted in this file saying: internal usage, do > not modify, etc.. > > WDYT? Sure, that would work. But, on further thought, I would like to amend my suggestion -- to change which ever is easier. I say that because ... "manifest" occurs ~600 times in the ./guix directory. I am guessing its use is deeply embedded with developers. If so, renaming it internally seems like a bad idea. And if we write our internal manifest into the profile and call it "specifications" it will only add the confusion. OTOH, "manifest" occurs only ~50 times in guix.info and the "user API" seems limited to the --manifest option and the functions: specifications->manifest and packages->manifest. Furthermore it is a new concept for new users. So I don't think users care what we call it. Bottom line: change whichever is more convenient for developers. HTH - George
Re: Using --manfistest with /manifest files
Dear, On Tue, 16 Jun 2020 at 20:33, elaexuo...@wilsonb.com wrote: >> 1. We can only approximate that actual profile content; storing >>an approximate ‘manifest.scm’ along with the profile would IMO be >>deceptive. > > Is this a technical barrier or a pragmatic one? [...] > If the problem is of pragmatics, then at the very least I would be interested > in hearing a delineation of the challenges. I think this could be helpful for > the discussion though. It is a pragmatic one. As any good Zen says: "Now is better than never. Although never is often better than *right* now." Going from imperative/sequential "install, pull, remove" way to the declarative manifest.scm way, in the general case, needs to change the format of /manifest (or add another file). Which means transition plan, etc.. Otherwise, on the technical level, all the material is there. So it is some work and it is not clear that it will pay off. >> Yeah, I think our goal is just to provide a tool to migrate from the >> “imperative” way to the declarative way. Once people have gotten >> started with manifests, they no longer need that migration tool. > > Would you mind commenting on the use case that I started this thread with? > Specifically, I was trying to `guix pack' a `guix environment'. The equivalent > is straightforward for purely --ad-hoc environments but not otherwise. I do not see why it is straightforward for some cases. > Personally, I have already encountered several instances where this would have > been useful. I also think it would be just plain cool to have the ability to > pack up, containerize, and share arbitrary profiles with non-guix users. Well, I have re-read your initial message and maybe miscommunication here. :-) Because your use case -- pack an existing profile for sharing -- is not really related to transform /manifest to a valid manifest.scm, if I understand correctly. And I agree with you that it should be possible to pack an existing profile (created by any mean). Does "pack --profile-name" fit your needs? If not, yes packing an existing profile could be a feature to "guix pack" -- doing transparently something similar to the Leo's suggestion -- because it is an internal consumption of these /manifest files. All the best, simon
Re: (propagated) 'inputs' depends on 'outputs'?
Bonjour Julien, On Sat, 13 Jun 2020 at 07:38, Julien Lepiller wrote: > Exactly, no. You cannot separate inputs from outputs, because they are > part of the same derivation. When you build an output, you actually > build the complete derivation and there's no way to separate that in > "this part builds out" and "this part builds doc", etc. So it means that I need to build all the outputs even if I am interested in only one, right? If I run "guix install foo:out --no-substitutes" then I potentially build any other "outputs"" of foo, e.g., "doc" i.e., potentially download a lot of TeX stuff, or in the case of Git, all the Subversion stuff. Right? Even if at the end, only the references used by "foo:out" will be tracked and all the others potentially garbage collected. Right? > It would make sense to only propagate for some outputs: suppose at > runtime only foo:bin requires the propagation of bar. Since foo and > bar are already built, it should be possible to restrict the > propagation behaviour to that output. Foo:out would not bring in bar > anymore, reducing the closure size. Yes, it seems making sense to only propagated if the output needs it. Well, if it is not implemented yet maybe it is because it is not really necessary. :-) Thank you for explaining. It is clearer for me now. Cheers, simon
Re: (propagated) 'inputs' depends on 'outputs'?
Hi Chris, Thank you for explaining. It still miss a point. On Sat, 13 Jun 2020 at 12:16, Christopher Baines wrote: > Looking at different outputs, the references are different. If you're > just using the "out" output, then you don't need subversion in your > store, but if you're using the "svn" output, then you do, as that output > references an output for subversion. > > The references for an output isn't something specified, but something > decided by what references that output actually contains. I understand. Back to the Git's example, there is still something I miss: I cannot build Git from source (e.g. no substitutes) using "guix build git" without downloading -- and possibly building too -- all the Subversion stuff. Other said, if I run "guix install git --no-substitutes" then I will build all the Git outputs, so build all Subversion and other, and finally only install the references of "out" (the rest could be garbage collected). Cheers, simon
“Reproducible research articles, from source code to PDF”
Hello! This new post introduces the work I did to have a fully reproducible replication (!) of a 13-year old article, using Guix to express the whole pipeline: https://hpc.guix.info/blog/2020/06/reproducible-research-articles-from-source-code-to-pdf/ Feedback welcome! Ludo’.
Re: Using --manfistest with /manifest files
> 1. We can only approximate that actual profile content; storing >an approximate ‘manifest.scm’ along with the profile would IMO be >deceptive. Is this a technical barrier or a pragmatic one? If it is the former, then I don't quite grok why. I explain my reasoning in great detail in a previous reply, but the gist is this: generating profiles is deterministic, no? That is, given the inputs of channel revision, command line invocation, and any referred-to external files, then `guix time-machine' is enough to recreate a profile, no? If the problem is of pragmatics, then at the very least I would be interested in hearing a delineation of the challenges. I think this could be helpful for the discussion though. > Yeah, I think our goal is just to provide a tool to migrate from the > “imperative” way to the declarative way. Once people have gotten > started with manifests, they no longer need that migration tool. Would you mind commenting on the use case that I started this thread with? Specifically, I was trying to `guix pack' a `guix environment'. The equivalent is straightforward for purely --ad-hoc environments but not otherwise. Personally, I have already encountered several instances where this would have been useful. I also think it would be just plain cool to have the ability to pack up, containerize, and share arbitrary profiles with non-guix users. Thoughts? signature.asc Description: PGP signature
Re: Reducing LLVM closure size
Efraim Flashner writes: > I see the include directory is about 20MB, can that be left in "out" or > is that needed in the "lib" output? It can be left out, I mentioned it in my first email. >> Where would we store the different backends? In different outputs? >> On which backend does Mesa depend for instance? > > I would assume llvmpipe OK, thanks! -- Pierre Neidhardt https://ambrevar.xyz/ signature.asc Description: PGP signature
Re: Latest download from website
Hi! Mathieu Othacehe skribis: > Here's a wip patch for the website. It adds a new "download/latest" > page allowing to download the latest Guix System images from Cuirass. > > I've merged all the required mechanisms in Guix and Cuirass, now most of > the remaining work is "cosmetic" (and that's not my cup of tea). > > Anyway, the patch and a screenshot are attached, please tell me what you > think. That looks nice to me! I think it should prominently say that these are “development snapshots” (probably these two words must appear) though, and perhaps contain a link to /download for those looking for releases. Other than that, I’d say go for it! Thanks, Ludo’.
Re: A script to check an edit does not break anything
Hi Edouard, Edouard Klein skribis: > Ludovic Courtès writes: [...] >> However, this is definitely something ‘guix lint’ could check with >> something along the lines of the patch below. > > Thank you for pushing profile-collisions, it certainly is helpful for > finding such problems, and it perfectly integrates within guix. > >> >>> - Secondly, it does not limit itself to the dependents (as listed by >>> guix refresh --list-dependents) of the packages one is meddling with, >>> but to the whole reverse bags (as listed by guix graph >>> --type=reverse-bag). >> >> I think it’s equivalent: ‘guix refresh -l’ simply shows the contour of >> the graph whereas ‘guix graph’ lists every node. > > The problem lies when the leafs are OK but the nodes in the middle are > not. See for example in the attached image, the failure of jupyter is > masked by r-irkernel being both buildable and installable. When you write “the failure of jupyter”, you mean failure to _install_ jupyter, right? ‘guix refresh -l’ returns the set of leaves whose closure encompasses all the dependents of the given package. Thus, if you build all the nodes returned by ‘guix refresh -l jupyter’, you’ll definitely notice a build failure in jupyter. > Now, the new tool you added to guix lint solves the discoverability > problem. It is now indeed reported that jupyter has a problem. > > Still, it takes around 10 minutes to run on my (admittedly underpowered) > machine, and one has to rummage to the output (or diff with a previous > run) to see if a specific action caused or solved problems. What takes 10 minutes? ‘guix lint -c profile-collisions’ without arguments? It runs in 2 minutes on my laptop, which is admittedly too much, but note that most of the time you’ll just run: guix lint -c profile-collisions jupyter That should take a few seconds at most. > gpwc.sh has a real time visual output that is specific to current > modifications (it could even be paired with Ricardo's automatic commit > message writer to automatically guess which root packages to start with) > that allows the developer to start investing a problem quicker, without > having to wait for the end of the run. Also, the visual output makes > seeing who depends on whom easier, the same information in text form > makes my head hurt. > > Provided I rewrite it in scheme, do you think gpwc could make it > into guix/etc ? I haven’t looked in detail at gwpc.sh but I guess we’d all like improved tooling. To have it in Guix, it’s better if it’s well integrated with existing tools and written in Scheme. Thanks! Ludo’.
Re: Using --manfistest with /manifest files
Hi, George Clemmer skribis: > Yes, only 'manifest.scm' is in the doc, but '.guix-profile/manifest' > smacks a user in the face pretty quickly which leads to these messy > questions. That’s something I had not anticipated. It’s an interesting lesson! Ludo’.
Re: Using --manfistest with /manifest files
Hi, zimoun skribis: > On Sun, 14 Jun 2020 at 17:24, Ludovic Courtès wrote: > >> I think there were several issues we discussed: >> >> 1. We can only approximate that actual profile content; storing >> an approximate ‘manifest.scm’ along with the profile would IMO be >> deceptive. >> >> 2. It’s easy to maintain compatibility over a data format, but it’s >> much harder to maintain compatibility for code. >> >> I think we discussed these issues the best we could in the megathread, >> so I’m personally in favor of moving forward in a pragmatic way. > > By pragmatic way, you mean: > > - let the format of /manifest as it is, > - write '--export-manifest' as an approximation > > right? For example, yes. > Well, I personally changed my workflow and now I always use manifest > files. And the situation that I described in the manifest about the > "Working Scientific" doing install, pull, install, pull, remove, > install, etc. is rooted in bad practises, so it should be avoided. > > Therefore, I agree that '--export-manifest' is the right approach, as an > helping tool; too bad for some corner cases. :-) Yeah, I think our goal is just to provide a tool to migrate from the “imperative” way to the declarative way. Once people have gotten started with manifests, they no longer need that migration tool. Thanks, Ludo’.
Re: Reducing LLVM closure size
On Tue, Jun 16, 2020 at 11:27:45AM +0200, Pierre Neidhardt wrote:
> Ludovic Courtès writes:
>
> >> - either move the libs to a "lib" output,
> >> - or move the "bin" and "include" folder to a new output.
> >>
> >> The second approach has the benefit of being less disruptive for
> >> dependents.
> >
> > I have a slight preference for a “lib” output since that’s more in line
> > with what we do for other packages.
>
> OK.
>
I see the include directory is about 20MB, can that be left in "out" or
is that needed in the "lib" output?
> > Nice! I looked for something like this when I packaged
> > ‘clang-tools-extra’ and didn’t find it. This should go into the next
> > ‘staging’ branch (or ‘core-updates’?).
>
> I can send a patch for llvm-10, but I guess many llvm-dependents will
> have to be updated accordingly.
>
> I suppose that the input
>
> --8<---cut here---start->8---
> ("llvm" ,llvm)
> --8<---cut here---end--->8---
>
> will need to be turned to
>
> --8<---cut here---start->8---
> ("llvm" ,llvm "lib")
> --8<---cut here---end--->8---
>
> for most packages. I have no experience with LLVM, so can someone
> confirm that this is the right way to go?
>
> >> All in all, it looks like we can save 52 MiB out of 140 MiB from the LLVM
> >> package (and 210 MiB from its closure).
> >
> > That’d be great.
>
> To clarify the ambiguous sentence I wrote above, we would save 52 MiB
> from the closure size of LLVM.
>
> > An additional option would be to have a package with fewer backends by
> > default (currently all of them are enabled and that takes up quite some
> > space). In particular, Mesa doesn’t depend to depend on an LLVM variant
> > with 15 backends when it’s only going to use one.
>
> Are you suggesting an alternative or a tweak to add on top of my
> suggestion?
>
> Where would we store the different backends? In different outputs?
> On which backend does Mesa depend for instance?
I would assume llvmpipe
>
> Cheers!
>
> --
> Pierre Neidhardt
> https://ambrevar.xyz/
--
Efraim Flashner אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
signature.asc
Description: PGP signature
Re: Using --manfistest with /manifest files
Hi Pierre, Pierre Neidhardt skribis: > Ludovic Courtès writes: > >> I think there were several issues we discussed: > > Allow me to reiterate, at the risk of bikeshedding... :) > >> 1. We can only approximate that actual profile content; storing >> an approximate ‘manifest.scm’ along with the profile would IMO be >> deceptive. > > Why is an approximate file more of a problem than a command producing > an approximate result? We could include a warning comment at the top of > the file to make it explicit that it's approximate. > >> 2. It’s easy to maintain compatibility over a data format, but it’s >> much harder to maintain compatibility for code. > > I'm confused: my suggestion is to add a new data format, so isn't it > better then? We seem to be talking past each other. IMO, exporting a file is easily done and better than nothing. I think we should be pragmatic. If you want, maybe you can prototype something that we could discuss? Thanks, Ludo’.
Re: Makefile logic to create Guix documentation
Hi, Chris Marusich skribis: > First, consider this snippet from doc/local.mk: > > # We cannot add new dependencies to `%D%/guix.pdf' & co. (info "(automake) > # Extending"). Using the `-local' rules is imperfect, because they may be > # triggered after the main rule. Oh, well. > pdf-local: $(DOT_FILES=%.dot=$(top_srcdir)/%.pdf) > info-local: $(DOT_FILES=%.dot=$(top_srcdir)/%.png) > ps-local: $(DOT_FILES=%.dot=$(top_srcdir)/%.eps) \ > $(top_srcdir)/%D%/images/coreutils-size-map.eps > dvi-local: ps-local > > > What is this syntax called? I checked the Make, Automake, and Autoconf > manuals, but I couldn't find anything. I'm talking about this syntax: > > info-local: $(DOT_FILES=%.dot=$(top_srcdir)/%.png) These are GNU Make substitutions (info "(make) Substitution Refs"). > It looks like when you added this, you intended to add a *.png > prerequisite to the info-local target for every equivalent *.dot file > that exists. My guess is you want to ensure that the PNG files are > generated before the info page gets created, since the PNG files are > required in order to build the info page. However, is it possible you > meant to write it like the following instead? (The first "=" has been > replaced with a ":".) > > info-local: $(DOT_FILES:%.dot=$(top_srcdir)/%.png) > > When using ":", I recognize this syntax as a "substitution reference" > (see: (make) Substitution Refs). However, I do not know what it is > supposed to be when the ":" is replaced with a "=". Is it a typo? Ah yes, definitely! I didn’t even notice when reading it above. You’re welcome to make that change if nothing bad happens! > Second, I noticed some rules like the following: > > .dot.eps: > $(AM_V_DOT)$(DOT) -Teps $(DOT_OPTIONS) < "$<" > "$(srcdir)/$@.tmp"; \ > mv "$(srcdir)/$@.tmp" "$(srcdir)/$@" Julien already replied; this is POSIX make notation. Thanks, Ludo’.
Re: generate commit messages for package updates
Ludovic Courtès writes: > Pierre Neidhardt skribis: > >> Is it worth including in guix/etc? > > I think it’d be nice! I fixed a bug and added etc/committer.scm.in. -- Ricardo
Re: Using --manfistest with /manifest files
Dear, On Mon, 15 Jun 2020 at 22:51, George Clemmer wrote: > ISTM we set ourselves up for confused users and a lot of explaining by > labeling two very different things with same name :-0 I think there is a confusion here. The file /manifest is an internal detail implementation and the user should *not* be exposed to. Well, there are a lot of internal files that the user is not aware. And sometimes, the names are more or less well-chosen. But hey! it is internal and naming is hard. :-) > Yes, only 'manifest.scm' is in the doc, but '.guix-profile/manifest' > smacks a user in the face pretty quickly which leads to these messy > questions. I am not convinced by "smacks in the face pretty quickly". But I agree that questions about this topic regularly come. For example, to pick an old one: https://lists.gnu.org/archive/html/guix-devel/2018-10/msg00011.htlm > IMO we could dramatically simplify the situation, and simplify our > lives, by simply renaming the .guix-profile/manifest file ;-) In contradiction with what I wrote above, I agree. :-) /manifest should be renamed /specifications or something like that. And a comment could be inserted in this file saying: internal usage, do not modify, etc.. WDYT? All the best, simon
Re: K of N trust in substitutes (related to reproducible builds)
Hi! Christopher Baines skribis: > My feeling is that making some initial step forward in this area is > going to be tricky, care needs to be taken around the security and > backwards compatibility aspects. I've now got around to actually > thinking about potential ways to make parts of this happen though, and > even changed some code [2] (although I haven't actually tried to run it > yet). Nice! > As I understand, the format for the ACL is based around [3] and I was > excited to see as part of that specification is something I think might > overlap with what I describe wanting above. Specifically, the k-of-n > bit. I think this could work something like this in an > ACL: > > (acl > (entry > (k-of-n >2 >3 >(public-key > (ecc > (curve Ed25519) > (q #5F5F4F321533D3A38F909785E682798933BA9BE257C97E5ABC07DD08F27B8DBF#) > ) > ) >(public-key > (ecc > (curve Ed25519) > (q #3AF2631C5E78F520CB1DC0D438D8D6F88EEF4B8E11097A62EE2DF390E946AED0#) > ) > ) >(public-key > (ecc > (curve Ed25519) > (q #1EEE5340C3AAD6E062A1395A88A86FC75982E8BC7DCBAE171858EEAAB14AAB77#) > ) > ) > ) > (tag >(guix import) >) > ) > ) > > 3: http://theworld.com/~cme/spki.txt > > Using the above ACL, you'd trust a substitute for a path with a specific > hash if you can find 2 narinfos for that path and hash if they're signed > with keys in that entry. Multiple entries would still be supported, and > you wouldn't need to specify the k-of-n bit if you don't want to. > > I'm not quite sure how expressive this is, or if there are some policies > that would be good to support that either can't be expressed, or can't > be expressed easily. There's probably other approaches, and how to > support trusting substitutes is an important part to consider. I would be tempted to not bake it into /etc/guix/acl. You would still authorize all the servers, but instead of choosing a policy that accepts anything signed by one of them, as is currently the case, you would choose a policy that only accepts something signed by two of them. The policy would be implemented in (guix scripts substitute). I haven’t put much thought into it but it could be something akin to ‘lookup-narinfos/diverse’, roughly. Thoughts? Ludo’.
Re: Reducing LLVM closure size
Ludovic Courtès writes:
>> - either move the libs to a "lib" output,
>> - or move the "bin" and "include" folder to a new output.
>>
>> The second approach has the benefit of being less disruptive for dependents.
>
> I have a slight preference for a “lib” output since that’s more in line
> with what we do for other packages.
OK.
> Nice! I looked for something like this when I packaged
> ‘clang-tools-extra’ and didn’t find it. This should go into the next
> ‘staging’ branch (or ‘core-updates’?).
I can send a patch for llvm-10, but I guess many llvm-dependents will
have to be updated accordingly.
I suppose that the input
--8<---cut here---start->8---
("llvm" ,llvm)
--8<---cut here---end--->8---
will need to be turned to
--8<---cut here---start->8---
("llvm" ,llvm "lib")
--8<---cut here---end--->8---
for most packages. I have no experience with LLVM, so can someone
confirm that this is the right way to go?
>> All in all, it looks like we can save 52 MiB out of 140 MiB from the LLVM
>> package (and 210 MiB from its closure).
>
> That’d be great.
To clarify the ambiguous sentence I wrote above, we would save 52 MiB
from the closure size of LLVM.
> An additional option would be to have a package with fewer backends by
> default (currently all of them are enabled and that takes up quite some
> space). In particular, Mesa doesn’t depend to depend on an LLVM variant
> with 15 backends when it’s only going to use one.
Are you suggesting an alternative or a tweak to add on top of my
suggestion?
Where would we store the different backends? In different outputs?
On which backend does Mesa depend for instance?
Cheers!
--
Pierre Neidhardt
https://ambrevar.xyz/
signature.asc
Description: PGP signature
Re: Reducing LLVM closure size
Hi, Pierre Neidhardt skribis: > For the first 2 links, click on > "View the file list for ..." link at the bottom to display the file > list. > > "bin" and "include" occupy some 35 MiB. First thing we can do is split > > - either move the libs to a "lib" output, > - or move the "bin" and "include" folder to a new output. > > The second approach has the benefit of being less disruptive for dependents. I have a slight preference for a “lib” output since that’s more in line with what we do for other packages. > Now if we look at the PKGBUILD, there are some interesting compilation > flags. If we remove > > "-DBUILD_SHARED_LIBS:BOOL=TRUE" > > and add > > "-DLLVM_BUILD_LLVM_DYLIB=ON" > "-DLLVM_LINK_LLVM_DYLIB=ON" > > it will produce a single libLLVM-10.so library. This reduces the "lib" > folder size from 107 MiB to 90 MiB. Nice! I looked for something like this when I packaged ‘clang-tools-extra’ and didn’t find it. This should go into the next ‘staging’ branch (or ‘core-updates’?). > All in all, it looks like we can save 52 MiB out of 140 MiB from the LLVM > package (and 210 MiB from its closure). That’d be great. An additional option would be to have a package with fewer backends by default (currently all of them are enabled and that takes up quite some space). In particular, Mesa doesn’t depend to depend on an LLVM variant with 15 backends when it’s only going to use one. Thanks, Ludo’.
Re: Using --manfistest with /manifest files
Dear, On Mon, 15 Jun 2020 at 19:08, elaexuo...@wilsonb.com wrote: > I went ahead and read through the threads that Pierre shared in a different > reply. For posterity and to collect my own thoughts, let me see if I can > distill the discussion so far: [...] > If the answer to the final question above is no, then that seems like a much > larger problem. However, if the answer is yes, then I would naively expect > profile reification to be mostly a matter of collecting together all the > sources of input that define a profile. Does forward-compatibility make this > less straightforward than I am thinking? What else am I missing? Thank you for your inputs. >From my understanding, what you are proposing is a variant of what Pierre proposes. To be precise, at Guix Days, Pierre and I discussed to change a bit the format of /manifest in order to unify the current situation of "manifest.scm" (code evaluated) and "/manifest" (flat data); as Pierre explained elsewhere in this thread. The change of internals will not happen, IMHO, because dealing with the general case adds too much burden, and the use-case discussed here -- recreate the exact same profile from another imperative profile -- does not deserve so much attention, again IMHO. >From my point of view, it is a technical problem of internal representation and then of UI. And Ludo expressed that he is not in favor for such internal change, because a lot of reasons he explained elsewhere (argh! I do not like what I am doing here: be imprecise without citing exactly, anyway!). The best, if I understand correctly, is simply to robustify the Pierre's script and provide a clean '--export-manifest' from a profile which should be an approximation. >> Sorry I am too lazy to search, but I think I remember that at the time >> Pierre sent -- probably in the mega thread :-) -- a small script to >> extract relevant information from /manifest; the preliminary >> for '--export-manifest'. :-) > > Perhaps you are thinking of Pierre's script here? > https://lists.gnu.org/archive/html/guix-devel/2020-02/msg00154.html Yes. All the best, simon
Re: Blog post on Further reduced bootstrap seed to 25%
Danny Milosavljevic writes: Hi Danny, > On Mon, 15 Jun 2020 14:54:39 +0200 > Jan Nieuwenhuizen wrote: > >> I’ve published a post about the second big reduction of the Guix >> bootstrap binaries >> >> https://guix.gnu.org/blog/2020/guix-further-reduces-bootstrap-seed-to-25/ > > "again decided to sponsor this work" link is broken > > Otherwise good :) Thanks, fix pushed! Janneke -- Jan Nieuwenhuizen | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
