Specify runtime dependencies with propagated-inputs or wrapper scripts
Hello! I often meet problems where some packages don't work out of the box because they have some runtime dependencies like themes or third party programs. I solved these problems on occasion by making commits such as this: https://git.savannah.gnu.org/cgit/guix.git/commit/?id=00c1793ce8e2210e48b18422ea3e76da10541874 - which adds a wrapper script to "bin/chromium" and includes xdg-utils in PATH variable. It works but it's tedious to do for each and every binary in every single package. I see we also have a propagated-inputs field, which looks nice but for some reason people advice against using it. For what reasons? It is not as tedious as wrappers and I would really like to be able to specify runtime dependencies of packages using it without problems. I think we must find a solution to this runtime dependencies problem that is better than wrapper scripts because they are very tedious to create for every single binary in every single package. Another recent example being that the caja package depends on dconf to change it's settings, which is not installed by default when users use window managers like sway. Let's find a convenient solution here that would allow us to put an end to these problems that affect many new users and remains obscure for them that they would need to add additional packages in their configuration (and which). Léo signature.asc Description: This is a digitally signed message part
Re: Specify runtime dependencies with propagated-inputs or wrapper scripts
On Fri, 2021-03-26 at 20:36 +0100, Léo Le Bouter wrote:
> Hello!
>
> I often meet problems where some packages don't work out of the box
> because they have some runtime dependencies like themes or third party
> programs.
>
> I solved these problems on occasion by making commits such as this:
> https://git.savannah.gnu.org/cgit/guix.git/commit/?id=00c1793ce8e2210e48b18422ea3e76da10541874
> - which adds a wrapper script to "bin/chromium" and includes xdg-utils
> in PATH variable.
>
> It works but it's tedious to do for each and every binary in every
> single package.
>
> I see we also have a propagated-inputs field, which looks nice but for
> some reason people advice against using it. For what reasons?
One reason is incompatibility.
Try "guix environment --ad-hoc pidgin eom", it will fail.
(At least, it used to, due to both gtk+@2 and gtk+@3 being in a
single profile, or perhaps I got the package names wrong.)
Another reason is that you lose ‘scoping’ (not sure how to name this).
Consider each project to be like a Scheme module (M). (M) can export
variables like:
(define-module (kitties))
(define-public binaries
`(("stuff" ,do-stuff)
("surf" ,search-kitties)))
(define do-stuff ...)
(define search-kitties ...)
(define-public documentation
`(("stuff" "TODO")
("surf" "In the beginning, there were kitties")))
[some library for adopting kittens]
Assume there is a module (puppies) as well, that also
has a "surf" binary. Now consider an adoption centre application
(animal-adopt), that uses the "surf" binaries from both (kitties)
and (puppies) for finding kitties and puppies. In Scheme-land,
this works just fine, without ambiguity:
(call-binary "surf" #:module '(kitties))
(call-binary "surf" #:module '(puppies))
However, this won't work if all binaries were in a single, global
hash table, instead of being defined per-module (or package in Guix-speak).
How could "animal-adopt" use both "kitties" and "puppies" if it were
to add "kitties" and "puppies" to propagated-inputs? Also, if
"animal-adopt" only has "kitties" as propagated-input, how could
a user install both "animal-adopt" and "a-browser-simply-named-surf" in their
profile?
> it is not as tedious as wrappers and I would really like to be able to specify
> runtime dependencies of packages using it without problems.
> I think we must find a solution to this runtime dependencies problem
> that is better than wrapper scripts because they are very tedious to
> create for every single binary in every single package.
This seems a difficult problem.
> Another recent example being that the caja package depends on dconf to
> change it's settings, which is not installed by default when users use
> window managers like sway.
>
> Let's find a convenient solution here that would allow us to put an end
> to these problems that affect many new users and remains obscure for
> them that they would need to add additional packages in their
> configuration (and which).
In order to discover these kind of issues earlier, it would be interesting
to create a "ambient-trouble" package, that has binaries named after
the common culprits "xdg-config", "which", "cat" (for shell scripts actually
requiring "coreutils"), maybe some others, to be used like this:
$ guix environment --ad-hoc software-to-test ambient-trouble
$ AMBIENT_TROUBLE_LOG=$HOME/ambient-trouble.log software-to-test
The fake "xdg-config", "which", "cat" ... would then write a line
to $AMBIENT_TROUBLE_LOG and return 1 (failure). In the package guidelines
we could recommend testing with "ambient-trouble". To find out current
violations, ‘someone’ could install "ambient-trouble" in their profile
and see what breaks.
Now I think of it, it should be sufficient to test with
"guix environment --pure", but it clears out too many variables
(e.g. DISPLAY is not unset, but XAUTHORITY is --- see
%precious-variables in guix/scripts/environment.scm).
Is there any reason why XAUTHORITY is unset but DISPLAY isn't?
After all, "guix environment --pure" is not that pure that it
removes process persona, removes the reference to the root & current
working directory from the process ... I don't see why keeping the
reference to the display server accessible would be any different from
keeping standard input/output/error open.
It's all I/O, the only (conceptual) difference is in representation
(text/bytes or pixels).
Greetings,
Maxime.
signature.asc
Description: This is a digitally signed message part
Re: Security patching and the branching workflow: a new security-updates branch
Léo Le Bouter writes: > There is two ways to ship security fixes to packages: > > 1. Update to a patched version if upstream provides one > 2. Apply or backport individual patches to fix the issues in the > shipped version > > Grafts are most reliable for 2. but there's cases where using 2. is > lots of work and we can't afford that right now. An example is > ImageMagick where not all security issues get a CVE so essentially the > only way of getting security fixes is to fetch master or get the latest > release. > > There's also some types of packages where we are not sure whether we > can use grafting or not, such as Python ones. > > For these reasons, I would like to propose a new branch called > security-updates that would be based on master where we queue security > fixes that introduce any arbitrary number of rebuilds without using > grafts. > > We would merge the security-updates branch as soon as there is complete > substitute availability for the branch and it's future merged version > within master. > > The downsides of this approach are that: > > 1. Substitutes availability does not mean we can ship the updates > quickly because this might mean hundreds of megabytes if not gigabytes > of new substitutes to fetch to actually get the update. > 2. Users that don't use substitutes will suffer big rebuilds on each > security update shipped through this branch. > > For these reasons, grafting should still be preferred when possible, > but there are cases where it cannot be used for technical reasons or > lack of resources reasons but we still must provide a fix quickly. > > What do you think? Can you clarify what specific problem or problems you're proposing this security-updates branch to address? You mention applying and backporting patches is lots of work, and uncertainty around whether grafts work in particular situations. Personally I think staging and core-updates are quite a bit of work, and adding more complexity to the process involves more work in my mind. Additionally, this isn't going to provide more information about areas where grafts can't be used (if those exist). Now the software involved is getting better at rapidly building things for substitutes, personally I see a way forward through trying to measure and potentially increase the rate of change for outputs in general. Going faster also involves more work probably, but in terms of the process, that might just mean that updates to more packages can be merged to master directly, without sitting on a non-master branch. signature.asc Description: PGP signature
Security patching and the branching workflow: a new security-updates branch
Hello! There is two ways to ship security fixes to packages: 1. Update to a patched version if upstream provides one 2. Apply or backport individual patches to fix the issues in the shipped version Grafts are most reliable for 2. but there's cases where using 2. is lots of work and we can't afford that right now. An example is ImageMagick where not all security issues get a CVE so essentially the only way of getting security fixes is to fetch master or get the latest release. There's also some types of packages where we are not sure whether we can use grafting or not, such as Python ones. For these reasons, I would like to propose a new branch called security-updates that would be based on master where we queue security fixes that introduce any arbitrary number of rebuilds without using grafts. We would merge the security-updates branch as soon as there is complete substitute availability for the branch and it's future merged version within master. The downsides of this approach are that: 1. Substitutes availability does not mean we can ship the updates quickly because this might mean hundreds of megabytes if not gigabytes of new substitutes to fetch to actually get the update. 2. Users that don't use substitutes will suffer big rebuilds on each security update shipped through this branch. For these reasons, grafting should still be preferred when possible, but there are cases where it cannot be used for technical reasons or lack of resources reasons but we still must provide a fix quickly. What do you think? Léo signature.asc Description: This is a digitally signed message part
Re: guix import hackage fails with errors, and failed tests
Hi, On Fri, 26 Mar 2021 at 17:21, c4t0 wrote: > We can consider the argument backwards. If a user is not using haskell, > is not even packaging software, why make him have import-code at all? Well, you are not comparing apples to apples: Scheme files (~KB) vs Haskell (~GB)… > or why shipping autotools when we can do a parser and macro transformers > in guile-scheme, and so on? I'm pushing it to make the point here, don't > get me wrong; it's obvious that we won't be reimplement gcc in guile > ever. …I get your point but as I said elsewhere, importers are not converters. > in emacs we have autoloads. Maybe 'guix import hackage ...' should > then, and only then install the necesary dependencies. Or print a > warning that it'll use a scheme parser and that you can have a native > one installing X. Yeah, that’s an idea. It makes sense to me. > we already do this with cpan: "If Perl is available in the store, then > the ‘corelist’ utility will be used to filter core modules out of the > list of dependencies." quoted from the manual. But it does not say for the Else case. ;-) > it does something really better that I did though, it checks the store: > > (define %corelist > (delay > (let* ((perl (with-store store >(derivation->output-path > (package-derivation store (perl-package) >(core (string-append perl "/bin/corelist"))) > (and (access? core X_OK) >core Yes and it is cheap to test. --8<---cut here---start->8--- $ guix gc --list-live | grep '\-hello-' finding garbage collector roots... determining live/dead paths... $ guix gc --list-dead | grep '\-hello-' finding garbage collector roots... determining live/dead paths... $ guix repl GNU Guile 3.0.5 Copyright (C) 1995-2021 Free Software Foundation, Inc. Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'. This program is free software, and you are welcome to redistribute it under certain conditions; type `,show c' for details. Enter `,help' for help. scheme@(guix-user)> ,use(gnu packages base) scheme@(guix-user)> ,use(guix store) scheme@(guix-user)> (with-store store (derivation->output-path (package-derivation store hello))) $1 = "/gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10" scheme@(guix-user)> ,q $ guix gc --list-live | grep '\-hello-' finding garbage collector roots... determining live/dead paths... $ guix gc --list-dead | grep '\-hello-' finding garbage collector roots... determining live/dead paths... /gnu/store/2bx5a99d6z3fn1905yjvzqy890kicfqm-hello-2.10.tar.gz.drv /gnu/store/kql8b2hbsabcmany4m3hfm3wzdiymliy-hello-2.10-guile-builder /gnu/store/sihk9hiiqhqkckjs4dzl2vdk5dfv2923-hello-2.10.drv --8<---cut here---end--->8--- >> Well, Guix provides an extension mechanism, see GUIX_EXTENSIONS_PATH. >> Maybe it is the entry point for what you would like to have. > > Are you refering to GUIX_PACKAGE_PATH? No, really GUIX_EXTENSIONS_PATH. :-) > It's been almost two years since this bug > > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=35743 > > this one has a delay of three years until it solved: > > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=23961 > > and this one, has been 4 years since open with out much progress. > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25138 > > I didn't do a statistical analysis, and it can be other reason -maybe > nobody cares about haskell in guix and they are using nix?- but I think > that these issues don't get fixed bacause we need to modify a parser. Well, maybe it is a good idea to fallback to Cabal thing if it is already in the store. It would avoid to reinvent the wheel as you said. And it would probably fix a lot of corner cases. > btw I'm not arguing to push my patch, it's rather hacky, but to initiate > a discussion that can address this to the future. > I'm even willing to try to tackle the parser. Discussion and (new) ideas to address (old) issues are always welcome. :-) Cheers, simon
Re: guix import hackage fails with errors, and failed tests
Hello! zimoun writes: > Hi, > > On Thu, 25 Mar 2021 at 16:23, c4t0 wrote: >> zimoun writes: > >>> The issue with this is that Guix would somehow depend on Haskell. And >>> it would not happen: GHC is not bootstrappable, is huge, etc. >> >> I get the point of bootstrappable guix, and yes GHC is really huge, but >> if you're importing a haskell package, doesn't make sense to require the >> haskell ecosystem? you'll be having it installed anyways... >> >> I don't see a problem from the user pov, but I agree with you that is >> bad to require a haskell package at testing time. > > Your mean from the Guix user using Haskell point of view where they > already has this huge “guix size cabal-install”. yes it's big; but mostly ghc storage totalself /gnu/store/wkhglgmlz28kpkd3ky7f3kfjkxmvyb10-ghc-8.6.5 1639.2 1358.0 79.7% ... /gnu/store/h9lmnjq7ip6fgpxyj3akjybx3dam218d-cabal-install-2.4.0.0 1703.1 10.2 0.6% I'll have to ammend my self, it's no really that bad at testing time, the problem is as you correctly said: > What about the Guix > users point of view who are not using Haskell? That’s why Guix does not > depend on per language tools for the importers. We can consider the argument backwards. If a user is not using haskell, is not even packaging software, why make him have import-code at all? or why shipping autotools when we can do a parser and macro transformers in guile-scheme, and so on? I'm pushing it to make the point here, don't get me wrong; it's obvious that we won't be reimplement gcc in guile ever. in emacs we have autoloads. Maybe 'guix import hackage ...' should then, and only then install the necesary dependencies. Or print a warning that it'll use a scheme parser and that you can have a native one installing X. btw, I've made that cabal.scm use a command line utility for the patch, and it fails with a warning if something is wrong and continues to parse with the scheme parser. That because it only formats a file. But the idea is similar, it only uses it when it available. For testing, well, you should have it to address the bugs that it solves; otherwise you can do without. we already do this with cpan: "If Perl is available in the store, then the ‘corelist’ utility will be used to filter core modules out of the list of dependencies." quoted from the manual. it does something really better that I did though, it checks the store: (define %corelist (delay (let* ((perl (with-store store (derivation->output-path (package-derivation store (perl-package) (core (string-append perl "/bin/corelist"))) (and (access? core X_OK) core > Without speaking that > Guix runs on many architectures other than x86_64. > the same applies, if is using haskell chances are that it has x86. The status for arm is not so bad now a days https://www.haskell.org/ghc/blog/20200515-ghc-on-arm.html otherwise it won't make much sense to use 'guix import hackage' but it could use the scheme imp. It's true though that we couldn't do it with FFI, at least not without much of a hassle. > >> so... maybe a package that leverages the import system *and* the haskell >> ecosystem to aid the haskell package mantainers? >> >> what would be a good idea to leverage the other packaging system tools >> to import packages to guix with out making guix humongous? >> >> can we have a module with importers that augment the guix 'native' ones >> with foreign tools? > > Well, Guix provides an extension mechanism, see GUIX_EXTENSIONS_PATH. > Maybe it is the entry point for what you would like to have. > Are you refering to GUIX_PACKAGE_PATH? Lastly, programmers that know about parsers are a subset of the ones that can do api plumbering. It's been almost two years since this bug https://debbugs.gnu.org/cgi/bugreport.cgi?bug=35743 this one has a delay of three years until it solved: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=23961 and this one, has been 4 years since open with out much progress. https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25138 I didn't do a statistical analysis, and it can be other reason -maybe nobody cares about haskell in guix and they are using nix?- but I think that these issues don't get fixed bacause we need to modify a parser. btw I'm not arguing to push my patch, it's rather hacky, but to initiate a discussion that can address this to the future. I'm even willing to try to tackle the parser. Saludos! COD.
Re: Specify runtime dependencies with propagated-inputs or wrapper scripts
Hello, Am Freitag, den 26.03.2021, 20:36 +0100 schrieb Léo Le Bouter: > Hello! > > I often meet problems where some packages don't work out of the box > because they have some runtime dependencies like themes or third > party > programs. > > I solved these problems on occasion by making commits such as this: > https://git.savannah.gnu.org/cgit/guix.git/commit/?id=00c1793ce8e2210e48b18422ea3e76da10541874 > - which adds a wrapper script to "bin/chromium" and includes xdg- > utils > in PATH variable. Including xdg-utils in PATH works, but the preferred solution in Guix is to use absolute paths where possible. > It works but it's tedious to do for each and every binary in every > single package. You could argue the same about bootstrapping :P > I see we also have a propagated-inputs field, which looks nice but > for > some reason people advice against using it. For what reasons? It is > not > as tedious as wrappers and I would really like to be able to specify > runtime dependencies of packages using it without problems. Using propagated-inputs is a double-edged sword. On the surface, it seems to solve all your problems, but underneath it introduced back all the dependency shenanigans, that Guix seeks to avoid. In particular, you can't upgrade just one of N libraries, that all (indirectly) propagate the same input. > I think we must find a solution to this runtime dependencies problem > that is better than wrapper scripts because they are very tedious to > create for every single binary in every single package. Le me presenting an even more tedious solution :) > Another recent example being that the caja package depends on dconf > to > change it's settings, which is not installed by default when users > use > window managers like sway. Shouldn't there already be a library for this? *sigh* > Let's find a convenient solution here that would allow us to put an > end > to these problems that affect many new users and remains obscure for > them that they would need to add additional packages in their > configuration (and which). We could probably do our best to search for variants of exec in all kinds of programming languages automatically and then do our substitute*s with (which "program"), but I personally believe explicitly doing this is less surprising overall. Regards, Leo
Re: Security patching and the branching workflow: a new security-updates branch
On Fri, 2021-03-26 at 22:13 +, Christopher Baines wrote: > Can you clarify what specific problem or problems you're proposing > this > security-updates branch to address? Substitute availability of security updates when they are released, without causing big rebuilds on master for users before the build farm had time to produce substitutes. > You mention applying and backporting patches is lots of work, and > uncertainty around whether grafts work in particular > situations. Also that some times backporting is just not possible because security fixes are not properly labeled upstream as security-relevant and manual review of each and every commit is just not viable. > Personally I think staging and core-updates are quite a bit of work, > and > adding more complexity to the process involves more work in my > mind. Additionally, this isn't going to provide more information > about > areas where grafts can't be used (if those exist). I understand. There's lot of uncertainty on how grafts work exactly for me and in what situations they work and what situations they do not. The only way I am certain some security fix is correctly applied in GNU Guix is when the vulnerable version of the package is not packaged at all anymore in GNU Guix. > Now the software involved is getting better at rapidly building > things > for substitutes, personally I see a way forward through trying to > measure and potentially increase the rate of change for outputs in > general. Going faster also involves more work probably, but in terms > of > the process, that might just mean that updates to more packages can > be > merged to master directly, without sitting on a non-master branch. I would like this, merging things to master directly when we feel it is the right thing would be what I would like to do. Even if it causes big world rebuilds, when we can't graft. There's another thing I saw that was ongoing but can't remember where, that 'guix pull' could hold off updating to newer revisions unless substitutes are available. Thanks for your input. Léo signature.asc Description: This is a digitally signed message part
Cuirass not processing core-updates (or weirdly)
Hello! If you look at https://ci.guix.gnu.org/eval/13652 you can see that the evaluation of the derivation seems completed but there's no pending builds. What is happening here? Thank you signature.asc Description: This is a digitally signed message part
Re: Discrepancy between manual and channels.scm
Hi, On Sat, 20 Mar 2021 at 18:02, Ludovic Courtès wrote: > ison skribis: > >> Under "6.7 Declaring Channel Dependencies" the examples show names being >> quoted, such as (name 'some-collection) however this causes guix pull to >> fail. I don't remember the exact error but it was failing the "match" >> statement under "resolve-dependencies" in "guix/channels.scm". >> >> (name "some-collection") also fails. >> (name some-collection) succeeds, without any quoting. > > Oops you’re right; I’ll push a fix to the manual shortly. Indeed, it can be confusing to have: (name some-collection) where elsewhere it is quoted. Related to below, since it is data, does you make sense to have (name “some-collection”) for .guix-channel and a ’string->symbol’ somewhere. >> I'm not sure if it's the manual which needs fixing or the code, but >> specifying the name without quoting would be inconsistent with how >> channel names are declared in ~/.config/guix/channels.scm. > > It’s necessarily different because ‘.guix-channel’ is data whereas > ‘channels.scm’ is code. Right, option ’--channels/-C’ of “guix pull” says: «file must contain Scheme code» then section “Specifying Additional Channels” says «Note that the snippet above is (as always!) Scheme code». However, what is maybe less clear is that ’.guix-channel’ is data. Even if it is often said «meta-data file .guix-channel». Maybe the “meta” is confusing. Why simply not say «data file». Or «meta-data file .guix-channel (which is not code)». Well, it is confusing because code is data. :-) And all is using parenthesis. And the ’.guix-channel’ looks very similar as channels.scm. Cheers, simon
Re: cuirass
Hello, > * The "Home" & "Guix logo" buttons bothlink back to the home page (duplicate). Yes, but I'm not sure what to do about that. Maybe remove the logo link. > > * The search combobox help popup window > is not on top level, probably on all pages. > So other page elements are displayed over > it. I applied your patch, thanks! > * Overdirve1 does not have infos (missing > zabbix) Yes, that's on purpose, the Overdrive resources are limited, and running Zabbix and its PostgreSQL database wouldn't help. > I have collected a buch of ideas for additions. > > Should I create bugs for those or just the > above list will suffice ? We can discuss them here I think. Don't hesitate if you have other improvements ideas. Thanks, Mathieu
ARM64 hardware
Hello, I stumbled upon the following blog post: https://www.mininodes.com/arm-server-update-fall-2020/ Which was a summary of what should be available. I say "should" because some links are already dead now. There are mentions of non-FLOSS things there. -- Vincent Legoll
