Re: guix --container is RAM hungry

[Maxim Cournoyer]

Hi Ludovic,

Ludovic Courtès  writes:

> Hi Edouard,
>
> Edouard Klein  skribis:
>
>> I'm a huge fan of guix --container, and I created a system to use those
>> by default for network services. But the VPS these services run on has
>> only 2GB of RAM, and I just realized that a container, by default,
>> requires at least 200MB.
>
> Ouch, confirmed:
>
> $ \time -v guix shell -C coreutils -- uname 2>&1 |grep 'Maximum resident'
> Maximum resident set size (kbytes): 283048
> $ \time -v guix shell coreutils -- uname 2>&1 |grep 'Maximum resident'
> Maximum resident set size (kbytes): 56588
> $ guix describe
> Generation 297  Mar 24 2024 23:12:25(current)
>   guix 28bc0e8
> repository URL: https://git.savannah.gnu.org/git/guix.git
> branch: master
> commit: 28bc0e870b4d48b8e3e773382bb0e999df2e3611
>
>
> As raingloom and Ricardo wrote, there’s a Guile process that keeps
> waiting.

Is there a technical reason for this?  Couldn't we replace the
current Guix process with 'exec', as hinted by Edouard?

If possible, that'd be the most direct way to avoid any of the memory
cost incurred by Guile/Guix.

-- 
Thanks,
Maxim

Re: Backdoor in upstream xz-utils

[John Kehayias]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512


-BEGIN PGP SIGNATURE-

iQJRBAEBCgA7FiEEpCB7VsJVEJ8ssxV+SZCXrl6oFdkFAmYHK0sdHGpvaG4ua2Vo
YXlpYXNAcHJvdG9ubWFpbC5jb20ACgkQSZCXrl6oFdkFRA//WaJMegtHd88wlq0V
QovAYD7+d6zj5DxgVTiGKXckyKWx7AceVJ0WVp9MB+WxU8dEXepEnd9AHOA4v/Fb
HLy4prms+noIpXqHW5y6EDgbMiBUX2rk6UVq7qnLCPujfv3hrJl4S7B5fJxjLSM/
M++F40YKc6PNSjQHi9BH5+Vl70jGCIzXNcomvEanu4SAsXLSlEwvOlnAPD57mb4k
n4Tg4d7ExXjdi7/qdq/OnF2RGQjiLQ4qX7AeSu8kIaEaK3WdMy1JO1fy9vaZNuSg
oCuUGJYCFj60BEYDQdUM8NiNe76zVzXvP/wKrR1XpqsnK9keKKEZpuZCQmJApgCJ
dvVbrU8OfKPJ/B7CwNJu32FyrdgQt53ytYjNxs/cNNjB2ciDeIGszCzxwytRZz4k
JEbE8VZrUACNvQXCdRbr1Jse1+FuM2hjTwILdia/A8GcWn9tfmfGdqlqOuw6c8qG
hYX7l3+3t0c7VzLhgs2iE/BEKtUAYCrwRf+10J9dOm4TzmbEbg7+1j7FJcYhmIgJ
qeEXistWXx7FY2Yl0UjrNtxi3UGR5rnx2hAb3zEcMoqcHHKuKz/X8aeMfIHryn23
rQms/cVwAPeR908xwbJgqkzQhY5A9DrU+0VGssILyXKvMYp6xTXJ6cf2gGLyhAFF
VerlLVFCEHunNyWr94ZTeXr3p00=
=dUKI
-END PGP SIGNATURE-
Hi Ryan, Felix, and guix-devel,

On Fri, Mar 29, 2024 at 01:39 PM, Felix Lechner via Reports of security issues 
in Guix itself and in packages provided by Guix wrote:

> Hi Ryan,
>
> On Fri, Mar 29 2024, Ryan Prior wrote:
>
>> I'm reading today that a backdoor is present in xz's upstream tarball
>> (but not in git), starting at version 5.6.0. Source:
>> 
>
> Thanks for sending this!  This is an extremely serious vulnerability
> with criminal intent.  I cc'd guix-secur...@gnu.org just in case you
> haven't.
>

At least me (as part of guix-security) is aware and have been reading
the analysis and further investigation.

Both clever and interesting, but also worrisome. I think we were
rather lucky this was found relatively quickly, though it seems to
point to a bad actor and throws into question other projects (like
libarchive) which have contributions from the same identity. Likely
other accounts are involved too, so maybe on a positive side this
unravels other issues.

The discussion on Hacker News has also been informative (though rather
long now): 

>> Guix currently packages xz-utils 5.2.8 as "xz" using the upstream
>> tarball. [...] Should we switch from using upstream tarballs to some
>> fork with more responsible maintainers?
>
> Guix's habit of building from tarballs is a poor idea because tarballs
> often differ.  For example, maintainers may choose to ship a ./configure
> script that is otherwise not present in Git (although a configure.ac
> might be).  Guix should build from Git.
>

We discussed a bit on #guix today about this. A movement to sourcing
more directly from Git in general has been discussed before, though
has some hurdles. I will let someone more knowledgeable about the
details chime in, but yes, something we should do.

Unfortunately in this case, while it seems the older versions don't
have *this* exploit, given the perpetrator either is or has control over
a maintainer account, it throws into question a lot more than the most
recent version. We will have to keep a careful eye on this. I'm not
currently aware of anything untoward for our current version, so far.

>> Is there a way we can blacklist known bad versions?
>

I'm not sure what you mean, but I don't think so. The main danger is
in guix time-machine to the past, as you are (purposefully) going to
older versions of software. This is warned in the manual

though we should perhaps do this at runtime as well.

Even better would be if we can warn about known bad versions. Such a
tool was started (guix health) here:
 Anyone up for reviving it, now
that we have some changes that should make this more doable (based on
a quick glance of more recent messages)?

> Having said all that, I am not sure Guix is affected.
>
> On my systems, the 'detect.sh' script shows no referece to liblzma in
> sshd.  Everyone, please send additional reports.
>

Pretty sure we are not affected, at least with what is known: the
exploit targets particular systems and things like argv[0] being
/usr/sbin/sshd. A combination perhaps of who or what was being
targeted as well as trying to make this harder to discover.

Still, we should have an abundance of caution and pay close attention,
as there is much we don't know and a history of commits to go through.
As well as being suspicious in general of things like binary files
added to a release tarball (as a project we always try to make sure
there are no binary files anyway), this is a clear example of a
clever/malicious way of causing harm.

Please do feel free to report privately any concerns or potential
affected packages to guix-secur...@gnu.org as well. And if you are
interested in helping with these things, I'm sure we could rotate in
some people for that team.

Thanks all! An action-packed Friday.

John

Re: Backdoor in upstream xz-utils

[Tomas Volf]

Hello,

On 2024-03-29 13:39:59 -0700, Felix Lechner via Development of GNU Guix and the 
GNU System distribution. wrote:
> > Is there a way we can blacklist known bad versions?
>
> Having said all that, I am not sure Guix is affected.
>
> On my systems, the 'detect.sh' script shows no referece to liblzma in
> sshd.  Everyone, please send additional reports.

If nothing else, our xz is at 5.2.8.  I think the question was if there is a way
to blacklist specific known tarball to ensure no-one updates to it by accident.

(I do not believe Guix would be vulnerable even when built from the malicious
tarball, but that is a separate issue.)

Have a nice day,
Tomas

--
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.


signature.asc
Description: PGP signature

Re: Backdoor in upstream xz-utils

[Development of GNU Guix and the GNU System distribution.]

Hi Ryan,

On Fri, Mar 29 2024, Ryan Prior wrote:

> I'm reading today that a backdoor is present in xz's upstream tarball
> (but not in git), starting at version 5.6.0. Source:
> https://www.openwall.com/lists/oss-security/2024/03/29/4

Thanks for sending this!  This is an extremely serious vulnerability
with criminal intent.  I cc'd guix-secur...@gnu.org just in case you
haven't.

> Guix currently packages xz-utils 5.2.8 as "xz" using the upstream
> tarball. [...] Should we switch from using upstream tarballs to some
> fork with more responsible maintainers?

Guix's habit of building from tarballs is a poor idea because tarballs
often differ.  For example, maintainers may choose to ship a ./configure
script that is otherwise not present in Git (although a configure.ac
might be).  Guix should build from Git.

> Is there a way we can blacklist known bad versions?

Having said all that, I am not sure Guix is affected.

On my systems, the 'detect.sh' script shows no referece to liblzma in
sshd.  Everyone, please send additional reports.

Kind regards
Felix

Backdoor in upstream xz-utils

[Ryan Prior]

I'm reading today that a backdoor is present in xz's upstream tarball (but not 
in git), starting at version 5.6.0. Source: 
https://www.openwall.com/lists/oss-security/2024/03/29/4

Guix currently packages xz-utils 5.2.8 as "xz" using the upstream tarball. Is 
there a way we can blacklist known bad versions? Should we switch from using 
upstream tarballs to some fork with more responsible maintainers?

Ryan

Re: Error handling when 'guix substitute' dies

[Ludovic Courtès]

Hello,

Ada Stevenson  skribis:

>> diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
>> index 37cd08e289..3af0bf0019 100755
>> --- a/guix/scripts/substitute.scm
>> +++ b/guix/scripts/substitute.scm
>> @@ -494,7 +494,9 @@ (define* (download-nar narinfo destination
>> (define (try-fetch choices)
>>   (match choices
>> (((uri compression file-size) rest ...)
>> -   (guard (c ((and (pair? rest) (http-get-error? c))
>> +   (guard (c ((and (pair? rest)
>> +   (or (http-get-error? c)
>> +   (network-error? c)))
>> (warning (G_ "download from '~a' failed, trying next 
>> URL~%")
>>  (uri->string uri))
>> (try-fetch rest)))
>>
>> I’ll go ahead with this change if there are no objections.
> Looks good to me! Thanks for looking into this :)

OK, I’ll push it shortly, but…

Lars Bilke  skribis:

> thanks Ada for bringing this issue up again. I get the same error on
> `guix pull` almost always when I am on my enterprise
> network. Re-running `guix pull` a second time also almost always then
> runs fine. I checked with our IT: nothing suspicious on the network,
> i.e. no firewall blocking.
>
> I never experienced the error on my home network.

… your reports make me think there’s a bug lurking somewhere that
perhaps only manifests under some precise networking or timing
conditions.

Could the two of you run the following command in a loop to see whether
it’s easy to reproduce that GnuTLS error?

  guile -c '(use-modules (guix http-client) (ice-9 binary-ports))
(get-bytevector-all (http-fetch "https://ci.guix.gnu.org/nix-cache-info;))'

If you can reproduce it, could you capture the strace output of the
process?  You would run the command above prefixed by:

  strace -o log.strace -s 300 …

Thanks in advance!

Ludo’.

Re: March update on bordeaux.guix.gnu.org

[Christopher Baines]

Ludovic Courtès  writes:

> Hi!
>
> Christopher Baines  skribis:
>
>> Related to this, I've added options to the nar-herder to help change the
>> TTL being used, and reduced the TTL for bordeaux.guix.gnu.org to 10
>> minutes (from 180 days) [4]. This will at least mean that in the future,
>> the nar-herder on bordeaux will be able to delete zstd compressed nars
>> that it's generated more quickly.
>
> It’s not 10mn right now:
>
> $ wget --debug -qO/dev/null 
> https://bordeaux.guix.gnu.org/yr39rh6wihd1wv6gzf7w4w687dwzf3vb.narinfo 2>&1 
> |grep Cache
> Cache-Control: max-age=15502374
>
>
> Or maybe that’s just for newly created nars?

The max-age of that narinfo is currently based on the scheduled removal
of the zstd compressed nar, which is going to happen quite far in the
future.

I did think of a number of ways to approach this, and I'm not sure I've
settled on the right one yet. Maybe the TTL should be capped at 600, and
then drop to 0 as the time to remove the zstd nar approaches?

This narinfo for example currently has a max-age of 600:

  https://bordeaux.guix.gnu.org/ganfjbgy75r31bwzgddpnpswwjrrffvj.narinfo

> But then again, that’s the advertised TTL; the real TTL is still
> infinite, right?

As you probably know, the situation is more complex.

The problems caused when the nar-herder started removing zstd compressed
nars shows the difference between retention of the nar in some form, and
whether a cached narinfo response can be considered fresh or stale.

Users might also not notice the availability of zstd nars if they cache
responses forever, since currently there will be a lag between the nar
becoming available, and a zstd compression being created (although we
could generate zstd compressed nars for everything).

As described below, I also do want to start removing some nars, and that
requires not having an infinite TTL.

>> I'm really unsure about the need/usefulness of narinfo caching in
>> general, the cost of storing all these narinfos locally is quite high I
>> think and I don't really know why it's done.
>
> It’s a cache.  It’s useful to have this cache because in “typical” Guix
> usage you’re likely to ask repeatedly for the same substitutes.
>
> Regarding the cost, 3f5e14182931f123c10513a3e1e2abaebfb52279 made things
> more reasonable by putting a higher bound on narinfo retention.  On my
> laptop, I have:
>
> $ ls -lrt 
> /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7kiciw2pfhaf26a/
>  |wc -l
> 11549
> $ du -h 
> /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7kiciw2pfhaf26a/
>  
> 50M 
> /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7kiciw2pfhaf26a/
>
> Maybe that’s still excessive and we could further reduce the maximum
> caching time.

Having played around with this a bit (e.g. hacking guix weather not to
cache), I'm a bit sceptical. Given maintaining the cache takes time that
could be spent doing network I/O, and does potentially slow disk I/O, I
think it would be interesting to try and work out in what situations the
cache speeds things up overall, and in what situations it slows things
down overall..

>> 6: https://lists.gnu.org/archive/html/guix-devel/2023-05/msg00290.html
>
> BTW, should we document this mirror somewhere (and also ensure that Guix
> Foundation pays the bills), or do you view it more as an experiment for
> now?

If the project does want to provide mirrors, I think that would be
great. From this experiment, I think we have some evidence that there
are people using Guix outside of Europe, and in some cases they struggle
with the European based infrastructure. It also seems like these mirrors
do help, and the monetary cost isn't too high in my view.

I think we should probably wait until the project starts managing them
before documenting/advertising them more widely though.

>> Apart from that, the main thing on my mind for the next year regarding
>> bordeaux substitutes specifically is storage space. We're at 90%
>> capacity on hatysa (one of the two machines storing all the nars) so
>> this will need looking at shortly. I'd also like to finally get removing
>> nars that don't relate to the guix master branch happening, as that
>> should free up a little bit of space at least.
>
> Nice (the difficulty, I guess, is that some substitutes that we not
> initially for ‘master’ eventually get used on ‘master’).

Yep, my plan is to wait some long amount of time (say 6 months) before
scheduling things for removal, to check that they haven't started being
used by the master branch in this time.

We could also add other criteria as well, like tracking which nars are
generated by fixed output derivations and never removing them.

> On this issue, I think we should learn from fellow NixOS hackers.  They
> kept substitutes for ~20 years I think and are now in a difficult
> position because they cannot afford, financially, to keep that.  So one
> of the solutions envisioned was to figure out which of these millions 

Re: Google Summer of Code Inquiry

[Ekaitz Zarraga]

Hi all,

Yes, I proposed the project because that was something I wanted to do 
myself, but I didn't have the time for.


As Ludo suggests, start getting familiar with Guix's codebase and usage. 
I can help you with contribution later.


When digging on the code, this talk by Josselin Poiret is very helpful:

https://10years.guix.gnu.org/program/#guixy-guile-the-derivation-factory-a-tour-of-the-guix-source-tree

Feel free to contact us at any time and as questions. I more than open 
to answer all I can.


Also, I recommend you take a look to what an AppImage is, if you are not 
sure about it and how `guix pack` works, i.e. what contents do the 
`pack`s have. Once you get that, it shouldn't be too hard to combine both.


Cheers,
Ekaitz

On 2024-03-29 10:52, Ludovic Courtès wrote:

Hi Zachary,

Zachary Liebl  skribis:


I am interested in taking on one of your Google Summer of Code projects. I have 
been a long time NixOS user, and I need an excuse to finally get involved with 
Guix, and I think this is it.

I am particularly interested in the project "Add support for AppImage in guix 
pack." What should my next steps be in my application for your GSoC project?


Thanks for reaching out to us!  I’m Cc’ing Ekaitz who’s willing to
mentor this effort according to
 (other people
familiar with ‘guix pack’, myself included, can give a hand during the
project).

The first steps for GSoC candidates is usually to get familiar with the
project and its code base.  I would first recommend installing Guix and
playing around with it, including with ‘guix pack’.  You could try
making a first contribution, for example by adding a package that you
need but is missing.  Here’s a good entry point:

   https://guix.gnu.org/manual/devel/en/html_node/Contributing.html

These are the first things that come to mind but Ekaitz might have other
ideas!

Cheers,
Ludo’.

Re: March update on bordeaux.guix.gnu.org

[Ludovic Courtès]

Hi!

Christopher Baines  skribis:

> I've finally got around to starting to address the problems with
> disappearing nars discussed in [3]. The nar-herder now schedules nars
> which it's generated for removal and the time for removal is based on
> the TTL in use.
>
> 3: https://issues.guix.gnu.org/63634

Neat!  Yeah it’s important to honor the TTL that’s advertised in
narinfos.

> Related to this, I've added options to the nar-herder to help change the
> TTL being used, and reduced the TTL for bordeaux.guix.gnu.org to 10
> minutes (from 180 days) [4]. This will at least mean that in the future,
> the nar-herder on bordeaux will be able to delete zstd compressed nars
> that it's generated more quickly.

It’s not 10mn right now:

--8<---cut here---start->8---
$ wget --debug -qO/dev/null 
https://bordeaux.guix.gnu.org/yr39rh6wihd1wv6gzf7w4w687dwzf3vb.narinfo 2>&1 
|grep Cache
Cache-Control: max-age=15502374
--8<---cut here---end--->8---

Or maybe that’s just for newly created nars?

But then again, that’s the advertised TTL; the real TTL is still
infinite, right?

> I'm really unsure about the need/usefulness of narinfo caching in
> general, the cost of storing all these narinfos locally is quite high I
> think and I don't really know why it's done.

It’s a cache.  It’s useful to have this cache because in “typical” Guix
usage you’re likely to ask repeatedly for the same substitutes.

Regarding the cost, 3f5e14182931f123c10513a3e1e2abaebfb52279 made things
more reasonable by putting a higher bound on narinfo retention.  On my
laptop, I have:

--8<---cut here---start->8---
$ ls -lrt 
/var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7kiciw2pfhaf26a/
 |wc -l
11549
$ du -h 
/var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7kiciw2pfhaf26a/
 
50M 
/var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7kiciw2pfhaf26a/
--8<---cut here---end--->8---

Maybe that’s still excessive and we could further reduce the maximum
caching time.

> The mishandling of these zstd nars available from bordeaux was the last
> thing on my mind blocking proposing to switch the default substitute
> server ordering, so I've now gone ahead and done that in [5].
>
> 5: https://issues.guix.gnu.org/70028
>
> Obviously the differences to having ci.guix.gnu.org first in the list of
> substitute URLs is subtle, but I'm hoping this will help with substitute
> speed issues (as discussed in [6]) plus increase the impact of mirroring
> work for the bordeaux substitutes in the future.

Good!

> 6: https://lists.gnu.org/archive/html/guix-devel/2023-05/msg00290.html

BTW, should we document this mirror somewhere (and also ensure that Guix
Foundation pays the bills), or do you view it more as an experiment for
now?

> Apart from that, the main thing on my mind for the next year regarding
> bordeaux substitutes specifically is storage space. We're at 90%
> capacity on hatysa (one of the two machines storing all the nars) so
> this will need looking at shortly. I'd also like to finally get removing
> nars that don't relate to the guix master branch happening, as that
> should free up a little bit of space at least.

Nice (the difficulty, I guess, is that some substitutes that we not
initially for ‘master’ eventually get used on ‘master’).

On this issue, I think we should learn from fellow NixOS hackers.  They
kept substitutes for ~20 years I think and are now in a difficult
position because they cannot afford, financially, to keep that.  So one
of the solutions envisioned was to figure out which of these millions of
substitutes were “important” (for instance, source code), which turns
out to be very hard if you don’t have that info already at hand.

  https://discourse.nixos.org/t/nixos-s3-long-term-resolution-phase-1/36493

Do you think the Data Service or another source of info would let us
make such decisions?

If we take it to the extreme, we could have a sophisticated retention
policy like: drop all fixed-output derivations known to be available
from disarchive.guix + SWH, drop substitutes for packages that have less
than 100 dependents, etc.

Thanks for the update!

Ludo’.

Re: Losing signing keys for custom Guix channel

[elaexuotee]

Ludovic Courtès  wrote:
> elaexuo...@wilsonb.com skribis:
> 
> > Well, the catch 22 is that I've lost the original key and so can only sign
> > .guix-authorizations with the new one.
> 
> Ah sorry, I misread the thing I quoted.  :-)
> 
> So, you have your new key.  You add it to ‘.guix-authorizations’ in a
> commit signed with that new key.  And then, you make this commit the new
> introduction of your channel.
> 
> Does that make sense?
> 
> Ludo’.

Makes perfect sense! It's also exactly what I tried and what ends up failing
authorization on guix pull.

Re: Google Summer of Code Inquiry

[Ludovic Courtès]

Hi Zachary,

Zachary Liebl  skribis:

> I am interested in taking on one of your Google Summer of Code projects. I 
> have been a long time NixOS user, and I need an excuse to finally get 
> involved with Guix, and I think this is it.
>
> I am particularly interested in the project "Add support for AppImage in guix 
> pack." What should my next steps be in my application for your GSoC project?

Thanks for reaching out to us!  I’m Cc’ing Ekaitz who’s willing to
mentor this effort according to
 (other people
familiar with ‘guix pack’, myself included, can give a hand during the
project).

The first steps for GSoC candidates is usually to get familiar with the
project and its code base.  I would first recommend installing Guix and
playing around with it, including with ‘guix pack’.  You could try
making a first contribution, for example by adding a package that you
need but is missing.  Here’s a good entry point:

  https://guix.gnu.org/manual/devel/en/html_node/Contributing.html

These are the first things that come to mind but Ekaitz might have other
ideas!

Cheers,
Ludo’.

Re: Losing signing keys for custom Guix channel

[Ludovic Courtès]

elaexuo...@wilsonb.com skribis:

> Well, the catch 22 is that I've lost the original key and so can only sign
> .guix-authorizations with the new one.

Ah sorry, I misread the thing I quoted.  :-)

So, you have your new key.  You add it to ‘.guix-authorizations’ in a
commit signed with that new key.  And then, you make this commit the new
introduction of your channel.

Does that make sense?

Ludo’.