Re: [aarch64] sbcl build failure

[Leo Famulari]

On Wed, Mar 31, 2021 at 12:15:01AM +0200, Vincent Legoll wrote:
> I'm looking at the following cl-zstd build failure:
> https://ci.guix.gnu.org/build/133326/details
> which is caused by sbcl build failure that I
> reproduced locally.

During the recent staging cycle, I noticed a huge number of SBCL build
failures, especially when emulating aarch64 on x86_64.
 
> I diffed the CI build log output with my local
> attempt and got the same thing modulo timings
> [1] and other harmless stuff.
> 
> [1] the Odroid N2 looks roughly 3x faster than the CI
> on this specific build / test (total 30 mins vs 90).

If it was an emulated build, that's more or less expected. The emulation
is very slow.

Comparing the Overdrive 1000 (4 x Cortex-A57, 25 watts TDP) to the
Odroid N2 (4 x Cortex-A73 and 2 x Cortex-A53, only 5 watts TDP), I'd
guess the Overdrive will complete the compilation more quickly, despite
the older design and lower core count.

I'm retrying the build now on one of the Overdrives.

Re: hikari build failure

[Leo Famulari]

On Tue, Mar 30, 2021 at 11:58:57PM +0200, Vincent Legoll wrote:
> > I've restarted .
> 
> I'm seeing overdrive1 as idle (as a lot of hydra workers) yet the
> restart is still in scheduled state.
> 
> What am I missing ?

I don't know exactly what happened here.

In general, most of the aarch64 substitutes are currently built using
(very slow) QEMU emulation on x86_64 machines.

Half of the x86_64 machines are reserved for this, and it seems like
they can never finish their tasks, compared to the machines building for
x86_64 / i686. This creates the distinctive "checkerboard" pattern often
seen at .

So, maybe your job was being handled by one of the x86_64-based
machines.

Anyways, I retried the bmake derivation on an Overdrive, and it
succeeded there.

The failure log doesn't say why it failed, but it does say 'command
"make" "-j" "16" "INSTALL=install" failed with status 2'.  Who knows,
maybe it's not designed to run with so many threads. Our overdrives only
have 4. Or maybe the emulation is just not good enough.

The hikari derivation also built successfully on an Overdrive. It should
be possible to retrieve substitutes now.

Re: GNOME 40 work should be done on Savannah

[Léo Le Bouter]

On Tue, 2021-03-30 at 21:55 -0400, Mark H Weaver wrote:
> I'm sorry if that happened.  It was not my intent.  Can you show me
> what
> I wrote that misrepresents your position?

I don't have the energy now I feel already bad enough this discussion
even happened, in a way that even Ludo, the creator of the GNU Guix
project, ended up saying that they think I don't understand the review
process. I think I do understand it and I wonder why anyone made me a
GNU Guix committer if they think any other way.

> I answered essentially the same question in my last message, so I'm
> not
> sure why you're asking again.  We've been successfully accepting
> contributions from non-committers for years.

I think one has to realize that being a non-committer is a really
frustrating experience even if your work is of good quality and that
frustrating experience is demotivating and makes people give up. And
having a branch and special relationship with a GNU Guix committer
(like what we're doing with Raghav) is helpful to capture Raghav's
motivation and energy in a way they do not get frustrated and give up.

> Yes, of course.  To be clear: you are well within your *rights* to do
> so, regardless of what anyone else thinks.
> 
> My main concerns are:
> 
> (1) The primary branch for GNOME 40 work should be on Savannah, and
> that's where we should encourage Guix developers to do this work.
> No Guix developer should be asked to work on an external site in
> order to contribute to the GNOME 40 effort.

That's where it felt it was the best place to collaborate for me and
Raghav (non-committer).

> 
> (2) Any work that you and Raghav do on an external site should be
> _regularly_ merged into Savannah, in manageable pieces, after
> being
> reviewed of course.  If you do this, my concerns over review
> quality
> would be addressed.

We already seek to do this.

> 
> (3) If changes are made on the 'wip' branch on Savannah that conflict
> with your preliminary work on an external site, it would be your
> responsibility to rebase your work as needed, just as anyone
> proposing a patch would be expected to do.  That should be an
> incentive to submit your work to Savannah early and often.

If changes were to be made in a way that rebasing work and solving
conflicts becomes too much trouble we will probably give up (Raghav
probably) and there wont be any GNOME 40 upgrade.

> Would this be acceptable to you?

I don't even know anymore.

> Regards,
>   Mark

Léo


signature.asc
Description: This is a digitally signed message part

Re: GNOME 40 work should be done on Savannah

[Mark H Weaver]

Hi Léo,

Léo Le Bouter  writes:
> I think Mark misrepresents my sayings in their messages.

I'm sorry if that happened.  It was not my intent.  Can you show me what
I wrote that misrepresents your position?

> We are ONLY using git to exchange work and collaborate on the patchset
> with Raghav, how else do you want us to work with Raghav on some git
> branch on Savannah if Raghav doesnt have commit access?

I answered essentially the same question in my last message, so I'm not
sure why you're asking again.  We've been successfully accepting
contributions from non-committers for years.

> Don't we have the right with Raghav to cooperate together on some
> patchset however we want?

Yes, of course.  To be clear: you are well within your *rights* to do
so, regardless of what anyone else thinks.

My main concerns are:

(1) The primary branch for GNOME 40 work should be on Savannah, and
that's where we should encourage Guix developers to do this work.
No Guix developer should be asked to work on an external site in
order to contribute to the GNOME 40 effort.

(2) Any work that you and Raghav do on an external site should be
_regularly_ merged into Savannah, in manageable pieces, after being
reviewed of course.  If you do this, my concerns over review quality
would be addressed.

(3) If changes are made on the 'wip' branch on Savannah that conflict
with your preliminary work on an external site, it would be your
responsibility to rebase your work as needed, just as anyone
proposing a patch would be expected to do.  That should be an
incentive to submit your work to Savannah early and often.

Would this be acceptable to you?

Regards,
  Mark

Re: Question about compile packages

[raingloom]

On Tue, 30 Mar 2021 07:12:27 -0400
Julien Lepiller  wrote:

> Not sure about your first question. Maybe create your own fork and
> regularly rebase it? Guix will not be very happy with that I think,
> but it should work.
> 
> To pass --no-substitutes, you can pass it to guix-daemon, or remove
> the authorized keys from /etc/guix/acl. The second option lets you
> download substitutes for fixed output derivations (basically source
> tarballs), so it's easier in case some of those tarballs disappear
> from their original location.
> 
> I think the first option is selectable from the installer, but I
> might be wrong. Otherwise, stop guix-daemon from herd and run it
> manually with your options.
> 
> Le 29 mars 2021 15:05:08 GMT-04:00, Charles Direg
>  a écrit :
> > Dear,
> >
> >How can I modify the flags that any program is compiled with within
> >guix? That
> >is, I can allow in the gnu-build-system to modify it globally so
> >that I can
> >add the build flags to any package, for example, add the flags -O2
> >-march=native -mtune=native so global as I already mentioned, so that
> >these
> >are added to each package at the time of compilation, this would not
> >be within the guix development environment, because what I want is
> >that this
> >compilation is natively for my pc.
> >
> >As a second question, how could I set the --no-susbtitutes option
> >when installing the guix system from ISO, since I would like all
> >installed packages to be compiled natively first?
> >
> >I really appreciate your kind time and I look forward to your
> >responses.
> >
> >Sincerely,
> >~ Abraham Huerta  

I'm mostly guessing here, but wrapping gcc with a script that sets the
options you want might work. Could try creating a gcc-toolchain that
is set up the way you want it (don't even have to recompile, like i
said, just create a wrapper script that calls the original toolchain)
and then run guix build --with-toolchain=gcc-impure-native-toolchain

At least that's how I'd get started.

You could also just modify the toolchain package, but then you'd have
to recompile everything.

Anyways, if you want -march=native optimisations, there is some recent
discussion around some new GCC feature that detects CPU features at
runtime. The high performance computing (HPC) related developers are
very interested in using it in Guix.

meson-build-system cross compilation

[Danny Milosavljevic]

Hi Raghav,

since you have been using meson build system a lot, could you add support for
cross-compilation to guix/build-system/meson.scm please?  I would do it myself
if I knew how.

See https://issues.guix.gnu.org/44244

See also https://mesonbuild.com/Cross-compilation.html

As it is now, a lot of basic guix packages will break when cross-compiling
since they have been switched to meson-build-system.



pgpMcZkEc5YAW.pgp
Description: OpenPGP digital signature

Re: Question about compile packages

[Mark H Weaver]

Hi,

Charles Direg  asked:

> How can I modify the flags that any program is compiled with within guix? That
> is, I can allow in the gnu-build-system to modify it globally so that I can
> add the build flags to any package, for example, add the flags -O2
> -march=native -mtune=native so global as I already mentioned, so that these
> are added to each package at the time of compilation, this would not be
> within the guix development environment, because what I want is that this
> compilation is natively for my pc.

Julien Lepiller  replied:

> Not sure about your first question.  Maybe create your own fork and
> regularly rebase it?  Guix will not be very happy with that I think,
> but it should work.

I've been doing this all along, since the early days of Guix.  No one
seems to be unhappy with me about it, and I'm not sure why they would.

I build everything locally, never use substitutes, and never use "guix
pull".  I always run guix via ./pre-inst-env from a git checkout of my
personal private branch of Guix.  I use a little script in ~/bin/guix
that lets me simply type "guix ..." like everyone else:

--8<---cut here---start->8---
#!/bin/sh

source /var/guix/gcroots/per-user/mhw/environment-guix/etc/profile
exec /home/mhw/guix/pre-inst-env guix "$@"
--8<---cut here---end--->8---

I periodically rebase my private branch onto 'master', because I want to
keep my private modifications in the form of a small number of patches,
but if I didn't care about that, periodically merging master into my
branch would also work.

This approach gives *enormous* flexibility -- you can easily change just
about any aspect of Guix, while still benefitting from our upstream
work, and with nearly automatic merges thanks to git -- but there are
some rough edges.

One is that if you're not careful to keep a suitable build environment
for building Guix, and your git checkout gets into a broken state, you
can get stuck.  Also, sometimes Guix adds additional build requirements,
and so you can get into a state where you've updated your git checkout
to a version with new requirements, but you cannot build those new
requirements because your git checkout (from which you run guix) is not
yet built.

These problems are manageable, although it's slightly tedious.  First,
I'm very careful to always keep a GC root to a known working build
environment: a profile created by "guix environment guix", as referenced
by $GUIX_ENVIRONMENT, which I make the target of a symlink
/var/guix/gcroots/per-user/mhw/environment-guix.  When I update this GC
root, I always keep the older GC root around until I'm absolutely
certain that the new one works.

With this GC root in hand, I can always load this build environment,
without running guix, by simply sourcing its etc/profile file (using the
first command of the script I quoted above).

If Guix adds a new requirement, and I fail to notice before updating my
git checkout, I can always roll back to an earlier checkout, rebuild it,
and then use that to run "guix environment guix --ad-hoc
".

Also, beware that when you build everything locally, which takes a
*long* time (a couple of weeks of continuous building for a GNOME system
on a Thinkpad X200), it's important to pass "--gc-keep-derivations=yes"
and "--gc-keep-outputs=yes" to the Guix daemon.

I have something close to this in the 'services' field of my OS config:

  (modify-services %desktop-services
(guix-service-type config =>
   (guix-configuration
 (inherit config)
 (use-substitutes? #f)
 (authorize-key?   #f)
 (authorized-keys '())
 (substitute-urls '())
 (extra-options '("--gc-keep-derivations=yes"
  "--gc-keep-outputs=yes")

I can say from long experience that with the methods outlined here, you
can make arbitrary local changes to your Guix that affects *every*
package built in Guix.  I've gone through long periods of running Guix
with additional patches deep in the early bootstrap.

If more people are interested in using Guix in this way, the experience
could probably be made much nicer.

* * *

In theory, the "--url", "--branch", and "--commit" to "guix pull" should
allow this level of flexibility without the gotchas listed above, at
least for those who are willing to put their branch of Guix on a public
Git repo.

However, I'm not sure that "guix pull" will actually work correctly if
you make "rebuild-the-world" changes like the ones that Charles is
asking about.  I suspect this because of the bug I reported here:
.  Since I seem to be the only one using
Guix in this way, the bug report didn't get attention.  That bug was
about "make as-derivation", but I guess it's likely that "guix pull"
would be affected as well.

Mark

Re: GNOME 40 work should be done on Savannah

[Léo Le Bouter]

On Tue, 2021-03-30 at 14:12 +0200, Ludovic Courtès wrote:
> > I don't feel like people should be barred to contribute to that
> > GNOME
> > 40 upgrade because they arent an approved committer. That doesnt
> > feel
> > inclusive to me.
> 
> I respectfully think you misunderstand the review process.  Review is
> about sharing responsibilities and reducing the likelihood of
> mistakes.

Additionally, please put that quote into it's surrounding context and
don't assume things I have not said, I was explicitly speaking about
wip branches on Savannah excluding non-committers from collaborating on
them easily.

If I created a wip-gnome-40 branch on Savannah then Raghav cannot push
and we cannot work together easily, that way the GNOME 40 upgrade
probably wont happen because Raghav is already tired I felt in some
way.

I think the way we work here is the best conditions to capture Raghav's
motivation and energy right now and invest it within this GNOME 40
upgrade before they give up and we don't have this GNOME 40 upgrade
worked on. We need git to collaborate on patches together, that's all. 
But those patches of course do get through the review process, first
with me reviewing changes as they happen, then as a whole also, because
we wont push anything to core-updates, master or else without review.

Léo


signature.asc
Description: This is a digitally signed message part

Re: GNOME 40 work should be done on Savannah

[Léo Le Bouter]

On Tue, 2021-03-30 at 14:12 +0200, Ludovic Courtès wrote:
> I respectfully think you misunderstand the review process.  Review is
> about sharing responsibilities and reducing the likelihood of
> mistakes.
> 
> It’s crucial from many different perspectives: security-wise,
> socially
> (the process is transparent, everyone gets a chance to chime in or to
> just see what’s cooking), and technically (we all learn and build
> better
> software, and we reduce the risk that users receive broken software
> on
> their next ‘guix pull’).
> 
> The process to get commit access is documented and it revolves around
> two criteria: the person must have experience with the project and
> must
> know the rules.
> 
> That’s not “bureaucracy”: it’s about building a community around
> mutual
> understanding of what each one of us may or may not expect from their
> peers.
> 
> 
> All that said, it’s of course fine to collaborate on external
> repositories, even though I agree with Mark that getting “closer” to
> Guix folks may be profitable to everyone.  In the end, work should be
> submitted for review, and for a patch set like this, it’ll probably
> be a
> ‘wip-’ branch on Savannah.
> 
> Thanks,
> Ludo’.

I think Mark misrepresents my sayings in their messages. We are ONLY
using git to exchange work and collaborate on the patchset with Raghav,
how else do you want us to work with Raghav on some git branch on
Savannah if Raghav doesnt have commit access? Don't we have the right
with Raghav to cooperate together on some patchset however we want? It
never meant that review process had to be skipped in ANY way.

Only Raghav and myself has showed up to cooperate on GNOME 40 upgrade
for now, and that's also why we are doing it this way.

I feel bad now that it seems like everyone assumes we are doing things
we are not.

Léo


signature.asc
Description: This is a digitally signed message part

Re: Security patching and the branching workflow: a new security-updates branch

[Léo Le Bouter]

On Tue, 2021-03-30 at 13:48 +0200, zimoun wrote:
> Ahah, I am happy to know it.  I hope it is because a
> “miscommunication»
> and not because you do not carefully read or because maybe you only
> see
> through the tiny lens of known security vulnerabilities.  From my
> opinion, your point of view to tackle the issue is wrong.  That’s
> said.

I feel harassed by your comments because you obsessed on this zstd
issue and try to make it the cause of some other problems you saw
without any evidence.

I don't think I look through the "tiny lens of known security
vulnerabilities", every distro has prioritization for security updates,
I think it is the right thing to do, that's how I work with it and I
think it is right. If you don't agree then OK but I wont stop thinking
it's the right way to do things and that not prioritizing security
issues does not work.

> 
> Best regards,
> simon

Léo


signature.asc
Description: This is a digitally signed message part

Re: GNOME 40 work should be done on Savannah (was: Re: GNOME 40)

[Léo Le Bouter]

On Tue, 2021-03-30 at 02:41 -0400, Mark H Weaver wrote:
> Sorry, but that's simply false.  You _do_ have a choice.  You can do
> what we've been doing in the Guix community for years: as a
> committer,
> _you_ can commit the work of non-committers on their behalf.  If not
> you, then any of the other ~64 Guix committers can do so.
> 
> Needless to say, before committing, you must review the proposed
> patches, for the sake of your reputation.  The fact that you must do
> this is a *feature*, not a bug.

Nobody is talking about skipping the review process, as I said, it's
just about collaborating over git rather than with patches (good for
little patches but very troublesome for larger patchsets)

> No one is "barred" from contributing.  Raghav and many others without
> commit access have been successfully contributing to Guix for years.
> 
> I understand that it's inconvenient.  Naturally, you would like to
> eliminate that inconvenience.
> 
> The thing is, the work of non-committers *must* be reviewed at some
> point, anyway.  Moreover, a committer must take responsibility by
> digitally signing it.  To eliminate either of these steps would put
> us
> at risk.
> 
> There's no guarantee that the work of Guix committers will be
> reviewed
> by anyone else, because no one else's reputation is on the
> line.  Some
> of us try to keep an eye on things, but I would not bet on that
> oversight being comprehensive.  I'm certainly not doing it
> comprehensively.
> 
> With this in mind, I think that we *should* have a high standard for
> committers.  The security of our systems, as well as Guix's
> reputation
> as a project, depends upon the good judgment of _every_ Guix
> committer.
> 
> Observe what can happen with projects that are too lax:
> 
>   
> https://arstechnica.com/gadgets/2021/03/buffer-overruns-license-violations-and-bad-code-freebsd-13s-close-call/
> 

I don't think we are on the same page.

> Upgrading GNOME is not trivial.  It will be a large patch set.  A
> large
> patch set presented to guix-patches when the branch is ready to merge
> is
> far less likely to get careful review than if the review is done a
> few
> commits at a time.  That's because, at any given time, it's easier to
> find Guix developers with a few minutes available to carefully review
> a
> small handful of commits, than to find developers prepared to review
> a
> non-trivial branch merge.  If they're reviewed at all, reviews of
> larger
> code drops are more likely to be superficial.

We also go by steps and review things as they appear with Raghav,
collaborating over git is just easier than exchanging patches for
testing/review back

> * * *
> 
> In summary: it seems to me that working in an external repository
> with a
> larger set of committers would not actually save time, because it
> would
> merely postpone the required review work until the end of the process
> when the branch is ready to be merged into Savannah.  Moreover, it
> would
> likely reduce the quality of that review work.
> 
> Does that make sense?

Not to me, the git vector is just a way to collaborate more easily but
things would still get reviewed (in smaller patchsets if necessary
also) before getting merged.

> Regards,
>   Mark

Léo


signature.asc
Description: This is a digitally signed message part

Re: Will 2021 be the year of build systems on gexps?

[Timothy Sample]

Hi Ludo,

Ludovic Courtès  writes:

> So this is it, 2021 *is* the year of build systems on gexps!

Wow!  Congratulations!  I am very happy to see this finally come
together.

> It’s crazy that it took 6 years (six!) to get past the finish line.

I think I’ve only known what a gexp is for three years... and that’s
about how long I’ve been excited by this feature.

Thanks for sticking with it for six years!


-- Tim

Re: [PATCHES] ImageMagick security updates without grafting

[Mark H Weaver]

Mark H Weaver  writes:

> Maxime Devos  writes:
>
>> guix build $PACKAGES
>> # maybe guix build $PACKAGES --no-grafts?
>> guix graph --type=references $PACKAGES
>> # ^ look in output for "imagemagick".
>
> For the record, it seems that this command gives false positives.

Sorry, I was mistaken here.  That command appears to be reliable for
this purpose.

> As pointed out in , the output of that
> command suggests that 'inkscape' retains references to 'imagemagick',
> but that turns out to be false, at least on my system.

It turns out we were talking about two different versions of 'inkscape'.
I was confused by the fact that our 'inkscape' variable points to an
older version of inkscape than "inkscape" selects on the command line.

Anyway, it turns out that inkscape@1.0.2 improperly retains a reference
to its native-input 'imagemagick', but inkscape@0.92.4 does not.
See  for more.

> I suppose the behavior of "guix graph" here makes sense, and is likely
> _not_ a bug, because IIUC "guix graph" does its work without requiring
> 'imagemagick' to be built,

What I wrote is true for many of the graph types supported by "guix
graph", but not when "--type=references" is passed.
Sorry for the confusion.

   Mark

Re: Let's include powerpc64le-linux in the next release

[Vincent Legoll]

Hello,

On Tue, Mar 30, 2021 at 10:24 AM Ludovic Courtès  wrote:
> Plus, I’m really grateful to OSUOSL for giving
> us access to this machine!

Yes, it's really nice to have access to those, and in the future, if
we want to really support the ppc64le with substitutes, we may
ask for a CI-class VM which should have better performance.

I've also asked for details of the openstack setup, and the virtual
disks provided to the VMs are networked ceph storage which is
quite slow, they are investigating the problem. I also asked if they
can try to provide hypervisor-local scratch storage (way faster but
not durable, and would make the VM non migratable) to the VMs and
they may be able to do so in the future.

Tchuss

--
Vincent Legoll

[aarch64] sbcl build failure

[Vincent Legoll]

Hello,

I'm looking at the following cl-zstd build failure:
https://ci.guix.gnu.org/build/133326/details
which is caused by sbcl build failure that I
reproduced locally.

I diffed the CI build log output with my local
attempt and got the same thing modulo timings
[1] and other harmless stuff.

[1] the Odroid N2 looks roughly 3x faster than the CI
on this specific build / test (total 30 mins vs 90).

-- 
Vincent Legoll

Re: hikari build failure

[Vincent Legoll]

> I've restarted .

I'm seeing overdrive1 as idle (as a lot of hydra workers) yet the
restart is still in scheduled state.

What am I missing ?

-- 
Vincent Legoll

Re: Will 2021 be the year of build systems on gexps?

[Ludovic Courtès]

Hi Maxim and all,

So this is it, 2021 *is* the year of build systems on gexps!

I’ve rebased it and pushed on ‘core-updates’ with
2eafeb2f3d661061bc412c3f27c90202e4532532 as the tip.

The build farm has a bit of work to do now it seems.  Hmm actually the
evaluation at  just failed
with a yellow cross, not sure what that means.

Anyway, I’ll be monitoring in the coming days, but do ping me when you
encounter problems!

It’s crazy that it took 6 years (six!) to get past the finish line.  The
initial reason for not merging it was that I was worried about
performance regressions; there’s still work to do in this area, as I
wrote earlier, but it’s looking good.  Another reason is that I was
afraid of unlocking features that would break assumptions we make about
packages—for instance, that all the inputs are specified in the various
inputs fields, and that these fields are all we need to care about when
doing graph rewriting.  (I now think this fear was unwarranted.)

And then, rebasing the branch became harder every time, there were more
and more build systems that needed to be converted… and there were
always more pressing issues to work on.

What’s the lesson here?  I don’t know!  Had it been done six years ago,
it would have had to go through a number of iterations:
‘with-imported-modules’ and ‘with-extensions’ didn’t exist, the template
mechanism in (guix monads) didn’t exist, self-referential records
(‘this-package’, etc.) weren’t there.  It would have worked though, and
perhaps we’d have explored other parts of the design space, who knows.

Ludo’.

Re: hikari build failure

[Tobias Geerinckx-Rice]

Vincent Legoll writes:

What should we do ? just wait for next build ?


I've restarted .

Kind regards,

T G-R

hikari build failure

[Vincent Legoll]

Hello,

I saw the aarch64 build failure for hikari on cuirass
https://ci.guix.gnu.org/build/133895/details
it looked like the failure was because bmake (a dep)
failed, I tried to build it locally which succeded, then
also retried hikari with success.

I'm on a binary install above armbian on odroid N2.

vince@odroidn2:~$ guix describe
Génération 230 mars 2021 22:59:41(actuelle)
  guix 634d984
URL du dépôt : https://git.savannah.gnu.org/git/guix.git
branche : master
commit : 634d9845a6b4e362f32ba369ae42851719455ba3

The retry button on cuirass gave me a 403 forbidden.

What should we do ? just wait for next build ?

-- 
Vincent Legoll

Re: Deep vs Shallow trace: Removing the tradeoff?

[Bengt Richter]

Hi ilmu,

On +2021-03-28 23:16:23 +, ilmu wrote:
> First of all, thanks for engaging with me, sorry for not putting more work 
> into the initial email. Note that I have attached inline images from the 
> paper, the paper is also attached to this email.
>

An interesting paper..

I have long been wondering if reproducibility is too exclusively focused
on the source transformation chain and the tools implementing that.

E.g., for programs that execute like side-effect-free functions,
why not an early-cutoff based on a functionally well-tested upgrade
written in a different language, with a totally different source?

If you source the following in a subshell

cat ./early-cutoff.txt
--8<---cut here---start->8---

export HELLO_TO=World

strace -qqxs120 -e write \
guile -c '(let* ((msg (string-append "Hello " (getenv "HELLO_TO") "\n"))) 
(display msg))' |& tr , $'\n'|grep '^ *"'

strace -qqxs120 -e write \
echo "Hello $HELLO_TO" |& tr , $'\n'|grep '^ *"'

--8<---cut here---end--->8---

Like,

(. early-cutoff.txt)
--8<---cut here---start->8---
 "Hello World\n"
 "Hello World\n"
--8<---cut here---end--->8---

It informally shows that in a particular context,

--8<---cut here---start->8---
guile -c '(let* ((msg (string-append "Hello " (getenv "HELLO_TO") "\n"))) 
(display msg))'
--8<---cut here---end--->8---

can substitute for
--8<---cut here---start->8---
echo "Hello $HELLO_TO"
--8<---cut here---end--->8---

So, what kinds of build sequences could be cut off early if you ignore how
you produced the code (i.e. just compile whatever source you like with whatever
tools you like) so long as the resultant code passed the functionality tests?

> > What I understand from the above is that there is not so much of a 
> > difference between Bazel and Nix/Guix. The "shallow" tracing is a deep 
> > tracing. It just records something different.
> 
> [classification.png]
> 
> The framework they present is that a build system can be classified based on 
> how it decides whether a rebuild is necessary and how it schedules the 
> necessary rebuilds (i.e. how it sorts the work to be done). They consider 
> these orthogonal strategies a complete description of "what a build system 
> is".
> 
> [constructive_traces.png]
> 
> The constructive trace allows you to keep intermediate artifacts but creates 
> the extra overhead of managing these.
> 
> [deep_constructive_traces.png]
> 
> Since a deep constructive trace only looks at the terminal dependencies you 
> have something of a merkle-dag. What I was suggesting is to be able to do 
> "shallow builds" while still supporting "early cutoff". For reference:
> 
> [early_cutoff.png]
> 
> Early cutoff is a very desirable property that nix does not have (and I 
> assume therefore that neither does guix).
> 
> [shallow_builds.png]
> 
> Basically, if we have a proof that the immediate dependencies are complete 
> then we don't need to fetch anything deeper in the dependency tree.
> 
> > In Nix/Guix, we recursively compute the hash of packages from their inputs, 
> > receipe and other stuff. This hash is unrelated to the checksum of the 
> > result and changes as soon as one element changes. It essentially records 
> > the whole source dependency graph.
> > From what you write, Bazel records the checksum of inputs and receipe, so 
> > it essentially encodes the whole binary dependency graph. It's not 
> > "shallow", as a real change down the graph can have repercussions higher up.
> > Actually Eelco described something similar for Nix in his thesis, the 
> > intensionnal model. As you noticed, different receipes or inputs can lead 
> > to the same output (bitwise). The idea is to declare the expected checksum 
> > of the result, and use this to compute the store item name. As long as you 
> > apply changes that do not affect the output, different receipes (updates) 
> > can be used to generate the same item. So if you update a package, it might 
> > affect its output checksum, so you need to change it. Dependents might not 
> > be affected, in which case you can block update propagation there.
> > The question being how to monitor that packages need to be rebuilt or have 
> > the same hash?
> 
> I think the pictures above should clarify, I should have looked at the paper 
> again before sending my previous email, I read it quite a long time ago.. The 
> property I was trying to recover in the nix-style build systems is this idea 
> of an early cutoff. The strategy for achieving that is to use constructive 
> traces with depth 1 (what I was calling "shallow traces") and then recursive 
> succinct arguments of knowledge for retaining the shallow build property that 
> you get from the "infinitely deep 

rust-tempfile-3 update to 3.2.0 breaks sequoia build

[Hartmut Goebel]

Hi Nicolas,

building sequoia is currently broken in master with
"syn::export::ToTokens" not found.

I tracked this down to 6513650d40f74 "gnu: rust-tempfile-3: Update to
3.2.0." (2021-02-16). The updated package also updates some
dependency-requirements: cfg-if 0.1 -> 1, rand 0.7 -> 0.8 and
redox-syscall 0.1 -> 0.2.

While in theory - according to semantic versioning -, updating 3.1.x to
3.2.x should not any effect on others packages, this updates has :-(

I tried building with rust-tempfile-3.1, with no success - this fails
with "rand::rngs::OsRng" missing. Updating rust-rand-core-0.6 to 0.6.2
(since 0.6.1 was yanked), did not help either - then I gave up.

Any ideas how to make sequoia build again?

-- 
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: [PATCH] display grafted package

[Ludovic Courtès]

Hi,

zimoun  skribis:

> The attached patch does that only for ’package->recutils’ (show and
> search).  For instance, note the ’replaced’ field for the grafted
> package.  (Obviously, it could be any other word than ’replaced’
> compatible with the recutils record.)

[...]

> name: zstd
> version: 1.4.4
> replaced: 1.4.9

I’d make it “replacement” rather than “replaced”.

> How display such information for ’package -A’?  The (selected) output
> looks like:
>
> zstd  1.4.9   out,lib,static  gnu/packages/compression.scm:1473:2
> zstd  1.4.4   out,lib,static  gnu/packages/compression.scm:1402:2

Ultimately, you’d want ‘fold-packages’ to traverse replacements, but I
don’t think that’s a good idea because it would break the assumption
that ‘fold-packages’ only traverses public packages.

All in all, I’m in favor of either the status quo or the
‘package->recutils’ patch you propose.

Thanks,
Ludo’.

Re: GNOME 40 work should be done on Savannah

[Ludovic Courtès]

Hi Léo,

Léo Le Bouter  skribis:

> I don't feel like people should be barred to contribute to that GNOME
> 40 upgrade because they arent an approved committer. That doesnt feel
> inclusive to me.

I respectfully think you misunderstand the review process.  Review is
about sharing responsibilities and reducing the likelihood of mistakes.

It’s crucial from many different perspectives: security-wise, socially
(the process is transparent, everyone gets a chance to chime in or to
just see what’s cooking), and technically (we all learn and build better
software, and we reduce the risk that users receive broken software on
their next ‘guix pull’).

The process to get commit access is documented and it revolves around
two criteria: the person must have experience with the project and must
know the rules.

That’s not “bureaucracy”: it’s about building a community around mutual
understanding of what each one of us may or may not expect from their
peers.


All that said, it’s of course fine to collaborate on external
repositories, even though I agree with Mark that getting “closer” to
Guix folks may be profitable to everyone.  In the end, work should be
submitted for review, and for a patch set like this, it’ll probably be a
‘wip-’ branch on Savannah.

Thanks,
Ludo’.

Re: Security patching and the branching workflow: a new security-updates branch

[zimoun]

On Sat, 27 Mar 2021 at 15:14, Léo Le Bouter  wrote:

>   but you
> cannot put forward the arguments you've made, they do not work.

Ahah, I am happy to know it.  I hope it is because a “miscommunication»
and not because you do not carefully read or because maybe you only see
through the tiny lens of known security vulnerabilities.  From my
opinion, your point of view to tackle the issue is wrong.  That’s said.

Best regards,
simon

Re: GNOME 40 work should be done on Savannah (was: Re: GNOME 40)

[zimoun]

Hi,

> The thing is, the work of non-committers *must* be reviewed at some
> point, anyway.  Moreover, a committer must take responsibility by
> digitally signing it.  To eliminate either of these steps would put us
> at risk.
>
> There's no guarantee that the work of Guix committers will be reviewed
> by anyone else, because no one else's reputation is on the line.  Some
> of us try to keep an eye on things, but I would not bet on that
> oversight being comprehensive.  I'm certainly not doing it
> comprehensively.

Reviewing does not require commit access.  Examples [1,2] among many
others.  The recent (trivial) addition of Julia packages [3] is
interesting in this regard, IMHO.  It is a chain of trust.  Committer
has the final word.

And to my taste, there is too much non-trivial patches pushed without
going through guix-patches first.  Another story.

1: 
2: 
3: 

>> The people that work on it now are Raghav and me, and Raghav does not
>> have commit access yet, so that's the only way we can work and
>> cooperate now. We don't have a choice.
>
> Sorry, but that's simply false.  You _do_ have a choice.  You can do
> what we've been doing in the Guix community for years: as a committer,
> _you_ can commit the work of non-committers on their behalf.  If not
> you, then any of the other ~64 Guix committers can do so.

[...]

>> I don't feel like people should be barred to contribute to that GNOME
>> 40 upgrade because they arent an approved committer. That doesnt feel
>> inclusive to me.
>
> No one is "barred" from contributing.  Raghav and many others without
> commit access have been successfully contributing to Guix for years.
>
> I understand that it's inconvenient.  Naturally, you would like to
> eliminate that inconvenience.

I miss something.  Is the Git ’remote’ not fitting the need?

Well, for instance, I have currently 4 remotes, some where I fetch, some
where I push.

For example, to avoid to overflow guix-patch when updating Bioconductor
R packages, Ricardo (committer) pushed the work on the Savannah branch
’wip-r’, i.e, I fetched from Savannah, tweaked, pushed to my personal
repo, Ricardo fetched from it, etc. with a simple synchronisation on
#guix or #guix-hpc.

Another example is the recent Outreachy.  Magali (intern) pushed their
work on their own repo, I (non-committer) fetched from it, commented,
etc.  Then once ready, I do not remember who (committer) pushed to the
Savannah branch ’wip-guix-log’ (help with review welcome ;-)).

Another example is the recent Cuirass / new offloading thing.  Mathieu
did some work on a branch in their personal repo, asked me to give a
look, so I fetched, commented, etc. then they pushed to ’master’
Savannah a part of it, still improving other part on their personal
branch, etc.

Well, I should miss something.  In my understanding, Git is designed to
allow collaboration without a central repo.  Is it not what
“distributed” means in DVCS?

If having a central repo—–where a large number of people can write
in––eases the work, why not.  But the key point is to regularly push to
a ’wip-gnome’ branch or ’core-updates’ on Savannah.  Savannah must be
the reference.  For 2 practical reasons: 1) it is more discoverable,
i.e., inclusive, for newcomers (clone the Guix repo Savannah, press ’y’
with Magit, see ’wip-gnome’, contribute!) and 2) it increases the chance
that other Guixers give a look time to time.

Last, I agree with Mark, regularly pushing to Savannah is the guarantee
that the final work is fully respecting the Guix standards.  By doing
so, it is the responsibility of the committer by signing off to ensure
that the standards are respected.

Somehow, it is the plan, right?  And a “miscommunication” about the word
«flexible» and about how to exchange large numbers of patches without
’format-patch+send-email’?

Cheers,
simon

Re: Question about compile packages

[Julien Lepiller]

Not sure about your first question. Maybe create your own fork and regularly 
rebase it? Guix will not be very happy with that I think, but it should work.

To pass --no-substitutes, you can pass it to guix-daemon, or remove the 
authorized keys from /etc/guix/acl. The second option lets you download 
substitutes for fixed output derivations (basically source tarballs), so it's 
easier in case some of those tarballs disappear from their original location.

I think the first option is selectable from the installer, but I might be 
wrong. Otherwise, stop guix-daemon from herd and run it manually with your 
options.

Le 29 mars 2021 15:05:08 GMT-04:00, Charles Direg  a 
écrit :
> Dear,
>
>How can I modify the flags that any program is compiled with within
>guix? That
>is, I can allow in the gnu-build-system to modify it globally so that I
>can
>add the build flags to any package, for example, add the flags -O2
>-march=native -mtune=native so global as I already mentioned, so that
>these
>are added to each package at the time of compilation, this would not be
>within the guix development environment, because what I want is that
>this
>compilation is natively for my pc.
>
>As a second question, how could I set the --no-susbtitutes option when
>installing the guix system from ISO, since I would like all installed
>packages to be compiled natively first?
>
>I really appreciate your kind time and I look forward to your
>responses.
>
>Sincerely,
>~ Abraham Huerta

Re: Deep vs Shallow trace: Removing the tradeoff?

[Ludovic Courtès]

Hi,

ilmu  skribis:

> Early cutoff is a very desirable property that nix does not have (and I 
> assume therefore that neither does guix).

The “intensional model” that Julien mentioned, or what the Nix folks now
refer to as “content-addressed derivations”, have this early cutoff
property.  It’s one of the main motivations for this whole endeavors.

This document shows how Nix folks are working on retrofitting
“content-addressed derivations” into Nix:

  
https://github.com/tweag/rfcs/blob/cas-rfc/rfcs/0062-content-addressed-paths.md

Thanks,
Ludo’.

Re: Needed: tooling to detect references to buggy */stable packages

[Ludovic Courtès]

Hi,

Mark H Weaver  skribis:

> It occurs to me that we will need some tooling to ensure that no
> references to these buggy "*/stable" packages end up in package outputs
> that users actually use.  Otherwise, it is likely that sooner or later,
> a runtime reference to one of these buggy packages will sneak in to our
> systems.

Couldn’t we use #:disallowed-references for this?

Thanks,
Ludo’.

Re: Document our WIP

[Ludovic Courtès]

Hi!

Vincent Legoll  skribis:

> I'd like to reiterate my proposal to document our
> ongoing projects, maybe with a "WIP" page on the
> web site (even if I'm not a web guy, I volunteer
> the maintenance of it).

To me, a good way to make sure work remains “in progress” is to post
regular updates to this list, and then to write blog posts for the web
site whenever an important milestone is reached.

I think a web page is likely to quickly become outdated… unless said
work transitions from “in progress” to “stalled”.  :-)

Ludo’.

Re: I have merged wip-ppc64le-for-master to master

[Ludovic Courtès]

Hi!

Chris Marusich  skribis:

> FYI, I have merged wip-ppc64le-for-master to master and closed bug
> 47182.  I have also deleted the wip-ppc64le-for-master branch.

Yay, congrats!  \o/

I now have access to the OSUOSL POWER9 machine.  I’ve set up offloading
and it seems to work well.

Thanks to all the hackers involved + OSUOSL!

Ludo’.

Buying AArch64 hardware?

[Ludovic Courtès]

Hi Leo,

Leo Famulari  skribis:

> I volunteer to host one or two workstation-type 64-bit ARM machines.
>
> Concretely, this would mean a Honeycomb LX2 or Ampere ALTRA workstation,
> since I don't believe there are any other aarch64 workstations available
> for sale.
>
> https://www.solid-run.com/arm-servers-networking-platforms/honeycomb-workstation/
> https://store.avantek.co.uk/ampere-altra-64bit-arm-workstation.html

Let’s do that!  We can already buy one or two and ship it to your place.
If it works well, we can later expand and ship it to the other people
who volunteers.

We can work out practical details on guix-sysadmin, such as finding who
does the initial payment (currently the process to access money held at
the FSF is through reimbursement, with an invoice).

Thanks,
Ludo’.

Re: A proposal for better quality in maintenance of packages by reducing scope

[Ludovic Courtès]

Hi Léo,

Léo Le Bouter  skribis:

> I would like to propose that we reduce the scope of the maintenance we
> do in GNU Guix and establish a list of packages that we more or less
> commit to maintaining because this is something that we can do and is
> attainable, for example, we could remove desktop environments that we
> can't maintain to good standards realistically and focus our efforts on
> upstreams that don't go against our way of doing things, that are
> cooperative, that provide good build systems we can rely on for our
> purposes, etc.
>
> I propose we also add some requirements before packages can go into
> such a maintained state, like a working and reliable updater/refresher
> with notifications directed to some mailing list when that one finds a
> new release, a reduced amount of downstream patches and a cooperative
> upstream with who we preferably have some point of contact to solve
> issues or gather more insider knowledge about the software if we need,
> a working and reliable CVE linter with proper cpe-name/vendor and
> notifications going to a mailing list we all subscribe to, etc..
> probably lots of other things are relevant but you see the idea.
>
> It should also be possible to filter out packages that are not declared
> to be in this maintained state, for example, in the GNU Guix System
> configuration.

I think most would agree with the general ideas.  What’s more
complicated is the implementation.  What’s “good standards”?  What’s
“realistically”?  How do we tell whether “upstream is cooperative”?
Whether a package is “maintained”?

However, concrete actions we can take is identify shortcomings of
existing tools (I’m glad you reported a bunch of ‘guix refresh’
failures!) and missing tools (a tool that would automatically
refresh/build and push patches to a branch would be great), and work on
them incrementally.

Thanks,
Ludo’.

Re: Secure GNU Guix offloading

[Ludovic Courtès]

Hi!

Léo Le Bouter  skribis:

> I don't want to give more access than what SSH non-root access would
> give, and I think it would be possible to do something helpful in GNU
> Guix offloading so it can work even without the offload machine
> trusting the client's store public signing key.

One possibility would be to give SSH access and nothing more.  That
would allow hackers to run:

  GUIX_DAEMON_SOCKET=ssh://leo.example.org guix build whatever

Users would still be able to retrieve build results from your machine
via ‘guix copy’ or an instance of ‘guix publish’ running on the machine.

HTH!

Ludo’.

Re: Let's include powerpc64le-linux in the next release

[Ludovic Courtès]

Hi Tobias,

Tobias Geerinckx-Rice  skribis:

> Ludovic Courtès 写道：
>> Like Efraim wrote, the person who makes the release (I’m afraid
>> it’ll be
>> me) needs to be able to offload to a “trustworthy” machine for that
>> platform.
>
> Define trustworthy?

I put quotes because it’s informally defined, at best, and it’s not all
or nothing.

To me one criterion would be that only known project contributors are
admins, physical access to the machine is limited to them and/or a
well-identified group of people, and that no build results flow from the
outside into this machine.

I realize we’re not there yet, but it’s OK: we’re just getting started
with this architecture.  Plus, I’m really grateful to OSUOSL for giving
us access to this machine!

Ludo’.

Re: GNOME 40 work should be done on Savannah (was: Re: GNOME 40)

[Christopher Baines]

Mark H Weaver  writes:

> Christopher Baines  writes:
>
>> Mark H Weaver  writes:
>>> How is it more flexible than a "wip-*" branch on Savannah?
>>
>> I wouldn't use quite the same words as Léo, but from my perspective,
>> controlling access to particular branches (master, staging,
>> core-updates, ...) on Savannah is a good thing, as it reduces risk.
>
> I don't see much risk here.  You're talking about a 'wip' branch that
> almost no one will be using anyway.  We already trust all Guix
> committers with our master branch, which directly and immediately
> affects any Guix user who updates their system at the right time.

No, I was talking about particular branches, master, staging,
core-updates, ... and controlling access to those more sensitive
branches.

I mention this as context for discussing acesss control to wip-*
branches, because currently as I understand it, if someone wants access
to work on a specific wip- branch, the only way to do that is grant
access to all branches in all repositories in the Guix Savannah project.

...

> I'd strongly prefer for this work to be done on Savannah.  If this were
> a fringe branch of marginal interest, it might make sense to have it
> elsewhere, but this is core Guix desktop work that's likely to be of
> interest to a large segment (plausibly a majority) of our community.
> IMO, it belongs in our official git repository.

I'm not commenting on this Gnome 40 related work, as I'm not really
involved, but I do think there's some potential for improvement
regarding how wip- branches are handled.

Having them on Savannah is great as you say, but that makes these
branches more difficult to use for people who don't have commit access.


signature.asc
Description: PGP signature

Re: GNOME 40 work should be done on Savannah (was: Re: GNOME 40)

[Mark H Weaver]

Hi Léo,

Léo Le Bouter  writes:
> The people that work on it now are Raghav and me, and Raghav does not
> have commit access yet, so that's the only way we can work and
> cooperate now. We don't have a choice.

Sorry, but that's simply false.  You _do_ have a choice.  You can do
what we've been doing in the Guix community for years: as a committer,
_you_ can commit the work of non-committers on their behalf.  If not
you, then any of the other ~64 Guix committers can do so.

Needless to say, before committing, you must review the proposed
patches, for the sake of your reputation.  The fact that you must do
this is a *feature*, not a bug.

> I don't feel like people should be barred to contribute to that GNOME
> 40 upgrade because they arent an approved committer. That doesnt feel
> inclusive to me.

No one is "barred" from contributing.  Raghav and many others without
commit access have been successfully contributing to Guix for years.

I understand that it's inconvenient.  Naturally, you would like to
eliminate that inconvenience.

The thing is, the work of non-committers *must* be reviewed at some
point, anyway.  Moreover, a committer must take responsibility by
digitally signing it.  To eliminate either of these steps would put us
at risk.

There's no guarantee that the work of Guix committers will be reviewed
by anyone else, because no one else's reputation is on the line.  Some
of us try to keep an eye on things, but I would not bet on that
oversight being comprehensive.  I'm certainly not doing it
comprehensively.

With this in mind, I think that we *should* have a high standard for
committers.  The security of our systems, as well as Guix's reputation
as a project, depends upon the good judgment of _every_ Guix committer.

Observe what can happen with projects that are too lax:

  
https://arstechnica.com/gadgets/2021/03/buffer-overruns-license-violations-and-bad-code-freebsd-13s-close-call/

> Why would it not get adequate oversight? It's just an easier way to
> collaborate on patches, but the patchset would be sent over to guix-
> patches before getting merged to master or else.

Upgrading GNOME is not trivial.  It will be a large patch set.  A large
patch set presented to guix-patches when the branch is ready to merge is
far less likely to get careful review than if the review is done a few
commits at a time.  That's because, at any given time, it's easier to
find Guix developers with a few minutes available to carefully review a
small handful of commits, than to find developers prepared to review a
non-trivial branch merge.  If they're reviewed at all, reviews of larger
code drops are more likely to be superficial.

* * *

In summary: it seems to me that working in an external repository with a
larger set of committers would not actually save time, because it would
merely postpone the required review work until the end of the process
when the branch is ready to be merged into Savannah.  Moreover, it would
likely reduce the quality of that review work.

Does that make sense?

Regards,
  Mark