Re: Deep vs Shallow trace: Removing the tradeoff?

[Julien Lepiller]

Le 27 mars 2021 12:56:08 GMT-04:00, ilmu  a écrit :
>Hi,
>
>I had this idea while reading about authenticated datastructures, I
>would be very thankful if you could tell me why it doesn't work.
>
>Bear in mind the following disclaimer as you read this:
>
>My experience with these things is mostly theoretical, I have never
>used Bazel and although I was a user of Nix and am now moving to Guix I
>have not contributed to nixpkgs and only written simple expressions.
>
>Without further ado.
>
>
>The premise I am assuming is the framework introduced in Build Systems
>a la Carte. They talk about Bazel and Nix as representatives of two
>different dependency tracing strategies:
>
>- Shallow tracing :: You hash immediate dependencies.
>- Deep tracing :: You hash the whole transitive closure.
>
>Now the tradeoff is basically the following:
>
>- In Nix when you put a comment in curl you need to rebuild every
>single package in nixpkgs because they more or less all depend on curl
>in one way or another and therefore the curl source is in the
>transitive closure for almost every package.
>- In Bazel when you put a comment in curl then the direct dependents
>need to be rebuilt but if they are the same as before after being
>rebuilt then the propagation is killed and nothing else needs to
>change.
>
>However, in Bazel you will need to traverse the whole dependency tree
>all of the time to verify that everything is as it should be.

What I understand from the above is that their is not so much of a difference 
between Bazel and Nix/Guix. The "shallow" tracing *is* a deep tracing. It just 
records something different.

In Nix/Guix, we recursively compute the hash of packages from their inputs, 
receipe and other stuff. This hash is unrelated to the checksum of the result 
and changes as soon as one element changes. It essentially records the whole 
source dependency graph.

From what you write, Bazel records the checksum of inputs and receipe, so it 
essentially encodes the whole binary dependency graph. It's not "shallow", as a 
real change down the graph can have repercussions higher up.

Actually Eelco described something similar for Nix in his thesis, the 
intensionnal model. As you noticed, different receipes or inputs can lead to 
the same output (bitwise). The idea is to declare the expected checksum of the 
result, and use this to compute the store item name. As long as you apply 
changes that do not affect the output, different receipes (updates) can be used 
to generate the same item. So if you update a package, it might affect its 
output checksum, so you need to change it. Dependents might not be affected, in 
which case you can block update propagation there.

The question being how to monitor that packages need to be rebuilt or have the 
same hash? 

>
>
>Now the idea I have is very simple:
>
>We use recursive zero knowledge proofs with shallow traces, the rzkp
>caches the traversal and provides the same guarantee as the deep traces
>do (transitive closure is verified to be as it should be). Now if
>someone puts a comment in curl there is a small amount of packages that
>need to be rebuilt and then we redo only the proofs all the way up.
>This way we save ourselves a potentially massive amount of compilation.

I don't really understand why it must be zero-knowledge. My understanding is 
that to have a zero-knowledge proof, you first need a proof of the claim, then 
find a clever way to proove you have the proof, without disclosing it. Not sure 
it's helpful here. If you have a proof that you don't need to rebuild, why not 
share that proof to everyone in the first place?

>
>As I said before I do not have much experience with the real
>implementations of these ideas so I am sure this is not as simple as it
>is in my head. However the distri experimental operating system (which
>implements a similar model to guix and nixos) does not put the hash in
>the store path but rather keeps a small metadata file for each path and
>then has a natural number suffix for the path of concurrent versions of
>the same package. This gives a better UX imho and is probably also
>easier to extend with more appropriate authenticated datastructures as
>they are discovered.
>
>
>I hope I am not a raving madman and that this actually makes at least a
>slight amount of sense. Very much looking forward to takedowns :)
>
>
>Kind regards,
>- Ilmu

Re: [PATCHES] ImageMagick security updates without grafting

[Mark H Weaver]

Hi Maxime,

Maxime Devos  writes:
> This approach (& patches) look good to me.

Thanks for looking.

> What does ‘guix refresh --list-dependent imagemagick@6.9.11-48’
> output now?

When I last checked, it reported on the order of 2400 dependent package
rebuilds.

> If it there are many dependent packages, could some
> of them use imagemagick/stable, dblatex/stable or gtk-doc/stable
> as well?

Yes, that's exactly the purpose of this patch set.  Although at present,
the only user of 'imagemagick/stable' is 'dblatex/stable', and the only
user of 'dblatex/stable' is 'gtk-doc/stable'.

> Maybe add a comment to imagemagick/stable on why there is a 
> /stable variant, for future reference.

Good idea.  I added comments similar to what you had suggested.

Thanks for the review!  I went ahead and pushed a revised version of
these commits to 'master', starting with commit
7c2b840d6c586f80fe22a862ce4e362c997559a5, but if anyone has further
input on this approach, it's still not too late to change things.

   Thanks,
 Mark

Re: Document our WIP

[Luis Felipe]

On Saturday, March 27, 2021 6:12 PM, Vincent Legoll  
wrote:

> The first bits are in, look:
> https://libreplanet.org/wiki/Group:Guix/WorkInProgress
> Add / enhance to tell what is running, where to find good recipes...
>
> To create pages you have to score a few edits (I went
> typo-hunting for a bit). I think about 5 should do.

Oh, I didn't know that. I had edited the wiki before, so I didn't find any 
restrictions.

Re: Document our WIP

[Vincent Legoll]

On Sat, Mar 27, 2021 at 7:11 PM Luis Felipe
 wrote:
> I see a newly created page for Works in Progress, though

Yep, my announce and your email crossed past each other...

-- 
Vincent Legoll

Re: Document our WIP

[Vincent Legoll]

The first bits are in, look:
https://libreplanet.org/wiki/Group:Guix/WorkInProgress
Add / enhance to tell what is running, where to find good recipes...

To create pages you have to score a few edits (I went
typo-hunting for a bit). I think about 5 should do.

-- 
Vincent Legoll

Re: Document our WIP

[Luis Felipe]

On Saturday, March 27, 2021 5:02 PM, Vincent Legoll  
wrote:


> BTW, there's no rush to change this, as Leo, I can't create a page on
> libreplanet's
> wiki

I see a newly created page for Works in Progress, though:

https://libreplanet.org/wiki/Group:Guix/WorkInProgress

And creating subpages seems to work too. For example, if I visit the following 
URL, there is the option to create the page and clicking on it takes me to the 
page creation view:

https://libreplanet.org/wiki?title=Group:Guix/WorkInProgress/XYZ

Re: gnu: imagemagick/fixed: Redirect old sonames to new sonames.

[Leo Famulari]

On Sat, Mar 27, 2021 at 05:36:48AM -0400, Mark H Weaver wrote:
> > gtk+@3 -> at-spi2-atk -> at-spi2-core -> gtk-doc -> dblatex -> imagemagick
> 
> It occurs to me that we could add "stable" variants of the
> 'imagemagick', 'dblatex', and 'gtk-doc' packages.  The stable variants
> would be used as native-inputs by our core packages, thus allowing the
> normal variants of these packages to be updated freely on our 'master'
> branch.  In particular, we could keep our 'imagemagick' package
> up-to-date without grafting and without having to backport fixes.
> 
> What do you think?

I think it's the right approach, and I saw your patch that implements
it. I'll reply in more detail there.

Deep vs Shallow trace: Removing the tradeoff?

[ilmu]

Hi,

I had this idea while reading about authenticated datastructures, I would be 
very thankful if you could tell me why it doesn't work.

Bear in mind the following disclaimer as you read this:

My experience with these things is mostly theoretical, I have never used Bazel 
and although I was a user of Nix and am now moving to Guix I have not 
contributed to nixpkgs and only written simple expressions.

Without further ado.


The premise I am assuming is the framework introduced in Build Systems a la 
Carte. They talk about Bazel and Nix as representatives of two different 
dependency tracing strategies:

- Shallow tracing :: You hash immediate dependencies.
- Deep tracing :: You hash the whole transitive closure.

Now the tradeoff is basically the following:

- In Nix when you put a comment in curl you need to rebuild every single 
package in nixpkgs because they more or less all depend on curl in one way or 
another and therefore the curl source is in the transitive closure for almost 
every package.
- In Bazel when you put a comment in curl then the direct dependents need to be 
rebuilt but if they are the same as before after being rebuilt then the 
propagation is killed and nothing else needs to change.

However, in Bazel you will need to traverse the whole dependency tree all of 
the time to verify that everything is as it should be.


Now the idea I have is very simple:

We use recursive zero knowledge proofs with shallow traces, the rzkp caches the 
traversal and provides the same guarantee as the deep traces do (transitive 
closure is verified to be as it should be). Now if someone puts a comment in 
curl there is a small amount of packages that need to be rebuilt and then we 
redo only the proofs all the way up. This way we save ourselves a potentially 
massive amount of compilation.

As I said before I do not have much experience with the real implementations of 
these ideas so I am sure this is not as simple as it is in my head. However the 
distri experimental operating system (which implements a similar model to guix 
and nixos) does not put the hash in the store path but rather keeps a small 
metadata file for each path and then has a natural number suffix for the path 
of concurrent versions of the same package. This gives a better UX imho and is 
probably also easier to extend with more appropriate authenticated 
datastructures as they are discovered.


I hope I am not a raving madman and that this actually makes at least a slight 
amount of sense. Very much looking forward to takedowns :)


Kind regards,
- Ilmu

Re: Document our WIP

[Vincent Legoll]

On Sat, Mar 27, 2021 at 5:50 PM Léo Le Bouter  wrote:
> After looking more closely at the help page, I think it could be
> acceptable to have it there too. But is the wiki an help resource?

Even if it may not be today, we are proposing something that could
turn it into a helpful resource. So OK with me.

-- 
Vincent Legoll

Re: Document our WIP

[Luis Felipe]

---
Luis Felipe López Acevedo
https://luis-felipe.gitlab.io/

Sent with ProtonMail Secure Email.

‐‐‐ Original Message ‐‐‐
On Saturday, March 27, 2021 4:50 PM, Léo Le Bouter  wrote:

> On Sat, 2021-03-27 at 16:42 +, Luis Felipe wrote:
>
> > I'm fine with that too (for now). I can send that patch.
> > The reason I didn't suggest that, though, is that the primary menu

[...]

> After looking more closely at the help page, I think it could be
> acceptable to have it there too. But is the wiki an help resource?

Well, right now most information seems more related to planning, but, with some 
sectioning, it could be used for additional help documentation. There is at 
least one page in that category 
https://libreplanet.org/wiki/Group:Guix/KeyboardLayout.

Re: Document our WIP

[Vincent Legoll]

On Sat, Mar 27, 2021 at 5:42 PM Luis Felipe
 wrote:
> The reason I didn't suggest that, though, is that the primary menu has already
> grown too big in my opinion. And, with the current design, the visibility of 
> the
> primary menu changes depending on screen width: The primary menu is hidden
> behind a primary menu button for screens narrower than 920 px (which may be
> too soon already).

I can hear that.

OK, no more top bar pollution, Let's get it in "Help", but we could
make this one
a submenu with direct links (the same list as the sections from the help page).

There's something strange, the default link to the doc is for the /en/ language
when I selected the french web site, this could go directly to the
french version.

BTW, there's no rush to change this, as Leo, I can't create a page on
libreplanet's
wiki

-- 
Vincent Legoll

Re: Document our WIP

[Léo Le Bouter]

On Sat, 2021-03-27 at 16:42 +, Luis Felipe wrote:
> I'm fine with that too (for now). I can send that patch.
> 
> The reason I didn't suggest that, though, is that the primary menu
> has already grown too big in my opinion. And, with the current
> design, the visibility of the primary menu changes depending on
> screen width: The primary menu is hidden behind a primary menu button
> for screens narrower than 920 px (which may be too soon already).
> 
> So, adding more primary items to that menu means the menu has to be
> hidden for even wider screens than 920. An that's from the English
> point of view only; I haven't checked how the header bar is behaving
> for other locales...
> 
> That said, I have a pending proposal to redesign the header bar to
> behave similar to headers in GNOME desktop applications (it could be
> simpler and follow already used conventions that people may be
> familiar with).

After looking more closely at the help page, I think it could be
acceptable to have it there too. But is the wiki an help resource?


signature.asc
Description: This is a digitally signed message part

Re: Document our WIP

[Luis Felipe]

On Saturday, March 27, 2021 4:04 PM, Léo Le Bouter  wrote:

> On Sat, 2021-03-27 at 15:54 +, Luis Felipe wrote:
>
> > Or that, yes. I can send a patch to add a Wiki entry to the Help page
> > instead of adding a "Wiki" item to the "About" menu.
>
> I think we should be looking forward to including it in the primary
> menu and not hidden in some submenu.

I'm fine with that too (for now). I can send that patch.

The reason I didn't suggest that, though, is that the primary menu has already 
grown too big in my opinion. And, with the current design, the visibility of 
the primary menu changes depending on screen width: The primary menu is hidden 
behind a primary menu button for screens narrower than 920 px (which may be too 
soon already).

So, adding more primary items to that menu means the menu has to be hidden for 
even wider screens than 920. An that's from the English point of view only; I 
haven't checked how the header bar is behaving for other locales...

That said, I have a pending proposal to redesign the header bar to behave 
similar to headers in GNOME desktop applications (it could be simpler and 
follow already used conventions that people may be familiar with).

Re: Cuirass not processing core-updates (or weirdly)

[Vincent Legoll]

Maybe we can have both: a core-updates:'core with
a sufficient priority so that it is built often enough,
and then a core-updates:'all with the lowest priority
(batch) so that it does not steal any processing power
from the other more important stuff...

-- 
Vincent Legoll

Re: Document our WIP

[Vincent Legoll]

On Sat, Mar 27, 2021 at 4:54 PM Luis Felipe
 wrote:
> Or that, yes. I can send a patch to add a Wiki entry to the Help page instead
> of adding a "Wiki" item to the "About" menu.

Or even better a "Wiki" title bar entry of its own, like we have one
for "Blog"...

-- 
Vincent Legoll

Re: Document our WIP

[Léo Le Bouter]

On Sat, 2021-03-27 at 15:54 +, Luis Felipe wrote:
> Or that, yes. I can send a patch to add a Wiki entry to the Help page
> instead of adding a "Wiki" item to the "About" menu.

I think we should be looking forward to including it in the primary
menu and not hidden in some submenu.


signature.asc
Description: This is a digitally signed message part

Re: Document our WIP

[Léo Le Bouter]

On Sat, 2021-03-27 at 16:41 +0100, Vincent Legoll wrote:
> I don't know if libreplanet's wiki meets Léo's requirements,
> but this is probably OK from a PoV of spam management.

I could create an FSF account with automated approval by email.

It seems I cannot create new pages, however it seems we can set page
protection rules arbitrarily:

Page protection
EditAllow all users (infinite)
MoveAllow all users (infinite)

This is good, however creating new pages under WorkInProgress for
information on specific WIP projects or else looks like it would be
required for me.

Does someone know if we can do that on Libreplanet? Else I am afraid
this an arbitrary limitation harming our goal here.


signature.asc
Description: This is a digitally signed message part

Re: Document our WIP

[Luis Felipe]

On Saturday, March 27, 2021 3:41 PM, Vincent Legoll  
wrote:

> On Sat, Mar 27, 2021 at 3:44 PM Ricardo Wurmus rek...@elephly.net wrote:
>
> > > I'd like to reiterate my proposal to document our
> > > ongoing projects, maybe with a "WIP" page on the
> > > web site (even if I'm not a web guy, I volunteer
> > > the maintenance of it).
> >
> > There’s a wiki that could be used for this purpose:
> > https://libreplanet.org/wiki/Group:Guix
>
> I'm OK with hosting that page there, but I still think
> discoverability from a link on:
> https://guix.gnu.org/en/help/
> will help...

Or that, yes. I can send a patch to add a Wiki entry to the Help page instead 
of adding a "Wiki" item to the "About" menu.

Re: Document our WIP

[Léo Le Bouter]

On Sat, 2021-03-27 at 15:44 +, Luis Felipe wrote:
> What do you think about adding a "Wiki" item to the "About" menu of
> the website linking to that Guix group on LibrePlanet? At least as a
> quick solution to try out.

I think this would be the best thing to do, however I don't know if I
can do that myself, nor that I would have authority to make such
change.


signature.asc
Description: This is a digitally signed message part

Re: Document our WIP

[Vincent Legoll]

Looks like we have a plan.

Now for the hard part, how do we name it ?

"Guix/WIP" or "Guix/WorkInProgress"
or
"Guix/Hacking", "Guix/CoolStuff", "Guix/BleedingEdge", "Guix/NewShiny"

Some of those may be half-jokes, I'd personally go with WorkInProgress.

-- 
Vincent Legoll

Re: imagemagick@6.9.11-48 to graft or not to graft with 6.9.12-2

[Andreas Enge]

Hello Mark,

Am Tue, Mar 23, 2021 at 07:42:48PM -0400 schrieb Mark H Weaver:
> To my mind this suggests a bug, or at least suboptimal behavior, in
> "guix package".  I don't think it's appropriate to set grafting policy
> to work around it.
> 
> How about changing "guix package -A" and "guix package -s" to display
> information about the package's replacement, if it has one?

I am not familiar enough with grafts and replacements to judge whether
this is a reasonable approach or whether it just adds more duct tape on
top of duct tape... Note that everything would have to be consistent,
from "guix package" to "guix build". And the "--no-grafts" also behaves
strangely, where it is not even clear what the optimal behaviour should be.
I still think it is a situation we should try to avoid as much as possible,
but maybe it is not worth to spend too much time fixing it.

Andreas

Re: Document our WIP

[Luis Felipe]

On Saturday, March 27, 2021 3:34 PM, Léo Le Bouter  wrote:

> On Sat, 2021-03-27 at 11:32 -0400, Joshua Branson wrote:
>
> > Good point. Perhaps we should link to this wiki from the guix
> > website?
>
> I think that we should do that for this wiki resource to be really
> useful. Widespread knowledge of the location is a must.

What do you think about adding a "Wiki" item to the "About" menu of the website 
linking to that Guix group on LibrePlanet? At least as a quick solution to try 
out.

Re: Document our WIP

[Vincent Legoll]

On Sat, Mar 27, 2021 at 3:44 PM Ricardo Wurmus  wrote:
> > I'd like to reiterate my proposal to document our
> > ongoing projects, maybe with a "WIP" page on the
> > web site (even if I'm not a web guy, I volunteer
> > the maintenance of it).
>
> There’s a wiki that could be used for this purpose:
> https://libreplanet.org/wiki/Group:Guix

I'm OK with hosting that page there, but I still think
discoverability from a link on:
https://guix.gnu.org/en/help/
will help...

I don't know if libreplanet's wiki meets Léo's requirements,
but this is probably OK from a PoV of spam management.

-- 
Vincent Legoll

Re: Document our WIP

[Léo Le Bouter]

On Sat, 2021-03-27 at 11:32 -0400, Joshua Branson wrote:
> Good point.  Perhaps we should link to this wiki from the guix
> website?

I think that we should do that for this wiki resource to be really
useful. Widespread knowledge of the location is a must.


signature.asc
Description: This is a digitally signed message part

Re: Document our WIP

[Joshua Branson]

Ricardo Wurmus  writes:

> Vincent Legoll  writes:
>
>> I'd like to reiterate my proposal to document our
>> ongoing projects, maybe with a "WIP" page on the
>> web site (even if I'm not a web guy, I volunteer
>> the maintenance of it).
>
> There’s a wiki that could be used for this purpose:
> https://libreplanet.org/wiki/Group:Guix

Good point.  Perhaps we should link to this wiki from the guix website?

--
Joshua Branson (joshuaBPMan in #guix)
Sent from Emacs and Gnus
  https://gnucode.me
  https://video.hardlimit.com/accounts/joshua_branson/video-channels
  https://propernaming.org
  "You can have whatever you want, as long as you help
enough other people get what they want." - Zig Ziglar

Re: Packaging

[Maxime Devos]

Hi,

On Sat, 2021-03-27 at 07:37 +, mecqor labi wrote:
> Please package shadowsocks-rust for Guix!

Is there any reason the "shadowsocks" package is not
usable for you?  Also, feel free to package it yourself
(See the manual for how) and note the folks at guix-devel@gnu.org
don't (often) take packaging requests.  (But see
.)

> Many users have a big problem in access to free internet, and have to go 
> through censorship wall.
> Please!!!

Would Tor suit your purposes? 
It's packaged in guix (the tor browser itself isn't, however).

Greetings,
Maxime


signature.asc
Description: This is a digitally signed message part

Re: Document our WIP

[Ricardo Wurmus]

Vincent Legoll  writes:

> I'd like to reiterate my proposal to document our
> ongoing projects, maybe with a "WIP" page on the
> web site (even if I'm not a web guy, I volunteer
> the maintenance of it).

There’s a wiki that could be used for this purpose:
https://libreplanet.org/wiki/Group:Guix

-- 
Ricardo

Re: [PATCHES] ImageMagick security updates without grafting

[Maxime Devos]

On Sat, 2021-03-27 at 09:09 -0400, Mark H Weaver wrote:
> Hello Guix,
> 
> Here's a proposed patch set that will henceforth enable us to freely
> update ImageMagick (and dblatex, and gtk-doc) on our 'master' branch
> without grafts.  This is done by adding variables 'imagemagick/stable',
> 'dblatex/stable', and 'gtk-doc/stable', which are then used as
> 'native-inputs' in selected packages.
> 
> The idea here is that the overwhelming majority of dependencies on
> 'imagemagick' are via references to 'gtk-doc' in the 'native-inputs' of
> GNOME libraries.  The risk of running buggy imagemagick code within Guix
> build containers is presumably quite limited, and in any case, grafting
> is no better in this regard.

This approach (& patches) look good to me.

> [...]
> Are there any comments or objections to this approach?

What does ‘guix refresh --list-dependent imagemagick@6.9.11-48’
output now?  If it there are many dependent packages, could some
of them use imagemagick/stable, dblatex/stable or gtk-doc/stable
as well?

Maybe add a comment to imagemagick/stable on why there is a 
/stable variant, for future reference.  Perhaps something along
the lines of:

;; This is a variant of the 'imagemagick' package that is not
;; updated often.  Where possible, use this variant instead of
;; the updated 'imagemagick' package to avoid large rebuilds
;; each time 'imagemagick' is updated (e.g. with security fixes),
;; unless this causes security issues.
;;
;; Normally the grafts mechanism would be used instead, but
;; imagemagick is a complicated package to graft.  See
;; .

>   Mark
> 
> Note: I haven't yet fully tested these commits.

Note: I haven't tested your patches.

Greetings,
Maxime.


signature.asc
Description: This is a digitally signed message part

Re: Cuirass not processing core-updates (or weirdly)

[Léo Le Bouter]

On Sat, 2021-03-27 at 04:24 +0100, Léo Le Bouter wrote:
> Hello!
> 
> If you look at https://ci.guix.gnu.org/eval/13652 you can see that
> the
> evaluation of the derivation seems completed but there's no pending
> builds.
> 
> What is happening here?
> 
> Thank you

I think it may be because core-updates is setup to only build 'core and
not 'all.


signature.asc
Description: This is a digitally signed message part

Re: Document our WIP

[Léo Le Bouter]

On Sat, 2021-03-27 at 11:07 +0100, Vincent Legoll wrote:
> Hello,
> 
> I'd like to reiterate my proposal to document our
> ongoing projects, maybe with a "WIP" page on the
> web site (even if I'm not a web guy, I volunteer
> the maintenance of it).
> 
> * CI-built pinebook-pro images [1]
> * other ARM boards
> * ppc64 & ppc
> * Hurd VM
> * Full source bootstrap
> 
> And probably other things I missed.
> 
> This should complement the ML archives, IRC logs or
> blog, with pointers to (hopefully) more current infos
> on those subjects.
> 
> WDYT ?
> 
> [1] I only (accidentally) discovered today that those
> exist
> 

That would be really awesome, I suggest that we introduce such a
mechanism in a way that ANYONE can edit and that we moderate or filter
spam after the fact, not *before*. As gatekeeping in this way will make
the WIP documents edits grind to a halt and not be a viable solution.

We could use a wiki for this, but inspired by Wikipedia's model where
one does not need to register, or that registration is approved with an
easy process where many people have power to approve registrations so
registrations are never stuck waiting for approval. After such
registrations, users must be able to make unlimited edits to the wiki
as they wish. If there's problematic edits, we can moderate and filter
spam after the fact.

These are to me requirements for such an initiative to succeed.


signature.asc
Description: This is a digitally signed message part

Re: Security patching and the branching workflow: a new security-updates branch

[Léo Le Bouter]

On Sat, 2021-03-27 at 14:56 +0100, zimoun wrote:
> Oh, I am a big boy and I can think whatever I want! :-)
> 
> Kidding aside.

...

> 
> First, what does it mean «risk»?  How do you evaluate it?  Is it a
> relative evaluation or an absolute one?

Most if not all users do not want their machines to be compromised or
any side-effects of that.

> Second, I am not arguing that security is not important.  I am saying
> that security is important, as important as everything else that is
> also
> important.  What does it mean «important»?  How do you evaluate
> it?  Is it a
> relative evaluation or an absolute one?

Having security-updates branch or any other mechanism to ship security
updates promptly does not mean that the rest is not important.

> Third, I am aligned with Leo’s words [1].  And probably with yours
> too. :-) To me, a better security is not implied by special
> treatments for security fixes but instead a better treatment for the
> updates in general.

Security updates *need* special treatment. We already specially treat
them with grafts because it's an absolute necessity. We already have
private disclosure mailing lists in GNU Guix because security updates
need special treatment.

> 
> You are proposing a new branch and Chris and I are saying that this
> branch already exists and is staging.  The real question is to know
> how
> staging currently behaves: how many time between 2 merges?  how many
> time to rebuild?  how many packages are rebuilt between 2
> merges?  etc.
> Is it enough?  If not, what could be done to improve?  etc.

The question whether this is solved by a branch or by making pushing to
master directly big rebuilds more viable, that I do not know, but you
cannot put forward the arguments you've made, they do not work.

Léo


signature.asc
Description: This is a digitally signed message part

Re: Security patching and the branching workflow: a new security-updates branch

[zimoun]

On Sat, 27 Mar 2021 at 13:42, Léo Le Bouter  wrote:
> On Sat, 2021-03-27 at 13:29 +0100, zimoun wrote:
>> And as I said elsewhere, “to me, security is important. But it's
>> no less important than everything *else* that is also important!“, so
>> personally I am not convinced that security updates deserve a special
>> treatment compared to a regular update.  That’s my opinion. :-)
>
> You can't think this, security updates have prioritized channel of
> distribution in every other GNU/Linux distribution, we in GNU Guix
> created grafts for it, it's not possible to not ship security updates
> promptly, it puts all users at risk.

Oh, I am a big boy and I can think whatever I want! :-)

Kidding aside.

First, what does it mean «risk»?  How do you evaluate it?  Is it a
relative evaluation or an absolute one?

Second, I am not arguing that security is not important.  I am saying
that security is important, as important as everything else that is also
important.  What does it mean «important»?  How do you evaluate it?  Is it a
relative evaluation or an absolute one?

Third, I am aligned with Leo’s words [1].  And probably with yours
too. :-) To me, a better security is not implied by special
treatments for security fixes but instead a better treatment for the
updates in general.

You are proposing a new branch and Chris and I are saying that this
branch already exists and is staging.  The real question is to know how
staging currently behaves: how many time between 2 merges?  how many
time to rebuild?  how many packages are rebuilt between 2 merges?  etc.
Is it enough?  If not, what could be done to improve?  etc.


1: 

Cheers,
simon

[PATCHES] ImageMagick security updates without grafting

[Mark H Weaver]

Hello Guix,

Here's a proposed patch set that will henceforth enable us to freely
update ImageMagick (and dblatex, and gtk-doc) on our 'master' branch
without grafts.  This is done by adding variables 'imagemagick/stable',
'dblatex/stable', and 'gtk-doc/stable', which are then used as
'native-inputs' in selected packages.

The idea here is that the overwhelming majority of dependencies on
'imagemagick' are via references to 'gtk-doc' in the 'native-inputs' of
GNOME libraries.  The risk of running buggy imagemagick code within Guix
build containers is presumably quite limited, and in any case, grafting
is no better in this regard.

The last 3 commits of this series apply more bug fixes beyond what we
currently have in 'master', including for CVE-2020-27829, as well as a
few other recent upstream commits that look to me potentially security
relevant.

Are there any comments or objections to this approach?

  Mark

Note: I haven't yet fully tested these commits.


>From eaecf83224fdae115a533d03b6fe949794835d43 Mon Sep 17 00:00:00 2001
From: Mark H Weaver 
Date: Sat, 27 Mar 2021 07:07:32 -0400
Subject: [PATCH 1/8] gnu: imagemagick: Remove graft.

Note that this commit does *not* integrate the fixes that were previously
applied via the graft.  This commit simply discards those fixes.  We will
address those security flaws, without grafting, in subsequent commits.

* gnu/packages/imagemagick.scm (imagemagick)[replacement]: Remove field.
(imagemagick/fixed): Remove variable.
---
 gnu/packages/imagemagick.scm | 40 
 1 file changed, 40 deletions(-)

diff --git a/gnu/packages/imagemagick.scm b/gnu/packages/imagemagick.scm
index a3562f2e13..cc5f1de4bf 100644
--- a/gnu/packages/imagemagick.scm
+++ b/gnu/packages/imagemagick.scm
@@ -51,7 +51,6 @@
 ;; maintained. Don't update to 7 until we've made sure that the ImageMagick
 ;; users are ready for the 7-series API.
 (version "6.9.11-48")
-(replacement imagemagick/fixed)
 (source (origin
  (method url-fetch)
  (uri (string-append "mirror://imagemagick/ImageMagick-"
@@ -128,45 +127,6 @@ transform images, adjust image colors, apply various special effects, or draw
 text, lines, polygons, ellipses and Bézier curves.")
 (license (license:fsf-free "http://www.imagemagick.org/script/license.php;
 
-(define-public imagemagick/fixed
-  (package
-(inherit imagemagick)
-(name "imagemagick")
-;; 'g' for 'guix', appended character to retain version length so grafting
-;; works properly.
-(version "6.9.12-2g")
-(source (origin
-  (method url-fetch)
-  (uri (string-append "mirror://imagemagick/ImageMagick-"
-  ;; Hardcode the version here since we had to
-  ;; change it above.
-  "6.9.12-2.tar.xz"))
-  (sha256
-   (base32
-"17da5zihz58qm41y61sbvw626m5xfwr2nzszlikrvxyq1j1q7asa"
-(arguments
- (substitute-keyword-arguments (package-arguments imagemagick)
-   ((#:phases phases)
-`(modify-phases ,phases
-   (add-after 'install 'fix-compat-cheat-rename-so
- (lambda* (#:key outputs #:allow-other-keys)
-   (with-directory-excursion
-   (string-append (assoc-ref outputs "out")
-  "/lib")
- (symlink "libMagick++-6.Q16.so.9.0.0"
-  "libMagick++-6.Q16.so.8.0.0")
- (symlink "libMagick++-6.Q16.so.9"
-  "libMagick++-6.Q16.so.8")
- (symlink "libMagickCore-6.Q16.so.7.0.0"
-  "libMagickCore-6.Q16.so.6.0.0")
- (symlink "libMagickCore-6.Q16.so.7"
-  "libMagickCore-6.Q16.so.6")
- (symlink "libMagickWand-6.Q16.so.7.0.0"
-  "libMagickWand-6.Q16.so.6.0.0")
- (symlink "libMagickWand-6.Q16.so.7"
-  "libMagickWand-6.Q16.so.6"))
-   #t
-
 (define-public perl-image-magick
   (package
 (name "perl-image-magick")
-- 
2.31.0

>From 370089473506c800cf3480f67a00860400fbed18 Mon Sep 17 00:00:00 2001
From: Mark H Weaver 
Date: Sat, 27 Mar 2021 07:16:23 -0400
Subject: [PATCH 2/8] gnu: imagemagick: Add 'imagemagick/stable' variant.

* gnu/packages/imagemagick.scm (imagemagick/stable): New variable.
(imagemagick): This is now an alias to 'imagemagick/stable'.
---
 gnu/packages/imagemagick.scm | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/imagemagick.scm b/gnu/packages/imagemagick.scm
index cc5f1de4bf..6d4649fbac 100644
--- a/gnu/packages/imagemagick.scm
+++ b/gnu/packages/imagemagick.scm
@@ -44,7 +44,7 @@
   #:use-module (gnu packages xml)
   #:use-module (gnu packages xorg))
 
-(define-public imagemagick
+(define-public imagemagick/stable
   (package
   

Re: Security patching and the branching workflow: a new security-updates branch

[Léo Le Bouter]

Thanks for your feedback.

On Sat, 2021-03-27 at 13:29 +0100, zimoun wrote:
> And as I said elsewhere, “to me, security is important. But it's
> no less important than everything *else* that is also important!“, so
> personally I am not convinced that security updates deserve a special
> treatment compared to a regular update.  That’s my opinion. :-)

You can't think this, security updates have prioritized channel of
distribution in every other GNU/Linux distribution, we in GNU Guix
created grafts for it, it's not possible to not ship security updates
promptly, it puts all users at risk.


signature.asc
Description: This is a digitally signed message part

Re: Security patching and the branching workflow: a new security-updates branch

[zimoun]

Hi Léo,

On Fri, 26 Mar 2021 at 21:10, Léo Le Bouter  wrote:

> For these reasons, I would like to propose a new branch called
> security-updates that would be based on master where we queue security
> fixes that introduce any arbitrary number of rebuilds without using
> grafts.
>
> We would merge the security-updates branch as soon as there is complete
> substitute availability for the branch and it's future merged version
> within master.

Basically, what you are proposing is more or less staging.  The issue is
that the cycles are sometimes longer than expected.  The cycle is
defined as 6 weeks for staging.  Is it really 6 weeks?  Well, It could
be shorter if there is more man power *and* if the build farm is working
better.

About the man power, it is already a good job.  Let plot roughly the
number of days between 2 merges between staging and master.

--8<---cut here---start->8---
$ git log --format="%cd|%s" --date=format:"%Y-%m-%d" --after=2019-01-01 \
  | grep Merge | grep staging | grep master \
  | cut -d'|' -f1   \
  > /tmp/staging.txt 
--8<---cut here---end--->8---

then this tiny R script:

--8<---cut here---start->8---
$ cat /tmp/dates.R
library("ggplot2")

raw <- read.table("/tmp/staging.txt")
dat <- raw$V1
dates <- data.frame(start=strptime(dat[1:length(dat)-1], format="%Y-%m-%d"),
end=strptime(dat[2:length(dat)], format="%Y-%m-%d"))
dates$diff <- difftime(dates$start, dates$end, units="days")

p <- ggplot(dates, aes(x=start, y=diff)) + geom_point(color='red') + 
geom_smooth()
ggsave("/tmp/number-of-days-staging-merges.pdf")
--8<---cut here---end--->8---

shows the attached graph.  Well, it is variable and the variance and
standard deviation should be considered.  The mean is less then 8 days.

Obviously, I can have done a mistake and this should be taken with grain
of salt.  My point is: we should root the practise on what we are
already doing and see what are the incremental changes to improve.

For example, staging is not merged since a couple of months. Why?
Because none of us is taking the time to make it.  Hard to be close to
the oven and in the same to the mill. :-)

About the build farm, it is doing better and better.  Mathieu introduced
some plots [1] and again I think we should explore a bit the data to see
what is the average time to build staging, how many packages are built
on average, etc.

Moreover, priorities have recently been introduced.  I do know if we
have enough time elapse to see their impacts.  They are per branch,
IIRC, and maybe we could imagine a priority ’properties’ applied for a
package because it is a security update.  IIUC, it is not implemented
yet.  And as I said elsewhere, “to me, security is important. But it's
no less important than everything *else* that is also important!“, so
personally I am not convinced that security updates deserve a special
treatment compared to a regular update.  That’s my opinion. :-)

That’s said, from my point of view, staging or security-updates are only
names but somehow they cover the same idea: update a package with many
dependants and provide the substitutes as soon as it is available.

The key for a better scaling is, IMHO, to tweak what is going to master,
staging and core-updates; as I wrote in [2].

But the real hard part is to collectively make this tough work of
merging branches on the long run.


Cheers,
simon

1: 
2: 



number-of-days-staging-merges.pdf
Description: graph.pdf

Packaging

[mecqor labi]

Please package shadowsocks-rust for Guix! Many users have a big problem in
access to free internet, and have to go through censorship wall. Please!!!

Re: Discrepancy between manual and channels.scm

[Ozhap]

Ludovic Courtès wrote:

> Oops you’re right; I’ll push a fix to the manual shortly.

In that case please also close bug#46769 I had raised regarding this
issue.

--
Ozhap

Re: 03/163: build/python: Add a new guix-pythonpath procedure.

[Hartmut Goebel]

Am 14.03.21 um 01:58 schrieb Maxim Cournoyer:
 6) Please add some more comments to the code explaining the idea.
>>> I was under the impression the code was concise enough to forego with
>>> verbose explanations; I'd rather keep it this way.
>> Please add some comments. I had a hard time understanding it - and I was
>> not even sure I understood, see my question (1).
> I'm spread thin right now, so if you could prepare a patch addressing
> the above for me to review, that'd be much appreciated.  Otherwise I'll
> get to it, but it won't be before some time.
>
Sorry, I'm too short in time ATM - and I'm afraid this will not change
anytime soon. Just take my questions as a guidance what the comments
should answer to make others (me :-) understand what is going on.

Many thanks.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |

Document our WIP

[Vincent Legoll]

Hello,

I'd like to reiterate my proposal to document our
ongoing projects, maybe with a "WIP" page on the
web site (even if I'm not a web guy, I volunteer
the maintenance of it).

* CI-built pinebook-pro images [1]
* other ARM boards
* ppc64 & ppc
* Hurd VM
* Full source bootstrap

And probably other things I missed.

This should complement the ML archives, IRC logs or
blog, with pointers to (hopefully) more current infos
on those subjects.

WDYT ?

[1] I only (accidentally) discovered today that those
exist

-- 
Vincent Legoll

Re: gnu: imagemagick/fixed: Redirect old sonames to new sonames.

[Mark H Weaver]

Mark H Weaver  writes:
> The following dependency chain seems to be responsible for most of the
> imagemagick-dependent packages:
>
> gtk+@3 -> at-spi2-atk -> at-spi2-core -> gtk-doc -> dblatex -> imagemagick

It occurs to me that we could add "stable" variants of the
'imagemagick', 'dblatex', and 'gtk-doc' packages.  The stable variants
would be used as native-inputs by our core packages, thus allowing the
normal variants of these packages to be updated freely on our 'master'
branch.  In particular, we could keep our 'imagemagick' package
up-to-date without grafting and without having to backport fixes.

What do you think?

  Mark

Re: Adding Substitute Mirrors page to installer

[Mathieu Othacehe]

Hello,

I plan to have a look to your patch in the next coming days. Sorry for
the long delay.

Thanks,

Mathieu

Re: Security patching and the branching workflow: a new security-updates branch

[Christopher Baines]

Léo Le Bouter  writes:

> On Fri, 2021-03-26 at 22:13 +, Christopher Baines wrote:
>> Can you clarify what specific problem or problems you're proposing
>> this
>> security-updates branch to address?
>
> Substitute availability of security updates when they are released,
> without causing big rebuilds on master for users before the build farm
> had time to produce substitutes.

Ok.

>> You mention applying and backporting patches is lots of work, and
>> uncertainty around whether grafts work in particular
>> situations.
>
> Also that some times backporting is just not possible because security
> fixes are not properly labeled upstream as security-relevant and manual
> review of each and every commit is just not viable.
>
>> Personally I think staging and core-updates are quite a bit of work,
>> and
>> adding more complexity to the process involves more work in my
>> mind. Additionally, this isn't going to provide more information
>> about
>> areas where grafts can't be used (if those exist).
>
> I understand. There's lot of uncertainty on how grafts work exactly for
> me and in what situations they work and what situations they do not.
> The only way I am certain some security fix is correctly applied in GNU
> Guix is when the vulnerable version of the package is not packaged at
> all anymore in GNU Guix.
>
>> Now the software involved is getting better at rapidly building
>> things
>> for substitutes, personally I see a way forward through trying to
>> measure and potentially increase the rate of change for outputs in
>> general. Going faster also involves more work probably, but in terms
>> of
>> the process, that might just mean that updates to more packages can
>> be
>> merged to master directly, without sitting on a non-master branch.
>
> I would like this, merging things to master directly when we feel it is
> the right thing would be what I would like to do. Even if it causes big
> world rebuilds, when we can't graft.
>
> There's another thing I saw that was ongoing but can't remember where,
> that 'guix pull' could hold off updating to newer revisions unless
> substitutes are available.

I think approaching the substitute availability problem this way, and
supporting users delaying updates if they want to is the way to go, at
least to avoid process on the side of making changes.

I'm hoping things will get better through a combination of these
factors:

 - Measuring the "churn" in Guix, which will hopefully allow for
   intentionally increaseing this

 - Keeping packages more up to date generally

 - Allowing users to choose whether they want the latest packages, or
   want to avoid building things (this means important security updates
   that cause rebuilds can be merged to master)


signature.asc
Description: PGP signature

Re: Cuirass not processing core-updates (or weirdly)

[Mathieu Othacehe]

Hello Leo,

> If you look at https://ci.guix.gnu.org/eval/13652 you can see that the
> evaluation of the derivation seems completed but there's no pending
> builds.

If you have a look to
https://ci.guix.gnu.org/specification/edit/core-updates you will see
that this specification is configured to build to "core"[1] subset of
the "core-updates" branch with the "4" priority.

The 13652 evaluation does not trigger any new derivation in the core
packages subset, hence there are no triggered builds.

When there are some new derivations to build, like for the 12656
evaluation, because of the low priority, they can stay pending for a
long time.

Thanks,

Mathieu

[1]: 
http://guix.gnu.org/cuirass/manual/html_node/Specifications.html#Specifications

Re: Adding Substitute Mirrors page to installer

[raid5atemyhomework]

Bump


Sent with ProtonMail Secure Email.

‐‐‐ Original Message ‐‐‐
On Tuesday, March 16, 2021 11:55 PM, raid5atemyhomework 
 wrote:

> Hi all,
>
> Below is the new patch version.
>
> In this version, the installer now also reads the generated 
> `operating-system` file to extract the `guix-configuration-substitute-urls`, 
> in order to pass it into the `start` action of `guix-daemon`. The `start` 
> action also now supports a second argument, the space-separated list of 
> substitute URLs. I'm wary of this technique as I feel it is unclean, but it 
> works and does not require significant changes to the existing software 
> architecture of the installer.
>
> Tested in this manner:
>
> -   Created an installer image by `./pre-inst-env guix system image -t 
> iso9660 gnu/system/install.scm`.
> -   Started a new VM with the installer image and selected the SJTUG mirror.
> -   Confirmed that during installation the installer downloaded substitutes 
> from the SJTUG mirror.
> -   After installation completed on the VM, did a `guix pull` on the new VM 
> instance and confirmed it downloaded substitutes from the SJTUG mirror.
>
> I haven't tested for the use of the normal Berlin Cuirass, as that would 
> be ridiculously slow right now from my network, but I expect it would 
> continue to work.
>
> Thanks
> raid5atemyhomework
>
> From 68a42cce2b4ae876cbbd1911aaa2a5bc8348bf15 Mon Sep 17 00:00:00 2001
> From: raid5atemyhomework raid5atemyhomew...@protonmail.com
>
>
> Date: Tue, 16 Mar 2021 23:45:37 +0800
> Subject: [PATCH] gnu: Add substitute mirrors page to installer.
>
> -   gnu/installer/services.scm (system-service) [snippet-type]: New field.
> (%system-services): Add substitute mirrors.
> (service-list-service?): New procedure.
> (modify-services-service?): New procedure.
> (system-services->configuration): Add support for services with
>
>
> `'modify-services` snippets.
>
> -   gnu/installer/newt/services.scm (run-substitute-mirror-page): New
> procedure.
> (run-services-page): Call `run-substitute-mirror-page`.
>
> -   gnu/services/base.scm (guix-shepherd-service)[start]: Accept second
> argument, a space-separated list of substitute URLs.
>
> -   gnu/installer/final.scm (%user-modules): New variable.
> (read-operating-system): New procedure.
> (install-system): Read the installation configuration file and extract
> substitute URLs to pass to `guix-daemon` start action.
>
>
> gnu/installer/final.scm | 36 ++-
> gnu/installer/newt/services.scm | 26 +-
> gnu/installer/services.scm | 62 -
> gnu/services/base.scm | 15 ++--
> 4 files changed, 125 insertions(+), 14 deletions(-)
>
> diff --git a/gnu/installer/final.scm b/gnu/installer/final.scm
> index fc0b7803fa..6eca3ec606 100644
> --- a/gnu/installer/final.scm
> +++ b/gnu/installer/final.scm
> @@ -22,9 +22,13 @@
> #:use-module (gnu installer steps)
> #:use-module (gnu installer utils)
> #:use-module (gnu installer user)
>
> -   #:use-module (gnu services)
> -   #:use-module (gnu services base)
> #:use-module (gnu services herd)
>
> -   #:use-module (gnu system)
> #:use-module (guix build syscalls)
> #:use-module (guix build utils)
>
> -   #:use-module (guix ui)
> #:use-module (gnu build accounts)
> #:use-module (gnu build install)
> #:use-module (gnu build linux-container)
> @@ -38,6 +42,20 @@
> #:use-module (ice-9 rdelim)
> #:export (install-system))
>
> +;; XXX duplicated from guix/scripts/system.scm, but that pulls in
> +;; (guix store database), which requires guile-sqlite which is not
> +;; available in the installation environment.
> +(define %user-module
>
> -   ;; Module in which the machine description file is loaded.
> -   (make-user-module '((gnu system)
> -(gnu services)
>
>
> -(gnu system shadow
>
>
> -
>
> +(define (read-operating-system file)
>
> -   "Read the operating-system declaration from FILE and return it."
> -   (load* file %user-module))
> +;; XXX
>
> -
>
> (define %seed
> (seed->random-state
>
> (logxor (getpid) (car (gettimeofday)
>
>
> @@ -174,6 +192,15 @@ or #f. Return #t on success and #f on failure."
> options
> (list (%installer-configuration-file)
> (%installer-target-dir
>
> -   ;; Extract the substitute URLs of the user configuration.
>
>
> -   (os  (read-operating-system 
> (%installer-configuration-file)))
>
>
> -   (substitute-urls (and=> (find
>
>
> - (lambda (service)
>
>
> -   (eq? guix-service-type
>
>
> -(service-kind service)))
>
>
> - (operating-system-services os))
>
>
> -   (compose 
> guix-configuration-substitute-urls
>
>
> -