Re: substitute derivation: also substitute grafts?

[Ricardo Wurmus]

Maxime Devos  writes:

> On 15-09-2022 16:46, Csepp wrote:
>> Ricardo Wurmus  writes:
>> 
>>> [...]
>>> Did I say *all items*? Well, … grafts are not included, because graft
>>> derivations are marked as not substitutable.
>>>
>>> Can we change that conditionally? I would really like to avoid having
>>> to build grafts on B when they have already been built on A.
>> I would love this too, because IO can be incredibly slow on HDDs and
>> large packages.  My netbook would be thankful.
>>
>
> There are some some opportunities for optimizations in the grafting
> code before substituting more -- for example, to avoid seek times, it
> would be possible to rewrite multiple files concurrently (maybe using
> 'par-for-each' to process each file in a directory in parallel).

While this is also an interesting development for grafts, my question is
much narrowed.  I’d be very happy if we could have a little switch that
made my daemon(s) ignore the “substitutable?” property and just
substitute everything.

I’m not looking for performance improvements in grafting, but in an
option to totally avoid doing the work twice when substituting a
derivation from a server in the same network.

-- 
Ricardo

Re: Translation: Would regional translation fall back to primary language?

[Liliana Marie Prikler]

Am Montag, dem 19.09.2022 um 13:53 +0200 schrieb pelzflorian (Florian
Pelz):
> But still, from an organizational view, I suggest not doing a partial
> translation.  I don’t know though.
I'd like to weaken this a little: Don't *intentionally* do a partial
translation.  With the insights you provided, I think it'd be fine to
fall back to regular Spanish if the Colombian translator is unavailable
for some while (due to vacation or what have you).

Cheers

Re: substitute derivation: also substitute grafts?

[Maxime Devos]

On 19-09-2022 18:26, Josselin Poiret wrote:

Hi everyone,

Maxime Devos  writes:

Fallbacks might be necessary (not every store item is constructed from a
package), but it all sounds doable and efficient.  Also the union could
needs to be modified to ignore the .graft-offsets of the union'ed things.


If I understand the whole thing properly, the daemon already scans for
references and could register their location in the database, avoiding
the need for a potentially brittle in-store format as you highlighted,
although that would require a non-trivial change to both the daemon and
the database format.


If you do this, fallbacks are still required for old daemons.

Daemon database changes sound much more brittle to me than the simple
.graft-offsets -- the only brittleness I see is the unioning code, but 
that's easy to address.


OTOH, in the ".graft-offsets" proposal, the build process has to do 
reference scanning (even though the daemon will do so as ell later), 
which adds some inefficiency -- so perhaps performance-wise, the 
additional testing for the fallbacks and brittleness might be worth it.



In any case, I don't think it's necessary to treat packages in a special
way: any derivation can hold references and thus be grafted.


(Corrected a likely typo derivations->references.)

They indeed don't need to be treated specially -- in my proposal, the 
grafting code wouldn't care whether a store item is of a package or 
something else, the only thing it cares about, is the presence of a 
.graft-offsets file.


However (unless we go for daemon changes), this .graft-offsets still 
needs to be made somewhere -- my proposal was to do so this, in case of 
packages, in an additional phase (which would call a 
'make-graft-offsets' procedure '(make-graft-offsets #$output ...)'.


Then, I suppose this could be extended to non-packages as well (e.g. 
modify gexp->derivation and sexp->derivation to do the 
make-graft-offsets automatically).  However, packages should cover 
almost everything (profiles are mostly a bunch of symlinks and thus 
don't benefit from grafting optimisations much IIUC), and then 
gexp->derivation would insert an additional file without it being asked 
for, which would sound rather surprising to me, so I think that covering 
packages would be sufficient.


Greetings,
Maxime.


OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: substitute derivation: also substitute grafts?

[Josselin Poiret]

Hi everyone,

Maxime Devos  writes:
> Fallbacks might be necessary (not every store item is constructed from a 
> package), but it all sounds doable and efficient.  Also the union could 
> needs to be modified to ignore the .graft-offsets of the union'ed things.

If I understand the whole thing properly, the daemon already scans for
references and could register their location in the database, avoiding
the need for a potentially brittle in-store format as you highlighted,
although that would require a non-trivial change to both the daemon and
the database format.

In any case, I don't think it's necessary to treat packages in a special
way: any derivation can hold derivations and thus be grafted.

Best,
-- 
Josselin Poiret

Re: What 'sh' should 'system' use?

[Maxime Devos]

On 19-09-2022 02:13, Philip McGrath wrote:

1) If we want to continue to hard-code a specific shell into Glibc,


We do, for reproducibility -- otherwise, the behaviour of the 'system' 
function depends on whatever is the current /bin/sh, and sometimes 
/bin/sh is updated (and on some foreign systems it might not even be the 
bash we are used to).


 I think we

should document the decision (for example, why 'bash-static' vs. 'bash-
minimal'?)


Because cycles -- bash-minimal is linked to a (shared) glibc, which is a 
separate package from bash-minimal, so glibc cannot use bash-minimal, it 
uses bash-static instead which is linked to a (static) glibc (which 
might use a bootstrap bash (not 100% sure), but it's statically linked, 
so no reference to the bootstrap bash remains IIUC).


Also, why?  This is an implementation detail.  Who would the target 
audience be for this documentation?



and recommendations for how packages should use it: '_PATH_BSHELL'
is the best mechanism I've heard of so far, though I wish it were
standardized, and the fact that it can't be portably assumed to be a string
constant could be surprising.


I consider _not_ using it, and using (4) instead, to be best.
If not suitable (for example, because a shell is needed to run an actual 
shell script), then a plain "sh" looked up in the $PATH (like other 
binaries) and substitute*-ed by Guix should suffice.




2) If we want to make 'sh' a weak/dynamic reference, I think we should
strongly consider arranging to make it available at '/bin/sh' when present. I
expect this option would require less patching of other packages*by far*  than
any other approach.


See (1) (reproducibility) -- also, you would need to modify the daemon 
for that, so there are compatibility concerns, and then we're stuck with 
the /bin/sh special case forever (unless breaking compatibility would 
later be considered acceptable).




3) If we want a dynamic 'sh' not located at '/bin/sh', I think we should
implement a function similar to '__bionic_get_shell_path()' and use it for
'_PATH_BSHELL', 'system', etc. That begs the question of how the function
should find 'sh', and I don't have an answer for that.


How about $PATH?


In principle, we could
design a configuration mechanism for 'confstr(_CS_PATH, buf, sizeof(buf))' and
use it to find the shell: that has some appeal, but making the mechanism
extensible enough to support "all of the standard utilities of POSIX.1-2017"
seems like a challenge.
What do you think?


(4) Stop using 'system' in applications -- instead use whatever the 
language's equivalent of Guile's system*, execl ... or Guix' 'invoke'. 
Why?  Because 'system'-like functions requires quoting the command line 
arguments whereas in 'system*'-like functions you could just pass a list 
of command line arguments, and it's easy to get the quoting wrong, 
especially if some of the arguments are generated dynamically.


As a bonus, this could often remove a dependency on bash{-minimal,-static,}.

Maybe we can eventually remove _PATH_BSHELL and 'system' from our glibc 
(and Guile, ...).


Greetings,
Maxime.


OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: Translation: Would regional translation fall back to primary language?

[pelzflorian (Florian Pelz)]

Hello Luis/sirgazil.  Translation help sounds great.

That said, I believe that, because others will add to the translation,
that it would be best if you Downloaded the Spain Spanish file from
Weblate and then uploaded that PO file as the first translation in
Colombian Spanish.  Then change it from that base.

Luis Felipe  writes:
> So, assuming an incomplete es-CO translation catalog already exists,
> would the translation system (gettext?) retrieve missing translations
> from the catalog corresponding to the primary language (es) or would
> it retrieve the source string in English?

I just tested adding a dummy de_CH.po translation file to Guix and yes,
it falls back to de translations if the msgstr in de_CH is empty, even
without changing the LANGUAGE environment variable and only setting
LC_ALL=de_CH.utf8.

But still, from an organizational view, I suggest not doing a partial
translation.  I don’t know though.

Regards,
Florian

Re: [bootstrappable] Re: Mes Bootstrapping demonstration

[Andrius Štikonas]

Hi Aret,

> If you don't want to use Guix, you may have a look at the live bootstrap
> project
> 
>https://github.com/fosslinux/live-bootstrap

You might want to take a look at the following files:

https://github.com/fosslinux/live-bootstrap/blob/master/sysa/run.kaem
https://github.com/fosslinux/live-bootstrap/blob/master/sysa/mes-0.24/mes-0.24.kaem

The first file sets up some environmental variables that are used by both 
mes-0.24 and tcc-0.9.26 steps.
I think GUILE_LOAD_PATH and MES_PREFIX are required by mes and rest (NYACC_PKG, 
MES_VERSION, MES_PKG_DIR)
are for scripting convenience.

The second file sets up MES_ARENA, MES_MAX_ARENA, MES_STACK and the rest of 
variables are just for scripting convenience.

Kind regards,
Andrius

> 
> Greetings,
> Janneke
> 
> 



signature.asc
Description: This is a digitally signed message part.

Re: Progress with automating testing of patches

[Christopher Baines]

Christopher Baines  writes:

> Since it's easier to iterate once there's something visible, I've just
> stuck what I've got so far on the internet, it's available at:
>
>   https://qa.guix.gnu.org/
>
> The code is here [3] and I've put a list of ideas in the README.
>
> 3: https://git.cbaines.net/guix/qa-frontpage/about/
>
> Currently, there's a page which lists issues, and pages for individual
> issues that show build and lint warning changes. Behind the scenes, it's
> also submitting builds to the build coordinator for the packages
> affected by patches (for x86_64-linux, i686-linux, aarch64-linux and
> armhf-linux).
>
> As I've been developing it, I've been looking at various recent patch
> series, and it seems like this is already useful. It's reassuring when
> reviewing patches to see that packages still build on the architectures
> being tested. Also, unlike earlier prototype patch testing setups,
> because the builds are now happening on a default substitute server,
> there should be some benefits already with substitutes being available
> at the point the patches are merged.
>
> This is very much still at the prototype stage though, many pages will
> timeout or just fail to load due to an error.
>
> Let me know if you have any comments or questions? If you want to try
> and work on adding any features, I should also have time to try and help
> out as well, so just let me know you're interested.

There's been some more progress now since I sent this email, so I
thought I'd send an update.

I've started to try and show some overall status for each issue, that's
the green circle which can appear on the patches page. I want to try and
make this available as an image as well, so that it can be included on
the issue page on issues.guix.gnu.org, linking up qa.guix.gnu.org and
issues.guix.gnu.org.

I've also managed to use the GraphQL API for issues.guix.gnu.org to
fetch tags, which is very exciting as it makes it possible to do all
kinds of things I think. Starting with showing specific tags on the
qa.guix.gnu.org pages, and ending maybe with replacing Patchwork with
data provided by Mumi.

Also, while builds for patches were happening, there's now support for
submitting builds for a branch. This is the case for staging, so these
builds are now happening. It'll probably take many days, maybe even
weeks for all the builds to happen, but at this point I'm mostly
interested in getting the software working so that qa.guix.gnu.org is
really useful for working with branches going forward.

I also covered this qa.guix.gnu.org thing in the talk I gave yesterday
at the 10 Years of Guix event. See the email with this subject [1] for
some very rough notes from the discussion that happened in the late
afternoon. There's a list of seemingly actionable things in there, some
of which relate to the qa.guix.gnu.org site. In particular, I'll try and
push the repository to Savannah at some point this week, so let me know
if you have thoughts on changing the name (currently qa-frontpage)
before I do.

1: Notes from discussion on Quality Assurance from the 10 Years of Guix event

Do let me know if you have any comments or questions!

Thanks,

Chris


signature.asc
Description: PGP signature

Re: Store Functor and Store Applicative

[zimoun]

Hi,

On Sun, 18 Sep 2022 at 09:51, jgart  wrote:

> Why does Guix implement a monad but not a functor and an applicative?

>From my understanding, because an appropriate and straightforward
abstraction for the store is the State Monad.

Maybe the implementator of the monadic stuff in Guix read right before
this [1]. :-)

1: 


Cheers,
simon

Re: What 'sh' should 'system' use?

[Liliana Marie Prikler]

Am Sonntag, dem 18.09.2022 um 20:13 -0400 schrieb Philip McGrath:
> On the other hand, even Nix puts '/bin/sh' at its usual path: we are
> really quite an outlier in this respect. (IIUC, Nix also has
> '/usr/bin/env', but no other utilities at FHS paths.)
We are not.  We provide both /bin/sh and /usr/bin/env.  If you're
talking about the build container then that's a much smaller
distinction.

> First, a test program I tried in [9] seemed to indicate that
> '_PATH_BSHELL' refered to 'bash-static', but 'system("echo $BASH")'
> referred to 'bash-minimal'. It's possible that my test gave an
> incorrect answer: I just tried 'guix size glibc' (I hadn't thought of
> that earlier), and it doesn't list a reference to 'bash-minimal'.
> But, if we are embedding references in libc to two different copies
> of Bash, that seems clearly bad.
We aren't embedding two references though; if we did, you'd see bash-
minimal in the closure.

> More broadly, I now think it would be better in we embedded zero
> references to copies of Bash in libc.
I don't think we can do that without breaking system.

> I have changed my mind on this before, and I could be persuaded
> otherwise.   When I wrote the Racket patch for '/bin/sh' that had
> been in place before the latest change, I initially was going to use
> a hard-coded Bash only when '/bin/sh' did not exist, but the
> discussion persuaded me it would make more sense to always use the
> 'sh' from the package's inputs.[10] For Racket, a dependency on 
> 'sh' didn't seem too unreasonable.
It certainly isn't the largest package racket pulls in, no.

> However, giving every program using Glibc a hard dependency on
> Bash—and on a particular Bash executable—seems like a much bigger
> imposition.
We're talking 1.7 MiB here.  Certainly a "big" imposition, but nothing
in comparison to the things you need in the store for bootstrapping
purposes.  Also note that bash-minimal, while only taking up 1.0 MiB
for itself, requires both glibc and gcc:lib, which apart from creating
a cycle does blow up its closure size quite a bit.

> I now think it would be better to find 'sh' dynamically at run time
Stop it.  Get some help.
> >    In versions of glibc before 2.1.3, [...] system()
> >    always returned 1 [...].
Note that always returning non-zero is required by POSIX 2017.

> >    [E]ven though POSIX.1-2001 requires a conforming
> >    implementation to provide a shell, that shell may not be
> >    available or executable if the calling program has
> >previously called chroot(2) [...].
Which does nothing to aid us in actually shelling out.

> Finally, some possible courses of action:
> 
> 1) If we want to continue to hard-code a specific shell into Glibc, I
> think we should document the decision (for example, why 'bash-static'
> vs. 'bash-minimal'?) and recommendations for how packages should use
> it: '_PATH_BSHELL' 
> is the best mechanism I've heard of so far, though I wish it were 
> standardized, and the fact that it can't be portably assumed to be a
> string constant could be surprising.
Note, that _PATH_BSHELL is only required for programs that want to be
portable to other *nix systems.  For most programs written with only
the common Linux distros in mind, substituting "/bin/sh" is more than
enough in terms of compatibility.

> 2) If we want to make 'sh' a weak/dynamic reference, I think we
> should strongly consider arranging to make it available at '/bin/sh'
> when present. I expect this option would require less patching of
> other packages *by far* than any other approach.
How about no?

> 3) If we want a dynamic 'sh' not located at '/bin/sh', I think we
> should implement a function similar to '__bionic_get_shell_path()'
> and use it for '_PATH_BSHELL', 'system', etc. That begs the question
> of how the function should find 'sh', and I don't have an answer for
> that. In principle, we could design a configuration mechanism for
> 'confstr(_CS_PATH, buf, sizeof(buf))' and use it to find the shell:
> that has some appeal, but making the mechanism extensible enough to
> support "all of the standard utilities of POSIX.1-2017" seems like a
> challenge.
This sounds like a very long and convoluted way to hard-code a string.
Remember that _PATH_BSHELL ought to be resolved to a compile-time
pseudo-constant string.

> What do you think?
If you're really annoyed by the confstr thing, make it so that it hard-
codes the #$bash-static/bin.

Cheers