Re: Fwd: Building Guile with ‘-j1’?

[Leo Prikler]

Am Mittwoch, den 20.01.2021, 16:08 -0500 schrieb Mark H Weaver:
> Leo Prikler  writes:
> 
> > There could potentially be another workaround by synchronizing
> > inside
> > guild, i.e. claiming a lock before reading and evaling any given
> > source
> > file.  This would have the advantage of applying to all guile
> > packages,
> > not just the ones that use guile-build-system, and it would also
> > allow
> > non-Guile stuff, e.g. C extensions to be built in parallel.  WDYT?
> 
> I don't think this would solve the problem, because it wouldn't
> ensure
> that the Guile modules are compiled in a deterministic order, and
> therefore the issue I raised in  would
> lead to nondeterministic results.
> 
>Mark
Because the OS might choose to schedule the tasks in a different order
than spawned, e.g. compilation tasks 1-4 start "simultaneously and get
reordered 1, 3, 2, 4?  Indeed, that would be a problem.  I don't
suppose file-lock based queues exist, do they?

Re: Fwd: Building Guile with ‘-j1’?

[Mark H Weaver]

Leo Prikler  writes:

> There could potentially be another workaround by synchronizing inside
> guild, i.e. claiming a lock before reading and evaling any given source
> file.  This would have the advantage of applying to all guile packages,
> not just the ones that use guile-build-system, and it would also allow
> non-Guile stuff, e.g. C extensions to be built in parallel.  WDYT?

I don't think this would solve the problem, because it wouldn't ensure
that the Guile modules are compiled in a deterministic order, and
therefore the issue I raised in  would
lead to nondeterministic results.

   Mark

Re: Building Guile with ‘-j1’?

[Mark H Weaver]

Hi Ludovic,

Ludovic Courtès  writes:

> As the saying goes, “the cobbler’s children go barefoot”.  Guile/Guix
> are no exception since Guile builds are non-reproducible, despite work
> done a few years ago:
>
>   https://issues.guix.gnu.org/20272
>
> Until it’s fixed in Guile proper, what do you think of building Guile
> 2.0/2.2/3.0 with #:parallel-build? #f ?  We could do that in
> ‘core-updates’ now.

Sounds good to me.  Given the nature of Guile's module system, it's
still not clear to me how to ensure reproducible builds of Guile modules
unless the order of compilation is deterministic.  I haven't thought
about it in a few years, though.

   Mark

Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Timothy Sample]

Hi janneke,

Jan Nieuwenhuizen  writes:

> I have reset Guix' wip-full-source-bootstrap branch with a first working
> implementation of the, well, "Full Source Bootstrap" for x86-linux (and
> x86_64-linux).  This bootstrap is rooted in the 357-byte hex0-seed from
> the Stage0 project (https://savannah.gnu.org/projects/stage0).

The dream is alive!  Congratulations on this big leap forward!

> When you look at the bottom of the graph (see attached), you will notice
> "%bootstrap-guile": the driver that we use for the Guix build and also
> for "bootar", "gash", and "gash-utils".  This "%bootstrap-guile" is not
> used as a seed in anything that is built, "%bootstrap-guile", "bootar",
> "gash", and "gash-utils" could be replaced with any other driver.

I never mentioned it, but a few months ago I took a little look at
porting Gash & friends to Mes.  The big issue that I ran into is that
Mes doesn’t really have a module system.  My plan was to build up Mes
modules and strip down Gash requirements until they met in the middle.
Sometime (probably not worth derailing this thread right now) we should
discuss what needs to be done for Mes modules.  (It looked like
something I could do with a little guidance on the design.)

> XXX TODO:
>* wip-full-source-bootstrap
>  [...]
>* wip-arm-bootstrap
>  - finish; currently stuck on gawk-mesboot0
>  [...]

It looks like you’ve made a lot of progress on this already (judging by
the rest of this thread).  However, if it helps, the current Gash-Utils
awk could _probably_ be used to skip most (all?) of the old versions of
Gawk.

Sorry I can’t be more helpful ATM.  I appreciate the work you do to keep
this project rolling!


-- Tim

Re: Building Guile with ‘-j1’?

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

Hello,

> As the saying goes, “the cobbler’s children go barefoot”.  Guile/Guix
> are no exception since Guile builds are non-reproducible, despite work
> done a few years ago:
>
>   https://issues.guix.gnu.org/20272

Ugh.  Thanks for bringing this up again.  Would you consider disabling
parallel builds in Guile by default, adding something like a
"--enable-somewhat-faster-non-reproducible-build" configure option until
this is fixed?

> Until it’s fixed in Guile proper, what do you think of building Guile
> 2.0/2.2/3.0 with #:parallel-build? #f ?  We could do that in
> ‘core-updates’ now.
>
> That would work around the problem for Guile itself.  It would increase
> build times, but probably not that much since the most expensive part
> (compiling the first few files) is sequential anyway.  IIRC this is what
> Vagrant did for the Debian packages.

Yes, I would support at least doing this.  

> We could also disable parallel builds in ‘guile-build-system’.  It’s
> only used for small packages so the extra build time is probably OK.

Are packages using guile-build-system expected to build non-reproducible
too?  In that case, I would certainly support doing it there.

Greetings,
Janneke

-- 
Jan Nieuwenhuizen  | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Fwd: Building Guile with ‘-j1’?

[Leo Prikler]

Messed up ML address.
--- Begin Message ---
Hi Ludo,

IIUC this issue potentially affects all packages, that use Guile to
build more than two modules, no?

There could potentially be another workaround by synchronizing inside
guild, i.e. claiming a lock before reading and evaling any given source
file.  This would have the advantage of applying to all guile packages,
not just the ones that use guile-build-system, and it would also allow
non-Guile stuff, e.g. C extensions to be built in parallel.  WDYT?

Regards,
Leo
--- End Message ---

Re: Building Guile with ‘-j1’?

[Gábor Boskovits]

Hello,

Ludovic Courtès  ezt írta (időpont: 2021. jan. 20., Sze, 9:42):
>
> Hi!
>
> As the saying goes, “the cobbler’s children go barefoot”.  Guile/Guix
> are no exception since Guile builds are non-reproducible, despite work
> done a few years ago:
>
>   https://issues.guix.gnu.org/20272
>
> Until it’s fixed in Guile proper, what do you think of building Guile
> 2.0/2.2/3.0 with #:parallel-build? #f ?  We could do that in
> ‘core-updates’ now.
>
> That would work around the problem for Guile itself.  It would increase
> build times, but probably not that much since the most expensive part
> (compiling the first few files) is sequential anyway.  IIRC this is what
> Vagrant did for the Debian packages.
>
> We could also disable parallel builds in ‘guile-build-system’.  It’s
> only used for small packages so the extra build time is probably OK.
>
> Thoughts?

Let's go for it.

>
> Ludo’.
>

Regards,
g_bor
-- 
OpenPGP Key Fingerprint: 7988:3B9F:7D6A:4DBF:3719:0367:2506:A96C:CF63:0B21

Re: Building Guile with ‘-j1’?

[zimoun]

Hi Ludo,

On Wed, 20 Jan 2021 at 09:41, Ludovic Courtès  wrote:

> Until it’s fixed in Guile proper, what do you think of building Guile
> 2.0/2.2/3.0 with #:parallel-build? #f ?  We could do that in
> ‘core-updates’ now.

[...]

> We could also disable parallel builds in ‘guile-build-system’.  It’s
> only used for small packages so the extra build time is probably OK.

Good idea.


One question: will “guix pull” be slowed?  (if it makes sense)


Cheers,
simon

Building Guile with ‘-j1’?

[Ludovic Courtès]

Hi!

As the saying goes, “the cobbler’s children go barefoot”.  Guile/Guix
are no exception since Guile builds are non-reproducible, despite work
done a few years ago:

  https://issues.guix.gnu.org/20272

Until it’s fixed in Guile proper, what do you think of building Guile
2.0/2.2/3.0 with #:parallel-build? #f ?  We could do that in
‘core-updates’ now.

That would work around the problem for Guile itself.  It would increase
build times, but probably not that much since the most expensive part
(compiling the first few files) is sequential anyway.  IIRC this is what
Vagrant did for the Debian packages.

We could also disable parallel builds in ‘guile-build-system’.  It’s
only used for small packages so the extra build time is probably OK.

Thoughts?

Ludo’.