Re: Building Guile with ‘-j1’?

[Maxim Cournoyer]

Hi,

Ludovic Courtès  writes:

> Hi Maxim,
>
> Maxim Cournoyer  skribis:
>
>> Mark H Weaver  writes:
>>
>>> Hi Ludovic,
>>>
>>> Ludovic Courtès  writes:
>>>
 As the saying goes, “the cobbler’s children go barefoot”.  Guile/Guix
 are no exception since Guile builds are non-reproducible, despite work
 done a few years ago:

   https://issues.guix.gnu.org/20272

 Until it’s fixed in Guile proper, what do you think of building Guile
 2.0/2.2/3.0 with #:parallel-build? #f ?  We could do that in
 ‘core-updates’ now.
>
> I pushed that as 2ea52f90141974fb5d88b8c6b1785817c4203da4.
>
>> I'm not too found of it.  It'll make the already slow Guile build much
>> slower, making the lower strata of core-updates packages more painfully
>> slow to build and test.
>
> Yes, it’s suboptimal.  OTOH, as I wrote, I think the slowdown is not as
> important as one might think: a large fraction of the Guile build time
> goes into building ice-9/eval.scm and the first few files, which is
> already sequential (enforced by Guile’s makefile).

I've built Guile many times on core-updates since and it seems to be
true, that the build time didn't change noticeably.

>> It'll also enable us to overlook the issue for years to come
>> (similarly to the fact that the testsuite of the Guix package itself
>> hasn't been run in parallel for the last 6 years or so :-)).
>
> It remains and bug to fix in Guile, no doubt.  But we also have to be
> pragmatic IMO.

I had the feeling this partial fix would cause build the build time to
explode, but given it didn't, I'm happy with it.

Thanks,

Maxim

Re: Building Guile with ‘-j1’?

[Ludovic Courtès]

Hi Maxim,

Maxim Cournoyer  skribis:

> Mark H Weaver  writes:
>
>> Hi Ludovic,
>>
>> Ludovic Courtès  writes:
>>
>>> As the saying goes, “the cobbler’s children go barefoot”.  Guile/Guix
>>> are no exception since Guile builds are non-reproducible, despite work
>>> done a few years ago:
>>>
>>>   https://issues.guix.gnu.org/20272
>>>
>>> Until it’s fixed in Guile proper, what do you think of building Guile
>>> 2.0/2.2/3.0 with #:parallel-build? #f ?  We could do that in
>>> ‘core-updates’ now.

I pushed that as 2ea52f90141974fb5d88b8c6b1785817c4203da4.

> I'm not too found of it.  It'll make the already slow Guile build much
> slower, making the lower strata of core-updates packages more painfully
> slow to build and test.

Yes, it’s suboptimal.  OTOH, as I wrote, I think the slowdown is not as
important as one might think: a large fraction of the Guile build time
goes into building ice-9/eval.scm and the first few files, which is
already sequential (enforced by Guile’s makefile).

> It'll also enable us to overlook the issue for years to come
> (similarly to the fact that the testsuite of the Guix package itself
> hasn't been run in parallel for the last 6 years or so :-)).

It remains and bug to fix in Guile, no doubt.  But we also have to be
pragmatic IMO.

> It also won't fix the issue of Guix modules compiled in parallel
> suffering from the same problem.

Definitely.

> So I'd keep it as it is for now, as a reminder that this is a serious
> problem in need of a fix.

There’s also the problem that a fix in Guile takes time to deploy
(basically a ‘core-updates’ cycle).  It’s not the first time we work
around a Guile bug until it’s fixed in Guile proper.

Thanks,
Ludo’.

Re: Building Guile with ‘-j1’?

[Maxim Cournoyer]

Hi,

Mark H Weaver  writes:

> Hi Ludovic,
>
> Ludovic Courtès  writes:
>
>> As the saying goes, “the cobbler’s children go barefoot”.  Guile/Guix
>> are no exception since Guile builds are non-reproducible, despite work
>> done a few years ago:
>>
>>   https://issues.guix.gnu.org/20272
>>
>> Until it’s fixed in Guile proper, what do you think of building Guile
>> 2.0/2.2/3.0 with #:parallel-build? #f ?  We could do that in
>> ‘core-updates’ now.

I'm not too found of it.  It'll make the already slow Guile build much
slower, making the lower strata of core-updates packages more painfully
slow to build and test.  It'll also enable us to overlook the issue for
years to come (similarly to the fact that the testsuite of the Guix
package itself hasn't been run in parallel for the last 6 years or so
:-)).

It also won't fix the issue of Guix modules compiled in parallel
suffering from the same problem.

So I'd keep it as it is for now, as a reminder that this is a serious
problem in need of a fix.

Thanks,

Maxim

Re: Fwd: Building Guile with ‘-j1’?

[Leo Prikler]

Am Mittwoch, den 20.01.2021, 16:08 -0500 schrieb Mark H Weaver:
> Leo Prikler  writes:
> 
> > There could potentially be another workaround by synchronizing
> > inside
> > guild, i.e. claiming a lock before reading and evaling any given
> > source
> > file.  This would have the advantage of applying to all guile
> > packages,
> > not just the ones that use guile-build-system, and it would also
> > allow
> > non-Guile stuff, e.g. C extensions to be built in parallel.  WDYT?
> 
> I don't think this would solve the problem, because it wouldn't
> ensure
> that the Guile modules are compiled in a deterministic order, and
> therefore the issue I raised in  would
> lead to nondeterministic results.
> 
>Mark
Because the OS might choose to schedule the tasks in a different order
than spawned, e.g. compilation tasks 1-4 start "simultaneously and get
reordered 1, 3, 2, 4?  Indeed, that would be a problem.  I don't
suppose file-lock based queues exist, do they?

Re: Fwd: Building Guile with ‘-j1’?

[Mark H Weaver]

Leo Prikler  writes:

> There could potentially be another workaround by synchronizing inside
> guild, i.e. claiming a lock before reading and evaling any given source
> file.  This would have the advantage of applying to all guile packages,
> not just the ones that use guile-build-system, and it would also allow
> non-Guile stuff, e.g. C extensions to be built in parallel.  WDYT?

I don't think this would solve the problem, because it wouldn't ensure
that the Guile modules are compiled in a deterministic order, and
therefore the issue I raised in  would
lead to nondeterministic results.

   Mark

Re: Building Guile with ‘-j1’?

[Mark H Weaver]

Hi Ludovic,

Ludovic Courtès  writes:

> As the saying goes, “the cobbler’s children go barefoot”.  Guile/Guix
> are no exception since Guile builds are non-reproducible, despite work
> done a few years ago:
>
>   https://issues.guix.gnu.org/20272
>
> Until it’s fixed in Guile proper, what do you think of building Guile
> 2.0/2.2/3.0 with #:parallel-build? #f ?  We could do that in
> ‘core-updates’ now.

Sounds good to me.  Given the nature of Guile's module system, it's
still not clear to me how to ensure reproducible builds of Guile modules
unless the order of compilation is deterministic.  I haven't thought
about it in a few years, though.

   Mark

Re: Building Guile with ‘-j1’?

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

Hello,

> As the saying goes, “the cobbler’s children go barefoot”.  Guile/Guix
> are no exception since Guile builds are non-reproducible, despite work
> done a few years ago:
>
>   https://issues.guix.gnu.org/20272

Ugh.  Thanks for bringing this up again.  Would you consider disabling
parallel builds in Guile by default, adding something like a
"--enable-somewhat-faster-non-reproducible-build" configure option until
this is fixed?

> Until it’s fixed in Guile proper, what do you think of building Guile
> 2.0/2.2/3.0 with #:parallel-build? #f ?  We could do that in
> ‘core-updates’ now.
>
> That would work around the problem for Guile itself.  It would increase
> build times, but probably not that much since the most expensive part
> (compiling the first few files) is sequential anyway.  IIRC this is what
> Vagrant did for the Debian packages.

Yes, I would support at least doing this.  

> We could also disable parallel builds in ‘guile-build-system’.  It’s
> only used for small packages so the extra build time is probably OK.

Are packages using guile-build-system expected to build non-reproducible
too?  In that case, I would certainly support doing it there.

Greetings,
Janneke

-- 
Jan Nieuwenhuizen  | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Fwd: Building Guile with ‘-j1’?

[Leo Prikler]

Messed up ML address.
--- Begin Message ---
Hi Ludo,

IIUC this issue potentially affects all packages, that use Guile to
build more than two modules, no?

There could potentially be another workaround by synchronizing inside
guild, i.e. claiming a lock before reading and evaling any given source
file.  This would have the advantage of applying to all guile packages,
not just the ones that use guile-build-system, and it would also allow
non-Guile stuff, e.g. C extensions to be built in parallel.  WDYT?

Regards,
Leo
--- End Message ---

Re: Building Guile with ‘-j1’?

[Gábor Boskovits]

Hello,

Ludovic Courtès  ezt írta (időpont: 2021. jan. 20., Sze, 9:42):
>
> Hi!
>
> As the saying goes, “the cobbler’s children go barefoot”.  Guile/Guix
> are no exception since Guile builds are non-reproducible, despite work
> done a few years ago:
>
>   https://issues.guix.gnu.org/20272
>
> Until it’s fixed in Guile proper, what do you think of building Guile
> 2.0/2.2/3.0 with #:parallel-build? #f ?  We could do that in
> ‘core-updates’ now.
>
> That would work around the problem for Guile itself.  It would increase
> build times, but probably not that much since the most expensive part
> (compiling the first few files) is sequential anyway.  IIRC this is what
> Vagrant did for the Debian packages.
>
> We could also disable parallel builds in ‘guile-build-system’.  It’s
> only used for small packages so the extra build time is probably OK.
>
> Thoughts?

Let's go for it.

>
> Ludo’.
>

Regards,
g_bor
-- 
OpenPGP Key Fingerprint: 7988:3B9F:7D6A:4DBF:3719:0367:2506:A96C:CF63:0B21

Re: Building Guile with ‘-j1’?

[zimoun]

Hi Ludo,

On Wed, 20 Jan 2021 at 09:41, Ludovic Courtès  wrote:

> Until it’s fixed in Guile proper, what do you think of building Guile
> 2.0/2.2/3.0 with #:parallel-build? #f ?  We could do that in
> ‘core-updates’ now.

[...]

> We could also disable parallel builds in ‘guile-build-system’.  It’s
> only used for small packages so the extra build time is probably OK.

Good idea.


One question: will “guix pull” be slowed?  (if it makes sense)


Cheers,
simon

Building Guile with ‘-j1’?

[Ludovic Courtès]

Hi!

As the saying goes, “the cobbler’s children go barefoot”.  Guile/Guix
are no exception since Guile builds are non-reproducible, despite work
done a few years ago:

  https://issues.guix.gnu.org/20272

Until it’s fixed in Guile proper, what do you think of building Guile
2.0/2.2/3.0 with #:parallel-build? #f ?  We could do that in
‘core-updates’ now.

That would work around the problem for Guile itself.  It would increase
build times, but probably not that much since the most expensive part
(compiling the first few files) is sequential anyway.  IIRC this is what
Vagrant did for the Debian packages.

We could also disable parallel builds in ‘guile-build-system’.  It’s
only used for small packages so the extra build time is probably OK.

Thoughts?

Ludo’.