Re: compiling guix is too slow?

[Chris Marusich]

"Feng Shu"  writes:

> Now I have found that 'guix pull' is too slow,
> I need 3 hours to compile guix, is it possible to speed it ?

3 hours is longer than it usually takes on my Lenovo X200 laptop with 2
cores, an SSD, and 8 GB of RAM.  Usually, for me, it seems to take about
15 minutes.

I once tried to speed up 'guix pull' by offloading to another machine
with comparable specs.  Although I did manage to set up offloading
successfully, the result was that 'guix pull' with offloading was many
times slower than 'guix pull' without offloading.  In fact, it was so
slow, I gave up before it completed.  Instead, I just ran 'guix pull'
without offloading.

-- 
Chris


signature.asc
Description: PGP signature

Re: compiling guix is too slow?

[Alex Vong]

Alex Kost  writes:

> Alex Vong (2017-07-04 16:16 +0800) wrote:
>
> [...]
>> Is there an intermediate approch? Is is possible for Emacs to signal a
>> warning when the required features are not found, but still continue
>> loading the rest of the config?
>
> If you wish to have a message in *Messages* buffer, you can do it like
> this:
>
> (unless (require 'foo nil t)
>   (message "Error during loading 'foo'!!!"))
>
> or like this:
>
> (with-demoted-errors "%S" (require 'foo))
>
> If you want to have a warning in a pop-up buffer, then:
>
> (unless (require 'foo nil t)
>   (display-warning 'oops "Error during loading 'foo'!!!"))
>
OK.

> [...]
>> Thanks for the suggestion! It turns out setting NIX_STATE_DIR to
>> "/usr/local/var/guix/" and making "~/.config/guix/latest" pointing to
>> "~/scm/guix/" completely resolves the problem. So now only a oneliner is
>> required:
F>>
>>   (require 'guix-autoloads '() t)
>
> Actually, why do you need this line at all?  Such autoloads should be
> loaded automatically.  Do you also require autoloads of other emacs
> packages?  Do you use Emacs from Guix or from your operating system (not
> GuixSD)?
>
Indeed, even that line is not needed. I think that was needed when
Emacs-Guix was still part of Guix.

>> Do you think it is a bug that Emacs-Guix does not set
>> `guix-state-directory' correctly?
>
> I'm not sure what you mean.  It sets this variable to "/var/guix" by
> default (or to NIX_STATE_DIR if you use it).  If you use Guix with a
> non-standard state directory, how can Emacs-Guix define it?
>
> BTW, don't you have problems using "/usr/local/var" as the state
> directory?
>
It works on my laptop so far...

> If you use guix from a git checkout, I think it is a standard practice
> to configure it with "--localstatedir=/var".  It is even mentioned in
> the manual a couple of times:
>
>   (info "(guix) Building from Git")
>   (info "(guix) Requirements")

Hmm, I am unaware of this practice. I'll re-install Guix with
"--localstatedir=/var".

Indeed, the manual mentions that, "LOCALSTATEDIR is the state directory
specified via ‘--localstatedir’ at configure time, usually ‘/var’"
in:

  (info "(guix) The Store")


signature.asc
Description: PGP signature

Re: compiling guix is too slow?

[Alex Kost]

Alex Vong (2017-07-04 16:16 +0800) wrote:

[...]
> Is there an intermediate approch? Is is possible for Emacs to signal a
> warning when the required features are not found, but still continue
> loading the rest of the config?

If you wish to have a message in *Messages* buffer, you can do it like
this:

(unless (require 'foo nil t)
  (message "Error during loading 'foo'!!!"))

or like this:

(with-demoted-errors "%S" (require 'foo))

If you want to have a warning in a pop-up buffer, then:

(unless (require 'foo nil t)
  (display-warning 'oops "Error during loading 'foo'!!!"))

[...]
> Thanks for the suggestion! It turns out setting NIX_STATE_DIR to
> "/usr/local/var/guix/" and making "~/.config/guix/latest" pointing to
> "~/scm/guix/" completely resolves the problem. So now only a oneliner is
> required:
>
>   (require 'guix-autoloads '() t)

Actually, why do you need this line at all?  Such autoloads should be
loaded automatically.  Do you also require autoloads of other emacs
packages?  Do you use Emacs from Guix or from your operating system (not
GuixSD)?

> Do you think it is a bug that Emacs-Guix does not set
> `guix-state-directory' correctly?

I'm not sure what you mean.  It sets this variable to "/var/guix" by
default (or to NIX_STATE_DIR if you use it).  If you use Guix with a
non-standard state directory, how can Emacs-Guix define it?

BTW, don't you have problems using "/usr/local/var" as the state
directory?

If you use guix from a git checkout, I think it is a standard practice
to configure it with "--localstatedir=/var".  It is even mentioned in
the manual a couple of times:

  (info "(guix) Building from Git")
  (info "(guix) Requirements")

-- 
Alex

Re: compiling guix is too slow?

[ng0]

Ricardo Wurmus transcribed 1.6K bytes:
> 
> ng0  writes:
> 
> > I'd say we split up:
> >
> > 2.0 MiB and bigger:
> > python (3.7 MiB go)
> > perl   (2.3 MiB go)
> > haskell(2.1 MiB go)
> > bioinformatics (2.0 MiB go)
> 
> Modules named after languages would probably benefit from splitting
> them.  A *lot* of software is written in those languages, and we ought
> to move packages to where they make sense (e.g. web, documentation,
> admin, etc).
> 
> I disagree on splitting up “bioinformatics”; it’s already a “topic
> module”.  It would be hard to split by field (e.g. “RNA-seq”), because
> many tools cover a wide range.  I’m against some arbitrary split, which
> would give us something silly like “(gnu packages bioinformatics a)”,
> “(gnu packages bioinformatics b)”, etc.
> 
> > And those we can consider to split up as they are bigger than 1MiB
> > and smaller than 2MiB:
> > statistics (1.4 MiB go)
> > gnome  (1.4 MiB go)
> > xorg   (1.3 MiB go)
> > emacs  (1.2 MiB go)
> > web(1.2 MiB go)
> > ruby   (1.1 MiB go)
> 
> The only module I’d split from this list is “ruby” (for the same reasons
> as above).  “statistics” will change as soon as we get started with the
> much bigger “cran” module — but that’s going to cause more problems, not
> solve them :)
> 
> Anyway, Andy has already identified a problem with the compilation, so
> I’d defer any work on these other modules.  Independent of how this
> goes, however, (gnu packages python) ought to be split up.
> 
> --
> Ricardo
> 
> GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
> https://elephly.net

Okay.

I would like to be able to run guix pull on servers with
512 MB RAM again, any way this happens at some point is
okay for me. If it is only splitting up python plus
solving the problem with compilation, that's okay.
-- 
ng0
GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588
GnuPG: https://n0is.noblogs.org/my-keys
https://www.infotropique.org https://krosos.org


signature.asc
Description: PGP signature

Re: compiling guix is too slow?

[Alex Vong]

Alex Kost  writes:

> Alex Vong (2017-06-29 12:28 +0800) wrote:
>
>> "Feng Shu"  writes:
>>
>>> Now I have found that 'guix pull' is too slow,
>>> I need 3 hours to compile guix, is it possible to speed it ?
>>
>> Maybe you can try building from git instead? I used to run
>> '$ guix pull && guix package --upgrade', but it gets slower as the
>> number of packages of guix increases. So now I use
>> '$ git pull && make -j`nproc` && ./pre-inst-env guix package --upgrade'.
>> You can read the manual[0] for more info.
>>
>> If you use emacs-guix, you need to tell emacs-guix the location of your
>> git repository as well. I am unware of how others do it. Here is how I
>> do it:
>>
>>   (require 'guix-autoloads)
>
> I always recommend avoid such "hard" requiring, it is better to use
> (require 'foo nil t) instead.  With (require 'foo), your emacs config
> will be really fragile: once 'foo' feature will disappear (renamed or
> 'foo' package will be uninstalled, etc.), Emacs will fail loading your
> config!  So I would use (require 'guix-autoloads nil t) instead.
>
>>   (guix-prettify-global-mode)
>
> For the same reason, I never call functions from external packages in my
> emacs config: what if this function disappear one day? (I don't mean I'm
> going to remove it).  If I need to load something on Emacs start, I add
> it to 'after-init-hook'.  If it is going to fail, at least it will fail
> after your config will be fully loaded.
>
Is there an intermediate approch? Is is possible for Emacs to signal a
warning when the required features are not found, but still continue
loading the rest of the config?

>>   (setq-default guix-current-profile
>> (file-chase-links "~/.guix-profile" 1))
>
> Hm, I don't think this is needed.  Doesn't Emacs-Guix work with the
> default 'guix-current-profile' setting?
>
>>   (require 'guix-build-config)
>>   (let ((guix-src-dir (expand-file-name "~/scm/guix/")))
>> (setq-default guix-config-image-directory guix-src-dir)
>> (setq-default guix-config-guix-scheme-compiled-directory guix-src-dir))
>
> I have never had a need to set these variables.  Setting image directory
> is definitely not needed (if the default value doesn't work for you,
> it's a bug).
>
> As for the directory with Guix compiled files, although you can set it,
> it is not really intended for customizing.  Do you have
> "~/.config/guix/latest" point to "~/scm/guix"?  Or do you add
> "~/scm/guix" to GUILE_LOAD_PATH and GUILE_LOAD_COMPILED_PATH?  Or do you
> add it to 'geiser-guile-load-path'?  If anything of this is true, then
> you don't need to set that variable.

Thanks for the suggestion! It turns out setting NIX_STATE_DIR to
"/usr/local/var/guix/" and making "~/.config/guix/latest" pointing to
"~/scm/guix/" completely resolves the problem. So now only a oneliner is
required:

  (require 'guix-autoloads '() t)

Do you think it is a bug that Emacs-Guix does not set
`guix-state-directory' correctly?


signature.asc
Description: PGP signature

Re: compiling guix is too slow?

[ng0]

Ludovic Courtès transcribed 0.3K bytes:
> ng0  skribis:
> 
> > What about the sledgehammer approach:
> > Would it help if we just split the incredible long modules into
> > smaller modules like python-web, python-library, etc?
> 
> We can always do that, yes (and it’s probably a good idea independently
> of this.)

I'd say we split up:

2.0 MiB and bigger:
python (3.7 MiB go)
perl   (2.3 MiB go)
haskell(2.1 MiB go)
bioinformatics (2.0 MiB go)

And those we can consider to split up as they are bigger than 1MiB
and smaller than 2MiB:
statistics (1.4 MiB go)
gnome  (1.4 MiB go)
xorg   (1.3 MiB go)
emacs  (1.2 MiB go)
web(1.2 MiB go)
ruby   (1.1 MiB go)

games.go and linux.go are border cases for consideration.

> Any volunteer to get started?
> 
> Ludo’.
> 
> 

-- 
ng0
GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588
GnuPG: https://n0is.noblogs.org/my-keys
https://www.infotropique.org https://krosos.org


signature.asc
Description: PGP signature

Re: compiling guix is too slow?

[Ludovic Courtès]

ng0  skribis:

> What about the sledgehammer approach:
> Would it help if we just split the incredible long modules into
> smaller modules like python-web, python-library, etc?

We can always do that, yes (and it’s probably a good idea independently
of this.)

Any volunteer to get started?

Ludo’.

Re: compiling guix is too slow?

[Ludovic Courtès]

Andy Wingo  skribis:

> On Sat 01 Jul 2017 15:33, l...@gnu.org (Ludovic Courtès) writes:
>
>> Leo Famulari  skribis:
>>
>>> My understanding is that Guile 2.2 traded slower compilation for faster
>>> execution of compiled programs. Hopefully the performance of the
>>> compiler will be improved in later updates to Guile.
>>
>> Yes, that’s a good summary.
>>
>> Most of the code we compile is package definitions that don’t benefit
>> from all the fancy optimizations, so build-aux/build-self.scm (run by
>> ‘guix pull’) turns those off.
>>
>> Unfortunately even with this change compilation is slower than with
>> Guile 2.0, and IMO unacceptably slow for ‘guix pull’ (it takes only a
>> few minutes at most on my laptop, but that’s a lot if we compare to
>> ‘apt-get update’.)
>>
>> Andy, do you have other approaches in mind that we could explore to
>> improve on this?  Evaluating all of gnu/packages instead of compiling it
>> AOT would probably be bad for memory and CPU consumption.
>
> I agree that this is a correct summary.  I think it's possible that even
> with improvements that 2.2 would remain slower than 2.0 to compile, but
> the current situation is clearly not good and needs improvement.  I
> think we need to revisit that synthetic python.scm test you had (can't
> find it atm), see where the compiler is spending time, and focus effort
> there.

It’s here:
.

This one was focusing on memory consumption but there’s obviously a
connection between the two.

I’ll see if I can at least provide you with more precise data and
benchmarks so we have an idea of where the problem lies.

Thank you,
Ludo’.

Re: compiling guix is too slow?

[ng0]

Andy Wingo transcribed 1.5K bytes:
> On Sat 01 Jul 2017 15:33, l...@gnu.org (Ludovic Courtès) writes:
> 
> > Leo Famulari  skribis:
> >
> >> My understanding is that Guile 2.2 traded slower compilation for faster
> >> execution of compiled programs. Hopefully the performance of the
> >> compiler will be improved in later updates to Guile.
> >
> > Yes, that’s a good summary.
> >
> > Most of the code we compile is package definitions that don’t benefit
> > from all the fancy optimizations, so build-aux/build-self.scm (run by
> > ‘guix pull’) turns those off.
> >
> > Unfortunately even with this change compilation is slower than with
> > Guile 2.0, and IMO unacceptably slow for ‘guix pull’ (it takes only a
> > few minutes at most on my laptop, but that’s a lot if we compare to
> > ‘apt-get update’.)
> >
> > Andy, do you have other approaches in mind that we could explore to
> > improve on this?  Evaluating all of gnu/packages instead of compiling it
> > AOT would probably be bad for memory and CPU consumption.
> 
> I agree that this is a correct summary.  I think it's possible that even
> with improvements that 2.2 would remain slower than 2.0 to compile, but
> the current situation is clearly not good and needs improvement.  I
> think we need to revisit that synthetic python.scm test you had (can't
> find it atm), see where the compiler is spending time, and focus effort
> there.
> 
> As far as what Guix can do -- there I don't know :/  Nothing immediate
> comes to mind.
> 
> I would like to help here but can't promise much over the summertime...
> 
> Andy
> 
> 

What about the sledgehammer approach:
Would it help if we just split the incredible long modules into
smaller modules like python-web, python-library, etc?
-- 
ng0
GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588
GnuPG: https://n0is.noblogs.org/my-keys
https://www.infotropique.org https://krosos.org


signature.asc
Description: PGP signature

Re: compiling guix is too slow?

[Andy Wingo]

On Sat 01 Jul 2017 15:33, l...@gnu.org (Ludovic Courtès) writes:

> Leo Famulari  skribis:
>
>> My understanding is that Guile 2.2 traded slower compilation for faster
>> execution of compiled programs. Hopefully the performance of the
>> compiler will be improved in later updates to Guile.
>
> Yes, that’s a good summary.
>
> Most of the code we compile is package definitions that don’t benefit
> from all the fancy optimizations, so build-aux/build-self.scm (run by
> ‘guix pull’) turns those off.
>
> Unfortunately even with this change compilation is slower than with
> Guile 2.0, and IMO unacceptably slow for ‘guix pull’ (it takes only a
> few minutes at most on my laptop, but that’s a lot if we compare to
> ‘apt-get update’.)
>
> Andy, do you have other approaches in mind that we could explore to
> improve on this?  Evaluating all of gnu/packages instead of compiling it
> AOT would probably be bad for memory and CPU consumption.

I agree that this is a correct summary.  I think it's possible that even
with improvements that 2.2 would remain slower than 2.0 to compile, but
the current situation is clearly not good and needs improvement.  I
think we need to revisit that synthetic python.scm test you had (can't
find it atm), see where the compiler is spending time, and focus effort
there.

As far as what Guix can do -- there I don't know :/  Nothing immediate
comes to mind.

I would like to help here but can't promise much over the summertime...

Andy

Re: compiling guix is too slow?

[Alex Kost]

Alex Vong (2017-06-29 12:28 +0800) wrote:

> "Feng Shu"  writes:
>
>> Now I have found that 'guix pull' is too slow,
>> I need 3 hours to compile guix, is it possible to speed it ?
>
> Maybe you can try building from git instead? I used to run
> '$ guix pull && guix package --upgrade', but it gets slower as the
> number of packages of guix increases. So now I use
> '$ git pull && make -j`nproc` && ./pre-inst-env guix package --upgrade'.
> You can read the manual[0] for more info.
>
> If you use emacs-guix, you need to tell emacs-guix the location of your
> git repository as well. I am unware of how others do it. Here is how I
> do it:
>
>   (require 'guix-autoloads)

I always recommend avoid such "hard" requiring, it is better to use
(require 'foo nil t) instead.  With (require 'foo), your emacs config
will be really fragile: once 'foo' feature will disappear (renamed or
'foo' package will be uninstalled, etc.), Emacs will fail loading your
config!  So I would use (require 'guix-autoloads nil t) instead.

>   (guix-prettify-global-mode)

For the same reason, I never call functions from external packages in my
emacs config: what if this function disappear one day? (I don't mean I'm
going to remove it).  If I need to load something on Emacs start, I add
it to 'after-init-hook'.  If it is going to fail, at least it will fail
after your config will be fully loaded.

>   (setq-default guix-current-profile
> (file-chase-links "~/.guix-profile" 1))

Hm, I don't think this is needed.  Doesn't Emacs-Guix work with the
default 'guix-current-profile' setting?

>   (require 'guix-build-config)
>   (let ((guix-src-dir (expand-file-name "~/scm/guix/")))
> (setq-default guix-config-image-directory guix-src-dir)
> (setq-default guix-config-guix-scheme-compiled-directory guix-src-dir))

I have never had a need to set these variables.  Setting image directory
is definitely not needed (if the default value doesn't work for you,
it's a bug).

As for the directory with Guix compiled files, although you can set it,
it is not really intended for customizing.  Do you have
"~/.config/guix/latest" point to "~/scm/guix"?  Or do you add
"~/scm/guix" to GUILE_LOAD_PATH and GUILE_LOAD_COMPILED_PATH?  Or do you
add it to 'geiser-guile-load-path'?  If anything of this is true, then
you don't need to set that variable.

-- 
Alex

Re: compiling guix is too slow?

[Ludovic Courtès]

Hi,

Leo Famulari  skribis:

> My understanding is that Guile 2.2 traded slower compilation for faster
> execution of compiled programs. Hopefully the performance of the
> compiler will be improved in later updates to Guile.

Yes, that’s a good summary.

Most of the code we compile is package definitions that don’t benefit
from all the fancy optimizations, so build-aux/build-self.scm (run by
‘guix pull’) turns those off.

Unfortunately even with this change compilation is slower than with
Guile 2.0, and IMO unacceptably slow for ‘guix pull’ (it takes only a
few minutes at most on my laptop, but that’s a lot if we compare to
‘apt-get update’.)

Andy, do you have other approaches in mind that we could explore to
improve on this?  Evaluating all of gnu/packages instead of compiling it
AOT would probably be bad for memory and CPU consumption.

Thanks,
Ludo’.

Re: compiling guix is too slow?

[Alex Vong]

"Feng Shu"  writes:

> Now I have found that 'guix pull' is too slow,
> I need 3 hours to compile guix, is it possible to speed it ?

Maybe you can try building from git instead? I used to run
'$ guix pull && guix package --upgrade', but it gets slower as the
number of packages of guix increases. So now I use
'$ git pull && make -j`nproc` && ./pre-inst-env guix package --upgrade'.
You can read the manual[0] for more info.

If you use emacs-guix, you need to tell emacs-guix the location of your
git repository as well. I am unware of how others do it. Here is how I
do it:

  (require 'guix-autoloads)
  (guix-prettify-global-mode)
  (setq-default guix-current-profile
(file-chase-links "~/.guix-profile" 1))
  (setq-default guix-directory "~/scm/guix/")

  (require 'guix-build-config)
  (let ((guix-src-dir (expand-file-name "~/scm/guix/")))
(setq-default guix-config-image-directory guix-src-dir)
(setq-default guix-config-guix-scheme-compiled-directory guix-src-dir))


[0]: https://www.gnu.org/software/guix/manual/html_node/Building-from-Git.html


signature.asc
Description: PGP signature

Re: compiling guix is too slow?

[Leo Famulari]

On Thu, Jun 29, 2017 at 06:58:28AM +0800, Feng Shu wrote:
> 
> Now I have found that 'guix pull' is too slow,
> I need 3 hours to compile guix, is it possible to speed it ?

Wow, that's really slow :(

I don't know of any easy way to speed it up. Maybe you could build Guix
on a faster computer and use `guix copy` to move it to this slower
computer.

My understanding is that Guile 2.2 traded slower compilation for faster
execution of compiled programs. Hopefully the performance of the
compiler will be improved in later updates to Guile.

I'm curious, what machine are you using where it takes 3 hours? My
slowest machine takes 100 minutes, which is also too slow, but it's much
better than 3 hours.


signature.asc
Description: PGP signature

Re: compiling guix is too slow?

[Adonay Felipe Nogueira]

I'm also trying to understand the cause of this.

I still don't have enough time to begin investigating it or to suggest
improvements, but I notice that it gets worse and with *more* resource
usage if you pass the `--cores`/`-c` option with any value to `guix
pull`. Without this option the resource usage is short (in terms of how
long it lasts) and causes less heat.

-- 
- [[https://libreplanet.org/wiki/User:Adfeno]]
- Palestrante e consultor sobre /software/ livre (não confundir com
  gratis).
- "WhatsApp"? Ele não é livre, por isso não uso. Iguais a ele prefiro
  GNU Ring, ou Tox. Quer outras formas de contato? Adicione o vCard
  que está no endereço acima aos teus contatos.
- Pretende me enviar arquivos .doc, .ppt, .cdr, ou .mp3? OK, eu
  aceito, mas não repasso. Entrego apenas em formatos favoráveis ao
  /software/ livre. Favor entrar em contato em caso de dúvida.