Re: performance issue with TeX Live

[Emmanuel Beffara]

De Maxim Cournoyer le 16/10/2023 à 15:46:
> My next question would be; how is this ls-R file produced in Guix?

That I do not know, but I would assume mktexlsr is called at some point when
building the profile.

> Does it matches the expectations of upstream for producing such file?

I don't know about those expectations, but after a quick look at the file in
Guix and in Debian, they look really similar. The only noticable difference is
that in Debian's file, the entries are sorted alphabetically in each folder
and the subfolders are browsed in alphabetical order.

-- 
Emmanuel

Re: performance issue with TeX Live

[Emmanuel Beffara]

Hello,

De Maxim Cournoyer le 14/10/2023 à 18:06:
> > I tried to explore this but I see no reason why the ls-R files would be
> > ignored and I don't know how to explore this further. I do want to 
> > contribute
> > to a solution, because right now texlive is practically unusable in Guix.
> 
> Have you tried tracing execution with strace?  Maybe try 'strace -e
> file -f -s800 $texlive-cmd' and confirm whether the ls-R file gets used;
> if it does, you could then introspect said file to see if it provides
> what is being looked up or not.

I did try strace, that is how I observed that apparently all installed files
are stated (with newfstatat) several times in each compilation.

The ls-R file is actually read:

$ grep /ls-R log
28308 
access("/run/current-system/profile/share/texmf-dist/../texmf-dist/ls-R", R_OK) 
= 0
28308 newfstatat(AT_FDCWD, 
"/run/current-system/profile/share/texmf-dist/../texmf-dist/ls-R", 
{st_mode=S_IFREG|0444, st_size=4053996, ...}, 0) = 0
28308 openat(AT_FDCWD, 
"/run/current-system/profile/share/texmf-dist/../texmf-dist/ls-R", O_RDONLY) = 3

but aparently its contents are not taken into account.

-- 
Emmanuel

Re: performance issue with TeX Live

[Emmanuel Beffara]

De Nicolas Goaziou le 09/10/2023 à 16:03:
> Emmanuel Beffara  writes:
> > I do want to contribute to a solution, because right now texlive is
> > practically unusable in Guix.
> 
> FWIW, I use modular TeX Live regularly, so "unusable" is probably a bit
> strong. However, I agree the current situation is frustrating.

Ok, "unusable" may be an overstatement. But when compiling a simple document
takes one or two minutes, it certainly changes the workflow (in particular,
adjusting a slideshow at the last minute becomes a stressful experience, this
is what triggered my message above…)

-- 
Emmanuel

Re: performance issue with TeX Live

[Emmanuel Beffara]

Hello,

De Nicolas Goaziou le 13/09/2023 à 14:39:
> Emmanuel Beffara  writes:
> 
> > I am facing a severe performance issue with TeX Live: compilation of any
> > document is an order of magnitude slower with a Guix installed system as
> > compared to a manual installation. Is anyone confronted to this phenomenon, 
> > or
> > is there a way to fix this ?
> 
> I also experienced a noticeable slowdown; I don't know how to fix it yet.
[...]
> A ls-R file is generated during profile creation (see
> `texlive-font-maps' function in "guix/profiles.scm") but it seems it is
> not read.
> 
> It may be interesting to compare location and contents of the ls-R files
> in both installations.

I tried to explore this but I see no reason why the ls-R files would be
ignored and I don't know how to explore this further. I do want to contribute
to a solution, because right now texlive is practically unusable in Guix.

-- 
Emmanuel

Re: performance issue with TeX Live

[Emmanuel Beffara]

De Olivier Dion le 12/09/2023 à 19:11:
> On Tue, 12 Sep 2023, Emmanuel Beffara  wrote:
> > Hello Guix devel,
> >
> > I am facing a severe performance issue with TeX Live: compilation of any
> > document is an order of magnitude slower with a Guix installed system as
> > compared to a manual installation. Is anyone confronted to this phenomenon, 
> > or
> > is there a way to fix this ?
> 
> I had a similar issue not long ago.  I think recent patches fix this and
> now it is smooth as before.

Apparently no. I did a pull and reconfigure and the behaviour is the same as I
described.

$ guix describe
Generation 53   Sep 12 2023 20:49:03(current)
  guix 4cb2e08
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: 4cb2e08b8eb1acc824b72ec17b483d7d85b5af68

Running my compilations in a pure environment with

guix shell --pure texlive-scheme-medium texlive-collection-...

leads to the same behaviour, and the same with --container. Not sure how to
investigate this further.

-- 
Emmanuel

performance issue with TeX Live

[Emmanuel Beffara]

Hello Guix devel,

I am facing a severe performance issue with TeX Live: compilation of any
document is an order of magnitude slower with a Guix installed system as
compared to a manual installation. Is anyone confronted to this phenomenon, or
is there a way to fix this ?

I suspect the problem comes from kpathsea, but I may be misinterpeting my
observations. The fact is that there is an enormous amount of file accesses
for any compilation.

As an experiment, I compared two installations.

- One is a pure Guix installation, managed with `guix home`, that contains a
  rather large collection of packages that I include through a custom package:

(define-public texlive-scheme-eb
  (package
(name "texlive-scheme-eb")
(version (number->string %texlive-revision))
(source #f)
(build-system trivial-build-system)
(arguments (list #:builder #~(mkdir #$output)))
(propagated-inputs
  (list texlive-scheme-medium
texlive-collection-fontsextra
texlive-collection-latexextra
texlive-collection-pictures))
(home-page "https://www.tug.org/texlive/;)
(synopsis "EB's custom installation scheme")
(description "This my a TeX Live scheme with what I use.")
(license (license:fsf-free 
"https://www.tug.org/texlive/copying.html;

- The other is manual, I set up a container with

 $ guix shell --container -F --network --share=$HOME/opt=/opt bash 
coreutils curl grep gzip ncurses perl sed tar wget

  in which I installed TeX Live using `install-tl` as found on
  , in the folder `/opt` exposed in the container.
  I installed the same package set by selecting the medium scheme in
  `install-tl` and installing the collections above with the provided `tlmgr`.

On my machine (a rather recent and powerful Dell Latitude 7410 with 16GiB RAM
and 12 cores), I observe the following timings:

- For a minimal LaTeX document (title, author and one word of body text)
  running `time pdflatex doc.tex` I get the following timings in the Guix
  version:

real0m4,288s
user0m3,140s
sys 0m1,148s

  and in the manual version:

real0m0.773s
user0m0.729s
sys 0m0.044s

- For a small beamer slideshow with nothing exotic (474 source lines, 17
  slides, no tikz graphics or anything), I get the following timings in the
  Guix version:

real1m0,122s
user0m14,337s
sys 0m44,279s

  and in the manual version:

real0m4.554s
user0m4.446s
sys 0m0.108s

I tried doing `strace` on the pdflatex calls to investigate further and it
appears that the behaviours of the two versions are largely different.
Counting the number of system calls of each kind gives the following, for the
most frequent calls:

- minimal document, Guix version:

   112860 newfstatat
10491 getdents64
 5247 openat
 5246 close
 4397 access
 3141 read

- minimal document, manual version:

 2772 read
   90 openat
   72 access
   64 close
   64 newfstatat
   60 getdents64

- slideshow, Guix version:

  2831722 getdents64
  1538560 newfstatat
  1498287 access
  1415296 openat
  1415295 close
 4283 read

- slideshow, Guix version:

 3913 read
 1288 getdents64
 1136 access
  960 openat
  925 close
  920 newfstatat

So apparently no file hash is used in the Guix version and a large part of the
`texmf-dist` folder is browsed, probably several times.

-- 
Emmanuel

Re: documentation in TeX Live collections

[Emmanuel Beffara]

Hello,

Thanks for the instructive feedback!

De Nicolas Goaziou le 28/08/2023 à 13:01:
> Every texlive-* package comes with its documentation, in a separate
> output. "doc" output are not uncommon at all in Guix. Therefore,
> I disagree with the inconsistency you're talking about.

Ok, I admit I didn't investigate much before asserting that! I just observed
that most of the tools I use come with their man or info pages in the main
output and extrapolated from that.

> > I think it would make much more sense to have "doc" outputs also for
> > collections and schemes. It would be consistent with the structure of
> > individual packages and would not require artificial packages.
> 
> I disagree. Collections are meta-packages. There is no documentation,
> nor content, attached to them. Moreover Guix meta-packages do nothing
> special about the documentation of packages they propagate. This would
> be inconsistent.

I don't understand how "out" and "doc" are different in this respect. The
"out" output of a collection meta-package has no content of its own and it
only serves to gather the "out" outputs of its inputs. Similarly, the "doc"
output would have no content of its own and only gather the "doc" outputs of
its inputs. How is that inconsistent?

There may be something I misunderstand about how Guix packages work here.

> > Having individual package documentations in one's manifests is of course
> > doable but it is contradictory with the approach of collections.
> 
> How so?

The point of a collection is to bring a meaningful set of packages on a
general topic without having to worry about its specific contents: the exact
list of packages may evolve from one version to another, it may contain many
things that are indirectly required, etc. I think it makes sense to be able to
request such an collection of packages with its documentation, and that should
not involve listing all the packages individually.

I realise it could also be possible to program that with something like

(map (lambda (p) (list p "doc"))
 (filter (lambda (p) (member "doc" (package-outputs p)))
 (map cadr (package-transitive-inputs
 texlive-scheme-medium

in the definition of a manifest but it feels a bit low-level. But it is nice
that doing this kind of thing possible, by the way!

> In any case, I suggest to write a proper bug report for this. Hopefully,
> someone with better understanding about the implications of GUIX_TEXMF
> will be able to solve this.

I can do that for the texdoc behaviour.

-- 
Emmanuel

Re: documentation in TeX Live collections

[Emmanuel Beffara]

Hellon

De Andreas Enge le 28/08/2023 à 20:05:
> if I understand things correctly, we would like the following behaviour
> for propagated inputs in the texlive context:
> We have these metapackages with propagated inputs; all of these inputs
> have "out" and "doc". Then we would like to automatically create "out"
> and "doc" for the metapackage, into which the corresponding "out" and
> "doc" of their "ingredients" are propagated.
> 
> Well, more precisely, the metapackages are empty, so it is a bit fuzzy
> what I mean by "into which" above.
> 
> We would like the following:
> - If a user installs metapackage:out, they get all the ingredient:out
>   in their profile.
> - If a user installs metapackage:doc, they get all the ingredient:doc
>   in their profile.
> I am quite certain this is not how propagated inputs work, and I do not
> know whether their behaviour could be changed in this way.

Indeed, that is what I imagine. Actually, it does not feel specific to the
texlive context and could apply to propagated inputs in general. 

[...]
> See also
>https://issues.guix.gnu.org/65550

Interesting! The case of development outputs referred to in this issue
suggests similar concerns.

> I do not really understand what is happening. All outputs are downloaded,
> but only the out outputs are propagated?

It does look like that: all outputs of the propagated inputs are downloaded
but only the "out" outputs are actually propagated to the profile. And indeed,
after guix gc, the other outputs are removed from the store. Is that the
intended behaviour?

> If this is true, then I think it would make sense to not split into two
> outputs, but to always include the documentation in the texlive packages.

I'm not sure about that. Splitting the documentation to a different output
does make sense in itself (for considerations of space, mostly). This thing
about propagating only "out" outputs looks more like an issue with package
definitions, or even the package model if propagating other outputs happens to
be unsupported.

-- 
Emmanuel

Re: documentation in TeX Live collections

[Emmanuel Beffara]

Hello,

De Nicolas Goaziou le 27/08/2023 à 12:13:
> > Apparently, all individual packages have a specific "doc" output but
> > collections and schemes do not and they don't have them as inputs either. So
> > we end up with an installation with no documentation
> 
> It sounds like a good default to me. I doubt anyone reads all
> documentation for every TeX Live package they install.

Yet, as far as I know, most packages in Guix (apart from texlive-* ones) come
with their documentation, so it feels somewhat inconsistent. But I agree that
not including the docs in the main outputs can make sense, especially given
the volume it represents. Anyway, given that there is extensive documentation
in TeX Live, it seems only natural to have it easily accessible.

> > I tried to explicity include documentation in a sub-shell but this changes
> > nothing:
> >
> > $ guix shell texlive-latex:doc -- texdoc inputenc
> 
> Note that you also need to install texlive-texdoc (or some collection/scheme 
> including it).

Indeed, but this command was run in a profile where texlive-scheme-medium is
present, so texdoc was already available. Adding texlive-texdoc to the guix
shell command does not change anything.

> > Apparently the fact that this GUIX_TEXMF variable contains several paths is
> > problematic for texdoc.
> 
> Would the following definition for texlive-texdoc solve both issues
> mentioned above? (the warning and the error.)
[...]
>(add-after 'link-scripts 'wrap-programs
>  (lambda _
>(wrap-program (string-append #$output "/bin/texdoc")
>  '("GUIX_TEXMF" = ("${GUIX_TEXMF%:*}"

It would certainly remove the warning but it would make only the first path
usable by texdoc, while other tools seem to support having several paths in
GUIX_TEXMF. Besides, I don't understand how GUIX_TEXMF is taken into account
by the various tools. Web2c and co don't know them, so there must be some
wrapping or patching somewhere in the package definitions?

> > Apparently something has triggered the download of documentation for all
> > packages `texlive-scheme-medium` depends on but only the one I explicitly
> > requested is made available in the profile (which is expected). All these
> > other documentation were downloaded but not used and `guix gc` actually
> > deletes them all!
> 
> I noticed that, too, but I don't have any explanation for it at the
> moment.

Is there a way to diagnose that kind of thing? I stumbled on a similar
behaviour in other contexts and was unable to investigate it (in my case, big
debug versions of Qt libraries are often downloaded, although I neved
requested any debugging version of anything).

> > So what would be the proper way to install `texlive-scheme-medium` in a home
> > profile with the documentation of the packages it includes ?
> 
> If that's a common request, we could add a `texlive-collection-foo-doc'
> package that would propagate all "doc" outputs from all packages
> included in `texlive-collection-foo'.
> 
> However, I'm a bit reluctant to add more artificial packages (i.e., not
> known to TeX Live distribution). Also, it might be as simple to do it in
> one's own manifest.

I think it would make much more sense to have "doc" outputs also for
collections and schemes. It would be consistent with the structure of
individual packages and would not require artificial packages.

Having individual package documentations in one's manifests is of course
doable but it is contradictory with the approach of collections.

-- 
Emmanuel Beffara

Re: documentation in TeX Live collections

[Emmanuel Beffara]

Hi,
 
De Nicolas Goaziou le 28/08/2023 à 20:01:
> Nicolas Goaziou  writes:
> >>> In any case, I suggest to write a proper bug report for this. Hopefully,
> >>> someone with better understanding about the implications of GUIX_TEXMF
> >>> will be able to solve this.
> >>
> >> I can do that for the texdoc behaviour.
> 
> By the way, I think I fixed `texdoc' utility. Would you care testing it
> after a fresh `guix pull'?

It seems to work fine: now texdoc does find all documentation in the various
items in GUIX_TEXMF. So there seems to be no need for a texdoc bug report
anymore, thanks! 

The only subtlety is that any call to guix shell with extra texlive packages
should also include texlive-bin (directly or not) so that the proper hooks are
called, otherwise GUIX_TEXMF is not updated. This goes beyond the point of
documentation, of course.

-- 
Emmanuel