bug#27217: texlive is too big

[Ludovic Courtès]

Pierre Neidhardt  skribis:

> Ricardo Wurmus  writes:
>
>>> So TeX Live packages would use ‘texlive-fetch’ instead of ‘svn-fetch’ or
>>> similar, right?
>>>
>>> If that’s the case, then merely computing the derivation of one of these
>>> would require fetching the tldb database, which would be problematic.
>>> Is this correct?
>>
>> I think there has been a misunderstanding here.  Pierre seems to be
>> using the name “texlive-fetch” for two different things.
>
> No, the "texlive-fetch" I meant was Ricardo's idea.  We are talking
> about the same thing.
>
>> The extraction of information from the plain text database (which is
>> included with “texlive-bin”) is done “offline” when using the importer,
>> i.e. when generating package definitions.
>
> Exactly, I think this is the part that Ludovic misunderstood.

OK, got it, thanks for explaining.

Ludo’.

bug#27217: texlive is too big

[Ricardo Wurmus]

Ludovic Courtès  writes:

>> I'm suggesting those following steps:
>>
>> - Write the package definition generator.  We can take inspiration from
>>   opam.scm, it seems to be well written and very similar.
>>   `texlive-fetch' is already aligned with `opam-fetch'.
>>
>>   For testing, we can try to recreate a simple package definition.
>>   texlive-latex-xcolor for instance.
>>   There is an immediate difference already: the name generated by the 
>> importer
>>   will be texlive-xcolor and not texlive-latex-xcolor.
>>
>> - Write texlive-fetch so that we can write definitions with multiple
>>   directories.
>>
>> - Update the texlive-build-system to accommodate the importer.  For 
>> instance, we
>>   will probably have to build fonts automatically.  We will also have to be
>>   smart about the files that need to be generated, and those that need to be 
>> discarded.
>>
>> - Write an updater (again, following opam.scm, should be trivial).
>>
>> I'll get started with the first step today.  Let me know what you think!
>
> So TeX Live packages would use ‘texlive-fetch’ instead of ‘svn-fetch’ or
> similar, right?
>
> If that’s the case, then merely computing the derivation of one of these
> would require fetching the tldb database, which would be problematic.
> Is this correct?

I think there has been a misunderstanding here.  Pierre seems to be
using the name “texlive-fetch” for two different things.  As it stands
it is not a source code downloader.  It is a procedure that extracts a
package’s information from the parsed plain text database.

The thing I proposed some days ago (also named “texlive-fetch”) would be
a variant of “svn-fetch” that supports more than one location in the
target SVN repository so that disjoint subsets of the SVN tree can be
downloaded at the same time.  This is unrelated.

The extraction of information from the plain text database (which is
included with “texlive-bin”) is done “offline” when using the importer,
i.e. when generating package definitions.

--
Ricardo

bug#27217: texlive is too big

[Ludovic Courtès]

Hello!

Pierre Neidhardt  skribis:

> Yesterday Jelle and I started getting down to it.
>
> I've created a wip-texlive branch.  For now, we can run
>
>   (texlive-fetch "xcolor") 
>
> and it returns the appropriate parsed entry from the tlpdb from texlive-bin.

Nice!

> The first call to texlive-fetch takes some 10-30 seconds because the tlpdb is
> some 11MB+ big.
>
> I'm caching the parsed DB, so that should not be a problem during development.

You can use ‘http-fetch/cached’, if that’s not what you’re already
doing.

> I'm suggesting those following steps:
>
> - Write the package definition generator.  We can take inspiration from
>   opam.scm, it seems to be well written and very similar.
>   `texlive-fetch' is already aligned with `opam-fetch'.
>   
>   For testing, we can try to recreate a simple package definition.
>   texlive-latex-xcolor for instance.
>   There is an immediate difference already: the name generated by the importer
>   will be texlive-xcolor and not texlive-latex-xcolor.
>
> - Write texlive-fetch so that we can write definitions with multiple
>   directories.
>   
> - Update the texlive-build-system to accommodate the importer.  For instance, 
> we
>   will probably have to build fonts automatically.  We will also have to be
>   smart about the files that need to be generated, and those that need to be 
> discarded.
>
> - Write an updater (again, following opam.scm, should be trivial).
>
> I'll get started with the first step today.  Let me know what you think!

So TeX Live packages would use ‘texlive-fetch’ instead of ‘svn-fetch’ or
similar, right?

If that’s the case, then merely computing the derivation of one of these
would require fetching the tldb database, which would be problematic.
Is this correct?

Thank you,
Ludo’.

bug#27217: texlive is too big

[Pierre Neidhardt]

Fantastic!

I have time to work on it this week, so let me know if you'd like some help :)

Can't wait to get this sorted once and for all :D

-- 
Pierre Neidhardt
https://ambrevar.xyz/


signature.asc
Description: PGP signature

bug#27217: texlive is too big

[Jelle Licht]

Pierre Neidhardt  writes:

> For posterity, here is a python importer for tlpdb:
>
> https://github.com/amaxwell/tlutility/blob/master/parse_tlpdb.py
>
> (Thanks to Henkjan for sharing this.)
>
> Jelle, did you work on the importer any further after FOSDEM?
> Can you share the file, I'm afraid I lost the link :p

As a matter of fact, I did! I am not at home with my guix-related work
right now. The short of it, is that I created a simple record type
(e.g. `tlpdb-package'). The tlpdb-parser we worked on at FOSDEM was able
to generate these structures from the tlpdb file, although translating
tlpdb-package records into actual guix packages was a step I did not
work on yet.

One of the problems I ran into was exactly what Ricardo described; the
fact that sources for texlive packages currently have a slight semantic
mismatch with what we have for most guix packages. If we have a solution
for this, the result of *any* of our importers would look a lot nicer.

bug#27217: texlive is too big

[Ricardo Wurmus]

Pierre Neidhardt  writes:

> What I meant is that with the tlpdb approach, we would base our build system 
> on
> a centralized authority with regard to what constitute a package and which 
> files
> belong to them.  Then we would not have to worry about file conflicts.

Ah, I see.  I was struggling to see how this would help us in this case,
so I looked up “palatino” in the texlive.tlpdb.

We can find this snippet in the texlive.tlpdb (which is part of the
texlive-bin package):

--8<---cut here---start->8---
name palatino
category Package
revision 31835
catalogue urw-base35
shortdesc URW "Base 35" font pack for LaTeX
longdesc A set of fonts for use as "drop-in" replacements for Adobe's
longdesc basic set, comprising: Century Schoolbook (substituting for
longdesc Adobe's New Century Schoolbook); Dingbats (substituting for
longdesc Adobe's Zapf Dingbats); Nimbus Mono L (substituting for Abobe's
longdesc Courier); Nimbus Roman No9 L (substituting for Adobe's Times);
longdesc Nimbus Sans L (substituting for Adobe's Helvetica); Standard
longdesc Symbols L (substituting for Adobe's Symbol); URW Bookman; URW
longdesc Chancery L Medium Italic (substituting for Adobe's Zapf
longdesc Chancery); URW Gothic L Book (substituting for Adobe's Avant
longdesc Garde); and URW Palladio L (substituting for Adobe's Palatino).
execute addMap upl.map
runfiles size=388
 texmf-dist/dvips/palatino/config.upl
 texmf-dist/fonts/afm/adobe/palatino/pplb8a.afm
 texmf-dist/fonts/afm/adobe/palatino/pplbi8a.afm
 texmf-dist/fonts/afm/adobe/palatino/pplr8a.afm
 texmf-dist/fonts/afm/adobe/palatino/pplri8a.afm
 texmf-dist/fonts/afm/urw/palatino/uplb8a.afm
 texmf-dist/fonts/afm/urw/palatino/uplbi8a.afm
 texmf-dist/fonts/afm/urw/palatino/uplr8a.afm
 texmf-dist/fonts/afm/urw/palatino/uplri8a.afm
 texmf-dist/fonts/map/dvips/palatino/upl.map
 texmf-dist/fonts/tfm/adobe/palatino/eurbo10.tfm
 texmf-dist/fonts/tfm/adobe/palatino/eurmo10.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplb.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplb7t.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplb8c.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplb8r.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplb8t.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplb9c.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplb9d.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplb9e.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplb9o.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplb9t.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplbc.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplbc7t.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplbc8t.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplbi.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplbi7t.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplbi8c.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplbi8r.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplbi8t.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplbi9c.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplbi9d.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplbi9e.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplbi9o.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplbi9t.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplbij8r.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplbj8r.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplbo.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplbo7t.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplbo8c.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplbo8r.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplbo8t.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplbu.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplbu8r.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplr.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplr7t.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplr8c.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplr8r.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplr8rn.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplr8t.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplr9c.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplr9d.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplr9e.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplr9o.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplr9t.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplrc.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplrc7t.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplrc8r.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplrc8t.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplrc9d.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplrc9e.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplrc9o.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplrc9t.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplri.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplri7t.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplri8c.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplri8r.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplri8t.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplri9c.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplri9d.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplri9e.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplri9o.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplri9t.tfm
 texmf-dist/fonts/tfm/adobe/palatino/pplrij8r.tfm
 texmf-dist/fonts/tfm/adobe

bug#27217: texlive is too big

[Ricardo Wurmus]

Ludovic Courtès  writes:

> Hello!
>
> Ricardo Wurmus  skribis:
>
>> Ludovic Courtès  writes:
>>
>>> Now I can’t find the ‘pplr’ font (actually Adobe’s Palatino).
>>
>> In the texmf-dist tree these fonts are found under
>> “fonts/{tfm,afm,vf/adobe/palatino”.  I don’t know if there are any
>> sources for generating these files, or if they are simply font blobs.
>
> Does that mean we don’t have a ‘texlive-’ package for these?

We don’t have one yet.  But we can add one similar to
“texlive-fonts-txfonts”.

--
Ricardo

bug#27217: texlive is too big

[Ricardo Wurmus]

Pierre Neidhardt  writes:

> And this highlights another big issue with our current build system: it's very
> hard to know which file is already packaged.  It could very well be that some
> texlive packages have conflicts simply because we've accidentally packaged the
> same files in different packages.

It’s not a problem of the build system.  The build system works fine for
ins/dtx files.

This is a problem with the TeX Live distribution.  It does not consist
of packages for the most part, but represents a directory tree that
is meant to be unpacked in place.

In some cases we use upstream packages as in the case of
“texlive-fonts-lm”, which provides a zip file that can be unpacked into
a texmf-dist tree.  In other cases all we can do is take the TeX Live
upstream sources from various SVN directories.

It sucks, but I don’t see how you could do much better given this
distribution method.

--
Ricardo

bug#27217: texlive is too big

[Pierre Neidhardt]

As far as I know, we don't, and I could not figure the package out (I could have
tried harder...).

And this highlights another big issue with our current build system: it's very
hard to know which file is already packaged.  It could very well be that some
texlive packages have conflicts simply because we've accidentally packaged the
same files in different packages.

-- 
Pierre Neidhardt
https://ambrevar.xyz/


signature.asc
Description: PGP signature

bug#27217: texlive is too big

[Ludovic Courtès]

Hello!

Ricardo Wurmus  skribis:

> Ludovic Courtès  writes:
>
>> Now I can’t find the ‘pplr’ font (actually Adobe’s Palatino).
>
> In the texmf-dist tree these fonts are found under
> “fonts/{tfm,afm,vf/adobe/palatino”.  I don’t know if there are any
> sources for generating these files, or if they are simply font blobs.

Does that mean we don’t have a ‘texlive-’ package for these?

Thanks,
Ludo’.

bug#27217: texlive is too big

[Ricardo Wurmus]

Ricardo Wurmus  writes:

> We may also want to
> support installation of individual packages by providing a profile hook
> (e.g. when a user only wants texlive-minimal with the gbrief package).

A profile hook has been added with commit
743497b5650713e082f4775a3b7dfd03babc8191.  Users may want to start by
installing “texlive-base” and then add packages to their profile as
needed.

I’m closing this bug now.

--
Ricardo

bug#27217: texlive is too big

[Ricardo Wurmus]

Ludovic Courtès  writes:

> Now I can’t find the ‘pplr’ font (actually Adobe’s Palatino).

In the texmf-dist tree these fonts are found under
“fonts/{tfm,afm,vf/adobe/palatino”.  I don’t know if there are any
sources for generating these files, or if they are simply font blobs.
In the latter case they would be installed just like the fonts provided
by “texlive-fonts-charter”.

--
Ricardo

bug#27217: texlive is too big

[Ricardo Wurmus]

Pierre Neidhardt  writes:

>> I don’t think that’s correct.  The .ins/.dtx files contain instructions
>> for generating files, including the .sty files, which are extracted from
>> the .dtx files.
>
> What I meant is that the .sty is always already generated in the SVN repo.

Yes, the SVN repo does include generated files.  That’s not great, but
we can choose to ignore those files (as we do in other projects).

-- 
Ricardo

bug#27217: texlive is too big

[Ricardo Wurmus]

Pierre Neidhardt  writes:

>> Yes, the SVN repo does include generated files.  That’s not great, but
>> we can choose to ignore those files (as we do in other projects).
>
> And the builder should take care not to discard .sty / .cls files that are not
> going to be generated.  I believe that in all cases the basename remains the
> same between .dtx/.ins and .sty/.cls.  Then it's easy to know which one to 
> keep and
> which one to discards.

Well, we do ignore the generated files already whenever we can.

> So what do you think of the tlpdb approach for a new tex-build-system?

I don’t know what exactly this entails.  I’m happy to see patches for
this.  Anything that improves the current state of TeXlive support in
Guix would be worth looking into.

--
Ricardo

bug#27217: texlive is too big

[Pierre Neidhardt]

> Yes, the SVN repo does include generated files.  That’s not great, but
> we can choose to ignore those files (as we do in other projects).

And the builder should take care not to discard .sty / .cls files that are not
going to be generated.  I believe that in all cases the basename remains the
same between .dtx/.ins and .sty/.cls.  Then it's easy to know which one to keep 
and
which one to discards.

So what do you think of the tlpdb approach for a new tex-build-system?

bug#27217: texlive is too big

[Pierre Neidhardt]

> I don’t think that’s correct.  The .ins/.dtx files contain instructions
> for generating files, including the .sty files, which are extracted from
> the .dtx files.

What I meant is that the .sty is always already generated in the SVN repo.
Can you find a counter-example?

> We have both kinds of packages in tex.scm.  Some where the .sty or .tex source
> files are copied to the target location and some where the .sty or .tex files
> are generated from the .ins/.dtx sources.

And some where it's a mix of both :p

> Whether a .sty or .tex file is a source file isn’t always obvious, but
> sometimes they mention that they are generated from other files.

> For some fonts the provided format *is* the source.

Hmm, maybe you are right and I'm wrong about this.

TeXlive has many formats: mf (source), afm, tfm, type1, several bitmap
formats...  I'm a bit suspicious about some fonts.  Lots of them are under
"other-free", while some are under "unknown".  Hmm...
Well, let's take care about this later.

bug#27217: texlive is too big

[Ricardo Wurmus]

Pierre Neidhardt  writes:

> TeXlive packages are provided "ready to use", they are not meant to be built.
> The .ins/.dtx are only here for potential package contributors or as a source 
> of
> documentation, but when it comes to TeXlive, they are not used to build the
> resulting package.  The .sty is (I think) always parachuted into the SVN
> repository as well.

I don’t think that’s correct.  The .ins/.dtx files contain instructions
for generating files, including the .sty files, which are extracted from
the .dtx files.

> (Actually, sometimes there is no .ins/.dtx, just a .sty.)

Correct.  For some “packages” there’s really just a .sty source file.
But often enough .sty files are generated.

We have both kinds of packages in tex.scm.  Some where the .sty or .tex source
files are copied to the target location and some where the .sty or .tex files
are generated from the .ins/.dtx sources.

Whether a .sty or .tex file is a source file isn’t always obvious, but
sometimes they mention that they are generated from other files.

> More worrisome: some fonts don't provide their source.

For some fonts the provided format *is* the source.

> In fact, some of them
> have confusing licenses, and since the source is missing, I wouldn't call that
> "free software".  But TeXlive is.  That's not very consistent and a lot of 
> FOSS
> TeXlive packages effectively depend on closed-source fonts.

I haven’t found any such cases yet.  Could you show us cases where the
font license makes the font non-free?

>> I don’t see this file in the texlive SVN repository.  Where is it
>> hosted?
>
> It's in Master/tlpkg/texlive.tlpdb.
> Or from CTAN:
> http://mirror.ctan.org/tex-archive/systems/texlive/tlnet/tlpkg/texlive.tlpdb.xz.

Ah, thanks.

-- 
Ricardo

bug#27217: texlive is too big

[Pierre Neidhardt]

> What I find most troubling is that sources are littered across the SVN
> repository.  Sometimes we’ve got simple .ins and .dtx files, but very
> often we have a stray .sty or .tex file in some seemingly arbitary
> directory and one needs to manually take care of adding these extra
> source files to the native-inputs.

I was puzzled like you, but last time I investigated it became a bit clearer:
there is simply no general recipe for building TeXlive packages.

TeXlive packages are provided "ready to use", they are not meant to be built.
The .ins/.dtx are only here for potential package contributors or as a source of
documentation, but when it comes to TeXlive, they are not used to build the
resulting package.  The .sty is (I think) always parachuted into the SVN
repository as well.

(Actually, sometimes there is no .ins/.dtx, just a .sty.)

More worrisome: some fonts don't provide their source.  In fact, some of them
have confusing licenses, and since the source is missing, I wouldn't call that
"free software".  But TeXlive is.  That's not very consistent and a lot of FOSS
TeXlive packages effectively depend on closed-source fonts.

What shall do then?

> I don’t see this file in the texlive SVN repository.  Where is it
> hosted?

It's in Master/tlpkg/texlive.tlpdb.
Or from CTAN:
http://mirror.ctan.org/tex-archive/systems/texlive/tlnet/tlpkg/texlive.tlpdb.xz.

> So, it’s a map of packages to file names?  That would probably simplify
> the importer.  I don’t think it would help with the build system.  Am I
> missing something?

I believe the tlpdb might actually be necessary to build every package reliably.

As I said above, TeXlive packages are not meant to be built, they are meant to
be copy/pasted, so it seems.  (Correct me if I'm wrong.)

Sure we can rebuild the .dtx/.ins, but that only works for some packages and it
does not suffice, we still need to include the extra files (e.g. fonts) if any.
We can only know this file list from the tlpdb.

So here is what I suggest: the texlive-build-system looks up the file list in
the tlpdb and copies everything.  If some of those files include .dtx/.ins,
it could build them (but that should not change anything since the .sty is
always provided).

Question: that would produce another .sty file beside the one existing
in the SVN.  Should we replace it?  Should we print a warning if it does not
match?

With such a build system, the only thing the importer would have to do is get
the synopsys/license information from the tlpdb.

Does that make sense?

bug#27217: texlive is too big

[Ricardo Wurmus]

Hey Pierre,

> I'm so sorry for the poor wording, Ricardo, it was uncalled for.  (Wrote in a
> haste.)  What I meant is that it does not solve the current issue of "what 
> file
> belongs to what package."

No worries :)

> It's not just the importer but our current approach to TeXlive that we've got 
> to
> work out.

I agree.  It’s always been messy and currently it’s pretty frustrating
to package or update TeXlive packages.

What I find most troubling is that sources are littered across the SVN
repository.  Sometimes we’ve got simple .ins and .dtx files, but very
often we have a stray .sty or .tex file in some seemingly arbitary
directory and one needs to manually take care of adding these extra
source files to the native-inputs.  This could be improved even before a
full overhaul of our TeXlive handling: add a convenience procedure that
takes a list of file names in the repository and collects them via SVN
as the source tree.  Beats having to add extra build phases and the
like.

>> : I'm more and more convinced that rewriting the texlive-build-system 
>> centered
>> : around texlive.tlpdb would work and is the right approach.
>>
>> Could you please outline what this would mean?
>
> Sure: if you look at the file, you'll see it's a textual database of all
> packages with their respective file.  A possible solution that we could
> implement either as a build-system or an importer: lookup the package in the 
> tlpdb
> (e.g. mflogo) and package all the corresponding file from the svn repo.  
> Sounds
> simple enough.  What do you think?

I don’t see this file in the texlive SVN repository.  Where is it
hosted?

So, it’s a map of packages to file names?  That would probably simplify
the importer.  I don’t think it would help with the build system.  Am I
missing something?

--
Ricardo

bug#27217: texlive is too big

[Ricardo Wurmus]

Hi Pierre,

> I think I know what's happening: some packages are missing files because they
> were not properly packaged.  With our current build system, this is almost
> impossible to fix.
>
> I've never been able to use the importer.  Anyways, as my previous email
> pointed out, I think it's quite useless as it is.

I wouldn’t say it’s useless.  Although I’m certainly prone to writing
useless code, this importer serves a need and allowed me to successfully
create the many texlive-* packages we have now.

I’ll admit, though, that it’s the one importer that I like the least :)

> I'm more and more convinced that rewriting the texlive-build-system centered
> around texlive.tlpdb would work and is the right approach.

Could you please outline what this would mean?

--
Ricardo

bug#27217: texlive is too big

[Ludovic Courtès]

Ricardo Wurmus  skribis:

> Ludovic Courtès  writes:
>
>> Hello!
>>
>> I was also looking at the remaining dependencies on ‘texlive’.  For
>> ‘teximpatient’, it looks like we’d need
>>  but:

[...]

>> Throw to key `srfi-34' with args `(#> arguments: ("checkout" "--non-interactive" "--trust-server-cert" "-r" 
>> "44591" 
>> "svn://mwww.tug.org/texlive/tags/texlive-2017.1/Master/texmf-dist/source/latex/mflogo-font"
>>  "/tmp/guix-directory.lVf6gO") exit-status: 1 term-signal: #f stop-signal: 
>> #f] 2219000>)'.
>
> So it fetches the description from ctan.org just fine, but then cannot
> download the directory from the SVN repository.
>
> That’s because there is no such directory in the SVN repository; see:
>
>   
> https://www.tug.org/svn/texlive/tags/texlive-2017.1/Master/texmf-dist/source/latex/
>
> There’s “mflogo”, but no “mflogo-font”.

“mflogo” appears to be a different thing though, it doesn’t contain the
font.  I found the ‘logo’ font in ‘texlive-fonts-knuth-lib’ though!

Now I can’t find the ‘pplr’ font (actually Adobe’s Palatino).

Ludo’.

bug#27217: texlive is too big

[Ricardo Wurmus]

Ludovic Courtès  writes:

> Hello!
>
> I was also looking at the remaining dependencies on ‘texlive’.  For
> ‘teximpatient’, it looks like we’d need
>  but:
>
> --8<---cut here---start->8---
> $ guix import texlive mflogo-font
> ni sekvas la redirektigon al 'https://ctan.org/xml/1.2/pkg/mflogo-font'...
> Backtrace:
>   10 (primitive-load "/home/ludo/.config/guix/current/bin/guix")
> In guix/ui.scm:
>   1644:12  9 (run-guix-command _ . _)
> In guix/scripts/import.scm:
>115:11  8 (guix-import . _)
> In guix/scripts/import/texlive.scm:
> 91:19  7 (guix-import-texlive . _)
> In guix/memoization.scm:
>  98:0  6 (_ # ("mflogo-font" "latex") _)
> In unknown file:
>5 (_ # 
> # #)
> In guix/import/texlive.scm:
>146:23  4 (sxml->package (*TOP* (entry (@ (id "mflogo-font")) (name 
> "mflogo-font") (caption "Metafont…") …)) _)
> In guix/utils.scm:
> 632:8  3 (call-with-temporary-directory # guix/svn-download.scm:92:3 (temp)>)
> In guix/svn-download.scm:
> 95:14  2 (_ "/tmp/guix-directory.lVf6gO")
> In guix/build/svn.scm:
>  38:2  1 (svn-fetch 
> "svn://www.tug.org/texlive/tags/texlive-2017.1/Master/texmf-dist/source/latex/mflogo-fo…"
>  …)
> In guix/build/utils.scm:
> 616:6  0 (invoke _ . _)
>
> guix/build/utils.scm:616:6: In procedure invoke:
> Throw to key `srfi-34' with args `(# arguments: ("checkout" "--non-interactive" "--trust-server-cert" "-r" "44591" 
> "svn://mwww.tug.org/texlive/tags/texlive-2017.1/Master/texmf-dist/source/latex/mflogo-font"
>  "/tmp/guix-directory.lVf6gO") exit-status: 1 term-signal: #f stop-signal: 
> #f] 2219000>)'.

So it fetches the description from ctan.org just fine, but then cannot
download the directory from the SVN repository.

That’s because there is no such directory in the SVN repository; see:

  
https://www.tug.org/svn/texlive/tags/texlive-2017.1/Master/texmf-dist/source/latex/

There’s “mflogo”, but no “mflogo-font”.

That said, this kind of common error should be caught by the importer.

--
Ricardo

bug#27217: texlive is too big

[Ludovic Courtès]

Hello!

I was also looking at the remaining dependencies on ‘texlive’.  For
‘teximpatient’, it looks like we’d need
 but:

--8<---cut here---start->8---
$ guix import texlive mflogo-font
ni sekvas la redirektigon al 'https://ctan.org/xml/1.2/pkg/mflogo-font'...
Backtrace:
  10 (primitive-load "/home/ludo/.config/guix/current/bin/guix")
In guix/ui.scm:
  1644:12  9 (run-guix-command _ . _)
In guix/scripts/import.scm:
   115:11  8 (guix-import . _)
In guix/scripts/import/texlive.scm:
91:19  7 (guix-import-texlive . _)
In guix/memoization.scm:
 98:0  6 (_ # ("mflogo-font" "latex") _)
In unknown file:
   5 (_ # 
# #)
In guix/import/texlive.scm:
   146:23  4 (sxml->package (*TOP* (entry (@ (id "mflogo-font")) (name 
"mflogo-font") (caption "Metafont…") …)) _)
In guix/utils.scm:
632:8  3 (call-with-temporary-directory #)
In guix/svn-download.scm:
95:14  2 (_ "/tmp/guix-directory.lVf6gO")
In guix/build/svn.scm:
 38:2  1 (svn-fetch 
"svn://www.tug.org/texlive/tags/texlive-2017.1/Master/texmf-dist/source/latex/mflogo-fo…"
 …)
In guix/build/utils.scm:
616:6  0 (invoke _ . _)

guix/build/utils.scm:616:6: In procedure invoke:
Throw to key `srfi-34' with args `(#)'.
ludo@ribbon ~/src/guix$ guix describe
Generacio 51Jan 08 2019 10:45:12(nuna)
  guix b509381
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: b50938108947d47c9e3d37608df83a11020a4d8d
--8<---cut here---end--->8---

WIP patch:

--- a/gnu/packages/tex.scm
+++ b/gnu/packages/tex.scm
@@ -4399,9 +4399,16 @@ develop documents with LaTeX, in a single application.")
(assoc-ref inputs "automake")
"^install-sh$"))
  srcdir)
-   (chdir srcdir)))
+   (chdir srcdir
+ (add-after 'unpack 'set-home
+   (lambda _
+ ;; XXX: Currently tex wants to generate fonts in
+ ;; ~/.texlive2018/texmf-var/fonts, hence this hack.
+ (setenv "HOME" (getcwd))
+ #t)
 (native-inputs
- `(("texlive" ,texlive)
+ `(("texlive" ,(texlive-union (list texlive-fonts-amsfonts
+texlive-fonts-ec)))
("automake" ,automake)))
 (home-page "https://www.gnu.org/software/teximpatient/";)
 (synopsis "Book on TeX, plain TeX and Eplain")

Ludo’.

bug#27217: texlive is too big

[Pierre Neidhardt]

I've dug into this a little deeper (sadly not too much, I don't think I'll have
time to work on this before a while).

Let me summarize the issue: packages can be fetched from two main resources: 

- the TeXlive subversion repository (what we are doing now) which bundles
  everything into a single massive folder.
  
- CTAN which distributes everything as separate packages.

The problem with CTAN is that it's not versioned and there is no "stable" URL
for the packages.

To paraphrase
https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/typesetting/tex/texlive/default.nix:

--8<---cut here---start->8---
  # Upstream refuses to distribute stable tarballs,
  # so we host snapshots on IPFS or on our own servers.
  # Common packages should get served from the binary cache anyway.
  # See discussions, e.g. https://github.com/NixOS/nixpkgs/issues/24683
  urlPrefixes = args.urlPrefixes or [
# A snapshot temporarily hosted by @xeji.
# TODO: remove when there is a reliable long-term solution
https://cat3.de/texlive-2018/tlnet/archive

# TODO: Add second, faster and more reliable snapshot mirror,
# maybe on one of our project's servers

# IPFS seeded by the mirror above - this may be quite slow

https://ipfs.io/ipfs/QmT4Z67wXin1Z9DhvqwSSkSZSuu8hT6LgDyMu6CBm9Tb7t/tlnet/archive

# The canonical source moves quickly and will be broken almost 
immediately
http://mirror.ctan.org/tex-archive/systems/texlive/tlnet/archive

# Should be stable for historic, archived releases
# 
http://ftp.math.utah.edu/pub/tex/historic/systems/texlive/2018/tlnet-final/archive
# TODO: use this later when 2018 is archived

  ];
--8<---cut here---end--->8---

The core question for us is: "Can we reconstruct a TeXlive package from the
subversion repository?"

Corollary: Was NixOS right to discard the repository as a source for packages?

If we can't reconstruct packages from the subversion repository, then our
current texlive-build-system is not very useful when centered around
texlive-ref.

Nix auto-generates all packages from this file:
http://mirror.ctan.org/tex-archive/systems/texlive/tlnet/tlpkg/texlive.tlpdb.xz.
This could be our redeemer :)

-- 
Pierre Neidhardt
https://ambrevar.xyz/


signature.asc
Description: PGP signature

bug#27217: texlive is too big

[Peter Neidhardt]

Also it seems that the texlive importer is broken:

--8<---cut here---start->8---
> guix import texlive fontspec
following redirection to `https://ctan.org/xml/1.2/pkg/fontspec'...
Backtrace:
  10 (apply-smob/1 #)
In ice-9/boot-9.scm:
705:2  9 (call-with-prompt _ _ #)
In ice-9/eval.scm:
619:8  8 (_ #(#(#)))
In guix/ui.scm:
  1535:12  7 (run-guix-command _ . _)
In guix/scripts/import.scm:
   114:11  6 (guix-import . _)
In guix/scripts/import/texlive.scm:
91:19  5 (guix-import-texlive . _)
In guix/memoization.scm:
 98:0  4 (_ # ("fontspec" "latex") _)
In unknown file:
   3 (_ # 
# #)
In guix/import/texlive.scm:
   157:25  2 (sxml->package (*TOP* (entry (@ (id "fontspec")) (name "fontspec") 
(# #) …)) _)
In guix/serialization.scm:
   270:25  1 (write-file #f # #:select? _)
In unknown file:
   0 (lstat #f)

ERROR: In procedure lstat:
Wrong type (expecting string): #
--8<---cut here---end--->8---

-- 
Peter Neidhardt

"How to make a million dollars:  First, get a million dollars."
-- Steve Martin


signature.asc
Description: PGP signature

bug#27217: texlive is too big

[Ricardo Wurmus]

Mathieu Lirzin  writes:

> Ricardo Wurmus  writes:
>
>> What’s missing?
>>
>> Currently, one needs to set a lot of environment variables to use these
>> separate packages.  For “fastcap” I needed to set “DVIPSHEADERS” (to
>> find “tex.pro”), “TFMFONTS” (for compiled metafont files), “TEXFORMATS”
>> (for the compiled “latex.fmt”), and “TEXINPUTS” (for all directories
>> containing tex source files).  Setting these variables manually is
>> really tedious.
>>
>> How can we set them automatically?  A simple idea is to provide a
>> procedure “texlive-union” that takes texlive packages and produces a
>> wrapped variant of the tools in “texlive-bin” that run inside an
>> environment where these variables are set.
>
> After reading, it is not clear to me why having all the environment
> variables set in “~/.guix-profile/etc/profile” is not good enough, or
> not possible?

IIRC the format of these variables is somewhat peculiar and does not
correspond to the way environment variables are commonly specified, so
we probably cannot use the search-path mechanism that Guix provides.
(For example, there is syntax for indicating that a directory is
supposed to be searched recursively.)

Instead we would generate a configuration file that contains all of the
environment variables and then only set the variables that are required
to look up this configuration file.  That’s roughly what “texlive-union”
does.  (It also builds a union directory, but we do this anyway when
building profiles, so that’s nothing special.)

--
Ricardo

bug#27217: texlive is too big

[Ricardo Wurmus]

Peter Neidhardt  writes:

> What's the current status on this?

No progress yet.  I’ve been meaning to work on this, but it’s pretty
tedious and demands concentration, which is a quality I haven’t been
able to maintain.

For most *packages* needing a subset of TeX live we can use
“texlive-union” as an input instead.  For user profiles, “texlive-union”
is not the right tool.

> What's the intended TeXlive installation process then?

Currently, the recommended way to get TeX live is to install the
huge “texlive” package.  “texlive-union” was only meant to be used as an
input in packages that require a subset of TeX live.  It cannot yet be
used in a profile.

> Let me know if I there is anything I can help with.

We need a profile hook that builds the configuration file and sets
environment variables whenever a texlive-* package is contained in a
profile.  The behaviour would be similar to “texlive-union”.

--
Ricardo

bug#27217: texlive is too big

[Peter Neidhardt]

What's the current status on this?

I've tried processing an Org letter
(http://orgmode.org/worg/exporters/koma-letter-export.html) and I get
the following error:

--8<---cut here---start->8---
(/gnu/store/m6ff2qqh0s1s8dy2ip5d757lmf5ai31x-texlive-tiny-44591/share/texmf-dis
t/tex/latex/graphics-def/pdftex.def)))

! LaTeX Error: File `grffile.sty' not found.

Type X to quit or  to proceed,
or enter new name. (Default extension: sty)

Enter file name:
! Emergency stop.


l.8 \usepackage
   {longtable}^^M
!  ==> Fatal error occurred, no output PDF file produced!
Transcript written on .//koma-letter-new-example.log.
--8<---cut here---end--->8---

I have installed some packages, among which

- texlive-bin
- Some texlive-latex-*
- texlive-latex-oberdiek (which contains grffile.sty)
- texlive-tiny

I noticed that texlive-latex-oberdiek is not in ~(texlive-union)~ (which is
used to generate texlive-tiny), so I suppose this is the root of the
issue.

Without texlive-tiny, pdflatex does not even start, failing on some
missing Perl module.

What's the intended TeXlive installation process then?
Would Guix provide a 1-to-1 map of the CTAN packages and it would be up
to the user to define their own texlive collection which calls
~(texlive-union ...)~ over a custom set of packages?

If so, I assume that koma-script does not have its own package yet.
Correct?

Let me know if I there is anything I can help with.

-- 
Pierre Neidhardt


signature.asc
Description: PGP signature

bug#27217: texlive is too big

[Mathieu Lirzin]

Ricardo Wurmus  writes:

> What’s missing?
>
> Currently, one needs to set a lot of environment variables to use these
> separate packages.  For “fastcap” I needed to set “DVIPSHEADERS” (to
> find “tex.pro”), “TFMFONTS” (for compiled metafont files), “TEXFORMATS”
> (for the compiled “latex.fmt”), and “TEXINPUTS” (for all directories
> containing tex source files).  Setting these variables manually is
> really tedious.
>
> How can we set them automatically?  A simple idea is to provide a
> procedure “texlive-union” that takes texlive packages and produces a
> wrapped variant of the tools in “texlive-bin” that run inside an
> environment where these variables are set.

After reading, it is not clear to me why having all the environment
variables set in “~/.guix-profile/etc/profile” is not good enough, or
not possible?

-- 
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

bug#27217: texlive is too big

[Ricardo Wurmus]

Currently, we offer the complete texlive distribution in a single
package that weighs several gigabytes.  This causes several problems:

* it’s much too heavy a dependency for packages
* it takes a long time to download
* it takes a long time to compress the substitute
* a user who only wants to compile a simple PDF needs to put up with
  installing the complete texlive distribution.

The purpose of this bug report is to keep track of the remaining issues
in splitting up texlive.

This is the current status (in my local branch):

* we have a texlive importer that fetches description and version info
  from CTAN but downloads from the texlive SVN.  This is because CTAN
  does not store versioned tarballs.

* we have a new texlive-build-system that can compile TeX packages in
  “.ins” + “.dtx” format

* the build system works fine for creating individual packages for the
  LaTeX required package set.

* these packages are sufficient to build the documentation of the
  package “fastcap” in DVI and PDF format

What’s missing?

Currently, one needs to set a lot of environment variables to use these
separate packages.  For “fastcap” I needed to set “DVIPSHEADERS” (to
find “tex.pro”), “TFMFONTS” (for compiled metafont files), “TEXFORMATS”
(for the compiled “latex.fmt”), and “TEXINPUTS” (for all directories
containing tex source files).  Setting these variables manually is
really tedious.

How can we set them automatically?  A simple idea is to provide a
procedure “texlive-union” that takes texlive packages and produces a
wrapped variant of the tools in “texlive-bin” that run inside an
environment where these variables are set.

It is also not clear how users should install texlive from countless
separate packages.  We should provide different sets of packages for
variants like texlive-minimal up to texlive-most.  We may also want to
support installation of individual packages by providing a profile hook
(e.g. when a user only wants texlive-minimal with the gbrief package).

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net