Re: [luatex] This is LuaHBTeX, Version 1.12.0 -- lualatex.fmt was written by luatex

[Knut Petersen]

Hi Norbert,

After updating my openSuSE Tumbleweed system I found lualatex broken. "lualatex somefile" led to an error message:  LuaHBTeX, Version 1.12.0 complained that "lualatex.fmt was written by luatex". Removing lualatex.fmt did not fix the problem. 

My guess is you have a format for luahbtex hanging around in below your $HOME. 
If you would have shown the actual log file, one could see it.


There definitely was no old format file in the cwd or somewhere else in or 
below $HOME, it really were outdated fmtutil.cnf / lualatex.fmt in the texmf 
tree. I agree that an error in the update process is a case for a bugreport to 
openSuSE.
But I was surprised that fixing fmtutil.cnf and removing the obsolete format 
file did not trigger the creation of a new format file , so I removed the 
lualatex.fmt file and executed lualatex again:

   knut@golem:~/foo> sudo rm /var/lib/texmf/web2c/lualatex/lualatex.fmt
   [sudo] Passwort für root:
   knut@golem:~/foo> rm TRACE*; strace -v -f -ff -s 8192 -o TRACE lualatex 
Teilnehmer20200627.tex
   This is LuaHBTeX, Version 1.12.0 (TeX Live 2020/TeX Live for SUSE Linux)
 restricted system commands enabled.

   kpathsea: Running mktexfmt lualatex.fmt
   mktexfmt [ERROR]: -user mode but path setup is -sys type, bailing out.
   I can't find the format file `lualatex.fmt'!
   knut@golem:~/foo>

The trace shows that"/usr/bin/mktexfmt" was used with the single option 
"lualatex.fmt".

Let's summarize: The engine fails if there is a broken or outdated format file.
The engine tries to recover if it does not find format file,  but that fails 
too.
I would suggest to actually use a command that generates a format file in a 
place that normally is writable and searched first in those cases.

   fmtutil --no-engine-subdir --fmtdir=.  --byfmt lualatex.fmt'

seems to be a better idea than

   '/usr/bin/mktexfmt lualatex.fmt'.

Knut

[luatex] This is LuaHBTeX, Version 1.12.0 -- lualatex.fmt was written by luatex

[Knut Petersen]

Hi everybody!

After updating my openSuSE Tumbleweed system I found lualatex broken. "lualatex somefile" 
led to an error message:  LuaHBTeX, Version 1.12.0 complained that "lualatex.fmt was written 
by luatex".

Removing lualatex.fmt did not fix the problem.

"fmtutil --all" did not fix the problem.

Then I changed the lualtex entry in /etc/texmf/web2c/fmtutil.cnf  to "lualatex lua*hb*tex 
language.dat,language.dat.lua lualatex.ini" and started "fmtutil --all" again. That 
fixed the problem, but I suspect that I only found a quick workaround.

Is this a general TeXLive 2020 problem or an openSuSE bug?

Knut

[luatex] pdf inclusion: extended font handling

[Knut Petersen]

Hi everybody!

I often write documents that mix text and music, a short example is given below:

   \NeedsTeXFormat{LaTeX2e}
   \documentclass[12pt]{letter}
   \usepackage{geometry}
   \geometry{papersize={14.85cm,21cm},noheadfoot,nomarginpar,
  left=1.5cm,right=1.5cm,top=1.5cm,bottom=1.5cm}
   \usepackage{fontspec}
   \defaultfontfeatures{Ligatures=TeX}
   \setmainfont{cmunrm.otf}[BoldFont=cmunbx.otf,
 ItalicFont=cmunti.otf,
 BoldItalicFont=cmunbi.otf]
   \usepackage{lipsum}
   \usepackage{lyinla}
   \begin{document}
   \pagestyle{empty}
   \small\lipsum[1]
   {\bfseries O du fröhliche}\hspace{\fill}EG 44, 1-3\par
   \begin{lilyfrag}[staffsize=16.666,width=0.92\textwidth,smalllatexfont]
  \include "deutsch.ly"
    \relative a' {
  \key d \major
  a2 h a4. g8 fis4 (g) \breathe a2 h a4. g8 fis4 (g) \breathe \break
  a2 2 h cis4 d cis2 h a1
    }
    \addlyrics {
  O du fröh -- li -- che, __ o du se -- li -- ge, __
  gna -- den -- brin -- gen -- de Weih -- nachts -- zeit!
    }
   \end{lilyfrag}
   \lipsum[2]
   \end{document}

The example above is intended to be translated with lualatex --shell-escape.

lyinla.sty provides the necessary code for the lilyfrag environement.
The text inside of a lilyfrag environment is extended and passed to the lilypond
music engraver. The result is a number of pdfs without any embedded
fonts, these pdf are included using \includegraphics. At the end of the lualatex
run (triggered by the stop_run callback) ghostscript is used to postprocess
the pdf generated by lualatex. A bit if postscript code provides the fonts to
ghostscript, the result is a final pdf that contains only one copy of every font
used in all the included pdfs, not one copy per usage of \includegraphics.

Often the mechanism pointed out above reduces file size by more than 80 percent.

Unfortunately recent changes to ghostscript (commit 04a517f39cc) broke
that mechanism, and it's not the first time that changes to gs broke my code.
I'd prefer to have a lualatex (or pdflatex/xelatex) that would be able to detect
usage of unembedded normal and CID fonts in included pdfs and to embed
exactly one subsetted copy of all needed fonts in the output pdf.


Is there already a solution somewhere in the bit bucket?

If not: Do you think it is possible to extend lua/xe/pdflatex with reasonable 
effort?

Knut

Re: [luatex] finish_pdffile callback

[Knut Petersen]

Am 09.02.2018 um 15:07 schrieb Br. Samuel Springuel:

I almost always use synctex, so that's a personal possibility, but I'm also 
programming for others who might not.

What about the `stop_run` callback?  Are the files closed by the time the final 
message is being spit out?


No, it is not. But everything really relevant is written at that point. It is possible to copy the file, complete it using some shell commands, and to process that copy.  See: http://tug.org/pipermail/luatex/2017-December/006692.html for an example. Luatex 1.04 does not complain if you delete the 
output pdf during the stop_run callback, so it is possible to reuse the original filename. Be aware that the stop_run callback is not used in draftmode.


In http://tug.org/pipermail/luatex/2017-December/006690.html I proposed a place 
for a pdf_closed callback.

Knut

Re: [luatex] First question: Retrieve *family* name of a font / the three main families

[Knut Petersen]

Am 19.01.2018 um 09:37 schrieb Urs Liska:

I'm contributing to a LuaLaTeX package (https://github.com/jperon/lyluatex) 
that uses an external program (LilyPond) to generate images to be included in 
the LaTeX document. I need to pass the used fonts to LilyPond, and I didn't 
manage to retrieve the necessary information in Lua.

What I need is

a) which families are currently set as roman, sans and mono fonts
b) the "family" name of these. [...]


So you want to teach lilypond to use the fonts latex uses.

In a similar project I use this code fragment to get information about the 
current TeX font and to pass it to lilypond:

  \else
    {
  \mdseries\@llftmp
  \immediate\write18{
    LANG=c
    FONT=`echo '\fontname\font'
    | sed -e 's/"//g'
    | sed -e 
's/\@esc[\@esc([[:print:]]*\@esc)\@esc][[:print:]]*/\@esc1/'`
    ;
    otfinfo --info `kpsewhich $FONT`
    | grep "Full name:"
    | sed -e 's/Full 
name:[[:blank:]]*\@esc([[:print:]]*\@esc)/\@esc\@esc
  def \@esc\@esc fontfullname\@cbopen\@esc 1\@cbclose/'
    > ourfont.tex}
  \input "ourfont.tex"
  \@imwrtof{\@ht(define lyinlaFontName "\fontfullname")}
  \@imwrtof{\@ht(define lyinlaFontSize (absFontSize \f@size))}
    }
  \fi


\@imwrtof is defined to immediately write to an output file that will be 
processed by lilypond,
\@esc gives a literal "\",
\@ht gives a literal "#",
\@cbopen gives a literal "{",
\@cbclose gives a  literal "}"

Obviously it does not compile, it requires --shell-escape, and it is not 
OS-independent. Bt it might be a starting point for you.

The idea of the code is to take \fontname\font,
to process it as shown,
to get the font filename from kpsewhich,
and to use otfinfo to get the needed information.

The output of otfinfo is filtered and written to a TeX file,
that file is read after return from the shell,
and two scheme expression (for fontname and fontsize) are
written to a file that later will be processed by lilypond.

Knut

Re: [luatex] Is there a hook that could be used by a pdf postprocessor?

[Knut Petersen]

Am 11.12.2017 um 21:27 schrieb Hans Hagen:



A "stat lyInLatex.pdf" after luatex finished shows that 40 bytes are not 
written at the time of callback execution.


Which doesn't mean that there cannot be something written via lua scripts. 
Also, there is be more to be wrapped up: the log file, file recording, closing 
of the synctex file, etc.



Yes. In currently stop_run is documented as cited below:

   8.5.3 stop_run
   function()
   end
   This callback replaces the code that prints LuaTEX’s statistics and ‘output 
written to’ messages.

In 2012  Patrick Gundlach discussed on this list if the sentence "The output file is not complete at this point and thus you cannot use this callback to post process the output file."  should be added to the documentation. The subject that 
stop_run is not executed in draftmode, the subject of a postprocessing hook and that stop_run cannot be used for that purpose also came up on stackexchange. It might be a good idea to extend the documentation, I propose:


   8.5.3 stop_run
   function()
   end
   This callback replaces the code that prints LuaTEX’s statistics and ‘output 
written to’ messages.
   It is not executed in draftmode, and the output file is not complete at the 
point of execution.
   Thus you cannot use this callback to post process the output file.


Anyway, managing your workflow can best be done with a wrapper (make was 
already suggested but many other solutions are possible).


Anything might be done using wrappers, but it's a pitty that some tasks 
currently need a wrapper. A few examples where a pdf_closed callback would be 
usefull:

 * I do not consider it to be good programming practice to remove temporary 
files that are still opened by the lualatex process, but I would like to remove 
those files from within the .sty file. [Ok, it is possible to remove files used 
by \includegraphics in the stop_run callback. They are still
   open, but they will not be used anymore and the library will probably 
silently handle the situation. But is this good programming practice?]
 * I take the pdf produced by *TeX, split it into pages and combine those pages 
and an audio recording to produce a final x.264 video. That could be done from 
within the .sty file, but with current luatex the best I can do is to write a 
script from within the style and typeout a message that tells
   the user to execute that script. [There is a hackish solution at the end of 
the mail]
 * If \includegraphics is used to include a lot of pdfs with embedded fonts the 
pdfs produced by luatex and other tex engines often produce big files that 
waste a lot of disk space because of duplicated font data. A solution is to 
include pdfs without embedded fonts and to post process the output
   with ghostscript. [There is a hackish solution at the end of the mail]

So there really would be reasonable use for a pdf_closed callback

   8.5.x pdf_closed
   function()
   end
   This callback is executed immediately after the output pdf has been finished 
and closed.
   It might be used topost process the output file.

In 2012 Taco Hoekwater wrote 
:

   At this point I am not sure any more but there definitely is a use case
   for a callback that runs after the pdf has been closed already. I am not
   sure whether we should change stop_run or create a new callback.


Pre- or postprocessing as part of the run is not part of the concept. What 
works for you can fail for someone else as timing is very application specific.


I don't understand your last sentence about timing, but I hope that you rethink 
about the implementation of an additional pdf_closed() callback.

Now here is the hackish solution to the problem that works on my system. I really don't like it, but it works here. Obviously it depends on the availability of a unix environment and the tools used, no error handling is implemented. The unfinished pdf is copied and fixed, mutool is used to obtain 
the correct xref offset. After that tmp.pdf should be identical to the final pdf and might be used as desired.


   \def\tmpScriptName{tmpfixpdf.sh}
   \newwrite\tmpScript
   \immediate\openout\tmpScript=\tmpScriptName
   \immediate\write\tmpScript{cp \jobname.pdf tmp.pdf}
   \immediate\write\tmpScript{echo endstream>> tmp.pdf}
   \immediate\write\tmpScript{echo endobj>> tmp.pdf}
   \immediate\write\tmpScript{echo startxref>> tmp.pdf}
   \immediate\write\tmpScript{\unexpanded{echo $((`mutool show tmp.pdf xref 
2>/dev/null
    | grep :
    | tail -1
    | sed -e 's/[[:digit:]]*: 
0*\([[:digit:]]*\)
   [[:print:]]*/\1/'` + 1))
    >> tmp.pdf}}
   \immediate\write\tmpScript{echo \%\%EOF>> tmp.pdf}
   

Re: [luatex] Is there a hook that could be used by a pdf postprocessor?

[Knut Petersen]

Am 11.12.2017 um 15:56 schrieb David Shourabi Porcel:

Hi!

If it's possible to carry out the postprocessing you need from within
LuaTeX instead of with ghostscript, the callback finish_pdffile might
be an option.


finish_pdffile is called very early.

Unfortunately even the stop_run callback is executed before the complete pdf 
has been written and closed:

   \directlua {
  function my_stop_run()
    os.execute("stat lyInLatex.pdf")
  end
  luatexbase.add_to_callback('stop_run', my_stop_run, 'TEST')
   }

A "stat lyInLatex.pdf" after luatex finished shows that 40 bytes are not 
written at the time of callback execution.

Let's have a look at the source code:

   void finish_pdf_file(PDF pdf, int luatexversion, str_number luatexrevision)
   {
   [...]
   if (total_pages == 0) {
  [...]
    } else {
    if (pdf->draftmode == 0) {
   [...]
    } else {
    if (callback_id > 0) {
    run_callback(callback_id, "->");
    }
    }
    libpdffinish(pdf);
    if (pdf->draftmode == 0)
    close_file(pdf->file);
   *%**
   **    % I think that pdf_closed callback called at this place would 
be a good idea!**
   **    %*
    else
    normal_warning("pdf backend","draftmode enabled, not changing output 
pdf");
    }

Knut

[luatex] Is there a hook that could be used by a pdf postprocessor?

[Knut Petersen]

Hi everybody!

I include a lot of pdfs without embedded fonts and use ghostscript (together 
with extractpdfmark if necessary) as a pdf postprocessor to insert one 
optimized subset of every font instead of one subset of every font for every 
included pdf.

Is there a hook that would allow a program to be executed after \jobname.pdf 
has been closed?

Knut

Re: [luatex] lualatex / includegraphics / open files

[Knut Petersen]

Am 04.12.2017 um 09:50 schrieb Hans Hagen:

i have no clue what your code is supposed to be maybe something


The code is part of an interface used to embed lilypond code in TeX documents.
One example:

   \begin{lilyfrag}{10 cm}{18 cm}
    \relative { c'4 d e f g a b c \bar "\." }
   \end{lilyfrag}

generates and includes a pdf that shows a c major scale.



\def\oneline{Grüße}

\unexpanded\expandafter{\oneline}



Yes, that does work. Thanks.

so far my experience with tex is that there's always a solution 


That sounds like an invitation to present some unsolved problems ;-)

Knut

Re: [luatex] lualatex / includegraphics / open files

[Knut Petersen]

Am 03.12.2017 um 17:59 schrieb David Carlisle:

On 3 December 2017 at 12:52, Hans Hagen  wrote:

...


Hans,
sounds quite reasonable, thanks, once it's there we'll try to expose
it via the existing  latex image inclusion options so
\includegraphics[keepopen=false]{..}
or something (details to be decided:-)

Yes. Thanks in advance.

Knut

Re: [luatex] lualatex / includegraphics / open files

[Knut Petersen]

Am 03.12.2017 um 10:19 schrieb Knut Petersen:


[...]  and/or does anyone know a solution of the "temorarily disable inputenc" 
problem?


I found out how to temporarily disable inputenc:

   \usepackage[utf8]{inputenc}
   % \stopUTFeight temporarily disables inputenc to handle 2-byte utf8 codes
   % \restartUTFeight temporarily reenables inputencs handling of 2-byte utf8 
codes
   % \savedUTFviii contains the original definition of \UTFviii@two@octets
   \makeatletter
   \let\savedUTFviii\UTFviii@two@octets
   \def\stopUTFeight{\gdef\UTFviii@two@octets##1##2{##1\string##2}}
   \def\restartUTFeight{\global\let\UTFviii@two@octets\savedUTFviii}
   \makeatother

Of course that does not handle the longer codes, but adding those is easy if 
they are needed.

Knut

Re: [luatex] lualatex / includegraphics / open files

[Knut Petersen]

Am 01.12.2017 um 15:44 schrieb Ulrike Fischer:

This has also already been discussed in tex.sx

https://tex.stackexchange.com/questions/258127/why-lualatex-gives-erorr-too-many-open-files-but-pdflatex-does-not-on-same-lat


Thanks for that link.

I tested pdflatex with a 100.000 lines input and found that pdftex correctly 
included all the 14.000+ pdfs it was expected to include.

So

 * pdflatex works correctly,
 * xelatex breaks after 5.000 included pdfs,
 * lualatex breaks after whatever the soft limit for file descriptors is set to.

luatex is the only engine that keeps all included pdfs open, and this is known 
since at least 2015.
I think this is a serious regression compared to pdflatex.

Increasing the fd limit beyond 1024 does not help in  my case (see this message on 
bug-lilypond 
).
I could use pdflatex ... if I would know how to temporarily disable and 
reenable inputenc.
Lilypond is a music engraving program, it expects utf8 input. I use something 
like

   \NeedsTeXFormat{LaTeX2e}
   [...]
   \usepackage[utf8]{inputenc} % Needed if pdflatex is used
   [ ...
   Code to implement the lilyfrag environment
   that
  - writes the content of a lilyfrag environment to a lilypond source 
file,
  - executes lilypond using write18 and
  - includes the pdfs produced by lilypond
   ...]
   \begin{document}
   [...]
   \begin{lilyfrag}{10 cm}{18 cm}
  % needs to be written to an utf8 encoded file
    \relative c'{ c d e f g a h c }
    \addlyrics { utf8 Lyrics }
   \end{lilyfrag}
   [...]
   \end{document}

Is there any chance that lualatex will be fixed soon and/or does anyone know a solution 
of the "temorarily disable inputenc" problem?

Knut

[luatex] lualatex / includegraphics / open files

[Knut Petersen]

Hi everybody!

Assume that the files testa.pdf, testb.pdf and testc.pdf exist.

Further assume there is a file mwe.tex:

   \NeedsTeXFormat{LaTeX2e}
   \documentclass[12pt]{letter}
   \usepackage{shellesc,graphicx}
   \begin{document}
   \includegraphics{testa.pdf}\newpage
   \includegraphics{testb.pdf}\newpage
   \includegraphics{testc.pdf}\newpage
   \immediate\write18{ lsof | grep latex | grep test | less -S }
   \end{document}

If (on a linux system) you execute

   lualatex --shell-escape mwe


and

   xelatex --shell-escape mwe

you see a difference: the included pdf files are open if you use
lualatex, they are _not_open_ when xelatex is used.

That difference is not a problem if you want to include only a few
files, but it is a problem if you want to include a few thousands
of pdfs - on most sytems there is a limit of about 1024 open files.

That limit can be raised, but at least in my case that does
not really help: using write18 to start a program when many file
descriptors are open might expose a bug in the c++ library if "many"
is "more than 1023". If that child uses execve() to start another child,
that execve fails with a "*** buffer overflow detected ***" message.

Is this is a problem of luatex? I don't know, but I think it is.

Keeping all files that are included by \includegraphics open is a
wasteful use of limited system resources (file descriptors) , and xetex
demonstrates that it is not necessary.

BTW: In my tests xelatex fails to include more than 5005 pdfs correctly.
Pdfs 5005+ are read,  white space is reserved, but the actual content of
the included pdfs is missing. Nothing unusual is written to the log.

System used:

 * Linux, 64bit (i4790K, 32GB)
 * OpenSuSE Tumbleweed
 * LuaTeX, Version 1.0.4 (TeX Live 2017/TeX Live for SUSE Linux)
 * XeTeX 3.14159265-2.6-0.8 (TeX Live 2017/TeX Live for SUSE Linux)


Knut