[luatex] lualatex / includegraphics / open files

[Akira Kakuto]

Dear Knut,


BTW: In my tests xelatex fails to include more than 5005
pdfs correctly.


This is a bug, which I introduced years ago, in xdvipdfmx.
I had fixed the bug this year. Please wait for the TL 2018.

Best,
Akira

Re: [luatex] lualatex / includegraphics / open files

[Ulrike Fischer]

Am Fri, 1 Dec 2017 15:18:50 +0100 schrieb luigi scarso:

> Under linux bash (if you have the rights) you can rise
> the maximum number of open file descriptors with ulimit -n

This has also already been discussed in tex.sx

https://tex.stackexchange.com/questions/258127/why-lualatex-gives-erorr-too-many-open-files-but-pdflatex-does-not-on-same-lat


-- 
Ulrike Fischer 
http://www.troubleshooting-tex.de/

Re: [luatex] lualatex / includegraphics / open files

[Patrick Gundlach]

>> wasteful use of limited system resources (file descriptors) , and xetex
>> demonstrates that it is not necessary.
> I have tested with 1 jpg and png, no problem.

the problem is only with PDF. This is an old limitation, already present in 0.79

(I have read your explanation)

Patrick

Re: [luatex] lualatex / includegraphics / open files

[luigi scarso]

On Fri, Dec 1, 2017 at 1:49 PM, Knut Petersen  wrote:
> Hi everybody!
>
> Assume that the files testa.pdf, testb.pdf and testc.pdf exist.
>
> Further assume there is a file mwe.tex:
>
> \NeedsTeXFormat{LaTeX2e}
> \documentclass[12pt]{letter}
> \usepackage{shellesc,graphicx}
> \begin{document}
> \includegraphics{testa.pdf}\newpage
> \includegraphics{testb.pdf}\newpage
> \includegraphics{testc.pdf}\newpage
> \immediate\write18{ lsof | grep latex | grep test | less -S }
> \end{document}
>
> If (on a linux system) you execute
>
> lualatex --shell-escape mwe
>
>
> and
>
> xelatex --shell-escape mwe
>
> you see a difference: the included pdf files are open if you use
> lualatex, they are _not_open_ when xelatex is used.
>
> That difference is not a problem if you want to include only a few
> files, but it is a problem if you want to include a few thousands
> of pdfs - on most sytems there is a limit of about 1024 open files.
>
> That limit can be raised, but at least in my case that does
> not really help: using write18 to start a program when many file
> descriptors are open might expose a bug in the c++ library if "many"
> is "more than 1023". If that child uses execve() to start another child,
> that execve fails with a "*** buffer overflow detected ***" message.
>
> Is this is a problem of luatex? I don't know, but I think it is.
>
> Keeping all files that are included by \includegraphics open is a
> wasteful use of limited system resources (file descriptors) , and xetex
> demonstrates that it is not necessary.
I have tested with 1 jpg and png, no problem.
In case of a pdf image, it can be that the same reference is used several times
--- e.g  the pdf can have more pages, or you can operate  with
clipping  on the same pdf etc ---
so internally the reference is kept alive until the end of run.
Under linux bash (if you have the rights) you can rise
the maximum number of open file descriptors with ulimit -n
 --- after ulimit -12000 I have tested with  1 pdf and  it works .
Under Windows 10 you can run a test with 25000 pdf without problem
(bash on windows has the same limit of 1024)

-- 
luigi

Re: [luatex] lualatex / includegraphics / open files

[luigi scarso]

On Fri, Dec 1, 2017 at 1:49 PM, Knut Petersen  wrote:
> Hi everybody!
>
> Assume that the files testa.pdf, testb.pdf and testc.pdf exist.
>
> Further assume there is a file mwe.tex:
>
> \NeedsTeXFormat{LaTeX2e}
> \documentclass[12pt]{letter}
> \usepackage{shellesc,graphicx}
> \begin{document}
> \includegraphics{testa.pdf}\newpage
> \includegraphics{testb.pdf}\newpage
> \includegraphics{testc.pdf}\newpage
> \immediate\write18{ lsof | grep latex | grep test | less -S }
> \end{document}
>
> If (on a linux system) you execute
>
> lualatex --shell-escape mwe
>
>
> and
>
> xelatex --shell-escape mwe
>
> you see a difference: the included pdf files are open if you use
> lualatex, they are _not_open_ when xelatex is used.
>
> That difference is not a problem if you want to include only a few
> files, but it is a problem if you want to include a few thousands
> of pdfs - on most sytems there is a limit of about 1024 open files.
>
> That limit can be raised, but at least in my case that does
> not really help: using write18 to start a program when many file
> descriptors are open might expose a bug in the c++ library if "many"
> is "more than 1023". If that child uses execve() to start another child,
> that execve fails with a "*** buffer overflow detected ***" message.
>
> Is this is a problem of luatex? I don't know, but I think it is.
>
> Keeping all files that are included by \includegraphics open is a
> wasteful use of limited system resources (file descriptors) , and xetex
> demonstrates that it is not necessary.
>
> BTW: In my tests xelatex fails to include more than 5005 pdfs  correctly.
> Pdfs 5005+ are read,  white space is reserved, but the actual content of
> the included pdfs is missing. Nothing unusual is written to the log.
>
> System used:
>
> Linux, 64bit (i4790K, 32GB)
> OpenSuSE Tumbleweed
> LuaTeX, Version 1.0.4 (TeX Live 2017/TeX Live for SUSE Linux)
> XeTeX 3.14159265-2.6-0.8 (TeX Live 2017/TeX Live for SUSE Linux)
>
>
> Knut
>
Thank you for the report, we are testing.




-- 
luigi

[luatex] lualatex / includegraphics / open files

[Knut Petersen]

Hi everybody!

Assume that the files testa.pdf, testb.pdf and testc.pdf exist.

Further assume there is a file mwe.tex:

   \NeedsTeXFormat{LaTeX2e}
   \documentclass[12pt]{letter}
   \usepackage{shellesc,graphicx}
   \begin{document}
   \includegraphics{testa.pdf}\newpage
   \includegraphics{testb.pdf}\newpage
   \includegraphics{testc.pdf}\newpage
   \immediate\write18{ lsof | grep latex | grep test | less -S }
   \end{document}

If (on a linux system) you execute

   lualatex --shell-escape mwe


and

   xelatex --shell-escape mwe

you see a difference: the included pdf files are open if you use
lualatex, they are _not_open_ when xelatex is used.

That difference is not a problem if you want to include only a few
files, but it is a problem if you want to include a few thousands
of pdfs - on most sytems there is a limit of about 1024 open files.

That limit can be raised, but at least in my case that does
not really help: using write18 to start a program when many file
descriptors are open might expose a bug in the c++ library if "many"
is "more than 1023". If that child uses execve() to start another child,
that execve fails with a "*** buffer overflow detected ***" message.

Is this is a problem of luatex? I don't know, but I think it is.

Keeping all files that are included by \includegraphics open is a
wasteful use of limited system resources (file descriptors) , and xetex
demonstrates that it is not necessary.

BTW: In my tests xelatex fails to include more than 5005 pdfs correctly.
Pdfs 5005+ are read,  white space is reserved, but the actual content of
the included pdfs is missing. Nothing unusual is written to the log.

System used:

 * Linux, 64bit (i4790K, 32GB)
 * OpenSuSE Tumbleweed
 * LuaTeX, Version 1.0.4 (TeX Live 2017/TeX Live for SUSE Linux)
 * XeTeX 3.14159265-2.6-0.8 (TeX Live 2017/TeX Live for SUSE Linux)


Knut