On 2010-11-19 23:03, Khaled Hosny wrote:
> On Fri, Nov 19, 2010 at 09:23:46PM +0100, Pander wrote:
>> On 2010-11-19 21:15, Khaled Hosny wrote:
>>>> The other thing is the performance. Now it could be that the current
>>>> executable is not optimised and is doing loads of debug logging.
>>>> However, my font catalogue is a good test case.
>>>>
>>>> XeLaTeX can process it in two parts because other it runs out of memory.
>>>> However, LuaLaTeX is hitting some sort of performance barrier halfway my
>>>> document and after using up almost 6 GB of memory I have to kill off the
>>>> process. Please see attached screenshots of memory usage.
>>>
>>> LuaTeX will happily use all memory available to it (unlike most other
>>> TeX engines which have hard-coded memory limit, IIUC). The current font
>>> loading code is not very memory efficient and for certain fonts e.g.
>>> fonts with huge kerning tables (some font developers like to say their
>>> fonts have tens of thousands of kerning pairs, like if it is a feature
>>> or something). I think post-texlive2010 releases of luatex made some
>>> progress in this area, but I'm not sure if Hans started making use of it
>>> or not.
>>>
>>>> How can LuaLaTeX's performance be improved? Is the current executable in
>>>> TLContrib a debug build with maximum logging?
>>>
>>> I don't think so, I don't think it would be that relevant either. If you
>>> can isolate the most offending fonts it might help in identifying and
>>> possibly fixing the cause.
>>
>> It it difficult to isolate since gradually memory gets eaten. Can I run
>> it with profiling or logging to report which font is using how much memory?
>
> Here is a quickly hacked script that takes a list of fonts and reports,
> in kb, the memory used for each:
>
> for _,v in ipairs(arg) do
> local b, m, f, t
> b = string.explode(v, "/")
> b = b[#b]
> m = collectgarbage("count")
> f = fontloader.open(v)
> if f then
> t = fontloader.to_table(f)
> fontloader.close(f)
> print(math.floor(collectgarbage("count") - m), b)
> t = nil
> collectgarbage('collect')
> end
> end
>
> Save it to some file and cal it something like:
>
> $ texlua foo.lua `find /path/to/fonts`
There are some pretty big ones:
20208 CODE2001.ttf
37006 DoulosSILCR.ttf
37122 DoulosSILR.ttf
20479 DejaVuSerif-Bold.ttf
20479 DejaVuSerifCondensed-Bold.ttf
20480 DejaVuSerif-BoldItalic.ttf
20480 DejaVuSerifCondensed-BoldItalic.ttf
20550 DejaVuSerifCondensed.ttf
20550 DejaVuSerif.ttf
20558 DejaVuSerifCondensed-Italic.ttf
20558 DejaVuSerif-Italic.ttf
21014 DejaVuSansMono.ttf
21014 mplus-1m-bold.ttf
21014 mplus-1m-light.ttf
21014 mplus-1m-medium.ttf
21014 mplus-1mn-bold.ttf
21014 mplus-1mn-light.ttf
21014 mplus-1mn-medium.ttf
21014 mplus-1mn-regular.ttf
21014 mplus-1mn-thin.ttf
21014 mplus-1m-regular.ttf
21014 mplus-1m-thin.ttf
21014 mplus-2m-bold.ttf
21014 mplus-2m-light.ttf
21014 mplus-2m-medium.ttf
21014 mplus-2m-regular.ttf
21014 mplus-2m-thin.ttf
22583 DejaVuSans-BoldOblique.ttf
22583 DejaVuSansCondensed-BoldOblique.ttf
22638 DejaVuSans-Oblique.ttf
22639 DejaVuSansCondensed-Oblique.ttf
22657 mplus-1c-black.ttf
22657 mplus-1c-bold.ttf
22657 mplus-1c-heavy.ttf
22657 mplus-1c-light.ttf
22657 mplus-1c-medium.ttf
22657 mplus-1c-regular.ttf
22657 mplus-1c-thin.ttf
22657 mplus-1p-black.ttf
22657 mplus-1p-bold.ttf
22657 mplus-1p-heavy.ttf
22657 mplus-1p-light.ttf
22657 mplus-1p-medium.ttf
22657 mplus-1p-regular.ttf
22657 mplus-1p-thin.ttf
22687 mplus-2c-regular.ttf
22687 mplus-2p-black.ttf
22687 mplus-2p-bold.ttf
22687 mplus-2p-heavy.ttf
22687 mplus-2p-light.ttf
22687 mplus-2p-medium.ttf
22687 mplus-2p-regular.ttf
22687 mplus-2p-thin.ttf
22688 mplus-2c-black.ttf
22688 mplus-2c-bold.ttf
22688 mplus-2c-heavy.ttf
22688 mplus-2c-light.ttf
22688 mplus-2c-medium.ttf
22688 mplus-2c-thin.ttf
23970 DejaVuSans-Bold.ttf
24004 DejaVuSansCondensed-Bold.ttf
24175 DejaVuSansCondensed.ttf
24175 DejaVuSans.ttf
37110 CharisSILB.ttf
37110 CharisSILR.ttf
38445 CharisSILI.ttf
38447 CharisSILBI.ttf
18546 UMTypewriter-Regular.otf
18646 STIXGeneralItalic.otf
18689 STIXGeneralBolIta.otf
19196 STIXGeneralBol.otf
19266 cmunit.otf
19308 cmuntx.otf
19311 cmunbtl.otf
19311 cmunbto.otf
19316 cmuntb.otf
19481 OldStandard-Italic.otf
19520 OldStandard-Bold.otf
19569 OldStandard-Regular.otf
19675 cmunst.otf
19830 cmuntt.otf
20339 FreeSerifBoldItalic.otf
20628 STIXGeneral.otf
20771 FreeSerifBold.otf
21101 FreeSerifItalic.otf
21667 xits-math.otf
22912 cmunssdc.otf
22917 cmunso.otf
23083 cmunbso.otf
23087 cmunbmo.otf
23087 cmunbsr.otf
23088 cmunbmr.otf
23302 cmunbxo.otf
23303 cmunbbx.otf
23303 cmunsi.otf
23455 cmunsx.otf
23456 cmunss.otf
23614 cmunoti.otf
23615 cmunobi.otf
23643 FreeSans.otf
23665 cmunobx.otf
24009 cmunci.otf
24072 cmunui.otf
24076 cmunbi.otf
24118 cmunti.otf
24761 cmunbl.otf
24761 cmunsl.otf
24784 cmunorm.otf
24919 cmunbx.otf
25018 cmunrm.otf
28229 FreeSerif.otf
20341 FreeSerifBoldItalic.ttf
20772 FreeSerifBold.ttf
21070 FreeSerifItalic.ttf
23580 FreeSans.ttf
28262 FreeSerif.ttf
But can I run a profiler because the way memory is eaten and is not
being released is the limiting factor.
When I run XeLaTeX of LuaLaTeX on my font catalogue, the first one
reports to run out of memeory but the second just continues to eat up
all my real and virtual memory and eventually I have to kill it.
When I split my document in two parts, each about 135 fonts almost all
in regular, bold, bolditalic and italic, XeLaTeX is able to process it
and write about 10MB for each part. It takes my Ubuntu system to use 900
MB RAM and 200 MB swap to process one part (1/2). (Memory use increases
over time, does XeLaTeX use dynamic allocation?)
LuaLaTeX is unable to process one of these parts as it needs much much
more memory to do the job, so I have to kill it off again. When I split
it in 4 parts with each 65 fonts, it works. Each PDF is about 4 or 5 MB.
It takes my Ubuntu system to use 3500 MB RAM and 200 MB swap to process
this part (1/4). (That dynamic memory allocation occurs here is expected.)
In short, LuaLaTeX uses (for this font intensive document on this system
etc etc) approximately (2*(3500+200))/(900+200) = 6.7 times more space
in memory then XeLaTeX.
Regards,
Pander
> Regards,
> Khaled
>
