Re: [NTG-context] distributed / parallel TeX?
Hi Lars, Lars Huttar wrote: Hello, We've been using TeX to typeset a 1200-page book, and at that size, the time it takes to run becomes a big issue (especially with multiple passes... about 8 on average). It takes us anywhere from 80 minutes on our fastest machine, to 9 hours on our slowest laptop. You should not need an average of 8 runs unless your document is ridiculously complex and I am curious what you are doing (but that is a different issue from what you are asking). So the question comes up, can TeX runs take advantage of parallelized or distributed processing? No. For the most part, this is because of another requisite: for applications to make good use of threads, they have to deal with a problem that can be parallelized well. And generally speaking, typesetting does not fall in this category. A seemingly small change on page 4 can easily affect each and every page right to the end of the document. About the only safe things that can be threaded is the reading of resources (images and fonts) and, mostly because of the small gain, nobody has been working on that that I know of. parallel pieces so that you could guarantee that you would get the same result for section B whether or not you were typesetting the whole book at the same time? if you are willing to promiss yourself that all chapters will be exactly 20 pages - no more, no less - they you can split the work off into separate job files yourself and take advantage of a whole server farm. If you can't ... Best wishes, Taco ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___
Re: [NTG-context] Some ligatures cannot be copied and pasted
Taco Hoekwater wrote: Yue Wang wrote: Hi, see the attachment Probably because these ligatures are not in Unicode. I do not know whether it is possible to support cutpaste of these, I have to investigate. maybe if we use the a_b_c naming for such ligature - Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl - ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___
Re: [NTG-context] markiv, output written on...
Yue Wang wrote: and I also find that MKIV does not take terminal width into consideration. (pdftex output has a max line length limit) it does, but i've set it to a higher value (i prefer to let the terminal do the wrapping itself) Hans - Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl - ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___
Re: [NTG-context] distributed / parallel TeX?
Lars Huttar wrote: Hello, We've been using TeX to typeset a 1200-page book, and at that size, the time it takes to run becomes a big issue (especially with multiple passes... about 8 on average). It takes us anywhere from 80 minutes on our fastest machine, to 9 hours on our slowest laptop. often there are ways to optimize the document style btw, in my laptop mk.tex which loads a bunch of real big fonts (esp punk slows down) does 10 pages/second (242 pages currently) so your setu opif probably not that efficient So the question comes up, can TeX runs take advantage of parallelized or distributed processing? As many in the computer industries are aware, processor speed (clock rate) has plateaued; it is not going to continue rising at the rate it had been. Hence the common move to dual-core, quad-core, etc. machines. But applications in general cannot take advantage of multiple cores to speed their work unless they are architected to take advantage of them. We googled around a bit but were surprised not to find any real references to efforts at running TeX in parallel or on distributed networks or clusters. Wouldn't this be something that a lot of people would find useful? Or does everyone only use TeX for typesetting short papers? it all depends on what you process; for simple docs tex is rather fast: \starttext \setuplayout[page] \dorecurse{1000}{\null\page} \stoptext such a run takes in mkiv: 5.944 seconds, 1000 processed pages, 1000 shipped pages, 168.237 pages/second Sure, you can use manual tricks to speed up TeX processing. You can comment out sections of a document, or select them via modes. But then you have to remember where you did the commenting out, so you can reverse it. And you have no guarantees as to whether the inclusion/exclusion of section B will affect the layout of section C or not. often it's inefficient font calls that slow down the job (or big graphics that one can skip including in a first pass) Wouldn't it be nice if TeX (or a TeX wrapper, or macro package, or typesetting system) could take care of this for you? mayne in the future we can do some parallization also, i'm thinking of 'one run with prerolls' but it has no high priority (maybe i'll do it when i need it for a project) What if you had a language -- or a few extensions to existing languages -- to give your typesetting engine hints or commands about where to split up your long document into fairly-independent chunks? What if you designed your document specifically to be typeset in independent, parallel pieces so that you could guarantee that you would get the same result for section B whether or not you were typesetting the whole book at the same time? there are quite some dependencies of pages on each other (take cross refs and how they might influence a next run) What if the typesetting system automated the stitching-back-together process of the chunks, gathering page reference info from each chunk to inform the next iteration of typesetting the other chunks? this is an option when you have to reprocess parts of the documents often Has anyone been working on this already? It seems like it must have been discussed, but I don't know where to go to look for that discussion. if i were you i'd first look into optimizing the style Hans - Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl - ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___
Re: [NTG-context] PDF page numbers
Best Regards Chen Zhi-chu Chen | Shanghai Synchrotron Radiation Facility No. 2019 | Jialuo Rd. | Jiading | Shanghai | P.R. China tel: 086 21 5955 3405 | zhichu.chen.googlepages.com | www.sinap.ac.cn On Tue, Dec 16, 2008 at 4:33 PM, Taco Hoekwater t...@elvenkind.com wrote: Mohamed Bana wrote: Derek CORDEIRO wrote: In latex + book/memoir + hyperref, if \frontmatter \mainmatter are used, the pdf files generated use the roman numbers in the pdf itself. It appears as ii (2 of 500) in the pdfviewer. Is it possible to do the same with ConTeXt? It would be great if ConTeXt could. Doesn't sound like rocket science. Perhaps someone could dig up the latex code that does this ? Does the following code work for you? == \pdfcompresslevel 0 \setupinteraction [state=start, focus=width, page=yes, openaction=firstpage, click=no, title=TEST, author=Neals, subtitle=Test, keyword={test}] \setupcolors [state=start] \newcount\FrontPageNumbers \setupsubpagenumber[way=bytext,state=start] \setuppagenumbering[location=] \startsectionblockenvironment[frontpart] \setupfootertexts[pagenumber] \setuppagenumbering[conversion=romannumerals] \stopsectionblockenvironment \startsectionblockenvironment[bodypart] \global\FrontPageNumbers=\pagenumber \global\advance\FrontPageNumbers\minusone \setuppagenumber[number=1] \resetsubpagenumber % \setupfootertexts[\the\FrontPageNumbers] \setupfootertexts[pagenumber] \stopsectionblockenvironment \startsectionblockenvironment[backpart] \resetsubpagenumber \stopsectionblockenvironment \def\PageLabels{\pdfcatalog{/PageLabels /Nums [ 0 /S /r \the\FrontPageNumbers /S /D ] }} \prependtoks \PageLabels \to \everystoptext \starttext \startfrontmatter % \chapter{Table of Contents} \completecontent[interaction=all] \chapter{Preface} \dorecurse{10}{\input zapf\par} % \chapter{Preface} % \dorecurse{10}{\input zapf\par} % \chapter{Preface} % \dorecurse{10}{\input zapf\par} \stopfrontmatter \startbodymatter \chapter{Test} \dorecurse{15}{\input tufte\par} \chapter{TestAgain} \dorecurse{15}{\input ward\par} \stopbodymatter \startbackmatter Index maybe. \stopbackmatter \stoptext == Taco ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___ ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___
Re: [NTG-context] distributed / parallel TeX?
Aditya Mahajan wrote: On Tue, 16 Dec 2008, Hans Hagen wrote: Lars Huttar wrote: Hello, We've been using TeX to typeset a 1200-page book, and at that size, the time it takes to run becomes a big issue (especially with multiple passes... about 8 on average). It takes us anywhere from 80 minutes on our fastest machine, to 9 hours on our slowest laptop. often there are ways to optimize the document style In Latex, there is a package called mylatex, which allows you to create a format consisting of your preamble. Then you can call latex with your own format and this speeds up things. not on this document since the processing takes way more than the startup in jelle's doc it's the many many many local font definitions and the some 800 metapost graphics that are the culprit - define fonts beforehand - use unique mpgraphic when possible i changes the definitions a bit and now get 5 pages per second on my laptop in luatex; xetex processes the pages a bit faster but spends way more time on the mp part (of course 1450 pages of columnsets and multipage bTABLE's also cost a bit of time) This approach did not provide a significant speed improvement in latex for me, and I don't know whether it will do so in context. Hans and Taco, do you think that creating a personal format and possibly also dumping some font related info could provide a tradeoff between processing speed and disk space? it depends, you might win maybe a second per run when on startup in cases where you have thousands of small runs but often using a small tree (like the minimals) pays of more Hans - Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl - ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___
Re: [NTG-context] Some ligatures cannot be copied and pasted
1 (page number) also has On Tue, Dec 16, 2008 at 6:24 PM, Hans Hagen pra...@wxs.nl wrote: Yue Wang wrote: Hi, see the attachment only Th has a problem; if you run with \nopdfcompression you can peek into the file and see what unicode mapping is used; however, i would not be surprised if there's a bug in the font's resource Hans - Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl - ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___ ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___
Re: [NTG-context] distributed / parallel TeX?
On Tue, 16 Dec 2008, Hans Hagen wrote: Lars Huttar wrote: Hello, We've been using TeX to typeset a 1200-page book, and at that size, the time it takes to run becomes a big issue (especially with multiple passes... about 8 on average). It takes us anywhere from 80 minutes on our fastest machine, to 9 hours on our slowest laptop. often there are ways to optimize the document style In Latex, there is a package called mylatex, which allows you to create a format consisting of your preamble. Then you can call latex with your own format and this speeds up things. This approach did not provide a significant speed improvement in latex for me, and I don't know whether it will do so in context. Hans and Taco, do you think that creating a personal format and possibly also dumping some font related info could provide a tradeoff between processing speed and disk space? Aditya ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___
Re: [NTG-context] markiv, output written on...
Patrick Gundlach wrote: Hi, when I run a markiv file I don't get any message of the resulting pdf file. Is this done on purpose? probably (i overload some things) Hans - Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl - ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___
Re: [NTG-context] about LuaTools | error unable to identify cnf file
On Mon, Dec 15, 2008 at 9:16 PM, Diego Depaoli wrote: On Mon, Dec 15, 2008 at 12:13 PM, Dalyoung Jeong wrote: Dear Mojca, texmfstart texexec --make (--xtx) --all run smoothly, but luatools still has a cnf error. Stupid question... did you run/source setuptex before? In pure theory it should not be needed. I told you to update TL, but I forgot to tell you that you need to update *both* from TeX Live *and* update LuaTeX and ConTeXt afterwards manually with some more recent version. Updating with tlmgr only serves the purpose to get the right version of luatools, mtxrun and texmfstart. They used to be copies of scripts, while they are symbolic links now. If you woud update ConTeXt without updating the version from TL, it would not update the scripts. Specifically on MacTeX you need some more patches that have been written after ConTeXt and LuaTeX have been frozen on TeX Live. So you could either set all the environmental variables or simply update everything. (See http://wiki.contextgarden.net/Running_Mark_IV where I forgot to mention that one should update with tlmgr first.) Mojca ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___
Re: [NTG-context] distributed / parallel TeX?
Lars Huttar wrote: We have close to 7000 mpgraphics, and they add about 15 minutes to the run time. most of them are the same so reusing them made sense But the run time was already quite long before we started using those. - define fonts beforehand OK, we will look into this. I'm sure Jelle knows about this but I'm a noob. I'm pretty sure we are not *loading* fonts every time, but maybe we're scaling fonts an unnecessary number of times. For example, we have the following macro, which we use thousands of times: \def\LN#1{{\switchtobodyfont[SansB,\LNfontsize]{#1}}} indeed this will define the scaled ones again and again (whole sets of them since you use a complete switch); internall tex reuses them but it only know so when they're defined Would it help much to instead use \definefont[SansBLN][... at \LNfontsize] and then \def\LN#1{{\SansBLN{#1}}} ? indeed: \definefont[SansBLN][... at \LNfontsize] but no extra { } needed: \def\LN#1{{\SansBLN#1}} - use unique mpgraphic when possible I would be interested to know if this is possible in our situation. Most of our mpgraphics are due to wanting thick-and-thin or single-and-double borders on tables, which are not natively supported by the ConTeXt table model. i sent jelle the patched files The advice I received said to define each mpgraphic using \startuseMPgraphic (we have about 18 of these), associate them with overlays using \defineoverlay (again, we have 18), and then use them in table cells using statements like \setupTABLE[c][first][background={LRtb}] Empirically, this seems to end up using one mpgraphic per table cell, hence our thousands of mpgraphics. I don't know why a new mpgraphic would be created for each cell. Can someone suggest a way to avoid this? metafun manual: unique mp graphics i changes the definitions a bit and now get 5 pages per second on my laptop in luatex; xetex processes the pages a bit faster but spends way more time on the mp part My last run gave about 0.25 pages per second on our fastest server, when taking into account multiple passes; that comes out to about 2 pps for --once. the patched files do 5-10 pps on my laptop (was 1 sec pp) so an improvement factor of at least 5 is possible there are probably other optimizations possible but i cannot spent too much time on it Hans - Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl - ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___
Re: [NTG-context] markiv, output written on...
Hi Hans, when I run a markiv file I don't get any message of the resulting pdf file. Is this done on purpose? probably (i overload some things) May I then ask for a change in behaviour? I'd like to parse the messages and search for the final pdf. Patrick -- ConTeXt wiki and more: http://contextgarden.net ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___
Re: [NTG-context] distributed / parallel TeX?
On 12/16/2008 2:08 AM, Taco Hoekwater wrote: Hi Lars, Lars Huttar wrote: Hello, We've been using TeX to typeset a 1200-page book, and at that size, the time it takes to run becomes a big issue (especially with multiple passes... about 8 on average). It takes us anywhere from 80 minutes on our fastest machine, to 9 hours on our slowest laptop. You should not need an average of 8 runs unless your document is ridiculously complex and I am curious what you are doing (but that is a different issue from what you are asking). So the question comes up, can TeX runs take advantage of parallelized or distributed processing? No. For the most part, this is because of another requisite: for applications to make good use of threads, they have to deal with a problem that can be parallelized well. And generally speaking, typesetting does not fall in this category. A seemingly small change on page 4 can easily affect each and every page right to the end of the document. Thank you for your response. Certainly this is true in general and in the worst case, as things stand currently. But I don't think it has to be that way. The following could greatly mitigate that problem: - You could design your document *specifically* to make the parts independent, so that the true and authoritative way to typeset them is to typeset the parts independently. (You can do this part now without modifying TeX at all... you just have the various sections' .tex files input common headers / macro defs.) Then, by definition, a change in one section cannot affect another section (except for page numbers, and possibly left/right pages, q.v. below). - Most large works are divisible into chunks separated by page breaks and possibly page breaks that force a recto. This greatly limits the effects that any section can have on another. The division (chunking) of the whole document into fairly-separate parts could either be done manually, or if there are clear page breaks, automatically. - The remaining problem, as you noted, is how to fix page references from one section to another. Currently, TeX resolves forward references by doing a second (or third, ...) pass, which uses page information from the previous pass. The same technique could be used for resolving inter-chunk references and determining on what page each chunk should start. After one pass on of the independent chunks (ideally performed simultaneously by separate processing nodes), page information is sent from each node to a coordinator process. E.g. the node that processed section two tells the coordinator that chapter 11 starts 37 pages after the beginning of section two. The coordinator knows in what sequence the chunks are to be concatenated, thanks to a config file. It uses this information together with info from each of the nodes to build a table of what page each chunk should start on, and a table giving the absolute page number of each page reference. If pagination has changed, or is new, this info is sent back to the various nodes for another round of processing. If this distributed method of typesetting a document takes 1 additional iteration compared to doing it in series, but you get to split the document into say 5 roughly equal parts, you could presumably get the job done a lot quicker in spite of the extra iteration. This is a crude description but hopefully the idea is clear enough. parallel pieces so that you could guarantee that you would get the same result for section B whether or not you were typesetting the whole book at the same time? if you are willing to promiss yourself that all chapters will be exactly 20 pages - no more, no less - they you can split the work off into separate job files yourself and take advantage of a whole server farm. If you can't ... Yes, the splitting can be done manually now, and when the pain point gets high enough, we do some manual separate TeX runs. However, I'm thinking that for large works, there is enough gain to be had that it would be worth systematizing the splitting process and especially the recombining process, since the later is more error-prone. I think people would do it a lot more if there were automation support for it. I know we would. But then, maybe our situation of having a large book with dual columns and multipage tables is not common enough in the TeX world. Maybe others who are typesetting similar books just use commercial WYSIWYG typesetting tools, as we did in the previous edition of this book. Lars ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net
Re: [NTG-context] PDF page numbers
Taco Hoekwater wrote: Mohamed Bana wrote: Derek CORDEIRO wrote: In latex + book/memoir + hyperref, if \frontmatter \mainmatter are used, the pdf files generated use the roman numbers in the pdf itself. It appears as ii (2 of 500) in the pdfviewer. Is it possible to do the same with ConTeXt? It would be great if ConTeXt could. sure, since it's just a page resource; i think i even played with it and dropped it when i got funny printing side effects (when selecting pages) in older versions of teh viewer Doesn't sound like rocket science. Perhaps someone could dig up the latex code that does this ? something for the mkiv backend Hans - Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl - ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___
Re: [NTG-context] markiv, output written on...
Patrick Gundlach wrote: Hi Hans, when I run a markiv file I don't get any message of the resulting pdf file. Is this done on purpose? probably (i overload some things) May I then ask for a change in behaviour? I'd like to parse the messages and search for the final pdf. keep in mind that context --result might overload the name so we cannot use jobname here. to cont-new.mkiv add: \ctxlua{ ctx.register_statistics(result saved in file, \letterpercent s.\letterpercent s, function() return \outputfilename, (tex.pdfoutput0 and pdf) or dvi end) } then you can parse for mkiv lua stats : result saved in file - x.pdf (i'll add this) Hans - Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl - ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___
Re: [NTG-context] distributed / parallel TeX?
On 12/16/2008 11:37 AM, Hans Hagen wrote: Lars Huttar wrote: We have close to 7000 mpgraphics, and they add about 15 minutes to the run time. most of them are the same so reusing them made sense But the run time was already quite long before we started using those. - define fonts beforehand OK, we will look into this. I'm sure Jelle knows about this but I'm a noob. I'm pretty sure we are not *loading* fonts every time, but maybe we're scaling fonts an unnecessary number of times. For example, we have the following macro, which we use thousands of times: \def\LN#1{{\switchtobodyfont[SansB,\LNfontsize]{#1}}} indeed this will define the scaled ones again and again (whole sets of them since you use a complete switch); internall tex reuses them but it only know so when they're defined Would it help much to instead use \definefont[SansBLN][... at \LNfontsize] and then \def\LN#1{{\SansBLN{#1}}} ? indeed: \definefont[SansBLN][... at \LNfontsize] but no extra { } needed: \def\LN#1{{\SansBLN#1}} Thanks, we will try this. (Jelle, since you have worked with this a lot longer than I have, please stop me if you have concerns about my making this sort of change.) - use unique mpgraphic when possible I would be interested to know if this is possible in our situation. Most of our mpgraphics are due to wanting thick-and-thin or single-and-double borders on tables, which are not natively supported by the ConTeXt table model. i sent jelle the patched files OK, I'll look to hear from him. Are these patches to support these kinds of borders on tables, thus no longer needing to use MPgraphics? The advice I received said to define each mpgraphic using \startuseMPgraphic (we have about 18 of these), associate them with overlays using \defineoverlay (again, we have 18), and then use them in table cells using statements like \setupTABLE[c][first][background={LRtb}] Empirically, this seems to end up using one mpgraphic per table cell, hence our thousands of mpgraphics. I don't know why a new mpgraphic would be created for each cell. Can someone suggest a way to avoid this? metafun manual: unique mp graphics Great... I converted our useMPgraphics to uniqueMPgraphics. This reduced our number of mpgraphics from 7000 to 800! Unfortunately the result doesn't look quite right... but since we may not need to use mpgraphics anyway thanks to your patches, I'll hold off on debugging the result. i changes the definitions a bit and now get 5 pages per second on my laptop in luatex; xetex processes the pages a bit faster but spends way more time on the mp part My last run gave about 0.25 pages per second on our fastest server, when taking into account multiple passes; that comes out to about 2 pps for --once. the patched files do 5-10 pps on my laptop (was 1 sec pp) so an improvement factor of at least 5 is possible there are probably other optimizations possible but i cannot spent too much time on it Thanks for all your help thus far. Lars ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___
Re: [NTG-context] distributed / parallel TeX?
On Tue, Dec 16, 2008 at 6:25 PM, Lars Huttar wrote: Most of our mpgraphics are due to wanting thick-and-thin or single-and-double borders on tables, which are not natively supported by the ConTeXt table model. The advice I received said to define each mpgraphic using \startuseMPgraphic (we have about 18 of these), associate them with overlays using \defineoverlay (again, we have 18), and then use them in table cells using statements like \setupTABLE[c][first][background={LRtb}] In that case I would suggest you to use vrules/hrules to achieve the same. As long as you don't need to use complicated graphics (like Hans uses random frames) this should be doable and would speed up the process enormously. What's the compile time if you omit the MP graphics? Mojca ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___
Re: [NTG-context] distributed / parallel TeX?
2008/12/16 Lars Huttar lars_hut...@sil.org: - You could design your document *specifically* to make the parts independent, so that the true and authoritative way to typeset them is to typeset the parts independently. (You can do this part now without modifying TeX at all... you just have the various sections' .tex files input common headers / macro defs.) Then, by definition, a change in one section cannot affect another section (except for page numbers, and possibly left/right pages, q.v. below). True. Also with TeX if your paragraphs are independent of each other (i.e. they don't include references to others), they could the typeset in parallel and then handed over to the page builder. - Most large works are divisible into chunks separated by page breaks and possibly page breaks that force a recto. This greatly limits the effects that any section can have on another. The division (chunking) of the whole document into fairly-separate parts could either be done manually, or if there are clear page breaks, automatically. pdfTeX 1.50 knows about the page diversions (analogue to m4's divert and undivert). They have a lot of potential. page number of each page reference. If pagination has changed, or is new, this info is sent back to the various nodes for another round of processing. Hopefully stopping at some point. If you use something like varioref, you can end with infinite circles. :-) Best Martin ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___
Re: [NTG-context] setupdelimitedtext: linespacing in final quoted paragraph
Thanks for the responses. I've wikified a short example, putting it at end of this page: http://wiki.contextgarden.net/Long_quotations_and_extracts Feel free to move or improve. Will LuaTeX's paragraph builder have the same restriction -- perhaps for compatability -- that linespacing be constant throughout the paragraph? -Sanjoy `Until lions have their historians, tales of the hunt shall always glorify the hunters.' --African Proverb ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___
Re: [NTG-context] distributed / parallel TeX?
On 12/16/2008 3:31 PM, Martin Schröder wrote: 2008/12/16 Lars Huttar lars_hut...@sil.org: - You could design your document *specifically* to make the parts independent, so that the true and authoritative way to typeset them is to typeset the parts independently. (You can do this part now without modifying TeX at all... you just have the various sections' .tex files input common headers / macro defs.) Then, by definition, a change in one section cannot affect another section (except for page numbers, and possibly left/right pages, q.v. below). True. Also with TeX if your paragraphs are independent of each other (i.e. they don't include references to others), they could the typeset in parallel and then handed over to the page builder. Good point... although doesn't the page optimization feed back into paragraph layout? - Most large works are divisible into chunks separated by page breaks and possibly page breaks that force a recto. This greatly limits the effects that any section can have on another. The division (chunking) of the whole document into fairly-separate parts could either be done manually, or if there are clear page breaks, automatically. pdfTeX 1.50 knows about the page diversions (analogue to m4's divert and undivert). They have a lot of potential. Sounds useful. It's impressive if you can get a correct table of contents in the first run (says http://www.gust.org.pl/BachoTeX/2008/presentations/ms/handout.pdf) page number of each page reference. If pagination has changed, or is new, this info is sent back to the various nodes for another round of processing. Hopefully stopping at some point. If you use something like varioref, you can end with infinite circles. :-) But this is just a problem of typesetting with TeX in general, not particular to parallel/distributed typesetting, right? IIRC, Knuth even says in the TeXbook that a really pathological case might never stabilize. Lars ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___
Re: [NTG-context] distributed / parallel TeX?
2008/12/16 Lars Huttar lars_hut...@sil.org: Good point... although doesn't the page optimization feed back into paragraph layout? No. :-( Best Martin ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___
Re: [NTG-context] distributed / parallel TeX?
On Tue, Dec 16, 2008 at 9:08 AM, Taco Hoekwater t...@elvenkind.com wrote: Hi Lars, Lars Huttar wrote: Hello, We've been using TeX to typeset a 1200-page book, and at that size, the time it takes to run becomes a big issue (especially with multiple passes... about 8 on average). It takes us anywhere from 80 minutes on our fastest machine, to 9 hours on our slowest laptop. You should not need an average of 8 runs unless your document is ridiculously complex and I am curious what you are doing (but that is a different issue from what you are asking). So the question comes up, can TeX runs take advantage of parallelized or distributed processing? No. For the most part, this is because of another requisite: for applications to make good use of threads, they have to deal with a problem that can be parallelized well. And generally speaking, typesetting does not fall in this category. A seemingly small change on page 4 can easily affect each and every page right to the end of the document. Also 3.11 Theory of page breaking www.cs.utk.edu/~eijkhout/594-LaTeX/handouts/TeX%20LaTeX%20*course*.pdf -- luigi ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___
[NTG-context] tiny reshuffle of ConTeXt versions on the Minimals
Hello, Just a short notice before it causes confusion. Since Hans is preparing some really big changes in mkiv (basically rewriting almost the whole ConTeXt core if I understood correctly), he is still waiting to update ConTeXt on his website until the changes are finished or at least working again. Since neither current nor beta are compatible with the latest LuaTeX, I have applied some patches that he sent me to official beta and now both current and beta are pointing to the same patched version of beta. (Maybe I should finally modify the timestamp ...) Else current on minimals would be broken, or we would be unable to upload luatex for quite a while. In theory everything should still work normally, beta has also been around for quite a while now, but before anyone asks me why are the versions on minimals different from the ones that Hans provides or it works with context beta/context current on my machine, why doesn't it work on yours, here you have some short explanation. This deviation from the official version will be dropped as soon as ConTeXt gets stable again. Mojca ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___
Re: [NTG-context] distributed / parallel TeX?
On 12/16/2008 3:15 PM, luigi scarso wrote: On Tue, Dec 16, 2008 at 9:08 AM, Taco Hoekwater t...@elvenkind.com mailto:t...@elvenkind.com wrote: Hi Lars, Lars Huttar wrote: ... So the question comes up, can TeX runs take advantage of parallelized or distributed processing? No. For the most part, this is because of another requisite: for applications to make good use of threads, they have to deal with a problem that can be parallelized well. And generally speaking, typesetting does not fall in this category. A seemingly small change on page 4 can easily affect each and every page right to the end of the document. Also 3.11 Theory of page breaking www.cs.utk.edu/~eijkhout/594-LaTeX/handouts/TeX%20LaTeX%20 http://www.cs.utk.edu/~eijkhout/594-LaTeX/handouts/TeX%20LaTeX%20*course*.pdf Certainly that is a tough problem (particularly in regard to laying out figures near the references to them). But again, if you can break down the document into chunks that are fairly independent of each other (and you almost always can for large texts), this problem seems no worse for distributed processing than for sequential processing. For example, the difficult part of laying out figures in Section 1 is confined to Section 1; it does not particularly interact with Section 2. References in Section 2 to Section 1 figures are going to be relatively distant from those figures regardless of page breaking decisions. Thus the difficult problem of page breaking is reduced to the sequential-processing case... still a hard problem, but one that can be attacked in chunks. Indeed, the greater amount of CPU time per page that is made available through distributed processing may mean that the algorithms can do a better job of page breaking than through sequential processing. Lars ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___
Re: [NTG-context] tiny reshuffle of ConTeXt versions on the Minimals
Hello, I also forgot to mention ... TeX Gyre is now the default font when using postscript/times/helvetica/palatino. In case you experience any problems, in particular with math, please report. Mojca ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___