Re: [XeTeX] XeTeX : request for enhancement

[David Carlisle]

On Sun, 15 Oct 2023 at 19:49, Philip Taylor <
philip.tay...@hellenic-institute.uk> wrote:

> At the moment, one can include a PDF file using the following syntax :
>
> \XeTeXpdffile ⟨filename⟩ [ page ⟨int⟩ ] [ crop | media | bleed | trim |
> art ]
> [ scaled ⟨int⟩ | xscaled ⟨int⟩ | yscaled ⟨int⟩ | width ⟨dimen⟩ |
> height ⟨dimen⟩ | rotated ⟨decimal⟩ ]
> Insert (pages of) a PDF. See below for explanation
>
> where [ crop | media | bleed | trim | art ] specify to which of the four
> boxes known to PDF the inserted file is to be cropped.  I frequently have
> the need to crop a PDF file during inclusion to arbitrary dimensions, but
> at the moment that would appear to be impossible (short of pre-massaging
> the PDF to be inserted).  Would it be possible to enhance XeTeX to
> recognise and honour the following (or similar) syntax :
>
> \XeTeXpdffile ⟨filename⟩ [ page ⟨int⟩ ] [ crop | media | bleed | trim |
> art | custom [xmin ymin xmax ymax]]
> [ scaled ⟨int⟩ | xscaled ⟨int⟩ | yscaled ⟨int⟩ | width ⟨dimen⟩ |
> height ⟨dimen⟩ | rotated ⟨decimal⟩ ]
> Insert (pages of) a PDF. See below for explanation
>
> --
> *Philip Taylor*
>


You can clip an arbitrary box so no need to have this as an image inclusion
feature.


\input{graphicx}

\includegraphics{example-image.pdf}

\bigskip

\includegraphics[trim=20pt 50pt 50pt 100pt, clip]{example-image.pdf}

\bye


Produces this (with plain xetex, for you)

[image: image.png]

Re: [XeTeX] XeTeX \openin — bug or feature

[David Carlisle]

Like most things, it's an option see

try_std_extension_first

in section 6.2 of texdoc kpathsea

On Tue, 20 Sept 2022 at 17:33, Philip Taylor (Hellenic Institute) <
p.tay...@hellenic-institute.uk> wrote:

> On 20/09/2022 17:29, David Carlisle wrote:
>
> feature, and not xetex specific
>
> Thank you, David.  Evil, nonetheless, and highly counter-intuitive.
>
> --
> *** Phil.*
>

Re: [XeTeX] XeTeX \openin — bug or feature

[David Carlisle]

feature, and not xetex specific

$ ls MM*
MM3.dat  MM3.dat.tex

$ kpsewhich MM3.dat
./MM3.dat.tex

$ kpsewhich --all  MM3.dat
./MM3.dat.tex
./MM3.dat

If both are available kpsearch prefers the .tex as shown by the first call
to kpsewhich

David


On Tue, 20 Sept 2022 at 16:40, Philip Taylor (Hellenic Institute) <
p.tay...@hellenic-institute.uk> wrote:

> Given
>
> \newread \standin
> \openin \standin = MM3.dat
>
> I had expected XeTeX to open MM3.dat for input — instead, it opened
> MM3.dat.tex
>
> Bug or feature ?
>
> --
> *Philip Taylor*
>

Re: [XeTeX] XeTeX development, issues?

[David Carlisle]

On Thu, 28 Jul 2022 at 08:32, Nico Schlömer 
wrote:

> Hi everyone,
>
> Where can I file bugs about XeTeX? Is there a GitHub repo perhaps?
>

sourceforge git, there are bug and feature reporting links at

http://xetex.sourceforge.net/

David

>
> Cheers,
> Nico
>

Re: [XeTeX] Support of variable OpenType fonts

[David Carlisle]

On Tue, 12 Jul 2022 at 22:03, David Jones  wrote:

> STIX Two is distributed in OTF, TTF, and WOFF2 formats:
>
>  https://github.com/stipub/stixfonts
>
> Only the TTF files are variable fonts.
>
> Only the OTF fonts are distributed via CTAN since none of the TeX engines
> can use the (variable) TTF fonts.
>

luatex (luaotfload) can use variable fonts (as shown in the original
message in this thread) see also https://tex.stackexchange.com/a/630608/1090


> The changes in version 2.13 only affected the variable fonts, not the
> "static" OTF files (although the version numbers were bumped up to match
> the ttf files).  That's why we didn't push a new version of stix2-otf
> package to CTAN.
>
> Cheers,
> David.
> 
> From: XeTeX  on behalf of Bruno Voisin
> via XeTeX 
> Sent: Tuesday, July 12, 2022 10:17 AM
> To: xetex@tug.org
> Subject: [XeTeX] Support of variable OpenType fonts
>
> Does XeTeX support variable OpenType fonts, or is the addition of their
> support planned at some point?
>
> There does not seem to be any clear statement on the net about this, but
> based on
>
>
> https://tex.stackexchange.com/questions/355104/tex-luatex-xetex-fontspec-support-for-opentype-variable-fonts
> https://sourceforge.net/p/xetex/feature-requests/28
>
> I have the impression there's no such support yet.
>
> I'm asking because the next version of macOS, currently in public beta,
> includes the STIX Two fonts in a slightly more recent version than TeX Live
> (2.13 b171 for macOS vs 2.12 b168 for TL).
>
> The math font is in OTF format, same as in TL, but the text fonts are in
> variable OpenType format, actually the files
>
> STIXTwoText[wght].ttf
> STIXTwoText-Italic[wght].ttf
>
> from the official STIX Two distribution renamed as
>
> STIXTwoText.ttf
> STIXTwoText-Italic.ttf
>
> respectively.
>
> Loading the fonts with LuaTeX and luaotfload, use of
>
> \font\testfont="STIX Two Text"
> \font\testfont="STIX Two Text Medium"
>
> works and gets the fonts from TeX Live, while
>
> \font\testfont="[STIXTwoText.ttf]"
> \font\testfont="[STIXTwoText.ttf]:instance=medium"
>
> gets the font from macOS. Similarly
>
> \fonttest{STIX Two Text Italic}
> \fonttest{STIX Two Text Medium Italic}
>
> for TL becomes
>
> \fonttest{[STIXTwoText-Italic.ttf]}
> \fonttest{[STIXTwoText-Italic.ttf]:axis={weight=medium}}
>
> for macOS (for some reason instance= doesn't work in this latter case).
> See the attached test file.
>
> So I was wondering whether the same could be achieved with XeTeX.
>
> The question is futile regarding STIX Two (the fonts in TeX Live are
> perfectly fine). But this was my first encounter with variable OpenType
> fonts ever in the real world, and I imagine the use of such fonts will
> become more widespread as time goes.
>
> Bruno Voisin
>
> PS Please cc possible answers to me, as I'm no longer subscribed to the
> list.
>
>
>
>
>

Re: [XeTeX] Uppercase in Armenian

[David Carlisle]

the input uses a ligature character which has no corresponding uppercase,
you need to decompose the ligature before uppercasing, which \uppercase
can't do, you see the same in Latin script ff ligature

\uppercase{diff diff}

looks like DIFF DIff as the second one uses U+FB00 which has no uppercase.

 U+0587 ARMENIAN SMALL LIGATURE ECH YIWN

would be better input as  U+0565 U+0582

David




On Sun, 1 May 2022 at 00:09, Zdenek Wagner  wrote:

> Yes, it looks better but the uppercase version should contain ԵՎ, not ԵՒ.
> Վ is capital vew (U+54E) while Ւ is capital yiwn (U+552).
>
> Zdeněk Wagner
> http://ttsm.icpf.cas.cz/team/wagner.shtml
>
>
> ne 1. 5. 2022 v 0:53 odesílatel David Carlisle 
> napsal:
>
>> Something like this, I think.
>>
>> [image: image.png]
>>
>> \documentclass{article}
>> \usepackage{polyglossia}
>>
>> \setdefaultlanguage{armenian}
>> \setmainfont{DejaVu Sans}
>> \ExplSyntaxOn
>> \let\tuppercase\text_uppercase:n
>> \ExplSyntaxOff
>> \pagestyle{empty}
>> \begin{document}
>> Երևան $\rightarrow$ \uppercase{Երևան}
>>
>> Երևան $\rightarrow$ \tuppercase{Երևան}
>>
>> \end{document}
>>
>> David
>>
>>
>> On Sat, 30 Apr 2022 at 22:15, Zdenek Wagner 
>> wrote:
>>
>>> Hi all,
>>>
>>> first I should mention that I do not know Armenian at all and can just
>>> recognize a few characters. Anyway I came across a problem which
>>> probably cannot be solved by the standard \lccode / \uccode method.
>>> What I mean is Yerevan which is written as Երևան but YEREVAN (all
>>> caps) is ԵՐԵՎԱՆ because և has no uppercase variant and must be
>>> replaced by two characters ԵՎ. At least, it is visible in Elena
>>> Yerevan's songs shot in the city of Yerevan. The following file
>>>
>>> \documentclass{article}
>>> \usepackage{polyglossia}
>>> \setdefaultlanguage{armenian}
>>> \setmainfont{DejaVu Sans}
>>> \pagestyle{empty}
>>> \begin{document}
>>> Երևան $\rightarrow$ \uppercase{Երևան}
>>> \end{document}
>>>
>>> shows that all characters are converted correctly to uppercase but և
>>> remains in lovercase. Is there a solution?
>>>
>>> Zdeněk Wagner
>>> http://ttsm.icpf.cas.cz/team/wagner.shtml
>>>
>>>

Re: [XeTeX] Uppercase in Armenian

[David Carlisle]

Something like this, I think.

[image: image.png]

\documentclass{article}
\usepackage{polyglossia}

\setdefaultlanguage{armenian}
\setmainfont{DejaVu Sans}
\ExplSyntaxOn
\let\tuppercase\text_uppercase:n
\ExplSyntaxOff
\pagestyle{empty}
\begin{document}
Երևան $\rightarrow$ \uppercase{Երևան}

Երևան $\rightarrow$ \tuppercase{Երևան}

\end{document}

David


On Sat, 30 Apr 2022 at 22:15, Zdenek Wagner  wrote:

> Hi all,
>
> first I should mention that I do not know Armenian at all and can just
> recognize a few characters. Anyway I came across a problem which
> probably cannot be solved by the standard \lccode / \uccode method.
> What I mean is Yerevan which is written as Երևան but YEREVAN (all
> caps) is ԵՐԵՎԱՆ because և has no uppercase variant and must be
> replaced by two characters ԵՎ. At least, it is visible in Elena
> Yerevan's songs shot in the city of Yerevan. The following file
>
> \documentclass{article}
> \usepackage{polyglossia}
> \setdefaultlanguage{armenian}
> \setmainfont{DejaVu Sans}
> \pagestyle{empty}
> \begin{document}
> Երևան $\rightarrow$ \uppercase{Երևան}
> \end{document}
>
> shows that all characters are converted correctly to uppercase but և
> remains in lovercase. Is there a solution?
>
> Zdeněk Wagner
> http://ttsm.icpf.cas.cz/team/wagner.shtml
>
>

Re: [XeTeX] Guaranteed Unicode replacement glyph in every TeX installation?

[David Carlisle]

On Sat, 21 Aug 2021, 13:46 Janusz S. Bień,  wrote:

> On Sat, Aug 21 2021 at 10:48 +02, Ulrike Fischer wrote:
> > Am Fri, 20 Aug 2021 19:45:37 -0600 (MDT) schrieb Doug McKenna:
> >
> >> Using XeTeX, I want to typeset a LaTeX document into a PDF file. The
> LaTeX source code in UTF-8 expressly includes the Unicode Replacement
> character (� = U+FFFD)
> >
> >> So ... What fixed-width and variable-width OpenType (or other)
> >> fonts, if any, are always distributed with TeX or TeXLive or
> >> whatever that one can rely upon to be available for placing this
> >> particular glyph in a final PDF file? What would be the correct
> >> incantation to doing so?
> >
> > With a current texlive you can use albatross to find out which fonts
> > on your system support this
> >
> >albatross -d 0xFFFD
>
> A nice replacement for fc-search-codepoint, as it accept also glyphs as
> the arguments.
>

note that fc-list will only find fonts known to fontconfig so by default
this won't include fonts distributed with texlive

David

>
> Best regard
>
> Janusz
>
> --
>  ,
> Janusz S. Bien
> emeryt (emeritus)
> https://sites.google.com/view/jsbien
>
>

[XeTeX] internal error using opentype font on mathaccent

[David Carlisle]

\font\f="TeX Gyre Termes"

\textfont0=\f
\scriptfont0=\f
\scriptscriptfont0=\f


$ \mathaccent"0017 c $

\bye

This is a plain version of a latex example posted by a user to
tex.stackexchange.
It shouldn't really do anything sensible (if the font load is adjusted
for luatex
you get the standard missing character warning)

But it gives a fatal internal error

This is XeTeX, Version 3.141592653-2.6-0.93 (TeX Live 2021)
(preloaded format=xetex)
 restricted \write18 enabled.
entering extended mode
(./aa539a.tex
! Internal error: bad native font flag in `map_char_to_glyph'


It doesn't seem to matter much which font is used, I just used TeX
Gyre above as a specific example.

Could this be adjusted to give a normal TeX warning (or error) ?

David

Re: [XeTeX] \XeTeXpdffile causes over-long line unless wrapped in \hbox

[David Carlisle]

Bruno wrote
> David got mislead.

Yes sorry too many programming languages in head at once today:-)

Re: [XeTeX] \XeTeXpdffile causes over-long line unless wrapped in \hbox

[David Carlisle]

The command is scanning ahead for keywords such as  page 2 in the unhboxed
case  but in the boxed case the } stops the scan but it seems having
decided that  test is not a keyword it is inserting a space before it which
seems wrong. More surprising is that it eats ! here even though  ! isn't
part of the allowed syntax as far as I can see


\def\imga{\hbox{\XeTeXpdffile "image.pdf"\relax!}}
\def\imgb{\XeTeXpdffile "image.pdf"\relax!}



test test test test test test test test test test test test test
test test test test test test test test test test test test test
test test test test test test test test test test test test test
test test test test test test test test test test test test test
\imga
test test test test test test test test test test test test test
test test test test test test test test test test test test test
test test test test test test test test test test test test test
test test test test test test test test test test test test test


test test test test test test test test test test test test test
test test test test test test test test test test test test test
test test test test test test test test test test test test test
test test test test test test test test test test test test test
\imgb
test test test test test test test test test test test test test
test test test test test test test test test test test test test
test test test test test test test test test test test test test
test test test test test test test test test test test test test

\end


The effect on linebreaking is more or less accidental "test" can't be
hyphenated so it's not surprising that some lines can not be justified, and
the extra space or not space after the image is changing the linebreaking
in the paragraph.

David



On Thu, 15 Apr 2021 at 14:24, Gavin Smith  wrote:

> If I XeTeX to display an image in text, like
>
> \def\img{\XeTeXpdffile "image.pdf"\relax}
>
> test test test test test test test test test test test test test
> test test test test test test test test test test test test test
> test test test test test test test test test test test test test
> test test test test test test test test test test test test test
> \img
> test test test test test test test test test test test test test
> test test test test test test test test test test test test test
> test test test test test test test test test test test test test
> test test test test test test test test test test test test test
>
> \end
>
> and then process with "xetex test.tex" on the command line, then the line
> after the image is too long and goes into the margin.  You can see this
> in the attached file.
>
> I found wrapping the \XeTeXpdffile in an \hbox, thus:
>
> \def\img{\hbox{\XeTeXpdffile "image.pdf"\relax}}
>
> fixed the problem, but I don't see why this should be necessary.
>
> Is this a bug?
>
> XeTeX reports its version as follows:
>
> $ xetex test.texi
> This is XeTeX, Version 3.14159265-2.6-0.8 (TeX Live 2017/Debian)
> (preloaded format=xetex)
>  restricted \write18 enabled.
> entering extended mode
> (./test.texi [1] )
> Output written on test.pdf (1 page).
> Transcript written on test.log.
>
>

Re: [XeTeX] Coloured fonts

[David Carlisle]

Sigh, I thought you'd like the challenge:-)

with luahbtex:



\input luaotfload.sty
\font\chess=BabelStoneXiangqiColour.ttf:mode=harf


testing {\chess ^^01fa64}

\bye



On Thu, 18 Mar 2021 at 20:44, Philip Taylor 
wrote:

> David Carlisle wrote:
>
> Not sure if xetex can do colour fonts currently, You can always experiment
> with luatex which gets this if using harfbuzz
>
> [image: image.png]
>
> \documentclass{article}
>
> \usepackage{fontspec}
>
> \newfontfamily\chess[Renderer=HarfBuzz]{BabelStoneXiangqiColour.ttf}
> \begin{document}
>
> testing {\chess ^^01fa64}
>
> \end{document}
>
>
> OK, thanks David.  I will see if LuaTeX can handle my existing work on the
> book (which of course assumed plain XeTeX), and if it can, I will switch to
> LuaTeX for this project.  But do you know how one does the plain LuaTeX
> equivalent of :
>
> \usepackage{fontspec}
> \newfontfamily\chess[Renderer=HarfBuzz]{BabelStoneXiangqiColour.ttf}
>
>
> I *really* don't want to have to use packages unnecessarily.
> --
> *** Phil.*
>

Re: [XeTeX] Coloured fonts

[David Carlisle]

Not sure if xetex can do colour fonts currently, You can always experiment
with luatex which gets this if using harfbuzz

[image: image.png]

\documentclass{article}

\usepackage{fontspec}

\newfontfamily\chess[Renderer=HarfBuzz]{BabelStoneXiangqiColour.ttf}
\begin{document}

testing {\chess ^^01fa64}

\end{document}



On Thu, 18 Mar 2021 at 18:39, Philip Taylor  wrote:

> Seeking to re-typeset a long out-of-print classic on Xiang-Qi ("Chinese
> Chess"), but with the pieces shewn as they really are rather than as
> upper-case Latin letters requiring a gloss (the presentation chosen by the
> original author), I downloaded and installed Andrew West's BabelStone
> Xiangqi Colour font .  I
> then wrote a short piece of XeTeX code to check that the glyphs/pieces
> appear in the PDF as they should, and very sadly they do not, coming out as
> monochrome rather than in colour (see attached PDF).
>
> The red pieces are described by Andrew as *red Chinese characters on a
> sandy yellow background*, and the black pieces as *black Chinese
> characters on a sandy yellow background.*  In the resulting PDF, however,
> they appear as white Hanzi on a black ground and black Hanzi on a white
> ground.  Does XeTeX support coloured fonts, and if so, how do I persuade it
> to render these glyphs as intended rather than in monochrome ?
>
> I can, of course, load \font \redpieces = "BabelStone Xiangqi
> Colour":color=FF scaled \magstep 5 (see code below), but that still
> does not give me the sandy yellow ground that each glyph was designed to
> have.
>
> 'opentype-info.tex', when run against BabelStone Xiangqi Colour, tells me
> that the font does not provide any Opentype layout features, so it does not
> look as if XeTeX's "/ICU:+abcd" convention would allow me to indicate that
> I require colour support.
>
> % !TeX Program=XeTeX
>
> \font \pieces = "BabelStone Xiangqi Colour" scaled \magstep 5
> \font \redpieces = "BabelStone Xiangqi Colour":color=FF scaled
> \magstep 5
> \font \blackpieces = "BabelStone Xiangqi Colour" scaled \magstep 5
> \pieces
> \centerline {\char "1FA60\relax \ \char "1FA61\relax \ \char "1FA62\relax
> \ \char "1FA63\relax \ \char "1FA64\relax \ \char "1FA65\relax \ \char
> "1FA66\relax}
> \centerline {\strut}
> \centerline {\char "1FA67\relax \ \char "1FA68\relax \ \char "1FA69\relax
> \ \char "1FA6A\relax \ \char "1FA6B\relax \ \char "1FA6C\relax \ \char
> "1FA6D\relax}
> \centerline {\strut}
> \centerline {\strut}
> \centerline {\redpieces \char "1FA60\relax \ \char "1FA61\relax \ \char
> "1FA62\relax \ \char "1FA63\relax \ \char "1FA64\relax \ \char "1FA65\relax
> \ \char "1FA66\relax}
> \centerline {\strut}
> \centerline {\blackpieces \char "1FA67\relax \ \char "1FA68\relax \ \char
> "1FA69\relax \ \char "1FA6A\relax \ \char "1FA6B\relax \ \char "1FA6C\relax
> \ \char "1FA6D\relax}
> \end
>
> --
> *Philip Taylor*
>
> This email, its contents and any attachments are intended solely for the
> addressee and may contain confidential information. In certain
> circumstances, it may also be subject to legal privilege. Any unauthorised
> use, disclosure, or copying is not permitted. If you have received this
> email in error, please notify us and immediately and permanently delete it.
> Any views or opinions expressed in personal emails are solely those of the
> author and do not necessarily represent those of Royal Holloway, University
> of London. It is your responsibility to ensure that this email and any
> attachments are virus free.
>
>

Re: [XeTeX] latin-1 encoded characters in commented out parts trigger log warnings

[David Carlisle]

On Mon, 22 Feb 2021 at 01:28, Ross Moore  wrote:

> Hi Jonathan, and others.
>
>
> There’s actually a pretty easy fix, at least for XeLaTeX.
> The package contains 2 files only:   xstring.sty  and  xstring.tex .
> The .sty is just a 1-liner to load the .tex .
>
> It could be beefed up with:
>
>  \RequirePackage{ifxetex} %   is this still the best package for
>  \ifxetex   ?
> \ifxetex
>   \XeTeXdefaultencoding "iso-8859-1"
> \input{xstring.tex}
>   \XeTeXdefaultencoding "utf8"
> \else
>  \input{xstring.tex}
> \fi
>
>
That would sort of work but be a suboptimal fix, it imposes a run-time test
on everyone just to save the file in a legacy encoding when saving the file
as
utf-8 (or as ascii with accents shown as commands) has no run time cost and
places the file in the default text encoding used by almost all current
systems.

David

Re: [XeTeX] latin-1 encoded characters in commented out parts trigger log warnings

[David Carlisle]

On Sun, 21 Feb 2021 at 20:27, Ross Moore  wrote:

> Hi David,
>
> Surely the line-end characters are already known, and the bits
> have been read up to that point *before* tokenisation.
>

This is not a pdflatex inputenc style utf-8 error failing to map a stream
of tokens.

It is at the file reading stage and if you have the file encoding wrong you
do not know reliably what are the ends of lines and you haven't interpreted
it as tex at all, so the comment character really can't have an effect
here. This mapping is invisible to the tex macro layer just as you can
change the internal character code mapping in classic tex to take an ebcdic
stream, if you do that then read an ascii file you get rubbish with no hope
to recover.


So provided the tokenisation of the comment character has occurred before
> tackling what comes after it, why would there be a problem?
>
> ... just guessing the encoding (which means guessing where the line and so
> the comment ends)
> is just guesswork.
>
>
> No guesswork intended.
>
>
>> The file encoding specifies the byte stream interpretation before any tex
>> tokenization
>> If the file can not be interpreted as utf-8 then it can't be interpreted
>> at all.
>>
>>
>> Why not?
>> Why can you not have a macro — presumably best on a single line by itself
>> –
>>
>
> there is an xetex   primitive that switches the encoding as Jonathan
> showed, but  guessing a different encoding
> if a file fails to decode properly against a specified encoding is a
> dangerous game to play.
>
>
> I don’t think anyone is asking for that.
>
> I can imagine situations where coding for packages that used to work well
> without UTF-8 may well be commented involving non-UTF-8 characters.
> (Indeed, there could even be binary bit-mapped images within comment
> sections;
> having bytes not intended to represent any characters at all, in any
> encoding.)
>

That really isn't possible. You are decoding a byte stream as UTF-8, once
you get to a section that does not decode you could delete it or replace it
byte by byte by the Unicode replacement character but after that everything
is guesswork and heuristics: just because some later section happens to
decode without error doesn't mean it was correctly decoded as intended.
Imagine if the section had been in UTF-16 rather than latin-1 it is quite
possible to have a stream of bytes that is valid utf8 and valid utf-16
there is no way to step over a commented out utf-16 section and know when
to switch back to utf-8.



> If such files are now subjected to constraints that formerly did not exist,
> then this is surely not a good thing.
>

That is not what happened here.  the constraints always existed. It is not
that the processing changed, the file, which used to be distributed in
UTF-8, is now distributed in latin-1 so gives warnings if read as UTF-8.



>
> Besides, not all the information required to build PDFs need be related to
> putting characters onscreen, through the typesetting engine.
>
> For example, when building fully-tagged PDFs, there can easily be more
> information
> overall within the tagging (both structure and content) than in the visual
> content itself.
> Thank goodness for Heiko’s packages that allow for re-encoding strings
> between
> different formats that are valid for inclusion within parts of a PDF.
>

But the packages require the files to be read correctly, and that is what
is not happening.


> I’m thinking here about how a section-title appears in:
>  bookmarks, ToC entries, tag-titles, /Alt strings, annotation text for
> hyperlinking, etc.
> as well as visually typeset for on-screen.
> These different representations need to be either derivable from a common
> source,
> or passed in as extra information, encoded appropriately (and not
> necessarily UTF-8).
>
> Sure but that is not related to the problem here, which is that the source
file  can not be read or rather that it is being incorrectly read as UTF-8
when it is latin-1.

So I don't think such a switch should be automatic to avoid reporting
> encoding errors.
>
> I reported the issue at xstring here
> https://framagit.org/unbonpetit/xstring/-/issues/4
>
>
> David
>
>
> that says what follows next is to be interpreted in a different way to
>> what came previously?
>> Until the next switch that returns to UTF-8 or whatever?
>>
>>
>> If XeTeX is based on eTeX, then this should be possible in that setting.
>>
>>
>> Even replacing by U+FFFD
>> is being lenient.
>>
>>
> Why has the mouth not realised that this information is to be discarded?
> Then no replacement is required at all.
>

The file reading has failed  before any tex accessible processing has
happened (see the ebcdic example in the TeXBook)

\danger \TeX\ always uses the internal character code of Appendix~C
for the standard ASCII characters,
regardless of what external coding scheme actually appears in the files
being read.  Thus, |b| is 98 inside of \TeX\ even when your computer
normally deals with ^{EBCDIC} or some 

Re: [XeTeX] latin-1 encoded characters in commented out parts trigger log warnings

[David Carlisle]

On Sun, 21 Feb 2021 at 11:47, Ross Moore  wrote:

> Hi David.
>
> On 21 Feb 2021, at 10:12 pm, David Carlisle 
> wrote:
>
> I think that should be taken up with the xstring maintainers.
>
>
> Is  xstring  intended for use with XeTeX ?
> I suspect not.
> But anyway, there are still issues with this.
>
> (BTW, I wrote this before Jonathan Kew’s response.)
>
>
> I don't think there is any reasonable way to say you can comment out parts
> of a file in a different encoding.
>
>
> I’m not convinced that this ought to be correct for TeX-based software.
>
> TeX (not necessarily XeTeX) has always operated as a finite-state machine.
> It *should* be possible to say that this part is encoded as such-and-such,
> and a later part encoded differently.
>
> I fully understand that editor software external to TeX might well have
> difficulties
> with files that mix encodings this way, but TeX itself has always been
> byte-based
> and should remain that way.
>
> A comment character is meant to be viewed as saying that:
>  *everything else on this line is to be ignored*
> – that’s the impression given by TeX documentation.
>


But you only know it is a comment character if you can interpret the
incoming byte stream
If there are encoding errors in that byte stream then everything ls is
guess work.

In this particular case with mostly ascii text and a few latin-1 characters
it may be that you can guess that
the invalid utf-8 is in fact valid latin1 and interpret it that way, and
the guess would be right for this file
but what if the non-utf8 file were utf-16 or latin-2  or ... just guessing
the encoding (which means guessing where the line and so the comment ends)
is just guesswork.



> If it is the documentation that is incorrect, then it should certainly be
> clarified.
>
> For XeTeX and this particular example, it’s probably just a matter of
> checking
> that the non-UTF8 characters occur *after* a UTF-8  ‘%' , and not issuing
> an error message under these conditions.
> A warning, maybe, but not an error.
>



>
> The file encoding specifies the byte stream interpretation before any tex
> tokenization
> If the file can not be interpreted as utf-8 then it can't be interpreted
> at all.
>
>
> Why not?
> Why can you not have a macro — presumably best on a single line by itself –
>

there is an xetex   primitive that switches the encoding as Jonathan
showed, but  guessing a different encoding
if a file fails to decode properly against a specified encoding is a
dangerous game to play.
So I don't think such a switch should be automatic to avoid reporting
encoding errors.

I reported the issue at xstring here
https://framagit.org/unbonpetit/xstring/-/issues/4


David


that says what follows next is to be interpreted in a different way to what
> came previously?
> Until the next switch that returns to UTF-8 or whatever?
>
>
> If XeTeX is based on eTeX, then this should be possible in that setting.
>
>
> Even replacing by U+FFFD
> is being lenient.
>
> David
>
>
>
>
> On Sun, 21 Feb 2021 at 11:04, jfbu  wrote:
>
>> Hi,
>>
>> consider this
>>
>> \documentclass{article}
>> \usepackage{xstring}
>> \begin{document}
>> \end{document}
>>
>> and call it xexstring.tex
>>
>> Then xelatex xexstring triggers 136 warnings of the type
>>
>> Invalid UTF-8 byte or sequence at line 35 replaced by U+FFFD.
>>
>> Looking at file
>>
>> /usr/local/texlive/2020/texmf-dist/tex/generic/xstring/xstring.tex
>>
>> I see that this matches with use of latin-1 encoded characters in
>> comments.
>>
>> Notice that it is a not a user decision here to use a latin-1
>> encoded file.
>>
>> In fact I encountered this in a file I was given where
>> xstring package was loaded by another package.
>>
>> Regards,
>>
>> Jean-François
>>
>
>
> Cheers.
>
> Ross
>
>
> Dr Ross Moore
> Department of Mathematics and Statistics
> 12 Wally’s Walk, Level 7, Room 734
> Macquarie University, NSW 2109, Australia
> T: +61 2 9850 8955  |  F: +61 2 9850 8114
> M:+61 407 288 255  |  E: ross.mo...@mq.edu.au
> http://www.maths.mq.edu.au
>
> CRICOS Provider Number 2J. Think before you print.
> Please consider the environment before printing this email.
>
> This message is intended for the addressee named and may
> contain confidential information. If you are not the intended
> recipient, please delete it and notify the sender. Views expressed
> in this message are those of the individual sender, and are not
> necessarily the views of Macquarie University. <http://mq.edu.au/>
> <http://mq.edu.au/>
>
>

Re: [XeTeX] latin-1 encoded characters in commented out parts trigger log warnings

[David Carlisle]

I think that should be taken up with the xstring maintainers.

I don't think there is any reasonable way to say you can comment out parts
of a file in a different encoding.

The file encoding specifies the byte stream interpretation before any tex
tokenization
If the file can not be interpreted as utf-8 then it can't be interpreted at
all. Even replacing by U+FFFD
is being lenient.

David




On Sun, 21 Feb 2021 at 11:04, jfbu  wrote:

> Hi,
>
> consider this
>
> \documentclass{article}
> \usepackage{xstring}
> \begin{document}
> \end{document}
>
> and call it xexstring.tex
>
> Then xelatex xexstring triggers 136 warnings of the type
>
> Invalid UTF-8 byte or sequence at line 35 replaced by U+FFFD.
>
> Looking at file
>
> /usr/local/texlive/2020/texmf-dist/tex/generic/xstring/xstring.tex
>
> I see that this matches with use of latin-1 encoded characters in comments.
>
> Notice that it is a not a user decision here to use a latin-1
> encoded file.
>
> In fact I encountered this in a file I was given where
> xstring package was loaded by another package.
>
> Regards,
>
> Jean-François
>
>
>

Re: [XeTeX] asterisk in font name - font error

[David Carlisle]

 >if that doesn't go anywhere, I will try at latex2e's (their bug tracker
is much more active than fontspec's).

Please don't do that, it just increases our administrative overload.

If it is fixable on our side then all relevant people will already have
seen your report here and at fontspec github, posting it again at the 2e
github (where there is no relevant code) will just mean that we see it
again.

David

Re: [XeTeX] asterisk in font name - font error

[David Carlisle]

On Mon, 14 Dec 2020 at 16:10, Apostolos Syropoulos via XeTeX 
wrote:

> >Works for me, Ulrike :
> >
> >
> > \font \Jost = "Jost*"
> > \Jost Hallo
> >
> > \end
> >
> >
> >Of course, one has to install the font in the first place.
>
> The above code works just fine in Solaris.
> A.S.
>
> --
> Apostolos Syropoulos
> Xanthi, Greece
>
>
>
For me (testing on cygwin and installing the fonts in ~/.fonts)  the Phil's
plain tex example works but I think it is somewhat underspecified which
font you actually get (I got Jost-Black which is a bit.. black...)


$ pdffonts phil.pdf
name type  encoding emb
sub uni object ID
 -  ---
--- --- -
HMHLTQ+Jost-BlackCID TrueType  Identity-H   yes
yes yes  5  0
JRRHPJ+CMR10 Type 1C   Builtin  yes
yes yes  6  0



fontspec does get confused by the  * (probably while trying to figure out
the bold/italic etc variants of the main font) and I  get the

! Package fontspec Error: The font "JostJost*" cannot be found.

error

Which is arguably a fontspec bug but since the "main" font is of a rather
arbitrary weight if loaded as  Jost* it is probably better to specify the
weights as in Ulrike's example anyway.

I did try to avoid the wildcard using a catcode 11 *,  interestingly that
gives a different error, so gets closer but doesn't actually work (for some
definition of "interesting")

\documentclass{article}
\usepackage{fontspec}

\setmainfont{Jost\Ucharcat`\* 11 }

\begin{document}

Hallo

\end{document}

Re: [XeTeX] problem with ~ in Mapping=velthuis-sanskrit

[David Carlisle]

>\catcode`\~=12%

changing catcodes in an environment looks wrong (the % after the 12 is
definitely wrong)  if you change catcodes then the definition cannot work
in the argument of any command (for the same reasons that \verb does not
work in command arguments) . Leave ~ active but make it expand to the
inactive one eg

\edef~{\string~}%

Re: [XeTeX] Problem with ifpdf switch

[David Carlisle]

I get no spurious output from this document with xetex

\input ifpdf.sty
\input eplain

xxx

\bye


However you do not have the current version of ifpdf.sty your text is
showing a 2016 date which presumably means you have an old copy somewhere
on a local input path that is masking the one you would have installed with
a current installation.



The current  version of ifpdf.sty as distributed with texlive2020 has the
line

  \ProvidesPackage{ifpdf}[2019/10/25 v3.4 ifpdf legacy package. Use iftex
instead.]

So will not have made the text that you show.

Some other macro that you have loaded has apparently defined
\ProvidesPackage so that instead of sending the final argument to the log
it typesets it, but impossible to say what that is from the information
posted.

David


On Mon, 6 Jul 2020 at 18:31, Julian Bradfield 
wrote:

> On 2020-07-06, John Was  wrote:
> > On acquiring a new PC I decided to install the latest TeXLive
> (previously I
> > was on the 2013 version).  I use plain XeTeX.  To my surprise, the first
> > page now has at the top the text:
> >
> > ifpdf [2016/04/04 v3.0 Provides the ifpdf switch]
> >
> > See the attached one-page PDF.
> >
> > Below I give some lines from my log file - it's the last line that is of
> > interest, viz.:
>
> No, that's not the line of interest. The lines of interest are all the
> other files you might have loaded that might cause this breakage.
> The line you see is, naturally, coming from ifpdf.sty, and to see it,
> something must have broken the \ProvidesPackage, messed with catcodes,
> or done something else to cause the break, unless the (rather old)
> ifpdf.sty itself is buggy, which seems unlikely.
>
> Please construct a minimal (non-)working example, and post the full
> log file, with links to all non-standard files loaded.
>
> Unless of course somebody jumps in to say they recognize the problem!
>

Re: [XeTeX] Colour specials for XeTeX

[David Carlisle]

Phil wrote
> However, it would seem that while the "xdv2pdf" driver was happy with
\special {color push}, the "xdvipdfmx" driver is not.

I doubt that ever worked, the semantics of "push" on any stack mean you
have to say what you are putting on the stack, unlike pop where you just
take off the top element (or underflow)

most of the color specials are in section 7 of texdoc dvipdfm, but that has
a different syntax for its colour stack push/pop (the begincolor (bc)
special which isn't used in the latex back end at all)

I don't suppose you count the most reliable documentation?

https://github.com/TeX-Live/texlive-source/blob/trunk/texk/dvipdfm-x/spc_color.c#L167

Re: [XeTeX] Colour specials for XeTeX

[David Carlisle]

On Sat, 6 Jun 2020 at 13:40, Philip Taylor 
wrote:

> Thank you all for your comments and explanations.  One question to
> Akira-san, though :
>
> > If you use -no-pdf option and apply xdvipdfmx to a created
> > .xdv file, you can see messages like
> >
> > xdvipdfmx:warning: xetex-style \special{x:rulecolorpush} is
> > not supported by this driver;
>
> Is there any way (perhaps through a command-line option to XeTeX) that one
> could obtain those most usedful diagnostics using XeTeX in "standard" mode,
> i.e. without using the "-no-pdf" option and without running xdvipdfmx
> explicitly ?
>


xetex --output-driver=dvipdfmx


will make it use the default dvipdfmx settings rather than quiet mode


$ xetex --output-driver=dvipdfmx cc942
This is XeTeX, Version 3.14159265-2.6-0.92 (TeX Live 2020) (preloaded
format=xetex)
 restricted \write18 enabled.
entering extended mode
(./cc942.tex [1]stdin -> stdout
[1
dvipdfmx:warning: xetex-style \special{x:textcolor} is not supported by
this driver;
update document or driver to use \special{color} instead.
]
1687 bytes written
 )
Output written on cc942.pdf (1 page).
Transcript written on cc942.log.



/Philip Taylor/
>

Re: [XeTeX] Xetex, Latex3, unicode-math and Math font not found

[David Carlisle]

 > \exp_args:Nxx \setmathfont
>   { \tl_use:N \c__test_MathFont_tl }
>   { [ \tl_use:N \c__test_MathFontSpec_tl ] }
>
> Thanks for the explanations but this form does not work

the x expansion leaves in the braces so this ends up as

\setmainfont{somefontname}{[expansionof \c_...]}

but the braces prevent the optional argument from being seen.

it's not really an expl3 (or xetex) question, you would see the same with

\edef\foo{\noexpand\setmainfont{\zzz}{[\qqq]}}
\foo

(which is more or less what x-expansion is)

Re: [XeTeX] macro-problem

[David Carlisle]

On Tue, 12 May 2020 at 16:16, Herman Grootaers 
wrote:

> On 05-05-20 08:34, Herman Grootaers wrote:
> > I am flabbergasted with this macro I am setting up. I must miss
> > something from reading the documentation.
> >
> >
> > It is a part of a macro that is much longer, only a conditional
> > partially printing of a header, indexing on a parameter and
> > color-formatting - just the nice things to do in a macro - , but this
> > misbehaves on the typesetting of the main event of the macro.
> >
> >
> > The output is one page long, there are a number of underfull
> > verticals, which are resolved in the final version with all bells and
> > whistles.
> >
> >
> > I hope someone can help me with this strange behaviour.
> >
> >
> > Thanks,
> >
> > Herman Grootaers
> >
> >
> > P.S. Please reply to the channel, I do not like it when someone finds
> > only the question but not the answer.
> >
> Well, I did not find out what went wrong, but changing the call to
> lettrine with a default parameter did the trick I was looking for:
>
>
> \newcommand{\BLAPRNT}[3][4]{\lettrine[lines=#1]{#2}{}#3} in the preamble;
>
> \blaprint{I}{\blatext} in the main body or
>
> \blaprint[2]{I}{\blatext} to change the default number of used lines to
> 2 or the used parameter.
>
>
> It took me a week to dig through google to get the result I wanted. And
> also I sometime have to remind that LaTeX sometimes forgets that the
> \parshape should be 0 as it sometimes keeps holding the lettrine space
> in memory.
>
>
> Thanks.
>
> Herman Grootaers
>
> Your original code was calling \lettrine inside a group which you have
removed here which accounts for the changed behaviour.

in

{Normal call of the package lettrine so 4 lines are used\\
 \lettrine{#2}{}#3}

\lettrine is inside the {} group and the paragraph doesn't end until after
the  group finishes, by which point the \lettrine settings were lost.

David

Re: [XeTeX] macro-problem

[David Carlisle]

I don't have the fonts so couldn't get quite the result you want but you
are using \lettrine mid-paragraph, and keeping the paragraph text in a
group that ends before the paragraph ends, both of these are likely to have
strange effects with macros like \lettrine that rely on setting \parshape

The
Underfull \hbox (badness 1) in paragraph at lines 33--35

are due to mis-placed \\ which should never be at the end of the paragraph


You don't really say what output you expect but if I remove the suspicious
looking groups and \\ I get the attached which seems to work


David


P_WIP_SMLL_EN.tex
Description: Binary data

Re: [XeTeX] A LaTeX Unicode initialization desire/question/suggestion

[David Carlisle]

On Mon, 13 Jan 2020 at 03:41, Doug McKenna  wrote:

> >| Are there architectural reasons preventing you from having a
> >| format file, or is it simply that you hope to make loading quick
> >| enough that you do not need it?
>
> Yes, and yes.  JSBox does not depend on an internal array of integers (mem[] 
> or whatever).  Doing so makes it essentially impossible to use a modern-day 
> debugger to examine data structures.  Every data structure that is allocated 
> is done so (indirectly) via malloc() or whatever the equivalent might be on 
> some system.  This makes it harder to create a \dump format file, though not 
> impossible.  But it wouldn't be (or need to be) compatible with anything in 
> the official TeX world.  Regardless, my goal is to see how far one can get 
> without needing format files.

You could probably look at what luatex is doing as it is possibly
closer to you here than xetex: the sources are in C and use dynamic
memory allocation replacing almost all of tex's fixed arrays, but they
do manage to dump formats.

> Also, see below.
>
> >| The pressure to load more into a
> >| format is likely to increase rather than decrease, people often
> >| routinely make custom formats preloading large packages like tikz or
> >| pstricks for example.
>
> True, but there is a fundamental difference between what I'm working toward, 
> and what the TeX infrastructure does.  In the TeX world, every job is a 
> single process.  Every time a TeX job is done, a process is launched, the job 
> gets done, and the program ends.  It's the Unix/command-line way.  So the 
> format has to be loaded (fast) on every job.  Makes perfect sense.



This seems really quite strange to me I can think of very few user
level programs that are compiled from source on every use, it is
placing a big extra burden on the programmer (the latex team in this
case) to write the program with all code paths optimised for run time
speed rather than being able to offload expensive testing to compile
time so it is only done once.

David

Re: [XeTeX] A LaTeX Unicode initialization desire/question/suggestion

[David Carlisle]

On Mon, 13 Jan 2020 at 03:41, Doug McKenna  wrote:
>
> David Carlisle wrote:
>
> >| Note this list is for the xetex extended tex,
> >| but the issues you raise are unrelated to xetex
> >| but to the latex format initialisation so this
> >| is not really the right list.
>
> I checked all the lists before posting, at
>
>   <https://tug.org/mailman/listinfo>
>
> and wasn't sure which would be best, because there is no mailing list listed 
> there re LaTeX that seemed appropriate.  I thought that because it involved 
> Unicode and XeTeX's primitives, it was on-topic for here.  Apologies.  What 
> is the appropriate list?

I doubt the list owners will mind the occasional non-xetex thread,
mainly it was a marker in case there were going to be more jsbox
discussions.
probably texhax (on this server) or latex-l
(https://listserv.uni-heidelberg.de/cgi-bin/wa?A0=LATEX-L)  would be
more suitable or to actually suggest code changes an issue raised at
https://github.com/latex3/unicode-data/issues
would be could as then that can be automatically referenced in any
code commit logs.

>
> >| I'm happy to see that you imply that the name might change,
> >| I think the JS prefix would prove very confusing should you
> >| distribute it under that name, as it isn't JavaScript based,
> >| especially as there are JavaScript ports of the texlive stack
> >| these days. Not to mention existing JavaScript apps called
> >| jsbox https://apps.apple.com/sg/app/jsbox-learn-to-code/id1312014438
>
> Agreed.  For now, I just need a word to refer to the thing.
>
> >| Not quite sure why you say that is unlike xetex? xetex and luatex
> >| expose unicode characters as single characters before TeX tokenisation
> >| happens, perhaps you are thinking of the TeX macro based UTF-8 decoder
> >| we use with (pdf)teX ?
>
> Quite possible.  I've only focused on some of XeTeX's source code.
>
> >| Note that you are looking at the older releases.
> >| For testing it would be better to test against the
> >| latex-dev releases, which preload more into the format
> >| (again saving time by not having to read the Unicode
> >| data when the expl3 package is loaded eg by fontspec).
> >|
> >| xelatex-dev is currently Pre release 2, but Pre release 3 is expected
> >| to be released in a few days, or the first full release with expl3
> >| preloaded in the format is due in the first week of February.
>
> >| Note the latex macros will still need to read the Unicode Data files
> >| to initialise structures held in TeX macros even if the lccode and
> >| uccode tables were pre-initialised, so it would not be possible to
> >| avoid all reading of the Unicode files.
>
> I don't really understand this ("structures held in TeX macros").  But see 
> below.
>
> >| load-unicode-data handles some of the reading, but there is additional
> >| reading  (see l3unicode.dtx) that is in expl3.sty (in current xelatex
> >| fomats) but will be preloaded in future releases and in the current
> >| xelatex-dev release as noted above.
>
> I tried looking at, e.g., l3unicode.dtx, and it's still using TeX (or 
> impenetrable LaTeX3 kernel language built on top) to parse the official 
> Unicode data files.  It's hard for me to imagine how any of that isn't at 
> least an order of magnitude slower than scanning through a mere 20K block of 
> bytes with a machine pointer in C, and installing into all pertinent 
> character mapping tables every piece of information that XeTeX says it's 
> interested in on a per character or per character range basis.  When I use 
> the term "preloaded" I'm not talking about parsing anything inside TeX's 
> virtual machine using the TeX language (or whatever's built on top of it).

Obviously in C you can parse it quicker but the final structure isn't
a simple array of integers like the lccode table, so at the engine
level you have nowhere to put the data, the Unicode files have lots of
additional information (notably from CaseFolding.txt SpecialCasing.txt
and similar files. The information needs to be stored in
latex-specified macros so the engine can't pre-populate that other
than by the traditional way to initialise a state of macro definitions
which is a format file (or something functionally equivalent to that).

>
> >| It wouldn't be appropriate to have a primitive just to control the
> >| behaviour of one higher level macro system but there could be tests
> >| there for additional cases, as you say.
>
> See my response regarding a \catcode test to Phil Taylor also in this thread.
>
> >| While jsbox is just in your private development code you could also
> >| simply arrange modified or empty tex file

Re: [XeTeX] A LaTeX Unicode initialization desire/question/suggestion

[David Carlisle]

latex-dev releases you are still going to need
the unicode data files to be read using tex macros. Before making any
changes to the tex macros you may want to do timings with the these
versions. It may be that you choose to reconsider not making (the
equivalent of) format files, as just saving the time for setting the
lccodes may be a less significant proportion of the startup time.

>
> Are there any downsides to this minor change that I'm missing?  Is there a 
> better name for the primitive?  What can I do to encourage that the above 
> test be officially added to "load-unicode-data.tex"?

To be in the core tex macros we would need to have the engine
incorporated into texlive so that it could be tested as part of our
test suite and continuous integration tests.
However as already discussed in this thread there are several
possibilities for you to build something along those lines without
requiring any changes to the core macro files, so lack of change here
shouldn't be seen as a discouragement and anyway gives you more
flexibility with changing names etc while jsbox is being developed.

Returning to your original question as to what constitutes a "Unicode"
TeX for LaTeX, we have put some data on the requirements  for extended
TeX features in the draft ltnews31 which will be part of next week's
latex-dev release, but you can see the sources now at

Primitive Requirements:
https://github.com/latex3/latex2e/blob/develop/base/doc/ltnews31.tex#L596

see also

Improved load-times for expl3:
https://github.com/latex3/latex2e/blob/develop/base/doc/ltnews31.tex#L169

on the additional items preloaded in the format.



>
>
> Doug McKenna
> Mathemaesthetics, Inc.


David Carlisle
for the LaTeX3 Project

Re: [XeTeX] Lowercase Unicode code points in hyphenation patterns

[David Carlisle]

On Sun, 24 Nov 2019 at 19:10, Doug McKenna  wrote:
>
> Is xgreek.sty loaded as part of creating the LaTeX format?

No

> If not, my understanding is that its corrections wouldn't affect any of the 
> hyphenation patterns installed from xetex.ini during the format build.
>
> Perhaps this doesn't matter.
>
> - Doug McKenna
>
>

as long as the lowercase letters that appear in the patterns have non
zero lccodes when the patterns are read then adjusting them after is
fine, and the uppercase codes don't affect hyphenation at all.

David

Re: [XeTeX] Lowercase Unicode code points in hyphenation patterns

[David Carlisle]

On Sun, 24 Nov 2019 at 18:41, Apostolos Syropoulos via XeTeX
 wrote:
> Of course these tables are all wrong but this is another problem.

Yes there is that.

However it seems better to start from a known standardised base shared
with basically everyone then fix as needed rather than try to come up
with a tex-specific set of mappings covering the whole Unicode code
range and having to document and maintain them and extend each year as
more characters are added.

> I have added the correct \uccodes and \lccodes in xgreek.sty

thanks:-)

David

Re: [XeTeX] Lowercase Unicode code points in hyphenation patterns

[David Carlisle]

the lccode tables are set by the macro layer not the engine code, it
reads in The Unicode consortium data file
tex/generic/unicode-data/UnicodeData.txt
and sets the lccode values and catcode values according to the data there.

see

tex/generic/unicode-data/load-unicode-data.tex


which is loaded at format creation

tex/generic/tex-ini-files/xetex.ini

David

On Sun, 24 Nov 2019 at 02:12, Doug McKenna  wrote:
>
> When the LaTeX format is built, there are tests for whether or not a 
> Unicode-aware TeX engine is doing the work.  I presume that XeTeX is such a 
> Unicode-aware engine, though I'm not familiar with what the definition of 
> "Unicode-aware TeX engine" actually is (separate issue).
>
> During the input of various hyphenation pattern files (a group for each 
> language code), the first such file that uses non-ASCII Unicode code points 
> is for Ancient Greek, in the file
>
> /usr/local/texlive/2017/texmf-dist/tex/generic/hyph-utf8/patterns/texhyph-grc.tex
>
> at line 61, which starts out
>
> α1 ε1 η1 ι1 ο1 υ1 ω1 ϊ1 ...
>
> TeX's code and specification says that only lowercase letters can appear in 
> pattern words, and the definition within TeX's source code of a lowercase 
> letter is any entry in the \lccode table that, when indexed by a character, 
> delivers itself.
>
> But as near as I can tell, during the building of the LaTeX format (i.e., 
> running "latex.ltx") there is no TeX source code that installs any of these 
> Greek letters into the \lccode table.  Therefore, I'm concluding that the 
> XeTeX engine does this itself when it initializes, rather than awaiting any 
> TeX source code to do it.
>
> But there are a whole lot of lowercase letters in Unicode, so I'm wondering 
> how XeTeX determines legal lowercase letters for initial pattern files?
>
> I've tried looking at some version of the xetex.web code, but without 
> illumination, I'm afraid.
>
> TIA,
>
> Doug McKenna
>
>
>

Re: [XeTeX] Use of BOM in XeTeX and TeXworks

[David Carlisle]

On Sat, 13 Jul 2019 at 21:21, Zdenek Wagner  wrote:

> Hi all,
>
> I received files from other users with BOM in UTF-8 ans UTF-16 and
>
XeTeX correctly recognized them. Tested on Linux and Mac.
>
>
Yes sure, when I said an encoding comment was more readable I meant "more
readable for humans to understand what is happening" not that software
finds it easier to read the comment than a BOM.


David

Re: [XeTeX] Use of BOM in XeTeX and TeXworks

[David Carlisle]

more readable than a BOM you can use an encoding  comment as far as I can
see for your use case

eg as listed here

https://www.texdev.net/2011/03/24/texworks-magic-comments/

On Wed, 10 Jul 2019 at 12:07, Taylor, P  wrote:

> Two related questions, the second dependent on the answer to the first.
>
>1. Is XeTeX happy with a BOM in UTF-8 files
>2. Could TeXworks be enhanced to (a) recognise, and (b) insert if
>absent, a BOM in UTF-8 files ?
>
> The reason for my request is that while, on a day-to-day basis, I work
> solely in UTF-8, I have a suite of legacy (ISO-8859-1) files on which I
> need to continue to work.  No matter whether I configure TeXworks to
> operate by default in UTF-8 or ISO-8859-1, there invariably comes a time
> when I open a file of the other type, make a change (without noticing that
> some of the on-screen characters are wrong) and attempt to compile.  At
> that point, the file is saved with the wrong encoding, and unless I have an
> archival copy, all is lost.
>
> Philip Taylor
>

Re: [XeTeX] setting math parameters via fontdimens (or other means)

[David Carlisle]

I wrote

\ifx\Umathchar\undefined
\usepackage{lmodern}
\else
\usepackage{unicode-math}
\fi


..

Khaled kindly corrected me off list.

This test file is faulty, prior to 2013 \Umathchar wasn't defined
(just \XeTeX...) so the test file takes the classic tfm font route on
old installations, which accounts for the different output shown on
old releases.

So... this isn't a regression, sorry about the misleading initial message.

But the feature request remains, could it be possible to change the
font parameters after the initial font load so as to adjust the math
typesetting without having to use fontforge or similar to change the
parameters in a copy of the font.

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

[XeTeX] setting math parameters via fontdimens (or other means)

[David Carlisle]

This document

\documentclass{article}
\ifx\Umathchar\undefined
\usepackage{lmodern}
\else
\usepackage{unicode-math}
\fi
\begin{document}


\sbox0{$x$}


\typeout{\expandafter\meaning\the\textfont2}
\fontdimen16\textfont2=20pt

  $a_b$

\end{document}


If processed with pdflatex has an excessively dropped subscript (as
specified by fontdimen16)

xelatex prior to 2013 did the same.

But post 2013 (presumably related to using the harfbuzz math layout)
you can set the fontdimen but it has no effect.

When experimenting with matching math fonts and text fonts it is not
at all uncommon to need to tweak the math fontdimens.

Does xetex have any way to get similar effects or do you have to
modify the values in the OpenType  Math table in fontforge or similar,
so modifying the font external to TeX?

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] semantic nest size

[David Carlisle]

well yes \text makes the thing be set four times as many times as
\mbox so does stress the semantic nest a bit but not as much as that
usually, but without a reproducible test hard to say exactly what is
wrong.

On 12 April 2018 at 19:14, Apostolos Syropoulos  wrote:
>>I don't have those fonts and couldn't reproduce the error when I tried
>>switching to other fonts so I can't trace the exact cause of the
>>error, but the error is not entirely surprising, unicode-math does
>>document that it must be called after amsmath
>
> I think the error is related to the \text command. When I change
> \text to \mbox, compilation proceeds with no error, even if
> amsmath is called before unicode-math.
>
> A.S.
>
> --
> Apostolos Syropoulos
> Xanthi, Greece
>
>


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] semantic nest size

[David Carlisle]

I don't have those fonts and couldn't reproduce the error when I tried
switching to other fonts so I can't trace the exact cause of the
error, but the error is not entirely surprising, unicode-math does
document that it must be called after amsmath

I get no error from the file below (using random TeX  Gyre fonts) do
you just get the error with the fonts you used, or can you reproduce
with fonts in texlive?

David

\documentclass[a4paper]{article}
%\usepackage{xltxtra}
\usepackage{xgreek}
\usepackage{unicode-math}
\usepackage{amsmath}
\begin{document}
\setmainfont[Mapping=tex-text,Ligatures=Common]{TeX Gyre Termes}
\setsansfont[Mapping=tex-text,Scale=MatchLowercase]{TeX Gyre Heros}
\setmonofont[Scale=MatchLowercase]{TeX Gyre Cursor}
\setmathfont[Scale=MatchUppercase]{Asana Math}
\begin{displaymath}
\alpha^{\nu}=\underbrace{\alpha\cdot\alpha\cdots\alpha\cdot\alpha}_{\text{$\nu$
παράγοντες}}.
\end{displaymath}
\end{document}

On 12 April 2018 at 17:57, Apostolos Syropoulos  wrote:
> Hello,
>
> I have this code
>
> documentclass[a4paper]{article}
> \usepackage{xltxtra}
> \usepackage{xgreek}
> \usepackage{unicode-math}
> \usepackage{amsmath}
> \begin{document}
> \setmainfont[Mapping=tex-text,Ligatures=Common]{Garamond Premier Pro}
> \setsansfont[Mapping=tex-text,Scale=MatchLowercase]{Myriad Pro}
> \setmonofont[Scale=MatchLowercase]{Courier}
> \setmathfont[Scale=MatchUppercase]{Asana Math}
> \begin{displaymath}
> \alpha^{\nu}=\underbrace{\alpha\cdot\alpha\cdots\alpha\cdot\alpha}_{\text{$\nu$
> παράγοντες}}.
> \end{displaymath}
> \end{document}
>
> I try to process it with XeLaTeX but it fails:
>
> ! TeX capacity exceeded, sorry [semantic nest size=500].
>  {
>
> l.14 ...{\alpha\cdot\alpha\cdots\alpha\cdot\alpha}
>   _{\text{$\nu$
> παράγοντες}}.
> No pages of output.
>
> When I have
>
> \usepackage{amsmath}
> \usepackage{unicode-math}
>
> It compiles just fine. What is wrong? Any ideas?
>
> A.S.
>
> --
> Apostolos Syropoulos
> Xanthi, Greece
>
>
>
> --
> Subscriptions, Archive, and List information, etc.:
>   http://tug.org/mailman/listinfo/xetex
>



--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Xetex equiv to luatex's \directlua{}

[David Carlisle]

The format is described in the TeXBook but in short

.\hbox(7.71974+2.25569)x360.0,

is a box of height 7.71974pt, depth 2.25569 and width 360.0pt  nested
one level inside some other box

\hbox(0.0+0.0)x17.0

is a box of height and depth 0 and width 17pt (it will be the
indentation box) that is nested 4 levels deep from the outer page box.


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Xetex equiv to luatex's \directlua{}

[David Carlisle]

there are several ways to get the box output in classic tex (or xetex)
although perhaps the easiest (and safest in terms of not accidentally
affecting the typeset positions)  is to use \showoutput so all boxes
are (somewhat verbosely) logged in the log file, and then parse that
with perl or python or whatever to get whatever lengths you need,

David


On 23 March 2018 at 20:33, maxwell  wrote:
> I'm just finishing up a project that involved typesetting text in several
> languages, while outputting an XML file that defined in X/Y coordinates the
> position and size of the bounding box surrounding each line of text in the
> PDF.  I used Luatex, because that made it possible to call Lua from Luatex
> using the \directlua{} command) to pass information to Lua, and to return
> information from Lua to Luatex using tex.print().  I also used Lua to write
> the XML file.
>
> Too late, I discovered that LuaTeX botches the rendering of one of the
> languages, Tamil.  Tamil has a complex script, with some typical Indic
> script features; so presumably LuaTeX would also mess up on other languages
> with complex scripts.  XeTeX of course does just fine at rendering text in
> complex scripts.
>
> As I say, it's too late to change now, but is there any way I could have
> done something similar using xetex?  That is, called another programming
> language to output box positions and sizes.  I suppose it's possible to
> write to an XML file in xetex natively, but I'm not sure how I could get the
> positions and sizes of boxes.  My style sheet defines a command,
> \outputpara{}, that requires the user to specify the X-position of the
> paragraph, and hence of lines in the paragraph, where line breaks are of
> course decided on the fly.  The command optionally specifies the Y-position
> of the paragraph, but the Y-position of each line in the paragraph--except
> the first--is determined by the usual TeX algorithms.  Getting TeX to tell
> me those Y-positions, as well as the vertical size of the box, was the
> difficult part.  But maybe I was missing something obvious?
>
>Mike Maxwell
>University of Maryland
>
>
> --
> Subscriptions, Archive, and List information, etc.:
>  http://tug.org/mailman/listinfo/xetex


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] "typeset by bidi" message

[David Carlisle]

On 28 January 2018 at 01:25,  wrote:

> > ...
> I think there is a human issue here that will not be solved by a purely
> technical fix.
>
>
Exactly. in this case, I asked that the package flip its default to off so
it only adds the text if requested and Vafa kindly agreed and has just sent
an updated package to ctan.

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] "typeset by bidi" message

[David Carlisle]

Getting the timing right with passing this option to bidi via polyglossia
is a bit tricky see

https://tex.stackexchange.com/questions/411907/suppress-bidis-logo-when-using-polyglossia

the bidi author gives some justification for this here

https://github.com/tex-xet/bidi/issues/60

but to be honest I think that this is a really bad default, and polyglossia
probably should disable this while loading bidi rather than rely on the
user to find an undocumented option in an internally loaded package.

David

On 26 January 2018 at 02:50, Bruno Le Floch  wrote:

> Hello Kamal,
>
> On 01/25/2018 09:10 PM, Kamal Abdali wrote:
> > My xelatex-processed document has this message on the title page of the
> > document: *"Typeset by the bidi package."*
> > *​* ​
> > I understand that bidi is being called by polyglossia which I am using.
> >
> > I hadn't
> > ​seen ​
> > such a message before. Is there a way to avoid
> > ​getting ​
> > this message?
> >
> > Kamal Abdali
>
> After loading the polyglossia package (but before selecting a language
> that reads right-to-left) try adding
>
> \usepackage[logo=off]{bidi}
>
> Strangely the default is logo=on and the option is not documented.
>
> Bruno
>
>
> --
> Subscriptions, Archive, and List information, etc.:
>   http://tug.org/mailman/listinfo/xetex
>


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Conflict among XeLaTeX, LaTeX.mk, and pdfpages

[David Carlisle]

(apparently) you need to run

extractbb testa.pdf

after processing testa.tex and before testb.tex

not sure if pdfpages should not require this but in any case the issue
seems to be at the macro layer not
in the xetex or xdvipdfmx binaries.

David


On 1 January 2018 at 23:22,   wrote:
> I've run into a problem when using all three of XeLaTeX, LaTeX.mk, and
> pdfpages, in their current versions from the latest TeXLive.  It appears
> that in order to determine file dependencies, LaTeX.mk runs the TeX engine
> with the texdepends package (which is part of LaTeX.mk) wedged into the
> input file; then texdepends intercepts a bunch of internal macros used by
> various graphics packages including indirectly by pdfpages, and the result
> is a failure with "arithmetic overflow" when trying to include PDF files.
>
> All three elements seem to be necessary:  it doesn't happen with other TeX
> engines I've tried; when invoking XeLaTeX manually instead of through
> LaTeX.mk; nor without using pdfpages.  However, I can also reproduce the
> problem by loading texdepends in my .tex file (with \RequirePackage - it
> must be loaded before \documentclass and \usepackage cannot be used then)
> and running xelatex from the command line instead of through make.
>
> The relevant messages in the log file when it fails look like:
>
> Package texdepends Warning: No 'testa.xbb' file
> (texdepends)using 1 for graphic dimensions on input line 31.
>
> File: testa.pdf Graphic file (type pdf)
> 
> ! Arithmetic overflow.
>  \calc@Acount
>
> l.31 \includepdf[pages=-]{testa.pdf}
>
> Any thoughts on how this might be fixable?
>
> --
> Matthew Skala
> msk...@ansuz.sooke.bc.ca People before principles.
> http://ansuz.sooke.bc.ca/
>
>
> --
> Subscriptions, Archive, and List information, etc.:
>   http://tug.org/mailman/listinfo/xetex
>


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Problem involving \includegraphics after texlive update on Debian Tesing

[David Carlisle]

\Gin@log is defined in the core graphics.sty so it would seem that you
have updated  xetex.def but not graphics itself, or perhaps have an
old copy in your path

David


On 7 July 2017 at 13:09, Joseph Wright  wrote:
> On 07/07/2017 12:54, Johann Spies wrote:
>> After a recent upgrade of texlive to 2017.20170629-1 on Debian I am
>> experiencing a problem compiling a longstanding document and I can
>> replicate the problem with the following code:
>>
>> \documentclass[12pt,a4paper]{article}
>> \usepackage{fontspec} % Gebruik met xelatex
>> \usepackage{graphicx} % Gebruik met xelatex
>> \usepackage[hyperindex=true,colorlinks=true,bookmarks]{hyperref}
>> \usepackage{colortbl}
>> \setmainfont[Ligatures=TeX,Mapping=tex-text]{Linux Libertine O}
>> \begin{document}
>>
>>
>>
>> \includegraphics{fruits.jpg}
>>
>> \end{document}
>>
>> %%% Local Variables:
>> %%% mode: latex
>> %%% TeX-engine: xetex
>> %%% TeX-master: t
>> %%% End:
>>
>>
>> results in
>>
>> ERROR: Undefined control sequence.
>>
>> --- TeX said ---
>> \Ginclude@bmp #1->\Gin@log
>>{<#1>}\bgroup \def \@tempa {!}\special 
>> {pdf:image...l.11 \includegraphics{fruits.jpg}
>>
>> --- HELP ---
>> TeX encountered an unknown command name. You probably misspelled the
>> name. If this message occurs when a LaTeX command is being processed,
>> the command is probably in the wrong place---for example, the error
>> can be produced by an \item command that's not inside a list-making
>> environment. The error can also be caused by a missing \documentclass
>> command.
>>
>> You can replace "fruit.jpg" with any jpg to replicate the problem.
>>
>> It will help me if someone on this list can identify what is causing
>> this.
>>
>> Regards
>> Johann
>
> Could you add \listfiles to your input and post the resulting *File
> list* from the .log?
>
> Joseph
>
>
>
> --
> Subscriptions, Archive, and List information, etc.:
>   http://tug.org/mailman/listinfo/xetex


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] ".ttf" automatically added to font names

[David Carlisle]

> So the only question remaining is: Will the xetex compiler developers
keep it like this


Note that mostly this is not a xetex question as you are using the syntax
of the fontspec macro package rather than the xetex primitive font loading
syntax (although of course these are not unrelated)

it might be more portable to use the opentype cm Unicode distributed with
tex than the ttf ones that you are pulling from /usr/share/fonts eg I have

/usr/local/texlive/2017/texmf-dist/fonts/opentype/public/cm-unicode/cmunrm.otf

I don't have them as .ttf  as far as I can see in tl2016 or tl2017.

David




On 12 May 2017 at 15:02, Dirk Hünniger  wrote:

> Hi,
>
> this works for me:
>
> \setmainfont[Path=/usr/share/fonts/truetype/cmu/,UprightFont
> =cmunrm,BoldFont=cmunbx,ItalicFont=cmunti,BoldItalicFo
> nt=cmunbi]{cmunbx.ttf}
>
> but this
>
> \setmainfont[Path=/usr/share/fonts/truetype/cmu/,UprightFont
> =cmunrm.ttf,BoldFont=cmunbx,ItalicFont=cmunti,
> BoldItalicFont=cmunbi]{cmunbx.ttf}
>
> causes the following error
>
>>
>> kpathsea:make_tex: Invalid fontname 
>> `[/usr/share/fonts/truetype/cmu/cmunrm.ttf.ttf]/OT',
>> contains '['
>>
>>
>> 
>> !
>> ! fontspec error: "font-not-found"
>> !
>> ! The font "cmunrm.ttf" cannot be found.
>> !
>> ! See the fontspec documentation for further information.
>> !
>> ! For immediate help type H .
>> !...
>>
>> l.23 ...=cmunti,BoldItalicFont=cmunbi]{cmunbx.ttf}
>>
> So the only question remaining is: Will the xetex compiler developers keep
> it like this will a later version again require to write out the extensions
> like in
>
> \setmainfont[Path=/usr/share/fonts/truetype/cmu/,UprightFont
> =cmunrm.ttf,BoldFont=cmunbx.ttf,ItalicFont=cmunti.ttf,Bold
> ItalicFont=cmunbi.ttf]{cmunbx.ttf}
>
> I just need to know so I can use the right form when generation latex code
> in my system.
> Yours Dirk
>
>
>
>
>
> --
> Subscriptions, Archive, and List information, etc.:
>  http://tug.org/mailman/listinfo/xetex
>


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Marathi Booklet for Polyglossia

[David Carlisle]

I think you could put it on ctan, they take documentation uploads as well
as tex code eg this entry for tex-by-topic

https://www.ctan.org/pkg/texbytopic

David

On 3 May 2017 at 18:20, RD Holkar  wrote:

> Hello!
>
> I have been working on a booklet explaining how to use package polyglossia
> for Indian languages, especially, Marathi, and then Hindi and Sanskrit.
> This booklet discusses basic writing, page layout, referencing,
> bibliographies, xindy, and some minute issues with polyglossia in the above
> mentioned Devanagari languages. The nearly finished booklet is written in
> Marathi, and some people in Pune and Mumbai are using it.
> I have been thinking if I can publish it somewhere or if I can be put as
> information document in the LaTeX repository once finished. Or if there is
> something else I can do with it.
>
> Indeed, no new work is done in the booklet, and its in Marathi hence
> "publication" is not really a good choice (and a appropriate word to use
> here).
>
> Any suggestion(s), help or recommendation(s) are welcome!
>
> With best regards,
> -Rohit.
>
>
>
> --
> Subscriptions, Archive, and List information, etc.:
>   http://tug.org/mailman/listinfo/xetex
>
>


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] are the intercharclasses mentioned in interchar.pdf correct?

[David Carlisle]

On 27 March 2017 at 19:37, jfbu  wrote:
> ..

ah, I don't think I'd noticed xetex was using _quite_ so many digits
in its version number:-)
as you say the test is a bit suspect but it works in the one case it
needs to work so D guess it's OK!

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] are the intercharclasses mentioned in interchar.pdf correct?

[David Carlisle]

On 27 March 2017 at 17:18, Mike "Pomax" Kamermans
 wrote:
> Are the predefined classes from
> http://mirror.utexas.edu/ctan/macros/xetex/latex/interchar/interchar.pdf
> still correct? are classes 255 (for boundary) and 256 (for unused) still the
> right ones, are those now 4095 and 4096? (based on some answers I've run
> into on stackoverflow)
>
>
> - Mike "Pomax" Kamermtna

For newer xetex the package needs to be updated for the larger range,
the test used in teh latex format

is

\ifdim\the\XeTeXversion\XeTeXrevision\p@>0.3\p@
  \chardef\e@alloc@intercharclass@top=4095
\else
  \chardef\e@alloc@intercharclass@top=255
\fi



with \e@alloc@intercharclass@top being the maximum value allocated by
\newXeTeXintercharclass

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] emdash and endash

[David Carlisle]

On 18 February 2017 at 09:45, Zdenek Wagner  wrote:
> No, -- and --- are handled by tfm in traditional TeX, in XeTeX it is handled
> by the teckit mapping. The keywords are either Mapping=tex-text or
> Ligatures=TeX. I have not examined the packages in depth but it may be added
> automaticaly. The mapping is defined in file
> texmf-dist/fonts/misc/xetex/fontmapping/base/tex-text.map (*.map is compiled
> to *.tec by teckit-compile).
>
>

As I said, it is handled by the mapping file (which is applied by
default if the font is loaded via fontspec)
but if classic fonts are being used then this is a feature of the tfm
(even in xetex, not just in traditional tex)
so it depends what fonts the OP is using in the document.

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] emdash and endash

[David Carlisle]

You could try updating (tuenc is 2017/01/24 v2.0c in the current
release) but I don't think it's likely to be a macro issue ---
ligatures are handled by tfm metrics or the tex-text mapping depending
on the font type. Others on this list would be better able to suggest
ways of debugging a failure for the tex-text mapping to kick in.

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] emdash and endash

[David Carlisle]

It should work

\documentclass{article}
\listfiles
\usepackage{fontspec}
\begin{document}


a---b

\end{document}


makes an emdash for me with current texlive.

If you have an example that fails can you show log from such a test
(\listfiles makes a list of package versions at the end of the log)


On 17 February 2017 at 21:22, Johann Spies  wrote:
> I suspect it was after the last update of texlive (now 2016.20170123-2)
> on my Debian Testing system that it happened that
>
> --- no longer compiles like \textemdash{}
> and
> -- no longer like \textendash{}
>
> It stays as typed.
>
> I do not want to go through all my files to change --- to \textemdash{}
> etc.
>
> Is there some setting that I can insert to force xelatex to compile it
> correctly?
>
> Regards
> Johann
> --
> J.H. Spies - Tel. 021-982 2694 / 082 782 0336 / 021-808 4699(w)
>  Posbus 4668, Tygervallei 7536
>
>  "Surely he hath borne our griefs, and carried our sorrows:
>   yet we did esteem him stricken, smitten of God, and
>   afflicted. But he was wounded for our transgressions,
>   he was bruised for our iniquities: the chastisement of
>   our peace was upon him; and with his stripes we are
>   healed."Isaiah 53:4,5
>
>
> --
> Subscriptions, Archive, and List information, etc.:
>   http://tug.org/mailman/listinfo/xetex


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

[XeTeX] misaligned math if \smash used in subscripts

[David Carlisle]

The following first reported at the unicode-math github
https://github.com/wspr/unicode-math/issues/383
but the problem can be seem with plain tex.


In the tex below there are six repeated instances of a subscripted bracket
they should all be the same but the first one is misaligned.

It appears to be a misbehaviour of \mathchoice as if I change the size of the
scriptscript font the effect changes, even though there is no visible
script script text.

The attached images show the output with the file as below and with
the scriptscript font changed to

at 25pt

I've tried it with fam 0,1,2 all set up as the same font, or as here
with just fam0
using a opentype font.

David


\font\m= "[latinmodern-math.otf]/OT:script=math;language=DFLT;" at 10pt
\font\sm= "[latinmodern-math.otf]/OT:script=math;language=DFLT;" at 7pt
\font\ssm= "[latinmodern-math.otf]/OT:script=math;language=DFLT;" at 5pt

\textfont0=\m
\scriptfont0=\sm
\scriptscriptfont0=\ssm


\Umathcode`.= 0 0 `.

\hrule
$$
\hbox{$\left)\vbox to15pt{}\right.$}_{\smash.}
\hbox{$\left)\vbox to15pt{}\right.$}_{\smash.}
\hbox{$\left)\vbox to15pt{}\right.$}_{\smash.}
$$
\hrule
$$
\hbox{$\left)\vbox to15pt{}\right.$}_{\smash.}
\hbox{$\left)\vbox to15pt{}\right.$}_{\smash.}
\hbox{$\left)\vbox to15pt{}\right.$}_{\smash.}
$$
\hrule


\bye


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] unicode-math and ucharcat

[David Carlisle]

ucharcat.sty isn't needed for xetex as it's already a primitive but if
you have texlive 2016 you should have ucharcat.sty (or tlmgr install
ucharcat should install it if for some reason it isn't installed
already). That said unicode-matth could move the luatex-specific part,
but really there's no reason not to have ucharcat installed.

David




On 8 February 2017 at 21:38, Kamal Abdali  wrote:
> When using the package unicode-math, I get the XeLaTex error:
> File `ucharcat.sty' not found.
>
> XeLaTex then asks for a new filename. When I enter an empty filename,
> XeLaTex proceeds normally. But to do this every time is annoying.
>
> The conclusion from some googling was that this error should no longer
> appear. Specifically, (1) unicode-math requires ucharcat, and (2) XeTeX now
> incorporates ucharcat.
>
> But I am encountering the problem in XeTeX of TeX Live 2016/W32TeX. Any help
> would be appreciated.
>
> Kamal Abdali
>
>
>
>
>
>
>
> --
> Subscriptions, Archive, and List information, etc.:
>   http://tug.org/mailman/listinfo/xetex
>


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] why does Latin Modern Mono have some stretch and shrink with XeTeX ?

[David Carlisle]

> Using XeTeX without fontspec is a relatively unusual case and it's
> unsurprising there hasn't been a lot of time spent on testing that.  The
> engine and package go together.

It may become slightly less unusual after this week's latex release as now
latex defaults to TU (Unicode) encoding with xetex and luatex
so the initial state for \ttfamiliy is OpenType latin modern monospace

However we will put out a patch level 1 release in the next day or so that
sets the font parameters 3 and 4 to zero in the fd file as in my
previous message
so the default setting of typewriter text will not stretch word spaces
in xelatex
just as in pdf and lua latex.

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] why does Latin Modern Mono have some stretch and shrink with XeTeX ?

[David Carlisle]

For latex at least I think the thing to do is amend tulmtt.fd in base
so that it has

\DeclareFontFamily{TU}{lmtt}{%
\hyphenchar \font\m@ne
\fontdimen3\font\z@%<<
\fontdimen4\font\z@%<<
}

with the two extra lines ensuring that these two font dimens are set to 0.

(If you try this with a latex 2017/01/01 release you need to remake
the format as it is input during format creation)

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] [tex-live] Problem with XeTeX's (and perhaps other binaries -- not investigated) handling of --output-directory

[David Carlisle]

On 15 January 2017 at 09:20, Philip Taylor  wrote:

>
>
> But all of this is besides the point :  surely
>
> o XeTeX --output-directory=../dynamic-content foo (or foo.tex)
>
> should compile foo.tex, not ../dynamic-content/foo.tex
>
> ** Phil.
>


No I think not,  if you do not give a full path to a file tex (usually)
looks in . first
then looks along the path. --output-directory --output-directory is
documented
as using the specified directory before the path

texdoc web2c page 8 says

`-output-directory=dirname'
   Specify the directory dirname to which output files are
written. Also look for
   input files in dirname first, before looking along the
normal search path. See
   Section 3.4 [Output file location], page 9.


but as I said this thread just joins the many previous ones, it seems like
anyone who
uses --output-directory eventually asks about some confusing aspect of its
behaviour.

I don't think it would be better here if tex looked in the current
directory, then the output
directory then the path, as that would mean that if you add
-output-directory a document
would keep inputting stale tables of contents and aux files from the
current directory
not new ones being written elsewhere.

You can always use a full path to the input file to specify a specific file.

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] [tex-live] Problem with XeTeX's (and perhaps other binaries -- not investigated) handling of --output-directory

[David Carlisle]

Phil,

> E:\TeX\Projects\WBH\Welcome>xetex foo --output-directory=../dynamic-
content
> (./foo.tex This is the intended foo.tex )

isn't that just because the flags after the filename are ignored?

Personally I'd just add it to the end of the list of reasons never to use
--output-directory,
although that list is so long, it's hard to find the end:-)

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Are the new etex changes reflected in XeTeX? (was XeTeX bugs in bidirectional typesetting)

[David Carlisle]

On 21 November 2016 at 12:28, Vafa Khalighi  wrote:
> Hi David
>
>
> Sure \textdir works in vertical mode in luatex but luatex's behavior is
> buggy. For instance, consider the following minimal example
>
>
>
> {\textdir TRT This} is just a test.
>
>
> \bye
>
>
> As you can see there is no indent at the beginning of the paragraph and that
> indent is shifted as a space between `This' and `is' which is completely
> wrong. So to fix it, you have to issue a \leavevmode at the start of the
> paragraph but by doing that, you are already entering horizontal mode.
>
>
>
No, that was the issue you raised on the luatex list thread

http://tug.org/mailman/htdig/luatex/2011-July/003192.html

it's not buggy, just not as you expected. See Taco's comment towards
the end of that thread.


The interactions of the various direction primitives are not always
intuitive (but that possibly can't be helped) and doesn't imply a bug.

See also the thread around

http://tug.org/mailman/htdig/luatex/2012-December/003957.html



David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Are the new etex changes reflected in XeTeX? (was XeTeX bugs in bidirectional typesetting)

[David Carlisle]

On 20 November 2016 at 23:28, Arthur Reutenauer
 wrote:
>> If you do find time to look at this... the other main issue besides \specials
>> where the current model causes problems is the inability to specify direction
>> while in vertical mode. The restriction to hmode was probably needed in 
>> tex-xet
>> so the extra nodes got added at safe places but in tex--xet it's less
>> clearly needed.
>
>   Do you want to make a bug report for this?  It’s good to discuss these
> issues between us, but when it comes to implementing and distributing
> changes, it’s best to have a record of them in the bug tracker; XeTeX’s
> is on Sourceforge as you know: https://sourceforge.net/p/xetex/bugs/.
>
> Best,
>
> Arthur


Done

https://sourceforge.net/p/xetex/bugs/134/

David



--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Are the new etex changes reflected in XeTeX? (was XeTeX bugs in bidirectional typesetting)

[David Carlisle]

On 20 November 2016 at 18:19, Jonathan Kew  wrote:

> This is very interesting, I didn't know Peter had done that. I think this is
> the right way forward (for an etex-based engine such as xetex), and had been
> hoping to find the time to work on essentially the same idea in xetex, but
> so far the code exists only in my head.
>

If you do find time to look at this... the other main issue besides \specials
where the current model causes problems is the inability to specify direction
while in vertical mode. The restriction to hmode was probably needed in tex-xet
so the extra nodes got added at safe places but in tex--xet it's less
clearly needed.

The restriction to horizontal mode forces fragile code trying to get
the \beginR into \everypar
at the right time, and with tables there basically is no workaround:
you can not construct the tables in vertical mode while stacking the
columns from the right.
(You can wrap them in a box but then they won't split)

It would be really good if xetex could have commands usable in vmode
that produced the effects of luatex
below (doesn't have to be "\textdir TRT" it could be
"\addBeginRByDefaultInHlists"  it doesn't really matter...
Of course it if _was_ "\textdir TRT" that would simplify cross engine
support, but the missing functionality is the main problem, not the
differing syntax.

David


\vsize=5\baselineskip
\topskip=\baselineskip

\ifx\directlua\undefined
\TeXXeTstate=1
%
\else
%
\textdir TRT%columns assembled right to left with RTL text
% \bodydir TRT \pagedir TRT % columns added to right of page
\fi

Re: [XeTeX] XeTeX bugs in bidirectional typesetting

[David Carlisle]

Vafa Khalighi wrote


>  A) Drop the TeX--XeT model entirely and switch to LuaTeX (Omega)
> bidi model
>
>
> Unfortunately Omega's bidi model has its own set of bugs so even if
> XeTeX goes this route, we need to fix quite some issues in its bidi
> model.

Long term I think that this would be a good development for xetex.
Having tex engines with two completely different models for directionality
causes real issues for formats and packages that need to build on the
base provided by the engines. It is clear that of the currently implemented
possibilities the omega/luatex model is more functional, having more
complete
support for right to left text, but also supporting vertical text as well.

If you have a list of issues it would be good to raise them (on the luatex
list
initially would presumably be best).


>
>
> B) Switch to the original TeX-XeT model by Knuth
>
>
> I would prefer this approach (considering my over 10 years of
> experience with both TeX--XeT and Omega bidi model) as it is more
> stable than Omega's bidi model. If I recall correctly, Khaled Hosny
> did this in 2013 and when I was testing most of the problems were
> fixed. The only issue was that the original TeX-XeT model adds \beginL
> ... \endL and that would cause some problem in math typesetting
> (sorry, I can not remember what exactly it was about but I remember it
> was reported by David Carlisle). For this problem, one can look at
> TeX--XeT and see how display math are done there and we could apply
> the same approach to TeX-XeT model.
>

I really could not recommend this, the extra nodes added by tex-xet
interfere with typesetting in ways that are very hard to work around.
Particularly but not limited to math mode.
The only way shown so far to avoid those issues is not to add the nodes,
which essentially leads to tex--xet  rather than tex-xet.


Again, if you have an explicit list of issues with tex--xet that are
not shared with tex-xet it would be good if you could post them here.


David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Random number primitives

[David Carlisle]

On 14 November 2016 at 09:22, Apostolos Syropoulos
 wrote:
>>
>> Changing the source isn't the issue, are you offering also to help
>> with user support to manage the change
>> and explain to people why it's not possible to re-create documents
>> that they created before they updated
>> their tex system?
>>
>
> By following this conservative line of thought, we should
> not use OpenType fonts and stick to plain old but good Type 1
> fonts!

No, not at all. There is a big difference between offering new functionality
and offering existing functionality. when implementing existing tex features.

xetex (far more than luatex) follows tex very closely, so that in the
main, if you are
not using the extended features it works exactly like tex. All I am
suggesting is that
design philosophy should apply here.

> In fact, I would recommend dropping support for Type 1
> fonts from XeTeX. If you want Type 1 fonts, use the ancient
> engines.

>  Also, each TeX destribution comes with all these
> TeX-based engines. So these people can always use their
> ancient files with no problem!

Er no, That was entirely the point of my message.
I was replying to your offer to change the sources of pdftex. If that were done
people would _not_ be able to use their ancient (or even not ancient)
files and get the same results.

David


>
>
> A.S.
> --
> Apostolos Syropoulos
> Xanthi, Greece


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Random number primitives

[David Carlisle]

On 13 November 2016 at 16:50, Apostolos Syropoulos
 wrote:

>
> OK please let me know how the pdftex thing is implemented and I will do the
> thing with drand48. Then people can check the implementation, etc.
>
> A.S.
>

People using a fixed seed to get reproducible results would not be
well served by a change in the random number generator. Just because
something would be easy to change doesn't mean that change is always a
good idea.

A document using a fixed seed would have been generating reproducible
results with pdftex since these functions were added more than ten
years ago.

Changing the source isn't the issue, are you offering also to help
with user support to manage the change
and explain to people why it's not possible to re-create documents
that they created before they updated
their tex system?

That said, if you want to look, the version in pdftex is

@* \[7b] Random numbers.

starting at line 2507 of  source/texk/web2c/pdftexdir/pdftex.web

Sometimes change is necessary but here it really isn't clear that the
statistical properties of drand48 are superior to Knuth's algorithm
from metafont  (I've never seen any comparison of these, only between
drand48 and C rand)
Or whether any of these algorithms have any real flaws when used with
sequence lengths of the sort realistically
practical in a tex document.

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Random number primitives

[David Carlisle]

On 13 November 2016 at 16:47, Apostolos Syropoulos
 wrote:
>>
>> That's true except that as pdftex and luatex have this already and have had
>> for years, in this particular instance it's playing catchup that is
>> being suggested.
>> Not promoting novel functionality.
>>
>
> So just because the design in pdftex and luatex was conservative, if not 
> flawed,
> it doesn't mean we should follow their "lead".

that is the model in much of xetex as far as features not relating to
unicode and font support are concerned:
that something that works in (pdf)tex and xetex should work the same
way in the two systems.

> The point here is to produce
> pseudo-random numbers. How they will be produced is something that the end 
> user
> should not worry about.

It depends. currently, if you set the seed you get the same numbers
from luatex and pdftex.
if xetex uses a different RNG then you would get different numbers in
xetex. that complicates some
things not least testing cross platform compatibility.



> In addition, it would be a nice opportunity for pdftex
> and luatex to "upgrade".

Perhaps. But that wasn't the current request.
>
>
>>
>> Not really. If the main use cases are met by the existing code then engine
>> differences are just extra work for the maintainers of the macro format
>> to hide to provide a consistent interface to the end user.
>>
>
> This is a fallacy! Just becauce LaTeX can process UTF-8 encoded input files,
> it doesn't mean XeTeX is useless!

That isn't what I said.

> And there is no extra code.

Oh there would be _less_ code probably as you'd use a system call rather than
implement the function inline, that isn't the issue.

> It would be just
> a simple system call. So instead of implementing random with a call to
>
>
> long random(void)
>
>
> one should implement it with a call to
>
>
> long lrand48(void)
>
> A.S.


David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Random number primitives

[David Carlisle]

On 13 November 2016 at 15:46, Apostolos Syropoulos
 wrote:

> Let me say that I find this point of view far too conservative. OK METAFONT
> lives in the previous century but luatex and xetex are products of this
> century so they must utilize what is better. Also, the point of introducing
> XeTeX was the need to handle things TeX had no provision.

That's true except that as pdftex and luatex have this already and have had
for years, in this particular instance it's playing catchup that is
being suggested.
Not promoting novel functionality.

>  Finally, it would
> an excellent opportunity for XeTeX to do something better that the others

Not really. If the main use cases are met by the existing code then engine
differences are just extra work for the maintainers of the macro format
to hide to provide a consistent interface to the end user.

>
> Regards,
>
> Apostolos
>

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Random number primitives

[David Carlisle]

On 13 November 2016 at 13:58, Apostolos Syropoulos
 wrote:
>
>
> Thank you very much for your prompt response. I would say that
> this feature makes the typesetting engine more independent and
> this is definitely a good thing. Also, I think it should be
> implemented using the drand48 function as Neelson has already
> suggested.
>
>
> A.S.
>

the statistical properties of drand48 are almost certainly better but
metafont/pdftex/luatex all use the code from mf.web (or for example as
C in luatex sources arithemtic.w)

@ And now let's complete our collection of numeric utility routines
by considering random number generation.
\MF\ generates pseudo-random numbers with the additive scheme recommended
in Section 3.6 of {\sl The Art of Computer Programming};
...
@ Finally, a normal deviate with mean zero and unit standard deviation
can readily be obtained with the ratio method (Algorithm 3.4.1R in
{\sl The Art of Computer Programming\/}).


A major aim of the original request was to bring the functionality of
the xetex engine into line with pdftex and luatex to help in
developing cross engine macro code. So there are arguments for using
the same code here unless the existing code is really broken and
luatex/metafont/pdftex should all change.

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

[XeTeX] atopwithdelims

[David Carlisle]

I've been looking to see if I can make \atopwithdelims and related
commands a little more consistent between the tex engines
(specifically in relation to the amsmath \genfrac usage)

Short version: would it be possible to make \atopwithdelims scale
\Udelimiter specified delimiters to a size determined by font
parameters 20 and 21?

Long version:

The attached test file xetexatop.tex shows (I think) that xetex obeys
fontdimens 20 and 21 to size the delimiters used with "withdelims" but
only if the supplied delimiters have \delcode settings and tfm metrics
specifying two slots. If the delimiters have \Udelcode settings just
specifying a single slot in the OTF file
then (I think) they are just set natural size.

(I set the delcode and Udelcode to strange values in the test so ()
typeset as {} just to check the values were being picked up)

You can compare the xetex result against pdftex or against

xetex \\let\\Ucharcat\\undefined\\input xetexatop

which makes it take the "classic tex" branch.

The OpenType Math table doesn't have entries corresponding to 20/21
but these values could be set from the tex side while loading the font
so it would still be useful if xetex supported this use (luatex has
two additional named parameters for the same issue
FractionDelimiterDisplayStyleSize and FractionDelimiterSize)

For the specific use in latex where the use (should) always use an argument form
\binom{a}{b} not {a \choose b}  it is I think possible to define
things using \left\right
and get a compatible definition see the attached
genfrac-vialeftright.tex but the definition is a bit of a mess using
\mathchoice and magic numbers (2.39, 1.35 etc) designed to make the
scaling for computer modern be exactly the same whether this
definition or the primitive \atopwithdelims is used.

For the primitive syntax used as an infix operator it is much harder
to make a working definition just from tex macro redefinitions. hence
the question if the underlying engine could help here...

David


xetexatop.tex
Description: TeX document


genfrac-vialeftright.tex
Description: TeX document


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Devanagari, xelatex, and \footnote

[David Carlisle]

If you base it on catcode changes it will never work in the argument
of another command, not just footnotes. There is no need to switch
catcodes.

leave ~ as active always and when you want it locally to be a normal ~ do

\edef~{\string~}

so the active ~ just expands to a non-active one.

David



On 15 August 2016 at 09:58, ShreeDevi Kumar  wrote:
> See
> http://cikitsa.blogspot.in/2010/09/xelatex-velthuis-encoding-and-palatal.html
> and the last comment .
>
> Here are modified tex and pdf files based on the above, tested with
> texlive2016. However, tilde does not get displayed in the english text.
>
>
>
> ShreeDevi
> 
> भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com
>
> On Mon, Aug 15, 2016 at 9:12 AM, ShreeDevi Kumar 
> wrote:
>>
>> You may need to use a footnotefont that supports devanagari.
>>
>> Or try using ucharclasses to automatically switch fonts based on unicode
>> blocks.
>>
>> ShreeDevi
>> 
>> भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com
>>
>> On Mon, Aug 15, 2016 at 6:16 AM, Hueckstedt, Robert A. (rah2k)
>>  wrote:
>>>
>>> I attach a simple tex file and the pdf it produces to illustrate the
>>> problem.
>>>
>>> We set the ~ character to catcode 12 so that it can function normally in
>>> devanagari text (typed in the tex file in roman). When j and ~n come
>>> together in a word, then they produce the conjunct consonant as seen in the
>>> pdf file, but when the same j and ~n come together in the same word in a
>>> footnote, the catcode for ~ seems to revert to 13 and it causes the
>>> interword space. How do I make that ~ function in the footnote as it does in
>>> the text?
>>>
>>> Thanks in advance for any help or for any suggestions for other fora to
>>> use to solve this problem.
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Subscriptions, Archive, and List information, etc.:
>>>   http://tug.org/mailman/listinfo/xetex
>>>
>>
>
>
>
>
> --
> Subscriptions, Archive, and List information, etc.:
>   http://tug.org/mailman/listinfo/xetex
>



--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] How to reduce terminal output from xelatex?

[David Carlisle]

>   xelatex "\def\istwocol{1} \input{./tmpmyfile.tex}  -max-print-line=2048 
> -c-style-errors -interaction=batchmode"

You are applying the options too late so they are ignored.

  xelatex  -max-print-line=2048 -c-style-errors -interaction=batchmode
"\def\istwocol{1} \input{./tmpmyfile.tex}"

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] [tex-live] Seemingly inexplicable shift in page origin between TL 2014 and TL 2016

[David Carlisle]

but if you have an up to date system it is in the documented code
(texdoc color.pdf)  I should probably update grfguide at some point.

David


On 15 July 2016 at 09:24, David Carlisle <d.p.carli...@gmail.com> wrote:
> On 15 July 2016 at 09:17, Philip Taylor <p.tay...@rhul.ac.uk> wrote:
>> P.S.  Option "nosetpagesize" does not seem to appear in the PDF output of 
>> "TeXdoc color", for the version of 2016/05/22.
>
> well no, it wouldnt.
>
> 2016-06-02  David Carlisle  <latex-b...@latex-project.org>
>
> * graphics.dtx, color.dtx, drivers.dtx: add pagesize special
> support to dvips
> option to match pdftex behaviour and add setpagesize and
> nosetpagesize options to color and graphics packages to enable or
> disable this feature for all drivers.
>
>> ** Phil.
>>
>>
>> --
>> Subscriptions, Archive, and List information, etc.:
>>   http://tug.org/mailman/listinfo/xetex


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] [tex-live] Seemingly inexplicable shift in page origin between TL 2014 and TL 2016

[David Carlisle]

On 15 July 2016 at 09:17, Philip Taylor <p.tay...@rhul.ac.uk> wrote:
> P.S.  Option "nosetpagesize" does not seem to appear in the PDF output of 
> "TeXdoc color", for the version of 2016/05/22.

well no, it wouldnt.

2016-06-02  David Carlisle  <latex-b...@latex-project.org>

* graphics.dtx, color.dtx, drivers.dtx: add pagesize special
support to dvips
option to match pdftex behaviour and add setpagesize and
nosetpagesize options to color and graphics packages to enable or
disable this feature for all drivers.

> ** Phil.
>
>
> --
> Subscriptions, Archive, and List information, etc.:
>   http://tug.org/mailman/listinfo/xetex


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] [tex-live] Seemingly inexplicable shift in page origin between TL 2014 and TL 2016

[David Carlisle]

Phil,

> And why does :
>
 >\input miniltx
  >   \input color

> /not/ change my page size ?

as stated earlier, if used under latex the specified page size is
known and set in the latex allocated registers
\paperheight/paperwidth, but if used under plain these are not set and
that section of xetex.def is skipped (really if you insist on using
plain you should be prepared to look at the fairly simple tests in
xetex.def) in the eplain/\beginpackages use enough of latex internals
are emulated to send xetex.def down the latex code path.

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] [tex-live] Seemingly inexplicable shift in page origin between TL 2014 and TL 2016

[David Carlisle]

On 15 July 2016 at 00:19, Ross Moore <ross.mo...@mq.edu.au> wrote:

> Hi David,
>
> On Jul 15, 2016, at 8:49 AM, David Carlisle <d.p.carli...@gmail.com>
> wrote:
>
>
> well that'll be the page size special change as mentioned earlier I assume.
>
>
> Hmm. In which version of  color.sty  was this introduced?
>

pdftex.def (and luatex.def) have _always_ set \pdfpage(height|width) with
no option not to do that,
conversely dvips.def did not by default set \ pagesize \special, which mean
that if you used color or graphics
then the pdf-specified page size either was or was not affected by latex
options such as
\documentclass[a4paper]{article}
depending if you used pdftex or tex+dvips (or luatex or xetex)
as of this year, after some discussion on the texlive list color and
graphics acquired  setpagesize and nosetpagesize options and then
dvips/pdftex/luatex/xetex/dvipdfmx.def either set the page size or not.



> Presumably later than:
>
> Package: color 2016/05/09 v1.1c Standard LaTeX Color (DPC)
>
> There is a potential conflict with  pdfx.sty  setting the  /MediaBox .
>
>
> what does \usepackage [nosetpagesize]{color} do?
>
>
> Is there a mechanism similar to  \hypersetup  that allows the options to
> be changed
> *after* the package has been loaded?
>

Not really, although the actual setting is in \AtBeginDocument{\AtBeginDVI
so as long
as you set the relevant registers to the right value it'll get set up (but
let me know if you need more hooks)


>
> Alternatively, can I detect whether the  pagesize  special has been done
> already?
> Then not repeat specifying  /MediaBox  when setting the other boxes:
>  Bleed/Crop/Trim
> which are required for PDF/X validation.
>
> If not loaded yet, I can do  \PassOptionsToPackage{nosetpagesize}{color} .
> But I’ll want to catch the case also if it is loaded.
>
>
> Cheers
>
> Ross
>


> David
>


>
> * Dr Ross Moore*
>
> *Mathematics Dept **|* Level 2, S2.638 AHH
> Macquarie University, NSW 2109, Australia
>
> *T:* +61 2 9850 *8955  |  F:* +61 2 9850 8114 <%2B61%202%209850%209695>
> *M:*+61 407 288 255 <%2B61%20409%20125%20670>*  |  *E:
> ross.mo...@mq.edu.au <rick.min...@mq.edu.au>
>
> http://www.maths.mq.edu.au <http://mq.edu.au/>
>
>
> <http://mq.edu.au/>
>
>
> CRICOS Provider Number 2J. Think before you print.
> Please consider the environment before printing this email.
> <http://mq.edu.au/>
>
> This message is intended for the addressee named and may
> contain confidential information. If you are not the intended
> recipient, please delete it and notify the sender. Views expressed
> in this message are those of the individual sender, and are not
> necessarily the views of Macquarie University. <http://mq.edu.au/>
>
>
>
>
> --
> Subscriptions, Archive, and List information, etc.:
>   http://tug.org/mailman/listinfo/xetex
>
>


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] [tex-live] Seemingly inexplicable shift in page origin between TL 2014 and TL 2016

[David Carlisle]

On 14 July 2016 at 23:40, Philip Taylor  wrote:
> It transpires that it is /not/ the PDF/X-1A \specials that are the root cause 
> of the problem.  It is the use of
>
> \input eplain
> \beginpackages
> \usepackage {color}
> \endpackages
>
> If I replace this with what I believe to be the functionally equivalent :
>
> \input miniltx
> \input color
>
> all works as in 2014 !
>
> ** Phil.
>


well that'll be the page size special change as mentioned earlier I assume.

what does \usepackage [nosetpagesize]{color} do?


>
> --
> Subscriptions, Archive, and List information, etc.:
>   http://tug.org/mailman/listinfo/xetex


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] [tex-live] Seemingly inexplicable shift in page origin between TL 2014 and TL 2016

[David Carlisle]

On 12 July 2016 at 16:17, Philip Taylor  wrote:



 > \advance \voffset by 1 true in % REQUIRED ONLY FOR TL 2016

If it's really a difference in texlive check that you specified the same
default driver page size (presumably A4)
in both installations, not A4 in one and letter in the other


or it may be a change in the driver files that now use a pagestyle special
by default but it wouldn't do that unless the latex \paperwidth register is
set (look at the end of xetex.def)

although neither of these would have the effect of exactly 1in shift.

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Grey fonts with XeTeX

[David Carlisle]

On 29 June 2016 at 13:35, Philip Taylor <p.tay...@rhul.ac.uk> wrote:
>
>
> David Carlisle wrote:
>
> see the color package documentation
>
> I tried this again today, David, and came up with a new problem which I do
> not understand.  Are you able to explain why the insertion of l.~12 :

possibly because sometime last century I spent about half an hour
stealing bits of the latex format code into miniltx.tex until the color
and graphics packages loaded without error into plain, and no one
has tried \pagecolor with plain xetex since then?

>
> \pagecolor {black}
>
> causes XeTeX to complain
>
> ! Output routine didn't use all of \box255.
>
> 
>
> \end
>
> l.50 \end
>
> ?
>
> Adding \nopagecolor before \end does not help, but removing \pagecolor
> {black} removes the error message.
>
> Minimal code to demonstrate follows, logo attached.

you don't need the logo, an empty \centerline is enough


More exactly the definition of

\def\set@page@color

in xetex.def looks wrong, but no time to debug now may look later but
anyway I'm not sure why it doesn't use the one from dvips.def which
seems to work ok with xetex/xdvipdfmx

\def\set@page@color{\special{%
background \current@color}}


David



> 
> % !TeX Program=XeTeX
>
>
> \input eplain
>
> \beginpackages
>
> \usepackage {color}
>
> \endpackages
>
>
> \pagecolor {black}
>
>
> \newbox \logobox
>
> \setbox \logobox = \hbox {\XeTeXpdffile westberry-logo.pdf }
>
> \centerline {\box \logobox}
>
>
> \end



-- 
http://dpcarlisle.blogspot.com/


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] xelatex, hyperref, and new TeXLive

[David Carlisle]

On 16 June 2016 at 00:25, Ross Moore  wrote:

> Hi Mike, David, Herb,
>
> I’m seeing the same behaviour, but for me the packages are as follows:
>
> (/usr/local/texlive/2016/texmf-dist/tex/latex/latexconfig/hyperref.cfg)
>
> /usr/local/texlive/2016/texmf-dist/tex/latex/hyperref/hyperref.sty:4322:
> Packag
> e hyperref Error: Wrong DVI mode driver option `dvipdfmx',
> (hyperref)because XeTeX is running.
>
>
>
>


Yes that's with the file with an explicit [dvipdfmx] option, as far as I
can see
hyperref has always (or at least for some years) given that error on xetex
it insists on xetex option (which would be the default if the correct cfg
is loaded)


>
> Here is the relevant coding from  hyperref.sty  with annotations added by
> me.
>
> \newif\ifHy@DviMode
>
> This defines  \ifHy@DviMode and switches, leaves it as  \iffalse
>
> \let\Hy@DviErrMsg\ltx@empty
> \ifpdf
>   \def\Hy@DviErrMsg{pdfTeX or LuaTeX is running in PDF mode}%
> \else
>   \ifxetex
>
> This is already  \iftrue
>
> \def\Hy@DviErrMsg{XeTeX is running}%
>
> … but surely we should be setting  \Hy@DviModetrue  here !!!
>



Hmm perhaps...
 although the end result is that it just turns the error into a warning and
ignores the specified option so you'd get

Package hyperref Warning: Wrong driver `hdvipdfm.def';
(hyperref)XeTeX is running.
(hyperref)Forcing driver `hxetex.def' for XeTeX.

probably giving an error and making the user fix the document is as clear,
and certainly what it's always done.


>   \else
> \ifvtex
>   \ifvtexdvi
> \Hy@DviModetrue
>   \else
> \def\Hy@DviErrMsg{VTeX is running, but not in DVI mode}%
>   \fi
> \else
>   \Hy@DviModetrue
> \fi
>   \fi
> \fi
>
>
> The result is that when you subsequently request   [dvipdfmx]  or  any
> other driver,
> hyperref thinks that we are in non-dvi mode, so  *incorrectly* throws the
> error.
>
> So it’s surely an omission in  hyperref.sty .
>
> But you don’t actually need to specify a driver option,
> and everything works OK anyway.
>

It only works with no option if you are not using a hyperref.cfg that
specifies incompatible options:-)


>
>
>   Mike Maxwell
>
>
>
> Hope this helps,
>
> Ross
>
>
>
David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] xelatex, hyperref, and new TeXLive

[David Carlisle]

On 16 June 2016 at 00:08, Herb Schulz  wrote:
> Howdy,
>
> Again, sorry for the top post.
>
> Does
>
> kpsewhich hyperref.cfg
>
> really return a file in the .../doc/... branch?
>

Oh of course! well spotted.

So that is the error.

The file is correctly installed in the doc branch but the OP's
$TEXINPUTS is incorrectly including the
/groups/tools/texlive/2016/texmf-dist/doc tree and worse searching it
before the /groups/tools/texlive/2016/texmf-dist/tex tree.


David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] xelatex, hyperref, and new TeXLive

[David Carlisle]

On 15 June 2016 at 23:46, maxwell <maxw...@umiacs.umd.edu> wrote:
> With the help of David Carlisle and Herbert Schulz, I've found part of the
> problem.  For some reason, in the (our?) 2016 version, kpsewhich points to
> this hyperref.cfg file:
>...texlive/2016/texmf-dist/doc/latex/listings-ext/hyperref.cfg

looking at the listings-ext distribution on ctan that is intended to
be a local configuration for producing the package documentation, it
is not intended to be in the general tex input path (and is not in the
input path of my texlive 2015 or 2016 installations which have only
listings-ext.sty in the latex/listings-ext directory.


> This .cfg file contains a \hypersetup{...} command that specifies 'ps2pdf'.
> Changing that to 'xetex' fixes the problem, at least for xelatex (I'm not
> sure what would happen with other flavors of latex).  (Update: removing the
> line entirely, so it specifies neither xetex nor ps2pdf, works too, and
> presumably won't cause trouble for other latices.)

I think you should remove the file.

>
> But:
> 1) Why does kpsewhich find that file, instead of this one:
>...texlive/2016/texmf-dist/tex/latex/latexconfig/hyperref.cfg
>which does not have any \hypersetup{} command, and which would
>presumably not cause the same problem?

it finds it because it is there:-) if you have two files of the same
name in the same input tree then kpsearch will find one of them but
it's not a very stable setup.

> 2) Why did this change from 2015 to 2016?  We did a pretty vanilla
>install, I think the only non-default choice we made was to use
>'letter' instead of 'a4'.

I'm not sure. How come the listings-ext hyperref config got installed
on your system
It isn't in the texlive install I have (which installs everything,
including listings-ext)
it installs into the doc tree and is just used for the listings-ext package doc

/usr/local/texlive/2016/texmf-dist/doc/latex/listings-ext/hyperref.cfg



> 3) Is this a bug? (meaning should I report it?)

If texlive installed that cfg file then that would seem to be a bug in
tlmgr, but I can't reproduce it here.

>
>Mike Maxwell


David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] xelatex, hyperref, and new TeXLive

[David Carlisle]

I get the same error with texlive 2015 so I don't think there is any
recent change (since support for hyperref moved to github) that has
changed this, with texlive 2015:

(/usr/local/texlive/2015/texmf-dist/tex/latex/hyperref/pd1enc.def)
(/usr/local/texlive/2015/texmf-dist/tex/latex/latexconfig/hyperref.cfg)

! Package hyperref Error: Wrong DVI mode driver option `dvipdfmx',
(hyperref)because XeTeX is running.

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] xelatex, hyperref, and new TeXLive

[David Carlisle]

On 15 June 2016 at 21:56, maxwell  wrote:
> We've downloaded and installed the new 2016 TeXLive distribution.  I was all
> set to be happy...but.  There seems to be an incompatibility between the new
> xelatex and the new hyperref package.  No matter what I do, I get the error
> msg:
> 
> ! Package hyperref Error: Wrong DVI mode driver option `ps2pdf',
> (hyperref)because XeTeX is running.
> 
>
> We did not of course get this error with the TeXLive 2015 distribution.
>
> Here's the minimal file to trigger this error (ok, I could take out the word
> "test", but it's pretty minimal :-)):
> 
> \documentclass{article}
> \usepackage[dvipdfmx]{hyperref}
> \begin{document}
> test
> \end{document}
> 
>
> I've tried telling hyperref
>[driverfallback=dvipdfm]
> instead, and I've tried telling xelatex
> --output-driver=xdvipdfmx
> and a bunch of other things, with no effect.
>
> What do I yet lack?
>
>Mike Maxwell
>University of Maryland



odd: I didn't think anything had changed here, I'll test with tl2015
later, meanwhile your test file works for me with with texlive 2016
xelatex and

\usepackage{hyperref}

or equivalently with


\usepackage[xetex]{hyperref}


David

>
>
>
> --
> Subscriptions, Archive, and List information, etc.:
>  http://tug.org/mailman/listinfo/xetex



-- 
http://dpcarlisle.blogspot.com/


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Overfull boxes return status of 0 in XeTeX

[David Carlisle]

On 13 March 2016 at 10:14, Philip Taylor <p.tay...@rhul.ac.uk> wrote:
>
>
> David Carlisle wrote:
>
>> Philip Taylor <p.tay...@rhul.ac.uk> wrote:
>
>>> Also please consider the following text from Wikipædia :
>> (Well from Wikipedia actually)
>>
>> Yes the relevant part is
>>
>>> On many systems, the higher the value, the more severe the
>>> cause of the error.
>>
>> so the status code should be 0 unless an error is reported.
>
> I think we are arguing about semantics when we should be discussing
> functionality; an overfull box /is/ an error, whether or not it results
> in a TeX compilation being interrupted to solicit user input.

No, it is up to the person writing the code to classify things as
errors or warnings
overfull boxes are the latter.  An option to make then errors would
not be unreasonable.

>
>> I don't see why that would be a problem with an editor front end like 
>> texworks
>> most of them run tex in scrollmode so they don't stop on errors as far
>> as I can see.
>>
>> I think that you are really approaching the problem from the wrong direction.
>> editors hiding the console output is a problem but the fix should be that the
>> editors don't do that. As we see on tex.sx all the time people don't even 
>> notice
>> clear errors like undefined commands as they are using front ends that just
>> force tex to get to the end and show the pdf that results.
>
> But we are not discussing general front ends; we are discussing an
> intelligent front end such as TeXworks that does /not/ conceal the log
> file unless (a) the user has configured it so to do, or (b) it believes
> that no error has occurred.  All I am asking is that XeTeX be given the
> option to inform TeXworks that an error has occurred when an overfull
> box has been generated.

It should only inform any system that an error has occurred if
something classified
as an error has actually occurred.

>
> ** Phil.



--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Overfull boxes return status of 0 in XeTeX

[David Carlisle]

On 13 March 2016 at 09:43, Philip Taylor  wrote:
> Also please consider the following text from Wikipædia :
(Well from Wikipedia actually)

Yes the relevant part is

> On many systems, the higher the value, the more severe the
> cause of the error.

so the status code should be 0 unless an error is reported.

I don't see why that would be a problem with an editor front end like texworks
most of them run tex in scrollmode so they don't stop on errors as far
as I can see.

I think that you are really approaching the problem from the wrong direction.
editors hiding the console output is a problem but the fix should be that the
editors don't do that. As we see on tex.sx all the time people don't even notice
clear errors like undefined commands as they are using front ends that just
force tex to get to the end and show the pdf that results.

David



--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Handling of combining and variant selector characters in math

[David Carlisle]

On 10 February 2016 at 21:36, David J. Perry <hospes.pri...@verizon.net> wrote:
> I know nothing whatsoever about math, so perhaps I shouldn't even join this
> discussion, but I am curious.  I do have considerable experience in font
> development and supporting things like the use of combining marks and
> variation selectors outside of math contexts.
>
> I looked in a font editor at the font (XITS) that was used to produce the
> sample PDF.  As far as I can see, it has no support for combining marks or
> variation selectors of the sort that I would expect, based on my non-math
> experience.  (For instance, fonts that support combining marks usually have
> a Mark to Base lookup.)   Is there an expectation that in TeX the math
> typesetting can properly position combining characters and handle variation
> selectors without support from the font?
>
> David
>

Note I was using Khaled's xits-math variant not the original stix
version, xits-math has many improvements to the opentype internals.
The VS1 combinations in xits-math work for example in firefox.

David

>
> On 2/9/2016 8:54 AM, David Carlisle wrote:
>
> Sending this separately to xetex  luatex lists...
>
> The current Unicode math alphabets in U+1D400 ...include a "script"
> alphabet this was intended to cover both "caligraphic/chancery" and
> "script/roundhand" as font variants, in the hope that no document
> would need both.
>
> Unicode is considering adding separate markup for the two forms, see
> Murray's blog here
>
> http://blogs.msdn.com/b/murrays/archive/2016/02/05/unicode-math-calligraphic-alphabet.aspx
>
> As explained in the article at least two possible suggestions are
> being considered:  adding the new alphabet in a new code block range,
> or defining  "variant selector" characters that would force one or
> other interpretation.
>
> Unicode combining characters following the base have always been a bit
> tricky in TeX so I wondered whether the engine (or font) developers
> (as opposed to macro level hacker like myself) have a view on what is
> a reasonable input form here.
>
> You could either reply here or as a comment on the above Blog.
>
> I attach a (latex) text file that produces more or less the same
> output in xetex and luatex showing that by default neither a simple
> combining character like e-acute nor VS1 work but the combining acute
> at least can perhaps be made to work but VS1 seems tricky as the base
> is a \mathop atom so it's not really amenable to being combined with
> the following variant selector character.
>
> Since current combining character use seems tricky I'm worried about
> the suggestion to use that method for selecting the entire script
> alphabet.
>
> (The combining acute could be normalised away by running filter to NFC
> form but don't do that as it's just standing in for a possible new
> character to switch script forms:-)
>
> Tests 0 and A show that both combining forms work fine in text, but
> math is the issue here
>
> David
>
>
>
>
> --
> Subscriptions, Archive, and List information, etc.:
>   http://tug.org/mailman/listinfo/xetex
>
>
>
>
>
> --
> Subscriptions, Archive, and List information, etc.:
>   http://tug.org/mailman/listinfo/xetex
>



-- 
http://dpcarlisle.blogspot.com/


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Copyright page with \pagestyle{empty} clears all styles

[David Carlisle]

That seems to be a general latex question rather than anything xetex
related, but to affect just one page use \thispagestyle not \pagestyle

David

On 6 February 2016 at 11:26, Shakthi Kannan  wrote:
> Hi,
>
> I have released the Tamil-English XeTeX customized book template at:
>
>   https://github.com/shakthimaan/xetex-book-template
>
> I have been able to add a copyright page to \maketitle, but, the
> \pagestyle{empty} use in styles/config.sty clears all the fancyhdr
> settings in the rest of the pages.
>
>   
> https://github.com/shakthimaan/xetex-book-template/blob/master/styles/config.sty#L46
>
> For example, on the top of page 8, I don't see the page number and the
> book name. How can this be fixed?
>
> A PDF example generated is available at:
>
>   http://shakthimaan.com/dump/book.pdf
>
> Any other feedback or suggestions to improve the sources is welcome.
>
> Thanks!
>
> SK
>
> --
> Shakthi Kannan
> http://www.shakthimaan.com
>
>
> --
> Subscriptions, Archive, and List information, etc.:
>   http://tug.org/mailman/listinfo/xetex



-- 
http://dpcarlisle.blogspot.com/


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] not enough \XeTeXcharclass registers

[David Carlisle]

Thanks for the test sources,

It all seems to work for me (texlive 2015/cygwin 64 build), but..

I do wonder if this change is going in the right direction.

The main problem with the char classes is not the overall number, in fact
since the important thing as far as specifying code is the boundary between
different classes rather than the classes themselves, there are now around
300 million such boundaries that could be specified, which seems more than
enough!

The main problem is that each character can only be in one class which
means that it is very hard to use these for any generic code. If you have
already classified characters by (say) line breaking properties and then
another package wants to classify by unicode block, or by default writing
direction, then the only way to handle that is to enumerate all the
intersecting properties and assign a a unique character class to each
intersection, this leads to a combinatorial explosion in the number of
boundary tokens that need to be specified. Where you may have had a single
specification for the boundary between LTR and RTL if you also want to
classify each unicode block you need  separate classes for LTR and RTL
characters in each block and then need to specify the same boundary tokens
for all the possible changes of LTR in one block followed by RTL in another.

That limitation of course has always been there, but increasing the number
of classes available highlights it more strongly.

Would it be impossibly difficult to extend the concept so that a character
takes a list of character classes so that you can classify characters in
more than one way without needing impossibly many character classes to do
that?

Sorry for sounding ungrateful for the extension:-)

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] not enough \XeTeXcharclass registers

[David Carlisle]

On 1 February 2016 at 10:53, Jonathan Kew <jfkth...@gmail.com> wrote:

> On 1/2/16 10:25, David Carlisle wrote:
>
>> y.
>>
>
> You're right, of course; this is a limitation of the concept as currently
> implemented.
>
> In practice, I suppose I don't expect there to be all that many "generic
> purposes" for which intercharclass is really a useful tool. For example,
> it's hard to see how it could work well for bidi issues, because of the
> problem of resolving neutral characters -- especially run-initial neutrals.
>

Yes I hesitated about using bidi as the example there, but couldn't think
of many other generally applicable things:-)

>
>
>> Would it be impossibly difficult to extend the concept so that a
>> character takes a list of character classes so that you can classify
>> characters in more than one way without needing impossibly many
>> character classes to do that?
>>
>
> There would be two aspects to this: first, extending the character class
> storage so as to allow a list rather than a single number. Currently, it's
> stashed in the upper part of the word where sfcode already lives, making
> the implementation very simple and cheap.
>
> And second, checking for the existence of a token list for the current
> boundary would become significantly more expensive.


Yes I suspected as much, perhaps it's a non starter. If the extended number
as in the current test branch is "still cheap" then perhaps that's the way
to go although character classes always seem like they are almost a
solution to a problem but never quite powerful enough.


> Currently, we just combine the two classes at the boundary to get a single
> 32-bit number, and do a simple lookup (in a sparse array) to see if there's
> anything defined. With class lists, we'd need to do this for each of the
> classes in the two lists -- i.e. m * n sparse-array lookups. Or perhaps go
> at it from the other direction: iterate over a list of defined transitions,
> and check whether each of them applies.
>

make sense.


>
> Oh, and if there are multiple matches at a given boundary, what happens?
> Using an imaginary extension to support lists:
>
>   \XeTeXintercharclasses `A = { 1, 2 }
>   \XeTeXintercharclasses `B = { 3, 4 }
>
>   \XeTeXinterchartoks 1 3 = { foo }
>   \XeTeXinterchartoks 1 4 = { bar }
>   \XeTeXinterchartoks 2 3 = { xyzzy }
>   \XeTeXinterchartoks 2 4 = { plugh }
>
> What happens at the boundary in "AB"? Should it depend on the numerical
> values of the classes, or the order in which the transitions were
> specified, or what?
>
> (I'm not saying the idea is a bad one; I can imagine it might be quite
> useful. But I can also imagine it getting a bit hairy..)


Yes it certainly wasn't a fully worked proposal, but I thought it worth
commenting while you were looking at that area of the code.



>
>
> JK
>
>
David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] misplaced accents in printout only?

[David Carlisle]

On 20 January 2016 at 23:42, Zdenek Wagner  wrote:

> As far as I know nowadays with HarfBuzz it does nothing, HarfBuzz handles
> normalization in its own way.
>
> Zdeněk Wagner
> http://ttsm.icpf.cas.cz/team/wagner.shtml
> http://icebearsoft.euweb.cz
>
> 2016-01-21 0:35 GMT+01:00 maxwell :
>
>> On 2016-01-20 17:59, Dominik Wujastyk wrote:
>>
>>> ...
>>> *Input normalization*
>>>
>>> I left {CMU Serif Italic} as the document font, and added
>>> \XeTeXinputnormalization=1 to the preamble
>>>
>>> This also produced PDF that printed *correctly* in all cases.
>>>
>>
>> Hmmm, is there documentation for this \XeTeXinputnormalization?  I can
>> find some references to it in a web search, but no explanation of exactly
>> what it does.  What does "normalization" mean (NFC, NFD,...)?  Under what
>> circumstances (besides the one Dominik saw) should this setting be used?  I
>> don't want to go through printouts of 300 page documents looking for
>> misplaced diacritics...
>>
>>Mike Maxwell
>>
>>
It's in the xetex manual (texdoc xetex) 0 is off, 1 is NFC 2 is NFD

It affects tokenization so does have an effect whatever font library is in
use see



\def\zz#1#2{[#1]}

\zz á12

\zz á12


\XeTeXinputnormalization=1

\def\zz#1#2{[#1]}

\zz á12

\zz á12

\bye


where the first line produces [a] as it separates the ascii a from the
following combing character but the third line which is also input using a
combing character produces [] the same as lines 2 and 4 which use the
pre-composed U+00E1 (which is nothing here as I'm using cmr)

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] [6933] luatex output encoding

[David Carlisle]

In experiments with using the inputencoding for latex

https://github.com/wspr/xetex-inputenc/blob/master/xetex-inputenc.dtx

deal with that problem by writing

\XeTeXinputencoding "utf8"

at the top of the auxiliary file so that it is read back in utf8 even if
the main document is
using a legacy encoding.

David


On 12 January 2016 at 19:31, Gavin Smith  wrote:

> It has been suggested to me to let you know about a problem we had
> with Texinfo with XeTeX about character encoding. XeTeX reads and
> writes to files by default using a UTF-8 encoding. It's possible to
> override the input encoding with \XeTeXdefaultencoding and
> \XeTeXinputencoding, but as far as we can tell there's no
> corresponding command for the output encoding. We managed to fix this
> problem for LuaTeX for both input and output, but XeTeX appears to
> have a setting for input only.
>
> This is a much smaller problem than it would be it would be if the
> input encoding couldn't be set, but it is a problem when reading and
> writing to auxiliary files to handle indices, cross-references and
> tables of contents. For example, a chapter title may have a non-ASCII
> character in it, e.g. "ü". When we set \XeTeXinputencoding "bytes",
> this is read in as two tokens with hex values c3 bc. That's intended:
> texinfo.tex needs the individual byte values. But then when it's
> written out to build up the table of contents, this gets written out
> as two UTF-8 characters (ü, bytes c3 83 c2 bc) which isn't what is
> needed: we want to write out the two bytes, that is a single UTF-8
> character. As it is, this means when the table of contents is typeset,
> the character "ü" comes out as "ü".
>
> If we're right in thinking there's no way to set the output encoding
> (for \write), it might be a good idea to add one.
>
> Best wishes,
> Gavin
>
>
>
> --
> Subscriptions, Archive, and List information, etc.:
>   http://tug.org/mailman/listinfo/xetex
>



-- 
http://dpcarlisle.blogspot.com/


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] not enough \XeTeXcharclass registers

[David Carlisle]

On 13 December 2015 at 07:04, Werner LEMBERG  wrote:
>
> [XeTeX 3.14159265-2.6-0.2 (TeX Live 2015)]
>
>
> Folks,
>
>
> I'm updating the `ucharclasses.sty' to completely cover Unicode.  This
> style file maps Unicode character blocks to character classes, and
> I've hit the 256 entry limit of \XeTeXcharclass...
>
> Any chance to extend it to 16 bits?
>
>
> Werner
>

Don't you only need to allocate a class to a block for which a
transition is specified, so
256 should only be a problem if you need to specify that many transitions?

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Problem with bidi + longtable + caption packages

[David Carlisle]

On 21 October 2015 at 02:34, Herbert Schulz  wrote:
>> On Oct 20, 2015, at 7:55 PM, maxwell  wrote:
>>

> The longtable environment is like a tabular environment, not the same as a 
> floating environment like the teble environment. It's floats that take the 
> caption. Try
>

No! longtable has a caption command and can not be inside a float box
(otherwise it will not break over a page).

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Problem with bidi + longtable + caption packages

[David Carlisle]

On 21 October 2015 at 01:55, maxwell  wrote:
> (Apologies if this isn't the right mailing list--the documentation I have
> says I should send this to persian-...@tug.org, but I don't see that mailing
> list on the list of TUG mailing lists, nor am I subscribed to it... and if
> it does exist, I don't know how to subscribe to it.)

I believe that's a personal mailbox of the bidi maintainer not a list.

bidi and longtable is tricky in xetex as the tex-xet bidi model it uses
doesn't really support right to left \halign as \beginR has to be in
horizontal mode.
Vafa patches longtable as well as he can but the underlying support
isn't really there,
this is one place where the directional model in luatex (from omega)
is clearly superior.

I'll try to look later but it's a delicate area...

David

>
> I've encountered a problem in the interaction between the longtable, bidi,
> and caption packages.  I'll attach the minimal example at the bottom of this
> msg, but I have a question.  We're running the 2014 TeXLive distro; I never
> got around to downloading the 2015 distro.  We'll probably do that, but
> we're having some connectivity issues (serves me right for not having
> downloaded it months ago), and there are some hoops to jump through, so it
> won't be instant gratification.
>
> The issue we've found is that we want table captions to be ragged right,
> rather than centered.  Ordinarily, the caption package handles this fine.
> But when I have a longtable, and I use the bidi package as well, I can only
> get centered captions for long tables (ordinary floating table captions work
> fine).
>
> I've tried numerous variants of the code below, e.g. telling the caption pkg
> to do ragged right only in one or the other of the places, loading the bidi
> bpackage before or after the \captionsetup command (it cannot be loaded
> before the caption pkg is loaded), etc.
>
> Did anyone else notice this last year, and if so, has it been fixed in the
> 2015 distro?
>
> Mike Maxwell
> University of Maryland
>
> ---Minimal example follows-
> \documentclass{report}
>
> \usepackage{longtable}
> %Tell the caption pkg to do ragged right at load time:
> \usepackage[singlelinecheck=off
>,justification=raggedright
>]{caption}
> \usepackage{bidi}
> %Tell the caption pkg again:
> \captionsetup{singlelinecheck=off
>  ,justification=raggedright
>  }
>
> \begin{document}
>
> \begin{longtable}{}
> \caption{A long table}%Should be ragged right, but is centered instead
> \tabularnewline
>  {} & {} & {} & {}
> \tabularnewline
> \end{longtable}
>
> \end{document}
>
>
> --
> Subscriptions, Archive, and List information, etc.:
>  http://tug.org/mailman/listinfo/xetex



-- 
http://dpcarlisle.blogspot.com/


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Problem with bidi + longtable + caption packages

[David Carlisle]

The caption wasn't really centred (hard to tell as the text was short)
it is a justified paragraph (set in a centred box) the problem is that
bidi doesn't "patch" the caption code to add \beginR it redefines the
standard one, so over-writing the caption package changes.


The following simply preserves the longtable/caption package version
of \caption when bidi is loaded  (so it doesn't get patched to right
to left usage, so you may need to add \beginR \endR explicitly in a
right to left caption, patching automatically might be tricky as you
have to cope with caption, ltcaption, longtable and bidi being loaded
or not, in different package loading orders.

\documentclass{report}

\usepackage{longtable}
%Tell the caption pkg to do ragged right at load time:
\usepackage[singlelinecheck=off
   ,justification=raggedright
   ]{caption}
\usepackage{ltcaption}
\LTcapwidth=\linewidth
\makeatletter
\let\temp\LT@makecaption
\usepackage{bidi}
\let\LT@makecaption\temp
%Tell the caption pkg again:
\captionsetup{singlelinecheck=off
 ,justification=raggedright
 }

\begin{document}

\begin{table}
\caption{A normal table
     
     
     
     
}
\end{table}

\begin{longtable}{}
\caption{A long table
     
     
     
     
}%Should be ragged right, but is centered instead
\tabularnewline
 {} & {} & {} & {}
\tabularnewline
\end{longtable}

\end{document}


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Discretionary hyphens don't work in paragraphed footnotes

[David Carlisle]

On 8 October 2015 at 09:25, Philip Taylor  wrote:
>
>
> Zdenek Wagner wrote:
>
>> I have no time to look deep into it but does anybody know what is the
>> definition of \- if these packages is used? Isn't is possible that it is
>> a fragile macro? I remember one thing from Phil Taylor's tutorial held
>> on EuroTeX'92 in Prague. He said that the most frequent error is wrong
>> timing of expansion. Is it possible that \- is redefined in such a way
>> that it disappears somewhere during expansion of \footnote before t is
>> typeset?
>
> More than possible :-(  Why /does/ LaTeX have to meddle so ?!

To make the format suitable for typesetting document, probably.

>
> \-::macro:->\x@protect \-\protect \-
>
> Try :
>
> % !TeX Program=XeLaTeX
> \documentclass[a4paper]{article}
> \usepackage{fontspec}
> \usepackage[para]{footmisc}
> \begin{document}
> \def \-{\discretionary {}{}{}}
> \footnote{XX just a few normal words to fill up the line
>   up to my x x x hy\-phe\-n\nobreak a\-te\-me }
> \end{document}
>
> ** Phil.
>

You'll see that example is another instance of the bug, as that is you
get a hyphen
after hy-  which you should not as you have defined \- to have three
empty hlists,
you get the same if you remove the definition.

It seems to be there just that the \nobreak is making the first part
not be th elast word in the paragraph.

so your example is makes exactly the same output as

% !TeX Program=XeLaTeX
\documentclass[a4paper]{article}
\usepackage{fontspec}
\usepackage[para]{footmisc}
\begin{document}
%\def \-{\discretionary {}{}{}}
\footnote{XX just a few normal words to fill up the line
  up to my x x x hyphen\nobreak ateme }
\end{document}

actually  the trailing space is enough to make it not the last word,
as you noted originally.

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Discretionary hyphens don't work in paragraphed footnotes

[David Carlisle]

On 8 October 2015 at 09:50, Philip Taylor  wrote:
> Hmmm, completely stymied :  not even this works ...

It wouldn't be expected to really, this was asked on stackexchange initially
and it looks almost certainly due to the interface between tex's hyphenation and
the font libraries, that is, an issue in the engine not due to macros,
that's why I suggested it was posted here.

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Discretionary hyphens don't work in paragraphed footnotes

[David Carlisle]

> Yes, it's very possible this is an engine bug. What would be most useful 
> here, I think, would be a reduced (non-LaTeX!) > testcase demonstrating the 
> underlying incorrect behavior at the (Xe)TeX primitive level. Any takers?

I'll see what I can do

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Discretionary hyphens don't work in paragraphed footnotes

[David Carlisle]

Here is a plain tex example, not quite as minimal as I'd like but out
of time for now.

This hyphenates with pdftex but not xetex

\nopagenumbers

\catcode`\@=11

\ifx\Umathchar\@undefined
\font\x=cmr12
\else
\font\x="[lmroman12-regular]:mapping=tex-text"
\fi

\x



\newbox\FN@tempboxa
\newbox\FN@tempboxb
\newbox\FN@tempboxc

  \long\def\makefootnoteparagraph{\unvbox0 \makehboxofhboxes
\setbox\FN@tempboxa=\hbox{\unhbox\FN@tempboxa \removehboxes}
\noindent
\unhbox\FN@tempboxa\par
  }
  \def\makehboxofhboxes{\setbox\FN@tempboxa=\hbox{}%
\loop
  \setbox\FN@tempboxb=\lastbox
  \ifhbox\FN@tempboxb
  \setbox\FN@tempboxa=\hbox{\box\FN@tempboxb\unhbox\FN@tempboxa}%
\repeat
  }
  \def\removehboxes{\setbox\FN@tempboxa=\lastbox
\ifhbox
  \FN@tempboxa{\removehboxes}%
  \unhbox\FN@tempboxa
\fi
  }


\setbox0\vbox{XX just a few normal words to fill up the line
  up to my x x x zz\-z}

\global\setbox\FN@tempboxc\vbox{\hbox{XXX}\makefootnoteparagraph}%
\tracingonline1\showbox\FN@tempboxc



\unvbox\FN@tempboxc

\bye


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Discretionary hyphens don't work in paragraphed footnotes

[David Carlisle]

On 8 October 2015 at 17:53, Dominik Wujastyk  wrote:
> The original questioner said he was doing a critical edition.  If he's using
> EDMAC or LEDMAC then perhaps I can note that EDMAC does some work to avoid
> setting footnote text in "restricted horizontal mode".  Restricted
> horizontal mode does a number of annoying things to text, including (I
> think) things affecting hyphenation; Plain TeX goes to some lengths to set
> footnotes in unrestricted mode.  But LaTeX doesn't bother with these
> niceties.  However, EDMAC does.
>
> Best,
> Dominik
>
> The attached fnfix.sty file fixes LaTeX footnotes so that they're set in
> unrestricted horizontal mode.
>
>

??? both standard latex footnotes and the paragraphed footnotes used
from footmisc in the original question use unrestricted horizontal
mode, if you set them in an hbox (restricted hmode) they wouldn't work
at all if longer than a line.

The code posted allows catcode changes

%   LaTeX style to redefine the \footnote command such that it allows the
%   altering of catcodes.
%   From: Joerg-Martin Schwarz <@MVS.GSI.DE:UPH508@DDOHRZ11.BITNET>
%   August 1992


but that's unrelated (and by design not included in standard latex footnotes)

David


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

Re: [XeTeX] Does there exist the equivalent of testfont.tex/fonttable.tex for XeTeX?

[David Carlisle]

On 12 June 2015 at 16:05, roger ro...@herz-fischler.ca wrote:
 As François Charette states on page 4 of his ««ArabXeTeX Manual»» there is
 a  great advantage to being able to input a language via code rather than
 typing  the glyph. This is particularly true if the language is not one
 that you master. As a personal example I cite Cree $\^{1}$. It is far
 easier, and much faster, for a non-speaker to type in \mi  than trying to
 search through the syllabic table for the sign for mi and then
 figuring out which key to touch (provided that you can input from the
 keyboard, which is not always the case)..

 This also has a great advantage in the teaching or linquistic description
 of a language, especially since the same coding can be used to print a
 transliteration; see e.g., Daniels and Bright, ««The World's Writing
 Systems»»,  Oxford University Press, 1996 for the usefulness of such a
 capability.

 This brings up the question of UNICODE.In order to write:
 \def\mi{14a5}
 one has to search through the UNICODE table to find the  #776;mi #776;
 syllabic. The
 tables tend to be all inclusive and thus very large and difficult to search
 through.

 Then there is the questsion of particular fonts. Does the font contain ALL
 syllabics? just Cree syllabics? just Moose Cree?$\^{1}$ Does the font even
 follow  UNICODE (not always evident)?

 In view of the above a VERY useful tool would be the equivalent of
 [testfont.tex]/[fonttable.tex], which, for a given UNICODE-based font,
 would print the glyph and the corresponding UNICODE number.

 My attempts at using the latter (in PureTeX) with Cree resulted in a Cork
 type table. Is there such an tool or modification of [fontable.tex]?

 Roger Herz-Fischler

 1. See http://www.languagegeek.com/font/fontdownload.html .



I showed one way to get tex commands based on the Unicode names of all
the characters in the cree  range here

http://tex.stackexchange.com/questions/244131/how-to-use-truetype-fonts-in-latex-no-map-encodinggiven-but-they-follow-unic/244136#244136

It probably doesn't make sense to print off an entire font (normally)
there are rather a lot of characters:-) but it would be easy to have a
macro that took a range and a font and made a Unicode-style table in
the style of

http://unicode.org/charts/PDF/U1400.pdf

David



--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex

[XeTeX] (not) understanding XeTeXinterchartoks

[David Carlisle]

The following plain xetex document loops forever on \show\tmpb
the \show don't cause the looping, if they are replaced by
\def\zzzb{} xetex just hangs in a tight loop.

The fact that it loops isn't necessarily a bug.
 \def\zzz{\zzz}\zzz
does the same,  but are there any words that could be added to
the manual so that I could have predicted this?

I'm not sure why 255 is being triggered at all as
the X is being inserted into the middle of an existing hlist.

The manual says 255 represents

 a boundary between a `run' of characters and something else

So I guess I am asking what 'run' means in this context:-)



David



\XeTeXinterchartokenstate = 1

\newXeTeXintercharclass \Xclass
\XeTeXcharclass `\X \Xclass

\XeTeXinterchartoks 0 \Xclass = {\zza}
\XeTeXinterchartoks 255 \Xclass = {\zzb}


\def\zza{\futurelet\tmpa\zzza}
\def\zzza{\show\tmpa}

\def\zzb{\futurelet\tmpb\zzzb}
\def\zzzb{\show\tmpb}



xxxXxxx


\bye


--
Subscriptions, Archive, and List information, etc.:
  http://tug.org/mailman/listinfo/xetex