Re: [XeTeX] Guaranteed Unicode replacement glyph in every TeX installation?

[Doug McKenna]

George - 

Thanks! 

- Doug McK. 


From: "George N. White III"  
To: "xetex"  
Sent: Sunday, August 22, 2021 8:14:58 AM 
Subject: Re: [XeTeX] Guaranteed Unicode replacement glyph in every TeX 
installation? 

On Fri, 20 Aug 2021 at 22:46, Doug McKenna < [ mailto:d...@mathemaesthetics.com 
| d...@mathemaesthetics.com ] > wrote: 



Using XeTeX, I want to typeset a LaTeX document into a PDF file. The LaTeX 
source code in UTF-8 



BQ_BEGIN

expressly includes the Unicode Replacement character (� = U+FFFD) (a black 
diamond with a 

BQ_END

BQ_BEGIN

question mark in it). 

I want to typeset it in this single document using a monospaced font in one 
place, and in another 

BQ_END

BQ_BEGIN

place in a variable-width font. 

I understand that XeTeX can take advantage of one's system's installed fonts, 
but my LaTeX file 

BQ_END

BQ_BEGIN

is being generated by another program that doesn't know what those fonts are or 
what glyphs 

BQ_END

BQ_BEGIN

they support. I simply want to guarantee that the fonts used are always 
available when processing 

BQ_END

BQ_BEGIN

that LaTeX file. 

BQ_END

BQ_BEGIN


I also understand that it's possible to synthesize the glyph graphically 
without using a font, but I'd 

BQ_END

BQ_BEGIN

rather not go that route. 

So ... What fixed-width and variable-width OpenType (or other) fonts, if any, 
are always distributed 

BQ_END

BQ_BEGIN

with TeX or TeXLive or whatever that one can rely upon to be available for 
placing this particular 

BQ_END

BQ_BEGIN

glyph in a final PDF file? What would be the correct incantation to doing so? 

BQ_END

0xFFFD should be in any mainstream general use OpenType font. It might be 
better to ask which 
OT fonts to avoid due to low quality, bugs, lack of ongoing support, etc. There 
has been a lot of 
churn in the available fonts over the years, so the answer may be different if 
you need fonts that 
can be expected to have long-term support and availability. 

-- 
George N. White III 

Re: [XeTeX] Guaranteed Unicode replacement glyph in every TeX installation?

[Philip Taylor]

Ulrike Fischer wrote:


And sorry: the general case is that people have windows 10 (if they
have windows) and not windows 7.


If by "the general case", you mean "the majority", then statistically 
you are of course correct :



Desktop Windows Version Market Share Worldwide - July 2021
Win10   78.43%
Win716.03%
Win8.1  3.49%
Win81.17%
WinXP   0.59%
WinVista0.26%



but 16% still represents a /very/ large number of people, perhaps even 
in excess of 100x10^6 
.

--
/Philip Taylor/

Re: [XeTeX] Guaranteed Unicode replacement glyph in every TeX installation?

[Philip Taylor]

George N. White III wrote:

I have access to a couple Windows Systems (10 Enterprise and 10 Pro) 
with a full install
of TeX Live 2021 with current updates.  Both have 
"\bin\win32\albatross.exe"
and the default Windows TL2021 fontconfig setup.   Both work as 
advertised (but default Windows terminal
doesn't display UniCode and the output seems to be a fixed width so 
truncates the font file pathnames).

On linux the UniCode characters display properly.


Interesting.  Under Windows 7 64-bit (Enterprise) I find the same file 
in the same location; I wonder why a TeX Live 2021 CMD prompt did not 
recognise the command.

--
/Philip Taylor/

Re: [XeTeX] Guaranteed Unicode replacement glyph in every TeX installation?

[George N. White III]

On Sat, 21 Aug 2021 at 11:20, Philip Taylor (HI) <
p.tay...@hellenic-institute.uk> wrote:

> Ulrike Fischer wrote:
>
>
> With a current texlive you can use albatross to find out which fonts
> on your system support this
>
>albatross -d 0xFFFD
>
>
> I appreciate that this answer may have been targetted solely at Doug
> McKenna, but in the general case it does not work.  Here, using TeX Live
> 2021 under Windows 7 Ultimate, the command "albatross" is unknown —
>
> Microsoft Windows [Version 6.1.7601]
> Copyright (c) 2009 Microsoft Corporation.  All rights reserved.
>
> d:\Users\Philip Taylor>albatross -d 0xFFFD
> 'albatross' is not recognized as an internal or external command,
> operable program or batch file.
>
> d:\Users\Philip Taylor>
>
>
I have access to a couple Windows Systems (10 Enterprise and 10 Pro) with a
full install
of TeX Live 2021 with current updates.  Both have
"\bin\win32\albatross.exe"
and the default Windows TL2021 fontconfig setup.   Both work as advertised
(but default Windows terminal
doesn't display UniCode and the output seems to be a fixed width so
truncates the font file pathnames).
On linux the UniCode characters display properly.


-- 
George N. White III

Re: [XeTeX] Guaranteed Unicode replacement glyph in every TeX installation?

[George N. White III]

On Fri, 20 Aug 2021 at 22:46, Doug McKenna 
wrote:

> Using XeTeX, I want to typeset a LaTeX document into a PDF file.  The
> LaTeX source code in UTF-8
>
expressly includes the Unicode Replacement character (� = U+FFFD) (a black
> diamond with a
>
question mark in it).
>
> I want to typeset it in this single document using a monospaced font in
> one place, and in another
>
place in a variable-width font.
>
> I understand that XeTeX can take advantage of one's system's installed
> fonts, but my LaTeX file
>
is being generated by another program that doesn't know what those fonts
> are or what glyphs
>
they support.  I simply want to guarantee that the fonts used are always
> available when processing
>
that LaTeX file.
>

> I also understand that it's possible to synthesize the glyph graphically
> without using a font, but I'd
>
rather not go that route.
>
> So ... What fixed-width and variable-width OpenType (or other) fonts, if
> any, are always distributed
>
with TeX or TeXLive or whatever that one can rely upon to be available for
> placing this particular
>
glyph in a final PDF file?  What would be the correct incantation to doing
> so?
>

0xFFFD should be in any mainstream general use OpenType font.  It might be
better to ask which
OT fonts to avoid due to low quality, bugs, lack of ongoing support, etc.
There has been a lot of
churn in the available fonts over the years, so the answer may be different
if you need fonts that
can be expected to have long-term support and availability.

-- 
George N. White III

Re: [XeTeX] Guaranteed Unicode replacement glyph in every TeX installation?

[Ken Moffat via XeTeX]

On Sat, Aug 21, 2021 at 04:23:45PM -0600, Doug McKenna wrote:
> Ulrike -
> 
> Excellent.  Thank you!
> 
> Using \setmainfont{DejaVuSerif.ttf} works on my non-Linux machine, and it is 
> not listed as "installed" in my Mac's FontBook, which means it's being used 
> solely within the TeXosystem.
> 
> DejaVuSerif is a variable-width font.  Is there a similar fixed-width 
> OpenType/TrueType font distributed with TeXLive that would work?
> 
> 
> Doug McKenna
> Mathemaesthetics, Inc.
> 
Try DejaVuSansMono.ttf

ĸen
-- 
It might have helped Anathema understand what was going on if she
understood the very simple reason why she couldn't see Adam's
aura. It's for the same reason that people in Times Square can't
see America -- Good Omens, 'Hard Times'.

Re: [XeTeX] Guaranteed Unicode replacement glyph in every TeX installation?

[BPJ]

Den sön 22 aug. 2021 00:25Doug McKenna  skrev:

> Ulrike -
>
> Excellent.  Thank you!
>
> Using \setmainfont{DejaVuSerif.ttf} works on my non-Linux machine, and it
> is not listed as "installed" in my Mac's FontBook, which means it's being
> used solely within the TeXosystem.
>
> DejaVuSerif is a variable-width font.  Is there a similar fixed-width
> OpenType/TrueType font distributed with TeXLive that would work?
>

If there is DejaVu Serif there probably is DejaVu Sans Mono too. You may
want to look at the Noto Fonts from Google (that would be Noto Sans Mono
rather than the superseded Noto Mono!) which may have better Unicode
coverage, although the replacement character should be available in both.
Don't forget to use the Scale=MatchLowercase option!

/bpj



>
> Doug McKenna
> Mathemaesthetics, Inc.
>
>
>
> - Original Message -
> From: "news3" 
> To: "xetex" 
> Sent: Saturday, August 21, 2021 10:39:25 AM
> Subject: Re: [XeTeX] Guaranteed Unicode replacement glyph in every TeX
> installation?
>
> Am Sat, 21 Aug 2021 09:25:14 -0600 (MDT) schrieb Doug McKenna:
>
> > Thanks all for your interesting responses.
> >
> > Unfortunately, my possibly poorly worded question remains unanswered.
> Let me try again.
> >
> > Consider the short example just used:
> >
> > \documentclass{article}
> > \usepackage{fontspec}
> > \setmainfont{DejaVu Serif}
> >
> > \begin{document}
> > fffd
> > \end{document}
> >
> > When I run it, fontspec complains that it can't find the font. So
> obviously "DejaVu Serif" is not installed, either on my system or anywhere
> in the bowels of all the ~150,000 TeXLive (2019) files that have been
> installed in the TDS on my machine.
>
> No, it only says that it is not found by fontname. Something that
> happens often on linux. Try with \setmainfont{DejaVuSerif.ttf}
>
>
> > So, is there a font name I can use in the \setmainfont{} command
> > that is ALWAYS available (upon TeX installation) when processing
> > this LaTeX file with XeTeX? Or always available after a certain
> > version of a TeX installation?
>
> I have no idea when DejaVu was added but it is in texlive 2019.
> If you want to support also older systems try e.g. on overleaf.
>
>
>
> --
> Ulrike Fischer
> http://www.troubleshooting-tex.de/
>

Re: [XeTeX] Guaranteed Unicode replacement glyph in every TeX installation?

[Doug McKenna]

Ulrike -

Excellent.  Thank you!

Using \setmainfont{DejaVuSerif.ttf} works on my non-Linux machine, and it is 
not listed as "installed" in my Mac's FontBook, which means it's being used 
solely within the TeXosystem.

DejaVuSerif is a variable-width font.  Is there a similar fixed-width 
OpenType/TrueType font distributed with TeXLive that would work?


Doug McKenna
Mathemaesthetics, Inc.



- Original Message -
From: "news3" 
To: "xetex" 
Sent: Saturday, August 21, 2021 10:39:25 AM
Subject: Re: [XeTeX] Guaranteed Unicode replacement glyph in every TeX  
installation?

Am Sat, 21 Aug 2021 09:25:14 -0600 (MDT) schrieb Doug McKenna:

> Thanks all for your interesting responses. 
> 
> Unfortunately, my possibly poorly worded question remains unanswered. Let me 
> try again. 
> 
> Consider the short example just used: 
> 
> \documentclass{article} 
> \usepackage{fontspec} 
> \setmainfont{DejaVu Serif} 
> 
> \begin{document} 
> fffd 
> \end{document} 
> 
> When I run it, fontspec complains that it can't find the font. So obviously 
> "DejaVu Serif" is not installed, either on my system or anywhere in the 
> bowels of all the ~150,000 TeXLive (2019) files that have been installed in 
> the TDS on my machine. 

No, it only says that it is not found by fontname. Something that
happens often on linux. Try with \setmainfont{DejaVuSerif.ttf}


> So, is there a font name I can use in the \setmainfont{} command
> that is ALWAYS available (upon TeX installation) when processing
> this LaTeX file with XeTeX? Or always available after a certain
> version of a TeX installation? 

I have no idea when DejaVu was added but it is in texlive 2019.
If you want to support also older systems try e.g. on overleaf.



-- 
Ulrike Fischer 
http://www.troubleshooting-tex.de/

Re: [XeTeX] Guaranteed Unicode replacement glyph in every TeX installation?

[Ulrike Fischer]

Am Sat, 21 Aug 2021 09:25:14 -0600 (MDT) schrieb Doug McKenna:

> Thanks all for your interesting responses. 
> 
> Unfortunately, my possibly poorly worded question remains unanswered. Let me 
> try again. 
> 
> Consider the short example just used: 
> 
> \documentclass{article} 
> \usepackage{fontspec} 
> \setmainfont{DejaVu Serif} 
> 
> \begin{document} 
> fffd 
> \end{document} 
> 
> When I run it, fontspec complains that it can't find the font. So obviously 
> "DejaVu Serif" is not installed, either on my system or anywhere in the 
> bowels of all the ~150,000 TeXLive (2019) files that have been installed in 
> the TDS on my machine. 

No, it only says that it is not found by fontname. Something that
happens often on linux. Try with \setmainfont{DejaVuSerif.ttf}


> So, is there a font name I can use in the \setmainfont{} command
> that is ALWAYS available (upon TeX installation) when processing
> this LaTeX file with XeTeX? Or always available after a certain
> version of a TeX installation? 

I have no idea when DejaVu was added but it is in texlive 2019.
If you want to support also older systems try e.g. on overleaf.



-- 
Ulrike Fischer 
http://www.troubleshooting-tex.de/

Re: [XeTeX] Guaranteed Unicode replacement glyph in every TeX installation?

[Doug McKenna]

Thanks all for your interesting responses. 

Unfortunately, my possibly poorly worded question remains unanswered. Let me 
try again. 

Consider the short example just used: 

\documentclass{article} 
\usepackage{fontspec} 
\setmainfont{DejaVu Serif} 

\begin{document} 
fffd 
\end{document} 

When I run it, fontspec complains that it can't find the font. So obviously 
"DejaVu Serif" is not installed, either on my system or anywhere in the bowels 
of all the ~150,000 TeXLive (2019) files that have been installed in the TDS on 
my machine. 

So, is there a font name I can use in the \setmainfont{} command that is ALWAYS 
available (upon TeX installation) when processing this LaTeX file with XeTeX? 
Or always available after a certain version of a TeX installation? 

I want to automatedly create such a LaTeX file that permits its user to declare 
or override a default font for typesetting the Unicode Replacement character, 
but which doesn't require the user to search for or declare such fonts at all 
in the simplest case. One would think that every OpenType font supporting 
Unicode glyphs would include a glyph for U+FFFD, but it doesn't appear to me 
that that is the case. 

All I want is for the computer-generated LaTeX file "to just work" out of the 
box (so to speak), so that a naive user isn't faced with an error message the 
first time they typeset it with something like TeXShop. 

Doug McKenna 
Mathemaesthetics, Inc. 

Re: [XeTeX] Guaranteed Unicode replacement glyph in every TeX installation?

[Ulrike Fischer]

Am Sat, 21 Aug 2021 15:20:22 +0100 schrieb Philip Taylor (HI):

> I appreciate that this answer may have been targetted solely at Doug 
> McKenna, but in the general case it does not work.  Here, using TeX Live 
> 2021 under Windows 7 Ultimate, the command "albatross" is unknown —
> 
>> Microsoft Windows [Version 6.1.7601]
>> Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

Next time I will write "with a current vanilla texlive on a current
OS with everything installed" to avoid that you get confused. 

And sorry: the general case is that people have windows 10 (if they
have windows) and not windows 7. 


-- 
Ulrike Fischer 
http://www.troubleshooting-tex.de/

Re: [XeTeX] Guaranteed Unicode replacement glyph in every TeX installation?

[Janusz S . Bień]

On Sat, Aug 21 2021 at 15:20 +01, Philip Taylor (HI) wrote:
> Ulrike Fischer wrote:
>
>  With a current texlive you can use albatross to find out which fonts
> on your system support this
>
>albatross -d 0xFFFD
>
> I appreciate that this answer may have been targetted solely at Doug McKenna, 
> but in the general case it does not work.  Here, using TeX Live 2021 under
> Windows 7 Ultimate, the command "albatross" is unknown —

It's also absent in all verions of Debian, including testing, unstable
and experimental.

I've just downloaded it from CTAN (I had to circumvent a minor path
problem).

Best regards

Janusz

-- 
 ,   
Janusz S. Bien
emeryt (emeritus)
https://sites.google.com/view/jsbien

Re: [XeTeX] Guaranteed Unicode replacement glyph in every TeX installation?

[Philip Taylor (HI)]

Ulrike Fischer wrote:



With a current texlive you can use albatross to find out which fonts
on your system support this

albatross -d 0xFFFD


I appreciate that this answer may have been targetted solely at Doug 
McKenna, but in the general case it does not work.  Here, using TeX Live 
2021 under Windows 7 Ultimate, the command "albatross" is unknown —



Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

d:\Users\Philip Taylor>albatross -d 0xFFFD
'albatross' is not recognized as an internal or external command,
operable program or batch file.

d:\Users\Philip Taylor>



--
/Philip Taylor/

Re: [XeTeX] Guaranteed Unicode replacement glyph in every TeX installation?

[David Carlisle]

On Sat, 21 Aug 2021, 13:46 Janusz S. Bień,  wrote:

> On Sat, Aug 21 2021 at 10:48 +02, Ulrike Fischer wrote:
> > Am Fri, 20 Aug 2021 19:45:37 -0600 (MDT) schrieb Doug McKenna:
> >
> >> Using XeTeX, I want to typeset a LaTeX document into a PDF file. The
> LaTeX source code in UTF-8 expressly includes the Unicode Replacement
> character (� = U+FFFD)
> >
> >> So ... What fixed-width and variable-width OpenType (or other)
> >> fonts, if any, are always distributed with TeX or TeXLive or
> >> whatever that one can rely upon to be available for placing this
> >> particular glyph in a final PDF file? What would be the correct
> >> incantation to doing so?
> >
> > With a current texlive you can use albatross to find out which fonts
> > on your system support this
> >
> >albatross -d 0xFFFD
>
> A nice replacement for fc-search-codepoint, as it accept also glyphs as
> the arguments.
>

note that fc-list will only find fonts known to fontconfig so by default
this won't include fonts distributed with texlive

David

>
> Best regard
>
> Janusz
>
> --
>  ,
> Janusz S. Bien
> emeryt (emeritus)
> https://sites.google.com/view/jsbien
>
>

Re: [XeTeX] Guaranteed Unicode replacement glyph in every TeX installation?

[Janusz S . Bień]

On Sat, Aug 21 2021 at 10:48 +02, Ulrike Fischer wrote:
> Am Fri, 20 Aug 2021 19:45:37 -0600 (MDT) schrieb Doug McKenna:
>
>> Using XeTeX, I want to typeset a LaTeX document into a PDF file. The LaTeX 
>> source code in UTF-8 expressly includes the Unicode Replacement character (� 
>> = U+FFFD)
>
>> So ... What fixed-width and variable-width OpenType (or other)
>> fonts, if any, are always distributed with TeX or TeXLive or
>> whatever that one can rely upon to be available for placing this
>> particular glyph in a final PDF file? What would be the correct
>> incantation to doing so? 
>
> With a current texlive you can use albatross to find out which fonts
> on your system support this
>
>albatross -d 0xFFFD

A nice replacement for fc-search-codepoint, as it accept also glyphs as
the arguments.

Best regard

Janusz

-- 
 ,   
Janusz S. Bien
emeryt (emeritus)
https://sites.google.com/view/jsbien

Re: [XeTeX] Guaranteed Unicode replacement glyph in every TeX installation?

[Ulrike Fischer]

Am Fri, 20 Aug 2021 19:45:37 -0600 (MDT) schrieb Doug McKenna:

> Using XeTeX, I want to typeset a LaTeX document into a PDF file. The LaTeX 
> source code in UTF-8 expressly includes the Unicode Replacement character (� 
> = U+FFFD)

> So ... What fixed-width and variable-width OpenType (or other)
> fonts, if any, are always distributed with TeX or TeXLive or
> whatever that one can rely upon to be available for placing this
> particular glyph in a final PDF file? What would be the correct
> incantation to doing so? 

With a current texlive you can use albatross to find out which fonts
on your system support this

   albatross -d 0xFFFD

It will also show the pathes so identifying the fonts which are in
texlive is quite easy. 

\documentclass{article}
\usepackage{fontspec}
\setmainfont{DejaVu Serif}

\begin{document}

fffd 

\end{document}


-- 
Ulrike Fischer 
http://www.troubleshooting-tex.de/

Re: [XeTeX] Guaranteed Unicode replacement glyph in every TeX installation?

[Janusz S . Bień]

On Fri, Aug 20 2021 at 19:45 -06, Doug McKenna wrote:

[...]

> So ... What fixed-width and variable-width OpenType (or other) fonts,
> if any, are always distributed with TeX or TeXLive or whatever that
> one can rely upon to be available for placing this particular glyph in
> a final PDF file?  What would be the correct incantation to doing so?

The question is which fonts you have installed on your system, not
necessarily distributed with TeX.

For checking whether you have an appropriate there are useful
suggestions (Linux specific) at

https://unix.stackexchange.com/questions/162305/find-the-best-font-for-rendering-a-codepoint

The answer

fc-list ':charset= '

is still valid, but the very usefull script

fc-search-codepoint

was written in Python2 so is unusable e.g. on Debian bullseyey.

Another approach is to use "Font Support" at
https://www.fileformat.info/. e.g.

https://www.fileformat.info/info/unicode/char/fffd/fontsupport.htm

Regards

Janusz

-- 
 ,   
Janusz S. Bien
emeryt (emeritus)
https://sites.google.com/view/jsbien