Re: Demystifying “symbol” fonts

[Caolán McNamara]

On Fri, 2016-11-18 at 05:53 +0200, Khaled Hosny wrote:
> simplest fix is to find a suitably licensed font and copy the few
> missing glyphs (a quick look shows only 21 glyphs are missing from
> Adobe Symbol encoding). What is the current license of OpenSymbol? I
> remember is was a bit of a mess few years ago.

extras/source/truetype/symbol/README and
extras/source/truetype/symbol/OpenSymbol.sfd
for every addition add a new entry to the Copyright with the original
author of the addition, ideally IMO mentioning the range added.

C.
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Demystifying “symbol” fonts

[Khaled Hosny]

On Fri, Nov 18, 2016 at 02:28:59AM +0200, Khaled Hosny wrote:
> So I think the problem here is that we are not doing any glyph fallback
> for OpenSymbol. Looks like we skip glyph fallback for symbol fonts in
> vcl/unx/generic/fontmanager/fontsubst.cxx
> FcPreMatchSubstitution::FindFontSubstitute(), which makes sense generally,
> except when you need it…

I think messing with the font fallback is something I want to do, so the
simplest fix is to find a suitably licensed font and copy the few
missing glyphs (a quick look shows only 21 glyphs are missing from Adobe
Symbol encoding). What is the current license of OpenSymbol? I remember
is was a bit of a mess few years ago.

Regards,
Khaled
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Demystifying “symbol” fonts

[Khaled Hosny]

On Thu, Nov 17, 2016 at 09:10:28PM +, Caolán McNamara wrote:
> On Thu, 2016-11-17 at 14:00 +0200, Khaled Hosny wrote:
> > Hi all,
> > 
> > I’m looking into bug 103944, which is mainly about handling so-called
> > symbol fonts.
> > 
> > The bullets in the bug document use the “Symbol” font from Windows,
> > which has “symbol” cmap subtable that maps characters from Private
> > Use Area to its own glyphs. Because the document uses PUA symbols,
> > without this font installed or another font that has the exact same
> > glyphs for PUA characters we can get anything from boxes to totally
> > unrelated glyphs.
> > 
> > Now, AFAIK, all glyphs in “Symbol” font have proper characters in
> > Unicode (and the font itself supports them as well, in its non-symbol
> > cmap subtable), so my question is what is the expected behaviour
> > here, map the PUA to proper Unicode characters? Expect the font or a
> > compatible one to be installed and usable? Bundle a compatible font?
> 
> So, IIRC, as an aside, Font::IsSymbolFont is set to true for a font
> derived from the  SYMBOL_CHARSET/RTL_TEXTENODING_SYMBOL cases. So a
> font like OpenSymbol is !IsSymbolFont, which is why there is code like
> IsSymbolFont() || name == "OpenSymbol".
> 
> The standard font/glyph fallback stuff doesn't/shouldn't apply to
> IsSymbolFont fonts generally, there's another piece of machinery built
> around unotools/source/misc/fontcvt.cxx to map "well known" dingbat
> fonts to and from unicode or pua opensymbol.
> 
> One place in vcl where we make use of this is
> vcl/source/font/fontcache.cxx and search for ConvertChar::GetRecodeData
> and its use of the ConvertChar class. We have one extra special case
> for !IsSymbolFont for the "Symbol" font under Mac already.

OK, looks like the mystery is unfolding :) So IIUC, we are supposed to
remap the PUA. Indeed with https://gerrit.libreoffice.org/#/c/30932/ the
remapping happens even for the original bug document (U+F031 → U+0031).
However we insist on using OpenSymbol which does not have U+0031, and I
guess this is what the original commit was trying to fix.

So I think the problem here is that we are not doing any glyph fallback
for OpenSymbol. Looks like we skip glyph fallback for symbol fonts in
vcl/unx/generic/fontmanager/fontsubst.cxx
FcPreMatchSubstitution::FindFontSubstitute(), which makes sense generally,
except when you need it…

Regards,
Khaled
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Demystifying “symbol” fonts

[Caolán McNamara]

On Thu, 2016-11-17 at 14:00 +0200, Khaled Hosny wrote:
> Hi all,
> 
> I’m looking into bug 103944, which is mainly about handling so-called
> symbol fonts.
> 
> The bullets in the bug document use the “Symbol” font from Windows,
> which has “symbol” cmap subtable that maps characters from Private
> Use Area to its own glyphs. Because the document uses PUA symbols,
> without this font installed or another font that has the exact same
> glyphs for PUA characters we can get anything from boxes to totally
> unrelated glyphs.
> 
> Now, AFAIK, all glyphs in “Symbol” font have proper characters in
> Unicode (and the font itself supports them as well, in its non-symbol
> cmap subtable), so my question is what is the expected behaviour
> here, map the PUA to proper Unicode characters? Expect the font or a
> compatible one to be installed and usable? Bundle a compatible font?

So, IIRC, as an aside, Font::IsSymbolFont is set to true for a font
derived from the  SYMBOL_CHARSET/RTL_TEXTENODING_SYMBOL cases. So a
font like OpenSymbol is !IsSymbolFont, which is why there is code like
IsSymbolFont() || name == "OpenSymbol".

The standard font/glyph fallback stuff doesn't/shouldn't apply to
IsSymbolFont fonts generally, there's another piece of machinery built
around unotools/source/misc/fontcvt.cxx to map "well known" dingbat
fonts to and from unicode or pua opensymbol.

One place in vcl where we make use of this is
vcl/source/font/fontcache.cxx and search for ConvertChar::GetRecodeData
and its use of the ConvertChar class. We have one extra special case
for !IsSymbolFont for the "Symbol" font under Mac already.

My first instincts would be to poke around there and see if that's the
right place to fix it or not. i.e. let vcl use the unotools fontcvt
stuff to map the PUA to something we do have

It might even be that removing the #ifdef MACOSX line does something
useful.

C.
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Demystifying “symbol” fonts

[Kaganski Mike]

11/17/2016 11:08 PM, Khaled Hosny пишет:

We are dealing here with imported documented, not what we are exporting.
The use of PUA certainly is backwards, but that is how is it and we have
to deal with it already. Again I’m asking what is the expectation in our
context, not in general.


IMO, we should:
(1) bundle our font suitable for proper replacement of such symbol fonts, 
because doing otherwise (depending on system fonts) leaves users with different 
problems like absent fonts and wrong sizing because of different font metrics;
(2) use OpenSymbol for that, because that is its purpose from the beginning, 
and we already bundle it;
(3) use proper Unicode ranges in OpenSymbol, but provide mapping from PUA 
inside LO code (not alternative mappings in font itself).

--
Best regards,
Mike Kaganski
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Demystifying “symbol” fonts

[Khaled Hosny]

On Thu, Nov 17, 2016 at 07:46:40PM +, toki wrote:
> 
> 
> On 17/11/2016 12:00, Khaled Hosny wrote:
> 
> > The bullets in the bug document use the “Symbol” font from Windows,
> > which has “symbol” cmap subtable that maps characters from Private Use
> > Area to its own glyphs. Because the document uses PUA symbols, without
> 
> Anything in the PUA is up for grabs.

That is a general statement, and well understood. However I’m asking in
the context of office suits and document format, and the position in our
code base.

> For bullets, you should be using glyphs in the code point range Ux2600
> through Ux27BF and/or Ux1F300 through Ux1F67F.

We are dealing here with imported documented, not what we are exporting.
The use of PUA certainly is backwards, but that is how is it and we have
to deal with it already. Again I’m asking what is the expectation in our
context, not in general.

Regards,
Khaled


signature.asc
Description: PGP signature
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Demystifying “symbol” fonts

[toki]

On 17/11/2016 12:00, Khaled Hosny wrote:

> The bullets in the bug document use the “Symbol” font from Windows,
> which has “symbol” cmap subtable that maps characters from Private Use
> Area to its own glyphs. Because the document uses PUA symbols, without

Anything in the PUA is up for grabs.

If using glyphs from the PUA, the only safe assumption is that the
reader will see Tofu.

MUFI, CSUR, and UCSUR are very careful to point out that the code points
the writing systems within their specific registry, can clash with each
other writing systems within their specific registry, as well as with
writing systems from other organizations that have set up PUA registries.

Those organizations also point out that when using writing systems
within their specific registry, one also needs to use the font they
recommend for the specific writing system.

For bullets, you should be using glyphs in the code point range Ux2600
through Ux27BF and/or Ux1F300 through Ux1F67F. (OK, there are couple of
other ranges that have nice glyphs to use as bullets. My point is that
there is no legitimate reason to use PUA glyphs as bullets. That LibO
does not correctly handle some of the glyphs in the Ux1F6## range is a
separate issue.)

> so my question is what is the expected behaviour here, map the PUA to proper 
> Unicode characters? Expect the font or a compatible one to be installed and 
> usable? Bundle a compatible font?

If the original font is embedded in the document, then the expected
behaviour is to display the glyph in question.

If the original font is not embedded in the document, then the expected
behaviour is to display Tofu.

jonathon



signature.asc
Description: OpenPGP digital signature
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Demystifying “symbol” fonts

[Khaled Hosny]

On Thu, Nov 17, 2016 at 02:07:34PM +0100, Stephan Bergmann wrote:
> On 11/17/2016 01:00 PM, Khaled Hosny wrote:
> > I’m looking into bug 103944, which is mainly about handling so-called
> > symbol fonts.
> > 
> > The bullets in the bug document use the “Symbol” font from Windows,
> > which has “symbol” cmap subtable that maps characters from Private Use
> > Area to its own glyphs. Because the document uses PUA symbols, without
> > this font installed or another font that has the exact same glyphs for
> > PUA characters we can get anything from boxes to totally unrelated
> > glyphs.
> > 
> > Now, AFAIK, all glyphs in “Symbol” font have proper characters in
> > Unicode (and the font itself supports them as well, in its non-symbol
> > cmap subtable), so my question is what is the expected behaviour here,
> > map the PUA to proper Unicode characters? Expect the font or a
> > compatible one to be installed and usable? Bundle a compatible font?
> 
> Another interesting question would be whether LO makes it too easy to create
> a document with PUA instead of "proper" characters (by deceivingly offering
> PUA characters in "Insert - Special Character...", say), but looking at
>  that bug .doc
> at least is presumably created with something else?

I’m mostly concerned about “standardized” PUA like bullets and other
stuff generated by other office suits, not user input. If I user is
entering PUA directly, then it is not a bug when the original font is
missing and he gets garbage, he (or the document author) brought it on
himself.

Regards,
Khaled
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Demystifying “symbol” fonts

[Khaled Hosny]

On Thu, Nov 17, 2016 at 02:12:49PM +0100, Miklos Vajna wrote:
> Hi Khaled,
> 
> On Thu, Nov 17, 2016 at 02:00:13PM +0200, Khaled Hosny 
>  wrote:
> > Now, AFAIK, all glyphs in “Symbol” font have proper characters in
> > Unicode (and the font itself supports them as well, in its non-symbol
> > cmap subtable), so my question is what is the expected behaviour here,
> > map the PUA to proper Unicode characters? Expect the font or a
> > compatible one to be installed and usable? Bundle a compatible font?
> 
> We bundle opensymbol (extras/source/truetype/symbol/OpenSymbol.sfd),
> isn't that there exactly so that we have no problems when the symbol
> font is missing? Or does that have nothing to do with your problem?

It seems to not cover all the symbols in Symbol (what a great font
name), or the original bug fixed by the commit in
https://gerrit.libreoffice.org/#/c/30932/ wouldn’t happen in the first
place.

So my question is really what is the current indented behaviour, as I
see loots of mystifying mentions of symbol font, conversion tables and
all convoluted stuff I can’t warp my head around.

Regards,
Khaled


signature.asc
Description: PGP signature
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Demystifying “symbol” fonts

[Miklos Vajna]

Hi Khaled,

On Thu, Nov 17, 2016 at 02:00:13PM +0200, Khaled Hosny  
wrote:
> Now, AFAIK, all glyphs in “Symbol” font have proper characters in
> Unicode (and the font itself supports them as well, in its non-symbol
> cmap subtable), so my question is what is the expected behaviour here,
> map the PUA to proper Unicode characters? Expect the font or a
> compatible one to be installed and usable? Bundle a compatible font?

We bundle opensymbol (extras/source/truetype/symbol/OpenSymbol.sfd),
isn't that there exactly so that we have no problems when the symbol
font is missing? Or does that have nothing to do with your problem?

Regards,

Miklos


signature.asc
Description: Digital signature
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Demystifying “symbol” fonts

[Stephan Bergmann]

On 11/17/2016 01:00 PM, Khaled Hosny wrote:

I’m looking into bug 103944, which is mainly about handling so-called
symbol fonts.

The bullets in the bug document use the “Symbol” font from Windows,
which has “symbol” cmap subtable that maps characters from Private Use
Area to its own glyphs. Because the document uses PUA symbols, without
this font installed or another font that has the exact same glyphs for
PUA characters we can get anything from boxes to totally unrelated
glyphs.

Now, AFAIK, all glyphs in “Symbol” font have proper characters in
Unicode (and the font itself supports them as well, in its non-symbol
cmap subtable), so my question is what is the expected behaviour here,
map the PUA to proper Unicode characters? Expect the font or a
compatible one to be installed and usable? Bundle a compatible font?


Another interesting question would be whether LO makes it too easy to 
create a document with PUA instead of "proper" characters (by 
deceivingly offering PUA characters in "Insert - Special Character...", 
say), but looking at 
 that bug 
.doc at least is presumably created with something else?

___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Demystifying “symbol” fonts

[Khaled Hosny]

Hi all,

I’m looking into bug 103944, which is mainly about handling so-called
symbol fonts.

The bullets in the bug document use the “Symbol” font from Windows,
which has “symbol” cmap subtable that maps characters from Private Use
Area to its own glyphs. Because the document uses PUA symbols, without
this font installed or another font that has the exact same glyphs for
PUA characters we can get anything from boxes to totally unrelated
glyphs.

Now, AFAIK, all glyphs in “Symbol” font have proper characters in
Unicode (and the font itself supports them as well, in its non-symbol
cmap subtable), so my question is what is the expected behaviour here,
map the PUA to proper Unicode characters? Expect the font or a
compatible one to be installed and usable? Bundle a compatible font?

Regards,
Khaled

1. https://bugs.documentfoundation.org/show_bug.cgi?id=103944
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice