Re: Errors with vref on de/Additional.lyx with lualatex

[Guenter Milde]

On 2017-07-22, Kornel Benko wrote:
> Am Samstag, 22. Juli 2017 um 15:04:06, schrieb Guenter Milde 
> 
>> On 2017-07-21, Kornel Benko wrote:
>> > Am Freitag, 21. Juli 2017 um 22:08:48, schrieb Guenter Milde 
>> > 

...

>> The default fonts used with fontspec (LatinModern) miss ZWNJ in serif,
>> sans, and mono!

>> >> * Actually, in the case of ZWNJ, just dropping the missing character 
>> >> results
>> >>   in correct output. 
...
>> The core of the problem can be described as a combination of

>> a) Since TeXLive 2016, "fontspec" maps \textcompwordmark to the ZWNJ
>>character. This character is missing in many fonts, leading to a
>>warning in the output.
>> b) LyX turns "missing character" warnings into errors.

...

>> If there is no time for a correct fix, then the common procedure is to
>> file a bug report. 

Done. https://www.lyx.org/trac/ticket/10727.

Jürgen provided a patch, let's see if it works.
(OK, if the locale encoding is UTF8, fails here on Linux with LANG=en_GB).

>> ...

>> > No, DejaVu is selected because of endless latex loop, nothing to do
>> > with \\textcompwordmark

>> I don't get the loop with LatinModern, only a long list of

>>   Missing character: There is no  in font 
>> [lmroman12-regular]:mapping=tex-text;!

>> for lmroman, lmsans, and lmmono.

> Therefore lmodern is not good here to be used for tests, don't you think too?

Once #10727 is fixed, the document should compile fine with default
non-TeX fonts. 

> Try with Free-font. The error message is
> lualatex && xelatex:
>Package varioref Error: \vref at page boundary 14-15 (may loop).

...

> The underlying problem is the use of Free-font in the test machinery.

We have actually 2 independent problems with de/Additional.lyx:

  1. "missing ZWNJ treated as error": this is now LyX bug #10727

  2. \vref at page boundary with FreeFonts and fontspec.

 This is a "TeX issue", a generic shortcoming of the varioref
 package. In this document, it shows up with FreeFont because the
 combination of the content and the font metrics result in a \vref at
 a page boundary.

These problems only affect the secondary use (tests). None of them is a
problem for the primary use (documentation) of the file.
Therefore, there is no need to change the document.


> It was your suggestion to use Dejavu for this case,

Indeed, this was my first try, because the standard workaround for missing
characters (switching the font to Free* in a script) did not work and
there were cases where DejaVu helped. However, also with DejaVu
problem 1 prevents successfull compilation until #10727 is fixed.

> so why are you insisting?

If all three, the default fonts (Latin Modern), the standard replacement
fonts (Free*) and the custom replacement fonts (DejaVu) fail, it is IMO
better to do without special casing and use the default fonts.


>> If the bug cannot be fixed soon, I propose to invert affected tests as
>> "LyXbug" or to temporarily add them to the "missing chars" exception list.

> I dare to cite you:

>   "This is a rare running-condition that depends on the selected
>   font (because this influences page-breaks)"

This is why I suggest to solve problem 1 (and temporarily invert the test
if required), not to work around problem 2.

Günter

Re: Errors with vref on de/Additional.lyx with lualatex

[Kornel Benko]

Am Samstag, 22. Juli 2017 um 15:04:06, schrieb Guenter Milde 

> Dear Kornel,
> 
> On 2017-07-21, Kornel Benko wrote:
> > Am Freitag, 21. Juli 2017 um 22:08:48, schrieb Guenter Milde 
> > 
> >> On 2017-07-20, Kornel Benko wrote:
> >> > Am Mittwoch, 19. Juli 2017 um 22:23:32, schrieb Kornel Benko 
> >> > 
> >> >> Am Mittwoch, 19. Juli 2017 um 19:48:49, schrieb Guenter Milde 
> >> >> 
> >> >> > On 2017-07-19, Kornel Benko wrote:
> >> >> > > Am Mittwoch, 19. Juli 2017 um 15:00:16, schrieb Kornel Benko 
> >> >> > > 
> 
> >> >> I don't insist. I would insist on removing textcompwordmark in mono
> >> >> fonts, but did not see an easy way.
> 
> >> > It turned out, it is easy.
> 
> >> But it is still wrong: 
> 
> >> * It is LyX policy to keep LaTeX output stable (unless it turns out to be
> >>   wrong).
> 
> >> * \textcompwordmark is the correct LaTeX internal character
> >>   representation (LICR) for u200C (zero width non joiner, ZWNJ).
> 
> > OK
> 
> >>   If some fonts are missing this character this is a bug in the fonts
> >>   or in the code calling this missing character (here tu1enc.def in
> >>   latex/base/).
> 
> >> * The workaround should fix the command, not use a different
> >>   one. Power users are then still able to override our workaround with
> >>   their own re-definition of \textcompwordmark.
> 
> > Mark, that the change is done only for mono fonts.
> 
> The default fonts used with fontspec (LatinModern) miss ZWNJ in serif,
> sans, and mono!
> 
> >> * Actually, in the case of ZWNJ, just dropping the missing character 
> >> results
> >>   in correct output. 
> >>   --> We don't even need to add a workaround, if we remove
> >> the "warning to error conversion for missing characters" for this case.
> 
> 
> The core of the problem can be described as a combination of
> 
> a) Since TeXLive 2016, "fontspec" maps \textcompwordmark to the ZWNJ
>character. This character is missing in many fonts, leading to a
>warning in the output.
>
> b) LyX turns "missing character" warnings into errors.
> 
> We may report a) to "fontspec", but chances are high that they will just tell
> "this is just a warning that can be safely ignored". Although it is still
> problematic, because the same type of warning may hint to data loss in other
> cases, I don't expect we can force "fontspec" to change its behaviour.
> 
> This means the task of differentiation between harmless and harmfull
> "missing character events" remains at the LyX side.
> We already do this for the special case of "nullfont" (see
> http://www.lyx.org/trac/ticket/10394) to work around a similar "false
> positive" with the PNG package.
> 
> > Please not now. Who may do the work?
> 
> If there is no time for a correct fix, then the common procedure is to
> file a bug report. See https://www.lyx.org/trac/ticket/10727.
> 
> 
> 
> ...
> 
> > No, DejaVu is selected because of endless latex loop, nothing to do
> > with \\textcompwordmark
> 
> I don't get the loop with LatinModern, only a long list of
> 
>   Missing character: There is no  in font 
> [lmroman12-regular]:mapping=tex-text;!
> 
> for lmroman, lmsans, and lmmono.

Therefore lmodern is not good here to be used for tests, don't you think too?
Try with Free-font. The error message is
lualatex && xelatex:
 Package varioref Error: \vref at page boundary 14-15 (may loop).

See the varioref package documentation for explanation.
Type  H   for immediate help.
 ...  
  
l.809 ...t\ \vref{Literaturverzeichnisse-mit-BibTeX}
  
Please check the pages in question. You might need to replace the \vref
or \vpageref by a normal \(page)ref to stop LaTeX running forever.



> >> ...
> 
> >> > +\font_roman "lmodern" "DejaVu Serif [bitstream]"
> >> > +\font_sans "lmss" "DejaVu Sans [bitstream]"
> >> > +\font_typewriter "lmtt" "DejaVu Sans Mono [bitstream]"
> 
> 
> >> This fixes the test case but not the problem.
> 
> > This fixes the lyx file which we provide.
> 
> ... and by this hides the underlying problem.

The underlying problem is the use of Free-font in the test machinery.
It was your suggestion to use Dejavu for this case case, so why are you 
insisting?

> The LyX file we provide works fine as documentation. It sets the default
> output to pdflatex and this works without problems.
> 
> The secondary use as test case revealed a real problem: this is good.
> 
> If the bug cannot be fixed soon, I propose to invert affected tests as
> "LyXbug" or to temporarily add them to the "missing chars" exception list.

I dare to cite you:
"This is a rare running-condition that depends on the selected font 
(because
this influences page-breaks)"

> Günter

Kornel

signature.asc
Description: This is a 

Re: Errors with vref on de/Additional.lyx with lualatex

[Guenter Milde]

Dear Kornel,

On 2017-07-21, Kornel Benko wrote:
> Am Freitag, 21. Juli 2017 um 22:08:48, schrieb Guenter Milde 
> 
>> On 2017-07-20, Kornel Benko wrote:
>> > Am Mittwoch, 19. Juli 2017 um 22:23:32, schrieb Kornel Benko 
>> > 
>> >> Am Mittwoch, 19. Juli 2017 um 19:48:49, schrieb Guenter Milde 
>> >> 
>> >> > On 2017-07-19, Kornel Benko wrote:
>> >> > > Am Mittwoch, 19. Juli 2017 um 15:00:16, schrieb Kornel Benko 
>> >> > > 

>> >> I don't insist. I would insist on removing textcompwordmark in mono
>> >> fonts, but did not see an easy way.

>> > It turned out, it is easy.

>> But it is still wrong: 

>> * It is LyX policy to keep LaTeX output stable (unless it turns out to be
>>   wrong).

>> * \textcompwordmark is the correct LaTeX internal character
>>   representation (LICR) for u200C (zero width non joiner, ZWNJ).

> OK

>>   If some fonts are missing this character this is a bug in the fonts
>>   or in the code calling this missing character (here tu1enc.def in
>>   latex/base/).

>> * The workaround should fix the command, not use a different
>>   one. Power users are then still able to override our workaround with
>>   their own re-definition of \textcompwordmark.

> Mark, that the change is done only for mono fonts.

The default fonts used with fontspec (LatinModern) miss ZWNJ in serif,
sans, and mono!

>> * Actually, in the case of ZWNJ, just dropping the missing character results
>>   in correct output. 
>>   --> We don't even need to add a workaround, if we remove
>> the "warning to error conversion for missing characters" for this case.


The core of the problem can be described as a combination of

a) Since TeXLive 2016, "fontspec" maps \textcompwordmark to the ZWNJ
   character. This character is missing in many fonts, leading to a
   warning in the output.
   
b) LyX turns "missing character" warnings into errors.

We may report a) to "fontspec", but chances are high that they will just tell
"this is just a warning that can be safely ignored". Although it is still
problematic, because the same type of warning may hint to data loss in other
cases, I don't expect we can force "fontspec" to change its behaviour.

This means the task of differentiation between harmless and harmfull
"missing character events" remains at the LyX side.
We already do this for the special case of "nullfont" (see
http://www.lyx.org/trac/ticket/10394) to work around a similar "false
positive" with the PNG package.

> Please not now. Who may do the work?

If there is no time for a correct fix, then the common procedure is to
file a bug report. See https://www.lyx.org/trac/ticket/10727.



...

> No, DejaVu is selected because of endless latex loop, nothing to do
> with \\textcompwordmark

I don't get the loop with LatinModern, only a long list of

  Missing character: There is no  in font [lmroman12-regular]:mapping=tex-text;!

for lmroman, lmsans, and lmmono.

>> ...

>> > +\font_roman "lmodern" "DejaVu Serif [bitstream]"
>> > +\font_sans "lmss" "DejaVu Sans [bitstream]"
>> > +\font_typewriter "lmtt" "DejaVu Sans Mono [bitstream]"


>> This fixes the test case but not the problem.

> This fixes the lyx file which we provide.

... and by this hides the underlying problem.

The LyX file we provide works fine as documentation. It sets the default
output to pdflatex and this works without problems.

The secondary use as test case revealed a real problem: this is good.

If the bug cannot be fixed soon, I propose to invert affected tests as
"LyXbug" or to temporarily add them to the "missing chars" exception list.

Günter

Re: Errors with vref on de/Additional.lyx with lualatex

[Kornel Benko]

Am Freitag, 21. Juli 2017 um 22:08:48, schrieb Guenter Milde 

> On 2017-07-20, Kornel Benko wrote:
> > Am Mittwoch, 19. Juli 2017 um 22:23:32, schrieb Kornel Benko 
> > 
> >> Am Mittwoch, 19. Juli 2017 um 19:48:49, schrieb Guenter Milde 
> >> 
> >> > On 2017-07-19, Kornel Benko wrote:
> >> > 
> >> > > [-- Type: text/plain, Encoding: 7bit --]
> >> > 
> >> > > Am Mittwoch, 19. Juli 2017 um 15:00:16, schrieb Kornel Benko 
> >> > > 
> >> > >> So, maybe better we could omit \textcompwordmark in mono fonts.
> >> > 
> >> > > This patch works for me. It uses vphantom{}, but only between '<<' and 
> >> > > '>>'.
> >> > 
> >> > See my other post for alternatives solving the "missing character" error
> >> > with Unicode fonts for \textcompwordmark.
> 
> >> I have read it. I understand the usage and I don’t insist on removing
> >> textcompwordmark any longer.
> 
> >> I don't insist. I would insist on removing textcompwordmark in mono
> >> fonts, but did not see an easy way.
> 
> > It turned out, it is easy.
> 
> But it is still wrong: 
> 
> * It is LyX policy to keep LaTeX output stable (unless it turns out to be
>   wrong).
> 
> * \textcompwordmark is the correct LaTeX internal character
>   representation (LICR) for u200C (zero width non joiner, ZWNJ).

OK

>   If some fonts are missing this character this is a bug in the fonts or
>   in the code calling this missing character (here tu1enc.def in latex/base/).
>   
> * The workaround should fix the command, not use a different
>   one. Power users are then still able to override our workaround with
>   their own re-definition of \textcompwordmark.

Mark, that the change is done only for mono fonts.

> 
> * Actually, in the case of ZWNJ, just dropping the missing character results
>   in correct output. 
>   --> We don't even need to add a workaround, if we remove
> the "warning to error conversion for missing characters" for this case.
> 

Please not now. Who may do the work?

> 
> > Output for desired ligatures with the attached patch is:
> > "\\vphantom{}" if mono mono-fonts
> > "\\textcompwordmark " else.
> 
> This will only help for documents using DejaVu.  OTOH, it is not required
> for Free* and no help for LatinModern.
> 

No, DejaVu is selected because of endless latex loop, nothing to do with 
\\textcompwordmark

> 
> ...
> 
> > diff --git a/lib/doc/de/Additional.lyx b/lib/doc/de/Additional.lyx
> > index 5ccd07b..539aa6d 100644
> > --- a/lib/doc/de/Additional.lyx
> > +++ b/lib/doc/de/Additional.lyx
> > @@ -41,9 +41,9 @@ shapepar
> >  \language_package default
> >  \inputencoding auto
> >  \fontencoding global
> > -\font_roman "lmodern" "default"
> > -\font_sans "lmss" "default"
> > -\font_typewriter "lmtt" "default"
> > +\font_roman "lmodern" "DejaVu Serif [bitstream]"
> > +\font_sans "lmss" "DejaVu Sans [bitstream]"
> > +\font_typewriter "lmtt" "DejaVu Sans Mono [bitstream]"
> >  \font_math "auto" "auto"
> >  \font_default_family default
> >  \use_non_tex_fonts false
> 
> This fixes the test case but not the problem.

This fixes the lyx file which we provide.

> The correct fix is either an exception to the "error-on-missing-chars"
> feature, or providing the workaround (re-defining to the original default):
> 
>   % ZWNJ missing in many Unicode fonts:
>   \DeclareTextCommand{\textcompwordmark}{\UnicodeEncodingName}{%
>  \leavevmode\kern\z@}
> 
> in LaTeXFeatures.cpp and "require" it in case \textcompwordmark is used in
> documents with "non-TeX fonts" if configure detects TeXLive16 or later.
> 
> Günter

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: Errors with vref on de/Additional.lyx with lualatex

[Kornel Benko]

Am Freitag, 21. Juli 2017 um 21:30:35, schrieb Guenter Milde 

> On 2017-07-19, Kornel Benko wrote:
> 
> 
> 
> > I did as you proposed (Dejavu font), but maybe you are unaware that
> > With TL15 everything compiles here
> 
> TL15 does not bind \textcompwordmark to the ZWNJ character but uses the
> default definition also for Unicode fonts (EC1 and EC2).
> 
> > With TL16 I get missing glyphs if font is Dejavu, but not with Free*
> > Mark, that changing 'Dejavu Sans Mono' to e.g. 'Dejavu Sans' cures the 
> > compilation though
> > With TL17 I get missing glyphs for any font. I have to replace the mono 
> > font to
> > get it to compile.
> 
> In TL16, the font encoding used by "fontspec" (Xe/LuaTeX with "system
> fonts") changed to TU: 
> 
> * In this encoding, \textcompwordmark is mapped to the ZWNJ character.
> 
> * Only some fonts actually contain this character.
> 
> * LyX turns the "missing character" WARNING into an ERROR
>   This is usually a good thing (because printing characters that are missing
>   represent data loss).
>   
>   For ZWNJ, it would be better if the warning was *not*
>   turned into an error, because the ligature break works also if the
>   character is missing -- no data loss, correct output!
> 
> > So, maybe better we could omit \textcompwordmark in mono fonts.
> 
> This would only cure the special case tested in the export tests. It will
> not help in documents with the default LM or custom fonts where the serif
> or non-serif fonts miss the ZWNJ. 
> It would also not help in documents using the "ligature-break" special-char.
> 

But that is not our problem IMHO. If someone uses that glyph with a font not 
having it,
it is not lyx's or latex problem.
But for mono-fonts, who may expect the glyph to exist?
WE output \textcompwordmark also for mono fonts, and this feels very wrong.

> The better cure is one of
> 
> a) an exception list for the "turn missing character warning to error" code,
> 
> b) preamble code reverting the binding of \textcompwordmark to ZWNJ (at
>least for the next years until "system fonts" can be safely expected to
>ship with the ZWNJ character).
>
> I prefer a), because the missing character still breaks the ligature (i.e.
> the output looks as expected).

Hm, yes, but that would also be a change in command line for lyx.
ATM, lyx is responsible to filter the latex log if instructed so.
# lyx --ignore-error-message missing_glyphs
With this method (ATM) _any_ missing glyph is ignored unfortunately.

> Günter
> 
Kornel

signature.asc
Description: This is a digitally signed message part.

Re: Errors with vref on de/Additional.lyx with lualatex

[Guenter Milde]

On 2017-07-20, Kornel Benko wrote:
> Am Mittwoch, 19. Juli 2017 um 22:23:32, schrieb Kornel Benko 
>> Am Mittwoch, 19. Juli 2017 um 19:48:49, schrieb Guenter Milde 
>> 
>> > On 2017-07-19, Kornel Benko wrote:
>> > 
>> > > [-- Type: text/plain, Encoding: 7bit --]
>> > 
>> > > Am Mittwoch, 19. Juli 2017 um 15:00:16, schrieb Kornel Benko 
>> > > 
>> > >> So, maybe better we could omit \textcompwordmark in mono fonts.
>> > 
>> > > This patch works for me. It uses vphantom{}, but only between '<<' and 
>> > > '>>'.
>> > 
>> > See my other post for alternatives solving the "missing character" error
>> > with Unicode fonts for \textcompwordmark.

>> I have read it. I understand the usage and I don’t insist on removing
>> textcompwordmark any longer.

>> I don't insist. I would insist on removing textcompwordmark in mono
>> fonts, but did not see an easy way.

> It turned out, it is easy.

But it is still wrong: 

* It is LyX policy to keep LaTeX output stable (unless it turns out to be
  wrong).

* \textcompwordmark is the correct LaTeX internal character
  representation (LICR) for u200C (zero width non joiner, ZWNJ).

  If some fonts are missing this character this is a bug in the fonts or
  in the code calling this missing character (here tu1enc.def in latex/base/).
  
* The workaround should fix the command, not use a different
  one. Power users are then still able to override our workaround with
  their own re-definition of \textcompwordmark.


* Actually, in the case of ZWNJ, just dropping the missing character results
  in correct output. 
  --> We don't even need to add a workaround, if we remove
the "warning to error conversion for missing characters" for this case.



> Output for desired ligatures with the attached patch is:
>   "\\vphantom{}" if mono mono-fonts
>   "\\textcompwordmark " else.

This will only help for documents using DejaVu.  OTOH, it is not required
for Free* and no help for LatinModern.



...

> diff --git a/lib/doc/de/Additional.lyx b/lib/doc/de/Additional.lyx
> index 5ccd07b..539aa6d 100644
> --- a/lib/doc/de/Additional.lyx
> +++ b/lib/doc/de/Additional.lyx
> @@ -41,9 +41,9 @@ shapepar
>  \language_package default
>  \inputencoding auto
>  \fontencoding global
> -\font_roman "lmodern" "default"
> -\font_sans "lmss" "default"
> -\font_typewriter "lmtt" "default"
> +\font_roman "lmodern" "DejaVu Serif [bitstream]"
> +\font_sans "lmss" "DejaVu Sans [bitstream]"
> +\font_typewriter "lmtt" "DejaVu Sans Mono [bitstream]"
>  \font_math "auto" "auto"
>  \font_default_family default
>  \use_non_tex_fonts false

This fixes the test case but not the problem.

The correct fix is either an exception to the "error-on-missing-chars"
feature, or providing the workaround (re-defining to the original default):

  % ZWNJ missing in many Unicode fonts:
  \DeclareTextCommand{\textcompwordmark}{\UnicodeEncodingName}{%
 \leavevmode\kern\z@}

in LaTeXFeatures.cpp and "require" it in case \textcompwordmark is used in
documents with "non-TeX fonts" if configure detects TeXLive16 or later.

Günter

Re: Errors with vref on de/Additional.lyx with lualatex

[Guenter Milde]

On 2017-07-19, Kornel Benko wrote:



> I did as you proposed (Dejavu font), but maybe you are unaware that
> With TL15 everything compiles here

TL15 does not bind \textcompwordmark to the ZWNJ character but uses the
default definition also for Unicode fonts (EC1 and EC2).

> With TL16 I get missing glyphs if font is Dejavu, but not with Free*
>   Mark, that changing 'Dejavu Sans Mono' to e.g. 'Dejavu Sans' cures the 
> compilation though
> With TL17 I get missing glyphs for any font. I have to replace the mono font 
> to
> get it to compile.

In TL16, the font encoding used by "fontspec" (Xe/LuaTeX with "system
fonts") changed to TU: 

* In this encoding, \textcompwordmark is mapped to the ZWNJ character.

* Only some fonts actually contain this character.

* LyX turns the "missing character" WARNING into an ERROR
  This is usually a good thing (because printing characters that are missing
  represent data loss).
  
  For ZWNJ, it would be better if the warning was *not*
  turned into an error, because the ligature break works also if the
  character is missing -- no data loss, correct output!

> So, maybe better we could omit \textcompwordmark in mono fonts.

This would only cure the special case tested in the export tests. It will
not help in documents with the default LM or custom fonts where the serif
or non-serif fonts miss the ZWNJ. 
It would also not help in documents using the "ligature-break" special-char.


The better cure is one of

a) an exception list for the "turn missing character warning to error" code,

b) preamble code reverting the binding of \textcompwordmark to ZWNJ (at
   least for the next years until "system fonts" can be safely expected to
   ship with the ZWNJ character).
   
I prefer a), because the missing character still breaks the ligature (i.e.
the output looks as expected).

Günter

Re: Errors with vref on de/Additional.lyx with lualatex

[Kornel Benko]

Am Mittwoch, 19. Juli 2017 um 22:23:32, schrieb Kornel Benko 
> Am Mittwoch, 19. Juli 2017 um 19:48:49, schrieb Guenter Milde 
> 
> > On 2017-07-19, Kornel Benko wrote:
> >
> > > [-- Type: text/plain, Encoding: 7bit --]
> >
> > > Am Mittwoch, 19. Juli 2017 um 15:00:16, schrieb Kornel Benko 
> > > 
> > >> So, maybe better we could omit \textcompwordmark in mono fonts.
> >
> > > This patch works for me. It uses vphantom{}, but only between '<<' and 
> > > '>>'.
> >
> > See my other post for alternatives solving the "missing character" error
> > with Unicode fonts for \textcompwordmark.
>
> I have read it. I understand the usage and I don’t insist on removing 
> textcompwordmark any longer.
>
> > If you insist on changing away from textcompwordmark, this should be limited
> > to TU, i.e. done in
> > bool Paragraph::Private::latexSpecialTU(char_type const c, otexstream & os,
> >
> >
> > Günter
>
> I don't insist. I would insist on removing textcompwordmark in mono fonts, 
> but did not see an easy way.

It turned out, it is easy. Output for desired ligatures with the attached patch 
is:
"\\vphantom{}" if mono mono-fonts
"\\textcompwordmark " else.

To disable ligature for '>>' and '<<' the output is "\\vphantom{}"

Korneldiff --git a/lib/doc/de/Additional.lyx b/lib/doc/de/Additional.lyx
index 5ccd07b..539aa6d 100644
--- a/lib/doc/de/Additional.lyx
+++ b/lib/doc/de/Additional.lyx
@@ -41,9 +41,9 @@ shapepar
 \language_package default
 \inputencoding auto
 \fontencoding global
-\font_roman "lmodern" "default"
-\font_sans "lmss" "default"
-\font_typewriter "lmtt" "default"
+\font_roman "lmodern" "DejaVu Serif [bitstream]"
+\font_sans "lmss" "DejaVu Sans [bitstream]"
+\font_typewriter "lmtt" "DejaVu Sans Mono [bitstream]"
 \font_math "auto" "auto"
 \font_default_family default
 \use_non_tex_fonts false
diff --git a/src/Paragraph.cpp b/src/Paragraph.cpp
index 73157e6..622477b 100644
--- a/src/Paragraph.cpp
+++ b/src/Paragraph.cpp
@@ -1378,8 +1378,8 @@ bool Paragraph::Private::latexSpecialT1(char_type const c, otexstream & os,
 		// but we should avoid ligatures
 		if (i + 1 >= int(text_.size()) || text_[i + 1] != c)
 			return true;
-		os << "\\textcompwordmark" << termcmd;
-		column += 19;
+		os << "\\vphantom{}";
+		column += 16;
 		return true;
 	case '|':
 		os.put(c);
diff --git a/src/insets/InsetSpecialChar.cpp b/src/insets/InsetSpecialChar.cpp
index bbf54cf..293541c 100644
--- a/src/insets/InsetSpecialChar.cpp
+++ b/src/insets/InsetSpecialChar.cpp
@@ -424,7 +424,15 @@ void InsetSpecialChar::latex(otexstream & os,
 		os << "\\LyXZeroWidthSpace" << termcmd;
 		break;
 	case LIGATURE_BREAK:
-		os << "\\textcompwordmark" << termcmd;
+		if (rp.local_font != 0) {
+			FontFamily ff = rp.local_font->fontInfo().family();
+			if (ff == TYPEWRITER_FAMILY)
+os << "\\vphantom{}";
+			else
+os << "\\textcompwordmark" << termcmd;
+		}
+		else
+			os << "\\textcompwordmark" << termcmd;
 		break;
 	case END_OF_SENTENCE:
 		os << "\\@.";


signature.asc
Description: This is a digitally signed message part.

Re: Errors with vref on de/Additional.lyx with lualatex

[Kornel Benko]

Am Mittwoch, 19. Juli 2017 um 19:48:49, schrieb Guenter Milde 

> On 2017-07-19, Kornel Benko wrote:
> 
> > [-- Type: text/plain, Encoding: 7bit --]
> 
> > Am Mittwoch, 19. Juli 2017 um 15:00:16, schrieb Kornel Benko 
> > 
> >> So, maybe better we could omit \textcompwordmark in mono fonts.
> 
> > This patch works for me. It uses vphantom{}, but only between '<<' and '>>'.
> 
> See my other post for alternatives solving the "missing character" error
> with Unicode fonts for \textcompwordmark.

I have read it. I understand the usage and I don’t insist on removing 
textcompwordmark any longer.

> If you insist on changing away from textcompwordmark, this should be limited
> to TU, i.e. done in
> bool Paragraph::Private::latexSpecialTU(char_type const c, otexstream & os,
> 
> 
> Günter

I don't insist. I would insist on removing textcompwordmark in mono fonts, but 
did not see an easy way.
Instead, if you look into the lines around src/Paragraph.cpp:1378 you see the 
only reason to use
textcompwordmark there is to break the ligatures for '>>' and '<<'. But that 
can be done by vphantom{} too.

Kornel


signature.asc
Description: This is a digitally signed message part.

Re: Errors with vref on de/Additional.lyx with lualatex

[Guenter Milde]

On 2017-07-19, Kornel Benko wrote:

> [-- Type: text/plain, Encoding: 7bit --]

> Am Mittwoch, 19. Juli 2017 um 15:00:16, schrieb Kornel Benko 
>> So, maybe better we could omit \textcompwordmark in mono fonts.

> This patch works for me. It uses vphantom{}, but only between '<<' and '>>'.

See my other post for alternatives solving the "missing character" error
with Unicode fonts for \textcompwordmark.

If you insist on changing away from textcompwordmark, this should be limited
to TU, i.e. done in
bool Paragraph::Private::latexSpecialTU(char_type const c, otexstream & os,


Günter

Re: Errors with vref on de/Additional.lyx with lualatex

[Kornel Benko]

Am Mittwoch, 19. Juli 2017 um 15:00:16, schrieb Kornel Benko 
> So, maybe better we could omit \textcompwordmark in mono fonts.

This patch works for me. It uses vphantom{}, but only between '<<' and '>>'.

Korneldiff --git a/src/Paragraph.cpp b/src/Paragraph.cpp
index 73157e6..622477b 100644
--- a/src/Paragraph.cpp
+++ b/src/Paragraph.cpp
@@ -1378,8 +1378,8 @@ bool Paragraph::Private::latexSpecialT1(char_type const c, otexstream & os,
 		// but we should avoid ligatures
 		if (i + 1 >= int(text_.size()) || text_[i + 1] != c)
 			return true;
-		os << "\\textcompwordmark" << termcmd;
-		column += 19;
+		os << "\\vphantom{}";
+		column += 16;
 		return true;
 	case '|':
 		os.put(c);


signature.asc
Description: This is a digitally signed message part.

Re: Errors with vref on de/Additional.lyx with lualatex

[Kornel Benko]

Am Mittwoch, 19. Juli 2017 um 10:56:21, schrieb Guenter Milde 

> On 2017-07-18, Kornel Benko wrote:
> 
> ...
> 
> > there is also error with the system font 'FreeSerif,FreeSans,FreeMono'
> > when exporting to PDF(luatex)
> 
> >  !Package varioref Error: \vref at page boundary 14-15 (may loop).
> 
> > See the varioref package documentation for explanation.
> > Type  H   for immediate help.
> >  ...  
> 
> > l.810 ...t\ \vref{Literaturverzeichnisse-mit-BibTeX}
> 
> > Please check the pages in question. You might need to replace the \vref
> > or \vpageref by a normal \(page)ref to stop LaTeX running forever.
> 
> > Changing the font to say Dejavu, the error disappears. Also inserting a line
> > for instance before paragraph 3.1.2 cures the compilation.
> 
> This is a rare running-condition that depends on the selected font (because
> this influences page-breaks).
> 
> > 1.) Should we do anything here? Or is the user on its own if he/she
> > gets such a message?
> 
> Generally, the user is on its own when seeing such a message - we cannot
> predict it from within LyX.
> 
> In the given case, it does not appear with the original fonts as defined
> in the documentation file, only when replacing them with FreeSerif
> (either in a script or in the document).
> 
> We could define DejaVu in the document (to solve the \textcompwordmark error)
> or try a different wording in the sentence containing the offending \vref.
> 
> Günter

I did as you proposed (Dejavu font), but maybe you are unaware that
With TL15 everything compiles here
With TL16 I get missing glyphs if font is Dejavu, but not with Free*
Mark, that changing 'Dejavu Sans Mono' to e.g. 'Dejavu Sans' cures the 
compilation though
With TL17 I get missing glyphs for any font. I have to replace the mono font to
get it to compile.
So, maybe better we could omit \textcompwordmark in mono fonts.

Kornel


signature.asc
Description: This is a digitally signed message part.

Re: Errors with vref on de/Additional.lyx with lualatex

[Guenter Milde]

On 2017-07-18, Kornel Benko wrote:

...

> there is also error with the system font 'FreeSerif,FreeSans,FreeMono'
> when exporting to PDF(luatex)

>  !Package varioref Error: \vref at page boundary 14-15 (may loop).

>   See the varioref package documentation for explanation.
>   Type  H   for immediate help.
>...  

>   l.810 ...t\ \vref{Literaturverzeichnisse-mit-BibTeX}

>   Please check the pages in question. You might need to replace the \vref
>   or \vpageref by a normal \(page)ref to stop LaTeX running forever.

> Changing the font to say Dejavu, the error disappears. Also inserting a line
> for instance before paragraph 3.1.2 cures the compilation.

This is a rare running-condition that depends on the selected font (because
this influences page-breaks).

> 1.) Should we do anything here? Or is the user on its own if he/she
> gets such a message?

Generally, the user is on its own when seeing such a message - we cannot
predict it from within LyX.

In the given case, it does not appear with the original fonts as defined
in the documentation file, only when replacing them with FreeSerif
(either in a script or in the document).

We could define DejaVu in the document (to solve the \textcompwordmark error)
or try a different wording in the sentence containing the offending \vref.

Günter

Re: Errors with vref on de/Additional.lyx with lualatex

[Guenter Milde]

On 2017-07-18, Kornel Benko wrote:

...

> 2.) Can we replace outputting \textcompwordmark with \vphantom{} in
> exported tex file?
> I could not see any difference in the pdflatex output. Also
> lualatex and xelatex displayed correctly.

However, lib/unicodesymbols says:

  Do only add commands that give correct output, no hacks that look "similar".

but \textcompwordmark is defined as the LaTeX equivalent to
200C ZERO WIDTH NON-JOINER (ZWNJ) in the LaTeX base:

  t1enc.dfu:261:\DeclareUnicodeCharacter{200C}{\textcompwordmark}

  utf8.def:247:\DeclareUnicodeCharacter{200C}{\textcompwordmark}


The macro itself is defined in the LaTeX base as

  
latex.ltx:2036:\DeclareTextCommandDefault{\textcompwordmark}{\leavevmode\kern\z@}

For T1-encoded fonts, it is mapped to character 23:

  latex.ltx:8275:\lccode 23 =23% textcompwordmark in T1
  t1enc.def:112:\DeclareTextSymbol{\textcompwordmark}{T1}{23}

Since the recent additon of the standard Unicode text font encoding "TU",
there is a conversion to ZWNJ if Unicode fonts are used with fontspec:

  tuenc.def:169:\DeclareTextSymbol{\textcompwordmark}
\UnicodeEncodingName{"200C}

The problem is, that this character is missing in the default (LatinModern
Unicode) font. It works fine with, e.g., DejaVu.


> Moreover, using
> \textcompwordmark with xelatex displays wrong.

I cannot reproduce.  In my tests, the output is correct: \textcompwordmark
suppresses the ff ligature without other side-effects in the PDF.

However, there is a difference if you copy-paste from the PDF output:

With evince, I get

* for pdflatex and T1-encoded lmodern fonts:

test with textcompwordmark: ff
test with vphantom: ff

  \textcompwordmark translates to the unprintable character u0017 because
  this is the position of the ZWNJ character in T1 encoding

  (This is a LaTeX or Evince bug.)

* for xetex or luatax with fontspec and LatinModern

test with textcompwordmark: ff
test with vphantom: ff

  Output and copy-paste OK despite the "missing character" warning.
  The invisible ZWNJ is simply dropped but prevents the ligature.

* for xetex with DejaVu:

test with textcompwordmark: f f
test with vphantom: ff

  The ZWNJ is converted to an ordinary space when pasting into LyX

* for luatex with DejaVu:

   test with textcompwordmark: f‌f
   test with vphantom: ff

  The ZWNJ is kept (and correctly suppresses the ligature in GUI and output)
  when pasting into LyX

Options:

a) Do not turn the "missing character" warning into an error for
   characters where this is no data loss, e.g. 200C (ZWNJ)

   + helps also, if there are literal ZWNJ characters in the source,
   - side-effects for copy-paste with pdflatex and xetex

b) Re-set \textcompwordmark to the Default in the preamble if TU is
   used:

 \DeclareTextCommand{\textcompwordmark}{TU}{\leavevmode\kern\z@}

As immediate action, maybe also

c) Invert the test(s) with a comment declaring the reason.

d) Use DejaVu or FreeSerif in the affected tests
   (preferably added to the documents instead of the script-based
   replacement).


Günter