Re: doc/UserGuide systemF tests failing

[Guenter Milde]

On 2017-01-11, Kornel Benko wrote:
> Am Montag, 9. Januar 2017 um 21:33:42, schrieb Günter Milde 
> 

>> The attached file works fine here with ucharclasses under XeTeX.

> Does not compile here because of missing font.

I installed this font via the Debian package "fonts-linex". It has a wide
range of characters, from the content of the UserGuide, only the "visible
space" is missing. Therefore I started combining fonts. 

With the preamble code, any other CJK font should do.


> BUT, selecting a Korean font like UnBatang works. Even with empty preamble.

Yes, with a font that contains all characters present in UserGuide.lyx, this
is to be expected. The problem is rather finding a good and free font with
all required characters. 

The font choice becomes a lot easier and wider by combining fonts for
different scripts. This way should be supported by LyX natively - I'll post
an enhancement request.

Günter

Re: doc/UserGuide systemF tests failing

[Kornel Benko]

Am Montag, 9. Januar 2017 um 21:33:42, schrieb Günter Milde 

> The attached file works fine here with ucharclasses under XeTeX.
>

Does not compile here because of missing font.

BUT, selecting a Korean font like UnBatang works. Even with empty preamble.

Kornel

signature.asc
Description: This is a digitally signed message part.


CJK-quotes.lyx
Description: application/lyx

Re: doc/UserGuide systemF tests failing

[Guenter Milde]

On 2017-01-11, Jürgen Spitzmüller wrote:

> [-- Type: text/plain, Encoding: quoted-printable --]

> 2017-01-11 17:12 GMT+01:00 Scott Kostyshak :

>> On Wed, Jan 11, 2017 at 10:27:34AM +0100, Kornel Benko wrote:

>> > Thanks Jürgen.
>> > Committed my patch too.

>> Thanks.

>> > Now we have unwanted messages e.g. for the test
>> "export/doc/de/UserGuide_pdf4_texF".
>> > I think, this is unrelated to the last patches.

>> I did a bisect based on the pass/failure of "de/Additional_pdf4_texF"
>> and it lead to b487e2b0. If you don't have time to take a look, I can
>> take a look either tonight or tomorrow and make sure the bisect was
>> correct.


> This is related to http://www.lyx.org/trac/ticket/9740

Yes and no:

The warnings (and a subsequent error) are there, since
"pfd4_texF" (XeTeX and 8-bit TeX-Fonts) uses the ASCII input
encoding (see the ticket for the reasons).

Therefore, "invertedTests" contains the pattern

# Non-ASCII char in verbatim environment.
# Fails with ASCII (and hence also with XeTeX).
export/doc/(|de/|es/|fr/)UserGuide_pdf4_texF


However, the same warnings and error occure with PDFLaTeX when
setting Document>Settings>Language>Encoding to ASCII.

The real problem is bad support for ASCII inputenc. 
(But our tests don't test different encodings.)


We could drop support for XeTeX with TeX fonts (except the user
explicitely sets Language>Encoding to ASCII) and/or file a bug for better
Index etc. sorting with ASCII inputenc.


Günter

Re: doc/UserGuide systemF tests failing

[Jürgen Spitzmüller]

2017-01-11 17:12 GMT+01:00 Scott Kostyshak :

> On Wed, Jan 11, 2017 at 10:27:34AM +0100, Kornel Benko wrote:
>
> > Thanks Jürgen.
> > Committed my patch too.
>
> Thanks.
>
> > Now we have unwanted messages e.g. for the test
> "export/doc/de/UserGuide_pdf4_texF".
> > I think, this is unrelated to the last patches.
>
> I did a bisect based on the pass/failure of "de/Additional_pdf4_texF"
> and it lead to b487e2b0. If you don't have time to take a look, I can
> take a look either tonight or tomorrow and make sure the bisect was
> correct.
>

This is related to http://www.lyx.org/trac/ticket/9740

Jürgen


>
> Scott
>

Re: doc/UserGuide systemF tests failing

[Scott Kostyshak]

On Wed, Jan 11, 2017 at 10:27:34AM +0100, Kornel Benko wrote:

> Thanks Jürgen.
> Committed my patch too.

Thanks.

> Now we have unwanted messages e.g. for the test 
> "export/doc/de/UserGuide_pdf4_texF".
> I think, this is unrelated to the last patches.

I did a bisect based on the pass/failure of "de/Additional_pdf4_texF"
and it lead to b487e2b0. If you don't have time to take a look, I can
take a look either tonight or tomorrow and make sure the bisect was
correct.

Scott


signature.asc
Description: PGP signature

Re: doc/UserGuide systemF tests failing

[Kornel Benko]

Am Mittwoch, 11. Januar 2017 um 09:22:21, schrieb Jürgen Spitzmüller 

> Am Dienstag, den 10.01.2017, 20:40 + schrieb Guenter Milde:
> > As Kornel provided a use case and I can see other possible uses for
> > command-line export of documents, I withdraw my objection.
> > 
> > Please commit.
> 
> OK. Done.
> 
> Jürgen
> 
> > 
> > Günter

Thanks Jürgen.
Committed my patch too.

Now we have unwanted messages e.g. for the test 
"export/doc/de/UserGuide_pdf4_texF".
I think, this is unrelated to the last patches.

LyX's automatic index sorting algorithm faced
problems with the entry 'Suchen & Ersetzen'.
Please specify the sorting of this entry manually, as
explained in the User Guide.
Warning: Index sorting failed

LyX's automatic index sorting algorithm faced
problems with the entry 'Eingabevervollständigung'.
Please specify the sorting of this entry manually, as
explained in the User Guide.
Warning: Index sorting failed

...
-- Exporting de/UserGuide.lyx to pdf4
CMake Error at /usr2/src/lyx/lyx-git/development/autotests/export.cmake:205 
(message):
  Export failed


Same with "export/doc/es/UserGuide_pdf4_texF"

Warning: Index sorting failed

LyX's automatic index sorting algorithm faced
problems with the entry 'Reconfiguración de LyX'.
Please specify the sorting of this entry manually, as
explained in the User Guide.
Warning: Index sorting failed

LyX's automatic index sorting algorithm faced
problems with the entry 'Reconfiguración de LyX'.
Please specify the sorting of this entry manually, as
explained in the User Guide.
Warning: Index sorting failed

...
-- Exporting es/UserGuide.lyx to pdf4
CMake Error at /usr2/src/lyx/lyx-git/development/autotests/export.cmake:205 
(message):
  Export failed

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: doc/UserGuide systemF tests failing

[Jürgen Spitzmüller]

Am Dienstag, den 10.01.2017, 20:40 + schrieb Guenter Milde:
> As Kornel provided a use case and I can see other possible uses for
> command-line export of documents, I withdraw my objection.
> 
> Please commit.

OK. Done.

Jürgen

> 
> Günter
> 

signature.asc
Description: This is a digitally signed message part

Re: doc/UserGuide systemF tests failing

[Scott Kostyshak]

On Tue, Jan 10, 2017 at 08:40:00PM +, Guenter Milde wrote:
> On 2017-01-10, Jürgen Spitzmüller wrote:
> 
> > [-- Type: text/plain, Encoding: quoted-printable --]
> 
> > Am Dienstag, den 10.01.2017, 14:10 +0100 schrieb Kornel Benko:
> >> Jürgen, I have a patch ready to push which builds on top of your
> >> patch.
> >> Please commit.
> 
> > I don't have the impression there is general consensus on this.
> 
> As Kornel provided a use case and I can see other possible uses for
> command-line export of documents, I withdraw my objection.
> 
> Please commit.

+1

Thanks Jürgen and Kornel for the work.

Scott


signature.asc
Description: PGP signature

Re: doc/UserGuide systemF tests failing

[Scott Kostyshak]

On Mon, Jan 09, 2017 at 09:33:42PM +0100, Günter Milde wrote:
> On  9.01.17, Scott Kostyshak wrote:
> > On Mon, Jan 09, 2017 at 03:23:47PM +0100, Jürgen Spitzmüller wrote:
> > > Am Montag, den 09.01.2017, 09:12 -0500 schrieb Scott Kostyshak:
> > > > On Mon, Jan 09, 2017 at 10:41:41AM +0100, Jürgen Spitzmüller wrote:
> > > > > Am Sonntag, den 08.01.2017, 09:22 -0500 schrieb Scott Kostyshak:
> 
> > > > > Please try the attached patch. The lyx option 
> > > > > --ignore-error-message missing_glyphs
> > > > > does what you need.
> > > > 
> > > > The patch looks good.
> > > 
> > > So shall I commit?
> 
> > Let's first ask Kornel and Günter. Do you think we should turn this
> > option on by default for the tests?
> 
> No. The "missing glyphs" warning-turned-error helped to find a whole range
> of LyX bugs and LaTeX problems. We would throw away more than what we gain.
> Remember that missing glyphs means data loss.

I see now that this was a poor suggestion on my part.

> If Kornel thinks this is sensible and there is a straightforward way to turn
> this option on for selected tests, so be it.

OK it seems this is the case.

> Rather than ignoring the missing glyphs, I propose to solve the issue of
> "font combination" for Unicode fonts.
> 
> For CJK quotes, there are several options:
> 
> xeCJK: (only XeTeX, no LuaTeX)
> 
>   \usepackage{xeCJK}
>   \setCJKmainfont{BABEL Unicode}
>   
> swallows whitespace after closing quote. Try to override:
> 
>   % don't ignore spaces: 
>   \let\xeCJK@ignorespaces\relax
>   
> does not help for space after closing quote
> 
> 
> luatexja:   did not test
> 
>   %\usepackage{luatexja-fontspec}
>   %\setmainjfont{BABEL Unicode}
> 
> 
> 
> ucharclasses (only XeTeX, no LuaTeX):
> 
>   \usepackage[CJK]{ucharclasses}
>   \newfontfamily\cjkfont{BABEL Unicode}
>   \newfontfamily\defaultfont{Latin Modern Roman}
>   \setTransitionsForCJK{\cjkfont{}}{\defaultfont{}}
> 
> 
> The attached file works fine here with ucharclasses under XeTeX.
> 
> We can either add "cheating" preamble code with
> \@ifpackageloaded{fontspec}{
>   ...
> }{}
> 
> or, with an enhancement ticket for a "font combi GUI", we can invert
> 
>   export/doc/(|de/|es/|fr/)UserGuide_.*_systemF
>   
> in the 
> 
> Sublabel: lyxbugs
> # LyX bugs with a Trac number.
> 
> section and add a copy without CJK quotes to the autotests/export/latex/
> samples.

I don't know much about this stuff so I don't have a comment on the
above.

Scott


signature.asc
Description: PGP signature

Re: doc/UserGuide systemF tests failing

[Guenter Milde]

On 2017-01-10, Jürgen Spitzmüller wrote:

> [-- Type: text/plain, Encoding: quoted-printable --]

> Am Dienstag, den 10.01.2017, 14:10 +0100 schrieb Kornel Benko:
>> Jürgen, I have a patch ready to push which builds on top of your
>> patch.
>> Please commit.

> I don't have the impression there is general consensus on this.

As Kornel provided a use case and I can see other possible uses for
command-line export of documents, I withdraw my objection.

Please commit.

Günter

Re: doc/UserGuide systemF tests failing

[Jürgen Spitzmüller]

Am Dienstag, den 10.01.2017, 14:10 +0100 schrieb Kornel Benko:
> Jürgen, I have a patch ready to push which builds on top of your
> patch.
> Please commit.

I don't have the impression there is general consensus on this.

Jürgen

> 
>   Kornel

signature.asc
Description: This is a digitally signed message part

Re: doc/UserGuide systemF tests failing

[Kornel Benko]

Am Montag, 9. Januar 2017 um 16:33:45, schrieb Jürgen Spitzmüller 

> Am Montag, den 09.01.2017, 10:15 -0500 schrieb Scott Kostyshak:
> > Let's first ask Kornel and Günter. Do you think we should turn this
> > option on by default for the tests? I don't know too much about this,
> > but it makes sense to me because it will help make the tests less
> > sensitive to font choice.
> 
> Actually, I thought you would just turn it on for he UserGuide.
> 
> Jürgen
> 
> > 
> > If we don't turn it on by default for the ctests, it might still make
> > sense to commit the patch for other use cases.
> > 
> > Scott

Jürgen, I have a patch ready to push which builds on top of your patch.
Please commit.

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: doc/UserGuide systemF tests failing

[Günter Milde]

On  9.01.17, Scott Kostyshak wrote:
> On Mon, Jan 09, 2017 at 03:23:47PM +0100, Jürgen Spitzmüller wrote:
> > Am Montag, den 09.01.2017, 09:12 -0500 schrieb Scott Kostyshak:
> > > On Mon, Jan 09, 2017 at 10:41:41AM +0100, Jürgen Spitzmüller wrote:
> > > > Am Sonntag, den 08.01.2017, 09:22 -0500 schrieb Scott Kostyshak:

> > > > Please try the attached patch. The lyx option 
> > > > --ignore-error-message missing_glyphs
> > > > does what you need.
> > > 
> > > The patch looks good.
> > 
> > So shall I commit?

> Let's first ask Kornel and Günter. Do you think we should turn this
> option on by default for the tests?

No. The "missing glyphs" warning-turned-error helped to find a whole range
of LyX bugs and LaTeX problems. We would throw away more than what we gain.
Remember that missing glyphs means data loss.

If Kornel thinks this is sensible and there is a straightforward way to turn
this option on for selected tests, so be it.

> If we don't turn it on by default for the ctests, it might still make
> sense to commit the patch for other use cases.

I am against an additional option without a convincing use case.

**

Rather than ignoring the missing glyphs, I propose to solve the issue of
"font combination" for Unicode fonts.

For CJK quotes, there are several options:

xeCJK: (only XeTeX, no LuaTeX)

  \usepackage{xeCJK}
  \setCJKmainfont{BABEL Unicode}
  
swallows whitespace after closing quote. Try to override:

  % don't ignore spaces: 
  \let\xeCJK@ignorespaces\relax
  
does not help for space after closing quote


luatexja:   did not test

  %\usepackage{luatexja-fontspec}
  %\setmainjfont{BABEL Unicode}



ucharclasses (only XeTeX, no LuaTeX):

  \usepackage[CJK]{ucharclasses}
  \newfontfamily\cjkfont{BABEL Unicode}
  \newfontfamily\defaultfont{Latin Modern Roman}
  \setTransitionsForCJK{\cjkfont{}}{\defaultfont{}}


The attached file works fine here with ucharclasses under XeTeX.

We can either add "cheating" preamble code with
\@ifpackageloaded{fontspec}{
  ...
}{}

or, with an enhancement ticket for a "font combi GUI", we can invert

  export/doc/(|de/|es/|fr/)UserGuide_.*_systemF
  
in the 

Sublabel: lyxbugs
# LyX bugs with a Trac number.

section and add a copy without CJK quotes to the autotests/export/latex/
samples.


Günter


CJK-quotes.lyx
Description: application/lyx

Re: doc/UserGuide systemF tests failing

[Kornel Benko]

Am Montag, 9. Januar 2017 um 17:07:01, schrieb Kornel Benko 
> Am Montag, 9. Januar 2017 um 10:55:06, schrieb Scott Kostyshak 
> 
> > On Mon, Jan 09, 2017 at 04:50:55PM +0100, Kornel Benko wrote:
> >
> > > I'd prefer to make the behaviour be document specific. Ignoring errors in 
> > > tests
> > > does not feel right if used in general.
> >
> > Sounds good to me. Seems like you and Jürgen agree on this. Kornel,
> > would it be difficult to incorporate this into our ctest framework in a
> > document-specific way? Would we just use a special label? Do you mind
> > doing the work?
>
> I will look into this.
>

Something like attached should work. Contents of the new file 
"ignoreMissingGlyphTests" is
"export/doc/UserGuide_(dvi3|pdf4|pdf5)_systemF"

Kornel
diff --git a/development/autotests/ExportTests.cmake b/development/autotests/ExportTests.cmake
index 7184994..837eec5 100644
--- a/development/autotests/ExportTests.cmake
+++ b/development/autotests/ExportTests.cmake
@@ -186,7 +186,7 @@ macro(maketestname testname inverted listinverted listignored listunreliable lis
   endif()
 endmacro()

-macro(loadTestList filename resList depth)
+macro(loadTestList filename resList depth splitlangs)
   # Create list of strings from a file without comments
   # ENCODING parameter is a new feature in cmake 3.1
   initLangVars(${resList})
@@ -218,8 +218,12 @@ macro(loadTestList filename resList depth)
 list(REMOVE_DUPLICATES mylabels)
 set(sublabel ${_newl})
   else()
-string(REGEX REPLACE "(\\/|\\||\\(|\\))" "  " _vxx ${_newl})
-string(REGEX MATCHALL " ([a-z][a-z](_[A-Z][A-Z])?) " _vx ${_vxx})
+	if (splitlangs)
+	  string(REGEX REPLACE "(\\/|\\||\\(|\\))" "  " _vxx ${_newl})
+	  string(REGEX MATCHALL " ([a-z][a-z](_[A-Z][A-Z])?) " _vx ${_vxx})
+	else()
+	  set(_vx OFF)
+	endif()
 if(_vx)
   foreach(_v ${_vx})
 string(REGEX REPLACE " " "" _v ${_v})
@@ -276,10 +280,11 @@ assignLabelDepth(1 "unreliable" "inverted")
 assignLabelDepth(2 "suspended")
 assignLabelDepth(-1 "examples" "manuals" "mathmacros" "templates" "autotests")

-loadTestList(invertedTests invertedTests 7)
-loadTestList(ignoredTests ignoredTests 0)
-loadTestList(suspendedTests suspendedTests 6)
-loadTestList(unreliableTests unreliableTests 5)
+loadTestList(invertedTests invertedTests 7 ON)
+loadTestList(ignoredTests ignoredTests 0 ON)
+loadTestList(suspendedTests suspendedTests 6 ON)
+loadTestList(unreliableTests unreliableTests 5 ON)
+loadTestList(ignoreMissingGlyphTests ignoreMissingGlyphTests 8 OFF)

 foreach(libsubfolderx autotests/export lib/doc lib/examples lib/templates autotests/mathmacros)
   set(testlabel "export")
@@ -417,6 +422,13 @@ foreach(libsubfolderx autotests/export lib/doc lib/examples lib/templates autote
   set(mytestlabel ${testlabel})
   maketestname(TestName inverted invertedTests ignoredTests unreliableTests mytestlabel)
   if(TestName)
+	set(missingLabels )	# will be ignored
+	findexpr(mfound TestName ignoreMissingGlyphTests missingLabels)
+	if (mfound)
+	  set(_ignoreMissingGlyphsError ON)
+	else()
+	  set(_ignoreMissingGlyphsError OFF)
+	endif()
 add_test(NAME ${TestName}
   WORKING_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/${LYX_HOME}"
   COMMAND ${CMAKE_COMMAND} -DLYX_ROOT=${LIBSUB_SRC_DIR}
@@ -429,6 +441,7 @@ foreach(libsubfolderx autotests/export lib/doc lib/examples lib/templates autote
   -Dfile=${f}
   -Dinverted=${inverted}
   -DTOP_SRC_DIR=${TOP_SRC_DIR}
+	  -DIgnoreMissingGlyphsError=${_ignoreMissingGlyphsError}
   -DPERL_EXECUTABLE=${PERL_EXECUTABLE}
   -DXMLLINT_EXECUTABLE=${XMLLINT_EXECUTABLE}
   -DENCODING=${_enc2}
diff --git a/development/autotests/check_load.cmake b/development/autotests/check_load.cmake
index 0a0d351..860f1ff 100644
--- a/development/autotests/check_load.cmake
+++ b/development/autotests/check_load.cmake
@@ -28,7 +28,7 @@ execute_process(
   ERROR_VARIABLE lyxerr)

 message(STATUS "Error code of lyx = ${_err}")
-
+message(STATUS "lyxerr=${lyxerr}")
 #
 # Ignore messages between "reconfiguring user directory" and "LyX: Done!"
 # (Reconfigure-messages are not symptom of an error)
diff --git a/development/autotests/export.cmake b/development/autotests/export.cmake
index 08fd86e..74cf9d6 100755
--- a/development/autotests/export.cmake
+++ b/development/autotests/export.cmake
@@ -24,6 +24,7 @@
 #   -Dfile=xxx \
 #   -Dinverted=[01] \
 #   -DTOP_SRC_DIR=${TOP_SRC_DIR} \
+#   -DIgnoreMissingGlyphsError=(ON/OFF) \
 #   -DPERL_EXECUTABLE=${PERL_EXECUTABLE} \
 #   -DXMLLINT_EXECUTABLE=${XMLLINT_EXECUTABLE} \
 #   -DENCODING=xxx \
@@ -143,6 +144,9 @@ else()
   else()
 set(LatexDebugParam)
   endif()
+  if(IgnoreMissingGlyphsError)
+list(APPEND LatexDebugParam --ignore-error-message missing_glyphs)
+  

Re: doc/UserGuide systemF tests failing

[Kornel Benko]

Am Montag, 9. Januar 2017 um 10:55:06, schrieb Scott Kostyshak 

> On Mon, Jan 09, 2017 at 04:50:55PM +0100, Kornel Benko wrote:
> 
> > I'd prefer to make the behaviour be document specific. Ignoring errors in 
> > tests
> > does not feel right if used in general.
> 
> Sounds good to me. Seems like you and Jürgen agree on this. Kornel,
> would it be difficult to incorporate this into our ctest framework in a
> document-specific way? Would we just use a special label? Do you mind
> doing the work?

I will look into this.

> Scott

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: doc/UserGuide systemF tests failing

[Scott Kostyshak]

On Mon, Jan 09, 2017 at 04:50:55PM +0100, Kornel Benko wrote:

> I'd prefer to make the behaviour be document specific. Ignoring errors in 
> tests
> does not feel right if used in general.

Sounds good to me. Seems like you and Jürgen agree on this. Kornel,
would it be difficult to incorporate this into our ctest framework in a
document-specific way? Would we just use a special label? Do you mind
doing the work?

Scott


signature.asc
Description: PGP signature

Re: doc/UserGuide systemF tests failing

[Kornel Benko]

Am Montag, 9. Januar 2017 um 10:15:56, schrieb Scott Kostyshak 

> On Mon, Jan 09, 2017 at 03:23:47PM +0100, Jürgen Spitzmüller wrote:
> > Am Montag, den 09.01.2017, 09:12 -0500 schrieb Scott Kostyshak:
> > > On Mon, Jan 09, 2017 at 10:41:41AM +0100, Jürgen Spitzmüller wrote:
> > > > Am Sonntag, den 08.01.2017, 09:22 -0500 schrieb Scott Kostyshak:
> > > > > > Ignoring the
> > > > > > missing glyphs warning strikes me a better alternative.
> > > > > 
> > > > > Perhaps. Does anyone volunteer to do this work?
> > > > 
> > > > Please try the attached patch. The lyx option 
> > > > --ignore-error-message missing_glyphs
> > > > does what you need.
> > > 
> > > The patch looks good.
> > 
> > So shall I commit?
> 
> Let's first ask Kornel and Günter. Do you think we should turn this
> option on by default for the tests? I don't know too much about this,
> but it makes sense to me because it will help make the tests less
> sensitive to font choice.

I'd prefer to make the behaviour be document specific. Ignoring errors in tests
does not feel right if used in general.

> If we don't turn it on by default for the ctests, it might still make
> sense to commit the patch for other use cases.

Here I am not so sure. The user already has the possibility to use 'Show Output 
Anyway'.

> Scott

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: doc/UserGuide systemF tests failing

[Jürgen Spitzmüller]

Am Montag, den 09.01.2017, 10:15 -0500 schrieb Scott Kostyshak:
> Let's first ask Kornel and Günter. Do you think we should turn this
> option on by default for the tests? I don't know too much about this,
> but it makes sense to me because it will help make the tests less
> sensitive to font choice.

Actually, I thought you would just turn it on for he UserGuide.

Jürgen

> 
> If we don't turn it on by default for the ctests, it might still make
> sense to commit the patch for other use cases.
> 
> Scott

signature.asc
Description: This is a digitally signed message part

Re: doc/UserGuide systemF tests failing

[Scott Kostyshak]

On Mon, Jan 09, 2017 at 03:23:47PM +0100, Jürgen Spitzmüller wrote:
> Am Montag, den 09.01.2017, 09:12 -0500 schrieb Scott Kostyshak:
> > On Mon, Jan 09, 2017 at 10:41:41AM +0100, Jürgen Spitzmüller wrote:
> > > Am Sonntag, den 08.01.2017, 09:22 -0500 schrieb Scott Kostyshak:
> > > > > Ignoring the
> > > > > missing glyphs warning strikes me a better alternative.
> > > > 
> > > > Perhaps. Does anyone volunteer to do this work?
> > > 
> > > Please try the attached patch. The lyx option 
> > > --ignore-error-message missing_glyphs
> > > does what you need.
> > 
> > The patch looks good.
> 
> So shall I commit?

Let's first ask Kornel and Günter. Do you think we should turn this
option on by default for the tests? I don't know too much about this,
but it makes sense to me because it will help make the tests less
sensitive to font choice.

If we don't turn it on by default for the ctests, it might still make
sense to commit the patch for other use cases.

Scott


signature.asc
Description: PGP signature

Re: doc/UserGuide systemF tests failing

[Jürgen Spitzmüller]

Am Montag, den 09.01.2017, 09:12 -0500 schrieb Scott Kostyshak:
> On Mon, Jan 09, 2017 at 10:41:41AM +0100, Jürgen Spitzmüller wrote:
> > Am Sonntag, den 08.01.2017, 09:22 -0500 schrieb Scott Kostyshak:
> > > > Ignoring the
> > > > missing glyphs warning strikes me a better alternative.
> > > 
> > > Perhaps. Does anyone volunteer to do this work?
> > 
> > Please try the attached patch. The lyx option 
> > --ignore-error-message missing_glyphs
> > does what you need.
> 
> The patch looks good.

So shall I commit?

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: doc/UserGuide systemF tests failing

[Scott Kostyshak]

On Mon, Jan 09, 2017 at 10:41:41AM +0100, Jürgen Spitzmüller wrote:
> Am Sonntag, den 08.01.2017, 09:22 -0500 schrieb Scott Kostyshak:
> > > Ignoring the
> > > missing glyphs warning strikes me a better alternative.
> > 
> > Perhaps. Does anyone volunteer to do this work?
> 
> Please try the attached patch. The lyx option 
> --ignore-error-message missing_glyphs
> does what you need.

The patch looks good.

> --ignore-error-message can be expanded to other messages later, if this
>  is desired.

OK good.

> And the option is sufficiently painful to write to prevent people using
> it without thinking about what they are doing ;-)

Agreed.

Thanks,

Scott


signature.asc
Description: PGP signature

Re: doc/UserGuide systemF tests failing

[Jürgen Spitzmüller]

Am Sonntag, den 08.01.2017, 09:22 -0500 schrieb Scott Kostyshak:
> > Ignoring the
> > missing glyphs warning strikes me a better alternative.
> 
> Perhaps. Does anyone volunteer to do this work?

Please try the attached patch. The lyx option 
--ignore-error-message missing_glyphs
does what you need.

--ignore-error-message can be expanded to other messages later, if this
 is desired.

And the option is sufficiently painful to write to prevent people using
it without thinking about what they are doing ;-)

Jürgendiff --git a/src/LaTeX.cpp b/src/LaTeX.cpp
index c4f987014d..9faf20102e 100644
--- a/src/LaTeX.cpp
+++ b/src/LaTeX.cpp
@@ -18,6 +18,7 @@
 #include "BufferList.h"
 #include "LaTeX.h"
 #include "LyXRC.h"
+#include "LyX.h"
 #include "DepTable.h"
 
 #include "support/debug.h"
@@ -899,8 +900,9 @@ int LaTeX::scanLogFile(TeXErrors & terr)
 		 from_local8bit("pdfTeX Error"),
 		 from_local8bit(token),
 		 child_name);
-			} else if (prefixIs(token, "Missing character: There is no ")
-	   && !contains(token, "nullfont")) {
+			} else if (!ignore_missing_glyphs
+   && prefixIs(token, "Missing character: There is no ")
+   && !contains(token, "nullfont")) {
 // Warning about missing glyph in selected font
 // may be dataloss (bug 9610)
 // but can be ignored for 'nullfont' (bug 10394).
diff --git a/src/LyX.cpp b/src/LyX.cpp
index b10b60b122..e10fbebcb1 100644
--- a/src/LyX.cpp
+++ b/src/LyX.cpp
@@ -98,6 +98,13 @@ bool use_gui = true;
 bool verbose = false;
 
 
+// Do not treat the "missing glyphs" warning of fontspec as an error message.
+// The default is false and can be changed with the option
+// --ignore-error-message missing_glyphs
+// This is used in automated testing.
+bool ignore_missing_glyphs = false;
+
+
 // We default to open documents in an already running instance, provided that
 // the lyxpipe has been setup. This can be overridden either on the command
 // line or through preference settings.
@@ -1309,6 +1316,16 @@ int parse_verbose(string const &, string const &, string &)
 }
 
 
+int parse_ignore_error_message(string const & arg1, string const &, string &)
+{
+	if (arg1 == "missing_glyphs") {
+		ignore_missing_glyphs = true;
+		return 1;
+	}
+	return 0;
+}
+
+
 int parse_force(string const & arg, string const &, string &)
 {
 	if (arg == "all") {
@@ -1358,6 +1375,7 @@ void LyX::easyParse(int & argc, char * argv[])
 	cmdmap["--remote"] = parse_remote;
 	cmdmap["-v"] = parse_verbose;
 	cmdmap["--verbose"] = parse_verbose;
+	cmdmap["--ignore-error-message"] = parse_ignore_error_message;
 
 	for (int i = 1; i < argc; ++i) {
 		map::const_iterator it
diff --git a/src/LyX.h b/src/LyX.h
index 4b9c201e25..11c67a3118 100644
--- a/src/LyX.h
+++ b/src/LyX.h
@@ -52,6 +52,7 @@ enum OverwriteFiles {
 
 extern bool use_gui;
 extern bool verbose;
+extern bool ignore_missing_glyphs;
 extern RunMode run_mode;
 extern OverwriteFiles force_overwrite;
 


signature.asc
Description: This is a digitally signed message part

Re: doc/UserGuide systemF tests failing

[Jürgen Spitzmüller]

Am Sonntag, den 08.01.2017, 09:22 -0500 schrieb Scott Kostyshak:
> > I doubt you will find a free suitable font that is expected to be
> > installed on every user's box so that "a user just clicking "'use
> > non-
> > TeX fonts' and compiling will get the correct result."
> 
> Until now we pretty much had such a font.

Yes. But until now, we pretty much ignored the non-Western world (in
the English UG).

> In the meantime, tests are failing. I will plan to copy the old User
> Guide to a separate test, invert the tests of the current User Guide
> for
> system fonts, and open a bug report that corresponds to potential
> improvements discussed here.

OK.

Jürgen

> 
> Scott

signature.asc
Description: This is a digitally signed message part

Re: doc/UserGuide systemF tests failing

[Scott Kostyshak]

On Wed, Dec 28, 2016 at 01:08:44PM +0100, Jürgen Spitzmüller wrote:
> Am Mittwoch, den 28.12.2016, 11:41 + schrieb Guenter Milde:
> > I see. But as the tests should correspond to "normal" LyX behaviour,
> > we
> > should rather
> > 
> > a) fix the document: we can set a custom non-tex font that contains
> > the
> >    missing characters. This way a user just clicking "use non-TeX
> > fonts" and
> >    compiling will get the correct result.
> >    There are no side-effects with the default (TeX-fonts) setting.
> >    
> >    If there is a free suitable font, this is my preference.
> 
> I doubt you will find a free suitable font that is expected to be
> installed on every user's box so that "a user just clicking "'use non-
> TeX fonts' and compiling will get the correct result."

Until now we pretty much had such a font.

> > b) invert the test: as the document is not required to work with non-
> > TeX
> >    fonts but only reused for tests of all export formats, the failing
> > test
> >    does not mean there is a problem in LyX and can be ignored.
> 
> However, the UG is a quite good test file. It would be a pity to take
> it off the set just because the glyphs are missing.

I agree it is unfortunate.

> Ignoring the
> missing glyphs warning strikes me a better alternative.

Perhaps. Does anyone volunteer to do this work?

> >    -1 the document can no longer be used to test export with Unicode
> > fonts.
> >    
> >    If there is no suitable Unicode font, this is the correct way.
> >    For test with Unicode fonts, we could add a modified copy to the
> >    "dedicated test samples folder" autotests/export/latex/.
> 
> Then you would need to maintain that copy and update it every time the
> UG is updated. I doubt this will work.

For testing the full User Guide, yes. But for testing that the "old"
user guide works (which I think is still very useful), we do not have to
maintain anything.

In the meantime, tests are failing. I will plan to copy the old User
Guide to a separate test, invert the tests of the current User Guide for
system fonts, and open a bug report that corresponds to potential
improvements discussed here.

Scott


signature.asc
Description: PGP signature

Re: doc/UserGuide systemF tests failing

[Jürgen Spitzmüller]

Am Mittwoch, den 28.12.2016, 11:41 + schrieb Guenter Milde:
> I see. But as the tests should correspond to "normal" LyX behaviour,
> we
> should rather
> 
> a) fix the document: we can set a custom non-tex font that contains
> the
>    missing characters. This way a user just clicking "use non-TeX
> fonts" and
>    compiling will get the correct result.
>    There are no side-effects with the default (TeX-fonts) setting.
>    
>    If there is a free suitable font, this is my preference.

I doubt you will find a free suitable font that is expected to be
installed on every user's box so that "a user just clicking "'use non-
TeX fonts' and compiling will get the correct result."

But, after all, this is not something the UG is designed for (and
neither the CJK quotation marks).

> b) invert the test: as the document is not required to work with non-
> TeX
>    fonts but only reused for tests of all export formats, the failing
> test
>    does not mean there is a problem in LyX and can be ignored.

However, the UG is a quite good test file. It would be a pity to take
it off the set just because the glyphs are missing. Ignoring the
missing glyphs warning strikes me a better alternative.

>    
>    -1 the document can no longer be used to test export with Unicode
> fonts.
>    
>    If there is no suitable Unicode font, this is the correct way.
>    For test with Unicode fonts, we could add a modified copy to the
>    "dedicated test samples folder" autotests/export/latex/.

Then you would need to maintain that copy and update it every time the
UG is updated. I doubt this will work.

Jürgen

>    
> Günter   
> 

signature.asc
Description: This is a digitally signed message part

Re: doc/UserGuide systemF tests failing

[Guenter Milde]

On 2016-12-28, Jürgen Spitzmüller wrote:

> [-- Type: text/plain, Encoding: quoted-printable --]

> Am Mittwoch, den 28.12.2016, 05:20 -0500 schrieb Scott Kostyshak:
>> > Note, though, that the missing glyphs message is a warning, not an
>> > error. We treat it as an error in the GUI, for good reasons (since
>> it
>> > severely can impact the output), but it does not stop LaTeX from
>> > processing.

>> I do remember this. But since the default is to treat it as an error,
>> I'm not sure we should treat it differently from any LaTeX error.

> Since it does not point to a LaTeX error, but to the fact that the
> selected font is not suitable for the content. Which is probably not
> something that is of interest for the tests (as opposed to LaTeX
> errors).

I see. But as the tests should correspond to "normal" LyX behaviour, we
should rather

a) fix the document: we can set a custom non-tex font that contains the
   missing characters. This way a user just clicking "use non-TeX fonts" and
   compiling will get the correct result.
   There are no side-effects with the default (TeX-fonts) setting.
   
   If there is a free suitable font, this is my preference.

b) invert the test: as the document is not required to work with non-TeX
   fonts but only reused for tests of all export formats, the failing test
   does not mean there is a problem in LyX and can be ignored.
   
   -1 the document can no longer be used to test export with Unicode fonts.
   
   If there is no suitable Unicode font, this is the correct way.
   For test with Unicode fonts, we could add a modified copy to the
   "dedicated test samples folder" autotests/export/latex/.
   
Günter   

Re: doc/UserGuide systemF tests failing

[Scott Kostyshak]

On Wed, Dec 28, 2016 at 11:25:51AM +0100, Jürgen Spitzmüller wrote:
> Am Mittwoch, den 28.12.2016, 05:20 -0500 schrieb Scott Kostyshak:
> > > Note, though, that the missing glyphs message is a warning, not an
> > > error. We treat it as an error in the GUI, for good reasons (since
> > it
> > > severely can impact the output), but it does not stop LaTeX from
> > > processing.
> > 
> > I do remember this. But since the default is to treat it as an error,
> > I'm not sure we should treat it differently from any LaTeX error.
> 
> Since it does not point to a LaTeX error, but to the fact that the
> selected font is not suitable for the content. Which is probably not
> something that is of interest for the tests (as opposed to LaTeX
> errors).

Ah I see what you mean.

Scott


signature.asc
Description: PGP signature

Re: doc/UserGuide systemF tests failing

[Jürgen Spitzmüller]

Am Mittwoch, den 28.12.2016, 05:20 -0500 schrieb Scott Kostyshak:
> > Note, though, that the missing glyphs message is a warning, not an
> > error. We treat it as an error in the GUI, for good reasons (since
> it
> > severely can impact the output), but it does not stop LaTeX from
> > processing.
> 
> I do remember this. But since the default is to treat it as an error,
> I'm not sure we should treat it differently from any LaTeX error.

Since it does not point to a LaTeX error, but to the fact that the
selected font is not suitable for the content. Which is probably not
something that is of interest for the tests (as opposed to LaTeX
errors).

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: doc/UserGuide systemF tests failing

[Scott Kostyshak]

On Wed, Dec 28, 2016 at 11:13:59AM +0100, Jürgen Spitzmüller wrote:
> Am Mittwoch, den 28.12.2016, 04:27 -0500 schrieb Scott Kostyshak:
> > I know nothing about this so I'll go along with whatever others
> > decide.
> > Günter might have an opinion.
> > 
> > If we do implement the above we might want a general solution for
> > ignoring other errors. I think there might be an enhancement request
> > about this somewhere.
> 
> Note, though, that the missing glyphs message is a warning, not an
> error. We treat it as an error in the GUI, for good reasons (since it
> severely can impact the output), but it does not stop LaTeX from
> processing.

I do remember this. But since the default is to treat it as an error,
I'm not sure we should treat it differently from any LaTeX error.

> I've actually been interested in the opposite: I would like a way to
> > turn warnings into errors. I suppose that only the warnings on the
> > last
> > compilation should be considered (from what I understand, the
> > intermediate compilations often produce expected warnings).
> 
> What do you mean by "errors"? Just make LyX stop processing (that would
> be a bad idea, IMHO) or report the warnings somewhat more loudly?

Report the warnings more loudly.

Scott


signature.asc
Description: PGP signature

Re: doc/UserGuide systemF tests failing

[Jürgen Spitzmüller]

Am Mittwoch, den 28.12.2016, 04:27 -0500 schrieb Scott Kostyshak:
> I know nothing about this so I'll go along with whatever others
> decide.
> Günter might have an opinion.
> 
> If we do implement the above we might want a general solution for
> ignoring other errors. I think there might be an enhancement request
> about this somewhere.

Note, though, that the missing glyphs message is a warning, not an
error. We treat it as an error in the GUI, for good reasons (since it
severely can impact the output), but it does not stop LaTeX from
processing.

I've actually been interested in the opposite: I would like a way to
> turn warnings into errors. I suppose that only the warnings on the
> last
> compilation should be considered (from what I understand, the
> intermediate compilations often produce expected warnings).

What do you mean by "errors"? Just make LyX stop processing (that would
be a bad idea, IMHO) or report the warnings somewhat more loudly?

Jürgen

> 
> Scott

signature.asc
Description: This is a digitally signed message part

Re: doc/UserGuide systemF tests failing

[Scott Kostyshak]

On Wed, Dec 28, 2016 at 10:38:02AM +0100, Jürgen Spitzmüller wrote:

> Probably not. But there are free alternatives that might suffice:
> https://en.wikipedia.org/wiki/Open-source_Unicode_typefaces

OK thanks.

Scott


signature.asc
Description: PGP signature

Re: doc/UserGuide systemF tests failing

[Jürgen Spitzmüller]

Am Mittwoch, den 28.12.2016, 04:09 -0500 schrieb Scott Kostyshak:
> > Note that the
> > CJK quotation marks used in sec. 3.9.3.2 are not covered in some
> fonts.
> > You will probably need a rather comprehensive font (such as Arial
> > Unicode MS).
> 
> I don't have this font. Is it free?

Probably not. But there are free alternatives that might suffice:
https://en.wikipedia.org/wiki/Open-source_Unicode_typefaces

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: doc/UserGuide systemF tests failing

[Scott Kostyshak]

On Wed, Dec 28, 2016 at 10:11:16AM +0100, Jürgen Spitzmüller wrote:

> A more general and probably generally useful solution is to add a
> command line option that makes LyX (again) proceed despite the missing
> glyphs warning (and just report the warning). Then the tests could use
> this option.

I know nothing about this so I'll go along with whatever others decide.
Günter might have an opinion.

If we do implement the above we might want a general solution for
ignoring other errors. I think there might be an enhancement request
about this somewhere.

I've actually been interested in the opposite: I would like a way to
turn warnings into errors. I suppose that only the warnings on the last
compilation should be considered (from what I understand, the
intermediate compilations often produce expected warnings).

Scott


signature.asc
Description: PGP signature

Re: doc/UserGuide systemF tests failing

[Jürgen Spitzmüller]

Am Mittwoch, den 28.12.2016, 09:58 +0100 schrieb Jürgen Spitzmüller:
> For the sake of the tests, we could just add something like the
> following to the preamble that intentionally ignores these missing
> chars:
> 
> \AtBeginDocument{%
> \usepackage{newunicodechar}
> \newcommand\lyxbypasschar[2]{%
> \newunicodechar{#1}{\iffontchar\font`#1 #1\else{\textbf{ char:
> #2>}}\fi}}
> \lyxbypasschar{「}{0x300c}
> \lyxbypasschar{」}{0x300d}
> \lyxbypasschar{『}{0x300e}
> \lyxbypasschar{』}{0x300f}
> \lyxbypasschar{《}{0x300a}
> \lyxbypasschar{》}{0x300b}
> \lyxbypasschar{〈}{0x3008}
> \lyxbypasschar{〉}{0x3009}
> }
> 
> However, I am reluctant to include it in the UG's preamble just for
> the
> sake of making the tests happy.

A more general and probably generally useful solution is to add a
command line option that makes LyX (again) proceed despite the missing
glyphs warning (and just report the warning). Then the tests could use
this option.

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: doc/UserGuide systemF tests failing

[Scott Kostyshak]

On Wed, Dec 28, 2016 at 09:58:08AM +0100, Jürgen Spitzmüller wrote:

> However, I am reluctant to include it in the UG's preamble just for the
> sake of making the tests happy.

I agree.

Scott


signature.asc
Description: PGP signature

Re: doc/UserGuide systemF tests failing

[Scott Kostyshak]

On Wed, Dec 28, 2016 at 09:07:45AM +0100, Jürgen Spitzmüller wrote:
> Am Dienstag, den 27.12.2016, 16:02 -0500 schrieb Scott Kostyshak:
> > The following tests on current master are failing:
> > 
> >   export/doc/UserGuide_dvi3_systemF (Failed)
> >   export/doc/UserGuide_pdf4_systemF (Failed)
> >   export/doc/UserGuide_pdf5_systemF (Failed)
> 
> Does "systemF" mean non-TeX fonts?

Yes. See the Development manual for more information.

> If so, is it due to "missing glyphs"
> errors?

Yes.

> And if this is the case, what font are you using?

The tests for English documents use DejaVu.

> Note that the
> CJK quotation marks used in sec. 3.9.3.2 are not covered in some fonts.
> You will probably need a rather comprehensive font (such as Arial
> Unicode MS).

I don't have this font. Is it free?

> Alternatively, we can define a cjk font for fontspec, but
> I don't know which font is supposed to be available everywhere.

No idea.

Scott


signature.asc
Description: PGP signature

Re: doc/UserGuide systemF tests failing

[Jürgen Spitzmüller]

Am Mittwoch, den 28.12.2016, 09:07 +0100 schrieb Jürgen Spitzmüller:
> Am Dienstag, den 27.12.2016, 16:02 -0500 schrieb Scott Kostyshak:
> > The following tests on current master are failing:
> > 
> >   export/doc/UserGuide_dvi3_systemF (Failed)
> >   export/doc/UserGuide_pdf4_systemF (Failed)
> >   export/doc/UserGuide_pdf5_systemF (Failed)
> 
> Does "systemF" mean non-TeX fonts? If so, is it due to "missing
> glyphs"
> errors? And if this is the case, what font are you using? Note that
> the
> CJK quotation marks used in sec. 3.9.3.2 are not covered in some
> fonts.
> You will probably need a rather comprehensive font (such as Arial
> Unicode MS). Alternatively, we can define a cjk font for fontspec,
> but
> I don't know which font is supposed to be available everywhere.

For the sake of the tests, we could just add something like the
following to the preamble that intentionally ignores these missing
chars:

\AtBeginDocument{%
\usepackage{newunicodechar}
\newcommand\lyxbypasschar[2]{%
\newunicodechar{#1}{\iffontchar\font`#1 #1\else{\textbf{}}\fi}}
\lyxbypasschar{「}{0x300c}
\lyxbypasschar{」}{0x300d}
\lyxbypasschar{『}{0x300e}
\lyxbypasschar{』}{0x300f}
\lyxbypasschar{《}{0x300a}
\lyxbypasschar{》}{0x300b}
\lyxbypasschar{〈}{0x3008}
\lyxbypasschar{〉}{0x3009}
}

However, I am reluctant to include it in the UG's preamble just for the
sake of making the tests happy.

Jürgen

> 
> Jürgen
> 
> > 
> > Scott

signature.asc
Description: This is a digitally signed message part

Re: doc/UserGuide systemF tests failing

[Jürgen Spitzmüller]

Am Dienstag, den 27.12.2016, 16:02 -0500 schrieb Scott Kostyshak:
> The following tests on current master are failing:
> 
>   export/doc/UserGuide_dvi3_systemF (Failed)
>   export/doc/UserGuide_pdf4_systemF (Failed)
>   export/doc/UserGuide_pdf5_systemF (Failed)

Does "systemF" mean non-TeX fonts? If so, is it due to "missing glyphs"
errors? And if this is the case, what font are you using? Note that the
CJK quotation marks used in sec. 3.9.3.2 are not covered in some fonts.
You will probably need a rather comprehensive font (such as Arial
Unicode MS). Alternatively, we can define a cjk font for fontspec, but
I don't know which font is supposed to be available everywhere.

Jürgen

> 
> Scott

signature.asc
Description: This is a digitally signed message part

doc/UserGuide systemF tests failing

[Scott Kostyshak]

The following tests on current master are failing:

  export/doc/UserGuide_dvi3_systemF (Failed)
  export/doc/UserGuide_pdf4_systemF (Failed)
  export/doc/UserGuide_pdf5_systemF (Failed)

Scott


signature.asc
Description: PGP signature