Re: [LyX/master] ctests: update labeling patterns.

[Guenter Milde]

On 2016-09-23, Kornel Benko wrote:
> Am Freitag, 23. September 2016 um 07:29:14, schrieb Guenter Milde 
> 

...

> If we could considerably reduce the number of failings, we can also
> omit regexes and instead use full test names.

1. We cannot significantly reduce the number of failings.

   This may have worked for the original ~300 test cases but now we have
   5407. Even with the same failure-rate we would get 10 times more
   problematic cases.
   
   In addition, there is a large number of failing exports due to the
   non-standard export routes tested but not supported by LaTeX for
   several packages and document classes. (We may ignore them if we do
   not expect future support.)
   See below for statistics.

2. Full test names would not solve the problem of a test case with two (or
   more) problems making it both, unreliable and failing.

> Yes, unfortunately we cannot make tests selective to specific failure.

There is also no need to do this.

However, we could create independent labels for unreliable and inverted
tests or just "allow" matches in both files (actually, there is nothing in
the documentation telling that this is not allowed).


Statistics:

Currently, we have 265 problematic tests 
(ctest -N -L "inverted|suspended|unreliable")

146 tests that are known to fail, including

 57 tests requiring attention -L todo
 14 tests with a track number -L lyxbugs
 61 tests we cannot fix   -L texissues
 
119 unreliable tests, including
 
 62 tests with extra requirements -L nonstandard
 46 passing test with wrong output-L wrong_output
 10 with result depending on TeXLive version  -L varying_versions
 
The 57 "todo" tests are to be investigated and sorted -- only a few
of them will be "easyfix".

This means we have about 200 problematic test cases that will stay so for a
longer time and should adapt to this.

Günter

Re: [LyX/master] ctests: update labeling patterns.

[Guenter Milde]

On 2016-09-23, Kornel Benko wrote:
> Am Freitag, 23. September 2016 um 07:29:14, schrieb Guenter Milde 
> 
>> On 2016-09-22, Kornel Benko wrote:
>> > Am Donnerstag, 22. September 2016 um 21:25:07, schrieb Guenter Milde 
>> > 
>> >> On 2016-09-22, Kornel Benko wrote:

>> >> > Now, export/doc/es/Additional_pdf4_texF in under label lyxbugs and
>> >> > under label ert in invertedTests. At the same time in
>> >> > unreliableTests (mask = export/doc/es/.*_(pdf[45]|dvi3)_texF) under
>> >> > label wrong_output.

...

>> > You could use for instance
>> >Sublabel: wrong_output ert lyxbugs

>> This would mean, 3 regexp patterns instead of 2 simpler ones
>> to single out the test case that currently matches for both:

> Yes, you have your points ...

> In ideal case however, the (inverted|suspended|unreliable)Tests should
> be empty.

Don't strife for this "ideal"
Rather, the (inverted|suspended|unreliable)Tests is comparable to track
issues: no open track issues would usually signalize the project is
dead...

As long as we are working on LyX, both track and *Tests-files are a valuable
buffer that allows us to prioritise which problems to solve first.


>> Currently, we have two "orthogonal" clauses:

>> 1. for a problem common to all "Additional.lyx" exports in
>>invertedTests:

>>  #9871 LyX sends invalid Unicode to iconv when converting to ASCII
>>  # most probably due to BabelPreamble code (language specific headings 
>> for
>>  # theorems, problems , ... are written in the language's default 
>> encoding 
>>  # if they contain non-ASCII characters)
>>  export/doc/(es|fr)/Additional_pdf4_texF

>> 2. for a problem common to all Spanish documents with 8-bit fonts and
>>Xe/LuaTeX:

>>  # Babel-Spanish uses Babel's "strings" feature to define
>>  # separate auto-strings using UTF-8 literals.
>>  # Babel uses the "unicode" strings if it detects XeTeX or LuaTeX.
>>  # This is wrong for Xe/Lua with 8-bit TeX-fonts.
>>  # set inputenc to utf8?
>>  # (Changing the default in lib/languages requires more tests for utf8 
>> first.)
>>  export/examples/es/linguistics_pdf4_texF
>>  export/doc/es/.*_(pdf[45]|dvi3)_texF


>> -1 Making an exception for Additonal.lyx in the second pattern complicates
>>the regexp considerably.

> If we could considerably reduce the number of failings, we can also
> omit regexes and instead use full test names.

If we could solve all track issues within two weeks, we would not need
keywords, importance levels and the like...


Considerably reducing the number of failings would require one of the
following ugly options:

  * spend most developing time on minor cases instead of moving forward
  
  * use ugly hacks (either in documentation or in the LyX code) just to get
export working

  * reduce the number of tests (don't test corner cases like *.texF)
  
  * don't support "tricky" or rarely used packages, combinations, ...

I strongly prefer to adapt the test system to work well with a large
number sorted and well-commented lists of "known failures".



>> -1 When one of the two problems is solved, there must be edits at two
>> places.

> True :(

>> -1 The two independent problems belong do different "*Test" labeling files.
>>Where should the third clause for wrong_output ert lyxbugs go?


>> It is a quite common case, that documents have more than one "orthogonal"
>> problem. 

> Yes, unfortunately we cannot make tests selective to specific failure.

>> It is also a quite common case that a document currently fails to export
>> but we know that even if it compiles fine the output will be wrong.

>> I think we need to change the test system to care for this.

>> One possibility would be run the "invertedtests" filter also for test-cases
>> matching "unreliableTests" and give them two labels.

> This contradicts our intent for unreliableTests. To be clear, I have no
> better idea.

It is exactly what I have in mind with unreliableTests:

  # Regular expressions for tests that do not work as expected
  # (either unreliable or invalid).

When handling "unreliable" independent of inversion, we can still:

-> Skip these tests with `-LE unreliable`

-> Run with e.g. `-L nonstandard` if you have the extra requirements 
   or with -L wrong_output if you want to check...

-> If running the complete suite, don't care for failures of unreliable tests
   unless you know what you are doing.
   
   A report to the list would say something like
   
There are 13 failures: 1 regression, 2 inverted tests that now work,
and 10 unreliable tests.


Günter  

Re: [LyX/master] ctests: update labeling patterns.

[Kornel Benko]

Am Freitag, 23. September 2016 um 07:29:14, schrieb Guenter Milde 

> Dear Kornel,
> 
> On 2016-09-22, Kornel Benko wrote:
> > Am Donnerstag, 22. September 2016 um 21:25:07, schrieb Guenter Milde 
> > 
> >> On 2016-09-22, Kornel Benko wrote:
> 
> >> > Now, export/doc/es/Additional_pdf4_texF in under label lyxbugs and
> >> > under label ert in invertedTests. At the same time in
> >> > unreliableTests (mask = export/doc/es/.*_(pdf[45]|dvi3)_texF) under
> >> > label wrong_output.
> 
> >> > Which of the labels should it be? (now it is wrong_output)
> 
> >> Actually both, because this are independent issues:
> 
> >> * we know export fails currently (like for de/Additional_pdf4_texF)
> >>   due to nonASCII in an ERT inset
> 
> >> * we know Spanish has problems with pdf[45]|dvi3)_texF leading to wrong
> >>   output without (but not export failure).
> 
> >>   # Babel uses the "unicode" strings if it detects XeTeX or LuaTeX.
> >>   # This is wrong for Xe/Lua with 8-bit TeX-fonts.
> 
> >> Please keep both patterns, even if currently it is not possible to have
> >> both labels. This way, the correct label will be applied if one of the
> >> issues is solved and the related pattern removed.
> 
> 
> > But that is simply not correct.
> 
> > You could use for instance
> > Sublabel: wrong_output ert lyxbugs
> 
> This would mean, 3 regexp patterns instead of 2 simpler ones
> to single out the test case that currently matches for both:

Yes, you have your points ...
In ideal case however, the (inverted|suspended|unreliable)Tests should be empty.

> Currently, we have two "orthogonal" clauses:
> 
> 1. for a problem common to all "Additional.lyx" exports in
>invertedTests:
> 
>  #9871 LyX sends invalid Unicode to iconv when converting to ASCII
>  # most probably due to BabelPreamble code (language specific headings for
>  # theorems, problems , ... are written in the language's default 
> encoding 
>  # if they contain non-ASCII characters)
>  export/doc/(es|fr)/Additional_pdf4_texF
> 
> 2. for a problem common to all Spanish documents with 8-bit fonts and
>Xe/LuaTeX:
> 
>  # Babel-Spanish uses Babel's "strings" feature to define
>  # separate auto-strings using UTF-8 literals.
>  # Babel uses the "unicode" strings if it detects XeTeX or LuaTeX.
>  # This is wrong for Xe/Lua with 8-bit TeX-fonts.
>  # set inputenc to utf8?
>  # (Changing the default in lib/languages requires more tests for utf8 
> first.)
>  export/examples/es/linguistics_pdf4_texF
>  export/doc/es/.*_(pdf[45]|dvi3)_texF
> 
> 
> -1 Making an exception for Additonal.lyx in the second pattern complicates
>the regexp considerably.

If we could considerably reduce the number of failings, we can also omit 
regexes and instead use full test names.

> -1 When one of the two problems is solved, there must be edits at two places.

True :(

> -1 The two independent problems belong do different "*Test" labeling files.
>Where should the third clause for wrong_output ert lyxbugs go?
>
> 
> It is a quite common case, that documents have more than one "orthogonal"
> problem. 

Yes, unfortunately we cannot make tests selective to specific failure.

> It is also a quite common case that a document currently fails to export
> but we know that even if it compiles fine the output will be wrong.
> 
> I think we need to change the test system to care for this.
> 
> One possibility would be run the "invertedtests" filter also for test-cases
> matching "unreliableTests" and give them two labels.

This contradicts our intent for unreliableTests. To be clear, I have no better 
idea.

> Günter
> 

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: [LyX/master] ctests: update labeling patterns.

[Guenter Milde]

Dear Kornel,

On 2016-09-22, Kornel Benko wrote:
> Am Donnerstag, 22. September 2016 um 21:25:07, schrieb Guenter Milde 
> 
>> On 2016-09-22, Kornel Benko wrote:

>> > Now, export/doc/es/Additional_pdf4_texF in under label lyxbugs and
>> > under label ert in invertedTests. At the same time in
>> > unreliableTests (mask = export/doc/es/.*_(pdf[45]|dvi3)_texF) under
>> > label wrong_output.

>> > Which of the labels should it be? (now it is wrong_output)

>> Actually both, because this are independent issues:

>> * we know export fails currently (like for de/Additional_pdf4_texF)
>>   due to nonASCII in an ERT inset

>> * we know Spanish has problems with pdf[45]|dvi3)_texF leading to wrong
>>   output without (but not export failure).

>>   # Babel uses the "unicode" strings if it detects XeTeX or LuaTeX.
>>   # This is wrong for Xe/Lua with 8-bit TeX-fonts.

>> Please keep both patterns, even if currently it is not possible to have
>> both labels. This way, the correct label will be applied if one of the
>> issues is solved and the related pattern removed.


> But that is simply not correct.

> You could use for instance
>   Sublabel: wrong_output ert lyxbugs

This would mean, 3 regexp patterns instead of 2 simpler ones
to single out the test case that currently matches for both:

Currently, we have two "orthogonal" clauses:

1. for a problem common to all "Additional.lyx" exports in
   invertedTests:

 #9871 LyX sends invalid Unicode to iconv when converting to ASCII
 # most probably due to BabelPreamble code (language specific headings for
 # theorems, problems , ... are written in the language's default encoding 
 # if they contain non-ASCII characters)
 export/doc/(es|fr)/Additional_pdf4_texF

2. for a problem common to all Spanish documents with 8-bit fonts and
   Xe/LuaTeX:

 # Babel-Spanish uses Babel's "strings" feature to define
 # separate auto-strings using UTF-8 literals.
 # Babel uses the "unicode" strings if it detects XeTeX or LuaTeX.
 # This is wrong for Xe/Lua with 8-bit TeX-fonts.
 # set inputenc to utf8?
 # (Changing the default in lib/languages requires more tests for utf8 
first.)
 export/examples/es/linguistics_pdf4_texF
 export/doc/es/.*_(pdf[45]|dvi3)_texF


-1 Making an exception for Additonal.lyx in the second pattern complicates
   the regexp considerably.

-1 When one of the two problems is solved, there must be edits at two places.

-1 The two independent problems belong do different "*Test" labeling files.
   Where should the third clause for wrong_output ert lyxbugs go?
   

It is a quite common case, that documents have more than one "orthogonal"
problem. 

It is also a quite common case that a document currently fails to export
but we know that even if it compiles fine the output will be wrong.

I think we need to change the test system to care for this.

One possibility would be run the "invertedtests" filter also for test-cases
matching "unreliableTests" and give them two labels.

Günter

Re: [LyX/master] ctests: update labeling patterns.

[Kornel Benko]

Am Donnerstag, 22. September 2016 um 21:25:07, schrieb Guenter Milde 

> On 2016-09-22, Kornel Benko wrote:
> > Am Donnerstag, 22. September 2016 um 12:10:50, schrieb Günter Milde 
> > 
> >> commit 6b60e6811bdfe73f0e0cb05dd793b5b5267aa6cc
> >> Author: Günter Milde 
> >> Date:   Thu Sep 22 12:09:39 2016 +0200
> 
> >>  Sublabel: ert
> >>  #
> >>  # Export failures due to "raw" LaTeX use in ERT or preamble code.
> >>  # "Wontfix" if demonstrating correct use and OK in the default output 
> >> format.
> 
> >>  # Non-ASCII in ERT, fails with inputenc==ASCII (e.g. XeTeX with tex-fonts)
> >> -export/doc/(de|es|fr)/Additional_pdf4_texF
> >> +export/doc/(de|es)/Additional_pdf4_texF
> 
> 
> > Now, export/doc/es/Additional_pdf4_texF in under label lyxbugs and under 
> > label ert in invertedTests. At the same time
> > in unreliableTests (mask = export/doc/es/.*_(pdf[45]|dvi3)_texF) under 
> > label wrong_output.
> 
> > Which of the labels should it be? (now it is wrong_output)
> 
> Actually both, because this are independent issues:
> 
> * we know export fails currently (like for de/Additional_pdf4_texF)
>   due to nonASCII in an ERT inset
>   
> * we know Spanish has problems with pdf[45]|dvi3)_texF leading to wrong
>   output without (but not export failure).
>   
>   # Babel uses the "unicode" strings if it detects XeTeX or LuaTeX.
>   # This is wrong for Xe/Lua with 8-bit TeX-fonts.
> 
> Please keep both patterns, even if currently it is not possible to have
> both labels. This way, the correct label will be applied if one of the
> issues is solved and the related pattern removed.
> 

But that is simply not correct.
You could use for instance
Sublabel: wrong_output ert lyxbugs

> Günter

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: [LyX/master] ctests: update labeling patterns.

[Guenter Milde]

On 2016-09-22, Kornel Benko wrote:
> Am Donnerstag, 22. September 2016 um 12:10:50, schrieb Günter Milde 
> 
>> commit 6b60e6811bdfe73f0e0cb05dd793b5b5267aa6cc
>> Author: Günter Milde 
>> Date:   Thu Sep 22 12:09:39 2016 +0200

>>  Sublabel: ert
>>  #
>>  # Export failures due to "raw" LaTeX use in ERT or preamble code.
>>  # "Wontfix" if demonstrating correct use and OK in the default output 
>> format.

>>  # Non-ASCII in ERT, fails with inputenc==ASCII (e.g. XeTeX with tex-fonts)
>> -export/doc/(de|es|fr)/Additional_pdf4_texF
>> +export/doc/(de|es)/Additional_pdf4_texF


> Now, export/doc/es/Additional_pdf4_texF in under label lyxbugs and under 
> label ert in invertedTests. At the same time
> in unreliableTests (mask = export/doc/es/.*_(pdf[45]|dvi3)_texF) under label 
> wrong_output.

> Which of the labels should it be? (now it is wrong_output)

Actually both, because this are independent issues:

* we know export fails currently (like for de/Additional_pdf4_texF)
  due to nonASCII in an ERT inset
  
* we know Spanish has problems with pdf[45]|dvi3)_texF leading to wrong
  output without (but not export failure).
  
  # Babel uses the "unicode" strings if it detects XeTeX or LuaTeX.
  # This is wrong for Xe/Lua with 8-bit TeX-fonts.

Please keep both patterns, even if currently it is not possible to have
both labels. This way, the correct label will be applied if one of the
issues is solved and the related pattern removed.


Günter

Re: [LyX/master] ctests: update labeling patterns.

[Kornel Benko]

Am Donnerstag, 22. September 2016 um 17:39:22, schrieb Kornel Benko 

> 
> Now, export/doc/es/Additional_pdf4_texF in under label lyxbugs and under 
> label ert in invertedTests. At the same time
> in unreliableTests (mask = export/doc/es/.*_(pdf[45]|dvi3)_texF) under label 
> wrong_output.
> 
> Which of the labels should it be? (now it is wrong_output)

Don't get me wrong, I am happy someone takes care for the tests :)

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: [LyX/master] ctests: update labeling patterns.

[Kornel Benko]

Am Donnerstag, 22. September 2016 um 12:10:50, schrieb Günter Milde 

> commit 6b60e6811bdfe73f0e0cb05dd793b5b5267aa6cc
> Author: Günter Milde 
> Date:   Thu Sep 22 12:09:39 2016 +0200
> 
> ctests: update labeling patterns.
> ---
>  development/autotests/invertedTests   |9 -
>  development/autotests/unreliableTests |2 +-
>  2 files changed, 9 insertions(+), 2 deletions(-)
> 
> diff --git a/development/autotests/invertedTests 
> b/development/autotests/invertedTests
> index 8dab991..f31a913 100644
> --- a/development/autotests/invertedTests
> +++ b/development/autotests/invertedTests
> @@ -93,13 +93,20 @@ export/export/WrongDfnTagHandling_xhtml
>  # contains underscores etc.
>  export/export/latex/LongestLabelWithUnderscore.*
>  
> +#9871 LyX sends invalid Unicode to iconv when converting to ASCII
> +# most probably due to BabelPreamble code (language specific headings for
> +# theorems, problems , ... are written in the language's default encoding 
> +# if they contain non-ASCII characters)
> +export/doc/(es|fr)/Additional_pdf4_texF
> +
> +
>  Sublabel: ert
>  #
>  # Export failures due to "raw" LaTeX use in ERT or preamble code.
>  # "Wontfix" if demonstrating correct use and OK in the default output format.
>  
>  # Non-ASCII in ERT, fails with inputenc==ASCII (e.g. XeTeX with tex-fonts)
> -export/doc/(de|es|fr)/Additional_pdf4_texF
> +export/doc/(de|es)/Additional_pdf4_texF


Now, export/doc/es/Additional_pdf4_texF in under label lyxbugs and under label 
ert in invertedTests. At the same time
in unreliableTests (mask = export/doc/es/.*_(pdf[45]|dvi3)_texF) under label 
wrong_output.

Which of the labels should it be? (now it is wrong_output)

Kornel


signature.asc
Description: This is a digitally signed message part.