Re: status update on the export tests
Am Donnerstag, 5. November 2015 um 16:56:32, schrieb Guenter Milde > On 2015-11-05, Kornel Benko wrote: > > Am Donnerstag, 5. November 2015 um 12:43:12, schrieb Guenter Milde > > > ... > > No, the tests do not have any special naming. As they are a subset of > > revertedTests their name starts with 'INVERTED'. > > Of course, it could be arranged. > > I was still explaining how my concept of "suspension" could work. Ah, sorry. > Arranging "suspended" tests to start with SUSPENDED would ensure the -R > parameter normally skips them as well... > > > But one can use e.g. > > #ctest -R someregex -LE suspended > > to get the desired. > > OK > > ... > > >> However, this is basically what you currently do with "inverted tests" > >> there is basically just a different naming scheme. > > >> The "measure for deviation from a clean state" would be the number of > >> "suspended" tests with my proposal and "inverted" tests in yours. > > >> The only difference are export tests we want to return an error. > >> How are they managed in your scheme? > > > I am not sure I understand. > > > ATM export tests are expected to pass. All other tests are coverer be > > revertedTests. We (Scott and me) would add them to revertedTests. > > I see that there is one more ambiguity: > > * I divide LyX testing into > > unit tests: > testing specific functions and classes > > currently missing We will see, how it evolves. > functional tests: > testing processing (import/export) against a known reference > > currently implemented as tex2lyx tests There is also a label "lyx2lyx", but I agree that are only rudimentary tests. > export tests: > testing the export status of sample documents (most of them taken from > documentation and examples, i.e. with a different main purpose). > > * In contrast, the label "export" inside the "automatic export test framework" > applies only to a subset of test cases. > The others are "inverted/reverted" or "ignored". > > -> "export test" can be either the complete set of the "autotests" or just >the subset where we currently expect the export to pass. > > I understand that this naming is historically motivated und "natural" for > the ones that work with the automatic tests from start. However, it makes > understanding what is going on complicated for newbies. > > > The next problem is, that we must be very careful when speaking about a > test that fails: > > -> inverted tests pass, when the export fails (i.e. returns an error) > > > So I'll try to re-state my question: > > I would like to have a simple number measuring the "cleanness" of LyX > code and shipped documents. For this, I propose to use the number > of test cases that do not give the "correct" result. > > With the current implementation, how can I distinguish: > > 1. correct results: > >a) tests where the export should succeed and really does > >b) tests where the export fails for "good resons" (i.e. correct error > reporting for "impossible" actions) Yes. > 2. wrong results: > >a) tests where the export should succeed but fails There is no such test known. But of course, might exist. If it fails, then something in our preparing lyx file for export is wrong (perl script useSystemFonts.pl+lyxStatus.pm) >b) tests where the export does not report an error but we know it should > (because the exported document is corrupt). This is the hard part. There are so many tests, that no one had time (or pleasure) to check them all visually. > As far as I understand, "revertedTests" reflect the currently expected > result, not the desired result. The number of inverted tests includes > 1b) and 2a). > > Conclusion: > > The result of running the tests is good at telling about regressions or > "surfaced bugs" (i.e. old problems that lead to export errors after changing > either code or sample documents). Yes. > However, neither the number of "failing tests", nor the number of > "inverted" tests can be used to tell how "good" LyX currently is. > Yes. > Günter Kornel signature.asc Description: This is a digitally signed message part.
Re: status update on the export tests
On 2015-11-05, Kornel Benko wrote: > Am Donnerstag, 5. November 2015 um 12:43:12, schrieb Guenter Milde > ... >> > (from here on it matches revertedTests) >> iv. Prepend INVERTED... to testname. Revert test verification. >> > Now calling ctest with '-L export' selects tests with label 'export', >> > thus skipping suspended tests. The same is valid for '-L reverted'. >> > The other possibilities, e.g. using '-R' parameter, does only check for >> > testnames. But the suspended tests must also have a name, so we cannot >> > skip them automatically. >> Then, with a regexp not starting with SUSPENDED, -R would not include >> suspended tests, right? > No, the tests do not have any special naming. As they are a subset of > revertedTests their name starts with 'INVERTED'. > Of course, it could be arranged. I was still explaining how my concept of "suspension" could work. Arranging "suspended" tests to start with SUSPENDED would ensure the -R parameter normally skips them as well... > But one can use e.g. > #ctest -R someregex -LE suspended > to get the desired. OK ... >> However, this is basically what you currently do with "inverted tests" >> there is basically just a different naming scheme. >> The "measure for deviation from a clean state" would be the number of >> "suspended" tests with my proposal and "inverted" tests in yours. >> The only difference are export tests we want to return an error. >> How are they managed in your scheme? > I am not sure I understand. > ATM export tests are expected to pass. All other tests are coverer be > revertedTests. We (Scott and me) would add them to revertedTests. I see that there is one more ambiguity: * I divide LyX testing into unit tests: testing specific functions and classes currently missing functional tests: testing processing (import/export) against a known reference currently implemented as tex2lyx tests export tests: testing the export status of sample documents (most of them taken from documentation and examples, i.e. with a different main purpose). * In contrast, the label "export" inside the "automatic export test framework" applies only to a subset of test cases. The others are "inverted/reverted" or "ignored". -> "export test" can be either the complete set of the "autotests" or just the subset where we currently expect the export to pass. I understand that this naming is historically motivated und "natural" for the ones that work with the automatic tests from start. However, it makes understanding what is going on complicated for newbies. The next problem is, that we must be very careful when speaking about a test that fails: -> inverted tests pass, when the export fails (i.e. returns an error) So I'll try to re-state my question: I would like to have a simple number measuring the "cleanness" of LyX code and shipped documents. For this, I propose to use the number of test cases that do not give the "correct" result. With the current implementation, how can I distinguish: 1. correct results: a) tests where the export should succeed and really does b) tests where the export fails for "good resons" (i.e. correct error reporting for "impossible" actions) 2. wrong results: a) tests where the export should succeed but fails b) tests where the export does not report an error but we know it should (because the exported document is corrupt). As far as I understand, "revertedTests" reflect the currently expected result, not the desired result. The number of inverted tests includes 1b) and 2a). Conclusion: The result of running the tests is good at telling about regressions or "surfaced bugs" (i.e. old problems that lead to export errors after changing either code or sample documents). However, neither the number of "failing tests", nor the number of "inverted" tests can be used to tell how "good" LyX currently is. Günter
Re: status update on the export tests
Am Donnerstag, 5. November 2015 um 12:43:12, schrieb Guenter Milde > Dear Kornel, > > thank you for the explanation. I come a bit nearer to a generic > understanding now. > > On 2015-11-05, Kornel Benko wrote: > > Am Donnerstag, 5. November 2015 um 07:34:47, schrieb Guenter Milde > > > > > ... > >> > It is suspended _only_ if you select testcases with the '-L' parameter. > > >> OK. My idea was that suspended testcases are skipped by default. > > > Let me sketch our use of '-L' parameter. > > > We have a number of lyx-files under lib/doc, lib/examples etc directories. > > There are export types like lyx16, xhtml, pdf, dvi etc > > TeX or non TeX fonts. > > For each combination we create a testname containing a hint to > > a.) the relative path of the lyx file (without extension) > > b.) export type (restricted to the specified output format(in > > lyx-file), if not 'default') > > c.) TeX or non TeX font (texF, systemF) > > (Is there a documentation of the testnames? If not, could this be added to > Development.lyx, maybe or in a README for the test machinery? With examples?) > I was too lazy. (But probably I try to omit documenting because of lack of enough English knowledge) > Then: for my concept of "suspension": > > > If the name > > > i. matches regex in ignoreTests, the testcase is discarded. Handle to > >next testname. > > ii. matches regex in suspendedTests: Prepend SUSPENDED to testname. If > there can be more than one label, the test gets the label "suspended". > > iii. Not matching revertedTests, the test gets the label "export". >Handle next testname. > > > (from here on it matches revertedTests) > iv. Prepend INVERTED... to testname. Revert test verification. > > > > Now calling ctest with '-L export' selects tests with label 'export', > > thus skipping suspended tests. The same is valid for '-L reverted'. > > > The other possibilities, e.g. using '-R' parameter, does only check for > > testnames. But the suspended tests must also have a name, so we cannot > > skip them automatically. > > Then, with a regexp not starting with SUSPENDED, -R would not include > suspended tests, right? No, the tests do not have any special naming. As they are a subset of revertedTests their name starts with 'INVERTED'. Of course, it could be arranged. But one can use e.g. #ctest -R someregex -LE suspended to get the desired. > > This should answer also your remark in a previous post ( > > > So what do you propose for such tests (84% of export tests)? > > > ctest -L export -N | wc => 3719 > > > ctest -L export -N |egrep '/(doc|examples)/'| wc => 3153 > > > 3153 / 3719 => 84.78% > > >>I don't have any clue which tests are hidden behind these commands. > > This means we have about 3000 export tests with "real life" documents in doc > and examples? > > Basically, I would look at currently failing tests > (i.e. return value != expected return value) and > > * invert/uninvert test based on the "correct" return value I expect the suspended to pass (120 cases). If some of them fails (good news), it will be the minority listed by ctest. This is really easier to check. > * suspend test failing for a known reason, so they are not run nor reported > by a "normal" run. > > However, this is basically what you currently do with "inverted tests" there > is basically just a different naming scheme. > > The "measure for deviation from a clean state" would be the number of > "suspended" tests with my proposal and "inverted" tests in yours. > > The only difference are export tests we want to return an error. > How are they managed in your scheme? I am not sure I understand. ATM export tests are expected to pass. All other tests are coverer be revertedTests. We (Scott and me) would add them to revertedTests. > Günter Kornel signature.asc Description: This is a digitally signed message part.
Re: status update on the export tests
Dear Kornel, thank you for the explanation. I come a bit nearer to a generic understanding now. On 2015-11-05, Kornel Benko wrote: > Am Donnerstag, 5. November 2015 um 07:34:47, schrieb Guenter Milde > > ... >> > It is suspended _only_ if you select testcases with the '-L' parameter. >> OK. My idea was that suspended testcases are skipped by default. > Let me sketch our use of '-L' parameter. > We have a number of lyx-files under lib/doc, lib/examples etc directories. > There are export types like lyx16, xhtml, pdf, dvi etc > TeX or non TeX fonts. > For each combination we create a testname containing a hint to > a.) the relative path of the lyx file (without extension) > b.) export type (restricted to the specified output format(in > lyx-file), if not 'default') > c.) TeX or non TeX font (texF, systemF) (Is there a documentation of the testnames? If not, could this be added to Development.lyx, maybe or in a README for the test machinery? With examples?) Then: for my concept of "suspension": > If the name > i. matches regex in ignoreTests, the testcase is discarded. Handle to >next testname. ii. matches regex in suspendedTests: Prepend SUSPENDED to testname. If there can be more than one label, the test gets the label "suspended". iii. Not matching revertedTests, the test gets the label "export". Handle next testname. > (from here on it matches revertedTests) iv. Prepend INVERTED... to testname. Revert test verification. > Now calling ctest with '-L export' selects tests with label 'export', > thus skipping suspended tests. The same is valid for '-L reverted'. > The other possibilities, e.g. using '-R' parameter, does only check for > testnames. But the suspended tests must also have a name, so we cannot > skip them automatically. Then, with a regexp not starting with SUSPENDED, -R would not include suspended tests, right? > This should answer also your remark in a previous post ( > > So what do you propose for such tests (84% of export tests)? > > ctest -L export -N | wc => 3719 > > ctest -L export -N |egrep '/(doc|examples)/'| wc => 3153 > > 3153 / 3719 => 84.78% >> I don't have any clue which tests are hidden behind these commands. This means we have about 3000 export tests with "real life" documents in doc and examples? Basically, I would look at currently failing tests (i.e. return value != expected return value) and * invert/uninvert test based on the "correct" return value * suspend test failing for a known reason, so they are not run nor reported by a "normal" run. However, this is basically what you currently do with "inverted tests" there is basically just a different naming scheme. The "measure for deviation from a clean state" would be the number of "suspended" tests with my proposal and "inverted" tests in yours. The only difference are export tests we want to return an error. How are they managed in your scheme? Günter
Re: status update on the export tests
Am Donnerstag, 5. November 2015 um 07:34:47, schrieb Guenter Milde ... > > It is suspended _only_ if you select testcases with the '-L' parameter. > > OK. My idea was that suspended testcases are skipped by default. Let me sketch our use of '-L' parameter. This should answer also your remark in a previous post ( > So what do you propose for such tests (84% of export tests)? > ctest -L export -N | wc => 3719 > ctest -L export -N |egrep '/(doc|examples)/'| wc => 3153 > 3153 / 3719 => 84.78% I don't have any clue which tests are hidden behind these commands. The general recipe is: ) We have a number of lyx-files under lib/doc, lib/examples etc directories. There are export types like lyx16, xhtml, pdf, dvi etc TeX or non TeX fonts. For each combination we create a testname containing a hint to a.) the relative path of the lyx file (without extension) b.) export type (restricted to the specified output format(in lyx-file), if not 'default') c.) TeX or non TeX font (texF, systemF) Then: If the name i. matches regex in ignoreTests, the testcase is discarded. Handle to next testname. ii. Not matching revertedTests, the test gets the label "export". Handle next testname. (from here on it matches revertedTests) iii. Prepend INVERTED... to testname. Revert test verification. iv. Not matching suspendedTests, the test gets the label "reverted". Handle next testname. (test matches suspendedTests) v. test gets the label "suspended". Now calling ctest with '-L export' selects tests with label 'export', thus skipping suspended tests. The same is valid for '-L reverted'. The other possibilities, e.g. using '-R' parameter, does only check for testnames. But the suspended tests must also have a name, so we cannot skip them automatically. Kornel signature.asc Description: This is a digitally signed message part.
Re: status update on the export tests
On 2015-11-04, Kornel Benko wrote: > Am Mittwoch, 4. November 2015 um 19:27:06, schrieb Guenter Milde > >> On 2015-11-04, Kornel Benko wrote: >> > Am Mittwoch, 4. November 2015 um 15:36:45, schrieb Guenter Milde >> > >> ... >> >> Not only, with "suspending" I also mean "The outcome is of no value >> >> for finding new bugs or regressions until someone solves the known >> >> bug ...". >> >> However, we usually know what the outcome should be if the bug is >> >> solved: if the expected outcome is "pass", this test should not be >> >> inverted. >> > Here we disagree. Matter of taste I suppose. For me the test fails >> > _now_. We don't care now (because we know what's going on etc.). >> > Therefore the test is to be inverted as to not catch unwanted >> > attention. >> I still do not understand the reasoning. It will not catch attention if >> it is suspended, that is why it should be suspended. > It is suspended _only_ if you select testcases with the '-L' parameter. OK. My idea was that suspended testcases are skipped by default. ... > Sure, but look into suspendedTests, there is only 1 regex (for now, I > know) selecting from the inverted tests. Maybe it is indeed a matter of taste. The advantage of your approach is, that it is a small change to the exisisting/previous setup. As you are doing the work, you shall have the final say. Just two thoughts: * In your implementeation, the 1 regex (for now) rather points to "fragile" tests, some of which are "suspended" (i.e. temporarily ignored). Could you name the file "fragileTests" reduce confusion? * How can we distinguish a "good" inversion (the test should fail) from a "bad" inversion (the test should pass but currently does not). Günter
Re: status update on the export tests
Am Mittwoch, 4. November 2015 um 19:27:06, schrieb Guenter Milde > On 2015-11-04, Kornel Benko wrote: > > Am Mittwoch, 4. November 2015 um 15:36:45, schrieb Guenter Milde > > > > ... > > >> Not only, with "suspending" I also mean "The outcome is of no value for > >> finding new bugs or regressions until someone solves the known bug ...". > > > OK. > > >> However, we usually know what the outcome should be if the bug is > >> solved: if the expected outcome is "pass", this test should not be > >> inverted. > > > Here we disagree. Matter of taste I suppose. For me the test fails > > _now_. We don't care now (because we know what's going on etc.). > > Therefore the test is to be inverted as to not catch unwanted > > attention. > > I still do not understand the reasoning. It will not catch attention if > it is suspended, that is why it should be suspended. It is suspended _only_ if you select testcases with the '-L' parameter. > OTOH, if a test that should pass but does not is inverted & suspended (2 > actions), we need to uninvert and to unsuspend (again 2 actions) once the > problem causing the failure is solved. No, we need only uninvert. 'Suspend' has no effect or non-inverted tests. > In contrast, if the "inversion status" matches the expected test result, we > can run suspended tests from time to time and "unsuspend" the tests that now > give the expected output. No, only uninvert. > Also, when looking at inverted tests, we do not know whether this is a > "good" inversion (the test should fail) or a "bad" inversion (the test > should pass but currently does not). > Sure, but look into suspendedTests, there is only 1 regex (for now, I know) selecting from the inverted tests. > Günter > Kornel signature.asc Description: This is a digitally signed message part.
Re: status update on the export tests
On 2015-11-04, Kornel Benko wrote: > Am Mittwoch, 4. November 2015 um 15:36:45, schrieb Guenter Milde > ... >> Not only, with "suspending" I also mean "The outcome is of no value for >> finding new bugs or regressions until someone solves the known bug ...". > OK. >> However, we usually know what the outcome should be if the bug is >> solved: if the expected outcome is "pass", this test should not be >> inverted. > Here we disagree. Matter of taste I suppose. For me the test fails > _now_. We don't care now (because we know what's going on etc.). > Therefore the test is to be inverted as to not catch unwanted > attention. I still do not understand the reasoning. It will not catch attention if it is suspended, that is why it should be suspended. OTOH, if a test that should pass but does not is inverted & suspended (2 actions), we need to uninvert and to unsuspend (again 2 actions) once the problem causing the failure is solved. In contrast, if the "inversion status" matches the expected test result, we can run suspended tests from time to time and "unsuspend" the tests that now give the expected output. Also, when looking at inverted tests, we do not know whether this is a "good" inversion (the test should fail) or a "bad" inversion (the test should pass but currently does not). Günter
Re: status update on the export tests
Am Mittwoch, 4. November 2015 um 15:36:45, schrieb Guenter Milde > On 2015-11-04, Kornel Benko wrote: > > > [-- Type: text/plain, Encoding: 7bit --] > > > Am Mittwoch, 4. November 2015 um 14:36:19, schrieb Vincent van Ravesteijn > > > >> > In my view, suspension is orthogonal to reversion: > >> > > >> > - normal: we want the test to pass > >> > revert: we want the test to fail > > >> You mean: revert is that "it is known to fail", but we haven't fixed it > >> yet. > > > Yes. We hope to do it eventually. > > We have also tests that correctly fail: > "it is known to fail", there is nothing to fix, and we must ensure that it > continues to fail. > The third point is, what prevents us from ignoring this test. > > >> > - normal: run the test > >> > suspend: skip the test temporarily > >> > ignore: skip the test permanently > >> > > > >> Suspending means: "The outcome is noisy, so skip it until someone > >> looks into it and makes the test better." > > > Sort of, if we are careful enough about which test should go there. > > Not only, with "suspending" I also mean "The outcome is of no value for > finding new bugs or regressions until someone solves the known bug ...". OK. > However, we usually know what the outcome should be if the bug is solved: if > the expected outcome is "pass", this test should not be inverted. > Here we disagree. Matter of taste I suppose. For me the test fails _now_. We don't care now (because we know what's going on etc.). Therefore the test is to be inverted as to not catch unwanted attention. > > For the actual committed cmake build, suspended test means: > > a.) Test is one of export tests > > b.) Test is failing, and therefore it is part of revertedTests > > c.) In 'normal' use like 'ctest -L export' it is hidden > > d.) Test gets the ctest-label 'suspended' > > You can run all suspended tests with 'ctest -L suspended'. > > Or, for a known testname (say xyzzy) you can use 'ctest -R xyzzy'. > > >> Ignore: Skipping a test permanently is the same as just removing the > >> test.. ?? > > > This is effectively the same here. Because the testcase should be > > ignored (it is part of ignoredTests) it will not be added with > > add_test(). E.g. it is not known to ctest. > > However, as the test rules are something like: "test exporting all our > manuals to PDF (pdflatex)", we need to specify cases where a manual is > known not to compile, e.g. because it relies on non-TeX fonts or special > features of LuaTeX. I was describing the machinery how our ctests currently works. Not politics, that is how we organize our config files (ignoredTests, revertedTests, suspendedTests). > Günter > Kornel signature.asc Description: This is a digitally signed message part.
Re: status update on the export tests
On 2015-11-04, Kornel Benko wrote: > [-- Type: text/plain, Encoding: 7bit --] > Am Mittwoch, 4. November 2015 um 14:36:19, schrieb Vincent van Ravesteijn > >> > In my view, suspension is orthogonal to reversion: >> > >> > - normal: we want the test to pass >> > revert: we want the test to fail >> You mean: revert is that "it is known to fail", but we haven't fixed it yet. > Yes. We hope to do it eventually. We have also tests that correctly fail: "it is known to fail", there is nothing to fix, and we must ensure that it continues to fail. The third point is, what prevents us from ignoring this test. >> > - normal: run the test >> > suspend: skip the test temporarily >> > ignore: skip the test permanently >> > >> Suspending means: "The outcome is noisy, so skip it until someone >> looks into it and makes the test better." > Sort of, if we are careful enough about which test should go there. Not only, with "suspending" I also mean "The outcome is of no value for finding new bugs or regressions until someone solves the known bug ...". However, we usually know what the outcome should be if the bug is solved: if the expected outcome is "pass", this test should not be inverted. > For the actual committed cmake build, suspended test means: > a.) Test is one of export tests > b.) Test is failing, and therefore it is part of revertedTests > c.) In 'normal' use like 'ctest -L export' it is hidden > d.) Test gets the ctest-label 'suspended' > You can run all suspended tests with 'ctest -L suspended'. > Or, for a known testname (say xyzzy) you can use 'ctest -R xyzzy'. >> Ignore: Skipping a test permanently is the same as just removing the >> test.. ?? > This is effectively the same here. Because the testcase should be > ignored (it is part of ignoredTests) it will not be added with > add_test(). E.g. it is not known to ctest. However, as the test rules are something like: "test exporting all our manuals to PDF (pdflatex)", we need to specify cases where a manual is known not to compile, e.g. because it relies on non-TeX fonts or special features of LuaTeX. Günter
Re: status update on the export tests
Am Mittwoch, 4. November 2015 um 14:36:19, schrieb Vincent van Ravesteijn > > In my view, suspension is orthogonal to reversion: > > > > - normal: we want the test to pass > > revert: we want the test to fail > > You mean: revert is that "it is known to fail", but we haven't fixed it yet. Yes. We hope to do it eventually. > > > > - normal: run the test > > suspend: skip the test temporarily > > ignore: skip the test permanently > > > > Suspending means: "The outcome is noisy, so skip it until someone > looks into it and makes the test better." Sort of, if we are careful enough about which test should go there. For the actual committed cmake build, suspended test means: a.) Test is one of export tests b.) Test is failing, and therefore it is part of revertedTests c.) In 'normal' use like 'ctest -L export' it is hidden d.) Test gets the ctest-label 'suspended' You can run all suspended tests with 'ctest -L suspended'. Or, for a known testname (say xyzzy) you can use 'ctest -R xyzzy'. > Ignore: Skipping a test permanently is the same as just removing the test.. ?? This is effectively the same here. Because the testcase should be ignored (it is part of ignoredTests) it will not be added with add_test(). E.g. it is not known to ctest. > Vincent Kornel signature.asc Description: This is a digitally signed message part.
Re: status update on the export tests
> In my view, suspension is orthogonal to reversion: > > - normal: we want the test to pass > revert: we want the test to fail You mean: revert is that "it is known to fail", but we haven't fixed it yet. > > - normal: run the test > suspend: skip the test temporarily > ignore: skip the test permanently > Suspending means: "The outcome is noisy, so skip it until someone looks into it and makes the test better." Ignore: Skipping a test permanently is the same as just removing the test.. ?? Vincent
Re: status update on the export tests
On 2015-11-04, Kornel Benko wrote: > Am Mittwoch, 4. November 2015 um 09:30:03, schrieb Guenter Milde > >> On 2015-11-02, Kornel Benko wrote: >> > Am 2. November 2015 um 08:36:05, schrieb Guenter Milde >> >> On 2015-11-02, Scott Kostyshak wrote: >> >> > On Sun, Nov 01, 2015 at 10:36:17PM +0100, Kornel Benko wrote: >> >> >> Am 1. November 2015 um 21:27:11, schrieb Guenter Milde >> >> >> > Could we introduce a new status "suspended" - meaning just skip the >> >> >> > test until a known bug is solved as we know the result is >> >> >> > insignificant until then? ... >> >> Candidates for "suspended" tests are >> >> * "fragile" documents (many packages, ERT, heavy preamble, ...), >> >> * "fragile" export routes (XeTeX/LuaTeX with TeX fonts), >> >> * non-default export routes >> >> and especially combinations of these. >> > OK, here is my suggestion >> > 1.) We add the appropriate tests into revertedTests >> Why? This would undo one important benefit of "suspended" tests: > I was not clear about this. These tests will not be run with e.g. > 'ctest -L export' (see the '-L' param). In this sense we get 'clean' > export tests. May be I was not clear either: In my view, suspension is orthogonal to reversion: - normal: we want the test to pass revert: we want the test to fail - normal: run the test suspend: skip the test temporarily ignore: skip the test permanently In other words: while "normal" vs. "reverted" will be used to specify the expected test result, ignore and suspend will be used if we established the reason why a test currently gives the wrong result but cannot immediately solve the issue. Then we have * suspended reverted tests: should fail but currently passes, e.g. wrong output/dataloss that does not lead to error. * suspended normal tests: should pass but currently fails, e.g. ru/Intro.lyx with XeTeX+TeX-fonts With the benefit: >> >> Suspending instead of reverting frees us from the need to >> >> reassess them if a change in the "real life documents" or a fix >> >> makes them pass again. Instead, they could/should be revised at a >> >> time we have fixed major known problems and no release >> >> pressure... ... > So what do you propose for such tests (84% of export tests)? > ctest -L export -N | wc => 3719 > ctest -L export -N |egrep '/(doc|examples)/'| wc => 3153 > 3153 / 3719 => 84.78% I don't have any clue which tests are hidden behind these commands. The general recipe is: * inspect the file, * run the export command "by hand", * try to find the reason for the test failure result If the current "inversion state" does not match the expected test outcome (pass/fail), correct it. If it is "easyfix", fix. If fixing »the right way« takes time, suspend (stating the established reason). The idea is, to de-couple the test analysis (from failure to bug report) from the repair of found problems. This will hopefully allow us to get the test suite clean without hackish quick-fixes. > For now committed 1.b at 538228d Thanks Günter
Re: status update on the export tests
Am Mittwoch, 4. November 2015 um 09:30:03, schrieb Guenter Milde > On 2015-11-02, Kornel Benko wrote: > > Am 2. November 2015 um 08:36:05, schrieb Guenter Milde > >> On 2015-11-02, Scott Kostyshak wrote: > >> > On Sun, Nov 01, 2015 at 10:36:17PM +0100, Kornel Benko wrote: > >> >> Am 1. November 2015 um 21:27:11, schrieb Guenter Milde > > >> >> > Could we introduce a new status "suspended" - meaning just skip the > >> >> > test until a known bug is solved as we know the result is > >> >> > insignificant until then? > > >> >> We already have such (specified in file "ignoredTests"). But as this > >> >> tests are never executed, nobody cares for them anymore. > >> >> The tests here are such, that we know, we never resolve them. > > But even here, there are two cases: > > a) cases that fail for a good reason and should always fail >(e.g. export document with non-TeX fonts using 8-bit LaTeX) > > b) cases that should not fail but do (for reasons we cannot change). > > While a) should be "permanently inverted" (to give a warning if a change > makes the export pass but we expect the result to be faulty), b) is > correctly placed in "ignored" -- the test case does not help detect LyX > buts in any way. > > > >> Suggestion: > > >> Specified in file "suspendedTests") with the reason for suspending > >> (bug report, commit that turned hidden problems into export failure, ...) > > >> These tests are normally skipped, but they are not forgotten. > > >> The tests here are such, that we know, we can resolve them but their > >> failure is a minor issue that can be postponed (comparable to enhancement > >> requests in Trac). > > > >> Candidates for "suspended" tests are > > >> * "fragile" documents (many packages, ERT, heavy preamble, ...), > >> * "fragile" export routes (XeTeX/LuaTeX with TeX fonts), > >> * non-default export routes > > >> and especially combinations of these. > > > > OK, here is my suggestion > > 1.) We add the appropriate tests into revertedTests > > Why? This would undo one important benefit of "suspended" tests: I was not clear about this. These tests will not be run with e.g. 'ctest -L export' (see the '-L' param). In this sense we get 'clean' export tests. > >> Suspending instead of reverting also frees us from the need to reassess > >> them if a change in the "real life documents" or a fix makes them pass > >> again. Instead, they could/should be revised at a time we have fixed > >> major known problems and no release pressure... > > I'd only add the test to "revertedTests", too, if it is established that > the correct result for this test is a failure. > > > The content in suspendedTests may be (in our case) e.g. > > pdf4_texF > > > 1.a) > > If a test is to be inverted, we check suspendedTest, > > and if there, we ignore the testcase. > > Here, I'd change the priority: > > * with a normal run, all suspended tests are ignored, > > * with "run suspended tests" or "include suspended tests", > suspended test are run, taking into consideration their > "inversion state". > > > or > > 1.b) > > Such a test may get a label "suspended" instead of > > "export", so that 'ctest -L export' will be clean, but > > we also have the possibility to use 'ctest -L > > suspended'. > > > This does neither effect non-inverted nor ignored. > > My suggestion would affect non-inverted tests (the ones with the label > "suspended"). > > This means for failing test you will have 3 options: > > 1. if failing is the expected outcome: invert > 2. if failing for permanent reasons that are none of our business: ignore > 3. if failing for minor reasons or a known bug: suspend > > Motivation: > > Some bugs only lead to failures depending on document content. We do > "road testing" with real life documents (manuals, examples). Work on the > documents could easily change the export status of tests without any > progress on fixing the underlying problem! (Examples: adding/removing a > character not available with non-default export route, adding non-ASCII > character to the PDF Header Information with "inputenc=ascii", > Writing my name as G\"unter in the PDF Header Information before solving > #9830, ...) > > However, editing manuals or examples should not require immediate > revision of test cases using this document if the basic problem is not > solved. So what do you propose for such tests (84% of export tests)? ctest -L export -N | wc => 3719 ctest -L export -N |egrep '/(doc|examples)/'| wc => 3153 3153 / 3719 => 84.78% > OTOH, if bugs that triggered suspension of some test cases are solved, > it is time to revisit the set of suspended tests, reactivate the ones > that pass and re-label the ones that still fail. > > > > > I prefer 1.b. > > Whatever works best
Re: status update on the export tests
Am Montag, 2. November 2015 um 08:39:59, schrieb Guenter Milde > On 2015-11-02, Scott Kostyshak wrote: > > > Good point. On the other hand, if we make regex changes then it is > > harder to revert them. Problems have come up before of us > > misconstructing the regexes. > > If possible, make separate regexp patterns for separate "fail reasons". > Then, we just have to delete one regexp if the cause for the inversion is > gone... I don't understand. There is no such reference in testcase-names. > Günter Kornel signature.asc Description: This is a digitally signed message part.
Re: status update on the export tests
On 2015-11-02, Kornel Benko wrote: > Am 2. November 2015 um 08:36:05, schrieb Guenter Milde >> On 2015-11-02, Scott Kostyshak wrote: >> > On Sun, Nov 01, 2015 at 10:36:17PM +0100, Kornel Benko wrote: >> >> Am 1. November 2015 um 21:27:11, schrieb Guenter Milde >> >> > Could we introduce a new status "suspended" - meaning just skip the >> >> > test until a known bug is solved as we know the result is >> >> > insignificant until then? >> >> We already have such (specified in file "ignoredTests"). But as this >> >> tests are never executed, nobody cares for them anymore. >> >> The tests here are such, that we know, we never resolve them. But even here, there are two cases: a) cases that fail for a good reason and should always fail (e.g. export document with non-TeX fonts using 8-bit LaTeX) b) cases that should not fail but do (for reasons we cannot change). While a) should be "permanently inverted" (to give a warning if a change makes the export pass but we expect the result to be faulty), b) is correctly placed in "ignored" -- the test case does not help detect LyX buts in any way. >> Suggestion: >> Specified in file "suspendedTests") with the reason for suspending >> (bug report, commit that turned hidden problems into export failure, ...) >> These tests are normally skipped, but they are not forgotten. >> The tests here are such, that we know, we can resolve them but their >> failure is a minor issue that can be postponed (comparable to enhancement >> requests in Trac). >> Candidates for "suspended" tests are >> * "fragile" documents (many packages, ERT, heavy preamble, ...), >> * "fragile" export routes (XeTeX/LuaTeX with TeX fonts), >> * non-default export routes >> and especially combinations of these. > OK, here is my suggestion > 1.) We add the appropriate tests into revertedTests Why? This would undo one important benefit of "suspended" tests: >> Suspending instead of reverting also frees us from the need to reassess >> them if a change in the "real life documents" or a fix makes them pass >> again. Instead, they could/should be revised at a time we have fixed >> major known problems and no release pressure... I'd only add the test to "revertedTests", too, if it is established that the correct result for this test is a failure. > The content in suspendedTests may be (in our case) e.g. > pdf4_texF > 1.a) > If a test is to be inverted, we check suspendedTest, > and if there, we ignore the testcase. Here, I'd change the priority: * with a normal run, all suspended tests are ignored, * with "run suspended tests" or "include suspended tests", suspended test are run, taking into consideration their "inversion state". > or > 1.b) > Such a test may get a label "suspended" instead of > "export", so that 'ctest -L export' will be clean, but > we also have the possibility to use 'ctest -L > suspended'. > This does neither effect non-inverted nor ignored. My suggestion would affect non-inverted tests (the ones with the label "suspended"). This means for failing test you will have 3 options: 1. if failing is the expected outcome: invert 2. if failing for permanent reasons that are none of our business: ignore 3. if failing for minor reasons or a known bug: suspend Motivation: Some bugs only lead to failures depending on document content. We do "road testing" with real life documents (manuals, examples). Work on the documents could easily change the export status of tests without any progress on fixing the underlying problem! (Examples: adding/removing a character not available with non-default export route, adding non-ASCII character to the PDF Header Information with "inputenc=ascii", Writing my name as G\"unter in the PDF Header Information before solving #9830, ...) However, editing manuals or examples should not require immediate revision of test cases using this document if the basic problem is not solved. OTOH, if bugs that triggered suspension of some test cases are solved, it is time to revisit the set of suspended tests, reactivate the ones that pass and re-label the ones that still fail. > I prefer 1.b. Whatever works best in praxi. Günter
Re: status update on the export tests
On Mon, Nov 02, 2015 at 09:36:02PM +0100, Kornel Benko wrote: > Am Montag, 2. November 2015 um 21:31:12, schrieb Kornel Benko > > Am Montag, 2. November 2015 um 11:21:09, schrieb Kornel Benko > > > > > Am Montag, 2. November 2015 um 08:36:05, schrieb Guenter Milde > > > > > > > Dear Scott and Kornel, > > > > > > > > On 2015-11-02, Scott Kostyshak wrote: > > > > > On Sun, Nov 01, 2015 at 10:36:17PM +0100, Kornel Benko wrote: > > > > >> Am Sonntag, 1. November 2015 um 21:27:11, schrieb Guenter Milde > > > > >> > > > > ... > > > > > > Candidates for "suspended" tests are > > > > > > > > * "fragile" documents (many packages, ERT, heavy preamble, ...), > > > > * "fragile" export routes (XeTeX/LuaTeX with TeX fonts), > > > > * non-default export routes > > > > > > > > and especially combinations of these. > > > > > > > > > > > > Günter > > > > > > > > > > OK, here is my suggestion > > > 1.) We add the appropriate tests into revertedTests > > > > > > The content in suspendedTests may be (in our case) e.g. > > > pdf4_texF > > > > > > 1.a) > > > If a test is to be inverted, we check suspendedTest, and if > > > there, we ignore the testcase. > > > > > > or > > > 1.b) > > > Such a test may became a label "suspended" instead of "export", > > > so that > > > 'ctest -L export' will be clean, but we also have the > > > possibility to > > > use 'ctest -L suspended'. > > > > > > This does neither effect non-inverted nor ignored. > > > > > > I prefer 1.b. > > > > > > > Apparently nobody cares? > > The attached works here. > > > > Forgotten this one in the last post. > Therefore I resend all. I don't have much of an opinion on this. Part of me thinks that it adds extra complication but on the other hand I understand the reasoning. If you and Günter think it makes sense, then since no one else has an opinion I would say go ahead. But please document the suspended tests, what they mean, how to interpret, etc. in Development.lyx. I don't have enough of an understanding to document it. Thanks for your work on this! Soon hopefully the tests will be in a good state. Scott > > Kornel > diff --git a/development/cmake/modules/LyXMacros.cmake > b/development/cmake/modules/LyXMacros.cmake > index 2d20cb7..ea16243 100644 > --- a/development/cmake/modules/LyXMacros.cmake > +++ b/development/cmake/modules/LyXMacros.cmake > @@ -322,7 +322,7 @@ macro(settestlabel testname) > endmacro() > > macro(setmarkedtestlabel testname reverted) > - if(reverted) > + if(${reverted}) > settestlabel(${testname} "reverted" ${ARGN}) >else() > settestlabel(${testname} ${ARGN}) > diff --git a/development/autotests/ExportTests.cmake > b/development/autotests/ExportTests.cmake > index 50b4aa4..c3367a3 100644 > --- a/development/autotests/ExportTests.cmake > +++ b/development/autotests/ExportTests.cmake > @@ -137,8 +137,26 @@ endmacro() > > loadTestList(revertedTests revertedTests) > loadTestList(ignoredTests ignoredTests) > +loadTestList(suspendedTests suspendedTests) > > -foreach(libsubfolder doc examples templates) > +macro(handlesuspended TestName reverted testlabel) > + set(mylabel ${testlabel}) > + set(myreverted ${reverted}) > + if (${reverted}) > +# check suspension only for reverted tests > +findexpr(tfound TestName suspendedTests) > +if (tfound) > + set(mylabel "suspended") > + set(myreverted 0) # if test is to be suspended, remove the 'reverted' > flag > +endif() > + endif() > + setmarkedtestlabel(${TestName} ${myreverted} ${mylabel}) > +endmacro() > + > +# preparing to add e.g. development/mathmacro to the foreach() loop > +foreach(libsubfolderx lib/doc lib/examples lib/templates) > + set(LIBSUB_SRC_DIR "${TOP_SRC_DIR}/${libsubfolderx}") > + string(REGEX REPLACE "^(lib|development)/" "" libsubfolder > "${libsubfolderx}") >set(LIBSUB_SRC_DIR "${TOP_SRC_DIR}/lib/${libsubfolder}") >file(GLOB_RECURSE lyx_files RELATIVE "${LIBSUB_SRC_DIR}" > "${LIBSUB_SRC_DIR}/*.lyx") >list(SORT lyx_files) > @@ -173,7 +191,7 @@ foreach(libsubfolder doc examples templates) > -DTOP_SRC_DIR=${TOP_SRC_DIR} > -DPERL_EXECUTABLE=${PERL_EXECUTABLE} > -P "${TOP_SRC_DIR}/development/autotests/export.cmake") > - setmarkedtestlabel(${TestName} ${reverted} "export") > + handlesuspended(${TestName} ${reverted} "export") # checking for > suspended lyx16 exports > endif() > if(LYX_PYTHON_EXECUTABLE) ># For use of lyx2lyx we need the python executable > @@ -188,7 +206,7 @@ foreach(libsubfolder doc examples templates) >"-DLYX_TESTS_USERDIR=${LYX_TESTS_USERDIR}" >"-DLYXFILE=${LIBSUB_SRC_DIR}/${f}.lyx" >-P "${TOP_SRC_DIR}/development/autotests/lyx2lyxtest.cmake") > -setmarkedtestlabel(${TestName} ${reverted} "lyx2lyx") > +handlesuspended(${TestName} ${reverted} "lyx2lyx") >endif() > endif() > set(TestNa
Re: status update on the export tests
Am Montag, 2. November 2015 um 21:31:12, schrieb Kornel Benko > Am Montag, 2. November 2015 um 11:21:09, schrieb Kornel Benko > > Am Montag, 2. November 2015 um 08:36:05, schrieb Guenter Milde > > > > > Dear Scott and Kornel, > > > > > > On 2015-11-02, Scott Kostyshak wrote: > > > > On Sun, Nov 01, 2015 at 10:36:17PM +0100, Kornel Benko wrote: > > > >> Am Sonntag, 1. November 2015 um 21:27:11, schrieb Guenter Milde > > > >> > > ... > > > > Candidates for "suspended" tests are > > > > > > * "fragile" documents (many packages, ERT, heavy preamble, ...), > > > * "fragile" export routes (XeTeX/LuaTeX with TeX fonts), > > > * non-default export routes > > > > > > and especially combinations of these. > > > > > > > > > Günter > > > > > > > OK, here is my suggestion > > 1.) We add the appropriate tests into revertedTests > > > > The content in suspendedTests may be (in our case) e.g. > > pdf4_texF > > > > 1.a) > > If a test is to be inverted, we check suspendedTest, and if > > there, we ignore the testcase. > > > > or > > 1.b) > > Such a test may became a label "suspended" instead of "export", > > so that > > 'ctest -L export' will be clean, but we also have the > > possibility to > > use 'ctest -L suspended'. > > > > This does neither effect non-inverted nor ignored. > > > > I prefer 1.b. > > > > Apparently nobody cares? > The attached works here. > Forgotten this one in the last post. Therefore I resend all. Korneldiff --git a/development/cmake/modules/LyXMacros.cmake b/development/cmake/modules/LyXMacros.cmake index 2d20cb7..ea16243 100644 --- a/development/cmake/modules/LyXMacros.cmake +++ b/development/cmake/modules/LyXMacros.cmake @@ -322,7 +322,7 @@ macro(settestlabel testname) endmacro() macro(setmarkedtestlabel testname reverted) - if(reverted) + if(${reverted}) settestlabel(${testname} "reverted" ${ARGN}) else() settestlabel(${testname} ${ARGN}) diff --git a/development/autotests/ExportTests.cmake b/development/autotests/ExportTests.cmake index 50b4aa4..c3367a3 100644 --- a/development/autotests/ExportTests.cmake +++ b/development/autotests/ExportTests.cmake @@ -137,8 +137,26 @@ endmacro() loadTestList(revertedTests revertedTests) loadTestList(ignoredTests ignoredTests) +loadTestList(suspendedTests suspendedTests) -foreach(libsubfolder doc examples templates) +macro(handlesuspended TestName reverted testlabel) + set(mylabel ${testlabel}) + set(myreverted ${reverted}) + if (${reverted}) +# check suspension only for reverted tests +findexpr(tfound TestName suspendedTests) +if (tfound) + set(mylabel "suspended") + set(myreverted 0) # if test is to be suspended, remove the 'reverted' flag +endif() + endif() + setmarkedtestlabel(${TestName} ${myreverted} ${mylabel}) +endmacro() + +# preparing to add e.g. development/mathmacro to the foreach() loop +foreach(libsubfolderx lib/doc lib/examples lib/templates) + set(LIBSUB_SRC_DIR "${TOP_SRC_DIR}/${libsubfolderx}") + string(REGEX REPLACE "^(lib|development)/" "" libsubfolder "${libsubfolderx}") set(LIBSUB_SRC_DIR "${TOP_SRC_DIR}/lib/${libsubfolder}") file(GLOB_RECURSE lyx_files RELATIVE "${LIBSUB_SRC_DIR}" "${LIBSUB_SRC_DIR}/*.lyx") list(SORT lyx_files) @@ -173,7 +191,7 @@ foreach(libsubfolder doc examples templates) -DTOP_SRC_DIR=${TOP_SRC_DIR} -DPERL_EXECUTABLE=${PERL_EXECUTABLE} -P "${TOP_SRC_DIR}/development/autotests/export.cmake") - setmarkedtestlabel(${TestName} ${reverted} "export") + handlesuspended(${TestName} ${reverted} "export") # checking for suspended lyx16 exports endif() if(LYX_PYTHON_EXECUTABLE) # For use of lyx2lyx we need the python executable @@ -188,7 +206,7 @@ foreach(libsubfolder doc examples templates) "-DLYX_TESTS_USERDIR=${LYX_TESTS_USERDIR}" "-DLYXFILE=${LIBSUB_SRC_DIR}/${f}.lyx" -P "${TOP_SRC_DIR}/development/autotests/lyx2lyxtest.cmake") -setmarkedtestlabel(${TestName} ${reverted} "lyx2lyx") +handlesuspended(${TestName} ${reverted} "lyx2lyx") endif() endif() set(TestName "check_load/${libsubfolder}/${f}") @@ -202,7 +220,7 @@ foreach(libsubfolder doc examples templates) -DPARAMS_DIR=${TOP_SRC_DIR}/development/autotests -DWORKDIR=${CMAKE_CURRENT_BINARY_DIR}/${LYX_HOME} -P "${TOP_SRC_DIR}/development/autotests/check_load.cmake") - setmarkedtestlabel(${TestName} ${reverted} "load") + handlesuspended(${TestName} ${reverted} "load") set_tests_properties(${TestName} PROPERTIES RUN_SERIAL ON) endif() getoutputformats("${LIBSUB_SRC_DIR}/${f}.lyx" formatlist) @@ -234,7 +252,7 @@ foreach(libsubfolder doc examples templates) -DTOP_SRC_DIR=${TOP_SRC_DIR} -DPERL_EXECUTABLE=${PERL_EXECUTABLE} -P "${TOP_SRC_DIR}/development/autotests/export.cmake") - setmarkedtestlabel(${TestName} ${reverted
Re: status update on the export tests
Am Montag, 2. November 2015 um 11:21:09, schrieb Kornel Benko > Am Montag, 2. November 2015 um 08:36:05, schrieb Guenter Milde > > > Dear Scott and Kornel, > > > > On 2015-11-02, Scott Kostyshak wrote: > > > On Sun, Nov 01, 2015 at 10:36:17PM +0100, Kornel Benko wrote: > > >> Am Sonntag, 1. November 2015 um 21:27:11, schrieb Guenter Milde > > >> ... > > Candidates for "suspended" tests are > > > > * "fragile" documents (many packages, ERT, heavy preamble, ...), > > * "fragile" export routes (XeTeX/LuaTeX with TeX fonts), > > * non-default export routes > > > > and especially combinations of these. > > > > > > Günter > > > > OK, here is my suggestion > 1.) We add the appropriate tests into revertedTests > > The content in suspendedTests may be (in our case) e.g. > pdf4_texF > > 1.a) > If a test is to be inverted, we check suspendedTest, and if > there, we ignore the testcase. > > or > 1.b) > Such a test may became a label "suspended" instead of "export", > so that > 'ctest -L export' will be clean, but we also have the > possibility to > use 'ctest -L suspended'. > > This does neither effect non-inverted nor ignored. > > I prefer 1.b. > Apparently nobody cares? The attached works here. Korneldiff --git a/development/autotests/ExportTests.cmake b/development/autotests/ExportTests.cmake index 50b4aa4..c3367a3 100644 --- a/development/autotests/ExportTests.cmake +++ b/development/autotests/ExportTests.cmake @@ -137,8 +137,26 @@ endmacro() loadTestList(revertedTests revertedTests) loadTestList(ignoredTests ignoredTests) +loadTestList(suspendedTests suspendedTests) -foreach(libsubfolder doc examples templates) +macro(handlesuspended TestName reverted testlabel) + set(mylabel ${testlabel}) + set(myreverted ${reverted}) + if (${reverted}) +# check suspension only for reverted tests +findexpr(tfound TestName suspendedTests) +if (tfound) + set(mylabel "suspended") + set(myreverted 0) # if test is to be suspended, remove the 'reverted' flag +endif() + endif() + setmarkedtestlabel(${TestName} ${myreverted} ${mylabel}) +endmacro() + +# preparing to add e.g. development/mathmacro to the foreach() loop +foreach(libsubfolderx lib/doc lib/examples lib/templates) + set(LIBSUB_SRC_DIR "${TOP_SRC_DIR}/${libsubfolderx}") + string(REGEX REPLACE "^(lib|development)/" "" libsubfolder "${libsubfolderx}") set(LIBSUB_SRC_DIR "${TOP_SRC_DIR}/lib/${libsubfolder}") file(GLOB_RECURSE lyx_files RELATIVE "${LIBSUB_SRC_DIR}" "${LIBSUB_SRC_DIR}/*.lyx") list(SORT lyx_files) @@ -173,7 +191,7 @@ foreach(libsubfolder doc examples templates) -DTOP_SRC_DIR=${TOP_SRC_DIR} -DPERL_EXECUTABLE=${PERL_EXECUTABLE} -P "${TOP_SRC_DIR}/development/autotests/export.cmake") - setmarkedtestlabel(${TestName} ${reverted} "export") + handlesuspended(${TestName} ${reverted} "export") # checking for suspended lyx16 exports endif() if(LYX_PYTHON_EXECUTABLE) # For use of lyx2lyx we need the python executable @@ -188,7 +206,7 @@ foreach(libsubfolder doc examples templates) "-DLYX_TESTS_USERDIR=${LYX_TESTS_USERDIR}" "-DLYXFILE=${LIBSUB_SRC_DIR}/${f}.lyx" -P "${TOP_SRC_DIR}/development/autotests/lyx2lyxtest.cmake") -setmarkedtestlabel(${TestName} ${reverted} "lyx2lyx") +handlesuspended(${TestName} ${reverted} "lyx2lyx") endif() endif() set(TestName "check_load/${libsubfolder}/${f}") @@ -202,7 +220,7 @@ foreach(libsubfolder doc examples templates) -DPARAMS_DIR=${TOP_SRC_DIR}/development/autotests -DWORKDIR=${CMAKE_CURRENT_BINARY_DIR}/${LYX_HOME} -P "${TOP_SRC_DIR}/development/autotests/check_load.cmake") - setmarkedtestlabel(${TestName} ${reverted} "load") + handlesuspended(${TestName} ${reverted} "load") set_tests_properties(${TestName} PROPERTIES RUN_SERIAL ON) endif() getoutputformats("${LIBSUB_SRC_DIR}/${f}.lyx" formatlist) @@ -234,7 +252,7 @@ foreach(libsubfolder doc examples templates) -DTOP_SRC_DIR=${TOP_SRC_DIR} -DPERL_EXECUTABLE=${PERL_EXECUTABLE} -P "${TOP_SRC_DIR}/development/autotests/export.cmake") - setmarkedtestlabel(${TestName} ${reverted} "export") + handlesuspended(${TestName} ${reverted} "export") # check for suspended pdf/dvi exports endif() endforeach() endforeach() INV.*pdf4_texF signature.asc Description: This is a digitally signed message part.
Re: status update on the export tests
Am Montag, 2. November 2015 um 08:36:05, schrieb Guenter Milde > Dear Scott and Kornel, > > On 2015-11-02, Scott Kostyshak wrote: > > On Sun, Nov 01, 2015 at 10:36:17PM +0100, Kornel Benko wrote: > >> Am Sonntag, 1. November 2015 um 21:27:11, schrieb Guenter Milde > >> > > >> > Could we introduce a new status "suspended" - meaning just skip the > >> > test until a known bug is solved as we know the result is > >> > insignificant until then? > > > As Kornel mentions, we have "ignoredTests" for this. > > Not exactly, see below. > > >> We already have such (specified in file "ignoredTests"). But as this > >> tests are never executed, nobody cares for them anymore. > >> The tests here are such, that we know, we never resolve them. > > > I agree. We should not use ignored tests for temporary issues. Once we > > put something in ignoredTests chances are strong that we will forget > > about them for a long time. > > This is why I said "suspended": > > Suggestion: > > Specified in file "suspendedTests") with the reason for suspending > (bug report, commit that turned hidden problems into export failure, ...) > > These tests are normally skipped, but they are not forgotten. > > The tests here are such, that we know, we can resolve them but their > failure is a minor issue that can be postponed (comparable to enhancement > requests in Trac). > > Suspending instead of reverting also frees us from the need to reassess > them if a change in the "real life documents" or a fix makes them pass > again. Instead, they could/should be revised at a time we have fixed > major known problems and no release pressure... > > > Candidates for "suspended" tests are > > * "fragile" documents (many packages, ERT, heavy preamble, ...), > * "fragile" export routes (XeTeX/LuaTeX with TeX fonts), > * non-default export routes > > and especially combinations of these. > > > Günter > OK, here is my suggestion 1.) We add the appropriate tests into revertedTests The content in suspendedTests may be (in our case) e.g. pdf4_texF 1.a) If a test is to be inverted, we check suspendedTest, and if there, we ignore the testcase. or 1.b) Such a test may became a label "suspended" instead of "export", so that 'ctest -L export' will be clean, but we also have the possibility to use 'ctest -L suspended'. This does neither effect non-inverted nor ignored. I prefer 1.b. Kornel signature.asc Description: This is a digitally signed message part.
Re: status update on the export tests
On 2015-11-02, Scott Kostyshak wrote: > Good point. On the other hand, if we make regex changes then it is > harder to revert them. Problems have come up before of us > misconstructing the regexes. If possible, make separate regexp patterns for separate "fail reasons". Then, we just have to delete one regexp if the cause for the inversion is gone... Günter
Re: status update on the export tests
Dear Scott and Kornel, On 2015-11-02, Scott Kostyshak wrote: > On Sun, Nov 01, 2015 at 10:36:17PM +0100, Kornel Benko wrote: >> Am Sonntag, 1. November 2015 um 21:27:11, schrieb Guenter Milde >> >> > Could we introduce a new status "suspended" - meaning just skip the >> > test until a known bug is solved as we know the result is >> > insignificant until then? > As Kornel mentions, we have "ignoredTests" for this. Not exactly, see below. >> We already have such (specified in file "ignoredTests"). But as this >> tests are never executed, nobody cares for them anymore. >> The tests here are such, that we know, we never resolve them. > I agree. We should not use ignored tests for temporary issues. Once we > put something in ignoredTests chances are strong that we will forget > about them for a long time. This is why I said "suspended": Suggestion: Specified in file "suspendedTests") with the reason for suspending (bug report, commit that turned hidden problems into export failure, ...) These tests are normally skipped, but they are not forgotten. The tests here are such, that we know, we can resolve them but their failure is a minor issue that can be postponed (comparable to enhancement requests in Trac). Suspending instead of reverting also frees us from the need to reassess them if a change in the "real life documents" or a fix makes them pass again. Instead, they could/should be revised at a time we have fixed major known problems and no release pressure... Candidates for "suspended" tests are * "fragile" documents (many packages, ERT, heavy preamble, ...), * "fragile" export routes (XeTeX/LuaTeX with TeX fonts), * non-default export routes and especially combinations of these. Günter
Re: status update on the export tests
On Sun, Nov 01, 2015 at 10:31:32PM +, Guenter Milde wrote: > On 2015-11-01, Scott Kostyshak wrote: > > On Sun, Nov 01, 2015 at 07:37:41PM +, Guenter Milde wrote: > >> On 2015-10-30, Kornel Benko wrote: > >> > Am Freitag, 30. Oktober 2015 um 19:23:02, schrieb Scott Kostyshak > >> > > > >> > Besides we have ATM about 200 failing export test cases. > > >> Too many to be helpfull. > > > I don't understand this logic. > > For me, the tests are meant to find problems with LyX code. > Also, they should ensure that commits do not break anything important. > > With 200 failing test cases, the test suite starts to resemble the issue > tracker: we can not ensure there is no known bug in LyX, nor can we > ensure that the test suite runs without failure. > > We are classifying bug reports and start with the important ones. > > The same should be done for failing export tests. In order to concentrate > on the important stuff, we need a way to filter less important tests > (maybe just do not run them.) But like the less important bug reports, we > should not remove them altogether without convincing reason. I see what you mean. Thanks for explaining this. I agree. > >> How many of these are for the obscure combination of XeTeX and TeX fonts? > >> While there is a use case for LuaTeX and TeX fonts, I can't see a reason to > >> use XeTeX instead of pdflatex with TeX fonts! > > > The use case that I have in mind is simply that I expect our users might > > use such a combination. If we don't want to support the combination, we > > should output a useful error message explaining why, or just disable the > > combination. If not, users will use it. > > Or treat the combination XeTeX + TeX fonts as a non-standard use case > that we do not prevent but discourage. OK but how do we discourage? In the documentation? What about one of those warning boxes that has the checkbox "don't show this warning again". So the first time a user tries to compile with XeTeX and TeX fonts it warns them "It is not useful to use XeTeX and TeX fonts. You might prefer to use PDF (pdfTeX) or use PDF (XeTeX) but use system fonts by going to Document > Settings > Fonts. > This is similar to, e.g. the "language default (without inputenc)" option > for the inputencoding: this option fails inevitably without additional user > input in the LaTeX preamble. OK but I think it is a stranger case that a user goes into the settings and changes that than just click on the button for PDF (XeTeX). > > Many LyX users do not have the understanding that you do. They do not > > even understand what it means to use non-TeX fonts. They might just > > read something about XeTeX and start using it by clicking on our button > > that says PDF (XeTeX). > > The "inviting" toolbar button starting this obscure and bad-supported > export route is the real problem. Ah yes I agree with this. > This is, basically, what I want in Ticket #9744: the toolbar button would > call XeTeX with Unicode fonts by default. I see. That does seem like a good idea then. > > The purpose of the tests is just to prevent regressions. Sure, maybe it > > is true that something was working by luck before, but I still think it > > is reasonable to try to preserve it. > > Restoring behaviour that was half-broken before and will be of no use > later (except for ill guided one) should not be a precondition for > getting the test suite in a usable shape again. > > The follow-up fixes led to even more "regressions" (or to surfacing of > hidden problems). I see. I did not know it was a similar situation. > >> Solving all issues that arise from this combination is diverting > >> attention and resources from more important tasks. > > > Agreed. > > >> Also, we identified some generic problems with this combination that are > >> not solvable in the short term: third party packages well as documents > >> not prepared for this use case. > > >> Just reverting failing XeTeX export tests for the moment would allow us to > >> concentrate on the remaining failing test and get the test suite in a > >> usable state again. > > > Fine with me. Kornel, OK with you? We could create a bug report so that > > we do not forget about the tests. Then we could just invert all of the > > pdf4_texF tests that are failing. I have in mind just pasting the long > > list of individual tests, rather than using regexes. > > Actually, I'd rather suspend than revert - whether they fail or pass has > currently no predictive value regarding regression or progress in solving > the underlying problems. I see your reasoning now. I will start a new thread about testing where we can discuss this. > I am all for the additon of special test cases. > We should find a way to specify the to-be-tested export route(s) (special > naming or sub-directories, say) for these test documents. I think we all like these special test cases. Hopefully we can implement more of them to avoid many of the fragile tests we have. Sc
Re: status update on the export tests
On Sun, Nov 01, 2015 at 09:39:24PM +0100, Kornel Benko wrote: > I have to be convinced. Fair. I think that whenever tests fail the burden is on those who wish to invert or temporarily ignore the tests. I am not proud to be on the dark side of wanting to invert the tests, but I think that is where I am right now because I think that we need to get clean output so we can catch future regressions. > It is not OK with me, see my other post. > Remark to regexes: > 1.) it is easier for us to read such a file > 2.) but is takes longer time to interpret. > > #ctest -j12 -R 'export/.*(pdf|dvi)' -N | wc > 3085 > ATM, we have 59 lines to identify 300 inverted tests. > Adding the new 200 we have 500 individual lines. > That means we have to check (500/2) * 3085 combinations > to determine which test is inverted and which not. Good point. On the other hand, if we make regex changes then it is harder to revert them. Problems have come up before of us misconstructing the regexes. This is easily fixed though. I might add the lines locally just to see how many clock seconds it increases the cmake call. Scott
Re: status update on the export tests
On Sun, Nov 01, 2015 at 11:05:06PM +0100, Kornel Benko wrote: > My argument is not that they do not belong into the set of reverted, but I'd > like them > to be checked first. > They will be included with the appropriate comment. +1 Scott
Re: status update on the export tests
On Sun, Nov 01, 2015 at 09:48:28PM +, Guenter Milde wrote: > On 2015-11-01, Kornel Benko wrote: > > Most of these tests were working some time ago. > > Many of them "by chance": not failing but with incorrect output (Missing > characters, wrong characters) or "fragile" (working, but would fail by > addition of one non-ASCII character, say). Agreed. I don't think "by chance" correctly describes the missing characters situation, but only the "fragile" documents. > > We already have many inverted test. > > This also adds to the impression, that this is an area where test > failures are to be expected for many reasons. I.e. the signal to noise ratio > is rather bad for XeTeX+TeX fonts and we would be better of without these > tests. This might be. I admit I have very little knowledge of XeTeX + TeX fonts. > > I am strongly against such policy. First one has to check if the reason is > > really > > babel/polyglossia conflict. > > There are many more reasons, mhchem, babel language files, font problems, ... > > The combination XeTeX + TeX-fonts is not supported by many packages, > including standard LaTeX packages like "inputenc"! OTOH, it is so obscure > that it is really not worth wasting much time to work around all the > problems. Rather treat/document it as: avoid if possible, use at your own > risk. I am starting to be open to this idea. If it is true that performing a simple, correct operation in a document leads to a significant chance of it breaking compilation of XeTeX with TeX fonts, then it is true that this could cause a lot of noise. On one hand, our manuals were pretty darn complicated and compiled just fine with TeX fonts. This might be partly due to 'luck' but since they use so many packages and ERT and weird constructs, that would be a lot of luck. I guess the main issue is the missing characters. Scott
Re: status update on the export tests
On Sun, Nov 01, 2015 at 10:36:17PM +0100, Kornel Benko wrote: > Am Sonntag, 1. November 2015 um 21:27:11, schrieb Guenter Milde > > > Could we introduce a new status "suspended" - meaning just skip the test > > until a known bug is solved as we know the result is insignificant until > > then? As Kornel mentions, we have "ignoredTests" for this. > > This way, known problems will not mask new problems. I agree with this goal. The current situation is a problem because like you say it might mask new problems. I agree with Kornel that ignored tests are not the solution. But I think that inverting the tests might be a temporary solution. The only way to get clean working tests would be to rever the commits and I don't think that is the right action for this situation. > We already have such (specified in file "ignoredTests"). But as this tests > are never executed, > nobody cares for them anymore. > The tests here are such, that we know, we never resolve them. > Example: > We write a lyx file for odt/word/whatever output only. There is sense to > expect > valid latex. I agree. We should not use ignored tests for temporary issues. Once we put something in ignoredTests chances are strong that we will forget about them for a long time. Scott
Re: status update on the export tests
On Sun, Nov 01, 2015 at 10:07:02PM +0100, Georg Baum wrote: > Kornel Benko wrote: > > > Does it mean 'delay 2.2 alpha'? > > I don't think so. 2.1 has language nesting bugs as well, and the more simple > cases do work. But we should probably mention that there are known language > nesting bugs in the release notes. It is alpha after all (and I still hope > to get some safe improvement for the beta). I agree with Georg. I think these issues should not block alpha. It might be reasonable to have a policy of "no failing tests" for beta in the future. This doesn't mean that we fix all bugs. It just means that every test failure was looked at. Maybe we looked at a test and decided it is OK for it to fail so we invert it and thus it no longer fails. Scott
Re: status update on the export tests
On Sun, Nov 01, 2015 at 09:14:53PM +0100, Kornel Benko wrote: > Am Sonntag, 1. November 2015 um 19:37:41, schrieb Guenter Milde > > > On 2015-10-30, Kornel Benko wrote: > > Solving all issues that arise from this combination is diverting attention > > and ressources from more important tasks. > > Most of these tests were working some time ago. We already have many inverted > test. > I am strongly against such policy. First one has to check if the reason is > really > babel/polyglossia conflict. I agree that we should check before reverting. Actually I don't think babel/polyglossia plays a role in most of the failing tests. There are only a few that fail because of this (you can see this by reverting the polyglossia commit, edd37de8). Georg has already done a good job of improving things and only a few affected tests remain. The main issue is the missing characters commit, I think. And this is a tricky issue because the commit that broke the tests was actually a good commit---it did not introduce a bug. What to do in such a case? I'm not sure. Ideally we would have a separate branch where we try to address the tests (either fixing the underlying bugs or fixing the .lyx files or inverting the tests). This way the tests on master continue to be clean and it is easy to see if there are any regressions. But I fear this is too much of an inconvenience to people who already are annoyed by the tests. > > Also, we identified some generic problems with this combination that are > > not solvable in the short term: third party packages well as documents > > not prepared for this use case. > > > > Just reverting failing Xetex export tests for the moment would allow us to > > concentrfate on the remaining failing test and get the test suite in a > > usable state again. > > Optimist (I mean 'usable state'). > I am strongly against such policy. First one has to check if the reason is > really > babel/polyglossia conflict. > There are already too many tests inverted, no one cares anymore. Actually the last few weeks have seen the most interest in the tests yet. Other developers are trying to run the tests, which is already an improvement. I would say this is the height of interest in the export tests :) Scott
Re: status update on the export tests
On 2015-11-01, Scott Kostyshak wrote: > On Sun, Nov 01, 2015 at 07:37:41PM +, Guenter Milde wrote: >> On 2015-10-30, Kornel Benko wrote: >> > Am Freitag, 30. Oktober 2015 um 19:23:02, schrieb Scott Kostyshak >> > >> > Besides we have ATM about 200 failing export test cases. >> Too many to be helpfull. > I don't understand this logic. For me, the tests are meant to find problems with LyX code. Also, they should ensure that commits do not break anything important. With 200 failing test cases, the test suite starts to resemble the issue tracker: we can not ensure there is no known bug in LyX, nor can we ensure that the test suite runs without failure. We are classifying bug reports and start with the important ones. The same should be done for failing export tests. In order to concentrate on the important stuff, we need a way to filter less important tests (maybe just do not run them.) But like the less important bug reports, we should not remove them altogether without convincing reason. ... >> How many of these are for the obscure combination of XeTeX and TeX fonts? >> While there is a use case for LuaTeX and TeX fonts, I can't see a reason to >> use XeTeX instead of pdflatex with TeX fonts! > The use case that I have in mind is simply that I expect our users might > use such a combination. If we don't want to support the combination, we > should output a useful error message explaining why, or just disable the > combination. If not, users will use it. Or treat the combination XeTeX + TeX fonts as a non-standard use case that we do not prevent but discourage. This is similar to, e.g. the "language default (without inputenc)" option for the inputencoding: this option fails inevitably without additional user input in the LaTeX preamble. > Many LyX users do not have the understanding that you do. They do not > even understand what it means to use non-TeX fonts. They might just > read something about XeTeX and start using it by clicking on our button > that says PDF (XeTeX). The "inviting" toolbar button starting this obscure and bad-supported export route is the real problem. This is, basically, what I want in Ticket #9744: the toolbar button would call XeTeX with Unicode fonts by default. > The purpose of the tests is just to prevent regressions. Sure, maybe it > is true that something was working by luck before, but I still think it > is reasonable to try to preserve it. Restoring behaviour that was half-broken before and will be of no use later (except for ill guided one) should not be a precondition for getting the test suite in a usable shape again. > So I defend the tests in general. The current situation is a special > case. You fixed a bug (the missing characters bug) I just reported it but did not fix it. > and that is why so many tests are failing. I do not imagine such a > situation coming up often. The follow-up fixes led to even more "regressions" (or to surfacing of hidden problems). >> Solving all issues that arise from this combination is diverting >> attention and resources from more important tasks. > Agreed. >> Also, we identified some generic problems with this combination that are >> not solvable in the short term: third party packages well as documents >> not prepared for this use case. >> Just reverting failing XeTeX export tests for the moment would allow us to >> concentrate on the remaining failing test and get the test suite in a >> usable state again. > Fine with me. Kornel, OK with you? We could create a bug report so that > we do not forget about the tests. Then we could just invert all of the > pdf4_texF tests that are failing. I have in mind just pasting the long > list of individual tests, rather than using regexes. Actually, I'd rather suspend than revert - whether they fail or pass has currently no predictive value regarding regression or progress in solving the underlying problems. I am all for the additon of special test cases. We should find a way to specify the to-be-tested export route(s) (special naming or sub-directories, say) for these test documents. Günter
Re: status update on the export tests
Am Sonntag, 1. November 2015 um 21:48:28, schrieb Guenter Milde > On 2015-11-01, Kornel Benko wrote: > > Am Sonntag, 1. November 2015 um 19:37:41, schrieb Guenter Milde > > > >> On 2015-10-30, Kornel Benko wrote: > > ... > > >> > Besides we have ATM about 200 failing export test cases. > > ... > > >> How many of these are for the obscure combination of Xetex and Tex fonts? > >> While there is a use case for LuaTeX and TeX fonts, I can't see a reason to > >> use Xetex instead of pdflatex with TeX fonts! > > > 316 tests are XeTex + system font, 52 (not inverted) of them fail, 42 > > inverted > > 316 tests are XeTex + tex font, 12 (not inverted) of them fail, 54 inverted > > >> Solving all issues that arise from this combination is diverting attention > >> and ressources from more important tasks. > > > Most of these tests were working some time ago. > > Many of them "by chance": not failing but with incorrect output (Missing > characters, wrong characters) or "fragile" (working, but would fail by > addition of one non-ASCII character, say). Sure, but we have to check, ... It is not done by simply declaring them to be inverted. > > We already have many inverted test. > > This also adds to the impression, that this is an area where test > failures are to be expected for many reasons. I.e. the signal to noise ratio > is rather bad for XeTeX+TeX fonts and we would be better of without these > tests. > > > I am strongly against such policy. First one has to check if the reason is > > really > > babel/polyglossia conflict. > > There are many more reasons, mhchem, babel language files, font problems, ... Yes, and I am grateful too for what you have done in this direction. > The combination XeTeX + TeX-fonts is not supported by many packages, > including standard LaTeX packages like "inputenc"! OTOH, it is so obscure > that it is really not worth wasting much time to work around all the > problems. Rather treat/document it as: avoid if possible, use at your own > risk. My argument is not that they do not belong into the set of reverted, but I'd like them to be checked first. They will be included with the appropriate comment. > Günter Kornel signature.asc Description: This is a digitally signed message part.
Re: status update on the export tests
On 2015-11-01, Kornel Benko wrote: > Am Sonntag, 1. November 2015 um 19:37:41, schrieb Guenter Milde > >> On 2015-10-30, Kornel Benko wrote: ... >> > Besides we have ATM about 200 failing export test cases. ... >> How many of these are for the obscure combination of Xetex and Tex fonts? >> While there is a use case for LuaTeX and TeX fonts, I can't see a reason to >> use Xetex instead of pdflatex with TeX fonts! > 316 tests are XeTex + system font, 52 (not inverted) of them fail, 42 inverted > 316 tests are XeTex + tex font, 12 (not inverted) of them fail, 54 inverted >> Solving all issues that arise from this combination is diverting attention >> and ressources from more important tasks. > Most of these tests were working some time ago. Many of them "by chance": not failing but with incorrect output (Missing characters, wrong characters) or "fragile" (working, but would fail by addition of one non-ASCII character, say). > We already have many inverted test. This also adds to the impression, that this is an area where test failures are to be expected for many reasons. I.e. the signal to noise ratio is rather bad for XeTeX+TeX fonts and we would be better of without these tests. > I am strongly against such policy. First one has to check if the reason is > really > babel/polyglossia conflict. There are many more reasons, mhchem, babel language files, font problems, ... The combination XeTeX + TeX-fonts is not supported by many packages, including standard LaTeX packages like "inputenc"! OTOH, it is so obscure that it is really not worth wasting much time to work around all the problems. Rather treat/document it as: avoid if possible, use at your own risk. Günter
Re: status update on the export tests
Am Sonntag, 1. November 2015 um 21:27:11, schrieb Guenter Milde > On 2015-11-01, Georg Baum wrote: > > Kornel Benko wrote: > > >> Optimist (I mean 'usable state'). > >> I am strongly against such policy. First one has to check if the reason is > >> really babel/polyglossia conflict. > >> There are already too many tests inverted, no one cares anymore. > > > We have a chicken and egg problem here. I started to work on the language > > nesting stuff, and fixing this without introducing regressions is > > impossible > > without creating specific test cases first. So, for this particular area of > > the code, the tests are already unusable, and we cannot get it into a good > > state again without usable tests... > > > In principle I am with Kornel here, but in the current situation I think > > the > > only option we have is to set up more specific tests, fix the code one bug > > at a time, and then begin to look at the automatic export tests again. In > > the mean time, I don't care too much whether the failing tests are inverted > > or not. > > Could we introduce a new status "suspended" - meaning just skip the test > until a known bug is solved as we know the result is insignificant until then? > > This way, known problems will not mask new problems. We already have such (specified in file "ignoredTests"). But as this tests are never executed, nobody cares for them anymore. The tests here are such, that we know, we never resolve them. Example: We write a lyx file for odt/word/whatever output only. There is sense to expect valid latex. > Günter Kornel signature.asc Description: This is a digitally signed message part.
Re: status update on the export tests
On 2015-11-01, Georg Baum wrote: > Kornel Benko wrote: >> Optimist (I mean 'usable state'). >> I am strongly against such policy. First one has to check if the reason is >> really babel/polyglossia conflict. >> There are already too many tests inverted, no one cares anymore. > We have a chicken and egg problem here. I started to work on the language > nesting stuff, and fixing this without introducing regressions is impossible > without creating specific test cases first. So, for this particular area of > the code, the tests are already unusable, and we cannot get it into a good > state again without usable tests... > In principle I am with Kornel here, but in the current situation I think the > only option we have is to set up more specific tests, fix the code one bug > at a time, and then begin to look at the automatic export tests again. In > the mean time, I don't care too much whether the failing tests are inverted > or not. Could we introduce a new status "suspended" - meaning just skip the test until a known bug is solved as we know the result is insignificant until then? This way, known problems will not mask new problems. Günter
Re: status update on the export tests
Kornel Benko wrote: > Am Sonntag, 1. November 2015 um 21:46:00, schrieb Georg Baum > >> >> We have a chicken and egg problem here. I started to work on the language >> nesting stuff, and fixing this without introducing regressions is >> impossible without creating specific test cases first. So, for this >> particular area of the code, the tests are already unusable, and we >> cannot get it into a good state again without usable tests... > > It is not only you. If our tests were usable at that time, then it would > have been easy to check and set the appropriate tests in reverted drawer. Definitely. >> In principle I am with Kornel here, but in the current situation I think >> the only option we have is to set up more specific tests, fix the code >> one bug at a time, and then begin to look at the automatic export tests >> again. In the mean time, I don't care too much whether the failing tests >> are inverted or not. > > > Does it mean 'delay 2.2 alpha'? I don't think so. 2.1 has language nesting bugs as well, and the more simple cases do work. But we should probably mention that there are known language nesting bugs in the release notes. It is alpha after all (and I still hope to get some safe improvement for the beta). Georg
Re: status update on the export tests
Am Sonntag, 1. November 2015 um 21:46:00, schrieb Georg Baum > Kornel Benko wrote: > > > Optimist (I mean 'usable state'). > > I am strongly against such policy. First one has to check if the reason is > > really babel/polyglossia conflict. > > There are already too many tests inverted, no one cares anymore. > > We have a chicken and egg problem here. I started to work on the language > nesting stuff, and fixing this without introducing regressions is impossible > without creating specific test cases first. So, for this particular area of > the code, the tests are already unusable, and we cannot get it into a good > state again without usable tests... It is not only you. If our tests were usable at that time, then it would have been easy to check and set the appropriate tests in reverted drawer. > In principle I am with Kornel here, but in the current situation I think the > only option we have is to set up more specific tests, fix the code one bug > at a time, and then begin to look at the automatic export tests again. In > the mean time, I don't care too much whether the failing tests are inverted > or not. Does it mean 'delay 2.2 alpha'? > Georg Kornel signature.asc Description: This is a digitally signed message part.
Re: status update on the export tests
Kornel Benko wrote: > Optimist (I mean 'usable state'). > I am strongly against such policy. First one has to check if the reason is > really babel/polyglossia conflict. > There are already too many tests inverted, no one cares anymore. We have a chicken and egg problem here. I started to work on the language nesting stuff, and fixing this without introducing regressions is impossible without creating specific test cases first. So, for this particular area of the code, the tests are already unusable, and we cannot get it into a good state again without usable tests... In principle I am with Kornel here, but in the current situation I think the only option we have is to set up more specific tests, fix the code one bug at a time, and then begin to look at the automatic export tests again. In the mean time, I don't care too much whether the failing tests are inverted or not. Georg
Re: status update on the export tests
Am Sonntag, 1. November 2015 um 15:00:25, schrieb Scott Kostyshak > On Sun, Nov 01, 2015 at 07:37:41PM +, Guenter Milde wrote: > > On 2015-10-30, Kornel Benko wrote: > > > Am Freitag, 30. Oktober 2015 um 19:23:02, schrieb Scott Kostyshak > > > > > > Besides we have ATM about 200 failing export test cases. > > > > Too many to be helpfull. > > I don't understand this logic. > > > > The summary contains lines like: > > > Label Time Summary: > > > export = 59316.83 sec (3753 tests) > > > key = 0.26 sec (1 test) > > > reverted= 5631.52 sec (312 tests) > > > > > Even if we label some tests, the summary does not specify > > > how mane tests went wrong for a specified label. > > > > How many of these are for the obscure combination of Xetex and Tex fonts? > > While there is a use case for LuaTeX and TeX fonts, I can't see a reason to > > use Xetex instead of pdflatex with TeX fonts! > > The use case that I have in mind is simply that I expect our users might > use such a combination. If we don't want to support the combination, we > should output a useful error message explaining why, or just disable the > combination. If not, users will use it. Many LyX users do not have the > understanding that you do. They do not even understand what it means to > use non-TeX fonts. They might just read something about XeTeX and start > using it by clicking on our button that says PDF (XeTeX). The purpose of > the tests is just to prevent regressions. Sure, maybe it is true that > something was working by luck before, but I still think it is reasonable > to try to preserve it. So I defend the tests in general. The current > situation is a special case. You fixed a bug (the missing characters > bug) and that is why so many tests are failing. I do not imagine such a > situation coming up often. > > > Solving all issues that arise from this combination is diverting attention > > and ressources from more important tasks. > > Agreed. > > > Also, we identified some generic problems with this combination that are > > not solvable in the short term: third party packages well as documents > > not prepared for this use case. > > > > Just reverting failing Xetex export tests for the moment would allow us to > > concentrfate on the remaining failing test and get the test suite in a > > usable state again. > > Fine with me. Kornel, OK with you? We could create a bug report so that > we do not forget about the tests. Then we could just invert all of the > pdf4_texF tests that are failing. I have in mind just pasting the long > list of individual tests, rather than using regexes. I have to be convinced. It is not OK with me, see my other post. Remark to regexes: 1.) it is easier for us to read such a file 2.) but is takes longer time to interpret. #ctest -j12 -R 'export/.*(pdf|dvi)' -N | wc 3085 ATM, we have 59 lines to identify 300 inverted tests. Adding the new 200 we have 500 individual lines. That means we have to check (500/2) * 3085 combinations to determine which test is inverted and which not. ATM, we have 59 lines to identify 300 inverted tests. > Scott Kornel signature.asc Description: This is a digitally signed message part.
Re: status update on the export tests
Am Sonntag, 1. November 2015 um 19:37:41, schrieb Guenter Milde > On 2015-10-30, Kornel Benko wrote: > > Am Freitag, 30. Oktober 2015 um 19:23:02, schrieb Scott Kostyshak > > > >> On Fri, Oct 30, 2015 at 11:11:52PM +, Guenter Milde wrote: > >> > On 2015-10-28, Scott Kostyshak wrote: > >> > > On Mon, Oct 26, 2015 at 03:33:22PM +, Guenter Milde wrote: > >> > > >> > ... > >> > > >> > >> However, if the exit status of an export test changes (from fail > >> > >> to pass or vice versa), we should check whether this is due to a > >> > >> new bug, a fix or just exposing previously hidden problems. > >> > > >> > > Agreed. In fact, sometimes it is an improvement when a test fails. > >> > > When I check manually, it was passing before but showing garbled > >> > > text in a PDF output. Now it might fail with a clear message of > >> > > "language xyz not supported". It is always good to check manually > >> > > why something fails and if it passes if the PDF output is good. > >> > > >> > And, instead of re-establishing this with every status change, we > >> > could have "tags" for inverted tests, distinguishing: > >> > > >> > * failure because of known permanent incompatiblity (good failure) > >> > e.g. lyx -e latex ... for a document using non-TeX fonts > >> > > >> > * failure because of ERT or preamble code (not LyX's fault) > >> > > >> > * failure because of upstream bugs > >> > > >> > * failure because of known LyX bugs > >> > > >> > * failure for unknown reason with non-standard export route > >> > e.g. XeTeX and TeX-fonts > > >> Yes this would be nice. Right now I just try to put that information > >> as a comment for why we invert a test, but it would be nice to have > >> that information more easily available in the test summary. > > > I don't know how such an info can go to a summary. > > We could have separate files for the tests that we revert for different > reasons. Or use a set of keywords. > > > Besides we have ATM about 200 failing export test cases. > > Too many to be helpfull. > > > The summary contains lines like: > > Label Time Summary: > > export = 59316.83 sec (3753 tests) > > key = 0.26 sec (1 test) > > reverted= 5631.52 sec (312 tests) > > > Even if we label some tests, the summary does not specify > > how mane tests went wrong for a specified label. > > How many of these are for the obscure combination of Xetex and Tex fonts? > While there is a use case for LuaTeX and TeX fonts, I can't see a reason to > use Xetex instead of pdflatex with TeX fonts! 316 tests are XeTex + system font, 52 (not inverted) of them fail, 42 inverted 316 tests are XeTex + tex font, 12 (not inverted) of them fail, 54 inverted > Solving all issues that arise from this combination is diverting attention > and ressources from more important tasks. Most of these tests were working some time ago. We already have many inverted test. I am strongly against such policy. First one has to check if the reason is really babel/polyglossia conflict. > Also, we identified some generic problems with this combination that are > not solvable in the short term: third party packages well as documents > not prepared for this use case. > > Just reverting failing Xetex export tests for the moment would allow us to > concentrfate on the remaining failing test and get the test suite in a > usable state again. Optimist (I mean 'usable state'). I am strongly against such policy. First one has to check if the reason is really babel/polyglossia conflict. There are already too many tests inverted, no one cares anymore. > Günter > Kornel signature.asc Description: This is a digitally signed message part.
Re: status update on the export tests
On Sun, Nov 01, 2015 at 07:37:41PM +, Guenter Milde wrote: > On 2015-10-30, Kornel Benko wrote: > > Am Freitag, 30. Oktober 2015 um 19:23:02, schrieb Scott Kostyshak > > > > Besides we have ATM about 200 failing export test cases. > > Too many to be helpfull. I don't understand this logic. > > The summary contains lines like: > > Label Time Summary: > > export = 59316.83 sec (3753 tests) > > key = 0.26 sec (1 test) > > reverted= 5631.52 sec (312 tests) > > > Even if we label some tests, the summary does not specify > > how mane tests went wrong for a specified label. > > How many of these are for the obscure combination of Xetex and Tex fonts? > While there is a use case for LuaTeX and TeX fonts, I can't see a reason to > use Xetex instead of pdflatex with TeX fonts! The use case that I have in mind is simply that I expect our users might use such a combination. If we don't want to support the combination, we should output a useful error message explaining why, or just disable the combination. If not, users will use it. Many LyX users do not have the understanding that you do. They do not even understand what it means to use non-TeX fonts. They might just read something about XeTeX and start using it by clicking on our button that says PDF (XeTeX). The purpose of the tests is just to prevent regressions. Sure, maybe it is true that something was working by luck before, but I still think it is reasonable to try to preserve it. So I defend the tests in general. The current situation is a special case. You fixed a bug (the missing characters bug) and that is why so many tests are failing. I do not imagine such a situation coming up often. > Solving all issues that arise from this combination is diverting attention > and ressources from more important tasks. Agreed. > Also, we identified some generic problems with this combination that are > not solvable in the short term: third party packages well as documents > not prepared for this use case. > > Just reverting failing Xetex export tests for the moment would allow us to > concentrfate on the remaining failing test and get the test suite in a > usable state again. Fine with me. Kornel, OK with you? We could create a bug report so that we do not forget about the tests. Then we could just invert all of the pdf4_texF tests that are failing. I have in mind just pasting the long list of individual tests, rather than using regexes. Scott
Re: status update on the export tests
On 2015-10-30, Kornel Benko wrote: > Am Freitag, 30. Oktober 2015 um 19:23:02, schrieb Scott Kostyshak > >> On Fri, Oct 30, 2015 at 11:11:52PM +, Guenter Milde wrote: >> > On 2015-10-28, Scott Kostyshak wrote: >> > > On Mon, Oct 26, 2015 at 03:33:22PM +, Guenter Milde wrote: >> > >> > ... >> > >> > >> However, if the exit status of an export test changes (from fail >> > >> to pass or vice versa), we should check whether this is due to a >> > >> new bug, a fix or just exposing previously hidden problems. >> > >> > > Agreed. In fact, sometimes it is an improvement when a test fails. >> > > When I check manually, it was passing before but showing garbled >> > > text in a PDF output. Now it might fail with a clear message of >> > > "language xyz not supported". It is always good to check manually >> > > why something fails and if it passes if the PDF output is good. >> > >> > And, instead of re-establishing this with every status change, we >> > could have "tags" for inverted tests, distinguishing: >> > >> > * failure because of known permanent incompatiblity (good failure) >> > e.g. lyx -e latex ... for a document using non-TeX fonts >> > >> > * failure because of ERT or preamble code (not LyX's fault) >> > >> > * failure because of upstream bugs >> > >> > * failure because of known LyX bugs >> > >> > * failure for unknown reason with non-standard export route >> > e.g. XeTeX and TeX-fonts >> Yes this would be nice. Right now I just try to put that information >> as a comment for why we invert a test, but it would be nice to have >> that information more easily available in the test summary. > I don't know how such an info can go to a summary. We could have separate files for the tests that we revert for different reasons. Or use a set of keywords. > Besides we have ATM about 200 failing export test cases. Too many to be helpfull. > The summary contains lines like: > Label Time Summary: > export = 59316.83 sec (3753 tests) > key = 0.26 sec (1 test) > reverted= 5631.52 sec (312 tests) > Even if we label some tests, the summary does not specify > how mane tests went wrong for a specified label. How many of these are for the obscure combination of Xetex and Tex fonts? While there is a use case for LuaTeX and TeX fonts, I can't see a reason to use Xetex instead of pdflatex with TeX fonts! Solving all issues that arise from this combination is diverting attention and ressources from more important tasks. Also, we identified some generic problems with this combination that are not solvable in the short term: third party packages well as documents not prepared for this use case. Just reverting failing Xetex export tests for the moment would allow us to concentrfate on the remaining failing test and get the test suite in a usable state again. Günter
Re: status update on the export tests
On Sat, Oct 31, 2015 at 12:14:39PM +0100, Jürgen Spitzmüller wrote: > Am Mittwoch 28 Oktober 2015, 11:31:38 schrieb Scott Kostyshak: > > > It may be that we need a new keyword for lib/languages, to tag > > > languages where Babel is superior to Polyglossia also with non-TeX fonts. > > > > > > Then, the "automatic" setting (which is also the default "default") could > > > take into account that some languages should use Babel also with non-TeX > > > fonts even if Polyglossia is available for them. > > > > Jürgen what do you think about this idea? > > Yes, this could work, but it is certainly not 2.2 stuff. Sounds good. I created a ticket for milestone 2.3: http://www.lyx.org/trac/ticket/9832 I don't think we would want to change between Polyglossia and Babel during 2.2.x (except I suppose if one is completely incompatible with a language). Scott
Re: status update on the export tests
Am Mittwoch 28 Oktober 2015, 11:31:38 schrieb Scott Kostyshak: > > It may be that we need a new keyword for lib/languages, to tag > > languages where Babel is superior to Polyglossia also with non-TeX fonts. > > > > Then, the "automatic" setting (which is also the default "default") could > > take into account that some languages should use Babel also with non-TeX > > fonts even if Polyglossia is available for them. > > Jürgen what do you think about this idea? Yes, this could work, but it is certainly not 2.2 stuff. Jürgen
Re: status update on the export tests
Am Freitag, 30. Oktober 2015 um 19:23:02, schrieb Scott Kostyshak > On Fri, Oct 30, 2015 at 11:11:52PM +, Guenter Milde wrote: > > On 2015-10-28, Scott Kostyshak wrote: > > > On Mon, Oct 26, 2015 at 03:33:22PM +, Guenter Milde wrote: > > > > ... > > > > >> However, if the exit status of an export test changes (from fail to > > >> pass or vice versa), we should check whether this is due to a new bug, > > >> a fix or just exposing previously hidden problems. > > > > > Agreed. In fact, sometimes it is an improvement when a test fails. When > > > I check manually, it was passing before but showing garbled text in a > > > PDF output. Now it might fail with a clear message of "language xyz not > > > supported". It is always good to check manually why something fails and > > > if it passes if the PDF output is good. > > > > And, instead of re-establishing this with every status change, we could > > have "tags" for inverted tests, distinguishing: > > > > * failure because of known permanent incompatiblity (good failure) > > e.g. lyx -e latex ... for a document using non-TeX fonts > > > > * failure because of ERT or preamble code (not LyX's fault) > > > > * failure because of upstream bugs > > > > * failure because of known LyX bugs > > > > * failure for unknown reason with non-standard export route > > e.g. XeTeX and TeX-fonts > > Yes this would be nice. Right now I just try to put that information as > a comment for why we invert a test, but it would be nice to have that > information more easily available in the test summary. I don't know how such an info can go to a summary. Besides we have ATM about 200 failing export test cases. The summary contains lines like: Label Time Summary: export = 59316.83 sec (3753 tests) key = 0.26 sec (1 test) reverted= 5631.52 sec (312 tests) Even if we label some tests, the summary does not specify how mane tests went wrong for a specified label. > Scott Kornel signature.asc Description: This is a digitally signed message part.
Re: status update on the export tests
On Fri, Oct 30, 2015 at 11:11:52PM +, Guenter Milde wrote: > On 2015-10-28, Scott Kostyshak wrote: > > On Mon, Oct 26, 2015 at 03:33:22PM +, Guenter Milde wrote: > > ... > > >> However, if the exit status of an export test changes (from fail to > >> pass or vice versa), we should check whether this is due to a new bug, > >> a fix or just exposing previously hidden problems. > > > Agreed. In fact, sometimes it is an improvement when a test fails. When > > I check manually, it was passing before but showing garbled text in a > > PDF output. Now it might fail with a clear message of "language xyz not > > supported". It is always good to check manually why something fails and > > if it passes if the PDF output is good. > > And, instead of re-establishing this with every status change, we could > have "tags" for inverted tests, distinguishing: > > * failure because of known permanent incompatiblity (good failure) > e.g. lyx -e latex ... for a document using non-TeX fonts > > * failure because of ERT or preamble code (not LyX's fault) > > * failure because of upstream bugs > > * failure because of known LyX bugs > > * failure for unknown reason with non-standard export route > e.g. XeTeX and TeX-fonts Yes this would be nice. Right now I just try to put that information as a comment for why we invert a test, but it would be nice to have that information more easily available in the test summary. Scott
Re: status update on the export tests
On 2015-10-28, Scott Kostyshak wrote: > On Mon, Oct 26, 2015 at 03:33:22PM +, Guenter Milde wrote: ... >> However, if the exit status of an export test changes (from fail to >> pass or vice versa), we should check whether this is due to a new bug, >> a fix or just exposing previously hidden problems. > Agreed. In fact, sometimes it is an improvement when a test fails. When > I check manually, it was passing before but showing garbled text in a > PDF output. Now it might fail with a clear message of "language xyz not > supported". It is always good to check manually why something fails and > if it passes if the PDF output is good. And, instead of re-establishing this with every status change, we could have "tags" for inverted tests, distinguishing: * failure because of known permanent incompatiblity (good failure) e.g. lyx -e latex ... for a document using non-TeX fonts * failure because of ERT or preamble code (not LyX's fault) * failure because of upstream bugs * failure because of known LyX bugs * failure for unknown reason with non-standard export route e.g. XeTeX and TeX-fonts Günter
Re: status update on the export tests
On Thu, Oct 29, 2015 at 10:36:57PM +0100, Georg Baum wrote: > Unit tests are almost always worth the effort. When I implement one, it does > almost always find a bug right away. See also the check_lstrings failure > Vincent reported and http://www.lyx.org/trac/ticket/9799. Having these bugs > found by users is a much worse alternative. > > The problem we have with unit tests is that we do not have a good > infrastructure: It is too much overhead to add one. But once this is > improved, it would be really worth it to add lots of unit tests. Note that Vincent has already done some work on this. See: https://www.mail-archive.com/search?l=mid&q=CAO7dr0gJu1hz%2Bmg7c1N74DH-bqiEAaY0tkS4wK%2BFUEEMmFBFSw%40mail.gmail.com or for a quick link: http://git.lyx.org/?p=developers/vfr/lyx.git;a=shortlog;h=refs/heads/tests Scott
Re: status update on the export tests
Scott Kostyshak wrote: > On Mon, Oct 26, 2015 at 03:33:22PM +, Guenter Milde wrote: > >> I think we have to distinguish several test methods: >> >> a) automatic export tests with lots of "real life" documents (manuals, >>templates, examples) >> >> b) functional tests: import/export of a complete test document and >> comparing >>it to the expected output >> >> c) unit tests: test of certain features or sub-units. >> >> Unit tests (c) are currently not implemented, AFAIK. We have some, but they are not really documented. See the test subdirectories in src/ and below. > There has been some little talk of implementing them but nothing in the > end was done. From what I understand, it would be a huge change and it > wasn't clear whether it was worth the effort and if everyone would use > the tests. Unit tests are almost always worth the effort. When I implement one, it does almost always find a bug right away. See also the check_lstrings failure Vincent reported and http://www.lyx.org/trac/ticket/9799. Having these bugs found by users is a much worse alternative. The problem we have with unit tests is that we do not have a good infrastructure: It is too much overhead to add one. But once this is improved, it would be really worth it to add lots of unit tests. Georg
Re: status update on the export tests
On Tue, Oct 27, 2015 at 10:46:58AM +0100, Stephan Witt wrote: > > Yes, I understood this. But for the first try I didn't want to run it in > parallel. > I've observed that e.g. kpsewhich is used heavily while testing and one call > took ~10ms. > Perhaps there is some tuning possible? My guess is that there's a lot of tuning to be done. Kornel has already improved a lot (thanks to him we can run tests in parallel) but I'm guessing that you're right that some tweaks could improve things. > But I'd guess, there is a significant overhead > for the first time to generate all the required pieces like compiled fonts > etc. > To check this now I'm starting the test again. Ah interesting. I would be interested in whether the second run was faster. I find that 11 hours is very slow even with just one core. If it is expected to be slow on the first run, it would be nice to document this (I'm currently working on a patch for Development.lyx). > 1st question: wouldn't it be useful to split the tests into groups and > enable the tests which should pass on a basic system automatically? > How could this be done? I don't think it's good to have to enable the > export tests explicitly. Yes this would be nice. In addition to Kornel's point that it increases configuration time, it increases the time it takes to run the tests, which is understandably a reason why many do not run the tests. > 2nd question: which tests should succeed with a "normal" TeX installation? Would be good to know. If you give me an idea of what you mean by a "normal" TeX installation I can tell you (I'll set up a virtual machine). Would a full TeX Live installation be what you have in mind or do you mean something more basic? Scott
Re: status update on the export tests
On Mon, Oct 26, 2015 at 09:24:23AM +, Guenter Milde wrote: > On 2015-10-26, Scott Kostyshak wrote: > > > OK. I just want to make sure we define "there are no regressions". These > > tests passed before edd37de8. So it is quite possible that documents > > that users have that compile with LyX 2.1.x suddenly fail to compile > > with LyX 2.2.0 because of edd37de8. In this sense, there is a > > regression. > > > So there is a trade-off: the bad side is that there is the possibility > > that user's documents suddenly fail to compile. The good side is that we > > now encourage the use of polyglossia by default which apparently has > > advantages over babel. Jürgen knowingly made this decision and he knows > > more than I do to be able to judge this tradeoff so I hesitantly accept > > it. > > Polyglossia has been the better (and sometimes only) choice for use with > Xe/LuaTeX Unicode fonts for a long time, but Babel is catching up in the > last years. Good to know. > It may be that we need a new keyword for lib/languages, to tag > languages where Babel is superior to Polyglossia also with non-TeX fonts. > > Then, the "automatic" setting (which is also the default "default") could > take into account that some languages should use Babel also with non-TeX > fonts even if Polyglossia is available for them. Jürgen what do you think about this idea? > Could this prevent some of the regressions? Yes, but possibly only because there are underlying bugs in our language handling. Once those are fixed, I'm not sure if this would fix current regressions or not. The following are the regressions (as indicated by the tests) that I'm not sure will be fixed once LyX's language handling is fixed. In other words, I'm not sure if these fail because we use polyglossia incorrectly or because there is no way to use polyglossia correctly in these situations: export/examples/seminar_pdf5_systemF export/examples/fr/seminar_pdf5_systemF export/doc/fr/UserGuide_pdf5_systemF export/examples/fr/Foils_pdf5_systemF # kluwer gives the following error with polyglossia: # LaTeX Error: Command \ifdef already defined. export/templates/kluwer_pdf5_systemF Regardless of fixing regressions, it still seems like a good idea though. > (We need to look carefully, not > only if the relevant documents compile without error, but also if the > exported document is OK.) Agreed. Scott
Re: status update on the export tests
On Mon, Oct 26, 2015 at 03:33:22PM +, Guenter Milde wrote: > On 2015-10-26, Liviu Andronic wrote: > > On Mon, Oct 26, 2015 at 10:24 AM, Guenter Milde wrote: > >> On 2015-10-26, Scott Kostyshak wrote: > > ... > > >> Could this prevent some of the regressions? (We need to look carefully, not > >> only if the relevant documents compile without error, but also if the > >> exported document is OK.) > > > By exported document do you mean .tex or .pdf? If it is .tex, would it > > be a good idea to check whether the latest export is identical to a > > reference .tex generated when creating the test; if not, display a > > diff. There are similar things Kornel and I have discussed. We would like to have tests that do what you suggest for lyx2lyx. For example, we first export to .tex, then we convert the .lyx file to an older version then convert it back and export to .tex again and see if the files differ. This is nice because it is quicker than compilation, and as you say it might catch bugs that relying on exit code would not. > > Simply relying on the exit code seems like an easy way to miss > > non-error generating regressions... Agreed. > I think we have to distinguish several test methods: > > a) automatic export tests with lots of "real life" documents (manuals, >templates, examples) > > b) functional tests: import/export of a complete test document and comparing >it to the expected output > > c) unit tests: test of certain features or sub-units. > > Unit tests (c) are currently not implemented, AFAIK. There has been some little talk of implementing them but nothing in the end was done. From what I understand, it would be a huge change and it wasn't clear whether it was worth the effort and if everyone would use the tests. > The tex2lyx tests are "functional tests" (b), where we keep track of the > expected output. They include also export tests (in the "round trip" suite). > Here, we have to manually check and update the "expected" output documents > discriminating between intended changes and regressions/bug-indicators. > > For a), it would be a pain to keep track of and update all the output > documents, because this would not only be required for different export > routines but also for changes in the input docs. However, if the exit > status of a test changes (from fail to pass or vice versa), we should > check whether this is due to a new bug, a fix or just exposing previously > hidden problems. Agreed. In fact, sometimes it is an improvement when a test fails. When I check manually, it was passing before but showing garbled text in a PDF output. Now it might fail with a clear message of "language xyz not supported". It is always good to check manually why something fails and if it passes if the PDF output is good. Scott
Re: status update on the export tests
On 2015-10-27, Georg Baum wrote: > Guenter Milde wrote: >> A proper fix would be to use lib/unicodesymbols for the content of the >> hypersetup. > It has been discussed in the past to use lib/unicodesymbols replacements for > the preamble, but there was some important reason not to do it (which I > forgot unfortunately). Searching the list archives or trac should for > unicodesymbols and preamble should hopefully find the discussion. OK. For XeTeX, this is not required, as the PDF-Info does not use TeX-fonts and hence utf8 works fine. For "latexencoding"="ascii", we need one of the tree options: a) \usepackage[ascii]{inputenc} (and \inputencoding{utf8} for the PDF-info). +1 simple -1 breaks documents that clash with inputenc b) use lib/unicodesymbols +1 no loading of "inputenc" package, as currently +1 support for more characters than \inputenc{utf8} with hyperref (e.g. polytonic Greek). -1 pdf strings expect a different input, many lib/unicodesymbols replacements do not work! c) tell the user to use LICR in the preamble, e.g. PDF title: Gru\ss aus Dresden +1 consistent with the idea: preamble is "raw" (kind of) +1 simple (for us) -1 fails for \" (Gr\"u\ss e) (different bug: any ASCII-" terminates the string!) Günter
Re: status update on the export tests
Guenter Milde wrote: > A proper fix would be to use lib/unicodesymbols for the content of the > hypersetup. It has been discussed in the past to use lib/unicodesymbols replacements for the preamble, but there was some important reason not to do it (which I forgot unfortunately). Searching the list archives or trac should for unicodesymbols and preamble should hopefully find the discussion. Georg
Re: status update on the export tests
On 2015-10-27, Stephan Witt wrote: > Am 27.10.2015 um 13:42 schrieb Kornel Benko : >> Am Dienstag, 27. Oktober 2015 um 13:29:35, schrieb Stephan Witt >> >>> One interesting example of a failure is as follows: >>> == >>> 350/4621 Testing: export/doc/Math_pdf4_texF >> It is simply not compilable with xelatex. At least here. > Because of this: > $ file lib/doc/Math.lyx > lib/doc/Math.lyx: UTF-8 Unicode English text > ... > \pdf_author "LyX Team, Uwe Stöhr" > ... Mind, that this is also currently a problem with "normal" latex or pdflatex, if you set the "latex encoding" to "ASCII". The problem is, that the hypersetup is wrapped in: \inputencoding{utf8} ... \inputencoding{ascii} which leads to; ! Undefined control sequence. l.35 \inputencoding {utf8} ! LaTeX Error: Missing \begin{document}. See the LaTeX manual or LaTeX Companion for explanation. Type H for immediate help. ... l.35 \inputencoding{u tf8} > This cannot be converted from UTF-8 to ASCII. Of course it can: \pdf_author "LyX Team, Uwe St\"ohr" (Hyperref supports the LaTeX Internal Character Representation (LICR).) If I export the "ascii"-version to latex, change as above and comment out the \inputencoding changes, the file compiles fine here. A proper fix would be to use lib/unicodesymbols for the content of the hypersetup. Günter
Re: status update on the export tests
Am 27.10.2015 um 13:42 schrieb Kornel Benko : > Am Dienstag, 27. Oktober 2015 um 13:29:35, schrieb Stephan Witt > >> One interesting example of a failure is as follows: >> >> == >> 350/4621 Testing: export/doc/Math_pdf4_texF > > It is simply not compilable with xelatex. At least here. Because of this: $ file lib/doc/Math.lyx lib/doc/Math.lyx: UTF-8 Unicode English text ... \pdf_author "LyX Team, Uwe Stöhr" ... This cannot be converted from UTF-8 to ASCII. Stephan
Re: status update on the export tests
Am Dienstag, 27. Oktober 2015 um 13:29:35, schrieb Stephan Witt > One interesting example of a failure is as follows: > > == > 350/4621 Testing: export/doc/Math_pdf4_texF It is simply not compilable with xelatex. At least here. Kornel signature.asc Description: This is a digitally signed message part.
Re: status update on the export tests
Am 27.10.2015 um 11:15 schrieb Kornel Benko : > Am Dienstag, 27. Oktober 2015 um 10:46:58, schrieb Stephan Witt > >> Am 27.10.2015 um 10:13 schrieb Kornel Benko : >> >>> Am Dienstag, 27. Oktober 2015 um 00:48:44, schrieb Stephan Witt >>> Am 26.10.2015 um 12:18 schrieb Kornel Benko : > Am Montag, 26. Oktober 2015 um 11:43:07, schrieb Stephan Witt > >> Am 26.10.2015 um 09:46 schrieb Kornel Benko : >> >>> They are started, because otherwise you would not see 'out of 199'. >>> What was your exact test command? >> >> This is the command: >> >> $ (cd /Users/stephan/git/lyx-build/cmake/2.2.0dev;ctest -C Debug) > > Wow ... never used it this way ... > But then I wonder, why only 199 tests ... > Apparently you have not configured cmake for export tests (cmake ... > -DLYX_ENABLE_EXPORT_TESTS=ON …) Yes, this make a huge difference - now I've 4621 tests to run :) >>> >>> :) >>> After 11 hours I'm at test 1405 of 4621… (running with one job at a time). >> >> The result is: >> >> 79% tests passed, 954 tests failed out of 4621 >> >> Label Time Summary: >> cmplyx = 214.12 sec >> export = 72802.49 sec >> layout = 94.51 sec >> load = 700.28 sec >> lyx2lyx = 67.98 sec >> module = 9.83 sec >> reverted = 19013.35 sec >> roundtrip= 186.85 sec >> >> Total Test time (real) = 74085.89 sec >> >>> For that many export tests, I would use 'ctest -j8' (in case you have >>> multiple cores). >>> I normally run 'ctest -j12 -R "export.*pdf"'. Having 8 cores makes the run >>> significantly faster. >> >> Yes, I understood this. But for the first try I didn't want to run it in >> parallel. >> I've observed that e.g. kpsewhich is used heavily while testing and one call >> took ~10ms. >> Perhaps there is some tuning possible? But I'd guess, there is a significant >> overhead >> for the first time to generate all the required pieces like compiled fonts >> etc. >> To check this now I'm starting the test again. >> >>> >>> Some tests may fail because of this parallel handling (I did not found why >>> yet), so I repeat >>> the failed tests without the '-j12' param with the command 'ctest >>> --rerun-failed'. >> >> 1st question: wouldn't it be useful to split the tests into groups and >> enable the tests which should pass on a basic system automatically? >> How could this be done? I don't think it's good to have to enable the >> export tests explicitly. > > Enabling the export tests increase the configuration time ..., so it felt > reasonable to set the > default to OFF. > And not 'many' developers tested at that time. > >> 2nd question: which tests should succeed with a "normal" TeX installation? > > 1.) Some export tests do not depend on TeX > "ctest -R 'export.*_(xhtml|lyx16)$'" > 2.) Probably you want to test TeX without system fonts > "ctest -R 'export.*_texF$'" With $ (cd /Users/stephan/git/lyx-build/cmake/2.2.0dev;ctest -C Debug -R 'export/doc/.*_texF$' -j8) I get this: 77% tests passed, 70 tests failed out of 302 Label Time Summary: export = 4782.73 sec reverted= 538.92 sec Total Test time (real) = 612.37 sec One interesting example of a failure is as follows: == 350/4621 Testing: export/doc/Math_pdf4_texF 350/4621 Test: export/doc/Math_pdf4_texF Command: "/opt/local/bin/cmake" "-DLYX_ROOT=/Users/stephan/git/lyx/lib/doc" "-DLYX_TESTS_USERDIR=/Users/stephan/git/lyx-build/cmake/2.2.0dev/Testing/.lyx" "-Dlyx=/Users/stephan/g it/lyx-build/cmake/2.2.0dev/bin/Debug/LyX" "-DWORKDIR=/Users/stephan/git/lyx-build/cmake/2.2.0dev/autotests/out-home" "-Dformat=pdf4" "-Dfonttype=texF" "-Dextension=pdf4" "-Dfile =Math" "-Dreverted=0" "-DTOP_SRC_DIR=/Users/stephan/git/lyx" "-DPERL_EXECUTABLE=/opt/local/bin/perl" "-P" "/Users/stephan/git/lyx/development/autotests/export.cmake" Directory: /Users/stephan/git/lyx-build/cmake/2.2.0dev/autotests/out-home "export/doc/Math_pdf4_texF" start time: Oct 27 13:13 CET Output: -- -- using fonttype = texF -- LYX_TESTS_USERDIR = /Users/stephan/git/lyx-build/cmake/2.2.0dev/Testing/.lyx -- Converting with perl /Users/stephan/git/lyx/development/autotests/useSystemFonts.pl -- Using source "/Users/stephan/git/lyx/lib/doc/Math.lyx" -- Using dest "/Users/stephan/git/lyx-build/cmake/2.2.0dev/autotests/out-home/AbC_N1_hGi/Math_pdf4_texF.lyx" Added (1) file "/Users/stephan/git/lyx/lib/doc/../images/math/phantom.png" to be copied to "/Users/stephan/git/lyx-build/cmake/2.2.0dev/autotests/out-home/AbC_N1_hGi/tmp_pdf4_main_Math/xx_phantom_wncljUBI.png" Added (1) file "/Users/stephan/git/lyx/lib/doc/../images/math/vphantom.png" to be copied to "/Users/stephan/git/lyx-build/cmake/2.2.0dev/autotests/out-home/AbC_N1_hGi/tmp_pdf4_main_Math/xx_vphantom_j7I4wBMv.png" Added (1) file "/Users/stephan/git/lyx/lib/doc/../images/math/hphantom.png" to be copied to "/Users/stephan
Re: status update on the export tests
Am Dienstag, 27. Oktober 2015 um 10:46:58, schrieb Stephan Witt > Am 27.10.2015 um 10:13 schrieb Kornel Benko : > > > Am Dienstag, 27. Oktober 2015 um 00:48:44, schrieb Stephan Witt > > > >> Am 26.10.2015 um 12:18 schrieb Kornel Benko : > >> > >>> Am Montag, 26. Oktober 2015 um 11:43:07, schrieb Stephan Witt > >>> > Am 26.10.2015 um 09:46 schrieb Kornel Benko : > > > They are started, because otherwise you would not see 'out of 199'. > > What was your exact test command? > > This is the command: > > $ (cd /Users/stephan/git/lyx-build/cmake/2.2.0dev;ctest -C Debug) > >>> > >>> Wow ... never used it this way ... > >>> But then I wonder, why only 199 tests ... > >>> Apparently you have not configured cmake for export tests (cmake ... > >>> -DLYX_ENABLE_EXPORT_TESTS=ON …) > >> > >> Yes, this make a huge difference - now I've 4621 tests to run :) > > > > :) > > > >> After 11 hours I'm at test 1405 of 4621… (running with one job at a time). > > The result is: > > 79% tests passed, 954 tests failed out of 4621 > > Label Time Summary: > cmplyx = 214.12 sec > export = 72802.49 sec > layout = 94.51 sec > load = 700.28 sec > lyx2lyx = 67.98 sec > module = 9.83 sec > reverted = 19013.35 sec > roundtrip= 186.85 sec > > Total Test time (real) = 74085.89 sec > > > For that many export tests, I would use 'ctest -j8' (in case you have > > multiple cores). > > I normally run 'ctest -j12 -R "export.*pdf"'. Having 8 cores makes the run > > significantly faster. > > Yes, I understood this. But for the first try I didn't want to run it in > parallel. > I've observed that e.g. kpsewhich is used heavily while testing and one call > took ~10ms. > Perhaps there is some tuning possible? But I'd guess, there is a significant > overhead > for the first time to generate all the required pieces like compiled fonts > etc. > To check this now I'm starting the test again. > > > > > Some tests may fail because of this parallel handling (I did not found why > > yet), so I repeat > > the failed tests without the '-j12' param with the command 'ctest > > --rerun-failed'. > > 1st question: wouldn't it be useful to split the tests into groups and > enable the tests which should pass on a basic system automatically? > How could this be done? I don't think it's good to have to enable the > export tests explicitly. Enabling the export tests increase the configuration time ..., so it felt reasonable to set the default to OFF. And not 'many' developers tested at that time. > 2nd question: which tests should succeed with a "normal" TeX installation? 1.) Some export tests do not depend on TeX "ctest -R 'export.*_(xhtml|lyx16)$'" 2.) Probably you want to test TeX without system fonts "ctest -R 'export.*_texF$'" > Stephan Kornel signature.asc Description: This is a digitally signed message part.
Re: status update on the export tests
Am 27.10.2015 um 10:13 schrieb Kornel Benko : > Am Dienstag, 27. Oktober 2015 um 00:48:44, schrieb Stephan Witt > >> Am 26.10.2015 um 12:18 schrieb Kornel Benko : >> >>> Am Montag, 26. Oktober 2015 um 11:43:07, schrieb Stephan Witt >>> Am 26.10.2015 um 09:46 schrieb Kornel Benko : > They are started, because otherwise you would not see 'out of 199'. > What was your exact test command? This is the command: $ (cd /Users/stephan/git/lyx-build/cmake/2.2.0dev;ctest -C Debug) >>> >>> Wow ... never used it this way ... >>> But then I wonder, why only 199 tests ... >>> Apparently you have not configured cmake for export tests (cmake ... >>> -DLYX_ENABLE_EXPORT_TESTS=ON …) >> >> Yes, this make a huge difference - now I've 4621 tests to run :) > > :) > >> After 11 hours I'm at test 1405 of 4621… (running with one job at a time). The result is: 79% tests passed, 954 tests failed out of 4621 Label Time Summary: cmplyx = 214.12 sec export = 72802.49 sec layout = 94.51 sec load = 700.28 sec lyx2lyx = 67.98 sec module = 9.83 sec reverted = 19013.35 sec roundtrip= 186.85 sec Total Test time (real) = 74085.89 sec > For that many export tests, I would use 'ctest -j8' (in case you have > multiple cores). > I normally run 'ctest -j12 -R "export.*pdf"'. Having 8 cores makes the run > significantly faster. Yes, I understood this. But for the first try I didn't want to run it in parallel. I've observed that e.g. kpsewhich is used heavily while testing and one call took ~10ms. Perhaps there is some tuning possible? But I'd guess, there is a significant overhead for the first time to generate all the required pieces like compiled fonts etc. To check this now I'm starting the test again. > > Some tests may fail because of this parallel handling (I did not found why > yet), so I repeat > the failed tests without the '-j12' param with the command 'ctest > --rerun-failed'. 1st question: wouldn't it be useful to split the tests into groups and enable the tests which should pass on a basic system automatically? How could this be done? I don't think it's good to have to enable the export tests explicitly. 2nd question: which tests should succeed with a "normal" TeX installation? Stephan
Re: status update on the export tests
Am Dienstag, 27. Oktober 2015 um 00:48:44, schrieb Stephan Witt > Am 26.10.2015 um 12:18 schrieb Kornel Benko : > > > Am Montag, 26. Oktober 2015 um 11:43:07, schrieb Stephan Witt > > > >> Am 26.10.2015 um 09:46 schrieb Kornel Benko : > >> > >>> They are started, because otherwise you would not see 'out of 199'. > >>> What was your exact test command? > >> > >> This is the command: > >> > >> $ (cd /Users/stephan/git/lyx-build/cmake/2.2.0dev;ctest -C Debug) > > > > Wow ... never used it this way ... > > But then I wonder, why only 199 tests ... > > Apparently you have not configured cmake for export tests (cmake ... > > -DLYX_ENABLE_EXPORT_TESTS=ON …) > > Yes, this make a huge difference - now I've 4621 tests to run :) :) > After 11 hours I'm at test 1405 of 4621… (running with one job at a time). For that many export tests, I would use 'ctest -j8' (in case you have multiple cores). I normally run 'ctest -j12 -R "export.*pdf"'. Having 8 cores makes the run significantly faster. Some tests may fail because of this parallel handling (I did not found why yet), so I repeat the failed tests without the '-j12' param with the command 'ctest --rerun-failed'. > Stephan Kornel signature.asc Description: This is a digitally signed message part.
Re: status update on the export tests
Am 26.10.2015 um 12:18 schrieb Kornel Benko : > Am Montag, 26. Oktober 2015 um 11:43:07, schrieb Stephan Witt > >> Am 26.10.2015 um 09:46 schrieb Kornel Benko : >> >>> They are started, because otherwise you would not see 'out of 199'. >>> What was your exact test command? >> >> This is the command: >> >> $ (cd /Users/stephan/git/lyx-build/cmake/2.2.0dev;ctest -C Debug) > > Wow ... never used it this way ... > But then I wonder, why only 199 tests ... > Apparently you have not configured cmake for export tests (cmake ... > -DLYX_ENABLE_EXPORT_TESTS=ON …) Yes, this make a huge difference - now I've 4621 tests to run :) After 11 hours I'm at test 1405 of 4621… (running with one job at a time). Stephan
Re: status update on the export tests
Scott Kostyshak wrote: > OK. Do you agree that this is not a blocker for alpha? Yes. Georg
Re: status update on the export tests
On Mon, Oct 26, 2015 at 09:40:06PM +0100, Georg Baum wrote: > Scott Kostyshak wrote: > > > OK. I just want to make sure we define "there are no regressions". These > > tests passed before edd37de8. So it is quite possible that documents > > that users have that compile with LyX 2.1.x suddenly fail to compile > > with LyX 2.2.0 because of edd37de8. In this sense, there is a > > regression. > > Sorry, then I mixed this up. Well it is a confusing situation with the tests. I might make a new ticket that makes it clear what the current state of this "regression" is and target it for 2.2.0. > This should of course be fixed for the release > IMO. I'll try to find out what is still missing. OK. Do you agree that this is not a blocker for alpha? Scott
Re: status update on the export tests
Scott Kostyshak wrote: > OK. I just want to make sure we define "there are no regressions". These > tests passed before edd37de8. So it is quite possible that documents > that users have that compile with LyX 2.1.x suddenly fail to compile > with LyX 2.2.0 because of edd37de8. In this sense, there is a > regression. Sorry, then I mixed this up. This should of course be fixed for the release IMO. I'll try to find out what is still missing. Georg
Re: status update on the export tests
On 2015-10-26, Liviu Andronic wrote: > On Mon, Oct 26, 2015 at 10:24 AM, Guenter Milde wrote: >> On 2015-10-26, Scott Kostyshak wrote: ... >> Could this prevent some of the regressions? (We need to look carefully, not >> only if the relevant documents compile without error, but also if the >> exported document is OK.) > By exported document do you mean .tex or .pdf? If it is .tex, would it > be a good idea to check whether the latest export is identical to a > reference .tex generated when creating the test; if not, display a > diff. Simply relying on the exit code seems like an easy way to miss > non-error generating regressions... I think we have to distinguish several test methods: a) automatic export tests with lots of "real life" documents (manuals, templates, examples) b) functional tests: import/export of a complete test document and comparing it to the expected output c) unit tests: test of certain features or sub-units. Unit tests (c) are currently not implemented, AFAIK. The tex2lyx tests are "functional tests" (b), where we keep track of the expected output. They include also export tests (in the "round trip" suite). Here, we have to manually check and update the "expected" output documents discriminating between intended changes and regressions/bug-indicators. For a), it would be a pain to keep track of and update all the output documents, because this would not only be required for different export routines but also for changes in the input docs. However, if the exit status of a test changes (from fail to pass or vice versa), we should check whether this is due to a new bug, a fix or just exposing previously hidden problems. Günter
Re: status update on the export tests
Am Montag, 26. Oktober 2015 um 11:43:07, schrieb Stephan Witt > Am 26.10.2015 um 09:46 schrieb Kornel Benko : > > > Am Montag, 26. Oktober 2015 um 09:13:44, schrieb Stephan Witt > > > > ... > >>> These tests fails because of language switching done with polyglossia, > >>> but I'm not sure what the root problem is: > >>> export/doc/fr/UserGuide_pdf5_systemF > >>> export/examples/fr/Foils_pdf5_systemF > >>> # kluwer gives the following error with polyglossia: > >>> # LaTeX Error: Command \ifdef already defined. > >>> export/templates/kluwer_pdf5_systemF > >>> > >>> Please let me (or Kornel) know if you have any questions. > >> > >> I tried to test these exports on my Mac and cannot see how it should be > >> triggered. ... > >> On my system there is no X11-utility installed and LyX doesn't belong to > >> X11. > >> Is this the reason why these tests are not started? > >> > >> Stephan > > > > They are started, because otherwise you would not see 'out of 199'. > > What was your exact test command? > > This is the command: > > $ (cd /Users/stephan/git/lyx-build/cmake/2.2.0dev;ctest -C Debug) Wow ... never used it this way ... But then I wonder, why only 199 tests ... Apparently you have not configured cmake for export tests (cmake ... -DLYX_ENABLE_EXPORT_TESTS=ON ...) > > > > To see what went wrong, look in file "Testing/Temporary/LastTest.log". > > > One example is: > > 6/199 Testing: tex2lyx/roundtrip/test.ltx > 6/199 Test: tex2lyx/roundtrip/test.ltx ... > -- Error output of /Users/stephan/git/lyx/src/tex2lyx/test/runtests.py = 0 > > Test time = 2.79 sec > -- > Test Passed. > "tex2lyx/roundtrip/test.ltx" end time: Oct 26 11:05 CET > "tex2lyx/roundtrip/test.ltx" time elapsed: 00:00:02 > -- Apparently this test passed ... > Stephan Kornel signature.asc Description: This is a digitally signed message part.
Re: status update on the export tests
Am 26.10.2015 um 09:46 schrieb Kornel Benko : > Am Montag, 26. Oktober 2015 um 09:13:44, schrieb Stephan Witt > > ... >>> These tests fails because of language switching done with polyglossia, but >>> I'm not sure what the root problem is: >>> export/doc/fr/UserGuide_pdf5_systemF >>> export/examples/fr/Foils_pdf5_systemF >>> # kluwer gives the following error with polyglossia: >>> # LaTeX Error: Command \ifdef already defined. >>> export/templates/kluwer_pdf5_systemF >>> >>> Please let me (or Kornel) know if you have any questions. >> >> I tried to test these exports on my Mac and cannot see how it should be >> triggered. >> >> Below I'm listing the summary output: >> == >> 100% tests passed, 0 tests failed out of 199 >> >> Label Time Summary: >> cmplyx = 120.88 sec >> layout = 107.01 sec >> lyx2lyx = 0.05 sec >> module = 12.17 sec >> roundtrip= 76.86 sec >> == >> >> On my system there is no X11-utility installed and LyX doesn't belong to X11. >> Is this the reason why these tests are not started? >> >> Stephan > > They are started, because otherwise you would not see 'out of 199'. > What was your exact test command? This is the command: $ (cd /Users/stephan/git/lyx-build/cmake/2.2.0dev;ctest -C Debug) > > To see what went wrong, look in file "Testing/Temporary/LastTest.log". One example is: 6/199 Testing: tex2lyx/roundtrip/test.ltx 6/199 Test: tex2lyx/roundtrip/test.ltx Command: "/opt/local/bin/cmake" "-DLYX_TESTS_USERDIR=/Users/stephan/git/lyx-build/cmake/2.2.0dev/Testing/.lyx" "-DLYX_USERDIR_VER=LYX_USERDIR_22x" "-DLYX_PYTHON_EXECUTABLE=/opt/local/bin/python2.7" "-DPY_SCRIPT=/Users/stephan/git/lyx/src/tex2lyx/test/runtests.py" "-DTEX2LYX_EXE=/Users/stephan/git/lyx-build/cmake/2.2.0dev/bin/Debug/tex2lyx" "-DSCRIPT_DIR=/Users/stephan/git/lyx/lib/scripts" "-DWORKDIR=/Users/stephan/git/lyx-build/cmake/2.2.0dev/src/tex2lyx/test" "-DTESTFILE=test.ltx" "-P" "/Users/stephan/git/lyx/src/tex2lyx/test/runtests.cmake" Directory: /Users/stephan/git/lyx-build/cmake/2.2.0dev/src/tex2lyx/test "tex2lyx/roundtrip/test.ltx" start time: Oct 26 11:05 CET Output: -- -- SCRIPT_DIR = /Users/stephan/git/lyx/lib/scripts -- Executing: /Users/stephan/git/lyx-build/cmake/2.2.0dev/bin/Debug/tex2lyx -roundtrip -copyfiles -f /Users/stephan/git/lyx/src/tex2lyx/test/test.ltx /Users/stephan/git/lyx-build/cmake/2.2.0dev/src/tex2lyx/test/test.lyx Overwriting existing file /Users/stephan/git/lyx-build/cmake/2.2.0dev/src/tex2lyx/test/test.lyx Error: New binding for 'Kontroll-T 1' is overriding old binding... Error: New binding for 'Kontroll-T 2' is overriding old binding... Error: New binding for 'Kontroll-T O' is overriding old binding... Error: New binding for 'Kontroll-T T' is overriding old binding... Error: New binding for 'Kontroll-T X' is overriding old binding... Warning: Die Dokumentklasse ist nicht verfügbar Die ausgewählte Dokumentklasse Article (Standardklasse) benötigt externe Dateien, welche nicht verfügbar sind. Die Dokumentklasse kann zwar verwendet werden, das Dokument kann aber nicht ausgegeben werden, solange die folgenden Dateien nicht installiert sind: article.cls In Abschnitt 3.1.2.2 des Benutzerhandbuchs finden Sie weitere Hilfe. -- Error output of /Users/stephan/git/lyx/src/tex2lyx/test/runtests.py = 0 Test time = 2.79 sec -- Test Passed. "tex2lyx/roundtrip/test.ltx" end time: Oct 26 11:05 CET "tex2lyx/roundtrip/test.ltx" time elapsed: 00:00:02 -- Stephan
Re: status update on the export tests
On Mon, Oct 26, 2015 at 10:24 AM, Guenter Milde wrote: > On 2015-10-26, Scott Kostyshak wrote: > >> OK. I just want to make sure we define "there are no regressions". These >> tests passed before edd37de8. So it is quite possible that documents >> that users have that compile with LyX 2.1.x suddenly fail to compile >> with LyX 2.2.0 because of edd37de8. In this sense, there is a >> regression. > >> So there is a trade-off: the bad side is that there is the possibility >> that user's documents suddenly fail to compile. The good side is that we >> now encourage the use of polyglossia by default which apparently has >> advantages over babel. Jürgen knowingly made this decision and he knows >> more than I do to be able to judge this tradeoff so I hesitantly accept >> it. > > Polyglossia has been the better (and sometimes only) choice for use with > Xe/LuaTeX Unicode fonts for a long time, but Babel is catching up in the > last years. > > It may be that we need a new keyword for lib/languages, to tag > languages where Babel is superior to Polyglossia also with non-TeX fonts. > > Then, the "automatic" setting (which is also the default "default") could > take into account that some languages should use Babel also with non-TeX > fonts even if Polyglossia is available for them. > > Could this prevent some of the regressions? (We need to look carefully, not > only if the relevant documents compile without error, but also if the > exported document is OK.) > By exported document do you mean .tex or .pdf? If it is .tex, would it be a good idea to check whether the latest export is identical to a reference .tex generated when creating the test; if not, display a diff. Simply relying on the exit code seems like an easy way to miss non-error generating regressions... Liviu > Günter > -- Do you think you know what math is? http://www.ideasroadshow.com/issues/ian-stewart-2013-08-02 Or what it means to be intelligent? http://www.ideasroadshow.com/issues/john-duncan-2013-08-30 Think again: http://www.ideasroadshow.com/library
Re: status update on the export tests
On 2015-10-26, Scott Kostyshak wrote: > OK. I just want to make sure we define "there are no regressions". These > tests passed before edd37de8. So it is quite possible that documents > that users have that compile with LyX 2.1.x suddenly fail to compile > with LyX 2.2.0 because of edd37de8. In this sense, there is a > regression. > So there is a trade-off: the bad side is that there is the possibility > that user's documents suddenly fail to compile. The good side is that we > now encourage the use of polyglossia by default which apparently has > advantages over babel. Jürgen knowingly made this decision and he knows > more than I do to be able to judge this tradeoff so I hesitantly accept > it. Polyglossia has been the better (and sometimes only) choice for use with Xe/LuaTeX Unicode fonts for a long time, but Babel is catching up in the last years. It may be that we need a new keyword for lib/languages, to tag languages where Babel is superior to Polyglossia also with non-TeX fonts. Then, the "automatic" setting (which is also the default "default") could take into account that some languages should use Babel also with non-TeX fonts even if Polyglossia is available for them. Could this prevent some of the regressions? (We need to look carefully, not only if the relevant documents compile without error, but also if the exported document is OK.) Günter
Re: status update on the export tests
Am Montag, 26. Oktober 2015 um 09:13:44, schrieb Stephan Witt ... > > These tests fails because of language switching done with polyglossia, but > > I'm not sure what the root problem is: > > export/doc/fr/UserGuide_pdf5_systemF > > export/examples/fr/Foils_pdf5_systemF > > # kluwer gives the following error with polyglossia: > > # LaTeX Error: Command \ifdef already defined. > > export/templates/kluwer_pdf5_systemF > > > > Please let me (or Kornel) know if you have any questions. > > I tried to test these exports on my Mac and cannot see how it should be > triggered. > > Below I'm listing the summary output: > == > 100% tests passed, 0 tests failed out of 199 > > Label Time Summary: > cmplyx = 120.88 sec > layout = 107.01 sec > lyx2lyx = 0.05 sec > module = 12.17 sec > roundtrip= 76.86 sec > == > > On my system there is no X11-utility installed and LyX doesn't belong to X11. > Is this the reason why these tests are not started? > > Stephan They are started, because otherwise you would not see 'out of 199'. What was your exact test command? To see what went wrong, look in file "Testing/Temporary/LastTest.log". Kornel signature.asc Description: This is a digitally signed message part.
Re: status update on the export tests
Am 25.10.2015 um 05:29 schrieb Scott Kostyshak : > Dear all, > > There seems to be more interest in running the export tests. This is great and > will encourage me to improve the (shamefully lacking) documentation for these > tests. > > First, I am focusing on the test results after reverting (locally) two > commits, > as follows: > > git revert --no-edit 664ef2c4 1523fc60 > > If you do not revert these two commits, many tests will fail and it will be > hard to figure out when there are regressions. > > To run all the tests successfully, you need to have a lot of dependencies > (even > if you have a full TeX Live installation). Thus, you might just want to limit > your running of the tests to our manuals, and exclude the templates and > examples. To do that, you can do something like: ctest -R "doc/" > > If you are interested in whether a certain patch breaks anything, feel free to > let me or Kornel know and we will do our best to run the tests before and > after > your change and see if any test suggests a regression. > > The following tests are currently expected to fail because of issues with > nested language handling and polyglossia. Georg, if you still have interest > and > time for fixing this type of issue, take a look at these: > > export/doc/nb/Intro_pdf5_systemF > export/doc/sk/Intro_pdf5_systemF > export/doc/es/Customization_pdf5_systemF > export/doc/de/Customization_pdf5_systemF > export/doc/fr/Customization_pdf5_systemF > > I'm not sure but these might also suffer from a similar problem: > export/examples/seminar_pdf5_systemF > export/examples/fr/seminar_pdf5_systemF > > These tests fails because of language switching done with polyglossia, but > I'm not sure what the root problem is: > export/doc/fr/UserGuide_pdf5_systemF > export/examples/fr/Foils_pdf5_systemF > # kluwer gives the following error with polyglossia: > # LaTeX Error: Command \ifdef already defined. > export/templates/kluwer_pdf5_systemF > > Please let me (or Kornel) know if you have any questions. I tried to test these exports on my Mac and cannot see how it should be triggered. Below I'm listing the summary output: == 100% tests passed, 0 tests failed out of 199 Label Time Summary: cmplyx = 120.88 sec layout = 107.01 sec lyx2lyx = 0.05 sec module = 12.17 sec roundtrip= 76.86 sec == On my system there is no X11-utility installed and LyX doesn't belong to X11. Is this the reason why these tests are not started? Stephan
Re: status update on the export tests
On Sun, Oct 25, 2015 at 11:10:56AM +0100, Georg Baum wrote: > Scott Kostyshak wrote: > > > The following tests are currently expected to fail because of issues with > > nested language handling and polyglossia. Georg, if you still have > > interest and time for fixing this type of issue, take a look at these: > > > > export/doc/nb/Intro_pdf5_systemF > > export/doc/sk/Intro_pdf5_systemF > > export/doc/es/Customization_pdf5_systemF > > export/doc/de/Customization_pdf5_systemF > > export/doc/fr/Customization_pdf5_systemF > > I am still interested, but don't know when I'll have time to look at these, > so don't wait for me. I think the current status is much better than with > 2.1, there are no regressions, and that this the most important thing. If > we'll find a safe improvement before the 2.2.0 release, then this is fine, > but if we don't then it is not a problem. OK. I just want to make sure we define "there are no regressions". These tests passed before edd37de8. So it is quite possible that documents that users have that compile with LyX 2.1.x suddenly fail to compile with LyX 2.2.0 because of edd37de8. In this sense, there is a regression. So there is a trade-off: the bad side is that there is the possibility that user's documents suddenly fail to compile. The good side is that we now encourage the use of polyglossia by default which apparently has advantages over babel. Jürgen knowingly made this decision and he knows more than I do to be able to judge this tradeoff so I hesitantly accept it. Are we all on the same page? Scott
Re: status update on the export tests
Scott Kostyshak wrote: > The following tests are currently expected to fail because of issues with > nested language handling and polyglossia. Georg, if you still have > interest and time for fixing this type of issue, take a look at these: > > export/doc/nb/Intro_pdf5_systemF > export/doc/sk/Intro_pdf5_systemF > export/doc/es/Customization_pdf5_systemF > export/doc/de/Customization_pdf5_systemF > export/doc/fr/Customization_pdf5_systemF I am still interested, but don't know when I'll have time to look at these, so don't wait for me. I think the current status is much better than with 2.1, there are no regressions, and that this the most important thing. If we'll find a safe improvement before the 2.2.0 release, then this is fine, but if we don't then it is not a problem. Georg
status update on the export tests
Dear all, There seems to be more interest in running the export tests. This is great and will encourage me to improve the (shamefully lacking) documentation for these tests. First, I am focusing on the test results after reverting (locally) two commits, as follows: git revert --no-edit 664ef2c4 1523fc60 If you do not revert these two commits, many tests will fail and it will be hard to figure out when there are regressions. To run all the tests successfully, you need to have a lot of dependencies (even if you have a full TeX Live installation). Thus, you might just want to limit your running of the tests to our manuals, and exclude the templates and examples. To do that, you can do something like: ctest -R "doc/" If you are interested in whether a certain patch breaks anything, feel free to let me or Kornel know and we will do our best to run the tests before and after your change and see if any test suggests a regression. The following tests are currently expected to fail because of issues with nested language handling and polyglossia. Georg, if you still have interest and time for fixing this type of issue, take a look at these: export/doc/nb/Intro_pdf5_systemF export/doc/sk/Intro_pdf5_systemF export/doc/es/Customization_pdf5_systemF export/doc/de/Customization_pdf5_systemF export/doc/fr/Customization_pdf5_systemF I'm not sure but these might also suffer from a similar problem: export/examples/seminar_pdf5_systemF export/examples/fr/seminar_pdf5_systemF These tests fails because of language switching done with polyglossia, but I'm not sure what the root problem is: export/doc/fr/UserGuide_pdf5_systemF export/examples/fr/Foils_pdf5_systemF # kluwer gives the following error with polyglossia: # LaTeX Error: Command \ifdef already defined. export/templates/kluwer_pdf5_systemF Please let me (or Kornel) know if you have any questions. Scott