On Sat, Aug 3, 2013 at 4:12 AM, Kornel Benko <[email protected]> wrote: > Am Samstag, 3. August 2013 um 03:54:46, schrieb Scott Kostyshak > <[email protected]> > >> On Fri, Aug 2, 2013 at 5:55 AM, Kornel Benko <[email protected]> wrote: > >> > Am Freitag, 2. August 2013 um 11:10:33, schrieb Vincent van Ravesteijn > >> > <[email protected]> > >> >> What about testing everything, and marking these tests as "known to >> >> fail". > >> > > >> >> In that way, if suddenly, LuaTex supports such a language, the tests >> >> will > >> > > >> >> tell us. By hardcoding that these languages are not tested, this > >> >> hardcoding > >> > > >> >> will be there for ever, even if some of the languages might get >> >> supported. > >> > > >> > > >> > > >> > The list could be read in CMakeLists.txt, then we can add a parameter to > >> > export.cmake > >> > > >> > to revert the meaning of success. > >> > >> I like these ideas. I created the file > >> development/expectedTestFailures partly for this reason (to know when > >> a test fails because of a regression and when a test file fails > >> because of something we already know), but I didn't think about > >> programmatically inputting such a file. > >> > >> My only hesitation with the inversion is that it is not intuitive: if > >> a Hebrew LuaTeX test passes, that means that it is broken, and if it > >> suddenly fails, that means that the export actually worked. This is > >> fine for you and me because we would know about the inversion. But if > >> anyone else ends up running the tests, how will they know? We could > >> have the LastTest.log explain this but I don't know if they would know > >> to look there. This could also be explained in > >> development/autotests/README, but again I don't know if someone will > >> read that. > >> > >> To be clear, I am in favor of this. But I wanted to explain my hesitation. > >> > >> Scott > > > > We could mark the test as "shouldFail" or "shouldNotSucceed" (in the > testname). > > > > like > > # ctest -R export/doc/attic/shouldNotSucceed_DocStyle_pdf
Do you mean instead of inversion or in addition to inversion? I think you mean instead of inversion. I like your idea of inversion better because it would be nice to have the list of failing tests as clean as possible. Also, this way we would not really be notified that a test went from failing to passing. If a test goes from passing to failing it will catch our attention (and we can fix the test and documentation of the now-working feature, etc). How about we combine the two ideas (perhaps this is what you were suggesting and I misunderstood)? We invert and change the name. For example: ctest -R export/doc/attic/INVERTED-TEST_FAILURE-IS-GOOD_DocStyle_pdf It's ugly but the name of the test is our only way to pass information. If someone sees this, either they will understand it's not bad or they will be confused and look for more information in README or LastTest.log Scott
