Re: insufficient output from LayoutEngineTestCase failures
s/doATTests/doATChecks/ On Fri, Nov 18, 2011 at 10:19 AM, Glenn Adams wrote: > one workaround for LayoutEngineTestCase.doATTests() would be to add the > test file name/path as a accessible property of LayoutResult, which would > then be accessible by check(LayoutResult); > > perhaps medhi can try this or something similar > > On Fri, Nov 18, 2011 at 3:26 AM, Vincent Hennebert > wrote: > >> Hi Glenn, >> >> On 16/11/11 23:47, Glenn Adams wrote: >> > With the recent transition to JUnit4 runner and changes to >> > LayoutEngineTestCase, it is now someone difficult to determine (from >> output >> > data alone) which layoutengine test file failed when a regression >> occurs. >> > In particular, the exception trace generated from EvalCheck, TrueCheck, >> > etc. does not include the test case input file name. So one ends up with >> > something like the following in >> > the TEST-org.apache.fop.layoutengine.LayoutEngineTestCase.txt file in >> > builds/test-reports: >> > >> > Testcase: runTest[342] took 0.017 sec >> > Caused an ERROR >> >> This is a known issue: >> https://github.com/KentBeck/junit/issues/44 >> And quite popular at that it seems: >> https://github.com/KentBeck/junit/issues/search?q=parameterized+name >> >> In the meantime, I wrote the attached patch to IFParserTestCase. The >> same could be done for LayoutEngineTestCase. >> >> >> > >> > Unfortunately, there is no way to correlate runTest[342] with a specific >> > test case input file. It would be very useful (for AT and IF tests) to >> also >> > include the test case input file name/path in this output. Otherwise, >> one >> > is forced to run junit in a debugger with a breakpoint on EvalCheck, >> > TrueCheck, etc. >> >> Vincent >> > >
Re: insufficient output from LayoutEngineTestCase failures
one workaround for LayoutEngineTestCase.doATTests() would be to add the test file name/path as a accessible property of LayoutResult, which would then be accessible by check(LayoutResult); perhaps medhi can try this or something similar On Fri, Nov 18, 2011 at 3:26 AM, Vincent Hennebert wrote: > Hi Glenn, > > On 16/11/11 23:47, Glenn Adams wrote: > > With the recent transition to JUnit4 runner and changes to > > LayoutEngineTestCase, it is now someone difficult to determine (from > output > > data alone) which layoutengine test file failed when a regression occurs. > > In particular, the exception trace generated from EvalCheck, TrueCheck, > > etc. does not include the test case input file name. So one ends up with > > something like the following in > > the TEST-org.apache.fop.layoutengine.LayoutEngineTestCase.txt file in > > builds/test-reports: > > > > Testcase: runTest[342] took 0.017 sec > > Caused an ERROR > > This is a known issue: > https://github.com/KentBeck/junit/issues/44 > And quite popular at that it seems: > https://github.com/KentBeck/junit/issues/search?q=parameterized+name > > In the meantime, I wrote the attached patch to IFParserTestCase. The > same could be done for LayoutEngineTestCase. > > > > > > Unfortunately, there is no way to correlate runTest[342] with a specific > > test case input file. It would be very useful (for AT and IF tests) to > also > > include the test case input file name/path in this output. Otherwise, one > > is forced to run junit in a debugger with a breakpoint on EvalCheck, > > TrueCheck, etc. > > Vincent >
Re: insufficient output from LayoutEngineTestCase failures
Hi Glenn, On 16/11/11 23:47, Glenn Adams wrote: > With the recent transition to JUnit4 runner and changes to > LayoutEngineTestCase, it is now someone difficult to determine (from output > data alone) which layoutengine test file failed when a regression occurs. > In particular, the exception trace generated from EvalCheck, TrueCheck, > etc. does not include the test case input file name. So one ends up with > something like the following in > the TEST-org.apache.fop.layoutengine.LayoutEngineTestCase.txt file in > builds/test-reports: > > Testcase: runTest[342] took 0.017 sec > Caused an ERROR This is a known issue: https://github.com/KentBeck/junit/issues/44 And quite popular at that it seems: https://github.com/KentBeck/junit/issues/search?q=parameterized+name In the meantime, I wrote the attached patch to IFParserTestCase. The same could be done for LayoutEngineTestCase. > > Unfortunately, there is no way to correlate runTest[342] with a specific > test case input file. It would be very useful (for AT and IF tests) to also > include the test case input file name/path in this output. Otherwise, one > is forced to run junit in a debugger with a breakpoint on EvalCheck, > TrueCheck, etc. Vincent Index: test/java/org/apache/fop/intermediate/IFParserTestCase.java === --- test/java/org/apache/fop/intermediate/IFParserTestCase.java (revision 1187300) +++ test/java/org/apache/fop/intermediate/IFParserTestCase.java (working copy) @@ -48,6 +48,9 @@ @RunWith(Parameterized.class) public class IFParserTestCase extends AbstractIFTestCase { +/** Set this to true to get the correspondence between test number and test file. */ +private static final boolean DEBUG = false; + /** * Gets the parameters for this test * @@ -56,9 +59,21 @@ */ @Parameters public static Collection getParameters() throws IOException { -return LayoutEngineTestUtils.getLayoutTestFiles(); +Collection testFiles = LayoutEngineTestUtils.getLayoutTestFiles(); +if (DEBUG) { +printFiles(testFiles); +} +return testFiles; } +private static void printFiles(Collection files) { +int index = 0; +for (File[] file : files) { +assert file.length == 1; +System.out.println(String.format("%3d %s", index++, file[0])); +} +} + /** * Constructor for the test suite that is used for each test file. * @param testFile the test file to run
Re: insufficient output from LayoutEngineTestCase failures
Hi Glenn, Yeah, JUnit4 doesn't allow one to parametrize test names this is a known issue. I did spend some time looking into how it would be possible, see https://issues.apache.org/bugzilla/show_bug.cgi?id=51928. Hopefully it will be rectified soon. Mehdi On 16 November 2011 23:47, Glenn Adams wrote: > With the recent transition to JUnit4 runner and changes to > LayoutEngineTestCase, it is now someone difficult to determine (from output > data alone) which layoutengine test file failed when a regression occurs. In > particular, the exception trace generated from EvalCheck, TrueCheck, etc. > does not include the test case input file name. So one ends up with > something like the following in > the TEST-org.apache.fop.layoutengine.LayoutEngineTestCase.txt file in > builds/test-reports: > Testcase: runTest[342] took 0.017 sec > Caused an ERROR > Expected XPath expression to evaluate to '2', but got '' (XPath: > //pageViewport[@nr=1]/page/regionViewport/regionBody/mainReference/span/flow/block/lineArea/inlineparent/text) > java.lang.RuntimeException: Expected XPath expression to evaluate to '2', > but got '' (XPath: > //pageViewport[@nr=1]/page/regionViewport/regionBody/mainReference/span/flow/block/lineArea/inlineparent/tex\ > t) > at org.apache.fop.layoutengine.EvalCheck.doCheck(EvalCheck.java:86) > at org.apache.fop.layoutengine.EvalCheck.check(EvalCheck.java:60) > at > org.apache.fop.layoutengine.LayoutEngineTestCase.doATChecks(LayoutEngineTestCase.java:258) > at > org.apache.fop.layoutengine.LayoutEngineTestCase.checkAll(LayoutEngineTestCase.java:191) > at > org.apache.fop.layoutengine.LayoutEngineTestCase.runTest(LayoutEngineTestCase.java:172) > Testcase: runTest[343] took 0.012 sec > Unfortunately, there is no way to correlate runTest[342] with a specific > test case input file. It would be very useful (for AT and IF tests) to also > include the test case input file name/path in this output. Otherwise, one is > forced to run junit in a debugger with a breakpoint on EvalCheck, TrueCheck, > etc. > >
insufficient output from LayoutEngineTestCase failures
With the recent transition to JUnit4 runner and changes to LayoutEngineTestCase, it is now someone difficult to determine (from output data alone) which layoutengine test file failed when a regression occurs. In particular, the exception trace generated from EvalCheck, TrueCheck, etc. does not include the test case input file name. So one ends up with something like the following in the TEST-org.apache.fop.layoutengine.LayoutEngineTestCase.txt file in builds/test-reports: Testcase: runTest[342] took 0.017 sec Caused an ERROR Expected XPath expression to evaluate to '2', but got '' (XPath: //pageViewport[@nr=1]/page/regionViewport/regionBody/mainReference/span/flow/block/lineArea/inlineparent/text) java.lang.RuntimeException: Expected XPath expression to evaluate to '2', but got '' (XPath: //pageViewport[@nr=1]/page/regionViewport/regionBody/mainReference/span/flow/block/lineArea/inlineparent/tex\ t) at org.apache.fop.layoutengine.EvalCheck.doCheck(EvalCheck.java:86) at org.apache.fop.layoutengine.EvalCheck.check(EvalCheck.java:60) at org.apache.fop.layoutengine.LayoutEngineTestCase.doATChecks(LayoutEngineTestCase.java:258) at org.apache.fop.layoutengine.LayoutEngineTestCase.checkAll(LayoutEngineTestCase.java:191) at org.apache.fop.layoutengine.LayoutEngineTestCase.runTest(LayoutEngineTestCase.java:172) Testcase: runTest[343] took 0.012 sec Unfortunately, there is no way to correlate runTest[342] with a specific test case input file. It would be very useful (for AT and IF tests) to also include the test case input file name/path in this output. Otherwise, one is forced to run junit in a debugger with a breakpoint on EvalCheck, TrueCheck, etc.