Thanks Jose! this is an improvement. In my experience, broken tests are introduced and fixed in mesa on a daily basis. This has a few consequences:
- On a daily basis, I look at failures and update the expected pass/fails depending on whether it is a new test or a regression. Much of this process is automated. - Branches quickly diverge on the basis of passing/failing tests. Having separate pass/fail configs on release branches is unmanageable. To account for this, my automation records the relevant commit sha as the value in the config file (the key is the test name). I post-process the junit xml to filter out test failures with commits that occurred after the branch point. - for platforms that are too slow to build each checkin, I run an automated bisect which builds/tests in jenkins, then updates config files. - Our platform matrix generates over 350k unskipped tests for each build. We filter out skipped tests due to the memory consumption on jenkins when displaying this many tests. I am interested in learning more about your test system, and sharing lessons learned / techniques. -Mark Reviewed-by: Mark Janes <[email protected]> Jose Fonseca <[email protected]> writes: > I recently tried the junit backend's ability to ignore expected > failures/crashes and found it a godsend -- instead of having to look as > test graph results periodically, I can just tell jenkins to email me > when things go south. > > The only drawback is that by reporting the expected issues as passing it > makes it too easy to forget about them and misinterpret the pass-rates. > So this change modifies the junit backend to report the expected issues > as skipped, making it more obvious when looking at the test graphs that > these tests are not really passing, and that whatever functionality they > target is not being fully covered. > > This change also makes use of the junit `message` attribute to explain > the reason of the skip. (In fact, we could consider using the `message` > attribute on other kind of failures to inform the piglit result, instead > of using the non-standard `type`.) > --- > framework/backends/junit.py | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > > diff --git a/framework/backends/junit.py b/framework/backends/junit.py > index 82f9c29..53b6086 100644 > --- a/framework/backends/junit.py > +++ b/framework/backends/junit.py > @@ -129,17 +129,19 @@ class JUnitBackend(FileBackend): > # Add relevant result value, if the result is pass then it > doesn't > # need one of these statuses > if data['result'] == 'skip': > - etree.SubElement(element, 'skipped') > + res = etree.SubElement(element, 'skipped') > > elif data['result'] in ['warn', 'fail', 'dmesg-warn', > 'dmesg-fail']: > if expected_result == "failure": > err.text += "\n\nWARN: passing test as an expected > failure" > + res = etree.SubElement(element, 'skipped', > message='expected failure') > else: > res = etree.SubElement(element, 'failure') > > elif data['result'] == 'crash': > if expected_result == "error": > err.text += "\n\nWARN: passing test as an expected crash" > + res = etree.SubElement(element, 'skipped', > message='expected crash') > else: > res = etree.SubElement(element, 'error') > > -- > 2.1.0 _______________________________________________ Piglit mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/piglit
