Title: [271955] trunk/Tools
- Revision
- 271955
- Author
- [email protected]
- Date
- 2021-01-27 09:14:14 -0800 (Wed, 27 Jan 2021)
Log Message
[GTK] run-gtk-tests: Crashing and non-existent tests should not count as a pass
https://bugs.webkit.org/show_bug.cgi?id=220863
Reviewed by Michael Catanzaro.
The current implementation doesn't account for test binaries that have
not emitted any subtests. This is the case when a test binary doesn't
exist or it crashes.
Also, in the latter case, the stderr of the crashed processes was not
being outputted to the user, masking a crashing test binary as a
passing test.
This patch fixes both issues: It adds several warning prints when
binaries can't be run or fail to add any subtest, emitting the stderr
generated by the failing or crashing binary.
* glib/api_test_runner.py:
(TestRunner._run_test):
(TestRunner.run_tests):
* glib/glib_test_runner.py:
(GLibTestRunner.run):
Modified Paths
Diff
Modified: trunk/Tools/ChangeLog (271954 => 271955)
--- trunk/Tools/ChangeLog 2021-01-27 16:12:27 UTC (rev 271954)
+++ trunk/Tools/ChangeLog 2021-01-27 17:14:14 UTC (rev 271955)
@@ -1,3 +1,28 @@
+2021-01-27 Alicia Boya García <[email protected]>
+
+ [GTK] run-gtk-tests: Crashing and non-existent tests should not count as a pass
+ https://bugs.webkit.org/show_bug.cgi?id=220863
+
+ Reviewed by Michael Catanzaro.
+
+ The current implementation doesn't account for test binaries that have
+ not emitted any subtests. This is the case when a test binary doesn't
+ exist or it crashes.
+
+ Also, in the latter case, the stderr of the crashed processes was not
+ being outputted to the user, masking a crashing test binary as a
+ passing test.
+
+ This patch fixes both issues: It adds several warning prints when
+ binaries can't be run or fail to add any subtest, emitting the stderr
+ generated by the failing or crashing binary.
+
+ * glib/api_test_runner.py:
+ (TestRunner._run_test):
+ (TestRunner.run_tests):
+ * glib/glib_test_runner.py:
+ (GLibTestRunner.run):
+
2021-01-27 Youenn Fablet <[email protected]>
Enable GPU WebRTC codecs in WebKitTestRunner
Modified: trunk/Tools/glib/api_test_runner.py (271954 => 271955)
--- trunk/Tools/glib/api_test_runner.py 2021-01-27 16:12:27 UTC (rev 271954)
+++ trunk/Tools/glib/api_test_runner.py 2021-01-27 17:14:14 UTC (rev 271955)
@@ -276,6 +276,7 @@
if self.is_qt_test(test_program):
return self._run_test_qt(test_program)
+ sys.stderr.write("WARNING: %s doesn't seem to be a supported test program.\n" % test_program)
return {}
def run_tests(self):
@@ -302,6 +303,11 @@
skipped_subtests = self._test_cases_to_skip(test)
number_of_total_tests += len(skipped_subtests)
results = self._run_test(test, subtests, skipped_subtests)
+ if len(results) == 0:
+ # No subtests were emitted, either the test binary didn't exist, or we don't know how to run it, or it crashed.
+ sys.stderr.write("ERROR: %s failed to run, as it didn't emit any subtests.\n" % test)
+ crashed_tests[test] = ["(problem in test executable)"]
+ continue
number_of_executed_subtests_for_test = len(results)
if number_of_executed_subtests_for_test > 1:
number_of_executed_tests += number_of_executed_subtests_for_test
Modified: trunk/Tools/glib/glib_test_runner.py (271954 => 271955)
--- trunk/Tools/glib/glib_test_runner.py 2021-01-27 16:12:27 UTC (rev 271954)
+++ trunk/Tools/glib/glib_test_runner.py 2021-01-27 17:14:14 UTC (rev 271955)
@@ -284,6 +284,12 @@
self._results['afterAll'] = 'CRASH'
return self._results
+ if len(self._results) == 0:
+ # Normally stderr is checked after a subtest has been parsed. If no subtests have been parsed
+ # chances are something went wrong with the test executable itself and we should print stderr
+ # to the user.
+ sys.stderr.write(self._read_from_stderr(self._stderr_fd))
+
self._stderr_fd = None
if need_restart:
_______________________________________________
webkit-changes mailing list
[email protected]
https://lists.webkit.org/mailman/listinfo/webkit-changes
