Title: [136437] trunk/Tools
Revision
136437
Author
[email protected]
Date
2012-12-03 13:59:12 -0800 (Mon, 03 Dec 2012)

Log Message

nrwt: clean up exception propagation / handling for interrupts and early exits
https://bugs.webkit.org/show_bug.cgi?id=103830

Reviewed by Ojan Vafai.

The code for propagating early exits out of the test run was
convoluted and hard to follow. By adjusting the WorkerException to
derive from BaseException rather than Exception (which is a fair
way of representing an error that was really unexpected/unrecoverable
anyway) and pushing the TestRunInterruptedException into the
ResultSummary (which is also fair since it was more of a result
than an exception) this allows us to consolidate and clean up
a bunch of the try/catch logic and pass fewer state flags around.

There are no real functional changes to this patch; there's an
API change in that we stop exporting TestRunInterruptedException
and WorkerException out of the Manager, and the existence of the
MessagePool is now completely hidden.

* Scripts/webkitpy/common/message_pool.py:
(WorkerException):
* Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py:
(LayoutTestRunner.run_tests):
* Scripts/webkitpy/layout_tests/controllers/manager.py:
(summarize_results):
(Manager.run):
* Scripts/webkitpy/layout_tests/controllers/manager_unittest.py:
(ResultSummaryTest.summarized_results):
* Scripts/webkitpy/layout_tests/models/result_summary.py:
(ResultSummary.__init__):
* Scripts/webkitpy/layout_tests/run_webkit_tests.py:
(run):
(main):
* Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py:
(MainTest.test_exception_raised.of):
(MainTest.test_exception_raised):
(MainTest.test_run_force):
* Scripts/webkitpy/layout_tests/views/printing_unittest.py:
(Testprinter.test_print_unexpected_results.get_unexpected_results):
(Testprinter):

Modified Paths

Diff

Modified: trunk/Tools/ChangeLog (136436 => 136437)


--- trunk/Tools/ChangeLog	2012-12-03 21:48:41 UTC (rev 136436)
+++ trunk/Tools/ChangeLog	2012-12-03 21:59:12 UTC (rev 136437)
@@ -1,3 +1,46 @@
+2012-12-03  Dirk Pranke  <[email protected]>
+
+        nrwt: clean up exception propagation / handling for interrupts and early exits
+        https://bugs.webkit.org/show_bug.cgi?id=103830
+
+        Reviewed by Ojan Vafai.
+
+        The code for propagating early exits out of the test run was
+        convoluted and hard to follow. By adjusting the WorkerException to
+        derive from BaseException rather than Exception (which is a fair
+        way of representing an error that was really unexpected/unrecoverable
+        anyway) and pushing the TestRunInterruptedException into the
+        ResultSummary (which is also fair since it was more of a result
+        than an exception) this allows us to consolidate and clean up
+        a bunch of the try/catch logic and pass fewer state flags around.
+
+        There are no real functional changes to this patch; there's an
+        API change in that we stop exporting TestRunInterruptedException
+        and WorkerException out of the Manager, and the existence of the
+        MessagePool is now completely hidden.
+
+        * Scripts/webkitpy/common/message_pool.py:
+        (WorkerException):
+        * Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py:
+        (LayoutTestRunner.run_tests):
+        * Scripts/webkitpy/layout_tests/controllers/manager.py:
+        (summarize_results):
+        (Manager.run):
+        * Scripts/webkitpy/layout_tests/controllers/manager_unittest.py:
+        (ResultSummaryTest.summarized_results):
+        * Scripts/webkitpy/layout_tests/models/result_summary.py:
+        (ResultSummary.__init__):
+        * Scripts/webkitpy/layout_tests/run_webkit_tests.py:
+        (run):
+        (main):
+        * Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py:
+        (MainTest.test_exception_raised.of):
+        (MainTest.test_exception_raised):
+        (MainTest.test_run_force):
+        * Scripts/webkitpy/layout_tests/views/printing_unittest.py:
+        (Testprinter.test_print_unexpected_results.get_unexpected_results):
+        (Testprinter):
+
 2012-12-03  Julien BRIANCEAU   <[email protected]>
 
         Add Qt/SH4 build slave to build.webkit.org

Modified: trunk/Tools/Scripts/webkitpy/common/message_pool.py (136436 => 136437)


--- trunk/Tools/Scripts/webkitpy/common/message_pool.py	2012-12-03 21:48:41 UTC (rev 136436)
+++ trunk/Tools/Scripts/webkitpy/common/message_pool.py	2012-12-03 21:59:12 UTC (rev 136437)
@@ -184,7 +184,7 @@
             pass
 
 
-class WorkerException(Exception):
+class WorkerException(BaseException):
     """Raised when we receive an unexpected/unknown exception from a worker."""
     pass
 

Modified: trunk/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py (136436 => 136437)


--- trunk/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py	2012-12-03 21:48:41 UTC (rev 136436)
+++ trunk/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py	2012-12-03 21:59:12 UTC (rev 136437)
@@ -81,16 +81,13 @@
         self._filesystem = self._port.host.filesystem
 
     def run_tests(self, test_inputs, expectations, result_summary, num_workers, needs_http, needs_websockets, retrying):
-        """Returns a tuple of (interrupted, keyboard_interrupted, thread_timings, test_timings, individual_test_timings):
-            interrupted is whether the run was interrupted
-            keyboard_interrupted is whether the interruption was because someone typed Ctrl^C
+        """Returns a tuple of (thread_timings, test_timings, individual_test_timings):
             thread_timings is a list of dicts with the total runtime
                 of each thread with 'name', 'num_tests', 'total_time' properties
             test_timings is a list of timings for each sharded subdirectory
                 of the form [time, directory_name, num_tests]
             individual_test_timings is a list of run times for each test
                 in the form {filename:filename, test_run_time:test_run_time}
-            result_summary: summary object to populate with the results
         """
         self._current_result_summary = result_summary
         self._expectations = expectations
@@ -107,9 +104,6 @@
         self._has_http_lock = False
         self._remaining_locked_shards = []
 
-        keyboard_interrupted = False
-        interrupted = False
-
         self._printer.write_update('Sharding tests ...')
         locked_shards, unlocked_shards = self._sharder.shard_tests(test_inputs, int(self._options.child_processes), self._options.fully_parallel)
 
@@ -130,28 +124,28 @@
         self._printer.print_workers_and_shards(num_workers, len(all_shards), len(locked_shards))
 
         if self._options.dry_run:
-            return (keyboard_interrupted, interrupted, self._worker_stats.values(), self._group_stats, self._all_results)
+            return (self._worker_stats.values(), self._group_stats, self._all_results)
 
         self._printer.write_update('Starting %s ...' % grammar.pluralize('worker', num_workers))
 
         try:
             with message_pool.get(self, self._worker_factory, num_workers, self._port.worker_startup_delay_secs(), self._port.host) as pool:
                 pool.run(('test_list', shard.name, shard.test_inputs) for shard in all_shards)
+        except TestRunInterruptedException, e:
+            _log.warning(e.reason)
+            result_summary.interrupted = True
         except KeyboardInterrupt:
             self._printer.flush()
             self._printer.writeln('Interrupted, exiting ...')
-            keyboard_interrupted = True
-        except TestRunInterruptedException, e:
-            _log.warning(e.reason)
-            interrupted = True
+            raise
         except Exception, e:
             _log.debug('%s("%s") raised, exiting' % (e.__class__.__name__, str(e)))
             raise
         finally:
             self.stop_servers_with_lock()
 
-        # FIXME: should this be a class instead of a tuple?
-        return (interrupted, keyboard_interrupted, self._worker_stats.values(), self._group_stats, self._all_results)
+        # FIXME: Move these stats into ResultSummary and return that.
+        return (self._worker_stats.values(), self._group_stats, self._all_results)
 
     def _worker_factory(self, worker_connection):
         results_directory = self._results_directory

Modified: trunk/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py (136436 => 136437)


--- trunk/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py	2012-12-03 21:48:41 UTC (rev 136436)
+++ trunk/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py	2012-12-03 21:59:12 UTC (rev 136437)
@@ -46,7 +46,7 @@
 
 from webkitpy.common import message_pool
 from webkitpy.layout_tests.controllers.layout_test_finder import LayoutTestFinder
-from webkitpy.layout_tests.controllers.layout_test_runner import LayoutTestRunner, TestRunInterruptedException, WorkerException
+from webkitpy.layout_tests.controllers.layout_test_runner import LayoutTestRunner
 from webkitpy.layout_tests.controllers.test_result_writer import TestResultWriter
 from webkitpy.layout_tests.layout_package import json_layout_results_generator
 from webkitpy.layout_tests.layout_package import json_results_generator
@@ -107,7 +107,7 @@
 # or split off from Manager onto another helper class, but should not be a free function.
 # Most likely this should be made into its own class, and this super-long function
 # split into many helper functions.
-def summarize_results(port_obj, expectations, result_summary, retry_summary, test_timings, only_unexpected, interrupted):
+def summarize_results(port_obj, expectations, result_summary, retry_summary, test_timings, only_unexpected):
     """Summarize failing results as a dict.
 
     FIXME: split this data structure into a separate class?
@@ -230,7 +230,7 @@
     results['num_missing'] = num_missing
     results['num_regressions'] = num_regressions
     results['uses_expectations_file'] = port_obj.uses_test_expectations_file()
-    results['interrupted'] = interrupted  # Does results.html have enough information to compute this itself? (by checking total number of results vs. total number of tests?)
+    results['interrupted'] = result_summary.interrupted  # Does results.html have enough information to compute this itself? (by checking total number of results vs. total number of tests?)
     results['layout_tests_dir'] = port_obj.layout_tests_dir()
     results['has_wdiff'] = port_obj.wdiff_available()
     results['has_pretty_patch'] = port_obj.pretty_patch_available()
@@ -402,46 +402,47 @@
             return -1
 
         start_time = time.time()
+        retry_summary = result_summary
 
-        interrupted, keyboard_interrupted, thread_timings, test_timings, individual_test_timings = \
-            self._run_tests(self._test_names, result_summary, int(self._options.child_processes), retrying=False)
+        # FIXME: move these fields into result_summary so we don't have to pass them around.
+        thread_timings = test_timings = {}
+        individual_test_timings = []
+        try:
+            thread_timings, test_timings, individual_test_timings = \
+                self._run_tests(self._test_names, result_summary, int(self._options.child_processes), retrying=False)
 
-        # We exclude the crashes from the list of results to retry, because
-        # we want to treat even a potentially flaky crash as an error.
+            # We exclude the crashes from the list of results to retry, because
+            # we want to treat even a potentially flaky crash as an error.
 
-        failures = self._failed_test_names(result_summary, include_crashes=self._port.should_retry_crashes(), include_missing=False)
-        if self._options.retry_failures and failures and not interrupted and not keyboard_interrupted:
-            _log.info('')
-            _log.info("Retrying %d unexpected failure(s) ..." % len(failures))
-            _log.info('')
-            # Note that we don't care about the values returned from the retry pass.
-            retry_summary = ResultSummary(self._expectations, failures, 1, set())
-            _ = self._run_tests(failures, retry_summary, 1, retrying=True)
-        else:
-            retry_summary = result_summary
+            failures = self._failed_test_names(result_summary, include_crashes=self._port.should_retry_crashes(), include_missing=False)
+            if self._options.retry_failures and failures and not result_summary.interrupted:
+                _log.info('')
+                _log.info("Retrying %d unexpected failure(s) ..." % len(failures))
+                _log.info('')
+                # Note that we don't care about the values returned from the retry pass.
+                retry_summary = ResultSummary(self._expectations, failures, 1, set())
+                _ = self._run_tests(failures, retry_summary, 1, retrying=True)
+        finally:
+            # If we are ctrl-c'ed; we still want to try and print the results we got, and clean up,
+            # but we don't want to upload anything.
+            end_time = time.time()
 
-        end_time = time.time()
+            # Some crash logs can take a long time to be written out so look
+            # for new logs after the test run finishes.
+            self._look_for_new_crash_logs(result_summary, start_time)
+            self._look_for_new_crash_logs(retry_summary, start_time)
+            self._clean_up_run()
 
-        # Some crash logs can take a long time to be written out so look
-        # for new logs after the test run finishes.
-        self._look_for_new_crash_logs(result_summary, start_time)
-        self._look_for_new_crash_logs(retry_summary, start_time)
-        self._clean_up_run()
+            unexpected_results = summarize_results(self._port, self._expectations, result_summary, retry_summary, individual_test_timings, _only_unexpected_=True)
 
-        unexpected_results = summarize_results(self._port, self._expectations, result_summary, retry_summary, individual_test_timings, _only_unexpected_=True, interrupted=interrupted)
+            self._printer.print_results(end_time - start_time, thread_timings, test_timings, individual_test_timings, result_summary, unexpected_results)
 
-        self._printer.print_results(end_time - start_time, thread_timings, test_timings, individual_test_timings, result_summary, unexpected_results)
-
-        # Re-raise a KeyboardInterrupt if necessary so the caller can handle it.
-        if keyboard_interrupted:
-            raise KeyboardInterrupt
-
         # FIXME: remove record_results. It's just used for testing. There's no need
         # for it to be a commandline argument.
-        if (self._options.record_results and not self._options.dry_run and not keyboard_interrupted):
+        if self._options.record_results and not self._options.dry_run:
             self._port.print_leaks_summary()
             # Write the same data to log files and upload generated JSON files to appengine server.
-            summarized_results = summarize_results(self._port, self._expectations, result_summary, retry_summary, individual_test_timings, _only_unexpected_=False, interrupted=interrupted)
+            summarized_results = summarize_results(self._port, self._expectations, result_summary, retry_summary, individual_test_timings, _only_unexpected_=False)
             self._upload_json_files(summarized_results, result_summary, individual_test_timings)
 
         # Write the summary to disk (results.html) and display it if requested.

Modified: trunk/Tools/Scripts/webkitpy/layout_tests/controllers/manager_unittest.py (136436 => 136437)


--- trunk/Tools/Scripts/webkitpy/layout_tests/controllers/manager_unittest.py	2012-12-03 21:48:41 UTC (rev 136436)
+++ trunk/Tools/Scripts/webkitpy/layout_tests/controllers/manager_unittest.py	2012-12-03 21:59:12 UTC (rev 136437)
@@ -178,8 +178,8 @@
             retry.add(self.get_result('passes/text.html'), True, test_is_slow)
             retry.add(self.get_result('failures/expected/timeout.html'), True, test_is_slow)
             retry.add(self.get_result('failures/expected/crash.html'), True, test_is_slow)
-        unexpected_results = summarize_results(port, exp, rs, retry, test_timings={}, _only_unexpected_=True, interrupted=False)
-        expected_results = summarize_results(port, exp, rs, retry, test_timings={}, _only_unexpected_=False, interrupted=False)
+        unexpected_results = summarize_results(port, exp, rs, retry, test_timings={}, _only_unexpected_=True)
+        expected_results = summarize_results(port, exp, rs, retry, test_timings={}, _only_unexpected_=False)
         return expected_results, unexpected_results
 
     def test_no_svn_revision(self):

Modified: trunk/Tools/Scripts/webkitpy/layout_tests/models/result_summary.py (136436 => 136437)


--- trunk/Tools/Scripts/webkitpy/layout_tests/models/result_summary.py	2012-12-03 21:48:41 UTC (rev 136436)
+++ trunk/Tools/Scripts/webkitpy/layout_tests/models/result_summary.py	2012-12-03 21:59:12 UTC (rev 136437)
@@ -56,6 +56,7 @@
         for timeline in TestExpectations.TIMELINES.values():
             self.tests_by_timeline[timeline] = expectations.get_tests_with_timeline(timeline)
         self.slow_tests = set()
+        self.interrupted = False
 
     def add(self, test_result, expected, test_is_slow):
         self.total_tests_by_expectation[test_result.type] += 1

Modified: trunk/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py (136436 => 136437)


--- trunk/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py	2012-12-03 21:48:41 UTC (rev 136436)
+++ trunk/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py	2012-12-03 21:59:12 UTC (rev 136437)
@@ -39,7 +39,7 @@
 
 from webkitpy.common.host import Host
 from webkitpy.common.system import stack_utils
-from webkitpy.layout_tests.controllers.manager import Manager, WorkerException, TestRunInterruptedException
+from webkitpy.layout_tests.controllers.manager import Manager
 from webkitpy.layout_tests.models import test_expectations
 from webkitpy.layout_tests.port import configuration_options, platform_options
 from webkitpy.layout_tests.views import printing
@@ -103,25 +103,15 @@
 
         # We wrap any parts of the run that are slow or likely to raise exceptions
         # in a try/finally to ensure that we clean up the logging configuration.
-        unexpected_result_count = -1
-
         manager = Manager(port, options, printer)
         printer.print_config(port.results_directory())
 
         unexpected_result_count = manager.run(args)
         _log.debug("Testing completed, Exit status: %d" % unexpected_result_count)
-    except Exception:
-        exception_type, exception_value, exception_traceback = sys.exc_info()
-        if exception_type not in (KeyboardInterrupt, TestRunInterruptedException, WorkerException):
-            print >> sys.stderr, '\n%s raised: %s' % (exception_type.__name__, exception_value)
-            stack_utils.log_traceback(_log.error, exception_traceback)
-        raise
+        return unexpected_result_count
     finally:
         printer.cleanup()
 
-    return unexpected_result_count
-
-
 def _set_up_derived_options(port, options):
     """Sets the options values that depend on other options values."""
     # We return a list of warnings to print after the printer is initialized.
@@ -426,36 +416,36 @@
 
 
 def main(argv=None):
+    options, args = parse_args(argv)
+    if options.platform and 'test' in options.platform:
+        # It's a bit lame to import mocks into real code, but this allows the user
+        # to run tests against the test platform interactively, which is useful for
+        # debugging test failures.
+        from webkitpy.common.host_mock import MockHost
+        host = MockHost()
+    else:
+        host = Host()
+
     try:
-        options, args = parse_args(argv)
-        if options.platform and 'test' in options.platform:
-            # It's a bit lame to import mocks into real code, but this allows the user
-            # to run tests against the test platform interactively, which is useful for
-            # debugging test failures.
-            from webkitpy.common.host_mock import MockHost
-            host = MockHost()
-        else:
-            host = Host()
         port = host.port_factory.get(options.platform, options)
     except NotImplementedError, e:
         # FIXME: is this the best way to handle unsupported port names?
         print >> sys.stderr, str(e)
         return EXCEPTIONAL_EXIT_STATUS
+
+    logging.getLogger().setLevel(logging.DEBUG if options.debug_rwt_logging else logging.INFO)
+    try:
+        return run(port, options, args)
     except Exception, e:
         print >> sys.stderr, '\n%s raised: %s' % (e.__class__.__name__, str(e))
         traceback.print_exc(file=sys.stderr)
         raise
 
-    logging.getLogger().setLevel(logging.DEBUG if options.debug_rwt_logging else logging.INFO)
-    return run(port, options, args)
 
-
 if '__main__' == __name__:
     try:
-        return_code = main()
-    except BaseException, e:
-        if e.__class__ in (KeyboardInterrupt, TestRunInterruptedException):
-            sys.exit(INTERRUPTED_EXIT_STATUS)
+        sys.exit(main())
+    except KeyboardInterrupt:
+        sys.exit(INTERRUPTED_EXIT_STATUS)
+    except:
         sys.exit(EXCEPTIONAL_EXIT_STATUS)
-
-    sys.exit(return_code)

Modified: trunk/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py (136436 => 136437)


--- trunk/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py	2012-12-03 21:48:41 UTC (rev 136436)
+++ trunk/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py	2012-12-03 21:59:12 UTC (rev 136437)
@@ -52,7 +52,6 @@
 
 from webkitpy.layout_tests import port
 from webkitpy.layout_tests import run_webkit_tests
-from webkitpy.layout_tests.controllers.manager import WorkerException
 from webkitpy.layout_tests.port import Port
 from webkitpy.layout_tests.port.test import TestPort, TestDriver
 from webkitpy.test.skip import skip_if
@@ -342,13 +341,13 @@
         # is actually useful in testing.
         #
         # Exceptions raised in a separate process are re-packaged into
-        # WorkerExceptions, which have a string capture of the stack which can
+        # WorkerExceptions (a subclass of BaseException), which have a string capture of the stack which can
         # be printed, but don't display properly in the unit test exception handlers.
-        self.assertRaises(ValueError, logging_run,
+        self.assertRaises(BaseException, logging_run,
             ['failures/expected/exception.html', '--child-processes', '1'], tests_included=True)
 
         if self.should_test_processes:
-            self.assertRaises(WorkerException, logging_run,
+            self.assertRaises(BaseException, logging_run,
                 ['--child-processes', '2', '--force', 'failures/expected/exception.html', 'passes/text.html'], tests_included=True, shared_port=False)
 
     def test_full_results_html(self):
@@ -499,7 +498,6 @@
         # This raises an exception because we run
         # failures/expected/exception.html, which is normally SKIPped.
 
-        # See also the comments in test_exception_raised() about ValueError vs. WorkerException.
         self.assertRaises(ValueError, logging_run, ['--force'])
 
     def test_run_part(self):

Modified: trunk/Tools/Scripts/webkitpy/layout_tests/views/printing_unittest.py (136436 => 136437)


--- trunk/Tools/Scripts/webkitpy/layout_tests/views/printing_unittest.py	2012-12-03 21:48:41 UTC (rev 136436)
+++ trunk/Tools/Scripts/webkitpy/layout_tests/views/printing_unittest.py	2012-12-03 21:59:12 UTC (rev 136437)
@@ -200,7 +200,7 @@
                 retry.add(self.get_result('passes/text.html'), True, test_is_slow)
                 retry.add(self.get_result('failures/expected/timeout.html'), True, test_is_slow)
                 retry.add(self.get_result('failures/expected/crash.html'), True, test_is_slow)
-            unexpected_results = manager.summarize_results(self._port, exp, rs, retry, test_timings={}, _only_unexpected_=True, interrupted=False)
+            unexpected_results = manager.summarize_results(self._port, exp, rs, retry, test_timings={}, _only_unexpected_=True)
             return unexpected_results
 
         tests = ['passes/text.html', 'failures/expected/timeout.html', 'failures/expected/crash.html']
_______________________________________________
webkit-changes mailing list
[email protected]
http://lists.webkit.org/mailman/listinfo/webkit-changes

Reply via email to