[webkit-changes] [109266] trunk/Tools

Wed, 29 Feb 2012 14:43:29 -0800

Title: [109266] trunk/Tools
Revision
109266
Author
[email protected]
Date
2012-02-29 14:43:27 -0800 (Wed, 29 Feb 2012)

Log Message

nrwt: implement simple 'virtual test suite' support
https://bugs.webkit.org/show_bug.cgi?id=79737

Reviewed by Adam Barth.

Add very basic support for 'virtual test suites' to NRWT:
a virtual suite is a directory that may contain new baselines,
contains a pointer to a "base" directory of tests, and an
optional list of command line arguments to pass to DRT;
it gives us a way to run all of the tests in a given directory
multiple times with multiple (potentially differing) sets of
results and baselines.

This patch implements the support needed for this feature, and
some basic tests, but no actual port will use the feature yet.

This is probably the simplest implementation possible - the
list of virtual suites will be hard-coded into the port's
implementation. One can imagine a more data-driven approach
where the suite instructions are stored in a manifest file
either in LayoutTests (all suites in one file), or per-port,
or per-directory. If this feature ends up being useful we
should probably add something like that so people don't have
to hunt around in the code to add suites.

* Scripts/webkitpy/layout_tests/port/base.py:
(Port.__init__):
(Port.expected_filename):
(Port.tests):
(Port):
(Port._real_tests):
(Port._driver_class):
(Port.virtual_test_suites):
(Port.virtual_suite):
(Port.virtual_suite.VirtualTestSuite):
(Port.virtual_suite.VirtualTestSuite.__init__):
(Port.virtual_suite.VirtualTestSuite.__repr__):
(Port.populated_virtual_test_suites):
(Port._virtual_tests):
(Port.lookup_virtual_test_base):
(Port.lookup_virtual_test_args):
* Scripts/webkitpy/layout_tests/port/base_unittest.py:
(PortTest.test_find_with_skipped_directories):
* Scripts/webkitpy/layout_tests/port/driver.py:
(DriverInput.__init__):
(DriverProxy.run_test):
* Scripts/webkitpy/layout_tests/port/test.py:
(TestPort.virtual_test_suites):
(TestDriver.run_test):
* Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py:
(MainTest.test_all):
(MainTest.test_virtual):

Modified Paths

Diff

Modified: trunk/Tools/ChangeLog (109265 => 109266)


--- trunk/Tools/ChangeLog	2012-02-29 22:43:05 UTC (rev 109265)
+++ trunk/Tools/ChangeLog	2012-02-29 22:43:27 UTC (rev 109266)
@@ -1,5 +1,60 @@
 2012-02-29  Dirk Pranke  <[email protected]>
 
+        nrwt: implement simple 'virtual test suite' support
+        https://bugs.webkit.org/show_bug.cgi?id=79737
+
+        Reviewed by Adam Barth.
+
+        Add very basic support for 'virtual test suites' to NRWT:
+        a virtual suite is a directory that may contain new baselines,
+        contains a pointer to a "base" directory of tests, and an
+        optional list of command line arguments to pass to DRT;
+        it gives us a way to run all of the tests in a given directory
+        multiple times with multiple (potentially differing) sets of
+        results and baselines.
+
+        This patch implements the support needed for this feature, and
+        some basic tests, but no actual port will use the feature yet.
+
+        This is probably the simplest implementation possible - the
+        list of virtual suites will be hard-coded into the port's
+        implementation. One can imagine a more data-driven approach
+        where the suite instructions are stored in a manifest file
+        either in LayoutTests (all suites in one file), or per-port,
+        or per-directory. If this feature ends up being useful we
+        should probably add something like that so people don't have
+        to hunt around in the code to add suites.
+
+        * Scripts/webkitpy/layout_tests/port/base.py:
+        (Port.__init__):
+        (Port.expected_filename):
+        (Port.tests):
+        (Port):
+        (Port._real_tests):
+        (Port._driver_class):
+        (Port.virtual_test_suites):
+        (Port.virtual_suite):
+        (Port.virtual_suite.VirtualTestSuite):
+        (Port.virtual_suite.VirtualTestSuite.__init__):
+        (Port.virtual_suite.VirtualTestSuite.__repr__):
+        (Port.populated_virtual_test_suites):
+        (Port._virtual_tests):
+        (Port.lookup_virtual_test_base):
+        (Port.lookup_virtual_test_args):
+        * Scripts/webkitpy/layout_tests/port/base_unittest.py:
+        (PortTest.test_find_with_skipped_directories):
+        * Scripts/webkitpy/layout_tests/port/driver.py:
+        (DriverInput.__init__):
+        (DriverProxy.run_test):
+        * Scripts/webkitpy/layout_tests/port/test.py:
+        (TestPort.virtual_test_suites):
+        (TestDriver.run_test):
+        * Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py:
+        (MainTest.test_all):
+        (MainTest.test_virtual):
+
+2012-02-29  Dirk Pranke  <[email protected]>
+
         nrwt: support more than two drivers in DriverProxy
         https://bugs.webkit.org/show_bug.cgi?id=79736

Modified: trunk/Tools/Scripts/webkitpy/layout_tests/port/base.py (109265 => 109266)


--- trunk/Tools/Scripts/webkitpy/layout_tests/port/base.py	2012-02-29 22:43:05 UTC (rev 109265)
+++ trunk/Tools/Scripts/webkitpy/layout_tests/port/base.py	2012-02-29 22:43:27 UTC (rev 109266)
@@ -379,6 +379,11 @@
         platform_dir, baseline_filename = self.expected_baselines(test_name, suffix)[0]
         if platform_dir:
             return self._filesystem.join(platform_dir, baseline_filename)
+
+        actual_test_name = self.lookup_virtual_test_base(test_name)
+        if actual_test_name:
+            return self.expected_filename(actual_test_name, suffix)
+
         return self._filesystem.join(self.layout_tests_dir(), baseline_filename)
 
     def expected_checksum(self, test_name):
@@ -459,6 +464,9 @@
 
     def tests(self, paths):
         """Return the list of tests found."""
+        return self._real_tests(paths).union(self._virtual_tests(paths, self.populated_virtual_test_suites()))
+
+    def _real_tests(self, paths):
         # When collecting test cases, skip these directories
         skipped_directories = set(['.svn', '_svn', 'resources', 'script-tests', 'reference', 'reftest'])
         files = find_files.find(self._filesystem, self.layout_tests_dir(), paths, skipped_directories, Port._is_test_file)
@@ -1043,3 +1051,53 @@
     def _driver_class(self):
         """Returns the port's driver implementation."""
         raise NotImplementedError('Port._driver_class')
+
+    def virtual_test_suites(self):
+        return []
+
+    @memoized
+    def populated_virtual_test_suites(self):
+        suites = self.virtual_test_suites()
+
+        # Sanity-check the suites to make sure they don't point to other suites.
+        suite_dirs = [suite.name for suite in suites]
+        for suite in suites:
+            assert suite.base not in suite_dirs
+
+        for suite in suites:
+            base_tests = self._real_tests([suite.base])
+            suite.tests = {}
+            for test in base_tests:
+                suite.tests[test.replace(suite.base, suite.name)] = test
+        return suites
+
+    def _virtual_tests(self, paths, suites):
+        virtual_tests = set()
+        for suite in suites:
+            for test in suite.tests:
+                if any(test.startswith(p) for p in paths):
+                    virtual_tests.add(test)
+        return virtual_tests
+
+    def lookup_virtual_test_base(self, test_name):
+        for suite in self.populated_virtual_test_suites():
+            if test_name.startswith(suite.name):
+                return suite.tests.get(test_name)
+        return None
+
+    def lookup_virtual_test_args(self, test_name):
+        for suite in self.populated_virtual_test_suites():
+            if test_name.startswith(suite.name):
+                return suite.args
+        return []
+
+
+class VirtualTestSuite(object):
+    def __init__(self, name, base, args, tests=None):
+        self.name = name
+        self.base = base
+        self.args = args
+        self.tests = tests or set()
+
+    def __repr__(self):
+        return "VirtualTestSuite('%s', '%s', %s)" % (self.name, self.base, self.args)

Modified: trunk/Tools/Scripts/webkitpy/layout_tests/port/base_unittest.py (109265 => 109266)


--- trunk/Tools/Scripts/webkitpy/layout_tests/port/base_unittest.py	2012-02-29 22:43:05 UTC (rev 109265)
+++ trunk/Tools/Scripts/webkitpy/layout_tests/port/base_unittest.py	2012-02-29 22:43:27 UTC (rev 109266)
@@ -312,7 +312,7 @@
 
     def test_find_with_skipped_directories(self):
         port = self.make_port(with_tests=True)
-        tests = port.tests('userscripts')
+        tests = port.tests(['userscripts'])
         self.assertTrue('userscripts/resources/iframe.html' not in tests)
 
     def test_find_with_skipped_directories_2(self):

Modified: trunk/Tools/Scripts/webkitpy/layout_tests/port/driver.py (109265 => 109266)


--- trunk/Tools/Scripts/webkitpy/layout_tests/port/driver.py	2012-02-29 22:43:05 UTC (rev 109265)
+++ trunk/Tools/Scripts/webkitpy/layout_tests/port/driver.py	2012-02-29 22:43:27 UTC (rev 109266)
@@ -26,6 +26,7 @@
 # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
+import copy
 import re
 import shlex
 
@@ -33,11 +34,12 @@
 
 
 class DriverInput(object):
-    def __init__(self, test_name, timeout, image_hash, is_reftest):
+    def __init__(self, test_name, timeout, image_hash, is_reftest, args=None):
         self.test_name = test_name
         self.timeout = timeout  # in ms
         self.image_hash = image_hash
         self.is_reftest = is_reftest
+        self.args = args or []
 
 
 class DriverOutput(object):
@@ -199,8 +201,15 @@
         return self._driver.uri_to_test(uri)
 
     def run_test(self, driver_input):
+        base = self._port.lookup_virtual_test_base(driver_input.test_name)
+        if base:
+            virtual_driver_input = copy.copy(driver_input)
+            virtual_driver_input.test_name = base
+            virtual_driver_input.args = self._port.lookup_virtual_test_args(driver_input.test_name)
+            return self.run_test(virtual_driver_input)
+
         pixel_tests_needed = self._pixel_tests or driver_input.is_reftest
-        cmd_line_key = self._cmd_line_as_key(pixel_tests_needed, [])
+        cmd_line_key = self._cmd_line_as_key(pixel_tests_needed, driver_input.args)
         if not cmd_line_key in self._running_drivers:
             self._running_drivers[cmd_line_key] = self._make_driver(pixel_tests_needed)

Modified: trunk/Tools/Scripts/webkitpy/layout_tests/port/test.py (109265 => 109266)


--- trunk/Tools/Scripts/webkitpy/layout_tests/port/test.py	2012-02-29 22:43:05 UTC (rev 109265)
+++ trunk/Tools/Scripts/webkitpy/layout_tests/port/test.py	2012-02-29 22:43:27 UTC (rev 109266)
@@ -32,6 +32,7 @@
 import time
 
 from webkitpy.layout_tests.port import Port, Driver, DriverOutput
+from webkitpy.layout_tests.port.base import VirtualTestSuite
 from webkitpy.layout_tests.models.test_configuration import TestConfiguration
 from webkitpy.common.system.filesystem_mock import MockFileSystem
 
@@ -159,6 +160,7 @@
     tests.add('http/tests/passes/text.html')
     tests.add('http/tests/passes/image.html')
     tests.add('http/tests/ssl/text.html')
+    tests.add('passes/args.html')
     tests.add('passes/error.html', error='stuff going to stderr')
     tests.add('passes/image.html')
     tests.add('passes/audio.html',
@@ -304,6 +306,7 @@
         add_file(test, '-expected.txt', test.expected_text)
         add_file(test, '-expected.png', test.expected_image)
 
+    filesystem.write_text_file(filesystem.join(LAYOUT_TEST_DIR, 'virtual', 'args-expected.txt'), 'args-txt --virtual-arg')
     # Clear the list of written files so that we can watch what happens during testing.
     filesystem.clear_written_files()
 
@@ -489,6 +492,10 @@
     def all_baseline_variants(self):
         return self.ALL_BASELINE_VARIANTS
 
+    def virtual_test_suites(self):
+        return [
+            VirtualTestSuite('virtual', 'passes', ['--virtual-arg']),
+        ]
 
 class TestDriver(Driver):
     """Test/Dummy implementation of the DumpRenderTree interface."""
@@ -500,6 +507,7 @@
     def run_test(self, test_input):
         start_time = time.time()
         test_name = test_input.test_name
+        test_args = test_input.args or []
         test = self._port._tests[test_name]
         if test.keyboard:
             raise KeyboardInterrupt
@@ -509,6 +517,10 @@
             time.sleep((float(test_input.timeout) * 4) / 1000.0)
 
         audio = None
+        actual_text = test.actual_text
+        if actual_text and test_args and test_name == 'passes/args.html':
+            actual_text = actual_text + ' ' + ' '.join(test_args)
+
         if test.actual_audio:
             audio = base64.b64decode(test.actual_audio)
         crashed_process_name = None
@@ -516,7 +528,7 @@
             crashed_process_name = self._port.driver_name()
         elif test.web_process_crash:
             crashed_process_name = 'WebProcess'
-        return DriverOutput(test.actual_text, test.actual_image,
+        return DriverOutput(actual_text, test.actual_image,
             test.actual_checksum, audio, crash=test.crash or test.web_process_crash,
             crashed_process_name=crashed_process_name,
             test_time=time.time() - start_time, timeout=test.timeout, error=test.error)

Modified: trunk/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py (109265 => 109266)


--- trunk/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py	2012-02-29 22:43:05 UTC (rev 109265)
+++ trunk/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py	2012-02-29 22:43:27 UTC (rev 109266)
@@ -273,6 +273,10 @@
         self.assertTrue(passing_run(['--accelerated-2d-canvas']))
         self.assertTrue(passing_run(['--no-accelerated-2d-canvas']))
 
+    def test_all(self):
+        res, out, err, user = logging_run([], tests_included=True)
+        self.assertEquals(res, unexpected_tests_count)
+
     def test_basic(self):
         self.assertTrue(passing_run())
 
@@ -748,6 +752,10 @@
         test_port = get_port_for_run(base_args)
         self.assertEqual(None, test_port.tolerance_used_for_diff_image)
 
+    def test_virtual(self):
+        self.assertTrue(passing_run(['passes/text.html', 'passes/args.html',
+                                     'virtual/text.html', 'virtual/args.html']))
+
     def test_worker_model__inline(self):
         self.assertTrue(passing_run(['--worker-model', 'inline']))
 

_______________________________________________
webkit-changes mailing list
[email protected]
http://lists.webkit.org/mailman/listinfo.cgi/webkit-changes

Reply via email to