Title: [115853] trunk/Tools
- Revision
- 115853
- Author
- [email protected]
- Date
- 2012-05-02 11:01:27 -0700 (Wed, 02 May 2012)
Log Message
Reftest mismatches are providing undefined diff percent in results.html
https://bugs.webkit.org/show_bug.cgi?id=85283
Reviewed by Ojan Vafai.
When writing test results for a reftest mismatch, also take into account the
difference percent that is produced when generating image diff. This value is
then used when creating test results in the form of dictionaries, stored under
the 'image_diff_percent' key. It ultimately ends up in JSON results data,
providing additional information about the reftest difference.
* Scripts/webkitpy/layout_tests/controllers/manager.py:
(interpret_test_failures): Store the diff percent for reftest failure under the
'image_diff_percent' key.
* Scripts/webkitpy/layout_tests/controllers/test_result_writer.py:
(write_test_result): Also set the new diff_percent attribute of the reftest
mismatch failure to the value of the difference percent ImageDiff outputs.
* Scripts/webkitpy/layout_tests/models/test_failures.py:
(FailureReftestMismatch.__init__): Add the new diff_percent attribute with the
default value of None.
* Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py:
(EndToEndTest.test_reftest_with_two_notrefs): Update the test case expected results.
Modified Paths
Diff
Modified: trunk/Tools/ChangeLog (115852 => 115853)
--- trunk/Tools/ChangeLog 2012-05-02 17:57:34 UTC (rev 115852)
+++ trunk/Tools/ChangeLog 2012-05-02 18:01:27 UTC (rev 115853)
@@ -1,3 +1,28 @@
+2012-05-02 Zan Dobersek <[email protected]>
+
+ Reftest mismatches are providing undefined diff percent in results.html
+ https://bugs.webkit.org/show_bug.cgi?id=85283
+
+ Reviewed by Ojan Vafai.
+
+ When writing test results for a reftest mismatch, also take into account the
+ difference percent that is produced when generating image diff. This value is
+ then used when creating test results in the form of dictionaries, stored under
+ the 'image_diff_percent' key. It ultimately ends up in JSON results data,
+ providing additional information about the reftest difference.
+
+ * Scripts/webkitpy/layout_tests/controllers/manager.py:
+ (interpret_test_failures): Store the diff percent for reftest failure under the
+ 'image_diff_percent' key.
+ * Scripts/webkitpy/layout_tests/controllers/test_result_writer.py:
+ (write_test_result): Also set the new diff_percent attribute of the reftest
+ mismatch failure to the value of the difference percent ImageDiff outputs.
+ * Scripts/webkitpy/layout_tests/models/test_failures.py:
+ (FailureReftestMismatch.__init__): Add the new diff_percent attribute with the
+ default value of None.
+ * Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py:
+ (EndToEndTest.test_reftest_with_two_notrefs): Update the test case expected results.
+
2012-05-02 Terry Anderson <[email protected]>
Addition of a contributor to committers.py
Modified: trunk/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py (115852 => 115853)
--- trunk/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py 2012-05-02 17:57:34 UTC (rev 115852)
+++ trunk/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py 2012-05-02 18:01:27 UTC (rev 115853)
@@ -88,6 +88,7 @@
elif isinstance(failure, test_failures.FailureReftestMismatch):
test_dict['is_reftest'] = True
test_dict['ref_file'] = port.relative_test_filename(failure.reference_filename)
+ test_dict['image_diff_percent'] = failure.diff_percent
elif isinstance(failure, test_failures.FailureReftestMismatchDidNotOccur):
test_dict['is_mismatch_reftest'] = True
test_dict['ref_file'] = port.relative_test_filename(failure.reference_filename)
Modified: trunk/Tools/Scripts/webkitpy/layout_tests/controllers/test_result_writer.py (115852 => 115853)
--- trunk/Tools/Scripts/webkitpy/layout_tests/controllers/test_result_writer.py 2012-05-02 17:57:34 UTC (rev 115852)
+++ trunk/Tools/Scripts/webkitpy/layout_tests/controllers/test_result_writer.py 2012-05-02 18:01:27 UTC (rev 115853)
@@ -68,9 +68,10 @@
# FIXME: This work should be done earlier in the pipeline (e.g., when we compare images for non-ref tests).
# FIXME: We should always have 2 images here.
if driver_output.image and expected_driver_output.image:
- image_diff = port.diff_image(driver_output.image, expected_driver_output.image, tolerance=0)[0]
- if image_diff:
- writer.write_image_diff_files(image_diff)
+ diff_result = port.diff_image(driver_output.image, expected_driver_output.image, tolerance=0)
+ if diff_result[0]:
+ writer.write_image_diff_files(diff_result[0])
+ failure.diff_percent = diff_result[1]
else:
_log.warn('Can not get image diff. ImageDiff program might not work correctly.')
writer.copy_file(failure.reference_filename)
Modified: trunk/Tools/Scripts/webkitpy/layout_tests/models/test_failures.py (115852 => 115853)
--- trunk/Tools/Scripts/webkitpy/layout_tests/models/test_failures.py 2012-05-02 17:57:34 UTC (rev 115852)
+++ trunk/Tools/Scripts/webkitpy/layout_tests/models/test_failures.py 2012-05-02 18:01:27 UTC (rev 115853)
@@ -186,6 +186,7 @@
def __init__(self, reference_filename=None):
self.reference_filename = reference_filename
+ self.diff_percent = None
def message(self):
return "Mismatch with reference"
Modified: trunk/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py (115852 => 115853)
--- trunk/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py 2012-05-02 17:57:34 UTC (rev 115852)
+++ trunk/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py 2012-05-02 18:01:27 UTC (rev 115853)
@@ -876,7 +876,7 @@
self.assertTrue("multiple-mismatch-success.html" not in json["tests"]["reftests"]["foo"])
self.assertTrue("multiple-both-success.html" not in json["tests"]["reftests"]["foo"])
self.assertEqual(json["tests"]["reftests"]["foo"]["multiple-match-failure.html"],
- {"expected": "PASS", "ref_file": "reftests/foo/second-mismatching-ref.html", "actual": "IMAGE", 'is_reftest': True})
+ {"expected": "PASS", "ref_file": "reftests/foo/second-mismatching-ref.html", "actual": "IMAGE", "image_diff_percent": 1, 'is_reftest': True})
self.assertEqual(json["tests"]["reftests"]["foo"]["multiple-mismatch-failure.html"],
{"expected": "PASS", "ref_file": "reftests/foo/matching-ref.html", "actual": "IMAGE", "is_mismatch_reftest": True})
self.assertEqual(json["tests"]["reftests"]["foo"]["multiple-both-failure.html"],
_______________________________________________
webkit-changes mailing list
[email protected]
http://lists.webkit.org/mailman/listinfo.cgi/webkit-changes
