Re: [webkit-dev] Process for making changes that affect layout test results
On Wed, Apr 11, 2012 at 4:58 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Apr 11, 2012 at 4:56 PM, Tony Payne tpa...@chromium.org wrote: Some of these test result changes turn out to be the effect of incorrect test cases. For example, compositing/color-matching/image-color-matching.html compares an image without a color profile with an image tagged with what it calls a default RGB profile. However, this default RGB profile is not sRGB, so the test is wrongly[1] assuming these images should be displayed the same. I'd like to replace all of the resources containing random RGB profiles with sRGB profiles instead. 1) Is this a good idea? Is there a reason why these image resources have non-standard profiles in them? I don't know the answer to this, but hopefully someone else will. The patch is here: https://bugs.webkit.org/attachment.cgi?id=136946action=review If anyone knows why these images had the color profiles they had or a reason not to assign them to sRGB, please let me know. 2) How do I coordinate this, as changing the image resources seems like it will affect all the ports? To the extent you can notify other port maintainers, I'm sure they would appreciate it. Otherwise, treat it the same way you would treat any other change ... It will just be more painful :). -- Dirk Tony [1] http://www.w3.org/Graphics/Color/sRGB.html On Wed, Apr 11, 2012 at 3:36 PM, Dirk Pranke dpra...@chromium.org wrote: use run-webkit-tests (or attempt to pull the results from the trybots by hand). You cannot pull new baselines from the try bots using rebaseline.py. -- Dirk On Wed, Apr 11, 2012 at 3:26 PM, Tony Payne tpa...@chromium.org wrote: Is the best way to do that to use run_webkit_tests to generate a new baseline or does Tools/Scripts/webkitpy/tool/commands/rebaseline.py support pulling new baselines from the trybots? Thanks, Tony On Wed, Apr 11, 2012 at 2:13 PM, Ojan Vafai o...@chromium.org wrote: Typically, if you're working on Chromium Linux or Win, you'd include the new expected results for that platform in your initial commit/code-review as well. On Wed, Apr 11, 2012 at 2:09 PM, Tony Payne tpa...@chromium.org wrote: All code I'm changing is inside of #if PLATFORM(CHROMIUM) blocks. Thanks for the quick answer. Tony On Wed, Apr 11, 2012 at 2:03 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Apr 11, 2012 at 1:57 PM, Tony Payne tpa...@chromium.org wrote: Given the recent discussion on test_expectations.txt, perhaps the answer to my question is still up in the air. I'm working on a change that I expect to require changing the expectations for about 75 tests on chromium win and linux. https://trac.webkit.org/wiki/Rebaseline seems to only cover the gardening work to rebaseline after the commit. I cannot find any wiki pages that describe what the original author is expected to do when making visual changes. Should I attempt to rebaseline manually? Should I mark the tests as failing? Should I just check in and let the bots go red? Just land the patch and rebaseline the tests. Please also coordinate with Chromium port's WebKit gardener when landing this patch. Also, does this patch only affect Chromium Windows and Linux, and not GTK, Qt, Windows, etc...? If the answer is no, and will affect other non-Chromium ports, then you're also responsible for rebaselining or coordinating with other ports to make sure you don't break tests on their ports as well. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Process for making changes that affect layout test results
On Wed, Apr 11, 2012 at 1:57 PM, Tony Payne tpa...@chromium.org wrote: Given the recent discussion on test_expectations.txt, perhaps the answer to my question is still up in the air. I'm working on a change that I expect to require changing the expectations for about 75 tests on chromium win and linux. https://trac.webkit.org/wiki/Rebaseline seems to only cover the gardening work to rebaseline after the commit. I cannot find any wiki pages that describe what the original author is expected to do when making visual changes. Should I attempt to rebaseline manually? Should I mark the tests as failing? Should I just check in and let the bots go red? Just land the patch and rebaseline the tests. Please also coordinate with Chromium port's WebKit gardener when landing this patch. Also, does this patch only affect Chromium Windows and Linux, and not GTK, Qt, Windows, etc...? If the answer is no, and will affect other non-Chromium ports, then you're also responsible for rebaselining or coordinating with other ports to make sure you don't break tests on their ports as well. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Process for making changes that affect layout test results
All code I'm changing is inside of #if PLATFORM(CHROMIUM) blocks. Thanks for the quick answer. Tony On Wed, Apr 11, 2012 at 2:03 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Apr 11, 2012 at 1:57 PM, Tony Payne tpa...@chromium.org wrote: Given the recent discussion on test_expectations.txt, perhaps the answer to my question is still up in the air. I'm working on a change that I expect to require changing the expectations for about 75 tests on chromium win and linux. https://trac.webkit.org/wiki/Rebaseline seems to only cover the gardening work to rebaseline after the commit. I cannot find any wiki pages that describe what the original author is expected to do when making visual changes. Should I attempt to rebaseline manually? Should I mark the tests as failing? Should I just check in and let the bots go red? Just land the patch and rebaseline the tests. Please also coordinate with Chromium port's WebKit gardener when landing this patch. Also, does this patch only affect Chromium Windows and Linux, and not GTK, Qt, Windows, etc...? If the answer is no, and will affect other non-Chromium ports, then you're also responsible for rebaselining or coordinating with other ports to make sure you don't break tests on their ports as well. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Process for making changes that affect layout test results
Typically, if you're working on Chromium Linux or Win, you'd include the new expected results for that platform in your initial commit/code-review as well. On Wed, Apr 11, 2012 at 2:09 PM, Tony Payne tpa...@chromium.org wrote: All code I'm changing is inside of #if PLATFORM(CHROMIUM) blocks. Thanks for the quick answer. Tony On Wed, Apr 11, 2012 at 2:03 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Apr 11, 2012 at 1:57 PM, Tony Payne tpa...@chromium.org wrote: Given the recent discussion on test_expectations.txt, perhaps the answer to my question is still up in the air. I'm working on a change that I expect to require changing the expectations for about 75 tests on chromium win and linux. https://trac.webkit.org/wiki/Rebaseline seems to only cover the gardening work to rebaseline after the commit. I cannot find any wiki pages that describe what the original author is expected to do when making visual changes. Should I attempt to rebaseline manually? Should I mark the tests as failing? Should I just check in and let the bots go red? Just land the patch and rebaseline the tests. Please also coordinate with Chromium port's WebKit gardener when landing this patch. Also, does this patch only affect Chromium Windows and Linux, and not GTK, Qt, Windows, etc...? If the answer is no, and will affect other non-Chromium ports, then you're also responsible for rebaselining or coordinating with other ports to make sure you don't break tests on their ports as well. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Process for making changes that affect layout test results
Is the best way to do that to use run_webkit_tests to generate a new baseline or does Tools/Scripts/webkitpy/tool/commands/rebaseline.py support pulling new baselines from the trybots? Thanks, Tony On Wed, Apr 11, 2012 at 2:13 PM, Ojan Vafai o...@chromium.org wrote: Typically, if you're working on Chromium Linux or Win, you'd include the new expected results for that platform in your initial commit/code-review as well. On Wed, Apr 11, 2012 at 2:09 PM, Tony Payne tpa...@chromium.org wrote: All code I'm changing is inside of #if PLATFORM(CHROMIUM) blocks. Thanks for the quick answer. Tony On Wed, Apr 11, 2012 at 2:03 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Apr 11, 2012 at 1:57 PM, Tony Payne tpa...@chromium.org wrote: Given the recent discussion on test_expectations.txt, perhaps the answer to my question is still up in the air. I'm working on a change that I expect to require changing the expectations for about 75 tests on chromium win and linux. https://trac.webkit.org/wiki/Rebaseline seems to only cover the gardening work to rebaseline after the commit. I cannot find any wiki pages that describe what the original author is expected to do when making visual changes. Should I attempt to rebaseline manually? Should I mark the tests as failing? Should I just check in and let the bots go red? Just land the patch and rebaseline the tests. Please also coordinate with Chromium port's WebKit gardener when landing this patch. Also, does this patch only affect Chromium Windows and Linux, and not GTK, Qt, Windows, etc...? If the answer is no, and will affect other non-Chromium ports, then you're also responsible for rebaselining or coordinating with other ports to make sure you don't break tests on their ports as well. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Process for making changes that affect layout test results
use run-webkit-tests (or attempt to pull the results from the trybots by hand). You cannot pull new baselines from the try bots using rebaseline.py. -- Dirk On Wed, Apr 11, 2012 at 3:26 PM, Tony Payne tpa...@chromium.org wrote: Is the best way to do that to use run_webkit_tests to generate a new baseline or does Tools/Scripts/webkitpy/tool/commands/rebaseline.py support pulling new baselines from the trybots? Thanks, Tony On Wed, Apr 11, 2012 at 2:13 PM, Ojan Vafai o...@chromium.org wrote: Typically, if you're working on Chromium Linux or Win, you'd include the new expected results for that platform in your initial commit/code-review as well. On Wed, Apr 11, 2012 at 2:09 PM, Tony Payne tpa...@chromium.org wrote: All code I'm changing is inside of #if PLATFORM(CHROMIUM) blocks. Thanks for the quick answer. Tony On Wed, Apr 11, 2012 at 2:03 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Apr 11, 2012 at 1:57 PM, Tony Payne tpa...@chromium.org wrote: Given the recent discussion on test_expectations.txt, perhaps the answer to my question is still up in the air. I'm working on a change that I expect to require changing the expectations for about 75 tests on chromium win and linux. https://trac.webkit.org/wiki/Rebaseline seems to only cover the gardening work to rebaseline after the commit. I cannot find any wiki pages that describe what the original author is expected to do when making visual changes. Should I attempt to rebaseline manually? Should I mark the tests as failing? Should I just check in and let the bots go red? Just land the patch and rebaseline the tests. Please also coordinate with Chromium port's WebKit gardener when landing this patch. Also, does this patch only affect Chromium Windows and Linux, and not GTK, Qt, Windows, etc...? If the answer is no, and will affect other non-Chromium ports, then you're also responsible for rebaselining or coordinating with other ports to make sure you don't break tests on their ports as well. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Process for making changes that affect layout test results
On Wed, Apr 11, 2012 at 4:56 PM, Tony Payne tpa...@chromium.org wrote: Some of these test result changes turn out to be the effect of incorrect test cases. For example, compositing/color-matching/image-color-matching.html compares an image without a color profile with an image tagged with what it calls a default RGB profile. However, this default RGB profile is not sRGB, so the test is wrongly[1] assuming these images should be displayed the same. I'd like to replace all of the resources containing random RGB profiles with sRGB profiles instead. 1) Is this a good idea? Is there a reason why these image resources have non-standard profiles in them? I don't know the answer to this, but hopefully someone else will. 2) How do I coordinate this, as changing the image resources seems like it will affect all the ports? To the extent you can notify other port maintainers, I'm sure they would appreciate it. Otherwise, treat it the same way you would treat any other change ... It will just be more painful :). -- Dirk Tony [1] http://www.w3.org/Graphics/Color/sRGB.html On Wed, Apr 11, 2012 at 3:36 PM, Dirk Pranke dpra...@chromium.org wrote: use run-webkit-tests (or attempt to pull the results from the trybots by hand). You cannot pull new baselines from the try bots using rebaseline.py. -- Dirk On Wed, Apr 11, 2012 at 3:26 PM, Tony Payne tpa...@chromium.org wrote: Is the best way to do that to use run_webkit_tests to generate a new baseline or does Tools/Scripts/webkitpy/tool/commands/rebaseline.py support pulling new baselines from the trybots? Thanks, Tony On Wed, Apr 11, 2012 at 2:13 PM, Ojan Vafai o...@chromium.org wrote: Typically, if you're working on Chromium Linux or Win, you'd include the new expected results for that platform in your initial commit/code-review as well. On Wed, Apr 11, 2012 at 2:09 PM, Tony Payne tpa...@chromium.org wrote: All code I'm changing is inside of #if PLATFORM(CHROMIUM) blocks. Thanks for the quick answer. Tony On Wed, Apr 11, 2012 at 2:03 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Apr 11, 2012 at 1:57 PM, Tony Payne tpa...@chromium.org wrote: Given the recent discussion on test_expectations.txt, perhaps the answer to my question is still up in the air. I'm working on a change that I expect to require changing the expectations for about 75 tests on chromium win and linux. https://trac.webkit.org/wiki/Rebaseline seems to only cover the gardening work to rebaseline after the commit. I cannot find any wiki pages that describe what the original author is expected to do when making visual changes. Should I attempt to rebaseline manually? Should I mark the tests as failing? Should I just check in and let the bots go red? Just land the patch and rebaseline the tests. Please also coordinate with Chromium port's WebKit gardener when landing this patch. Also, does this patch only affect Chromium Windows and Linux, and not GTK, Qt, Windows, etc...? If the answer is no, and will affect other non-Chromium ports, then you're also responsible for rebaselining or coordinating with other ports to make sure you don't break tests on their ports as well. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
