Re: [webkit-dev] Reflecting pixel delta distance in ImageDiff
Food for thought... Skia's skdiff tool generates two diff images for each image pair: - 1. every pixel which is different at all between images 1 2 shows up as white - 2. every pixel shows the color difference between images 1 2 (much more subtle) For an example of what this looks like, see http://skia.googlecode.com/svn/trunk/tools/tests/skdiff/test1/output-expected/index.html On Fri, Jun 15, 2012 at 7:24 PM, Tony Payne tpa...@chromium.org wrote: I would like to change chromium's ImageDiff to reflect the magnitude of pixel changes. Currently, if the pixel has any difference, the entire pixel is marked as 100% red. I'd like to change it so that miniscule difference are 20% red and large differences are 100% red. Looking at the code for CG, gtk and Win versions of ImageDiff, I think they already do something similar. Is this correct? Would anyone be opposed to such a change? See the attached image for an example of how this could assist in debugging regressions. It would certainly be enormously useful in my color management efforts. Thanks, Tony The CL in question is https://bugs.webkit.org/show_bug.cgi?id=89253 ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 6:53 PM, Dirk Pranke dpra...@chromium.org wrote: * We'll probably rename IMAGE+TEXT to IMAGE_AND_TEXT. Can someone please remind me why IMAGE+TEXT even exists? Wouldn't it be simpler to just mark a test as follows? - IMAGE : allow image failure; go red if there is a text failure - TEXT: allow text failure; go red if there is an image failure - IMAGE TEXT: allow text and/or image failure ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 4:44 PM, Peter Kasting pkast...@chromium.orgwrote: On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger epo...@chromium.org wrote: Can someone please remind me why IMAGE+TEXT even exists? Wouldn't it be simpler to just mark a test as follows? - IMAGE : allow image failure; go red if there is a text failure - TEXT: allow text failure; go red if there is an image failure - IMAGE TEXT: allow text and/or image failure The distinction is that IMAGE TEXT will allow image, text, or both to fail, thus making transitions among the three generate no events. IMAGE+TEXT says specifically that we expect both to fail and that if one starts passing, someone should do something. (For example, maybe someone checks in a partial rebaseline where they miss the image expectations.) Thanks for the explanation. That makes sense, although it seems to me that the problem of no-events-generated-by-changes-in-actual-images-while-IMAGE-failure-is-expected is about 100x worse for us. But that's not a reason to hide these particular transitions! :-) PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] please add EditBugs permission
Can someone please add EditBugs permission for my bugs.webkit.org account (
epo...@chromium.org)?
Another WebKit committer and I both tried it, using the steps Eric pasted
in April, but it failed... I think maybe the add-users-to-groups tool is
broken?
$ Tools/Scripts/webkit-patch add-users-to-groups epo...@chromium.org
Add users matching epo...@chromium.org which groups?
1. canconfirm
2. editbugs
Enter one or more numbers (comma-separated), or all: all
bugs.webkit.org login: epo...@google.com
bugs.webkit.org password for epo...@google.com:
Logging in as epo...@google.com...
Traceback (most recent call last):
File Tools/Scripts/webkit-patch, line 69, in module
main()
File Tools/Scripts/webkit-patch, line 64, in main
WebKitPatch(os.path.abspath(__file__)).main()
File
/usr/local/google/home/epoger/src/webkit/green/WebKit/Tools/Scripts/webkitpy/tool/multicommandtool.py,
line 311, in main
result = command.check_arguments_and_execute(options, args, self)
File
/usr/local/google/home/epoger/src/webkit/green/WebKit/Tools/Scripts/webkitpy/tool/multicommandtool.py,
line 120, in check_arguments_and_execute
return self.execute(options, args, tool) or 0
File
/usr/local/google/home/epoger/src/webkit/green/WebKit/Tools/Scripts/webkitpy/tool/commands/adduserstogroups.py,
line 50, in execute
login_userid_pairs =
tool.bugs.queries.fetch_login_userid_pairs_matching_substring(search_string)
File
/usr/local/google/home/epoger/src/webkit/green/WebKit/Tools/Scripts/webkitpy/common/net/bugzilla/bugzilla.py,
line 257, in fetch_login_userid_pairs_matching_substring
return
EditUsersParser().login_userid_pairs_from_edit_user_results(results_page)
File
/usr/local/google/home/epoger/src/webkit/green/WebKit/Tools/Scripts/webkitpy/common/net/bugzilla/bugzilla.py,
line 72, in login_userid_pairs_from_edit_user_results
login_userid_pairs = [self._login_and_uid_from_row(row) for row in
results_table('tr')]
TypeError: 'NoneType' object is not callable
On Thu, Apr 12, 2012 at 6:47 PM, Eric Seidel e...@webkit.org wrote:
Done.
% webkit-patch add-users-to-groups shezbaig...@gmail.com
Add users matching shezbaig...@gmail.com which groups?
1. canconfirm
2. editbugs
Enter one or more numbers (comma-separated), or all:
Reading Keychain for bugs.webkit.org account and password. Click
Allow to continue...
Logging in as e...@webkit.org...
Found 1 users matching shezbaig...@gmail.com:
shezbaig...@gmail.com (17597)
Are you sure you want add 1 users to groups ['canconfirm',
'editbugs']? (This action cannot be undone using webkit-patch.)
[Y/n]:
Adding shezbaig...@gmail.com to ['canconfirm', 'editbugs']
On Thu, Apr 12, 2012 at 3:17 PM, Shezan Baig shezbaig...@gmail.com
wrote:
Hello,
When I submit a patch to bugzilla, I get an error that EditBugs is not
set
on my account. Can someone add this permission to my bugzilla account
please? I sent a message to webkit-committ...@lists.webkit.org as
instructed, but that doesn't get through.
My login is: shezbaig...@gmail.com
My patch submitted to: https://bugs.webkit.org/show_bug.cgi?id=80382
Thanks,
Shezan
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] optimising png files in LayoutTests - an experiment
In deciding whether to just let the PNGs get shrunk over time or to do a
pass over all of them at once, I think we should decide what we are trying
to optimize for. (Download time for initial checkout of the webkit tree?
Time for existing developers to update their trees? Storage space in the
repository?)
I would think that, except for initial checkout of the webkit repository,
the others would be optimized by lazily converting the files (waiting until
we need to check in new baselines, instead of shrinking them all at once).
On Wed, Jan 25, 2012 at 1:12 PM, Dirk Pranke dpra...@chromium.org wrote:
I like the idea. I'm not sure I agree w/ Adam that I'd roll the code
into DRT, insofar as I don't know how big it is and I would definitely
want the code shared across all of the DRT implementations. I'd
probably also do a pass over all of the existing PNGs at some point
rather than wait for them to be updated as necessary.
-- Dirk
On Tue, Jan 24, 2012 at 10:43 PM, Mike Lawther mikelawt...@chromium.org
wrote:
Hi guys,
Just thought I'd share the results of an experiment I did in optimising
the
png files in LayoutTests.
I used a tool on Mac called ImageOptim (http://imageoptim.pornel.net/)
which
tries a set of different png lossless optimising tools (like pngcrush - I
also downloaded and included PNGOUT). Note that this strips out some of
the
stuff we need, like the hash, so if doing this for real we'd have to
watch
out for that.
My results were:
Before:
$ ls -laR LayoutTests/ | grep png$ | awk '{total = total + $5}END{print
total}'
1220535840
Test:
$ find LayoutTests/ | grep png$ | xargs open -a ImageOptim.app
After:
$ ls -laR LayoutTests/ | grep png$ | awk '{total = total + $5}END{print
total, 1220535840 - total}'
937198328 283337512
So this has saved ~280MB (~23% of the original size).
Based on this, it seems worthwhile to include a png optimiser somewhere
in
the patch upload/submit toolchain, and also (separately) to optimise the
existing pngs.
Thoughts?
mike
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] chromium-cg-mac results
I agree that it is reasonable (and a good idea) to remove the chromium-cg-mac expectations from WebKit now, and I am willing to take the lead on doing so (although I will need help from WebKit committers). Please let me know if anyone agrees/disagrees with the following steps to do so: 1. remove the following buildbots that rely on chromium-cg-mac expectations (otherwise, they will start failing once the CG expectations disappear): - http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac%20Builder%20%28CG%29%28deps%29 - http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.6%20%28CG%29%28deps%29 - http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac%20Builder%20%28CG%29 - http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.5%20%28CG%29 - http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.6%20%28CG%29 - http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac%20Builder%20%28CG%29%28dbg%29 - http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.5%20%28CG%29%28dbg%29%281%29 - http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.5%20%28CG%29%28dbg%29%282%29 - http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.6%20%28CG%29%28dbg%29 2. remove the LayoutTests/platform/*-cg-* directories from the WebKit repo 3. remove any CG-specific entries from LayoutTests/platform/chromium/test-expectations.txt 4. remove any CG-specific test code from the Tools/Scripts/webkitpy tree On Wed, Jan 4, 2012 at 12:01 AM, David Levin le...@chromium.org wrote: On Tue, Jan 3, 2012 at 8:33 PM, Dirk Pranke dpra...@chromium.org wrote: On Tue, Jan 3, 2012 at 3:28 PM, Adam Barth aba...@webkit.org wrote: On Tue, Jan 3, 2012 at 3:22 PM, Nico Weber tha...@chromium.org wrote: On Tue, Jan 3, 2012 at 3:00 PM, Adam Barth aba...@webkit.org wrote: It looks like Chromium Mac has successfully moved to Skia. I'd wait with this assessment until a version of Chrome with Skia has shipped to stable. Things are looking really good so that should be smooth sailing, but it's a bit early to say we're successfully moved :-) Fair enough. However, I believe the Skia transition plan called for removing the chromium-cg-mac expectation much earlier than a Skia build shipping to stable. Originally, we were only supposed to have to maintain both sets of expectations for about a month. The transition has taken longer than expected, but it seems like we have sufficient confidence in Skia now that we can remove the chromium-cg-mac expectations. Has the skia transition hit beta yet? It seems like as soon as we get it onto a version that is pointing to a branched version of webkit, we should be completely safe to remove the directories on trunk (frankly, I'd agree with Adam that it's probably safe to remove it now, since we can always add them back in if we have to, but I can compromise as well). Remove it. It is a cost on everyone who enlists in WebKit. Those of us who aren't creating new enlistments are not affected much but that doesn't mean it isn't costly. dave ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] chromium-cg-mac results
Filed https://bugs.webkit.org/show_bug.cgi?id=75548 ('[rollup] remove
chromium-cg-mac baselines')
On Wed, Jan 4, 2012 at 10:37 AM, Adam Barth aba...@webkit.org wrote:
On Wed, Jan 4, 2012 at 6:10 AM, Elliot Poger epo...@chromium.org wrote:
I agree that it is reasonable (and a good idea) to remove the
chromium-cg-mac expectations from WebKit now, and I am willing to take
the
lead on doing so (although I will need help from WebKit committers).
Sounds like a good plan. I'm happy to help. Let's coordinate off-list.
Thanks,
Adam
Please let me know if anyone agrees/disagrees with the following steps
to do
so:
1. remove the following buildbots that rely on chromium-cg-mac
expectations
(otherwise, they will start failing once the CG expectations disappear):
http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac%20Builder%20%28CG%29%28deps%29
http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.6%20%28CG%29%28deps%29
http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac%20Builder%20%28CG%29
http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.5%20%28CG%29
http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.6%20%28CG%29
http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac%20Builder%20%28CG%29%28dbg%29
http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.5%20%28CG%29%28dbg%29%281%29
http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.5%20%28CG%29%28dbg%29%282%29
http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.6%20%28CG%29%28dbg%29
2. remove the LayoutTests/platform/*-cg-* directories from the WebKit
repo
3. remove any CG-specific entries from
LayoutTests/platform/chromium/test-expectations.txt
4. remove any CG-specific test code from the Tools/Scripts/webkitpy tree
On Wed, Jan 4, 2012 at 12:01 AM, David Levin le...@chromium.org wrote:
On Tue, Jan 3, 2012 at 8:33 PM, Dirk Pranke dpra...@chromium.org
wrote:
On Tue, Jan 3, 2012 at 3:28 PM, Adam Barth aba...@webkit.org wrote:
On Tue, Jan 3, 2012 at 3:22 PM, Nico Weber tha...@chromium.org
wrote:
On Tue, Jan 3, 2012 at 3:00 PM, Adam Barth aba...@webkit.org
wrote:
It looks like Chromium Mac has successfully moved to Skia.
I'd wait with this assessment until a version of Chrome with Skia
has
shipped to stable. Things are looking really good so that should be
smooth sailing, but it's a bit early to say we're successfully
moved
:-)
Fair enough. However, I believe the Skia transition plan called for
removing the chromium-cg-mac expectation much earlier than a Skia
build shipping to stable. Originally, we were only supposed to have
to maintain both sets of expectations for about a month. The
transition has taken longer than expected, but it seems like we have
sufficient confidence in Skia now that we can remove the
chromium-cg-mac expectations.
Has the skia transition hit beta yet? It seems like as soon as we get
it onto a version that is pointing to a branched version of webkit, we
should be completely safe to remove the directories on trunk (frankly,
I'd agree with Adam that it's probably safe to remove it now, since we
can always add them back in if we have to, but I can compromise as
well).
Remove it.
It is a cost on everyone who enlists in WebKit. Those of us who aren't
creating new enlistments are not affected much but that doesn't mean it
isn't costly.
dave
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] layout tests: how are some compared against image, and others only text?
What is it that causes some tests to require baseline images (and not text files) for comparison, while others require text and not image baselines? (I know that I can specifically SKIP comparison against IMAGE and/or TEXT using test_expectations.txt... but even without the use of test_expectations, I believe that some tests are compared against only text or only image.) As an example, I see that this test has only image baselines and no text baselines: $ ls LayoutTests/platform/*/fast/canvas/canvas-text-baseline*.png | wc -l 10 $ ls LayoutTests/platform/*/fast/canvas/canvas-text-baseline*.txt | wc -l ls: cannot access LayoutTests/platform/*/fast/canvas/canvas-text-baseline*.txt: No such file or directory 0 while this test has only text baselines and no image baselines: $ ls LayoutTests/platform/*/fast/canvas/canvas-lineWidth*.png | wc -l ls: cannot access LayoutTests/platform/*/fast/canvas/canvas-lineWidth*.png: No such file or directory 0 $ ls LayoutTests/platform/*/fast/canvas/canvas-lineWidth*.txt | wc -l 5 Is there something inherent in each test that indicates whether its results will be compared against image and/or text baselines? Or is it simply a matter of what baseline files are found to compare against? ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Tools/Scripts/webkit-patch rebaseline-expectations does not launch html comparison page?
I finally got back to this and tried to use garden-o-matic. I launched it
(./Tools/Scripts/webkit-patch
garden-o-matic) and it did nothing. I opened a separate thread about
this:
http://groups.google.com/a/chromium.org/group/chromium-dev/browse_thread/thread/a64eabe35e59ac17#('garden-o-matic
does nothing?')
Thus I am still unable to rebaseline tests across multiple platforms.
Is there a technique that actually works, with tools that exist today?
On Tue, Nov 8, 2011 at 4:08 PM, Adam Barth aba...@webkit.org wrote:
On Tue, Nov 8, 2011 at 1:00 PM, Elliot Poger epo...@chromium.org wrote:
How do the gardeners do the rebaselining currently? It seems like what
I'm
looking for is pretty much akin to gardening...
They use garden-o-matic, which displays the diffs prior to conducting
the rebaseline.
I have looked at
http://www.chromium.org/developers/how-tos/webkit-gardening
, but I have no idea if it is current.
It is current.
Adam
On Tue, Nov 8, 2011 at 3:31 PM, Dirk Pranke dpra...@chromium.org
wrote:
On Tue, Nov 8, 2011 at 12:24 PM, Adam Barth aba...@webkit.org wrote:
On Tue, Nov 8, 2011 at 12:20 PM, Tony Chang t...@chromium.org
wrote:
On Tue, Nov 8, 2011 at 12:00 PM, Elliot Poger epo...@chromium.org
wrote:
Perhaps I should approach this from a different angle:
What is the recommended procedure for:
- generating new baseline images for a few dozen failing tests, on
various
platforms
webkit-patch rebaseline-expectations
- visually inspecting them to make sure they're not bogus
Would 'webkit-patch pretty-diff' work for you? It should show the
files
being added/deleted, but it won't generate a pixel diff.
The tricky part is that this view requires you to understand all the
fallback behavior among different ports. My sense is that this would
be easier if we had a smarter view that understood that and presented
it to the user in an understandable way. Unfortunately, no one has
built that view yet.
rebaseline-chromium-webkit-tests had some careful logging to stdout
that explained what files were (or weren't) being updated and why
(i.e., I claim that I had solved this problem in that script. Although
it wasn't presented in the HTML, that wouldn't have been that hard to
add).
I think if we could get the equivalent into the new tool, and if we
could separate the update and optimize steps, that would probably be
good enough. I think combining update and optimize makes it *very*
hard to determine the correctness of what you've done.
In other words, my ideal workflow would be update -- review approve
-- optimize -- [optionally review optimze?] -- land.
-- Dirk
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Tools/Scripts/webkit-patch rebaseline-expectations does not launch html comparison page?
Perhaps I should approach this from a different angle: What is the recommended procedure for: - generating new baseline images for a few dozen failing tests, on various platforms - visually inspecting them to make sure they're not bogus - committing them, along with test_expectations changes, to the WebKit repo On Tue, Nov 8, 2011 at 2:48 PM, Eric Seidel e...@webkit.org wrote: I believe that tool just got nuked. Seems the Wiki page should as well: https://bugs.webkit.org/show_bug.cgi?id=71833 On Tue, Nov 8, 2011 at 9:09 AM, Adam Barth aba...@webkit.org wrote: On Tue, Nov 8, 2011 at 7:06 AM, Elliot Poger epo...@chromium.org wrote: I have just run Tools/Scripts/webkit-patch rebaseline-expectations. There are plenty of image rebaseline changes (svn status yields 792 lines!), but I'm not seeing the promised html page to compare old and new baselines. Correct. It does not launch a comparison page. The docs at http://trac.webkit.org/wiki/Rebaseline indicate that such an html page will be launched by default. That's documentation for a different tool. Is there some flag I have to pass to make this happen? There is no flag. My plan was to add the ability to take an arbitrary patch in the working copy and show a pretty-diff like view of how the patch effects layout test results. I haven't had time to implement that yet. If you're interested in adding that feature, let me know and I'll help you get started. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Tools/Scripts/webkit-patch rebaseline-expectations does not launch html comparison page?
How do the gardeners do the rebaselining currently? It seems like what I'm looking for is pretty much akin to gardening... I have looked at http://www.chromium.org/developers/how-tos/webkit-gardening, but I have no idea if it is current. On Tue, Nov 8, 2011 at 3:31 PM, Dirk Pranke dpra...@chromium.org wrote: On Tue, Nov 8, 2011 at 12:24 PM, Adam Barth aba...@webkit.org wrote: On Tue, Nov 8, 2011 at 12:20 PM, Tony Chang t...@chromium.org wrote: On Tue, Nov 8, 2011 at 12:00 PM, Elliot Poger epo...@chromium.org wrote: Perhaps I should approach this from a different angle: What is the recommended procedure for: - generating new baseline images for a few dozen failing tests, on various platforms webkit-patch rebaseline-expectations - visually inspecting them to make sure they're not bogus Would 'webkit-patch pretty-diff' work for you? It should show the files being added/deleted, but it won't generate a pixel diff. The tricky part is that this view requires you to understand all the fallback behavior among different ports. My sense is that this would be easier if we had a smarter view that understood that and presented it to the user in an understandable way. Unfortunately, no one has built that view yet. rebaseline-chromium-webkit-tests had some careful logging to stdout that explained what files were (or weren't) being updated and why (i.e., I claim that I had solved this problem in that script. Although it wasn't presented in the HTML, that wouldn't have been that hard to add). I think if we could get the equivalent into the new tool, and if we could separate the update and optimize steps, that would probably be good enough. I think combining update and optimize makes it *very* hard to determine the correctness of what you've done. In other words, my ideal workflow would be update -- review approve -- optimize -- [optionally review optimze?] -- land. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] insanity of updating 4000+ baseline images due to font rendering change?
We finally have correct baseline images checked in for Chromium-Skia on
SnowLeopard (hooray). Now we face the daunting task of rebaselining for
Chromium-Skia on Leopard.
The problem is that chromium-mac-leopard has ~4500 image diffs compared to
chromium-mac. The vast majority of these diffs are due to very minor
differences in text rendering between Leopard and SnowLeopard. (Details are
in http://code.google.com/p/chromium/issues/detail?id=100904 )
I poked around in LayoutTests/platform (details are at the bottom of this
email) and found that indeed there are 4000-5000 baseline images that
distinguish between Leopard and SnowLeopard. This is presumably why layout
tests generally pass on both platforms.
Before Cary and I spend 2-3 days committing all these new baseline images so
that we can re-enable test_expectations for SnowLeopard... is this the right
approach? It seems pretty inefficient to clog the WebKit repository with
all these baseline images with minor pixel value differences.
Here are the various approaches I can think of... what's the
Hive-Mind-Approved approach?
- Commit 4500 new baseline images for SnowLeopard
- pro: known to work, will catch any regressions that come later
- con: takes a long time to commit, chews up disk space and bandwidth
for all developers, future minor changes may require yet another
set of new
baselines
- Leave all SnowLeopard tests marked as PASS FAIL (or maybe mark them
SKIP) in test_expectations
- pro: known to work, quick and easy, doesn't clog repo space and
developer update bandwidth, future minor changes won't break any bots
- con: will not catch any regressions that come later on SnowLeopard
- Remove descriptive text from all these tests, so that text rendering is
only evaluated in tests specifically for that purpose
- pro: prevents this problem for future OS versions, should allow for
lots more baseline images to be shared across platforms
- con: a lot of work to replace all existing baseline images, must
coordinate across community of Chromium/WebKit developers, tests will be
more difficult to interpret without text
- Figure out how our test pages can be rendered with a completely
cross-platform pixel-equivalent font
- pro: similar to above but tests keep their descriptive text
- con: similar to above but more technically challenging
- Augment our pixel-diff tools to allow for comparison masks (only pay
attention to pixel diffs within this rectangle)
- pro: existing baseline images can stay in place, and perhaps be
shared with new OS versions and platforms
- con: requires modification of pixel-diff tools, need to add
comparison mask to each test definition
Details on how I determined that there are already 4000-5000 baseline images
that distinguish between Leopard and SnowLeopard for existing platforms
(Safari and chromium-cg):
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ mkdir tests
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ find
platform -name *.png tests/all-pngs
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ wc -l
tests/all-pngs
45529 tests/all-pngs
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ cat
tests/all-pngs | awk -F '/' '{print $2}' | sort | uniq
tests/list-all-platforms
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ grep mac
tests/list-all-platforms tests/list-mac-platforms
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ mkdir
tests/by-platform
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ for PLATFORM
in $(cat tests/list-mac-platforms); do grep ^platform/$PLATFORM/
tests/all-pngs | sed s/platform\\/$PLATFORM\\///
tests/by-platform/$PLATFORM ; done
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ wc -l
tests/by-platform/*
1050 tests/by-platform/chromium-cg-mac
1575 tests/by-platform/chromium-cg-mac-leopard
114 tests/by-platform/chromium-cg-mac-snowleopard
114 tests/by-platform/chromium-gpu-cg-mac
135 tests/by-platform/chromium-gpu-mac
4909 tests/by-platform/chromium-mac
79 tests/by-platform/chromium-mac-leopard
131 tests/by-platform/chromium-mac-snowleopard
7631 tests/by-platform/mac
4435 tests/by-platform/mac-leopard
247 tests/by-platform/mac-snowleopard
7 tests/by-platform/mac-wk2
20427 total
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ cat
tests/by-platform/chromium-cg-mac
tests/by-platform/chromium-cg-mac-snowleopard tests/by-platform/mac
tests/by-platform/mac-snowleopard | sort | uniq
tests/list-snowleopard-cg-images
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ cat
tests/by-platform/chromium-cg-mac-leopard
tests/by-platform/mac-leopard | sort | uniq
tests/list-leopard-cg-images
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ cat
tests/list-leopard-cg-images tests/list-snowleopard-cg-images | sort |
uniq
[webkit-dev] layout tests cannot set Generic RGB Color Profile on headless MacPro?
I was having problems with lots of image mismatches running WebKit layout tests on a headless MacPro (10.6.8). (The image diffs were very minor adjustments in scroll bar shading.) Eventually, out of frustration, I tried running the same test on that headless MacPro as well as my desktop MacPro (also 10.6.8). The test succeeded on my desktop MacPro but not the headless MacPro. One difference I have noticed is that the Display Profile (under System Preferences Displays Color) is set to sRGB IEC61966-2.1 (my desktop was set to Generic RGB Profile). I tried setting it to Generic RGB Profile on the headless machine, but when I closed and reopened System Preferences it had reverted to sRGB IEC61966-2.1. I tried setting the Display Profile on my desktop to sRGB IEC61966-2.1, and the color scheme was noticeably brighter... but when I ran layout tests afterwards, I saw that layout tests automatically set the profile back to Generic RGB Profile temporarily for the test. When I run layout tests on the headless MacPro, it seems that it fails to change the color profile (and I don't see any error in the output of layout tests). I tried it on a headless Mac Mini (10.6.8) and there layout tests was able to change to Generic RGB Profile (and thus the tests passed). For now, my fix is going to be: don't run layout tests on headless MacPros; use headless MacMinis instead. Is this a Known Issue? Any suggestions? Thanks -Elliot ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] layout tests cannot set Generic RGB Color Profile on headless MacPro?
On Fri, Oct 7, 2011 at 3:06 PM, Ryosuke Niwa rn...@webkit.org wrote: Yeah, layout tests fail on Mac without a screen connected to it. Chromium buildbots use Mac minis connected to KVMs for that reason. I'm actually surprised to learn that tests passed on a headless Mac mini. Ah, then you will perhaps NOT be surprised to learn: that MacMini actually WAS plugged into a monitor (but the monitor was far away and hadn't been turned on for months--thus I forgot about it). I plugged a monitor into the MacPro, and unplugged the monitor from the MacMini, and established the following for both MacPro and MacMini (restarting the computer each time): - monitor plugged in and turned on: layout tests work fine - monitor plugged in and turned off: layout tests work fine - no monitor plugged in: layout tests break I also tried plugging various adapters (Mini DisplayPort to DVI, Mini DisplayPort to VGA, HDMI to DVI) into the computer but *not* connecting a monitor to them; in all cases, it was the same as plugging in no monitor cable at all (layout tests break). Presumably connecting to a KVM or other fake monitor, as Ryosuke mentions, would work. Thanks all for the information. - Ryosuke On Fri, Oct 7, 2011 at 11:53 AM, Simon Fraser simon.fra...@apple.comwrote: On Oct 7, 2011, at 11:45 AM, Elliot Poger wrote: I was having problems with lots of image mismatches running WebKit layout tests on a headless MacPro (10.6.8). (The image diffs were very minor adjustments in scroll bar shading.) Eventually, out of frustration, I tried running the same test on that headless MacPro as well as my desktop MacPro (also 10.6.8). The test succeeded on my desktop MacPro but not the headless MacPro. One difference I have noticed is that the Display Profile (under System Preferences Displays Color) is set to sRGB IEC61966-2.1 (my desktop was set to Generic RGB Profile). I tried setting it to Generic RGB Profile on the headless machine, but when I closed and reopened System Preferences it had reverted to sRGB IEC61966-2.1. I tried setting the Display Profile on my desktop to sRGB IEC61966-2.1, and the color scheme was noticeably brighter... but when I ran layout tests afterwards, I saw that layout tests automatically set the profile back to Generic RGB Profile temporarily for the test. When I run layout tests on the headless MacPro, it seems that it fails to change the color profile (and I don't see any error in the output of layout tests). I tried it on a headless Mac Mini (10.6.8) and there layout tests was able to change to Generic RGB Profile (and thus the tests passed). For now, my fix is going to be: don't run layout tests on headless MacPros; use headless MacMinis instead. Is this a Known Issue? Any suggestions? I'm aware that there are color profile issues when running pixel tests, but I was not aware of differences between hardware. We may be able to fix DumpRenderTree/WebKitTestRunner to change the color profile just for the test window, and not globally. I haven't tried that yet. Finally, I've noticed some changes in color profile behavior on Lion, so if you try running tests there, you may see a new set of problems. Simon ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
