Re: [webkit-dev] Reflecting pixel delta "distance" in ImageDiff
Food for thought... Skia's "skdiff" tool generates two diff images for each image pair: - 1. every pixel which is different at all between images 1 & 2 shows up as white - 2. every pixel shows the color difference between images 1 & 2 (much more subtle) For an example of what this looks like, see http://skia.googlecode.com/svn/trunk/tools/tests/skdiff/test1/output-expected/index.html On Fri, Jun 15, 2012 at 7:24 PM, Tony Payne wrote: > I would like to change chromium's ImageDiff to reflect the magnitude of > pixel changes. Currently, if the pixel has any difference, the entire pixel > is marked as 100% red. I'd like to change it so that miniscule difference > are 20% red and large differences are 100% red. Looking at the code for CG, > gtk and Win versions of ImageDiff, I think they already do something > similar. Is this correct? > > Would anyone be opposed to such a change? See the attached image for an > example of how this could assist in debugging regressions. It would > certainly be enormously useful in my color management efforts. > > Thanks, > Tony > > The CL in question is https://bugs.webkit.org/show_bug.cgi?id=89253 > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 4:44 PM, Peter Kasting wrote: > On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger wrote: > >> Can someone please remind me why IMAGE+TEXT even exists? >> >> Wouldn't it be simpler to just mark a test as follows? >> >>- IMAGE : allow image failure; go red if there is a text failure >>- TEXT: allow text failure; go red if there is an image failure >>- IMAGE TEXT: allow text and/or image failure >> >> The distinction is that IMAGE TEXT will allow image, text, or both to > fail, thus making transitions among the three generate no events. > IMAGE+TEXT says specifically that we expect both to fail and that if one > starts passing, someone should do something. (For example, maybe someone > checks in a partial rebaseline where they miss the image expectations.) > Thanks for the explanation. That makes sense, although it seems to me that the problem of no-events-generated-by-changes-in-actual-images-while-IMAGE-failure-is-expected is about 100x worse for us. But that's not a reason to hide these particular transitions! :-) > > PK > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 6:53 PM, Dirk Pranke wrote: > > * We'll probably rename IMAGE+TEXT to IMAGE_AND_TEXT. > > Can someone please remind me why IMAGE+TEXT even exists? Wouldn't it be simpler to just mark a test as follows? - IMAGE : allow image failure; go red if there is a text failure - TEXT: allow text failure; go red if there is an image failure - IMAGE TEXT: allow text and/or image failure ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] please add EditBugs permission
Can someone please add EditBugs permission for my bugs.webkit.org account (
epo...@chromium.org)?
Another WebKit committer and I both tried it, using the steps Eric pasted
in April, but it failed... I think maybe the add-users-to-groups tool is
broken?
$ Tools/Scripts/webkit-patch add-users-to-groups epo...@chromium.org
Add users matching "epo...@chromium.org" which groups?
1. canconfirm
2. editbugs
Enter one or more numbers (comma-separated), or "all": all
bugs.webkit.org login: epo...@google.com
bugs.webkit.org password for epo...@google.com:
Logging in as epo...@google.com...
Traceback (most recent call last):
File "Tools/Scripts/webkit-patch", line 69, in
main()
File "Tools/Scripts/webkit-patch", line 64, in main
WebKitPatch(os.path.abspath(__file__)).main()
File
"/usr/local/google/home/epoger/src/webkit/green/WebKit/Tools/Scripts/webkitpy/tool/multicommandtool.py",
line 311, in main
result = command.check_arguments_and_execute(options, args, self)
File
"/usr/local/google/home/epoger/src/webkit/green/WebKit/Tools/Scripts/webkitpy/tool/multicommandtool.py",
line 120, in check_arguments_and_execute
return self.execute(options, args, tool) or 0
File
"/usr/local/google/home/epoger/src/webkit/green/WebKit/Tools/Scripts/webkitpy/tool/commands/adduserstogroups.py",
line 50, in execute
login_userid_pairs =
tool.bugs.queries.fetch_login_userid_pairs_matching_substring(search_string)
File
"/usr/local/google/home/epoger/src/webkit/green/WebKit/Tools/Scripts/webkitpy/common/net/bugzilla/bugzilla.py",
line 257, in fetch_login_userid_pairs_matching_substring
return
EditUsersParser().login_userid_pairs_from_edit_user_results(results_page)
File
"/usr/local/google/home/epoger/src/webkit/green/WebKit/Tools/Scripts/webkitpy/common/net/bugzilla/bugzilla.py",
line 72, in login_userid_pairs_from_edit_user_results
login_userid_pairs = [self._login_and_uid_from_row(row) for row in
results_table('tr')]
TypeError: 'NoneType' object is not callable
On Thu, Apr 12, 2012 at 6:47 PM, Eric Seidel wrote:
> Done.
>
> % webkit-patch add-users-to-groups shezbaig...@gmail.com
> Add users matching "shezbaig...@gmail.com" which groups?
> 1. canconfirm
> 2. editbugs
> Enter one or more numbers (comma-separated), or "all":
> Reading Keychain for bugs.webkit.org account and password. Click
> "Allow" to continue...
> Logging in as e...@webkit.org...
> Found 1 users matching shezbaig...@gmail.com:
> shezbaig...@gmail.com (17597)
> Are you sure you want add 1 users to groups ['canconfirm',
> 'editbugs']? (This action cannot be undone using webkit-patch.)
> [Y/n]:
> Adding shezbaig...@gmail.com to ['canconfirm', 'editbugs']
>
> On Thu, Apr 12, 2012 at 3:17 PM, Shezan Baig
> wrote:
> > Hello,
> >
> > When I submit a patch to bugzilla, I get an error that EditBugs is not
> set
> > on my account. Can someone add this permission to my bugzilla account
> > please? I sent a message to as
> > instructed, but that doesn't get through.
> >
> > My login is: shezbaig...@gmail.com
> > My patch submitted to: https://bugs.webkit.org/show_bug.cgi?id=80382
> >
> > Thanks,
> > Shezan
> >
> >
> > ___
> > webkit-dev mailing list
> > webkit-dev@lists.webkit.org
> > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
> >
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] optimising png files in LayoutTests - an experiment
In deciding whether to just let the PNGs get shrunk over time or to do a
pass over all of them at once, I think we should decide what we are trying
to optimize for. (Download time for initial checkout of the webkit tree?
Time for existing developers to update their trees? Storage space in the
repository?)
I would think that, except for initial checkout of the webkit repository,
the others would be optimized by lazily converting the files (waiting until
we need to check in new baselines, instead of shrinking them all at once).
On Wed, Jan 25, 2012 at 1:12 PM, Dirk Pranke wrote:
> I like the idea. I'm not sure I agree w/ Adam that I'd roll the code
> into DRT, insofar as I don't know how big it is and I would definitely
> want the code shared across all of the DRT implementations. I'd
> probably also do a pass over all of the existing PNGs at some point
> rather than wait for them to be updated as necessary.
>
> -- Dirk
>
> On Tue, Jan 24, 2012 at 10:43 PM, Mike Lawther
> wrote:
> > Hi guys,
> >
> > Just thought I'd share the results of an experiment I did in optimising
> the
> > png files in LayoutTests.
> >
> > I used a tool on Mac called ImageOptim (http://imageoptim.pornel.net/)
> which
> > tries a set of different png lossless optimising tools (like pngcrush - I
> > also downloaded and included PNGOUT). Note that this strips out some of
> the
> > stuff we need, like the hash, so if doing this for real we'd have to
> watch
> > out for that.
> >
> > My results were:
> >
> > Before:
> > $ ls -laR LayoutTests/ | grep png$ | awk '{total = total + $5}END{print
> > total}'
> > 1220535840
> >
> > Test:
> > $ find LayoutTests/ | grep png$ | xargs open -a ImageOptim.app
> >
> > After:
> > $ ls -laR LayoutTests/ | grep png$ | awk '{total = total + $5}END{print
> > total, 1220535840 - total}'
> > 937198328 283337512
> >
> > So this has saved ~280MB (~23% of the original size).
> >
> > Based on this, it seems worthwhile to include a png optimiser somewhere
> in
> > the patch upload/submit toolchain, and also (separately) to optimise the
> > existing pngs.
> >
> > Thoughts?
> >
> > mike
> >
> >
> > ___
> > webkit-dev mailing list
> > webkit-dev@lists.webkit.org
> > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
> >
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] layout tests: how are some compared against image, and others only text?
What is it that causes some tests to require baseline images (and not text files) for comparison, while others require text and not image baselines? (I know that I can specifically SKIP comparison against IMAGE and/or TEXT using test_expectations.txt... but even without the use of test_expectations, I believe that some tests are compared against only text or only image.) As an example, I see that this test has only image baselines and no text baselines: $ ls LayoutTests/platform/*/fast/canvas/canvas-text-baseline*.png | wc -l 10 $ ls LayoutTests/platform/*/fast/canvas/canvas-text-baseline*.txt | wc -l ls: cannot access LayoutTests/platform/*/fast/canvas/canvas-text-baseline*.txt: No such file or directory 0 while this test has only text baselines and no image baselines: $ ls LayoutTests/platform/*/fast/canvas/canvas-lineWidth*.png | wc -l ls: cannot access LayoutTests/platform/*/fast/canvas/canvas-lineWidth*.png: No such file or directory 0 $ ls LayoutTests/platform/*/fast/canvas/canvas-lineWidth*.txt | wc -l 5 Is there something inherent in each test that indicates whether its results will be compared against image and/or text baselines? Or is it simply a matter of what baseline files are found to compare against? ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] chromium-cg-mac results
Filed https://bugs.webkit.org/show_bug.cgi?id=75548 ('[rollup] remove
chromium-cg-mac baselines')
On Wed, Jan 4, 2012 at 10:37 AM, Adam Barth wrote:
> On Wed, Jan 4, 2012 at 6:10 AM, Elliot Poger wrote:
> > I agree that it is reasonable (and a good idea) to remove the
> > chromium-cg-mac expectations from WebKit now, and I am willing to take
> the
> > lead on doing so (although I will need help from WebKit committers).
>
> Sounds like a good plan. I'm happy to help. Let's coordinate off-list.
>
> Thanks,
> Adam
>
>
> > Please let me know if anyone agrees/disagrees with the following steps
> to do
> > so:
> >
> > 1. remove the following buildbots that rely on chromium-cg-mac
> expectations
> > (otherwise, they will start failing once the CG expectations disappear):
> >
> >
> http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac%20Builder%20%28CG%29%28deps%29
> >
> http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.6%20%28CG%29%28deps%29
> >
> http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac%20Builder%20%28CG%29
> >
> http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.5%20%28CG%29
> >
> http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.6%20%28CG%29
> >
> http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac%20Builder%20%28CG%29%28dbg%29
> >
> http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.5%20%28CG%29%28dbg%29%281%29
> >
> http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.5%20%28CG%29%28dbg%29%282%29
> >
> http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.6%20%28CG%29%28dbg%29
> >
> > 2. remove the LayoutTests/platform/*-cg-* directories from the WebKit
> repo
> >
> > 3. remove any CG-specific entries from
> > LayoutTests/platform/chromium/test-expectations.txt
> >
> > 4. remove any CG-specific test code from the Tools/Scripts/webkitpy tree
> >
> >
> > On Wed, Jan 4, 2012 at 12:01 AM, David Levin wrote:
> >>
> >>
> >>
> >> On Tue, Jan 3, 2012 at 8:33 PM, Dirk Pranke
> wrote:
> >>>
> >>> On Tue, Jan 3, 2012 at 3:28 PM, Adam Barth wrote:
> >>> > On Tue, Jan 3, 2012 at 3:22 PM, Nico Weber
> wrote:
> >>> >> On Tue, Jan 3, 2012 at 3:00 PM, Adam Barth
> wrote:
> >>> >>> It looks like Chromium Mac has successfully moved to Skia.
> >>> >>
> >>> >> I'd wait with this assessment until a version of Chrome with Skia
> has
> >>> >> shipped to stable. Things are looking really good so that should be
> >>> >> smooth sailing, but it's a bit early to say we're "successfully
> moved"
> >>> >> :-)
> >>> >
> >>> > Fair enough. However, I believe the Skia transition plan called for
> >>> > removing the chromium-cg-mac expectation much earlier than a Skia
> >>> > build shipping to stable. Originally, we were only supposed to have
> >>> > to maintain both sets of expectations for about a month. The
> >>> > transition has taken longer than expected, but it seems like we have
> >>> > sufficient confidence in Skia now that we can remove the
> >>> > chromium-cg-mac expectations.
> >>> >
> >>>
> >>> Has the skia transition hit beta yet? It seems like as soon as we get
> >>> it onto a version that is pointing to a branched version of webkit, we
> >>> should be completely safe to remove the directories on trunk (frankly,
> >>> I'd agree with Adam that it's probably safe to remove it now, since we
> >>> can always add them back in if we have to, but I can compromise as
> >>> well).
> >>
> >>
> >> Remove it.
> >>
> >> It is a cost on everyone who enlists in WebKit. Those of us who aren't
> >> creating new enlistments are not affected much but that doesn't mean it
> >> isn't costly.
> >>
> >> dave
> >>
> >
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] chromium-cg-mac results
I agree that it is reasonable (and a good idea) to remove the chromium-cg-mac expectations from WebKit now, and I am willing to take the lead on doing so (although I will need help from WebKit committers). Please let me know if anyone agrees/disagrees with the following steps to do so: 1. remove the following buildbots that rely on chromium-cg-mac expectations (otherwise, they will start failing once the CG expectations disappear): - http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac%20Builder%20%28CG%29%28deps%29 - http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.6%20%28CG%29%28deps%29 - http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac%20Builder%20%28CG%29 - http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.5%20%28CG%29 - http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.6%20%28CG%29 - http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac%20Builder%20%28CG%29%28dbg%29 - http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.5%20%28CG%29%28dbg%29%281%29 - http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.5%20%28CG%29%28dbg%29%282%29 - http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.6%20%28CG%29%28dbg%29 2. remove the LayoutTests/platform/*-cg-* directories from the WebKit repo 3. remove any CG-specific entries from LayoutTests/platform/chromium/test-expectations.txt 4. remove any CG-specific test code from the Tools/Scripts/webkitpy tree On Wed, Jan 4, 2012 at 12:01 AM, David Levin wrote: > > > On Tue, Jan 3, 2012 at 8:33 PM, Dirk Pranke wrote: > >> On Tue, Jan 3, 2012 at 3:28 PM, Adam Barth wrote: >> > On Tue, Jan 3, 2012 at 3:22 PM, Nico Weber wrote: >> >> On Tue, Jan 3, 2012 at 3:00 PM, Adam Barth wrote: >> >>> It looks like Chromium Mac has successfully moved to Skia. >> >> >> >> I'd wait with this assessment until a version of Chrome with Skia has >> >> shipped to stable. Things are looking really good so that should be >> >> smooth sailing, but it's a bit early to say we're "successfully moved" >> >> :-) >> > >> > Fair enough. However, I believe the Skia transition plan called for >> > removing the chromium-cg-mac expectation much earlier than a Skia >> > build shipping to stable. Originally, we were only supposed to have >> > to maintain both sets of expectations for about a month. The >> > transition has taken longer than expected, but it seems like we have >> > sufficient confidence in Skia now that we can remove the >> > chromium-cg-mac expectations. >> > >> >> Has the skia transition hit beta yet? It seems like as soon as we get >> it onto a version that is pointing to a branched version of webkit, we >> should be completely safe to remove the directories on trunk (frankly, >> I'd agree with Adam that it's probably safe to remove it now, since we >> can always add them back in if we have to, but I can compromise as >> well). >> > > Remove it. > > It is a cost on everyone who enlists in WebKit. Those of us who aren't > creating new enlistments are not affected much but that doesn't mean it > isn't costly. > > dave > > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] "Tools/Scripts/webkit-patch rebaseline-expectations" does not launch html comparison page?
I finally got back to this and tried to use garden-o-matic. I launched it
("./Tools/Scripts/webkit-patch
garden-o-matic") and it did nothing. I opened a separate thread about
this:
http://groups.google.com/a/chromium.org/group/chromium-dev/browse_thread/thread/a64eabe35e59ac17#('garden-o-matic
does nothing?')
Thus I am still unable to rebaseline tests across multiple platforms.
Is there a technique that actually works, with tools that exist today?
On Tue, Nov 8, 2011 at 4:08 PM, Adam Barth wrote:
> On Tue, Nov 8, 2011 at 1:00 PM, Elliot Poger wrote:
> > How do the gardeners do the rebaselining currently? It seems like what
> I'm
> > looking for is pretty much akin to gardening...
>
> They use garden-o-matic, which displays the diffs prior to conducting
> the rebaseline.
>
> > I have looked at
> http://www.chromium.org/developers/how-tos/webkit-gardening
> > , but I have no idea if it is current.
>
> It is current.
>
> Adam
>
>
> > On Tue, Nov 8, 2011 at 3:31 PM, Dirk Pranke
> wrote:
> >> On Tue, Nov 8, 2011 at 12:24 PM, Adam Barth wrote:
> >> > On Tue, Nov 8, 2011 at 12:20 PM, Tony Chang
> wrote:
> >> >> On Tue, Nov 8, 2011 at 12:00 PM, Elliot Poger
> >> >> wrote:
> >> >>> Perhaps I should approach this from a different angle:
> >> >>> What is the recommended procedure for:
> >> >>> - generating new baseline images for a few dozen failing tests, on
> >> >>> various
> >> >>> platforms
> >> >>
> >> >> webkit-patch rebaseline-expectations
> >> >>
> >> >>> - visually inspecting them to make sure they're not bogus
> >> >>
> >> >> Would 'webkit-patch pretty-diff' work for you? It should show the
> >> >> files
> >> >> being added/deleted, but it won't generate a pixel diff.
> >> >
> >> > The tricky part is that this view requires you to understand all the
> >> > fallback behavior among different ports. My sense is that this would
> >> > be easier if we had a smarter view that understood that and presented
> >> > it to the user in an understandable way. Unfortunately, no one has
> >> > built that view yet.
> >>
> >> rebaseline-chromium-webkit-tests had some careful logging to stdout
> >> that explained what files were (or weren't) being updated and why
> >> (i.e., I claim that I had solved this problem in that script. Although
> >> it wasn't presented in the HTML, that wouldn't have been that hard to
> >> add).
> >>
> >> I think if we could get the equivalent into the new tool, and if we
> >> could separate the update and optimize steps, that would probably be
> >> good enough. I think combining update and optimize makes it *very*
> >> hard to determine the correctness of what you've done.
> >>
> >> In other words, my ideal workflow would be update --> review & approve
> >> --> optimize --> [optionally review optimze?] --> land.
> >>
> >> -- Dirk
> >
> >
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] "Tools/Scripts/webkit-patch rebaseline-expectations" does not launch html comparison page?
How do the gardeners do the rebaselining currently? It seems like what I'm looking for is pretty much akin to gardening... I have looked at http://www.chromium.org/developers/how-tos/webkit-gardening, but I have no idea if it is current. On Tue, Nov 8, 2011 at 3:31 PM, Dirk Pranke wrote: > On Tue, Nov 8, 2011 at 12:24 PM, Adam Barth wrote: > > On Tue, Nov 8, 2011 at 12:20 PM, Tony Chang wrote: > >> On Tue, Nov 8, 2011 at 12:00 PM, Elliot Poger > wrote: > >>> Perhaps I should approach this from a different angle: > >>> What is the recommended procedure for: > >>> - generating new baseline images for a few dozen failing tests, on > various > >>> platforms > >> > >> webkit-patch rebaseline-expectations > >> > >>> - visually inspecting them to make sure they're not bogus > >> > >> Would 'webkit-patch pretty-diff' work for you? It should show the files > >> being added/deleted, but it won't generate a pixel diff. > > > > The tricky part is that this view requires you to understand all the > > fallback behavior among different ports. My sense is that this would > > be easier if we had a smarter view that understood that and presented > > it to the user in an understandable way. Unfortunately, no one has > > built that view yet. > > rebaseline-chromium-webkit-tests had some careful logging to stdout > that explained what files were (or weren't) being updated and why > (i.e., I claim that I had solved this problem in that script. Although > it wasn't presented in the HTML, that wouldn't have been that hard to > add). > > I think if we could get the equivalent into the new tool, and if we > could separate the update and optimize steps, that would probably be > good enough. I think combining update and optimize makes it *very* > hard to determine the correctness of what you've done. > > In other words, my ideal workflow would be update --> review & approve > --> optimize --> [optionally review optimze?] --> land. > > -- Dirk > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] "Tools/Scripts/webkit-patch rebaseline-expectations" does not launch html comparison page?
Perhaps I should approach this from a different angle:
What is the recommended procedure for:
- generating new baseline images for a few dozen failing tests, on various
platforms
- visually inspecting them to make sure they're not bogus
- committing them, along with test_expectations changes, to the WebKit repo
On Tue, Nov 8, 2011 at 2:48 PM, Eric Seidel wrote:
> I believe that tool just got nuked. Seems the Wiki page should as well:
> https://bugs.webkit.org/show_bug.cgi?id=71833
>
> On Tue, Nov 8, 2011 at 9:09 AM, Adam Barth wrote:
> > On Tue, Nov 8, 2011 at 7:06 AM, Elliot Poger
> wrote:
> >> I have just run "Tools/Scripts/webkit-patch rebaseline-expectations".
> There
> >> are plenty of image rebaseline changes ("svn status" yields 792
> lines!), but
> >> I'm not seeing the promised html page to compare old and new baselines.
> >
> > Correct. It does not launch a comparison page.
> >
> >> The docs at http://trac.webkit.org/wiki/Rebaseline indicate that such
> an
> >> html page will be launched by default.
> >
> > That's documentation for a different tool.
> >
> >> Is there some flag I have to pass to make this happen?
> >
> > There is no flag.
> >
> > My plan was to add the ability to take an arbitrary patch in the
> > working copy and show a pretty-diff like view of how the patch effects
> > layout test results. I haven't had time to implement that yet. If
> > you're interested in adding that feature, let me know and I'll help
> > you get started.
> >
> > Adam
> > ___
> > webkit-dev mailing list
> > webkit-dev@lists.webkit.org
> > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
> >
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] "Tools/Scripts/webkit-patch rebaseline-expectations" does not launch html comparison page?
Perhaps I should approach this from a different angle:
What is the recommended procedure for:
- generating new baseline images for a few dozen failing tests, on various
platforms
- visually inspecting them to make sure they're not bogus
- committing them, along with test_expectations changes, to the WebKit repo
On Tue, Nov 8, 2011 at 2:48 PM, Eric Seidel wrote:
> I believe that tool just got nuked. Seems the Wiki page should as well:
> https://bugs.webkit.org/show_bug.cgi?id=71833
>
> On Tue, Nov 8, 2011 at 9:09 AM, Adam Barth wrote:
> > On Tue, Nov 8, 2011 at 7:06 AM, Elliot Poger
> wrote:
> >> I have just run "Tools/Scripts/webkit-patch rebaseline-expectations".
> There
> >> are plenty of image rebaseline changes ("svn status" yields 792
> lines!), but
> >> I'm not seeing the promised html page to compare old and new baselines.
> >
> > Correct. It does not launch a comparison page.
> >
> >> The docs at http://trac.webkit.org/wiki/Rebaseline indicate that such
> an
> >> html page will be launched by default.
> >
> > That's documentation for a different tool.
> >
> >> Is there some flag I have to pass to make this happen?
> >
> > There is no flag.
> >
> > My plan was to add the ability to take an arbitrary patch in the
> > working copy and show a pretty-diff like view of how the patch effects
> > layout test results. I haven't had time to implement that yet. If
> > you're interested in adding that feature, let me know and I'll help
> > you get started.
> >
> > Adam
> > ___
> > webkit-dev mailing list
> > webkit-dev@lists.webkit.org
> > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
> >
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] insanity of updating 4000+ baseline images due to font rendering change?
We finally have correct baseline images checked in for Chromium-Skia on
SnowLeopard (hooray). Now we face the daunting task of rebaselining for
Chromium-Skia on Leopard.
The problem is that chromium-mac-leopard has ~4500 image diffs compared to
chromium-mac. The vast majority of these diffs are due to very minor
differences in text rendering between Leopard and SnowLeopard. (Details are
in http://code.google.com/p/chromium/issues/detail?id=100904 )
I poked around in LayoutTests/platform (details are at the bottom of this
email) and found that indeed there are 4000-5000 baseline images that
distinguish between Leopard and SnowLeopard. This is presumably why layout
tests generally pass on both platforms.
Before Cary and I spend 2-3 days committing all these new baseline images so
that we can re-enable test_expectations for SnowLeopard... is this the right
approach? It seems pretty inefficient to clog the WebKit repository with
all these baseline images with minor pixel value differences.
Here are the various approaches I can think of... what's the
Hive-Mind-Approved approach?
- Commit 4500 new baseline images for SnowLeopard
- pro: known to work, will catch any regressions that come later
- con: takes a long time to commit, chews up disk space and bandwidth
for all developers, future minor changes may require yet another
set of new
baselines
- Leave all SnowLeopard tests marked as "PASS FAIL" (or maybe mark them
"SKIP") in test_expectations
- pro: known to work, quick and easy, doesn't clog repo space and
developer update bandwidth, future minor changes won't break any bots
- con: will not catch any regressions that come later on SnowLeopard
- Remove descriptive text from all these tests, so that text rendering is
only evaluated in tests specifically for that purpose
- pro: prevents this problem for future OS versions, should allow for
lots more baseline images to be shared across platforms
- con: a lot of work to replace all existing baseline images, must
coordinate across community of Chromium/WebKit developers, tests will be
more difficult to interpret without text
- Figure out how our test pages can be rendered with a completely
cross-platform pixel-equivalent font
- pro: similar to above but tests keep their descriptive text
- con: similar to above but more technically challenging
- Augment our pixel-diff tools to allow for comparison masks (only pay
attention to pixel diffs within this rectangle)
- pro: existing baseline images can stay in place, and perhaps be
shared with new OS versions and platforms
- con: requires modification of pixel-diff tools, need to add
comparison mask to each test definition
Details on how I determined that there are already 4000-5000 baseline images
that distinguish between Leopard and SnowLeopard for existing platforms
(Safari and chromium-cg):
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ mkdir tests
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ find
platform -name *.png >tests/all-pngs
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ wc -l
tests/all-pngs
45529 tests/all-pngs
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ cat
tests/all-pngs | awk -F '/' '{print $2}' | sort | uniq >
tests/list-all-platforms
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ grep mac
tests/list-all-platforms >tests/list-mac-platforms
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ mkdir
tests/by-platform
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ for PLATFORM
in $(cat tests/list-mac-platforms); do grep ^platform/$PLATFORM/
tests/all-pngs | sed s/platform\\/$PLATFORM\\///
>tests/by-platform/$PLATFORM ; done
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ wc -l
tests/by-platform/*
1050 tests/by-platform/chromium-cg-mac
1575 tests/by-platform/chromium-cg-mac-leopard
114 tests/by-platform/chromium-cg-mac-snowleopard
114 tests/by-platform/chromium-gpu-cg-mac
135 tests/by-platform/chromium-gpu-mac
4909 tests/by-platform/chromium-mac
79 tests/by-platform/chromium-mac-leopard
131 tests/by-platform/chromium-mac-snowleopard
7631 tests/by-platform/mac
4435 tests/by-platform/mac-leopard
247 tests/by-platform/mac-snowleopard
7 tests/by-platform/mac-wk2
20427 total
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ cat
tests/by-platform/chromium-cg-mac
tests/by-platform/chromium-cg-mac-snowleopard tests/by-platform/mac
tests/by-platform/mac-snowleopard | sort | uniq
>tests/list-snowleopard-cg-images
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ cat
tests/by-platform/chromium-cg-mac-leopard
tests/by-platform/mac-leopard | sort | uniq
>tests/list-leopard-cg-images
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ cat
tests/list-leopard-cg-images tests/list-snowleopard-cg-images | sort |
uniq
Re: [webkit-dev] layout tests cannot set Generic RGB Color Profile on headless MacPro?
On Fri, Oct 7, 2011 at 3:06 PM, Ryosuke Niwa wrote: > Yeah, layout tests fail on Mac without a screen connected to it. Chromium > buildbots use Mac minis connected to KVMs for that reason. I'm actually > surprised to learn that tests passed on a headless Mac mini. > Ah, then you will perhaps NOT be surprised to learn: that MacMini actually WAS plugged into a monitor (but the monitor was far away and hadn't been turned on for months--thus I forgot about it). I plugged a monitor into the MacPro, and unplugged the monitor from the MacMini, and established the following for both MacPro and MacMini (restarting the computer each time): - monitor plugged in and turned on: layout tests work fine - monitor plugged in and turned off: layout tests work fine - no monitor plugged in: layout tests break I also tried plugging various adapters (Mini DisplayPort to DVI, Mini DisplayPort to VGA, HDMI to DVI) into the computer but *not* connecting a monitor to them; in all cases, it was the same as plugging in no monitor cable at all (layout tests break). Presumably connecting to a KVM or other "fake monitor", as Ryosuke mentions, would work. Thanks all for the information. > - Ryosuke > > On Fri, Oct 7, 2011 at 11:53 AM, Simon Fraser wrote: > >> On Oct 7, 2011, at 11:45 AM, Elliot Poger wrote: >> >> > I was having problems with lots of image mismatches running WebKit >> layout tests on a headless MacPro (10.6.8). (The image diffs were very >> minor adjustments in scroll bar shading.) >> > >> > Eventually, out of frustration, I tried running the same test on that >> headless MacPro as well as my desktop MacPro (also 10.6.8). The test >> succeeded on my desktop MacPro but not the headless MacPro. >> > >> > One difference I have noticed is that the Display Profile (under System >> Preferences > Displays > Color) is set to "sRGB IEC61966-2.1" (my desktop >> was set to "Generic RGB Profile"). I tried setting it to "Generic RGB >> Profile" on the headless machine, but when I closed and reopened System >> Preferences it had reverted to "sRGB IEC61966-2.1". >> > >> > I tried setting the Display Profile on my desktop to "sRGB >> IEC61966-2.1", and the color scheme was noticeably brighter... but when I >> ran layout tests afterwards, I saw that layout tests automatically set the >> profile back to "Generic RGB Profile" temporarily for the test. When I run >> layout tests on the headless MacPro, it seems that it fails to change the >> color profile (and I don't see any error in the output of layout tests). >> > >> > I tried it on a headless Mac Mini (10.6.8) and there layout tests was >> able to change to Generic RGB Profile (and thus the tests passed). >> > >> > For now, my fix is going to be: don't run layout tests on headless >> MacPros; use headless MacMinis instead. Is this a Known Issue? Any >> suggestions? >> >> I'm aware that there are color profile issues when running pixel tests, >> but I was not aware of differences between hardware. >> >> We may be able to fix DumpRenderTree/WebKitTestRunner to change the color >> profile just for the test window, and not globally. I haven't tried that >> yet. >> >> Finally, I've noticed some changes in color profile behavior on Lion, so >> if you try running tests there, you may see a new set of problems. >> >> Simon >> >> >> ___ >> webkit-dev mailing list >> webkit-dev@lists.webkit.org >> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >> > > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] layout tests cannot set Generic RGB Color Profile on headless MacPro?
I was having problems with lots of image mismatches running WebKit layout tests on a headless MacPro (10.6.8). (The image diffs were very minor adjustments in scroll bar shading.) Eventually, out of frustration, I tried running the same test on that headless MacPro as well as my desktop MacPro (also 10.6.8). The test succeeded on my desktop MacPro but not the headless MacPro. One difference I have noticed is that the Display Profile (under System Preferences > Displays > Color) is set to "sRGB IEC61966-2.1" (my desktop was set to "Generic RGB Profile"). I tried setting it to "Generic RGB Profile" on the headless machine, but when I closed and reopened System Preferences it had reverted to "sRGB IEC61966-2.1". I tried setting the Display Profile on my desktop to "sRGB IEC61966-2.1", and the color scheme was noticeably brighter... but when I ran layout tests afterwards, I saw that layout tests automatically set the profile back to "Generic RGB Profile" temporarily for the test. When I run layout tests on the headless MacPro, it seems that it fails to change the color profile (and I don't see any error in the output of layout tests). I tried it on a headless Mac Mini (10.6.8) and there layout tests was able to change to Generic RGB Profile (and thus the tests passed). For now, my fix is going to be: don't run layout tests on headless MacPros; use headless MacMinis instead. Is this a Known Issue? Any suggestions? Thanks -Elliot ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
