Re: [webkit-dev] "Tools/Scripts/webkit-patch rebaseline-expectations" does not launch html comparison page?
I finally got back to this and tried to use garden-o-matic. I launched it
("./Tools/Scripts/webkit-patch
garden-o-matic") and it did nothing. I opened a separate thread about
this:
http://groups.google.com/a/chromium.org/group/chromium-dev/browse_thread/thread/a64eabe35e59ac17#('garden-o-matic
does nothing?')
Thus I am still unable to rebaseline tests across multiple platforms.
Is there a technique that actually works, with tools that exist today?
On Tue, Nov 8, 2011 at 4:08 PM, Adam Barth wrote:
> On Tue, Nov 8, 2011 at 1:00 PM, Elliot Poger wrote:
> > How do the gardeners do the rebaselining currently? It seems like what
> I'm
> > looking for is pretty much akin to gardening...
>
> They use garden-o-matic, which displays the diffs prior to conducting
> the rebaseline.
>
> > I have looked at
> http://www.chromium.org/developers/how-tos/webkit-gardening
> > , but I have no idea if it is current.
>
> It is current.
>
> Adam
>
>
> > On Tue, Nov 8, 2011 at 3:31 PM, Dirk Pranke
> wrote:
> >> On Tue, Nov 8, 2011 at 12:24 PM, Adam Barth wrote:
> >> > On Tue, Nov 8, 2011 at 12:20 PM, Tony Chang
> wrote:
> >> >> On Tue, Nov 8, 2011 at 12:00 PM, Elliot Poger
> >> >> wrote:
> >> >>> Perhaps I should approach this from a different angle:
> >> >>> What is the recommended procedure for:
> >> >>> - generating new baseline images for a few dozen failing tests, on
> >> >>> various
> >> >>> platforms
> >> >>
> >> >> webkit-patch rebaseline-expectations
> >> >>
> >> >>> - visually inspecting them to make sure they're not bogus
> >> >>
> >> >> Would 'webkit-patch pretty-diff' work for you? It should show the
> >> >> files
> >> >> being added/deleted, but it won't generate a pixel diff.
> >> >
> >> > The tricky part is that this view requires you to understand all the
> >> > fallback behavior among different ports. My sense is that this would
> >> > be easier if we had a smarter view that understood that and presented
> >> > it to the user in an understandable way. Unfortunately, no one has
> >> > built that view yet.
> >>
> >> rebaseline-chromium-webkit-tests had some careful logging to stdout
> >> that explained what files were (or weren't) being updated and why
> >> (i.e., I claim that I had solved this problem in that script. Although
> >> it wasn't presented in the HTML, that wouldn't have been that hard to
> >> add).
> >>
> >> I think if we could get the equivalent into the new tool, and if we
> >> could separate the update and optimize steps, that would probably be
> >> good enough. I think combining update and optimize makes it *very*
> >> hard to determine the correctness of what you've done.
> >>
> >> In other words, my ideal workflow would be update --> review & approve
> >> --> optimize --> [optionally review optimze?] --> land.
> >>
> >> -- Dirk
> >
> >
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] "Tools/Scripts/webkit-patch rebaseline-expectations" does not launch html comparison page?
On Tue, Nov 8, 2011 at 1:00 PM, Elliot Poger wrote: > How do the gardeners do the rebaselining currently? It seems like what I'm > looking for is pretty much akin to gardening... They use garden-o-matic, which displays the diffs prior to conducting the rebaseline. > I have looked at http://www.chromium.org/developers/how-tos/webkit-gardening > , but I have no idea if it is current. It is current. Adam > On Tue, Nov 8, 2011 at 3:31 PM, Dirk Pranke wrote: >> On Tue, Nov 8, 2011 at 12:24 PM, Adam Barth wrote: >> > On Tue, Nov 8, 2011 at 12:20 PM, Tony Chang wrote: >> >> On Tue, Nov 8, 2011 at 12:00 PM, Elliot Poger >> >> wrote: >> >>> Perhaps I should approach this from a different angle: >> >>> What is the recommended procedure for: >> >>> - generating new baseline images for a few dozen failing tests, on >> >>> various >> >>> platforms >> >> >> >> webkit-patch rebaseline-expectations >> >> >> >>> - visually inspecting them to make sure they're not bogus >> >> >> >> Would 'webkit-patch pretty-diff' work for you? It should show the >> >> files >> >> being added/deleted, but it won't generate a pixel diff. >> > >> > The tricky part is that this view requires you to understand all the >> > fallback behavior among different ports. My sense is that this would >> > be easier if we had a smarter view that understood that and presented >> > it to the user in an understandable way. Unfortunately, no one has >> > built that view yet. >> >> rebaseline-chromium-webkit-tests had some careful logging to stdout >> that explained what files were (or weren't) being updated and why >> (i.e., I claim that I had solved this problem in that script. Although >> it wasn't presented in the HTML, that wouldn't have been that hard to >> add). >> >> I think if we could get the equivalent into the new tool, and if we >> could separate the update and optimize steps, that would probably be >> good enough. I think combining update and optimize makes it *very* >> hard to determine the correctness of what you've done. >> >> In other words, my ideal workflow would be update --> review & approve >> --> optimize --> [optionally review optimze?] --> land. >> >> -- Dirk > > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] "Tools/Scripts/webkit-patch rebaseline-expectations" does not launch html comparison page?
How do the gardeners do the rebaselining currently? It seems like what I'm looking for is pretty much akin to gardening... I have looked at http://www.chromium.org/developers/how-tos/webkit-gardening, but I have no idea if it is current. On Tue, Nov 8, 2011 at 3:31 PM, Dirk Pranke wrote: > On Tue, Nov 8, 2011 at 12:24 PM, Adam Barth wrote: > > On Tue, Nov 8, 2011 at 12:20 PM, Tony Chang wrote: > >> On Tue, Nov 8, 2011 at 12:00 PM, Elliot Poger > wrote: > >>> Perhaps I should approach this from a different angle: > >>> What is the recommended procedure for: > >>> - generating new baseline images for a few dozen failing tests, on > various > >>> platforms > >> > >> webkit-patch rebaseline-expectations > >> > >>> - visually inspecting them to make sure they're not bogus > >> > >> Would 'webkit-patch pretty-diff' work for you? It should show the files > >> being added/deleted, but it won't generate a pixel diff. > > > > The tricky part is that this view requires you to understand all the > > fallback behavior among different ports. My sense is that this would > > be easier if we had a smarter view that understood that and presented > > it to the user in an understandable way. Unfortunately, no one has > > built that view yet. > > rebaseline-chromium-webkit-tests had some careful logging to stdout > that explained what files were (or weren't) being updated and why > (i.e., I claim that I had solved this problem in that script. Although > it wasn't presented in the HTML, that wouldn't have been that hard to > add). > > I think if we could get the equivalent into the new tool, and if we > could separate the update and optimize steps, that would probably be > good enough. I think combining update and optimize makes it *very* > hard to determine the correctness of what you've done. > > In other words, my ideal workflow would be update --> review & approve > --> optimize --> [optionally review optimze?] --> land. > > -- Dirk > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] "Tools/Scripts/webkit-patch rebaseline-expectations" does not launch html comparison page?
On Tue, Nov 8, 2011 at 12:31 PM, Dirk Pranke wrote: > On Tue, Nov 8, 2011 at 12:24 PM, Adam Barth wrote: >> On Tue, Nov 8, 2011 at 12:20 PM, Tony Chang wrote: >>> On Tue, Nov 8, 2011 at 12:00 PM, Elliot Poger wrote: Perhaps I should approach this from a different angle: What is the recommended procedure for: - generating new baseline images for a few dozen failing tests, on various platforms >>> >>> webkit-patch rebaseline-expectations >>> - visually inspecting them to make sure they're not bogus >>> >>> Would 'webkit-patch pretty-diff' work for you? It should show the files >>> being added/deleted, but it won't generate a pixel diff. >> >> The tricky part is that this view requires you to understand all the >> fallback behavior among different ports. My sense is that this would >> be easier if we had a smarter view that understood that and presented >> it to the user in an understandable way. Unfortunately, no one has >> built that view yet. > > rebaseline-chromium-webkit-tests had some careful logging to stdout > that explained what files were (or weren't) being updated and why > (i.e., I claim that I had solved this problem in that script. Although > it wasn't presented in the HTML, that wouldn't have been that hard to > add). > > I think if we could get the equivalent into the new tool, and if we > could separate the update and optimize steps, that would probably be > good enough. I think combining update and optimize makes it *very* > hard to determine the correctness of what you've done. > > In other words, my ideal workflow would be update --> review & approve > --> optimize --> [optionally review optimze?] --> land. I'm not religious about the specifics of the workflow. The limiting factor here is someone deciding to spend time solving the problem. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] "Tools/Scripts/webkit-patch rebaseline-expectations" does not launch html comparison page?
On Tue, Nov 8, 2011 at 12:24 PM, Adam Barth wrote: > On Tue, Nov 8, 2011 at 12:20 PM, Tony Chang wrote: >> On Tue, Nov 8, 2011 at 12:00 PM, Elliot Poger wrote: >>> Perhaps I should approach this from a different angle: >>> What is the recommended procedure for: >>> - generating new baseline images for a few dozen failing tests, on various >>> platforms >> >> webkit-patch rebaseline-expectations >> >>> - visually inspecting them to make sure they're not bogus >> >> Would 'webkit-patch pretty-diff' work for you? It should show the files >> being added/deleted, but it won't generate a pixel diff. > > The tricky part is that this view requires you to understand all the > fallback behavior among different ports. My sense is that this would > be easier if we had a smarter view that understood that and presented > it to the user in an understandable way. Unfortunately, no one has > built that view yet. rebaseline-chromium-webkit-tests had some careful logging to stdout that explained what files were (or weren't) being updated and why (i.e., I claim that I had solved this problem in that script. Although it wasn't presented in the HTML, that wouldn't have been that hard to add). I think if we could get the equivalent into the new tool, and if we could separate the update and optimize steps, that would probably be good enough. I think combining update and optimize makes it *very* hard to determine the correctness of what you've done. In other words, my ideal workflow would be update --> review & approve --> optimize --> [optionally review optimze?] --> land. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] "Tools/Scripts/webkit-patch rebaseline-expectations" does not launch html comparison page?
-- Forwarded message -- From: Ryosuke Niwa Date: Tue, Nov 8, 2011 at 12:25 PM Subject: Re: [webkit-dev] "Tools/Scripts/webkit-patch rebaseline-expectations" does not launch html comparison page? To: Tony Chang Cc: Elliot Poger , James Kozianski , webkit-dev@lists.webkit.org, Mike Reed On Tue, Nov 8, 2011 at 12:20 PM, Tony Chang wrote: > Would 'webkit-patch pretty-diff' work for you? It should show the files > being added/deleted, but it won't generate a pixel diff. > Won't. It's really hard to tell what has changed especially because webkit-patch rebaseline-expectations optimize results. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] "Tools/Scripts/webkit-patch rebaseline-expectations" does not launch html comparison page?
On Tue, Nov 8, 2011 at 12:20 PM, Tony Chang wrote: > On Tue, Nov 8, 2011 at 12:00 PM, Elliot Poger wrote: >> Perhaps I should approach this from a different angle: >> What is the recommended procedure for: >> - generating new baseline images for a few dozen failing tests, on various >> platforms > > webkit-patch rebaseline-expectations > >> - visually inspecting them to make sure they're not bogus > > Would 'webkit-patch pretty-diff' work for you? It should show the files > being added/deleted, but it won't generate a pixel diff. The tricky part is that this view requires you to understand all the fallback behavior among different ports. My sense is that this would be easier if we had a smarter view that understood that and presented it to the user in an understandable way. Unfortunately, no one has built that view yet. Adam >> - committing them, along with test_expectations changes, to the WebKit >> repo > > webkit-patch land ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] "Tools/Scripts/webkit-patch rebaseline-expectations" does not launch html comparison page?
On Tue, Nov 8, 2011 at 12:00 PM, Elliot Poger wrote: > Perhaps I should approach this from a different angle: > > What is the recommended procedure for: > - generating new baseline images for a few dozen failing tests, on various > platforms > webkit-patch rebaseline-expectations > - visually inspecting them to make sure they're not bogus > Would 'webkit-patch pretty-diff' work for you? It should show the files being added/deleted, but it won't generate a pixel diff. > - committing them, along with test_expectations changes, to the WebKit repo > webkit-patch land ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] "Tools/Scripts/webkit-patch rebaseline-expectations" does not launch html comparison page?
Perhaps I should approach this from a different angle:
What is the recommended procedure for:
- generating new baseline images for a few dozen failing tests, on various
platforms
- visually inspecting them to make sure they're not bogus
- committing them, along with test_expectations changes, to the WebKit repo
On Tue, Nov 8, 2011 at 2:48 PM, Eric Seidel wrote:
> I believe that tool just got nuked. Seems the Wiki page should as well:
> https://bugs.webkit.org/show_bug.cgi?id=71833
>
> On Tue, Nov 8, 2011 at 9:09 AM, Adam Barth wrote:
> > On Tue, Nov 8, 2011 at 7:06 AM, Elliot Poger
> wrote:
> >> I have just run "Tools/Scripts/webkit-patch rebaseline-expectations".
> There
> >> are plenty of image rebaseline changes ("svn status" yields 792
> lines!), but
> >> I'm not seeing the promised html page to compare old and new baselines.
> >
> > Correct. It does not launch a comparison page.
> >
> >> The docs at http://trac.webkit.org/wiki/Rebaseline indicate that such
> an
> >> html page will be launched by default.
> >
> > That's documentation for a different tool.
> >
> >> Is there some flag I have to pass to make this happen?
> >
> > There is no flag.
> >
> > My plan was to add the ability to take an arbitrary patch in the
> > working copy and show a pretty-diff like view of how the patch effects
> > layout test results. I haven't had time to implement that yet. If
> > you're interested in adding that feature, let me know and I'll help
> > you get started.
> >
> > Adam
> > ___
> > webkit-dev mailing list
> > webkit-dev@lists.webkit.org
> > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
> >
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] "Tools/Scripts/webkit-patch rebaseline-expectations" does not launch html comparison page?
Perhaps I should approach this from a different angle:
What is the recommended procedure for:
- generating new baseline images for a few dozen failing tests, on various
platforms
- visually inspecting them to make sure they're not bogus
- committing them, along with test_expectations changes, to the WebKit repo
On Tue, Nov 8, 2011 at 2:48 PM, Eric Seidel wrote:
> I believe that tool just got nuked. Seems the Wiki page should as well:
> https://bugs.webkit.org/show_bug.cgi?id=71833
>
> On Tue, Nov 8, 2011 at 9:09 AM, Adam Barth wrote:
> > On Tue, Nov 8, 2011 at 7:06 AM, Elliot Poger
> wrote:
> >> I have just run "Tools/Scripts/webkit-patch rebaseline-expectations".
> There
> >> are plenty of image rebaseline changes ("svn status" yields 792
> lines!), but
> >> I'm not seeing the promised html page to compare old and new baselines.
> >
> > Correct. It does not launch a comparison page.
> >
> >> The docs at http://trac.webkit.org/wiki/Rebaseline indicate that such
> an
> >> html page will be launched by default.
> >
> > That's documentation for a different tool.
> >
> >> Is there some flag I have to pass to make this happen?
> >
> > There is no flag.
> >
> > My plan was to add the ability to take an arbitrary patch in the
> > working copy and show a pretty-diff like view of how the patch effects
> > layout test results. I haven't had time to implement that yet. If
> > you're interested in adding that feature, let me know and I'll help
> > you get started.
> >
> > Adam
> > ___
> > webkit-dev mailing list
> > webkit-dev@lists.webkit.org
> > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
> >
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] "Tools/Scripts/webkit-patch rebaseline-expectations" does not launch html comparison page?
I believe that tool just got nuked. Seems the Wiki page should as well:
https://bugs.webkit.org/show_bug.cgi?id=71833
On Tue, Nov 8, 2011 at 9:09 AM, Adam Barth wrote:
> On Tue, Nov 8, 2011 at 7:06 AM, Elliot Poger wrote:
>> I have just run "Tools/Scripts/webkit-patch rebaseline-expectations". There
>> are plenty of image rebaseline changes ("svn status" yields 792 lines!), but
>> I'm not seeing the promised html page to compare old and new baselines.
>
> Correct. It does not launch a comparison page.
>
>> The docs at http://trac.webkit.org/wiki/Rebaseline indicate that such an
>> html page will be launched by default.
>
> That's documentation for a different tool.
>
>> Is there some flag I have to pass to make this happen?
>
> There is no flag.
>
> My plan was to add the ability to take an arbitrary patch in the
> working copy and show a pretty-diff like view of how the patch effects
> layout test results. I haven't had time to implement that yet. If
> you're interested in adding that feature, let me know and I'll help
> you get started.
>
> Adam
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] "Tools/Scripts/webkit-patch rebaseline-expectations" does not launch html comparison page?
On Tue, Nov 8, 2011 at 7:06 AM, Elliot Poger wrote:
> I have just run "Tools/Scripts/webkit-patch rebaseline-expectations". There
> are plenty of image rebaseline changes ("svn status" yields 792 lines!), but
> I'm not seeing the promised html page to compare old and new baselines.
Correct. It does not launch a comparison page.
> The docs at http://trac.webkit.org/wiki/Rebaseline indicate that such an
> html page will be launched by default.
That's documentation for a different tool.
> Is there some flag I have to pass to make this happen?
There is no flag.
My plan was to add the ability to take an arbitrary patch in the
working copy and show a pretty-diff like view of how the patch effects
layout test results. I haven't had time to implement that yet. If
you're interested in adding that feature, let me know and I'll help
you get started.
Adam
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
