Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-14 Thread Dirk Pranke
On Wed, Feb 13, 2013 at 11:33 PM, Benjamin Poulain benja...@webkit.org wrote:
 On Wed, Feb 13, 2013 at 11:16 PM, Dirk Pranke dpra...@chromium.org wrote:

  Those changes are not harmless. There are people monitoring tests
  results
  full time in order to keep WebKit in good shape. No other part of WebKit
  require continuous attention.

 I'm sorry, but either I don't understand Dongsung's suggestion, or I
 don't understand your criticism. What does this sentence paragraph
 mean? Are you suggesting someone needs to look at this type of test
 full time in a way that we don't have to look at the other tests?


 My lack of sleep is likely to blame here :)

 All I am saying is tests have a hight cost when they start failing, we
 should be careful how we design them so we can easily triage bad tests,
 flaky tests, and real errors.


No argument here :).


  Case 1: CSS Filters  Shaders
  I wanted this test functionality when I commented
  http://webkit.org/b/97859#c19
  If I want to make gaussian blur test, I prefer using 'getPixel' test as
  follows,
 
 
  Why wasn't a ref-test a better solution in this particular case?
 

 I'm curious, what would you imagine the ref test contains?


 If I am not mistaken, the composition operations are parallel with the ones
 of SVG and Canvas (aren't they?).
 I would have attempted comparing the 3 implementations as it seems to me the
 pixels values should be the same.

 That was a genuine question about that case :)



  Case 2: Fixed Position Element
  [...]
  function repeatedlyCalledDuringScrolling() {
  ASSERT(getPixel(15, 9) == white);
  ASSERT(getPixel(15, 10) == green);
  ASSERT(getPixel(9, 15) == white);
  ASSERT(getPixel(10, 15) == green);
  
  }
 
 
  I think this shows what I said about correctness and readability:
  -Asserting the correctness of the test and the result becomes close to
  impossible for the reader. One has to review the full code to have a
  chance
  of understanding an error.
  -You cannot cover non trivial cases (images, text, form elements, etc).
  -And it is inefficient. You have to render each frame on the UIProcess,
  move
  it to the WebProcess, and box it for JavaScript to process (with pixel
  format conversions depending on your graphics system)
 
  Of the ideas raised, I think this is one of my least favorite for
  testing
  fixed positioning.
 

 Isn't his suggestion the equivalent of what we do today in text-only
 tests? i.e., printing pass or fail and making you have to look at
 the test itself to see what's being tested?

 If the correctness of the rendering depends on those 4 specific pixels
 having those four specific values, how exactly are you going to verify
 that by looking at it?

 Again, I think I'm just not understanding you here?


 When looking at a test test, you follow the flow to know what it is supposed
 to do and where it breaks.

 How are you supposed to know, _by reading the code_, that the color at
 position 15, 9 should be white?


is the code the test code, or the source code we're testing? Of
course, without any larger context, those assertions are impossible to
interpret. I figured he was just indicating what he meant by a
programmatic check, and a real test case would be more informative.

And of course it's easy to write bad tests that are neither readable
nor maintainable. But we also have lots of pixel tests that are
difficult for people not intimately familiar with the code being
tested to evaluate for correctness, too.

Lastly, I don't think he was suggesting that this would replace all
pixel tests; as you say, it is more well suited for some times of
tests than others.

As far as efficiency goes, even moving the picture back and forth
between two processes is still going to be far faster than running a
second reftest and comparing the output of the two pages.

-- Dirk
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-14 Thread Dongsung Huang


 I'm curious, what would you imagine the ref test contains?


 If I am not mistaken, the composition operations are parallel with the
 ones of SVG and Canvas (aren't they?).
 I would have attempted comparing the 3 implementations as it seems to me
 the pixels values should be the same.

 That was a genuine question about that case :)


Thank you for feedback! :)
I presented gaussian blur case to express my opinion effectively.

As I know, Adobe want to draw CSS Filters and SVG Filters using the same
code. And Canvas does not have something like Filters.
What I want to say is that there are some cases in which snapshot
functionality is more useful.


  Case 2: Fixed Position Element
  [...]
  function repeatedlyCalledDuringScrolling() {
  ASSERT(getPixel(15, 9) == white);
  ASSERT(getPixel(15, 10) == green);
  ASSERT(getPixel(9, 15) == white);
  ASSERT(getPixel(10, 15) == green);
  
  }
 
 
  I think this shows what I said about correctness and readability:
  -Asserting the correctness of the test and the result becomes close to
  impossible for the reader. One has to review the full code to have a
 chance
  of understanding an error.
  -You cannot cover non trivial cases (images, text, form elements, etc).
  -And it is inefficient. You have to render each frame on the UIProcess,
 move
  it to the WebProcess, and box it for JavaScript to process (with pixel
  format conversions depending on your graphics system)
 
  Of the ideas raised, I think this is one of my least favorite for
 testing
  fixed positioning.
 

 Isn't his suggestion the equivalent of what we do today in text-only
 tests? i.e., printing pass or fail and making you have to look at
 the test itself to see what's being tested?

 If the correctness of the rendering depends on those 4 specific pixels
 having those four specific values, how exactly are you going to verify
 that by looking at it?

 Again, I think I'm just not understanding you here?


 When looking at a test test, you follow the flow to know what it is
 supposed to do and where it breaks.

 How are you supposed to know, _by reading the code_, that the color at
 position 15, 9 should be white?

 Benjamin



I agree on Dirk's opinion. I supports 3 points as follows:
1. we can help to understand what's goal of test by comments
2. when we want to see the graphical result, we can use MiniBrowser like
other text based test.
3. philip's canvas tests (http://philip.html5.org/tests/canvas/suite/tests/)
already use well getImageData to test.

I agree that snapshot test can make hardly maintainable tests, but we have
review process and we can make succinct tests like philip canvas test.
And when pixel test is more suitable, we can use pixel test also.

Best regards,

Dongsung Huang
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-13 Thread Dongsung Huang
I like this idea. I cannot find any harm if we have this functionality.
When a pixel test is more succinct, we can make a pixel test. When a
'getPixel on Javascript' test is more succinct, we can make a 'getPixel'
test.
As
http://trac.webkit.org/wiki/Writing%20Layout%20Tests%20for%20DumpRenderTree
said
'Pixel tests are a burden on every ports as any small change in the engine
could lead to your test needing a new pixel result.', new test
functionality that can be an alternative of pixel test is good!
philip canvas tests (LayoutTests/canvas/philip) has no pixel test because
of the power of getImageData API in canvas spec.

I can put two examples for the power of functionality that Noam suggested,

Case 1: CSS Filters  Shaders
I wanted this test functionality when I commented
http://webkit.org/b/97859#c19
If I want to make gaussian blur test, I prefer using 'getPixel' test as
follows,

ASSERT(getPixel(10, 10) = rgba(255, 0, 0, 255));
ASSERT(getPixel(10, 15)  rgba(255, 0, 0, 98)  getPixel(10, 15) 
rgba(255, 0, 0, 102));

If we don't have 'getPixel', we need to capture all port's image results
because all port's painting engine would make a bit different result. As
you know, it is painful.


Case 2: Fixed Position Element
I guess it is one of reasons why Noam suggested.
Currently, Qt and EFL AC (a.k.a Coordinated Graphics) has a problem related
to fixed position element.
During panning animation, Fixed position element is slightly flickering.

If we have 'getPixel', we can make a succinct test case.

PRECONDITION : a green fixed position div is on the white body. The rect of
the div is rect(10, 10, 30, 30).
function repeatedlyCalledDuringScrolling() {
ASSERT(getPixel(15, 9) == white);
ASSERT(getPixel(15, 10) == green);
ASSERT(getPixel(9, 15) == white);
ASSERT(getPixel(10, 15) == green);

}

WDYT?

Best regards,

Dongsung Huang
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-13 Thread Benjamin Poulain
On Wed, Feb 13, 2013 at 9:38 PM, Dongsung Huang luxte...@company100.netwrote:

 I like this idea. I cannot find any harm if we have this functionality.


Those changes are not harmless. There are people monitoring tests results
full time in order to keep WebKit in good shape. No other part of WebKit
require continuous attention.


 Case 1: CSS Filters  Shaders
 I wanted this test functionality when I commented
 http://webkit.org/b/97859#c19
 If I want to make gaussian blur test, I prefer using 'getPixel' test as
 follows,


Why wasn't a ref-test a better solution in this particular case?

Case 2: Fixed Position Element
 [...]
 function repeatedlyCalledDuringScrolling() {
 ASSERT(getPixel(15, 9) == white);
 ASSERT(getPixel(15, 10) == green);
 ASSERT(getPixel(9, 15) == white);
 ASSERT(getPixel(10, 15) == green);
 
 }


I think this shows what I said about correctness and readability:
-Asserting the correctness of the test and the result becomes close to
impossible for the reader. One has to review the full code to have a chance
of understanding an error.
-You cannot cover non trivial cases (images, text, form elements, etc).
-And it is inefficient. You have to render each frame on the UIProcess,
move it to the WebProcess, and box it for JavaScript to process (with pixel
format conversions depending on your graphics system)

Of the ideas raised, I think this is one of my least favorite for testing
fixed positioning.

Benjamin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-13 Thread noam . rosenthal


From: ext Benjamin Poulain benja...@webkit.orgmailto:benja...@webkit.org
Date: Thursday, February 14, 2013 8:07 AM
To: Dongsung Huang luxte...@company100.netmailto:luxte...@company100.net
Cc: webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org 
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org
Subject: Re: [webkit-dev] Testing feature suggestion: animation/interaction 
pixel-results on the fly

On Wed, Feb 13, 2013 at 9:38 PM, Dongsung Huang 
luxte...@company100.netmailto:luxte...@company100.net wrote:
I like this idea. I cannot find any harm if we have this functionality.

Those changes are not harmless. There are people monitoring tests results full 
time in order to keep WebKit in good shape. No other part of WebKit require 
continuous attention.

Case 1: CSS Filters  Shaders
I wanted this test functionality when I commented http://webkit.org/b/97859#c19
If I want to make gaussian blur test, I prefer using 'getPixel' test as follows,

Why wasn't a ref-test a better solution in this particular case?
Because gaussian blurs on the GPU are not accurate, and look slightly different 
on different GPUs, but usually close enough.
We need a way to measure close enough for features where all you can get is 
close enough! Ref-tests and pixel-tests are way to rigid for that, and 
require constant rebaselines and headaches.


Case 2: Fixed Position Element
[...]
function repeatedlyCalledDuringScrolling() {
ASSERT(getPixel(15, 9) == white);
ASSERT(getPixel(15, 10) == green);
ASSERT(getPixel(9, 15) == white);
ASSERT(getPixel(10, 15) == green);

}

I think this shows what I said about correctness and readability:
-Asserting the correctness of the test and the result becomes close to 
impossible for the reader. One has to review the full code to have a chance of 
understanding an error.
We can generate PNGs when a snapshot is requested so that the tester can 
eyeball the results.

-You cannot cover non trivial cases (images, text, form elements, etc).
-And it is inefficient. You have to render each frame on the UIProcess, move it 
to the WebProcess, and box it for JavaScript to process (with pixel format 
conversions depending on your graphics system)
Another option is to do expose a partial comparison mechanism that runs in  the 
UI process.
E.g.
var snapshotHandle = testRunner.createSnapshot();
testRunner.comparePixel(snapshotHandle, 15, 20, 'white', { tolerance: 0.2 });

This way we can generate a visual representation of failures.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-13 Thread Benjamin Poulain
On Wed, Feb 13, 2013 at 11:16 PM, Dirk Pranke dpra...@chromium.org wrote:

  Those changes are not harmless. There are people monitoring tests results
  full time in order to keep WebKit in good shape. No other part of WebKit
  require continuous attention.

 I'm sorry, but either I don't understand Dongsung's suggestion, or I
 don't understand your criticism. What does this sentence paragraph
 mean? Are you suggesting someone needs to look at this type of test
 full time in a way that we don't have to look at the other tests?


My lack of sleep is likely to blame here :)

All I am saying is tests have a hight cost when they start failing, we
should be careful how we design them so we can easily triage bad tests,
flaky tests, and real errors.


 Case 1: CSS Filters  Shaders
  I wanted this test functionality when I commented
  http://webkit.org/b/97859#c19
  If I want to make gaussian blur test, I prefer using 'getPixel' test as
  follows,
 
 
  Why wasn't a ref-test a better solution in this particular case?
 

 I'm curious, what would you imagine the ref test contains?


If I am not mistaken, the composition operations are parallel with the ones
of SVG and Canvas (aren't they?).
I would have attempted comparing the 3 implementations as it seems to me
the pixels values should be the same.

That was a genuine question about that case :)



  Case 2: Fixed Position Element
  [...]
  function repeatedlyCalledDuringScrolling() {
  ASSERT(getPixel(15, 9) == white);
  ASSERT(getPixel(15, 10) == green);
  ASSERT(getPixel(9, 15) == white);
  ASSERT(getPixel(10, 15) == green);
  
  }
 
 
  I think this shows what I said about correctness and readability:
  -Asserting the correctness of the test and the result becomes close to
  impossible for the reader. One has to review the full code to have a
 chance
  of understanding an error.
  -You cannot cover non trivial cases (images, text, form elements, etc).
  -And it is inefficient. You have to render each frame on the UIProcess,
 move
  it to the WebProcess, and box it for JavaScript to process (with pixel
  format conversions depending on your graphics system)
 
  Of the ideas raised, I think this is one of my least favorite for testing
  fixed positioning.
 

 Isn't his suggestion the equivalent of what we do today in text-only
 tests? i.e., printing pass or fail and making you have to look at
 the test itself to see what's being tested?

 If the correctness of the rendering depends on those 4 specific pixels
 having those four specific values, how exactly are you going to verify
 that by looking at it?

 Again, I think I'm just not understanding you here?


When looking at a test test, you follow the flow to know what it is
supposed to do and where it breaks.

How are you supposed to know, _by reading the code_, that the color at
position 15, 9 should be white?

Benjamin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-13 Thread Benjamin Poulain
On Wed, Feb 13, 2013 at 11:27 PM, noam.rosent...@nokia.com wrote:

  Why wasn't a ref-test a better solution in this particular case?

  Because gaussian blurs on the GPU are not accurate, and look slightly
 different on different GPUs, but usually close enough.
 We need a way to measure close enough for features where all you can get
 is close enough! Ref-tests and pixel-tests are way to rigid for that, and
 require constant rebaselines and headaches.


There is _NO_ gaussian blur in that patch.

Benjamin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-13 Thread noam . rosenthal


From: ext Benjamin Poulain benja...@webkit.orgmailto:benja...@webkit.org
Date: Thursday, February 14, 2013 8:36 AM
To: Noam Rosenthal noam.rosent...@nokia.commailto:noam.rosent...@nokia.com
Cc: luxte...@company100.netmailto:luxte...@company100.net 
luxte...@company100.netmailto:luxte...@company100.net, 
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org 
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org
Subject: Re: [webkit-dev] Testing feature suggestion: animation/interaction 
pixel-results on the fly

On Wed, Feb 13, 2013 at 11:27 PM, 
noam.rosent...@nokia.commailto:noam.rosent...@nokia.com wrote:
Why wasn't a ref-test a better solution in this particular case?
Because gaussian blurs on the GPU are not accurate, and look slightly different 
on different GPUs, but usually close enough.
We need a way to measure close enough for features where all you can get is 
close enough! Ref-tests and pixel-tests are way to rigid for that, and 
require constant rebaselines and headaches.

There is _NO_ gaussian blur in that patch.
DongSung mentioned gaussian blur in the comment, which is what I was referring 
to.
But other bugs as well, like delegated scrolling, also have that problem of 
results not being 100% the same every time.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-10 Thread Ryosuke Niwa
On Sat, Feb 9, 2013 at 5:35 PM, Dirk Pranke dpra...@chromium.org wrote:

 Perhaps -- and this is a tangent to this thread -- we could also
 investigate ways of writing tests that did render pngs but told NRWT
 to ignore them (e.g., dumpAsTextAndIgnoredImage() or something), or
 conveyed which parts of the image were important ...


What's the point of that?

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-09 Thread noam . rosenthal


From: ext Benjamin Poulain benja...@webkit.orgmailto:benja...@webkit.org
Date: Saturday, February 9, 2013 12:52 AM
To: Noam Rosenthal noam.rosent...@nokia.commailto:noam.rosent...@nokia.com
Cc: rn...@webkit.orgmailto:rn...@webkit.org 
rn...@webkit.orgmailto:rn...@webkit.org, 
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org 
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org
Subject: Re: [webkit-dev] Testing feature suggestion: animation/interaction 
pixel-results on the fly

On Fri, Feb 8, 2013 at 3:16 PM, 
noam.rosent...@nokia.commailto:noam.rosent...@nokia.com wrote:
The problem with dynamic features of the web like animations/interactions is 
that they're non-deterministic, or at least a lot less deterministic than 
static features of the web like layouts.
Ref tests, pixel tests etc. are tools built for deterministic testing: load a 
file, take a snapshot, compare against a result. Testing an animation (or a 
filter) needs to feel a lot more dynamic and expressive: Animate green boxes, 
make sure that they're within a particular range at particular points in time.

The tests also have to be deterministic and comprehensive. I am afraid of 
loosing both with the Render-to-Canvas approach.

Can you give concrete examples of the kind of bugs you are hunting, and why 
testing cannot use the two methods suggested?
I did not say they cannot be tested with the two methods suggested :) It's more 
 about a preference to move some of those decisions from the test 
infrastructure to the test itself, but potentially those bugs could be tested 
in either way.
Examples for bugs I've encountered/reviewed and can use better in-motion 
testing (note that those are specific to Qt/EFL, but I'm sure there are tons of 
bugs like this that come up for Apple/Google as well)
http://trac.webkit.org/changeset/140825
http://trac.webkit.org/changeset/142112
http://trac.webkit.org/changeset/134953
https://bugs.webkit.org/show_bug.cgi?id=109179

Controlling the clock and programmatically sampling the end result would 
definitely make those more testable, but of course any progress in this area 
would be beneficial and my preference to a canvas-based API is more of an 
opinion.

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-09 Thread Benjamin Poulain
On Sat, Feb 9, 2013 at 1:24 AM, noam.rosent...@nokia.com wrote:

I did not say they cannot be tested with the two methods suggested :)
 It's more  about a preference to move some of those decisions from the test
 infrastructure to the test itself, but potentially those bugs could be
 tested in either way.
 Examples for bugs I've encountered/reviewed and can use better in-motion
 testing (note that those are specific to Qt/EFL, but I'm sure there are
 tons of bugs like this that come up for Apple/Google as well)
 http://trac.webkit.org/changeset/140825
 http://trac.webkit.org/changeset/142112
 http://trac.webkit.org/changeset/134953
 https://bugs.webkit.org/show_bug.cgi?id=109179

  Controlling the clock and programmatically sampling the end result would
 definitely make those more testable, but of course any progress in this
 area would be beneficial and my preference to a canvas-based API is more of
 an opinion.


To explain my concerns:
Sometime, I look at a failing test, and think: what the f**k is this
supposed to test?. Then I have to dig a ton of logs, and finally read the
change to understand a the single JS statement in the whole script that
make the test useful.

This is the situation I am afraid with a solution where pixels would be
evaluated from JavaScript. You can test, but you lack visibility why
something is correct or incorrect. Text tests, ref-tests and pixel tests
have a great readability, you can quickly evaluate their correctness. This
is important in my opinion, I don't think we want more opaque output like
the RenderTree dump.

I am not against your plan if the readability of the tests can be good. I'd
be happy to review patches toward testing dynamic changes in webpages.

Benjamin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-09 Thread noam . rosenthal


From: ext Benjamin Poulain benja...@webkit.orgmailto:benja...@webkit.org
This is the situation I am afraid with a solution where pixels would be 
evaluated from JavaScript. You can test, but you lack visibility why something 
is correct or incorrect. Text tests, ref-tests and pixel tests have a great 
readability, you can quickly evaluate their correctness. This is important in 
my opinion, I don't think we want more opaque output like the RenderTree dump.

I am not against your plan if the readability of the tests can be good. I'd be 
happy to review patches toward testing dynamic changes in webpages.
We're on the same page. I'm afraid of the same thing… Let's talk specifics when 
a patch is available to review.

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-09 Thread noam . rosenthal
Since people seem to agree that this is a good problem to solve, I've created a 
bug for it: https://bugs.webkit.org/show_bug.cgi?id=109356
I can't promise to fix it myself right this moment as I'm spread a bit too 
thin, but if someone wants to help or pick it up before I get to it please feel 
free :)
Noam

From: ext Benjamin Poulain benja...@webkit.orgmailto:benja...@webkit.org
Date: Saturday, February 9, 2013 10:57 AM
To: Noam Rosenthal noam.rosent...@nokia.commailto:noam.rosent...@nokia.com
Cc: rn...@webkit.orgmailto:rn...@webkit.org 
rn...@webkit.orgmailto:rn...@webkit.org, 
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org 
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org
Subject: Re: [webkit-dev] Testing feature suggestion: animation/interaction 
pixel-results on the fly

On Sat, Feb 9, 2013 at 1:24 AM, 
noam.rosent...@nokia.commailto:noam.rosent...@nokia.com wrote:
I did not say they cannot be tested with the two methods suggested :) It's more 
 about a preference to move some of those decisions from the test 
infrastructure to the test itself, but potentially those bugs could be tested 
in either way.
Examples for bugs I've encountered/reviewed and can use better in-motion 
testing (note that those are specific to Qt/EFL, but I'm sure there are tons of 
bugs like this that come up for Apple/Google as well)
http://trac.webkit.org/changeset/140825
http://trac.webkit.org/changeset/142112
http://trac.webkit.org/changeset/134953
https://bugs.webkit.org/show_bug.cgi?id=109179

Controlling the clock and programmatically sampling the end result would 
definitely make those more testable, but of course any progress in this area 
would be beneficial and my preference to a canvas-based API is more of an 
opinion.

To explain my concerns:
Sometime, I look at a failing test, and think: what the f**k is this supposed 
to test?. Then I have to dig a ton of logs, and finally read the change to 
understand a the single JS statement in the whole script that make the test 
useful.

This is the situation I am afraid with a solution where pixels would be 
evaluated from JavaScript. You can test, but you lack visibility why something 
is correct or incorrect. Text tests, ref-tests and pixel tests have a great 
readability, you can quickly evaluate their correctness. This is important in 
my opinion, I don't think we want more opaque output like the RenderTree dump.

I am not against your plan if the readability of the tests can be good. I'd be 
happy to review patches toward testing dynamic changes in webpages.

Benjamin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-09 Thread Simon Fraser
The existing pauseAnimation DRT API can stop an animation mid-flight, to allow 
for reliable snapshotting. Why isn't this enough?

Simon

On Feb 9, 2013, at 8:44 AM, noam.rosent...@nokia.com wrote:

 Since people seem to agree that this is a good problem to solve, I've created 
 a bug for it: https://bugs.webkit.org/show_bug.cgi?id=109356
 I can't promise to fix it myself right this moment as I'm spread a bit too 
 thin, but if someone wants to help or pick it up before I get to it please 
 feel free :)
 Noam
 
 From: ext Benjamin Poulain benja...@webkit.org
 Date: Saturday, February 9, 2013 10:57 AM
 To: Noam Rosenthal noam.rosent...@nokia.com
 Cc: rn...@webkit.org rn...@webkit.org, webkit-dev@lists.webkit.org 
 webkit-dev@lists.webkit.org
 Subject: Re: [webkit-dev] Testing feature suggestion: animation/interaction 
 pixel-results on the fly
 
 On Sat, Feb 9, 2013 at 1:24 AM, noam.rosent...@nokia.com wrote:
 
 I did not say they cannot be tested with the two methods suggested :) It's 
 more  about a preference to move some of those decisions from the test 
 infrastructure to the test itself, but potentially those bugs could be 
 tested in either way.
 Examples for bugs I've encountered/reviewed and can use better in-motion 
 testing (note that those are specific to Qt/EFL, but I'm sure there are 
 tons of bugs like this that come up for Apple/Google as well)
 http://trac.webkit.org/changeset/140825
 http://trac.webkit.org/changeset/142112
 http://trac.webkit.org/changeset/134953
 https://bugs.webkit.org/show_bug.cgi?id=109179
 
 Controlling the clock and programmatically sampling the end result would 
 definitely make those more testable, but of course any progress in this 
 area would be beneficial and my preference to a canvas-based API is more of 
 an opinion.
 
 To explain my concerns:
 Sometime, I look at a failing test, and think: what the f**k is this 
 supposed to test?. Then I have to dig a ton of logs, and finally read the 
 change to understand a the single JS statement in the whole script that make 
 the test useful. 
 
 This is the situation I am afraid with a solution where pixels would be 
 evaluated from JavaScript. You can test, but you lack visibility why 
 something is correct or incorrect. Text tests, ref-tests and pixel tests 
 have a great readability, you can quickly evaluate their correctness. This 
 is important in my opinion, I don't think we want more opaque output like 
 the RenderTree dump.
 
 I am not against your plan if the readability of the tests can be good. I'd 
 be happy to review patches toward testing dynamic changes in webpages.
 
 Benjamin
 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-09 Thread noam . rosenthal
Oops, forgot to reply to the list.

From: Noam Rosenthal noam.rosent...@nokia.commailto:noam.rosent...@nokia.com
From: ext Simon Fraser simon.fra...@apple.commailto:simon.fra...@apple.com
The existing pauseAnimation DRT API can stop an animation mid-flight, to allow 
for reliable snapshotting. Why isn't this enough?
I would say that the main things that are missing are:

  1.  Sample and compare several images in the same test
  2.  Sample images heuristically, e.g. sample that a green box is close 
enough to a particular location – A full ImageDiff is very rigid in that way.
  3.  Sample images mid-flight without pausing. Pausing can mess with animation 
timing and thus skew the results.

I don't have a full list of what's missing – but I know for sure that when 
trying to hunt mid-flight bugs in the past I was not able to create sufficient 
tests with the current DRT tools.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-09 Thread Dirk Pranke
On Sat, Feb 9, 2013 at 1:57 AM, Benjamin Poulain benja...@webkit.org wrote:
 On Sat, Feb 9, 2013 at 1:24 AM, noam.rosent...@nokia.com wrote:

 I did not say they cannot be tested with the two methods suggested :) It's
 more  about a preference to move some of those decisions from the test
 infrastructure to the test itself, but potentially those bugs could be
 tested in either way.
 Examples for bugs I've encountered/reviewed and can use better in-motion
 testing (note that those are specific to Qt/EFL, but I'm sure there are tons
 of bugs like this that come up for Apple/Google as well)
 http://trac.webkit.org/changeset/140825
 http://trac.webkit.org/changeset/142112
 http://trac.webkit.org/changeset/134953
 https://bugs.webkit.org/show_bug.cgi?id=109179

 Controlling the clock and programmatically sampling the end result would
 definitely make those more testable, but of course any progress in this area
 would be beneficial and my preference to a canvas-based API is more of an
 opinion.


 To explain my concerns:
 Sometime, I look at a failing test, and think: what the f**k is this
 supposed to test?. Then I have to dig a ton of logs, and finally read the
 change to understand a the single JS statement in the whole script that make
 the test useful.

 This is the situation I am afraid with a solution where pixels would be
 evaluated from JavaScript. You can test, but you lack visibility why
 something is correct or incorrect. Text tests, ref-tests and pixel tests
 have a great readability, you can quickly evaluate their correctness. This
 is important in my opinion, I don't think we want more opaque output like
 the RenderTree dump.

 I am not against your plan if the readability of the tests can be good. I'd
 be happy to review patches toward testing dynamic changes in webpages.


A counter-argument, of course, that there are a lot of pixel tests
that would be a lot better as text-only tests that were verifying
certain aspects of how the page rendered programmatically.

Perhaps -- and this is a tangent to this thread -- we could also
investigate ways of writing tests that did render pngs but told NRWT
to ignore them (e.g., dumpAsTextAndIgnoredImage() or something), or
conveyed which parts of the image were important ...

-- Dirk
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread Robert Kroeger
Questions inline.

On Fri, Feb 8, 2013 at 8:35 AM, noam.rosent...@nokia.com wrote:

 Hellos

 we don't currently have a solution in webkit's test infrastructure for
 testing animations on the fly, or in general for testing multiple image
 results in the same test.

I have been wishing for such a thing around how to test scroll animations.

 (If this was discussed before and I'm unaware of that discussion, please
 stop me here...)

 This is because ImageDiff works on the single-test level, and you can't
 explicitly make image comparisons in the same test at different points in
 time.
 This has before caused us several regressions that were not caught by
 layout tests.

 I'd like to propose a solution, and would welcome some feedback on whether
 it's a good one...
 The idea is that you would be able to programatically retrieve the current
 snapshot into a canvas ImageData, and then compare the pixel results with
 JavaScript in the LayoutTest. Something like:

 window.testRunner.getWindowSnapshot(function(snapshot) {
 if (snapshot.data[snapshot.width * 100 + 10] != 127)
 output.innerHTML = FAIL;
 });

 WDYT?

How would the test wait for the animation to happen? Tests that use
timeouts seem more likely to be flaky from what I've seen. Could
window.testRunner.getWindowSnapshot take a time delta of some kind?

Rob.

 Noam

 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread noam . rosenthal


From: webkit-dev-boun...@lists.webkit.org [webkit-dev-boun...@lists.webkit.org] 
on behalf of ext Robert Kroeger [rjkro...@chromium.org]

 How would the test wait for the animation to happen? Tests that use
 timeouts seem more likely to be flaky from what I've seen. Could
 window.testRunner.getWindowSnapshot take a time delta of some kind?
Most of the bugs I've encountered in this area can be caught by a test that 
would have obvious results that can buffer the flakiness, e.g. by moving a 
big green rectangle and testing only its center.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread Michelangelo De Simone
 I'd like to propose a solution, and would welcome some feedback on whether 
 it's a good one...
 The idea is that you would be able to programatically retrieve the current 
 snapshot into a canvas ImageData, and then compare the pixel results with 
 JavaScript in the LayoutTest. Something like:

+1
Lovely idea. For instance, we've been having some issue [1] testing 
CustomFilters, so having the ability to /sample/ the final result from JS would 
be great.

[1] https://bugs.webkit.org/show_bug.cgi?id=104012

--
Bye,
Michelangelo
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread Ryosuke Niwa
On Fri, Feb 8, 2013 at 5:35 AM, noam.rosent...@nokia.com wrote:

  we don't currently have a solution in webkit's test infrastructure for
 testing animations on the fly, or in general for testing multiple image
 results in the same test.
 (If this was discussed before and I'm unaware of that discussion, please
 stop me here...)

 This is because ImageDiff works on the single-test level, and you can't
 explicitly make image comparisons in the same test at different points in
 time.
 This has before caused us several regressions that were not caught by
 layout tests.

 I'd like to propose a solution, and would welcome some feedback on whether
 it's a good one...
 The idea is that you would be able to programatically retrieve the current
 snapshot into a canvas ImageData, and then compare the pixel results with
 JavaScript in the LayoutTest. Something like:


I had similar thoughts but my counter proposal is to let DRT/WTR generate
multiple actual results either in the form of multiple layers in PNG or
multiple PNG images. The advantage of this latter approach is that we can
have multiple reference files as well. e.g. if a green box is to be moved
from (0,0) to (50,0) and to (100, 0), we could create three reference files
that correspond to each state.

But perhaps there is a good reason you didn't choose this approach. Could
you elaborate on the reason you picked this particular API?

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread Benjamin Poulain
On Fri, Feb 8, 2013 at 11:12 AM, Ryosuke Niwa rn...@webkit.org wrote:

 I had similar thoughts but my counter proposal is to let DRT/WTR
 generate multiple actual results either in the form of multiple layers in
 PNG or multiple PNG images. The advantage of this latter approach is that
 we can have multiple reference files as well. e.g. if a green box is to be
 moved from (0,0) to (50,0) and to (100, 0), we could create three reference
 files that correspond to each state.


This sounds like a good idea for programmatic updates. I would just argue
that instead of multiple images, it would be more maintainable to have
multiple html references; a kind of multiple-frame-ref-test.


A problem with this approach is, it is still fragile for animations. The
position of a given frame is depends on timing. For animations, we could
use mask-based test. The reference file of a test is a mask which gives an
area where pixels can be.
For example:
-Take a block in a fixed position, scroll the page. The mask is the block
in its original position. If for any frame, the block moved outside the
mask, the test fails.
-A block animating from left to right, while at the same time the page is
scrolled. There is a deterministic path where the object can be. If for any
frame, the block moved outside the path, the test fails.

Making the mask would not always be trivial. The advantage is, if any
single frame is wrong, we have a chance to catch it, regardless of the
timing.

Benjamin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread Dirk Pranke
On Fri, Feb 8, 2013 at 11:12 AM, Ryosuke Niwa rn...@webkit.org wrote:
 On Fri, Feb 8, 2013 at 5:35 AM, noam.rosent...@nokia.com wrote:

 we don't currently have a solution in webkit's test infrastructure for
 testing animations on the fly, or in general for testing multiple image
 results in the same test.
 (If this was discussed before and I'm unaware of that discussion, please
 stop me here...)

 This is because ImageDiff works on the single-test level, and you can't
 explicitly make image comparisons in the same test at different points in
 time.
 This has before caused us several regressions that were not caught by
 layout tests.

 I'd like to propose a solution, and would welcome some feedback on whether
 it's a good one...
 The idea is that you would be able to programatically retrieve the current
 snapshot into a canvas ImageData, and then compare the pixel results with
 JavaScript in the LayoutTest. Something like:


 I had similar thoughts but my counter proposal is to let DRT/WTR generate
 multiple actual results either in the form of multiple layers in PNG or
 multiple PNG images. The advantage of this latter approach is that we can
 have multiple reference files as well. e.g. if a green box is to be moved
 from (0,0) to (50,0) and to (100, 0), we could create three reference files
 that correspond to each state.

 But perhaps there is a good reason you didn't choose this approach. Could
 you elaborate on the reason you picked this particular API?


I also appreciate the problem -- and would be interesting in solving
it -- but I'm a bit concerned that either proposal is at least
somewhat nondeterministic (or flaky), and I'm not sure how you'd be
able to control which frames of the animation you were getting well
enough. But I'm not much of expert here so I'm not sure how often it
matters.

-- Dirk
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread noam . rosenthal


From: ext Ryosuke Niwa rn...@webkit.orgmailto:rn...@webkit.org
I'd like to propose a solution, and would welcome some feedback on whether it's 
a good one...
The idea is that you would be able to programatically retrieve the current 
snapshot into a canvas ImageData, and then compare the pixel results with 
JavaScript in the LayoutTest. Something like:

I had similar thoughts but my counter proposal is to let DRT/WTR generate 
multiple actual results either in the form of multiple layers in PNG or 
multiple PNG images. The advantage of this latter approach is that we can have 
multiple reference files as well. e.g. if a green box is to be moved from (0,0) 
to (50,0) and to (100, 0), we could create three reference files that 
correspond to each state.

But perhaps there is a good reason you didn't choose this approach. Could you 
elaborate on the reason you picked this particular API?
The problem with dynamic features of the web like animations/interactions is 
that they're non-deterministic, or at least a lot less deterministic than 
static features of the web like layouts.
Ref tests, pixel tests etc. are tools built for deterministic testing: load a 
file, take a snapshot, compare against a result. Testing an animation (or a 
filter) needs to feel a lot more dynamic and expressive: Animate green boxes, 
make sure that they're within a particular range at particular points in time.
I think that when testing some of the dynamic features it's best to create a 
minimal mechanism such as retrieving snapshot pixels, giving more power to the 
tests themselves in a dynamic language like JavaScript, rather than try to 
force the tests to fit into a predefined set of features supported by the 
DRT/WTR infrastructure. Also this would allow some differences between the 
ports, which implement animations a bit differently, without constant 
rebaselines of reference results – e.g. A heuristic animation timing test may 
stay valid across platform even if the actual animation implementation is 
different.

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread Benjamin Poulain
On Fri, Feb 8, 2013 at 3:16 PM, noam.rosent...@nokia.com wrote:

  The problem with dynamic features of the web like
 animations/interactions is that they're non-deterministic, or at least a
 lot less deterministic than static features of the web like layouts.
 Ref tests, pixel tests etc. are tools built for deterministic testing:
 load a file, take a snapshot, compare against a result. Testing an
 animation (or a filter) needs to feel a lot more dynamic and expressive:
 Animate green boxes, make sure that they're within a particular range at
 particular points in time.


The tests also have to be deterministic and comprehensive. I am afraid of
loosing both with the Render-to-Canvas approach.

Can you give concrete examples of the kind of bugs you are hunting, and why
testing cannot use the two methods suggested?


 I think that when testing some of the dynamic features it's best to create
 a minimal mechanism such as retrieving snapshot pixels, giving more power
 to the tests themselves in a dynamic language like JavaScript, rather than
 try to force the tests to fit into a predefined set of features supported
 by the DRT/WTR infrastructure. Also this would allow some differences
 between the ports, which implement animations a bit differently, without
 constant rebaselines of reference results – e.g. A heuristic animation
 timing test may stay valid across platform even if the actual animation
 implementation is different.


Again, can you give concrete examples of this?

Benjamin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread Gregg Tavares
Can you expose a time or setTime function in DRT and some option that says
let JS control the clock?

Then a test could do something like

layoutTestController.overridePreference(JsControlledClock, 1);

   // Render 5 frames over 1 second
   var frame =0;
   function renderFrame() {
  layoutTestController.getWindowSnapshot(...);
  if (frame == 5) {
 finishTest();
 return;
  }
  ++frame;
  layoutTestController.setAnimationClock(frame * 1000 / 5);  // assumes
clock is in ms.
  requestAnimationFrame(renderFrame);
  }



On Fri, Feb 8, 2013 at 3:52 PM, Benjamin Poulain benja...@webkit.orgwrote:

 On Fri, Feb 8, 2013 at 3:16 PM, noam.rosent...@nokia.com wrote:

  The problem with dynamic features of the web like
 animations/interactions is that they're non-deterministic, or at least a
 lot less deterministic than static features of the web like layouts.
 Ref tests, pixel tests etc. are tools built for deterministic testing:
 load a file, take a snapshot, compare against a result. Testing an
 animation (or a filter) needs to feel a lot more dynamic and expressive:
 Animate green boxes, make sure that they're within a particular range at
 particular points in time.


 The tests also have to be deterministic and comprehensive. I am afraid of
 loosing both with the Render-to-Canvas approach.

 Can you give concrete examples of the kind of bugs you are hunting, and
 why testing cannot use the two methods suggested?


  I think that when testing some of the dynamic features it's best to
 create a minimal mechanism such as retrieving snapshot pixels, giving more
 power to the tests themselves in a dynamic language like JavaScript, rather
 than try to force the tests to fit into a predefined set of features
 supported by the DRT/WTR infrastructure. Also this would allow some
 differences between the ports, which implement animations a bit
 differently, without constant rebaselines of reference results – e.g. A
 heuristic animation timing test may stay valid across platform even if the
 actual animation implementation is different.


 Again, can you give concrete examples of this?

 Benjamin

 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 https://lists.webkit.org/mailman/listinfo/webkit-dev


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread Benjamin Poulain
On Fri, Feb 8, 2013 at 4:12 PM, Gregg Tavares g...@google.com wrote:

 Can you expose a time or setTime function in DRT and some option that says
 let JS control the clock?


Unfortunately, that only works in the simple cases.
We could cheat the WebCore clock, but that would ultimately be wrong for
animations running in threads or driven outside WebCore.

Benjamin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread Gregg Tavares
On Fri, Feb 8, 2013 at 4:16 PM, Benjamin Poulain benja...@webkit.orgwrote:

 On Fri, Feb 8, 2013 at 4:12 PM, Gregg Tavares g...@google.com wrote:

 Can you expose a time or setTime function in DRT and some option that
 says let JS control the clock?


 Unfortunately, that only works in the simple cases.
 We could cheat the WebCore clock, but that would ultimately be wrong for
 animations running in threads or driven outside WebCore.


The point was you need to be able to control all the clocks if you want
repeatable results. If you're saying some clocks can't be controlled that
sounds like something needs to be refactored so it's testable.



 Benjamin

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread Philip Rogers
I'm not sure if this is generally known: we use [svg
document].setCurrentTime(time) in the svg/animations tests for both script
and pixel tests. The SVG animation test harness leaves a lot to be desired,
but it may be a good place to start for a more general DRT approach:
http://trac.webkit.org/browser/trunk/LayoutTests/svg/animations/script-tests/animate-css-xml-attributeType.js#L62

Philip


On Fri, Feb 8, 2013 at 4:26 PM, Gregg Tavares g...@google.com wrote:




 On Fri, Feb 8, 2013 at 4:16 PM, Benjamin Poulain benja...@webkit.orgwrote:

 On Fri, Feb 8, 2013 at 4:12 PM, Gregg Tavares g...@google.com wrote:

 Can you expose a time or setTime function in DRT and some option that
 says let JS control the clock?


 Unfortunately, that only works in the simple cases.
 We could cheat the WebCore clock, but that would ultimately be wrong for
 animations running in threads or driven outside WebCore.


 The point was you need to be able to control all the clocks if you want
 repeatable results. If you're saying some clocks can't be controlled that
 sounds like something needs to be refactored so it's testable.



 Benjamin



 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 https://lists.webkit.org/mailman/listinfo/webkit-dev


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev