Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
On Wed, Feb 13, 2013 at 11:33 PM, Benjamin Poulain benja...@webkit.org wrote:
On Wed, Feb 13, 2013 at 11:16 PM, Dirk Pranke dpra...@chromium.org wrote:
Those changes are not harmless. There are people monitoring tests
results
full time in order to keep WebKit in good shape. No other part of WebKit
require continuous attention.
I'm sorry, but either I don't understand Dongsung's suggestion, or I
don't understand your criticism. What does this sentence paragraph
mean? Are you suggesting someone needs to look at this type of test
full time in a way that we don't have to look at the other tests?
My lack of sleep is likely to blame here :)
All I am saying is tests have a hight cost when they start failing, we
should be careful how we design them so we can easily triage bad tests,
flaky tests, and real errors.
No argument here :).
Case 1: CSS Filters Shaders
I wanted this test functionality when I commented
http://webkit.org/b/97859#c19
If I want to make gaussian blur test, I prefer using 'getPixel' test as
follows,
Why wasn't a ref-test a better solution in this particular case?
I'm curious, what would you imagine the ref test contains?
If I am not mistaken, the composition operations are parallel with the ones
of SVG and Canvas (aren't they?).
I would have attempted comparing the 3 implementations as it seems to me the
pixels values should be the same.
That was a genuine question about that case :)
Case 2: Fixed Position Element
[...]
function repeatedlyCalledDuringScrolling() {
ASSERT(getPixel(15, 9) == white);
ASSERT(getPixel(15, 10) == green);
ASSERT(getPixel(9, 15) == white);
ASSERT(getPixel(10, 15) == green);
}
I think this shows what I said about correctness and readability:
-Asserting the correctness of the test and the result becomes close to
impossible for the reader. One has to review the full code to have a
chance
of understanding an error.
-You cannot cover non trivial cases (images, text, form elements, etc).
-And it is inefficient. You have to render each frame on the UIProcess,
move
it to the WebProcess, and box it for JavaScript to process (with pixel
format conversions depending on your graphics system)
Of the ideas raised, I think this is one of my least favorite for
testing
fixed positioning.
Isn't his suggestion the equivalent of what we do today in text-only
tests? i.e., printing pass or fail and making you have to look at
the test itself to see what's being tested?
If the correctness of the rendering depends on those 4 specific pixels
having those four specific values, how exactly are you going to verify
that by looking at it?
Again, I think I'm just not understanding you here?
When looking at a test test, you follow the flow to know what it is supposed
to do and where it breaks.
How are you supposed to know, _by reading the code_, that the color at
position 15, 9 should be white?
is the code the test code, or the source code we're testing? Of
course, without any larger context, those assertions are impossible to
interpret. I figured he was just indicating what he meant by a
programmatic check, and a real test case would be more informative.
And of course it's easy to write bad tests that are neither readable
nor maintainable. But we also have lots of pixel tests that are
difficult for people not intimately familiar with the code being
tested to evaluate for correctness, too.
Lastly, I don't think he was suggesting that this would replace all
pixel tests; as you say, it is more well suited for some times of
tests than others.
As far as efficiency goes, even moving the picture back and forth
between two processes is still going to be far faster than running a
second reftest and comparing the output of the two pages.
-- Dirk
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
I'm curious, what would you imagine the ref test contains?
If I am not mistaken, the composition operations are parallel with the
ones of SVG and Canvas (aren't they?).
I would have attempted comparing the 3 implementations as it seems to me
the pixels values should be the same.
That was a genuine question about that case :)
Thank you for feedback! :)
I presented gaussian blur case to express my opinion effectively.
As I know, Adobe want to draw CSS Filters and SVG Filters using the same
code. And Canvas does not have something like Filters.
What I want to say is that there are some cases in which snapshot
functionality is more useful.
Case 2: Fixed Position Element
[...]
function repeatedlyCalledDuringScrolling() {
ASSERT(getPixel(15, 9) == white);
ASSERT(getPixel(15, 10) == green);
ASSERT(getPixel(9, 15) == white);
ASSERT(getPixel(10, 15) == green);
}
I think this shows what I said about correctness and readability:
-Asserting the correctness of the test and the result becomes close to
impossible for the reader. One has to review the full code to have a
chance
of understanding an error.
-You cannot cover non trivial cases (images, text, form elements, etc).
-And it is inefficient. You have to render each frame on the UIProcess,
move
it to the WebProcess, and box it for JavaScript to process (with pixel
format conversions depending on your graphics system)
Of the ideas raised, I think this is one of my least favorite for
testing
fixed positioning.
Isn't his suggestion the equivalent of what we do today in text-only
tests? i.e., printing pass or fail and making you have to look at
the test itself to see what's being tested?
If the correctness of the rendering depends on those 4 specific pixels
having those four specific values, how exactly are you going to verify
that by looking at it?
Again, I think I'm just not understanding you here?
When looking at a test test, you follow the flow to know what it is
supposed to do and where it breaks.
How are you supposed to know, _by reading the code_, that the color at
position 15, 9 should be white?
Benjamin
I agree on Dirk's opinion. I supports 3 points as follows:
1. we can help to understand what's goal of test by comments
2. when we want to see the graphical result, we can use MiniBrowser like
other text based test.
3. philip's canvas tests (http://philip.html5.org/tests/canvas/suite/tests/)
already use well getImageData to test.
I agree that snapshot test can make hardly maintainable tests, but we have
review process and we can make succinct tests like philip canvas test.
And when pixel test is more suitable, we can use pixel test also.
Best regards,
Dongsung Huang
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
I like this idea. I cannot find any harm if we have this functionality.
When a pixel test is more succinct, we can make a pixel test. When a
'getPixel on Javascript' test is more succinct, we can make a 'getPixel'
test.
As
http://trac.webkit.org/wiki/Writing%20Layout%20Tests%20for%20DumpRenderTree
said
'Pixel tests are a burden on every ports as any small change in the engine
could lead to your test needing a new pixel result.', new test
functionality that can be an alternative of pixel test is good!
philip canvas tests (LayoutTests/canvas/philip) has no pixel test because
of the power of getImageData API in canvas spec.
I can put two examples for the power of functionality that Noam suggested,
Case 1: CSS Filters Shaders
I wanted this test functionality when I commented
http://webkit.org/b/97859#c19
If I want to make gaussian blur test, I prefer using 'getPixel' test as
follows,
ASSERT(getPixel(10, 10) = rgba(255, 0, 0, 255));
ASSERT(getPixel(10, 15) rgba(255, 0, 0, 98) getPixel(10, 15)
rgba(255, 0, 0, 102));
If we don't have 'getPixel', we need to capture all port's image results
because all port's painting engine would make a bit different result. As
you know, it is painful.
Case 2: Fixed Position Element
I guess it is one of reasons why Noam suggested.
Currently, Qt and EFL AC (a.k.a Coordinated Graphics) has a problem related
to fixed position element.
During panning animation, Fixed position element is slightly flickering.
If we have 'getPixel', we can make a succinct test case.
PRECONDITION : a green fixed position div is on the white body. The rect of
the div is rect(10, 10, 30, 30).
function repeatedlyCalledDuringScrolling() {
ASSERT(getPixel(15, 9) == white);
ASSERT(getPixel(15, 10) == green);
ASSERT(getPixel(9, 15) == white);
ASSERT(getPixel(10, 15) == green);
}
WDYT?
Best regards,
Dongsung Huang
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
On Wed, Feb 13, 2013 at 9:38 PM, Dongsung Huang luxte...@company100.netwrote:
I like this idea. I cannot find any harm if we have this functionality.
Those changes are not harmless. There are people monitoring tests results
full time in order to keep WebKit in good shape. No other part of WebKit
require continuous attention.
Case 1: CSS Filters Shaders
I wanted this test functionality when I commented
http://webkit.org/b/97859#c19
If I want to make gaussian blur test, I prefer using 'getPixel' test as
follows,
Why wasn't a ref-test a better solution in this particular case?
Case 2: Fixed Position Element
[...]
function repeatedlyCalledDuringScrolling() {
ASSERT(getPixel(15, 9) == white);
ASSERT(getPixel(15, 10) == green);
ASSERT(getPixel(9, 15) == white);
ASSERT(getPixel(10, 15) == green);
}
I think this shows what I said about correctness and readability:
-Asserting the correctness of the test and the result becomes close to
impossible for the reader. One has to review the full code to have a chance
of understanding an error.
-You cannot cover non trivial cases (images, text, form elements, etc).
-And it is inefficient. You have to render each frame on the UIProcess,
move it to the WebProcess, and box it for JavaScript to process (with pixel
format conversions depending on your graphics system)
Of the ideas raised, I think this is one of my least favorite for testing
fixed positioning.
Benjamin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
From: ext Benjamin Poulain benja...@webkit.orgmailto:benja...@webkit.org
Date: Thursday, February 14, 2013 8:07 AM
To: Dongsung Huang luxte...@company100.netmailto:luxte...@company100.net
Cc: webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org
Subject: Re: [webkit-dev] Testing feature suggestion: animation/interaction
pixel-results on the fly
On Wed, Feb 13, 2013 at 9:38 PM, Dongsung Huang
luxte...@company100.netmailto:luxte...@company100.net wrote:
I like this idea. I cannot find any harm if we have this functionality.
Those changes are not harmless. There are people monitoring tests results full
time in order to keep WebKit in good shape. No other part of WebKit require
continuous attention.
Case 1: CSS Filters Shaders
I wanted this test functionality when I commented http://webkit.org/b/97859#c19
If I want to make gaussian blur test, I prefer using 'getPixel' test as follows,
Why wasn't a ref-test a better solution in this particular case?
Because gaussian blurs on the GPU are not accurate, and look slightly different
on different GPUs, but usually close enough.
We need a way to measure close enough for features where all you can get is
close enough! Ref-tests and pixel-tests are way to rigid for that, and
require constant rebaselines and headaches.
Case 2: Fixed Position Element
[...]
function repeatedlyCalledDuringScrolling() {
ASSERT(getPixel(15, 9) == white);
ASSERT(getPixel(15, 10) == green);
ASSERT(getPixel(9, 15) == white);
ASSERT(getPixel(10, 15) == green);
}
I think this shows what I said about correctness and readability:
-Asserting the correctness of the test and the result becomes close to
impossible for the reader. One has to review the full code to have a chance of
understanding an error.
We can generate PNGs when a snapshot is requested so that the tester can
eyeball the results.
-You cannot cover non trivial cases (images, text, form elements, etc).
-And it is inefficient. You have to render each frame on the UIProcess, move it
to the WebProcess, and box it for JavaScript to process (with pixel format
conversions depending on your graphics system)
Another option is to do expose a partial comparison mechanism that runs in the
UI process.
E.g.
var snapshotHandle = testRunner.createSnapshot();
testRunner.comparePixel(snapshotHandle, 15, 20, 'white', { tolerance: 0.2 });
This way we can generate a visual representation of failures.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
On Wed, Feb 13, 2013 at 11:16 PM, Dirk Pranke dpra...@chromium.org wrote:
Those changes are not harmless. There are people monitoring tests results
full time in order to keep WebKit in good shape. No other part of WebKit
require continuous attention.
I'm sorry, but either I don't understand Dongsung's suggestion, or I
don't understand your criticism. What does this sentence paragraph
mean? Are you suggesting someone needs to look at this type of test
full time in a way that we don't have to look at the other tests?
My lack of sleep is likely to blame here :)
All I am saying is tests have a hight cost when they start failing, we
should be careful how we design them so we can easily triage bad tests,
flaky tests, and real errors.
Case 1: CSS Filters Shaders
I wanted this test functionality when I commented
http://webkit.org/b/97859#c19
If I want to make gaussian blur test, I prefer using 'getPixel' test as
follows,
Why wasn't a ref-test a better solution in this particular case?
I'm curious, what would you imagine the ref test contains?
If I am not mistaken, the composition operations are parallel with the ones
of SVG and Canvas (aren't they?).
I would have attempted comparing the 3 implementations as it seems to me
the pixels values should be the same.
That was a genuine question about that case :)
Case 2: Fixed Position Element
[...]
function repeatedlyCalledDuringScrolling() {
ASSERT(getPixel(15, 9) == white);
ASSERT(getPixel(15, 10) == green);
ASSERT(getPixel(9, 15) == white);
ASSERT(getPixel(10, 15) == green);
}
I think this shows what I said about correctness and readability:
-Asserting the correctness of the test and the result becomes close to
impossible for the reader. One has to review the full code to have a
chance
of understanding an error.
-You cannot cover non trivial cases (images, text, form elements, etc).
-And it is inefficient. You have to render each frame on the UIProcess,
move
it to the WebProcess, and box it for JavaScript to process (with pixel
format conversions depending on your graphics system)
Of the ideas raised, I think this is one of my least favorite for testing
fixed positioning.
Isn't his suggestion the equivalent of what we do today in text-only
tests? i.e., printing pass or fail and making you have to look at
the test itself to see what's being tested?
If the correctness of the rendering depends on those 4 specific pixels
having those four specific values, how exactly are you going to verify
that by looking at it?
Again, I think I'm just not understanding you here?
When looking at a test test, you follow the flow to know what it is
supposed to do and where it breaks.
How are you supposed to know, _by reading the code_, that the color at
position 15, 9 should be white?
Benjamin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
On Wed, Feb 13, 2013 at 11:27 PM, noam.rosent...@nokia.com wrote: Why wasn't a ref-test a better solution in this particular case? Because gaussian blurs on the GPU are not accurate, and look slightly different on different GPUs, but usually close enough. We need a way to measure close enough for features where all you can get is close enough! Ref-tests and pixel-tests are way to rigid for that, and require constant rebaselines and headaches. There is _NO_ gaussian blur in that patch. Benjamin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
From: ext Benjamin Poulain benja...@webkit.orgmailto:benja...@webkit.org Date: Thursday, February 14, 2013 8:36 AM To: Noam Rosenthal noam.rosent...@nokia.commailto:noam.rosent...@nokia.com Cc: luxte...@company100.netmailto:luxte...@company100.net luxte...@company100.netmailto:luxte...@company100.net, webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org Subject: Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly On Wed, Feb 13, 2013 at 11:27 PM, noam.rosent...@nokia.commailto:noam.rosent...@nokia.com wrote: Why wasn't a ref-test a better solution in this particular case? Because gaussian blurs on the GPU are not accurate, and look slightly different on different GPUs, but usually close enough. We need a way to measure close enough for features where all you can get is close enough! Ref-tests and pixel-tests are way to rigid for that, and require constant rebaselines and headaches. There is _NO_ gaussian blur in that patch. DongSung mentioned gaussian blur in the comment, which is what I was referring to. But other bugs as well, like delegated scrolling, also have that problem of results not being 100% the same every time. ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
On Sat, Feb 9, 2013 at 5:35 PM, Dirk Pranke dpra...@chromium.org wrote: Perhaps -- and this is a tangent to this thread -- we could also investigate ways of writing tests that did render pngs but told NRWT to ignore them (e.g., dumpAsTextAndIgnoredImage() or something), or conveyed which parts of the image were important ... What's the point of that? - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
From: ext Benjamin Poulain benja...@webkit.orgmailto:benja...@webkit.org Date: Saturday, February 9, 2013 12:52 AM To: Noam Rosenthal noam.rosent...@nokia.commailto:noam.rosent...@nokia.com Cc: rn...@webkit.orgmailto:rn...@webkit.org rn...@webkit.orgmailto:rn...@webkit.org, webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org Subject: Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly On Fri, Feb 8, 2013 at 3:16 PM, noam.rosent...@nokia.commailto:noam.rosent...@nokia.com wrote: The problem with dynamic features of the web like animations/interactions is that they're non-deterministic, or at least a lot less deterministic than static features of the web like layouts. Ref tests, pixel tests etc. are tools built for deterministic testing: load a file, take a snapshot, compare against a result. Testing an animation (or a filter) needs to feel a lot more dynamic and expressive: Animate green boxes, make sure that they're within a particular range at particular points in time. The tests also have to be deterministic and comprehensive. I am afraid of loosing both with the Render-to-Canvas approach. Can you give concrete examples of the kind of bugs you are hunting, and why testing cannot use the two methods suggested? I did not say they cannot be tested with the two methods suggested :) It's more about a preference to move some of those decisions from the test infrastructure to the test itself, but potentially those bugs could be tested in either way. Examples for bugs I've encountered/reviewed and can use better in-motion testing (note that those are specific to Qt/EFL, but I'm sure there are tons of bugs like this that come up for Apple/Google as well) http://trac.webkit.org/changeset/140825 http://trac.webkit.org/changeset/142112 http://trac.webkit.org/changeset/134953 https://bugs.webkit.org/show_bug.cgi?id=109179 Controlling the clock and programmatically sampling the end result would definitely make those more testable, but of course any progress in this area would be beneficial and my preference to a canvas-based API is more of an opinion. ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
On Sat, Feb 9, 2013 at 1:24 AM, noam.rosent...@nokia.com wrote: I did not say they cannot be tested with the two methods suggested :) It's more about a preference to move some of those decisions from the test infrastructure to the test itself, but potentially those bugs could be tested in either way. Examples for bugs I've encountered/reviewed and can use better in-motion testing (note that those are specific to Qt/EFL, but I'm sure there are tons of bugs like this that come up for Apple/Google as well) http://trac.webkit.org/changeset/140825 http://trac.webkit.org/changeset/142112 http://trac.webkit.org/changeset/134953 https://bugs.webkit.org/show_bug.cgi?id=109179 Controlling the clock and programmatically sampling the end result would definitely make those more testable, but of course any progress in this area would be beneficial and my preference to a canvas-based API is more of an opinion. To explain my concerns: Sometime, I look at a failing test, and think: what the f**k is this supposed to test?. Then I have to dig a ton of logs, and finally read the change to understand a the single JS statement in the whole script that make the test useful. This is the situation I am afraid with a solution where pixels would be evaluated from JavaScript. You can test, but you lack visibility why something is correct or incorrect. Text tests, ref-tests and pixel tests have a great readability, you can quickly evaluate their correctness. This is important in my opinion, I don't think we want more opaque output like the RenderTree dump. I am not against your plan if the readability of the tests can be good. I'd be happy to review patches toward testing dynamic changes in webpages. Benjamin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
From: ext Benjamin Poulain benja...@webkit.orgmailto:benja...@webkit.org This is the situation I am afraid with a solution where pixels would be evaluated from JavaScript. You can test, but you lack visibility why something is correct or incorrect. Text tests, ref-tests and pixel tests have a great readability, you can quickly evaluate their correctness. This is important in my opinion, I don't think we want more opaque output like the RenderTree dump. I am not against your plan if the readability of the tests can be good. I'd be happy to review patches toward testing dynamic changes in webpages. We're on the same page. I'm afraid of the same thing… Let's talk specifics when a patch is available to review. ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
Since people seem to agree that this is a good problem to solve, I've created a bug for it: https://bugs.webkit.org/show_bug.cgi?id=109356 I can't promise to fix it myself right this moment as I'm spread a bit too thin, but if someone wants to help or pick it up before I get to it please feel free :) Noam From: ext Benjamin Poulain benja...@webkit.orgmailto:benja...@webkit.org Date: Saturday, February 9, 2013 10:57 AM To: Noam Rosenthal noam.rosent...@nokia.commailto:noam.rosent...@nokia.com Cc: rn...@webkit.orgmailto:rn...@webkit.org rn...@webkit.orgmailto:rn...@webkit.org, webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org Subject: Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly On Sat, Feb 9, 2013 at 1:24 AM, noam.rosent...@nokia.commailto:noam.rosent...@nokia.com wrote: I did not say they cannot be tested with the two methods suggested :) It's more about a preference to move some of those decisions from the test infrastructure to the test itself, but potentially those bugs could be tested in either way. Examples for bugs I've encountered/reviewed and can use better in-motion testing (note that those are specific to Qt/EFL, but I'm sure there are tons of bugs like this that come up for Apple/Google as well) http://trac.webkit.org/changeset/140825 http://trac.webkit.org/changeset/142112 http://trac.webkit.org/changeset/134953 https://bugs.webkit.org/show_bug.cgi?id=109179 Controlling the clock and programmatically sampling the end result would definitely make those more testable, but of course any progress in this area would be beneficial and my preference to a canvas-based API is more of an opinion. To explain my concerns: Sometime, I look at a failing test, and think: what the f**k is this supposed to test?. Then I have to dig a ton of logs, and finally read the change to understand a the single JS statement in the whole script that make the test useful. This is the situation I am afraid with a solution where pixels would be evaluated from JavaScript. You can test, but you lack visibility why something is correct or incorrect. Text tests, ref-tests and pixel tests have a great readability, you can quickly evaluate their correctness. This is important in my opinion, I don't think we want more opaque output like the RenderTree dump. I am not against your plan if the readability of the tests can be good. I'd be happy to review patches toward testing dynamic changes in webpages. Benjamin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
The existing pauseAnimation DRT API can stop an animation mid-flight, to allow for reliable snapshotting. Why isn't this enough? Simon On Feb 9, 2013, at 8:44 AM, noam.rosent...@nokia.com wrote: Since people seem to agree that this is a good problem to solve, I've created a bug for it: https://bugs.webkit.org/show_bug.cgi?id=109356 I can't promise to fix it myself right this moment as I'm spread a bit too thin, but if someone wants to help or pick it up before I get to it please feel free :) Noam From: ext Benjamin Poulain benja...@webkit.org Date: Saturday, February 9, 2013 10:57 AM To: Noam Rosenthal noam.rosent...@nokia.com Cc: rn...@webkit.org rn...@webkit.org, webkit-dev@lists.webkit.org webkit-dev@lists.webkit.org Subject: Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly On Sat, Feb 9, 2013 at 1:24 AM, noam.rosent...@nokia.com wrote: I did not say they cannot be tested with the two methods suggested :) It's more about a preference to move some of those decisions from the test infrastructure to the test itself, but potentially those bugs could be tested in either way. Examples for bugs I've encountered/reviewed and can use better in-motion testing (note that those are specific to Qt/EFL, but I'm sure there are tons of bugs like this that come up for Apple/Google as well) http://trac.webkit.org/changeset/140825 http://trac.webkit.org/changeset/142112 http://trac.webkit.org/changeset/134953 https://bugs.webkit.org/show_bug.cgi?id=109179 Controlling the clock and programmatically sampling the end result would definitely make those more testable, but of course any progress in this area would be beneficial and my preference to a canvas-based API is more of an opinion. To explain my concerns: Sometime, I look at a failing test, and think: what the f**k is this supposed to test?. Then I have to dig a ton of logs, and finally read the change to understand a the single JS statement in the whole script that make the test useful. This is the situation I am afraid with a solution where pixels would be evaluated from JavaScript. You can test, but you lack visibility why something is correct or incorrect. Text tests, ref-tests and pixel tests have a great readability, you can quickly evaluate their correctness. This is important in my opinion, I don't think we want more opaque output like the RenderTree dump. I am not against your plan if the readability of the tests can be good. I'd be happy to review patches toward testing dynamic changes in webpages. Benjamin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
Oops, forgot to reply to the list. From: Noam Rosenthal noam.rosent...@nokia.commailto:noam.rosent...@nokia.com From: ext Simon Fraser simon.fra...@apple.commailto:simon.fra...@apple.com The existing pauseAnimation DRT API can stop an animation mid-flight, to allow for reliable snapshotting. Why isn't this enough? I would say that the main things that are missing are: 1. Sample and compare several images in the same test 2. Sample images heuristically, e.g. sample that a green box is close enough to a particular location – A full ImageDiff is very rigid in that way. 3. Sample images mid-flight without pausing. Pausing can mess with animation timing and thus skew the results. I don't have a full list of what's missing – but I know for sure that when trying to hunt mid-flight bugs in the past I was not able to create sufficient tests with the current DRT tools. ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
On Sat, Feb 9, 2013 at 1:57 AM, Benjamin Poulain benja...@webkit.org wrote: On Sat, Feb 9, 2013 at 1:24 AM, noam.rosent...@nokia.com wrote: I did not say they cannot be tested with the two methods suggested :) It's more about a preference to move some of those decisions from the test infrastructure to the test itself, but potentially those bugs could be tested in either way. Examples for bugs I've encountered/reviewed and can use better in-motion testing (note that those are specific to Qt/EFL, but I'm sure there are tons of bugs like this that come up for Apple/Google as well) http://trac.webkit.org/changeset/140825 http://trac.webkit.org/changeset/142112 http://trac.webkit.org/changeset/134953 https://bugs.webkit.org/show_bug.cgi?id=109179 Controlling the clock and programmatically sampling the end result would definitely make those more testable, but of course any progress in this area would be beneficial and my preference to a canvas-based API is more of an opinion. To explain my concerns: Sometime, I look at a failing test, and think: what the f**k is this supposed to test?. Then I have to dig a ton of logs, and finally read the change to understand a the single JS statement in the whole script that make the test useful. This is the situation I am afraid with a solution where pixels would be evaluated from JavaScript. You can test, but you lack visibility why something is correct or incorrect. Text tests, ref-tests and pixel tests have a great readability, you can quickly evaluate their correctness. This is important in my opinion, I don't think we want more opaque output like the RenderTree dump. I am not against your plan if the readability of the tests can be good. I'd be happy to review patches toward testing dynamic changes in webpages. A counter-argument, of course, that there are a lot of pixel tests that would be a lot better as text-only tests that were verifying certain aspects of how the page rendered programmatically. Perhaps -- and this is a tangent to this thread -- we could also investigate ways of writing tests that did render pngs but told NRWT to ignore them (e.g., dumpAsTextAndIgnoredImage() or something), or conveyed which parts of the image were important ... -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
Hellos
we don't currently have a solution in webkit's test infrastructure for testing
animations on the fly, or in general for testing multiple image results in
the same test.
(If this was discussed before and I'm unaware of that discussion, please stop
me here...)
This is because ImageDiff works on the single-test level, and you can't
explicitly make image comparisons in the same test at different points in time.
This has before caused us several regressions that were not caught by layout
tests.
I'd like to propose a solution, and would welcome some feedback on whether it's
a good one...
The idea is that you would be able to programatically retrieve the current
snapshot into a canvas ImageData, and then compare the pixel results with
JavaScript in the LayoutTest. Something like:
window.testRunner.getWindowSnapshot(function(snapshot) {
if (snapshot.data[snapshot.width * 100 + 10] != 127)
output.innerHTML = FAIL;
});
WDYT?
Noam
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
Questions inline.
On Fri, Feb 8, 2013 at 8:35 AM, noam.rosent...@nokia.com wrote:
Hellos
we don't currently have a solution in webkit's test infrastructure for
testing animations on the fly, or in general for testing multiple image
results in the same test.
I have been wishing for such a thing around how to test scroll animations.
(If this was discussed before and I'm unaware of that discussion, please
stop me here...)
This is because ImageDiff works on the single-test level, and you can't
explicitly make image comparisons in the same test at different points in
time.
This has before caused us several regressions that were not caught by
layout tests.
I'd like to propose a solution, and would welcome some feedback on whether
it's a good one...
The idea is that you would be able to programatically retrieve the current
snapshot into a canvas ImageData, and then compare the pixel results with
JavaScript in the LayoutTest. Something like:
window.testRunner.getWindowSnapshot(function(snapshot) {
if (snapshot.data[snapshot.width * 100 + 10] != 127)
output.innerHTML = FAIL;
});
WDYT?
How would the test wait for the animation to happen? Tests that use
timeouts seem more likely to be flaky from what I've seen. Could
window.testRunner.getWindowSnapshot take a time delta of some kind?
Rob.
Noam
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
From: webkit-dev-boun...@lists.webkit.org [webkit-dev-boun...@lists.webkit.org] on behalf of ext Robert Kroeger [rjkro...@chromium.org] How would the test wait for the animation to happen? Tests that use timeouts seem more likely to be flaky from what I've seen. Could window.testRunner.getWindowSnapshot take a time delta of some kind? Most of the bugs I've encountered in this area can be caught by a test that would have obvious results that can buffer the flakiness, e.g. by moving a big green rectangle and testing only its center. ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
I'd like to propose a solution, and would welcome some feedback on whether it's a good one... The idea is that you would be able to programatically retrieve the current snapshot into a canvas ImageData, and then compare the pixel results with JavaScript in the LayoutTest. Something like: +1 Lovely idea. For instance, we've been having some issue [1] testing CustomFilters, so having the ability to /sample/ the final result from JS would be great. [1] https://bugs.webkit.org/show_bug.cgi?id=104012 -- Bye, Michelangelo ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
On Fri, Feb 8, 2013 at 5:35 AM, noam.rosent...@nokia.com wrote: we don't currently have a solution in webkit's test infrastructure for testing animations on the fly, or in general for testing multiple image results in the same test. (If this was discussed before and I'm unaware of that discussion, please stop me here...) This is because ImageDiff works on the single-test level, and you can't explicitly make image comparisons in the same test at different points in time. This has before caused us several regressions that were not caught by layout tests. I'd like to propose a solution, and would welcome some feedback on whether it's a good one... The idea is that you would be able to programatically retrieve the current snapshot into a canvas ImageData, and then compare the pixel results with JavaScript in the LayoutTest. Something like: I had similar thoughts but my counter proposal is to let DRT/WTR generate multiple actual results either in the form of multiple layers in PNG or multiple PNG images. The advantage of this latter approach is that we can have multiple reference files as well. e.g. if a green box is to be moved from (0,0) to (50,0) and to (100, 0), we could create three reference files that correspond to each state. But perhaps there is a good reason you didn't choose this approach. Could you elaborate on the reason you picked this particular API? - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
On Fri, Feb 8, 2013 at 11:12 AM, Ryosuke Niwa rn...@webkit.org wrote: I had similar thoughts but my counter proposal is to let DRT/WTR generate multiple actual results either in the form of multiple layers in PNG or multiple PNG images. The advantage of this latter approach is that we can have multiple reference files as well. e.g. if a green box is to be moved from (0,0) to (50,0) and to (100, 0), we could create three reference files that correspond to each state. This sounds like a good idea for programmatic updates. I would just argue that instead of multiple images, it would be more maintainable to have multiple html references; a kind of multiple-frame-ref-test. A problem with this approach is, it is still fragile for animations. The position of a given frame is depends on timing. For animations, we could use mask-based test. The reference file of a test is a mask which gives an area where pixels can be. For example: -Take a block in a fixed position, scroll the page. The mask is the block in its original position. If for any frame, the block moved outside the mask, the test fails. -A block animating from left to right, while at the same time the page is scrolled. There is a deterministic path where the object can be. If for any frame, the block moved outside the path, the test fails. Making the mask would not always be trivial. The advantage is, if any single frame is wrong, we have a chance to catch it, regardless of the timing. Benjamin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
On Fri, Feb 8, 2013 at 11:12 AM, Ryosuke Niwa rn...@webkit.org wrote: On Fri, Feb 8, 2013 at 5:35 AM, noam.rosent...@nokia.com wrote: we don't currently have a solution in webkit's test infrastructure for testing animations on the fly, or in general for testing multiple image results in the same test. (If this was discussed before and I'm unaware of that discussion, please stop me here...) This is because ImageDiff works on the single-test level, and you can't explicitly make image comparisons in the same test at different points in time. This has before caused us several regressions that were not caught by layout tests. I'd like to propose a solution, and would welcome some feedback on whether it's a good one... The idea is that you would be able to programatically retrieve the current snapshot into a canvas ImageData, and then compare the pixel results with JavaScript in the LayoutTest. Something like: I had similar thoughts but my counter proposal is to let DRT/WTR generate multiple actual results either in the form of multiple layers in PNG or multiple PNG images. The advantage of this latter approach is that we can have multiple reference files as well. e.g. if a green box is to be moved from (0,0) to (50,0) and to (100, 0), we could create three reference files that correspond to each state. But perhaps there is a good reason you didn't choose this approach. Could you elaborate on the reason you picked this particular API? I also appreciate the problem -- and would be interesting in solving it -- but I'm a bit concerned that either proposal is at least somewhat nondeterministic (or flaky), and I'm not sure how you'd be able to control which frames of the animation you were getting well enough. But I'm not much of expert here so I'm not sure how often it matters. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
From: ext Ryosuke Niwa rn...@webkit.orgmailto:rn...@webkit.org I'd like to propose a solution, and would welcome some feedback on whether it's a good one... The idea is that you would be able to programatically retrieve the current snapshot into a canvas ImageData, and then compare the pixel results with JavaScript in the LayoutTest. Something like: I had similar thoughts but my counter proposal is to let DRT/WTR generate multiple actual results either in the form of multiple layers in PNG or multiple PNG images. The advantage of this latter approach is that we can have multiple reference files as well. e.g. if a green box is to be moved from (0,0) to (50,0) and to (100, 0), we could create three reference files that correspond to each state. But perhaps there is a good reason you didn't choose this approach. Could you elaborate on the reason you picked this particular API? The problem with dynamic features of the web like animations/interactions is that they're non-deterministic, or at least a lot less deterministic than static features of the web like layouts. Ref tests, pixel tests etc. are tools built for deterministic testing: load a file, take a snapshot, compare against a result. Testing an animation (or a filter) needs to feel a lot more dynamic and expressive: Animate green boxes, make sure that they're within a particular range at particular points in time. I think that when testing some of the dynamic features it's best to create a minimal mechanism such as retrieving snapshot pixels, giving more power to the tests themselves in a dynamic language like JavaScript, rather than try to force the tests to fit into a predefined set of features supported by the DRT/WTR infrastructure. Also this would allow some differences between the ports, which implement animations a bit differently, without constant rebaselines of reference results – e.g. A heuristic animation timing test may stay valid across platform even if the actual animation implementation is different. ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
On Fri, Feb 8, 2013 at 3:16 PM, noam.rosent...@nokia.com wrote: The problem with dynamic features of the web like animations/interactions is that they're non-deterministic, or at least a lot less deterministic than static features of the web like layouts. Ref tests, pixel tests etc. are tools built for deterministic testing: load a file, take a snapshot, compare against a result. Testing an animation (or a filter) needs to feel a lot more dynamic and expressive: Animate green boxes, make sure that they're within a particular range at particular points in time. The tests also have to be deterministic and comprehensive. I am afraid of loosing both with the Render-to-Canvas approach. Can you give concrete examples of the kind of bugs you are hunting, and why testing cannot use the two methods suggested? I think that when testing some of the dynamic features it's best to create a minimal mechanism such as retrieving snapshot pixels, giving more power to the tests themselves in a dynamic language like JavaScript, rather than try to force the tests to fit into a predefined set of features supported by the DRT/WTR infrastructure. Also this would allow some differences between the ports, which implement animations a bit differently, without constant rebaselines of reference results – e.g. A heuristic animation timing test may stay valid across platform even if the actual animation implementation is different. Again, can you give concrete examples of this? Benjamin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
Can you expose a time or setTime function in DRT and some option that says
let JS control the clock?
Then a test could do something like
layoutTestController.overridePreference(JsControlledClock, 1);
// Render 5 frames over 1 second
var frame =0;
function renderFrame() {
layoutTestController.getWindowSnapshot(...);
if (frame == 5) {
finishTest();
return;
}
++frame;
layoutTestController.setAnimationClock(frame * 1000 / 5); // assumes
clock is in ms.
requestAnimationFrame(renderFrame);
}
On Fri, Feb 8, 2013 at 3:52 PM, Benjamin Poulain benja...@webkit.orgwrote:
On Fri, Feb 8, 2013 at 3:16 PM, noam.rosent...@nokia.com wrote:
The problem with dynamic features of the web like
animations/interactions is that they're non-deterministic, or at least a
lot less deterministic than static features of the web like layouts.
Ref tests, pixel tests etc. are tools built for deterministic testing:
load a file, take a snapshot, compare against a result. Testing an
animation (or a filter) needs to feel a lot more dynamic and expressive:
Animate green boxes, make sure that they're within a particular range at
particular points in time.
The tests also have to be deterministic and comprehensive. I am afraid of
loosing both with the Render-to-Canvas approach.
Can you give concrete examples of the kind of bugs you are hunting, and
why testing cannot use the two methods suggested?
I think that when testing some of the dynamic features it's best to
create a minimal mechanism such as retrieving snapshot pixels, giving more
power to the tests themselves in a dynamic language like JavaScript, rather
than try to force the tests to fit into a predefined set of features
supported by the DRT/WTR infrastructure. Also this would allow some
differences between the ports, which implement animations a bit
differently, without constant rebaselines of reference results – e.g. A
heuristic animation timing test may stay valid across platform even if the
actual animation implementation is different.
Again, can you give concrete examples of this?
Benjamin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
On Fri, Feb 8, 2013 at 4:12 PM, Gregg Tavares g...@google.com wrote: Can you expose a time or setTime function in DRT and some option that says let JS control the clock? Unfortunately, that only works in the simple cases. We could cheat the WebCore clock, but that would ultimately be wrong for animations running in threads or driven outside WebCore. Benjamin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
On Fri, Feb 8, 2013 at 4:16 PM, Benjamin Poulain benja...@webkit.orgwrote: On Fri, Feb 8, 2013 at 4:12 PM, Gregg Tavares g...@google.com wrote: Can you expose a time or setTime function in DRT and some option that says let JS control the clock? Unfortunately, that only works in the simple cases. We could cheat the WebCore clock, but that would ultimately be wrong for animations running in threads or driven outside WebCore. The point was you need to be able to control all the clocks if you want repeatable results. If you're saying some clocks can't be controlled that sounds like something needs to be refactored so it's testable. Benjamin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
I'm not sure if this is generally known: we use [svg document].setCurrentTime(time) in the svg/animations tests for both script and pixel tests. The SVG animation test harness leaves a lot to be desired, but it may be a good place to start for a more general DRT approach: http://trac.webkit.org/browser/trunk/LayoutTests/svg/animations/script-tests/animate-css-xml-attributeType.js#L62 Philip On Fri, Feb 8, 2013 at 4:26 PM, Gregg Tavares g...@google.com wrote: On Fri, Feb 8, 2013 at 4:16 PM, Benjamin Poulain benja...@webkit.orgwrote: On Fri, Feb 8, 2013 at 4:12 PM, Gregg Tavares g...@google.com wrote: Can you expose a time or setTime function in DRT and some option that says let JS control the clock? Unfortunately, that only works in the simple cases. We could cheat the WebCore clock, but that would ultimately be wrong for animations running in threads or driven outside WebCore. The point was you need to be able to control all the clocks if you want repeatable results. If you're saying some clocks can't be controlled that sounds like something needs to be refactored so it's testable. Benjamin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
