Re: [webkit-dev] Testing iOS/WK1?
On 23/10/2018 17:44, Ali Juma wrote: > Try passing the argument "-1": > > run-webkit-tests --ios-simulator -1 Great thanks for the hint, I think this --dump-render-tree could be useful indeed to check WK1. However, in the particular scrolling tests I'm considering, I'm relying a lot on testRunner.runUIScript to execute basic gestures or emulate the find UI and most of these APIs are not implemented (easily implementable?) in Tools/DumpRenderTree/ios/UIScriptControllerIOS.mm -- Frédéric Wang - frederic-wang.fr ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing iOS/WK1?
On Tue, Oct 23, 2018 at 11:39 AM Frédéric Wang wrote: > Hi all, > > During the WebKit Contributors Meeting, Simon Fraser suggested that I > should verify some of the my patches using WK1. Hence I did some > experiments (again): > > * When I run run-webkit-tests, whatever the value specified via > --platform, the selected expectation is always a "wk2" file and the code > executed is from WebKit (i.e. WK2 == non-legacy). > Try passing the argument "-1": run-webkit-tests --ios-simulator -1 ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] Testing iOS/WK1?
Hi all, During the WebKit Contributors Meeting, Simon Fraser suggested that I should verify some of the my patches using WK1. Hence I did some experiments (again): * When I run run-webkit-tests, whatever the value specified via --platform, the selected expectation is always a "wk2" file and the code executed is from WebKit (i.e. WK2 == non-legacy). * I wrote a dummy iOS app using the deprecated UIWebView [1] and tried to launch it with Tools/Scripts/run-webkit-app --ios-simulator /path/to/WebKitLegacyApp.app ; that seems to be working (code from WebKitLegacy is indeed executed). Also, page is loaded and properly rendered at startup so I can somewhat test it. However, the app does not reply to user interaction so it's difficult to do manual testing (not to mention features like the Find UI are not available). Also, I have not checked but IIRC the EWS bots actually only run iOS/WK2 tests, right? Any idea/suggestion regarding how one could test iOS/WK1? PS: I'm using the public Xcode SDK. In particular I only have access to the simulator. [1] https://github.com/fred-wang/WebKitLegacyApp -- Frédéric Wang - frederic-wang.fr ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] Testing feature suggestion: Custom measurable data @ Performance Tests
Hi all, I'd like to suggest us to provide ways to aggregate custom measurable data to Performance Test results (i.e. as seen on perf-o-matic tool). As example of a custom measurable data, I'd suggest counting the amount of OpenGL context-summarized calls per frame. This kind of data is useful when evaluating performance impact of changes related to hardware-accelerated drawing code (i.e. TextureMapperLayer). Along with this example, other OpenGL-related data also apply (no. of state changes - specially talking about the redundant ones). I'm aware each platform may or may not benefit much from lesser OpenGL calls, but as a general guide, "the lesser the better". This kind of information can be obtained with tools like apitrace [1], for instance (at least on Linux), but may also be provided from other tools depending on the platform. As done in WebCore/WebKit, platofrm-specific implementations of the data logger can share a common interface. I haven't analyzed the perftestsrunner code to check if a similar approach to custom data is already available, so I'd kindly ask for your feedback on the subject (whether it seems like a valid approach). Best regards, Links: [1] http://apitrace.github.com -- Bruno de Oliveira Abinader Software Engineer @ basysKom GmbH WebKit committer / Nokia Certified Qt Specialist ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
>
>
>> I'm curious, what would you imagine the ref test contains?
>>
>
> If I am not mistaken, the composition operations are parallel with the
> ones of SVG and Canvas (aren't they?).
> I would have attempted comparing the 3 implementations as it seems to me
> the pixels values should be the same.
>
> That was a genuine question about that case :)
>
Thank you for feedback! :)
I presented gaussian blur case to express my opinion effectively.
As I know, Adobe want to draw CSS Filters and SVG Filters using the same
code. And Canvas does not have something like Filters.
What I want to say is that there are some cases in which snapshot
functionality is more useful.
>> >> Case 2: Fixed Position Element
>> >> [...]
>> >> function repeatedlyCalledDuringScrolling() {
>> >> ASSERT(getPixel(15, 9) == white);
>> >> ASSERT(getPixel(15, 10) == green);
>> >> ASSERT(getPixel(9, 15) == white);
>> >> ASSERT(getPixel(10, 15) == green);
>> >>
>> >> }
>> >
>> >
>> > I think this shows what I said about correctness and readability:
>> > -Asserting the correctness of the test and the result becomes close to
>> > impossible for the reader. One has to review the full code to have a
>> chance
>> > of understanding an error.
>> > -You cannot cover non trivial cases (images, text, form elements, etc).
>> > -And it is inefficient. You have to render each frame on the UIProcess,
>> move
>> > it to the WebProcess, and box it for JavaScript to process (with pixel
>> > format conversions depending on your graphics system)
>> >
>> > Of the ideas raised, I think this is one of my least favorite for
>> testing
>> > fixed positioning.
>> >
>>
>> Isn't his suggestion the equivalent of what we do today in text-only
>> tests? i.e., printing "pass" or "fail" and making you have to look at
>> the test itself to see what's being tested?
>>
>> If the correctness of the rendering depends on those 4 specific pixels
>> having those four specific values, how exactly are you going to verify
>> that by looking at it?
>>
>> Again, I think I'm just not understanding you here?
>>
>
> When looking at a test test, you follow the flow to know what it is
> supposed to do and where it breaks.
>
> How are you supposed to know, _by reading the code_, that the color at
> position 15, 9 should be white?
>
> Benjamin
>
I agree on Dirk's opinion. I supports 3 points as follows:
1. we can help to understand what's goal of test by comments
2. when we want to see the graphical result, we can use MiniBrowser like
other text based test.
3. philip's canvas tests (http://philip.html5.org/tests/canvas/suite/tests/)
already use well getImageData to test.
I agree that snapshot test can make hardly maintainable tests, but we have
review process and we can make succinct tests like philip canvas test.
And when pixel test is more suitable, we can use pixel test also.
Best regards,
Dongsung Huang
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
On Wed, Feb 13, 2013 at 11:33 PM, Benjamin Poulain wrote:
> On Wed, Feb 13, 2013 at 11:16 PM, Dirk Pranke wrote:
>>
>> > Those changes are not harmless. There are people monitoring tests
>> > results
>> > full time in order to keep WebKit in good shape. No other part of WebKit
>> > require continuous attention.
>>
>> I'm sorry, but either I don't understand Dongsung's suggestion, or I
>> don't understand your criticism. What does this sentence paragraph
>> mean? Are you suggesting someone needs to look at this type of test
>> full time in a way that we don't have to look at the other tests?
>
>
> My lack of sleep is likely to blame here :)
>
> All I am saying is tests have a hight cost when they start failing, we
> should be careful how we design them so we can easily triage bad tests,
> flaky tests, and real errors.
>
No argument here :).
>
>> >> Case 1: CSS Filters & Shaders
>> >> I wanted this test functionality when I commented
>> >> http://webkit.org/b/97859#c19
>> >> If I want to make gaussian blur test, I prefer using 'getPixel' test as
>> >> follows,
>> >
>> >
>> > Why wasn't a ref-test a better solution in this particular case?
>> >
>>
>> I'm curious, what would you imagine the ref test contains?
>
>
> If I am not mistaken, the composition operations are parallel with the ones
> of SVG and Canvas (aren't they?).
> I would have attempted comparing the 3 implementations as it seems to me the
> pixels values should be the same.
>
> That was a genuine question about that case :)
>
>>
>>
>> >> Case 2: Fixed Position Element
>> >> [...]
>> >> function repeatedlyCalledDuringScrolling() {
>> >> ASSERT(getPixel(15, 9) == white);
>> >> ASSERT(getPixel(15, 10) == green);
>> >> ASSERT(getPixel(9, 15) == white);
>> >> ASSERT(getPixel(10, 15) == green);
>> >>
>> >> }
>> >
>> >
>> > I think this shows what I said about correctness and readability:
>> > -Asserting the correctness of the test and the result becomes close to
>> > impossible for the reader. One has to review the full code to have a
>> > chance
>> > of understanding an error.
>> > -You cannot cover non trivial cases (images, text, form elements, etc).
>> > -And it is inefficient. You have to render each frame on the UIProcess,
>> > move
>> > it to the WebProcess, and box it for JavaScript to process (with pixel
>> > format conversions depending on your graphics system)
>> >
>> > Of the ideas raised, I think this is one of my least favorite for
>> > testing
>> > fixed positioning.
>> >
>>
>> Isn't his suggestion the equivalent of what we do today in text-only
>> tests? i.e., printing "pass" or "fail" and making you have to look at
>> the test itself to see what's being tested?
>>
>> If the correctness of the rendering depends on those 4 specific pixels
>> having those four specific values, how exactly are you going to verify
>> that by looking at it?
>>
>> Again, I think I'm just not understanding you here?
>
>
> When looking at a test test, you follow the flow to know what it is supposed
> to do and where it breaks.
>
> How are you supposed to know, _by reading the code_, that the color at
> position 15, 9 should be white?
>
is "the code" the test code, or the source code we're testing? Of
course, without any larger context, those assertions are impossible to
interpret. I figured he was just indicating what he meant by a
programmatic check, and a real test case would be more informative.
And of course it's easy to write bad tests that are neither readable
nor maintainable. But we also have lots of pixel tests that are
difficult for people not intimately familiar with the code being
tested to evaluate for correctness, too.
Lastly, I don't think he was suggesting that this would replace all
pixel tests; as you say, it is more well suited for some times of
tests than others.
As far as efficiency goes, even moving the picture back and forth
between two processes is still going to be far faster than running a
second reftest and comparing the output of the two pages.
-- Dirk
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
From: ext Benjamin Poulain mailto:benja...@webkit.org>> Date: Thursday, February 14, 2013 8:36 AM To: Noam Rosenthal mailto:noam.rosent...@nokia.com>> Cc: "luxte...@company100.net<mailto:luxte...@company100.net>" mailto:luxte...@company100.net>>, "webkit-dev@lists.webkit.org<mailto:webkit-dev@lists.webkit.org>" mailto:webkit-dev@lists.webkit.org>> Subject: Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly" On Wed, Feb 13, 2013 at 11:27 PM, mailto:noam.rosent...@nokia.com>> wrote: Why wasn't a ref-test a better solution in this particular case? Because gaussian blurs on the GPU are not accurate, and look slightly different on different GPUs, but usually "close enough". We need a way to measure "close enough" for features where all you can get is "close enough"! Ref-tests and pixel-tests are way to rigid for that, and require constant rebaselines and headaches. There is _NO_ gaussian blur in that patch. DongSung mentioned gaussian blur in the comment, which is what I was referring to. But other bugs as well, like delegated scrolling, also have that problem of results not being 100% the same every time. ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
On Wed, Feb 13, 2013 at 11:27 PM, wrote: > Why wasn't a ref-test a better solution in this particular case? > > Because gaussian blurs on the GPU are not accurate, and look slightly > different on different GPUs, but usually "close enough". > We need a way to measure "close enough" for features where all you can get > is "close enough"! Ref-tests and pixel-tests are way to rigid for that, and > require constant rebaselines and headaches. > There is _NO_ gaussian blur in that patch. Benjamin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
On Wed, Feb 13, 2013 at 11:16 PM, Dirk Pranke wrote:
> > Those changes are not harmless. There are people monitoring tests results
> > full time in order to keep WebKit in good shape. No other part of WebKit
> > require continuous attention.
>
> I'm sorry, but either I don't understand Dongsung's suggestion, or I
> don't understand your criticism. What does this sentence paragraph
> mean? Are you suggesting someone needs to look at this type of test
> full time in a way that we don't have to look at the other tests?
>
My lack of sleep is likely to blame here :)
All I am saying is tests have a hight cost when they start failing, we
should be careful how we design them so we can easily triage bad tests,
flaky tests, and real errors.
>> Case 1: CSS Filters & Shaders
> >> I wanted this test functionality when I commented
> >> http://webkit.org/b/97859#c19
> >> If I want to make gaussian blur test, I prefer using 'getPixel' test as
> >> follows,
> >
> >
> > Why wasn't a ref-test a better solution in this particular case?
> >
>
> I'm curious, what would you imagine the ref test contains?
>
If I am not mistaken, the composition operations are parallel with the ones
of SVG and Canvas (aren't they?).
I would have attempted comparing the 3 implementations as it seems to me
the pixels values should be the same.
That was a genuine question about that case :)
>
> >> Case 2: Fixed Position Element
> >> [...]
> >> function repeatedlyCalledDuringScrolling() {
> >> ASSERT(getPixel(15, 9) == white);
> >> ASSERT(getPixel(15, 10) == green);
> >> ASSERT(getPixel(9, 15) == white);
> >> ASSERT(getPixel(10, 15) == green);
> >>
> >> }
> >
> >
> > I think this shows what I said about correctness and readability:
> > -Asserting the correctness of the test and the result becomes close to
> > impossible for the reader. One has to review the full code to have a
> chance
> > of understanding an error.
> > -You cannot cover non trivial cases (images, text, form elements, etc).
> > -And it is inefficient. You have to render each frame on the UIProcess,
> move
> > it to the WebProcess, and box it for JavaScript to process (with pixel
> > format conversions depending on your graphics system)
> >
> > Of the ideas raised, I think this is one of my least favorite for testing
> > fixed positioning.
> >
>
> Isn't his suggestion the equivalent of what we do today in text-only
> tests? i.e., printing "pass" or "fail" and making you have to look at
> the test itself to see what's being tested?
>
> If the correctness of the rendering depends on those 4 specific pixels
> having those four specific values, how exactly are you going to verify
> that by looking at it?
>
> Again, I think I'm just not understanding you here?
>
When looking at a test test, you follow the flow to know what it is
supposed to do and where it breaks.
How are you supposed to know, _by reading the code_, that the color at
position 15, 9 should be white?
Benjamin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
From: ext Benjamin Poulain mailto:benja...@webkit.org>>
Date: Thursday, February 14, 2013 8:07 AM
To: Dongsung Huang mailto:luxte...@company100.net>>
Cc: "webkit-dev@lists.webkit.org<mailto:webkit-dev@lists.webkit.org>"
mailto:webkit-dev@lists.webkit.org>>
Subject: Re: [webkit-dev] Testing feature suggestion: animation/interaction
pixel-results "on the fly"
On Wed, Feb 13, 2013 at 9:38 PM, Dongsung Huang
mailto:luxte...@company100.net>> wrote:
I like this idea. I cannot find any harm if we have this functionality.
Those changes are not harmless. There are people monitoring tests results full
time in order to keep WebKit in good shape. No other part of WebKit require
continuous attention.
Case 1: CSS Filters & Shaders
I wanted this test functionality when I commented http://webkit.org/b/97859#c19
If I want to make gaussian blur test, I prefer using 'getPixel' test as follows,
Why wasn't a ref-test a better solution in this particular case?
Because gaussian blurs on the GPU are not accurate, and look slightly different
on different GPUs, but usually "close enough".
We need a way to measure "close enough" for features where all you can get is
"close enough"! Ref-tests and pixel-tests are way to rigid for that, and
require constant rebaselines and headaches.
Case 2: Fixed Position Element
[...]
function repeatedlyCalledDuringScrolling() {
ASSERT(getPixel(15, 9) == white);
ASSERT(getPixel(15, 10) == green);
ASSERT(getPixel(9, 15) == white);
ASSERT(getPixel(10, 15) == green);
}
I think this shows what I said about correctness and readability:
-Asserting the correctness of the test and the result becomes close to
impossible for the reader. One has to review the full code to have a chance of
understanding an error.
We can generate PNGs when a snapshot is requested so that the tester can
eyeball the results.
-You cannot cover non trivial cases (images, text, form elements, etc).
-And it is inefficient. You have to render each frame on the UIProcess, move it
to the WebProcess, and box it for JavaScript to process (with pixel format
conversions depending on your graphics system)
Another option is to do expose a partial comparison mechanism that runs in the
UI process.
E.g.
var snapshotHandle = testRunner.createSnapshot();
testRunner.comparePixel(snapshotHandle, 15, 20, 'white', { tolerance: 0.2 });
This way we can generate a visual representation of failures.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
On Wed, Feb 13, 2013 at 11:07 PM, Benjamin Poulain wrote:
> On Wed, Feb 13, 2013 at 9:38 PM, Dongsung Huang
> wrote:
>>
>> I like this idea. I cannot find any harm if we have this functionality.
>
>
> Those changes are not harmless. There are people monitoring tests results
> full time in order to keep WebKit in good shape. No other part of WebKit
> require continuous attention.
>
I'm sorry, but either I don't understand Dongsung's suggestion, or I
don't understand your criticism. What does this sentence paragraph
mean? Are you suggesting someone needs to look at this type of test
full time in a way that we don't have to look at the other tests?
>>
>> Case 1: CSS Filters & Shaders
>> I wanted this test functionality when I commented
>> http://webkit.org/b/97859#c19
>> If I want to make gaussian blur test, I prefer using 'getPixel' test as
>> follows,
>
>
> Why wasn't a ref-test a better solution in this particular case?
>
I'm curious, what would you imagine the ref test contains?
>> Case 2: Fixed Position Element
>> [...]
>> function repeatedlyCalledDuringScrolling() {
>> ASSERT(getPixel(15, 9) == white);
>> ASSERT(getPixel(15, 10) == green);
>> ASSERT(getPixel(9, 15) == white);
>> ASSERT(getPixel(10, 15) == green);
>>
>> }
>
>
> I think this shows what I said about correctness and readability:
> -Asserting the correctness of the test and the result becomes close to
> impossible for the reader. One has to review the full code to have a chance
> of understanding an error.
> -You cannot cover non trivial cases (images, text, form elements, etc).
> -And it is inefficient. You have to render each frame on the UIProcess, move
> it to the WebProcess, and box it for JavaScript to process (with pixel
> format conversions depending on your graphics system)
>
> Of the ideas raised, I think this is one of my least favorite for testing
> fixed positioning.
>
Isn't his suggestion the equivalent of what we do today in text-only
tests? i.e., printing "pass" or "fail" and making you have to look at
the test itself to see what's being tested?
If the correctness of the rendering depends on those 4 specific pixels
having those four specific values, how exactly are you going to verify
that by looking at it?
Again, I think I'm just not understanding you here?
-- Dirk
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
On Wed, Feb 13, 2013 at 9:38 PM, Dongsung Huang wrote:
> I like this idea. I cannot find any harm if we have this functionality.
>
Those changes are not harmless. There are people monitoring tests results
full time in order to keep WebKit in good shape. No other part of WebKit
require continuous attention.
> Case 1: CSS Filters & Shaders
> I wanted this test functionality when I commented
> http://webkit.org/b/97859#c19
> If I want to make gaussian blur test, I prefer using 'getPixel' test as
> follows,
>
Why wasn't a ref-test a better solution in this particular case?
Case 2: Fixed Position Element
> [...]
> function repeatedlyCalledDuringScrolling() {
> ASSERT(getPixel(15, 9) == white);
> ASSERT(getPixel(15, 10) == green);
> ASSERT(getPixel(9, 15) == white);
> ASSERT(getPixel(10, 15) == green);
>
> }
>
I think this shows what I said about correctness and readability:
-Asserting the correctness of the test and the result becomes close to
impossible for the reader. One has to review the full code to have a chance
of understanding an error.
-You cannot cover non trivial cases (images, text, form elements, etc).
-And it is inefficient. You have to render each frame on the UIProcess,
move it to the WebProcess, and box it for JavaScript to process (with pixel
format conversions depending on your graphics system)
Of the ideas raised, I think this is one of my least favorite for testing
fixed positioning.
Benjamin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
I like this idea. I cannot find any harm if we have this functionality.
When a pixel test is more succinct, we can make a pixel test. When a
'getPixel on Javascript' test is more succinct, we can make a 'getPixel'
test.
As
http://trac.webkit.org/wiki/Writing%20Layout%20Tests%20for%20DumpRenderTree
said
'Pixel tests are a burden on every ports as any small change in the engine
could lead to your test needing a new pixel result.', new test
functionality that can be an alternative of pixel test is good!
philip canvas tests (LayoutTests/canvas/philip) has no pixel test because
of the power of getImageData API in canvas spec.
I can put two examples for the power of functionality that Noam suggested,
Case 1: CSS Filters & Shaders
I wanted this test functionality when I commented
http://webkit.org/b/97859#c19
If I want to make gaussian blur test, I prefer using 'getPixel' test as
follows,
ASSERT(getPixel(10, 10) = rgba(255, 0, 0, 255));
ASSERT(getPixel(10, 15) > rgba(255, 0, 0, 98) && getPixel(10, 15) <
rgba(255, 0, 0, 102));
If we don't have 'getPixel', we need to capture all port's image results
because all port's painting engine would make a bit different result. As
you know, it is painful.
Case 2: Fixed Position Element
I guess it is one of reasons why Noam suggested.
Currently, Qt and EFL AC (a.k.a Coordinated Graphics) has a problem related
to fixed position element.
During panning animation, Fixed position element is slightly flickering.
If we have 'getPixel', we can make a succinct test case.
PRECONDITION : a green fixed position div is on the white body. The rect of
the div is rect(10, 10, 30, 30).
function repeatedlyCalledDuringScrolling() {
ASSERT(getPixel(15, 9) == white);
ASSERT(getPixel(15, 10) == green);
ASSERT(getPixel(9, 15) == white);
ASSERT(getPixel(10, 15) == green);
}
WDYT?
Best regards,
Dongsung Huang
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
On Sun, Feb 10, 2013 at 1:30 AM, Ryosuke Niwa wrote: > On Sat, Feb 9, 2013 at 5:35 PM, Dirk Pranke wrote: >> >> Perhaps -- and this is a tangent to this thread -- we could also >> investigate ways of writing tests that did render pngs but told NRWT >> to ignore them (e.g., dumpAsTextAndIgnoredImage() or something), or >> conveyed which parts of the image were important ... > > > What's the point of that? > Oops. I think I deleted the part of my reply explaining that :). What I meant to say is that the downside of text-only/programmatic tests is that you don't actually get a rendered page to look at. The downside of pixel tests is that you can get a rendered page that shows you what's wrong, but the rendered page often contains a bunch of "extraneous" things like explanatory text that can cause unnecessary diffs. It might be nice if we had better ways to filter out the parts that caused the unnecessary diffs. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
On Sat, Feb 9, 2013 at 5:35 PM, Dirk Pranke wrote: > Perhaps -- and this is a tangent to this thread -- we could also > investigate ways of writing tests that did render pngs but told NRWT > to ignore them (e.g., dumpAsTextAndIgnoredImage() or something), or > conveyed which parts of the image were important ... > What's the point of that? - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
On Sat, Feb 9, 2013 at 1:57 AM, Benjamin Poulain wrote: > On Sat, Feb 9, 2013 at 1:24 AM, wrote: >> >> I did not say they cannot be tested with the two methods suggested :) It's >> more about a preference to move some of those decisions from the test >> infrastructure to the test itself, but potentially those bugs could be >> tested in either way. >> Examples for bugs I've encountered/reviewed and can use better in-motion >> testing (note that those are specific to Qt/EFL, but I'm sure there are tons >> of bugs like this that come up for Apple/Google as well) >> http://trac.webkit.org/changeset/140825 >> http://trac.webkit.org/changeset/142112 >> http://trac.webkit.org/changeset/134953 >> https://bugs.webkit.org/show_bug.cgi?id=109179 >> >> Controlling the clock and programmatically sampling the end result would >> definitely make those more testable, but of course any progress in this area >> would be beneficial and my preference to a canvas-based API is more of an >> opinion. > > > To explain my concerns: > Sometime, I look at a failing test, and think: "what the f**k is this > supposed to test?". Then I have to dig a ton of logs, and finally read the > change to understand a the single JS statement in the whole script that make > the test useful. > > This is the situation I am afraid with a solution where pixels would be > evaluated from JavaScript. You can test, but you lack visibility why > something is correct or incorrect. Text tests, ref-tests and pixel tests > have a great readability, you can quickly evaluate their correctness. This > is important in my opinion, I don't think we want more opaque output like > the RenderTree dump. > > I am not against your plan if the readability of the tests can be good. I'd > be happy to review patches toward testing dynamic changes in webpages. > A counter-argument, of course, that there are a lot of pixel tests that would be a lot better as text-only tests that were verifying certain aspects of how the page rendered programmatically. Perhaps -- and this is a tangent to this thread -- we could also investigate ways of writing tests that did render pngs but told NRWT to ignore them (e.g., dumpAsTextAndIgnoredImage() or something), or conveyed which parts of the image were important ... -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
Oops, forgot to reply to the list. From: Noam Rosenthal mailto:noam.rosent...@nokia.com>> From: ext Simon Fraser mailto:simon.fra...@apple.com>> The existing pauseAnimation DRT API can stop an animation mid-flight, to allow for reliable snapshotting. Why isn't this enough? I would say that the main things that are missing are: 1. Sample and compare several images in the same test 2. Sample images "heuristically", e.g. sample that a green box is "close enough" to a particular location – A full ImageDiff is very rigid in that way. 3. Sample images mid-flight without pausing. Pausing can mess with animation timing and thus skew the results. I don't have a full list of what's missing – but I know for sure that when trying to hunt mid-flight bugs in the past I was not able to create sufficient tests with the current DRT tools. ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
The existing pauseAnimation DRT API can stop an animation mid-flight, to allow for reliable snapshotting. Why isn't this enough? Simon On Feb 9, 2013, at 8:44 AM, noam.rosent...@nokia.com wrote: > Since people seem to agree that this is a good problem to solve, I've created > a bug for it: https://bugs.webkit.org/show_bug.cgi?id=109356 > I can't promise to fix it myself right this moment as I'm spread a bit too > thin, but if someone wants to help or pick it up before I get to it please > feel free :) > Noam > > From: ext Benjamin Poulain > Date: Saturday, February 9, 2013 10:57 AM > To: Noam Rosenthal > Cc: "rn...@webkit.org" , "webkit-dev@lists.webkit.org" > > Subject: Re: [webkit-dev] Testing feature suggestion: animation/interaction > pixel-results "on the fly" > >> On Sat, Feb 9, 2013 at 1:24 AM, wrote: >>> >>> I did not say they cannot be tested with the two methods suggested :) It's >>> more about a preference to move some of those decisions from the test >>> infrastructure to the test itself, but potentially those bugs could be >>> tested in either way. >>> Examples for bugs I've encountered/reviewed and can use better in-motion >>> testing (note that those are specific to Qt/EFL, but I'm sure there are >>> tons of bugs like this that come up for Apple/Google as well) >>> http://trac.webkit.org/changeset/140825 >>> http://trac.webkit.org/changeset/142112 >>> http://trac.webkit.org/changeset/134953 >>> https://bugs.webkit.org/show_bug.cgi?id=109179 >>> >>> Controlling the clock and programmatically sampling the end result would >>> definitely make those more testable, but of course any progress in this >>> area would be beneficial and my preference to a canvas-based API is more of >>> an opinion. >> >> To explain my concerns: >> Sometime, I look at a failing test, and think: "what the f**k is this >> supposed to test?". Then I have to dig a ton of logs, and finally read the >> change to understand a the single JS statement in the whole script that make >> the test useful. >> >> This is the situation I am afraid with a solution where pixels would be >> evaluated from JavaScript. You can test, but you lack visibility why >> something is correct or incorrect. Text tests, ref-tests and pixel tests >> have a great readability, you can quickly evaluate their correctness. This >> is important in my opinion, I don't think we want more opaque output like >> the RenderTree dump. >> >> I am not against your plan if the readability of the tests can be good. I'd >> be happy to review patches toward testing dynamic changes in webpages. >> >> Benjamin > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
Since people seem to agree that this is a good problem to solve, I've created a bug for it: https://bugs.webkit.org/show_bug.cgi?id=109356 I can't promise to fix it myself right this moment as I'm spread a bit too thin, but if someone wants to help or pick it up before I get to it please feel free :) Noam From: ext Benjamin Poulain mailto:benja...@webkit.org>> Date: Saturday, February 9, 2013 10:57 AM To: Noam Rosenthal mailto:noam.rosent...@nokia.com>> Cc: "rn...@webkit.org<mailto:rn...@webkit.org>" mailto:rn...@webkit.org>>, "webkit-dev@lists.webkit.org<mailto:webkit-dev@lists.webkit.org>" mailto:webkit-dev@lists.webkit.org>> Subject: Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly" On Sat, Feb 9, 2013 at 1:24 AM, mailto:noam.rosent...@nokia.com>> wrote: I did not say they cannot be tested with the two methods suggested :) It's more about a preference to move some of those decisions from the test infrastructure to the test itself, but potentially those bugs could be tested in either way. Examples for bugs I've encountered/reviewed and can use better in-motion testing (note that those are specific to Qt/EFL, but I'm sure there are tons of bugs like this that come up for Apple/Google as well) http://trac.webkit.org/changeset/140825 http://trac.webkit.org/changeset/142112 http://trac.webkit.org/changeset/134953 https://bugs.webkit.org/show_bug.cgi?id=109179 Controlling the clock and programmatically sampling the end result would definitely make those more testable, but of course any progress in this area would be beneficial and my preference to a canvas-based API is more of an opinion. To explain my concerns: Sometime, I look at a failing test, and think: "what the f**k is this supposed to test?". Then I have to dig a ton of logs, and finally read the change to understand a the single JS statement in the whole script that make the test useful. This is the situation I am afraid with a solution where pixels would be evaluated from JavaScript. You can test, but you lack visibility why something is correct or incorrect. Text tests, ref-tests and pixel tests have a great readability, you can quickly evaluate their correctness. This is important in my opinion, I don't think we want more opaque output like the RenderTree dump. I am not against your plan if the readability of the tests can be good. I'd be happy to review patches toward testing dynamic changes in webpages. Benjamin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
From: ext Benjamin Poulain mailto:benja...@webkit.org>> This is the situation I am afraid with a solution where pixels would be evaluated from JavaScript. You can test, but you lack visibility why something is correct or incorrect. Text tests, ref-tests and pixel tests have a great readability, you can quickly evaluate their correctness. This is important in my opinion, I don't think we want more opaque output like the RenderTree dump. I am not against your plan if the readability of the tests can be good. I'd be happy to review patches toward testing dynamic changes in webpages. We're on the same page. I'm afraid of the same thing… Let's talk specifics when a patch is available to review. ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
On Sat, Feb 9, 2013 at 1:24 AM, wrote: >I did not say they cannot be tested with the two methods suggested :) > It's more about a preference to move some of those decisions from the test > infrastructure to the test itself, but potentially those bugs could be > tested in either way. > Examples for bugs I've encountered/reviewed and can use better in-motion > testing (note that those are specific to Qt/EFL, but I'm sure there are > tons of bugs like this that come up for Apple/Google as well) > http://trac.webkit.org/changeset/140825 > http://trac.webkit.org/changeset/142112 > http://trac.webkit.org/changeset/134953 > https://bugs.webkit.org/show_bug.cgi?id=109179 > > Controlling the clock and programmatically sampling the end result would > definitely make those more testable, but of course any progress in this > area would be beneficial and my preference to a canvas-based API is more of > an opinion. > To explain my concerns: Sometime, I look at a failing test, and think: "what the f**k is this supposed to test?". Then I have to dig a ton of logs, and finally read the change to understand a the single JS statement in the whole script that make the test useful. This is the situation I am afraid with a solution where pixels would be evaluated from JavaScript. You can test, but you lack visibility why something is correct or incorrect. Text tests, ref-tests and pixel tests have a great readability, you can quickly evaluate their correctness. This is important in my opinion, I don't think we want more opaque output like the RenderTree dump. I am not against your plan if the readability of the tests can be good. I'd be happy to review patches toward testing dynamic changes in webpages. Benjamin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
From: ext Benjamin Poulain mailto:benja...@webkit.org>> Date: Saturday, February 9, 2013 12:52 AM To: Noam Rosenthal mailto:noam.rosent...@nokia.com>> Cc: "rn...@webkit.org<mailto:rn...@webkit.org>" mailto:rn...@webkit.org>>, "webkit-dev@lists.webkit.org<mailto:webkit-dev@lists.webkit.org>" mailto:webkit-dev@lists.webkit.org>> Subject: Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly" On Fri, Feb 8, 2013 at 3:16 PM, mailto:noam.rosent...@nokia.com>> wrote: The problem with dynamic features of the web like animations/interactions is that they're non-deterministic, or at least a lot less deterministic than static features of the web like layouts. Ref tests, pixel tests etc. are tools built for deterministic testing: load a file, take a snapshot, compare against a result. Testing an animation (or a filter) needs to feel a lot more dynamic and expressive: Animate green boxes, make sure that they're within a particular range at particular points in time". The tests also have to be deterministic and comprehensive. I am afraid of loosing both with the Render-to-Canvas approach. Can you give concrete examples of the kind of bugs you are hunting, and why testing cannot use the two methods suggested? I did not say they cannot be tested with the two methods suggested :) It's more about a preference to move some of those decisions from the test infrastructure to the test itself, but potentially those bugs could be tested in either way. Examples for bugs I've encountered/reviewed and can use better in-motion testing (note that those are specific to Qt/EFL, but I'm sure there are tons of bugs like this that come up for Apple/Google as well) http://trac.webkit.org/changeset/140825 http://trac.webkit.org/changeset/142112 http://trac.webkit.org/changeset/134953 https://bugs.webkit.org/show_bug.cgi?id=109179 Controlling the clock and programmatically sampling the end result would definitely make those more testable, but of course any progress in this area would be beneficial and my preference to a canvas-based API is more of an opinion. ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
I'm not sure if this is generally known: we use [svg document].setCurrentTime(time) in the svg/animations tests for both script and pixel tests. The SVG animation test harness leaves a lot to be desired, but it may be a good place to start for a more general DRT approach: http://trac.webkit.org/browser/trunk/LayoutTests/svg/animations/script-tests/animate-css-xml-attributeType.js#L62 Philip On Fri, Feb 8, 2013 at 4:26 PM, Gregg Tavares wrote: > > > > On Fri, Feb 8, 2013 at 4:16 PM, Benjamin Poulain wrote: > >> On Fri, Feb 8, 2013 at 4:12 PM, Gregg Tavares wrote: >> >>> Can you expose a time or setTime function in DRT and some option that >>> says "let JS control the clock"? >>> >> >> Unfortunately, that only works in the simple cases. >> We could cheat the WebCore clock, but that would ultimately be wrong for >> animations running in threads or driven outside WebCore. >> > > The point was you need to be able to control all the clocks if you want > repeatable results. If you're saying some clocks can't be controlled that > sounds like something needs to be refactored so it's testable. > > >> >> Benjamin >> > > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev > > ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
On Fri, Feb 8, 2013 at 4:16 PM, Benjamin Poulain wrote: > On Fri, Feb 8, 2013 at 4:12 PM, Gregg Tavares wrote: > >> Can you expose a time or setTime function in DRT and some option that >> says "let JS control the clock"? >> > > Unfortunately, that only works in the simple cases. > We could cheat the WebCore clock, but that would ultimately be wrong for > animations running in threads or driven outside WebCore. > The point was you need to be able to control all the clocks if you want repeatable results. If you're saying some clocks can't be controlled that sounds like something needs to be refactored so it's testable. > > Benjamin > ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
On Fri, Feb 8, 2013 at 4:12 PM, Gregg Tavares wrote: > Can you expose a time or setTime function in DRT and some option that says > "let JS control the clock"? > Unfortunately, that only works in the simple cases. We could cheat the WebCore clock, but that would ultimately be wrong for animations running in threads or driven outside WebCore. Benjamin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
Can you expose a time or setTime function in DRT and some option that says
"let JS control the clock"?
Then a test could do something like
layoutTestController.overridePreference("JsControlledClock", "1");
// Render 5 frames over 1 second
var frame =0;
function renderFrame() {
layoutTestController.getWindowSnapshot(...);
if (frame == 5) {
finishTest();
return;
}
++frame;
layoutTestController.setAnimationClock(frame * 1000 / 5); // assumes
clock is in ms.
requestAnimationFrame(renderFrame);
}
On Fri, Feb 8, 2013 at 3:52 PM, Benjamin Poulain wrote:
> On Fri, Feb 8, 2013 at 3:16 PM, wrote:
>
>> The problem with dynamic features of the web like
>> animations/interactions is that they're non-deterministic, or at least a
>> lot less deterministic than static features of the web like layouts.
>> Ref tests, pixel tests etc. are tools built for deterministic testing:
>> load a file, take a snapshot, compare against a result. Testing an
>> animation (or a filter) needs to feel a lot more dynamic and expressive:
>> Animate green boxes, make sure that they're within a particular range at
>> particular points in time".
>>
>
> The tests also have to be deterministic and comprehensive. I am afraid of
> loosing both with the Render-to-Canvas approach.
>
> Can you give concrete examples of the kind of bugs you are hunting, and
> why testing cannot use the two methods suggested?
>
>
>> I think that when testing some of the dynamic features it's best to
>> create a minimal mechanism such as retrieving snapshot pixels, giving more
>> power to the tests themselves in a dynamic language like JavaScript, rather
>> than try to force the tests to fit into a predefined set of features
>> supported by the DRT/WTR infrastructure. Also this would allow some
>> differences between the ports, which implement animations a bit
>> differently, without constant rebaselines of reference results – e.g. A
>> heuristic animation timing test may stay valid across platform even if the
>> actual animation implementation is different.
>>
>
> Again, can you give concrete examples of this?
>
> Benjamin
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
On Fri, Feb 8, 2013 at 3:16 PM, wrote: > The problem with dynamic features of the web like > animations/interactions is that they're non-deterministic, or at least a > lot less deterministic than static features of the web like layouts. > Ref tests, pixel tests etc. are tools built for deterministic testing: > load a file, take a snapshot, compare against a result. Testing an > animation (or a filter) needs to feel a lot more dynamic and expressive: > Animate green boxes, make sure that they're within a particular range at > particular points in time". > The tests also have to be deterministic and comprehensive. I am afraid of loosing both with the Render-to-Canvas approach. Can you give concrete examples of the kind of bugs you are hunting, and why testing cannot use the two methods suggested? > I think that when testing some of the dynamic features it's best to create > a minimal mechanism such as retrieving snapshot pixels, giving more power > to the tests themselves in a dynamic language like JavaScript, rather than > try to force the tests to fit into a predefined set of features supported > by the DRT/WTR infrastructure. Also this would allow some differences > between the ports, which implement animations a bit differently, without > constant rebaselines of reference results – e.g. A heuristic animation > timing test may stay valid across platform even if the actual animation > implementation is different. > Again, can you give concrete examples of this? Benjamin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
From: ext Ryosuke Niwa mailto:rn...@webkit.org>> I'd like to propose a solution, and would welcome some feedback on whether it's a good one... The idea is that you would be able to programatically retrieve the current snapshot into a canvas ImageData, and then compare the pixel results with JavaScript in the LayoutTest. Something like: I had similar thoughts but my counter "proposal" is to let DRT/WTR generate multiple actual results either in the form of multiple layers in PNG or multiple PNG images. The advantage of this latter approach is that we can have multiple reference files as well. e.g. if a green box is to be moved from (0,0) to (50,0) and to (100, 0), we could create three reference files that correspond to each state. But perhaps there is a good reason you didn't choose this approach. Could you elaborate on the reason you picked this particular API? The problem with dynamic features of the web like animations/interactions is that they're non-deterministic, or at least a lot less deterministic than static features of the web like layouts. Ref tests, pixel tests etc. are tools built for deterministic testing: load a file, take a snapshot, compare against a result. Testing an animation (or a filter) needs to feel a lot more dynamic and expressive: Animate green boxes, make sure that they're within a particular range at particular points in time". I think that when testing some of the dynamic features it's best to create a minimal mechanism such as retrieving snapshot pixels, giving more power to the tests themselves in a dynamic language like JavaScript, rather than try to force the tests to fit into a predefined set of features supported by the DRT/WTR infrastructure. Also this would allow some differences between the ports, which implement animations a bit differently, without constant rebaselines of reference results – e.g. A heuristic animation timing test may stay valid across platform even if the actual animation implementation is different. ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
On Fri, Feb 8, 2013 at 11:12 AM, Ryosuke Niwa wrote: > On Fri, Feb 8, 2013 at 5:35 AM, wrote: >> >> we don't currently have a solution in webkit's test infrastructure for >> testing animations "on the fly", or in general for testing multiple image >> results in the same test. >> (If this was discussed before and I'm unaware of that discussion, please >> stop me here...) >> >> This is because ImageDiff works on the single-test level, and you can't >> explicitly make image comparisons in the same test at different points in >> time. >> This has before caused us several regressions that were not caught by >> layout tests. >> >> I'd like to propose a solution, and would welcome some feedback on whether >> it's a good one... >> The idea is that you would be able to programatically retrieve the current >> snapshot into a canvas ImageData, and then compare the pixel results with >> JavaScript in the LayoutTest. Something like: > > > I had similar thoughts but my counter "proposal" is to let DRT/WTR generate > multiple actual results either in the form of multiple layers in PNG or > multiple PNG images. The advantage of this latter approach is that we can > have multiple reference files as well. e.g. if a green box is to be moved > from (0,0) to (50,0) and to (100, 0), we could create three reference files > that correspond to each state. > > But perhaps there is a good reason you didn't choose this approach. Could > you elaborate on the reason you picked this particular API? > I also appreciate the problem -- and would be interesting in solving it -- but I'm a bit concerned that either proposal is at least somewhat nondeterministic (or flaky), and I'm not sure how you'd be able to control which frames of the animation you were getting well enough. But I'm not much of expert here so I'm not sure how often it matters. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
On Fri, Feb 8, 2013 at 11:12 AM, Ryosuke Niwa wrote: > I had similar thoughts but my counter "proposal" is to let DRT/WTR > generate multiple actual results either in the form of multiple layers in > PNG or multiple PNG images. The advantage of this latter approach is that > we can have multiple reference files as well. e.g. if a green box is to be > moved from (0,0) to (50,0) and to (100, 0), we could create three reference > files that correspond to each state. > This sounds like a good idea for programmatic updates. I would just argue that instead of multiple images, it would be more maintainable to have multiple html references; a kind of multiple-frame-ref-test. A problem with this approach is, it is still fragile for animations. The position of a given frame is depends on timing. For animations, we could use mask-based test. The reference file of a test is a mask which gives an area where pixels can be. For example: -Take a block in a fixed position, scroll the page. The mask is the block in its original position. If for any frame, the block moved outside the mask, the test fails. -A block animating from left to right, while at the same time the page is scrolled. There is a deterministic path where the object can be. If for any frame, the block moved outside the path, the test fails. Making the mask would not always be trivial. The advantage is, if any single frame is wrong, we have a chance to catch it, regardless of the timing. Benjamin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
On Fri, Feb 8, 2013 at 5:35 AM, wrote: > we don't currently have a solution in webkit's test infrastructure for > testing animations "on the fly", or in general for testing multiple image > results in the same test. > (If this was discussed before and I'm unaware of that discussion, please > stop me here...) > > This is because ImageDiff works on the single-test level, and you can't > explicitly make image comparisons in the same test at different points in > time. > This has before caused us several regressions that were not caught by > layout tests. > > I'd like to propose a solution, and would welcome some feedback on whether > it's a good one... > The idea is that you would be able to programatically retrieve the current > snapshot into a canvas ImageData, and then compare the pixel results with > JavaScript in the LayoutTest. Something like: > I had similar thoughts but my counter "proposal" is to let DRT/WTR generate multiple actual results either in the form of multiple layers in PNG or multiple PNG images. The advantage of this latter approach is that we can have multiple reference files as well. e.g. if a green box is to be moved from (0,0) to (50,0) and to (100, 0), we could create three reference files that correspond to each state. But perhaps there is a good reason you didn't choose this approach. Could you elaborate on the reason you picked this particular API? - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
> I'd like to propose a solution, and would welcome some feedback on whether > it's a good one... > The idea is that you would be able to programatically retrieve the current > snapshot into a canvas ImageData, and then compare the pixel results with > JavaScript in the LayoutTest. Something like: +1 Lovely idea. For instance, we've been having some issue [1] testing CustomFilters, so having the ability to /sample/ the final result from JS would be great. [1] https://bugs.webkit.org/show_bug.cgi?id=104012 -- Bye, Michelangelo ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
From: webkit-dev-boun...@lists.webkit.org [webkit-dev-boun...@lists.webkit.org] on behalf of ext Robert Kroeger [rjkro...@chromium.org] > How would the test wait for the animation to happen? Tests that use > timeouts seem more likely to be flaky from what I've seen. Could > window.testRunner.getWindowSnapshot take a time delta of some kind? Most of the bugs I've encountered in this area can be caught by a test that would have "obvious" results that can buffer the flakiness, e.g. by moving a big green rectangle and testing only its center. ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
Questions inline.
On Fri, Feb 8, 2013 at 8:35 AM, wrote:
>
> Hellos
>
> we don't currently have a solution in webkit's test infrastructure for
> testing animations "on the fly", or in general for testing multiple image
> results in the same test.
I have been wishing for such a thing around how to test scroll animations.
> (If this was discussed before and I'm unaware of that discussion, please
> stop me here...)
>
> This is because ImageDiff works on the single-test level, and you can't
> explicitly make image comparisons in the same test at different points in
> time.
> This has before caused us several regressions that were not caught by
> layout tests.
>
> I'd like to propose a solution, and would welcome some feedback on whether
> it's a good one...
> The idea is that you would be able to programatically retrieve the current
> snapshot into a canvas ImageData, and then compare the pixel results with
> JavaScript in the LayoutTest. Something like:
>
> window.testRunner.getWindowSnapshot(function(snapshot) {
> if (snapshot.data[snapshot.width * 100 + 10] != 127)
> output.innerHTML = "FAIL";
> });
>
> WDYT?
How would the test wait for the animation to happen? Tests that use
timeouts seem more likely to be flaky from what I've seen. Could
window.testRunner.getWindowSnapshot take a time delta of some kind?
Rob.
> Noam
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"
Hellos
we don't currently have a solution in webkit's test infrastructure for testing
animations "on the fly", or in general for testing multiple image results in
the same test.
(If this was discussed before and I'm unaware of that discussion, please stop
me here...)
This is because ImageDiff works on the single-test level, and you can't
explicitly make image comparisons in the same test at different points in time.
This has before caused us several regressions that were not caught by layout
tests.
I'd like to propose a solution, and would welcome some feedback on whether it's
a good one...
The idea is that you would be able to programatically retrieve the current
snapshot into a canvas ImageData, and then compare the pixel results with
JavaScript in the LayoutTest. Something like:
window.testRunner.getWindowSnapshot(function(snapshot) {
if (snapshot.data[snapshot.width * 100 + 10] != 127)
output.innerHTML = "FAIL";
});
WDYT?
Noam
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] testing strategy? - CJK line breaking tests
On Tue, Sep 4, 2012 at 11:26 PM, Dan Bernstein wrote: > On Sep 4, 2012, at 5:50 AM, Glenn Adams wrote: > > Alternatively, you can compute the with of each character with script >> (without adding any new feature to WebKit). >> > > Unfortunately, that won't work since I can't access the width of > individual lines produced by formatting a block. > > You can use Range’s getClientRects to get the widths of individual > characters and to determine where a line breaks occur. > Thanks for that pointer. I wasn't aware of that feature. In any case, it looks like the reftest approach is going to work. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] testing strategy? - CJK line breaking tests
On Sep 4, 2012, at 5:50 AM, Glenn Adams wrote: >> >> Alternatively, you can compute the with of each character with script >> (without adding any new feature to WebKit). > > Unfortunately, that won't work since I can't access the width of individual > lines produced by formatting a block. You can use Range’s getClientRects to get the widths of individual characters and to determine where a line breaks occur.___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] testing strategy? - CJK line breaking tests
On Tue, Sep 4, 2012 at 8:30 PM, Ryosuke Niwa wrote:
> Can we use ref tests?
I'll try this first.
>
> Alternatively, you can compute the with of each character with script
> (without adding any new feature to WebKit).
>
Unfortunately, that won't work since I can't access the width of individual
lines produced by formatting a block. For example, XYZ, where XYZ are certain CJK line break sensitive
chars, may format to
line-break: loose
line1: XY
lline2: Z
line-break: strict
line1: X
line2: YZ
The getComputedValue('width'|'height'| of the two results doesn't vary
since one can't access the resulting lines.
> If neither is possible, then you should use (1) unless the test tests a
> platform specific feature; I.e. the feature isnt available on other
> platforms.
>
The feature is not platform specific (other than the fact different
platforms may use different versions of ICU and support different fonts).
If reftest doesn't work, I'll try (1).
>
> (3) seems like a terrible idea as it means that we'll be either exposing
> the WebKit internals to the Web without standardizing those properties or
> we'll be adding yet-another DRT-specific behavior
>
Certainly it is not be the intent to expose such props to web content in
general. But I could fathom exposing this to WK test content (without
necessarily being dependent on DRT framework).
>
>
> On Tuesday, September 4, 2012, Glenn Adams wrote:
>
>> What is the recommended approach to test cases when one needs to use a
>> CJK font that covers the test data? I could use DRT text results as
>> expected but given lack of common font across platforms, that doesn't seem
>> to be effective.
>>
>> From my somewhat limited (i.e., newbie) exposure to WK, I gather one can
>> take one of the following approaches:
>>
>> (1) put font (and thus platform) dependent test cases in non-platform
>> test directory, then add entries to Skipped for other platforms; this seems
>> the current approach with many platform/font dependent tests, especially
>> related to I18N features;
>> (2) put font (and thus platform) dependent test cases in platform test
>> directory, possibly ending up with separate tests per platform;
>> (3) what would be nice is to not generate rendering dump from DRT but
>> instead use script only; since I'm testing line-breaking logic, what would
>> do nicely is a set of internal styles available via getComputedStyle() on a
>> block element:
>>
>>- -webkit-line-count - numeric value indicating number of formatted
>>lines produced from rendering a block
>>- -webkit-line-chars-1 through -N, which returns the chars of line N
>>as formatted when rendering a block
>>
>> That is, with these properties, I could use script to determine which
>> line breaks occurred and where. I don't care about geometries or pixels in
>> these particular tests, just whether breaks occur in specific contexts.
>>
>> Any recommendations on whether to pursue one of (1) or (2) above, or try
>> the more ambitious (but more platform independent) third approach?
>>
>> I suppose I could start with (1) or (2) and then pursue (3) as a
>> subsequent task.
>>
>> Regards,
>> Glenn
>>
>>
>
> --
> Ryosuke Niwa
> Software Engineer
> Google Inc.
>
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] testing strategy? - CJK line breaking tests
Can we use ref tests? Alternatively, you can compute the with of each character with script (without adding any new feature to WebKit). If neither is possible, then you should use (1) unless the test tests a platform specific feature; I.e. the feature isnt available on other platforms. (3) seems like a terrible idea as it means that we'll be either exposing the WebKit internals to the Web without standardizing those properties or we'll be adding yet-another DRT-specific behavior On Tuesday, September 4, 2012, Glenn Adams wrote: > What is the recommended approach to test cases when one needs to use a CJK > font that covers the test data? I could use DRT text results as expected > but given lack of common font across platforms, that doesn't seem to be > effective. > > From my somewhat limited (i.e., newbie) exposure to WK, I gather one can > take one of the following approaches: > > (1) put font (and thus platform) dependent test cases in non-platform test > directory, then add entries to Skipped for other platforms; this seems the > current approach with many platform/font dependent tests, especially > related to I18N features; > (2) put font (and thus platform) dependent test cases in platform test > directory, possibly ending up with separate tests per platform; > (3) what would be nice is to not generate rendering dump from DRT but > instead use script only; since I'm testing line-breaking logic, what would > do nicely is a set of internal styles available via getComputedStyle() on a > block element: > >- -webkit-line-count - numeric value indicating number of formatted >lines produced from rendering a block >- -webkit-line-chars-1 through -N, which returns the chars of line N >as formatted when rendering a block > > That is, with these properties, I could use script to determine which line > breaks occurred and where. I don't care about geometries or pixels in these > particular tests, just whether breaks occur in specific contexts. > > Any recommendations on whether to pursue one of (1) or (2) above, or try > the more ambitious (but more platform independent) third approach? > > I suppose I could start with (1) or (2) and then pursue (3) as a > subsequent task. > > Regards, > Glenn > > -- Ryosuke Niwa Software Engineer Google Inc. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] testing strategy? - CJK line breaking tests
What is the recommended approach to test cases when one needs to use a CJK font that covers the test data? I could use DRT text results as expected but given lack of common font across platforms, that doesn't seem to be effective. >From my somewhat limited (i.e., newbie) exposure to WK, I gather one can take one of the following approaches: (1) put font (and thus platform) dependent test cases in non-platform test directory, then add entries to Skipped for other platforms; this seems the current approach with many platform/font dependent tests, especially related to I18N features; (2) put font (and thus platform) dependent test cases in platform test directory, possibly ending up with separate tests per platform; (3) what would be nice is to not generate rendering dump from DRT but instead use script only; since I'm testing line-breaking logic, what would do nicely is a set of internal styles available via getComputedStyle() on a block element: - -webkit-line-count - numeric value indicating number of formatted lines produced from rendering a block - -webkit-line-chars-1 through -N, which returns the chars of line N as formatted when rendering a block That is, with these properties, I could use script to determine which line breaks occurred and where. I don't care about geometries or pixels in these particular tests, just whether breaks occur in specific contexts. Any recommendations on whether to pursue one of (1) or (2) above, or try the more ambitious (but more platform independent) third approach? I suppose I could start with (1) or (2) and then pursue (3) as a subsequent task. Regards, Glenn ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing XHR
On Tue, Sep 21, 2010 at 8:43 AM, Alexey Proskuryakov wrote: > I don't think that the suite is testing much besides what we already have > regression tests for. For the most part, it just has different expectations. FWIW, if you find cases the test suite is missing I'm happy to add them. The only reason the W3C is not hosting this test suite at the moment is because they cannot handle server-side scripts on the test server at the moment. PLH is looking into a solution on their side as I understand things. -- Anne van Kesteren http://annevankesteren.nl/ ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] testing
-- Regards " Navigate Today For Better Tomorrow!!!" mEeT mE LivE wItH tEcH cAkE @ http://hitechi.19.forumer.com/ http://go4expert.com/ http://www.liveperson.com/asadullah-ansari/ http://go4hitech.wordpress.com/ ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing XHR
Yes, I have just gone through all the tests in this suite and categorized and filed some WebKit bugs. Some other test failures seem to be due to something wrong with tests themselves. I am pinging Anne about these issues. Thanks, Jian On Mon, Sep 20, 2010 at 11:43 PM, Alexey Proskuryakov wrote: > > I don't think that the suite is testing much besides what we already have > regression tests for. For the most part, it just has different expectations. > > There doesn't seem to be any harm in importing it as a whole now, but > looking over the failing tests, categorizing them and sending feedback to > public-webapps working groups is more important at this point. Jian Li from > Chromium team is currently filing bugs for tests that fail in WebKit. > > - WBR, Alexey Proskuryakov > > 20.09.2010, в 22:34, Ojan Vafai написал(а): > > Filed https://bugs.webkit.org/show_bug.cgi?id=46164 for the script to pull > the tests into our repo. > > On Tue, Sep 21, 2010 at 3:27 PM, Maciej Stachowiak wrote: > >> >> On Sep 20, 2010, at 10:19 PM, Ojan Vafai wrote: >> >> > >> > How about we create http/tests/xmlhttprequest/w3c-experimental or >> something like that? That can tide us over until the official version comes >> out, at which point, we can delete the w3c-experimental directory and just >> add a w3c directory. >> > >> > It would be nice if we could start fixing these things before they >> become part of the official test suite as a way of evaluating whether there >> are issues with the spec and/or test suite. >> >> That is in fact exactly what we should be doing at this stage of the >> standards process for XHR. >> >> > >> > Also, it seems to me like to does make sense to add an >> update-experimental-w3c-xhr-tests script or something until there is an >> official version. >> >> Indeed. And even after the test suite is official, it is likely to expand >> over time. We should also look over our own XHR tests to see if there are >> any that it would make sense to contribute to the W3C. >> >> Cheers, >> Maciej >> >> > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > > > > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing XHR
I don't think that the suite is testing much besides what we already have regression tests for. For the most part, it just has different expectations. There doesn't seem to be any harm in importing it as a whole now, but looking over the failing tests, categorizing them and sending feedback to public-webapps working groups is more important at this point. Jian Li from Chromium team is currently filing bugs for tests that fail in WebKit. - WBR, Alexey Proskuryakov 20.09.2010, в 22:34, Ojan Vafai написал(а): > Filed https://bugs.webkit.org/show_bug.cgi?id=46164 for the script to pull > the tests into our repo. > > On Tue, Sep 21, 2010 at 3:27 PM, Maciej Stachowiak wrote: > > On Sep 20, 2010, at 10:19 PM, Ojan Vafai wrote: > > > > > How about we create http/tests/xmlhttprequest/w3c-experimental or something > > like that? That can tide us over until the official version comes out, at > > which point, we can delete the w3c-experimental directory and just add a > > w3c directory. > > > > It would be nice if we could start fixing these things before they become > > part of the official test suite as a way of evaluating whether there are > > issues with the spec and/or test suite. > > That is in fact exactly what we should be doing at this stage of the > standards process for XHR. > > > > > Also, it seems to me like to does make sense to add an > > update-experimental-w3c-xhr-tests script or something until there is an > > official version. > > Indeed. And even after the test suite is official, it is likely to expand > over time. We should also look over our own XHR tests to see if there are any > that it would make sense to contribute to the W3C. > > Cheers, > Maciej > > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing XHR
Filed https://bugs.webkit.org/show_bug.cgi?id=46164 for the script to pull the tests into our repo. On Tue, Sep 21, 2010 at 3:27 PM, Maciej Stachowiak wrote: > > On Sep 20, 2010, at 10:19 PM, Ojan Vafai wrote: > > > > > How about we create http/tests/xmlhttprequest/w3c-experimental or > something like that? That can tide us over until the official version comes > out, at which point, we can delete the w3c-experimental directory and just > add a w3c directory. > > > > It would be nice if we could start fixing these things before they become > part of the official test suite as a way of evaluating whether there are > issues with the spec and/or test suite. > > That is in fact exactly what we should be doing at this stage of the > standards process for XHR. > > > > > Also, it seems to me like to does make sense to add an > update-experimental-w3c-xhr-tests script or something until there is an > official version. > > Indeed. And even after the test suite is official, it is likely to expand > over time. We should also look over our own XHR tests to see if there are > any that it would make sense to contribute to the W3C. > > Cheers, > Maciej > > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing XHR
On Sep 20, 2010, at 10:19 PM, Ojan Vafai wrote: > > How about we create http/tests/xmlhttprequest/w3c-experimental or something > like that? That can tide us over until the official version comes out, at > which point, we can delete the w3c-experimental directory and just add a w3c > directory. > > It would be nice if we could start fixing these things before they become > part of the official test suite as a way of evaluating whether there are > issues with the spec and/or test suite. That is in fact exactly what we should be doing at this stage of the standards process for XHR. > > Also, it seems to me like to does make sense to add an > update-experimental-w3c-xhr-tests script or something until there is an > official version. Indeed. And even after the test suite is official, it is likely to expand over time. We should also look over our own XHR tests to see if there are any that it would make sense to contribute to the W3C. Cheers, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing XHR
On Tue, Sep 21, 2010 at 2:55 PM, Maciej Stachowiak wrote: > On Sep 20, 2010, at 9:34 PM, Luke Macpherson wrote: > > > WebKit (or at least Chrome) is currently failing a bunch opera's tests > > located at: http://tc.labs.opera.com/apis/XMLHttpRequest/ > > > > It seems that it would be a good idea for us to make use of these > > tests in WebKit, so I was thinking of importing them into the > > codebase. This raises the question: > > Should we import the tests one at a time as we fix the bugs, or should > > we import the whole suite? > > The tests look like they will fit into the existing LayoutTests > > infrastructure pretty easily, but if we import the whole lot it would > > be good to figure out if it is possible to keep our copy in sync with > > Opera's, possibly along the lines of > > WebKitTools/Scripts/update-webgl-conformance-tests . > > > > Should they end up in a subdirectory of LayoutTests/http/tests? (They > > require php scripts on the server). > > It would be a good idea to drop in the test suite wholesale. On the other > hand, it's likely that most of this will turn into the official W3C test > suite for the XMLHttpRequest spec. So we should either wait a bit, or be > vigilant to avoid adding the same tests twice when we drop in the official > test suite. > > One possibility would be to make this a subdirectory of > http/tests/xmlhttprequest. > How about we create http/tests/xmlhttprequest/w3c-experimental or something like that? That can tide us over until the official version comes out, at which point, we can delete the w3c-experimental directory and just add a w3c directory. It would be nice if we could start fixing these things before they become part of the official test suite as a way of evaluating whether there are issues with the spec and/or test suite. Also, it seems to me like to does make sense to add an update-experimental-w3c-xhr-tests script or something until there is an official version. Ojan ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing XHR
On Sep 20, 2010, at 9:34 PM, Luke Macpherson wrote: > WebKit (or at least Chrome) is currently failing a bunch opera's tests > located at: http://tc.labs.opera.com/apis/XMLHttpRequest/ > > It seems that it would be a good idea for us to make use of these > tests in WebKit, so I was thinking of importing them into the > codebase. This raises the question: > Should we import the tests one at a time as we fix the bugs, or should > we import the whole suite? > The tests look like they will fit into the existing LayoutTests > infrastructure pretty easily, but if we import the whole lot it would > be good to figure out if it is possible to keep our copy in sync with > Opera's, possibly along the lines of > WebKitTools/Scripts/update-webgl-conformance-tests . > > Should they end up in a subdirectory of LayoutTests/http/tests? (They > require php scripts on the server). It would be a good idea to drop in the test suite wholesale. On the other hand, it's likely that most of this will turn into the official W3C test suite for the XMLHttpRequest spec. So we should either wait a bit, or be vigilant to avoid adding the same tests twice when we drop in the official test suite. One possibility would be to make this a subdirectory of http/tests/xmlhttprequest. Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] Testing XHR
WebKit (or at least Chrome) is currently failing a bunch opera's tests located at: http://tc.labs.opera.com/apis/XMLHttpRequest/ It seems that it would be a good idea for us to make use of these tests in WebKit, so I was thinking of importing them into the codebase. This raises the question: Should we import the tests one at a time as we fix the bugs, or should we import the whole suite? The tests look like they will fit into the existing LayoutTests infrastructure pretty easily, but if we import the whole lot it would be good to figure out if it is possible to keep our copy in sync with Opera's, possibly along the lines of WebKitTools/Scripts/update-webgl-conformance-tests . Should they end up in a subdirectory of LayoutTests/http/tests? (They require php scripts on the server). Thanks, Luke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing changes to CodeGenerator*.pm
On Thu, Apr 29, 2010 at 9:05 PM, Adam Barth wrote: > On Thu, Apr 29, 2010 at 12:43 PM, Maciej Stachowiak wrote: > > > It seems to me a better model would be to regenerate the bindings test > file > > automatically as part of the build. Then the changes can still be > reviewed > > by you, or as part of a diff, but there would be no extra manual steps > > involved. And people making behaviorally transparent changes to codegen > > output would perhaps feel a little less burdened. > > That sounds like a good improvement. I think it would be fine to > regenerate the output as part of the build. However, I think we > should still preserve the ability to run the script along it "test" > mode because that's about three orders of magnitude faster than > performing a build after touching CodeGeneratorJS. > >From reading this thread, this seems like the best solution and most easy to accomplish. We already have the standalone script, so all we need to do is hack it into the build. Isn't there some makefile that generates the derived sources for at least most of the platforms? Couldn't we just add it to that? People who don't care or value it could continue with their current work-flows unaffected. And those that it helps can continue with their workflows as well, as far as I can tell. And, optionally, if someone would like to create more robust testing frameworks in the future could. Am I totally missing something? I feel like I must since this discussion has continued on for a while beyond this point. J ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing changes to CodeGenerator*.pm
Since I'm in the bindings hall of shame, I guess I'm supposed to reply. ;) The twice that I've used it, it was very helpful. The few reviews I've done of Adam's it was much better than what we had before. However, I agree something better could be built. I'm just not sure what better looks like yet. I expect when Adam finishes getting rid of custom bindings code he'll have a better idea. :) I'm surprised this thread so much attention. -eric On Thu, Apr 29, 2010 at 1:38 PM, Maciej Stachowiak wrote: > > > On Apr 29, 2010, at 1:05 PM, Adam Barth wrote: > > On Thu, Apr 29, 2010 at 12:43 PM, Maciej Stachowiak wrote: > > It seems to me a better model would be to regenerate the bindings test file > > automatically as part of the build. Then the changes can still be reviewed > > by you, or as part of a diff, but there would be no extra manual steps > > involved. And people making behaviorally transparent changes to codegen > > output would perhaps feel a little less burdened. > > That sounds like a good improvement. I think it would be fine to > regenerate the output as part of the build. However, I think we > should still preserve the ability to run the script along it "test" > mode because that's about three orders of magnitude faster than > performing a build after touching CodeGeneratorJS. > > Alexey (or others who don't like the new tests), do you think this change > would address your concerns? > On Apr 29, 2010, at 1:05 PM, Adam Barth wrote: > > What I hear from this conversation is the following: > > 1) A bunch of people who've used the tool saying that they've found it > useful. > 2) A bunch of people who haven't used the tool suggesting improvements. > > This tool impacts the handful of people who work on > CodeGeneratorJS.pm. Everyone else in the project can safely ignore > it. I'm all for improvements, and I invite anyone interested to > either improve the tool or write a new tool that does the job better. > > If everyone has to use the tool for the tool to be useful, then ideally we > want a system where the people who change the bindings frequently mostly buy > into. Here is the list of people with more than 5 all-time commits in the > WebCore/bindings/scripts directory. Ideally I'd like to hear from more of > these what they think would be helpful and not burdensome. > 59 wei...@apple.com > 46 e...@webkit.org > 35 da...@apple.com > 32 jap...@chromium.org > 29 oli...@apple.com > 26 gga...@apple.com > 26 dglaz...@chromium.org > 16 aba...@webkit.org > 14 zimmerm...@webkit.org > 12 �...@webkit.org > 10 aro...@apple.com > 8 le...@chromium.org > 7 ...@apple.com > 7 da...@chromium.org > 6 timo...@apple.com > 6 s...@chromium.org > 6 jia...@chromium.org > 6 ddkil...@apple.com > 6 cwzwar...@webkit.org > Here is the command anyone can run to see the full list (assuming you have > an SVN checkout): > $ svn log WebCore/bindings/scripts | grep '|.*@' | sed -e 's/^[^|]* |//g; s/ > | .*$//g;' | sort | uniq -c | sort -rn > The long tail of people who have made only a few bindings changes is rather > large, so I suspect this tool affects more than a handful people, if it > becomes a mandatory part of the process. > Regards, > Maciej > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing changes to CodeGenerator*.pm
On 29.04.2010, at 13:38, Maciej Stachowiak wrote: Alexey (or others who don't like the new tests), do you think this change would address your concerns? It would definitely be better if changes weren't detected as failures. Changing a checked out file as part of build seems rather magical though. I'm not sure if everyone agrees that making a diff of actual generated files would be better than looking at a single custom test. That's something I wished for many times, but it calls for a different workflow. And of course, that's not something that is currently implemented. - WBR, Alexey Proskuryakov ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing changes to CodeGenerator*.pm
On Apr 29, 2010, at 1:05 PM, Adam Barth wrote: On Thu, Apr 29, 2010 at 12:43 PM, Maciej Stachowiak wrote: It seems to me a better model would be to regenerate the bindings test file automatically as part of the build. Then the changes can still be reviewed by you, or as part of a diff, but there would be no extra manual steps involved. And people making behaviorally transparent changes to codegen output would perhaps feel a little less burdened. That sounds like a good improvement. I think it would be fine to regenerate the output as part of the build. However, I think we should still preserve the ability to run the script along it "test" mode because that's about three orders of magnitude faster than performing a build after touching CodeGeneratorJS. Alexey (or others who don't like the new tests), do you think this change would address your concerns? On Apr 29, 2010, at 1:05 PM, Adam Barth wrote: What I hear from this conversation is the following: 1) A bunch of people who've used the tool saying that they've found it useful. 2) A bunch of people who haven't used the tool suggesting improvements. This tool impacts the handful of people who work on CodeGeneratorJS.pm. Everyone else in the project can safely ignore it. I'm all for improvements, and I invite anyone interested to either improve the tool or write a new tool that does the job better. If everyone has to use the tool for the tool to be useful, then ideally we want a system where the people who change the bindings frequently mostly buy into. Here is the list of people with more than 5 all-time commits in the WebCore/bindings/scripts directory. Ideally I'd like to hear from more of these what they think would be helpful and not burdensome. 59 wei...@apple.com 46 e...@webkit.org 35 da...@apple.com 32 jap...@chromium.org 29 oli...@apple.com 26 gga...@apple.com 26 dglaz...@chromium.org 16 aba...@webkit.org 14 zimmerm...@webkit.org 12 a...@webkit.org 10 aro...@apple.com 8 le...@chromium.org 7 m...@apple.com 7 da...@chromium.org 6 timo...@apple.com 6 s...@chromium.org 6 jia...@chromium.org 6 ddkil...@apple.com 6 cwzwar...@webkit.org Here is the command anyone can run to see the full list (assuming you have an SVN checkout): $ svn log WebCore/bindings/scripts | grep '|.*@' | sed -e 's/^[^|]* |// g; s/ | .*$//g;' | sort | uniq -c | sort -rn The long tail of people who have made only a few bindings changes is rather large, so I suspect this tool affects more than a handful people, if it becomes a mandatory part of the process. Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing changes to CodeGenerator*.pm
I see run-bindings-tests is primary a productivity tool: Faster development, easy debugging, better reviews. But it only works if the golden copies don't fall into disrepair => Hence the need to bake this tool into the general testing framework (run-webkit-tests, pre-submit checks, etc.). It will become a useful (but not perfect) regression tool as a side effect of that. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing changes to CodeGenerator*.pm
On Thu, Apr 29, 2010 at 12:43 PM, Maciej Stachowiak wrote: > To repeat, I think this is a useful tool, but not necessarily as a test. > The mode of operation is that when you run this test, if the generated > bindings for the text IDL file change in any way, it reports a failure. Then > you must manually take the step of regenerating the golden master file. It > doesn't seem like the failure report itself will result in a decision. It > doesn't provide interesting information until you regenerate the test file > and look at the diffs, except in the highly unusual case of changing the > output of the codegen script unintentially. Most changes to the codegen > script are to intentionally change the output. The failure report shows you the diff. In hacking on CodeGeneratorJS, I often run the script many times to see how my changes are affecting the generated code, much in the same way I would run all the LayoutTests constantly if they took <5s to run. > It seems to me a better model would be to regenerate the bindings test file > automatically as part of the build. Then the changes can still be reviewed > by you, or as part of a diff, but there would be no extra manual steps > involved. And people making behaviorally transparent changes to codegen > output would perhaps feel a little less burdened. That sounds like a good improvement. I think it would be fine to regenerate the output as part of the build. However, I think we should still preserve the ability to run the script along it "test" mode because that's about three orders of magnitude faster than performing a build after touching CodeGeneratorJS. > That makes more sense to me than treating it as a regression test. It also > seems like this model would still have all the benefits you cite above. What I hear from this conversation is the following: 1) A bunch of people who've used the tool saying that they've found it useful. 2) A bunch of people who haven't used the tool suggesting improvements. This tool impacts the handful of people who work on CodeGeneratorJS.pm. Everyone else in the project can safely ignore it. I'm all for improvements, and I invite anyone interested to either improve the tool or write a new tool that does the job better. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing changes to CodeGenerator*.pm
On Apr 29, 2010, at 11:38 AM, Ojan Vafai wrote: I don't really follow the argument against supporting this test. What is the maintenance overhead? Isn't this just like having a layout test with expected results? It's a small isolated test instead of testing everything. That seems like a good thing. More importantly, it lets you be sure that every feature of the code generator has some testing. In the real IDLs, a feature might stop getting used temporarily and then changes to the code generator would not be readily apparent. I don't think your comments above are responsive to my paragraph below: On Thu, Apr 29, 2010 at 11:05 AM, Maciej Stachowiak wrote: It also strikes me as odd to do testing by doing exact comparison of the generated source. But I can also see side benefits. I think the real issue here may be one of naming. If the use model is that you fully regenerate the results every time you make a change to the bindings generator, then it's not really a regression test. The purpose is not to catch unintentional changes but rather to be able to observe changes to generated code, and new kinds of generated code, while working on a change and when reviewing code. Perhaps the tool should have a name that reflects that, instead of implying that the purpose is to catch bugs accidentally introduced by changes. It doesn't seem like an efficient or effective way to do the latter. To repeat, I think this is a useful tool, but not necessarily as a test. The mode of operation is that when you run this test, if the generated bindings for the text IDL file change in any way, it reports a failure. Then you must manually take the step of regenerating the golden master file. It doesn't seem like the failure report itself will result in a decision. It doesn't provide interesting information until you regenerate the test file and look at the diffs, except in the highly unusual case of changing the output of the codegen script unintentially. Most changes to the codegen script are to intentionally change the output. It seems to me a better model would be to regenerate the bindings test file automatically as part of the build. Then the changes can still be reviewed by you, or as part of a diff, but there would be no extra manual steps involved. And people making behaviorally transparent changes to codegen output would perhaps feel a little less burdened. That makes more sense to me than treating it as a regression test. It also seems like this model would still have all the benefits you cite above. Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing changes to CodeGenerator*.pm
IMHO, run-webkit-tests should run all the various webkit testing scripts and we should have a run-layout-tests script that does what run-webkit-tests does today. I'd also settle for a run-tests scripts that was the ASAD testing script. Adam On Thu, Apr 29, 2010 at 12:16 PM, David Levin wrote: > Just curious, would it be less maintenance if the test run was integrated > with run-webkit-tests?/Is the concern about having lots of different tests > harness to run to verify a change? > dave > On Thu, Apr 29, 2010 at 12:11 PM, James Robinson wrote: >> >> As a concrete example, I found this test setup helpful for this >> patch: http://trac.webkit.org/changeset/58345. A nice side effect was that >> it revealed a bug in CodeGeneratorGObject.pm and let me fix it without >> having to set up build setup for whatever it is that uses the GObject >> bindings. >> I agree that golden file testing is a very high-maintenance fragile test >> method, but it's better than nothing. If this framework didn't exist then I >> would have likely made the change and relied on spot checking and our >> existing automated tests to catch any regressions which is less than ideal. >> - James >> >> On Thu, Apr 29, 2010 at 11:44 AM, Ojan Vafai wrote: >>> >>> On Thu, Apr 29, 2010 at 11:39 AM, Alexey Proskuryakov >>> wrote: On 29.04.2010, at 11:17, Yaar Schnitman wrote: > > I've been using the tool for a couple of patches in V8. It really boost > the development cycle, helps reviewers understanding what a cryptic perl > block of code actually does, and side effects are easy to find. Once you > start using it, its becoming hard to work without it. Give it a try! 'm thinking about how this tool could have helped with the CodeGenerator changes I made in the past, and it seems that it wouldn't have detected any changes, and could require me to find creative ways to test the new behavior. >>> >>> I don't really follow the what the maintenance overhead is. How does this >>> actually cause you more than a trivial amount of extra work? Maybe a >>> specific example would help. >>> Isn't this just like having a layout test with expected results? It's a >>> small isolated test instead of testing everything. That seems like a good >>> thing. >>> More importantly, it lets you be sure that every feature of the code >>> generator has some testing. In the real IDLs, a feature might stop getting >>> used temporarily and then changes to the code generator would not be readily >>> apparent. >>> Ojan >>> P.S. Sorry for the double-post some of you got. Sent from the wrong email >>> address at first. >>> ___ >>> webkit-dev mailing list >>> webkit-dev@lists.webkit.org >>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >>> >> >> >> ___ >> webkit-dev mailing list >> webkit-dev@lists.webkit.org >> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >> > > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing changes to CodeGenerator*.pm
Just curious, would it be less maintenance if the test run was integrated with run-webkit-tests?/Is the concern about having lots of different tests harness to run to verify a change? dave On Thu, Apr 29, 2010 at 12:11 PM, James Robinson wrote: > As a concrete example, I found this test setup helpful for this patch: > http://trac.webkit.org/changeset/58345. A nice side effect was that it > revealed a bug in CodeGeneratorGObject.pm and let me fix it without having > to set up build setup for whatever it is that uses the GObject bindings. > > I agree that golden file testing is a very high-maintenance fragile test > method, but it's better than nothing. If this framework didn't exist then I > would have likely made the change and relied on spot checking and our > existing automated tests to catch any regressions which is less than ideal. > > - James > > On Thu, Apr 29, 2010 at 11:44 AM, Ojan Vafai wrote: > >> On Thu, Apr 29, 2010 at 11:39 AM, Alexey Proskuryakov wrote: >> >>> On 29.04.2010, at 11:17, Yaar Schnitman wrote: >>> I've been using the tool for a couple of patches in V8. It really boost the development cycle, helps reviewers understanding what a cryptic perl block of code actually does, and side effects are easy to find. Once you start using it, its becoming hard to work without it. Give it a try! >>> >>> 'm thinking about how this tool could have helped with the CodeGenerator >>> changes I made in the past, and it seems that it wouldn't have detected any >>> changes, and could require me to find creative ways to test the new >>> behavior. >> >> >> I don't really follow the what the maintenance overhead is. How does this >> actually cause you more than a trivial amount of extra work? Maybe a >> specific example would help. >> >> Isn't this just like having a layout test with expected results? It's a >> small isolated test instead of testing everything. That seems like a good >> thing. >> >> More importantly, it lets you be sure that every feature of the code >> generator has some testing. In the real IDLs, a feature might stop getting >> used temporarily and then changes to the code generator would not be readily >> apparent. >> >> Ojan >> >> P.S. Sorry for the double-post some of you got. Sent from the wrong email >> address at first. >> >> ___ >> webkit-dev mailing list >> webkit-dev@lists.webkit.org >> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >> >> > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing changes to CodeGenerator*.pm
As a concrete example, I found this test setup helpful for this patch: http://trac.webkit.org/changeset/58345. A nice side effect was that it revealed a bug in CodeGeneratorGObject.pm and let me fix it without having to set up build setup for whatever it is that uses the GObject bindings. I agree that golden file testing is a very high-maintenance fragile test method, but it's better than nothing. If this framework didn't exist then I would have likely made the change and relied on spot checking and our existing automated tests to catch any regressions which is less than ideal. - James On Thu, Apr 29, 2010 at 11:44 AM, Ojan Vafai wrote: > On Thu, Apr 29, 2010 at 11:39 AM, Alexey Proskuryakov wrote: > >> On 29.04.2010, at 11:17, Yaar Schnitman wrote: >> >>> I've been using the tool for a couple of patches in V8. It really boost >>> the development cycle, helps reviewers understanding what a cryptic perl >>> block of code actually does, and side effects are easy to find. Once you >>> start using it, its becoming hard to work without it. Give it a try! >>> >> >> 'm thinking about how this tool could have helped with the CodeGenerator >> changes I made in the past, and it seems that it wouldn't have detected any >> changes, and could require me to find creative ways to test the new >> behavior. > > > I don't really follow the what the maintenance overhead is. How does this > actually cause you more than a trivial amount of extra work? Maybe a > specific example would help. > > Isn't this just like having a layout test with expected results? It's a > small isolated test instead of testing everything. That seems like a good > thing. > > More importantly, it lets you be sure that every feature of the code > generator has some testing. In the real IDLs, a feature might stop getting > used temporarily and then changes to the code generator would not be readily > apparent. > > Ojan > > P.S. Sorry for the double-post some of you got. Sent from the wrong email > address at first. > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing changes to CodeGenerator*.pm
On Thu, Apr 29, 2010 at 11:39 AM, Alexey Proskuryakov wrote: > On 29.04.2010, at 11:17, Yaar Schnitman wrote: > >> I've been using the tool for a couple of patches in V8. It really boost >> the development cycle, helps reviewers understanding what a cryptic perl >> block of code actually does, and side effects are easy to find. Once you >> start using it, its becoming hard to work without it. Give it a try! >> > > 'm thinking about how this tool could have helped with the CodeGenerator > changes I made in the past, and it seems that it wouldn't have detected any > changes, and could require me to find creative ways to test the new > behavior. I don't really follow the what the maintenance overhead is. How does this actually cause you more than a trivial amount of extra work? Maybe a specific example would help. Isn't this just like having a layout test with expected results? It's a small isolated test instead of testing everything. That seems like a good thing. More importantly, it lets you be sure that every feature of the code generator has some testing. In the real IDLs, a feature might stop getting used temporarily and then changes to the code generator would not be readily apparent. Ojan P.S. Sorry for the double-post some of you got. Sent from the wrong email address at first. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing changes to CodeGenerator*.pm
On 29.04.2010, at 11:17, Yaar Schnitman wrote: I've been using the tool for a couple of patches in V8. It really boost the development cycle, helps reviewers understanding what a cryptic perl block of code actually does, and side effects are easy to find. Once you start using it, its becoming hard to work without it. Give it a try! 'm thinking about how this tool could have helped with the CodeGenerator changes I made in the past, and it seems that it wouldn't have detected any changes, and could require me to find creative ways to test the new behavior. - WBR, Alexey Proskuryakov ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing changes to CodeGenerator*.pm
On 29.04.2010, at 10:54, Adam Barth wrote: You should feel free to develop a better testing harness. Since you're proposing run-bindings-tests as a required tool for everyone to use, I don't think that this resolves my concerns. The maintenance is super easy. I've been doing a lot of development work on the code generator in the past few days, and it amounts to typing a single command: ./WebKitTools/Scripts/run-bindings-tests --reset-results So, isn't once supposed to extend the tests when e.g. adding new flags? - WBR, Alexey Proskuryakov ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing changes to CodeGenerator*.pm
On Apr 29, 2010, at 10:54 AM, Adam Barth wrote: It would be great to have a tool that generates a diff of derived sources for inspection, but making it into a test for everyone to maintain feels like unnecessary burden. I certainly would feel bad about having to maintain a test that verifies source file content instead of behavior. You should feel free to develop a better testing harness. This one certainly isn't best conceivable tool, but it's better than what we had previously, which was essentially the C++ compiler. The maintenance is super easy. I've been doing a lot of development work on the code generator in the past few days, and it amounts to typing a single command: ./WebKitTools/Scripts/run-bindings-tests --reset-results The harness has been super useful in working on the code generator because the tests run in a few seconds. That lets me iterate on the script much more quickly compared to rebuilding the world every time I want to try a tweak. It also strikes me as odd to do testing by doing exact comparison of the generated source. But I can also see side benefits. I think the real issue here may be one of naming. If the use model is that you fully regenerate the results every time you make a change to the bindings generator, then it's not really a regression test. The purpose is not to catch unintentional changes but rather to be able to observe changes to generated code, and new kinds of generated code, while working on a change and when reviewing code. Perhaps the tool should have a name that reflects that, instead of implying that the purpose is to catch bugs accidentally introduced by changes. It doesn't seem like an efficient or effective way to do the latter. Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing changes to CodeGenerator*.pm
On Thu, Apr 29, 2010 at 10:54 AM, Adam Barth wrote: > On Thu, Apr 29, 2010 at 10:39 AM, Alexey Proskuryakov wrote: >> On 29.04.2010, at 10:27, Jeremy Orlow wrote: >>> It's great to test end-to-end behavior, and unit tests can also also >>> useful sometimes, but why test that source code stays byte to byte >>> identical? >>> >>> When you make a change to the code generator, you should make a >>> corresponding change to the generated test code. This allows the reviewer >>> to see how the change in CodeGenerator*.pm affects the generated code and >>> documents the change in SVN. The hope is that it'll be obvious if your >>> change is going to have unintended consequences on the generated code. >>> We've found this useful when working on the V8 bindings. >> >> For these goals to be achieved, we'd need to check all generated code, not >> just one custom test file for each language. > > The idea is that the test IDL file exercises the features of the code > generator. Sure, it's not 100% coverage, but it's much better not > testing it at all. > >> It would be great to have a tool that generates a diff of derived sources >> for inspection, but making it into a test for everyone to maintain feels >> like unnecessary burden. I certainly would feel bad about having to maintain >> a test that verifies source file content instead of behavior. > > You should feel free to develop a better testing harness. This one > certainly isn't best conceivable tool, but it's better than what we > had previously, which was essentially the C++ compiler. > > The maintenance is super easy. I've been doing a lot of development > work on the code generator in the past few days, and it amounts to > typing a single command: > > ./WebKitTools/Scripts/run-bindings-tests --reset-results > > The harness has been super useful in working on the code generator > because the tests run in a few seconds. That lets me iterate on the > script much more quickly compared to rebuilding the world every time I > want to try a tweak. As an additional datapoint on maintenance, it looks like you've touched CodeGeneratorJS.pm twice in the past year and neither of those change would have resulted in a large diff to the expected test results. By contrast, I've made five changes in the past two days and plan to made dozens more changes shortly to reduce the amount of custom bindings code we have in the tree, which is a serious maintenance problem because it is full of subtle, quirky behavior. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing changes to CodeGenerator*.pm
ext Alexey Proskuryakov wrote: On 29.04.2010, at 10:30, Adam Barth wrote: Yeah, without seeing the consequence on the generated code, it can be very difficult to review changes to the code generator. Yes, that's a useful goal. Since an old version of code generating scripts is always in .svn directory (or its git equivalent), it should be possible to generate "before" and "after" versions of DerivedSources directory, and diff these. Such a tool would be useful and exciting. Definitely. The generator for Qt Jambi (Qt bindings for Java, http://qt.gitorious.org/qt-jambi) has a --diff option that produces a colorized diff against the output of the previous run. It's a great, simple way of building confidence that the generator change only had the impact you intended (no guarantees of course, due to platform differences, feature permutations etc.). In my experience such a feature is like "turning on the lights" (or cranking up the generator, to use a direct analogy). Kent ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing changes to CodeGenerator*.pm
On Thu, Apr 29, 2010 at 10:39 AM, Alexey Proskuryakov wrote: > On 29.04.2010, at 10:27, Jeremy Orlow wrote: >> It's great to test end-to-end behavior, and unit tests can also also >> useful sometimes, but why test that source code stays byte to byte >> identical? >> >> When you make a change to the code generator, you should make a >> corresponding change to the generated test code. This allows the reviewer >> to see how the change in CodeGenerator*.pm affects the generated code and >> documents the change in SVN. The hope is that it'll be obvious if your >> change is going to have unintended consequences on the generated code. >> We've found this useful when working on the V8 bindings. > > For these goals to be achieved, we'd need to check all generated code, not > just one custom test file for each language. The idea is that the test IDL file exercises the features of the code generator. Sure, it's not 100% coverage, but it's much better not testing it at all. > It would be great to have a tool that generates a diff of derived sources > for inspection, but making it into a test for everyone to maintain feels > like unnecessary burden. I certainly would feel bad about having to maintain > a test that verifies source file content instead of behavior. You should feel free to develop a better testing harness. This one certainly isn't best conceivable tool, but it's better than what we had previously, which was essentially the C++ compiler. The maintenance is super easy. I've been doing a lot of development work on the code generator in the past few days, and it amounts to typing a single command: ./WebKitTools/Scripts/run-bindings-tests --reset-results The harness has been super useful in working on the code generator because the tests run in a few seconds. That lets me iterate on the script much more quickly compared to rebuilding the world every time I want to try a tweak. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing changes to CodeGenerator*.pm
On 29.04.2010, at 10:30, Adam Barth wrote: Yeah, without seeing the consequence on the generated code, it can be very difficult to review changes to the code generator. Yes, that's a useful goal. Since an old version of code generating scripts is always in .svn directory (or its git equivalent), it should be possible to generate "before" and "after" versions of DerivedSources directory, and diff these. Such a tool would be useful and exciting. - WBR, Alexey Proskuryakov ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing changes to CodeGenerator*.pm
On 29.04.2010, at 10:27, Jeremy Orlow wrote: It's great to test end-to-end behavior, and unit tests can also also useful sometimes, but why test that source code stays byte to byte identical? When you make a change to the code generator, you should make a corresponding change to the generated test code. This allows the reviewer to see how the change in CodeGenerator*.pm affects the generated code and documents the change in SVN. The hope is that it'll be obvious if your change is going to have unintended consequences on the generated code. We've found this useful when working on the V8 bindings. For these goals to be achieved, we'd need to check all generated code, not just one custom test file for each language. It would be great to have a tool that generates a diff of derived sources for inspection, but making it into a test for everyone to maintain feels like unnecessary burden. I certainly would feel bad about having to maintain a test that verifies source file content instead of behavior. - WBR, Alexey Proskuryakov ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing changes to CodeGenerator*.pm
On Thu, Apr 29, 2010 at 9:44 AM, Alexey Proskuryakov wrote: > On 26.04.2010, at 22:06, Adam Barth wrote: >> If you make changes to CodeGenerator*.pm, please test your >> change using the following command: >> >> ~/git/webkit$ ./WebKitTools/Scripts/run-bindings-tests > > > I don't understand the utility of such testing. When one edits > CodeGenerator*.pm, that's generally to change its output, so > run-bindings-tests tests will of course fail. > > It's great to test end-to-end behavior, and unit tests can also also useful > sometimes, but why test that source code stays byte to byte identical? I should say that there's also a class of changes to CodeGenerator*.pm where we don't want to change its output at all. For example, removing some of the ridiculously deeply nested if statements. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing changes to CodeGenerator*.pm
On Thu, Apr 29, 2010 at 10:27 AM, Jeremy Orlow wrote: > On Thu, Apr 29, 2010 at 5:44 PM, Alexey Proskuryakov wrote: >> On 26.04.2010, at 22:06, Adam Barth wrote: >>> If you make changes to CodeGenerator*.pm, please test your >>> change using the following command: >>> >>> ~/git/webkit$ ./WebKitTools/Scripts/run-bindings-tests >> >> >> I don't understand the utility of such testing. When one edits >> CodeGenerator*.pm, that's generally to change its output, so >> run-bindings-tests tests will of course fail. >> >> It's great to test end-to-end behavior, and unit tests can also also >> useful sometimes, but why test that source code stays byte to byte >> identical? > > When you make a change to the code generator, you should make > a corresponding change to the generated test code. This allows the reviewer > to see how the change in CodeGenerator*.pm affects the generated code and > documents the change in SVN. The hope is that it'll be obvious if your > change is going to have unintended consequences on the generated code. > We've found this useful when working on the V8 bindings. Yeah, without seeing the consequence on the generated code, it can be very difficult to review changes to the code generator. Imagine if you were only able to review patches to WebCore by looking at the LayoutTests. The tests have already let us find and fix a handful of bugs and we've only had them for three days. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing changes to CodeGenerator*.pm
On Thu, Apr 29, 2010 at 5:44 PM, Alexey Proskuryakov wrote: > > On 26.04.2010, at 22:06, Adam Barth wrote: > > If you make changes to CodeGenerator*.pm, please test your >> change using the following command: >> >> ~/git/webkit$ ./WebKitTools/Scripts/run-bindings-tests >> > > > I don't understand the utility of such testing. When one edits > CodeGenerator*.pm, that's generally to change its output, so > run-bindings-tests tests will of course fail. > > It's great to test end-to-end behavior, and unit tests can also also useful > sometimes, but why test that source code stays byte to byte identical? > When you make a change to the code generator, you should make a corresponding change to the generated test code. This allows the reviewer to see how the change in CodeGenerator*.pm affects the generated code and documents the change in SVN. The hope is that it'll be obvious if your change is going to have unintended consequences on the generated code. We've found this useful when working on the V8 bindings. J ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing changes to CodeGenerator*.pm
On 26.04.2010, at 22:06, Adam Barth wrote: If you make changes to CodeGenerator*.pm, please test your change using the following command: ~/git/webkit$ ./WebKitTools/Scripts/run-bindings-tests I don't understand the utility of such testing. When one edits CodeGenerator*.pm, that's generally to change its output, so run- bindings-tests tests will of course fail. It's great to test end-to-end behavior, and unit tests can also also useful sometimes, but why test that source code stays byte to byte identical? - WBR, Alexey Proskuryakov ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] Testing changes to CodeGenerator*.pm
People of webkit-dev: We now have a testing harness in place for CodeGeneratorJS.pm and friends. If you make changes to CodeGenerator*.pm, please test your change using the following command: ~/git/webkit$ ./WebKitTools/Scripts/run-bindings-tests If you add a new feature to CodeGenerator*.pm, please exercise that feature by adding a declaration in our test IDL file: http://trac.webkit.org/browser/trunk/WebCore/bindings/scripts/test/TestObj.idl If your patch changes the expected results, you can generate new results as follows: ~/git/webkit$ ./WebKitTools/Scripts/run-bindings-tests --overwrite That way folks can review the change to the expected results along side your change to the generator. Happy code generation hacking! Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] testing client implementations
Yes. LayoutTests test all manner of things. Looks for tests that call the dump...Callbacks method of layoutTestController. Adam On Tue, Jan 26, 2010 at 10:27 PM, Joe Mason wrote: > I'm not too familiar with the test framework, but my impression is that > the LayoutTests are all about the results of page rendering. Are there > any tests which validate that callbacks on classes like > FrameLoaderClient are called at the expected time? > > Joe > > - > This transmission (including any attachments) may contain confidential > information, privileged material (including material protected by the > solicitor-client or other applicable privileges), or constitute non-public > information. Any use of this information by anyone other than the intended > recipient is prohibited. If you have received this transmission in error, > please immediately reply to the sender and delete this information from your > system. Use, dissemination, distribution, or reproduction of this > transmission by unintended recipients is not authorized and may be unlawful. > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] testing client implementations
I'm not too familiar with the test framework, but my impression is that the LayoutTests are all about the results of page rendering. Are there any tests which validate that callbacks on classes like FrameLoaderClient are called at the expected time? Joe - This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing harness for Geolocation
Hi Adam, On Wed, Sep 2, 2009 at 2:59 PM, Adam Barth wrote: > Improvements to the geolocation feature have stalled out because we > don't have a way to test these patches. It seems the existing > geolocation feature has been implemented without testing. (Boo!) > > Testing geolocation seems different than testing other WebKit features > because geolocation lives in WebCore/platform. One approach is to > create a fake (or mock) geolocation service that DRT can control. > This has been tried a couple of times: > > https://bugs.webkit.org/show_bug.cgi?id=21717 > https://bugs.webkit.org/show_bug.cgi?id=28264 > The approach taken in 28264 uses a static factory method for creating GeolocationService instances. It also allows DRT to set the specific factory method that the Geolocation framework will use to one that creates mock GeolocationService instances. The layout tests can then use the layoutTestController to configure the mock GeolocationService as needed. This seems reasonable to me. Thanks, Andrei ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] Testing harness for Geolocation
Improvements to the geolocation feature have stalled out because we don't have a way to test these patches. It seems the existing geolocation feature has been implemented without testing. (Boo!) Testing geolocation seems different than testing other WebKit features because geolocation lives in WebCore/platform. One approach is to create a fake (or mock) geolocation service that DRT can control. This has been tried a couple of times: https://bugs.webkit.org/show_bug.cgi?id=21717 https://bugs.webkit.org/show_bug.cgi?id=28264 I couldn't find find any precedent for testing this kind of feature before, so I wanted to run this by the list to see if anyone has strong opinions about the right way to test geolocation. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing worker lifecycle
On Thu, Jun 25, 2009 at 2:33 PM, Maciej Stachowiak wrote:
>
> On Jun 25, 2009, at 2:27 PM, Drew Wilson wrote:
>
> (resending to a wider audience - apologies to those of you who receive
>> this twice)
>> Hi all,
>>
>> The HTML5 worker spec has changed significantly from its earlier
>> incarnations, in that it's no longer externally visible whether a given
>> worker thread has shut down or not (there are no more close events surfaced
>> on the Worker).
>>
>
> What's the motivation for this particular change?
>
Primarily to avoid requiring specific GC behavior in the HTML5 spec. If you
expose when a worker has shut down, then for compatibility reasons you need
to try to make all user agents behave identically wrt garbage collection and
reachability, which puts an unnecessary burden on implementors. We've also
removed things like MessagePort.active as well as throwing exceptions if you
try to post a MessagePort that's been closed, for similar reasons. All of
this is motivated by the fact that it's prohibitively difficult to determine
cross-thread/cross-process reachability.
>
>
> This means that there's not currently a good way to write tests to verify
>> that workers are actually shutting down when they are unreachable/idle. Do
>> you guys have any ideas about how to address this?
>>
>
> One obvious way would be to add a nonstandard equivalent to the "close"
> event ("webkitClose"). The downside is that this would be exposed to Web
> content.
>
Exactly, which I'd like to avoid for the reasons mentioned above.
>
>
>
>> One idea I had would be to expose a WorkerController for layout tests,
>> similar to GCController - the WorkerController could expose a numWorkers
>> attribute which is incremented/decremented as worker threads startup/exit. I
>> haven't started looking into how this would be done, but I figured I'd ask
>> you guys if this was a good approach or if there's a cleaner way to test
>> these cases.
>>
>
> That's also possible, the downside of this approach would be that the tests
> wouldn't work in a normal browser that doesn't have the special
> DumpRenderTree APIs.
Indeed. That feels kind of lame, but the current alternative (no tests at
all) seems bad too.
>
>
> - Maciej
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing worker lifecycle
On Jun 25, 2009, at 2:27 PM, Drew Wilson wrote:
(resending to a wider audience - apologies to those of you who
receive this twice)
Hi all,
The HTML5 worker spec has changed significantly from its earlier
incarnations, in that it's no longer externally visible whether a
given worker thread has shut down or not (there are no more close
events surfaced on the Worker).
What's the motivation for this particular change?
This means that there's not currently a good way to write tests to
verify that workers are actually shutting down when they are
unreachable/idle. Do you guys have any ideas about how to address
this?
One obvious way would be to add a nonstandard equivalent to the
"close" event ("webkitClose"). The downside is that this would be
exposed to Web content.
One idea I had would be to expose a WorkerController for layout
tests, similar to GCController - the WorkerController could expose a
numWorkers attribute which is incremented/decremented as worker
threads startup/exit. I haven't started looking into how this would
be done, but I figured I'd ask you guys if this was a good approach
or if there's a cleaner way to test these cases.
That's also possible, the downside of this approach would be that the
tests wouldn't work in a normal browser that doesn't have the special
DumpRenderTree APIs.
- Maciej
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] Testing worker lifecycle
(resending to a wider audience - apologies to those of you who receive this twice) Hi all, The HTML5 worker spec has changed significantly from its earlier incarnations, in that it's no longer externally visible whether a given worker thread has shut down or not (there are no more close events surfaced on the Worker). This means that there's not currently a good way to write tests to verify that workers are actually shutting down when they are unreachable/idle. Do you guys have any ideas about how to address this? One idea I had would be to expose a WorkerController for layout tests, similar to GCController - the WorkerController could expose a numWorkers attribute which is incremented/decremented as worker threads startup/exit. I haven't started looking into how this would be done, but I figured I'd ask you guys if this was a good approach or if there's a cleaner way to test these cases. -atw ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] testing question for code that calls abstract methods (like ChromeClient)
On May 27, 2009, at 4:11 PM, John Gregg wrote: What's a good strategy for testing WebKit code that calls out to the ChromeClient interface or some other abstract interface? I can't find anything obvious in the codebase-- is there any way to mock out those calls for unit tests? Typically we add features to DumpRenderTree that use the public interfaces on the various platforms to test such things. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] testing question for code that calls abstract methods (like ChromeClient)
What's a good strategy for testing WebKit code that calls out to the ChromeClient interface or some other abstract interface? I can't find anything obvious in the codebase-- is there any way to mock out those calls for unit tests? Thanks, -John ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] testing changes to UString/String
On Jan 12, 2009, at 1:40 PM, Geoffrey Garen wrote: Hi Dave. We tend to write unit tests in HTML or JavaScript. If it's possible to trigger all of the relevant test conditions in HTML or JavaScript, I'd recommend that. The next best thing to do is to write a C++ test function, add it to WebKitTools/DumpRenderTree, and invoke it during DumpRenderTree's initialization. This solution is not as good because you can't run it inside a web browser, and a failure in one test will cause all unit tests to appear to fail. As a last resort, you could create a new build target specifically for your unit test. This solution is not as good because most WebKit developers will not run your test as a part of their daily development. We've discussed the possibility of adopting a C++ unit testing framework in the past. String and UString are cases where this kind of approach would really shine. It would also be great for basic data structure classes like Vector and HashMap. We haven't made much progress on this front, although Eric Seidel made a preliminary patch a while back to adopt the same unit test framework that Chrome uses, although it got hung up on build issues. Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] testing changes to UString/String
Hi Dave. We tend to write unit tests in HTML or JavaScript. If it's possible to trigger all of the relevant test conditions in HTML or JavaScript, I'd recommend that. The next best thing to do is to write a C++ test function, add it to WebKitTools/DumpRenderTree, and invoke it during DumpRenderTree's initialization. This solution is not as good because you can't run it inside a web browser, and a failure in one test will cause all unit tests to appear to fail. As a last resort, you could create a new build target specifically for your unit test. This solution is not as good because most WebKit developers will not run your test as a part of their daily development. Cheers, Geoff On Jan 11, 2009, at 12:36 PM, David Levin wrote: I'm refactoring and improving UString and String. I want to write tests for these classes. What's the best way to do this? Thanks, Dave ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] testing changes to UString/String
I'm refactoring and improving UString and String. I want to write tests for these classes. What's the best way to do this? Thanks, Dave ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing - Compilating Webcore
Hi all, I have 2 questions in the compilation :- Q1: I downloaded yesterday the CVS source of webkittools, called: webkittools/checkout called: update-webkit and started the compilation by: build-webkit 4 errors came in Javascriptcore Q2: if i replace the downloaded Javascriptcore and webcore with the ones from iphone and start the build, is it the correct way to get the newly compiled webcore? or is there a mistake. thanks in advace, regards, Sherif ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Testing - Compilating Webcore
On Wed, 10/8/08, Sherif <[EMAIL PROTECTED]> wrote: > Could any body please help me in compiling the webcore? > I ve macos, 10.5 and xcode. Only the souce code of Webcore > in a folder. > > Q1: why Should i compile javascriptcore first before > compiling webcore? > This is a note in the readme but i don't understand > why. JavaScriptCore must be built first because WebCore depends on it. > Q2: What is the correct parameter to compile the webcore > using xcodebuild? See WebKitTools/Scripts/build-webkit or run "xcodebuild -help" or build the project from within Xcode. Dave ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] Testing - Compilating Webcore
Hello guys, Could any body please help me in compiling the webcore? I ve macos, 10.5 and xcode. Only the souce code of Webcore in a folder. Q1: why Should i compile javascriptcore first before compiling webcore? This is a note in the readme but i don't understand why. Q2: What is the correct parameter to compile the webcore using xcodebuild? thanks in advance, regards, Sherif ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
