Re: [webkit-dev] Testing iOS/WK1?

2018-10-24 Thread Frédéric Wang
On 23/10/2018 17:44, Ali Juma wrote:
> Try passing the argument "-1":
>
> run-webkit-tests --ios-simulator -1

Great thanks for the hint, I think this --dump-render-tree could be
useful indeed to check WK1.

However, in the particular scrolling tests I'm considering, I'm relying
a lot on testRunner.runUIScript to execute basic gestures or emulate the
find UI and most of these APIs are not implemented (easily
implementable?) in Tools/DumpRenderTree/ios/UIScriptControllerIOS.mm

-- 
Frédéric Wang - frederic-wang.fr

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing iOS/WK1?

2018-10-23 Thread Ali Juma
On Tue, Oct 23, 2018 at 11:39 AM Frédéric Wang  wrote:

> Hi all,
>
> During the WebKit Contributors Meeting, Simon Fraser suggested that I
> should verify some of the my patches using WK1. Hence I did some
> experiments (again):
>
> * When I run run-webkit-tests, whatever the value specified via
> --platform, the selected expectation is always a "wk2" file and the code
> executed is from WebKit (i.e. WK2 == non-legacy).
>

Try passing the argument "-1":

run-webkit-tests --ios-simulator -1
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


[webkit-dev] Testing iOS/WK1?

2018-10-23 Thread Frédéric Wang
Hi all,

During the WebKit Contributors Meeting, Simon Fraser suggested that I
should verify some of the my patches using WK1. Hence I did some
experiments (again):

* When I run run-webkit-tests, whatever the value specified via
--platform, the selected expectation is always a "wk2" file and the code
executed is from WebKit (i.e. WK2 == non-legacy).

* I wrote a dummy iOS app using the deprecated UIWebView [1] and tried
to launch it with Tools/Scripts/run-webkit-app --ios-simulator
/path/to/WebKitLegacyApp.app ; that seems to be working (code from
WebKitLegacy is indeed executed). Also, page is loaded and properly
rendered at startup so I can somewhat test it. However, the app does not
reply to user interaction so it's difficult to do manual testing (not to
mention features like the Find UI are not available).

Also, I have not checked but IIRC the EWS bots actually only run iOS/WK2
tests, right?

Any idea/suggestion regarding how one could test iOS/WK1?

PS: I'm using the public Xcode SDK. In particular I only have access to
the simulator.

[1] https://github.com/fred-wang/WebKitLegacyApp

-- 
Frédéric Wang - frederic-wang.fr


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


[webkit-dev] Testing feature suggestion: Custom measurable data @ Performance Tests

2013-02-15 Thread Bruno Abinader
Hi all,

I'd like to suggest us to provide ways to aggregate custom measurable data
to Performance Test results (i.e. as seen on perf-o-matic tool). As example
of a custom measurable data, I'd suggest counting the amount of OpenGL
context-summarized calls per frame. This kind of data is useful when
evaluating performance impact of changes related to hardware-accelerated
drawing code (i.e. TextureMapperLayer). Along with this example, other
OpenGL-related data also apply (no. of state changes - specially talking
about the redundant ones). I'm aware each platform may or may not benefit
much from lesser OpenGL calls, but as a general guide, the lesser the
better.

This kind of information can be obtained with tools like apitrace [1], for
instance (at least on Linux), but may also be provided from other tools
depending on the platform. As done in WebCore/WebKit, platofrm-specific
implementations of the data logger can share a common interface. I haven't
analyzed the perftestsrunner code to check if a similar approach to custom
data is already available, so I'd kindly ask for your feedback on the
subject (whether it seems like a valid approach).

Best regards,

Links:
[1] http://apitrace.github.com

-- 
Bruno de Oliveira Abinader
Software Engineer @ basysKom GmbH
WebKit committer / Nokia Certified Qt Specialist
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-14 Thread Dirk Pranke
On Wed, Feb 13, 2013 at 11:33 PM, Benjamin Poulain benja...@webkit.org wrote:
 On Wed, Feb 13, 2013 at 11:16 PM, Dirk Pranke dpra...@chromium.org wrote:

  Those changes are not harmless. There are people monitoring tests
  results
  full time in order to keep WebKit in good shape. No other part of WebKit
  require continuous attention.

 I'm sorry, but either I don't understand Dongsung's suggestion, or I
 don't understand your criticism. What does this sentence paragraph
 mean? Are you suggesting someone needs to look at this type of test
 full time in a way that we don't have to look at the other tests?


 My lack of sleep is likely to blame here :)

 All I am saying is tests have a hight cost when they start failing, we
 should be careful how we design them so we can easily triage bad tests,
 flaky tests, and real errors.


No argument here :).


  Case 1: CSS Filters  Shaders
  I wanted this test functionality when I commented
  http://webkit.org/b/97859#c19
  If I want to make gaussian blur test, I prefer using 'getPixel' test as
  follows,
 
 
  Why wasn't a ref-test a better solution in this particular case?
 

 I'm curious, what would you imagine the ref test contains?


 If I am not mistaken, the composition operations are parallel with the ones
 of SVG and Canvas (aren't they?).
 I would have attempted comparing the 3 implementations as it seems to me the
 pixels values should be the same.

 That was a genuine question about that case :)



  Case 2: Fixed Position Element
  [...]
  function repeatedlyCalledDuringScrolling() {
  ASSERT(getPixel(15, 9) == white);
  ASSERT(getPixel(15, 10) == green);
  ASSERT(getPixel(9, 15) == white);
  ASSERT(getPixel(10, 15) == green);
  
  }
 
 
  I think this shows what I said about correctness and readability:
  -Asserting the correctness of the test and the result becomes close to
  impossible for the reader. One has to review the full code to have a
  chance
  of understanding an error.
  -You cannot cover non trivial cases (images, text, form elements, etc).
  -And it is inefficient. You have to render each frame on the UIProcess,
  move
  it to the WebProcess, and box it for JavaScript to process (with pixel
  format conversions depending on your graphics system)
 
  Of the ideas raised, I think this is one of my least favorite for
  testing
  fixed positioning.
 

 Isn't his suggestion the equivalent of what we do today in text-only
 tests? i.e., printing pass or fail and making you have to look at
 the test itself to see what's being tested?

 If the correctness of the rendering depends on those 4 specific pixels
 having those four specific values, how exactly are you going to verify
 that by looking at it?

 Again, I think I'm just not understanding you here?


 When looking at a test test, you follow the flow to know what it is supposed
 to do and where it breaks.

 How are you supposed to know, _by reading the code_, that the color at
 position 15, 9 should be white?


is the code the test code, or the source code we're testing? Of
course, without any larger context, those assertions are impossible to
interpret. I figured he was just indicating what he meant by a
programmatic check, and a real test case would be more informative.

And of course it's easy to write bad tests that are neither readable
nor maintainable. But we also have lots of pixel tests that are
difficult for people not intimately familiar with the code being
tested to evaluate for correctness, too.

Lastly, I don't think he was suggesting that this would replace all
pixel tests; as you say, it is more well suited for some times of
tests than others.

As far as efficiency goes, even moving the picture back and forth
between two processes is still going to be far faster than running a
second reftest and comparing the output of the two pages.

-- Dirk
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-14 Thread Dongsung Huang


 I'm curious, what would you imagine the ref test contains?


 If I am not mistaken, the composition operations are parallel with the
 ones of SVG and Canvas (aren't they?).
 I would have attempted comparing the 3 implementations as it seems to me
 the pixels values should be the same.

 That was a genuine question about that case :)


Thank you for feedback! :)
I presented gaussian blur case to express my opinion effectively.

As I know, Adobe want to draw CSS Filters and SVG Filters using the same
code. And Canvas does not have something like Filters.
What I want to say is that there are some cases in which snapshot
functionality is more useful.


  Case 2: Fixed Position Element
  [...]
  function repeatedlyCalledDuringScrolling() {
  ASSERT(getPixel(15, 9) == white);
  ASSERT(getPixel(15, 10) == green);
  ASSERT(getPixel(9, 15) == white);
  ASSERT(getPixel(10, 15) == green);
  
  }
 
 
  I think this shows what I said about correctness and readability:
  -Asserting the correctness of the test and the result becomes close to
  impossible for the reader. One has to review the full code to have a
 chance
  of understanding an error.
  -You cannot cover non trivial cases (images, text, form elements, etc).
  -And it is inefficient. You have to render each frame on the UIProcess,
 move
  it to the WebProcess, and box it for JavaScript to process (with pixel
  format conversions depending on your graphics system)
 
  Of the ideas raised, I think this is one of my least favorite for
 testing
  fixed positioning.
 

 Isn't his suggestion the equivalent of what we do today in text-only
 tests? i.e., printing pass or fail and making you have to look at
 the test itself to see what's being tested?

 If the correctness of the rendering depends on those 4 specific pixels
 having those four specific values, how exactly are you going to verify
 that by looking at it?

 Again, I think I'm just not understanding you here?


 When looking at a test test, you follow the flow to know what it is
 supposed to do and where it breaks.

 How are you supposed to know, _by reading the code_, that the color at
 position 15, 9 should be white?

 Benjamin



I agree on Dirk's opinion. I supports 3 points as follows:
1. we can help to understand what's goal of test by comments
2. when we want to see the graphical result, we can use MiniBrowser like
other text based test.
3. philip's canvas tests (http://philip.html5.org/tests/canvas/suite/tests/)
already use well getImageData to test.

I agree that snapshot test can make hardly maintainable tests, but we have
review process and we can make succinct tests like philip canvas test.
And when pixel test is more suitable, we can use pixel test also.

Best regards,

Dongsung Huang
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-13 Thread Dongsung Huang
I like this idea. I cannot find any harm if we have this functionality.
When a pixel test is more succinct, we can make a pixel test. When a
'getPixel on Javascript' test is more succinct, we can make a 'getPixel'
test.
As
http://trac.webkit.org/wiki/Writing%20Layout%20Tests%20for%20DumpRenderTree
said
'Pixel tests are a burden on every ports as any small change in the engine
could lead to your test needing a new pixel result.', new test
functionality that can be an alternative of pixel test is good!
philip canvas tests (LayoutTests/canvas/philip) has no pixel test because
of the power of getImageData API in canvas spec.

I can put two examples for the power of functionality that Noam suggested,

Case 1: CSS Filters  Shaders
I wanted this test functionality when I commented
http://webkit.org/b/97859#c19
If I want to make gaussian blur test, I prefer using 'getPixel' test as
follows,

ASSERT(getPixel(10, 10) = rgba(255, 0, 0, 255));
ASSERT(getPixel(10, 15)  rgba(255, 0, 0, 98)  getPixel(10, 15) 
rgba(255, 0, 0, 102));

If we don't have 'getPixel', we need to capture all port's image results
because all port's painting engine would make a bit different result. As
you know, it is painful.


Case 2: Fixed Position Element
I guess it is one of reasons why Noam suggested.
Currently, Qt and EFL AC (a.k.a Coordinated Graphics) has a problem related
to fixed position element.
During panning animation, Fixed position element is slightly flickering.

If we have 'getPixel', we can make a succinct test case.

PRECONDITION : a green fixed position div is on the white body. The rect of
the div is rect(10, 10, 30, 30).
function repeatedlyCalledDuringScrolling() {
ASSERT(getPixel(15, 9) == white);
ASSERT(getPixel(15, 10) == green);
ASSERT(getPixel(9, 15) == white);
ASSERT(getPixel(10, 15) == green);

}

WDYT?

Best regards,

Dongsung Huang
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-13 Thread Benjamin Poulain
On Wed, Feb 13, 2013 at 9:38 PM, Dongsung Huang luxte...@company100.netwrote:

 I like this idea. I cannot find any harm if we have this functionality.


Those changes are not harmless. There are people monitoring tests results
full time in order to keep WebKit in good shape. No other part of WebKit
require continuous attention.


 Case 1: CSS Filters  Shaders
 I wanted this test functionality when I commented
 http://webkit.org/b/97859#c19
 If I want to make gaussian blur test, I prefer using 'getPixel' test as
 follows,


Why wasn't a ref-test a better solution in this particular case?

Case 2: Fixed Position Element
 [...]
 function repeatedlyCalledDuringScrolling() {
 ASSERT(getPixel(15, 9) == white);
 ASSERT(getPixel(15, 10) == green);
 ASSERT(getPixel(9, 15) == white);
 ASSERT(getPixel(10, 15) == green);
 
 }


I think this shows what I said about correctness and readability:
-Asserting the correctness of the test and the result becomes close to
impossible for the reader. One has to review the full code to have a chance
of understanding an error.
-You cannot cover non trivial cases (images, text, form elements, etc).
-And it is inefficient. You have to render each frame on the UIProcess,
move it to the WebProcess, and box it for JavaScript to process (with pixel
format conversions depending on your graphics system)

Of the ideas raised, I think this is one of my least favorite for testing
fixed positioning.

Benjamin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-13 Thread noam . rosenthal


From: ext Benjamin Poulain benja...@webkit.orgmailto:benja...@webkit.org
Date: Thursday, February 14, 2013 8:07 AM
To: Dongsung Huang luxte...@company100.netmailto:luxte...@company100.net
Cc: webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org 
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org
Subject: Re: [webkit-dev] Testing feature suggestion: animation/interaction 
pixel-results on the fly

On Wed, Feb 13, 2013 at 9:38 PM, Dongsung Huang 
luxte...@company100.netmailto:luxte...@company100.net wrote:
I like this idea. I cannot find any harm if we have this functionality.

Those changes are not harmless. There are people monitoring tests results full 
time in order to keep WebKit in good shape. No other part of WebKit require 
continuous attention.

Case 1: CSS Filters  Shaders
I wanted this test functionality when I commented http://webkit.org/b/97859#c19
If I want to make gaussian blur test, I prefer using 'getPixel' test as follows,

Why wasn't a ref-test a better solution in this particular case?
Because gaussian blurs on the GPU are not accurate, and look slightly different 
on different GPUs, but usually close enough.
We need a way to measure close enough for features where all you can get is 
close enough! Ref-tests and pixel-tests are way to rigid for that, and 
require constant rebaselines and headaches.


Case 2: Fixed Position Element
[...]
function repeatedlyCalledDuringScrolling() {
ASSERT(getPixel(15, 9) == white);
ASSERT(getPixel(15, 10) == green);
ASSERT(getPixel(9, 15) == white);
ASSERT(getPixel(10, 15) == green);

}

I think this shows what I said about correctness and readability:
-Asserting the correctness of the test and the result becomes close to 
impossible for the reader. One has to review the full code to have a chance of 
understanding an error.
We can generate PNGs when a snapshot is requested so that the tester can 
eyeball the results.

-You cannot cover non trivial cases (images, text, form elements, etc).
-And it is inefficient. You have to render each frame on the UIProcess, move it 
to the WebProcess, and box it for JavaScript to process (with pixel format 
conversions depending on your graphics system)
Another option is to do expose a partial comparison mechanism that runs in  the 
UI process.
E.g.
var snapshotHandle = testRunner.createSnapshot();
testRunner.comparePixel(snapshotHandle, 15, 20, 'white', { tolerance: 0.2 });

This way we can generate a visual representation of failures.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-13 Thread Benjamin Poulain
On Wed, Feb 13, 2013 at 11:16 PM, Dirk Pranke dpra...@chromium.org wrote:

  Those changes are not harmless. There are people monitoring tests results
  full time in order to keep WebKit in good shape. No other part of WebKit
  require continuous attention.

 I'm sorry, but either I don't understand Dongsung's suggestion, or I
 don't understand your criticism. What does this sentence paragraph
 mean? Are you suggesting someone needs to look at this type of test
 full time in a way that we don't have to look at the other tests?


My lack of sleep is likely to blame here :)

All I am saying is tests have a hight cost when they start failing, we
should be careful how we design them so we can easily triage bad tests,
flaky tests, and real errors.


 Case 1: CSS Filters  Shaders
  I wanted this test functionality when I commented
  http://webkit.org/b/97859#c19
  If I want to make gaussian blur test, I prefer using 'getPixel' test as
  follows,
 
 
  Why wasn't a ref-test a better solution in this particular case?
 

 I'm curious, what would you imagine the ref test contains?


If I am not mistaken, the composition operations are parallel with the ones
of SVG and Canvas (aren't they?).
I would have attempted comparing the 3 implementations as it seems to me
the pixels values should be the same.

That was a genuine question about that case :)



  Case 2: Fixed Position Element
  [...]
  function repeatedlyCalledDuringScrolling() {
  ASSERT(getPixel(15, 9) == white);
  ASSERT(getPixel(15, 10) == green);
  ASSERT(getPixel(9, 15) == white);
  ASSERT(getPixel(10, 15) == green);
  
  }
 
 
  I think this shows what I said about correctness and readability:
  -Asserting the correctness of the test and the result becomes close to
  impossible for the reader. One has to review the full code to have a
 chance
  of understanding an error.
  -You cannot cover non trivial cases (images, text, form elements, etc).
  -And it is inefficient. You have to render each frame on the UIProcess,
 move
  it to the WebProcess, and box it for JavaScript to process (with pixel
  format conversions depending on your graphics system)
 
  Of the ideas raised, I think this is one of my least favorite for testing
  fixed positioning.
 

 Isn't his suggestion the equivalent of what we do today in text-only
 tests? i.e., printing pass or fail and making you have to look at
 the test itself to see what's being tested?

 If the correctness of the rendering depends on those 4 specific pixels
 having those four specific values, how exactly are you going to verify
 that by looking at it?

 Again, I think I'm just not understanding you here?


When looking at a test test, you follow the flow to know what it is
supposed to do and where it breaks.

How are you supposed to know, _by reading the code_, that the color at
position 15, 9 should be white?

Benjamin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-13 Thread Benjamin Poulain
On Wed, Feb 13, 2013 at 11:27 PM, noam.rosent...@nokia.com wrote:

  Why wasn't a ref-test a better solution in this particular case?

  Because gaussian blurs on the GPU are not accurate, and look slightly
 different on different GPUs, but usually close enough.
 We need a way to measure close enough for features where all you can get
 is close enough! Ref-tests and pixel-tests are way to rigid for that, and
 require constant rebaselines and headaches.


There is _NO_ gaussian blur in that patch.

Benjamin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-13 Thread noam . rosenthal


From: ext Benjamin Poulain benja...@webkit.orgmailto:benja...@webkit.org
Date: Thursday, February 14, 2013 8:36 AM
To: Noam Rosenthal noam.rosent...@nokia.commailto:noam.rosent...@nokia.com
Cc: luxte...@company100.netmailto:luxte...@company100.net 
luxte...@company100.netmailto:luxte...@company100.net, 
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org 
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org
Subject: Re: [webkit-dev] Testing feature suggestion: animation/interaction 
pixel-results on the fly

On Wed, Feb 13, 2013 at 11:27 PM, 
noam.rosent...@nokia.commailto:noam.rosent...@nokia.com wrote:
Why wasn't a ref-test a better solution in this particular case?
Because gaussian blurs on the GPU are not accurate, and look slightly different 
on different GPUs, but usually close enough.
We need a way to measure close enough for features where all you can get is 
close enough! Ref-tests and pixel-tests are way to rigid for that, and 
require constant rebaselines and headaches.

There is _NO_ gaussian blur in that patch.
DongSung mentioned gaussian blur in the comment, which is what I was referring 
to.
But other bugs as well, like delegated scrolling, also have that problem of 
results not being 100% the same every time.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-10 Thread Ryosuke Niwa
On Sat, Feb 9, 2013 at 5:35 PM, Dirk Pranke dpra...@chromium.org wrote:

 Perhaps -- and this is a tangent to this thread -- we could also
 investigate ways of writing tests that did render pngs but told NRWT
 to ignore them (e.g., dumpAsTextAndIgnoredImage() or something), or
 conveyed which parts of the image were important ...


What's the point of that?

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-09 Thread noam . rosenthal


From: ext Benjamin Poulain benja...@webkit.orgmailto:benja...@webkit.org
Date: Saturday, February 9, 2013 12:52 AM
To: Noam Rosenthal noam.rosent...@nokia.commailto:noam.rosent...@nokia.com
Cc: rn...@webkit.orgmailto:rn...@webkit.org 
rn...@webkit.orgmailto:rn...@webkit.org, 
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org 
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org
Subject: Re: [webkit-dev] Testing feature suggestion: animation/interaction 
pixel-results on the fly

On Fri, Feb 8, 2013 at 3:16 PM, 
noam.rosent...@nokia.commailto:noam.rosent...@nokia.com wrote:
The problem with dynamic features of the web like animations/interactions is 
that they're non-deterministic, or at least a lot less deterministic than 
static features of the web like layouts.
Ref tests, pixel tests etc. are tools built for deterministic testing: load a 
file, take a snapshot, compare against a result. Testing an animation (or a 
filter) needs to feel a lot more dynamic and expressive: Animate green boxes, 
make sure that they're within a particular range at particular points in time.

The tests also have to be deterministic and comprehensive. I am afraid of 
loosing both with the Render-to-Canvas approach.

Can you give concrete examples of the kind of bugs you are hunting, and why 
testing cannot use the two methods suggested?
I did not say they cannot be tested with the two methods suggested :) It's more 
 about a preference to move some of those decisions from the test 
infrastructure to the test itself, but potentially those bugs could be tested 
in either way.
Examples for bugs I've encountered/reviewed and can use better in-motion 
testing (note that those are specific to Qt/EFL, but I'm sure there are tons of 
bugs like this that come up for Apple/Google as well)
http://trac.webkit.org/changeset/140825
http://trac.webkit.org/changeset/142112
http://trac.webkit.org/changeset/134953
https://bugs.webkit.org/show_bug.cgi?id=109179

Controlling the clock and programmatically sampling the end result would 
definitely make those more testable, but of course any progress in this area 
would be beneficial and my preference to a canvas-based API is more of an 
opinion.

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-09 Thread Benjamin Poulain
On Sat, Feb 9, 2013 at 1:24 AM, noam.rosent...@nokia.com wrote:

I did not say they cannot be tested with the two methods suggested :)
 It's more  about a preference to move some of those decisions from the test
 infrastructure to the test itself, but potentially those bugs could be
 tested in either way.
 Examples for bugs I've encountered/reviewed and can use better in-motion
 testing (note that those are specific to Qt/EFL, but I'm sure there are
 tons of bugs like this that come up for Apple/Google as well)
 http://trac.webkit.org/changeset/140825
 http://trac.webkit.org/changeset/142112
 http://trac.webkit.org/changeset/134953
 https://bugs.webkit.org/show_bug.cgi?id=109179

  Controlling the clock and programmatically sampling the end result would
 definitely make those more testable, but of course any progress in this
 area would be beneficial and my preference to a canvas-based API is more of
 an opinion.


To explain my concerns:
Sometime, I look at a failing test, and think: what the f**k is this
supposed to test?. Then I have to dig a ton of logs, and finally read the
change to understand a the single JS statement in the whole script that
make the test useful.

This is the situation I am afraid with a solution where pixels would be
evaluated from JavaScript. You can test, but you lack visibility why
something is correct or incorrect. Text tests, ref-tests and pixel tests
have a great readability, you can quickly evaluate their correctness. This
is important in my opinion, I don't think we want more opaque output like
the RenderTree dump.

I am not against your plan if the readability of the tests can be good. I'd
be happy to review patches toward testing dynamic changes in webpages.

Benjamin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-09 Thread noam . rosenthal


From: ext Benjamin Poulain benja...@webkit.orgmailto:benja...@webkit.org
This is the situation I am afraid with a solution where pixels would be 
evaluated from JavaScript. You can test, but you lack visibility why something 
is correct or incorrect. Text tests, ref-tests and pixel tests have a great 
readability, you can quickly evaluate their correctness. This is important in 
my opinion, I don't think we want more opaque output like the RenderTree dump.

I am not against your plan if the readability of the tests can be good. I'd be 
happy to review patches toward testing dynamic changes in webpages.
We're on the same page. I'm afraid of the same thing… Let's talk specifics when 
a patch is available to review.

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-09 Thread noam . rosenthal
Since people seem to agree that this is a good problem to solve, I've created a 
bug for it: https://bugs.webkit.org/show_bug.cgi?id=109356
I can't promise to fix it myself right this moment as I'm spread a bit too 
thin, but if someone wants to help or pick it up before I get to it please feel 
free :)
Noam

From: ext Benjamin Poulain benja...@webkit.orgmailto:benja...@webkit.org
Date: Saturday, February 9, 2013 10:57 AM
To: Noam Rosenthal noam.rosent...@nokia.commailto:noam.rosent...@nokia.com
Cc: rn...@webkit.orgmailto:rn...@webkit.org 
rn...@webkit.orgmailto:rn...@webkit.org, 
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org 
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org
Subject: Re: [webkit-dev] Testing feature suggestion: animation/interaction 
pixel-results on the fly

On Sat, Feb 9, 2013 at 1:24 AM, 
noam.rosent...@nokia.commailto:noam.rosent...@nokia.com wrote:
I did not say they cannot be tested with the two methods suggested :) It's more 
 about a preference to move some of those decisions from the test 
infrastructure to the test itself, but potentially those bugs could be tested 
in either way.
Examples for bugs I've encountered/reviewed and can use better in-motion 
testing (note that those are specific to Qt/EFL, but I'm sure there are tons of 
bugs like this that come up for Apple/Google as well)
http://trac.webkit.org/changeset/140825
http://trac.webkit.org/changeset/142112
http://trac.webkit.org/changeset/134953
https://bugs.webkit.org/show_bug.cgi?id=109179

Controlling the clock and programmatically sampling the end result would 
definitely make those more testable, but of course any progress in this area 
would be beneficial and my preference to a canvas-based API is more of an 
opinion.

To explain my concerns:
Sometime, I look at a failing test, and think: what the f**k is this supposed 
to test?. Then I have to dig a ton of logs, and finally read the change to 
understand a the single JS statement in the whole script that make the test 
useful.

This is the situation I am afraid with a solution where pixels would be 
evaluated from JavaScript. You can test, but you lack visibility why something 
is correct or incorrect. Text tests, ref-tests and pixel tests have a great 
readability, you can quickly evaluate their correctness. This is important in 
my opinion, I don't think we want more opaque output like the RenderTree dump.

I am not against your plan if the readability of the tests can be good. I'd be 
happy to review patches toward testing dynamic changes in webpages.

Benjamin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-09 Thread Simon Fraser
The existing pauseAnimation DRT API can stop an animation mid-flight, to allow 
for reliable snapshotting. Why isn't this enough?

Simon

On Feb 9, 2013, at 8:44 AM, noam.rosent...@nokia.com wrote:

 Since people seem to agree that this is a good problem to solve, I've created 
 a bug for it: https://bugs.webkit.org/show_bug.cgi?id=109356
 I can't promise to fix it myself right this moment as I'm spread a bit too 
 thin, but if someone wants to help or pick it up before I get to it please 
 feel free :)
 Noam
 
 From: ext Benjamin Poulain benja...@webkit.org
 Date: Saturday, February 9, 2013 10:57 AM
 To: Noam Rosenthal noam.rosent...@nokia.com
 Cc: rn...@webkit.org rn...@webkit.org, webkit-dev@lists.webkit.org 
 webkit-dev@lists.webkit.org
 Subject: Re: [webkit-dev] Testing feature suggestion: animation/interaction 
 pixel-results on the fly
 
 On Sat, Feb 9, 2013 at 1:24 AM, noam.rosent...@nokia.com wrote:
 
 I did not say they cannot be tested with the two methods suggested :) It's 
 more  about a preference to move some of those decisions from the test 
 infrastructure to the test itself, but potentially those bugs could be 
 tested in either way.
 Examples for bugs I've encountered/reviewed and can use better in-motion 
 testing (note that those are specific to Qt/EFL, but I'm sure there are 
 tons of bugs like this that come up for Apple/Google as well)
 http://trac.webkit.org/changeset/140825
 http://trac.webkit.org/changeset/142112
 http://trac.webkit.org/changeset/134953
 https://bugs.webkit.org/show_bug.cgi?id=109179
 
 Controlling the clock and programmatically sampling the end result would 
 definitely make those more testable, but of course any progress in this 
 area would be beneficial and my preference to a canvas-based API is more of 
 an opinion.
 
 To explain my concerns:
 Sometime, I look at a failing test, and think: what the f**k is this 
 supposed to test?. Then I have to dig a ton of logs, and finally read the 
 change to understand a the single JS statement in the whole script that make 
 the test useful. 
 
 This is the situation I am afraid with a solution where pixels would be 
 evaluated from JavaScript. You can test, but you lack visibility why 
 something is correct or incorrect. Text tests, ref-tests and pixel tests 
 have a great readability, you can quickly evaluate their correctness. This 
 is important in my opinion, I don't think we want more opaque output like 
 the RenderTree dump.
 
 I am not against your plan if the readability of the tests can be good. I'd 
 be happy to review patches toward testing dynamic changes in webpages.
 
 Benjamin
 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-09 Thread noam . rosenthal
Oops, forgot to reply to the list.

From: Noam Rosenthal noam.rosent...@nokia.commailto:noam.rosent...@nokia.com
From: ext Simon Fraser simon.fra...@apple.commailto:simon.fra...@apple.com
The existing pauseAnimation DRT API can stop an animation mid-flight, to allow 
for reliable snapshotting. Why isn't this enough?
I would say that the main things that are missing are:

  1.  Sample and compare several images in the same test
  2.  Sample images heuristically, e.g. sample that a green box is close 
enough to a particular location – A full ImageDiff is very rigid in that way.
  3.  Sample images mid-flight without pausing. Pausing can mess with animation 
timing and thus skew the results.

I don't have a full list of what's missing – but I know for sure that when 
trying to hunt mid-flight bugs in the past I was not able to create sufficient 
tests with the current DRT tools.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-09 Thread Dirk Pranke
On Sat, Feb 9, 2013 at 1:57 AM, Benjamin Poulain benja...@webkit.org wrote:
 On Sat, Feb 9, 2013 at 1:24 AM, noam.rosent...@nokia.com wrote:

 I did not say they cannot be tested with the two methods suggested :) It's
 more  about a preference to move some of those decisions from the test
 infrastructure to the test itself, but potentially those bugs could be
 tested in either way.
 Examples for bugs I've encountered/reviewed and can use better in-motion
 testing (note that those are specific to Qt/EFL, but I'm sure there are tons
 of bugs like this that come up for Apple/Google as well)
 http://trac.webkit.org/changeset/140825
 http://trac.webkit.org/changeset/142112
 http://trac.webkit.org/changeset/134953
 https://bugs.webkit.org/show_bug.cgi?id=109179

 Controlling the clock and programmatically sampling the end result would
 definitely make those more testable, but of course any progress in this area
 would be beneficial and my preference to a canvas-based API is more of an
 opinion.


 To explain my concerns:
 Sometime, I look at a failing test, and think: what the f**k is this
 supposed to test?. Then I have to dig a ton of logs, and finally read the
 change to understand a the single JS statement in the whole script that make
 the test useful.

 This is the situation I am afraid with a solution where pixels would be
 evaluated from JavaScript. You can test, but you lack visibility why
 something is correct or incorrect. Text tests, ref-tests and pixel tests
 have a great readability, you can quickly evaluate their correctness. This
 is important in my opinion, I don't think we want more opaque output like
 the RenderTree dump.

 I am not against your plan if the readability of the tests can be good. I'd
 be happy to review patches toward testing dynamic changes in webpages.


A counter-argument, of course, that there are a lot of pixel tests
that would be a lot better as text-only tests that were verifying
certain aspects of how the page rendered programmatically.

Perhaps -- and this is a tangent to this thread -- we could also
investigate ways of writing tests that did render pngs but told NRWT
to ignore them (e.g., dumpAsTextAndIgnoredImage() or something), or
conveyed which parts of the image were important ...

-- Dirk
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


[webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread noam . rosenthal
Hellos

we don't currently have a solution in webkit's test infrastructure for testing 
animations on the fly, or in general for testing multiple image results in 
the same test.
(If this was discussed before and I'm unaware of that discussion, please stop 
me here...)

This is because ImageDiff works on the single-test level, and you can't 
explicitly make image comparisons in the same test at different points in time.
This has before caused us several regressions that were not caught by layout 
tests.

I'd like to propose a solution, and would welcome some feedback on whether it's 
a good one...
The idea is that you would be able to programatically retrieve the current 
snapshot into a canvas ImageData, and then compare the pixel results with 
JavaScript in the LayoutTest. Something like:

window.testRunner.getWindowSnapshot(function(snapshot) {
if (snapshot.data[snapshot.width * 100 + 10] != 127)
output.innerHTML = FAIL;
});

WDYT?
Noam
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread Robert Kroeger
Questions inline.

On Fri, Feb 8, 2013 at 8:35 AM, noam.rosent...@nokia.com wrote:

 Hellos

 we don't currently have a solution in webkit's test infrastructure for
 testing animations on the fly, or in general for testing multiple image
 results in the same test.

I have been wishing for such a thing around how to test scroll animations.

 (If this was discussed before and I'm unaware of that discussion, please
 stop me here...)

 This is because ImageDiff works on the single-test level, and you can't
 explicitly make image comparisons in the same test at different points in
 time.
 This has before caused us several regressions that were not caught by
 layout tests.

 I'd like to propose a solution, and would welcome some feedback on whether
 it's a good one...
 The idea is that you would be able to programatically retrieve the current
 snapshot into a canvas ImageData, and then compare the pixel results with
 JavaScript in the LayoutTest. Something like:

 window.testRunner.getWindowSnapshot(function(snapshot) {
 if (snapshot.data[snapshot.width * 100 + 10] != 127)
 output.innerHTML = FAIL;
 });

 WDYT?

How would the test wait for the animation to happen? Tests that use
timeouts seem more likely to be flaky from what I've seen. Could
window.testRunner.getWindowSnapshot take a time delta of some kind?

Rob.

 Noam

 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread noam . rosenthal


From: webkit-dev-boun...@lists.webkit.org [webkit-dev-boun...@lists.webkit.org] 
on behalf of ext Robert Kroeger [rjkro...@chromium.org]

 How would the test wait for the animation to happen? Tests that use
 timeouts seem more likely to be flaky from what I've seen. Could
 window.testRunner.getWindowSnapshot take a time delta of some kind?
Most of the bugs I've encountered in this area can be caught by a test that 
would have obvious results that can buffer the flakiness, e.g. by moving a 
big green rectangle and testing only its center.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread Michelangelo De Simone
 I'd like to propose a solution, and would welcome some feedback on whether 
 it's a good one...
 The idea is that you would be able to programatically retrieve the current 
 snapshot into a canvas ImageData, and then compare the pixel results with 
 JavaScript in the LayoutTest. Something like:

+1
Lovely idea. For instance, we've been having some issue [1] testing 
CustomFilters, so having the ability to /sample/ the final result from JS would 
be great.

[1] https://bugs.webkit.org/show_bug.cgi?id=104012

--
Bye,
Michelangelo
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread Ryosuke Niwa
On Fri, Feb 8, 2013 at 5:35 AM, noam.rosent...@nokia.com wrote:

  we don't currently have a solution in webkit's test infrastructure for
 testing animations on the fly, or in general for testing multiple image
 results in the same test.
 (If this was discussed before and I'm unaware of that discussion, please
 stop me here...)

 This is because ImageDiff works on the single-test level, and you can't
 explicitly make image comparisons in the same test at different points in
 time.
 This has before caused us several regressions that were not caught by
 layout tests.

 I'd like to propose a solution, and would welcome some feedback on whether
 it's a good one...
 The idea is that you would be able to programatically retrieve the current
 snapshot into a canvas ImageData, and then compare the pixel results with
 JavaScript in the LayoutTest. Something like:


I had similar thoughts but my counter proposal is to let DRT/WTR generate
multiple actual results either in the form of multiple layers in PNG or
multiple PNG images. The advantage of this latter approach is that we can
have multiple reference files as well. e.g. if a green box is to be moved
from (0,0) to (50,0) and to (100, 0), we could create three reference files
that correspond to each state.

But perhaps there is a good reason you didn't choose this approach. Could
you elaborate on the reason you picked this particular API?

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread Benjamin Poulain
On Fri, Feb 8, 2013 at 11:12 AM, Ryosuke Niwa rn...@webkit.org wrote:

 I had similar thoughts but my counter proposal is to let DRT/WTR
 generate multiple actual results either in the form of multiple layers in
 PNG or multiple PNG images. The advantage of this latter approach is that
 we can have multiple reference files as well. e.g. if a green box is to be
 moved from (0,0) to (50,0) and to (100, 0), we could create three reference
 files that correspond to each state.


This sounds like a good idea for programmatic updates. I would just argue
that instead of multiple images, it would be more maintainable to have
multiple html references; a kind of multiple-frame-ref-test.


A problem with this approach is, it is still fragile for animations. The
position of a given frame is depends on timing. For animations, we could
use mask-based test. The reference file of a test is a mask which gives an
area where pixels can be.
For example:
-Take a block in a fixed position, scroll the page. The mask is the block
in its original position. If for any frame, the block moved outside the
mask, the test fails.
-A block animating from left to right, while at the same time the page is
scrolled. There is a deterministic path where the object can be. If for any
frame, the block moved outside the path, the test fails.

Making the mask would not always be trivial. The advantage is, if any
single frame is wrong, we have a chance to catch it, regardless of the
timing.

Benjamin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread Dirk Pranke
On Fri, Feb 8, 2013 at 11:12 AM, Ryosuke Niwa rn...@webkit.org wrote:
 On Fri, Feb 8, 2013 at 5:35 AM, noam.rosent...@nokia.com wrote:

 we don't currently have a solution in webkit's test infrastructure for
 testing animations on the fly, or in general for testing multiple image
 results in the same test.
 (If this was discussed before and I'm unaware of that discussion, please
 stop me here...)

 This is because ImageDiff works on the single-test level, and you can't
 explicitly make image comparisons in the same test at different points in
 time.
 This has before caused us several regressions that were not caught by
 layout tests.

 I'd like to propose a solution, and would welcome some feedback on whether
 it's a good one...
 The idea is that you would be able to programatically retrieve the current
 snapshot into a canvas ImageData, and then compare the pixel results with
 JavaScript in the LayoutTest. Something like:


 I had similar thoughts but my counter proposal is to let DRT/WTR generate
 multiple actual results either in the form of multiple layers in PNG or
 multiple PNG images. The advantage of this latter approach is that we can
 have multiple reference files as well. e.g. if a green box is to be moved
 from (0,0) to (50,0) and to (100, 0), we could create three reference files
 that correspond to each state.

 But perhaps there is a good reason you didn't choose this approach. Could
 you elaborate on the reason you picked this particular API?


I also appreciate the problem -- and would be interesting in solving
it -- but I'm a bit concerned that either proposal is at least
somewhat nondeterministic (or flaky), and I'm not sure how you'd be
able to control which frames of the animation you were getting well
enough. But I'm not much of expert here so I'm not sure how often it
matters.

-- Dirk
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread noam . rosenthal


From: ext Ryosuke Niwa rn...@webkit.orgmailto:rn...@webkit.org
I'd like to propose a solution, and would welcome some feedback on whether it's 
a good one...
The idea is that you would be able to programatically retrieve the current 
snapshot into a canvas ImageData, and then compare the pixel results with 
JavaScript in the LayoutTest. Something like:

I had similar thoughts but my counter proposal is to let DRT/WTR generate 
multiple actual results either in the form of multiple layers in PNG or 
multiple PNG images. The advantage of this latter approach is that we can have 
multiple reference files as well. e.g. if a green box is to be moved from (0,0) 
to (50,0) and to (100, 0), we could create three reference files that 
correspond to each state.

But perhaps there is a good reason you didn't choose this approach. Could you 
elaborate on the reason you picked this particular API?
The problem with dynamic features of the web like animations/interactions is 
that they're non-deterministic, or at least a lot less deterministic than 
static features of the web like layouts.
Ref tests, pixel tests etc. are tools built for deterministic testing: load a 
file, take a snapshot, compare against a result. Testing an animation (or a 
filter) needs to feel a lot more dynamic and expressive: Animate green boxes, 
make sure that they're within a particular range at particular points in time.
I think that when testing some of the dynamic features it's best to create a 
minimal mechanism such as retrieving snapshot pixels, giving more power to the 
tests themselves in a dynamic language like JavaScript, rather than try to 
force the tests to fit into a predefined set of features supported by the 
DRT/WTR infrastructure. Also this would allow some differences between the 
ports, which implement animations a bit differently, without constant 
rebaselines of reference results – e.g. A heuristic animation timing test may 
stay valid across platform even if the actual animation implementation is 
different.

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread Benjamin Poulain
On Fri, Feb 8, 2013 at 3:16 PM, noam.rosent...@nokia.com wrote:

  The problem with dynamic features of the web like
 animations/interactions is that they're non-deterministic, or at least a
 lot less deterministic than static features of the web like layouts.
 Ref tests, pixel tests etc. are tools built for deterministic testing:
 load a file, take a snapshot, compare against a result. Testing an
 animation (or a filter) needs to feel a lot more dynamic and expressive:
 Animate green boxes, make sure that they're within a particular range at
 particular points in time.


The tests also have to be deterministic and comprehensive. I am afraid of
loosing both with the Render-to-Canvas approach.

Can you give concrete examples of the kind of bugs you are hunting, and why
testing cannot use the two methods suggested?


 I think that when testing some of the dynamic features it's best to create
 a minimal mechanism such as retrieving snapshot pixels, giving more power
 to the tests themselves in a dynamic language like JavaScript, rather than
 try to force the tests to fit into a predefined set of features supported
 by the DRT/WTR infrastructure. Also this would allow some differences
 between the ports, which implement animations a bit differently, without
 constant rebaselines of reference results – e.g. A heuristic animation
 timing test may stay valid across platform even if the actual animation
 implementation is different.


Again, can you give concrete examples of this?

Benjamin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread Gregg Tavares
Can you expose a time or setTime function in DRT and some option that says
let JS control the clock?

Then a test could do something like

layoutTestController.overridePreference(JsControlledClock, 1);

   // Render 5 frames over 1 second
   var frame =0;
   function renderFrame() {
  layoutTestController.getWindowSnapshot(...);
  if (frame == 5) {
 finishTest();
 return;
  }
  ++frame;
  layoutTestController.setAnimationClock(frame * 1000 / 5);  // assumes
clock is in ms.
  requestAnimationFrame(renderFrame);
  }



On Fri, Feb 8, 2013 at 3:52 PM, Benjamin Poulain benja...@webkit.orgwrote:

 On Fri, Feb 8, 2013 at 3:16 PM, noam.rosent...@nokia.com wrote:

  The problem with dynamic features of the web like
 animations/interactions is that they're non-deterministic, or at least a
 lot less deterministic than static features of the web like layouts.
 Ref tests, pixel tests etc. are tools built for deterministic testing:
 load a file, take a snapshot, compare against a result. Testing an
 animation (or a filter) needs to feel a lot more dynamic and expressive:
 Animate green boxes, make sure that they're within a particular range at
 particular points in time.


 The tests also have to be deterministic and comprehensive. I am afraid of
 loosing both with the Render-to-Canvas approach.

 Can you give concrete examples of the kind of bugs you are hunting, and
 why testing cannot use the two methods suggested?


  I think that when testing some of the dynamic features it's best to
 create a minimal mechanism such as retrieving snapshot pixels, giving more
 power to the tests themselves in a dynamic language like JavaScript, rather
 than try to force the tests to fit into a predefined set of features
 supported by the DRT/WTR infrastructure. Also this would allow some
 differences between the ports, which implement animations a bit
 differently, without constant rebaselines of reference results – e.g. A
 heuristic animation timing test may stay valid across platform even if the
 actual animation implementation is different.


 Again, can you give concrete examples of this?

 Benjamin

 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 https://lists.webkit.org/mailman/listinfo/webkit-dev


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread Benjamin Poulain
On Fri, Feb 8, 2013 at 4:12 PM, Gregg Tavares g...@google.com wrote:

 Can you expose a time or setTime function in DRT and some option that says
 let JS control the clock?


Unfortunately, that only works in the simple cases.
We could cheat the WebCore clock, but that would ultimately be wrong for
animations running in threads or driven outside WebCore.

Benjamin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread Gregg Tavares
On Fri, Feb 8, 2013 at 4:16 PM, Benjamin Poulain benja...@webkit.orgwrote:

 On Fri, Feb 8, 2013 at 4:12 PM, Gregg Tavares g...@google.com wrote:

 Can you expose a time or setTime function in DRT and some option that
 says let JS control the clock?


 Unfortunately, that only works in the simple cases.
 We could cheat the WebCore clock, but that would ultimately be wrong for
 animations running in threads or driven outside WebCore.


The point was you need to be able to control all the clocks if you want
repeatable results. If you're saying some clocks can't be controlled that
sounds like something needs to be refactored so it's testable.



 Benjamin

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly

2013-02-08 Thread Philip Rogers
I'm not sure if this is generally known: we use [svg
document].setCurrentTime(time) in the svg/animations tests for both script
and pixel tests. The SVG animation test harness leaves a lot to be desired,
but it may be a good place to start for a more general DRT approach:
http://trac.webkit.org/browser/trunk/LayoutTests/svg/animations/script-tests/animate-css-xml-attributeType.js#L62

Philip


On Fri, Feb 8, 2013 at 4:26 PM, Gregg Tavares g...@google.com wrote:




 On Fri, Feb 8, 2013 at 4:16 PM, Benjamin Poulain benja...@webkit.orgwrote:

 On Fri, Feb 8, 2013 at 4:12 PM, Gregg Tavares g...@google.com wrote:

 Can you expose a time or setTime function in DRT and some option that
 says let JS control the clock?


 Unfortunately, that only works in the simple cases.
 We could cheat the WebCore clock, but that would ultimately be wrong for
 animations running in threads or driven outside WebCore.


 The point was you need to be able to control all the clocks if you want
 repeatable results. If you're saying some clocks can't be controlled that
 sounds like something needs to be refactored so it's testable.



 Benjamin



 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 https://lists.webkit.org/mailman/listinfo/webkit-dev


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


[webkit-dev] testing strategy? - CJK line breaking tests

2012-09-04 Thread Glenn Adams
What is the recommended approach to test cases when one needs to use a CJK
font that covers the test data? I could use DRT text results as expected
but given lack of common font across platforms, that doesn't seem to be
effective.

From my somewhat limited (i.e., newbie) exposure to WK, I gather one can
take one of the following approaches:

(1) put font (and thus platform) dependent test cases in non-platform test
directory, then add entries to Skipped for other platforms; this seems the
current approach with many platform/font dependent tests, especially
related to I18N features;
(2) put font (and thus platform) dependent test cases in platform test
directory, possibly ending up with separate tests per platform;
(3) what would be nice is to not generate rendering dump from DRT but
instead use script only; since I'm testing line-breaking logic, what would
do nicely is a set of internal styles available via getComputedStyle() on a
block element:

   - -webkit-line-count - numeric value indicating number of formatted
   lines produced from rendering a block
   - -webkit-line-chars-1 through -N, which returns the chars of line N as
   formatted when rendering a block

That is, with these properties, I could use script to determine which line
breaks occurred and where. I don't care about geometries or pixels in these
particular tests, just whether breaks occur in specific contexts.

Any recommendations on whether to pursue one of (1) or (2) above, or try
the more ambitious (but more platform independent) third approach?

I suppose I could start with (1) or (2) and then pursue (3) as a subsequent
task.

Regards,
Glenn
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] testing strategy? - CJK line breaking tests

2012-09-04 Thread Ryosuke Niwa
Can we use ref tests?

Alternatively, you can compute the with of each character with script
(without adding any new feature to WebKit).

If neither is possible, then you should use (1) unless the test tests a
platform specific feature; I.e. the feature isnt available on other
platforms.

(3) seems like a terrible idea as it means that we'll be either exposing
the WebKit internals to the Web without standardizing those properties or
we'll be adding yet-another DRT-specific behavior

On Tuesday, September 4, 2012, Glenn Adams wrote:

 What is the recommended approach to test cases when one needs to use a CJK
 font that covers the test data? I could use DRT text results as expected
 but given lack of common font across platforms, that doesn't seem to be
 effective.

 From my somewhat limited (i.e., newbie) exposure to WK, I gather one can
 take one of the following approaches:

 (1) put font (and thus platform) dependent test cases in non-platform test
 directory, then add entries to Skipped for other platforms; this seems the
 current approach with many platform/font dependent tests, especially
 related to I18N features;
 (2) put font (and thus platform) dependent test cases in platform test
 directory, possibly ending up with separate tests per platform;
 (3) what would be nice is to not generate rendering dump from DRT but
 instead use script only; since I'm testing line-breaking logic, what would
 do nicely is a set of internal styles available via getComputedStyle() on a
 block element:

- -webkit-line-count - numeric value indicating number of formatted
lines produced from rendering a block
- -webkit-line-chars-1 through -N, which returns the chars of line N
as formatted when rendering a block

 That is, with these properties, I could use script to determine which line
 breaks occurred and where. I don't care about geometries or pixels in these
 particular tests, just whether breaks occur in specific contexts.

 Any recommendations on whether to pursue one of (1) or (2) above, or try
 the more ambitious (but more platform independent) third approach?

 I suppose I could start with (1) or (2) and then pursue (3) as a
 subsequent task.

 Regards,
 Glenn



-- 
Ryosuke Niwa
Software Engineer
Google Inc.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] testing strategy? - CJK line breaking tests

2012-09-04 Thread Glenn Adams
On Tue, Sep 4, 2012 at 8:30 PM, Ryosuke Niwa rn...@webkit.org wrote:

 Can we use ref tests?


I'll try this first.



 Alternatively, you can compute the with of each character with script
 (without adding any new feature to WebKit).


Unfortunately, that won't work since I can't access the width of individual
lines produced by formatting a block. For example, p
style=width:2emXYZ/p:, where XYZ are certain CJK line break sensitive
chars, may format to

line-break: loose

line1: XY
lline2: Z

line-break: strict

line1: X
line2: YZ

The getComputedValue('width'|'height'| of the two results doesn't vary
since one can't access the resulting lines.


 If neither is possible, then you should use (1) unless the test tests a
 platform specific feature; I.e. the feature isnt available on other
 platforms.


The feature is not platform specific (other than the fact different
platforms may use different versions of ICU and support different fonts).
If reftest doesn't work, I'll try (1).


 (3) seems like a terrible idea as it means that we'll be either exposing
 the WebKit internals to the Web without standardizing those properties or
 we'll be adding yet-another DRT-specific behavior


Certainly it is not be the intent to expose such props to web content in
general. But I could fathom exposing this to WK test content (without
necessarily being dependent on DRT framework).




 On Tuesday, September 4, 2012, Glenn Adams wrote:

 What is the recommended approach to test cases when one needs to use a
 CJK font that covers the test data? I could use DRT text results as
 expected but given lack of common font across platforms, that doesn't seem
 to be effective.

 From my somewhat limited (i.e., newbie) exposure to WK, I gather one can
 take one of the following approaches:

 (1) put font (and thus platform) dependent test cases in non-platform
 test directory, then add entries to Skipped for other platforms; this seems
 the current approach with many platform/font dependent tests, especially
 related to I18N features;
 (2) put font (and thus platform) dependent test cases in platform test
 directory, possibly ending up with separate tests per platform;
 (3) what would be nice is to not generate rendering dump from DRT but
 instead use script only; since I'm testing line-breaking logic, what would
 do nicely is a set of internal styles available via getComputedStyle() on a
 block element:

- -webkit-line-count - numeric value indicating number of formatted
lines produced from rendering a block
- -webkit-line-chars-1 through -N, which returns the chars of line N
as formatted when rendering a block

 That is, with these properties, I could use script to determine which
 line breaks occurred and where. I don't care about geometries or pixels in
 these particular tests, just whether breaks occur in specific contexts.

 Any recommendations on whether to pursue one of (1) or (2) above, or try
 the more ambitious (but more platform independent) third approach?

 I suppose I could start with (1) or (2) and then pursue (3) as a
 subsequent task.

 Regards,
 Glenn



 --
 Ryosuke Niwa
 Software Engineer
 Google Inc.



___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] testing strategy? - CJK line breaking tests

2012-09-04 Thread Dan Bernstein


On Sep 4, 2012, at 5:50 AM, Glenn Adams gl...@skynav.com wrote:

 
 Alternatively, you can compute the with of each character with script 
 (without adding any new feature to WebKit).
 
 Unfortunately, that won't work since I can't access the width of individual 
 lines produced by formatting a block.

You can use Range’s getClientRects to get the widths of individual characters 
and to determine where a line breaks occur.___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] testing strategy? - CJK line breaking tests

2012-09-04 Thread Glenn Adams
On Tue, Sep 4, 2012 at 11:26 PM, Dan Bernstein m...@apple.com wrote:

 On Sep 4, 2012, at 5:50 AM, Glenn Adams gl...@skynav.com wrote:

 Alternatively, you can compute the with of each character with script
 (without adding any new feature to WebKit).


 Unfortunately, that won't work since I can't access the width of
 individual lines produced by formatting a block.

 You can use Range’s getClientRects to get the widths of individual
 characters and to determine where a line breaks occur.


Thanks for that pointer. I wasn't aware of that feature. In any case, it
looks like the reftest approach is going to work.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Testing XHR

2010-09-21 Thread Alexey Proskuryakov

I don't think that the suite is testing much besides what we already have 
regression tests for. For the most part, it just has different expectations.

There doesn't seem to be any harm in importing it as a whole now, but looking 
over the failing tests, categorizing them and sending feedback to 
public-webapps working groups is more important at this point. Jian Li from 
Chromium team is currently filing bugs for tests that fail in WebKit.

- WBR, Alexey Proskuryakov

20.09.2010, в 22:34, Ojan Vafai написал(а):

 Filed https://bugs.webkit.org/show_bug.cgi?id=46164 for the script to pull 
 the tests into our repo.
 
 On Tue, Sep 21, 2010 at 3:27 PM, Maciej Stachowiak m...@apple.com wrote:
 
 On Sep 20, 2010, at 10:19 PM, Ojan Vafai wrote:
 
 
  How about we create http/tests/xmlhttprequest/w3c-experimental or something 
  like that? That can tide us over until the official version comes out, at 
  which point, we can delete the w3c-experimental directory and just add a 
  w3c directory.
 
  It would be nice if we could start fixing these things before they become 
  part of the official test suite as a way of evaluating whether there are 
  issues with the spec and/or test suite.
 
 That is in fact exactly what we should be doing at this stage of the 
 standards process for XHR.
 
 
  Also, it seems to me like to does make sense to add an 
  update-experimental-w3c-xhr-tests script or something until there is an 
  official version.
 
 Indeed. And even after the test suite is official, it is likely to expand 
 over time. We should also look over our own XHR tests to see if there are any 
 that it would make sense to contribute to the W3C.
 
 Cheers,
 Maciej
 
 
 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing XHR

2010-09-21 Thread Jian Li
Yes, I have just gone through all the tests in this suite
and categorized and filed some WebKit bugs. Some other test failures seem to
be due to something wrong with tests themselves. I am pinging Anne about
these issues.

Thanks,

Jian

On Mon, Sep 20, 2010 at 11:43 PM, Alexey Proskuryakov a...@webkit.org wrote:


 I don't think that the suite is testing much besides what we already have
 regression tests for. For the most part, it just has different expectations.

 There doesn't seem to be any harm in importing it as a whole now, but
 looking over the failing tests, categorizing them and sending feedback to
 public-webapps working groups is more important at this point. Jian Li from
 Chromium team is currently filing bugs for tests that fail in WebKit.

 - WBR, Alexey Proskuryakov

 20.09.2010, в 22:34, Ojan Vafai написал(а):

 Filed https://bugs.webkit.org/show_bug.cgi?id=46164 for the script to pull
 the tests into our repo.

 On Tue, Sep 21, 2010 at 3:27 PM, Maciej Stachowiak m...@apple.com wrote:


 On Sep 20, 2010, at 10:19 PM, Ojan Vafai wrote:

 
  How about we create http/tests/xmlhttprequest/w3c-experimental or
 something like that? That can tide us over until the official version comes
 out, at which point, we can delete the w3c-experimental directory and just
 add a w3c directory.
 
  It would be nice if we could start fixing these things before they
 become part of the official test suite as a way of evaluating whether there
 are issues with the spec and/or test suite.

 That is in fact exactly what we should be doing at this stage of the
 standards process for XHR.

 
  Also, it seems to me like to does make sense to add an
 update-experimental-w3c-xhr-tests script or something until there is an
 official version.

 Indeed. And even after the test suite is official, it is likely to expand
 over time. We should also look over our own XHR tests to see if there are
 any that it would make sense to contribute to the W3C.

 Cheers,
 Maciej


 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev




 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


[webkit-dev] testing

2010-09-21 Thread asadulla ansari
-- 
Regards
 Navigate Today For Better Tomorrow!!!

mEeT mE LivE wItH tEcH cAkE @
http://hitechi.19.forumer.com/
http://go4expert.com/
http://www.liveperson.com/asadullah-ansari/
http://go4hitech.wordpress.com/
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing XHR

2010-09-21 Thread Anne van Kesteren
On Tue, Sep 21, 2010 at 8:43 AM, Alexey Proskuryakov a...@webkit.org wrote:
 I don't think that the suite is testing much besides what we already have
 regression tests for. For the most part, it just has different expectations.

FWIW, if you find cases the test suite is missing I'm happy to add
them. The only reason the W3C is not hosting this test suite at the
moment is because they cannot handle server-side scripts on the test
server at the moment. PLH is looking into a solution on their side as
I understand things.


-- 
Anne van Kesteren
http://annevankesteren.nl/
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


[webkit-dev] Testing XHR

2010-09-20 Thread Luke Macpherson
WebKit (or at least Chrome) is currently failing a bunch opera's tests
located at: http://tc.labs.opera.com/apis/XMLHttpRequest/

It seems that it would be a good idea for us to make use of these
tests in WebKit, so I was thinking of importing them into the
codebase. This raises the question:
Should we import the tests one at a time as we fix the bugs, or should
we import the whole suite?
The tests look like they will fit into the existing LayoutTests
infrastructure pretty easily, but if we import the whole lot it would
be good to figure out if it is possible to keep our copy in sync with
Opera's, possibly along the lines of
WebKitTools/Scripts/update-webgl-conformance-tests .

Should they end up in a subdirectory of LayoutTests/http/tests? (They
require php scripts on the server).

Thanks,
Luke
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing XHR

2010-09-20 Thread Ojan Vafai
On Tue, Sep 21, 2010 at 2:55 PM, Maciej Stachowiak m...@apple.com wrote:

 On Sep 20, 2010, at 9:34 PM, Luke Macpherson wrote:

  WebKit (or at least Chrome) is currently failing a bunch opera's tests
  located at: http://tc.labs.opera.com/apis/XMLHttpRequest/
 
  It seems that it would be a good idea for us to make use of these
  tests in WebKit, so I was thinking of importing them into the
  codebase. This raises the question:
  Should we import the tests one at a time as we fix the bugs, or should
  we import the whole suite?
  The tests look like they will fit into the existing LayoutTests
  infrastructure pretty easily, but if we import the whole lot it would
  be good to figure out if it is possible to keep our copy in sync with
  Opera's, possibly along the lines of
  WebKitTools/Scripts/update-webgl-conformance-tests .
 
  Should they end up in a subdirectory of LayoutTests/http/tests? (They
  require php scripts on the server).

 It would be a good idea to drop in the test suite wholesale. On the other
 hand, it's likely that most of this will turn into the official W3C test
 suite for the XMLHttpRequest spec. So we should either wait a bit, or be
 vigilant to avoid adding the same tests twice when we drop in the official
 test suite.

 One possibility would be to make this a subdirectory of
 http/tests/xmlhttprequest.


How about we create http/tests/xmlhttprequest/w3c-experimental or something
like that? That can tide us over until the official version comes out, at
which point, we can delete the w3c-experimental directory and just add a w3c
directory.

It would be nice if we could start fixing these things before they become
part of the official test suite as a way of evaluating whether there are
issues with the spec and/or test suite.

Also, it seems to me like to does make sense to add an
update-experimental-w3c-xhr-tests script or something until there is an
official version.

Ojan
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing XHR

2010-09-20 Thread Maciej Stachowiak

On Sep 20, 2010, at 10:19 PM, Ojan Vafai wrote:

 
 How about we create http/tests/xmlhttprequest/w3c-experimental or something 
 like that? That can tide us over until the official version comes out, at 
 which point, we can delete the w3c-experimental directory and just add a w3c 
 directory.
 
 It would be nice if we could start fixing these things before they become 
 part of the official test suite as a way of evaluating whether there are 
 issues with the spec and/or test suite.

That is in fact exactly what we should be doing at this stage of the standards 
process for XHR.

 
 Also, it seems to me like to does make sense to add an 
 update-experimental-w3c-xhr-tests script or something until there is an 
 official version.

Indeed. And even after the test suite is official, it is likely to expand over 
time. We should also look over our own XHR tests to see if there are any that 
it would make sense to contribute to the W3C.

Cheers,
Maciej

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing XHR

2010-09-20 Thread Ojan Vafai
Filed https://bugs.webkit.org/show_bug.cgi?id=46164 for the script to pull
the tests into our repo.

On Tue, Sep 21, 2010 at 3:27 PM, Maciej Stachowiak m...@apple.com wrote:


 On Sep 20, 2010, at 10:19 PM, Ojan Vafai wrote:

 
  How about we create http/tests/xmlhttprequest/w3c-experimental or
 something like that? That can tide us over until the official version comes
 out, at which point, we can delete the w3c-experimental directory and just
 add a w3c directory.
 
  It would be nice if we could start fixing these things before they become
 part of the official test suite as a way of evaluating whether there are
 issues with the spec and/or test suite.

 That is in fact exactly what we should be doing at this stage of the
 standards process for XHR.

 
  Also, it seems to me like to does make sense to add an
 update-experimental-w3c-xhr-tests script or something until there is an
 official version.

 Indeed. And even after the test suite is official, it is likely to expand
 over time. We should also look over our own XHR tests to see if there are
 any that it would make sense to contribute to the W3C.

 Cheers,
 Maciej


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing changes to CodeGenerator*.pm

2010-04-29 Thread Adam Barth
On Thu, Apr 29, 2010 at 9:44 AM, Alexey Proskuryakov a...@webkit.org wrote:
 On 26.04.2010, at 22:06, Adam Barth wrote:
 If you make changes to CodeGenerator*.pm, please test your
 change using the following command:

 ~/git/webkit$ ./WebKitTools/Scripts/run-bindings-tests


 I don't understand the utility of such testing. When one edits
 CodeGenerator*.pm, that's generally to change its output, so
 run-bindings-tests tests will of course fail.

 It's great to test end-to-end behavior, and unit tests can also also useful
 sometimes, but why test that source code stays byte to byte identical?

I should say that there's also a class of changes to CodeGenerator*.pm
where we don't want to change its output at all.  For example,
removing some of the ridiculously deeply nested if statements.

Adam
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing changes to CodeGenerator*.pm

2010-04-29 Thread Alexey Proskuryakov


On 29.04.2010, at 10:27, Jeremy Orlow wrote:

It's great to test end-to-end behavior, and unit tests can also also  
useful sometimes, but why test that source code stays byte to byte  
identical?


When you make a change to the code generator, you should make a  
corresponding change to the generated test code.  This allows the  
reviewer to see how the change in CodeGenerator*.pm affects the  
generated code and documents the change in SVN.  The hope is that  
it'll be obvious if your change is going to have unintended  
consequences on the generated code.  We've found this useful when  
working on the V8 bindings.


For these goals to be achieved, we'd need to check all generated code,  
not just one custom test file for each language.


It would be great to have a tool that generates a diff of derived  
sources for inspection, but making it into a test for everyone to  
maintain feels like unnecessary burden. I certainly would feel bad  
about having to maintain a test that verifies source file content  
instead of behavior.


- WBR, Alexey Proskuryakov

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing changes to CodeGenerator*.pm

2010-04-29 Thread Alexey Proskuryakov


On 29.04.2010, at 10:30, Adam Barth wrote:


Yeah, without seeing the consequence on the generated code, it can be
very difficult to review changes to the code generator.



Yes, that's a useful goal. Since an old version of code generating  
scripts is always in .svn directory (or its git equivalent), it should  
be possible to generate before and after versions of  
DerivedSources directory, and diff these. Such a tool would be useful  
and exciting.


- WBR, Alexey Proskuryakov

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing changes to CodeGenerator*.pm

2010-04-29 Thread Kent Hansen

ext Alexey Proskuryakov wrote:

On 29.04.2010, at 10:30, Adam Barth wrote:

  

Yeah, without seeing the consequence on the generated code, it can be
very difficult to review changes to the code generator.




Yes, that's a useful goal. Since an old version of code generating  
scripts is always in .svn directory (or its git equivalent), it should  
be possible to generate before and after versions of  
DerivedSources directory, and diff these. Such a tool would be useful  
and exciting.
  


Definitely. The generator for Qt Jambi (Qt bindings for Java, 
http://qt.gitorious.org/qt-jambi) has a --diff option that produces a 
colorized diff against the output of the previous run. It's a great, 
simple way of building confidence that the generator change only had the 
impact you intended (no guarantees of course, due to platform 
differences, feature permutations etc.).
In my experience such a feature is like turning on the lights (or 
cranking up the generator, to use a direct analogy).


Kent
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing changes to CodeGenerator*.pm

2010-04-29 Thread Adam Barth
On Thu, Apr 29, 2010 at 10:54 AM, Adam Barth aba...@webkit.org wrote:
 On Thu, Apr 29, 2010 at 10:39 AM, Alexey Proskuryakov a...@webkit.org wrote:
 On 29.04.2010, at 10:27, Jeremy Orlow wrote:
 It's great to test end-to-end behavior, and unit tests can also also
 useful sometimes, but why test that source code stays byte to byte
 identical?

 When you make a change to the code generator, you should make a
 corresponding change to the generated test code.  This allows the reviewer
 to see how the change in CodeGenerator*.pm affects the generated code and
 documents the change in SVN.  The hope is that it'll be obvious if your
 change is going to have unintended consequences on the generated code.
  We've found this useful when working on the V8 bindings.

 For these goals to be achieved, we'd need to check all generated code, not
 just one custom test file for each language.

 The idea is that the test IDL file exercises the features of the code
 generator.  Sure, it's not 100% coverage, but it's much better not
 testing it at all.

 It would be great to have a tool that generates a diff of derived sources
 for inspection, but making it into a test for everyone to maintain feels
 like unnecessary burden. I certainly would feel bad about having to maintain
 a test that verifies source file content instead of behavior.

 You should feel free to develop a better testing harness.  This one
 certainly isn't best conceivable tool, but it's better than what we
 had previously, which was essentially the C++ compiler.

 The maintenance is super easy.  I've been doing a lot of development
 work on the code generator in the past few days, and it amounts to
 typing a single command:

 ./WebKitTools/Scripts/run-bindings-tests --reset-results

 The harness has been super useful in working on the code generator
 because the tests run in a few seconds.  That lets me iterate on the
 script much more quickly compared to rebuilding the world every time I
 want to try a tweak.

As an additional datapoint on maintenance, it looks like you've
touched CodeGeneratorJS.pm twice in the past year and neither of those
change would have resulted in a large diff to the expected test
results.

By contrast, I've made five changes in the past two days and plan to
made dozens more changes shortly to reduce the amount of custom
bindings code we have in the tree, which is a serious maintenance
problem because it is full of subtle, quirky behavior.

Adam
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing changes to CodeGenerator*.pm

2010-04-29 Thread Maciej Stachowiak


On Apr 29, 2010, at 10:54 AM, Adam Barth wrote:



It would be great to have a tool that generates a diff of derived  
sources
for inspection, but making it into a test for everyone to maintain  
feels
like unnecessary burden. I certainly would feel bad about having to  
maintain

a test that verifies source file content instead of behavior.


You should feel free to develop a better testing harness.  This one
certainly isn't best conceivable tool, but it's better than what we
had previously, which was essentially the C++ compiler.

The maintenance is super easy.  I've been doing a lot of development
work on the code generator in the past few days, and it amounts to
typing a single command:

./WebKitTools/Scripts/run-bindings-tests --reset-results

The harness has been super useful in working on the code generator
because the tests run in a few seconds.  That lets me iterate on the
script much more quickly compared to rebuilding the world every time I
want to try a tweak.


It also strikes me as odd to do testing by doing exact comparison of  
the generated source. But I can also see side benefits. I think the  
real issue here may be one of naming. If the use model is that you  
fully regenerate the results every time you make a change to the  
bindings generator, then it's not really a regression test. The  
purpose is not to catch unintentional changes but rather to be able to  
observe changes to generated code, and new kinds of generated code,  
while working on a change and when reviewing code. Perhaps the tool  
should have a name that reflects that, instead of implying that the  
purpose is to catch bugs accidentally introduced by changes. It  
doesn't seem like an efficient or effective way to do the latter.


Regards,
Maciej

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing changes to CodeGenerator*.pm

2010-04-29 Thread Alexey Proskuryakov


On 29.04.2010, at 10:54, Adam Barth wrote:


You should feel free to develop a better testing harness.


Since you're proposing run-bindings-tests as a required tool for  
everyone to use, I don't think that this resolves my concerns.



The maintenance is super easy.  I've been doing a lot of development
work on the code generator in the past few days, and it amounts to
typing a single command:

./WebKitTools/Scripts/run-bindings-tests --reset-results


So, isn't once supposed to extend the tests when e.g. adding new flags?

- WBR, Alexey Proskuryakov

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing changes to CodeGenerator*.pm

2010-04-29 Thread Alexey Proskuryakov


On 29.04.2010, at 11:17, Yaar Schnitman wrote:

I've been using the tool for a couple of patches in V8. It really  
boost the development cycle, helps reviewers understanding what a  
cryptic perl block of code actually does, and side effects are easy  
to find. Once you start using it, its becoming hard to work without  
it. Give it a try!



'm thinking about how this tool could have helped with the  
CodeGenerator changes I made in the past, and it seems that it  
wouldn't have detected any changes, and could require me to find  
creative ways to test the new behavior.


- WBR, Alexey Proskuryakov

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing changes to CodeGenerator*.pm

2010-04-29 Thread Ojan Vafai
On Thu, Apr 29, 2010 at 11:39 AM, Alexey Proskuryakov a...@webkit.org wrote:

 On 29.04.2010, at 11:17, Yaar Schnitman wrote:

 I've been using the tool for a couple of patches in V8. It really boost
 the development cycle, helps reviewers understanding what a cryptic perl
 block of code actually does, and side effects are easy to find. Once you
 start using it, its becoming hard to work without it. Give it a try!


 'm thinking about how this tool could have helped with the CodeGenerator
 changes I made in the past, and it seems that it wouldn't have detected any
 changes, and could require me to find creative ways to test the new
 behavior.


I don't really follow the what the maintenance overhead is. How does this
actually cause you more than a trivial amount of extra work? Maybe a
specific example would help.

Isn't this just like having a layout test with expected results? It's a
small isolated test instead of testing everything. That seems like a good
thing.

More importantly, it lets you be sure that every feature of the code
generator has some testing. In the real IDLs, a feature might stop getting
used temporarily and then changes to the code generator would not be readily
apparent.

Ojan

P.S. Sorry for the double-post some of you got. Sent from the wrong email
address at first.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing changes to CodeGenerator*.pm

2010-04-29 Thread James Robinson
As a concrete example, I found this test setup helpful for this patch:
http://trac.webkit.org/changeset/58345.  A nice side effect was that it
revealed a bug in CodeGeneratorGObject.pm and let me fix it without having
to set up build setup for whatever it is that uses the GObject bindings.

I agree that golden file testing is a very high-maintenance fragile test
method, but it's better than nothing.  If this framework didn't exist then I
would have likely made the change and relied on spot checking and our
existing automated tests to catch any regressions which is less than ideal.

- James

On Thu, Apr 29, 2010 at 11:44 AM, Ojan Vafai o...@chromium.org wrote:

 On Thu, Apr 29, 2010 at 11:39 AM, Alexey Proskuryakov a...@webkit.orgwrote:

 On 29.04.2010, at 11:17, Yaar Schnitman wrote:

 I've been using the tool for a couple of patches in V8. It really boost
 the development cycle, helps reviewers understanding what a cryptic perl
 block of code actually does, and side effects are easy to find. Once you
 start using it, its becoming hard to work without it. Give it a try!


 'm thinking about how this tool could have helped with the CodeGenerator
 changes I made in the past, and it seems that it wouldn't have detected any
 changes, and could require me to find creative ways to test the new
 behavior.


 I don't really follow the what the maintenance overhead is. How does this
 actually cause you more than a trivial amount of extra work? Maybe a
 specific example would help.

 Isn't this just like having a layout test with expected results? It's a
 small isolated test instead of testing everything. That seems like a good
 thing.

 More importantly, it lets you be sure that every feature of the code
 generator has some testing. In the real IDLs, a feature might stop getting
 used temporarily and then changes to the code generator would not be readily
 apparent.

 Ojan

 P.S. Sorry for the double-post some of you got. Sent from the wrong email
 address at first.

 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing changes to CodeGenerator*.pm

2010-04-29 Thread David Levin
Just curious, would it be less maintenance if the test run was integrated
with run-webkit-tests?/Is the concern about having lots of different tests
harness to run to verify a change?

dave

On Thu, Apr 29, 2010 at 12:11 PM, James Robinson jam...@google.com wrote:

 As a concrete example, I found this test setup helpful for this patch:
 http://trac.webkit.org/changeset/58345.  A nice side effect was that it
 revealed a bug in CodeGeneratorGObject.pm and let me fix it without having
 to set up build setup for whatever it is that uses the GObject bindings.

 I agree that golden file testing is a very high-maintenance fragile test
 method, but it's better than nothing.  If this framework didn't exist then I
 would have likely made the change and relied on spot checking and our
 existing automated tests to catch any regressions which is less than ideal.

 - James

 On Thu, Apr 29, 2010 at 11:44 AM, Ojan Vafai o...@chromium.org wrote:

 On Thu, Apr 29, 2010 at 11:39 AM, Alexey Proskuryakov a...@webkit.orgwrote:

 On 29.04.2010, at 11:17, Yaar Schnitman wrote:

 I've been using the tool for a couple of patches in V8. It really boost
 the development cycle, helps reviewers understanding what a cryptic perl
 block of code actually does, and side effects are easy to find. Once you
 start using it, its becoming hard to work without it. Give it a try!


 'm thinking about how this tool could have helped with the CodeGenerator
 changes I made in the past, and it seems that it wouldn't have detected any
 changes, and could require me to find creative ways to test the new
 behavior.


 I don't really follow the what the maintenance overhead is. How does this
 actually cause you more than a trivial amount of extra work? Maybe a
 specific example would help.

 Isn't this just like having a layout test with expected results? It's a
 small isolated test instead of testing everything. That seems like a good
 thing.

 More importantly, it lets you be sure that every feature of the code
 generator has some testing. In the real IDLs, a feature might stop getting
 used temporarily and then changes to the code generator would not be readily
 apparent.

 Ojan

 P.S. Sorry for the double-post some of you got. Sent from the wrong email
 address at first.

 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev



 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing changes to CodeGenerator*.pm

2010-04-29 Thread Adam Barth
IMHO, run-webkit-tests should run all the various webkit testing
scripts and we should have a run-layout-tests script that does what
run-webkit-tests does today.

I'd also settle for a run-tests scripts that was the ASAD testing script.

Adam


On Thu, Apr 29, 2010 at 12:16 PM, David Levin le...@chromium.org wrote:
 Just curious, would it be less maintenance if the test run was integrated
 with run-webkit-tests?/Is the concern about having lots of different tests
 harness to run to verify a change?
 dave
 On Thu, Apr 29, 2010 at 12:11 PM, James Robinson jam...@google.com wrote:

 As a concrete example, I found this test setup helpful for this
 patch: http://trac.webkit.org/changeset/58345.  A nice side effect was that
 it revealed a bug in CodeGeneratorGObject.pm and let me fix it without
 having to set up build setup for whatever it is that uses the GObject
 bindings.
 I agree that golden file testing is a very high-maintenance fragile test
 method, but it's better than nothing.  If this framework didn't exist then I
 would have likely made the change and relied on spot checking and our
 existing automated tests to catch any regressions which is less than ideal.
 - James

 On Thu, Apr 29, 2010 at 11:44 AM, Ojan Vafai o...@chromium.org wrote:

 On Thu, Apr 29, 2010 at 11:39 AM, Alexey Proskuryakov a...@webkit.org
 wrote:

 On 29.04.2010, at 11:17, Yaar Schnitman wrote:

 I've been using the tool for a couple of patches in V8. It really boost
 the development cycle, helps reviewers understanding what a cryptic perl
 block of code actually does, and side effects are easy to find. Once you
 start using it, its becoming hard to work without it. Give it a try!

 'm thinking about how this tool could have helped with the CodeGenerator
 changes I made in the past, and it seems that it wouldn't have detected any
 changes, and could require me to find creative ways to test the new
 behavior.

 I don't really follow the what the maintenance overhead is. How does this
 actually cause you more than a trivial amount of extra work? Maybe a
 specific example would help.
 Isn't this just like having a layout test with expected results? It's a
 small isolated test instead of testing everything. That seems like a good
 thing.
 More importantly, it lets you be sure that every feature of the code
 generator has some testing. In the real IDLs, a feature might stop getting
 used temporarily and then changes to the code generator would not be readily
 apparent.
 Ojan
 P.S. Sorry for the double-post some of you got. Sent from the wrong email
 address at first.
 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev



 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev



 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing changes to CodeGenerator*.pm

2010-04-29 Thread Maciej Stachowiak


On Apr 29, 2010, at 11:38 AM, Ojan Vafai wrote:

I don't really follow the argument against supporting this test.  
What is the maintenance overhead? Isn't this just like having a  
layout test with expected results? It's a small isolated test  
instead of testing everything. That seems like a good thing.


More importantly, it lets you be sure that every feature of the code  
generator has some testing. In the real IDLs, a feature might stop  
getting used temporarily and then changes to the code generator  
would not be readily apparent.


I don't think your comments above are responsive to my paragraph below:



On Thu, Apr 29, 2010 at 11:05 AM, Maciej Stachowiak m...@apple.com  
wrote:


It also strikes me as odd to do testing by doing exact comparison of  
the generated source. But I can also see side benefits. I think the  
real issue here may be one of naming. If the use model is that you  
fully regenerate the results every time you make a change to the  
bindings generator, then it's not really a regression test. The  
purpose is not to catch unintentional changes but rather to be able  
to observe changes to generated code, and new kinds of generated  
code, while working on a change and when reviewing code. Perhaps the  
tool should have a name that reflects that, instead of implying that  
the purpose is to catch bugs accidentally introduced by changes. It  
doesn't seem like an efficient or effective way to do the latter.




To repeat, I think this is a useful tool, but not necessarily as a test.

The mode of operation is that when you run this test, if the generated  
bindings for the text IDL file change in any way, it reports a  
failure. Then you must manually take the step of regenerating the  
golden master file. It doesn't seem like the failure report itself  
will result in a decision. It doesn't provide interesting information  
until you regenerate the test file and look at the diffs, except in  
the highly unusual case of changing the output of the codegen script  
unintentially. Most changes to the codegen script are to intentionally  
change the output.


It seems to me a better model would be to regenerate the bindings test  
file automatically as part of the build. Then the changes can still be  
reviewed by you, or as part of a diff, but there would be no extra  
manual steps involved. And people making behaviorally transparent  
changes to codegen output would perhaps feel a little less burdened.


That makes more sense to me than treating it as a regression test. It  
also seems like this model would still have all the benefits you cite  
above.


Regards,
Maciej

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing changes to CodeGenerator*.pm

2010-04-29 Thread Adam Barth
On Thu, Apr 29, 2010 at 12:43 PM, Maciej Stachowiak m...@apple.com wrote:
 To repeat, I think this is a useful tool, but not necessarily as a test.
 The mode of operation is that when you run this test, if the generated
 bindings for the text IDL file change in any way, it reports a failure. Then
 you must manually take the step of regenerating the golden master file. It
 doesn't seem like the failure report itself will result in a decision. It
 doesn't provide interesting information until you regenerate the test file
 and look at the diffs, except in the highly unusual case of changing the
 output of the codegen script unintentially. Most changes to the codegen
 script are to intentionally change the output.

The failure report shows you the diff.  In hacking on CodeGeneratorJS,
I often run the script many times to see how my changes are affecting
the generated code, much in the same way I would run all the
LayoutTests constantly if they took 5s to run.

 It seems to me a better model would be to regenerate the bindings test file
 automatically as part of the build. Then the changes can still be reviewed
 by you, or as part of a diff, but there would be no extra manual steps
 involved. And people making behaviorally transparent changes to codegen
 output would perhaps feel a little less burdened.

That sounds like a good improvement.  I think it would be fine to
regenerate the output as part of the build.  However, I think we
should still preserve the ability to run the script along it test
mode because that's about three orders of magnitude faster than
performing a build after touching CodeGeneratorJS.

 That makes more sense to me than treating it as a regression test. It also
 seems like this model would still have all the benefits you cite above.

What I hear from this conversation is the following:

1) A bunch of people who've used the tool saying that they've found it useful.
2) A bunch of people who haven't used the tool suggesting improvements.

This tool impacts the handful of people who work on
CodeGeneratorJS.pm.  Everyone else in the project can safely ignore
it.  I'm all for improvements, and I invite anyone interested to
either improve the tool or write a new tool that does the job better.

Adam
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing changes to CodeGenerator*.pm

2010-04-29 Thread Yaar Schnitman
I see run-bindings-tests is primary a productivity tool: Faster development,
easy debugging, better reviews. But it only works if the golden copies don't
fall into disrepair = Hence the need to bake this tool into the general
testing framework (run-webkit-tests,  pre-submit checks, etc.). It will
become a useful (but not perfect) regression tool as a side effect of that.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing changes to CodeGenerator*.pm

2010-04-29 Thread Maciej Stachowiak



On Apr 29, 2010, at 1:05 PM, Adam Barth wrote:

On Thu, Apr 29, 2010 at 12:43 PM, Maciej Stachowiak m...@apple.com  
wrote:


It seems to me a better model would be to regenerate the bindings  
test file
automatically as part of the build. Then the changes can still be  
reviewed
by you, or as part of a diff, but there would be no extra manual  
steps
involved. And people making behaviorally transparent changes to  
codegen

output would perhaps feel a little less burdened.


That sounds like a good improvement.  I think it would be fine to
regenerate the output as part of the build.  However, I think we
should still preserve the ability to run the script along it test
mode because that's about three orders of magnitude faster than
performing a build after touching CodeGeneratorJS.



Alexey (or others who don't like the new tests), do you think this  
change would address your concerns?


On Apr 29, 2010, at 1:05 PM, Adam Barth wrote:



What I hear from this conversation is the following:

1) A bunch of people who've used the tool saying that they've found  
it useful.
2) A bunch of people who haven't used the tool suggesting  
improvements.


This tool impacts the handful of people who work on
CodeGeneratorJS.pm.  Everyone else in the project can safely ignore
it.  I'm all for improvements, and I invite anyone interested to
either improve the tool or write a new tool that does the job better.


If everyone has to use the tool for the tool to be useful, then  
ideally we want a system where the people who change the bindings  
frequently mostly buy into. Here is the list of people with more than  
5 all-time commits in the WebCore/bindings/scripts directory. Ideally  
I'd like to hear from more of these what they think would be helpful  
and not burdensome.


  59  wei...@apple.com
  46  e...@webkit.org
  35  da...@apple.com
  32  jap...@chromium.org
  29  oli...@apple.com
  26  gga...@apple.com
  26  dglaz...@chromium.org
  16  aba...@webkit.org
  14  zimmerm...@webkit.org
  12  a...@webkit.org
  10  aro...@apple.com
   8  le...@chromium.org
   7  m...@apple.com
   7  da...@chromium.org
   6  timo...@apple.com
   6  s...@chromium.org
   6  jia...@chromium.org
   6  ddkil...@apple.com
   6  cwzwar...@webkit.org

Here is the command anyone can run to see the full list (assuming you  
have an SVN checkout):
$ svn log WebCore/bindings/scripts | grep '|.*@' | sed -e 's/^[^|]* |// 
g; s/ | .*$//g;' | sort | uniq -c | sort -rn


The long tail of people who have made only a few bindings changes is  
rather large, so I suspect this tool affects more than a handful  
people, if it becomes a mandatory part of the process.


Regards,
Maciej

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing changes to CodeGenerator*.pm

2010-04-29 Thread Alexey Proskuryakov


On 29.04.2010, at 13:38, Maciej Stachowiak wrote:

Alexey (or others who don't like the new tests), do you think this  
change would address your concerns?



It would definitely be better if changes weren't detected as failures.  
Changing a checked out file as part of build seems rather magical  
though.


I'm not sure if everyone agrees that making a diff of actual generated  
files would be better than looking at a single custom test. That's  
something I wished for many times, but it calls for a different  
workflow. And of course, that's not something that is currently  
implemented.


- WBR, Alexey Proskuryakov

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing changes to CodeGenerator*.pm

2010-04-29 Thread Eric Seidel
Since I'm in the bindings hall of shame, I guess I'm supposed to reply. ;)

The twice that I've used it, it was very helpful.  The few reviews
I've done of Adam's it was much better than what we had before.
However, I agree something better could be built.  I'm just not sure
what better looks like yet.  I expect when Adam finishes getting rid
of custom bindings code he'll have a better idea. :)

I'm surprised this thread so much attention.

-eric

On Thu, Apr 29, 2010 at 1:38 PM, Maciej Stachowiak m...@apple.com wrote:


 On Apr 29, 2010, at 1:05 PM, Adam Barth wrote:

 On Thu, Apr 29, 2010 at 12:43 PM, Maciej Stachowiak m...@apple.com wrote:

 It seems to me a better model would be to regenerate the bindings test file

 automatically as part of the build. Then the changes can still be reviewed

 by you, or as part of a diff, but there would be no extra manual steps

 involved. And people making behaviorally transparent changes to codegen

 output would perhaps feel a little less burdened.

 That sounds like a good improvement.  I think it would be fine to
 regenerate the output as part of the build.  However, I think we
 should still preserve the ability to run the script along it test
 mode because that's about three orders of magnitude faster than
 performing a build after touching CodeGeneratorJS.

 Alexey (or others who don't like the new tests), do you think this change
 would address your concerns?
 On Apr 29, 2010, at 1:05 PM, Adam Barth wrote:

 What I hear from this conversation is the following:

 1) A bunch of people who've used the tool saying that they've found it
 useful.
 2) A bunch of people who haven't used the tool suggesting improvements.

 This tool impacts the handful of people who work on
 CodeGeneratorJS.pm.  Everyone else in the project can safely ignore
 it.  I'm all for improvements, and I invite anyone interested to
 either improve the tool or write a new tool that does the job better.

 If everyone has to use the tool for the tool to be useful, then ideally we
 want a system where the people who change the bindings frequently mostly buy
 into. Here is the list of people with more than 5 all-time commits in the
 WebCore/bindings/scripts directory. Ideally I'd like to hear from more of
 these what they think would be helpful and not burdensome.
   59  wei...@apple.com
   46  e...@webkit.org
   35  da...@apple.com
   32  jap...@chromium.org
   29  oli...@apple.com
   26  gga...@apple.com
   26  dglaz...@chromium.org
   16  aba...@webkit.org
   14  zimmerm...@webkit.org
   12 �...@webkit.org
   10  aro...@apple.com
    8  le...@chromium.org
    7  ...@apple.com
    7  da...@chromium.org
    6  timo...@apple.com
    6  s...@chromium.org
    6  jia...@chromium.org
    6  ddkil...@apple.com
    6  cwzwar...@webkit.org
 Here is the command anyone can run to see the full list (assuming you have
 an SVN checkout):
 $ svn log WebCore/bindings/scripts | grep '|.*@' | sed -e 's/^[^|]* |//g; s/
 | .*$//g;' | sort | uniq -c | sort -rn
 The long tail of people who have made only a few bindings changes is rather
 large, so I suspect this tool affects more than a handful people, if it
 becomes a mandatory part of the process.
 Regards,
 Maciej

 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing changes to CodeGenerator*.pm

2010-04-29 Thread Jeremy Orlow
On Thu, Apr 29, 2010 at 9:05 PM, Adam Barth aba...@webkit.org wrote:

 On Thu, Apr 29, 2010 at 12:43 PM, Maciej Stachowiak m...@apple.com wrote:

  It seems to me a better model would be to regenerate the bindings test
 file
  automatically as part of the build. Then the changes can still be
 reviewed
  by you, or as part of a diff, but there would be no extra manual steps
  involved. And people making behaviorally transparent changes to codegen
  output would perhaps feel a little less burdened.

 That sounds like a good improvement.  I think it would be fine to
 regenerate the output as part of the build.  However, I think we
 should still preserve the ability to run the script along it test
 mode because that's about three orders of magnitude faster than
 performing a build after touching CodeGeneratorJS.


From reading this thread, this seems like the best solution and most easy
to accomplish.  We already have the standalone script, so all we need to do
is hack it into the build.  Isn't there some makefile that generates the
derived sources for at least most of the platforms?  Couldn't we just add it
to that?

People who don't care or value it could continue with their current
work-flows unaffected.  And those that it helps can continue with their
workflows as well, as far as I can tell.

And, optionally, if someone would like to create more robust testing
frameworks in the future could.

Am I totally missing something?  I feel like I must since this discussion
has continued on for a while beyond this point.

J
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


[webkit-dev] Testing changes to CodeGenerator*.pm

2010-04-26 Thread Adam Barth
People of webkit-dev:

We now have a testing harness in place for CodeGeneratorJS.pm and
friends.  If you make changes to CodeGenerator*.pm, please test your
change using the following command:

~/git/webkit$ ./WebKitTools/Scripts/run-bindings-tests

If you add a new feature to CodeGenerator*.pm, please exercise that
feature by adding a declaration in our test IDL file:

http://trac.webkit.org/browser/trunk/WebCore/bindings/scripts/test/TestObj.idl

If your patch changes the expected results, you can generate new
results as follows:

~/git/webkit$ ./WebKitTools/Scripts/run-bindings-tests --overwrite

That way folks can review the change to the expected results along
side your change to the generator.

Happy code generation hacking!
Adam
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


[webkit-dev] testing client implementations

2010-01-26 Thread Joe Mason
I'm not too familiar with the test framework, but my impression is that
the LayoutTests are all about the results of page rendering.  Are there
any tests which validate that callbacks on classes like
FrameLoaderClient are called at the expected time?

Joe

-
This transmission (including any attachments) may contain confidential 
information, privileged material (including material protected by the 
solicitor-client or other applicable privileges), or constitute non-public 
information. Any use of this information by anyone other than the intended 
recipient is prohibited. If you have received this transmission in error, 
please immediately reply to the sender and delete this information from your 
system. Use, dissemination, distribution, or reproduction of this transmission 
by unintended recipients is not authorized and may be unlawful.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] testing client implementations

2010-01-26 Thread Adam Barth
Yes.  LayoutTests test all manner of things.  Looks for tests that
call the dump...Callbacks method of layoutTestController.

Adam


On Tue, Jan 26, 2010 at 10:27 PM, Joe Mason jma...@rim.com wrote:
 I'm not too familiar with the test framework, but my impression is that
 the LayoutTests are all about the results of page rendering.  Are there
 any tests which validate that callbacks on classes like
 FrameLoaderClient are called at the expected time?

 Joe

 -
 This transmission (including any attachments) may contain confidential 
 information, privileged material (including material protected by the 
 solicitor-client or other applicable privileges), or constitute non-public 
 information. Any use of this information by anyone other than the intended 
 recipient is prohibited. If you have received this transmission in error, 
 please immediately reply to the sender and delete this information from your 
 system. Use, dissemination, distribution, or reproduction of this 
 transmission by unintended recipients is not authorized and may be unlawful.
 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


[webkit-dev] Testing harness for Geolocation

2009-09-02 Thread Adam Barth
Improvements to the geolocation feature have stalled out because we
don't have a way to test these patches.  It seems the existing
geolocation feature has been implemented without testing.  (Boo!)

Testing geolocation seems different than testing other WebKit features
because geolocation lives in WebCore/platform.  One approach is to
create a fake (or mock) geolocation service that DRT can control.
This has been tried a couple of times:

https://bugs.webkit.org/show_bug.cgi?id=21717
https://bugs.webkit.org/show_bug.cgi?id=28264

I couldn't find find any precedent for testing this kind of feature
before, so I wanted to run this by the list to see if anyone has
strong opinions about the right way to test geolocation.

Adam
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


[webkit-dev] Testing worker lifecycle

2009-06-25 Thread Drew Wilson
(resending to a wider audience - apologies to those of you who receive this
twice)
Hi all,
The HTML5 worker spec has changed significantly from its earlier
incarnations, in that it's no longer externally visible whether a given
worker thread has shut down or not (there are no more close events surfaced
on the Worker).

This means that there's not currently a good way to write tests to verify
that workers are actually shutting down when they are unreachable/idle. Do
you guys have any ideas about how to address this?

One idea I had would be to expose a WorkerController for layout tests,
similar to GCController - the WorkerController could expose a numWorkers
attribute which is incremented/decremented as worker threads startup/exit. I
haven't started looking into how this would be done, but I figured I'd ask
you guys if this was a good approach or if there's a cleaner way to test
these cases.

-atw
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing worker lifecycle

2009-06-25 Thread Maciej Stachowiak


On Jun 25, 2009, at 2:27 PM, Drew Wilson wrote:

(resending to a wider audience - apologies to those of you who  
receive this twice)

Hi all,

The HTML5 worker spec has changed significantly from its earlier  
incarnations, in that it's no longer externally visible whether a  
given worker thread has shut down or not (there are no more close  
events surfaced on the Worker).


What's the motivation for this particular change?

This means that there's not currently a good way to write tests to  
verify that workers are actually shutting down when they are  
unreachable/idle. Do you guys have any ideas about how to address  
this?


One obvious way would be to add a nonstandard equivalent to the  
close event (webkitClose). The downside is that this would be  
exposed to Web content.




One idea I had would be to expose a WorkerController for layout  
tests, similar to GCController - the WorkerController could expose a  
numWorkers attribute which is incremented/decremented as worker  
threads startup/exit. I haven't started looking into how this would  
be done, but I figured I'd ask you guys if this was a good approach  
or if there's a cleaner way to test these cases.


That's also possible, the downside of this approach would be that the  
tests wouldn't work in a normal browser that doesn't have the special  
DumpRenderTree APIs.


 - Maciej

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing worker lifecycle

2009-06-25 Thread Drew Wilson
On Thu, Jun 25, 2009 at 2:33 PM, Maciej Stachowiak m...@apple.com wrote:


 On Jun 25, 2009, at 2:27 PM, Drew Wilson wrote:

  (resending to a wider audience - apologies to those of you who receive
 this twice)
 Hi all,

 The HTML5 worker spec has changed significantly from its earlier
 incarnations, in that it's no longer externally visible whether a given
 worker thread has shut down or not (there are no more close events surfaced
 on the Worker).


 What's the motivation for this particular change?


Primarily to avoid requiring specific GC behavior in the HTML5 spec. If you
expose when a worker has shut down, then for compatibility reasons you need
to try to make all user agents behave identically wrt garbage collection and
reachability, which puts an unnecessary burden on implementors. We've also
removed things like MessagePort.active as well as throwing exceptions if you
try to post a MessagePort that's been closed, for similar reasons. All of
this is motivated by the fact that it's prohibitively difficult to determine
cross-thread/cross-process reachability.




  This means that there's not currently a good way to write tests to verify
 that workers are actually shutting down when they are unreachable/idle. Do
 you guys have any ideas about how to address this?


 One obvious way would be to add a nonstandard equivalent to the close
 event (webkitClose). The downside is that this would be exposed to Web
 content.


Exactly, which I'd like to avoid for the reasons mentioned above.





 One idea I had would be to expose a WorkerController for layout tests,
 similar to GCController - the WorkerController could expose a numWorkers
 attribute which is incremented/decremented as worker threads startup/exit. I
 haven't started looking into how this would be done, but I figured I'd ask
 you guys if this was a good approach or if there's a cleaner way to test
 these cases.


 That's also possible, the downside of this approach would be that the tests
 wouldn't work in a normal browser that doesn't have the special
 DumpRenderTree APIs.


Indeed. That feels kind of lame, but the current alternative (no tests at
all) seems bad too.




  - Maciej


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


[webkit-dev] testing question for code that calls abstract methods (like ChromeClient)

2009-05-27 Thread John Gregg
What's a good strategy for testing WebKit code that calls out to the
ChromeClient interface or some other abstract interface?  I can't find
anything obvious in the codebase-- is there any way to mock out those calls
for unit tests?

Thanks,
 -John
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] testing question for code that calls abstract methods (like ChromeClient)

2009-05-27 Thread Darin Adler

On May 27, 2009, at 4:11 PM, John Gregg wrote:

What's a good strategy for testing WebKit code that calls out to the  
ChromeClient interface or some other abstract interface? I can't  
find anything obvious in the codebase-- is there any way to mock out  
those calls for unit tests?


Typically we add features to DumpRenderTree that use the public  
interfaces on the various platforms to test such things.


-- Darin

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] testing changes to UString/String

2009-01-12 Thread Geoffrey Garen

Hi Dave.

We tend to write unit tests in HTML or JavaScript. If it's possible to  
trigger all of the relevant test conditions in HTML or JavaScript, I'd  
recommend that.


The next best thing to do is to write a C++ test function, add it to  
WebKitTools/DumpRenderTree, and invoke it during DumpRenderTree's  
initialization. This solution is not as good because you can't run it  
inside a web browser, and a failure in one test will cause all unit  
tests to appear to fail.


As a last resort, you could create a new build target specifically for  
your unit test. This solution is not as good because most WebKit  
developers will not run your test as a part of their daily development.


Cheers,
Geoff

On Jan 11, 2009, at 12:36 PM, David Levin wrote:

I'm refactoring and improving UString and String. I want to write  
tests for these classes. What's the best way to do this?


Thanks,
Dave
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


[webkit-dev] testing changes to UString/String

2009-01-11 Thread David Levin
I'm refactoring and improving UString and String. I want to write tests for
these classes. What's the best way to do this?
Thanks,
Dave
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing - Compilating Webcore

2008-10-10 Thread Sherif
Hi all,

I have 2 questions in the compilation :-

Q1:

I downloaded yesterday the CVS source of webkittools, 
called: webkittools/checkout
called: update-webkit
 and started the compilation by: build-webkit

4 errors came in Javascriptcore


Q2:

if i replace the downloaded Javascriptcore and webcore with the ones 
from iphone and start the build, is it the correct way to get the newly 
compiled webcore? or is there a mistake.


thanks in advace,

regards,
Sherif






___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


[webkit-dev] Testing - Compilating Webcore

2008-10-08 Thread Sherif
Hello guys,

Could any body please help me in compiling the webcore?
I ve macos, 10.5 and xcode. Only the souce code of Webcore in a folder.

Q1: why Should i compile javascriptcore first before compiling webcore? 
This is a note in the readme but i don't understand why.

Q2: What is the correct parameter to compile the webcore using  xcodebuild?

thanks in advance,

regards,
Sherif

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Testing - Compilating Webcore

2008-10-08 Thread David Kilzer
On Wed, 10/8/08, Sherif [EMAIL PROTECTED] wrote:

 Could any body please help me in compiling the webcore?
 I ve macos, 10.5 and xcode. Only the souce code of Webcore
 in a folder.
 
 Q1: why Should i compile javascriptcore first before
 compiling webcore? 
 This is a note in the readme but i don't understand
 why.

JavaScriptCore must be built first because WebCore depends on it.

 Q2: What is the correct parameter to compile the webcore
 using  xcodebuild?

See WebKitTools/Scripts/build-webkit or run xcodebuild -help or build the 
project from within Xcode.

Dave


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev