[webkit-dev] Building on Mountain Lion

2012-07-25 Thread Jacob Goldstein
Has any tried to build WebKit on Mountain Lion?

I just attempted with a new install of Mountain Lion and Xcode 4.4 and the 
build keeps failing while trying to compile AlternativeTextUIController.  If 
anyone has thoughts as to why this could be, please let me know.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Building on Mountain Lion

2012-07-25 Thread Jacob Goldstein


On 7/25/12 2:21 PM, Mark Rowe mr...@apple.com wrote:


On 2012-07-25, at 14:15, Jacob Goldstein jac...@adobe.com wrote:

 Has any tried to build WebKit on Mountain Lion?
 
 I just attempted with a new install of Mountain Lion and Xcode 4.4 and
the build keeps failing while trying to compile
AlternativeTextUIController.  If anyone has thoughts as to why this
could be, please let me know.

We've obviously been building on Mountain Lion for quite some time. The
WebKit nightly builds have been building for 10.6 through 10.8 against
the OS X 10.8 SDK for the last few weeks. What problems are you running
in to?


Ah, right, thanks for the reminder.  It must be something with my local
configuration - I'll have to take a look.

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Can we distinguish imported tests in LayoutTests/css3 ?

2012-06-11 Thread Jacob Goldstein
On 6/11/12 11:11 AM, Alexis Menard alexis.men...@openbossa.org wrote:

On Mon, Jun 11, 2012 at 3:09 PM, Simon Fraser simon.fra...@apple.com
wrote:
 I think imported tests should be outside of LayoutTests/fast.

I agree a dedicated folder seems more appropriate.

My two cents.


 Simon

 On Jun 11, 2012, at 1:57 AM, Ryosuke Niwa wrote:

 Hi,

 I realized that there are a whole bunch of tests in LayoutTests/css3
that
 use layoutTestController (e.g. css3/calc, css3/filters, etc...), which
 appears to mean that they're our own tests. However, css3/selectors3 is
an
 imported W3C test suite. It's very confusing to mix imported tests and
 WebKit's own tests. Can we put the imported W3C tests in imported-w3c
 directory as proposed earlier?


Can we just create an imported-w3c folder at the same level as LayoutTests?



 Best,
 Ryosuke Niwa
 Software Engineer
 Google Inc.

 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev



 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev




-- 
Alexis Menard (darktears)
Software Engineer
openBossa @ INdT - Instituto Nokia de Tecnologia
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Can we distinguish imported tests in LayoutTests/css3 ?

2012-06-11 Thread Jacob Goldstein
On 6/11/12 2:24 PM, Ryosuke Niwa rn...@webkit.orgmailto:rn...@webkit.org 
wrote:

On Mon, Jun 11, 2012 at 2:21 PM, Jacob Goldstein 
jac...@adobe.commailto:jac...@adobe.com wrote:
Can we just create an imported-w3c folder at the same level as LayoutTests?

You mean at trunk? I don't think that makes sense, and our testing 
infrastructure doesn't support that.

Ah, ok. In that case the next best option is to put it within LayoutTests.  Are 
there any objections to this idea?

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Can we distinguish imported tests in LayoutTests/css3 ?

2012-06-11 Thread Jacob Goldstein
On 6/11/12 2:37 PM, Ryosuke Niwa rn...@webkit.orgmailto:rn...@webkit.org 
wrote:

On Mon, Jun 11, 2012 at 2:33 PM, Dirk Pranke 
dpra...@chromium.orgmailto:dpra...@chromium.org wrote:
Technically, it does (at least NRWT does), but I wouldn't recommend
it. NRWT is designed to allow tests to live in multiple repos, but it
seems like all of the tests in the webkit repo should be under a
single top-level dir.

I don't think old-run-webkit-tests supports that, not that it'll be too hard to 
support it. Regardless I agree with your point that all tests should be in 
single directory although I can see we can separate tests into two groups: 
regression tests and conformance tests.

That sounds fine to me.  I believe that the idea floated at the contributor's 
meeting was to put an imported folder within each feature/topic area.  That way 
you can run the top level folder for a feature area and get both imported and 
WebKit-specific tests.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] testharness Wiki page added

2012-06-01 Thread Jacob Goldstein

On 5/31/12 7:55 PM, Maciej Stachowiak m...@apple.commailto:m...@apple.com 
wrote:


On May 31, 2012, at 5:51 PM, Jacob Goldstein 
jac...@adobe.commailto:jac...@adobe.com wrote:

I haven't found that to be the case for the tests I have written for each 
suite, the output from testharness can be as simple as PASS or FAIL, or 
include additional debug information defined by the test author.  That being 
said, my experience is likely more limited than yours.  Peter Linss, who wrote 
and maintains the W3C testharness, would likely be open to suggestions for 
improvement if there is anything specific that we want to change.

The source is more verbose. The output is uglier and less clear, at least by 
default. PASS and FAIL is not as good as what our test driver does by 
default, because you can't as easily tell what passed or failed or why. Our own 
harness tells you the expression the test evaluated that failed, and what the 
expected result was, so it's much quicker to debug a failure.

I have given my feedback to James Graham before, including showing him how our 
test harness works. At the time, he was not open to making changes, in part 
because of details of Opera's internal testing setup and in part because eval 
is evil. I really would rather not reduce all our DOM tests to the 
Opera-driven level of source legibility and output quality.

Let me give you an example. This zip file contains an actual w3c test case, and 
a webkit-style conversion of the same test:


I understand your concern, however the example you attached doesn't use the W3C 
testharness to make assertions.  Instead, it is using a custom assertEquals 
method from the file domtestcase.js.  The output from this example appears to 
be totally defined by the test author.

In the attached zip I added a new file, anchor_href-w3c_EDITED.html, that uses 
testharness and shows the results from its assert_equals method.  I think these 
results are more inline with what you want – i.e. the default results include 
both the actual and expected values for the assertion that failed.  If you 
launch this file, the output from testharness appears at the bottom of the 
page.  The rest of the content is from the original test file.

For example, the output from the failing test, using testharness.js, is as 
follows:

FAIL Test that a href attribute contains search string assert_equals: Optional 
debug info. expected 
http://www.anothersite.com/path/page.htm?parameter=this%20is%a%20parameter.x; 
but got 
http://www.anothersite.com/path/page.htm?parameter=this%20is%a%20parameter;

Two portions of the output above is defined by the test author and so can be 
customized or omitted altogether.  The test name or description, i.e. Test 
that a href attribute contains search string , and the debug message Optional 
debug info..  These fields can be made to show anything, including the 
expression being evaluated, if so desired.

By default, testharness outputs FAIL and the actual and expected value for 
any assertion that fails.

Compare this to the output from your example using js-test-pre.js:

FAIL document.getElementById('anchor_5').getAttribute('href') should be 
http://www.anothersite.com/path/page.htm?parameter=this%20is%a%20parameter.x. 
Was http://www.anothersite.com/path/page.htm?parameter=this%20is%a%20parameter.


And the two look much more similar.  That is not to say that testharness 
couldn't benefit from some tweaks, but I think the results are closer to what 
we already have than demonstrated by your example.


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] testharness Wiki page added

2012-06-01 Thread Jacob Goldstein
Apologies - my mail server appears to have blocked the zip file I tried to 
attach to my previous message (for some reason it doesn't like when anyone 
sends .js files).

If anyone is interested, the only changes I made to the anchor_href-w3c.html 
page were the following:

(1) I included testharness.js and testharnessreport.js (and also copied them to 
the local folder for simplicity)

script src=testharness.js/script
script src=testharnessreport.js/script

(2) I added a div with id=log, which is used as the container for the results 
from testharness.

div id=log/div

(3) Finally, I used testharness for the assert that appears on line 36, as 
follows:

test(function() {assert_equals(oAnchor.getAttribute('href'), value, Optional 
debug info.)}, description);




On 6/1/12 2:40 PM, Jacob Goldstein 
jac...@adobe.commailto:jac...@adobe.com wrote:


On 5/31/12 7:55 PM, Maciej Stachowiak m...@apple.commailto:m...@apple.com 
wrote:


On May 31, 2012, at 5:51 PM, Jacob Goldstein 
jac...@adobe.commailto:jac...@adobe.com wrote:

I haven't found that to be the case for the tests I have written for each 
suite, the output from testharness can be as simple as PASS or FAIL, or 
include additional debug information defined by the test author.  That being 
said, my experience is likely more limited than yours.  Peter Linss, who wrote 
and maintains the W3C testharness, would likely be open to suggestions for 
improvement if there is anything specific that we want to change.

The source is more verbose. The output is uglier and less clear, at least by 
default. PASS and FAIL is not as good as what our test driver does by 
default, because you can't as easily tell what passed or failed or why. Our own 
harness tells you the expression the test evaluated that failed, and what the 
expected result was, so it's much quicker to debug a failure.

I have given my feedback to James Graham before, including showing him how our 
test harness works. At the time, he was not open to making changes, in part 
because of details of Opera's internal testing setup and in part because eval 
is evil. I really would rather not reduce all our DOM tests to the 
Opera-driven level of source legibility and output quality.

Let me give you an example. This zip file contains an actual w3c test case, and 
a webkit-style conversion of the same test:


I understand your concern, however the example you attached doesn't use the W3C 
testharness to make assertions.  Instead, it is using a custom assertEquals 
method from the file domtestcase.js.  The output from this example appears to 
be totally defined by the test author.

In the attached zip I added a new file, anchor_href-w3c_EDITED.html, that uses 
testharness and shows the results from its assert_equals method.  I think these 
results are more inline with what you want – i.e. the default results include 
both the actual and expected values for the assertion that failed.  If you 
launch this file, the output from testharness appears at the bottom of the 
page.  The rest of the content is from the original test file.

For example, the output from the failing test, using testharness.js, is as 
follows:

FAIL Test that a href attribute contains search string assert_equals: Optional 
debug info. expected 
http://www.anothersite.com/path/page.htm?parameter=this%20is%a%20parameter.x; 
but got 
http://www.anothersite.com/path/page.htm?parameter=this%20is%a%20parameter;

Two portions of the output above is defined by the test author and so can be 
customized or omitted altogether.  The test name or description, i.e. Test 
that a href attribute contains search string , and the debug message Optional 
debug info..  These fields can be made to show anything, including the 
expression being evaluated, if so desired.

By default, testharness outputs FAIL and the actual and expected value for 
any assertion that fails.

Compare this to the output from your example using js-test-pre.js:

FAIL document.getElementById('anchor_5').getAttribute('href') should be 
http://www.anothersite.com/path/page.htm?parameter=this%20is%a%20parameter.x. 
Was http://www.anothersite.com/path/page.htm?parameter=this%20is%a%20parameter.


And the two look much more similar.  That is not to say that testharness 
couldn't benefit from some tweaks, but I think the results are closer to what 
we already have than demonstrated by your example.


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


[webkit-dev] testharness Wiki page added

2012-05-31 Thread Jacob Goldstein
I added the following Wiki page to provide some information on testharness.js 
(the JavaScript framework from W3C recently landed in WebKit):
http://trac.webkit.org/wiki/Writing%20testharness%20Tests

I also updated the text under Writing JavaScript-based DOM only test cases on 
this page 
http://trac.webkit.org/wiki/Writing%20Layout%20Tests%20for%20DumpRenderTree.

Note that I am recommending the use of testharness.js / testharnessreport.js 
over js-test-pre.js/js-test-post.js, when applicable, since tests written using 
testharness can be copied to the W3C test repository with only minor changes.

Please review and send me any feedback on these pages.  I am considering adding 
a detailed example to the testharness wiki page to demonstrate how the API 
should be used.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] We should rename layoutTestController to testController

2012-05-31 Thread Jacob Goldstein

On 5/31/12 11:56 AM, Darin Adler da...@apple.com wrote:

I am thinking we should rename layoutTestController to testController. Or
if you don¹t like that name, maybe testHarness or some even better name.



testHarness is probably not a good choice as the W3C JavaScript framework
that recently landed in WebKit, and that we are going to encourage people
to use for DOM-only tests, is called testharness.js




The old name is too long and the word ³layout² is so strange.

We could expose the object under the new name and the old one, and then
over time convert all the tests to the new name, then get rid of the old
one.

What do you all think?

-- Darin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] testharness Wiki page added

2012-05-31 Thread Jacob Goldstein
I haven't found that to be the case for the tests I have written for each 
suite, the output from testharness can be as simple as PASS or FAIL, or 
include additional debug information defined by the test author.  That being 
said, my experience is likely more limited than yours.  Peter Linss, who wrote 
and maintains the W3C testharness, would likely be open to suggestions for 
improvement if there is anything specific that we want to change.


From: Maciej Stachowiak m...@apple.commailto:m...@apple.com
To: Jacob Goldstein jac...@adobe.commailto:jac...@adobe.com
Cc: WebKit Development 
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org
Subject: Re: [webkit-dev] testharness Wiki page added


On May 31, 2012, at 2:18 PM, Jacob Goldstein 
jac...@adobe.commailto:jac...@adobe.com wrote:

I added the following Wiki page to provide some information on testharness.js 
(the JavaScript framework from W3C recently landed in WebKit):
http://trac.webkit.org/wiki/Writing%20testharness%20Tests

I also updated the text under Writing JavaScript-based DOM only test cases on 
this page 
http://trac.webkit.org/wiki/Writing%20Layout%20Tests%20for%20DumpRenderTree.

Note that I am recommending the use of testharness.js / testharnessreport.js 
over js-test-pre.js/js-test-post.js, when applicable, since tests written using 
testharness can be copied to the W3C test repository with only minor changes.

Please review and send me any feedback on these pages.  I am considering adding 
a detailed example to the testharness wiki page to demonstrate how the API 
should be used.

In my experience, the W3C's DOM test harness results in tests that are more 
verbose, are less readable, and which have less readable output, as compared to 
js-test-pre.hs. I wish we would improve the W3C's upstream test harness instead 
of degrading the quality of our own tests.

Regards,
Maciej

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] testharness Wiki page added

2012-05-31 Thread Jacob Goldstein
Actually, I misspoke, it's not Peter Linss who maintains testharness, it's 
James Graham.

From: Jacob Goldstein jac...@adobe.commailto:jac...@adobe.com
To: Maciej Stachowiak m...@apple.commailto:m...@apple.com
Cc: WebKit Development 
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org
Subject: Re: [webkit-dev] testharness Wiki page added

I haven't found that to be the case for the tests I have written for each 
suite, the output from testharness can be as simple as PASS or FAIL, or 
include additional debug information defined by the test author.  That being 
said, my experience is likely more limited than yours.  Peter Linss, who wrote 
and maintains the W3C testharness, would likely be open to suggestions for 
improvement if there is anything specific that we want to change.


From: Maciej Stachowiak m...@apple.commailto:m...@apple.com
To: Jacob Goldstein jac...@adobe.commailto:jac...@adobe.com
Cc: WebKit Development 
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org
Subject: Re: [webkit-dev] testharness Wiki page added


On May 31, 2012, at 2:18 PM, Jacob Goldstein 
jac...@adobe.commailto:jac...@adobe.com wrote:

I added the following Wiki page to provide some information on testharness.js 
(the JavaScript framework from W3C recently landed in WebKit):
http://trac.webkit.org/wiki/Writing%20testharness%20Tests

I also updated the text under Writing JavaScript-based DOM only test cases on 
this page 
http://trac.webkit.org/wiki/Writing%20Layout%20Tests%20for%20DumpRenderTree.

Note that I am recommending the use of testharness.js / testharnessreport.js 
over js-test-pre.js/js-test-post.js, when applicable, since tests written using 
testharness can be copied to the W3C test repository with only minor changes.

Please review and send me any feedback on these pages.  I am considering adding 
a detailed example to the testharness wiki page to demonstrate how the API 
should be used.

In my experience, the W3C's DOM test harness results in tests that are more 
verbose, are less readable, and which have less readable output, as compared to 
js-test-pre.hs. I wish we would improve the W3C's upstream test harness instead 
of degrading the quality of our own tests.

Regards,
Maciej

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] testharness Wiki page added

2012-05-31 Thread Jacob Goldstein
On Thu, May 31, 2012 at 11:18 AM, Jacob Goldstein 
jac...@adobe.commailto:jac...@adobe.com wrote:
I added the following Wiki page to provide some information on testharness.js 
(the JavaScript framework from W3C recently landed in WebKit):
http://trac.webkit.org/wiki/Writing%20testharness%20Tests

I also updated the text under Writing JavaScript-based DOM only test cases on 
this page 
http://trac.webkit.org/wiki/Writing%20Layout%20Tests%20for%20DumpRenderTree.

Note that I am recommending the use of testharness.js / testharnessreport.js 
over js-test-pre.js/js-test-post.js, when applicable, since tests written using 
testharness can be copied to the W3C test repository with only minor changes.

Note that this isn't true once we start using layoutTestController in the test.

Tests that we currently use LTC such as to invoke mouse events, keyboard 
events, etc... are currently manual tests in W3C test suites. They use meta 
elements to tag manual tests but the actual instruction is just regular English 
sentences. It would be good to codify this information so that we can automate 
some of W3C manual tests at least in WebKit.


Would the W3C WebDriver specification you mentioned in another thread replace 
the need to use LTC for these types of events?



- Ryosuke

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


[webkit-dev] Importing W3C tests to webkit

2012-05-23 Thread Jacob Goldstein
At the WebKit contributor's meeting in April, we discussed a process for 
importing third party tests into the WebKit repository (specifically, from the 
W3C test repository).

I documented the process that we came up with at the meeting on the WebKit 
wiki, here:
http://trac.webkit.org/wiki/ImportingThirdPartyTests

It would be great if we could get some feedback on this process.

Once we can finalize the details for how this should function, the effort can 
proceed - scripts can be written to help automate the effort and an actual 
import of select W3C test suites can be performed.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Importing W3C tests to webkit

2012-05-23 Thread Jacob Goldstein
I agree.  If nothing else, getting W3C tests into the WebKit repository will 
help catch regressions.

From: Ryosuke Niwa rn...@webkit.orgmailto:rn...@webkit.org
To: Jacob Goldstein jac...@adobe.commailto:jac...@adobe.com
Cc: WebKit Development 
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org
Subject: Re: [webkit-dev] Importing W3C tests to webkit

As I have said in the past, we should just import all tests, and treat 
non-text, non-ref tests as pixel tests. If we wanted to reduce the number of 
pixel tests we import, then we should submit those patches to W3C instead of 
directly submitting them to WebKit.

In general, I don't buy the argument that adding more pixel tests incurs more 
cost than the benefit we get from importing the tests. If that were the case, 
we can just get rid of the existing pixel tests we have.

The only sane argument I've heard so far to gate pixel tests is that the 
correctness of such tests need to be manually inspected, which requires a lot 
of manual labor and is very error prone.

For script-based tests that print PASS/FAIL, it seems better to just check in 
*-expected-failure.txt or even -expected.txt files because we'll see PASS/FAIL 
in the expected tests themselves.

- Ryosuke

On Wed, May 23, 2012 at 10:26 AM, Jacob Goldstein 
jac...@adobe.commailto:jac...@adobe.com wrote:
At the WebKit contributor's meeting in April, we discussed a process for 
importing third party tests into the WebKit repository (specifically, from the 
W3C test repository).

I documented the process that we came up with at the meeting on the WebKit 
wiki, here:
http://trac.webkit.org/wiki/ImportingThirdPartyTests

It would be great if we could get some feedback on this process.

Once we can finalize the details for how this should function, the effort can 
proceed - scripts can be written to help automate the effort and an actual 
import of select W3C test suites can be performed.

___
webkit-dev mailing list
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Importing W3C tests to webkit

2012-05-23 Thread Jacob Goldstein
Dirk, my apologies, I was on travel the week you replied and missed your
message.  I found it and will review / update now.



On 5/23/12 1:25 PM, Dirk Pranke dpra...@chromium.org wrote:

On Wed, May 23, 2012 at 11:56 AM, Ryosuke Niwa rn...@webkit.org wrote:
 As I have said in the past, we should just import all tests, and treat
 non-text, non-ref tests as pixel tests. If we wanted to reduce the
number of
 pixel tests we import, then we should submit those patches to W3C
instead of
 directly submitting them to WebKit.

 In general, I don't buy the argument that adding more pixel tests incurs
 more cost than the benefit we get from importing the tests. If that
were the
 case, we can just get rid of the existing pixel tests we have.


Not so: if the tests we had had 100% coverage, then importing more
tests would buy us nothing, but getting rid of the existing tests
would be quite unfortunate. Clearly adding tests incurs some costs and
probably provides some benefit; the question is when does the cost
exceed the benefit?

As I am not against importing more tests per se, I think this only
makes sense to evaluate on a case-by-case basis.

 The only sane argument I've heard so far to gate pixel tests is that the
 correctness of such tests need to be manually inspected, which requires
a
 lot of manual labor and is very error prone.


I'm assuming the above includes the ongoing maintenance cost of
keeping pixel tests up to date, as well as the cost at the initial
checkin.

There is also the fact that the more tests we have, the more tests we
have to run, and increasing cycle time by itself is a cost to
developer productivity. Of course, it's also potentially the case that
we have to update tests from time to time, although this doesn't
happen often.

Jacob, I gave you a bunch of feedback not long after you published the
initial writeup, but it doesn't look like any of that has been
incorporated?

-- Dirk

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Importing W3C tests to webkit

2012-05-23 Thread Jacob Goldstein


On 5/23/12 2:30 PM, Maciej Stachowiak m...@apple.com wrote:


On May 23, 2012, at 2:16 PM, Dirk Pranke dpra...@chromium.org wrote:

 On Wed, May 23, 2012 at 1:41 PM, Ryosuke Niwa rn...@webkit.org wrote:
 The only sane argument I've heard so far to gate pixel tests is that
the
 correctness of such tests need to be manually inspected, which
requires
 a
 lot of manual labor and is very error prone.
 
 I'm assuming the above includes the ongoing maintenance cost of
 keeping pixel tests up to date, as well as the cost at the initial
 checkin.
 
 I'm not concerned of those. Once the correct expected result is
checked in,
 it's pretty easy to rebaseline tests per rendering engine changes
assuming
 people who are rebaselining tests know what they're doing.
 
 You should be concerned; keeping pixel tests up-to-date is clearly a
 non-zero cost that only the chromium port thus far has been willing to
 bear, and I suspect that the cost of updating baselines is
 substantially higher than the cost of the initial review over time
 (since it's a recurring cost).

Are you concerned just about the actual pixel results or also about
keeping render tree dumps up to date? We can address the pixel result
issue by introducing a new test that dumps its render tree but does not
do pixel testing.

I think there is a high value to importing standards test suites
wholesale, even if they overlap with our existing coverage. Picking and
choosing subsets makes things  more complicated. If there are significant
externalities to adding particular kinds of tests, I would prefer we
mitigate those externalities rather than run fewer tests.


As a side note to this discussion, there is talk in the W3C community
regarding their test approval process.  At the recent working group
meetings in Germany the idea was floated to simply approve all tests that
are currently waiting for review (and doing this going forward, e.g. no
longer requiring approval upon submission).

Apparently, not enough people are reviewing tests, and as a result, tests
can linger for months (or longer) before ever being looked at.  Once
browser vendors start implementing features, associated tests will be
revisited for that area.

This has by no means been decided, but something we should consider if it
ultimately does come to fruition.  If W3C tests are no longer
reviewed, this would mean importing tests without any knowledge of their
accuracy - though that will still allow us to catch regressions.



Dirk, I've updated the process on the Wiki page with the feedback you
provided.  I hope I captured it all - I included everything that appeared
to have agreement between you and Ryosuke.  Feel free to modify it
directly if I missed anything, or let me know and I can refine it further.


Jacob



Regards,
Maciej

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


[webkit-dev] Process for importing third party tests

2012-04-23 Thread Jacob Goldstein
At the recent WebKit Contributors Meeting, a process was drafted for importing 
third party tests into WebKit.

I created a wiki page that captures the process that we came up with here:

http://trac.webkit.org/wiki/ImportingThirdPartyTests

We'd like to get more input from the community on all aspects of this process.

Please review and lets discuss further.


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Handling failing reftests

2012-04-12 Thread Jacob Goldstein
Isn't the goal of writing a ref test that it is not platform specific?

From: Ryosuke Niwa rn...@webkit.orgmailto:rn...@webkit.org
Date: Thu, 12 Apr 2012 12:29:42 -0700
To: Ojan Vafai o...@chromium.orgmailto:o...@chromium.org
Cc: Dirk Pranke dpra...@chromium.orgmailto:dpra...@chromium.org, WebKit 
Development webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org
Subject: Re: [webkit-dev] Handling failing reftests

On Thu, Apr 12, 2012 at 11:53 AM, Ojan Vafai 
o...@chromium.orgmailto:o...@chromium.org wrote:
I agree that it's hard sometimes to construct reftests that work, but once 
you've done so, the cost on the project of maintaining the test is usually 
considerably lower than pixel tests.

Not so sure. There are cases where we have these platform specific failures for 
ref tests, and they're much harder to fix than rebaselining pixel results.

- Ryosuke

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Handling failing reftests

2012-04-12 Thread Jacob Goldstein
Right, but when those differences come to light, should the goal be to adjust 
the ref test to make it platform-independent, or is having platform-specific 
ref tests acceptable?  Doesn't that put us in the same situation as having 
platform-specific pixel tests?

From: Ryosuke Niwa rn...@webkit.orgmailto:rn...@webkit.org
Date: Thu, 12 Apr 2012 13:43:26 -0700
To: Jacob Goldstein jac...@adobe.commailto:jac...@adobe.com
Cc: Ojan Vafai o...@chromium.orgmailto:o...@chromium.org, Dirk Pranke 
dpra...@chromium.orgmailto:dpra...@chromium.org, WebKit Development 
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org
Subject: Re: [webkit-dev] Handling failing reftests

Right but you wouldn't know platform-specific issues until you land them. e.g. 
rounding errors, subtle font differences in edge cases, etc...

On Thu, Apr 12, 2012 at 1:30 PM, Jacob Goldstein 
jac...@adobe.commailto:jac...@adobe.com wrote:
Isn't the goal of writing a ref test that it is not platform specific?

From: Ryosuke Niwa rn...@webkit.orgmailto:rn...@webkit.org
Date: Thu, 12 Apr 2012 12:29:42 -0700
To: Ojan Vafai o...@chromium.orgmailto:o...@chromium.org
Cc: Dirk Pranke dpra...@chromium.orgmailto:dpra...@chromium.org, WebKit 
Development webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org
Subject: Re: [webkit-dev] Handling failing reftests

On Thu, Apr 12, 2012 at 11:53 AM, Ojan Vafai 
o...@chromium.orgmailto:o...@chromium.org wrote:
I agree that it's hard sometimes to construct reftests that work, but once 
you've done so, the cost on the project of maintaining the test is usually 
considerably lower than pixel tests.

Not so sure. There are cases where we have these platform specific failures for 
ref tests, and they're much harder to fix than rebaselining pixel results.

- Ryosuke


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] importing test suites (was: CSS 2.1 Test Suite)

2012-04-11 Thread Jacob Goldstein
+1 on not introducing new pixel tests and allowing someone other than the
test author to create the -expected file.

We may also be able to streamline some of this process by implementing
some helper scripts.  Ultimately, someone will still have to review new
files manually, but scripts should be able to speed up the process.

Alan Stearns and I plan to present some ideas around W3C and WebKit test
integration at the contributors meeting to elicit more feedback and
recruit people to support the effort.




On 4/11/12 3:29 PM, Dirk Pranke dpra...@chromium.org wrote:

On Wed, Apr 11, 2012 at 12:31 PM, Robert Hogan li...@roberthogan.net
wrote:
 On Wednesday 11 April 2012 20:27:23 Dirk Pranke wrote:

 What does clean up the existing folder entail?


 Just move any -expected.html file out of there and generate pixel
results.
 Or move the test and its -expected.html into a folder in fast/css.

Okay, following a conversation on #irc, I think I understand this
better. There do appear to be differences of opinion over what to do
here, so I will try to describe this from ground zero.

The CSS test suite has ~9000 tests. the correctness of which is
established manually (and visually). We would like to import these
tests and run them automatically. Some small number of tests (a few
hundred?) do have reference html, but most don't. Since most tests do
not come with reference html or expected PNGs, we have to add one or
the other.

Currently we have added ~400 or so to the tree, and when we add them
we have been writing -expected.html reference results to avoid
generating pixel results where possible.

This raises at least two questions, which have been confusing the
discussion until now:

1) What is the best way to add these tests to the tree? Should we add
them one-at-a-time with -expected files that we have created? Should
we add them all at once and SKIP the tests until we have generated
-expected results for each test? Or should we only import subsets that
have official -expected files?

2) Is it acceptable for someone other than the test author to write
-expected html references? What bar do we need to have to ensure that
the reference file will catch the bug the initial test is intended to
catch? Do we (or should we) maintain a similar bar for pixel tests
today?

Personally, I don't care that much about (1), as long as we can track
the provenance of the tests (where they came from) ... I would be
happy with us importing tests one at a time and adding our own
results, and hopefully feeding them back up stream to the W3C.

I care much more about (2): I believe we should not be introducing new
tests that only have pixel tests for results unless absolutely
necessary - the cost of maintaining pixel test results vastly
outweighs the potential benefit we get by having a reference file that
won't break the same way the test breaks. I believe it is reasonably
for someone other than the test author to write a reference html for a
test and get it to be good enough that it will provide more value than
a pixel test result, and we should be encouraging that.

I believe we're hoping to discuss this more broadly at the
contributors meeting, but it might be good to discuss here first.
Thoughts?

-- Dirk
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] importing test suites (was: CSS 2.1 Test Suite)

2012-04-11 Thread Jacob Goldstein
I understand, all valid points.

Don't people currently fix tests that are failing and that they didn't author?  
If so, isn't that similar as generating a new ref file in that you first have 
to understand what is being tested?  There would be the old pixel file to go 
from, so in many cases this may not be too hard.

On the other hand, having the test author create the ref file is definitely 
preferable.  What would we do in the case where a test author is no longer 
available?




From: Ryosuke Niwa rn...@webkit.orgmailto:rn...@webkit.org
Date: Wed, 11 Apr 2012 16:00:44 -0700
To: Jacob Goldstein jac...@adobe.commailto:jac...@adobe.com
Cc: Dirk Pranke dpra...@chromium.orgmailto:dpra...@chromium.org, Robert 
Hogan li...@roberthogan.netmailto:li...@roberthogan.net, 
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org 
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org
Subject: Re: [webkit-dev] importing test suites (was: CSS 2.1 Test Suite)

On Wed, Apr 11, 2012 at 3:52 PM, Jacob Goldstein 
jac...@adobe.commailto:jac...@adobe.com wrote:
+1 on not introducing new pixel tests and allowing someone other than the
test author to create the -expected file.

We may also be able to streamline some of this process by implementing
some helper scripts.  Ultimately, someone will still have to review new
files manually, but scripts should be able to speed up the process.

-1 on that. As I said on other threads about this topic, determining whether a 
reference file adequately detect all bugs a test is intended to test is hard, 
and losing the test coverage at the cost of lowering maintenance cost is not 
necessary a good thing.

Also, adding a reference file would mean that either we're adding -ref.html / 
-noref.html files or modifying reftest.list. If doing the former, then we can't 
use this approach in any directory where we use reftest.list at the moment 
because we explicitly prohibit mixing naming convention and reftest.list.

Modifying reftest.list is essentially modifying the test suite, and it seems 
like there is a consensus that we don't want to do it.

- Ryosuke

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


[webkit-dev] Test conversion to use W3C testharness.js

2012-03-09 Thread Jacob Goldstein
I recently uploaded a patch to https://bugs.webkit.org/show_bug.cgi?id=80709 
which converted an existing JavaScript regions parsing test to use the W3C 
testharness.js in place of js-test-pre.js/js-test-post.js.  This patch also 
places testharness.js and a WebKit-specific testharnessreport.js file in 
LayoutTests/fast/js/resources.

In cases where tests need to be written for both the WebKit and W3C testing 
suites, having the ability to use testharness.js with WebKit tests would mean 
that the test file only needs to be written once, and yet can still rely on the 
functionality from both test harnesses.   As it stands now, if someone needs to 
write a test for both suites, they either have to implement all functionality 
from scratch, or write one version of the test to use js-test-pre.js and 
another to use testharness.js.  The inclusion of testharness.js in the WebKit 
repository alleviates the need for this duplication of effort.  The 
testharnessreport.js file was intended for customization of the capabilities 
provided by testharness.js, I've added a call to 
layoutTestController.dumpAsText() to this file to allow it to function as a 
WebKit JavaScript test.

There is some potential issues that I wanted feedback on.

testharness.js uses a css file to format the test output.  The path to this css 
file is assembled by the script.  For the time being, I have not included the 
css file in the patch I attached to 80709.  The CSS file has no impact when 
testharness.js is used along with dumpAsText as I have in the test I converted. 
 It may provide some value, but I am going to suggest to the W3C that the CSS 
styling of results should move from testharness.js to testharnessreport.js 
(where it can be overridden).

Another concern is that changes to testharness.js in the future that break 
backward compatibility could then break WebKit tests.  This is an issue I plan 
to discuss with W3C members to determine if backward compatibility can be 
ensured.

I'd appreciate any thoughts or feedback people have on this issue.

Thanks,
Jacob
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Test conversion to use W3C testharness.js

2012-03-09 Thread Jacob Goldstein
LayoutTests/resources is fine with me – that was the location I considered 
using originally and only moved them to LayoutTests/fast/js/resources because 
that is where js-test-pre and –post are.

I'll upload a new patch with the files in LayoutTests/resources.



From: Ryosuke Niwa rn...@webkit.orgmailto:rn...@webkit.org
Date: Fri, 9 Mar 2012 14:37:18 -0800
To: Jacob Goldstein jac...@adobe.commailto:jac...@adobe.com
Cc: webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org 
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org
Subject: Re: [webkit-dev] Test conversion to use W3C testharness.js

On Fri, Mar 9, 2012 at 2:28 PM, Jacob Goldstein 
jac...@adobe.commailto:jac...@adobe.com wrote:
I recently uploaded a patch to https://bugs.webkit.org/show_bug.cgi?id=80709 
which converted an existing JavaScript regions parsing test to use the W3C 
testharness.js in place of js-test-pre.js/js-test-post.js.  This patch also 
places testharness.js and a WebKit-specific testharnessreport.js file in 
LayoutTests/fast/js/resources.

Can we put them in LayoutTests/resources instead? I always find it hard to 
remember the path fast/js/resources.

In cases where tests need to be written for both the WebKit and W3C testing 
suites, having the ability to use testharness.js with WebKit tests would mean 
that the test file only needs to be written once, and yet can still rely on the 
functionality from both test harnesses.   As it stands now, if someone needs to 
write a test for both suites, they either have to implement all functionality 
from scratch, or write one version of the test to use js-test-pre.js and 
another to use testharness.js.  The inclusion of testharness.js in the WebKit 
repository alleviates the need for this duplication of effort.  The 
testharnessreport.js file was intended for customization of the capabilities 
provided by testharness.js, I've added a call to 
layoutTestController.dumpAsText() to this file to allow it to function as a 
WebKit JavaScript test.

I support the effort to make layout tests more compatible with W3C tests.

Is the plan to use testharness.js for all new tests? Or only tests that we 
intend to contribute back to W3C?

Another concern is that changes to testharness.js in the future that break 
backward compatibility could then break WebKit tests.  This is an issue I plan 
to discuss with W3C members to determine if backward compatibility can be 
ensured.

There is no such a guarantee at the moment? That concerns me. On other hand, we 
wouldn't be importing ToT version of testharness.js so if such an 
incompatibility is introduced, we can migrate our tests on time as well.

- Ryosuke

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Test conversion to use W3C testharness.js

2012-03-09 Thread Jacob Goldstein
I replied to your comment in the bug, but will also copy below
-

I understand your concern.  Some of this is user defined, and the output
could also be improved via customizations to the testharnessreport.js file.

The message you see after Pass, for example, testParse: Assigned none
to property, expected return = none, is the user defined string from the
description argument of the assert_equals method.   assert_equals looks
like this:

assert_equals(actual, expected, description)

All of the methods in testharness.js include a description argument that
is output after the test result.  In the W3C test suite, the
testharness.css file formats the output into a slightly more readable
output.  Since we are dumping as text, that isn't possible, but other
customizations in testharnessreport.js are.  I will look into this further.





On 3/9/12 3:21 PM, Adam Barth aba...@webkit.org wrote:

I looked at the patch you uploaded, but it wasn't clear from the text
dump whether the subtests passed or failed.  Maybe testharness.js uses
a table and/or colors to present that information?  It's important
that we can easily determine which subtests pass or fail from a text
dump.

Adam


On Fri, Mar 9, 2012 at 3:19 PM, Jacob Goldstein jac...@adobe.com wrote:
 LayoutTests/resources is fine with me ­ that was the location I
considered
 using originally and only moved them to LayoutTests/fast/js/resources
 because that is where js-test-pre and ­post are.

 I'll upload a new patch with the files in LayoutTests/resources.



 From: Ryosuke Niwa rn...@webkit.org
 Date: Fri, 9 Mar 2012 14:37:18 -0800
 To: Jacob Goldstein jac...@adobe.com
 Cc: webkit-dev@lists.webkit.org webkit-dev@lists.webkit.org
 Subject: Re: [webkit-dev] Test conversion to use W3C testharness.js

 On Fri, Mar 9, 2012 at 2:28 PM, Jacob Goldstein jac...@adobe.com
wrote:

 I recently uploaded a patch
 to https://bugs.webkit.org/show_bug.cgi?id=80709 which converted an
existing
 JavaScript regions parsing test to use the W3C testharness.js in place
of
 js-test-pre.js/js-test-post.js.  This patch also places testharness.js
and a
 WebKit-specific testharnessreport.js file in
LayoutTests/fast/js/resources.


 Can we put them in LayoutTests/resources instead? I always find it hard
to
 remember the path fast/js/resources.

 In cases where tests need to be written for both the WebKit and W3C
 testing suites, having the ability to use testharness.js with WebKit
tests
 would mean that the test file only needs to be written once, and yet
can
 still rely on the functionality from both test harnesses.   As it
stands
 now, if someone needs to write a test for both suites, they either
have to
 implement all functionality from scratch, or write one version of the
test
 to use js-test-pre.js and another to use testharness.js.  The
inclusion of
 testharness.js in the WebKit repository alleviates the need for this
 duplication of effort.  The testharnessreport.js file was intended for
 customization of the capabilities provided by testharness.js, I've
added a
 call to layoutTestController.dumpAsText() to this file to allow it to
 function as a WebKit JavaScript test.


 I support the effort to make layout tests more compatible with W3C
tests.

 Is the plan to use testharness.js for all new tests? Or only tests that
we
 intend to contribute back to W3C?

 Another concern is that changes to testharness.js in the future that
break
 backward compatibility could then break WebKit tests.  This is an
issue I
 plan to discuss with W3C members to determine if backward
compatibility can
 be ensured.


 There is no such a guarantee at the moment? That concerns me. On other
hand,
 we wouldn't be importing ToT version of testharness.js so if such an
 incompatibility is introduced, we can migrate our tests on time as well.

 - Ryosuke


 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


Re: [webkit-dev] Test conversion to use W3C testharness.js

2012-03-09 Thread Jacob Goldstein
Is your concern that the Pass is not in all caps?

The current output looks like this:

Result  Test Name   Message
PasstestParse: Assigned none to property, expected return = none


The pass at the beginning of the line is the test result.  Had the test
failed, this would read Fail.

The next portion of the line, in this case testParse: Assigned none to
property, expected return = none, is the description string specified in
the call to assert_equals.  The description can be any string that the
user specifies.

The spacing of the results when they are dumped as text obscures the
difference between the test result and description - this is something I
can address in a number of ways depending on everyone's preference.  I'll
investigate further and come up with some suggested fixes.





On 3/9/12 4:11 PM, Adam Barth aba...@webkit.org wrote:

Can we use some CSS tricks like :before to put the word PASS at the
beginning of the line for each subtest?

Adam


On Fri, Mar 9, 2012 at 3:51 PM, Jacob Goldstein jac...@adobe.com wrote:
 I replied to your comment in the bug, but will also copy below
 -

 I understand your concern.  Some of this is user defined, and the output
 could also be improved via customizations to the testharnessreport.js
file.

 The message you see after Pass, for example, testParse: Assigned none
 to property, expected return = none, is the user defined string from
the
 description argument of the assert_equals method.   assert_equals looks
 like this:

 assert_equals(actual, expected, description)

 All of the methods in testharness.js include a description argument that
 is output after the test result.  In the W3C test suite, the
 testharness.css file formats the output into a slightly more readable
 output.  Since we are dumping as text, that isn't possible, but other
 customizations in testharnessreport.js are.  I will look into this
further.





 On 3/9/12 3:21 PM, Adam Barth aba...@webkit.org wrote:

I looked at the patch you uploaded, but it wasn't clear from the text
dump whether the subtests passed or failed.  Maybe testharness.js uses
a table and/or colors to present that information?  It's important
that we can easily determine which subtests pass or fail from a text
dump.

Adam


On Fri, Mar 9, 2012 at 3:19 PM, Jacob Goldstein jac...@adobe.com
wrote:
 LayoutTests/resources is fine with me ­ that was the location I
considered
 using originally and only moved them to LayoutTests/fast/js/resources
 because that is where js-test-pre and ­post are.

 I'll upload a new patch with the files in LayoutTests/resources.



 From: Ryosuke Niwa rn...@webkit.org
 Date: Fri, 9 Mar 2012 14:37:18 -0800
 To: Jacob Goldstein jac...@adobe.com
 Cc: webkit-dev@lists.webkit.org webkit-dev@lists.webkit.org
 Subject: Re: [webkit-dev] Test conversion to use W3C testharness.js

 On Fri, Mar 9, 2012 at 2:28 PM, Jacob Goldstein jac...@adobe.com
wrote:

 I recently uploaded a patch
 to https://bugs.webkit.org/show_bug.cgi?id=80709 which converted an
existing
 JavaScript regions parsing test to use the W3C testharness.js in
place
of
 js-test-pre.js/js-test-post.js.  This patch also places
testharness.js
and a
 WebKit-specific testharnessreport.js file in
LayoutTests/fast/js/resources.


 Can we put them in LayoutTests/resources instead? I always find it
hard
to
 remember the path fast/js/resources.

 In cases where tests need to be written for both the WebKit and W3C
 testing suites, having the ability to use testharness.js with WebKit
tests
 would mean that the test file only needs to be written once, and yet
can
 still rely on the functionality from both test harnesses.   As it
stands
 now, if someone needs to write a test for both suites, they either
have to
 implement all functionality from scratch, or write one version of the
test
 to use js-test-pre.js and another to use testharness.js.  The
inclusion of
 testharness.js in the WebKit repository alleviates the need for this
 duplication of effort.  The testharnessreport.js file was intended
for
 customization of the capabilities provided by testharness.js, I've
added a
 call to layoutTestController.dumpAsText() to this file to allow it to
 function as a WebKit JavaScript test.


 I support the effort to make layout tests more compatible with W3C
tests.

 Is the plan to use testharness.js for all new tests? Or only tests
that
we
 intend to contribute back to W3C?

 Another concern is that changes to testharness.js in the future that
break
 backward compatibility could then break WebKit tests.  This is an
issue I
 plan to discuss with W3C members to determine if backward
compatibility can
 be ensured.


 There is no such a guarantee at the moment? That concerns me. On other
hand,
 we wouldn't be importing ToT version of testharness.js so if such an
 incompatibility is introduced, we can migrate our tests on time as
well.

 - Ryosuke


 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http

Re: [webkit-dev] Test conversion to use W3C testharness.js

2012-03-09 Thread Jacob Goldstein
Got it. I'll add more space after the Pass and see what other formatting makes 
sense and get back to you next week.



On Mar 9, 2012, at 5:15 PM, Adam Barth aba...@webkit.org wrote:

 Ah, I see now that the word Pass is there at the beginning of the
 line.  I missed it because it ran together with the next couple words.
 Having spaces between the words would address my concern.
 
 Thanks!
 Adam
 
 
 2012/3/9 Jacob Goldstein jac...@adobe.com:
 Is your concern that the Pass is not in all caps?
 
 The current output looks like this:
 
 Result  Test Name   Message
 PasstestParse: Assigned none to property, expected return = none
 
 
 The pass at the beginning of the line is the test result.  Had the test
 failed, this would read Fail.
 
 The next portion of the line, in this case testParse: Assigned none to
 property, expected return = none, is the description string specified in
 the call to assert_equals.  The description can be any string that the
 user specifies.
 
 The spacing of the results when they are dumped as text obscures the
 difference between the test result and description - this is something I
 can address in a number of ways depending on everyone's preference.  I'll
 investigate further and come up with some suggested fixes.
 
 
 
 
 
 On 3/9/12 4:11 PM, Adam Barth aba...@webkit.org wrote:
 
 Can we use some CSS tricks like :before to put the word PASS at the
 beginning of the line for each subtest?
 
 Adam
 
 
 On Fri, Mar 9, 2012 at 3:51 PM, Jacob Goldstein jac...@adobe.com wrote:
 I replied to your comment in the bug, but will also copy below
 -
 
 I understand your concern.  Some of this is user defined, and the output
 could also be improved via customizations to the testharnessreport.js
 file.
 
 The message you see after Pass, for example, testParse: Assigned none
 to property, expected return = none, is the user defined string from
 the
 description argument of the assert_equals method.   assert_equals looks
 like this:
 
 assert_equals(actual, expected, description)
 
 All of the methods in testharness.js include a description argument that
 is output after the test result.  In the W3C test suite, the
 testharness.css file formats the output into a slightly more readable
 output.  Since we are dumping as text, that isn't possible, but other
 customizations in testharnessreport.js are.  I will look into this
 further.
 
 
 
 
 
 On 3/9/12 3:21 PM, Adam Barth aba...@webkit.org wrote:
 
 I looked at the patch you uploaded, but it wasn't clear from the text
 dump whether the subtests passed or failed.  Maybe testharness.js uses
 a table and/or colors to present that information?  It's important
 that we can easily determine which subtests pass or fail from a text
 dump.
 
 Adam
 
 
 On Fri, Mar 9, 2012 at 3:19 PM, Jacob Goldstein jac...@adobe.com
 wrote:
 LayoutTests/resources is fine with me ­ that was the location I
 considered
 using originally and only moved them to LayoutTests/fast/js/resources
 because that is where js-test-pre and ­post are.
 
 I'll upload a new patch with the files in LayoutTests/resources.
 
 
 
 From: Ryosuke Niwa rn...@webkit.org
 Date: Fri, 9 Mar 2012 14:37:18 -0800
 To: Jacob Goldstein jac...@adobe.com
 Cc: webkit-dev@lists.webkit.org webkit-dev@lists.webkit.org
 Subject: Re: [webkit-dev] Test conversion to use W3C testharness.js
 
 On Fri, Mar 9, 2012 at 2:28 PM, Jacob Goldstein jac...@adobe.com
 wrote:
 
 I recently uploaded a patch
 to https://bugs.webkit.org/show_bug.cgi?id=80709 which converted an
 existing
 JavaScript regions parsing test to use the W3C testharness.js in
 place
 of
 js-test-pre.js/js-test-post.js.  This patch also places
 testharness.js
 and a
 WebKit-specific testharnessreport.js file in
 LayoutTests/fast/js/resources.
 
 
 Can we put them in LayoutTests/resources instead? I always find it
 hard
 to
 remember the path fast/js/resources.
 
 In cases where tests need to be written for both the WebKit and W3C
 testing suites, having the ability to use testharness.js with WebKit
 tests
 would mean that the test file only needs to be written once, and yet
 can
 still rely on the functionality from both test harnesses.   As it
 stands
 now, if someone needs to write a test for both suites, they either
 have to
 implement all functionality from scratch, or write one version of the
 test
 to use js-test-pre.js and another to use testharness.js.  The
 inclusion of
 testharness.js in the WebKit repository alleviates the need for this
 duplication of effort.  The testharnessreport.js file was intended
 for
 customization of the capabilities provided by testharness.js, I've
 added a
 call to layoutTestController.dumpAsText() to this file to allow it to
 function as a WebKit JavaScript test.
 
 
 I support the effort to make layout tests more compatible with W3C
 tests.
 
 Is the plan to use testharness.js for all new tests? Or only tests
 that
 we
 intend to contribute back to W3C?
 
 Another concern is that changes