[webkit-dev] Importing W3C tests to webkit

2012-05-23 Thread Jacob Goldstein
At the WebKit contributor's meeting in April, we discussed a process for importing third party tests into the WebKit repository (specifically, from the W3C test repository). I documented the process that we came up with at the meeting on the WebKit wiki, here:

Re: [webkit-dev] Importing W3C tests to webkit

2012-05-23 Thread Ryosuke Niwa
As I have said in the past, we should just import all tests, and treat non-text, non-ref tests as pixel tests. If we wanted to reduce the number of pixel tests we import, then we should submit those patches to W3C instead of directly submitting them to WebKit. In general, I don't buy the argument

Re: [webkit-dev] Importing W3C tests to webkit

2012-05-23 Thread Jacob Goldstein
@lists.webkit.org Subject: Re: [webkit-dev] Importing W3C tests to webkit As I have said in the past, we should just import all tests, and treat non-text, non-ref tests as pixel tests. If we wanted to reduce the number of pixel tests we import, then we should submit those patches to W3C instead

Re: [webkit-dev] Importing W3C tests to webkit

2012-05-23 Thread Dirk Pranke
On Wed, May 23, 2012 at 11:56 AM, Ryosuke Niwa rn...@webkit.org wrote: As I have said in the past, we should just import all tests, and treat non-text, non-ref tests as pixel tests. If we wanted to reduce the number of pixel tests we import, then we should submit those patches to W3C instead of

Re: [webkit-dev] Importing W3C tests to webkit

2012-05-23 Thread Jacob Goldstein
Dirk, my apologies, I was on travel the week you replied and missed your message. I found it and will review / update now. On 5/23/12 1:25 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, May 23, 2012 at 11:56 AM, Ryosuke Niwa rn...@webkit.org wrote: As I have said in the past, we should

Re: [webkit-dev] Importing W3C tests to webkit

2012-05-23 Thread Ryosuke Niwa
On Wed, May 23, 2012 at 1:25 PM, Dirk Pranke dpra...@chromium.org wrote: Not so: if the tests we had had 100% coverage, then importing more tests would buy us nothing, but getting rid of the existing tests would be quite unfortunate. We certainly don't have 100% test coverage. Clearly

Re: [webkit-dev] Importing W3C tests to webkit

2012-05-23 Thread Dirk Pranke
On Wed, May 23, 2012 at 1:41 PM, Ryosuke Niwa rn...@webkit.org wrote: The only sane argument I've heard so far to gate pixel tests is that the correctness of such tests need to be manually inspected, which requires a lot of manual labor and is very error prone. I'm assuming the above

Re: [webkit-dev] Importing W3C tests to webkit

2012-05-23 Thread Maciej Stachowiak
On May 23, 2012, at 2:16 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, May 23, 2012 at 1:41 PM, Ryosuke Niwa rn...@webkit.org wrote: The only sane argument I've heard so far to gate pixel tests is that the correctness of such tests need to be manually inspected, which requires a lot

Re: [webkit-dev] Importing W3C tests to webkit

2012-05-23 Thread Jacob Goldstein
On 5/23/12 2:30 PM, Maciej Stachowiak m...@apple.com wrote: On May 23, 2012, at 2:16 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, May 23, 2012 at 1:41 PM, Ryosuke Niwa rn...@webkit.org wrote: The only sane argument I've heard so far to gate pixel tests is that the correctness of

Re: [webkit-dev] Importing W3C tests to webkit

2012-05-23 Thread Dirk Pranke
On Wed, May 23, 2012 at 2:30 PM, Maciej Stachowiak m...@apple.com wrote: Are you concerned just about the actual pixel results or also about keeping render tree dumps up to date? Both are more maintenance than a text-only test. In my experience, maintaining pixel tests is more expensive, but

Re: [webkit-dev] Importing W3C tests to webkit

2012-05-23 Thread Maciej Stachowiak
On May 23, 2012, at 3:13 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, May 23, 2012 at 2:30 PM, Maciej Stachowiak m...@apple.com wrote: Are you concerned just about the actual pixel results or also about keeping render tree dumps up to date? Both are more maintenance than a

Re: [webkit-dev] Importing W3C tests to webkit

2012-05-23 Thread Dirk Pranke
On Wed, May 23, 2012 at 2:59 PM, Jacob Goldstein jac...@adobe.com wrote: As a side note to this discussion, there is talk in the W3C community regarding their test approval process.  At the recent working group meetings in Germany the idea was floated to simply approve all tests that are