Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?

[Ryosuke Niwa]

On Wed, Apr 11, 2012 at 4:45 PM, Ojan Vafai  wrote:
>
> I agree with the sentiment that we should be upstreaming these to the W3C,
> but I don't see why we would require upstreaming them first instead of
> committing them locally and then upstreaming them.
>

How do we know whether a reference file came from W3C repository or not.
(Maybe by the fact it's named *-expected.html?)

Also, there are directories with reftest.list but without reference files
for some tests. The last time I checked, you were opposed to having bot
reftest.list and *-expected.html / *-expected-mismatch.html files. Have you
changed your opinion on this?

- Ryosuke
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Process for making changes that affect layout test results

[Dirk Pranke]

On Wed, Apr 11, 2012 at 4:56 PM, Tony Payne  wrote:
> Some of these test result changes turn out to be the effect of incorrect
> test cases.
>
> For example, compositing/color-matching/image-color-matching.html compares
> an image without a color profile with an image tagged with what it calls a
> "default RGB" profile. However, this "default RGB" profile is not sRGB, so
> the test is wrongly[1] assuming these images should be displayed the same.
> I'd like to replace all of the resources containing random RGB profiles with
> sRGB profiles instead.
>
> 1) Is this a good idea? Is there a reason why these image resources have
> non-standard profiles in them?

I don't know the answer to this, but hopefully someone else will.

> 2) How do I coordinate this, as changing the image resources seems like it
> will affect all the ports?

To the extent you can notify other port maintainers, I'm sure they
would appreciate it. Otherwise, treat it the same way you would treat
any other change ... It will just be more painful :).

-- Dirk
>
> Tony
>
> [1] http://www.w3.org/Graphics/Color/sRGB.html
>
> On Wed, Apr 11, 2012 at 3:36 PM, Dirk Pranke  wrote:
>>
>> use run-webkit-tests (or attempt to pull the results from the trybots
>> by hand). You cannot pull new baselines from the try bots using
>> rebaseline.py.
>>
>> -- Dirk
>>
>> On Wed, Apr 11, 2012 at 3:26 PM, Tony Payne  wrote:
>> > Is the best way to do that to use run_webkit_tests to generate a new
>> > baseline or does Tools/Scripts/webkitpy/tool/commands/rebaseline.py
>> > support
>> > pulling new baselines from the trybots?
>> >
>> > Thanks,
>> > Tony
>> >
>> > On Wed, Apr 11, 2012 at 2:13 PM, Ojan Vafai  wrote:
>> >>
>> >> Typically, if you're working on Chromium Linux or Win, you'd include
>> >> the
>> >> new expected results for that platform in your initial
>> >> commit/code-review as
>> >> well.
>> >>
>> >> On Wed, Apr 11, 2012 at 2:09 PM, Tony Payne 
>> >> wrote:
>> >>>
>> >>> All code I'm changing is inside of #if PLATFORM(CHROMIUM) blocks.
>> >>>
>> >>> Thanks for the quick answer.
>> >>>
>> >>> Tony
>> >>>
>> >>>
>> >>> On Wed, Apr 11, 2012 at 2:03 PM, Ryosuke Niwa 
>> >>> wrote:
>> 
>>  On Wed, Apr 11, 2012 at 1:57 PM, Tony Payne 
>>  wrote:
>> >
>> > Given the recent discussion on test_expectations.txt, perhaps the
>> > answer to my question is still up in the air.
>> >
>> > I'm working on a change that I expect to require changing the
>> > expectations for about 75 tests on chromium win and
>> > linux. https://trac.webkit.org/wiki/Rebaseline seems to only cover
>> > the
>> > gardening work to rebaseline after the commit. I cannot find any
>> > wiki pages
>> > that describe what the original author is expected to do when making
>> > visual
>> > changes. Should I attempt to rebaseline manually? Should I mark the
>> > tests as
>> > failing? Should I just check in and let the bots go red?
>> 
>> 
>>  Just land the patch and rebaseline the tests. Please also coordinate
>>  with Chromium port's WebKit gardener when landing this patch.
>> 
>>  Also, does this patch only affect Chromium Windows and Linux, and not
>>  GTK, Qt, Windows, etc...? If the answer is no, and will affect other
>>  non-Chromium ports, then you're also responsible for rebaselining or
>>  coordinating with other ports to make sure you don't break tests on
>>  their
>>  ports as well.
>> 
>>  - Ryosuke
>> 
>> >>>
>> >>>
>> >>> ___
>> >>> webkit-dev mailing list
>> >>> webkit-dev@lists.webkit.org
>> >>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>> >>>
>> >>
>> >
>> >
>> > ___
>> > webkit-dev mailing list
>> > webkit-dev@lists.webkit.org
>> > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>> >
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Process for making changes that affect layout test results

[Tony Payne]

Some of these test result changes turn out to be the effect of incorrect
test cases.

For example, compositing/color-matching/image-color-matching.html compares
an image without a color profile with an image tagged with what it calls a
"default RGB" profile. However, this "default RGB" profile is not sRGB, so
the test is wrongly[1] assuming these images should be displayed the same.
I'd like to replace all of the resources containing random RGB profiles
with sRGB profiles instead.

1) Is this a good idea? Is there a reason why these image resources have
non-standard profiles in them?
2) How do I coordinate this, as changing the image resources seems like it
will affect all the ports?

Tony

[1] http://www.w3.org/Graphics/Color/sRGB.html

On Wed, Apr 11, 2012 at 3:36 PM, Dirk Pranke  wrote:

> use run-webkit-tests (or attempt to pull the results from the trybots
> by hand). You cannot pull new baselines from the try bots using
> rebaseline.py.
>
> -- Dirk
>
> On Wed, Apr 11, 2012 at 3:26 PM, Tony Payne  wrote:
> > Is the best way to do that to use run_webkit_tests to generate a new
> > baseline or does Tools/Scripts/webkitpy/tool/commands/rebaseline.py
> support
> > pulling new baselines from the trybots?
> >
> > Thanks,
> > Tony
> >
> > On Wed, Apr 11, 2012 at 2:13 PM, Ojan Vafai  wrote:
> >>
> >> Typically, if you're working on Chromium Linux or Win, you'd include the
> >> new expected results for that platform in your initial
> commit/code-review as
> >> well.
> >>
> >> On Wed, Apr 11, 2012 at 2:09 PM, Tony Payne 
> wrote:
> >>>
> >>> All code I'm changing is inside of #if PLATFORM(CHROMIUM) blocks.
> >>>
> >>> Thanks for the quick answer.
> >>>
> >>> Tony
> >>>
> >>>
> >>> On Wed, Apr 11, 2012 at 2:03 PM, Ryosuke Niwa 
> wrote:
> 
>  On Wed, Apr 11, 2012 at 1:57 PM, Tony Payne 
> wrote:
> >
> > Given the recent discussion on test_expectations.txt, perhaps the
> > answer to my question is still up in the air.
> >
> > I'm working on a change that I expect to require changing the
> > expectations for about 75 tests on chromium win and
> > linux. https://trac.webkit.org/wiki/Rebaseline seems to only cover
> the
> > gardening work to rebaseline after the commit. I cannot find any
> wiki pages
> > that describe what the original author is expected to do when making
> visual
> > changes. Should I attempt to rebaseline manually? Should I mark the
> tests as
> > failing? Should I just check in and let the bots go red?
> 
> 
>  Just land the patch and rebaseline the tests. Please also coordinate
>  with Chromium port's WebKit gardener when landing this patch.
> 
>  Also, does this patch only affect Chromium Windows and Linux, and not
>  GTK, Qt, Windows, etc...? If the answer is no, and will affect other
>  non-Chromium ports, then you're also responsible for rebaselining or
>  coordinating with other ports to make sure you don't break tests on
> their
>  ports as well.
> 
>  - Ryosuke
> 
> >>>
> >>>
> >>> ___
> >>> webkit-dev mailing list
> >>> webkit-dev@lists.webkit.org
> >>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
> >>>
> >>
> >
> >
> > ___
> > webkit-dev mailing list
> > webkit-dev@lists.webkit.org
> > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
> >
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] PSA: Chromium layout test fallbacks AKA down with graphs, long live trees

[Dirk Pranke]

This change has been made.

I will note, however, that the fallback order for all WebKit ports is
still not a tree :( The webkit2 ports will use results in platform/wk2
in addition to their non-wk2 fallback paths :(

-- Dirk

On Mon, Mar 26, 2012 at 12:00 PM, Ojan Vafai  wrote:
> Sometime this week or next, we plan to change the Chromium layout test
> fallback order. We'll be changing it so all chromium ports fallback to
> platform/chromium, then to platform/mac. They will never fallback to
> platform/win or platform/mac-* as they do today.
>
> The benefits of this change are that it makes the fallback order much easier
> to reason about for both human beings and code. I believe it makes the
> fallback order for all WebKit ports a tree instead of a complex graph. This
> will have minimal impact on the number of expected result files in the tree
> since only a few hundred chromium tests currently fallback to platform/win
> and/or platform/mac-*.
>
> Ojan
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?

[Ojan Vafai]

On Wed, Apr 11, 2012 at 3:48 PM, Dirk Pranke  wrote:

> On Wed, Apr 11, 2012 at 3:46 PM, Dirk Pranke  wrote:
> > On Fri, Mar 9, 2012 at 2:49 PM, Sam Weinig  wrote:
> >>
> >> On Mar 7, 2012, at 4:41 PM, Ojan Vafai wrote:
> >>
> >> I just did a first pass a greening the Chromium Lion
> >> bot: http://trac.webkit.org/changeset/110096. Of these hundreds of
> tests,
> >>  ~99% of them are perfect candidates for being reftests (e.g. they
> contain
> >> one line of text and a solid box or two under the text), but most of
> them
> >> are in the CSS imported test suites.
> >>
> >> Is it kosher to convert them to reftests or should we leave pixel tests
> from
> >> imported test suites alone?
> >>
> >>
> >> If we want to make these ref tests, it probably makes more sense to do
> that
> >> work with the CSS WG, so that they can be part of the standard test
> suite.
> >> Until then, I think we should keep them regular pixel tests.
> >>
> >
> > Note that this thread (to resurrect it just-after-easter because it's
> > timely) is directly relevant to the note I just sent out about
> > importing test suites.
> >
> > I, at least, would like some clarification ... if we are importing
> > tests that have no accompanying "expected result" (and are expected to
> > be inspected manually for correctness), is it acceptable to write
> > reference html for the tests, or do they have to be imported as pixel
> > tests?
> >
>
> Put differently, we either need to add an -expected html or an
> -expected png. Does it have to be the latter?
>

I strongly prefer adding -expected.html files. While there is a chance the
reference may not cover all the code paths the original test intended, I
believe our overall test coverage will be better with more reftests and
fewer pixel tests, not to mention our project-wide sanity from spending
less time doing expectations file management. Pixel tests accidentally let
bugs through all the time because not all ports run them and because it's
often hard to tell if a new result is correct.

I agree with the sentiment that we should be upstreaming these to the W3C,
but I don't see why we would require upstreaming them first instead of
committing them locally and then upstreaming them. If there are people
willing to do the work, lets allow them do it either way.


> > I personally think it's acceptable, but I understand that there might
> > be a difference of opinion.
> >
> > https://bugs.webkit.org/show_bug.cgi?id=72987 is an example of this.
> >
> > -- Dirk
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] importing test suites (was: CSS 2.1 Test Suite)

[Dirk Pranke]

On Wed, Apr 11, 2012 at 4:15 PM, Ryosuke Niwa  wrote:
> On Wed, Apr 11, 2012 at 4:05 PM, Jacob Goldstein  wrote:
>>
>> Don't people currently fix tests that are failing and that they didn't
>> author?
>
>
> We don't fix imported tests if you meant modifying html files, etc... to fix
> a broken test file.
>
>> If so, isn't that similar as generating a new ref file in that you first
>> have to understand what is being tested?  There would be the old pixel file
>> to go from, so in many cases this may not be too hard.
>
>
> Yes, but determining whether a pixel result is correct or not is easier than
> determining whether a reference file is correct AND adequately addresses all
> intended test purposes.

It may sometimes be easier to tell if a pixel result is correct or
not, but not always: I've stared at plenty of failing pixel tests
where there were no visible changes at all.

Moreover, I think that requiring a reference file to address "all
intended test purposes" is too high of a bar to allow it to replace a
PNG; I'm not even sure if it's a practical goal at all, especially
given the limited amount of experience we have with reference tests. I
think there is some lower bar of goodness that we can reach in
practice through experience.

-- Dirk
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] importing test suites (was: CSS 2.1 Test Suite)

[Ryosuke Niwa]

On Wed, Apr 11, 2012 at 4:05 PM, Jacob Goldstein  wrote:
>
> Don't people currently fix tests that are failing and that they didn't
> author?
>

We don't fix imported tests if you meant modifying html files, etc... to
fix a broken test file.

If so, isn't that similar as generating a new ref file in that you first
> have to understand what is being tested?  There would be the old pixel file
> to go from, so in many cases this may not be too hard.
>

Yes, but determining whether a pixel result is correct or not is easier
than determining whether a reference file is correct AND adequately
addresses all intended test purposes.

On the other hand, having the test author create the ref file is definitely
> preferable.  What would we do in the case where a test author is no longer
> available?
>

Someone needs to figure out the intended purpose of the test and make a
call. I've done quite few of those for editing tests we have (not imported
tests); I've converted them to dumpAsText and dump-as-markup tests.

- Ryosuke
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] importing test suites (was: CSS 2.1 Test Suite)

[Dirk Pranke]

On Wed, Apr 11, 2012 at 4:00 PM, Ryosuke Niwa  wrote:
> On Wed, Apr 11, 2012 at 3:52 PM, Jacob Goldstein  wrote:
>>
>> +1 on not introducing new pixel tests and allowing someone other than the
>> test author to create the -expected file.
>>
>> We may also be able to streamline some of this process by implementing
>> some helper scripts.  Ultimately, someone will still have to review new
>> files manually, but scripts should be able to speed up the process.
>
>
> -1 on that. As I said on other threads about this topic, determining whether
> a reference file adequately detect all bugs a test is intended to test is
> hard, and losing the test coverage at the cost of lowering maintenance cost
> is not necessary a good thing.
>
> Also, adding a reference file would mean that either we're adding -ref.html
> / -noref.html files or modifying reftest.list. If doing the former, then we
> can't use this approach in any directory where we use reftest.list at the
> moment because we explicitly prohibit mixing naming convention and
> reftest.list.
>
> Modifying reftest.list is essentially modifying the test suite, and it seems
> like there is a consensus that we don't want to do it.
>

This is a quibble compared to the first paragraph, but the last two
paragraphs are merely implementation details (especially if we think
we're generating reftest.list from  tags embedded in the tests).
I think if we agree that it's okay to add -ref.html / -noref.html
files for tests we can revisit what the best way to manage such a
process is. I think the initial guideline was established when we
thought that an imported test suite would come with all of the needed
reference files, and in such a case I agree we should leave it as
"stock" as possible.

-- Dirk
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] importing test suites (was: CSS 2.1 Test Suite)

[Jacob Goldstein]

I understand, all valid points.

Don't people currently fix tests that are failing and that they didn't author?  
If so, isn't that similar as generating a new ref file in that you first have 
to understand what is being tested?  There would be the old pixel file to go 
from, so in many cases this may not be too hard.

On the other hand, having the test author create the ref file is definitely 
preferable.  What would we do in the case where a test author is no longer 
available?




From: Ryosuke Niwa mailto:rn...@webkit.org>>
Date: Wed, 11 Apr 2012 16:00:44 -0700
To: Jacob Goldstein mailto:jac...@adobe.com>>
Cc: Dirk Pranke mailto:dpra...@chromium.org>>, Robert 
Hogan mailto:li...@roberthogan.net>>, 
"webkit-dev@lists.webkit.org" 
mailto:webkit-dev@lists.webkit.org>>
Subject: Re: [webkit-dev] importing test suites (was: CSS 2.1 Test Suite)

On Wed, Apr 11, 2012 at 3:52 PM, Jacob Goldstein 
mailto:jac...@adobe.com>> wrote:
+1 on not introducing new pixel tests and allowing someone other than the
test author to create the -expected file.

We may also be able to streamline some of this process by implementing
some helper scripts.  Ultimately, someone will still have to review new
files manually, but scripts should be able to speed up the process.

-1 on that. As I said on other threads about this topic, determining whether a 
reference file adequately detect all bugs a test is intended to test is hard, 
and losing the test coverage at the cost of lowering maintenance cost is not 
necessary a good thing.

Also, adding a reference file would mean that either we're adding -ref.html / 
-noref.html files or modifying reftest.list. If doing the former, then we can't 
use this approach in any directory where we use reftest.list at the moment 
because we explicitly prohibit mixing naming convention and reftest.list.

Modifying reftest.list is essentially modifying the test suite, and it seems 
like there is a consensus that we don't want to do it.

- Ryosuke

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] importing test suites (was: CSS 2.1 Test Suite)

[Ryosuke Niwa]

On Wed, Apr 11, 2012 at 3:52 PM, Jacob Goldstein  wrote:

> +1 on not introducing new pixel tests and allowing someone other than the
> test author to create the -expected file.
>
> We may also be able to streamline some of this process by implementing
> some helper scripts.  Ultimately, someone will still have to review new
> files manually, but scripts should be able to speed up the process.
>

-1 on that. As I said on other threads about this topic, determining
whether a reference file adequately detect all bugs a test is intended to
test is hard, and losing the test coverage at the cost of lowering
maintenance cost is not necessary a good thing.

Also, adding a reference file would mean that either we're adding -ref.html
/ -noref.html files or modifying reftest.list. If doing the former, then we
can't use this approach in any directory where we use reftest.list at the
moment because we explicitly prohibit mixing naming convention and
reftest.list.

Modifying reftest.list is essentially modifying the test suite, and it
seems like there is a consensus that we don't want to do it.

- Ryosuke
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] importing test suites (was: CSS 2.1 Test Suite)

[Jacob Goldstein]

+1 on not introducing new pixel tests and allowing someone other than the
test author to create the -expected file.

We may also be able to streamline some of this process by implementing
some helper scripts.  Ultimately, someone will still have to review new
files manually, but scripts should be able to speed up the process.

Alan Stearns and I plan to present some ideas around W3C and WebKit test
integration at the contributors meeting to elicit more feedback and
recruit people to support the effort.




On 4/11/12 3:29 PM, "Dirk Pranke"  wrote:

>On Wed, Apr 11, 2012 at 12:31 PM, Robert Hogan 
>wrote:
>> On Wednesday 11 April 2012 20:27:23 Dirk Pranke wrote:
>>>
>>> What does "clean up the existing folder" entail?
>>>
>>
>> Just move any -expected.html file out of there and generate pixel
>>results.
>> Or move the test and its -expected.html into a folder in fast/css.
>
>Okay, following a conversation on #irc, I think I understand this
>better. There do appear to be differences of opinion over what to do
>here, so I will try to describe this from ground zero.
>
>The CSS test suite has ~9000 tests. the correctness of which is
>established manually (and visually). We would like to import these
>tests and run them automatically. Some small number of tests (a few
>hundred?) do have reference html, but most don't. Since most tests do
>not come with reference html or expected PNGs, we have to add one or
>the other.
>
>Currently we have added ~400 or so to the tree, and when we add them
>we have been writing -expected.html reference results to avoid
>generating pixel results where possible.
>
>This raises at least two questions, which have been confusing the
>discussion until now:
>
>1) What is the best way to add these tests to the tree? Should we add
>them one-at-a-time with -expected files that we have created? Should
>we add them all at once and SKIP the tests until we have generated
>-expected results for each test? Or should we only import subsets that
>have official -expected files?
>
>2) Is it acceptable for someone other than the test author to write
>-expected html references? What bar do we need to have to ensure that
>the reference file will catch the bug the initial test is intended to
>catch? Do we (or should we) maintain a similar bar for pixel tests
>today?
>
>Personally, I don't care that much about (1), as long as we can track
>the provenance of the tests (where they came from) ... I would be
>happy with us importing tests one at a time and adding our own
>results, and hopefully feeding them back up stream to the W3C.
>
>I care much more about (2): I believe we should not be introducing new
>tests that only have pixel tests for results unless absolutely
>necessary - the cost of maintaining pixel test results vastly
>outweighs the potential benefit we get by having a reference file that
>won't break the same way the test breaks. I believe it is reasonably
>for someone other than the test author to write a reference html for a
>test and get it to be good enough that it will provide more value than
>a pixel test result, and we should be encouraging that.
>
>I believe we're hoping to discuss this more broadly at the
>contributors meeting, but it might be good to discuss here first.
>Thoughts?
>
>-- Dirk
>___
>webkit-dev mailing list
>webkit-dev@lists.webkit.org
>http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] importing test suites (was: CSS 2.1 Test Suite)

[Robert Hogan]

On Wednesday 11 April 2012 23:29:56 Dirk Pranke wrote:
> 
> 1) What is the best way to add these tests to the tree? Should we add
> them one-at-a-time with -expected files that we have created? Should
> we add them all at once and SKIP the tests until we have generated
> -expected results for each test? Or should we only import subsets that
> have official -expected files?

Here is a good example of this in action:

http://trac.webkit.org/changeset/113076

The tests all come unmodified from the CSS test suite, the -expected.html 
files were created by me.

The CSS test suite does not used the -expected.html semantic.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?

[Dirk Pranke]

On Wed, Apr 11, 2012 at 3:46 PM, Dirk Pranke  wrote:
> On Fri, Mar 9, 2012 at 2:49 PM, Sam Weinig  wrote:
>>
>> On Mar 7, 2012, at 4:41 PM, Ojan Vafai wrote:
>>
>> I just did a first pass a greening the Chromium Lion
>> bot: http://trac.webkit.org/changeset/110096. Of these hundreds of tests,
>>  ~99% of them are perfect candidates for being reftests (e.g. they contain
>> one line of text and a solid box or two under the text), but most of them
>> are in the CSS imported test suites.
>>
>> Is it kosher to convert them to reftests or should we leave pixel tests from
>> imported test suites alone?
>>
>>
>> If we want to make these ref tests, it probably makes more sense to do that
>> work with the CSS WG, so that they can be part of the standard test suite.
>> Until then, I think we should keep them regular pixel tests.
>>
>
> Note that this thread (to resurrect it just-after-easter because it's
> timely) is directly relevant to the note I just sent out about
> importing test suites.
>
> I, at least, would like some clarification ... if we are importing
> tests that have no accompanying "expected result" (and are expected to
> be inspected manually for correctness), is it acceptable to write
> reference html for the tests, or do they have to be imported as pixel
> tests?
>

Put differently, we either need to add an -expected html or an
-expected png. Does it have to be the latter?

> I personally think it's acceptable, but I understand that there might
> be a difference of opinion.
>
> https://bugs.webkit.org/show_bug.cgi?id=72987 is an example of this.
>
> -- Dirk
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?

[Dirk Pranke]

On Fri, Mar 9, 2012 at 2:49 PM, Sam Weinig  wrote:
>
> On Mar 7, 2012, at 4:41 PM, Ojan Vafai wrote:
>
> I just did a first pass a greening the Chromium Lion
> bot: http://trac.webkit.org/changeset/110096. Of these hundreds of tests,
>  ~99% of them are perfect candidates for being reftests (e.g. they contain
> one line of text and a solid box or two under the text), but most of them
> are in the CSS imported test suites.
>
> Is it kosher to convert them to reftests or should we leave pixel tests from
> imported test suites alone?
>
>
> If we want to make these ref tests, it probably makes more sense to do that
> work with the CSS WG, so that they can be part of the standard test suite.
> Until then, I think we should keep them regular pixel tests.
>

Note that this thread (to resurrect it just-after-easter because it's
timely) is directly relevant to the note I just sent out about
importing test suites.

I, at least, would like some clarification ... if we are importing
tests that have no accompanying "expected result" (and are expected to
be inspected manually for correctness), is it acceptable to write
reference html for the tests, or do they have to be imported as pixel
tests?

I personally think it's acceptable, but I understand that there might
be a difference of opinion.

https://bugs.webkit.org/show_bug.cgi?id=72987 is an example of this.

-- Dirk
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Process for making changes that affect layout test results

[Dirk Pranke]

use run-webkit-tests (or attempt to pull the results from the trybots
by hand). You cannot pull new baselines from the try bots using
rebaseline.py.

-- Dirk

On Wed, Apr 11, 2012 at 3:26 PM, Tony Payne  wrote:
> Is the best way to do that to use run_webkit_tests to generate a new
> baseline or does Tools/Scripts/webkitpy/tool/commands/rebaseline.py support
> pulling new baselines from the trybots?
>
> Thanks,
> Tony
>
> On Wed, Apr 11, 2012 at 2:13 PM, Ojan Vafai  wrote:
>>
>> Typically, if you're working on Chromium Linux or Win, you'd include the
>> new expected results for that platform in your initial commit/code-review as
>> well.
>>
>> On Wed, Apr 11, 2012 at 2:09 PM, Tony Payne  wrote:
>>>
>>> All code I'm changing is inside of #if PLATFORM(CHROMIUM) blocks.
>>>
>>> Thanks for the quick answer.
>>>
>>> Tony
>>>
>>>
>>> On Wed, Apr 11, 2012 at 2:03 PM, Ryosuke Niwa  wrote:

 On Wed, Apr 11, 2012 at 1:57 PM, Tony Payne  wrote:
>
> Given the recent discussion on test_expectations.txt, perhaps the
> answer to my question is still up in the air.
>
> I'm working on a change that I expect to require changing the
> expectations for about 75 tests on chromium win and
> linux. https://trac.webkit.org/wiki/Rebaseline seems to only cover the
> gardening work to rebaseline after the commit. I cannot find any wiki 
> pages
> that describe what the original author is expected to do when making 
> visual
> changes. Should I attempt to rebaseline manually? Should I mark the tests 
> as
> failing? Should I just check in and let the bots go red?


 Just land the patch and rebaseline the tests. Please also coordinate
 with Chromium port's WebKit gardener when landing this patch.

 Also, does this patch only affect Chromium Windows and Linux, and not
 GTK, Qt, Windows, etc...? If the answer is no, and will affect other
 non-Chromium ports, then you're also responsible for rebaselining or
 coordinating with other ports to make sure you don't break tests on their
 ports as well.

 - Ryosuke

>>>
>>>
>>> ___
>>> webkit-dev mailing list
>>> webkit-dev@lists.webkit.org
>>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>>
>>
>
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

[webkit-dev] importing test suites (was: CSS 2.1 Test Suite)

[Dirk Pranke]

On Wed, Apr 11, 2012 at 12:31 PM, Robert Hogan  wrote:
> On Wednesday 11 April 2012 20:27:23 Dirk Pranke wrote:
>>
>> What does "clean up the existing folder" entail?
>>
>
> Just move any -expected.html file out of there and generate pixel results.
> Or move the test and its -expected.html into a folder in fast/css.

Okay, following a conversation on #irc, I think I understand this
better. There do appear to be differences of opinion over what to do
here, so I will try to describe this from ground zero.

The CSS test suite has ~9000 tests. the correctness of which is
established manually (and visually). We would like to import these
tests and run them automatically. Some small number of tests (a few
hundred?) do have reference html, but most don't. Since most tests do
not come with reference html or expected PNGs, we have to add one or
the other.

Currently we have added ~400 or so to the tree, and when we add them
we have been writing -expected.html reference results to avoid
generating pixel results where possible.

This raises at least two questions, which have been confusing the
discussion until now:

1) What is the best way to add these tests to the tree? Should we add
them one-at-a-time with -expected files that we have created? Should
we add them all at once and SKIP the tests until we have generated
-expected results for each test? Or should we only import subsets that
have official -expected files?

2) Is it acceptable for someone other than the test author to write
-expected html references? What bar do we need to have to ensure that
the reference file will catch the bug the initial test is intended to
catch? Do we (or should we) maintain a similar bar for pixel tests
today?

Personally, I don't care that much about (1), as long as we can track
the provenance of the tests (where they came from) ... I would be
happy with us importing tests one at a time and adding our own
results, and hopefully feeding them back up stream to the W3C.

I care much more about (2): I believe we should not be introducing new
tests that only have pixel tests for results unless absolutely
necessary - the cost of maintaining pixel test results vastly
outweighs the potential benefit we get by having a reference file that
won't break the same way the test breaks. I believe it is reasonably
for someone other than the test author to write a reference html for a
test and get it to be good enough that it will provide more value than
a pixel test result, and we should be encouraging that.

I believe we're hoping to discuss this more broadly at the
contributors meeting, but it might be good to discuss here first.
Thoughts?

-- Dirk
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Process for making changes that affect layout test results

[Tony Payne]

Is the best way to do that to use run_webkit_tests to generate a new
baseline or does Tools/Scripts/webkitpy/tool/commands/rebaseline.py support
pulling new baselines from the trybots?

Thanks,
Tony

On Wed, Apr 11, 2012 at 2:13 PM, Ojan Vafai  wrote:

> Typically, if you're working on Chromium Linux or Win, you'd include the
> new expected results for that platform in your initial commit/code-review
> as well.
>
> On Wed, Apr 11, 2012 at 2:09 PM, Tony Payne  wrote:
>
>> All code I'm changing is inside of #if PLATFORM(CHROMIUM) blocks.
>>
>> Thanks for the quick answer.
>>
>> Tony
>>
>>
>> On Wed, Apr 11, 2012 at 2:03 PM, Ryosuke Niwa  wrote:
>>
>>> On Wed, Apr 11, 2012 at 1:57 PM, Tony Payne  wrote:
>>>
 Given the recent discussion on test_expectations.txt, perhaps the
 answer to my question is still up in the air.

 I'm working on a change that I expect to require changing the
 expectations for about 75 tests on chromium win and linux.
 https://trac.webkit.org/wiki/Rebaseline seems to only cover the
 gardening work to rebaseline after the commit. I cannot find any wiki pages
 that describe what the original author is expected to do when making visual
 changes. Should I attempt to rebaseline manually? Should I mark the tests
 as failing? Should I just check in and let the bots go red?

>>>
>>> Just land the patch and rebaseline the tests. Please also coordinate
>>> with Chromium port's WebKit gardener when landing this patch.
>>>
>>> Also, does this patch only affect Chromium Windows and Linux, and not
>>> GTK, Qt, Windows, etc...? If the answer is no, and will affect other
>>> non-Chromium ports, then you're also responsible for rebaselining or
>>> coordinating with other ports to make sure you don't break tests on their
>>> ports as well.
>>>
>>> - Ryosuke
>>>
>>>
>>
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>
>>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] NPAPI plugin may be broken in mac latest sources?

[Eric Seidel]

The first step is to file a bug.  Then we can CC the right people and
triage your troubles.

http://www.webkit.org/quality/reporting.html

On Wed, Apr 11, 2012 at 3:02 PM, Rudi Sherry  wrote:
> Yesterday I updated to the latest webkit sources for the mac (`Tools\Scripts 
> update-webkit`) and rebuilt.
>
> Now, when I run Safari using them (`Tools\Scripts\run-safari --debug`) and 
> open a file for my plug-in (which works fine), then go Back/Forward QUICKLY, 
> NPP_NewStream() is never called for Forward.  If I go Back/Forward SLOWLY, 
> then it gets called for Forward.
>
> In between Back and Forward, the NP_Shutdown() .. NP_Initialize, NPP_New is 
> called in both cases, and I've verified they're all returning no error (or 
> valid values).
>
> The release of Safari I have (5.1.5) does not exhibit this behavior (I'm on 
> Lion).
>
> I don't see anything in the bug database about this; before I enter a bug, is 
> this known?
>
> Thanks,
> Rudi
>
>
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

[webkit-dev] NPAPI plugin may be broken in mac latest sources?

[Rudi Sherry]

Yesterday I updated to the latest webkit sources for the mac (`Tools\Scripts 
update-webkit`) and rebuilt.

Now, when I run Safari using them (`Tools\Scripts\run-safari --debug`) and open 
a file for my plug-in (which works fine), then go Back/Forward QUICKLY, 
NPP_NewStream() is never called for Forward.  If I go Back/Forward SLOWLY, then 
it gets called for Forward.

In between Back and Forward, the NP_Shutdown() .. NP_Initialize, NPP_New is 
called in both cases, and I've verified they're all returning no error (or 
valid values).

The release of Safari I have (5.1.5) does not exhibit this behavior (I'm on 
Lion).

I don't see anything in the bug database about this; before I enter a bug, is 
this known?

Thanks,
Rudi



___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Process for making changes that affect layout test results

[Ojan Vafai]

Typically, if you're working on Chromium Linux or Win, you'd include the
new expected results for that platform in your initial commit/code-review
as well.

On Wed, Apr 11, 2012 at 2:09 PM, Tony Payne  wrote:

> All code I'm changing is inside of #if PLATFORM(CHROMIUM) blocks.
>
> Thanks for the quick answer.
>
> Tony
>
>
> On Wed, Apr 11, 2012 at 2:03 PM, Ryosuke Niwa  wrote:
>
>> On Wed, Apr 11, 2012 at 1:57 PM, Tony Payne  wrote:
>>
>>> Given the recent discussion on test_expectations.txt, perhaps the answer
>>> to my question is still up in the air.
>>>
>>> I'm working on a change that I expect to require changing the
>>> expectations for about 75 tests on chromium win and linux.
>>> https://trac.webkit.org/wiki/Rebaseline seems to only cover the
>>> gardening work to rebaseline after the commit. I cannot find any wiki pages
>>> that describe what the original author is expected to do when making visual
>>> changes. Should I attempt to rebaseline manually? Should I mark the tests
>>> as failing? Should I just check in and let the bots go red?
>>>
>>
>> Just land the patch and rebaseline the tests. Please also coordinate with
>> Chromium port's WebKit gardener when landing this patch.
>>
>> Also, does this patch only affect Chromium Windows and Linux, and not
>> GTK, Qt, Windows, etc...? If the answer is no, and will affect other
>> non-Chromium ports, then you're also responsible for rebaselining or
>> coordinating with other ports to make sure you don't break tests on their
>> ports as well.
>>
>> - Ryosuke
>>
>>
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Process for making changes that affect layout test results

[Tony Payne]

All code I'm changing is inside of #if PLATFORM(CHROMIUM) blocks.

Thanks for the quick answer.

Tony

On Wed, Apr 11, 2012 at 2:03 PM, Ryosuke Niwa  wrote:

> On Wed, Apr 11, 2012 at 1:57 PM, Tony Payne  wrote:
>
>> Given the recent discussion on test_expectations.txt, perhaps the answer
>> to my question is still up in the air.
>>
>> I'm working on a change that I expect to require changing the
>> expectations for about 75 tests on chromium win and linux.
>> https://trac.webkit.org/wiki/Rebaseline seems to only cover the
>> gardening work to rebaseline after the commit. I cannot find any wiki pages
>> that describe what the original author is expected to do when making visual
>> changes. Should I attempt to rebaseline manually? Should I mark the tests
>> as failing? Should I just check in and let the bots go red?
>>
>
> Just land the patch and rebaseline the tests. Please also coordinate with
> Chromium port's WebKit gardener when landing this patch.
>
> Also, does this patch only affect Chromium Windows and Linux, and not GTK,
> Qt, Windows, etc...? If the answer is no, and will affect other
> non-Chromium ports, then you're also responsible for rebaselining or
> coordinating with other ports to make sure you don't break tests on their
> ports as well.
>
> - Ryosuke
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Process for making changes that affect layout test results

[Ryosuke Niwa]

On Wed, Apr 11, 2012 at 1:57 PM, Tony Payne  wrote:

> Given the recent discussion on test_expectations.txt, perhaps the answer
> to my question is still up in the air.
>
> I'm working on a change that I expect to require changing the expectations
> for about 75 tests on chromium win and linux.
> https://trac.webkit.org/wiki/Rebaseline seems to only cover the gardening
> work to rebaseline after the commit. I cannot find any wiki pages that
> describe what the original author is expected to do when making visual
> changes. Should I attempt to rebaseline manually? Should I mark the tests
> as failing? Should I just check in and let the bots go red?
>

Just land the patch and rebaseline the tests. Please also coordinate with
Chromium port's WebKit gardener when landing this patch.

Also, does this patch only affect Chromium Windows and Linux, and not GTK,
Qt, Windows, etc...? If the answer is no, and will affect other
non-Chromium ports, then you're also responsible for rebaselining or
coordinating with other ports to make sure you don't break tests on their
ports as well.

- Ryosuke
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

[webkit-dev] Process for making changes that affect layout test results

[Tony Payne]

Given the recent discussion on test_expectations.txt, perhaps the answer to
my question is still up in the air.

I'm working on a change that I expect to require changing the expectations
for about 75 tests on chromium win and linux.
https://trac.webkit.org/wiki/Rebaseline seems to only cover the gardening
work to rebaseline after the commit. I cannot find any wiki pages that
describe what the original author is expected to do when making visual
changes. Should I attempt to rebaseline manually? Should I mark the tests
as failing? Should I just check in and let the bots go red?

Thanks,
Tony
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] CSS 2.1 Test Suite

[Robert Hogan]

On Wednesday 11 April 2012 20:27:23 Dirk Pranke wrote:
> 
> What does "clean up the existing folder" entail?
> 

Just move any -expected.html file out of there and generate pixel results. 
Or move the test and its -expected.html into a folder in fast/css.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] CSS 2.1 Test Suite

[Robert Hogan]

On Tuesday 10 April 2012 22:35:13 Ryosuke Niwa wrote:
> As we have previous discussed following
> https://lists.webkit.org/pipermail/webkit-dev/2012-March/019782.html,
> it's hard to judge whether a given reference result is enough to cover
> everything the test intends to test. e.g. you can have a bug such that
> both the test and the reference file ends up having the same rendering
> result.
> 

Definitely - and this will certainly be the case with some of the css tests, 
but a minority. A *lot* of them are along the lines of 'this text/box is 
green'. 

For the tests we write to fix failing CSS tests this will definitely be 
something to watch out for in review.

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] CSS 2.1 Test Suite

[Ryosuke Niwa]

On Wed, Apr 11, 2012 at 12:25 PM, Robert Hogan wrote:

> On Wednesday 11 April 2012 20:11:23 Alan Stearns wrote:
> > The best way to judge whether a reference result is correct is to submit
> > the result to the W3C CSS 2.1 test suite and have it reviewed. The only
> > way this test suite will get more reference results is if people like us
> > volunteer to submit references. If it's useful to us to have these
> > 'homebrew reference results' then it will be useful to everyone else who
> > uses the suite.
>
> Agreed. This will help us land the tests that already pass and won't slow
> down the effort to fix the css tests that we don't. We can agree to only
> import css tests with reftests and get NRWT working on them. I hope to do
> this in:
>
> https://bugs.webkit.org/show_bug.cgi?id=83048
>
> However I do think we need a decision on how we:
>
> 1. Land fixes for currently broken tests that don't have a reftest.
> 2. Clean up the existing css2.1/20110323 folder.
>
> If it's just a case of living with pixel results for now I'm happy with
> that. But I think allowing them to live outside css2.1/20110323 would
> encourage me and others to write reftests while we're fixing the tests'
> results on WebKit.
>

I'd argue for just using pixel results for now, and submit patches (to W3C)
to convert them to reftests.

- Ryosuke
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] CSS 2.1 Test Suite

[Dirk Pranke]

On Wed, Apr 11, 2012 at 12:25 PM, Robert Hogan  wrote:
> On Wednesday 11 April 2012 20:11:23 Alan Stearns wrote:
>> The best way to judge whether a reference result is correct is to submit
>> the result to the W3C CSS 2.1 test suite and have it reviewed. The only
>> way this test suite will get more reference results is if people like us
>> volunteer to submit references. If it's useful to us to have these
>> 'homebrew reference results' then it will be useful to everyone else who
>> uses the suite.
>
> Agreed. This will help us land the tests that already pass and won't slow
> down the effort to fix the css tests that we don't. We can agree to only
> import css tests with reftests and get NRWT working on them. I hope to do
> this in:
>
> https://bugs.webkit.org/show_bug.cgi?id=83048
>
> However I do think we need a decision on how we:
>
> 1. Land fixes for currently broken tests that don't have a reftest.
> 2. Clean up the existing css2.1/20110323 folder.
>
> If it's just a case of living with pixel results for now I'm happy with
> that. But I think allowing them to live outside css2.1/20110323 would
> encourage me and others to write reftests while we're fixing the tests'
> results on WebKit.
>
> From listening to Maciej, Ojan and Ryosuke it is not an option to keep
> homebrew reftests in css2.1/20110323 for two good reasons: it breaks
> important assumptions in the way the reftest harness works, and it is
> better to keep imported test suites clean and unmodified.
>
> So it's either 1 or 2 above I think. I would prefer 2 as it won't bloat git
> checkouts so much and will make fixes easier to land.

What does "clean up the existing folder" entail?

-- Dirk
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] CSS 2.1 Test Suite

[Robert Hogan]

On Wednesday 11 April 2012 20:11:23 Alan Stearns wrote:
> The best way to judge whether a reference result is correct is to submit
> the result to the W3C CSS 2.1 test suite and have it reviewed. The only
> way this test suite will get more reference results is if people like us
> volunteer to submit references. If it's useful to us to have these
> 'homebrew reference results' then it will be useful to everyone else who
> uses the suite.

Agreed. This will help us land the tests that already pass and won't slow 
down the effort to fix the css tests that we don't. We can agree to only 
import css tests with reftests and get NRWT working on them. I hope to do 
this in:

https://bugs.webkit.org/show_bug.cgi?id=83048

However I do think we need a decision on how we:

1. Land fixes for currently broken tests that don't have a reftest.
2. Clean up the existing css2.1/20110323 folder.

If it's just a case of living with pixel results for now I'm happy with 
that. But I think allowing them to live outside css2.1/20110323 would 
encourage me and others to write reftests while we're fixing the tests' 
results on WebKit.

>From listening to Maciej, Ojan and Ryosuke it is not an option to keep 
homebrew reftests in css2.1/20110323 for two good reasons: it breaks 
important assumptions in the way the reftest harness works, and it is 
better to keep imported test suites clean and unmodified.

So it's either 1 or 2 above I think. I would prefer 2 as it won't bloat git 
checkouts so much and will make fixes easier to land.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] CSS 2.1 Test Suite

[Ryosuke Niwa]

On Wed, Apr 11, 2012 at 12:11 PM, Alan Stearns  wrote:

> The best way to judge whether a reference result is correct is to submit
> the result to the W3C CSS 2.1 test suite and have it reviewed. The only
> way this test suite will get more reference results is if people like us
> volunteer to submit references.


Right. We might want to add some wiki page on how to do this (step by step
instruction) since not everyone contributing to the WebKit is familiar with
their procedure.

The previous thread mentioned checking Mozilla to see if their test suite
> had references for particular tests. If that's the case, then we should
> either encourage them to submit the references to the W3C, or just do that
> ourselves on their behalf.
>

That'll be nice.

- Ryosuke
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] CSS 2.1 Test Suite

[Alan Stearns]

The best way to judge whether a reference result is correct is to submit
the result to the W3C CSS 2.1 test suite and have it reviewed. The only
way this test suite will get more reference results is if people like us
volunteer to submit references. If it's useful to us to have these
'homebrew reference results' then it will be useful to everyone else who
uses the suite.

The previous thread mentioned checking Mozilla to see if their test suite
had references for particular tests. If that's the case, then we should
either encourage them to submit the references to the W3C, or just do that
ourselves on their behalf.

Alan

From:  Ryosuke Niwa 

As we have previous discussed following
https://lists.webkit.org/pipermail/webkit-dev/2012-March/019782.html, it's
hard to judge whether a given reference result is enough to cover
everything the test intends to test. e.g. you can have a bug such that
both the test and the reference file ends up having the same rendering
result.
- Ryosuke

On Tue, Apr 10, 2012 at 2:26 PM, Robert Hogan 
wrote:


Hi there,

We currently add tests from the CSS 2.1 suite as we fix them. They get
added
to the css2.1/20110323 folder in LayoutTests. Most of them don't have
native reference tests yet in the suite so we (mostly I) have been adding
homebrew reference results to the folder to avoid generating pixel results
on all platforms. (see http://webkitmemes.tumblr.com/post/20788159625 !)

These reference-results are easily removed once superseded but it might be
cleaner just to move them, and the associated css tests, to a folder of
their own in fast/css. That will allow css2.1/20110323 to be a clean import
that the 8500 or so passing tests can be added to in 20 or 30 batches.[1]
It will also make NRWT's reftests harness work with the suite.

Does anyone object to that approach? The only thing going against it seems
to be the principle that imported tests should be stored separately and
together but this is more a case of using them to fix bugs and prevent
future regressions while allowing a clean import of the CSS 2.1 test suite
to take place in parallel.

The problem this does not solve is how we avoid creating pixel results for
tests that already pass but which do not have reftests of their own. Again
I would be in favour of putting these in fast/css and keeping them there
until reftests are added to the suite. This would allow us to prevent them
regressing and come up with a reftest for them that can be submitted to the
CSS test suite guys.

The end result would be that we only directly import to the css2.1 folder
those tests in the CSS test suite that have ref tests native to the suite.

Thanks,
Robert


[1] I keep a local and relatively up to date copy of the passing and
failing tests in separate folders in my checkout.  Yes I know I should
create bugs for them and get started on landing the passes.

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev





___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] trac.webkit.org seems down?

[William Siegrist]

The trac/svn server was overloaded for about an hour and recovered on its own 
around 1am PDT. 

-Bill


On Apr 11, 2012, at 12:50 AM, Ryosuke Niwa wrote:

> Resent from the right address.
> 
> On Wed, Apr 11, 2012 at 12:47 AM, Ryosuke Niwa <*> wrote:
> Observing the same. Seems like it's time to go sleep...
> 
> 
> On Wed, Apr 11, 2012 at 12:35 AM, Eric Seidel  wrote:
> http://trac.webkit.org/ is failing to load for me.
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
> 
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] trac.webkit.org seems down?

[Ryosuke Niwa]

Resent from the right address.

On Wed, Apr 11, 2012 at 12:47 AM, Ryosuke Niwa <*> wrote:

> Observing the same. Seems like it's time to go sleep...
>
>
> On Wed, Apr 11, 2012 at 12:35 AM, Eric Seidel  wrote:
>
>> http://trac.webkit.org/ is failing to load for me.
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] trac.webkit.org seems down?

[Osztrogonac Csaba]

Take it easy, Eric!
No trac, no SVN server, no commit, no regression, no problem. ;-)

Eric Seidel írta:

http://trac.webkit.org/ is failing to load for me.


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] webkitPostMessage

[Adam Barth]

[+dslomov]

I found a thread on public-webapps that seems related to this
question.  Dmitry, do you know what current status is here?  I'd like
to make sure we're on a path towards interoperability with other
browsers.

Thanks,
Adam


On Tue, Apr 10, 2012 at 9:08 PM, Adam Barth  wrote:
> I'm trying to understand why we have both DOMWindow.webkitPostMessage
> and DOMWindow.postMessage.  I'm also trying to understand the
> following comment in {JS,V8}DOMWindowCustom.cpp:
>
>    // This function has variable arguments and can be:
>    // Per current spec:
>    //   postMessage(message, targetOrigin)
>    //   postMessage(message, targetOrigin, {sequence of transferrables})
>    // Legacy non-standard implementations in webkit allowed:
>    //   postMessage(message, {sequence of transferrables}, targetOrigin);
>
> Specifically:
>
> 1) Can we remove webkitPostMessage?  If we can't remove it now, is
> there a time in the future at which we can remove it?
>
> 2) Can we adopt the behavior in the specification (and drop the
> non-standard behavior)?  If not, should we change the specification to
> match our behavior?
>
> Many thanks,
> Adam
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

[webkit-dev] trac.webkit.org seems down?

[Eric Seidel]

http://trac.webkit.org/ is failing to load for me.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev