Re: [webkit-dev] Where do we put WPT tests to be exported

2017-05-16 Thread Ryosuke Niwa
On Tue, May 16, 2017 at 8:42 AM, Konstantin Tokarev  wrote:
>
>
> 16.05.2017, 17:54, "Michael Catanzaro" :
>> On Mon, May 15, 2017 at 11:01 PM, Ryosuke Niwa  wrote:
>>>  Hi all,
>>>
>>>  Youenn is working on a patch to automatically create a GitHub PR to
>>>  export tests from a WebKit patch which includes changes to
>>>  web-platform-tests [1].
>>>
>>>  That raises a question as to where we should put new tests or modified
>>>  tests intended to be exported to web-platform-tests from WebKit.
>>
>> Whatever you wind up doing, it would be helpful to add a README
>> somewhere that explains the workflow, so the process doesn't turn into
>> folklore only known to the developers who work on it regularly and
>> those who read this mailing list thread.
>
> I think it would also be useful to add wiki page to "Layout Tests" section

With all likelihood, we'd just update webkit.org or the wiki. We don't
typically write a README for things like this.

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Where do we put WPT tests to be exported

2017-05-16 Thread Konstantin Tokarev


16.05.2017, 17:54, "Michael Catanzaro" :
> On Mon, May 15, 2017 at 11:01 PM, Ryosuke Niwa  wrote:
>>  Hi all,
>>
>>  Youenn is working on a patch to automatically create a GitHub PR to
>>  export tests from a WebKit patch which includes changes to
>>  web-platform-tests [1].
>>
>>  That raises a question as to where we should put new tests or modified
>>  tests intended to be exported to web-platform-tests from WebKit.
>
> Whatever you wind up doing, it would be helpful to add a README
> somewhere that explains the workflow, so the process doesn't turn into
> folklore only known to the developers who work on it regularly and
> those who read this mailing list thread.

I think it would also be useful to add wiki page to "Layout Tests" section

>
> Thanks,
>
> Michael
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

-- 
Regards,
Konstantin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Where do we put WPT tests to be exported

2017-05-16 Thread youenn fablet
That is a fair point!
For importing tests, effort was done to ease the use of the
import-w3c-tests script.
Documentation should be in "Tools/Scripts/import-w3c-tests --help"
I will be happy to beef it up based on suggestions.
y

Le mar. 16 mai 2017 à 07:55, Michael Catanzaro  a
écrit :

> On Mon, May 15, 2017 at 11:01 PM, Ryosuke Niwa  wrote:
> > Hi all,
> >
> > Youenn is working on a patch to automatically create a GitHub PR to
> > export tests from a WebKit patch which includes changes to
> > web-platform-tests [1].
> >
> > That raises a question as to where we should put new tests or modified
> > tests intended to be exported to web-platform-tests from WebKit.
>
> Whatever you wind up doing, it would be helpful to add a README
> somewhere that explains the workflow, so the process doesn't turn into
> folklore only known to the developers who work on it regularly and
> those who read this mailing list thread.
>
> Thanks,
>
> Michael
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Where do we put WPT tests to be exported

2017-05-16 Thread youenn fablet
Le lun. 15 mai 2017 à 23:15, Maciej Stachowiak  a écrit :

>
> On May 15, 2017, at 9:08 PM, youenn fablet  wrote:
>
> I see two main cases:
> - Writer of the patch is making sure to upstream WPT test changes at
> WebKit landing time. It is ok to make the changes directly in
> LayoutTests/imported/w3c/web-platform-tests/
> - Writer plans to upstream WPT test changes at some point but wants more
> time. It is better to develop the tests in LayoutTests/http/wpt and then
> migrate them later on.
>
>
> My proposal was different from either of these, it was to have a directory
> specifically for tests meant to be upstreamed (LayoutTests/http/wpt should
> contain only tests not meant
>

The possibility to have LayoutTests/http/wpt/to-be-imported was mentionned
previously.
I think it makes sense to run to-be-imported tests behind WPT as this is
the way they will end up being run when upstreamed.


>
> I think adding new tests directly to
> LayoutTests/imported/w3c/web-platform-tests/ is needlessly messy. Most
> stuff in the imported/ directory is an exact copy of an upstream test
> suite, so if you run only a specific directory of tests, you know you are
> running an official conformance suite. With this proposal, it might also
> contain random tests that will hopefully be upstreamed but maybe not, or
> might be changed before the PR lands upstream, or might get renamed, or
> whatever. There's no guarantee that updating from the official version will
> ever fully resolve the delta.
>

LayoutTests/imported/w3c/web-platform-tests is mostly about regression
testing.
For conformance testing, it is probably more accurate to either use
w3c-test.org (probably not reliable enough though) or make a clone of W3C
WPT, do the set up as specified in their README (etc/hosts...) and use
wptrunner.


>
> I think it would be more elegant to have a parallel directory
> (LayoutTests/for-export/w3c/web-platform-tests). Then when something is
> actually upstreamed and then pulled back down, we could delete the staged
> version. Directly modifying our local copy seems like it could easily lead
> to long-term divergences slipping through the cracks.
>

A parallel directory is fine when starting a test suite.
Often, changes are limited to modifying a test, which might be best done
in-place.
Often, adding a test is easier by adding it in an already existing file.

Tooling should protect us from diverging.
Authoring in LayoutTests/for-export/w3c/web-platform-tests might not allow
you to use WPT tools and common resources.
At upstream time, reviewers will probably tell to use that and that
existing resource.

There is agreement that a imported/w3c/web-platform-test layout test change
that lands in WebKit can be merged upstreamed.
I am not sure what would be the process with
LayoutTests/for-export/w3c/web-platform-tests tests.


>
> Of course, people could always go run the official copy from w3c-test.org .
> But we usually leave imported conformance test suites unmodified except the
> minimum necessary to make them run.
>
> I would start with an experimental phase with some of us making direct
> changes in LayoutTests/imported/w3c/web-platform-tests/.
> When we are happy with the tools and think the risk for issues is low
> enough (or when the bots can handle most of it for us), hacking
> LayoutTests/imported/w3c/web-platform-tests/ could be the default.
>
>
> I think the problem with this plan is not tools risk, it's the fact that a
> directory of imported tests can no longer be trusted to actually just be
> imported tests. So a smaller number of people doing it to start would not
> address my concern. It might be that other people don't care about this. My
> opinion counts no more than anyone else's, and I'd be interested in hearing
> from others.
>

I am not sure to understand precisely what the consequences of "imported
tests can no longer be trusted to actually just be imported tests" are.
Ideally, a WPT test change landing in WebKit should be merged at the same
time on W3C WPT.

There was a suggestion that LayoutTests/imported/w3c/web-platform-tests be
moved to a shorter path like LayoutTests/web-platform-tests. That would
also make it clear that this folder is not only about one-way-sync.


>
> Regards,
> Maciej
>
>
> Le lun. 15 mai 2017 à 21:02, Ryosuke Niwa  a écrit :
>
>> Hi all,
>>
>> Youenn is working on a patch to automatically create a GitHub PR to
>> export tests from a WebKit patch which includes changes to
>> web-platform-tests [1].
>>
>> That raises a question as to where we should put new tests or modified
>> tests intended to be exported to web-platform-tests from WebKit.
>>
>> I think the most obvious option is to use
>> LayoutTests/imported/w3c/web-platform-tests/.  However, in the other
>> thread about adopting testharness.js (titled Another WPT bite), Maciej
>> briefly expressed the preference for creating a new directory:
>> 

Re: [webkit-dev] Another WPT bite

2017-05-16 Thread Rick Byers
On Tue, May 16, 2017 at 4:54 AM, Maciej Stachowiak  wrote:

>
>
> > On May 16, 2017, at 1:16 AM, Ryosuke Niwa  wrote:
> >
> > On Mon, May 15, 2017 at 11:57 PM, Anne van Kesteren 
> wrote:
> >> On Tue, May 16, 2017 at 7:29 AM, Ryosuke Niwa  wrote:
> >>> Given we're talking about how these tests are ran inside WebKit,
> >>> whether there is an agreement about this or not is sort of irrelevant.
> >>> If a test doesn't run as expected, we can run it inside a HTTP server.
> >>
> >> I was just trying to help clarify why what makes sense for WebKit,
> >> might not make sense for tests designed to run on all engines. If
> >> that's not desired here, I'll stop.
> >
> > Sure, I don't think we're interested in changing the way tests are ran
> > elsewhere.
> >
> > It would be great if upstream web-platform-tests could use relative
> > paths whenever possible or allowed annotation (e.g. we'd add such
> > annotation) as to which tests could be ran locally via file URL.
> >
> > However, that's more or less a secondary concern (nice-to-have)
> > whereas how web-platform-tests are imported and ran in WebKit are a
> > lot more important to us due to the impact it has on our development
> > process, tooling, as well as the maintenance cost.
>
> I agree with Ryosuke. We don't want to impose on others, but these two
> changes would be convenient for WebKit's use. Perhaps somewhere in WPT
> space (a GitHub issue?) would be the appropriate venue to discuss it. I am
> assuming here the tradeoff is just the maintenance burden of keeping the
> relative paths up to date, but maybe there is some deeper reason not to do
> it.
>

Filed https://github.com/w3c/web-platform-tests/issues/5945

Even though (after much debate) chromium has switched to using wptserve all
the time, there are still some situations where I'd find it handy to be
able to load a test (which I know doesn't depend on wptserve) from some
other server or filesystem.  So I'd support changing this upstream.


>  - Maciej
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Where do we put WPT tests to be exported

2017-05-16 Thread Michael Catanzaro

On Mon, May 15, 2017 at 11:01 PM, Ryosuke Niwa  wrote:

Hi all,

Youenn is working on a patch to automatically create a GitHub PR to
export tests from a WebKit patch which includes changes to
web-platform-tests [1].

That raises a question as to where we should put new tests or modified
tests intended to be exported to web-platform-tests from WebKit.


Whatever you wind up doing, it would be helpful to add a README 
somewhere that explains the workflow, so the process doesn't turn into 
folklore only known to the developers who work on it regularly and 
those who read this mailing list thread.


Thanks,

Michael

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Another WPT bite

2017-05-16 Thread Ben Kelly
On Sat, May 13, 2017 at 8:55 PM, Michael[tm] Smith  wrote:

> Anne van Kesteren , 2017-05-13 06:20 +0200:
> >
> > On Fri, May 12, 2017 at 11:49 PM, Maciej Stachowiak 
> wrote:
> > > It seems like there's two unusual things about WPT:
> > > - At least according to Alexey, WPT tests are somewhat prone to
> flakiness in Safari.
> >
> > Although they haven't always been working perfectly, changes to
> > web-platform-tests run through some kind of stability check in both
> > Chrome and Firefox (run the new tests 10 times and fail if the results
> > are inconsistent). Ideally Safari is added to that mix, though I think
> > the last time folks looked into that it wasn't possible for some
> > reason. That might be something to put some resources on.
>
> I think we already have that now un-blocked and we will by the end of this
> week
> (May 19) start having Travis running the stability checker for Safari too
> (and
> Edge too) and providing the results as comments to PRs (just as we now do
> for Chrome and Firefox).
>

FWIW safari results are now showing up:

https://github.com/w3c/web-platform-tests/pull/5921#issuecomment-301517865
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Another WPT bite

2017-05-16 Thread Maciej Stachowiak


> On May 16, 2017, at 1:16 AM, Ryosuke Niwa  wrote:
> 
> On Mon, May 15, 2017 at 11:57 PM, Anne van Kesteren  wrote:
>> On Tue, May 16, 2017 at 7:29 AM, Ryosuke Niwa  wrote:
>>> Given we're talking about how these tests are ran inside WebKit,
>>> whether there is an agreement about this or not is sort of irrelevant.
>>> If a test doesn't run as expected, we can run it inside a HTTP server.
>> 
>> I was just trying to help clarify why what makes sense for WebKit,
>> might not make sense for tests designed to run on all engines. If
>> that's not desired here, I'll stop.
> 
> Sure, I don't think we're interested in changing the way tests are ran
> elsewhere.
> 
> It would be great if upstream web-platform-tests could use relative
> paths whenever possible or allowed annotation (e.g. we'd add such
> annotation) as to which tests could be ran locally via file URL.
> 
> However, that's more or less a secondary concern (nice-to-have)
> whereas how web-platform-tests are imported and ran in WebKit are a
> lot more important to us due to the impact it has on our development
> process, tooling, as well as the maintenance cost.

I agree with Ryosuke. We don't want to impose on others, but these two changes 
would be convenient for WebKit's use. Perhaps somewhere in WPT space (a GitHub 
issue?) would be the appropriate venue to discuss it. I am assuming here the 
tradeoff is just the maintenance burden of keeping the relative paths up to 
date, but maybe there is some deeper reason not to do it.

 - Maciej
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Another WPT bite

2017-05-16 Thread Ryosuke Niwa
On Mon, May 15, 2017 at 11:57 PM, Anne van Kesteren  wrote:
> On Tue, May 16, 2017 at 7:29 AM, Ryosuke Niwa  wrote:
>> Given we're talking about how these tests are ran inside WebKit,
>> whether there is an agreement about this or not is sort of irrelevant.
>> If a test doesn't run as expected, we can run it inside a HTTP server.
>
> I was just trying to help clarify why what makes sense for WebKit,
> might not make sense for tests designed to run on all engines. If
> that's not desired here, I'll stop.

Sure, I don't think we're interested in changing the way tests are ran
elsewhere.

It would be great if upstream web-platform-tests could use relative
paths whenever possible or allowed annotation (e.g. we'd add such
annotation) as to which tests could be ran locally via file URL.

However, that's more or less a secondary concern (nice-to-have)
whereas how web-platform-tests are imported and ran in WebKit are a
lot more important to us due to the impact it has on our development
process, tooling, as well as the maintenance cost.

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Another WPT bite

2017-05-16 Thread Anne van Kesteren
On Tue, May 16, 2017 at 7:29 AM, Ryosuke Niwa  wrote:
> Given we're talking about how these tests are ran inside WebKit,
> whether there is an agreement about this or not is sort of irrelevant.
> If a test doesn't run as expected, we can run it inside a HTTP server.

I was just trying to help clarify why what makes sense for WebKit,
might not make sense for tests designed to run on all engines. If
that's not desired here, I'll stop.


-- 
https://annevankesteren.nl/
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Where do we put WPT tests to be exported

2017-05-16 Thread Maciej Stachowiak


> On May 15, 2017, at 9:08 PM, youenn fablet  wrote:
> 
> I see two main cases:
> - Writer of the patch is making sure to upstream WPT test changes at WebKit 
> landing time. It is ok to make the changes directly in 
> LayoutTests/imported/w3c/web-platform-tests/
> - Writer plans to upstream WPT test changes at some point but wants more 
> time. It is better to develop the tests in LayoutTests/http/wpt and then 
> migrate them later on.

My proposal was different from either of these, it was to have a directory 
specifically for tests meant to be upstreamed (LayoutTests/http/wpt should 
contain only tests not meant

I think adding new tests directly to 
LayoutTests/imported/w3c/web-platform-tests/ is needlessly messy. Most stuff in 
the imported/ directory is an exact copy of an upstream test suite, so if you 
run only a specific directory of tests, you know you are running an official 
conformance suite. With this proposal, it might also contain random tests that 
will hopefully be upstreamed but maybe not, or might be changed before the PR 
lands upstream, or might get renamed, or whatever. There's no guarantee that 
updating from the official version will ever fully resolve the delta. 

I think it would be more elegant to have a parallel directory 
(LayoutTests/for-export/w3c/web-platform-tests). Then when something is 
actually upstreamed and then pulled back down, we could delete the staged 
version. Directly modifying our local copy seems like it could easily lead to 
long-term divergences slipping through the cracks.

Of course, people could always go run the official copy from w3c-test.org 
 . But we usually leave imported conformance test suites 
unmodified except the minimum necessary to make them run.

> I would start with an experimental phase with some of us making direct 
> changes in LayoutTests/imported/w3c/web-platform-tests/.
> When we are happy with the tools and think the risk for issues is low enough 
> (or when the bots can handle most of it for us), hacking 
> LayoutTests/imported/w3c/web-platform-tests/ could be the default.

I think the problem with this plan is not tools risk, it's the fact that a 
directory of imported tests can no longer be trusted to actually just be 
imported tests. So a smaller number of people doing it to start would not 
address my concern. It might be that other people don't care about this. My 
opinion counts no more than anyone else's, and I'd be interested in hearing 
from others.

Regards,
Maciej

> 
> Le lun. 15 mai 2017 à 21:02, Ryosuke Niwa  > a écrit :
> Hi all,
> 
> Youenn is working on a patch to automatically create a GitHub PR to
> export tests from a WebKit patch which includes changes to
> web-platform-tests [1].
> 
> That raises a question as to where we should put new tests or modified
> tests intended to be exported to web-platform-tests from WebKit.
> 
> I think the most obvious option is to use
> LayoutTests/imported/w3c/web-platform-tests/.  However, in the other
> thread about adopting testharness.js (titled Another WPT bite), Maciej
> briefly expressed the preference for creating a new directory:
> https://lists.webkit.org/pipermail/webkit-dev/2017-May/029022.html 
> 
> 
> Do other people have strong opinions about this?
> 
> - R. Niwa
> 
> [1] https://bugs.webkit.org/show_bug.cgi?id=169462 
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org 
> https://lists.webkit.org/mailman/listinfo/webkit-dev 
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev