Re: [webkit-dev] Another WPT bite

2017-05-16 Thread Rick Byers
On Tue, May 16, 2017 at 4:54 AM, Maciej Stachowiak wrote: > > > > On May 16, 2017, at 1:16 AM, Ryosuke Niwa wrote: > > > > On Mon, May 15, 2017 at 11:57 PM, Anne van Kesteren > wrote: > >> On Tue, May 16, 2017 at 7:29 AM, Ryosuke Niwa

Re: [webkit-dev] Another WPT bite

2017-05-16 Thread Ben Kelly
On Sat, May 13, 2017 at 8:55 PM, Michael[tm] Smith wrote: > Anne van Kesteren , 2017-05-13 06:20 +0200: > > > > On Fri, May 12, 2017 at 11:49 PM, Maciej Stachowiak > wrote: > > > It seems like there's two unusual things about WPT: > > > - At least

Re: [webkit-dev] Another WPT bite

2017-05-16 Thread Maciej Stachowiak
> On May 16, 2017, at 1:16 AM, Ryosuke Niwa wrote: > > On Mon, May 15, 2017 at 11:57 PM, Anne van Kesteren wrote: >> On Tue, May 16, 2017 at 7:29 AM, Ryosuke Niwa wrote: >>> Given we're talking about how these tests are ran inside WebKit,

Re: [webkit-dev] Another WPT bite

2017-05-16 Thread Ryosuke Niwa
On Mon, May 15, 2017 at 11:57 PM, Anne van Kesteren wrote: > On Tue, May 16, 2017 at 7:29 AM, Ryosuke Niwa wrote: >> Given we're talking about how these tests are ran inside WebKit, >> whether there is an agreement about this or not is sort of irrelevant. >>

Re: [webkit-dev] Another WPT bite

2017-05-16 Thread Anne van Kesteren
On Tue, May 16, 2017 at 7:29 AM, Ryosuke Niwa wrote: > Given we're talking about how these tests are ran inside WebKit, > whether there is an agreement about this or not is sort of irrelevant. > If a test doesn't run as expected, we can run it inside a HTTP server. I was just

Re: [webkit-dev] Another WPT bite

2017-05-15 Thread Ryosuke Niwa
On Mon, May 15, 2017 at 10:07 PM, Anne van Kesteren wrote: > On Tue, May 16, 2017 at 5:45 AM, Ryosuke Niwa wrote: >>> I think the main problem with not running a server is that behavior >>> for file URLs is not defined. And browsers tend to impose different

Re: [webkit-dev] Another WPT bite

2017-05-15 Thread youenn fablet
It makes sense to run WPT tests as HTTP URLs for conformance/regression purposes. It is fine to run WPT tests as file based URLs for development purposes. Tooling should make it possible to run WPT tests as HTTP URLs for development purposes with minimum to no cost. We are not there yet. Le lun.

Re: [webkit-dev] Another WPT bite

2017-05-15 Thread Anne van Kesteren
On Tue, May 16, 2017 at 5:45 AM, Ryosuke Niwa wrote: >> I think the main problem with not running a server is that behavior >> for file URLs is not defined. And browsers tend to impose different >> restrictions there. So you might end up debugging something only to >> later

Re: [webkit-dev] Another WPT bite

2017-05-15 Thread Ryosuke Niwa
On Sun, May 14, 2017 at 12:35 AM, Anne van Kesteren wrote: > On Sun, May 14, 2017 at 8:24 AM, Maciej Stachowiak wrote: >> For the engineer use case, we can make a command-line tool to launch the >> server and load the test. But it's kind of sad that in ~95% of

Re: [webkit-dev] Another WPT bite

2017-05-14 Thread Anne van Kesteren
On Sun, May 14, 2017 at 8:24 AM, Maciej Stachowiak wrote: > For the engineer use case, we can make a command-line tool to launch the > server and load the test. But it's kind of sad that in ~95% of cases, the > only value provided by the server is resolving the path to

Re: [webkit-dev] Another WPT bite

2017-05-14 Thread Maciej Stachowiak
> On May 13, 2017, at 5:26 PM, Michael[tm] Smith wrote: > > Maciej Stachowiak , 2017-05-13 14:58 -0700: >> ... From what I gather, there are a lot of tests where only the paths to >> the test harness end up requiring the server. > > Yeah that’s the case for the

Re: [webkit-dev] Another WPT bite

2017-05-13 Thread Michael[tm] Smith
Anne van Kesteren , 2017-05-13 06:20 +0200: > > On Fri, May 12, 2017 at 11:49 PM, Maciej Stachowiak wrote: > > It seems like there's two unusual things about WPT: > > - At least according to Alexey, WPT tests are somewhat prone to flakiness > > in Safari. > >

Re: [webkit-dev] Another WPT bite

2017-05-13 Thread Michael[tm] Smith
Brady Eidson , 2017-05-13 17:11 -0700: > > > > On May 12, 2017, at 7:43 PM, Ryosuke Niwa wrote: > > > > On Fri, May 12, 2017 at 7:31 PM, Alexey Proskuryakov > > wrote: > >> > >> When there is a test failure that I need to communicate to

Re: [webkit-dev] Another WPT bite

2017-05-13 Thread Michael[tm] Smith
Maciej Stachowiak , 2017-05-13 14:58 -0700: > ... From what I gather, there are a lot of tests where only the paths to > the test harness end up requiring the server. Yeah that’s the case for the vast majority of tests. Relatively few — less than 5% altogether, I’d estimate —

Re: [webkit-dev] Another WPT bite

2017-05-13 Thread Brady Eidson
> On May 12, 2017, at 7:43 PM, Ryosuke Niwa wrote: > > On Fri, May 12, 2017 at 7:31 PM, Alexey Proskuryakov wrote: >> >> When there is a test failure that I need to communicate to others, I say >> something "please open >>

Re: [webkit-dev] Another WPT bite

2017-05-13 Thread Maciej Stachowiak
> On May 12, 2017, at 7:49 PM, a...@webkit.org wrote: > > >> 12 мая 2017 г., в 19:38, Brian Burg > > написал(а): >> >>> I think that I explained it very clearly, but let me try again. >>> >>> When there is a test failure that I need to communicate

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread youenn fablet
I filed https://bugs.webkit.org/show_bug.cgi?id=172068 to track the need for some extra tooling for HTTP/WPT served tests. We already gathered information about related requirements & workflows here. Let's add more there! Le ven. 12 mai 2017 à 19:50, a écrit : > > 12 мая 2017

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread Anne van Kesteren
On Fri, May 12, 2017 at 11:49 PM, Maciej Stachowiak wrote: > It seems like there's two unusual things about WPT: > - At least according to Alexey, WPT tests are somewhat prone to flakiness in > Safari. Although they haven't always been working perfectly, changes to

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread Brian Burg
> On May 12, 2017, at 7:31 PM, Alexey Proskuryakov wrote: > >> >> 12 мая 2017 г., в 16:12, Sam Weinig > > написал(а): >> >> >> >>> On May 12, 2017, at 2:49 PM, Alexey Proskuryakov >> >

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread ap
> 12 мая 2017 г., в 19:38, Brian Burg написал(а): > >> I think that I explained it very clearly, but let me try again. >> >> When there is a test failure that I need to communicate to others, I say >> something "please open >>

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread Ryosuke Niwa
On Fri, May 12, 2017 at 7:38 PM, Brian Burg wrote: > > If the person works on WebKit, then it seems unreasonable that they would do > work without a checkout. This is not entirely true. If I have an iOS device, it's a lot easier to load up a web page on a public server

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread Ryosuke Niwa
On Fri, May 12, 2017 at 7:31 PM, Alexey Proskuryakov wrote: > > 12 мая 2017 г., в 16:12, Sam Weinig написал(а): > > > > On May 12, 2017, at 2:49 PM, Alexey Proskuryakov wrote: > > > 12 мая 2017 г., в 14:38, Sam Weinig

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread Alexey Proskuryakov
> 12 мая 2017 г., в 16:12, Sam Weinig написал(а): > > > >> On May 12, 2017, at 2:49 PM, Alexey Proskuryakov > > wrote: >> >> >>> 12 мая 2017 г., в 14:38, Sam Weinig >> > написал(а): >>>

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread youenn fablet
> On the topic of LayoutTest/imported tests, can someone describe the > current process of working with LayoutTest/imported? > > How do we handle a broken test in our tree? > > • Do we modify our expectations? > > - If so, how do we remember to change the expectations in a later import? > > If a

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread Joseph Pecoraro
> On May 12, 2017, at 2:39 PM, Ryosuke Niwa wrote: > > On Fri, May 12, 2017 at 12:04 PM, Alexey Proskuryakov > wrote: > >> We don't have a concept of "first class", but I hope that when choosing >> between looking into a simple test

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread Sam Weinig
> On May 12, 2017, at 2:49 PM, Alexey Proskuryakov wrote: > > >> 12 мая 2017 г., в 14:38, Sam Weinig > > написал(а): >> >> I regret piling on here, as I think this thread has diverged from it’s >> original purpose, but…I

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread youenn fablet
I would guess Chromium and Mozilla to have the same issues there. Incidentally, some work is being done right now to ease the run-a-server-then-launch-a-browser thing. We should be able to piggy-back on that effort and also handle the same thing for LayoutTetsts/http/tests.

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread Alexey Proskuryakov
> 12 мая 2017 г., в 14:39, Ryosuke Niwa написал(а): > > This is absolutely not how I operate at all. Since almost all custom > elements and shadow DOM API tests I wrote are written using > testharness.js and upstreamed to web-platform-tests, they have been > deleted from

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread Maciej Stachowiak
> On May 12, 2017, at 2:39 PM, Ryosuke Niwa wrote: > > On Fri, May 12, 2017 at 12:04 PM, Alexey Proskuryakov wrote: >> >> 12 мая 2017 г., в 11:52, Ben Kelly написал(а): >> >> On Fri, May 12, 2017 at 2:26 PM, Rick Byers

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread Alexey Proskuryakov
> 12 мая 2017 г., в 14:38, Sam Weinig написал(а): > > I regret piling on here, as I think this thread has diverged from it’s > original purpose, but…I understand this frustration. That said, perhaps this > is something we can solve with some tooling. For instance, a >

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread Brian Burg
> On May 12, 2017, at 2:10 PM, Simon Fraser wrote: > >> >> On May 12, 2017, at 12:04 PM, Alexey Proskuryakov > > wrote: >> >> >>> 12 мая 2017 г., в 11:52, Ben Kelly >> >

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread Sam Weinig
> On May 12, 2017, at 2:10 PM, Simon Fraser wrote: > > >> On May 12, 2017, at 12:04 PM, Alexey Proskuryakov > > wrote: >> >> >>> 12 мая 2017 г., в 11:52, Ben Kelly >> >

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread Ryosuke Niwa
On Fri, May 12, 2017 at 2:10 PM, Simon Fraser wrote: > > On May 12, 2017, at 12:04 PM, Alexey Proskuryakov wrote: > > > 12 мая 2017 г., в 11:52, Ben Kelly написал(а): > > On Fri, May 12, 2017 at 2:26 PM, Rick Byers

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread Chris Dumez
Our test importer script is perfectly able to rewrite those paths to use relative paths. However, Youenn, who imports and re-syncs most tests does not like this option I believe. I think, part of the issue is that *some* tests do not do the right thing when loading over file:// (e.g. security

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread Christian Biesinger
For what it's worth, my personal solution to this was to put a symlink in / for the resources directory: lrwxrwxrwx 1 root root 55 Dec 11 2015 /resources -> /home/cbiesinger/csswg-test/resources/ Hm, I guess I should update that to web-platform-tests now! Christian On Fri, May 12, 2017 at

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread Simon Fraser
> On May 12, 2017, at 12:04 PM, Alexey Proskuryakov wrote: > > >> 12 мая 2017 г., в 11:52, Ben Kelly > > написал(а): >> >> On Fri, May 12, 2017 at 2:26 PM, Rick Byers > >

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread youenn fablet
Filed https://github.com/w3c/web-platform-tests/issues/5909 and https://github.com/w3c/web-platform-tests/issues/5910. Le ven. 12 mai 2017 à 12:08, Rick Byers a écrit : > On Fri, May 12, 2017 at 2:43 PM, youenn fablet wrote: > >> >> Le ven. 12 mai 2017

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread Rick Byers
On Fri, May 12, 2017 at 2:43 PM, youenn fablet wrote: > > Le ven. 12 mai 2017 à 11:07, Alexey Proskuryakov a écrit : > >> >> 9 мая 2017 г., в 11:27, Simon Fraser написал(а): >> >> >> Another consideration here is "would my test be

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread Alexey Proskuryakov
> 12 мая 2017 г., в 11:52, Ben Kelly написал(а): > > On Fri, May 12, 2017 at 2:26 PM, Rick Byers > wrote: > On Fri, May 12, 2017 at 2:06 PM, Alexey Proskuryakov > wrote: > Since

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread Ben Kelly
On Fri, May 12, 2017 at 2:26 PM, Rick Byers wrote: > On Fri, May 12, 2017 at 2:06 PM, Alexey Proskuryakov > wrote: > >> Since imported WPT tests are very flaky, and are not necessarily written >> to defend against important regressions, investigating issues

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread youenn fablet
Le ven. 12 mai 2017 à 11:07, Alexey Proskuryakov a écrit : > > 9 мая 2017 г., в 11:27, Simon Fraser написал(а): > > > Another consideration here is "would my test be useful for other browser > vendors". I don't think the answer is a unanimous "yes", so I

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread Rick Byers
On Fri, May 12, 2017 at 2:06 PM, Alexey Proskuryakov wrote: > > 9 мая 2017 г., в 11:27, Simon Fraser написал(а): > > Another consideration here is "would my test be useful for other browser > vendors". I don't think the answer is a unanimous "yes", so I

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread Alexey Proskuryakov
> 9 мая 2017 г., в 11:27, Simon Fraser написал(а): > > Another consideration here is "would my test be useful for other browser > vendors". I don't think the answer is a unanimous "yes", so I think we should > only use WPT for tests that will think are worth sharing.

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread Ryosuke Niwa
On Thu, May 11, 2017 at 11:32 PM, Anne van Kesteren wrote: > On Fri, May 12, 2017 at 6:46 AM, Simon Fraser wrote: >> I was under the impression that tests upstreamed from vendor repositories >> would land in WPT tests >> with minimal review, based on

Re: [webkit-dev] Another WPT bite

2017-05-12 Thread Anne van Kesteren
On Fri, May 12, 2017 at 6:46 AM, Simon Fraser wrote: > I was under the impression that tests upstreamed from vendor repositories > would land in WPT tests > with minimal review, based on the fact that they had been reviewed when > landed in the vendor repo. I think

Re: [webkit-dev] Another WPT bite

2017-05-11 Thread Simon Fraser
> On May 11, 2017, at 9:30 PM, Anne van Kesteren wrote: > > On Tue, May 9, 2017 at 8:27 PM, Simon Fraser wrote: >> I'm also concerned that with 4 vendors upstreaming their WPT tests, the WPT >> repo will just become a morass of partially overlapping

Re: [webkit-dev] Another WPT bite

2017-05-11 Thread Anne van Kesteren
On Tue, May 9, 2017 at 8:27 PM, Simon Fraser wrote: > I'm also concerned that with 4 vendors upstreaming their WPT tests, the WPT > repo will just become a morass of partially overlapping tests that takes 4x > longer to run than a curated repo. Why do you think WPT is not

Re: [webkit-dev] Another WPT bite

2017-05-11 Thread youenn fablet
Patch just landed. Location is LayoutTests/http/wpt. I also forgot to say that, should you want to write http served tests, it might make sense to use this folder instead of http/tests. Y Le jeu. 11 mai 2017 à 19:05, Sam Weinig a écrit : > > On May 8, 2017, at 9:31 PM,

Re: [webkit-dev] Another WPT bite

2017-05-11 Thread Sam Weinig
> On May 8, 2017, at 9:31 PM, youenn fablet wrote: > > Hi all, > > Discussing with some WebKittens, testharness.js is more and more used in > WebKit. > Is it time to make testharness.js the recommended way of writing LayoutTests? I am in favor of this. If we simplified

Re: [webkit-dev] Another WPT bite

2017-05-09 Thread youenn fablet
> Another consideration here is "would my test be useful for other browser > vendors". I don't think the answer is a unanimous "yes", so I think we > should only use WPT for tests that will think are worth sharing. > Agreed that some tests, especially the ones dedicated to WebKit specificities

Re: [webkit-dev] Another WPT bite

2017-05-09 Thread Mike Pennisi
> One question I have is whether web platform tests can run under a regular > HTTP server (maybe with appropriate configuration) or do we need something > special? Is the WPT server more than just a web server with specific > configuration settings? While many tests can probably be run in this

Re: [webkit-dev] Another WPT bite

2017-05-09 Thread Ryosuke Niwa
Forgot to CC webkit-dev. - R. Niwa On Tue, May 9, 2017 at 12:36 PM, Ryosuke Niwa wrote: > On Tue, May 9, 2017 at 1:12 AM, Maciej Stachowiak wrote: >> >> >> On May 8, 2017, at 11:15 PM, Ryosuke Niwa wrote: >> >> On Mon, May 8, 2017 at 11:01

Re: [webkit-dev] Another WPT bite

2017-05-09 Thread Ryosuke Niwa
On Tue, May 9, 2017 at 11:06 AM, Maciej Stachowiak > > If we run all the w3c-imported web platform tests under a web server, then > obviously we only need one directory. My understanding is that we don't run > them under a server at all. So it seems like one part of this proposal

Re: [webkit-dev] Another WPT bite

2017-05-09 Thread Maciej Stachowiak
> On May 9, 2017, at 8:11 AM, Geoffrey Garen wrote: > >> What we're suggesting is to give preferential treatments to >> testharness.js over js-test.js / js-test-pre.js when you were already >> planning to write a test with the latter two scripts. > > OK, I think this makes

Re: [webkit-dev] Another WPT bite

2017-05-09 Thread Maciej Stachowiak
> On May 9, 2017, at 8:44 AM, youenn fablet wrote: > > > Besides other issues mentioned, testharness tends to result in more verbose > tests compared to js-test, at least for simple cases. > > For synchronous tests, I am not sure there is any big difference one way or >

Re: [webkit-dev] Another WPT bite

2017-05-09 Thread Geoffrey Garen
> Another concern is the ease of running tests for developers: drag > tests into a browser instead of running a server. Yeah, it’s a pretty big concern if you can’t just drop a simple test case into a browser. > We can partially accommodate this by rewriting >

Re: [webkit-dev] Another WPT bite

2017-05-09 Thread youenn fablet
> > > Besides other issues mentioned, testharness tends to result in more > verbose tests compared to js-test, at least for simple cases. > For synchronous tests, I am not sure there is any big difference one way or the other. With asynchronous tests, it might be true, but using

Re: [webkit-dev] Another WPT bite

2017-05-09 Thread Geoffrey Garen
> What we're suggesting is to give preferential treatments to > testharness.js over js-test.js / js-test-pre.js when you were already > planning to write a test with the latter two scripts. OK, I think this makes sense. But I still think the very best kind of test is a flat file with 10-20 lines

Re: [webkit-dev] Another WPT bite

2017-05-09 Thread Maciej Stachowiak
> On May 8, 2017, at 11:15 PM, Ryosuke Niwa wrote: > > On Mon, May 8, 2017 at 11:01 PM, Brady Eidson > wrote: > > On May 8, 2017, at 10:44 PM, Ryosuke Niwa < > rn...@webkit.org > > wrote: >>> On

Re: [webkit-dev] Another WPT bite

2017-05-09 Thread Ryosuke Niwa
On Mon, May 8, 2017 at 11:01 PM, Brady Eidson wrote: > >> On May 8, 2017, at 10:44 PM, Ryosuke Niwa wrote: >> >> On Mon, May 8, 2017 at 10:17 PM, Brady Eidson wrote: >> >>> But now talking about testharness.js directly, I object on the

Re: [webkit-dev] Another WPT bite

2017-05-09 Thread Brady Eidson
> On May 8, 2017, at 10:44 PM, Ryosuke Niwa wrote: > > On Mon, May 8, 2017 at 10:17 PM, Brady Eidson wrote: > >> But now talking about testharness.js directly, I object on the grounds of "a >> file:// regression test is dirt easy to hack on and work with,

Re: [webkit-dev] Another WPT bite

2017-05-08 Thread Brady Eidson
> On May 8, 2017, at 10:42 PM, youenn fablet wrote: > > testharness.js does not need an http server. Some WPT goodies need the WPT > server. I misunderstood since we were also discussing: >> To continue moving forward, some of us are proposing to serve all tests in >>

Re: [webkit-dev] Another WPT bite

2017-05-08 Thread Ryosuke Niwa
On Mon, May 8, 2017 at 10:17 PM, Brady Eidson wrote: > On May 8, 2017, at 9:31 PM, youenn fablet wrote: > > Hi all, > > Discussing with some WebKittens, testharness.js is more and more used in > WebKit. > Is it time to make testharness.js the recommended way

Re: [webkit-dev] Another WPT bite

2017-05-08 Thread youenn fablet
testharness.js does not need an http server. Some WPT goodies need the WPT server. I agree different frameworks offer different benefits. There is no reason we should mandate one framework in particular. In case there is no specific needs, it makes sense to me to recommend using testharness.js,

Re: [webkit-dev] Another WPT bite

2017-05-08 Thread Brady Eidson
> On May 8, 2017, at 9:31 PM, youenn fablet wrote: > > Hi all, > > Discussing with some WebKittens, testharness.js is more and more used in > WebKit. > Is it time to make testharness.js the recommended way of writing LayoutTests? Setting aside the pros or cons of

Re: [webkit-dev] Another WPT bite

2017-05-08 Thread Chris Dumez
> On May 8, 2017, at 9:44 PM, Geoffrey Garen wrote: > >> Is it time to make testharness.js the recommended way of writing LayoutTests? > > What are the costs and benefits of testharness.js? Benefit: - Tests would be more easily upstreamable to web-platform-tests, which are

Re: [webkit-dev] Another WPT bite

2017-05-08 Thread Geoffrey Garen
> Is it time to make testharness.js the recommended way of writing LayoutTests? What are the costs and benefits of testharness.js? We usually try to make regression tests reductions of some larger problem to aid debugging and to make testing fast. But testharness.js is 95kB. That's kind of the