> On May 9, 2017, at 8:44 AM, youenn fablet <youe...@gmail.com> wrote: > > > Besides other issues mentioned, testharness tends to result in more verbose > tests compared to js-test, at least for simple cases. > > For synchronous tests, I am not sure there is any big difference one way or > the other. > With asynchronous tests, it might be true, but using > testharness.js/promise_test usually improves things there. > I personally find it easier to not wrap code-to-be-tested into quotes.
It seems like it's a matter of personal preference to some extent. So I am not sure we should recommend one or the other (besides for tests that are intended to be contributed to web-platform-tests). > > Another concern is the lack of verbose output which reduces the ability to > debug failing tests. > This can be partially fixed by authoring tests with that issue in mind. > For instance, having a big promise_test to handle the asynchronous aspect of > a test and nested test() inside it. I don't think we've ever had a problem with debugging js-test.js tests when they fail. > >> The thing I specifically asked Youenn to ask is, whether we should >> place a test inside LayoutTests/wpt like LayoutTests/http/tests when >> we want to write a test using testharness.js which requires some sort >> of network code. >> >> Since people have had some opinions about directory structures in the past. > > > It seems like we need a few different directories, here are my opinions on > them: > > (1) Imported web platform tests that don't need a server > Currently LayoutTests/imported/w3c/web-platform-tests, which seems fine. > > All WPT tests are expected to run behind the WPT server. > That is the way tests are authored and tested elsewhere. > If we have an issue with that, it is best to bring that and fix that directly > in WPT. > I encountered several times small issues due to file:// based origins which > makes me think defaulting to http is a safe choice. > > One concern is efficiency. We should study that and improve on that. > Another concern is the ease of running tests for developers: drag&dropping > tests into a browser instead of running a server. > We can partially accommodate this by rewriting > testharness.js/testharnessreport.js urls. > A significant and growing amount of wpt tests will not behave as expected > (other resources loaded, origins, need for specific headers, need for > https...) > > (2) Imported web platform tests that do need a server > Probably should be under LayoutTests/imported/w3c/ somewhere, or maybe > under http/tests/ per point (4) > > I don't think this will work, web-platform-tests is organized in terms of > features. > There is no clear separation between file based compatible tests and http > based tests like in WebKit. If we run all the w3c-imported web platform tests under a web server, then obviously we only need one directory. My understanding is that we don't run them under a server at all. So it seems like one part of this proposal is "run everything under LayoutTests/imported/w3c/ from a web server". > > (3) Custom testharness.js tests that don't need a server > Probably these should just go in their normal topic-specific directories > and should not need a special directory > > Right. > The only case where it might make sense to put such tests in a specific > WPT-enabled directory is if the plan is to upstream these tests at some point. > Such tests could be added in imported/w3c/web-platform-tests directly but > this requires coordination with resyncing tests at the moment. > In a not-too-far-future, I hope such tests would directly be authored in > imported/w3c/web-platform-tests. I think it would be cleaner to have a separate directory of tests intended for import, separate from imported tests. It could be right next to imported/w3c/web-platform-tests. I think mixing tests that are imported from upstream and tests intended for eventual upstreaming is more confusing than helpful. > > (4) Custom testharness.js tests that do need a server > Can these just be a subdirectory of http/tests/? We have websocket and > ssl/tls tests in there too. Would be nice to not need a separate directory > for networking tests that to use a particular test framework. > > I do not have strong feelings there, http/wpt might make sense if it is found > easier to understand to everybody. > I'll update the patch accordingly and will land it sometimes this week if > there is no additional feedback. One question I have is whether web platform tests can run under a regular HTTP server (maybe with appropriate configuration) or do we need something special? Is the WPT server more than just a web server with specific configuration settings? Regards, Maciej
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
