Re: [webkit-dev] Rename LayoutTests

[Maciej Stachowiak]

> On May 10, 2017, at 2:51 AM, Konstantin Tokarev  wrote:
> 
> 
> 
> 10.05.2017, 12:42, "Ryosuke Niwa"  >:
>   On Wed, May 
> 10, 2017 at 2:14 AM, Ryosuke Niwa < 
> rn...@webkit.org > 
> wrote:
>>>  On Wed, May 10, 2017 at 12:02 AM, Maciej Stachowiak >> > wrote:
  IntegrationTests doesn't distinguish them from performance tests, or API
  tests. And most test integration only incidentally. FunctionalTests 
 doesn't
  distinguish them from any of the other kinds of tests besides performance
  tests.
  I think just plain Tests is better than calling out a characteristic that
  isn't actually unique. It's by far the most common type of test we have, 
 so
  it's ok for it to be the unmarked category.
>>>  The problem with Tests is that it competes with Tools.
>>>  (Renaming Tools is a lot harder not to mention it's a pretty
>>>  descriptive name already).
>>>  How about CorrectnessTests or CoreTests then?
>> I guess another name along this line would be RegressionTests.
> 
>> On Wed, May 10, 2017 at 2:14 AM, Ryosuke Niwa > > wrote:
>>>  On Wed, May 10, 2017 at 12:02 AM, Maciej Stachowiak >> > wrote:
  IntegrationTests doesn't distinguish them from performance tests, or API
  tests. And most test integration only incidentally. FunctionalTests 
 doesn't
  distinguish them from any of the other kinds of tests besides performance
  tests.
 
  I think just plain Tests is better than calling out a characteristic that
  isn't actually unique. It's by far the most common type of test we have, 
 so
  it's ok for it to be the unmarked category.
>>> 
>>>  The problem with Tests is that it competes with Tools.
>>>  (Renaming Tools is a lot harder not to mention it's a pretty
>>>  descriptive name already).

It has a two-letter disambiguation at least, which is better than what you get 
for WebCore / WebKit / WebKit2.

>>> 
>>>  How about CorrectnessTests or CoreTests then?
>> 
>> I guess another name along this line would be RegressionTests.

Most of our other tests are also regression tests.

> 
> Why not just keep historical name, so nobody is confused what is what?

CoreTests is brief and descriptive IMO. But I agree that LayoutTests is ok, 
even though it's not quite accurate any more.

> 
>> 
  I think the only names that are both accurate and unique are likely to be
  bad for autocomplete (WebTests, WebContentTests, etc).
>>> 
>>>  I guess we can also move WebKit.xcworkspace into Source, and rename
>>>  WebKitLibraries to SystemInterfaceLibraries (and maybe move into
>>>  Source), and then move Websites into Tools.
>>> 
>>>  - R. Niwa
>> 
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org 
>> https://lists.webkit.org/mailman/listinfo/webkit-dev 
>> 
> 
> -- 
> Regards,
> Konstantin

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Rename LayoutTests

[Konstantin Tokarev]

10.05.2017, 12:42, "Ryosuke Niwa" :
> On Wed, May 10, 2017 at 2:14 AM, Ryosuke Niwa  wrote:
>>  On Wed, May 10, 2017 at 12:02 AM, Maciej Stachowiak  wrote:
>>>  IntegrationTests doesn't distinguish them from performance tests, or API
>>>  tests. And most test integration only incidentally. FunctionalTests doesn't
>>>  distinguish them from any of the other kinds of tests besides performance
>>>  tests.
>>>
>>>  I think just plain Tests is better than calling out a characteristic that
>>>  isn't actually unique. It's by far the most common type of test we have, so
>>>  it's ok for it to be the unmarked category.
>>
>>  The problem with Tests is that it competes with Tools.
>>  (Renaming Tools is a lot harder not to mention it's a pretty
>>  descriptive name already).
>>
>>  How about CorrectnessTests or CoreTests then?
>
> I guess another name along this line would be RegressionTests.

Why not just keep historical name, so nobody is confused what is what?

>
>>>  I think the only names that are both accurate and unique are likely to be
>>>  bad for autocomplete (WebTests, WebContentTests, etc).
>>
>>  I guess we can also move WebKit.xcworkspace into Source, and rename
>>  WebKitLibraries to SystemInterfaceLibraries (and maybe move into
>>  Source), and then move Websites into Tools.
>>
>>  - R. Niwa
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

-- 
Regards,
Konstantin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Rename LayoutTests

[Ryosuke Niwa]

On Wed, May 10, 2017 at 2:14 AM, Ryosuke Niwa  wrote:
> On Wed, May 10, 2017 at 12:02 AM, Maciej Stachowiak  wrote:
>>
>> IntegrationTests doesn't distinguish them from performance tests, or API
>> tests. And most test integration only incidentally. FunctionalTests doesn't
>> distinguish them from any of the other kinds of tests besides performance
>> tests.
>>
>> I think just plain Tests is better than calling out a characteristic that
>> isn't actually unique. It's by far the most common type of test we have, so
>> it's ok for it to be the unmarked category.
>
> The problem with Tests is that it competes with Tools.
> (Renaming Tools is a lot harder not to mention it's a pretty
> descriptive name already).
>
> How about CorrectnessTests or CoreTests then?

I guess another name along this line would be RegressionTests.

>> I think the only names that are both accurate and unique are likely to be
>> bad for autocomplete (WebTests, WebContentTests, etc).
>
> I guess we can also move WebKit.xcworkspace into Source, and rename
> WebKitLibraries to SystemInterfaceLibraries (and maybe move into
> Source), and then move Websites into Tools.
>
> - R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Rename LayoutTests

[Ryosuke Niwa]

On Wed, May 10, 2017 at 12:02 AM, Maciej Stachowiak  wrote:
>
> IntegrationTests doesn't distinguish them from performance tests, or API
> tests. And most test integration only incidentally. FunctionalTests doesn't
> distinguish them from any of the other kinds of tests besides performance
> tests.
>
> I think just plain Tests is better than calling out a characteristic that
> isn't actually unique. It's by far the most common type of test we have, so
> it's ok for it to be the unmarked category.

The problem with Tests is that it competes with Tools.
(Renaming Tools is a lot harder not to mention it's a pretty
descriptive name already).

How about CorrectnessTests or CoreTests then?

> I think the only names that are both accurate and unique are likely to be
> bad for autocomplete (WebTests, WebContentTests, etc).

I guess we can also move WebKit.xcworkspace into Source, and rename
WebKitLibraries to SystemInterfaceLibraries (and maybe move into
Source), and then move Websites into Tools.

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Rename LayoutTests

[Maciej Stachowiak]

A few more coats of paint for the bike shed:

> On May 9, 2017, at 10:45 PM, Ryosuke Niwa  wrote:
> 
> On Tue, May 9, 2017 at 10:23 PM, Maciej Stachowiak  > wrote:
>
> On May 9, 2017, at 9:07 PM, Michael Catanzaro 
> < mcatanz...@igalia.com 
> > wrote:
>>> On Tue, May 9, 2017 at 11:01 PM, Maciej Stachowiak >> > wrote:
 How about just Tests?
 Or alternately, RegressionTests. But I like just plain Tests.
>>> Then we should move ManualTests
>> I'd be in favor of burying this somewhere deeper. As it is, people are still 
>> adding tests here, which is kind of a disaster. These tests are very rarely 
>> run, so a manual test is often barely better than no test at all. We should 
>> also put a file in this directory strongly discouraging the addition of new 
>> manual tests IMO.
>>> , PerformanceTests,
>> Could be renamed Benchmarks.
> 
>> 
>> 
>>> On May 9, 2017, at 9:07 PM, Michael Catanzaro >> > wrote:
>> 
>>> , PerformanceTests,
>> 
>> Could be renamed Benchmarks.
> 
> I'm not sure benchmarks would be a good description given that
> directory also contains perf tests that are written to test specific
> feature like line layout and DOM bindings.

Those are still benchmarks (albeit microbenchmarks). But I think it would be OK 
to still call it "PerformanceTests", since unqualified "Tests" connotes 
functional tests. But "benchmark" and "performance test" are basically synonyms.

> 
> I find it much nicer to have a separate test directory under which the
> source code structure is mirrored such as UnitTests/WTFTests/,
> UnitTests/WebCoreTests/, etc...

Having UnitTests and APITests directories at top level might be better than 
having them under Tools/TestWebKitAPI/Tests.


>> If we did add any special directories to Tests with different semantics, 
>> they could just be special directories that are peers to the others, much 
>> like the http/ directory.
>> 
>> What are now called LayoutTests have the distinction (along with 
>> PerformanceTests) of being tests that can cover things up and down the 
>> stack. Most other tests could be assigned to one of the subdirectories of 
>> Source.
> 
> This is why I think IntegrationTests or FunctionalTests most
> accurately describe these tests.

IntegrationTests doesn't distinguish them from performance tests, or API tests. 
And most test integration only incidentally. FunctionalTests doesn't 
distinguish them from any of the other kinds of tests besides performance tests.

I think just plain Tests is better than calling out a characteristic that isn't 
actually unique. It's by far the most common type of test we have, so it's ok 
for it to be the unmarked category.

I also think LayoutTests is ok; it's not totally accurate but it's a historical 
name that most people understand at this point.

I think the only names that are both accurate and unique are likely to be bad 
for autocomplete (WebTests, WebContentTests, etc).

Regards,
Maciej

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Rename LayoutTests

[Maciej Stachowiak]

> On May 9, 2017, at 10:43 PM, Dan Bernstein  wrote:
> 
> 
>> On May 9, 2017, at 10:23 PM, Maciej Stachowiak > > wrote:
>> 
>>> JSTests, and
>> 
>> Could go under JavaScriptCore, since these by design don't test anything 
>> outside of JavaScriptCore.
> 
> There is an advantage to Apple’s internal production build system in keeping 
> large amounts of test content out of projects’ source directories (which is 
> why we moved it out of JavaScriptCore in 
> https://bugs.webkit.org/show_bug.cgi?id=160180 
> ), but if that’s the 
> preferred directory layout, then we can make some project changes to 
> accommodate the aforementioned build system.

My first preference would be to leave these tests in JSTests, even if we 
renamed the main tests directory to Tests.

 - Maciej

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Rename LayoutTests

[Ryosuke Niwa]

On Tue, May 9, 2017 at 10:23 PM, Maciej Stachowiak  wrote:
>
>
>> On May 9, 2017, at 9:07 PM, Michael Catanzaro  wrote:
>>
>> On Tue, May 9, 2017 at 11:01 PM, Maciej Stachowiak  wrote:
>>> How about just Tests?
>>> Or alternately, RegressionTests. But I like just plain Tests.
>>
>> Then we should move ManualTests
>
> I'd be in favor of burying this somewhere deeper. As it is, people are still 
> adding tests here, which is kind of a disaster. These tests are very rarely 
> run, so a manual test is often barely better than no test at all. We should 
> also put a file in this directory strongly discouraging the addition of new 
> manual tests IMO.
>
>> , PerformanceTests,
>
> Could be renamed Benchmarks.

I'm not sure benchmarks would be a good description given that
directory also contains perf tests that are written to test specific
feature like line layout and DOM bindings.

>
>> JSTests, and
>
> Could go under JavaScriptCore, since these by design don't test anything 
> outside of JavaScriptCore. But I also think having Tests, JSTests and 
> Benchmarks at top level would be totally understandable.

This used to be the case but it got moved out of JavaScriptCore in
https://trac.webkit.org/changeset/203972 to be aligned with
LayoutTests and ManualTests and ease the pain of grep'ing
Source/JavaScriptCore, which used to find a whole bunch of things
inside Source/JavaScriptCote/Tests

>> TestWebKitAPI
>
> That might be reasonable for TestWebKitAPI/Tests but probably not 
> TestWebKitAPI itself, since the part other than the tests is a harness like 
> WebKitTestRunner. And I'm not sure it's practical to move just the tests. 
> It's probably easier to have them contained inside the tool in the source 
> tree.
>
> Also this is a weird mix of native code tests of public APIs for different 
> frameworks, and native code unit tests for some internal classes (mostly 
> low-level data structures).
>
> If these didn't need to be linked into a single tool, I might suggest that 
> each framework should contain its own units and API tests.

I'm not sure if that would be a good idea for the same reason JSTests
got moved out of Source/JavaScriptCore. Chromium use a similar
directory structure where tests are located next to code, and I find
that to be extremely annoying when grep'ing the code because I end up
finding a whole bunch of code in tests while looking for some keyword.

I find it much nicer to have a separate test directory under which the
source code structure is mirrored such as UnitTests/WTFTests/,
UnitTests/WebCoreTests/, etc...

>> underneath it, because it would be weird to have tests outside of the Tests 
>> directory. Right?
>>
>> And then we would probably want to move all the layout tests to a new 
>> subdirectory, to separate them from the ManualTests, PerformanceTests, 
>> JSTests, and API tests. Then we have to find a name that subdirectory
>
> If we did add any special directories to Tests with different semantics, they 
> could just be special directories that are peers to the others, much like the 
> http/ directory.
>
> What are now called LayoutTests have the distinction (along with 
> PerformanceTests) of being tests that can cover things up and down the stack. 
> Most other tests could be assigned to one of the subdirectories of Source.

This is why I think IntegrationTests or FunctionalTests most
accurately describe these tests.

> But I'd also be ok with having special subdirectories under Source.

I'd be strongly against that approach given my prior experience
working with (sub-)projects which used that structure including but
not limited to JavaScriptCore, webkitpy, and Chromium.

> BTW we also have:
>  bindings tests (under WebCore/bindings/scripts/tests)
>  perl tests (under Tools)
>  python tests (under Tools)
>  builtins-generator-tests (not sure what or where these are)
>  dashboard-tests (also not sure what or where these are)
>
> In general, I think it's good for more specific kinds of tests like this to 
> be next to their relevant code.

I think these are small enough that they don't suffer from the same
issue of making grep'ing much harder although I've found binding tests
to be annoying for the same reason.

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Rename LayoutTests

[Dan Bernstein]

> On May 9, 2017, at 10:23 PM, Maciej Stachowiak  wrote:
> 
>> JSTests, and
> 
> Could go under JavaScriptCore, since these by design don't test anything 
> outside of JavaScriptCore.

There is an advantage to Apple’s internal production build system in keeping 
large amounts of test content out of projects’ source directories (which is why 
we moved it out of JavaScriptCore in 
https://bugs.webkit.org/show_bug.cgi?id=160180 
), but if that’s the preferred 
directory layout, then we can make some project changes to accommodate the 
aforementioned build system.___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Rename LayoutTests

[Manuel Rego Casasnovas]

On 10/05/17 04:23, Ryosuke Niwa wrote:
> Continuing the tradition of a massive rename in
> https://lists.webkit.org/pipermail/webkit-dev/2012-August/021746.html,
> I suggest we rename the top-level LayoutTests directory to something
> more descriptive of the current state.
> 
> Some ideas:
> IntegrationTests - Represents what they are for WebCore.
> AutomatedTests - As opposed to ManualTests.
> ContentTests - Not great for tab completion.
> WebTests - Probably a bad idea for tab completion.

The very same discussion is happening right now on blink-dev:
https://groups.google.com/a/chromium.org/d/topic/blink-dev/KKNbuzj-3HY/discussion

A document with some ideas and pros/cons of each one:
https://docs.google.com/document/d/1wLSfod8apDy2sZsmXpTLRacfjweXP86C-V0wx5koJBg/edit

BTW, on a first sight I like "WebTests/" or even "Tests/".

Bye,
  Rego
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Rename LayoutTests

[Maciej Stachowiak]

> On May 9, 2017, at 9:07 PM, Michael Catanzaro  wrote:
> 
> On Tue, May 9, 2017 at 11:01 PM, Maciej Stachowiak  wrote:
>> How about just Tests?
>> Or alternately, RegressionTests. But I like just plain Tests.
> 
> Then we should move ManualTests

I'd be in favor of burying this somewhere deeper. As it is, people are still 
adding tests here, which is kind of a disaster. These tests are very rarely 
run, so a manual test is often barely better than no test at all. We should 
also put a file in this directory strongly discouraging the addition of new 
manual tests IMO.

> , PerformanceTests,

Could be renamed Benchmarks.

> JSTests, and

Could go under JavaScriptCore, since these by design don't test anything 
outside of JavaScriptCore. But I also think having Tests, JSTests and 
Benchmarks at top level would be totally understandable.

> TestWebKitAPI

That might be reasonable for TestWebKitAPI/Tests but probably not TestWebKitAPI 
itself, since the part other than the tests is a harness like WebKitTestRunner. 
And I'm not sure it's practical to move just the tests. It's probably easier to 
have them contained inside the tool in the source tree.

Also this is a weird mix of native code tests of public APIs for different 
frameworks, and native code unit tests for some internal classes (mostly 
low-level data structures).

If these didn't need to be linked into a single tool, I might suggest that each 
framework should contain its own units and API tests.

> underneath it, because it would be weird to have tests outside of the Tests 
> directory. Right?
> 
> And then we would probably want to move all the layout tests to a new 
> subdirectory, to separate them from the ManualTests, PerformanceTests, 
> JSTests, and API tests. Then we have to find a name that subdirectory

If we did add any special directories to Tests with different semantics, they 
could just be special directories that are peers to the others, much like the 
http/ directory.

What are now called LayoutTests have the distinction (along with 
PerformanceTests) of being tests that can cover things up and down the stack. 
Most other tests could be assigned to one of the subdirectories of Source. But 
I'd also be ok with having special subdirectories under Source.

BTW we also have:
 bindings tests (under WebCore/bindings/scripts/tests)
 perl tests (under Tools)
 python tests (under Tools)
 builtins-generator-tests (not sure what or where these are)
 dashboard-tests (also not sure what or where these are)

In general, I think it's good for more specific kinds of tests like this to be 
next to their relevant code.

Regards,
MAciej

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Rename LayoutTests

[Michael Catanzaro]

On Tue, May 9, 2017 at 11:01 PM, Maciej Stachowiak  
wrote:

How about just Tests?

Or alternately, RegressionTests. But I like just plain Tests.


Then we should move ManualTests, PerformanceTests, JSTests, and 
TestWebKitAPI underneath it, because it would be weird to have tests 
outside of the Tests directory. Right?


And then we would probably want to move all the layout tests to a new 
subdirectory, to separate them from the ManualTests, PerformanceTests, 
JSTests, and API tests. Then we have to find a name that 
subdirectory


Michael

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Rename LayoutTests

[Ryosuke Niwa]

On Tue, May 9, 2017 at 9:01 PM, Maciej Stachowiak  wrote:
> How about just Tests?
>
> Or alternately, RegressionTests. But I like just plain Tests.

But we also have JSTests, ManualTests, and PerformanceTests so I think
it's nice to convey that what they're testing. e.g. WebContentTests
would work but that's not great for tab completion.

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Rename LayoutTests

[Maciej Stachowiak]

How about just Tests?

Or alternately, RegressionTests. But I like just plain Tests.

> On May 9, 2017, at 8:51 PM, Michael Catanzaro  wrote:
> 
>> On Tue, May 9, 2017 at 9:23 PM, Ryosuke Niwa  wrote:
>> AutomatedTests - As opposed to ManualTests.
> 
> The API tests under Tools/TestWebKitAPI (which never seemed to me like a good 
> location for tests) are also automated, so this doesn't seem like the best 
> name unless we want to move the API tests there too.
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Rename LayoutTests

[Michael Catanzaro]

On Tue, May 9, 2017 at 9:23 PM, Ryosuke Niwa  wrote:

AutomatedTests - As opposed to ManualTests.


The API tests under Tools/TestWebKitAPI (which never seemed to me like 
a good location for tests) are also automated, so this doesn't seem 
like the best name unless we want to move the API tests there too.


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev