> On Dec 4, 2018, at 4:43 PM, Ryosuke Niwa <rn...@webkit.org> wrote:
> 
> 
> On Tue, Dec 4, 2018 at 12:55 PM Jonathan Bedard <jbed...@apple.com 
> <mailto:jbed...@apple.com>> wrote:
> These directories would be along-side tests.
> 
> I don't like that platform-specific results are under LayoutTests/platform 
> and device-specific results are next to the tests. We should stick with 
> either convention, not mix them up.
> 
> If we were to match LayoutTests/platforms, we should probably put it under 
> LayoutTests/devices, or alternatively inside each platform's test directory. 
> Alternatively, I'd be fine if we moved platform specific results to those 
> subdirectories. But having both conventions used throughout would be an 
> insane mess.
> 
> It is different from how we treat platforms because platforms have 
> historically been used to differentiate between different binary types, which 
> doesn’t really apply here. We also have platform versions which we use for 
> inheriting expectations between versions, that idea also doesn’t transfer 
> well to the case.
> 
> I don't think these differences are all that interesting. The problem I have 
> with this approach is that this will be yet another concept new contributor 
> has to learn, and yet another layer of abstraction & a new set of 
> configurations we'd have to worry about when figuring out which test 
> expectation will be used & needs to be updated when WebKit's behavior changes.

The platform solution to this has similar problems (as mentioned bellow, 
combining the results from multiple platforms into a single logical platform). 
The best we will be able to do here is come up with a system that effects as 
few tests as possible. The reason I like optionally splitting expected results 
for a test into a directory is because this idiom transfers well to displaying 
test failures when a test has run multiple times. I don’t see a good solution 
to that problem through the platform technique.

> 
> Creating new device-platforms would make EWS and running layout tests locally 
> very confusing
> 
> Why?
>  
> and wouldn’t really solve the run-a-test-twice problem.
> 
> Again, why?

Because if devices are platforms, then 'run-webkit-tests —ios’ needs to run 2 
platforms. This is a huge departure from what a ‘platform’ conceptually is. We 
would need to invent a whole new way to display results, because what a 
contributor is thinking of as a platform is, in fact, multiple platforms.

> 
> We would have to have an entirely different set of results for each device 
> type, and when running on-device tests, the user would have to be aware 
> enough to use the platform which corresponds to the device (or devices) they 
> had attached. This would be a pretty radical departure from the way things 
> currently work, were iPhone 7 and iPad tests run with the rest of the iOS 
> tests.
> 
> I don't see how treating iPhone 7 as a platform would pose any issue here. 
> It's like a variant of iOS port like Mountain Lion is a variant of macOS port.
> 
> What I also don't understand is what happens when iOS behavior changes 
> between iOS 10 and iOS 11 on iPhone 7. Where would the expected result live 
> for those two operating systems and how are they named?

I think that example illustrates why the platform idiom is not right for 
handling device specific expected results. Under my proposal, you could place a 
platform + device specific expectation like so:

        some-dir/
                some-test.html
                some-test-expected.txt
        platform/ios-10/some-dir/some-test-expected/iPhone-7.txt
        platform/ios-11/some-dir/some-test-expected/iPhone-7.txt

Jonathan

> 
> - R. Niwa
> 
>> On Dec 3, 2018, at 6:25 PM, Ryosuke Niwa <rn...@webkit.org 
>> <mailto:rn...@webkit.org>> wrote:
>> 
>> Are those directories along side tests? If so, that seems very different 
>> from the approach we take with platforms. Why the discrepancy?
>> 
>> Can't we just treat them as specific type of iOS platforms?
>> 
>> - R. Niwa
>> 
>> 
>> On Mon, Dec 3, 2018 at 4:05 PM Jonathan Bedard <jbed...@apple.com 
>> <mailto:jbed...@apple.com>> wrote:
>> Hello everyone,
>> 
>> I have a proposal to make writing layout tests for specific types of devices 
>> more straight-forward.
>> 
>> Currently, we name directories with the beginning of a device name, such as 
>> ‘iphone7’ or ‘ipad’ and hope that those directory names match the name of a 
>> device type in the associated port’s CUSTOM_DEVICE_CLASS. This causes some 
>> problems, however, in a few circumstances. The first problem arises when two 
>> ports have associated devices (such as watchOS and iOS), in this case, a 
>> directory which maps to a device type on one port might not on another. The 
>> second problem is that our current code for managing devices assumes that 
>> devices can be booted on-demand, which is not the case in on-device testing, 
>> where we need to run different sets of tests depending on the type of device 
>> attached. The last (and most important) problem is that we cannot run the 
>> same test twice on two different devices in a single test run (ie, run 
>> test.html once on an iPhone and once on an iPad).
>> 
>> To resolve this, I propose 3 changes to how we organize layout tests.
>> 
>> 1) Allow *-expected.txt files to be device-specific 
>> <https://bugs.webkit.org/show_bug.cgi?id=192162 
>> <https://bugs.webkit.org/show_bug.cgi?id=192162>>
>> 
>> The idea here would be to optionally allow *-expected.txt files to be a 
>> directory looking something like this:
>> 
>> dir/
>>     test.html
>>     test-expected/
>>         iPhone-7.txt
>> 
>> Under this scheme, the name of the .txt file would map directly to a device 
>> type. Multiple device-specific expectations would be permitted, but making 
>> sense of these results requires change #2.
>> 
>> 2) Support multiple results for a specific test 
>> <https://bugs.webkit.org/show_bug.cgi?id=192163 
>> <https://bugs.webkit.org/show_bug.cgi?id=192163>>
>> 
>> Similar to #1, we need to allow test results to be attributed to a device 
>> type. This would be done with a similar scheme, where results for a test 
>> would optionally be saved as a directory looking like this:
>> 
>> dir/
>>     test/
>>         iPhone-7-actual.txt
>>         iPhone-7-diff.txt
>>         iPhone-7-expected.txt
>>         iPhone-7-diff.html
>>         iPhone-8-actual.txt
>>         iPhone-8-diff.txt
>>         iPhone-8-expected.txt
>>         iPhone-8-diff.html
>> 
>> When reporting, results would be reported to as a new ‘queue’ of the form 
>> '<queue>-<device>’.
>> 
>> 3) Provide device specific tags in test expectations 
>> <https://bugs.webkit.org/show_bug.cgi?id=192164 
>> <https://bugs.webkit.org/show_bug.cgi?id=192164>>
>> 
>> We already have Debug/Release as well as version tags in our test 
>> expectations. The addition here would be device-type tags, those tags would 
>> look like this:
>> 
>> [ iPhone 7 ] dir/test.html [ Pass ]
>> [ iPad ] dir/other-test.html [ Pass ]
>> 
>> These tags would allow for 2 features. The first is similar to existing 
>> tags, allowing a certain test (or group of tests) to have different 
>> expectations or to be skipped entirely on certain device types. The second 
>> feature allows for tests and groups of tests to be run on multiple different 
>> devices in a single instantiation of run-webkit-tests. For example,
>> 
>> [ iPhone 7, iPad ] dir/some-dir/ [ Pass ]
>> 
>> would indicate that every test in dir/some-dir/ will be run on BOTH iPhone 7 
>> and iPad.
>> 
>> I would welcome anyone else’s thoughts on this.
>> 
>> Jonathan
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org <mailto:webkit-dev@lists.webkit.org>
>> https://lists.webkit.org/mailman/listinfo/webkit-dev 
>> <https://lists.webkit.org/mailman/listinfo/webkit-dev>
> 

_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Reply via email to