> On Dec 4, 2018, at 4:43 PM, Ryosuke Niwa <rn...@webkit.org> wrote:
>
>
> On Tue, Dec 4, 2018 at 12:55 PM Jonathan Bedard <jbed...@apple.com
> <mailto:jbed...@apple.com>> wrote:
> These directories would be along-side tests.
>
> I don't like that platform-specific results are under LayoutTests/platform
> and device-specific results are next to the tests. We should stick with
> either convention, not mix them up.
>
> If we were to match LayoutTests/platforms, we should probably put it under
> LayoutTests/devices, or alternatively inside each platform's test directory.
> Alternatively, I'd be fine if we moved platform specific results to those
> subdirectories. But having both conventions used throughout would be an
> insane mess.
>
> It is different from how we treat platforms because platforms have
> historically been used to differentiate between different binary types, which
> doesn’t really apply here. We also have platform versions which we use for
> inheriting expectations between versions, that idea also doesn’t transfer
> well to the case.
>
> I don't think these differences are all that interesting. The problem I have
> with this approach is that this will be yet another concept new contributor
> has to learn, and yet another layer of abstraction & a new set of
> configurations we'd have to worry about when figuring out which test
> expectation will be used & needs to be updated when WebKit's behavior changes.
The platform solution to this has similar problems (as mentioned bellow,
combining the results from multiple platforms into a single logical platform).
The best we will be able to do here is come up with a system that effects as
few tests as possible. The reason I like optionally splitting expected results
for a test into a directory is because this idiom transfers well to displaying
test failures when a test has run multiple times. I don’t see a good solution
to that problem through the platform technique.
>
> Creating new device-platforms would make EWS and running layout tests locally
> very confusing
>
> Why?
>
> and wouldn’t really solve the run-a-test-twice problem.
>
> Again, why?
Because if devices are platforms, then 'run-webkit-tests —ios’ needs to run 2
platforms. This is a huge departure from what a ‘platform’ conceptually is. We
would need to invent a whole new way to display results, because what a
contributor is thinking of as a platform is, in fact, multiple platforms.
>
> We would have to have an entirely different set of results for each device
> type, and when running on-device tests, the user would have to be aware
> enough to use the platform which corresponds to the device (or devices) they
> had attached. This would be a pretty radical departure from the way things
> currently work, were iPhone 7 and iPad tests run with the rest of the iOS
> tests.
>
> I don't see how treating iPhone 7 as a platform would pose any issue here.
> It's like a variant of iOS port like Mountain Lion is a variant of macOS port.
>
> What I also don't understand is what happens when iOS behavior changes
> between iOS 10 and iOS 11 on iPhone 7. Where would the expected result live
> for those two operating systems and how are they named?
I think that example illustrates why the platform idiom is not right for
handling device specific expected results. Under my proposal, you could place a
platform + device specific expectation like so:
some-dir/
some-test.html
some-test-expected.txt
platform/ios-10/some-dir/some-test-expected/iPhone-7.txt
platform/ios-11/some-dir/some-test-expected/iPhone-7.txt
Jonathan
>
> - R. Niwa
>
>> On Dec 3, 2018, at 6:25 PM, Ryosuke Niwa <rn...@webkit.org
>> <mailto:rn...@webkit.org>> wrote:
>>
>> Are those directories along side tests? If so, that seems very different
>> from the approach we take with platforms. Why the discrepancy?
>>
>> Can't we just treat them as specific type of iOS platforms?
>>
>> - R. Niwa
>>
>>
>> On Mon, Dec 3, 2018 at 4:05 PM Jonathan Bedard <jbed...@apple.com
>> <mailto:jbed...@apple.com>> wrote:
>> Hello everyone,
>>
>> I have a proposal to make writing layout tests for specific types of devices
>> more straight-forward.
>>
>> Currently, we name directories with the beginning of a device name, such as
>> ‘iphone7’ or ‘ipad’ and hope that those directory names match the name of a
>> device type in the associated port’s CUSTOM_DEVICE_CLASS. This causes some
>> problems, however, in a few circumstances. The first problem arises when two
>> ports have associated devices (such as watchOS and iOS), in this case, a
>> directory which maps to a device type on one port might not on another. The
>> second problem is that our current code for managing devices assumes that
>> devices can be booted on-demand, which is not the case in on-device testing,
>> where we need to run different sets of tests depending on the type of device
>> attached. The last (and most important) problem is that we cannot run the
>> same test twice on two different devices in a single test run (ie, run
>> test.html once on an iPhone and once on an iPad).
>>
>> To resolve this, I propose 3 changes to how we organize layout tests.
>>
>> 1) Allow *-expected.txt files to be device-specific
>> <https://bugs.webkit.org/show_bug.cgi?id=192162
>> <https://bugs.webkit.org/show_bug.cgi?id=192162>>
>>
>> The idea here would be to optionally allow *-expected.txt files to be a
>> directory looking something like this:
>>
>> dir/
>> test.html
>> test-expected/
>> iPhone-7.txt
>>
>> Under this scheme, the name of the .txt file would map directly to a device
>> type. Multiple device-specific expectations would be permitted, but making
>> sense of these results requires change #2.
>>
>> 2) Support multiple results for a specific test
>> <https://bugs.webkit.org/show_bug.cgi?id=192163
>> <https://bugs.webkit.org/show_bug.cgi?id=192163>>
>>
>> Similar to #1, we need to allow test results to be attributed to a device
>> type. This would be done with a similar scheme, where results for a test
>> would optionally be saved as a directory looking like this:
>>
>> dir/
>> test/
>> iPhone-7-actual.txt
>> iPhone-7-diff.txt
>> iPhone-7-expected.txt
>> iPhone-7-diff.html
>> iPhone-8-actual.txt
>> iPhone-8-diff.txt
>> iPhone-8-expected.txt
>> iPhone-8-diff.html
>>
>> When reporting, results would be reported to as a new ‘queue’ of the form
>> '<queue>-<device>’.
>>
>> 3) Provide device specific tags in test expectations
>> <https://bugs.webkit.org/show_bug.cgi?id=192164
>> <https://bugs.webkit.org/show_bug.cgi?id=192164>>
>>
>> We already have Debug/Release as well as version tags in our test
>> expectations. The addition here would be device-type tags, those tags would
>> look like this:
>>
>> [ iPhone 7 ] dir/test.html [ Pass ]
>> [ iPad ] dir/other-test.html [ Pass ]
>>
>> These tags would allow for 2 features. The first is similar to existing
>> tags, allowing a certain test (or group of tests) to have different
>> expectations or to be skipped entirely on certain device types. The second
>> feature allows for tests and groups of tests to be run on multiple different
>> devices in a single instantiation of run-webkit-tests. For example,
>>
>> [ iPhone 7, iPad ] dir/some-dir/ [ Pass ]
>>
>> would indicate that every test in dir/some-dir/ will be run on BOTH iPhone 7
>> and iPad.
>>
>> I would welcome anyone else’s thoughts on this.
>>
>> Jonathan
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org <mailto:webkit-dev@lists.webkit.org>
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
>> <https://lists.webkit.org/mailman/listinfo/webkit-dev>
>
_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
