> On Dec 12, 2018, at 10:07 AM, Jonathan Bedard <jbed...@apple.com> wrote:
> 
> Ryosuke and I discussed this on Monday, and in passing, Ryosuke mentioned 
> that he personally finds something like this:
> 
>       <test-name>.html
>       <test-name>-expected.txt
>       <test-name>-expected-<device-type>.txt
> 
> more clear than the directory method I proposed. After implementing the above 
> approach in the patch uploaded to 
> <https://bugs.webkit.org/show_bug.cgi?id=192162 
> <https://bugs.webkit.org/show_bug.cgi?id=192162>>, I’m inclined to agree. 
> Ryosuke’s approach achieves everything we need for device-type specific 
> expected results.
> 
> This still doesn’t solve disagreements about how to organize test results 
> when a single test is run on multiple device types, but it seems like a step 
> in the right direction.

In my opinion, we should think about what kind of device and platform 
differences we expect, and see if it can be organized into a single model. It 
strikes me as odd to have two totally different ways to organize variant 
results. And we don’t necessarily need to consider different platforms to be 
only targets with different binaries.

One issue with these flat device names is that they have no hierarchy. It was 
hard for me to tell if your iPhone 7 vs iPhone 8 example of a difference was 
real or just imaginary, but I’d expect more tests to be different for iPhone vs 
iPad than different for iPhone 7 vs iPhone 8, so it would be nice to have a 
hierarchy for iPhone as a device class with different types of iPhones under it.

But of course, while a device hierarchy could be fit under a flat notion of OS, 
the trick is how to fit it with a sequence of OS versions. Using directories 
for OS versions but filename variations for devices classes seems weird to me, 
and maybe almost backwards. I’d expect many device class differences to be 
permanent (iPhone vs iPad for example), while OS version differences may be 
transitory, in that they are often quirks of an older OS that will not matter 
once we no longer support that OS.

I think it’s worth thinking through all the variations that would be needed for 
a few real tests (ideally ones that already vary by OS version but which would 
also vary by device type) and make a single model that makes sense.

Maybe we should just use filename variations for everything, since that 
naturally expresses independent variation along multiple dimensions, while 
directories can only represent a single hierarchy. The trick then would be 
figuring out the priority order. If I have <test-name>-expected-ios-ipad.txt 
and <test-name>-expected-ios12.txt, then which is the right one to use on an 
iOS 12 iPad? Maybe we could have a convention to make ambiguous variation like 
this an error, or else decide whether OS version or device should take priority.

Regards,
Maciej



> 
> Jonathan
> 
>> On Dec 4, 2018, at 5:41 PM, Alexey Proskuryakov <a...@webkit.org 
>> <mailto:a...@webkit.org>> wrote:
>> 
>> 
>> 
>>> 4 дек. 2018 г., в 16:43, Ryosuke Niwa <rn...@webkit.org 
>>> <mailto:rn...@webkit.org>> написал(а):
>>> 
>>> 
>>> On Tue, Dec 4, 2018 at 12:55 PM Jonathan Bedard <jbed...@apple.com 
>>> <mailto:jbed...@apple.com>> wrote:
>>> These directories would be along-side tests.
>>> 
>>> I don't like that platform-specific results are under LayoutTests/platform 
>>> and device-specific results are next to the tests. We should stick with 
>>> either convention, not mix them up.
>>> 
>>> If we were to match LayoutTests/platforms, we should probably put it under 
>>> LayoutTests/devices, or alternatively inside each platform's test 
>>> directory. Alternatively, I'd be fine if we moved platform specific results 
>>> to those subdirectories. But having both conventions used throughout would 
>>> be an insane mess.
>> 
>> I'm not ready to have an opinion about which approach is best, however I 
>> wanted to make a general comment about unification.
>> 
>> I think that the attempt to abstract all sorts of differences behind the 
>> "port" and "platform" concepts in webkitpy was short sighted. There are many 
>> subtle and not so subtle variations of behaviors (OS, OS family, specific OS 
>> version, build style, specific API used from the build when there are 
>> multiple options). Trying to represent these by god objects is creating a 
>> lot more inconsistency and confusion than necessary.
>> 
>> One good thing about platform directories is that we can specify inheritance 
>> order with these in a way that is somewhat meaningful. If we are to have an 
>> inheritance order for device types, then continuing with the approach would 
>> make sense. But if the rule is exact match, or best match, then storing them 
>> as -expected directories makes good sense to me.
>> 
>> - Alexey
>> 
> 
> _______________________________________________
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Reply via email to