Re: wpt CSS tests now running on Linux

[Gregory Szorc]

On Thu, Jul 20, 2017 at 9:33 AM, James Graham 
wrote:

> Bug 1341078 and dependencies just landed on inbound, which means we now
> have the W3C/web-platform-tests CSS tests in-tree and running in
> automation. This adds about 12,000 reftests for CSS features to the
> web-platform-tests suite. They are currently enabled in CI, but only on
> linux*, due to limited capacity on OSX, and issues with the harness on
> Windows. The tests will be enabled on other platforms once these problems
> are resolved.
>
> Servo was already running many of these tests in their automation, so this
> landing plugs a gap in the stylo testing vs upstream.
>
> Note that, as usual with wpt landings, these tests have been vetted for
> stability, but not for the actual results.
>
> Changes to the css tests in this directory will be upstreamed to
> web-platform-tests in the same way as for any other web-platform-test. Note
> that the reftest harness versions of the Mozilla submitted tests are still
> running, so if you want to edit or add to those it is recommended to use
> the copy in layout/reftests/w3c-css/submitted/ since that will be
> correctly synchronised and currently runs on more platforms in CI.
>
> The number of tests and nature of reftests means that this change added a
> large number of files to the repository (around 37,000). Apologies for any
> inconvenience caused by such a large changeset. I'm told that narrow clones
> are just around the corner and may make this kind of thing more tolerable
> in the future.
>

"Around the corner" is a bit optimistic. Mercurial 4.3 has experimental
support for sparse checkouts via an extension. (The feature has been part
of a Facebook extension for years.) I plan to leverage this in automation
to speed up VCS interactions. But end-user support will likely be
experimental until the feature stabilizes upstream, hopefully in 4.4 in 3
months.

In the mean time, be sure to install Watchman (https://facebook.github.io/
watchman/) and the fsmonitor Mercurial extension to make operations faster.
`mach mercurial-setup` should help with this.

Git doesn't yet have filesystem watcher integration, so you are pretty much
at the mercy of your filesystem and I/O subsystem. However, an engineer at
Twitter is currently trying to land Watchman support into the upstream Git
project! The first version of the patches was emailed in March. The 14th
iteration was emailed last week. But there still appears to be active
discussion on some high-level details of the patch series. So I'm not
holding my breath waiting for it to land.

For reference, the best timings I could obtain are as follows. These are
under ideal conditions with as much as possible already in filesystem
caches. If you actually need to perform I/O due to a cache miss (which is
pretty common because common build system activity touches tons of files
and tends to cause cache eviction), the differences with Watchman are even
more pronounced. So it isn't uncommon for Watchman to save a second or two
for random operations.


Linux
$ hg status
no watchman: ~1.00s
w/ watchman: ~0.11s

$ git status
~0.28s

MacOS
$ hg status
no watchman: ~1.78s
w/ watchman: ~0.23s

Windows
$ hg status
no watchman: ~2.31s
w/ watchman: ~0.35s


Linux
$ hg status
no watchman: ~1.16s
w/ watchman: ~0.12s

$ git status
~0.33s

MacOS
$ hg status
no watchman: ~2.04s
w/ watchman: ~0.26s

Windows
$ hg status
no watchman: ~2.56s
w/ watchman: ~0.40s
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: wpt CSS tests now running on Linux

[James Graham]

On 20/07/17 18:26, Emilio Cobos Álvarez wrote:

Thanks for this James! \o/

One question, do we run the CSS test linter on automation, or are there
any plans for it?


Yes, that should be run as part of the W lint job (e.g. [1]), which is 
run on pushes (including to try) that change files under 
testing/web-platform/tests/. We don't run it for changes under 
layout/reftests/w3c-css/submitted/ and it's not clear how easy that 
would be, since the lint is rather tied to the structure of the wpt 
repository.


Note that there are other reasons that a push might be blocked upstream 
but land in m-c (e.g. unstable tests). We are working to create an 
upstream PR earlier in the cycle, and improve the communication with 
test authors about problems upstreaming their changes (as well as 
corresponding improvements to downstreaming that should allow us to 
notify relevant people when "interesting" test changes are going to land).


[1]https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound=60fa00c73b05bd2bc0e7485826a86ffed47627c9=testfailed=busted=exception=runnable=pending=running=success=lint=115937895
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: wpt CSS tests now running on Linux

[Emilio Cobos Álvarez]

Thanks for this James! \o/

One question, do we run the CSS test linter on automation, or are there
any plans for it?

We probably should, because otherwise we may only notice when trying to
upstream, like in [1], which is more work for everyone.

[1]:
https://github.com/w3c/web-platform-tests/pull/6357#issuecomment-311778265

On 07/20/2017 06:33 PM, James Graham wrote:
> Bug 1341078 and dependencies just landed on inbound, which means we now
> have the W3C/web-platform-tests CSS tests in-tree and running in
> automation. This adds about 12,000 reftests for CSS features to the
> web-platform-tests suite. They are currently enabled in CI, but only on
> linux*, due to limited capacity on OSX, and issues with the harness on
> Windows. The tests will be enabled on other platforms once these
> problems are resolved.
> 
> Servo was already running many of these tests in their automation, so
> this landing plugs a gap in the stylo testing vs upstream.
> 
> Note that, as usual with wpt landings, these tests have been vetted for
> stability, but not for the actual results.
> 
> Changes to the css tests in this directory will be upstreamed to
> web-platform-tests in the same way as for any other web-platform-test.
> Note that the reftest harness versions of the Mozilla submitted tests
> are still running, so if you want to edit or add to those it is
> recommended to use the copy in layout/reftests/w3c-css/submitted/ since
> that will be correctly synchronised and currently runs on more platforms
> in CI.
> 
> The number of tests and nature of reftests means that this change added
> a large number of files to the repository (around 37,000). Apologies for
> any inconvenience caused by such a large changeset. I'm told that narrow
> clones are just around the corner and may make this kind of thing more
> tolerable in the future.
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

wpt CSS tests now running on Linux

[James Graham]

Bug 1341078 and dependencies just landed on inbound, which means we now 
have the W3C/web-platform-tests CSS tests in-tree and running in 
automation. This adds about 12,000 reftests for CSS features to the 
web-platform-tests suite. They are currently enabled in CI, but only on 
linux*, due to limited capacity on OSX, and issues with the harness on 
Windows. The tests will be enabled on other platforms once these 
problems are resolved.


Servo was already running many of these tests in their automation, so 
this landing plugs a gap in the stylo testing vs upstream.


Note that, as usual with wpt landings, these tests have been vetted for 
stability, but not for the actual results.


Changes to the css tests in this directory will be upstreamed to 
web-platform-tests in the same way as for any other web-platform-test. 
Note that the reftest harness versions of the Mozilla submitted tests 
are still running, so if you want to edit or add to those it is 
recommended to use the copy in layout/reftests/w3c-css/submitted/ since 
that will be correctly synchronised and currently runs on more platforms 
in CI.


The number of tests and nature of reftests means that this change added 
a large number of files to the repository (around 37,000). Apologies for 
any inconvenience caused by such a large changeset. I'm told that narrow 
clones are just around the corner and may make this kind of thing more 
tolerable in the future.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform