Re: [webkit-dev] WebKit Transition to Git

[Robert Ma]

On Mon, Oct 5, 2020 at 6:22 AM Frédéric Wang  wrote:

> One thing to take into account is that WebKit's repository is big and
> public GitHub/GitLab prevent creating large repository by default. This
> means it might not be possible for contributors to actually fork
> WebKit's repository on their account and then create a pull request
> (which is the standard way IIUC). Instead, we would probably end up
> doing like web-platform-tests and give contributors the permission to
> create branches to the WebKit account and make Pull Request to the
> master branch. Probably, we should forbid people to commit to the master
> branch directly (I think someone broke WPT's master branch that way last
> year)...
>

Note that this is not exactly the reason we give people write access in
WPT. The WPT repo is not too big to impede forking. Rather, this was to
work around some CI setup constraints (e.g. some secrets not accessible
from forks), most of which have been resolved so the write access is
largely for convenience for active developers (e.g. avoid having to sync
the fork

).

And yeah, do remember to protect the master branch :)
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Supporting for finding ref tests

[Robert Ma]

MANIFEST.json mostly stays out of the way / happens under the hood. It is
cached as GitHub releases, and ignored by .gitignore. One of its purposes
is a cache of the reftest graph.

Once you run any test or explicitly run `./wpt manifest`, the manifest will
be generated/updated in the root of your WPT checkout.

On Tue, Nov 12, 2019 at 2:06 PM Simon Fraser  wrote:

> > On Nov 12, 2019, at 4:52 AM, Frédéric Wang  wrote:
> >
> > On 09/11/2019 04:02, Ryosuke Niwa wrote:
> >>
> >  - Requires us modifying each port's DRT to support this format
> >
> > No, it just requires webkitpy hacking which I've done in the patch.
>  I'm not certain writing a bunch of regular expressions in webkitpy is
>  a reliable mechanism to find expected results. Another issue I found
>  back then was that it significantly slowed run-webkit-tests' startup
>  time because WPT has a workflow to find all tests & their expected
>  results upfront before any tests could run.
> >>> The patch uses html5lib (via BeautifulSoup), which is exactly what
> WPT, and our importer use to find the ref tests.
> >>>
> >>> We don't find references up-front; only when running each test. This
> patch does add some overhead for parsing each test file,
> >>> which I measured to be about 1-2 sec on a directory which took 30s to
> run. I think this slight slowdown is worthwhile (we could
> >>> probably eliminate it with some webkitpy optimizations).
> >> Hm... that's ~3% overhead.
> >
> > @Simon: I agree with Ryosuke that 3% sounds big. IIUC you are parsing
> > the HTML file when running each test? I thought that there is a
> > MANIFEST.json file which is supposed to cache that information, why
> > can't we use it?
>
> I don't see any files call MANIFEST.json in the wpt repo.
>
> There are reftest.list files but these are obsolete.
>
> I hope to get that 3% back by other webkitpy perf optimizations (python
> optimization hints welcome).
>
> Simon
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Supporting for finding ref tests

[Robert Ma]

WPT has recently passed an RFC

trying to simplify the reftest graph (although it has not been implemented
yet), which eliminates a lot of the complexities and concerns.

On Fri, Nov 8, 2019 at 5:07 PM Ryosuke Niwa  wrote:

> On Fri, Nov 8, 2019 at 2:01 PM Simon Fraser 
> wrote:
> >
> > I'd like to land a patch to support finding test references via  rel="match/mismatch">:
> > https://bugs.webkit.org/show_bug.cgi?id=203784
> >
> > There has been some discussion about this in the past:
> > https://lists.webkit.org/pipermail/webkit-dev/2011-November/018470.html
> >
> > But I think the benefits outweigh the drawbacks. As that mail states:
> >
> > *Link element approach*
> > Pros:
> >
> >- Can reuse same ref. file for multiple tests
> >
> > Still true.
> >
> >- Can have multiple ref. files for single test
> >
> > True but no something that we support, and I haven't see any WPT use
> this (our importer throws an error if it sees this)
> >
> >- Information is self-contained in the test file
> >
> > Still true
> >
> >- We may get away with test suite build step
> >
> > It certainly simplifies WPT test import.
> >
> > Currently importing some CSS suites (e.g. css-backgrounds) results in
> broken -expected.html files because copying them breaks references to sub
> resources.
> >
> > (It turns out that we can't convert W3C ref tests to use WebKit
> conventions
> > due to the first two points.)
> >
> > We're doing this much more now, and the "multiple references" point is
> moot, so I think we can import WPT tests mostly as-is.
> >
> > Cons:
> >
> >- Requires us modifying each port's DRT to support this format
> >
> > No, it just requires webkitpy hacking which I've done in the patch.
>
> I'm not certain writing a bunch of regular expressions in webkitpy is
> a reliable mechanism to find expected results. Another issue I found
> back then was that it significantly slowed run-webkit-tests' startup
> time because WPT has a workflow to find all tests & their expected
> results upfront before any tests could run.
> >
> >- Adding link elements itself may affect tests (all W3C tests are
> >required to have link elements at the moment)
> >
> > I haven't seen this be an issue.
>
> Another issue is that if you were to modify a test which happens to be
> also used as a reference or a mismatch result (worse) for some other
> test, then you may not notice that without inspecting every other test
> in existence.
>
> >- Hard to understand relationship between files. e.g. if we want to
> >figure out which tests use ref.html, we must look at all test files
> >
> > This is true, but I don't really see it being a problem in practice.
>
> This definitely is an issue. It's possible WPT has improved things but
> we've definitely had an experience where tests were used as reference
> for other tests, etc... and having to think about this issue every
> time I touch test drove me nuts.
>
> > What I have seen is us importing CSS 2.1 tests that have foo.html and
> foo-ref.html, and treating foo-ref.html as a test so generating
> foo-expected.txt and foo-ref-expected.txt. That seems worse.
>
> Seems like we can treat "-ref" as a special suffix like we already do
> with support directory and resources directory.
>
> > So now that WPT is heavily invested in  I think we should
> follow suite. It will simplify WPT import, and reduced the number of cloned
> -expected.html files significantly.
>
> I really don't want to deal with tests being used as references for
> other tests. I'm okay with this approach if we forbid that.
>
> - R. Niwa
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Robert Ma]

Any thoughts on bytes and Unicode strings, especially the string literals
in the code base?

On Fri, Jul 12, 2019 at 3:38 PM Tim Horton  wrote:

>
>
> On Jul 12, 2019, at 12:18 PM, Jonathan Bedard  wrote:
>
> Hello WebKit developers,
>
> Now that the Catalina developer seeds are available, it is official that
> the new Mac developer tools come with Python 3. As a result, we need to
> continue the ongoing discussion about migrating our Python 2.7 scripts to
> Python 3.
>
> I propose that, over the next 9 months, we do the following:
>
> 1. Make any no-cost Python 3 compatibility changes, in particular
> - print foo -> print(foo)
> - import .foo -> import webkitpy.foo
> 2. Convert any scripts not used in automation to Python 3 ASAP (scripts
> like bisect-builds, block-spammers, compare-results)
> 3. Make most Python 3 compatibility changes which sacrifice efficiency,
> subject to a case-by-case audit. These would be things like:
> - dict.iteritems() -> dict.items()
> - dict.items() -> list(dict.items())
> 4. Install Python 3 on macOS Sierra and Mojave bots
> 5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts like
> clean-webkit, merge-results-json, webkit-patch)
> 6. Convert testing scripts and webkitpy to Python 3 in a single change
>
> The trouble I foresee us encountering with any scheme which attempts a
> conversion which retains both Python 2.7 and Python 3 compatibility is code
> like this:
>
> for expectation_string, expectation_enum in
> test_expectations.TestExpectations.EXPECTATIONS.iteritems():
> ...
>
> In this code, the EXPECTATIONS dictionary is thousands of elements long.
> In Python 2.7, iteritems() gives us an iterator instead of creating a new
> list, like items() would. In Python 3, iteritems() doesn’t exist, but
> items() does, and now gives us an iterator instead of creating a new list.
> The trouble here is that, in this case, creating a new list will be very
> expensive, expensive enough that we might manage to impact the testing run.
> There isn’t really an elegant way around this problem if we want to support
> both Python 2.7 and Python 3, other than defining different code paths for
> each language.
>
>
> The official Python 3 transition documentation has a fairly elegant
> solution to this, actually??
>
> https://legacy.python.org/dev/peps/pep-0469/
>
> See "Migrating to the common subset of Python 2 and 3” — you define
> different iteritems() helpers in the two cases. Seems pretty reasonable to
> me.
>
> There are other small gotchas as well. For example, ‘%’ is no longer a
> protected character, which can actually change the behavior of regexes.
> That’s why I think it’s better to just try and directly convert things
> instead of attempting to be compatible with both Python 2.7 and Python 3.
>
> Jonathan
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Standard process for exporting new WPT tests?

[Robert Ma]

Current maintainer of Chromium's WPT sync chiming in:

Because of the lack of atomic operations across two separate repos, some
kind of transient inconsistency and race conditions are unavoidable.

FWIW, when Chromium's WPT sync was originally designed, there was concern
about increasing the wait time of landing a Chromium change, which partly
led to our design choice of landing the Chromium change first and then the
GitHub export PR. One important mechanism to avoid losing the Chromium
change (because of being overwritten by the next import) that hasn't been
mentioned is that *we re-apply landed-in-Chromium-but-not-merged-on-Github
patches during import*. Of course, this mechanism caused quite some extra
complexities, but has been working well for us.

The approach being taken by the WebKit community (merging GitHub PRs before
landing WebKit patches) is sound and should be simpler. And I'm very
excited to see more contributions to WPT!

On Fri, May 25, 2018 at 4:16 AM Philip Jägenstedt 
wrote:

> On Fri, May 25, 2018, 00:52 Ryosuke Niwa  wrote:
>
>>
>> On Thu, May 24, 2018 at 3:22 AM, Philip Jägenstedt 
>> wrote:
>>
>>> On Wed, May 23, 2018, 23:43 youenn fablet  wrote:
>>>
 Le mer. 23 mai 2018 à 14:11, Frédéric Wang  a écrit :

> On 23/05/2018 22:50, Ryosuke Niwa wrote:
> > As we have preciously discussed, we should NEVER commit new tests
> into
> > LayoutTests/imported/w3c/web-platform-tests.
>

 Ryosuke, correct me if I am wrong, I think you are pointing out the
 following rule:
 Changes to LayoutTests/imported/w3c/web-platform-tests tests should
 land first in WPT repository, then in WebKit repository.

>>>
>>> Oh, that is surprising.
>>>
>>> https://github.com/w3c/web-platform-tests/pull/10964 is a recent WebKit
>>> export, and https://trac.webkit.org/changeset/231788/webkit did modify
>>> the test in place. Do you mean that the WPT PR was merged first, or should
>>> be in general? Chromium and Gecko do it in the other order, and I'd be
>>> interested to understand the trade-offs of flipping the order.
>>>
>>
>> Yes, it's okay for a WebKit commit to merge the change which got merged
>> into WPT but it's never okay to first commit the test into WebKit and then
>> later upstream it to WPT...
>>
>> Any process like this where changes end up in WebKit trunk via anything
>>> except a full WPT import will mean that divergence is possible.
>>>
>>
>> Precisely to avoid these problems.
>>
>
> I'm saying that there's temporary divergence with the WPT-then-WebKit
> order as well.
>
> Whichever side is merged first, the other side can fail to merge because
> any number of reasons, and something/someone then needs to deal with that.
> (See question in reply to Youenn.)
>
> However, one thing I really like about your order is that it makes it more
> straightforward to make WPT repo failures block browser repo merging. In
> the WPT Travis job, we already detect flakiness for Chrome and Firefox and
> I think we should also detect harness errors, and eventually also flag when
> a test goes from passing everywhere to failing everywhere, which is
> probably a mistake. We haven't really figured out how to make that block
> Chromium's CQ yet, and maybe flipping the order would do it.
>
>> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Robert Ma]

On Fri, Nov 24, 2017 at 9:09 AM, Frédéric WANG  wrote:
> Right, running all the tests is nice. But sometimes I personally feel
> guilty to have to execute everything on EWS for example to retrieve
> results for macOS/iOS (when I did not have access to a mac) or to fix
> compilation failures on Windows ports. So better granularity would be
> helpful in my opinion. In any cases, this was just one idea and people
> working on CI and infrastructure are in better position to know how to
> improve running time of WPT tests. I'm surprised that I don't hear such
> complaints from Mozilla and Chromium (but maybe I'm not aware of them).

Chromium has build bots that are similar to EWS. Some bots run before every
patch lands (CQ bots), while the other bots can be requested to run any
time with a WIP patch (try bots). CQ bots run *all* layout tests + WPT,
among other tests, on all major platforms. Try bots are similar, but one
select a subset of bots (platforms) to run. This is made possible by the
infrastructure team with the fancy LUCI system, especially the distributed
test execution system Swarming

.

Running all layout tests + WPT on a single machine takes hours, so I don't
think Blink engineers often do that locally. Generally speaking, people
will use try bots liberally to test 'em all whenever they want to run more
than just a handful of tests.

That said, we can of course benefit from WPT performance improvement as
well.

Robert
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev